Recording h264 RTSP stream from surveillance camera using ffmpeg

| January 15, 2018 | Updated: March 14, 2020

For several years I used Vstarcam software (IP Camera Super Client) on Windows to record video stream from my cameras. This software (I preferred specific version 1.1.4.557) is quite reliable, though it produces video files that can’t be played by other video players. You can watch them only using this application.
It was also painful to restore archived video after reinstallation of the system or after changing the directory to which files are recorded.
After some time problems, which were caused by this software, became much significant than conveniences and I decided to move all recording processes to dedicated Ubuntu 16.04 server.

Unfortunately, it turned out that streams from my cameras, after they were recorded by direct copying using ffmpeg, can’t be played by third-party players such as MPC-HC or VLC ( players actually can’t seek in these files ), so I couldn’t just write stream but needed to recompress it.
At first I thought it is impossible to recompress three 720p streams on the fly, but I was wrong 🙂

For the purpose of video recording I have very interesting machine – Asus EB1505. It’s a nettop with low-end Celeron 847 CPU, but considering the price by which I have gotten it (50$ including 500GB HDD and 2GB RAM) and the possibility to install up to 16GB RAM into 2 slots (I tested 12GB) and the possibility to install two hard drives (not mentioning external eSATA) this is a very interesting device, though its CPU frequency 1.1GHz is somewhat low.
Anyway, I think this CPU is much better than any old or new Atom and the level of extensibility of this device is unusually high for nettop.
I have been cursing Asus for its crappy EeePC 701 and Fonepad 7″, which I have unluckily stumbled upon, but EB1505 is very interesting device at least for 50$ 😀

Celeron 847 has integrated Intel HD Graphics with hardware h264 decoder. It is important. I spent quite a bit time digging through ffmpeg docs and the Internet before I understood that this CPU has decoder only.
Hardware acceleration for video in Intel CPUs is called QuickSync and it seemed that HD Graphics 3000 should have hardware h264 encoding but it turned out that QuickSync in Celerons is somewhat abridged.
Anyway, I grateful even for hardware decoding because it reduces CPU usage for about 30%.

Later I tried to move my recording facility on a laptop with Radeon Xpress 1250 integrated video, but I failed. I couldn’t find any support for hardware acceleration in ffmpeg for this graphics card at all.
Even user interface on desktop linux was glitchy on that laptop, so old cheap AMD video cards are supported very poorly.

Moving back to the topic.
Here is my steps:

1) Install ffmpeg (I think that standard installation from official repo will be sufficient, though I have installed and checked a number of PPAs and even compiled ffmpeg myself in violent attempts to use hardware encoding where it is absent. 🙂 So I am not quite sure which ffmpeg version I use now…).
Upon installation check your hardware video acceleration capabilities:

# vainfo
error: XDG_RUNTIME_DIR not set in the environment.
error: can't connect to X server!
libva info: VA-API version 0.39.4
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_0_39
libva info: va_openDriver() returns 0
vainfo: VA-API version: 0.39 (libva 1.7.3)
vainfo: Driver version: Intel i965 driver for Intel(R) Sandybridge Mobile - 1.7.0
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            : VAEntrypointVLD
      VAProfileMPEG2Main              : VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointVLD
      VAProfileH264High               : VAEntrypointVLD
      VAProfileH264StereoHigh         : VAEntrypointVLD
      VAProfileVC1Simple              : VAEntrypointVLD
      VAProfileVC1Main                : VAEntrypointVLD
      VAProfileVC1Advanced            : VAEntrypointVLD
      VAProfileNone                   : VAEntrypointVideoProc

This command tests VA-API capabilities and string like “VAProfileH264Main : VAEntrypointVLD” means that hardware is able to decode video with this profile.
If I had a GPU with encoding capabilities I would see lines similar to “VAProfileH264Main : VAEntrypointEncSlice”.
As you can see, my hardware have no any encoding capabilities so only software encoding is available.

2) next step is to ensure that the user, who will run ffmpeg, has access to hardware encoding device /dev/dri/renderD128
To do so your user has to belong to group video.
Personally I needed to do system restart after this manipulation or maybe reboot is all that were needed after installation of ffmpeg 🙂

3) next you can try some variations of ffmpeg command from terminal to ensure that everything works as it should. That hardware acceleration works, that ffmpeg is able to catch the stream from you camera and so on. You also have to choose the balance between quality+compression level and CPU resources according to this page

4) Personally I use ffmpeg segmenter to create 30min video files from live h264 stream. Unfortunately there is a slight loss of frames during the switching to the new segment, but it seems to be less than one second.
I use command like this:
– hardware decoding
– RTSP source
– h264 encoding with superfast preset and rate factor of 36 which gives me about 1.5 – 2 GB video data per day from one camera.
– use segmenter
– use internal strftime function
– segment size 30min
– reset timestamps from stream
– save to mp4 file with name that includes camera’s id and date.

#
ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128  -i rtsp://admin:888888@192.168.100.102:10554/udp/av0_0  -an -codec:v libx264 -preset superfast -crf 36  -f ssegment -strftime 1 -segment_time 1800 -reset_timestamps 1 /HDD/CamVideo/CAM02/cam2_%Y-%m-%d_%H-%M-%S.mp4
#

5) I arranged my ffmpeg recording commands as systemd services (one per stream) so they can be automatically started after system startup and restarted after ffmpeg crash (it crashes about 10 times per day in average mostly because it is unable to catch the stream.). It is also convenient, that after correct stopping or restarting of such a service, ffmpeg closes current video file properly so it can be played.
Example of the daemon for one stream:

Perl script 
#! /usr/bin/perl
 
use strict;
use utf8;
use IO::File;
 
$SIG{__WARN__}  = \&alert_sysadmin;
$SIG{__DIE__}   = \&alert_sysadmin;
$SIG{HUP}       = \&do_reload;
 
my $PIDFILE     = '/tmp/camrec02.pid';
 
open(STDIN, '</dev/null');
open(STDOUT, '>/dev/null');
open(STDERR, '>&STDOUT');
chdir '/path/to/daemon';
 
my $pidfh = IO::File->new($PIDFILE, O_WRONLY|O_CREAT|O_EXCL, 0644);
if($pidfh){
    print $pidfh $$;
    close $pidfh;
}else{
    alert_sysadmin("first open of the pidfile failed $!");
    open my $pidfile, '<', $PIDFILE or die "unable to open existing pid file for reading";
    my $pid = <$pidfile>;
    if( $pid =~ /^\d+$/){
      if(kill(0, $pid)){# process exists
        alert_sysadmin("daemon process already exists"), die;
      }else{# process doesn't exists
        create_pid_file();
      }
    }else{# file corrupted
      create_pid_file();
    }
}
 
 
alert_sysadmin('daemon started');
 
my $r = `/usr/bin/ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128  -i rtsp://admin:888888@192.168.100.102:10554/udp/av0_0  -an -codec:v libx264 -preset superfast -crf 36  -f ssegment -strftime 1 -segment_time 1800 -reset_timestamps 1 /HDD/CamVideo/CAM02/cam2_%Y-%m-%d_%H-%M-%S.mp4`;
 
sub do_reload{
    alert_sysadmin("reload requested");
    unlink $PIDFILE;
    exec './camrec02.pl';
}
 
sub alert_sysadmin{
    my $message = shift;
    system('/bin/jabbersend.sh', "camrec stream 02 ALERT: $message");
}
 
sub create_pid_file{
  unlink $PIDFILE or die "unable to unlink old pidfile";
  my $pidfh = IO::File->new($PIDFILE, O_WRONLY|O_CREAT|O_EXCL, 0644);
  die "unable to create pidfile exlusively" unless $pidfh;
  print $pidfh $$;
  close $pidfh;
  alert_sysadmin("pidfile recreated");
}

To make this systemd service you should create a unit file in /etc/systemd/system/camrec02.service

[Unit]

Description=CAM2 stream recorder
After=syslog.target
After=network.target

[Service]
Type=simple
PIDFile=/tmp/camrec02.pid
WorkingDirectory=/path/to/daemon
User=myusername
Group=myusername

OOMScoreAdjust=-100
ExecStart=/path/to/daemon/camrec02.pl
ExecStop=/bin/kill -TERM  $MAINPID
ExecReload=/bin/kill -HUP $MAINPID

TimeoutSec=15
RestartSec=15
Restart=always

[Install]
WantedBy=multi-user.target

and then

systemctl enable camrec02
systemctl start camrec02
systemctl status camrec02

6) I’ve made a few auxiliary cron-scripts that control the process of recording.
– script that moves records from the previous day to the specific folder (daemon writes video chunks to the root of the /HDD/CamVideo/$CAMID folder and then this script moves these chunks to /HDD/CamVideo/$CAMID/PreviousDate)
this script also deletes folders older than 35 days to remove old records
It runs once per day at night.

Bash script 
#! /bin/bash
 
PREV_DATE=$(date -d "yesterday 13:00" '+%Y-%m-%d')
 
for CAMID in CAM01 CAM02 CAM03
do
  cd "/HDD/CamVideo/$CAMID"
  mkdir $PREV_DATE
  if [ $? -eq 0 ]; then
    find ./ -maxdepth 1 -type f -daystart -mtime 1 -exec mv {} ./$PREV_DATE \;
    find ./ -maxdepth 1 -type d -ctime +35 -exec rm -rf {} \;
  else
    echo "unable to create directory /HDD/CamVideo/$CAMID for video files"
    exit 1
  fi
done

– script that checks that recording process is alive (two types of checks)
it runs each 5 minutes

Bash script 
#! /bin/bash
 
for CAMID in CAM01 CAM02 CAM03
do
  ERROR=0
  cd "/HDD/CamVideo/$CAMID"
 
# there must be a file that has been modifyed during the last minute
  FINDOUT=`find ./ -maxdepth 1 -type f  -mmin -1 -print`
  if [ -z "$FINDOUT" ]; then
    /bin/jabbersend.sh "$CAMID => no live file. daemon must be restarted"
    ERROR=1
  fi
# files older that 10 minutes must be at least 2MB, or there is no actual record is going
  FINDOUT=`find ./ -maxdepth 1 -type f  -size -2000k -mmin +10 -print`
  if ! [ -z "$FINDOUT" ]; then
    /bin/jabbersend.sh "$CAMID => there is an old files with size less than 2MB"
    ERROR=2
  fi
 
 
  if [ $ERROR -gt 0 ]; then
    CAMNUM=`echo $CAMID | grep -Po '\d+'`
    /bin/systemctl stop camrec$CAMNUM
 
    if [ $ERROR -eq 2 ]; then
      find ./ -maxdepth 1 -type f  -size -2000k  -delete
    fi
 
    /bin/systemctl start camrec$CAMNUM
  fi
 
done

7) All records are now accessible on the LAN via SMB protocol and can be viewed using standard player from each computer on local network which is very convenient to me, though rarely some records are corrupted and my auxiliary scripts can’t prevent all possible errors.

With my parameters for ffmpeg I have a load average about 1.3 but it depends…
If there is a lot of chaotic movement on the camera (a noise in a low-light conditions for instance) – one camera can raise CPU usage up to 2.0 which is dangerous condition for my 2-core CPU.

Update 2018-09-14
8) After a few months of recording RTSP streams from my cameras using ffmpeg I noticed one more glitch that should be taken care of.
About once a day one of the recording processes begins to work improperly. It doesn’t actually record anything and consumes high amount of CPU time.
Usually CPU load by such a process is above 90% (my system allows CPU load up to 200%).
Because this process doesn’t record any file it will be killed as soon as one of the mentioned above watchdogs notice that video files are too small, or maybe it will be restarted together with the beginning of the new ffmpeg segment. Anyway, it is too much time to keep my server overloaded.
This is why I was forced to write one more watchdog service…

This watchdog script checks CPU load for particular processes. I decided to make a check once in 10 seconds but it could be adjusted.
Script works as a daemon so it could remember previous measurements for each process in which we are interested.
If some quantity (in my case – five) measurements in a row for certain process are higher than selected threshold (90% in my case) – the process is restarted.

The structure of the daemon is the same as in paragraph (5), so I show here only the main cycle and necessary variables.

Perl script 
#! /usr/bin/perl
 
our $THRESHOLD          = 90; # threshold system load in percent
our $QUEUE_SIZE         = 5;  # how many consecutive records we consider as failure
our $QUEUE_POS          = 0;  # current position in cyclic queue
our $CHECK_TIMEOUT      = 10; # timeout in seconds between sequential checks
 
our $watch_queues = {
  "camrec01" => {  # arbitrary name for observed process
      command   => "top -c -b -n1 -w1024| grep 192.168.100.101 | grep ffmpeg", # command to select needed process from the 'top' command
      queue     => [(0) x $QUEUE_SIZE]  # initialize queue for this process
  },
  "camrec02" => {
      command   => "top -c -b -n1 -w1024| grep 192.168.100.102 | grep ffmpeg",
      queue     => [(0) x $QUEUE_SIZE]
  },
  "camrec03" => {
      command   => "top -c -b -n1 -w1024| grep 192.168.100.103 | grep ffmpeg",
      queue     => [(0) x $QUEUE_SIZE]
  },
};
 
while(1){
 
  foreach my $rec_proc( keys %{$watch_queues}){
    # call appropriate command, remove leading spaces, split it by spaces and so get 'top'-command columns
    my($pr_pid, $pr_user, $pr_prio, $pr_nice, $pr_virt, $pr_res, $pr_shr, $pr_s, $pr_cpu, $pr_mem, $pr_time) = split(/\s+/, `$$watch_queues{$rec_proc}{command}` =~ s/^\s+(.*)/$1/r);
 
    $$watch_queues{$rec_proc}{queue}[$QUEUE_POS] = $pr_cpu; # record gathered cpu load in queue
 
    my $threshold_reached = 1;
    foreach (@{$$watch_queues{$rec_proc}{queue}}){ # compare each queue's value to threshold
      $threshold_reached = 0 if $_ < $THRESHOLD; 
    }
 
 
    if($threshold_reached){ # all sequential measuremens yeild high cpu usage
        alert_sysadmin("restarting daemon $rec_proc load records are  " . join(' ', @{$$watch_queues{$rec_proc}{queue}}));
        `/bin/systemctl restart $rec_proc`; # issuing command to restart service which gone nuts
        sleep(1);
        $_ = 0 foreach(@{$$watch_queues{$rec_proc}{queue}}); # clear measurement queue to start new series of measurements
        my $res = `/bin/systemctl status $rec_proc | grep Active`; # checking service uptime to be sure that it has been restarted
        alert_sysadmin("result: $res");
    }
 
  }
 
  $QUEUE_POS = 0 if ++$QUEUE_POS >= $QUEUE_SIZE;
 
  sleep($CHECK_TIMEOUT);
}

1 Comment