I wanted to add timestamps to the video that's captured while using my picam, opencv-based motion detection application. The easiest way would probably be to read and modify every frame in userspace on the Pi, before letting the GPU encode it to h264, however I'd like to keep frame rates as high as possible - so would rather the Pi did the minimum amount of work it needs to do. I only really need the frame number and a timestamp and some way to merge them with the video.

The MMAL API for the Pi's camera has a bunch of flags in the 'buffer header' objects that get passed around as the image data flows from component to component. By counting the instances of one of these - MMAL_BUFFER_HEADER_FLAG_FRAME_END - you can track frame numbers. I've modified the code to emit the frame number and an epoch timestamp when motion is detected. To prevent it from producing hundreds of these events, it only emits when another flag: MMAL_BUFFER_HEADER_FLAG_KEYFRAME - is set at the start of a keyframe. This pegs the rate a the keyframe rate; every two seconds. An example keyframe event looks like this in the logs:

  KEYFRAME (1397291533:1)
  

The logs are written to stderr and the h264 video stream is written to stdout. To record them both, the app is run like this:

  ./mmal_opencv_modect > video.h264 2> video.log
  

As it turns out the h264 file it spits out is less than perfect, some things play it at a high frame rate and some things don't play it at all. To clean it up I've been pushing it through ffmpeg to create a nice avi:

  ./ffmpeg -r 30 -i video.h264 -q:v 1 -r 30 video.avi
  

This step doesn't need to happen on the pi, and can be offloaded to faster machine - and while we're post processing we may as well be adding timestamp overlays too :) Initially I tried to use ffmpeg's 'drawtext' filter and to start with it looked promising; it has expressions so that text strings can be positioned across exact frame ranges. Unfortunately each of these are specified individually on the command line and more than a thousand exceeds the maximum command line length - at least here on my machine.

Instead I've written a python script that can take the log output from mmal_opencv_modect and convert it into a MicroDVD subtitle file. The MicroDVD format uses frame numbers, so is easy to translate from the keyframe events in the logs. The epoch timestamp is also converted to something a little more human friendly. The script is run like this:

  ./scripts/modect2sub.py video.log > video.sub
  

Now we have the subtitle file it just needs to be merged with the h264 stream. Using ffmpeg, compiled with libass support:

  ./ffmpeg -r 30 -i video.h264  -vf "subtitles=video.sub" -q:v 1 -r 30 video.avi
  

Github repo can be found here.

I've just pushed some code for the raspberry pi and it's camera - based on tasanakorn's work - up to github that uses some very simple OpenCV to do background subtraction and then detect movement. When it detects motion, the camera image data is sent on to the h264 encoder in the VideoCore and the h264 stream is written to stdout. The detection and encoding happen in different threads, so that even though we can't do motion detection at full frame rate, that doesn't affect the recording frame rate. Here's some frame rates and CPU utilisation numbers taken at various camera modes (incl. the new 640x480 at 90 FPS!):

 720p30 : OpenCV = 15.05, Video = 30.51, ~60% CPU
1080p30 : OpenCV = 14.90, Video = 30.02, ~75% CPU
 VGAp90 : OpenCV = 21.57, Video = 91.12, ~90% CPU
  

To build, follow the README instructions in the github repo. Once it's built you can run it like this:

  ./mmal_opencv_modect > video.h264
  

I'm hoping to use some more raspberry pi's (and camera modules) to replace the analogue system we're currently using for birdcam. The resolution is much higher and the 90 FPS mode should show the very fast movement of some of the smaller birds. The pi based system should be more distributed and scalable than the current system too.