Surveillance VideoWall with OpenVidu (part 2)

Today we talk about the las challenges we faced on the Renacen’s Surveillance VideoWall project. Until here we have overcame the concurrence and partially the transcoding challenges, but… You can read the first part here. 

Already overcame challenges thanks to OpenVidu PRO:

• User concurrent access.
• Transcoding

New challenges: 

• Still many cameras couldn’t be played or played with errors.
• Grey screens.
• Incompatible codecs.

Still many cameras couldn’t be played or played with errors

After the inclusion of OpenVidu PRO to the system we reduced the cameras that weren’t displayed properly from about a 60% to about 30%. Obviously in a surveillance platform this percentage is still to high to be acceptable. From this 30% about a third couldn’t be played and the rest are played with errors. As we already have a transcoding system (Kurento running behind OpenVidu PRO) we need to review those video flows that even after the transcoding in Kurento aren’t properly showing and review the specific properties to and compare them with the specifications of Kurento compabilities. And we detected that many of the cameras that wheren played properly have this specifications (obtained with ffprobe)

Codec H264

index=0
codec_name=h264
codec_long_name=H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10
profile=Main
codec_type=video
codec_time_base=1/20
[...]

o

index=0
codec_name=h264
codec_long_name=H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10
profile=High
codec_type=video
codec_time_base=1/20
[...]

Doing a little research we find that Kurento as most used web browser and other media servers are compatibles with H264 -as stated by the WebRTC standard-. However if we looked deeper you can see that the H264 codec has plenty of variable properties such as “profile” or “level” which would indicate kinds of compression and additional transmission properties. On top of that we would see that H264 is not a “free” codec, in fact it has royalties and because of this many of browsers and OpenSource projects base their compatibility in the most basic profiles. Thus whenever we find Medium or High profiles is highly possible that there is some incompatibility with Kurento – OpenVidu – and the browsers, resulting in bad quality of the video, pixeled images, green frames or even no flow at all.

Solution

In order to visualise these video flows we need to add an additional transcoding step where the flow is transcoded to a profile known to Kurento. But this extra transcoding must be applied only when it is really necessary.

To realise this we designed a component that is able to discriminate the video flows taking into account its properties and decides which is the better transcoding parameters to a media flow -if necessary- . This component will provide video flows 100% compatible with Kurento but maintaining the process as light as possible.

Grey screen

The grey screen are usually shown in the beginning of the connection and are a nuissance. However this is not really an error. This is caused due the method of compressing media data.

As it is shown in the image we can see there are different types of frames I, B and P. If we simplify -a lot- we can say that only frames I have the whole image -like a photo- and frames B and P have only the pixels that have changed- the difference-. This way the amount of data it is sent.

What does it happen when reproducing the flow?

Simplifying -again a lot-, when any player receives an I frame it gets “painted” and then over the reception of B or P frames we “paint” the differences until a new I frame arrives and possible accumulated errors get corrected.  When the first frame received is not an I frame there are two strategies to how to proceed -until the first I frame is received-. The first option is to wait to the next I frame so the media will always be played nicely but this option will add some latency to the connexion (up to 2 min approx.). The second option is “paint” any frame usually a grey frame with some “painted” pixels an the “grey screen” will disappear as soon there are many accumulated differences or an I frame is received.

How can it be solved?

In this case transcoding won’t improve the flow neither other middle component will. One way to solve or reduce at least the nuisance is acting upon the camera. You may configure the camera -the codec- to send I frames at faster rate. However, be that as it might, it will have impact on the bandwidth as the video data will be much bigger.

Incompatible codecs

In addition to the challenges already explained we found minimal amount of cameras completely incompatible and would need a heavy transcoding process. This cameras where H265 emitting with large resolutions.

The H265 codec is not on the compatibility list of the current WebRTC definition, therfore most browsers and Kurento aren’t compatibe. What is more, we have found that even when Kurento is capable of receive an H265 flow and even transcode it with moderate success once the media flow stops Kurento finds itself in an inconsistent state and stops working properly for that flow or any others and can recover itself.  This represents a critical failure -thankfully very rarely found-.

The solution

The solution is simple, never allow any to reach Kurento by intercepting all H265 flows and eithe raise an error or trancode it. In this case taking into account that the number of H265 flows were and the associated resolutions are huge, the decision that was taken was to raise errors. The option of transcoding such huge H265 flows would have taken big amount of computing resources and would have affected the transcoding of other flows.

Conclusion

When talking of media delivery there are many variables that have unexpected effects. In this project we are working with more than 1000 real-time cameras where it isn’t possible to know the properties of the cameras and produced flows in advance. Thus a detailed analysis of a sample of the available flows and a custom configurable “on the go” solution has been our main objective.

We can read more about this project with Renacen in Live RTSP cameras integration in web Services for public administrations