Replace missing video frames to ensure a consistent frame rate with Wowza Streaming Engine

Prevent streams from buffering and displaying artifacts for smoother, more continuous video playback by replacing missing video frames and ensuring a consistent frame rate using the Transcoder in Wowza Streaming Engine™ media server software. Replacing missing frames with the last successfully decoded frames allows for maintaining a specific output frame rate even when the source video is missing frames and drops below the desired output frame rate. You can enable a consistent frame rate using either a strict frame rate for constant frame rate video or minimum frame rate for variable frame rate video. When transcoding a stream from one protocol to another protocol for playback, configuring the type of frame rate that is compliant with the playback protocol enhances stability of the stream.

Note: Wowza Streaming Engine 4.8.11 or later is required.

How the Transcoder replaces missing video frames


Streams sent to Wowza Streaming Engine can have missing video frames because frames aren't sent to Wowza Streaming Engine from the video source or frames arrive late or out or order and are dropped. When streams are sent over poor network conditions or over UDP—which is often the case with WebRTC, SRT, and RTSP/RTP—they are more likely to have missing frames.

The Wowza Streaming Engine Transcoder identifies missing video frames when the decoder receives two consecutive frames with a difference in timecodes that is at least two times the duration of a single frame or when a frame does not arrive within a specified amount of time since the last received frame (InputBufferTime). The Transcoder uses either a specified frame rate (MinimumFrameRate) or the frame rate it detects from the incoming source stream as the output frame rate, adjusting frame timecodes accordingly, and as the basis for when to replace frames. The Transcoder buffers the last successfully decoded frame in memory so that it can be used in place of missing frames.

If a late frame arrives after the input buffer time and it has already been replaced, Transcoder does not use the late frame in its original place but may buffer it and use it to replace a subsequent frame. If a stream drops, Transcoder replaces missing frames with the last successfully decoded frame until the stream reconnects or times out.

Configure video frame replacement and frame rate


To enable replacing missing frames and ensure a consistent frame rate, you must configure the InputBufferTime property. We recommend also configuring MinimumFrameRate to specify the desired output frame rate and StrictFrameRate if you need constant frame rate video.

To configure the following properties, add them to your Transcoder template XML using a text editor. Transcoder templates are located in [install-dir]/transcoder/templates. See the following example configuration for an example of these properties implemented in a Transcoder template.

Path Name Description
Root/Transcode/Decode/Video InputBufferTime
(required)
Enables replacing missing frames with the last successfully decoded frame when set to a value higher than 0 ms. The default value is 0. This property specifies, in milliseconds, the duration that the Transcoder will wait for late or out-of-order frames. This allows you to control the amount of latency that is added to streams to replace missing frames. We recommend setting the value to at least 100 ms but not higher than 1000 ms.
 
Note: If you configure this property but not MinimumFrameRate or StrictFrameRate, the Transcoder uses the frame rate it detects from the incoming source stream as the minimum frame rate and the basis for when to replace frames.
Root/Transcode/Decode/Video MinimumFrameRate
(recommended)

Specifies the desired minimum or strict frame rate of the output video in frames per second (fps). Transcoder uses this to determine when to replace missing frames. By default, the value is the frame rate of the incoming source stream.

The value of this property is used in two different ways depending on the value of StrictFrameRate.

  • When StrictFrameRate is set to false (default), this property allows for variable frame rate output and specifies a frame rate that the output stream should not go below. For example, if you set this property to 15, the frame rate of the output stream can vary but will not fall below 15 fps. Some protocols, such as WebRTC and other RTP-based protocols, call for or allow variable frame rate video.
  • When StrictFrameRate is set to true, this property specifies a constant output frame rate. Some protocols, such as HLS and MPEG-DASH, call for constant frame rate video.

In either case, the value of this property should be less than or equal to the source video's frame rate.

Root/Transcode/Decode/Video StrictFrameRate
(optional)

If true, enables a strict output frame rate that is constant. The default value is false. When this property is set to true, the value of MinimumFrameRate is used as the constant output frame rate. Some protocols, such as HLS and MPEG-DASH, call for constant frame rate video.

Root/Transcode/Decode Debug
(optional)

Enables debug logging for the video decoding process and logs when frames are missing and have been replaced to wowzastreamingengine_access.log, for example: TranscoderFrameManager.onFrame: Inserting 1 frames due to not receiving a frame in over 500ms. The default value is false.

Example configuration

In this example, if the decoder receives two consecutive frames with a difference in timecodes that is at least two times the duration of a single frame or does not receive frames for 500 ms, Transcoder replaces missing video frames at a constant frame rate output of 30 fps.

<Decode>
   <Debug>false</Debug>
   <Video>
      <Implementation>NVCUVID</Implementation>
      <Deinterlace>false</Deinterlace>
      <InputBufferTime>500</InputBufferTime>
      <StrictFrameRate>true</StrictFrameRate>
      <MinimumFrameRate>30</MinimumFrameRate>
      ...
   </Video>
   ...
</Decode>

More resources