Wowza Community

Transcoded video lagging

I have problem with transcoded streams. Original (source) streams are smooth and we still use them as for HD quality playback on website:

http://stream10.idnes.cz:1935/live/slowtv7/manifest.mpd (or http://stream10.idnes.cz:1935/live/slowtv5/manifest.mpd)

But when we transcode it, it chuckle or lag (I don’t know how to describe it better, but picture stops after few secs and then start again). It is very disturbing:

http://stream10.idnes.cz:1935/live/slowtv7_720p/manifest.mpd

or: http://stream10.idnes.cz:1935/live/slowtv5_720p/manifest.mpd

and it’s same in 360p.

(it’s MPEG DASH but same situation is in HLS and even in RTMP)

We have machine with GPU and both CPU and GPU is at 10 %. Wowza version is 4.6.0

xml settings:

true

720p

mp4:${SourceStreamName}_720p

and logo:

But we have another stream without logo at result is same:

http://stream10.idnes.cz:1935/live/reuters_720p/manifest.mpd

Hello Pavel,

I know that you have already settled this issue with another member of our Support Team, but I wanted to post the recommendations for anyone else that comes across the same issue:

The GPU is being overloaded which is causing the transcoder to skip frames. The transcoder does this to reduce the load when the CPU or GPU requirement is more than what’s available. You can see this in the logs below showing some of your streams skipping frames to the point of only having keyframes.

2017-01-26    00:03:55    CET    comment    server    INFO    200    -    TranscodingSession.updateBehindFilter[live/_definst_/slowtv10]: Video behind filter state change. New state: KEYFRAMESONLY
2017-01-26    00:13:01    CET    comment    server    INFO    200    -    TranscodingSession.updateBehindFilter[live/_definst_/slowtv7]: Video behind filter state change. New state: KEYFRAMESONLY
2017-01-26    05:22:03    CET    comment    server    INFO    200    -    TranscodingSession.updateBehindFilter[live/_definst_/slowtv6]: Video behind filter state change. New state: KEYFRAMESONLY
2017-01-26    07:50:03    CET    comment    server    INFO    200    -    TranscodingSession.updateBehindFilter[live/_definst_/slowtv5]: Video behind filter state change. New state: KEYFRAMESONLY
2017-01-26    09:59:54    CET    comment    server    INFO    200    -    TranscodingSession.updateBehindFilter[live/_definst_/jvc2.stream]: Video behind filter state change. New state: KEYFRAMESONLY 

Below you can see that some of the transcoded streams no longer contain video due to the lack of resources required to transcode them:

 2017-01-26    00:22:37    CET    comment    server    INFO    200    -    TranscodingSession.updateBehindFilter[live/_definst_/slowtv7]: Video behind filter state change. New state: ALLFRAMESOFF
2017-01-26    11:42:57    CET    comment    server    INFO    200    -    TranscodingSession.updateBehindFilter[live/_definst_/slowtv10]: Video behind filter state change. New state: ALLFRAMESOFF
2017-01-26    13:02:28    CET    comment    server    INFO    200    -    TranscodingSession.updateBehindFilter[live/_definst_/slowtv2]: Video behind filter state change. New state: ALLFRAMESOFF

This also explains why the bitrate of these streams is much lower than when using the CPU which will likely have not been overloaded or as overloaded during the test performed.

When using the Nvidia SMI to review the load on the GPU(s) this can be very inaccurate when the GPU is being used for transcoding as transcoding uses a fixed point function which is not detected in the Nvidia SMI report and so we recommend reviewing the wowzastreamingengine_access.log file(s) for the " Video behind filter state change. New state: [state]" messages. When overloaded you will first see SKIPXFRAME messages with either a 1, 2 or 4 replacing the X. A small number of SKIP1FRAME followed by ALLFRAMESON indicates you’re at the limitation of the CPU or GPU. If you see any SKIP4FRAME, KEYFRAMESONLY or ALLFRAMESOFF messages this is overloaded which will either result in a poor quality transcoded stream or no video in the stream. These messages are often missed as they’re INFO messages and do not appear in the wowzastreamingengine_error.log file.

To resolve this problem you would need to either reduce the number of streams being transcoded or reduce the number of encodes being created for each stream.

Note: A higher quality encode requires more resources than a lower quality encode. For example a 1080p encode requires more CPU/GPU than a 360p encode created from the same source stream.

Regards,

Jason Hatchett

@Jason_Hatchett,
i have also this Problem, but we do only CPU-.Transcoding, no GPU. I have the Problem above on 3-4 Machines with a Dual Xeon E5-2630v4 (2,20 Ghz) 20 Cores / 40 Threads / 128GB RAM, 960GB Nvme SSD with Wowza 4.8.15+3 or 4.8.17+1. And also on a Ryzen 9 5950X with 128GB RAM and SSD´s. All Server run with Windows Server 2012, 2016 or 2019.

Wowza Setup is nothing special with Standard-Settings. The Servers are only for Transcoding and HLS-Push to the Akamai-Network. NO deliver to the Endusers. Ingest-Streams are 720p50 / 5Mbit. We make 4 Renderings (600, 480,360 and 240p). The Servers run sporadically into this Problem. If i switch the Ingest-Stream to another Server with same Setup, already is fine. I only noticed the Problem since we updated to Wowza 4.8.14 or higher on this Machines but not previously.

I can’t imagine that the server cpu’s should be overloaded. CPU-Load was with this one Stream only near 10%.