Hello Pavel,
I know that you have already settled this issue with another member of our Support Team, but I wanted to post the recommendations for anyone else that comes across the same issue:
The GPU is being overloaded which is causing the transcoder to skip frames. The transcoder does this to reduce the load when the CPU or GPU requirement is more than what’s available. You can see this in the logs below showing some of your streams skipping frames to the point of only having keyframes.
2017-01-26 00:03:55 CET comment server INFO 200 - TranscodingSession.updateBehindFilter[live/_definst_/slowtv10]: Video behind filter state change. New state: KEYFRAMESONLY
2017-01-26 00:13:01 CET comment server INFO 200 - TranscodingSession.updateBehindFilter[live/_definst_/slowtv7]: Video behind filter state change. New state: KEYFRAMESONLY
2017-01-26 05:22:03 CET comment server INFO 200 - TranscodingSession.updateBehindFilter[live/_definst_/slowtv6]: Video behind filter state change. New state: KEYFRAMESONLY
2017-01-26 07:50:03 CET comment server INFO 200 - TranscodingSession.updateBehindFilter[live/_definst_/slowtv5]: Video behind filter state change. New state: KEYFRAMESONLY
2017-01-26 09:59:54 CET comment server INFO 200 - TranscodingSession.updateBehindFilter[live/_definst_/jvc2.stream]: Video behind filter state change. New state: KEYFRAMESONLY
Below you can see that some of the transcoded streams no longer contain video due to the lack of resources required to transcode them:
2017-01-26 00:22:37 CET comment server INFO 200 - TranscodingSession.updateBehindFilter[live/_definst_/slowtv7]: Video behind filter state change. New state: ALLFRAMESOFF
2017-01-26 11:42:57 CET comment server INFO 200 - TranscodingSession.updateBehindFilter[live/_definst_/slowtv10]: Video behind filter state change. New state: ALLFRAMESOFF
2017-01-26 13:02:28 CET comment server INFO 200 - TranscodingSession.updateBehindFilter[live/_definst_/slowtv2]: Video behind filter state change. New state: ALLFRAMESOFF
This also explains why the bitrate of these streams is much lower than when using the CPU which will likely have not been overloaded or as overloaded during the test performed.
When using the Nvidia SMI to review the load on the GPU(s) this can be very inaccurate when the GPU is being used for transcoding as transcoding uses a fixed point function which is not detected in the Nvidia SMI report and so we recommend reviewing the wowzastreamingengine_access.log file(s) for the " Video behind filter state change. New state: [state]" messages. When overloaded you will first see SKIPXFRAME messages with either a 1, 2 or 4 replacing the X. A small number of SKIP1FRAME followed by ALLFRAMESON indicates you’re at the limitation of the CPU or GPU. If you see any SKIP4FRAME, KEYFRAMESONLY or ALLFRAMESOFF messages this is overloaded which will either result in a poor quality transcoded stream or no video in the stream. These messages are often missed as they’re INFO messages and do not appear in the wowzastreamingengine_error.log file.
To resolve this problem you would need to either reduce the number of streams being transcoded or reduce the number of encodes being created for each stream.
Note: A higher quality encode requires more resources than a lower quality encode. For example a 1080p encode requires more CPU/GPU than a 360p encode created from the same source stream.
Regards,
Jason Hatchett
