Transcoder CPU 100% and CUDA + Cupertino problem
First I'd like to describe my setup, and then the problems.
So we take an RTMP stream from a source, and transcode it to 11 output streams (6 of which are used for Cupertino and Smooth adaptive streaming) of various bitrates and resolution. The transcoder box is a Windows 7 64bit (this is for testing purposes only, and the production will be on Windows Server 2008 R2) on an i7 950 processor, 6GB RAM with a nvidia 460 SE GPU running Wowza 3.0.5. The transcoder also has the "Performance Tuning" done on it (Java JDK 1.6 update 31 with -server option), and nvidia drivers are latest (296.10). The JAVA_OPTS=-server -Xmx4768M are set in setenv.bat. The RAM usage never goes above 2GB, so memory shouldn't be an issue.
The transcoder template is set using "CUDA" on 5 of the streams and "default" on 6 streams that will be used to make adaptive Cupertino and Smoothstreaming (I will explain why, in my first problem), and in Application.xml; PingTimeout is set to 12000, StreamType is live and no packetizers are enabled.
Then we take these 11 streams to a splitter (liverepeater-origin) where Cupertino and Smoothstreaming packetizers are enabled, from there it goes to our front (liverepeater-edge) servers where Cupertino and Smoothstreaming repeater is enabled. And at first everything works perfectly fine, with CPU not exceeding 40%~ average load on all 8 cores. I can play all 11 streams on the edge servers with no problem or performance degradation, smil is made on origin + edge for adaptive streaming which also works fine.
Problem #1:
When I enable CUDA in the transcoder template for the 6 streams that are used for making adaptive Cupertino and Smoothstreaming, the Cupertino streaming does not work, it just gets stuck on loading; while the Smoothstreaming works perfectly fine (keyframe GOP is 50, ie. 2 sec in transcoder template). When I use "default" instead of CUDA on these 6 streams, Cupertino works as well as Smoothstreaming. I have no idea why this problem occurs. And I would really like to use CUDA for all my streams to relieve the CPU.
Problem #2:
The client "cuts" the source stream which becomes only a black screen (ie. no moving video) at certain time of day (let's say around midnight), but there it is still a stream coming from the source, it just so happens to be a black screen.
Now, when the source stream is being published again (moving video), the transcoder starts choking. CPU goes to 100%, there appears SKIPFRAME errors in the transcoder Error Log and output streams become very choppy etc.
And also a very particular thing happens: all the streams that use "CUDA" in the transcoder no longer have video in output, but only audio (the video is not being transcoded I'm guessing).
Solution for problem #2 is that I stop/start the source stream in the StreamManager on the transcoder, and everything starts working fine again, CPU goes down to 40%~ and the CUDA streams start reproducing video again. But it is a pain that I have to monitor the stream non-stop and restart the source stream on Transcoder, every time the customer "cuts" and "reenables" the source stream.
Here are some of the error log messages that might be helpful to you, the timecode is around the time the source streams is "cut" and "enabled" (ie. from static black screen to moving video).
Any help would be much appreciated, as I was not able to find much on this particular issue on the forums.
So we take an RTMP stream from a source, and transcode it to 11 output streams (6 of which are used for Cupertino and Smooth adaptive streaming) of various bitrates and resolution. The transcoder box is a Windows 7 64bit (this is for testing purposes only, and the production will be on Windows Server 2008 R2) on an i7 950 processor, 6GB RAM with a nvidia 460 SE GPU running Wowza 3.0.5. The transcoder also has the "Performance Tuning" done on it (Java JDK 1.6 update 31 with -server option), and nvidia drivers are latest (296.10). The JAVA_OPTS=-server -Xmx4768M are set in setenv.bat. The RAM usage never goes above 2GB, so memory shouldn't be an issue.
The transcoder template is set using "CUDA" on 5 of the streams and "default" on 6 streams that will be used to make adaptive Cupertino and Smoothstreaming (I will explain why, in my first problem), and in Application.xml; PingTimeout is set to 12000, StreamType is live and no packetizers are enabled.
Then we take these 11 streams to a splitter (liverepeater-origin) where Cupertino and Smoothstreaming packetizers are enabled, from there it goes to our front (liverepeater-edge) servers where Cupertino and Smoothstreaming repeater is enabled. And at first everything works perfectly fine, with CPU not exceeding 40%~ average load on all 8 cores. I can play all 11 streams on the edge servers with no problem or performance degradation, smil is made on origin + edge for adaptive streaming which also works fine.
Problem #1:
When I enable CUDA in the transcoder template for the 6 streams that are used for making adaptive Cupertino and Smoothstreaming, the Cupertino streaming does not work, it just gets stuck on loading; while the Smoothstreaming works perfectly fine (keyframe GOP is 50, ie. 2 sec in transcoder template). When I use "default" instead of CUDA on these 6 streams, Cupertino works as well as Smoothstreaming. I have no idea why this problem occurs. And I would really like to use CUDA for all my streams to relieve the CPU.
Problem #2:
The client "cuts" the source stream which becomes only a black screen (ie. no moving video) at certain time of day (let's say around midnight), but there it is still a stream coming from the source, it just so happens to be a black screen.
Now, when the source stream is being published again (moving video), the transcoder starts choking. CPU goes to 100%, there appears SKIPFRAME errors in the transcoder Error Log and output streams become very choppy etc.
And also a very particular thing happens: all the streams that use "CUDA" in the transcoder no longer have video in output, but only audio (the video is not being transcoded I'm guessing).
Solution for problem #2 is that I stop/start the source stream in the StreamManager on the transcoder, and everything starts working fine again, CPU goes down to 40%~ and the CUDA streams start reproducing video again. But it is a pain that I have to monitor the stream non-stop and restart the source stream on Transcoder, every time the customer "cuts" and "reenables" the source stream.
Here are some of the error log messages that might be helpful to you, the timecode is around the time the source streams is "cut" and "enabled" (ie. from static black screen to moving video).
WARN server comment 2012-04-03 01:38:23 - - - - - 11908.058 - - - - - - - - LiveMediaStreamReceiver.doWatchdog: streamTimeout: Resetting connection INFO server comment 2012-04-03 01:38:23 - - - - - 11908.058 - - - - - - - - LiveMediaStreamReceiver.resetConnection: (SOCKET, R: gffstream.fc.llnwd.net/87.248.221.29:1935, L: /10.42.90.102:1069, S: gffstream.fc.llnwd.net/87.248.221.29:1935) INFO server comment 2012-04-03 01:38:23 - - - - - 11908.06 - - - - - - - - LiveMediaStreamReceiver.sessionClosed: INFO server comment 2012-04-03 01:38:23 - - - - - 11908.06 - - - - - - - - TranscodingSession.resetStream[transcodewdr-test/_definst_/wdr2.stream] INFO server comment 2012-04-03 01:38:23 - - - - - 11908.06 - - - - - - - - LiveMediaStreamReceiver.sessionClosed: reconnect: isCurrentSession:false tryConnect:true INFO server comment 2012-04-03 01:38:23 - - - - - 11908.061 - - - - - - - - TranscodingSession.close[transcodewdr-test/_definst_/wdr2.stream]
WARN server comment 2012-04-03 09:39:38 - - - - - 40782.581 - - - - - - - - LiveMediaStreamReceiver.doWatchdog: streamTimeout: Resetting connection INFO server comment 2012-04-03 09:39:38 - - - - - 40782.581 - - - - - - - - LiveMediaStreamReceiver.resetConnection: (SOCKET, R: gffstream.fc.llnwd.net/95.140.238.34:1935, L: /10.42.90.102:1290, S: gffstream.fc.llnwd.net/95.140.238.34:1935) INFO server comment 2012-04-03 09:39:38 - - - - - 40782.582 - - - - - - - - TranscodingSession.resetStream[transcodewdr-test/_definst_/wdr2.stream] INFO server comment 2012-04-03 09:39:38 - - - - - 40782.582 - - - - - - - - LiveMediaStreamReceiver.sessionClosed: INFO server comment 2012-04-03 09:39:38 - - - - - 40782.585 - - - - - - - - TranscodingSession.close[transcodewdr-test/_definst_/wdr2.stream] INFO server comment 2012-04-03 09:39:38 - - - - - 40782.585 - - - - - - - - LiveMediaStreamReceiver.sessionClosed: reconnect: isCurrentSession:false tryConnect:true
Any help would be much appreciated, as I was not able to find much on this particular issue on the forums.
Thanks for the thorough write-up, excuse a short answer: per my understanding (not experience), consumer based CUDA cards can be unreliable. It seems like both problems are related to CUDA.
If re-start in StreamManager solves problem 2, take a look at the ModuleMediaCasterStreamMonitorAdvanced:
http://www.wowza.com/forums/content.php?80-MediaCaster-Monitor-ModuleMediaCasterStreamMonitorAdvanced
which will more closely monitor and reset MediaCaster streams.
Richard
If re-start in StreamManager solves problem 2, take a look at the ModuleMediaCasterStreamMonitorAdvanced:
http://www.wowza.com/forums/content.php?80-MediaCaster-Monitor-ModuleMediaCasterStreamMonitorAdvanced
which will more closely monitor and reset MediaCaster streams.
Richard
I have tried without using CUDA at all, and only using "default" on all the streams.
Problem #2 will still occur (when the stream is cut and resumed from source), specifically that the CPU is stuck at 100%, Wowza Error Log starts producing SKIPFRAME error messages (due to CPU overload), and the reproduction of streams is very laggy/choppy, thus needing a StreamManager restart of the stream.
Regarding problem #1, I have not been able to find anyone with similar problem on the forums. Do you have any more resources on this kind of issue, and maybe the specific GPU's that produce this problem?
Regards,
Problem #2 will still occur (when the stream is cut and resumed from source), specifically that the CPU is stuck at 100%, Wowza Error Log starts producing SKIPFRAME error messages (due to CPU overload), and the reproduction of streams is very laggy/choppy, thus needing a StreamManager restart of the stream.
Regarding problem #1, I have not been able to find anyone with similar problem on the forums. Do you have any more resources on this kind of issue, and maybe the specific GPU's that produce this problem?
Regards,
Right, the SKIPFRAME log messages are associated with overloaded transcoding, the server can't keep up. Try cutting back on the number of output streams -- for comparison. It sounds like there is only one source stream, if there are more reduce there too.
Richard
Richard
Yes you are correct, there is only 1 source stream. The issue (and mystery) for me is that the CPU does not exceed 50% load (when all streams are done using "default", CPU only setting), when everything is running normally. The mystery is that when the source is transitioned from static black screen to moving video and vice-versa, thats when the CPU load skyrockets to 100% and stays as such constantly (thus producing all kinds of performance issues with the output streams).
I have tried this with only 5 output streams transcoded on "default" setting, and 11 streams transcoded on "default" setting. The issue is always the same when that "static black -> moving video" transition occurs on the source stream.
Regards,
I have tried this with only 5 output streams transcoded on "default" setting, and 11 streams transcoded on "default" setting. The issue is always the same when that "static black -> moving video" transition occurs on the source stream.
Regards,
Just an update on the Problem #1, CUDA + HLS not working.
I have tested now the streams that are transcoded with CUDA, they only play audio in HLS (Cupertino) streaming, while those transcoded with "default" setting work fine.
I've tried to check the Error and Access logs on my origin server, but I see nothing unusual, the playlist.m3u8 I also see nothing wrong on both versions of the stream (CUDA and default).
Error log on origin gives no report, access log is the following on the origin (from where I test the HLS streaming), for the working stream:
and for the not working (CUDA encoded) stream:
Contents of playlist.m3u8 for working ("default") stream:
Contents of playlist.m3u8 for not working (CUDA) stream:
So I'm really baffled why HLS is not working when transcoded with CUDA :S
Regards,
I have tested now the streams that are transcoded with CUDA, they only play audio in HLS (Cupertino) streaming, while those transcoded with "default" setting work fine.
I've tried to check the Error and Access logs on my origin server, but I see nothing unusual, the playlist.m3u8 I also see nothing wrong on both versions of the stream (CUDA and default).
Error log on origin gives no report, access log is the following on the origin (from where I test the HLS streaming), for the working stream:
178.239.23.46 2012-04-03 14:50:57 [B]<name of working "default" encoded stream>[/B] 200 http (cupertino) 0 0 _defaultVHost_ [B]Application-name[/B] _definst_ 34.19 [B]<origin wowza ip>[/B] 1935 - Mozilla/5.0 (iPad; CPU OS 5_0_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9A405 Safari/7534.48.3 882472058
[B]<name of working "default" encoded stream>[/B] - - - - - http://[B]<origin wowza ip>[/B]:1935/[B]Application-name[/B]/[B]<name of working "default" encoded stream>[/B]/playlist.m3u8 - -
178.239.23.46 2012-04-03 14:51:22 [B]<name of working "default" encoded stream>[/B] 200 http (cupertino) 6689604 0 _defaultVHost_ [B]Application-name[/B] _definst_ 58.146 [B]<origin wowza ip>[/B] 1935 - AppleCoreMedia/1.0.0.9A405 (iPad; U; CPU OS 5_0_1 like Mac OS X; en_us) 2050295285 [B]<name of working "default" encoded stream>[/B] - - - -
- http://[B]<origin wowza ip>[/B]:1935/[B]Application-name[/B]/[B]<name of working "default" encoded stream>[/B]/playlist.m3u8 - -
and for the not working (CUDA encoded) stream:
178.239.23.46 2012-04-03 14:51:31 [B]<name of not working "CUDA" encoded stream>[/B] 200 http (cupertino) 0 0 _defaultVHost_ [B]Application-name[/B] _definst_ 34.005 [B]<origin wowza ip>[/B]t 1935 - Mozilla/5.0 (iPad; CPU OS 5_0_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9A405 Safari/7534.48.3 848239401
[B]<name of not working "CUDA" encoded stream>[/B] - - - - - http://[B]<origin wowza ip>[/B]:1935/[B]Application-name[/B]/[B]<name of not working "CUDA" encoded stream>[/B]/playlist.m3u8 - -
178.239.23.46 2012-04-03 14:52:17 [B]<name of not working "CUDA" encoded stream>[/B] 200 http (cupertino) 5679480 0 _defaultVHost_ [B]Application-name[/B] _definst_ 78.774 [B]<origin wowza ip>[/B] 1935 - AppleCoreMedia/1.0.0.9A405 (iPad; U; CPU OS 5_0_1 like Mac OS X; en_us) 1457864229 [B]<name of not working "CUDA" encoded stream>[/B] - - - -
- http://[B]<origin wowza ip>[/B]:1935/[B]Application-name[/B]/[B]<name of not working "CUDA" encoded stream>[/B]/playlist.m3u8 - -
Contents of playlist.m3u8 for working ("default") stream:
#EXTM3U #EXT-X-VERSION:2 #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=1468610,CODECS="avc1.66.30, mp4a.40.2",RESOLUTION=636x360 playlist.m3u8?wowzasessionid=1308020767
Contents of playlist.m3u8 for not working (CUDA) stream:
#EXTM3U #EXT-X-VERSION:2 #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=801213,CODECS="avc1.66.30, mp4a.40.2",RESOLUTION=636x360 playlist.m3u8?wowzasessionid=1997380418
So I'm really baffled why HLS is not working when transcoded with CUDA :S
Regards,
Can you provide more details about the CUDA card you are using?
Richard
Richard
Please open a support ticket:
Zip up conf, logs and transcoder folders and send them to support@wowza.com.
Thanks,
Richard
Zip up conf, logs and transcoder folders and send them to support@wowza.com.
Thanks,
Richard
The CUDA + HLS problem has been fixed in Wowza 3.0.5 patch6. Just wanted to post feedback in case other people run into similar issues, to know that the fix is there :).
Thanks to the developers for very fast reaction on fixing this issue.
Regards,
Thanks to the developers for very fast reaction on fixing this issue.
Regards,
