First I’d like to describe my setup, and then the problems.
So we take an RTMP stream from a source, and transcode it to 11 output streams (6 of which are used for Cupertino and Smooth adaptive streaming) of various bitrates and resolution. The transcoder box is a Windows 7 64bit (this is for testing purposes only, and the production will be on Windows Server 2008 R2) on an i7 950 processor, 6GB RAM with a nvidia 460 SE GPU running Wowza 3.0.5. The transcoder also has the “Performance Tuning” done on it (Java JDK 1.6 update 31 with -server option), and nvidia drivers are latest (296.10). The JAVA_OPTS=-server -Xmx4768M are set in setenv.bat. The RAM usage never goes above 2GB, so memory shouldn’t be an issue.
The transcoder template is set using “CUDA” on 5 of the streams and “default” on 6 streams that will be used to make adaptive Cupertino and Smoothstreaming (I will explain why, in my first problem), and in Application.xml; PingTimeout is set to 12000, StreamType is live and no packetizers are enabled.
Then we take these 11 streams to a splitter (liverepeater-origin) where Cupertino and Smoothstreaming packetizers are enabled, from there it goes to our front (liverepeater-edge) servers where Cupertino and Smoothstreaming repeater is enabled. And at first everything works perfectly fine, with CPU not exceeding 40%~ average load on all 8 cores. I can play all 11 streams on the edge servers with no problem or performance degradation, smil is made on origin + edge for adaptive streaming which also works fine.
Problem #1:
When I enable CUDA in the transcoder template for the 6 streams that are used for making adaptive Cupertino and Smoothstreaming, the Cupertino streaming does not work, it just gets stuck on loading; while the Smoothstreaming works perfectly fine (keyframe GOP is 50, ie. 2 sec in transcoder template). When I use “default” instead of CUDA on these 6 streams, Cupertino works as well as Smoothstreaming. I have no idea why this problem occurs. And I would really like to use CUDA for all my streams to relieve the CPU.
Problem #2:
The client “cuts” the source stream which becomes only a black screen (ie. no moving video) at certain time of day (let’s say around midnight), but there it is still a stream coming from the source, it just so happens to be a black screen.
Now, when the source stream is being published again (moving video), the transcoder starts choking. CPU goes to 100%, there appears SKIPFRAME errors in the transcoder Error Log and output streams become very choppy etc.
And also a very particular thing happens: all the streams that use “CUDA” in the transcoder no longer have video in output, but only audio (the video is not being transcoded I’m guessing).
Solution for problem #2 is that I stop/start the source stream in the StreamManager on the transcoder, and everything starts working fine again, CPU goes down to 40%~ and the CUDA streams start reproducing video again. But it is a pain that I have to monitor the stream non-stop and restart the source stream on Transcoder, every time the customer “cuts” and “reenables” the source stream.
Here are some of the error log messages that might be helpful to you, the timecode is around the time the source streams is “cut” and “enabled” (ie. from static black screen to moving video).
WARN server comment 2012-04-03 01:38:23 - - - - - 11908.058 - - - - - - - - LiveMediaStreamReceiver.doWatchdog:
streamTimeout: Resetting connection
INFO server comment 2012-04-03 01:38:23 - - - - - 11908.058 - - - - - - - - LiveMediaStreamReceiver.resetConnection:
(SOCKET, R: gffstream.fc.llnwd.net/87.248.221.29:1935, L:
/10.42.90.102:1069, S: gffstream.fc.llnwd.net/87.248.221.29:1935)
INFO server comment 2012-04-03 01:38:23 - - - - - 11908.06 - - - - - - - - LiveMediaStreamReceiver.sessionClosed:
INFO server comment 2012-04-03 01:38:23 - - - - - 11908.06 - - - - - - - - TranscodingSession.resetStream[transcodewdr-test/_definst_/wdr2.stream]
INFO server comment 2012-04-03 01:38:23 - - - - - 11908.06 - - - - - - - - LiveMediaStreamReceiver.sessionClosed:
reconnect: isCurrentSession:false tryConnect:true
INFO server comment 2012-04-03 01:38:23 - - - - - 11908.061 - - - - - - - - TranscodingSession.close[transcodewdr-test/_definst_/wdr2.stream]
WARN server comment 2012-04-03 09:39:38 - - - - - 40782.581 - - - - - - - - LiveMediaStreamReceiver.doWatchdog:
streamTimeout: Resetting connection
INFO server comment 2012-04-03 09:39:38 - - - - - 40782.581 - - - - - - - - LiveMediaStreamReceiver.resetConnection:
(SOCKET, R: gffstream.fc.llnwd.net/95.140.238.34:1935, L:
/10.42.90.102:1290, S: gffstream.fc.llnwd.net/95.140.238.34:1935)
INFO server comment 2012-04-03 09:39:38 - - - - - 40782.582 - - - - - - - - TranscodingSession.resetStream[transcodewdr-test/_definst_/wdr2.stream]
INFO server comment 2012-04-03 09:39:38 - - - - - 40782.582 - - - - - - - - LiveMediaStreamReceiver.sessionClosed:
INFO server comment 2012-04-03 09:39:38 - - - - - 40782.585 - - - - - - - - TranscodingSession.close[transcodewdr-test/_definst_/wdr2.stream]
INFO server comment 2012-04-03 09:39:38 - - - - - 40782.585 - - - - - - - - LiveMediaStreamReceiver.sessionClosed:
reconnect: isCurrentSession:false tryConnect:true
Any help would be much appreciated, as I was not able to find much on this particular issue on the forums.