Stream Manager hangs: Requires Server Restart

[garywiz]

We are experiencing a regular, reproducible problem:

• We use stream manager to create an RTSP re-broadcast stream from a Darwin Streaming Server
• For several hours, things are normal. We can stop and start the stream, we can "reset receiving stream". No problems.
• Sometime later, usually at least 10-12 hours later, we find that we can no longer reset the stream in the Stream Manager. In fact, the Stream Manager will not respond to any requests on existing streams, but hangs with the browser "Loading...", which never finishes (we have let it sit for 15 minutes waiting).
• The only recovery we have found is to do a "service WowzaMediaServer restart", which does not restart the server normally, but rather prints out "waiting for processes to exit" about 20 times followed by "killing processes which didn't stop after 20 seconds", followed by normal restart.
• We can then restart our streams using the stream manager and everything is OK until it happens again.

There appear to be no unusual errors in the log file before, during, or after one of these "hangs".

Our streams are set up according to the instructions (more or less!) in the thread http://www.wowzamedia.com/forums/showthread.php?t=4403.

Note however that we have found that the warning about Darwin dropping the stream after 2 minutes does not appear to be true. Even if there are no viewers on the stream, it appears to run continually, receiving the stream from our Darwin distribution server. This is the behavior we want, however. Also, for at least 5 to 6 hours, the streams work great. We have had over 100 viewers regularly using the stream, and there appear to be no technical problems whatsoever, until the above "hang" occurs.

We are running Wowza 2, patch level 9 on a Centos 5.4 64-bit platform (VM under KVM). The system has plenty of resources and never seems to be under any stress, has plenty of CPU and memory headroom.

The stream being served is at http://treet.tv/chat (among other places).

What additional information can I provide to attempt to solve this problem?

[charlie]

Zip up your conf and logs folders and point to the time in the logs where the stream/server stopped working. Send this info to support@wowzamedia.com. Be sure to reference this forum thread.

I would also suggest that you configure the JMX interface to the server (the process is described in the User's Guide). You can use VisualVM to monitor the server. The next time this happens I would like you to take a thread dump. The instructions are here:

http://www.wowzamedia.com/forums/showthread.php?t=8415

Charlie

[cloudsurfin]

Gary,
How did you resolve this? I have the same exact issue w/ 3.1.1 on Ubuntu.

Thx.

[laurosdorneles]

Hello, similar thing is happening to us, since 3.1.1 update, last week, we´re using a wowza instance on EC2. It simply does not accept any encoder connection, but the OS is ok, we even can connect using ssh and so, we have to restart the wowzamediaserver to temporarily fix the problem. It happened at least 5 times on the last 5 days. Thanks.

[rrlanham]

First thing I would do is replace that instance.

Richard