Wowza Community

Wowza Freezes

Any ideas why wowza seem to stop responding after an hour or 2 of operations. The service seems to be still running but it does not respond to connect request, no connect failed or reject being returned as well. I checked the logs and no error is logged or anything. As I was observing it seems that when it reaches 20 instances of an application and around 100 clients it would freeze. After restarting the service it will go back to normal. Already followed the tuning guide for Java Memory heap size and GC but the problem still appears, it is intermittent too, sometimes it takes 3 hours before it freezes. Problem is there are no hints why it does that, any inputs?

Hi wolfza,

Are you running any custom modules or addons of any sort.

What is the platform, java version, Wowza version memory settings etc. Normally it will run for ages unless there is something consuming resources.

Also, what sort of application are you running that requires so many appInstances? Just trying to get a handle on what your setup is.

Can / have you tried to access the server using the jmx interface when it freezes up?

See if this link helps (Java Timer utility class):

http://www.java2s.com/Code/Java/Development-Class/UsejavautilTimertoscheduleatasktoexecuteonce5secondshavepassed.htm

Charlie

No custom modules or add-ons, just using mysql-connector-java-5.1.8 for mysql based authentication,

You must be running a custom module or an add-on as there is nothing in Wowza core that will talk to the mysql connector directly.

You say you are monitoring it with jmx. Does jmx become un-responsive when Wowza stops accepting connections?

If not then there are a couple of thing that you can look at.

I had a similar problem a while back where a module went a bit haywire and used up all the threads in the VHost/HandlerThreadPool.

In jmx, navigate to VHosts/defaultVHost/HandlerThreadPool/Attributes and this will show you how many threads are active and if there are any threads waiting. Double click the values to get a graph up for each and leave it running. You can also right click the graph and save it.

When it does happen, perform a Heap Dump. In the jmx interface, com.sun.management/HotSpotDiagnostic/Operations. dumpHeap. p0 is the path & file name.

you can connect to the heap dump to view it with the following command as root.

[java-path]/bin/jhat [dump-file]

This will open a http interface on port 7000 which will need to be open in your firewall to view it from another location.

pm me with the address and we can take a look.

It depends if that is the problem. Normally there would be plenty of threads available.

What is using the mysql connector?

If the mysql calls are taking too long then this may be the cause of the problem. If so then increasing the size of the thread pool will only extend the time before failure. You may have to re-jig your module to make it more efficient.

Let it run it’s course and see if that is the problem first before making any changes.

No custom modules or add-ons, just using mysql-connector-java-5.1.8 for mysql based authentication, I also close db objects and connections as soon as it is not needed, the server is also a load-balancer sender. Already followed the tuning guide for Java Memory heap size and GC.

Server Specs:


CPU: 2 x Dual Core Xeon, Memory: 4GB, Harddrive: SSD, Raid: RAID0

Java: 1.6.0_22 Beta 4

Wowza Media Server 2.1.2 build24878

Application is many to many that why it requires many appInstances, it is a text chat with live video streaming.

I am monitoring using JMX while the server is running, based on the status in JMX, it shows less than 10% CPU usage, Memory heap size is less than 20% of the max 3GB and it drops from time to time when the GC triggers I guess, Classes shows 2900 loaded and live threads around 234, in that status for an hour or two then suddenly I will notice it will not respond when I am trying to connect via rtmp, no connect failed being returned, it seems to get stucked, also, the currently connected users will not notice because it will not issue a connection closed or fail either, it just sits there, will only notice the video stream and text chat will freeze as well. When this happens, I will immediately check the error logs and nothing there either, it still seem to send status to the load balancer too because I can see the updating but the connectcount stays as is, JMX will then show memory heap size increasing to the max which sounds like it stops doing GC too.

Everything goes back to normal only when I restart the wowza service. Any ideas?

Thanks a lot for this info, I will monitor it again and will take a look at that part when it starts to freeze connections. By the way, will it help when the HandlerThreadPool size is increased in the VHost.xml config?

Yes, I have my custom application that is using the mysql connector, I also forgot to mention, 3 applications running to split the chat text, video stream and the main authentication part, so all in all when a client connects it is maintaining 3 connections each, could this be causing it to eat up all the thread pool?

thanks Roger, I will observe it more.

Is there some sort of a timer class (e.g. like setInterval) in Wowza2?

I am using the custom runnable class to implement a timer while the application instance is active but it seem to be the reason why my ThreadPool maxing out crashing wowza service, as I was observing the HandlerThreadPool in JMX, it seem to increase little by little over time that would cause it to reach the max after couple of hours.

My timer is something like this:

payTimer payingTimer = new payTimer();

client.getVHost().getThreadPool().execute(payingTimer);

class payTimer implements Runnable

{

private boolean done = false;

public void run()

{

while (true)

{

try

{

// interval in milliseconds

Thread.currentThread().sleep(6000);

// do action here and set done to true to exit…

Thanks !

Issue with the threadpool flooding fixed, it was the timer class, replaced with the java util timer and everything is fine now, thanks a lot!