Has anyone done some serious research in Garbage Collection for Wowza?
We’ve been struck by a big problem now. One of our client’s uses Wowza for live broadcasts - it’s running Linux 2.6.26-2-amd64 on 2 quad-core CPU server with 4GB of RAM and. We use Flash Media Live Encoder (version 3) to encode mixed down video and stream it to Wowza server (version 1.7.0_patch21) which has an application with stream type “live” to which all other viewers connect and watch the broadcast. After running the broadcast for about 30 minutes heap suddenly sky-rockets to the maximum and after few more minutes Wowza server stops responding.
Wowza server has been configured with settings shown in “General tuning instructions” and running with thease settings:
-Xmx3000M -Djava.net.preferIPv4Stack=true -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:NewSize=1024m
As you can see in this image taken from JConsole - there we started up Wowza server and after it reached heap limit we had to kill it and start it over again. The difference is that at first we started it up with thease settings:
-d64 -server -Xmx3000M -Djava.net.preferIPv4Stack=true -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:NewSize=1024m
But the -server mode was actualy performing worse.
So my question here is what could be the problem with GC? I also must add that this high peak is actualy “CMS old gen” - so as far as my understanding for GC goes - it’s those objects which live through “new” age, “medium” age and then finaly are considered “old”, but nobody goes up there to clean them out when they are un-referenced in “old gen” heap (or atleast somebody goes there when it’s already too late). Is there some specific java options to specify GC life-time for each of those generations?
Gusts
PS. I forgot to add that wowza stops responding and flooded error log full with thease:
- - - 2009-11-08 20:35:00 - - - - - 0.902 - - - - - - - -
java.lang.OutOfMemoryError: Java heap space
- - - 2009-11-08 20:35:00 - - - - - 0.902 - - - - - - - -
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
at org.apache.mina.common.SimpleByteBufferAllocator.allocate(Unknown Source)
at org.apache.mina.common.ByteBuffer.allocate(Unknown Source)
at org.apache.mina.transport.socket.nio.SocketIoProcessor.read(Unknown Source)
at org.apache.mina.transport.socket.nio.SocketIoProcessor.process(Unknown Source)
at org.apache.mina.transport.socket.nio.SocketIoProcessor.access$600(Unknown Source)
at org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run(Unknown Source)
at org.apache.mina.util.NamePreservingRunnable.run(Unknown Source)
at java.lang.Thread.run(Thread.java:619)