Results 1 to 8 of 8

Thread: Origin memory error with GC

  1. #1

    Default Origin memory error with GC

    Hello All,
    I have origin-edge loadbalancer setup with about 20 servers and origin server was constantly crashing with following errors just before I have to restart wowza again. When this happens all the connection counts in the edges are reduced by about 75%. If I try to play a stream it just goes to stop.

    Errors in the log file just before the restart.
    -	-	-	2011-10-30	09:59:41	-	-	-	-	-	1.598	-	-	-	-	-	-	-	-	-
    java.util.concurrent.RejectedExecutionException
    	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1768)
    	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
    	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
    	at org.apache.mina.filter.executor.ExecutorExecutor.execute(Unknown Source)
    	at org.apache.mina.filter.executor.ExecutorFilter.fireEvent(Unknown Source)
    	at org.apache.mina.filter.executor.ExecutorFilter.exceptionCaught(Unknown Source)
    	at org.apache.mina.common.support.AbstractIoFilterChain.callNextExceptionCaught(Unknown Source)
    	at org.apache.mina.common.support.AbstractIoFilterChain.access$800(Unknown Source)
    	at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.exceptionCaught(Unknown Source)
    	at org.apache.mina.common.support.AbstractIoFilterChain$1.exceptionCaught(Unknown Source)
    	at org.apache.mina.common.support.AbstractIoFilterChain.callNextExceptionCaught(Unknown Source)
    	at org.apache.mina.common.support.AbstractIoFilterChain.fireExceptionCaught(Unknown Source)
    	at org.apache.mina.transport.socket.nio.SocketIoProcessor.read(Unknown Source)
    	at org.apache.mina.transport.socket.nio.SocketIoProcessor.process(Unknown Source)
    	at org.apache.mina.transport.socket.nio.SocketIoProcessor.access$600(Unknown Source)
    	at org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run(Unknown Source)
    	at org.apache.mina.util.NamePreservingRunnable.run(Unknown Source)
    	at java.lang.Thread.run(Thread.java:662)
    ERROR	server	comment	2011-10-30	12:12:14	-	-	-	-	-	7954.697	-	-	-	-	-	-	-	-	execute: java.util.concurrent.RejectedExecutionException
    ERROR	server	comment	2011-10-30	12:12:14	-	-	-	-	-	7954.698	-	-	-	-	-	-	-	-	execute: java.util.concurrent.RejectedExecutionException
    I am monitoring the server with JMX and the memory graph looks odd since the GC seems to be not effective as time passes. CPU graphs looks normal and never goes above 40%

    Even when I restart wowza this error happens again after 2-3 hours and the JMX memory usage levels are same.

    Screenshot after restarting due to this issue


    Server specs for the origin



    At the time of crash server was only using about 400Mbps of available 1Gbps.

    My java is SUN JDK
    [root@30AB16 ~]# java -version
    java version "1.6.0_23"
    Java(TM) SE Runtime Environment (build 1.6.0_23-b05)
    Java HotSpot(TM) 64-Bit Server VM (build 19.0-b09, mixed mode)
    I am also using the latest jar files for wms-plugin-loadbalancer and wms-plugin-collection

    I have also followed the tune up steps mentioned in the forum thread.
    My current setenv.sh file is setup like this ( I am thinking of upgrading memory of the server from 8GB to 12GB in this case what would be a good JAVA_OPTS memory size?)
    #!/bin/sh
    
    _EXECJAVA=java
    JAVA_OPTS="-Xmx4096M"
    
    # Uncomment to run server environment (faster), Note: will only work if server VM install, comes with JDL
    JAVA_OPTS="$JAVA_OPTS -server"
    
    # Can be a better GC setting to avoid long pauses
    #JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC"
    
    # Uncomment to fix multicast crosstalk problem when streams share multicast port
    JAVA_OPTS="$JAVA_OPTS -Djava.net.preferIPv4Stack=true"
    
    WMSAPP_HOME=/usr/local/WowzaMediaServer
    WMSCONFIG_HOME=/usr/local/WowzaMediaServer
    WMSCONFIG_URL=
    
    export WMSAPP_HOME WMSCONFIG_HOME JAVA_OPTS _EXECJAVA
    Can anyone please suggest some fixes for this issue?

  2. #2

    Default

    Actually I am using Wowza 2. can someone please move this thread there?
    Thanks.

  3. #3

    Default

    Be sure to upgrade to the latest 2.2.4.* build of Wowza: http://www.wowza.com/forums/content.php?198

    Also, upgrade to JDK 1.7. I think this is your best option.

    You can set your Java heap to 5GB, since it doesn't appear you are using much RAM for other things. But I don't think this is the problem.

    You can try removing the GC settings you added and try the Experimental GC (Option 6b in the Tuning Guide. Or, you can try just removing the GC tuning alltogether.

  4. #4

    Default Origin java RAM increase & Zero sized function

    Thanks for the reply. I will looking into upgrade and I have already increased RAM to 5GB.

    Do you know is it okay to increase java RAM allocation to 8GB if I upgrade the total RAM in origin to 12GB?

    Also, when checking the error log it is filled with following type. We have about 40,000 of these errors for today.

    ERROR	server	comment	2011-10-30	12:12:11	-	-	-	-	-	7951.512	-	-	-	-	-	-	-	-	Zero sized function (client:1520568836:WIN 10,3,181,34): type:20 size:0:
    ERROR	server	comment	2011-10-30	12:12:11	-	-	-	-	-	7951.512	-	-	-	-	-	-	-	-	Zero sized function (client:1520568836:WIN 10,3,181,34): type:20 size:0:
    ERROR	server	comment	2011-10-30	12:12:11	-	-	-	-	-	7951.512	-	-	-	-	-	-	-	-	Zero sized function (client:1520568836:WIN 10,3,181,34): type:20 size:0:
    ERROR	server	comment	2011-10-30	12:12:11	-	-	-	-	-	7951.512	-	-	-	-	-	-	-	-	Zero sized function (client:1520568836:WIN 10,3,181,34): type:20 size:0:
    ERROR	server	comment	2011-10-30	12:12:11	-	-	-	-	-	7951.512	-	-	-	-	-	-	-	-	Zero sized function (client:1520568836:WIN 10,3,181,34): type:20 size:0:
    ERROR	server	comment	2011-10-30	12:12:11	-	-	-	-	-	7951.512	-	-	-	-	-	-	-	-	Zero sized function (client:1520568836:WIN 10,3,181,34): type:20 size:0:
    Is there anything i can do to prevent this type error?
    Thanks again.

  5. #5

    Default

    I don't see why 8GB would hurt. I also don't see why it would help, since it looks like you're only using about 3GB of RAM. Note that the tuning guide recommends 5GB.

    See the second post in this thread for info regarding the Zero sized function error: http://www.wowza.com/forums/showthre...acketSizeReset

  6. #6

    Default

    Quote Originally Posted by randall View Post
    I don't see why 8GB would hurt. I also don't see why it would help, since it looks like you're only using about 3GB of RAM. Note that the tuning guide recommends 5GB.

    See the second post in this thread for info regarding the Zero sized function error: http://www.wowza.com/forums/showthre...acketSizeReset

    For now I have increased the java to use 5.5GB of memory (same GC method) and upgrade wowza.
    This seems to have fixed the issue but I am not sure what exactly fixed the issue because I also did a server reboot.

    Does anyone who is doing lot of live streaming find it is beneficial to reboot the server once in a few days?

    Thanks randall for help.

  7. #7

    Default

    Great news. You should not have to reboot the server. If you do, there is a problem that needs to be resolved.

  8. #8

    Unhappy

    Hi All,

    Unfortunately problem starts happening again during our peak hours (only). When load is regular wowza runs very smooth.
    When I say our peak time it is also likely that it is also our Data Center's peak hours (so bandwdith/network issues are possible)



    Just like when I first posted this error we are again having these errors in the logs
    ERROR	server	comment	2011-11-06	13:14:09	-	3048.964 -	Zero sized function (client:719332946:WIN 10,2,152,26): type:20 size:0:
    ERROR	server	comment	2011-11-06	13:14:09	-       3048.964	-Data format incorrect, killing client connection.
    As you can see from the attached JMX heap monitor image a Wowza restart with the following setenv.sh changes have stabilize the issue for now. Unfortunately, we only made the change after suffering through the peak period so we have to wait and see how the fix performs in the peak hours

    JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:NewSize=512m"
    What I really like to find out is what would cause a GC disaster like this into Wowza. I know few other wowza farms in the same network and they run very smooth over a same (load/number of edges). When this happened today server just froze and JwPlayer displayed "origin server can not be found". Even did a cold boot but did not help.

    randall, I think I may have sent you an email outside of the forum through a Wowza recommendation.
    please let me know, I would not mind paying to fix/understand this issue since it is really effecting us.

Similar Threads

  1. Memory usage issue with liverepeater origin-edge setup
    By yee.woon2@gmail.com in forum Live Streaming and Encoder Discussion
    Replies: 4
    Last Post: 07-27-2014, 02:21 PM
  2. Java Out of memory error
    By rezilient in forum General Forum
    Replies: 7
    Last Post: 02-18-2014, 11:00 AM
  3. Replies: 1
    Last Post: 12-08-2013, 02:47 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •