Results 1 to 7 of 7

Thread: Need help with tuning - crashing when low free memory

  1. #1
    Join Date
    Jan 2012
    Posts
    13

    Default Need help with tuning - crashing when low free memory

    Seeing wowza 3.0.3 crash on our linux box pretty consistently. I've narrowed it down to what I think is lack of memory but I do not see any memory errors in the console/logs. I have wowza ingests 8 live h264 streams ~1Mpbs, and just under 48 clients attached via HLS (encrypted) and RTMP. After about an hour, the free memory widdles down to about 56MB free. Soon after that, the java thread dies, and all the client devices stop playing video.

    I have seen you optimization page and tried most things there. I have also tried updating the Xmx option to 1000 and 1400. But no help, not sure how low I can/should go. I just need to server no more than 200 clients devices with this hardware configuration. Tnks.

    == vitals ==

    cpu: Intel(R) Core(TM)2 CPU L7400 @ 1.50GHz
    memory: 2G
    swap: none

    # java -version -server
    java version "1.6.0_22"
    OpenJDK Runtime Environment (IcedTea6 1.10.3) (ArchLinux-6.b22_1.10.3-1-x86_64)
    OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)

    == typical console last line ==
    ./startup.sh: line 25: 13537 Killed $_EXECJAVA $JAVA_OPTS $JMXOPTIONS -Dcom.wowza.wms.runmode="standalone" -Dcom.wowza.wms.native.base="linux" -Dcom.wowza.wms.AppHome="$WMSAPP_HOME" -Dcom.wowza.wms.ConfigURL="$WMSCONFIG_URL" -Dcom.wowza.wms.ConfigHome="$WMSCONFIG_HOME" -cp $WMSAPP_HOME/bin/wms-bootstrap.jar com.wowza.wms.bootstrap.Bootstrap start


    == A couple intersting WARN's near the end (but not always) ==

    acketHandler.handlePacket[live/_definst_/versus.stream]: Timecode out of order [video]: 79361635:79361701
    wowzamediaserver_error.log:WARN server comment 2012-02-08 18:19:08 144.236 - - - - - - - - LiveStreamPacketizerSmoothStreaming.handlePacket[live/_definst_/versus.stream]: Timecode out of order [video]: 793616350000:793617010000
    wowzamediaserver_error.log:WARN server comment 2012-02-08 18:19:08 144.672 - - - - - - - - CupertinoPacketHandler.handlePacket[live/_definst_/versus.stream]: Timecode out of order [video]: 79361635:79361701

    == random info ==
    # grep -i memory *logwowza.log:INFO server comment - Hardware Physical Memory: 1232MB/2003MB
    wowzamediaserver_access.log:2012-02-08 16:22:11 UTC comment server INFO 200 - Hardware Physical Memory: 579MB/2003MB - - 1.455 - - - - - - - - - -
    wowzamediaserver_access.log:2012-02-08 16:22:32 UTC comment server INFO 200 - Hardware Physical Memory: 575MB/2003MB - - 1.143 - - - - - - - - - -
    wowzamediaserver_access.log:2012-02-08 17:01:14 UTC comment server INFO 200 - Hardware Physical Memory: 1311MB/2003MB - - 1.677 - - - - - - - - - -
    wowzamediaserver_access.log:2012-02-08 17:06:29 UTC comment server INFO 200 - Hardware Physical Memory: 1291MB/2003MB - - 1.59 - - - - - - - - - -
    wowzamediaserver_access.log:2012-02-08 18:16:45 UTC comment server INFO 200 - Hardware Physical Memory: 1312MB/2003MB - - 1.834 - - - - - - - - - -
    wowzamediaserver_access.log:2012-02-08 18:50:13 UTC comment server INFO 200 - Hardware Physical Memory: 1232MB/2003MB - - 1.437 - - - - - - - - - -

  2. #2

    Default

    Your garbage collection may not be emptying all of your RAM and slowly building up over time. Try this on a box with 2GB of RAM:

    -Xmx1200M -Xms1200M -XX:+UseConcMarkSweepGC -XX:NewRatio=1 -Xss256k -XX:+AlwaysPreTouch -XX:+CMSScavengeBeforeRemark -XX:+UseParNewGC -XX:ParallelCMSThreads=2 -XX:ParallelGCThreads=2

    Don't know if that will help and you may need to tweak a couple of values (such as the parallel threads) but it should be a good place to start.

    Thanks,

    Benjamin

  3. #3

    Default

    Try to run with Oracle JDK 1.7

  4. #4
    Join Date
    Jan 2012
    Posts
    13

    Default

    PMBencredible: Tnks. I tried those options tweaking Xmx and Xms to lower identical values down to 800M and it still crashed but too longer.

    PMcdgraff:I updated to:
    INFO server comment - Java Vendor: Oracle Corporation
    INFO server comment - Java Version: 1.7.0_03
    INFO server comment - Java VM Version: 22.1-b02

    And that was the ticket!!! Nice. Tnks.

  5. #5

    Default

    Thanks for the feedback, is good to know than changing the JVM engine to Oracle fix the issue, I think will be useful to other.

    BTW; into Amazon EC2 I see than use OpenJDK, Charlie? why is this? is not better use Oracle JVM?

    Thanks,
    Ale

  6. #6

    Default

    We use Oracle Java on EC2. The latest 3.0.4 AMIs are setup to use this version. We were not able to gracefully replace the OpenJDK that is installed on teh AMI so the java command is still linked to OpenJDK. If you look at the startup logs you will see that we are using a different JAva command to get the installed Oracle version:

    _EXECJAVA=java
    if [ -f /usr/java/latest/bin/java ] ; then
    	_EXECJAVA=/usr/java/latest/bin/java
    fi
    Charlie

  7. #7

    Default

    ooo, ok thanks for the response Charlie! I'll check the logs until de Java -version command line.

Similar Threads

  1. Transcoder errors when running 3.6.2 on low memory setting
    By pburke in forum AddOn: Transcoder
    Replies: 5
    Last Post: 09-09-2013, 09:34 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •