Results 1 to 10 of 10

Thread: stream repeater causing java.lang.OutOfMemoryError: Java heap space and carshing

  1. #1
    Join Date
    Oct 2012
    Posts
    7

    Default stream repeater causing java.lang.OutOfMemoryError: Java heap space and carshing

    hi
    we have a scenario where configured repeated streams(pulled) are down due to "no service" , the Wowza instance retry to pull those streams till he crash.
    the first warning is :LiveMediaStreamReceiver.doWatchdog: streamTimeout: Resetting connection
    after many retries with the above warning we get : Unexpected exception.java.lang.OutOfMemoryError: Java heap space
    java.lang.OutOfMemoryError: Java heap space
    at java.nio.HeapByteBuffer.<init>(Unknown Source)
    at java.nio.ByteBuffer.allocate(Unknown Source)
    at org.apache.mina.common.SimpleByteBufferAllocator.allocate(Unknown Source)
    at org.apache.mina.common.ByteBuffer.allocate(Unknown Source)
    at org.apache.mina.transport.socket.nio.SocketIoProcessor.read(Unknown Source)
    at org.apache.mina.transport.socket.nio.SocketIoProcessor.process(Unknown Source)
    at org.apache.mina.transport.socket.nio.SocketIoProcessor.access$600(Unknown Source)
    at org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run(Unknown Source)
    at org.apache.mina.util.NamePreservingRunnable.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)


    another warnings on this scenario are : NetConnectionConnection.connect: Failed to connect[xxx.xx.xxxx.net:1935]: org.apache.mina.common.RuntimeIOException: Failed to get the session[java.net.SocketException: Network is unreachable].
    LiveMediaStreamReceiver.internalConnect: [[xxx.xx.xxxx.net:1935]:java.nio.channels.UnresolvedAddressException



    the current setenv.sh settings
    #!/bin/sh

    _EXECJAVA=/opt/local/lvs/jre1.7.0_09/bin/java
    JAVA_OPTS="-Xmx3000M"

    # Uncomment to run server environment (faster), Note: will only work if server VM install, comes with JDL
    JAVA_OPTS="$JAVA_OPTS -server"

    # Can be a better GC setting to avoid long pauses
    #JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:NewSize=512m"

    # Uncomment to log debug garbage collection information
    #NOW=$(date +"%Y-%m-%d-%H-%M")
    #JAVA_OPTS="$JAVA_OPTS -verbose:gc -Xloggc:"/usr/local/WowzaMediaServer/logs/gc_$NOW.log" -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC"

    # Uncomment to write heap dump on OutOfMemoryError
    #JAVA_OPTS="$JAVA_OPTS -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/local/WowzaMediaServer"

    # Uncomment to force Java to use specific language settings
    #JAVA_OPTS="$JAVA_OPTS -Duser.language=en -Duser.country=US -Dfile.encoding=Cp1252"

    # Uncomment to fix multicast crosstalk problem when streams share multicast port
    JAVA_OPTS="$JAVA_OPTS -Djava.net.preferIPv4Stack=true"

    WMSAPP_HOME=/opt/local/lvs/WowzaMediaServer
    WMSCONFIG_HOME=/opt/local/lvs/WowzaMediaServer
    WMSCONFIG_URL=

    export WMSAPP_HOME WMSCONFIG_HOME JAVA_OPTS _EXECJAVA



    How can we avoid this from happening ?

    Yaniv

  2. #2

    Default

    Hi there, these messages are due to network problems.

    I am not sure if the outOfMemory issue is due to network problems. Do you have any custom modules enabled?

    Can you post your start-up messages?

    Salvadore

  3. #3
    Join Date
    Oct 2012
    Posts
    7

    Default

    Hi
    true those messages are due to network problems , we fail to pull stream since its temporary down.
    when we press "stop receiving streams" the warning are gone.
    when streams are up we get no warning and no crashes.

    ether way Wowza shouldn't crash or get out of memory due to failure on pulling streams.

    Yaniv

  4. #4

    Default

    Hi Yaniv,

    Can you show the startup lines from your access log?

    Salvadore

  5. #5
    Join Date
    Oct 2012
    Posts
    7

    Default

    Hi Salvadore

    below is the startup print , can u share what u looking for ?


    comment server INFO 200 - Server License Key: SVRM3-XXXXX-XXXXX-XXXXX-XXXXX-aCQDz - - - 0.647 - - - - - - - - - - - - - - - - - - - - - - - - -
    comment server INFO 200 - Maximum Connections: Unlimited - - - 0.649 - - - - - - - - - - - - - - - - - - - - - - - - -
    comment server INFO 200 - Transcoder Streams Available: Unlimited - - - 0.654 - - - - - - - - - - - - - - - - - - - - - - - - -
    comment server INFO 200 - Transcoder Watermark: No - - - 0.655 - - - - - - - - - - - - - - - - - - - - - - - - -
    comment server INFO 200 - nDVR Available: Yes - - - 0.655 - - - - - - - - - - - - - - - - - - - - - - - - -
    comment server INFO 200 - DRM Available: Yes - - - 0.656 - - - - - - - - - - - - - - - - - - - - - - - - -
    comment server INFO 200 - Hardware Available Processors: 4 - - - 0.657 - - - - - - - - - - - - - - - - - - - - - - - - -
    comment server INFO 200 - Hardware Physical Memory: 3247MB/3947MB - - - 0.658 - - - - - - - - - - - - - - - - - - - - - - - - -
    comment server INFO 200 - Hardware Swap Space: 511MB/511MB - - - 0.658 - - - - - - - - - - - - - - - - - - - - - - - - -
    comment server INFO 200 - Max File Descriptor Count: 20000 - - - 0.659 - - - - - - - - - - - - - - - - - - - - - - - - -
    comment server INFO 200 - Open File Descriptor Count: 47 - - - 0.66 - - - - - - - - - - - - - - - - - - - - - - - - -
    comment server INFO 200 - OS Name: Linux - - - 0.66 - - - - - - - - - - - - - - - - - - - - - - - - -
    comment server INFO 200 - OS Version: 2.6.18-308.16.1.el5 - - - 0.661 - - - - - - - - - - - - - - - - - - - - - - - - -
    comment server INFO 200 - OS Architecture: amd64 - - - 0.662 - - - - - - - - - - - - - - - - - - - - - - - - -
    comment server INFO 200 - Java Name: Java HotSpot(TM) 64-Bit Server VM - - - 0.662 - - - - - - - - - - - - - - - - - - - - - - - - -
    comment server INFO 200 - Java Vendor: Oracle Corporation - - - 0.663 - - - - - - - - - - - - - - - - - - - - - - - - -
    comment server INFO 200 - Java Version: 1.7.0_09 - - - 0.663 - - - - - - - - - - - - - - - - - - - - - - - - -
    comment server INFO 200 - Java VM Version: 23.5-b02 - - - 0.664 - - - - - - - - - - - - - - - - - - - - - - - - -
    comment server INFO 200 - Java Spec Version: 1.7 - - - 0.665 - - - - - - - - - - - - - - - - - - - - - - - - -
    comment server INFO 200 - Java Home: /opt/local/lvs/jre1.7.0_09 - - - 0.665 - - - - - - - - - - - - - - - - - - - - - - - - -
    comment server INFO 200 - Java Max Heap Size: 2666MB - - - 0.666 - - - - - - - - - - - - - - - - - - - - - - - - -
    comment server INFO 200 - Java Architecture: 64 - - - 0.666 - - - - - - - - - - - - - - - - - - - - - - - - -
    comment server INFO 200 - Java Locale[user.language]: en - - - 0.667 - - - - - - - - - - - - - - - - - - - - - - - - -
    comment server INFO 200 - Java Locale[user.country]: US - - - 0.667 - - - - - - - - - - - - - - - - - - - - - - - - -
    comment server INFO 200 - Java Locale[file.encoding]: UTF-8 - - - 0.668 - - - - - - - - - - - - - - - - - - - - - - - - -
    comment server INFO 200 - Java Timezone[user.timezone]: Etc/GMT - - - 0.669 - - - - - - - - - - - - - - - - - - - - - - - - -
    comment server INFO 200 - Java Args[0]: -Xmx3000M - - - 0.669 - - - - - - - - - - - - - - - - - - - - - - - - -
    comment server INFO 200 - Java Args[1]: -Djava.net.preferIPv4Stack=true - - - 0.67 - - - - - - - - - - - - - - - - - - - - - - - - -
    comment server INFO 200 - Java Args[2]: -Dcom.sun.management.jmxremote=true - - - 0.671 - - - - - - - - - - - - - - - - -
    comment server INFO 200 - Java Args[3]: -Dcom.wowza.wms.runmode=service - - - 0.671 - - - - - - - - - - - - - - - - - -
    comment server INFO 200 - Java Args[4]: -Dcom.wowza.wms.native.base=linux - - - 0.672 - - - - - - - - - - - - - - - - - -
    comment server INFO 200 - Java Args[5]: -Dcom.wowza.wms.AppHome=/opt/local/lvs/WowzaMediaServer - - - 0.672 - - - - - - - - - - - - - - -
    comment server INFO 200 - Java Args[6]: -Dcom.wowza.wms.ConfigURL= - - - 0.673 - - - - - - - - - - - - - - - - - -
    comment server INFO 200 - Java Args[7]: -Dcom.wowza.wms.ConfigHome=/opt/local/lvs/WowzaMediaServer - - - 0.673 - - - - - - - - - - - - - -
    comment server INFO 200 - Server runmode: service - - - 0.674 - - - - - - - - - - - - - - - - - - - - -
    comment server INFO 200 - Server native.platform: linux - - - 0.675 - - - - - - - - - - - - - - - - - - - -
    comment server INFO 200 - Server threads[h/t]: 10/10 - - - 0.675 - - - - - - - - - - - - - - - - - - - -
    comment server INFO 200 - JMX java.rmi.server.hostname: localhost/ - - - 1.824 - - - - - - - - - - - - - - - - - -
    comment server INFO 200 - JMX bind attempt service:jmx:rmi://localhost/:8084/jndi/rmi://localhost/:8085/jmxrmi - - - 1.824 - - - - - - - - - - - - -
    comment server INFO 200 - JMX password file: /opt/local/lvs/WowzaMediaServer/conf/jmxremote.password - - - 1.827 - - - - - - - - - - - - - -
    comment server INFO 200 - JMX access file: /opt/local/lvs/WowzaMediaServer/conf/jmxremote.access - - - 1.828 - - - - - - - - - - - - - - -
    comment server INFO 200 - JMX bind successful - - - 1.904 - - - - - - - - - - - - - - - - - - - - -
    comment server INFO 200 - CMDInterface now listening: [any]:8083 - - - 1.939 - - - - - - - - - - - - - - - - - - -
    vhost-start vhost INFO 200 _defaultVHost_ - - - - 2.693 - - - - - - - - - - - - - - - - - - - - -
    comment server INFO 200 - _defaultVHost_ threads[h/t]:120/80 home:/opt/local/lvs/WowzaMediaServer - - - 2.759 - - - - - - - - - - - - - - -
    comment vhost INFO 200 _defaultVHost_ Bind attempt ([any]:2255:4) - - - 2.764 - - - - - - - - - - - - - - - - - - -
    comment vhost INFO 200 _defaultVHost_ Bind successful ([any]:2255) - - - 2.77 - - - - - - - - - - - - - - - - - - -
    comment vhost INFO 200 _defaultVHost_ Bind attempt ([any]:8080:4) - - - 2.773 - - - - - - - - - - - - - - - - - - -
    comment vhost INFO 200 _defaultVHost_ Bind successful ([any]:8080) - - - 2.775 - - - - - - - - - - - - - - - - - - -
    comment vhost INFO 200 _defaultVHost_ Bind attempt ([any]:1935:4) - - - 2.776 - - - - - - - - - - - - - - - - - - -
    comment vhost INFO 200 _defaultVHost_ Bind successful ([any]:1935) - - - 2.777 - - - - - - - - - - - - - - - - - - -
    comment vhost INFO 200 _defaultVHost_ Bind attempt ([any]:8086:1) - - - 2.778 - - - - - - - - - - - - - - - - - - -
    comment vhost INFO 200 _defaultVHost_ Bind successful ([any]:8086) - - - 2.778 - - - - - - - - - - - - - - - - - - -
    comment server INFO 200 - ModuleRTMPAuthenticate.onAppStart: SecureToken is off - - - 2.945 - - - - - - - - - - - - - - - - -
    comment server INFO 200 - ModuleRTMPAuthenticate.onAppStart: Authorization password file: /opt/local/lvs/WowzaMediaServer/conf/publish.password - - - 2.945 - - - - - - - - -
    app-start application INFO 200 _definst_ secureliveHLS/_definst_ - - - 2.991 - - - - - - - - - - - - - - - - - -
    create stream INFO 200 - - - - - 0.011 - - - - - - - - - - 1 0 0 0 - - - - - - - - -
    comment server INFO 200 - LiveMediaStreamReceiver.connect: rtmp://xxxxxxxxxxxxxxxxxx_mobileHLS_480p_Opal_7_seats_Baccarat:rtmp://xxxxxxxxxxxxxxxxxx:1935/eurostech[854x480_mobileHLS_480p_Opal_7_seats_Baccarat] - - - 3.049 -
    comment server INFO 200 - VHost.startStartupStreams: Stream successfully started [secureliveHLS/_definst_]: flv:854x480_mobileHLS_480p_Opal_7_seats_Baccarat.stream - - - 3.058 - - - - - -
    create stream INFO 200 - - - - - 0.0 - - - - - - - - - - 2 0 0 0 - - - - - - - - -
    comment server INFO 200 - LiveMediaStreamReceiver.connect: rtmp://xxxxxxxxxxxxxxxxxx_mobileHLS_480p_Topaz_mini_Baccarat:rtmp://xxxxxxxxxxxxxxxxxx:1935/eurostech[854x480_mobileHLS_480p_Topaz_mini_Baccarat] - - - 3.06 -
    comment server INFO 200 - VHost.startStartupStreams: Stream successfully started [secureliveHLS/_definst_]: flv:854x480_mobileHLS_480p_Topaz_mini_Baccarat.stream - - - 3.062 - - - - - - -
    create stream INFO 200 - - - - - 0.0 - - - - - - - - - - 3 0 0 0 - - - - - - - - -
    comment server INFO 200 - LiveMediaStreamReceiver.connect: rtmp://xxxxxxxxxxxxxxxxxx_mobileHLS_480p_Diamond_Roulette:rtmp://xxxxxxxxxxxxxxxxxx:1935/eurostech[854x482_mobileHLS_480p_Diamond_Roulette] - - - 3.063 - -
    comment server INFO 200 - VHost.startStartupStreams: Stream successfully started [secureliveHLS/_definst_]: flv:854x482_mobileHLS_480p_Diamond_Roulette.stream - - - 3.065 - - - - - - -
    create stream INFO 200 - - - - - 0.0 - - - - - - - - - - 4 0 0 0 - - - - - - - - -
    comment server INFO 200 - LiveMediaStreamReceiver.connect: rtmp://xxxxxxxxxxxxxxxxxx_mobileHLS_480p_Pearl_Roulette:rtmp://xxxxxxxxxxxxxxxxxx:1935/eurostech[854x482_mobileHLS_480p_Pearl_Roulette] - - - 3.066 - -
    comment server INFO 200 - VHost.startStartupStreams: Stream successfully started [secureliveHLS/_definst_]: flv:854x482_mobileHLS_480p_Pearl_Roulette.stream - - - 3.068 - - - - - - -
    comment server INFO 200 - ModuleRTMPAuthenticate.onAppStart: SecureToken is off - - - 3.127 - - - - - - - - - - - - - - - - -
    comment server INFO 200 - ModuleRTMPAuthenticate.onAppStart: Authorization password file: /opt/local/lvs/WowzaMediaServer/conf/publish.password - - - 3.127 - - - - - - - - -
    app-start application INFO 200 _definst_ liveHLS/_definst_ - - - 3.133 - - - - - - - - - - - - - - - - - -
    create stream INFO 200 - - - - - 0.0 - - - - - - - - - - 1 0 0 0 - - - - - - - - -
    comment server INFO 200 - LiveMediaStreamReceiver.connect: rtmp://xxxxxxxxxxxxxxxxxxs_mobileHLS_480p_1495:rtmp://xxxxxxxxxxxxxxxxxx[s_mobileHLS_480p_1495] - - - 3.135 - - - - - - -
    comment server INFO 200 - VHost.startStartupStreams: Stream successfully started [liveHLS/_definst_]: flv:mobileHLS_480p_1495.stream - - - 3.136 - - - - - - - - - -
    comment server INFO 200 - Server.startShutdownHook: Start server shutdown hook - - - 3.137 - - - - - - - - - - - - - - - - -
    comment server INFO 200 - Wowza Media Server is started! - - - 3.138 - - - - - - - - - - - - - - - - - - - -
    comment server INFO 200 -

  6. #6

    Default

    I see that your Hardware Physical Memory is: 3247MB/3947MB
    With a Java Heap size of 3000MB, which is almost all of your available ram (3247MB) going to heap size, and JMX running you are right at maxed out on ram, not to mention if anything else is running.

    You might try lowering the Heap size to 2500MB, you may also want to consider a dedicated Wowza server if you are trying to use the machine for other activities.

    Salvadore

  7. #7

    Default

    Also, make sure you are utilizing all your CPU resources by tuning your VHost.xml file to use 4 cores, as shown in the tuning guide

    Salvadore

  8. #8
    Join Date
    Oct 2012
    Posts
    7

    Default

    Hi Salvadore thanks for u'r replay
    we have modified this value lately due to the crashes we have experienced , it was the default value till 8 days ago.
    my problem is still , if stream repeater have values that fail to pull streams my instance will crash after x hours.
    u can try it your self .. config some dummy/non-existed pulled streams and see your instance go out of memory , more streams that are failing the quicker the memory will run out.


    Yaniv

  9. #9

    Default

    Try making the .stream file on the edge that points to the origin like this:

    {
        uri:"rtmp://[wowza-address]:1935/liveorigin/myStream",
        lsrResetOnStreamNotFound: false
    }
    Salvadore

  10. #10
    Join Date
    Oct 2012
    Posts
    7

    Default

    10x will try and update ..


    UPDATE : well the above syntax generates an error.
    ERROR server comment 2013-07-08 07:18:18 - - - - - 430709.857 - - - - - - - - Invalid LiveMediaStream URL[{]: java.net.URISyntaxException: Illegal character in path at index 0: {
    WARN server comment 2013-07-08 07:18:18 - - - - - 430709.858 - - - - - - - - LiveMediaStreamReceiver.connect: Failure no URL: {

    Yaniv
    Last edited by janiv77; 07-08-2013 at 12:26 AM.

Similar Threads

  1. Unexpected exception.: java.lang.OutOfMemoryError: Java heap space
    By sriv.mayank in forum Performance and Tuning
    Replies: 1
    Last Post: 03-26-2014, 02:40 PM
  2. Error: java.lang.OutOfMemoryError: Java heap space
    By Denis_Ukraine in forum Server Administration Discussion
    Replies: 5
    Last Post: 05-20-2013, 03:32 PM
  3. Wowza failed after 10 days - java.lang.OutOfMemoryError: Java heap space
    By tribalhost in forum Performance Tuning Discussion
    Replies: 5
    Last Post: 04-10-2013, 06:30 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •