Results 1 to 6 of 6

Thread: Wowza server hanged "java.lang.OutOfMemoryError: Java heap space "

  1. #1

    Question Wowza server hanged "java.lang.OutOfMemoryError: Java heap space "

    Hi,

    We have setup wowza in origin-edge configuration. Earlier it was streaming 3 channels from origin and it was operating fine. Then we installed NVIDIA K4000 Card and we setup 8 channels at the origin. Everything was going fine as expected. But suddenly, yesterday, the origin wowza server just got hanged. By hanged, I mean the service was running, but it was not processing the stream.

    I checked the log there was message like:
    WARN server comment 2013-09-15 07:07:02 - - - - - 235821.696 - - - - - - - - Unexpected exception.
    java.lang.OutOfMemoryError: Java heap space
    at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
    at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
    at org.apache.mina.common.SimpleByteBufferAllocator.allocate(Unknown Source)
    at org.apache.mina.common.ByteBuffer.allocate(Unknown Source)
    at org.apache.mina.transport.socket.nio.SocketIoProcessor.read(Unknown Source)
    at org.apache.mina.transport.socket.nio.SocketIoProcessor.process(Unknown Source)
    at org.apache.mina.transport.socket.nio.SocketIoProcessor.access$600(Unknown Source)
    at org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run(Unknown Source)
    at org.apache.mina.util.NamePreservingRunnable.run(Unknown Source)
    at java.lang.Thread.run(Thread.java:662)
    WARN server comment 2013-09-15 07:07:05 - - - - - 235841.725 - - - - - - - - Unexpected exception.
    java.lang.OutOfMemoryError: Java heap space
    at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
    at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
    at org.apache.mina.common.SimpleByteBufferAllocator.allocate(Unknown Source)
    at org.apache.mina.common.ByteBuffer.allocate(Unknown Source)
    at org.apache.mina.transport.socket.nio.SocketIoProcessor.read(Unknown Source)
    at org.apache.mina.transport.socket.nio.SocketIoProcessor.process(Unknown Source)
    at org.apache.mina.transport.socket.nio.SocketIoProcessor.access$600(Unknown Source)
    at org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run(Unknown Source)
    at org.apache.mina.util.NamePreservingRunnable.run(Unknown Source)
    at java.lang.Thread.run(Thread.java:662)
    WARN server comment 2013-09-15 07:07:05 - - - - - 235829.413 - - - - - - - - Unexpected exception.


    I checked the forum and found this issue is similar to the issue mentioned at;
    http://www.wowza.com/forums/showthre...ava-heap-space

    But, didn't find any solution in that thread.

    I have already tuned my Wowza as per :
    http://www.wowza.com/forums/content....general-tuning



    Any solution of this?

    Thanks,
    Birendra

  2. #2
    Join Date
    May 2013
    Posts
    680

    Default

    Hello Birendra,

    You should try to grab a heap dump the next time you receive this exception and send it to support@wowza.com along with a zipped file of your conf & logs.

    Matt

  3. #3
    Join Date
    Nov 2013
    Posts
    3

    Default

    We are having the same issue running Wowza on Amazon EC2

    java.lang.OutOfMemoryError: Java heap space

    appears in the logs repeatedly, CPU goes to 100%, wowza does not stream and becomes unresponsive.

    we also get this stack trace in the error log

    java.lang.OutOfMemoryError: Java heap space
    at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
    at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
    at org.apache.mina.common.SimpleByteBufferAllocator.allocate(Unknown Source)
    at org.apache.mina.common.ByteBuffer.allocate(Unknown Source)
    at org.apache.mina.transport.socket.nio.SocketIoProcessor.read(Unknown Source)
    at org.apache.mina.transport.socket.nio.SocketIoProcessor.process(Unknown Source)
    at org.apache.mina.transport.socket.nio.SocketIoProcessor.access$600(Unknown Source)
    at org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run(Unknown Source)
    at org.apache.mina.util.NamePreservingRunnable.run(Unknown Source)
    at java.lang.Thread.run(Thread.java:724)

    The Amazon AMI ID is ami-4669142f.

    We have not done any special tuning (no adjusting OS, Java, or Wowza memory or other limits), on the assumption that the pre-built AMI would be correctly configured "out-of-the-box".

    We have enabled RTMP PushPublishing according to the instructions available on the Internet and in general it seems to be working as designed.

    The OutOfMemory errors are coming under a light server load, for example 1 stream with 3 bitrates, 700k, 400k, 200k, and an occasional VOD from S3.

    This has happened twice so far.

  4. #4

    Default

    The best I can do here is suggest you follow matt_y's advice as stated above:

    "You should try to grab a heap dump the next time you receive this exception and send it to support@wowza.com along with a zipped file of your conf & logs."

    Here is a guide for taking a heap dump:
    How to take a Java heap dump

    Salvadore

  5. #5
    Join Date
    Jun 2010
    Posts
    8

    Default

    I can report a similar OOM problem.

    Running Software: Wowza Media Server 3.6.3 build8031

    java version "1.7.0_25"
    OpenJDK Runtime Environment (IcedTea 2.3.10) (7u25-2.3.10-1~deb7u1)
    OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)

    On a Linux Debian Wheezy server, Intel 1270 CPU , 16 Gb of memory.

    Wowza has 8 Gb (XmX) allocated. Very low load (about 2%). The server has a server-sided playlist , running an 700 Mb , 12 minute, +/- 5 Mbit MP4 file for 24/7

    An external process (ffmpeg) reads that (RTMP) stream and feeds it back into Wowza (after transcoding).

    At startup the Wowza process uses around 630 Mb (RES). At this moment, about 6 hours after starting, the process is using 702 Mb (RES). This is normal and not causing the final OOM ; if Wowza would "leak" 10 Mb per hour then even after 48 hours the total leakage would not be more than 500 Mb (which is very acceptable to us).

    However at some point, in 1 or 2 days suddenly memory usage will go up a lot ( to 8000 Mb ) and there will be OOM errors in the log.

    This origin also feeds 2 edges, but the load is low (about 2-5 concurrent viewers maximum).

    This is a test-environment but we can't get it stable. The Edges are stable. The edges are located at about 40ms and 190ms distance.

    I've prepared the options to dump the heap, when that has happened we'll send it to Wowza if they want to have a look.

  6. #6
    Join Date
    Jun 2010
    Posts
    8

    Default

    The problem was 'solved' by setting the StreamType of the Origin from repeater to "live". According to the feedback from Wowza the origin was buffering video for the edge and due to a possible underperforming network it would keep buffering all the way to the maximum available memory and then crash. Now the Origin is 'stable' at around 900 Mb of memory usage after 4 days of operation under the same load.

Similar Threads

  1. Unexpected exception.: java.lang.OutOfMemoryError: Java heap space
    By sriv.mayank in forum Performance and Tuning
    Replies: 1
    Last Post: 03-26-2014, 02:40 PM
  2. Replies: 9
    Last Post: 07-03-2013, 08:31 AM
  3. Error: java.lang.OutOfMemoryError: Java heap space
    By Denis_Ukraine in forum Server Administration Discussion
    Replies: 5
    Last Post: 05-20-2013, 03:32 PM
  4. Wowza failed after 10 days - java.lang.OutOfMemoryError: Java heap space
    By tribalhost in forum Performance Tuning Discussion
    Replies: 5
    Last Post: 04-10-2013, 06:30 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •