Wowza Community

Wowza Heap and Memory is getting full

Hello,

We are running WowzaStreamingEngine 4.0.4. We are using Java 1.8.0_05, Java Heap Size is set to “Production” level.

We reload Wowza service daily but every few days Wowza service is out of memory even if the number of connection is low. How can we identify what is causing this?

Thank you,

I. P.

Hello there and welcome to the Wowza support forum.

Do you see any errors in the logs?

Do you have any custom modules enabled? If so does removing them solve the issue?

Can you please provide your server specs, and tuning settings? Or are you using the “auto tune” feature?

Performance Tuning

Is this a dedicated Wowza box? If not, it would help if it were.

Kind regards,

Salvadore

Hi,

This is now being handled in ticket #111073

Daren

Hi,

As you have tuned the server and you’re already using the “G1 (Garbage First) collector” which I would have recommended, please edit the [Wowza-Install]/conf/Tune.xml file to remove the comments “” from around the VMOptions/HeapDumpOnOutOfMemoryError which will create a heapdump when Wowza runs out of memory. The result should look like this.

<VMOption>-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="${com.wowza.wms.AppHome}/logs"</VMOption>

You will need to save the Tune.xml file after editing and restart Wowza for the changes to take effect.

After you have done this, wait until Wowza runs out of memory and it will create the heapdump which you will need to upload to a sharing site or FTP or web server for download and analysis.

Create a ticket with Wowza support by emailing support@wowza.com

When creating a ticket please attach a ZIP file which contains the following directories and files:

[Wowza-Install]/conf/

[Wowza-Install]/logs/

[Wowza-Install]/transcoder/

[Wowza-Install]/manager/logs/

Please make sure you provide logs which show Wowza server starting.

If you are not sure how to get this information please see the following tutorial.

How to create a compressed zip file in Windows, OS X, and Linux

As for the other message, please see it below followed by the cause.

WARN server comment LiveMediaStreamReceiver.doWatchdog: streamTimeout: Resetting connection

Cause: Wowza can’t connect to a MediaCaster stream. It could be a problem with the network or a problem with the source. For example, the camera was reached, but rejected the connection. Wowza will continue trying to reconnect until the stream is available again.

You can use the How to troubleshoot error messages article for referencing common messages in the error log.

Regards,

Jason

Hello All,

Has there been a conclusion after this point? Is there a known memory leak in Wowza 4?

We are also experiencing a possible memory leak. Should we open a new case or can you update this thread?

Regards,

Bora.

Hi,

There are currently no known memory leaks in Wowza so please do open a support ticket if you are having issues.

Andrew

Hi,

The heap file in that case is generated when the Java JVM experiences an out of memory critical event. You can also trigger a heap dump if this is required. Please see this article which shows a few methods to do this.

Paul

Hello,

I’ve removed comments for

in /conf/Tune.xml

This error occurred again, but I don’t see any new logs except access and error logs. Is this suppose to generate a new log file or are the errors added to existing logs?

Thank you,

Ionatan

I have followed the instructions given here, but I don’t see a heap dump file being created anywhere on my server. Can anyone help?

I have followed the instructions on that article, to automatically create a heap dump on out of memory errors.

This is the error as it appears on the log:

WARN server comment 2016-03-10 22:59:48 - - - - - 641832.52 - - - - - - - - Unexpected exception.: java.lang.OutOfMemoryError: Java heap space|

WARN server comment 2016-03-10 22:59:48 - - - - - 641832.52 - - - - - - - - Unexpected exception.: java.lang.OutOfMemoryError: Java heap space|

And the heapdump file is nowhere to be seen. Is there any other configuration needed?

I did follow that article, and my tune.xml file has this line uncommented:

-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="${com.wowza.wms.AppHome}/logs"

And still I see no heapdumps on that location.

Hello,

I’ve found this in logfile in the same timeframe when Wowza ran out of memory:

Unexpected exception.: java.lang.OutOfMemoryError: Java heap space|at java.util.HashMap.keySet(HashMap.java:906)|at java.util.HashSet.iterator(HashSet.java:172)|at sun.nio.ch.EPollSelectorImpl.implClose(EPollSelectorImpl.java:144)|at sun.nio.ch.SelectorImpl.implCloseSelector(SelectorImpl.java:113)|at java.nio.channels.spi.AbstractSelector.close(AbstractSelector.java:111)|

Also these errors are in error log

          • LiveMediaStreamReceiver.doWatchdog: streamTimeout[…]: Resetting connection: […]

WARN server comment 2014-11-27 11:53:08 - - - - - 42463.098 - - - - - - - -

These are Performance Tuning settings:

Memory

Total: 15.66 GB

Java Heap Size

Production level

10000 MB

Java Garbage Collection Settings

G1 (Garbage First) collector

Hello,

I’ve removed comments for

-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="${com.wowza.wms.AppHome}/logs"

in /conf/Tune.xml

This error occurred again, but I don’t see any new logs except access and error logs. Is this suppose to generate a new log file or are the errors added to existing logs?

Thank you,

Ionatan

Hi,

You can opt to generate a heap dump manually when you notice the heap increasing/peaking using the commands outlined in that post. Do you also see any hs_err_pid.log files in the bin/ directory?

Another option you can look into is to run a JFR on your Wowza instance. This is built-in to Java 8, and should be available for free when testing.

Michelle

Hello,

In the article How to take a Java heap dump, it mentions that the dump save location should be specified in your JAVA_OPTS command. For example:

-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=[path-to-heapdump]

Where [path-to-heapdump] is the path to where the heap dump file is stored (such as /opt).

Best regards,

Andrew

Hello peers having low heap,

I am trying to narrow down this issue with heap space not getting reclaimed until OOM or service affect the server due to uneffective garbage collection at critical almost full level.

Do you have many or just a few channels INPUT?

Do you suffer signal losses between your headend/encoder and Wowza Streaming Engine (WSE)?

Please check “Incoming Streams” one by one whether uptime is same as application uptime and ideally server uptime (if yes, you would confuse my theory, if no you qualify to actually suffer from this issue)

What input decoder is your most popular?

Do you transcode any channels?

What ouput packetizer is your most popular one?