Wowza Community

WowzaMediaServer too much use memories.

Version: Wowza Media Server 2 Perpetual 2.2.4.07 build27628

OS: CentOS release 5.7 (Final) 64bit

RAM: 8GB

Java version: 1.6.0_20

Java options (WowzaMediaSever’s setenv): -Xmx5000M -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC -XX:NewSize=256m -Djava.net.preferIPv4Stack=true

The media server too much use memories. ( CPU total usage is usually under 12%)

This media server used only live streaming and VOD server is seperated.

I set a FMLE(windows 7 64bit) PC and FMLE sends stream(RTMP, H.264, AAC) to WowzaMediaServer.

At first, there was no problem. But there is freezing 1 or 2 days after.(memory usage is over 75%, the concurrent users are under 50. )

Then, decoder clients can’t play any streams.(RTSP, RTMP, HLS)

That’s why I restart WowzaMediaServer almost every 2 days.

I tested many times to solve this problem as performance tuning guide. (https://www.wowza.com/docs/how-to-do-performance-tuning)

The test has changed options.

But there still has same problem.

Is there the solution or hints?

If I upgrade WowzaMediaSever from 2 to 3, these problem will not appear?

Do you have any custom modules?

Are there any other applications on that server using substantial memory?

Show the start up lines in the access log.

Upgrading could help

https://www.wowza.com/docs/how-to-upgrade-wowza-media-server-software-to-wowza-streaming-engine

Richard

Try reducing java heap size to 4000M or 3500M, because you only have about 5g available when the server is started.

Hardware Physical Memory: 5311MB/7000MB

d

Richard

You are using advanced GC options which might not be helping. As stated in step #5 of the tuning guide, we suggest not using these options unless there is already a problem as they might cause problems. Do you have the “-server” flag turned on following step #4?

Have you followed tuning step #6 for a 12 core machine, which is what you have?

It is also possible that you are just over-loading the machine, in which case you might need to setup Liverepeater (origin/edge) cluster to handle the load.

Richard

Do you mean the /conf/[app-name]/Application.xml /RTP /Property “streamTimeout”?

<Property>
	<Name>streamTimeout</Name>
	<Value>12000</Value>
	<Type>Integer</Type>
</Property>

Are there outOfMemory errors in the access and error log?

Is it possible for you to upgrade to Wowza 3.5? The current version is 3.5.0.02, with the latest patch applied.

https://www.wowza.com/docs/how-to-upgrade-wowza-media-server-software-to-wowza-streaming-engine

It might improve the issue at least, there has been much improvement since 2.2.4, though I cannot guarantee that

Richard

Richard, thank you for answer.

This machine dedicated for WowzaMediaServer. (Not VOD, but olny live with FMLE)

And I use only 1 module HTTPProvider, there is no other custom module.

Here is Start up lines.

2012-12-25	09:01:02	KST	server-start	server	INFO	200	Wowza Media Server 2 Perpetual 2.2.4.07 build27628	-	-	-	-	0.711	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2012-12-25	09:01:02	KST	comment	server	INFO	200	-	Serial number: XXXXX-XXXXX-XXXXX-XXXXX-UCNRH	-	-	-	0.713	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2012-12-25	09:01:02	KST	comment	server	INFO	200	-	Maximum connections: Unlimited	-	-	-	0.714	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2012-12-25	09:01:02	KST	comment	server	INFO	200	-	Hardware Available Processors: 12	-	-	-	0.719	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2012-12-25	09:01:02	KST	comment	server	INFO	200	-	Hardware Physical Memory: 5311MB/7000MB	-	-	-	0.72	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2012-12-25	09:01:02	KST	comment	server	INFO	200	-	Hardware Swap Space: 2047MB/2047MB	-	-	-	0.721	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2012-12-25	09:01:02	KST	comment	server	INFO	200	-	Max File Descriptor Count: 1024	-	-	-	0.722	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2012-12-25	09:01:02	KST	comment	server	INFO	200	-	Open File Descriptor Count: 44	-	-	-	0.722	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2012-12-25	09:01:02	KST	comment	server	INFO	200	-	OS Name: Linux	-	-	-	0.723	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2012-12-25	09:01:02	KST	comment	server	INFO	200	-	OS Version: 2.6.18-274.7.1.el5xen	-	-	-	0.724	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2012-12-25	09:01:02	KST	comment	server	INFO	200	-	OS Architecture: amd64	-	-	-	0.724	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2012-12-25	09:01:02	KST	comment	server	INFO	200	-	Java Name: Java HotSpot(TM) 64-Bit Server VM	-	-	-	0.725	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2012-12-25	09:01:02	KST	comment	server	INFO	200	-	Java Vendor: Oracle Corporation	-	-	-	0.725	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2012-12-25	09:01:02	KST	comment	server	INFO	200	-	Java Version: 1.7.0_03	-	-	-	0.726	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2012-12-25	09:01:02	KST	comment	server	INFO	200	-	Java VM Version: 22.1-b02	-	-	-	0.726	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2012-12-25	09:01:02	KST	comment	server	INFO	200	-	Java Spec Version: 1.7	-	-	-	0.727	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2012-12-25	09:01:02	KST	comment	server	INFO	200	-	Java Home: /usr/java/jdk1.7.0_03/jre	-	-	-	0.728	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2012-12-25	09:01:02	KST	comment	server	INFO	200	-	Java Max Heap Size: 4916MB	-	-	-	0.728	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2012-12-25	09:01:02	KST	comment	server	INFO	200	-	Java Architecture: 64	-	-	-	0.729	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2012-12-25	09:01:02	KST	comment	server	INFO	200	-	Java Encoding[file.encoding]: UTF-8	-	-	-	0.729	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2012-12-25	09:01:02	KST	comment	server	INFO	200	-	Java Args[0]: -Xmx5000M	-	-	-	0.73	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2012-12-25	09:01:02	KST	comment	server	INFO	200	-	Java Args[1]: -XX:+UseConcMarkSweepGC	-	-	-	0.731	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2012-12-25	09:01:02	KST	comment	server	INFO	200	-	Java Args[2]: -XX:+CMSIncrementalMode	-	-	-	0.731	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2012-12-25	09:01:02	KST	comment	server	INFO	200	-	Java Args[3]: -XX:+UseParNewGC	-	-	-	0.732	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2012-12-25	09:01:02	KST	comment	server	INFO	200	-	Java Args[4]: -XX:NewSize=256m	-	-	-	0.732	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2012-12-25	09:01:02	KST	comment	server	INFO	200	-	Java Args[5]: -Djava.net.preferIPv4Stack=true	-	-	-	0.733	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2012-12-25	09:01:02	KST	comment	server	INFO	200	-	Java Args[6]: -Dcom.sun.management.jmxremote=true	-	-	-	0.733	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2012-12-25	09:01:02	KST	comment	server	INFO	200	-	Java Args[7]: -Dcom.wowza.wms.AppHome=/WowzaMediaServer	-	-	-	0.734	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2012-12-25	09:01:02	KST	comment	server	INFO	200	-	Java Args[8]: -Dcom.wowza.wms.ConfigURL=	-	-	-	0.734	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2012-12-25	09:01:02	KST	comment	server	INFO	200	-	Java Args[9]: -Dcom.wowza.wms.ConfigHome=/WowzaMediaServer	-	-	-	0.735	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2012-12-25	09:01:02	KST	comment	server	INFO	200	-	Java GC[0]: ParNew	-	-	-	0.735	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2012-12-25	09:01:02	KST	comment	server	INFO	200	-	Java GC[1]: ConcurrentMarkSweep	-	-	-	0.736	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2012-12-25	09:01:02	KST	comment	server	INFO	200	-	Server threads[h/t]: 10/10	-	-	-	0.737	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2012-12-25	09:01:04	KST	comment	server	INFO	200	-	JMX java.rmi.server.hostname: xxx.xxx.xxx.xxx	-	-	-	2.346	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2012-12-25	09:01:04	KST	comment	server	INFO	200	-	JMX bind attempt service:jmx:rmi://127.0.0.1:8084/jndi/rmi://127.0.0.1:8085/jmxrmi	-	-	-	2.346	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2012-12-25	09:01:04	KST	comment	server	INFO	200	-	JMX password file: /WowzaMediaServer/conf/jmxremote.password	-	-	-	2.35	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2012-12-25	09:01:04	KST	comment	server	INFO	200	-	JMX access file: /WowzaMediaServer/conf/jmxremote.access	-	-	-	2.35	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2012-12-25	09:01:04	KST	comment	server	INFO	200	-	JMX bind successful	-	-	-	2.549	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2012-12-25	09:01:04	KST	comment	server	INFO	200	-	CMDInterface now listening: [any]:8083	-	-	-	2.598	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2012-12-25	09:01:05	KST	vhost-start	vhost	INFO	200	_defaultVHost_	-	-	-	-	3.261	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2012-12-25	09:01:05	KST	comment	server	INFO	200	-	_defaultVHost_ threads[h/t]:360/240 home:/WowzaMediaServer	-	-	-	3.325	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2012-12-25	09:01:05	KST	comment	vhost	INFO	200	_defaultVHost_	Bind attempt ([any]:1935:6)	-	-	-	3.331	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2012-12-25	09:01:05	KST	comment	vhost	INFO	200	_defaultVHost_	Bind successful ([any]:1935)	-	-	-	3.375	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2012-12-25	09:01:05	KST	comment	vhost	INFO	200	_defaultVHost_	Bind attempt ([any]:554:6)	-	-	-	3.378	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2012-12-25	09:01:05	KST	comment	vhost	INFO	200	_defaultVHost_	Bind successful ([any]:554)	-	-	-	3.379	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2012-12-25	09:01:05	KST	comment	vhost	INFO	200	_defaultVHost_	Bind attempt ([any]:8086:1)	-	-	-	3.379	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2012-12-25	09:01:05	KST	comment	vhost	INFO	200	_defaultVHost_	Bind successful ([any]:8086)	-	-	-	3.38	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-

Richard, thank you for advice.

I reduced java heap size to 4000M as your advice. (option: -Xmx4000M )

There was an exactly same problem after 3 days.

Is there any other hints for this problem?

I will try anything to solve this problem.

Richard, thanks for your advice.

I checked and test again as your advice.

  1. Of course this WowzaMediaServer runs as “-server” mode.

  2. I removed this GC option, but it still has same problem.

  3. I set step #6 for 12 core, such as ProcessorCount, PoolSize. -> It has same problem.

  4. I increase RAM from 7G to 11G and set 8G JAVA heap size and I added “NewSize=512” option. -> It has same problem.

  5. I don’t try origin/edge cluster not yet. This will be final step in this situation.

I found suspicious option “RTSPSessionTimeout=0” that is in “Application.xml”.

I want to try test as “RTSPSessionTimeout=90000” in “Applicaiont.xml”.

But now some Android apps can’t keep connection without option “RTSPSessionTimeout=0”.

Is it possible “RTSPSessionTimeout=0” can be cause of this memory problem?

So, I tried analyzing java heap memory dump.

Please look around the capture image.

No, it means the /conf/[app-name]/Application.xml /RTP /RTSPSessionTimeout

                <RTP>
                        <AVSyncMethod>senderreport</AVSyncMethod>
                        <MaxRTCPWaitTime>12000</MaxRTCPWaitTime>
                        <IdleFrequency>75</IdleFrequency>
                        [COLOR="#B22222"][B]<RTSPSessionTimeout>0</RTSPSessionTimeout>[/B][/COLOR]
                        <RTSPMaximumPendingWriteBytes>0</RTSPMaximumPendingWriteBytes>
                        <RTSPBindIpAddress></RTSPBindIpAddress>
                        <RTSPConnectionIpAddress>0.0.0.0</RTSPConnectionIpAddress>
                        <RTSPOriginIpAddress>127.0.0.1</RTSPOriginIpAddress>
                        <IncomingDatagramPortRanges>*</IncomingDatagramPortRanges>
                        <!-- Properties defined here will override any properties defined in conf/RTP.xml for any depacketizers loaded by this application -->
                        <Properties>
                        </Properties>
                </RTP>

Actually, I can’t upgrade Wowza version to 3.5 right now.

And there are no errors in access or error log.

It just freeze.

I’m trying test as “7200000”.

May be I can get the result 1 or 2 days later.

Here is the result of testing with option “RTSPSessionTimeout” in /conf/[app-name]/Application.xml /RTP

Before: 0 (The WowzaMediaServer was freezing in every 1 or 2 days. Only the solution is restarting.)

Now: 7200000

The WowzaMediaServer doesn’t freeze for 8 days and it works stable yet.

It was effective setting in compare with before.

I attached some graph of memory usage

1. WowzaMediaServer freezing pattern.

2. WowzaMediaServer works stable pattern.

I’m not sure the problem has been solved yet. Because of the red box in this graph.

Anyway, It works stable till now.

Please let me know your opinion of this result.