Java/Wowza high cpu load (1200%)

[Srx291]

Hello,
this is a system running wowza. At random times (not always under high traffic) it goes fully overloaded due java/wowza.

Only way to fix it is restart wowza.

Versions:
- Wowza Media Server 2.2.4 build27452 (25 April 2011)
- java version "1.6.0_25" - Java(TM) SE Runtime Environment (build 1.6.0_25-b06) - Java HotSpot(TM) 64-Bit Server VM (build 20.0-b11, mixed mode)

Streaming:
- live streaming (webcams)

top - 11:53:52 up 5 days, 1:37, 1 user, load average: 11.35, 12.00, 12.29
Tasks: 205 total, 2 running, 203 sleeping, 0 stopped, 0 zombie
Cpu0 : 0.0%us,100.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu1 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu2 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu3 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu4 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu5 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu6 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu7 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu8 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu9 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu10 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu11 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu12 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu13 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu14 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu15 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 24674040k total, 19957468k used, 4716572k free, 455124k buffers
Swap: 6144856k total, 0k used, 6144856k free, 14172028k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
17083 root 20 0 7513m 1.4g 10m S 1281.2 5.9 9251:06 java

This is now with 6 or 7 live streams and about 50 viewers. (Only restarting wowza would set load to 0.xx again)

[shamrock]

I would post your Application.xml and possibly your Server.xml along with finding a point in your logs where the issue starts, may show an error/output.

Shamrock

[Srx291]

Access and Error Logs are clean of errors.
System logs are also clean.


Application.xml

<Root>
        <Application>
                <!-- Uncomment to set application level timeout values
                <ApplicationTimeout>60000</ApplicationTimeout>
                <PingTimeout>12000</PingTimeout>
                <ValidationFrequency>8000</ValidationFrequency>
                <MaximumPendingWriteBytes>0</MaximumPendingWriteBytes>
                <MaximumSetBufferTime>60000</MaximumSetBufferTime>
                <MaximumStorageDirDepth>25</MaximumStorageDirDepth>
                -->
                <Connections>
                        <AutoAccept>true</AutoAccept>
                        <AllowDomains></AllowDomains>
                </Connections>
                <!--
                        StorageDir path variables

                        ${com.wowza.wms.AppHome} - Application home directory
                        ${com.wowza.wms.ConfigHome} - Configuration home directory
                        ${com.wowza.wms.context.VHost} - Virtual host name
                        ${com.wowza.wms.context.VHostConfigHome} - Virtual host config directory
                        ${com.wowza.wms.context.Application} - Application name
                        ${com.wowza.wms.context.ApplicationInstance} - Application instance name

                -->
                <Streams>
                        <StreamType>default</StreamType>
                        <StorageDir>${com.wowza.wms.context.VHostConfigHome}/content</StorageDir>
                        <KeyDir>${com.wowza.wms.context.VHostConfigHome}/keys</KeyDir>
                        <!-- LiveStreamPacketizers (separate with commas): cupertinostreamingpacketizer, smoothstreamingpacketizer, sanjosestreamingpacketizer, cupertinostreamingrepeater, smoothstreamingrepeater, sanjosestreamingrepeater -->
                        <LiveStreamPacketizers></LiveStreamPacketizers>
                        <!-- Properties defined here will override any properties defined in conf/Streams.xml for any streams types loaded by this application -->
                        <Properties>
                        </Properties>
                </Streams>
                <!-- HTTPStreamers (separate with commas): cupertinostreaming, smoothstreaming, sanjosestreaming -->
                <HTTPStreamers>cupertinostreaming,smoothstreaming,sanjosestreaming</HTTPStreamers>
                <SharedObjects>
                        <StorageDir></StorageDir>
                </SharedObjects>
                <Client>
                        <IdleFrequency>-1</IdleFrequency>
                        <Access>
                                <StreamReadAccess>*</StreamReadAccess>
                                <StreamWriteAccess>*</StreamWriteAccess>
                                <StreamAudioSampleAccess></StreamAudioSampleAccess>
                                <StreamVideoSampleAccess></StreamVideoSampleAccess>
                                <SharedObjectReadAccess>*</SharedObjectReadAccess>
                                <SharedObjectWriteAccess>*</SharedObjectWriteAccess>
                        </Access>
                </Client>
                <RTP>
                        <!-- RTP/Authentication/[type]Methods defined in Authentication.xml. Default setup includes; none, basic, digest -->
                        <Authentication>
                                <PublishMethod>digest</PublishMethod>
                                <PlayMethod>digest</PlayMethod>
                        </Authentication>
                        <!-- RTP/AVSyncMethod. Valid values are: senderreport, systemclock, rtptimecode -->
                        <AVSyncMethod>senderreport</AVSyncMethod>
                        <MaxRTCPWaitTime>12000</MaxRTCPWaitTime>
                        <IdleFrequency>75</IdleFrequency>
                        <RTSPSessionTimeout>90000</RTSPSessionTimeout>
                        <RTSPMaximumPendingWriteBytes>0</RTSPMaximumPendingWriteBytes>
                        <RTSPBindIpAddress></RTSPBindIpAddress>
                        <RTSPConnectionIpAddress>0.0.0.0</RTSPConnectionIpAddress>
                        <RTSPOriginIpAddress>127.0.0.1</RTSPOriginIpAddress>
                        <IncomingDatagramPortRanges>*</IncomingDatagramPortRanges>
                        <!-- Properties defined here will override any properties defined in conf/RTP.xml for any depacketizers loaded by this application -->
                        <Properties>
                        </Properties>
                </RTP>
                <MediaCaster>
                        <!-- Properties defined here will override any properties defined in conf/MediaCasters.xml for any MediaCasters loaded by this applications -->
                        <Properties>
                        </Properties>
                </MediaCaster>
                <MediaReader>
                        <!-- Properties defined here will override any properties defined in conf/MediaReaders.xml for any MediaReaders loaded by this applications -->
                        <Properties>
                        </Properties>
                </MediaReader>
                <MediaWriter>
                        <!-- Properties defined here will override any properties defined in conf/MediaWriter.xml for any MediaWriter loaded by this applications -->
                        <Properties>
                        </Properties>
                </MediaWriter>
                <LiveStreamPacketizer>
                        <!-- Properties defined here will override any properties defined in conf/LiveStreamPacketizers.xml for any LiveStreamPacketizers loaded by this applications -->
                        <Properties>
                        </Properties>
                </LiveStreamPacketizer>
                <HTTPStreamer>
                        <!-- Properties defined here will override any properties defined in conf/HTTPStreamers.xml for any HTTPStreamer loaded by this applications -->
                        <Properties>
                        </Properties>
                </HTTPStreamer>
                <Repeater>
                        <OriginURL></OriginURL>
                        <QueryString><![CDATA[]]></QueryString>
                </Repeater> 
                <Modules>
                        <Module>
                                <Name>base</Name>
                                <Description>Base</Description>
                                <Class>com.wowza.wms.module.ModuleCore</Class>
                        </Module>
                        <Module>
                                <Name>properties</Name>
                                <Description>Properties</Description>
                                <Class>com.wowza.wms.module.ModuleProperties</Class>
                        </Module>
                        <Module>
                                <Name>logging</Name>
                                <Description>Client Logging</Description>
                                <Class>com.wowza.wms.module.ModuleClientLogging</Class>
                        </Module>
                        <Module>
                                <Name>flvplayback</Name>
                                <Description>FLVPlayback</Description>
                                <Class>com.wowza.wms.module.ModuleFLVPlayback</Class>
                        </Module> 
                </Modules>
                <!-- Properties defined here will be added to the IApplication.getProperties() and IApplicationInstance.getProperties() collections -->
                <Properties>
                </Properties>
        </Application>
</Root>

Server.xml

<Root>
        <Server>
                <CommandInterface>
                        <HostPort>
                                <IpAddress>127.0.0.1</IpAddress>
                                <Port>8083</Port>
                        </HostPort>
                </CommandInterface>
                <AdminInterface>
                        <!-- Objects exposed through JMX interface: Server, VHost, VHostItem, Application, ApplicationInstance, MediaCaster, Module, Client, MediaStream, SharedObject, Acceptor, IdleWorker -->
                        <ObjectList>Server,VHost,VHostItem,Application,ApplicationInstance,MediaCaster,Module,IdleWorker</ObjectList>
                </AdminInterface>
                <!-- JMXUrl: service:jmx:rmi://localhost:8084/jndi/rmi://localhost:8085/jmxrmi -->
                <JMXRemoteConfiguration>
                        <Enable>false</Enable>
                        <IpAddress>localhost</IpAddress> <!-- set to localhost or internal ip address if behind NAT -->
                        <RMIServerHostName>xxx.xxx.xxx.xxx</RMIServerHostName> <!-- set to external ip address or domain name if behind NAT -->
                        <RMIConnectionPort>8084</RMIConnectionPort>
                        <RMIRegistryPort>8085</RMIRegistryPort>
                        <Authenticate>true</Authenticate>
                        <PasswordFile>${com.wowza.wms.ConfigHome}/conf/jmxremote.password</PasswordFile>
                        <AccessFile>${com.wowza.wms.ConfigHome}/conf/jmxremote.access</AccessFile>
                        <SSLSecure>false</SSLSecure>
                </JMXRemoteConfiguration>
                <UserAgents>Shockwave Flash|CFNetwork|MacNetwork/1.0 (Macintosh)</UserAgents>
                <ServerListeners>
                        <!--
                        <ServerListener>
                                <BaseClass>com.wowza.wms.plugin.loadbalancer.ServerListenerLoadBalancerListener</BaseClass>
                        </ServerListener>
                        -->
                        <!--
                        <ServerListener>
                                <BaseClass>com.wowza.wms.plugin.loadbalancer.ServerListenerLoadBalancerSender</BaseClass>
                        </ServerListener>
                        -->
                </ServerListeners>
                <VHostListeners>
                        <!--
                        <VHostListener>
                                <BaseClass></BaseClass>
                        </VHostListener>
                        -->
                        <VHostListener>
                                <BaseClass>xxx.xxx.wms.common.listeners.ApplicationStartupListener</BaseClass>
                        </VHostListener>
                </VHostListeners> 
                <HandlerThreadPool>
                        <PoolSize>10</PoolSize>
                </HandlerThreadPool>
                <TransportThreadPool>
                        <PoolSize>10</PoolSize>
                </TransportThreadPool>
                <RTP>
                        <DatagramStartingPort>6970</DatagramStartingPort>
                        <DatagramPortSharing>false</DatagramPortSharing>
                </RTP>
                <!-- Properties defined here will be added to the IServer.getProperties() collection -->
                <Properties>
                </Properties>
        </Server>
</Root>

[rrlanham]

A Wowza production server has to be tuned:
http://www.wowzamedia.com/forums/con...ormance-Tuning

Monitor with JConsole, because Wowza/Java reserves ram with java heap size (see tuning guide).

http://www.wowzamedia.com/forums/con...tart-guide#jmx

Richard

[Srx291]

All settings are tunned but java goes to 1200% and load +10.00 with only ~1500 viewers.
Ram is enough (24gb total, 6gb free now with 10.00 load)

Will check with jconsole.

[Srx291]

I haven't found any solution..
This is a wowza dedicated server with enough cpu power and ram, tuned with performance tuning settings, only running wowza, but after some hours of use always goes overload:

top - 17:12:39 up 1 day, 18:19, 1 user, load average: 23.85, 24.98, 25.00
Tasks: 152 total, 1 running, 151 sleeping, 0 stopped, 0 zombie
Cpu0 : 82.2%us, 4.4%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 13.3%si, 0.0%st
Cpu1 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu2 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu3 : 97.7%us, 2.3%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu4 : 88.9%us, 11.1%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu5 : 93.3%us, 6.7%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu6 : 95.7%us, 4.3%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu7 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 24728664k total, 7130528k used, 17598136k free, 210056k buffers
Swap: 525308k total, 0k used, 525308k free, 4603396k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
19979 root 20 0 13.2g 1.2g 9.9m S 796.9 5.0 2928:27 java
3531 root 20 0 12804 1056 740 R 2.2 0.0 0:01.10 top
1 root 20 0 10412 696 576 S 0.0 0.0 0:02.69 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:01.46 ksoftirqd/0