Different load average in the same configuration

**robertookla** · 06-16-2012, 01:36 PM

I am monitoring 5 Wowza servers, each has the same Configuration

2 Processors @ 2.13GHZ
24G of RAM
CentOS 5.7 64 bit
Java JDK 7.0.1

Each server is having 1600 connections live streaming (not VOD)

Among them, only 1 server (Wowza 3.0.1) seems to keep the load average at the lowest all the time (around 0.5) . The rest of the servers (mixed 3.0.1 and 3.0.3) have high load average and fluctuate, sometimes go up to 6 or 7, and this makes the streaming buffering and slow.

I tuned up all of the server exactly the same and now I changed the heapsize to different values, but It still won't help. The good server is always good. The rest are always bad or up and down. Here is my Vhost.xml

<Root>
<VHost>
<HostPortList>
<HostPort>
<ProcessorCount>16</ProcessorCount>
<IpAddress>*</IpAddress>



<Port>1935</Port>
<SocketConfiguration>
<ReuseAddress>true</ReuseAddress>

<ReceiveBufferSize>24000</ReceiveBufferSize>
<SendBufferSize>65000</SendBufferSize>

<KeepAlive>true</KeepAlive>




<AcceptorBackLog>100</AcceptorBackLog>
</SocketConfiguration>
<HTTPStreamerAdapterIDs>cupertinostreaming,smooths treaming,sanjosestreaming,dvrchunkstreaming</HTTPStreamerAdapterIDs>
<HTTPProviders>
<HTTPProvider>
<BaseClass>com.wowza.wms.http.HTTPCrossdomain</BaseClass>
<RequestFilters>*crossdomain.xml</RequestFilters>
<AuthenticationMethod>none</AuthenticationMethod>
</HTTPProvider>
<HTTPProvider>
<BaseClass>com.wowza.wms.http.HTTPClientAccessPoli cy</BaseClass>
<RequestFilters>*clientaccesspolicy.xml</RequestFilters>
<AuthenticationMethod>none</AuthenticationMethod>
</HTTPProvider>
<HTTPProvider>
<BaseClass>com.wowza.wms.http.HTTPProviderMediaLis t</BaseClass>
<RequestFilters>*jwplayer.rss|*medialist.smil|*man ifest-rtmp.f4m</RequestFilters>
<AuthenticationMethod>none</AuthenticationMethod>
</HTTPProvider>
<HTTPProvider>
<BaseClass>com.wowza.wms.http.HTTPServerVersion</BaseClass>
<RequestFilters>*</RequestFilters>
<AuthenticationMethod>none</AuthenticationMethod>
</HTTPProvider>
</HTTPProviders>
</HostPort>





<HostPort>
<ProcessorCount>1</ProcessorCount>
<IpAddress>*</IpAddress>
<Port>8086</Port>
<SocketConfiguration>
<ReuseAddress>true</ReuseAddress>
<ReceiveBufferSize>16000</ReceiveBufferSize>
<SendBufferSize>16000</SendBufferSize>
<KeepAlive>true</KeepAlive>
<AcceptorBackLog>100</AcceptorBackLog>
</SocketConfiguration>
<HTTPStreamerAdapterIDs></HTTPStreamerAdapterIDs>
<HTTPProviders>
<HTTPProvider>
<BaseClass>com.wowza.wms.http.streammanager.HTTPSt reamManager</BaseClass>
<RequestFilters>streammanager*</RequestFilters>
<AuthenticationMethod>admin-digest</AuthenticationMethod>
</HTTPProvider>
<HTTPProvider>
<BaseClass>com.wowza.wms.http.HTTPServerInfoXML</BaseClass>
<RequestFilters>serverinfo*</RequestFilters>
<AuthenticationMethod>admin-digest</AuthenticationMethod>
</HTTPProvider>
<HTTPProvider>
<BaseClass>com.wowza.wms.http.HTTPConnectionInfo </BaseClass>
<RequestFilters>connectioninfo*</RequestFilters>
<AuthenticationMethod>admin-digest</AuthenticationMethod>
</HTTPProvider>
<HTTPProvider>
<BaseClass>com.wowza.wms.http.HTTPConnectionCounts XML</BaseClass>
<RequestFilters>connectioncounts*</RequestFilters>
<AuthenticationMethod>admin-digest</AuthenticationMethod>
</HTTPProvider>
<HTTPProvider>
<BaseClass>com.wowza.wms.transcoder.httpprovider.H TTPTranscoderThumbnail</BaseClass>
<RequestFilters>transcoderthumbnail*</RequestFilters>
<AuthenticationMethod>admin-digest</AuthenticationMethod>
</HTTPProvider>
<HTTPProvider>
<BaseClass>com.wowza.wms.http.HTTPProviderMediaLis t</BaseClass>
<RequestFilters>medialist*</RequestFilters>
<AuthenticationMethod>admin-digest</AuthenticationMethod>
</HTTPProvider>
<HTTPProvider>
<BaseClass>com.wowza.wms.http.HTTPServerVersion</BaseClass>
<RequestFilters>*</RequestFilters>
<AuthenticationMethod>none</AuthenticationMethod>
</HTTPProvider>
</HTTPProviders>
</HostPort>

</HostPortList>

<HTTPStreamerAdapters>
<HTTPStreamerAdapter>
<ID>smoothstreaming</ID>
<Name>smoothstreaming</Name>
<Properties>
</Properties>
</HTTPStreamerAdapter>
<HTTPStreamerAdapter>
<ID>cupertinostreaming</ID>
<Name>cupertinostreaming</Name>
<Properties>
</Properties>
</HTTPStreamerAdapter>
<HTTPStreamerAdapter>
<ID>sanjosestreaming</ID>
<Name>sanjosestreaming</Name>
<Properties>
</Properties>
</HTTPStreamerAdapter>
<HTTPStreamerAdapter>
<ID>dvrchunkstreaming</ID>
<Name>dvrchunkstreaming</Name>
<Properties>
</Properties>
</HTTPStreamerAdapter>
</HTTPStreamerAdapters>

<HandlerThreadPool>
<PoolSize>480</PoolSize>
</HandlerThreadPool>
<TransportThreadPool>
<PoolSize>320</PoolSize>
</TransportThreadPool>
<IdleWorkers>
<WorkerCount>16</WorkerCount>
<CheckFrequency>100</CheckFrequency>
<MinimumWaitTime>5</MinimumWaitTime>
</IdleWorkers>
<NetConnections>
<ProcessorCount>16</ProcessorCount>
<IdleFrequency>500</IdleFrequency>
<SocketConfiguration>
<ReuseAddress>true</ReuseAddress>
<ReceiveBufferSize>65000</ReceiveBufferSize>
<SendBufferSize>24000</SendBufferSize>
<KeepAlive>true</KeepAlive>




<AcceptorBackLog>100</AcceptorBackLog>
</SocketConfiguration>
</NetConnections>
<HTTPTunnel>
<KeepAliveTimeout>2000</KeepAliveTimeout>
</HTTPTunnel>
<Client>
<ClientTimeout>90000</ClientTimeout>
<IdleFrequency>250</IdleFrequency>
</Client>

<RTP>
<IdleFrequency>75</IdleFrequency>
<DatagramConfiguration>
<Incoming>
<ReuseAddress>true</ReuseAddress>
<ReceiveBufferSize>1024000</ReceiveBufferSize>
<SendBufferSize>65000</SendBufferSize>



<MulticastTimeout>50</MulticastTimeout>
<DatagramMaximumPacketSize>4096</DatagramMaximumPacketSize>
</Incoming>
<Outgoing>
<ReuseAddress>true</ReuseAddress>
<ReceiveBufferSize>65000</ReceiveBufferSize>
<SendBufferSize>65000</SendBufferSize>



<MulticastTimeout>50</MulticastTimeout>
<DatagramMaximumPacketSize>4096</DatagramMaximumPacketSize>
</Outgoing>
</DatagramConfiguration>
<UnicastIncoming>
<ProcessorCount>8</ProcessorCount>
</UnicastIncoming>
<UnicastOutgoing>
<ProcessorCount>16</ProcessorCount>
</UnicastOutgoing>
<MulticastIncoming>
<ProcessorCount>8</ProcessorCount>
</MulticastIncoming>
<MulticastOutgoing>
<ProcessorCount>8</ProcessorCount>
</MulticastOutgoing>
</RTP>
<Application>
<ApplicationTimeout>60000</ApplicationTimeout>
<PingTimeout>12000</PingTimeout>
<ValidationFrequency>20000</ValidationFrequency>
<MaximumPendingWriteBytes>0</MaximumPendingWriteBytes>
<MaximumSetBufferTime>60000</MaximumSetBufferTime>
</Application>
<StartStartupStreams>true</StartStartupStreams>

<Properties>
</Properties>
</VHost>
</Root>

Originally I thought it could be java problem, so I used the same java 7.0.1 for testing, but it won't help. Then I downgraded Wowza from 3.0.3 to 3.0.1 but it won't help too much. Is there a way to find out why Wowza is taking too much CPU for those servers?

Thanks

**randall** · 06-16-2012, 11:47 PM

Hi Robert,

I can't identify anything from when you've shown so far. You'll have to do some analysis to see what the problem is.

Some thoughts:

1. It is not recommended to set a Java heap size greater than 10GB. You should lower this.
2. The goal is to figure out what is different about the two servers. This is likely a system administration issue rather than a Wowza specific issue. But, to investigate exactly what Wowza is doing, you can use jconsole, with the "TopThreads" plugin.
3. If the good server is the one running 15.4% total CPU, I see that Java has allocated 20GB. It appears you have set XMS (initial heap size) in addition to max heap size, unlike the other 3 servers. Possibly, because of this, Java has not yet started any garbage collection, and so Java is running fast.

**robertookla** · 06-17-2012, 12:10 AM

Thanks for the reply.

The good server is indeed the server that has 15.4% CPU. I am not sure about the XMS. Here is the code in the sentev.sh

#!/bin/sh

_EXECJAVA=java
JAVA_OPTS="-Xmx20000M"

# Uncomment to run server environment (faster), Note: will only work if server VM install, comes with JDL
JAVA_OPTS="$JAVA_OPTS -server"

# Can be a better GC setting to avoid long pauses
JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC"

# Uncomment to fix multicast crosstalk problem when streams share multicast port
JAVA_OPTS="$JAVA_OPTS -Djava.net.preferIPv4Stack=true"

WMSAPP_HOME=/usr/local/WowzaMediaServer
WMSCONFIG_HOME=/usr/local/WowzaMediaServer
WMSCONFIG_URL=

export WMSAPP_HOME WMSCONFIG_HOME JAVA_OPTS _EXECJAVA

and startup.sh

#!/bin/sh
. ./setenv.sh

#chmod 600 ../conf/jmxremote.password
#chmod 600 ../conf/jmxremote.access

# NOTE: Here you can configure the JVM's built in JMX interface.
# See the "Server Management Console and Monitoring" chapter
# of the "User's Guide" for more information on how to configure the
# remote JMX interface in the [install-dir]/conf/Server.xml file.

JMXOPTIONS=-Dcom.sun.management.jmxremote=true
#JMXOPTIONS="$JMXOPTIONS -Djava.rmi.server.hostname=192.168.1.7"
#JMXOPTIONS="$JMXOPTIONS -Dcom.sun.management.jmxremote.port=1099"
#JMXOPTIONS="$JMXOPTIONS -Dcom.sun.management.jmxremote.authenticate=true"
#JMXOPTIONS="$JMXOPTIONS -Dcom.sun.management.jmxremote.ssl=false"
#JMXOPTIONS="$JMXOPTIONS -Dcom.sun.management.jmxremote.password.file=$WMSCO NFIG_HOME/conf/jmxremote.password"
#JMXOPTIONS="$JMXOPTIONS -Dcom.sun.management.jmxremote.access.file=$WMSCONF IG_HOME/conf/jmxremote.access"

ulimit -n 20000 > /dev/null 2>&1

# log interceptor com.wowza.wms.logging.LogNotify - see Javadocs for ILogNotify

$_EXECJAVA $JAVA_OPTS $JMXOPTIONS -Dcom.wowza.wms.runmode="standalone" -Dcom.wowza.wms.native.base="linux" -Dcom.wowza.wms.AppHome="$WMSAPP_HOME" -Dcom.wowza.wms.ConfigURL="$WMSCONFIG_URL" -Dcom.wowza.wms.ConfigHome="$WMSCONFIG_HOME" -cp $WMSAPP_HOME/bin/wms-bootstrap.jar com.wowza.wms.bootstrap.Bootstrap start

Can you please tell me where to check the XMS? Also if we comment out this code in sentenv.sh

# Can be a better GC setting to avoid long pauses
JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC"

will it lower CPU usage?

My point is to have as many connections as possible so I am looking to lower the CPU to the lowest possible point. High latency is acceptable to me.

Thanks

**randall** · 06-17-2012, 01:29 AM

"Can you please tell me where to check the XMS?"

-I don't see the -Xms setting in those files.

"Also if we comment out this code in sentenv.sh will it lower CPU usage?"

I'm note sure. There are different Garbage collection strategies. If you're running a large production server using Java, it is necessary to fully understand these concepts. You can try the suggestions I listed before to see why memory is being used. You can experiment with Ben's suggestions to see if they help.

Be sure to follow the tuning guide.

Thread: Different load average in the same configuration

Thread Tools

Display

Different load average in the same configuration

Posting Permissions