Results 1 to 4 of 4

Thread: Different load average in the same configuration

  1. #1

    Default Different load average in the same configuration

    I am monitoring 5 Wowza servers, each has the same Configuration

    2 Processors @ 2.13GHZ
    24G of RAM
    CentOS 5.7 64 bit
    Java JDK 7.0.1


    Each server is having 1600 connections live streaming (not VOD)

    Among them, only 1 server (Wowza 3.0.1) seems to keep the load average at the lowest all the time (around 0.5) . The rest of the servers (mixed 3.0.1 and 3.0.3) have high load average and fluctuate, sometimes go up to 6 or 7, and this makes the streaming buffering and slow.

    I tuned up all of the server exactly the same and now I changed the heapsize to different values, but It still won't help. The good server is always good. The rest are always bad or up and down. Here is my Vhost.xml

    <Root>
    <VHost>
    <HostPortList>
    <HostPort>
    <ProcessorCount>16</ProcessorCount>
    <IpAddress>*</IpAddress>
    <!-- Separate multiple ports with commas -->
    <!-- 80: HTTP, RTMPT -->
    <!-- 554: RTSP -->
    <Port>1935</Port>
    <SocketConfiguration>
    <ReuseAddress>true</ReuseAddress>
    <!-- suggested settings for video on demand applications -->
    <ReceiveBufferSize>24000</ReceiveBufferSize>
    <SendBufferSize>65000</SendBufferSize>
    <!-- suggest settings for low latency chat and video recording applications
    <ReceiveBufferSize>16000</ReceiveBufferSize>
    <SendBufferSize>16000</SendBufferSize>
    -->
    <KeepAlive>true</KeepAlive>
    <!-- <TrafficClass>0</TrafficClass> -->
    <!-- <OobInline>false</OobInline> -->
    <!-- <SoLingerTime>-1</SoLingerTime> -->
    <!-- <TcpNoDelay>false</TcpNoDelay> -->
    <AcceptorBackLog>100</AcceptorBackLog>
    </SocketConfiguration>
    <HTTPStreamerAdapterIDs>cupertinostreaming,smoothstreaming,sanjosestreaming,dvrchunkstreaming</HTTPStreamerAdapterIDs>
    <HTTPProviders>
    <HTTPProvider>
    <BaseClass>com.wowza.wms.http.HTTPCrossdomain</BaseClass>
    <RequestFilters>*crossdomain.xml</RequestFilters>
    <AuthenticationMethod>none</AuthenticationMethod>
    </HTTPProvider>
    <HTTPProvider>
    <BaseClass>com.wowza.wms.http.HTTPClientAccessPolicy</BaseClass>
    <RequestFilters>*clientaccesspolicy.xml</RequestFilters>
    <AuthenticationMethod>none</AuthenticationMethod>
    </HTTPProvider>
    <HTTPProvider>
    <BaseClass>com.wowza.wms.http.HTTPProviderMediaList</BaseClass>
    <RequestFilters>*jwplayer.rss|*medialist.smil|*manifest-rtmp.f4m</RequestFilters>
    <AuthenticationMethod>none</AuthenticationMethod>
    </HTTPProvider>
    <HTTPProvider>
    <BaseClass>com.wowza.wms.http.HTTPServerVersion</BaseClass>
    <RequestFilters>*</RequestFilters>
    <AuthenticationMethod>none</AuthenticationMethod>
    </HTTPProvider>
    </HTTPProviders>
    </HostPort>

    <!-- 443 with SSL -->
    <!--
    <HostPort>
    <ProcessorCount>16</ProcessorCount>
    <IpAddress>*</IpAddress>
    <Port>443</Port>
    <SSLConfig>
    <KeyStorePath>${com.wowza.wms.context.VHostConfigHome}/conf/keystore.cert</KeyStorePath>
    <KeyStorePassword>[password]</KeyStorePassword>
    <KeyStoreType>JKS</KeyStoreType>
    <SSLProtocol>TLS</SSLProtocol>
    <Algorithm>SunX509</Algorithm>
    <CipherSuites></CipherSuites>
    <Protocols></Protocols>
    </SSLConfig>
    <SocketConfiguration>
    <ReuseAddress>true</ReuseAddress>
    <ReceiveBufferSize>24000</ReceiveBufferSize>
    <SendBufferSize>65000</SendBufferSize>
    <KeepAlive>true</KeepAlive>
    <AcceptorBackLog>100</AcceptorBackLog>
    </SocketConfiguration>
    <HTTPStreamerAdapterIDs>cupertinostreaming,smoothstreaming,sanjosestreaming,dvrchunkstreaming</HTTPStreamerAdapterIDs>
    <HTTPProviders>
    <HTTPProvider>
    <BaseClass>com.wowza.wms.http.HTTPCrossdomain</BaseClass>
    <RequestFilters>*crossdomain.xml</RequestFilters>
    <AuthenticationMethod>none</AuthenticationMethod>
    </HTTPProvider>
    <HTTPProvider>
    <BaseClass>com.wowza.wms.http.HTTPClientAccessPolicy</BaseClass>
    <RequestFilters>*clientaccesspolicy.xml</RequestFilters>
    <AuthenticationMethod>none</AuthenticationMethod>
    </HTTPProvider>
    <HTTPProvider>
    <BaseClass>com.wowza.wms.http.HTTPProviderMediaList</BaseClass>
    <RequestFilters>*jwplayer.rss|*medialist.smil|*manifest-rtmp.f4m</RequestFilters>
    <AuthenticationMethod>none</AuthenticationMethod>
    </HTTPProvider>
    <HTTPProvider>
    <BaseClass>com.wowza.wms.http.HTTPServerVersion</BaseClass>
    <RequestFilters>*</RequestFilters>
    <AuthenticationMethod>none</AuthenticationMethod>
    </HTTPProvider>
    </HTTPProviders>
    </HostPort>
    -->

    <!-- Admin HostPort -->
    <HostPort>
    <ProcessorCount>1</ProcessorCount>
    <IpAddress>*</IpAddress>
    <Port>8086</Port>
    <SocketConfiguration>
    <ReuseAddress>true</ReuseAddress>
    <ReceiveBufferSize>16000</ReceiveBufferSize>
    <SendBufferSize>16000</SendBufferSize>
    <KeepAlive>true</KeepAlive>
    <AcceptorBackLog>100</AcceptorBackLog>
    </SocketConfiguration>
    <HTTPStreamerAdapterIDs></HTTPStreamerAdapterIDs>
    <HTTPProviders>
    <HTTPProvider>
    <BaseClass>com.wowza.wms.http.streammanager.HTTPStreamManager</BaseClass>
    <RequestFilters>streammanager*</RequestFilters>
    <AuthenticationMethod>admin-digest</AuthenticationMethod>
    </HTTPProvider>
    <HTTPProvider>
    <BaseClass>com.wowza.wms.http.HTTPServerInfoXML</BaseClass>
    <RequestFilters>serverinfo*</RequestFilters>
    <AuthenticationMethod>admin-digest</AuthenticationMethod>
    </HTTPProvider>
    <HTTPProvider>
    <BaseClass>com.wowza.wms.http.HTTPConnectionInfo</BaseClass>
    <RequestFilters>connectioninfo*</RequestFilters>
    <AuthenticationMethod>admin-digest</AuthenticationMethod>
    </HTTPProvider>
    <HTTPProvider>
    <BaseClass>com.wowza.wms.http.HTTPConnectionCountsXML</BaseClass>
    <RequestFilters>connectioncounts*</RequestFilters>
    <AuthenticationMethod>admin-digest</AuthenticationMethod>
    </HTTPProvider>
    <HTTPProvider>
    <BaseClass>com.wowza.wms.transcoder.httpprovider.HTTPTranscoderThumbnail</BaseClass>
    <RequestFilters>transcoderthumbnail*</RequestFilters>
    <AuthenticationMethod>admin-digest</AuthenticationMethod>
    </HTTPProvider>
    <HTTPProvider>
    <BaseClass>com.wowza.wms.http.HTTPProviderMediaList</BaseClass>
    <RequestFilters>medialist*</RequestFilters>
    <AuthenticationMethod>admin-digest</AuthenticationMethod>
    </HTTPProvider>
    <HTTPProvider>
    <BaseClass>com.wowza.wms.http.HTTPServerVersion</BaseClass>
    <RequestFilters>*</RequestFilters>
    <AuthenticationMethod>none</AuthenticationMethod>
    </HTTPProvider>
    </HTTPProviders>
    </HostPort>

    </HostPortList>

    <HTTPStreamerAdapters>
    <HTTPStreamerAdapter>
    <ID>smoothstreaming</ID>
    <Name>smoothstreaming</Name>
    <Properties>
    </Properties>
    </HTTPStreamerAdapter>
    <HTTPStreamerAdapter>
    <ID>cupertinostreaming</ID>
    <Name>cupertinostreaming</Name>
    <Properties>
    </Properties>
    </HTTPStreamerAdapter>
    <HTTPStreamerAdapter>
    <ID>sanjosestreaming</ID>
    <Name>sanjosestreaming</Name>
    <Properties>
    </Properties>
    </HTTPStreamerAdapter>
    <HTTPStreamerAdapter>
    <ID>dvrchunkstreaming</ID>
    <Name>dvrchunkstreaming</Name>
    <Properties>
    </Properties>
    </HTTPStreamerAdapter>
    </HTTPStreamerAdapters>

    <HandlerThreadPool>
    <PoolSize>480</PoolSize>
    </HandlerThreadPool>
    <TransportThreadPool>
    <PoolSize>320</PoolSize>
    </TransportThreadPool>
    <IdleWorkers>
    <WorkerCount>16</WorkerCount>
    <CheckFrequency>100</CheckFrequency>
    <MinimumWaitTime>5</MinimumWaitTime>
    </IdleWorkers>
    <NetConnections>
    <ProcessorCount>16</ProcessorCount>
    <IdleFrequency>500</IdleFrequency>
    <SocketConfiguration>
    <ReuseAddress>true</ReuseAddress>
    <ReceiveBufferSize>65000</ReceiveBufferSize>
    <SendBufferSize>24000</SendBufferSize>
    <KeepAlive>true</KeepAlive>
    <!-- <TrafficClass>0</TrafficClass> -->
    <!-- <OobInline>false</OobInline> -->
    <!-- <SoLingerTime>-1</SoLingerTime> -->
    <!-- <TcpNoDelay>false</TcpNoDelay> -->
    <AcceptorBackLog>100</AcceptorBackLog>
    </SocketConfiguration>
    </NetConnections>
    <HTTPTunnel>
    <KeepAliveTimeout>2000</KeepAliveTimeout>
    </HTTPTunnel>
    <Client>
    <ClientTimeout>90000</ClientTimeout>
    <IdleFrequency>250</IdleFrequency>
    </Client>
    <!-- RTP/Authentication/Methods defined in Authentication.xml. Default setup includes; none, basic, digest -->
    <RTP>
    <IdleFrequency>75</IdleFrequency>
    <DatagramConfiguration>
    <Incoming>
    <ReuseAddress>true</ReuseAddress>
    <ReceiveBufferSize>1024000</ReceiveBufferSize>
    <SendBufferSize>65000</SendBufferSize>
    <!-- <MulticastBindToAddress>true</MulticastBindToAddress> -->
    <!-- <MulticastInterfaceAddress>192.168.1.22</MulticastInterfaceAddress> -->
    <!-- <TrafficClass>0</TrafficClass> -->
    <MulticastTimeout>50</MulticastTimeout>
    <DatagramMaximumPacketSize>4096</DatagramMaximumPacketSize>
    </Incoming>
    <Outgoing>
    <ReuseAddress>true</ReuseAddress>
    <ReceiveBufferSize>65000</ReceiveBufferSize>
    <SendBufferSize>65000</SendBufferSize>
    <!-- <MulticastBindToAddress>true</MulticastBindToAddress> -->
    <!-- <MulticastInterfaceAddress>192.168.1.22</MulticastInterfaceAddress> -->
    <!-- <TrafficClass>0</TrafficClass> -->
    <MulticastTimeout>50</MulticastTimeout>
    <DatagramMaximumPacketSize>4096</DatagramMaximumPacketSize>
    </Outgoing>
    </DatagramConfiguration>
    <UnicastIncoming>
    <ProcessorCount>8</ProcessorCount>
    </UnicastIncoming>
    <UnicastOutgoing>
    <ProcessorCount>16</ProcessorCount>
    </UnicastOutgoing>
    <MulticastIncoming>
    <ProcessorCount>8</ProcessorCount>
    </MulticastIncoming>
    <MulticastOutgoing>
    <ProcessorCount>8</ProcessorCount>
    </MulticastOutgoing>
    </RTP>
    <Application>
    <ApplicationTimeout>60000</ApplicationTimeout>
    <PingTimeout>12000</PingTimeout>
    <ValidationFrequency>20000</ValidationFrequency>
    <MaximumPendingWriteBytes>0</MaximumPendingWriteBytes>
    <MaximumSetBufferTime>60000</MaximumSetBufferTime>
    </Application>
    <StartStartupStreams>true</StartStartupStreams>
    <!-- Properties defined here will be added to the IVHost.getProperties() collection -->
    <Properties>
    </Properties>
    </VHost>
    </Root>




    Originally I thought it could be java problem, so I used the same java 7.0.1 for testing, but it won't help. Then I downgraded Wowza from 3.0.3 to 3.0.1 but it won't help too much. Is there a way to find out why Wowza is taking too much CPU for those servers?

    Thanks

  2. #2

    Default

    Hi Robert,

    I can't identify anything from when you've shown so far. You'll have to do some analysis to see what the problem is.

    Some thoughts:

    1. It is not recommended to set a Java heap size greater than 10GB. You should lower this.
    2. The goal is to figure out what is different about the two servers. This is likely a system administration issue rather than a Wowza specific issue. But, to investigate exactly what Wowza is doing, you can use jconsole, with the "TopThreads" plugin.
    3. If the good server is the one running 15.4% total CPU, I see that Java has allocated 20GB. It appears you have set XMS (initial heap size) in addition to max heap size, unlike the other 3 servers. Possibly, because of this, Java has not yet started any garbage collection, and so Java is running fast.

  3. #3

    Default

    Thanks for the reply.

    The good server is indeed the server that has 15.4% CPU. I am not sure about the XMS. Here is the code in the sentev.sh

    #!/bin/sh

    _EXECJAVA=java
    JAVA_OPTS="-Xmx20000M"

    # Uncomment to run server environment (faster), Note: will only work if server VM install, comes with JDL
    JAVA_OPTS="$JAVA_OPTS -server"

    # Can be a better GC setting to avoid long pauses
    JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC"

    # Uncomment to fix multicast crosstalk problem when streams share multicast port
    JAVA_OPTS="$JAVA_OPTS -Djava.net.preferIPv4Stack=true"

    WMSAPP_HOME=/usr/local/WowzaMediaServer
    WMSCONFIG_HOME=/usr/local/WowzaMediaServer
    WMSCONFIG_URL=

    export WMSAPP_HOME WMSCONFIG_HOME JAVA_OPTS _EXECJAVA
    and startup.sh
    #!/bin/sh
    . ./setenv.sh

    #chmod 600 ../conf/jmxremote.password
    #chmod 600 ../conf/jmxremote.access

    # NOTE: Here you can configure the JVM's built in JMX interface.
    # See the "Server Management Console and Monitoring" chapter
    # of the "User's Guide" for more information on how to configure the
    # remote JMX interface in the [install-dir]/conf/Server.xml file.

    JMXOPTIONS=-Dcom.sun.management.jmxremote=true
    #JMXOPTIONS="$JMXOPTIONS -Djava.rmi.server.hostname=192.168.1.7"
    #JMXOPTIONS="$JMXOPTIONS -Dcom.sun.management.jmxremote.port=1099"
    #JMXOPTIONS="$JMXOPTIONS -Dcom.sun.management.jmxremote.authenticate=true"
    #JMXOPTIONS="$JMXOPTIONS -Dcom.sun.management.jmxremote.ssl=false"
    #JMXOPTIONS="$JMXOPTIONS -Dcom.sun.management.jmxremote.password.file=$WMSCONFIG_HOME/conf/jmxremote.password"
    #JMXOPTIONS="$JMXOPTIONS -Dcom.sun.management.jmxremote.access.file=$WMSCONFIG_HOME/conf/jmxremote.access"

    ulimit -n 20000 > /dev/null 2>&1

    # log interceptor com.wowza.wms.logging.LogNotify - see Javadocs for ILogNotify

    $_EXECJAVA $JAVA_OPTS $JMXOPTIONS -Dcom.wowza.wms.runmode="standalone" -Dcom.wowza.wms.native.base="linux" -Dcom.wowza.wms.AppHome="$WMSAPP_HOME" -Dcom.wowza.wms.ConfigURL="$WMSCONFIG_URL" -Dcom.wowza.wms.ConfigHome="$WMSCONFIG_HOME" -cp $WMSAPP_HOME/bin/wms-bootstrap.jar com.wowza.wms.bootstrap.Bootstrap start

    Can you please tell me where to check the XMS? Also if we comment out this code in sentenv.sh
    # Can be a better GC setting to avoid long pauses
    JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC"
    will it lower CPU usage?

    My point is to have as many connections as possible so I am looking to lower the CPU to the lowest possible point. High latency is acceptable to me.

    Thanks

  4. #4

    Default

    "Can you please tell me where to check the XMS?"

    -I don't see the -Xms setting in those files.

    "Also if we comment out this code in sentenv.sh will it lower CPU usage?"

    I'm note sure. There are different Garbage collection strategies. If you're running a large production server using Java, it is necessary to fully understand these concepts. You can try the suggestions I listed before to see why memory is being used. You can experiment with Ben's suggestions to see if they help.

    Be sure to follow the tuning guide.

Similar Threads

  1. Load balancing configuration
    By steveitl in forum Wowza Streaming Engine functionality
    Replies: 2
    Last Post: 07-29-2014, 10:44 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •