Hi,
Our one of the wowza server was running fine, but now a days it started getting heap memory issues, and cpu utilization was also getting high. Due to this streams are being stopped from streaming and requires a restart.
Earlier wowza was running with about 30 streams , but we enabled pulling from another wowza server (sitting on amazon) for distribution of streams for all of them, after that it started giving memory issues.
Below is the log on wowza o/p.
Configure logging: file:///usr/local/WowzaMediaServer/conf/log4j.properties
INFO server server-start Wowza Media Server 2 Subscription 2.2.4 build27452 -
INFO server comment - Serial number: XXXXX-XXXXX-XXXXX-XXXXX-xxxxx
INFO server comment - Maximum connections: Unlimited
INFO server comment - Hardware Available Processors: 4
INFO server comment - Hardware Physical Memory: 13233MB/16054MB
INFO server comment - Hardware Swap Space: 16386MB/16386MB
INFO server comment - Max File Descriptor Count: 20000
INFO server comment - Open File Descriptor Count: 41
INFO server comment - OS Name: Linux
INFO server comment - OS Version: 2.6.18-194.32.1.el5
INFO server comment - OS Architecture: amd64
INFO server comment - Java Name: Java HotSpot(TM) 64-Bit Server VM
INFO server comment - Java Vendor: Sun Microsystems Inc.
INFO server comment - Java Version: 1.6.0_21
INFO server comment - Java VM Version: 17.0-b16
INFO server comment - Java Spec Version: 1.6
INFO server comment - Java Home: /usr/java/jdk1.6.0_21/jre
INFO server comment - Java Max Heap Size: 3991MB
INFO server comment - Java Architecture: 64
INFO server comment - Java Encoding[file.encoding]: UTF-8
INFO server comment - Java Args[0]: -Xmx4000M
INFO server comment - Java Args[1]: -XX:+UseConcMarkSweepGC
INFO server comment - Java Args[2]: -XX:+CMSIncrementalMode
INFO server comment - Java Args[3]: -XX:+UseParNewGC
INFO server comment - Java Args[4]: -Djava.net.preferIPv4Stack=true
INFO server comment - Java Args[5]: -Dcom.sun.management.jmxremote=true
INFO server comment - Java Args[6]: -Dcom.wowza.wms.AppHome=/usr/local/WowzaMediaServer
INFO server comment - Java Args[7]: -Dcom.wowza.wms.ConfigURL=
INFO server comment - Java Args[8]: -Dcom.wowza.wms.ConfigHome=/usr/local/WowzaMediaServer
INFO server comment - Java GC[0]: ParNew
INFO server comment - Java GC[1]: ConcurrentMarkSweep
INFO server comment - Server threads[h/t]: 10/10
INFO server comment - JMX java.rmi.server.hostname: WOWZA_IP
INFO server comment - JMX bind attempt service:jmx:rmi://WOWZA_IP:8084/jndi/rmi://WOWZA_IP:8085/jmxrmi
INFO server comment - JMX password file: /usr/local/WowzaMediaServer/conf/jmxremote.password
INFO server comment - JMX access file: /usr/local/WowzaMediaServer/conf/jmxremote.access
INFO server comment - JMX bind successful
INFO server comment - CMDInterface now listening: [any]:8083
INFO vhost vhost-start _defaultVHost_ -
[ DELLSERVER_WOWZA_EXPAT root@localhost.localdomain:/usr/local/WowzaMediaServer/WOWZALOGS]
Below is the setenv.sh, I have increased heap memory to 4GB instead of 3Gb, still the issue persists.
#!/bin/sh
_EXECJAVA=java
#JAVA_OPTS="-Xmx3000M"
JAVA_OPTS="-Xmx4000M"
# Uncomment to run server environment (faster), Note: will only work if server VM install, comes with JDL
JAVA_OPTS="$JAVA_OPTS -server"
# Can be a better GC setting to avoid long pauses
JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC"
# Uncomment to fix multicast crosstalk problem when streams share multicast port
JAVA_OPTS="$JAVA_OPTS -Djava.net.preferIPv4Stack=true"
WMSAPP_HOME=/usr/local/WowzaMediaServer
WMSCONFIG_HOME=/usr/local/WowzaMediaServer
WMSCONFIG_URL=
export WMSAPP_HOME WMSCONFIG_HOME JAVA_OPTS _EXECJAVA
wms.sh has below lines,
#!/bin/sh
# NOTE: Here you can configure the JVM's built in JMX interface.
# See the "Server Management Console and Monitoring" chapter
# of the "User's Guide" for more information on how to configure the
# remote JMX interface in the [install-dir]/conf/Server.xml file.
JMXOPTIONS=-Dcom.sun.management.jmxremote=true
#JMXOPTIONS="$JMXOPTIONS -Dcom.sun.management.jmxremote.port=1099"
#JMXOPTIONS="$JMXOPTIONS -Dcom.sun.management.jmxremote.authenticate=true"
#JMXOPTIONS="$JMXOPTIONS -Dcom.sun.management.jmxremote.ssl=false"
#JMXOPTIONS="$JMXOPTIONS -Dcom.sun.management.jmxremote.password.file=$WMSCONFIG_HOME/conf/jmxremote.password"
#JMXOPTIONS="$JMXOPTIONS -Dcom.sun.management.jmxremote.access.file=$WMSCONFIG_HOME/conf/jmxremote.access"
#JMXOPTIONS="$JMXOPTIONS -Djava.rmi.server.hostname=192.168.1.7"
if [ "$WMSCOMMAND" != "start" ]; then
JMXOPTIONS=""
fi
ulimit -n 20000
# log interceptor com.wowza.wms.logging.LogNotify - see Javadocs for ILogNotify
$_EXECJAVA $JAVA_OPTS $JMXOPTIONS -Dcom.wowza.wms.AppHome="$WMSAPP_HOME" -Dcom.wowza.wms.ConfigURL="$WMSCONFIG_URL" -Dcom.wowza.wms.ConfigHome="$WMSCONFIG_HOME" -cp $WMSAPP_HOME/bin/wms-bootstrap.jar com.wowza.wms.bootstrap.Bootstrap $WMSCOMMAND >/dev/null 2>&1 &
if [ "$WMSCOMMAND" = "start" ]; then
echo $! > ${WMSPIDFILE}
wait $!
fi
exit 0
Server is of 16GB, only wowza is running there, does the 4GB is enough for heap (mentioned in sevenv.sh) or it needs to be increased?
Here is the Image of memory usage and cpu utilization
can anybody point out where the issues are present.
Thanks,
Shekar
