Iím currently tuning performance of wowza 4.0.3, with following hardware and test situation.
2*Intel(R) Xeon(R) CPU E5-2650 0 @ 2.60GHz(total 32 logic processors)
Only one vod file, 5Mbps, about 5 minutes, codec:H264, profile:High, level:3.1, frameSize:1280x720, displaySize:1280x720, frameRate:23.980000
Clients use HLS to access the VOD video file, hafe from NIC 1, and hafe from the other NIC 2, so the NIC is not the bottleneck.
Change cpu mode to "performance" by command "for CPUFREQ in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor; do [ -f $CPUFREQ ] || continue; echo -n performance > $CPUFREQ; done"
We have tested some case as following:
1、When we use all the 32 cpu cores, wowza only can support about 1600 connections, if we increase more connections, clients will get timeout, that is to say, clients can not download ts file in 10 seconds (each ts file is 10 second), the phenomenon is the same as "http://www.wowza.com/forums/showthread.php?37266-DirectRandomAccessReader-read()-and-seek()-generate-performance-bottleneck", VisualVM shows that DirectRandomAccessReader.read() and seek() used the most cpu time;
2、When we use only 16 cpu cores, wowza can support about 1700 connectins, and when increase more connections, clients will get timeout, but different from above, VisualVM shows that nio.SocketIoProcessor$Worker.run() used the most cpu time. We looked into the GC log, and found that with 1700 connections, each Eden Area GC left object need about 50M-100M to store in survivor area (by gc log's "age 1" keyword), but with 1800 connections, each Eden Area GC left much more objects, need 500M-2G memory in survivor area.
3、When we use only 8 cpu cores, wowza can support about 1900 connectins, and when increase more connections, clients will get timeout, VisualVM and gc log is similar to 16 cpu cores.
4、Base on the results of case 1 and case 3, we start 2 wowza on this server, each using 8 cpu cores, then first start 1800 connections to the first wowza, the wowza works well, and then we start about 200 connections to the second wowza, the clients connected to the first wowza get timeout, VisualVM and gc log is similar to case 2.
Base from above test cases, something looks like strange:
1、Why with 8 cpu cores can achieve more connections than 16 cpu cores and 32 cpu cores? They have the same momory, same configrations.
2、Why when the connection increase to the limits, Eden area GC left much more alive objects? We are sure that clients create connectons and get ts evenly.
3、In case 4, why the seconds wowza influence the first one? They use different cpu cores.
With 1Mbps vod file to test, wowza can achieve 4500 connections by using 8 cpu cores, so it seem that the configration is not the problem.
Following is one of our startup command, and we using command "taskset" to limit wowza only can use some cpu cores, for example, "taskset 0x0000FFFF java " will limit the java only can use cpu core 0 to 15, can not use cpu core 16 to 31.
java -server -Xms20g -Xmx20g -XX:PermSize=512m -XX:MaxPermSize=512m -XX:NewSize=12g -XX:MaxNewSize=12g -XX:+UseParNewGC -XX:MaxTenuringThreshold=15 -XX:SurvivorRatio=3 -XX:+UnlockDiagnosticVMOptions -XX:ParGCCardsPerStrideChunk=32768 -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -XX:+CMSClassUnloadingEnabled -XX:CMSInitiatingOccupancyFraction=20 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:-OmitStackTraceInFastThrow -Xloggc:/home/lid/cms_gc1935.log -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.wowza.wms.runmode=standalone -Dcom.wowza.wms.native.base=linux -Dcom.wowza.wms.AppHome=/usr/local/WowzaStreamingEngine -Dcom.wowza.wms.ConfigURL= -Dcom.wowza.wms.ConfigHome=/usr/local/WowzaStreamingEngine -cp /usr/local/WowzaStreamingEngine/bin/wms-bootstrap.jar com.wowza.wms.bootstrap.Bootstrap start
We want to achieve 20Gbps one server, and in our opinion, if you can help us to solve above problems, we can achieve our goal. Thanks.