Hi everyone,
Been using Wowza for a few years now, and finally seem to have the golden setup. So through my persperation, would like to share what I have learned off the forum, and through trial & testing in a live environment pushing multiple gigabits over the internet.
So here is my current setup & tips. I have been using FreeBSD since 1996, but in the last year have moved to CentOS due to updated drivers and hardware support.
(I love both)
Also, common sense would say to make each of these changes one at a time if you can. So if something goes wrong, you know which thing is responsible (Especially when your doing the threadpool tweaks.)
1st) Had some buffering issues, when there really shouldn’t be. As posted in another thread here, the Buffer settings are weak. After I changed this, I was personally able to view > 2Mbit streams without a problem.
@ VHost.xml for your VHosts, bump it to 120k
120000
120000
2nd) Another tip I got off this forum – For those running multiple VHosts, you can make it share against the main Thread Pool by setting the VHosts to 0, and boosting the main one.
So under WowzaMediaServer/conf/VHost.xml (Main)
1200
800
And then for each of your VHost’s in VHost.xml
0
0
So once again, a setting of 0 will cause it to piggy back off the main pool.
3rd) Upgrade your NIC drivers & Network tuning
This may not do much for you, but upgrading NIC drivers can help things. Especially if your on 10Gbit w/ ixgbe.
Also, NIC tuning recommendations from here:
http://fasterdata.es.net/host-tuning/linux/
There are also FreeBSD and other OS’ linked on the left side there.
I have updated all mine using the yum repo – elrepo – http://elrepo.org/tiki/tiki-index.php
Find your driver & confirm your hardware using their recommended steps, or you will find yourself without internet after a reboot
And also confirm with their DeviceIDs page.
4th) Java optimization. LargePages aka HugePages. This gave me a HUGE improvement in stability and performance after I put this guy into production.
I couldn’t get it going on FreeBSD, but CentOS was no problem.
By the way, I am running Java 7u7 (jdk) you can get from Oracle’s download page as an rpm.
in sysctl.conf, depending on how much memory you have.
You can echo it in – echo 1536 > /proc/sys/vm/nr_hugepages
But its recommended you do it at startup (via the sysctl.conf setting), otherwise memory is fragmented, atleast thats what I’m told.
On a server with 4GB of ram, probably 1536 is more than enough. 8GB can go with 2048 or so. And 16+ 3072.
sysctl.conf
Huge Pages
vm.nr_hugepages = 2048
Now make Wowza/Java use these.
in Wowza’s setenv.sh for my 10Gbit server (which was able to handle 2,3Gbit peak traffic this past weekend & just 1 complaint )
JAVA_OPTS="-Xmx6064M"
Uncomment to run server environment (faster), Note: will only work if server VM install, comes with JDL
JAVA_OPTS="$JAVA_OPTS -server"
Can be a better GC setting to avoid long pauses
JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseLargePages -XX:+UseParNewGC -XX:NewSize=512m"
So basically added the UseLargePages to the one normally commented in setenv.sh --^
You can view it in action by doing the following command. (Both servers doing ~100Mbit of traffic at time of writing).
cat /proc/meminfo | grep Huge
AnonHugePages: 475136 kB
HugePages_Total: 2048
HugePages_Free: 631
HugePages_Rsvd: 26
HugePages_Surp: 0
Hugepagesize: 2048 kB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2526 root 20 0 15.1g 999m 5948 S 77.3 12.7 9056:28 java
and another server
$ cat /proc/meminfo | grep Huge
AnonHugePages: 280576 kB
HugePages_Total: 3072
HugePages_Free: 2289
HugePages_Rsvd: 25
HugePages_Surp: 0
Hugepagesize: 2048 kB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
25563 root 20 0 12.2g 563m 10m S 62.8 7.2 2504:20 java
The result of this is no huge memory build up when you view the process in ‘top’, as java is using the pre-allocated memory you set above.
Before this, java/wowza would balloon to 3-4GB over a few days. Now relatively stable with LargePages.
I would also recommend you install ‘htop’ to get a different view of things as well. (rpmforge repo)
That’s all I have for now, happy streaming to all !