Wowza Community

CentOS Tuning Recommendations

Hi everyone,

Been using Wowza for a few years now, and finally seem to have the golden setup. So through my persperation, would like to share what I have learned off the forum, and through trial & testing in a live environment pushing multiple gigabits over the internet.

So here is my current setup & tips. I have been using FreeBSD since 1996, but in the last year have moved to CentOS due to updated drivers and hardware support.

(I love both)

Also, common sense would say to make each of these changes one at a time if you can. So if something goes wrong, you know which thing is responsible :slight_smile: (Especially when your doing the threadpool tweaks.)

1st) Had some buffering issues, when there really shouldn’t be. As posted in another thread here, the Buffer settings are weak. After I changed this, I was personally able to view > 2Mbit streams without a problem.

@ VHost.xml for your VHosts, bump it to 120k

120000

120000

2nd) Another tip I got off this forum – For those running multiple VHosts, you can make it share against the main Thread Pool by setting the VHosts to 0, and boosting the main one.

So under WowzaMediaServer/conf/VHost.xml (Main)

1200

800

And then for each of your VHost’s in VHost.xml

0

0

So once again, a setting of 0 will cause it to piggy back off the main pool.

3rd) Upgrade your NIC drivers & Network tuning

This may not do much for you, but upgrading NIC drivers can help things. Especially if your on 10Gbit w/ ixgbe.

Also, NIC tuning recommendations from here:

http://fasterdata.es.net/host-tuning/linux/

There are also FreeBSD and other OS’ linked on the left side there.

I have updated all mine using the yum repo – elrepo – http://elrepo.org/tiki/tiki-index.php

Find your driver & confirm your hardware using their recommended steps, or you will find yourself without internet after a reboot :slight_smile:

http://elrepo.org/tiki/FAQ #4

And also confirm with their DeviceIDs page.

4th) Java optimization. LargePages aka HugePages. This gave me a HUGE improvement in stability and performance after I put this guy into production.

I couldn’t get it going on FreeBSD, but CentOS was no problem.

By the way, I am running Java 7u7 (jdk) you can get from Oracle’s download page as an rpm.

in sysctl.conf, depending on how much memory you have.

You can echo it in – echo 1536 > /proc/sys/vm/nr_hugepages

But its recommended you do it at startup (via the sysctl.conf setting), otherwise memory is fragmented, atleast thats what I’m told.

On a server with 4GB of ram, probably 1536 is more than enough. 8GB can go with 2048 or so. And 16+ 3072.

sysctl.conf

Huge Pages

vm.nr_hugepages = 2048

Now make Wowza/Java use these.

in Wowza’s setenv.sh for my 10Gbit server (which was able to handle 2,3Gbit peak traffic this past weekend & just 1 complaint )

JAVA_OPTS="-Xmx6064M"

Uncomment to run server environment (faster), Note: will only work if server VM install, comes with JDL

JAVA_OPTS="$JAVA_OPTS -server"

Can be a better GC setting to avoid long pauses

JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseLargePages -XX:+UseParNewGC -XX:NewSize=512m"

So basically added the UseLargePages to the one normally commented in setenv.sh --^

You can view it in action by doing the following command. (Both servers doing ~100Mbit of traffic at time of writing).

cat /proc/meminfo | grep Huge

AnonHugePages: 475136 kB

HugePages_Total: 2048

HugePages_Free: 631

HugePages_Rsvd: 26

HugePages_Surp: 0

Hugepagesize: 2048 kB

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

2526 root 20 0 15.1g 999m 5948 S 77.3 12.7 9056:28 java

and another server

$ cat /proc/meminfo | grep Huge

AnonHugePages: 280576 kB

HugePages_Total: 3072

HugePages_Free: 2289

HugePages_Rsvd: 25

HugePages_Surp: 0

Hugepagesize: 2048 kB

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

25563 root 20 0 12.2g 563m 10m S 62.8 7.2 2504:20 java

The result of this is no huge memory build up when you view the process in ‘top’, as java is using the pre-allocated memory you set above.

Before this, java/wowza would balloon to 3-4GB over a few days. Now relatively stable with LargePages.

I would also recommend you install ‘htop’ to get a different view of things as well. (rpmforge repo)

That’s all I have for now, happy streaming to all !

Great info Matador. Just for completeness, can you tell us which version of CentOS you’re running?

Great info Matador. Just for completeness, can you tell us which version of CentOS you’re running?

Of course the latest 6.3 64-bit.