Results 1 to 3 of 3

Thread: CentOS Tuning Recommendations

  1. #1
    Join Date
    Jan 2010
    Posts
    24

    Lightbulb CentOS Tuning Recommendations

    Hi everyone,

    Been using Wowza for a few years now, and finally seem to have the golden setup. So through my persperation, would like to share what I have learned off the forum, and through trial & testing in a live environment pushing multiple gigabits over the internet.

    So here is my current setup & tips. I have been using FreeBSD since 1996, but in the last year have moved to CentOS due to updated drivers and hardware support.
    (I love both)

    Also, common sense would say to make each of these changes one at a time if you can. So if something goes wrong, you know which thing is responsible (Especially when your doing the threadpool tweaks.)

    1st) Had some buffering issues, when there really shouldn't be. As posted in another thread here, the Buffer settings are weak. After I changed this, I was personally able to view > 2Mbit streams without a problem.

    @ VHost.xml for your VHosts, bump it to 120k
    <ReceiveBufferSize>120000</ReceiveBufferSize>
    <SendBufferSize>120000</SendBufferSize>

    2nd) Another tip I got off this forum -- For those running multiple VHosts, you can make it share against the main Thread Pool by setting the VHosts to 0, and boosting the main one.

    So under WowzaMediaServer/conf/VHost.xml (Main)
    <HandlerThreadPool>
    <PoolSize>1200</PoolSize>
    </HandlerThreadPool>
    <TransportThreadPool>
    <PoolSize>800</PoolSize>
    </TransportThreadPool>

    And then for each of your VHost's in VHost.xml
    <HandlerThreadPool>
    <PoolSize>0</PoolSize>
    </HandlerThreadPool>
    <TransportThreadPool>
    <PoolSize>0</PoolSize>

    So once again, a setting of 0 will cause it to piggy back off the main pool.

    3rd) Upgrade your NIC drivers & Network tuning
    This may not do much for you, but upgrading NIC drivers can help things. Especially if your on 10Gbit w/ ixgbe.

    Also, NIC tuning recommendations from here:
    http://fasterdata.es.net/host-tuning/linux/
    There are also FreeBSD and other OS' linked on the left side there.

    I have updated all mine using the yum repo -- elrepo -- http://elrepo.org/tiki/tiki-index.php

    Find your driver & confirm your hardware using their recommended steps, or you will find yourself without internet after a reboot
    http://elrepo.org/tiki/FAQ #4
    And also confirm with their DeviceIDs page.

    4th) Java optimization. LargePages aka HugePages. This gave me a HUGE improvement in stability and performance after I put this guy into production.
    I couldn't get it going on FreeBSD, but CentOS was no problem.

    By the way, I am running Java 7u7 (jdk) you can get from Oracle's download page as an rpm.

    in sysctl.conf, depending on how much memory you have.

    You can echo it in -- echo 1536 > /proc/sys/vm/nr_hugepages
    But its recommended you do it at startup (via the sysctl.conf setting), otherwise memory is fragmented, atleast thats what I'm told.

    On a server with 4GB of ram, probably 1536 is more than enough. 8GB can go with 2048 or so. And 16+ 3072.

    sysctl.conf
    # Huge Pages
    vm.nr_hugepages = 2048

    Now make Wowza/Java use these.
    in Wowza's setenv.sh for my 10Gbit server (which was able to handle 2,3Gbit peak traffic this past weekend & just 1 complaint )

    JAVA_OPTS="-Xmx6064M"
    # Uncomment to run server environment (faster), Note: will only work if server VM install, comes with JDL
    JAVA_OPTS="$JAVA_OPTS -server"
    # Can be a better GC setting to avoid long pauses
    JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseLargePages -XX:+UseParNewGC -XX:NewSize=512m"

    So basically added the UseLargePages to the one normally commented in setenv.sh --^

    You can view it in action by doing the following command. (Both servers doing ~100Mbit of traffic at time of writing).

    # cat /proc/meminfo | grep Huge
    AnonHugePages: 475136 kB
    HugePages_Total: 2048
    HugePages_Free: 631
    HugePages_Rsvd: 26
    HugePages_Surp: 0
    Hugepagesize: 2048 kB
    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    2526 root 20 0 15.1g 999m 5948 S 77.3 12.7 9056:28 java

    and another server
    $ cat /proc/meminfo | grep Huge
    AnonHugePages: 280576 kB
    HugePages_Total: 3072
    HugePages_Free: 2289
    HugePages_Rsvd: 25
    HugePages_Surp: 0
    Hugepagesize: 2048 kB

    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    25563 root 20 0 12.2g 563m 10m S 62.8 7.2 2504:20 java

    The result of this is no huge memory build up when you view the process in 'top', as java is using the pre-allocated memory you set above.
    Before this, java/wowza would balloon to 3-4GB over a few days. Now relatively stable with LargePages.

    I would also recommend you install 'htop' to get a different view of things as well. (rpmforge repo)

    That's all I have for now, happy streaming to all !

  2. #2
    Join Date
    Feb 2012
    Posts
    4

    Default

    Great info Matador. Just for completeness, can you tell us which version of CentOS you're running?

  3. #3
    Join Date
    Jan 2010
    Posts
    24

    Default

    Quote Originally Posted by jland138 View Post
    Great info Matador. Just for completeness, can you tell us which version of CentOS you're running?
    Of course the latest 6.3 64-bit.

Similar Threads

  1. [Rant] SendBufferSize recommendations
    By Mantriur in forum Performance Tuning Discussion
    Replies: 4
    Last Post: 10-08-2012, 11:27 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •