• How to do performance tuning

    This article provides suggestions for tuning Wowza® server software on your production hardware.

    Contents


    Overview
    Tuning the server in Wowza Streaming Engine Manager
    Tuning the server with XML configuration files

    Overview


    You'll achieve best results with Wowza Streaming Engine software if you run the most recent Oracle Java Development Kit (JDK), which you can get from (Java SE Downloads). Your best option is to run a 64–bit OS with the 64–bit Java VM, which enables Java heap sizes greater than 2 GB.

    Notes:
    • For Windows operating systems, verify that the JAVA_HOME environment variable points to the latest Java installation.
    • If you're on a Mac with OS X 10.5.x (Snow Leopard) or later with 64–bit hardware, you can run the Java 6 VM. To change your default Java VM in OS X, open the Java preferences application (/Applications/Utilities/Java/Java Preferences) and drag the Java SE 6 item to the top of the Java Applications list.
    By default, Wowza Streaming Engine automatically tunes the server to a development state with the hardware available. The server can then be tuned to a production level by adjusting the Java settings. Knowing that the server is running to the best of its ability makes Wowza Streaming Engine easy to deploy in a production environment. Advanced users can alter the tuning further by using Wowza Streaming Engine Manager, if required.

    Tuning the server in Wowza Streaming Engine Manager


    Notes:
    • Wowza Streaming Engine™ 4.0 or later is required.
    • Streaming Engine Manager requires authentication, which is the user name and password specified in the [install-dir]/conf/admin.password file.
    By default, Wowza Streaming Engine Manager can be accessed by using the following URL:

    http://[wowza-ip-address]:8088/enginemanager


    After you access Streaming Engine Manager, click the Server tab at the top of the manager, and then select Performance Tuning in the contents pane. The following page shows the hardware available on this server and the Java version and architecture in use.



    Java settings

    This shows the current Java settings, which includes the Java Heap Size setting followed by the memory allocated to Wowza Streaming Engine and the Garbage Collection type, followed by the garbage collection code in use.



    To edit these settings, click Edit. When editing, both Java Heap Size and Java Garbage Collection Settings have three options to choose from. To run the server in a production environment, ensure that you choose Production level for the Java Heap Size and Concurrent collector for the Java Garbage Collection Settings if you're running a dedicated Wowza server, and then click Save. The default for the Java Heap Size is the Development level. If you're using this server to run other services that have high memory consumption, you may want to choose Custom level for the Java Heap Size and enter a specific value here accordingly.

    Restart Wowza Streaming Engine to complete the production level tuning. If you aren't using a dedicated server or you want to manually tune the server by using the Wowza Streaming Engine Manager, see the remaining options described below.

    Note: Leave the remaining Java settings options to Set automatically, which is the default choice.

    Thread pools

    This shows the current Handler Thread Pool Size and Transport Thread Pool Size.



    To edit these settings, click Edit. Leave both of these options as Set automatically. When setting the Handler Thread Pool Size and Transport Thread Pool Size automatically, the following calculations are made:
    Code:
    Handler Thread Pool Size 60x [total-core-count] 
    Transport Thread Pool Size 40x [total-core-count]
    Note: The [total-core-count] for the server is the figure below the Processor Cores in the Performance Tuning section above.

    Media Cache tuning

    The Media Cache Tuning section is only required when using a VOD Edge application. This shows the current Writer Thread Pool, Readahead Thread Pool, Maximum Pending Write Request Size, and Maximum Pending Readahead Request Size.



    Configure all the options in this section to Set automatically. The calculations for Writer Thread Pool and Readahead Thread Pool when configured to set automatically are listed below:
    Code:
    Writer Thread Pool - 2x [total-core-count]
    Readahead Thread Pool - 1x [total-core-count]
    Both Maximum Pending Write Request Size and Maximum Pending Readahead Request Size are calculated by the total memory allocated to Wowza Streaming Engine. The following values are used dependent on the heap size that's shown in the Performance Tuning.

    Code:
    1200M to 3999M heap size has the following values:
    Maximum Pending Write Request Size 160M
    Maximum Pending Readahead Request Size 80M
    
    4000M to 7999M heap size has the following values:
    Maximum Pending Write Request Size 500M
    Maximum Pending Readahead Request Size 250M
    
    8000M or above heap size has the following values:
    Maximum Pending Write Request Size 1000M
    Maximum Pending Readahead Request Size 500M

    Virtual host processors

    This shows the current configuration for this section.





    To edit the settings, click the Edit button above Setting and Values. Configure all these options to Set automatically. When configured to set automatically, the calculations for Net Connections Processor Count, Media Caster Processor Count, and Idle Worker Count are listed below:
    Code:
    Net Connections Processor Count 2x [total-core-count]
    Media Caster Processor Count 2x [total-core-count]
    Idle Worker Count 2x [total-core-count]
    When configured to set automatically, the calculations for Unicast Incoming Processor Count and Unicast Outgoing Processor Count are listed below:
    Code:
    Unicast Incoming Processor Count 2x [total-core-count]
    Unicast Outgoing Processor Count 2x [total-core-count]
    When configured to set automatically, the calculations for Multicast Incoming ProcessorCount and Multicast Outgoing ProcessorCount are listed below:
    Code:
    Multicast Incoming ProcessorCount 2x [total-core-count]
    Multicast Outgoing ProcessorCount 2x [total-core-count]
    The Client Idle Frequency value is the time (in milliseconds) between idle events for Adobe Flash client connections. For basic video on demand streaming, a value of 250 provides the best reliability-to-performance ratio. For live streaming, a value between 125 and 250 is better for low-latency streaming. If you aren't doing low-latency streaming, this idle frequency value can be increased to 500, which reduces the CPU usage and allows more concurrent connections. Values between 1 and 1000 are supported.

    The RTP Idle Frequency value is the time (in milliseconds) between idle events for RTP connections. Values between 1 and 1000 are supported.

    Virtual host ports

    This shows the current open ports followed by the processor count associated with each port.



    To edit the settings, click the Edit button above the listed ports. Configure the Processor Count associated with port 1935 and 8086 to Set automatically. The calculation made for both Processor Count values are listed below:
    Code:
    Port 1935 Processor Count 2x [total-core-count]
    Port 8086 Processor Count 2x [total-core-count]

    Virtual host threads

    The Virtual Host Threads are required when running multiple Virtual Hosts on the server.



    If you're running multiple Virtual Hosts, configure the Handler Thread Pool Size and Transport Thread Pool Size to Use server setting. When configured to Use server setting, the Handler and Transport threads are divided equally between all of the active Virtual Hosts that are running on the server.

    Tuning the server with XML configuration files


    Notes:
    • Applies to all Wowza server software versions. If you tuned the server in Wowza Streaming Engine Manager, some of the steps may not be needed.

    • After editing any of the configuration files described in this section, you must restart the server before the changes take effect.
    1. It's best to run the -server version of the Java VM. The -server option is enabled on Linux and Mac OS X by default. On Windows, this option isn't enabled by default. To enable the -server version of the Java VM on Windows, do the following:

      1. Install the latest version of the Oracle Java JDK, and verify that the JAVA_HOME environment variable is pointing to this Java installation.

      2. Edit the [install-dir]/bin/setenv.bat file to add the -server option to the startup scripts. For example:
        Code:
        set JAVA_OPTS=%JAVA_OPTS% -server
        Note: Affects both Windows Standalone and Windows Service.
    2. The default Java heap size for Windows is set to 768MB, and for Linux is set to 1200MB. These values are optimized for development, but they aren't large enough for high-volume production use. If you're running the 64–bit version of the Java VM and have 4GB or more of RAM in your machine, set your Java heap size between 3000 MB to 5000 MB. If you have 16 GB of RAM (or more), set your heap size to 8000 MB. Don't set your heap size above 10 GB, as this can lead to long garbage collection cycles/pauses. If you're running the 32–bit version of the Java VM or have less than 3GB of RAM in your machine, set your Java heap size to 1500MB. Don't try to set a heap size greater than 1800MB when you're using the 32–bit VM. On many Linux platforms, the server starts but fails with an OutOfMemory error after a short period of time. Memory settings can be adjusted in the following script files (based on platform):

      • [install-dir]/bin/setenv.sh (Linux and Mac OS X, around line 4)
        Code:
        JAVA_OPTS="-Xmx3000M"
      • [install-dir]/bin/setenv.bat (Windows Standalone and Windows Service, around line 4)
        Code:
        set JAVA_OPTS=-Xmx3000M
    3. Garbage collection (GC) tuning in Java is tricky. What works best in one server setup may not work in another. Through trial-and-error and customer feedback, we've developed several suggested approaches:

      Garbage Collection Options:

      We strongly recommend No Additional Settings. The default -server tuning (see step #2 above) works well for many streaming situations, so in most cases this tuning works great without additional settings.

      Only if you're experiencing problems with the Java Heap, such as a build-up of memory usage, should you try implementing one of the following Garbage Collection options:

      1. Concurrent collector: The concurrent collector uses a Java garbage collector that's designed for applications that prefer shorter GC pauses and that can share processor resources with the garbage collector while the application is running. The suggested additional settings are:

        If heap size is 5000 MB or greater, NewSize=512m

        If heap size is 3000 MB - 5000 MB, NewSize=256m

        If heap size is less than 3000 MB, NewSize=128m

        For example, with 64–bit OS, 8 GB RAM, and heap size of 6000 MB:
        Code:
        -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:NewSize=512m
      2. G1 (Garbage-First) collector: The G1 collector uses a Java garbage collector that's designed for low pause time, high-throughput applications. The G1 collector is a server-style garbage collector, targeted for multi-processor machines with large memories and is fully supported in Oracle JDK 7 Update 4 and later releases. The suggested additional settings are:
        Code:
        -XX:+UseG1GC -XX:MaxGCPauseMillis=100

      To change the garbage collection settings, modify the following scripts:

      • [install-dir]/bin/setenv.sh (Linux and Mac OS X, uncomment line 10). For example:
        Code:
        JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:NewSize=512m"
      • [install-dir]/bin/setenv.bat (Windows Standalone and Windows Service, uncomment line 10). For example:
        Code:
        set JAVA_OPTS=%JAVA_OPTS% -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:NewSize=512m
      • [install-dir]/bin/setenv.sh to monitor GC pause times for Linux, add:
        Code:
        -verbose:gc -Xloggc:"/usr/local/WowzaMediaServer/logs/gc_%UNIQUE%.log" -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC
      • [install-dir]/bin/setenv.sh to monitor GC pause times for Mac OS X, add:
        Code:
        -verbose:gc -Xloggc:"/Library/WowzaMediaServer/logs/gc_%UNIQUE%.log" -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC
      • [install-dir]/bin/setenv.bat to monitor GC pause times for Windows, add:
        Code:
        set JAVA_OPTS=%JAVA_OPTS% -verbose:gc -Xloggc:"c:\gc.log" -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime
        Note: Windows: This creates a file called gc.log in the C drive for debugging purposes and shouldn't be left running.
        Windows: The file C:\gc.log is only created when starting the Wowza server in service mode or if using the command line.

      There's more detailed information in the following articles:


    4. For low-latency chat applications, it's best to use smaller socket buffer sizes (16000 bytes for read and write). The socket buffer sizes are configured in [install-dir]/conf/VHost.xml:
      Code:
      <ReceiveBufferSize>32000</ReceiveBufferSize>
      <SendBufferSize>32000</SendBufferSize>
    5. In Linux, increase the maximum number of file descriptors. This addresses the Too many files open error message. To do this, edit the following two files:

      /usr/local/WowzaMediaServer/bin/wms.sh

      from:
      Code:
      #ulimit -n 20000
      to:
      Code:
      ulimit -n 20000
      and

      /usr/local/WowzaMediaServer/bin/startup.sh

      from:
      Code:
      #ulimit -n 20000
      to:
      Code:
      ulimit -n 20000
      Note: This may fail on some versions of Linux or if you don't have permission to change the descriptor limit. If after making this change Wowza server won't start, consult the documentation for your Linux distribution to see how to increase the descriptor limit.
      Note: On some versions of Linux, there's also a limit in the kernel that may need to be increased. Consult the documentation for your distribution. You may need to add the following line to your /etc/sysctl.conf file:
      Code:
      fs.file-max=20000
    6. In Linux, switch to using the Anticipatory elevator (as) elevator algorithm. It's described in the following article:


    7. In Linux, mount your disk with the noatime option. This operation differs based on your Linux distribution. Here's a basic description of the setting:


    8. To tune your server based on the available CPU resources of your server, use the following guidelines:

      The [total-core-count] refers to the total number of CPU cores in your server. For example, if you have dual quad-core processors (two quad-core processors), the [total-core-count] is:
      Code:
      2 (processor) x 4 (cores) = 8
      If your server supports hyper-threading, then use the total number of threads. In the above example, if hyper-threading is available, then the total number of threads is:
      Code:
      2 (processor) x 4 (cores) x 2 (threads per core) = 16
      With the number of cores and threads per physical processor continually growing, we suggest a maximum number of threads for each value below:

      In the configuration file [install-dir]/conf/VHost.xml:
      Code:
      HostPort/ProcessorCount: 2x[total-core-count] (maximum of 24) Note: The HostPort/ProcessorCount field in the Admin HostPort (/Port "8086") shouldn't be modified.
      IdleWorkers/WorkerCount: 2x[total-core-count] (maximum of 24)
      
      NetConnections/ProcessorCount: 2x[total-core-count]  (maximum of 24)
      
      RTP/UnicastIncoming/ProcessorCount: [total-core-count]  (maximum of 12)
      RTP/UnicastOutgoing/ProcessorCount: 2x[total-core-count]  (maximum of 24)
      
      RTP/MulticastIncoming/ProcessorCount: [total-core-count]  (maximum of 12)
      RTP/MulticastOutgoing/ProcessorCount: [total-core-count]  (maximum of 12)
      
      HandlerThreadPool/PoolSize: (60x[total-core-count]) (maximum of 480)
      TransportThreadPool/PoolSize: (40x[total-core-count]) (maximum of 320)
      These configuration calculations assume that you have at least 1GB of memory per core or—if you have four or more total cores—that you're running the 64–bit Java VM, and that you're using the suggested memory settings above.

    9. Running Multiple Virtual Hosts:

      If you're running more than one virtual host, the resource allocations must be distributed between each VHost. The simplest approach to running more than one virtual host is to divide the settings listed above (which are intended for a single VHost) and distribute the resources across each VHost on your system. The settings don't have to be evenly divided; however, the total should equal what you would allocate if you were configuring for a single VHost. If one of your VHosts is idle most of the time, you may allocate more memory than the combined total. Be careful with this setting because excessive allocations are risky. An outOfMemory error occurs if both VHosts exceed the combined, available resources.

      Alternatively, you can set each VHost/HandlerThreadPool/PoolSize to "0" and VHost/TransportThreadPool/PoolSize to "0" in [install-dir]/conf/VHost.xml, which causes the [install-dir]/conf/Server.xml settings for these properties to be used instead. This instructs the Wowza server to manage the PoolSize across all VHosts.

      A mixed approach can also be used with the [install-dir]/conf/VHost.xml file by setting the PoolSize to “0” for idle/minimal-use VHosts while using higher values for busy/high-performance VHosts with higher resource requirements.

    10. If you aren't doing low-latency streaming and you have a client side buffer of 3 or more seconds (NetStream.bufferTime), you can reduce the CPU load on the machine and handle more concurrent sessions by editing [install-dir]/conf/VHost.xml and changing the following two values:

      Code:
      IdleWorkers/CheckFrequency: 100
      Client/IdleFrequency: 500
    11. If you're having problems with multiple incoming multicast streams interfering with each other on Linux, you may need to set the Java property java.net.preferIPv4Stack to true. To do this, edit [install-dir]/bin/setenv.sh and uncomment the following line (line 13):

      Code:
      JAVA_OPTS="$JAVA_OPTS -Djava.net.preferIPv4Stack=true"
    12. When streaming to the Adobe Flash player, it's important to try and avoid RTMPT (the tunneling version of RTMP) as much as possible. RTMPT uses a polling mechanism that's very chatty and CPU intensive. Instead of using RTMPT, use a protocol rollover strategy so that only connections that need RTMPT use RTMPT. Such a strategy is described in the following article:


    13. In Windows 7, Server 2003 or 2008, turn off TCP auto tuning. There's information about how to do this in the following article:



    Updated: For Wowza Streaming Engine 4.0 on 02-11-2014

    Comments 23 Comments
    1. notreg -
      Why only 3-5GB for heap? How it could be calculated?
    1. rrlanham -
      You can allocate more to java heap if you have it, and if you have 64bit OS and 64bit Java. If server is dedicated Wowza server allocate most available ram.

      Richard
    1. alinutzelu -
      Yes, but "How it could be calculated?"
      It's any formula for occupied memory?
      I have 70 live http streams at 250kbps and wowza has reached at 2GB RAM (java process).

      Thank you
      Alin
    1. rrlanham -
      Take a look at the tuning guide:

      http://www.wowza.com/forums/content....ormance-Tuning

      The suggestions in this guide assume that the server is a dedicated Wowza server. If there are other applications that require a lot of memory, you would have to allow for that.

      Richard
    1. phamkv -
      Regarding point 9. :

      anticipatory scheduler has been removed from linux kernel as of kernel Linux 2.6.33 which was released on February 24th, 2010 .
    1. charlie -
      Thanks for the info. We will amend this step.

      Charlie
    1. donASKet -
      > If you have 16 GB or more RAM, we suggest a heap size of 8000 MB. We do not suggest a heap size above 10 GB as it can lead to long GC cycles/pauses
      >This assumes you have at least 1GB of memory per core and that in the case you have 4 or more total cores you are running the 64-bit Java VM using the suggested memory settings above.

      But i have server with 16 core's. How much memory must i set for JVM Heap? 8GB or 16GB (1GB per core)? Server use for CDN (10-20 stream's with 500-2000 user's per stream)
    1. dptekco -
      This is a followup to donASKet's as well, with larger machines and interfaces do the tuning guides recommended limits still apply? In particular:

      - 10GB heap limit
      - If heap size is 5000 MB or greater, NewSize=512m, but at 10GB or higher, still 512m?
      - Do the various VHost.xml maximum values still hold with a 16+ core machine?
      - With Java 7, do you still recommend the default GC with -server option? With a beefed up machine running huge loads, are any of the optional GC's , including new G1 superior at keeping garbage collection from affecting streams?

      I know each must test , but I'm curious in regards to best practices already posted and if they still apply with larger machines (16+ cores, 20GB+ ram, 10gbit interface, sustaining over 6gbps throughput)
    1. romild0 -
      I would like to know that too. Btw, is there a big difference in java version 1.6 and 1.7?
    1. rrlanham -
      If you have multiple VHosts you do have to divide allocations between them. In most cases you can use multiple applications in one VHost and avoid this complication.

      You will have to look at Oracle's release notes for details on different Java versions, but we are recommending the latest 1.7 release for Wowza

      Richard
    1. steveitl -
      hello,
      i not understand, where i need put the code :

      -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:NewSize=512m

      in the setenv.bat or WowzaMediaServer-Service.conf ?

      so, using Win2k8 x64 i need put :

      •[install-dir]/bin/setenv.bat (add at line 8, Windows).
      set JAVA_OPTS=%JAVA_OPTS% -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC

      and :

      •[install-dir]/bin/WowzaMediaServer-Service.conf (Windows).
      wrapper.java.additional.[n]=-XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC

      is correct ?

      thanks
      steve
    1. rrlanham -
      The GC lines go in /bin/setenv.bat

      Richard
    1. steveitl -
      Richard,
      for complete configuration, in setenv.bat, two line :

      set JAVA_OPTS=%JAVA_OPTS% -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC

      and :

      -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:NewSize=512m

      in WowzaMediaServer-Service.conf, one line :

      wrapper.java.additional.[n]=-XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC

      is correct ?

      steve
    1. Erwin Leon Pentzke -
      Richard and/or Charlie:
      In order to follow this guide and tune the right way EC2 Wowza servers I need you to help me with this question:
      Which is the number of threads in the Amazon Web Services EC2 most frequently used instances? (m1.small, m1.large, m1.xlarge, etc)
      I know that the m1.large has two cores, but there is no information about hyper-threading.
      Thanks in advance,
      Erwin
    1. oletinani -
      Hi,

      Some times my system taking 100% CPU, after changing default port(1935) to some other(7756) it should be fine. after some days again error occured then also we changed the port from 7756 to 1935, then it should be fine.

      Please help on this.

      Thanks
      Nani
    1. overture -
      I'm running multiple VHosts. CAn I set each VHost/HandlerThreadPool/PoolSize to "0" and VHost/TransportThreadPool/PoolSize to "0" and then configure only the /conf/Server.xml file?
      In this case How can I configure these settings?

      Daniele
    1. salvadore -
      Yes, you can set each VHost/HandlerThreadPool/PoolSize to "0" and VHost/TransportThreadPool/PoolSize to "0", which will cause the [install-dir]/conf/Server.xml settings for these properties to be used instead. This will instruct Wowza Media Server to manage the thread pool size across all VHosts.

      Although multiple applications is usually enough, consider that instead of multiple VHosts. Otherwise you should edit these files according to the tuning guide:
      Tuning Guide

      Salvadore
    1. overture -
      Ok, now I'm running 20 Virtual Hosts. Which is the better configuration in the Server.xml?
      Now both HandlerThreadPool/PoolSize and TransportThreadPool/PoolSize are set to 10 as default values.
    1. rrlanham -
      With many VHosts you are going to have to test. It is impossible to recommend exact settings. Some VHosts may have different loads. The section on multiple VHosts in the tuning guide also suggests a mixed approach where some VHosts have specific values for the settings and others are set to "0" to let Wowza manage between them, as it would with multiple applications anyway. In most cases you can make the necessary separate with Applications, then not have to worry about this tuning aspect.

      Richard
    1. tadung -
      Hi all,

      My Wowza streaming server with spec (using VPS):

      - 04 CPUs
      - RAM 16GB
      - HDD: 500GB, 7.2 rpm
      - NIC: 01Gbps
      - Internet (NIX): 512Mbps

      The only parameter that I configured:
      - Heapsize: 5000M (I just up to 8000M)

      After I checked this server on the system with tools: htop, top, free & others, I got the information:
      - CPU: ~5%
      - RAM: ~12-13GB used; ~2-3GB free
      - Each connection connects to this server, it occupied RAM ~600M
      - Connection concurrent users (CU): ~200
      (Sometimes, I must clear cahce/ram with "sync; echo 3 > /proc/sys/vm/drop_caches")

      So, I have questions for this issue:

      1. How to adjust the Heapsize with 16GBRAM? If I up to 32GBRAM, what's the Heapsize or other parameters?
      2. Can I decrease the memory that each connection occupy RAM? (Currently, it occupied ~600M)
      3. The system needs:
      - 500 CU at the same time, what about RAM for the streaming server?
      - 1000 CU at the same time, what about RAM for the streaming server?
      4. What parameters that I can tune for the streaming server?
      5. CPU is OK for this server if the system have more than 1000 CU at the same time?

      Please give me your advices about issues above.

      Thanks in advances.

      Tran Anh Dung.