• How to do performance tuning

    This article provides suggestions for tuning Wowza™ media server software on your production hardware.

    Contents


    Wowza Streaming Engine Performance Tuning
    Tuning Wowza Streaming Engine Java Settings
    Tuning Wowza Streaming Engine Server Thread Pools
    Tuning Wowza Streaming Engine Media Cache
    Tuning Wowza Streaming Engine Virtual Host Processors
    Tuning Wowza Streaming Engine Virtual Host Ports
    Tuning Wowza Streaming Engine Virtual Host Thread Pools
    Wowza Media Server Performance Tuning
    Additional tuning suggestions

    Wowza Streaming Engine Performance Tuning


    By default, Wowza Streaming Engine Manager can be accessed by using the following URL:

    http://[wowza-ip-address]:8088/enginemanager

    In Streaming Engine Manager, click the Server tab at the top of the page, and then click Performance Tuning in the contents panel. The Performance Tuning page shows the server's OS architecture, amount of memory available to Streaming Engine, the number of core processors on the system, and the Java version and architecture (bitness) in use.


    Tuning Wowza Streaming Engine Java Settings


    By default, Wowza Streaming Engine software (version 4.2.0 and later) is automatically installed with a supported version of Java and is tuned to a development state optimized for the hardware on which it's running. The server can be tuned to a production level by adjusting the Java settings. Knowing that the server is running to the best of its ability makes Wowza Streaming Engine easy to deploy in a production environment. Advanced users can alter the tuning further by using Wowza Streaming Engine Manager, if required.

    Note: If you're running Wowza Streaming Engine™ software version 4.1.2 (or earlier) or Wowza Media Server™ 3 software, you'll get best results if you run the most recent Oracle Java Development Kit (JDK). The best option is to run a 64-bit OS with the 64-bit Java VM, which enables Java heap sizes greater than 2 GB. For details, see How to manually install and troubleshoot Java with Wowza Streaming Engine.
    To access the Wowza Streaming Engine Java settings, click Java Settings in the contents panel. The Java Settings page shows the current Java settings, including the Java Heap Size, which is the amount of memory allocated to Wowza Streaming Engine, and the Java Garbage Collection Settings.



    To change these settings, click Edit.

    Java Heap Size

    The default for the Java Heap Size is the Development level. To run a dedicated server in a production environment, change the Java Heap Size setting to the Production level.

    If your server runs other services that have high memory consumption, you may want to change the Java Heap Size setting to the Custom level and enter a specific value. If you're running the 64-bit version of the Java VM and have 4 GB or more of RAM in your computer, set your Java heap size to between 3000 MB to 5000 MB. If you have at least 16 GB of RAM, set your heap size to 8000 MB. Don't set your heap size above 10 GB as this can lead to long garbage collection cycles/pauses.

    Click Save and then restart Wowza Streaming Engine so that your changes take effect.

    Java Garbage Collection Settings

    Garbage collection (GC) tuning in Java is tricky. What works best in one server setup may not work in another. Through trial-and-error and customer feedback, we recommend several approaches.

    G1 (Garbage First) collector


    First, we recommend that you use the default tuning G1 (Garbage-First) collector, which works well for many streaming situations and doesn't require modification. The G1 collector is designed for low pause time, high-throughput applications. It's a server-style garbage collector targeted for multi-processor computers with large memories, and it's fully supported in Oracle JDK 7 Update 4 and later releases.

    Note: You can adjust the GC pause time for the G1 collector in Custom collector settings. The following is an example of a suggested custom setting:

    -XX:+UseG1GC -XX:MaxGCPauseMillis=100

    Concurrent collector


    The Concurrent collector option is designed for applications that prefer shorter GC pauses and that can share processor resources with the garbage collector while the application is running.

    Note: If you experience problems with the Java heap, such as a build-up of memory usage, you can specify custom garbage collection settings for the concurrent collector in Custom collector settings. The following is an example of a suggested custom setting:

    -XX:+UseConcMarkSweepGC -XX:NewSize=512m

    The NewSize value varies based on Java heap size:

    • If Java heap size is 5000 MB or greater, NewSize=512m.

    • If Java heap size is 3000 MB to 5000 MB, NewSize=256m.

    • If Java heap size is less than 3000 MB, NewSize=128m

    NUMA-aware allocator


    A NUMA-aware allocator enables performance optimization of an application on computers with non-uniform memory architecture (NUMA) by increasing the application's use of lower latency memory. These are typically computers with multiple physical CPU sockets. By default, this option is disabled and no optimization for NUMA is made. The option is only available when the parallel garbage collector is used (-XX:+UseParallelGC).

    For example, to enable NUMA optimization on a multi-CPU socket system, enter the following in Custom collector settings:

    -XX:+UseParallelGC -XX:+UseNUMA

    Monitoring GC pause times


    To monitor GC pause times, open [install-dir]/conf/Tune.xml in a text editor and then add the following <VMOption> property:
    <VMOptions>
    	<VMOption>-server</VMOption>
    	<VMOption>-Djava.net.preferIPv4Stack=true</VMOption>
    	<VMOption>-verbose:gc -Xloggc:"${com.wowza.wms.AppHome}/logs/gc_${com.wowza.wms.StartupDateTime}.log" -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime</VMOption>
    </VMOptions>
    Then restart the Wowza Streaming Engine server to apply the changes.

    Notes:
    • This property is provided on line 33 in the Tune.xml file. You can remove the comment marks that surround the property to add it.

    • On Windows operating systems, this property creates a log file for debugging purposes and shouldn't be left running. The gc_${com.wowza.wms.StartupDateTime}.log file is only created when starting the Wowza media server in service mode or if using the command line.

    Tuning Wowza Streaming Engine Server Thread Pools


    Click Server Thread Pools in the contents panel. The Server Thread Pools page shows the current Handler Thread Pool Size and Transport Thread Pool Size.



    To change these settings, click Edit. When left at Set automatically, Wowza Streaming Engine calculates the Handler Thread Pool Size and Transport Thread Pool Size as follows:

    Handler Thread Pool Size = 60 x Processor Cores

    Transport Thread Pool Size = 40 x Processor Cores

    Note: The Processor Cores value appears on the Performance Tuning page.

    Tuning Wowza Streaming Engine Media Cache


    Click Media Cache Tuning in the contents panel. Media Cache Tuning settings apply to VOD Edge applications. They include the current Writer Thread Pool, Readahead Thread Pool, Maximum Pending Write Request Size, and Maximum Pending Readahead Request Size.



    To change these settings, click Edit. When left at Set automatically, Wowza Streaming Engine calculates the Writer Thread Pool and Readahead Thread Pool as follows:

    Writer Thread Pool = 2 x Processor Cores

    Readahead Thread Pool = 1 x Processor Cores

    Maximum Pending Write Request Size and Maximum Pending Readahead Request Size are calculated based on the Java Heap Size.

    Java Heap Size 1200 MB to 3999 MB 4000 MB to 7999 MB 8000 MB or greater
    Maximum Pending Write Request Size 160 MB 500 MB 1000 MB
    Maximum Pending Readahead Request Size 80 MB 250 MB 500 MB

    Tuning Wowza Streaming Engine Virtual Host Processors


    Click Virtual Host Processors in the contents panel. The Virtual Host Processors page shows the number of threads used at the VHost level to service various types of connections.



    To change the settings, click Edit. When left at Set automatically, Wowza Streaming Engine calculates the values as follows:

    Net Connections Processor Count = 2 x Processor Cores

    Media Caster Processor Count = 2 x Processor Cores

    Idle Worker Count = 2 x Processor Cores

    Unicast Incoming Processor Count = 2 x Processor Cores

    Unicast Outgoing Processor Count = 2 x Processor Cores

    Multicast Incoming Processor Count = 2 x Processor Cores

    Multicast Outgoing Processor Count = 2 x Processor Cores

    The Client Idle Frequency is the time (in milliseconds) between idle events for Adobe Flash client connections. For basic on-demand streaming, 250 ms provides the best reliability-to-performance ratio. For live, low-latency streaming, 125 to 250 ms is better. If you aren't doing low-latency streaming, Client Idle Frequency can be increased to 500, which reduces the CPU usage and allows more concurrent connections. Values between 1 and 1000 are supported.

    The RTP Idle Frequency value is the time (in milliseconds) between idle events for RTP connections. Values between 1 and 1000 are supported.

    Tuning Wowza Streaming Engine Virtual Host Ports


    The Virtual Host Ports page shows the current open ports and the processor count associated with each port.



    To change the settings, click Edit. When left at Set automatically, Wowza Streaming Engine sets the default processor counts as follows:

    Port 1935 Processor Count = 2 x Processor Cores

    Port 8086 Processor Count = 2 x Processor Cores

    Tuning Wowza Streaming Engine Virtual Host Thread Pools


    Click Virtual Host Thread Pools in the contents panel. Virtual Host Thread Pools are required when running multiple virtual hosts (VHosts) on the server.



    If you're running multiple VHosts, configure the Handler Thread Pool Size and Transport Thread Pool Size to Use Server Settings. When configured to Use Server Settings, the handler and transport threads are divided equally between all of the active VHosts that are running on the server.

    Wowza Media Server Performance Tuning


    Notes:
    • The instructions in this section are for use with Wowza Media Server™ 3 software. If you're running Wowza Streaming Engine™ software, see Wowza Streaming Engine Performance Tuning.

    • After editing any of the configuration files described in this section, restart the media server so that the changes take effect.

    Java Heap Size


    The default Java heap size is 768 MB for Windows and 1200 MB for Linux and OS X. These values are optimized for development and aren't large enough for high-volume production use. If you're running the 64-bit version of the Java VM and have 4 GB or more of RAM in your machine, set your Java heap size to between 3000 MB to 5000 MB. If you have at least 16 GB of RAM, set your heap size to 8000 MB. Don't set your heap size above 10 GB as this can lead to long garbage collection cycles/pauses. You can adjust memory settings in the following script files:

    • [install-dir]/bin/setenv.bat (Windows Standalone and Windows Service, around line 4)
      set JAVA_OPTS=-Xmx3000M
    • [install-dir]/bin/setenv.sh (Linux and OS X, around line 4)
      JAVA_OPTS="-Xmx3000M"

    Garbage Collection


    Garbage collection (GC) tuning in Java is tricky. What works best in one server setup may not work in another. Through trial-and-error and customer feedback, we've developed several suggested approaches.

    First, we strongly recommend that you use no additional settings. The default tuning works well for many streaming situations and doesn't require modification.

    If, however, you're experiencing problems with the Java heap, such as a build-up of memory usage, you can try implementing one of the following garbage collection options:

    • G1 (Garbage-First) collector is designed for low pause time, high-throughput applications. The G1 collector is a server-style garbage collector targeted for multi-processor machines with large memories, and it's fully supported in Oracle JDK 7 Update 4 and later releases. The suggested additional settings are:

      -XX:+UseG1GC -XX:MaxGCPauseMillis=100
    • Concurrent collector is designed for applications that prefer shorter GC pauses and that can share processor resources with the garbage collector while the application is running. The suggested additional settings are:

      If heap size is 5000 MB or greater, NewSize=512m

      If heap size is 3000 MB to 5000 MB, NewSize=256m

      If heap size is less than 3000 MB, NewSize=128m

      For example, with a 64-bit OS, 8 GB RAM, and heap size of 6000 MB:
      -XX:+UseConcMarkSweepGC -XX:NewSize=512m
    • NUMA-aware allocator enables performance optimization of an application on computers with non-uniform memory architecture (NUMA) by increasing the application's use of lower latency memory. These are typically computers with multiple physical CPU sockets. By default, this option is disabled and no optimization for NUMA is made. The option is only available when the parallel garbage collector is used (-XX:+UseParallelGC).

      For example, to enable NUMA optimization on a multi-CPU socket system:
      -XX:+UseParallelGC -XX:+UseNUMA
    To change the garbage collection settings, modify the following scripts:

    • [install-dir]/bin/setenv.sh (Linux and OS X, uncomment line 10). For example:
      JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:NewSize=512m"
    • [install-dir]/bin/setenv.bat (Windows Standalone and Windows Service, uncomment line 10). For example:
      set JAVA_OPTS=%JAVA_OPTS% -XX:+UseConcMarkSweepGC -XX:NewSize=512m
    To monitor GC pause time, modify the following scripts:

    • [install-dir]/bin/setenv.sh (Linux). Add:
      -verbose:gc -Xloggc:"/usr/local/WowzaMediaServer/logs/gc_%UNIQUE%.log" -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC
    • [install-dir]/bin/setenv.sh (OS X). Add:
      -verbose:gc -Xloggc:"/Library/WowzaMediaServer/logs/gc_%UNIQUE%.log" -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC
    • [install-dir]/bin/setenv.bat (Windows). Add:
      set JAVA_OPTS=%JAVA_OPTS% -verbose:gc -Xloggc:"c:\gc.log" -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime
      Note: On Windows, this creates a C:\gc.log file for debugging purposes and shouldn't be left running. The gc.log file is only created when starting the Wowza media server in service mode or if using the command line.

    Additional tuning suggestions


    Additional tuning for low-latency chat applications


    For low-latency chat applications, it's best to use smaller socket buffer sizes (16000 bytes for read and write). The socket buffer sizes are configured in [install-dir]/conf/VHost.xml:
    <ReceiveBufferSize>32000</ReceiveBufferSize>
    <SendBufferSize>32000</SendBufferSize>
    To adjust these properties in Wowza Streaming Engine Manager:

    1. Click the Server tab at the top of the page.

    2. In the Server contents panel, click Virtual Host Setup.

    3. On the Virtual Host Setup page Properties tab, click Net Connctions in the Quick Links bar.

      Note: Access to the Properties tab is limited to administrators with advanced permissions. For more information, see Manage credentials.
    4. In the Net Connections properties area, click Edit, adjust the receiveBufferSize and sendBufferSize property values, and then click Save.

    5. Restart the virtual host when prompted to apply the changes.

    Additional tuning for Linux computers


    • You can increase the maximum number of file descriptors to address the "Too many files open" error message. To do this, in /usr/local/WowzaMediaServer/bin/wms.sh, and in /usr/local/WowzaMediaServer/bin/startup.sh, change:
      #ulimit -n 20000
      To:
      ulimit -n 20000
      Notes:
      • If the Wowza media server won't start after making this change, consult the documentation for your Linux distribution to see how to increase the descriptor limit.

      • On some Linux versions, there's also a limit in the kernel that may need to be increased. Consult the documentation for your distribution. You may need to add the following line to your /etc/sysctl.conf file:
        fs.file-max=20000
    • Switch to using the Anticipatory (as) elevator algorithm.

    • Mount your disk with the noatime option. This operation differs based on your Linux distribution. A description of the setting is available at The atime and noatime attribute.

    CPU resource-based tuning


    To tune your media server based on the available CPU resources of your server, use the following guidelines:

    The [total-core-count] refers to the total number of CPU cores in your server. For example, if you have dual quad-core processors (two quad-core processors), your [total-core-count] is 8.

    If your server supports hyper-threading, then use the total number of threads. If hyper-threading is available on a system with dual quad-core processors, for example, the total number of threads is:
    2 processors x 4 cores x 2 threads per core = 16
    With the number of cores and threads per physical processor continually growing, we suggest a maximum number of threads for each value below:

    In the configuration file [install-dir]/conf/VHost.xml:
    HostPort/ProcessorCount: 2 x [total-core-count] (maximum of 24)
    
    IdleWorkers/WorkerCount: 2 x [total-core-count] (maximum of 24)
    
    NetConnections/ProcessorCount: 2 x [total-core-count] (maximum of 24)
    
    RTP/UnicastIncoming/ProcessorCount: [total-core-count] (maximum of 12)
    
    RTP/UnicastOutgoing/ProcessorCount: 2 x [total-core-count] (maximum of 24)
    
    RTP/MulticastIncoming/ProcessorCount: [total-core-count] (maximum of 12)
    
    RTP/MulticastOutgoing/ProcessorCount: [total-core-count] (maximum of 12)
    
    HandlerThreadPool/PoolSize: (60 x [total-core-count]) (maximum of 480)
    
    TransportThreadPool/PoolSize: (40 x [total-core-count]) (maximum of 320)
    Note: Do NOT modify the HostPort ProcessorCount field in the Admin HostPort (/Port "8086").
    These configuration calculations assume that you have at least 1 GB of memory per core or--if you have four or more total cores--that you're running the 64-bit Java VM, and that you're using the suggested memory settings above.

    Tuning for multiple VHosts


    If you're running more than one virtual host (VHost), resource allocations must be distributed between each VHost. The simplest approach to running more than one VHost is to divide the settings listed above (which are intended for a single VHost) and distribute the resources across each VHost on your system. The settings don't have to be evenly divided; however, the total should equal what you would allocate if you were configuring for a single VHost. If one of your VHosts is idle most of the time, you may allocate more memory than the combined total. Be careful with this setting because excessive allocations are risky. An out-of-memory error occurs if both VHosts exceed the combined, available resources.

    Alternatively, in [install-dir]/conf/VHost.xml, you can set each VHost/HandlerThreadPool/PoolSize and VHost/TransportThreadPool/PoolSize property to 0, which causes the [install-dir]/conf/Server.xml settings for these properties to be used instead. This instructs the Wowza media server to manage the pool size across all VHosts.

    A mixed approach can also be used with the [install-dir]/conf/VHost.xml file by setting the PoolSize properties to 0 for idle/minimal-use VHosts while using higher values for busy/high-performance VHosts with higher resource requirements.

    Tuning idle client system checks


    If you aren't doing low-latency streaming and you have a client-side buffer of 3 or more seconds (NetStream.bufferTime), you can reduce the CPU load on the computer and handle more concurrent sessions by changing the following values in [install-dir]/conf/VHost.xml:
    IdleWorkers/CheckFrequency: 50
    Client/IdleFrequency: 250

    Tuning on dual-stack computers


    If you're running Wowza Media Server software and having problems with multiple incoming multicast streams interfering with each other on Linux, you may need to set the Java property java.net.preferIPv4Stack to true. To do this, edit [install-dir]/bin/setenv.sh and uncomment the following line (line 13):
    JAVA_OPTS="$JAVA_OPTS -Djava.net.preferIPv4Stack=true"
    Note: On Wowza Streaming Engine software, this Java property is set to true by default in [install-dir]/conf/Tune.xml.

    TCP auto-tuning


    In Windows 7, Server 2003, or 2008, turn off TCP auto-tuning. For more information, see How to disable Windows Vista TCP/IP auto-tuning.

    Protocol rollover


    When streaming to the Adobe Flash player, it's important to try and avoid RTMPT (the tunneling version of RTMP). RTMPT uses a polling mechanism that's chatty and CPU intensive. Instead of using RTMPT, use a protocol rollover strategy so that only connections that need RTMPT use it. This strategy is described in How to set up protocol rollover with ActionScript.

    Originally Published: 10-01-2010.
    Updated: For Wowza Streaming Engine 4.2.0 on 06-16-2015.

    If you're having problems or want to discuss this article, post in our forum.