Page 1 of 2 12 LastLast
Results 1 to 10 of 11

Thread: Memory increase over time

  1. #1
    Join Date
    Aug 2013
    Posts
    19

    Default Memory increase over time

    We are trying to figure out why our Wowza servers memory usage increases until it finally maxes out.

    Scenerio
    ======
    We will eventually have nearly 1000 clients connecting to our 2, load balanced, Wowza servers for an IPTV setup that includes live streaming with DVR and vod capability. Right now, in testing, we don't even have 50 clients connected at once.

    Problem
    ======

    After a few days, the Java heap size maxes out and the streams will start to occasionally skip. By monitoring JConsole, I noticed that the MemoryPool "CMS Old gen" gradually increases until 'Committed Memory' hits 'Max' (normally about 2 days), at which point all the other MemoryPools stop working correctly and we start to see errors with the streams (frames being dropped, stutter in video).

    Setup
    ====

    Encoders
    --------
    (18) MakitoX2
    (36) streams
    Video Out: 720p
    FPS: 60
    GOP: 120
    UDP Port: 10000-10500
    Bitrate: 3000

    Servers
    -------
    Server: (2)HP DL380G8
    CPU: (2)XeonE5(8core, 32 threads)
    RAM: 32GB
    NIC: (4)1Gb Bonded mode 6
    Harddrive: (5)146GB 6G SAS 15k RPM RAID10
    OS: Ubuntu 12.04.3
    Java: jdk 1.7.0_45
    Wowza Media Server: 3.6.4

    Wowza Settings
    --------------
    VHost.xml
    <Root>
            <VHost>
                    <HostPortList>
                            <HostPort>
                                    <ProcessorCount>24</ProcessorCount>
                                    <IpAddress>*</IpAddress>
                                    <!-- Separate multiple ports with commas -->
                                    <!-- 80: HTTP, RTMPT -->
                                    <!-- 554: RTSP -->
                                    <Port>1935</Port>
                                    <HTTPIdent2Response></HTTPIdent2Response>
                                    <SocketConfiguration>
                                            <ReuseAddress>true</ReuseAddress>
                                            <!-- suggested settings for video on demand applications -->
                                            <ReceiveBufferSize>65000</ReceiveBufferSize>
                                            <ReadBufferSize>65000</ReadBufferSize>
                                            <SendBufferSize>65000</SendBufferSize>
                                            <!-- suggest settings for low latency chat and video recording applications
                                            <ReceiveBufferSize>32000</ReceiveBufferSize>
                                            <ReadBufferSize>32000</ReadBufferSize>
                                            <SendBufferSize>32000</SendBufferSize>
                                            -->
                                            <KeepAlive>true</KeepAlive>
                                            <!-- <TrafficClass>0</TrafficClass> -->
                                            <!-- <OobInline>false</OobInline> -->
                                            <!-- <SoLingerTime>-1</SoLingerTime> -->
                                            <!-- <TcpNoDelay>false</TcpNoDelay> -->
                                            <AcceptorBackLog>100</AcceptorBackLog>
                                    </SocketConfiguration>
                                    <HTTPStreamerAdapterIDs>cupertinostreaming,smoothstreaming,sanjosestreaming,dvrchunkstreaming,mpegdashstreaming</HTTPStreamerAdapterIDs>
                                    <HTTPProviders>
                                            <HTTPProvider>
                                                    <BaseClass>com.wowza.wms.http.HTTPCrossdomain</BaseClass>
                                                    <RequestFilters>*crossdomain.xml</RequestFilters>
                                                    <AuthenticationMethod>none</AuthenticationMethod>
                                            </HTTPProvider>
                                            <HTTPProvider>
                                                    <BaseClass>com.wowza.wms.http.HTTPClientAccessPolicy</BaseClass>
                                                    <RequestFilters>*clientaccesspolicy.xml</RequestFilters>
                                                    <AuthenticationMethod>none</AuthenticationMethod>
                                            </HTTPProvider>
                                            <HTTPProvider>
                                                    <BaseClass>com.wowza.wms.http.HTTPProviderMediaList</BaseClass>
                                                    <RequestFilters>*jwplayer.rss|*jwplayer.smil|*medialist.smil|*manifest-rtmp.f4m</RequestFilters>
                                                    <AuthenticationMethod>none</AuthenticationMethod>
                                            </HTTPProvider>
                                            <!-- load balancer listener -->
                                            <HTTPProvider>
                                                    <BaseClass>com.wowza.wms.plugin.loadbalancer.HTTPLoadBalancerRedirector</BaseClass>
                                                    <RequestFilters>*loadbalancer</RequestFilters>
                                                    <AuthenticationMethod>none</AuthenticationMethod>
                                                    <Properties>
                                                            <Property>
                                                                    <Name>enableServerInfoXML</Name>
                                                                    <Value>true</Value>
                                                                    <Type>Boolean</Type>
                                                            </Property>
                                                    </Properties>
                                            </HTTPProvider>
                                            <HTTPProvider>
                                                    <BaseClass>com.wowza.wms.http.HTTPServerVersion</BaseClass>
                                                    <RequestFilters>*</RequestFilters>
                                                    <AuthenticationMethod>none</AuthenticationMethod>
                                            </HTTPProvider>
                                    </HTTPProviders>
                            </HostPort>
    
                            <!-- 443 with SSL -->
                            <!--
                            <HostPort>
                                    <ProcessorCount>4</ProcessorCount>
                                    <IpAddress>*</IpAddress>
                                    <Port>443</Port>
                                    <HTTPIdent2Response></HTTPIdent2Response>
                                    <SSLConfig>
                                            <KeyStorePath>${com.wowza.wms.context.VHostConfigHome}/conf/keystore.jks</KeyStorePath>
                                            <KeyStorePassword>[password]</KeyStorePassword>
                                            <KeyStoreType>JKS</KeyStoreType>
                                            <SSLProtocol>TLS</SSLProtocol>
                                            <Algorithm>SunX509</Algorithm>
                                            <CipherSuites></CipherSuites>
                                            <Protocols></Protocols>
                                    </SSLConfig>
                                    <SocketConfiguration>
                                            <ReuseAddress>true</ReuseAddress>
                                            <ReceiveBufferSize>65000</ReceiveBufferSize>
                                            <ReadBufferSize>65000</ReadBufferSize>
                                            <SendBufferSize>65000</SendBufferSize>
                                            <KeepAlive>true</KeepAlive>
                                            <AcceptorBackLog>100</AcceptorBackLog>
                                    </SocketConfiguration>
                                    <HTTPStreamerAdapterIDs>cupertinostreaming,smoothstreaming,sanjosestreaming,dvrchunkstreaming,mpegdashstreaming</HTTPStreamerAdapterIDs>
                                    <HTTPProviders>
                                            <HTTPProvider>
                                                    <BaseClass>com.wowza.wms.http.HTTPCrossdomain</BaseClass>
                                                    <RequestFilters>*crossdomain.xml</RequestFilters>
                                                    <AuthenticationMethod>none</AuthenticationMethod>
                                            </HTTPProvider>
                                            <HTTPProvider>
                                                    <BaseClass>com.wowza.wms.http.HTTPClientAccessPolicy</BaseClass>
                                                    <RequestFilters>*clientaccesspolicy.xml</RequestFilters>
                                                    <AuthenticationMethod>none</AuthenticationMethod>
                                            </HTTPProvider>
                                            <HTTPProvider>
                                                    <BaseClass>com.wowza.wms.http.HTTPProviderMediaList</BaseClass>
                                                    <RequestFilters>*jwplayer.rss|*jwplayer.smil|*medialist.smil|*manifest-rtmp.f4m</RequestFilters>
                                                    <AuthenticationMethod>none</AuthenticationMethod>
                                            </HTTPProvider>
                                            <HTTPProvider>
                                                    <BaseClass>com.wowza.wms.http.HTTPServerVersion</BaseClass>
                                                    <RequestFilters>*</RequestFilters>
                                                    <AuthenticationMethod>none</AuthenticationMethod>
                                            </HTTPProvider>
                                    </HTTPProviders>
                            </HostPort>
                            -->
    
                            <!-- Admin HostPort -->
                            <HostPort>
                                    <ProcessorCount>1</ProcessorCount>
                                    <IpAddress>*</IpAddress>
                                    <Port>8086</Port>
                                    <HTTPIdent2Response></HTTPIdent2Response>
                                    <SocketConfiguration>
                                            <ReuseAddress>true</ReuseAddress>
                                            <ReceiveBufferSize>16000</ReceiveBufferSize>
                                            <ReadBufferSize>16000</ReadBufferSize>
                                            <SendBufferSize>16000</SendBufferSize>
                                            <KeepAlive>true</KeepAlive>
                                            <AcceptorBackLog>100</AcceptorBackLog>
                                    </SocketConfiguration>
                                    <HTTPStreamerAdapterIDs></HTTPStreamerAdapterIDs>
                                    <HTTPProviders>
                                            <HTTPProvider>
                                                    <BaseClass>com.wowza.wms.http.streammanager.HTTPStreamManager</BaseClass>
                                                    <RequestFilters>streammanager*</RequestFilters>
                                                    <AuthenticationMethod>admin-digest</AuthenticationMethod>
                                            </HTTPProvider>
                                            <HTTPProvider>
                                                    <BaseClass>com.wowza.wms.http.HTTPServerInfoXML</BaseClass>
                                                    <RequestFilters>serverinfo*</RequestFilters>
                                                    <AuthenticationMethod>admin-digest</AuthenticationMethod>
                                            </HTTPProvider>
                                            <HTTPProvider>
                                                    <BaseClass>com.wowza.wms.http.HTTPConnectionInfo</BaseClass>
                                                    <RequestFilters>connectioninfo*</RequestFilters>
                                                    <AuthenticationMethod>admin-digest</AuthenticationMethod>
                                            </HTTPProvider>
                                            <HTTPProvider>
                                                    <BaseClass>com.wowza.wms.http.HTTPConnectionCountsXML</BaseClass>
                                                    <RequestFilters>connectioncounts*</RequestFilters>
                                                    <AuthenticationMethod>admin-digest</AuthenticationMethod>
                                            </HTTPProvider>
                                            <HTTPProvider>
                                                    <BaseClass>com.wowza.wms.transcoder.httpprovider.HTTPTranscoderThumbnail</BaseClass>
                                                    <RequestFilters>transcoderthumbnail*</RequestFilters>
                                                    <AuthenticationMethod>admin-digest</AuthenticationMethod>
                                            </HTTPProvider>
                                            <HTTPProvider>
                                                    <BaseClass>com.wowza.wms.http.HTTPProviderMediaList</BaseClass>
                                                    <RequestFilters>medialist*</RequestFilters>
                                                    <AuthenticationMethod>admin-digest</AuthenticationMethod>
                                            </HTTPProvider>
                                            <HTTPProvider>
                                                    <BaseClass>com.wowza.wms.livestreamrecord.http.HTTPLiveStreamRecord</BaseClass>
                                                    <RequestFilters>livestreamrecord*</RequestFilters>
                                                    <AuthenticationMethod>admin-digest</AuthenticationMethod>
                                            </HTTPProvider>
                                            <HTTPProvider>
                                                    <BaseClass>com.wowza.wms.http.HTTPServerVersion</BaseClass>
                                                    <RequestFilters>*</RequestFilters>
                                                    <AuthenticationMethod>none</AuthenticationMethod>
                                            </HTTPProvider>
                                    </HTTPProviders>
                            </HostPort>
    
                    </HostPortList>
    
                    <HTTPStreamerAdapters>
                            <HTTPStreamerAdapter>
                                    <ID>smoothstreaming</ID>
                                    <Name>smoothstreaming</Name>
                                    <Properties>
                                    </Properties>
                            </HTTPStreamerAdapter>
                            <HTTPStreamerAdapter>
                                    <ID>cupertinostreaming</ID>
                                    <Name>cupertinostreaming</Name>
                                    <Properties>
                                    </Properties>
                            </HTTPStreamerAdapter>
                            <HTTPStreamerAdapter>
                                    <ID>sanjosestreaming</ID>
                                    <Name>sanjosestreaming</Name>
                                    <Properties>
                                    </Properties>
                            </HTTPStreamerAdapter>
                            <HTTPStreamerAdapter>
                                    <ID>dvrchunkstreaming</ID>
                                    <Name>dvrchunkstreaming</Name>
                                    <Properties>
                                    </Properties>
                            </HTTPStreamerAdapter>
                            <HTTPStreamerAdapter>
                                    <ID>mpegdashstreaming</ID>
                                    <Name>mpegdashstreaming</Name>
                                    <Properties>
                                    </Properties>
                            </HTTPStreamerAdapter>
                    </HTTPStreamerAdapters>
    
                    <HandlerThreadPool>
                            <PoolSize>480</PoolSize>
                    </HandlerThreadPool>
                    <TransportThreadPool>
                            <PoolSize>320</PoolSize>
                    </TransportThreadPool>
                    <IdleWorkers>
                            <WorkerCount>24</WorkerCount>
                            <CheckFrequency>100</CheckFrequency>
                            <MinimumWaitTime>5</MinimumWaitTime>
                    </IdleWorkers>
                    <NetConnections>
                            <ProcessorCount>24</ProcessorCount>
                            <IdleFrequency>250</IdleFrequency>
                            <SocketConfiguration>
                                    <ReuseAddress>true</ReuseAddress>
                                    <ReceiveBufferSize>65000</ReceiveBufferSize>
                                    <ReadBufferSize>65000</ReadBufferSize>
                                    <SendBufferSize>65000</SendBufferSize>
                                    <KeepAlive>true</KeepAlive>
                                    <!-- <TrafficClass>0</TrafficClass> -->
                                    <!-- <OobInline>false</OobInline> -->
                                    <!-- <SoLingerTime>-1</SoLingerTime> -->
                                    <!-- <TcpNoDelay>false</TcpNoDelay> -->
                                    <AcceptorBackLog>100</AcceptorBackLog>
                            </SocketConfiguration>
                    </NetConnections>
                    <HTTPTunnel>
                            <KeepAliveTimeout>2000</KeepAliveTimeout>
                    </HTTPTunnel>
                    <Client>
                            <ClientTimeout>90000</ClientTimeout>
                            <IdleFrequency>500</IdleFrequency>
                    </Client>
                    <!-- RTP/Authentication/Methods defined in Authentication.xml. Default setup includes; none, basic, digest -->
                    <RTP>
                            <IdleFrequency>75</IdleFrequency>
                            <DatagramConfiguration>
                                    <Incoming>
                                            <ReuseAddress>true</ReuseAddress>
                                            <ReceiveBufferSize>1024000</ReceiveBufferSize>
                                            <SendBufferSize>65000</SendBufferSize>
                                            <!-- <MulticastBindToAddress>true</MulticastBindToAddress> -->
                                            <!-- <MulticastInterfaceAddress>192.168.1.22</MulticastInterfaceAddress> -->
                                            <!-- <TrafficClass>0</TrafficClass> -->
                                            <MulticastTimeout>50</MulticastTimeout>
                                            <DatagramMaximumPacketSize>4096</DatagramMaximumPacketSize>
                                    </Incoming>
                                    <Outgoing>
                                            <ReuseAddress>true</ReuseAddress>
                                            <ReceiveBufferSize>65000</ReceiveBufferSize>
                                            <SendBufferSize>65000</SendBufferSize>
                                            <!-- <MulticastBindToAddress>true</MulticastBindToAddress> -->
                                            <!-- <MulticastInterfaceAddress>192.168.1.22</MulticastInterfaceAddress> -->
                                            <!-- <TrafficClass>0</TrafficClass> -->
                                            <MulticastTimeout>50</MulticastTimeout>
                                            <DatagramMaximumPacketSize>4096</DatagramMaximumPacketSize>
                                    </Outgoing>
                            </DatagramConfiguration>
                            <UnicastIncoming>
                                    <ProcessorCount>12</ProcessorCount>
                            </UnicastIncoming>
                            <UnicastOutgoing>
                                    <ProcessorCount>24</ProcessorCount>
                            </UnicastOutgoing>
                            <MulticastIncoming>
                                    <ProcessorCount>12</ProcessorCount>
                            </MulticastIncoming>
                            <MulticastOutgoing>
                                    <ProcessorCount>12</ProcessorCount>
                            </MulticastOutgoing>
                    </RTP>
                    <Application>
                            <ApplicationTimeout>60000</ApplicationTimeout>
                            <PingTimeout>12000</PingTimeout>
                            <UnidentifiedSessionTimeout>30000</UnidentifiedSessionTimeout>
                            <ValidationFrequency>20000</ValidationFrequency>
                            <MaximumPendingWriteBytes>0</MaximumPendingWriteBytes>
                            <MaximumSetBufferTime>60000</MaximumSetBufferTime>
                    </Application>
                    <StartStartupStreams>true</StartStartupStreams>
                    <!-- Properties defined here will be added to the IVHost.getProperties() collection -->
                    <Properties>
                    </Properties>
            </VHost>
    </Root>
    bin/setenv.sh
    #!/bin/sh
    
    _EXECJAVA=java
    
    # Run server environment
    JAVA_OPTS="$JAVA_OPTS -server"
    
    # Max heap size (-Xmx<max> -Xms<min>)
    JAVA_OPTS="$JAVA_OPTS -Xmx10000M"
    
    # Custom garbage collection
    #JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:NewRatio=3" 
    
    # Better garbage collection setting to avoid long pauses
    JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:NewSize=512m"
    
    # Uncomment to log debug garbage collection information
    NOW=$(date +"%Y-%m-%d-%H-%M")
    JAVA_OPTS="$JAVA_OPTS -verbose:gc -Xloggc:"/var/log/wowza/gc_$NOW.log" -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime"
    
    # Uncomment to write heap dump on OutOfMemoryError
    #JAVA_OPTS="$JAVA_OPTS -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/local/WowzaMediaServer"
    
    # Uncomment to force Java to use specific language settings
    #JAVA_OPTS="$JAVA_OPTS -Duser.language=en -Duser.country=US -Dfile.encoding=Cp1252"
    
    # Uncomment to fix multicast crosstalk problem when streams share multicast port
    JAVA_OPTS="$JAVA_OPTS -Djava.net.preferIPv4Stack=true"
    
    WMSAPP_HOME=/usr/local/WowzaMediaServer
    WMSCONFIG_HOME=/usr/local/WowzaMediaServer
    WMSCONFIG_URL=
    
    export WMSAPP_HOME WMSCONFIG_HOME JAVA_OPTS _EXECJAVA

    Applications
    ------------

    vod
    liveorigin
    liveedge


    What we have tried
    ==================

    We have tuned the server as recommended here;
    http://www.wowza.com/forums/content....general-tuning

    We have looked through everything we could find on tuning in the forums and have tried many different ways for GC in setenv.sh. Here are some posts that we looked into:
    http://www.wowza.com/forums/showthre...on-a-local-LAN
    http://www.wowza.com/forums/showthre...g-Help-Request
    http://www.wowza.com/forums/showthre...-increasing-on

    On some of the posts, users were experiancing similar issues but it was resolved with upgrading Wowza. We have done that, however, and are still getting the memory increase.

    If you have any ideas or suggestions on how we can resolve this, it would be very helpful. We have a huge deployment coming up that will include 6 more servers with nearly the exact same setup. I need to get these 2 working first so that we have an idea of how to deploy the rest. Thanks!
    Last edited by jordanrumph; 01-27-2014 at 06:49 AM.

  2. #2
    Join Date
    Aug 2013
    Posts
    19

    Default

    Update. I just realized that the VHost.xml file wasn't tuned. I had done it and double checked in the past, but I might have lost it during an upgrade. I will monitor with the new changes and post results. That's probably it =(.

  3. #3
    Join Date
    Aug 2013
    Posts
    19

    Default

    I have verified the servers are properly tuned but we are still getting the same result. I updated my original post to show what the current tuning is. Any ideas as to why the memory increases?

    I have noticed more specifically what happens with the garbage collection. It seems that the "Par Eden space" clears out completely when it does it's GC. The graph over time is exactly the same. It's "CMS Old Gen" that slowly builds up. When it hits it's ceiling, the "Par Eden Space" and the "Par survivor space" max out as well and we notice video errors. It seems that the big GC that is supposed to clean out "CMS Old Gen" can't get it all in the time alloted, or can't clear it all for some other reason.

    Thanks for your help!

  4. #4

    Default

    Please zip up your /conf and /logs folders and send them to support@wowza.com to open a ticket.
    Please also include a link to this thread as reference.

    Salvadore

  5. #5
    Join Date
    Aug 2013
    Posts
    19

    Default

    Ok sent. Let me know if you need anything else. Thanks Salvadore!

    Jordan

  6. #6
    Join Date
    May 2014
    Posts
    2

    Default

    *bump*

    Any update? I am seeing similar trends on my wowza install where over 2-3 day the memory is filled up and our streams become unstable and jittery.

  7. #7
    Join Date
    Jun 2011
    Posts
    1,037

    Default

    Hi,
    In the above case improvement was seen by using the G1 (Garbage first) collector.

    It's certainly worth a try, and may help if you're seeing exactly the same issue.

    Daren

  8. #8
    Join Date
    Aug 2013
    Posts
    19

    Default

    Hi everyone,

    Just a quick update. We have been running these tuned settings for about 9 months now with complete stability. Here is our GC:

    bin/setenv.sh
    #!/bin/sh
    
    _EXECJAVA=java
    
    # Run server environment
    JAVA_OPTS="$JAVA_OPTS -server"
    
    # Max heap size (-Xmx<max> -Xms<min>)
    JAVA_OPTS="$JAVA_OPTS -Xmx10000M -Xms10000M"
    
    # G1GC settings
    JAVA_OPTS="$JAVA_OPTS -XX:+UseG1GC -XX:MaxGCPauseMillis=100 -XX:InitiatingHeapOccupancyPercent=45"
    
    # Better garbage collection setting to avoid long pauses
    #JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:NewSize=512m"
    
    # Uncomment to log debug G1GC information
    #NOW=$(date +"%Y-%m-%d-%H-%M")
    #JAVA_OPTS="$JAVA_OPTS -Xloggc:"/var/log/wowza/gc_$NOW.log" -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps"
    
    # Uncomment to log debug garbage collection information
    #NOW=$(date +"%Y-%m-%d-%H-%M")
    #JAVA_OPTS="$JAVA_OPTS -verbose:gc -Xloggc:"/var/log/wowza/gc_$NOW.log" -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime"
    
    # Uncomment to write heap dump on OutOfMemoryError
    #JAVA_OPTS="$JAVA_OPTS -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/local/WowzaMediaServer"
    
    # Uncomment to force Java to use specific language settings
    #JAVA_OPTS="$JAVA_OPTS -Duser.language=en -Duser.country=US -Dfile.encoding=Cp1252"
    
    # Uncomment to fix multicast crosstalk problem when streams share multicast port
    JAVA_OPTS="$JAVA_OPTS -Djava.net.preferIPv4Stack=true"
    
    WMSAPP_HOME=/usr/local/WowzaMediaServer
    WMSCONFIG_HOME=/usr/local/WowzaMediaServer
    WMSCONFIG_URL=
    
    export WMSAPP_HOME WMSCONFIG_HOME JAVA_OPTS _EXECJAVA
    Hope that helps!

    Jordan

  9. #9
    Join Date
    Aug 2014
    Posts
    3

    Default memory issue with Streaming Engine

    Quote Originally Posted by jordanrumph View Post
    Hi everyone,

    Just a quick update. We have been running these tuned settings for about 9 months now with complete stability. Here is our GC:

    bin/setenv.sh
    #!/bin/sh
    
    _EXECJAVA=java
    
    # Run server environment
    JAVA_OPTS="$JAVA_OPTS -server"
    
    # Max heap size (-Xmx<max> -Xms<min>)
    JAVA_OPTS="$JAVA_OPTS -Xmx10000M -Xms10000M"
    
    # G1GC settings
    JAVA_OPTS="$JAVA_OPTS -XX:+UseG1GC -XX:MaxGCPauseMillis=100 -XX:InitiatingHeapOccupancyPercent=45"
    
    # Better garbage collection setting to avoid long pauses
    #JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:NewSize=512m"
    
    # Uncomment to log debug G1GC information
    #NOW=$(date +"%Y-%m-%d-%H-%M")
    #JAVA_OPTS="$JAVA_OPTS -Xloggc:"/var/log/wowza/gc_$NOW.log" -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps"
    
    # Uncomment to log debug garbage collection information
    #NOW=$(date +"%Y-%m-%d-%H-%M")
    #JAVA_OPTS="$JAVA_OPTS -verbose:gc -Xloggc:"/var/log/wowza/gc_$NOW.log" -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime"
    
    # Uncomment to write heap dump on OutOfMemoryError
    #JAVA_OPTS="$JAVA_OPTS -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/local/WowzaMediaServer"
    
    # Uncomment to force Java to use specific language settings
    #JAVA_OPTS="$JAVA_OPTS -Duser.language=en -Duser.country=US -Dfile.encoding=Cp1252"
    
    # Uncomment to fix multicast crosstalk problem when streams share multicast port
    JAVA_OPTS="$JAVA_OPTS -Djava.net.preferIPv4Stack=true"
    
    WMSAPP_HOME=/usr/local/WowzaMediaServer
    WMSCONFIG_HOME=/usr/local/WowzaMediaServer
    WMSCONFIG_URL=
    
    export WMSAPP_HOME WMSCONFIG_HOME JAVA_OPTS _EXECJAVA
    Hope that helps!

    Jordan
    I assume this is version 3.x. How about for Streaming Engine v4.x? I have the same problem with memory usage.

    I stream a live stream to the server and have the transcoder on. The server record all incoming stream and then send that to my S3 bucket. My memory get close to 80% by the time I finished a 2hr session live streaming and recording. I am using an amazon EC2 instance c3.xlarge. I know it is at the low end, but the memory usage is way too high and I want to find out why and how could I fix it.

    Thanks!

  10. #10
    Join Date
    Aug 2013
    Posts
    19

    Default

    Hi cymaknova,

    You're right, that was for Wowza 3. We recently migrated to Wowza 4, and after a LOT of load testing, we find that the default GC, CMS, works best for us. I'm not sure if our load testing just got better or there was a change to Wowza 4 tuning, but G1GC won't last as long as CMS.

    One thing you can check is the GC logs that are enabled in 'conf/Tune.xml'. You'll need to educate yourself on how to read them, but a quick google search of 'reading gc logs' will give you lots of places to start. A site I found that had a lot of good articles is:

    https://blogs.oracle.com/poonam/

    Once you learn how to read the GC logs, you can see what's going on. Hope that helps!

    Jordan

Page 1 of 2 12 LastLast

Similar Threads

  1. Increase the segmentDuration time dynamically
    By bluestone in forum Live Streaming and Encoders
    Replies: 1
    Last Post: 09-25-2014, 09:31 AM
  2. Memory increase over time
    By sykwon in forum Performance Tuning Discussion
    Replies: 1
    Last Post: 02-13-2014, 09:38 AM
  3. memory use over time
    By pburke in forum Performance Tuning Discussion
    Replies: 8
    Last Post: 09-24-2013, 02:55 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •