Results 1 to 9 of 9

Thread: memory use over time

  1. #1

    Default memory use over time

    I posted this over in the transcoder forum, because I figure it has something to do with that, however, it may just be a normal thing.

    thread over in the other forum is

    http://www.wowza.com/forums/showthre...memory-setting

    This sever (3.6.2, windows, 16 cores, 24GB ram with 8000M heap size) is running four live streams with little traffic. Two are single bitrate, two are transcoded from 720p to 360p, 240p and 160p. Starting up the server and transcoders has the system running at about 750MB to 1000MB of RAM in use. Since the transcoder mountpoints were started, the amount of memory Wowza is using has been steadily creeping up. Today, 6 days later, we are averaging 2.8GB of RAM, and the trend is a pretty linear line upwards.



    Is this growth in memory use normal? I am worried we'll run out of memory after a few weeks when it hits the 8GB allocated for java.

    server has "Java HotSpot(TM) 64-Bit Server VM version 23.0-b21", threads are around 900, CPU runs around 25% when transcoding two sources.

    conf files that may be relevant


    @echo off
    
    set _EXECJAVA=java
    set JAVA_OPTS=-server -Xmx8000M
    
    rem If running JDK, uncomment to run server environment (faster)
    rem set JAVA_OPTS=%JAVA_OPTS% -server
    
    rem Better garbage collection setting to avoid long pauses
    rem set JAVA_OPTS=%JAVA_OPTS% -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:NewSize=512m
    
    rem Uncomment to log debug garbage collection information
    rem set NOW=%date:~10,4%-%date:~4,2%-%date:~7,2%-%time:~0,2%-%time:~3,2%
    rem set JAVA_OPTS=%JAVA_OPTS% -verbose:gc -Xloggc:"%WMSAPP_HOME%/logs/gc_%NOW%.log" -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime
    
    rem Uncomment to write heap dump on OutOfMemoryError
    rem set JAVA_OPTS=%JAVA_OPTS% -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=%WMSAPP_HOME%
    
    rem Uncomment to fix multicast crosstalk problem when streams share multicast port
    rem set JAVA_OPTS=%JAVA_OPTS% -Djava.net.preferIPv4Stack=true
    
    rem Uncomment to force Java to use specific language settings
    rem set JAVA_OPTS=%JAVA_OPTS% -Duser.language=en -Duser.country=US -Dfile.encoding=Cp1252
    
    set EXECCSCRIPT=cscript
    if not exist "%SystemRoot%\SysWOW64\cscript.exe" goto skip64bitcscript
    set EXECCSCRIPT="%SystemRoot%\SysWOW64\cscript.exe"
    :skip64bitcscript
    
    set WMSCONFIG_URL=
    rem set WMSAPP_HOME=C:/wms2
    rem set WMSCONFIG_HOME=C:/wms2
    
    set WMSENVOK="false"
    
    rem Guess WMSAPP_HOME if not defined
    set CURRENT_DIR=%cd%
    if not "%WMSAPP_HOME%" == "" goto gotAppHome
    set WMSAPP_HOME=%CURRENT_DIR%
    if exist "%WMSAPP_HOME%\bin\startup.bat" goto okAppHome
    cd ..
    set WMSAPP_HOME=%cd%
    cd %CURRENT_DIR%
    :gotAppHome
    if exist "%WMSAPP_HOME%\bin\startup.bat" goto okAppHome
    %EXECSCRIPT% "displaymsg.vbs" "The WMSAPP_HOME environment variable is missing or not defined correctly (%WMSAPP_HOME%). This environment variable is needed to run the server."
    goto end
    :okAppHome
    
    rem Guess WMSCONFIG_HOME if not defined
    set CURRENT_DIR=%cd%
    if not "%WMSCONFIG_HOME%" == "" goto gotConfigHome
    set WMSCONFIG_HOME=%WMSAPP_HOME%
    if exist "%WMSCONFIG_HOME%\conf\Server.license" goto okConfigHome
    cd ..
    set WMSCONFIG_HOME=%cd%
    cd %CURRENT_DIR%
    :gotConfigHome
    if exist "%WMSCONFIG_HOME%\conf\Server.license" goto okConfigHome
    %EXECSCRIPT% "displaymsg.vbs" "The WMSCONFIG_HOME environment variable is missing or not defined correctly (%WMSCONFIG_HOME%). This environment variable is needed to run the server."
    goto end
    :okConfigHome
    
    %EXECSCRIPT% "checkjavahome.vbs" //Nologo
    if not %errorlevel% == 0 goto end
    
    set WMSENVOK="true"
    set _EXECJAVA=%JAVA_HOME%\bin\java.exe
    
    :end

    vhost.xml

    <Root>
    	<VHost>
    		<HostPortList>
    			<HostPort>
    				<ProcessorCount>16</ProcessorCount>
    				<IpAddress>128.104.nnn.nnn</IpAddress>
    				<!-- 128.104.144.134 is our fixed 10 gig number -->
    				<!-- Separate multiple ports with commas -->
    				<!-- 80: HTTP, RTMPT -->
    				<!-- 554: RTSP -->
    				<Port>1935,80,443,554</Port>
    				<HTTPIdent2Response></HTTPIdent2Response>
    				<SocketConfiguration>
    					<ReuseAddress>true</ReuseAddress>
    					<!-- suggested settings for video on demand applications -->
    					<ReceiveBufferSize>65000</ReceiveBufferSize>
    					<ReadBufferSize>65000</ReadBufferSize>
    					<SendBufferSize>65000</SendBufferSize>
    					<!-- suggest settings for low latency chat and video recording applications
    					<ReceiveBufferSize>32000</ReceiveBufferSize>
    					<ReadBufferSize>32000</ReadBufferSize>
    					<SendBufferSize>32000</SendBufferSize>
    					-->
    					<KeepAlive>true</KeepAlive>
    					<!-- <TrafficClass>0</TrafficClass> -->
    					<!-- <OobInline>false</OobInline> -->
    					<!-- <SoLingerTime>-1</SoLingerTime> -->
    					<!-- <TcpNoDelay>false</TcpNoDelay> -->
    					<AcceptorBackLog>100</AcceptorBackLog>
    				</SocketConfiguration>
    				<HTTPStreamerAdapterIDs>cupertinostreaming,smoothstreaming,sanjosestreaming,dvrchunkstreaming,mpegdashstreaming</HTTPStreamerAdapterIDs>
    				<HTTPProviders>
    					<HTTPProvider>
    						<BaseClass>com.wowza.wms.http.HTTPCrossdomain</BaseClass>
    						<RequestFilters>*crossdomain.xml</RequestFilters>
    						<AuthenticationMethod>none</AuthenticationMethod>
    					</HTTPProvider>
    					<HTTPProvider>
    						<BaseClass>com.wowza.wms.http.HTTPClientAccessPolicy</BaseClass>
    						<RequestFilters>*clientaccesspolicy.xml</RequestFilters>
    						<AuthenticationMethod>none</AuthenticationMethod>
    					</HTTPProvider>
    					<HTTPProvider>
    						<BaseClass>com.wowza.wms.http.HTTPProviderMediaList</BaseClass>
    						<RequestFilters>*jwplayer.rss|*jwplayer.smil|*medialist.smil|*manifest-rtmp.f4m</RequestFilters>
    						<AuthenticationMethod>none</AuthenticationMethod>
    					</HTTPProvider>
    					<HTTPProvider>
    						<BaseClass>com.wowza.wms.http.HTTPServerVersion</BaseClass>
    						<RequestFilters>*</RequestFilters>
    						<AuthenticationMethod>none</AuthenticationMethod>
    					</HTTPProvider>
    				</HTTPProviders>
    			</HostPort>
    			
    			<!-- 443 with SSL -->
    			<!--
    			<HostPort>
    				<ProcessorCount>4</ProcessorCount>
    				<IpAddress>*</IpAddress>
    				<Port>443</Port>
    				<HTTPIdent2Response></HTTPIdent2Response>
    				<SSLConfig>
    					<KeyStorePath>${com.wowza.wms.context.VHostConfigHome}/conf/keystore.jks</KeyStorePath>
    					<KeyStorePassword>[password]</KeyStorePassword>
    					<KeyStoreType>JKS</KeyStoreType>
    					<SSLProtocol>TLS</SSLProtocol>
    					<Algorithm>SunX509</Algorithm>
    					<CipherSuites></CipherSuites>
    					<Protocols></Protocols>
    				</SSLConfig>
    				<SocketConfiguration>
    					<ReuseAddress>true</ReuseAddress>
    					<ReceiveBufferSize>65000</ReceiveBufferSize>
    					<ReadBufferSize>65000</ReadBufferSize>
    					<SendBufferSize>65000</SendBufferSize>
    					<KeepAlive>true</KeepAlive>
    					<AcceptorBackLog>100</AcceptorBackLog>
    				</SocketConfiguration>
    				<HTTPStreamerAdapterIDs>cupertinostreaming,smoothstreaming,sanjosestreaming,dvrchunkstreaming,mpegdashstreaming</HTTPStreamerAdapterIDs>
    				<HTTPProviders>
    					<HTTPProvider>
    						<BaseClass>com.wowza.wms.http.HTTPCrossdomain</BaseClass>
    						<RequestFilters>*crossdomain.xml</RequestFilters>
    						<AuthenticationMethod>none</AuthenticationMethod>
    					</HTTPProvider>
    					<HTTPProvider>
    						<BaseClass>com.wowza.wms.http.HTTPClientAccessPolicy</BaseClass>
    						<RequestFilters>*clientaccesspolicy.xml</RequestFilters>
    						<AuthenticationMethod>none</AuthenticationMethod>
    					</HTTPProvider>
    					<HTTPProvider>
    						<BaseClass>com.wowza.wms.http.HTTPProviderMediaList</BaseClass>
    						<RequestFilters>*jwplayer.rss|*jwplayer.smil|*medialist.smil|*manifest-rtmp.f4m</RequestFilters>
    						<AuthenticationMethod>none</AuthenticationMethod>
    					</HTTPProvider>
    					<HTTPProvider>
    						<BaseClass>com.wowza.wms.http.HTTPServerVersion</BaseClass>
    						<RequestFilters>*</RequestFilters>
    						<AuthenticationMethod>none</AuthenticationMethod>
    					</HTTPProvider>
    				</HTTPProviders>
    			</HostPort>
    			-->
    			
    			<!-- Admin HostPort -->
    			<HostPort>
    				<ProcessorCount>16</ProcessorCount>
    				<IpAddress>128.104.nnn.nnn</IpAddress>
    				<Port>8086</Port>
    				<HTTPIdent2Response></HTTPIdent2Response>
    				<SocketConfiguration>
    					<ReuseAddress>true</ReuseAddress>
    					<ReceiveBufferSize>16000</ReceiveBufferSize>
    					<ReadBufferSize>16000</ReadBufferSize>
    					<SendBufferSize>16000</SendBufferSize>
    					<KeepAlive>true</KeepAlive>
    					<AcceptorBackLog>100</AcceptorBackLog>
    				</SocketConfiguration>
    				<HTTPStreamerAdapterIDs></HTTPStreamerAdapterIDs>
    				<HTTPProviders>
    					<HTTPProvider>
    						<BaseClass>com.wowza.wms.http.streammanager.HTTPStreamManager</BaseClass>
    						<RequestFilters>streammanager*</RequestFilters>
    						<AuthenticationMethod>admin-digest</AuthenticationMethod>
    					</HTTPProvider>
    					<HTTPProvider>
    						<BaseClass>com.wowza.wms.http.HTTPServerInfoXML</BaseClass>
    						<RequestFilters>serverinfo*</RequestFilters>
    						<AuthenticationMethod>admin-digest</AuthenticationMethod>
    					</HTTPProvider>
    					<HTTPProvider>
    						<BaseClass>com.wowza.wms.http.HTTPConnectionInfo</BaseClass>
    						<RequestFilters>connectioninfo*</RequestFilters>
    						<AuthenticationMethod>admin-digest</AuthenticationMethod>
    					</HTTPProvider>
    					<HTTPProvider>
    						<BaseClass>com.wowza.wms.http.HTTPConnectionCountsXML</BaseClass>
    						<RequestFilters>connectioncounts*</RequestFilters>
    						<AuthenticationMethod>admin-digest</AuthenticationMethod>
    					</HTTPProvider>
    					<HTTPProvider>
    						<BaseClass>com.wowza.wms.transcoder.httpprovider.HTTPTranscoderThumbnail</BaseClass>
    						<RequestFilters>transcoderthumbnail*</RequestFilters>
    						<AuthenticationMethod>admin-digest</AuthenticationMethod>
    					</HTTPProvider>
    					<HTTPProvider>
    						<BaseClass>com.wowza.wms.http.HTTPProviderMediaList</BaseClass>
    						<RequestFilters>medialist*</RequestFilters>
    						<AuthenticationMethod>admin-digest</AuthenticationMethod>
    					</HTTPProvider>
    					<HTTPProvider>
    						<BaseClass>com.wowza.wms.livestreamrecord.http.HTTPLiveStreamRecord</BaseClass>
    						<RequestFilters>livestreamrecord*</RequestFilters>
    						<AuthenticationMethod>admin-digest</AuthenticationMethod>
    					</HTTPProvider>
    					<HTTPProvider>
    						<BaseClass>com.wowza.wms.http.HTTPServerVersion</BaseClass>
    						<RequestFilters>*</RequestFilters>
    						<AuthenticationMethod>none</AuthenticationMethod>
    					</HTTPProvider>
    				</HTTPProviders>
    			</HostPort>
    		
    		</HostPortList>
    		
    		<HTTPStreamerAdapters>
    			<HTTPStreamerAdapter>
    				<ID>smoothstreaming</ID>
    				<Name>smoothstreaming</Name>
    				<Properties>
    				</Properties>
    			</HTTPStreamerAdapter>
    			<HTTPStreamerAdapter>
    				<ID>cupertinostreaming</ID>
    				<Name>cupertinostreaming</Name>
    				<Properties>
    				</Properties>
    			</HTTPStreamerAdapter>
    			<HTTPStreamerAdapter>
    				<ID>sanjosestreaming</ID>
    				<Name>sanjosestreaming</Name>
    				<Properties>
    				</Properties>
    			</HTTPStreamerAdapter>
    			<HTTPStreamerAdapter>
    				<ID>dvrchunkstreaming</ID>
    				<Name>dvrchunkstreaming</Name>
    				<Properties>
    				</Properties>
    			</HTTPStreamerAdapter>
    			<HTTPStreamerAdapter>
    				<ID>mpegdashstreaming</ID>
    				<Name>mpegdashstreaming</Name>
    				<Properties>
    				</Properties>
    			</HTTPStreamerAdapter>
    		</HTTPStreamerAdapters>
    
    		<HandlerThreadPool>
    			<PoolSize>480</PoolSize>
    		</HandlerThreadPool>
    		<TransportThreadPool>
    			<PoolSize>360</PoolSize>
    		</TransportThreadPool>
    		<IdleWorkers>
    			<WorkerCount>4</WorkerCount>
    			<CheckFrequency>50</CheckFrequency>
    			<MinimumWaitTime>5</MinimumWaitTime>
    		</IdleWorkers>
    		<NetConnections>
    			<ProcessorCount>16</ProcessorCount>
    			<IdleFrequency>250</IdleFrequency>
    			<SocketConfiguration>
    				<ReuseAddress>true</ReuseAddress>
    				<ReceiveBufferSize>65000</ReceiveBufferSize>
    				<ReadBufferSize>65000</ReadBufferSize>
    				<SendBufferSize>65000</SendBufferSize>
    				<KeepAlive>true</KeepAlive>
    				<!-- <TrafficClass>0</TrafficClass> -->
    				<!-- <OobInline>false</OobInline> -->
    				<!-- <SoLingerTime>-1</SoLingerTime> -->
    				<!-- <TcpNoDelay>false</TcpNoDelay> -->
    				<AcceptorBackLog>100</AcceptorBackLog>
    			</SocketConfiguration>
    		</NetConnections>
    		<HTTPTunnel>
    			<KeepAliveTimeout>2000</KeepAliveTimeout>
    		</HTTPTunnel>
    		<Client>
    			<ClientTimeout>90000</ClientTimeout>
    			<IdleFrequency>250</IdleFrequency>
    		</Client>
    		<!-- RTP/Authentication/Methods defined in Authentication.xml. Default setup includes; none, basic, digest -->
    		<RTP>
    			<IdleFrequency>75</IdleFrequency>
    			<DatagramConfiguration>
    				<Incoming>
    					<ReuseAddress>true</ReuseAddress>
    					<ReceiveBufferSize>1024000</ReceiveBufferSize>
    					<SendBufferSize>65000</SendBufferSize>
    					<!-- <MulticastBindToAddress>true</MulticastBindToAddress> -->
    					<!-- <MulticastInterfaceAddress>192.168.1.22</MulticastInterfaceAddress> -->
    					<!-- <TrafficClass>0</TrafficClass> -->
    					<MulticastTimeout>50</MulticastTimeout>
    					<DatagramMaximumPacketSize>4096</DatagramMaximumPacketSize>
    				</Incoming>
    				<Outgoing>
    					<ReuseAddress>true</ReuseAddress>
    					<ReceiveBufferSize>65000</ReceiveBufferSize>
    					<SendBufferSize>65000</SendBufferSize>
    					<!-- <MulticastBindToAddress>true</MulticastBindToAddress> -->
    					<!-- <MulticastInterfaceAddress>192.168.1.22</MulticastInterfaceAddress> -->
    					<!-- <TrafficClass>0</TrafficClass> -->
    					<MulticastTimeout>50</MulticastTimeout>
    					<DatagramMaximumPacketSize>4096</DatagramMaximumPacketSize>
    				</Outgoing>
    			</DatagramConfiguration>
    			<UnicastIncoming>
    				<ProcessorCount>8</ProcessorCount>
    			</UnicastIncoming>
    			<UnicastOutgoing>
    				<ProcessorCount>16</ProcessorCount>
    			</UnicastOutgoing>
    			<MulticastIncoming>
    				<ProcessorCount>4</ProcessorCount>
    			</MulticastIncoming>
    			<MulticastOutgoing>
    				<ProcessorCount>4</ProcessorCount>
    			</MulticastOutgoing>
    		</RTP>
    		<Application>
    			<ApplicationTimeout>60000</ApplicationTimeout>
    			<PingTimeout>12000</PingTimeout>
    			<ValidationFrequency>20000</ValidationFrequency>
    			<MaximumPendingWriteBytes>0</MaximumPendingWriteBytes>
    			<MaximumSetBufferTime>60000</MaximumSetBufferTime>
    		</Application>
    		<StartStartupStreams>true</StartStartupStreams>
    		<!-- Properties defined here will be added to the IVHost.getProperties() collection -->
    		<Properties>
    		</Properties>
    	</VHost>
    </Root>

  2. #2
    Join Date
    Sep 2011
    Posts
    1,933

    Default

    Hi,
    Looking at the [Wowza-Install]/bin/setenv.bat file I can see that you have not enabled any garbage collection (GC).
    Please find the tuning guide below which has instructions on how to enable the GC settings. The GC settings will reduce the memory usage.
    Tuning guide

    I can also see in the [Wowza-Install]/conf/VHost.xml you have configured the ProcessorCount as 16, if the server has 16 processors available this should be configured as 24 (the maximum).
    In the tuning guide you will see how the [total-core-count] should be calculated for tuning:

    Note: The HostPort/ProcessorCount field in the Admin HostPort (/Port "8086") should not be modified.
    HostPort/ProcessorCount: 2x[total-core-count] (maximum of 24)
    IdleWorkers/WorkerCount: 2x[total-core-count] (maximum of 24)
    
    NetConnections/ProcessorCount: 2x[total-core-count]  (maximum of 24)
    
    RTP/UnicastIncoming/ProcessorCount: [total-core-count]  (maximum of 12)
    RTP/UnicastOutgoing/ProcessorCount: 2x[total-core-count]  (maximum of 24)
    
    RTP/MulticastIncoming/ProcessorCount: [total-core-count]  (maximum of 12)
    RTP/MulticastOutgoing/ProcessorCount: [total-core-count]  (maximum of 12)
    
    HandlerThreadPool/PoolSize: (60x[total-core-count]) (maximum of 480)
    TransportThreadPool/PoolSize: (40x[total-core-count]) (maximum of 320)
    Jason

  3. #3

    Default

    thank you Jason - I will change the settings, just in time for our patch reboot for MS patches tonight.

    But let me ask this, because I thought I had garbage collection working: when I look at the Management console, I see memory dropping from 4GB to 2.5GB every 4 to 5 minutes - that's NOT garbage collection, or just general Java cleanup that doesn't affect wowza? The Java Monitoring console tells me

    Garbage collector:*
    Name = 'PS MarkSweep', Collections = 6,286, Total time spent = 39 minutes
    Garbage collector:*
    Name = 'PS Scavenge', Collections = 283,611, Total time spent = 58 minutes

    I guess I expected that this was built in and the line in the batch file was to be uncommented when you are having problems. Otherwise, why would you not enable that setting by default?


    Also - I am getting a headache looking at your documentation that goes from processor count to cores to threads. The system has two 4 core/dual threaded Xeon processors. That's 16 little graphs in Task Manager under performance. That means I should set things to 24 because that's the max (because the math otherwise would go to 32)?

    and a more general comment: something that could save you a lot of forum replies, because I have seen this "check the tuning guide" so many times here that it isn't just me having issues. This happens all the time, and it only happens because the numbers are hidden in bloated XML documents and cryptic command line batch lines nobody but people who work with java and wowza on a daily basis really understand fully. So why isn't there a simply configuration took (heck, a batch file could do it) that asks the user the key questions to configure the basics from processor count to memory? It seems the more advanced these servers get, the more difficult it gets to configure them. Or an online form done as a configurator that spits out ready to use XML files based on input to copy paste into the files in the conf folder? You could save yourself a lot of support time repeating the same old "read the tuning guide" to people like me who read it, but still didn't quite understand where to enter what in their scenario.

  4. #4
    Join Date
    Sep 2011
    Posts
    1,933

    Default

    Hi,
    The GC settings set in Wowza are specific to Wowza and will run to clean up memory usage as needed. The default configuration is not tuned in any way as each server is different and will require tuning based on the hardware available. The default figures are high enough for testing purposes but not suitable for use in a production environment.

    Yes that's correct, if the maximum was not 24, you would have used 32 as HostPort/ProcessorCount: 2x[16] = 32.

    and a more general comment: something that could save you a lot of forum replies, because I have seen this "check the tuning guide" so many times here that it isn't just me having issues. This happens all the time, and it only happens because the numbers are hidden in bloated XML documents and cryptic command line batch lines nobody but people who work with java and wowza on a daily basis really understand fully. So why isn't there a simply configuration took (heck, a batch file could do it) that asks the user the key questions to configure the basics from processor count to memory? It seems the more advanced these servers get, the more difficult it gets to configure them. Or an online form done as a configurator that spits out ready to use XML files based on input to copy paste into the files in the conf folder? You could save yourself a lot of support time repeating the same old "read the tuning guide" to people like me who read it, but still didn't quite understand where to enter what in their scenario.
    Thank you for your feedback, we do appreciate a general view.
    I will pass this information to the Wowza development team who I'm sure will also appreciate your comments and consider your views.

    An easy way to check what hardware you have is when Wowza starts, it prints the amount of processors and memory it can see and available to it in the access log.
    This number of processors is then used to calculate the [total-core-count] as seen above.

    Thanks again,
    Jason

  5. #5

    Default

    The GC seems to work now. I changed the settings as suggested in the java startup, plus corrected the processor count in the xml (16 according to Wowza - it's one of those hyper threaded dual quad core boxes). Memory after 36 hours holding steady between 750MB and 1.5GB, which is much better than it id before.

    Could I save memory or other resources by disabling some of the packetizers we aren't using (smooth, DASH?)?

    I have some config/performance questions regarding another live streaming/transcoding system we are planning, but I'll post that separately.

  6. #6
    Join Date
    Sep 2011
    Posts
    1,933

    Default

    Hi,
    Thanks for the update, I'm glad the server is stable and you have managed to reduce the memory usage.
    Yes, you're correct, if you're not streaming to Microsoft Silverlight or MPEG DASH you can turn off the packetization for these clients which will save you some memory but it's not much.

    Regards,
    Jason

  7. #7

    Default

    well, I may have been too quick posting about this being resolved. memory somehow decided to pick up 2 days into running the server

    Now, about 4 days into running with the GC settings suggested, this is what I am seeing:



    seems to be doing the same thing. I'll let it go and see if we get any closer to the 8GB limit over the coming days, but I will have to reset this before the weekend when a larger audience is expected.

  8. #8

    Default

    Still scratching my head about the memory use of the server.

    We changed the live encoder to make sure we were not seeing this memory drift because of the errors the Liveshell Pro box is throwing on frame rate with the transcoder (adding the sourceStreamFrameRate property made no difference) . Friday we reset the entire server, and all transcoder feeds were switched to Wirecast feeds. The server memory use is still increasing over time. The system is essentially idle, 4 live streams, 2 being transcoded, never more than 10 viewers this weekend, nobody by a tester or two looking at the transcoded streams. Memory went from 700MB used to over 2GB in less than 3 days.




    current server settings - tweaks suggested here in the past don't seem to make much of a difference:

    setenv.bat

    @echo off
    
    set _EXECJAVA=java
    set JAVA_OPTS=-server -Xmx8000M
    
    rem If running JDK, uncomment to run server environment (faster)
    set JAVA_OPTS=%JAVA_OPTS% -server
    
    rem Better garbage collection setting to avoid long pauses
    set JAVA_OPTS=%JAVA_OPTS% -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:NewSize=512m
    
    rem Uncomment to log debug garbage collection information
    rem set NOW=%date:~10,4%-%date:~4,2%-%date:~7,2%-%time:~0,2%-%time:~3,2%
    rem set JAVA_OPTS=%JAVA_OPTS% -verbose:gc -Xloggc:"%WMSAPP_HOME%/logs/gc_%NOW%.log" -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime
    
    rem Uncomment to write heap dump on OutOfMemoryError
    rem set JAVA_OPTS=%JAVA_OPTS% -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=%WMSAPP_HOME%
    
    rem Uncomment to fix multicast crosstalk problem when streams share multicast port
    rem set JAVA_OPTS=%JAVA_OPTS% -Djava.net.preferIPv4Stack=true
    
    rem Uncomment to force Java to use specific language settings
    rem set JAVA_OPTS=%JAVA_OPTS% -Duser.language=en -Duser.country=US -Dfile.encoding=Cp1252
    
    set EXECCSCRIPT=cscript
    if not exist "%SystemRoot%\SysWOW64\cscript.exe" goto skip64bitcscript
    set EXECCSCRIPT="%SystemRoot%\SysWOW64\cscript.exe"
    :skip64bitcscript
    
    set WMSCONFIG_URL=
    rem set WMSAPP_HOME=C:/wms2
    rem set WMSCONFIG_HOME=C:/wms2
    
    set WMSENVOK="false"
    
    rem Guess WMSAPP_HOME if not defined
    set CURRENT_DIR=%cd%
    if not "%WMSAPP_HOME%" == "" goto gotAppHome
    set WMSAPP_HOME=%CURRENT_DIR%
    if exist "%WMSAPP_HOME%\bin\startup.bat" goto okAppHome
    cd ..
    set WMSAPP_HOME=%cd%
    cd %CURRENT_DIR%
    :gotAppHome
    if exist "%WMSAPP_HOME%\bin\startup.bat" goto okAppHome
    %EXECSCRIPT% "displaymsg.vbs" "The WMSAPP_HOME environment variable is missing or not defined correctly (%WMSAPP_HOME%). This environment variable is needed to run the server."
    goto end
    :okAppHome
    
    rem Guess WMSCONFIG_HOME if not defined
    set CURRENT_DIR=%cd%
    if not "%WMSCONFIG_HOME%" == "" goto gotConfigHome
    set WMSCONFIG_HOME=%WMSAPP_HOME%
    if exist "%WMSCONFIG_HOME%\conf\Server.license" goto okConfigHome
    cd ..
    set WMSCONFIG_HOME=%cd%
    cd %CURRENT_DIR%
    :gotConfigHome
    if exist "%WMSCONFIG_HOME%\conf\Server.license" goto okConfigHome
    %EXECSCRIPT% "displaymsg.vbs" "The WMSCONFIG_HOME environment variable is missing or not defined correctly (%WMSCONFIG_HOME%). This environment variable is needed to run the server."
    goto end
    :okConfigHome
    
    %EXECSCRIPT% "checkjavahome.vbs" //Nologo
    if not %errorlevel% == 0 goto end
    
    set WMSENVOK="true"
    set _EXECJAVA=%JAVA_HOME%\bin\java.exe
    
    :end
    vhost.xml


    <Root>
    	<VHost>
    		<HostPortList>
    			<HostPort>
    				<ProcessorCount>16</ProcessorCount>
    				<IpAddress>128.104.nnn.nnn</IpAddress>
    				<!-- Separate multiple ports with commas -->
    				<!-- 80: HTTP, RTMPT -->
    				<!-- 554: RTSP -->
    				<Port>1935,80,443,554</Port>
    				<HTTPIdent2Response></HTTPIdent2Response>
    				<SocketConfiguration>
    					<ReuseAddress>true</ReuseAddress>
    					<!-- suggested settings for video on demand applications -->
    					<ReceiveBufferSize>65000</ReceiveBufferSize>
    					<ReadBufferSize>65000</ReadBufferSize>
    					<SendBufferSize>65000</SendBufferSize>
    					<!-- suggest settings for low latency chat and video recording applications
    					<ReceiveBufferSize>32000</ReceiveBufferSize>
    					<ReadBufferSize>32000</ReadBufferSize>
    					<SendBufferSize>32000</SendBufferSize>
    					-->
    					<KeepAlive>true</KeepAlive>
    					<!-- <TrafficClass>0</TrafficClass> -->
    					<!-- <OobInline>false</OobInline> -->
    					<!-- <SoLingerTime>-1</SoLingerTime> -->
    					<!-- <TcpNoDelay>false</TcpNoDelay> -->
    					<AcceptorBackLog>100</AcceptorBackLog>
    				</SocketConfiguration>
    				<HTTPStreamerAdapterIDs>cupertinostreaming,smoothstreaming,sanjosestreaming,dvrchunkstreaming,mpegdashstreaming</HTTPStreamerAdapterIDs>
    				<HTTPProviders>
    					<HTTPProvider>
    						<BaseClass>com.wowza.wms.http.HTTPCrossdomain</BaseClass>
    						<RequestFilters>*crossdomain.xml</RequestFilters>
    						<AuthenticationMethod>none</AuthenticationMethod>
    					</HTTPProvider>
    					<HTTPProvider>
    						<BaseClass>com.wowza.wms.http.HTTPClientAccessPolicy</BaseClass>
    						<RequestFilters>*clientaccesspolicy.xml</RequestFilters>
    						<AuthenticationMethod>none</AuthenticationMethod>
    					</HTTPProvider>
    					<HTTPProvider>
    						<BaseClass>com.wowza.wms.http.HTTPProviderMediaList</BaseClass>
    						<RequestFilters>*jwplayer.rss|*jwplayer.smil|*medialist.smil|*manifest-rtmp.f4m</RequestFilters>
    						<AuthenticationMethod>none</AuthenticationMethod>
    					</HTTPProvider>
    					<HTTPProvider>
    						<BaseClass>com.wowza.wms.http.HTTPServerVersion</BaseClass>
    						<RequestFilters>*</RequestFilters>
    						<AuthenticationMethod>none</AuthenticationMethod>
    					</HTTPProvider>
    				</HTTPProviders>
    			</HostPort>
    			
    			<!-- 443 with SSL -->
    			<!--
    			<HostPort>
    				<ProcessorCount>4</ProcessorCount>
    				<IpAddress>*</IpAddress>
    				<Port>443</Port>
    				<HTTPIdent2Response></HTTPIdent2Response>
    				<SSLConfig>
    					<KeyStorePath>${com.wowza.wms.context.VHostConfigHome}/conf/keystore.jks</KeyStorePath>
    					<KeyStorePassword>[password]</KeyStorePassword>
    					<KeyStoreType>JKS</KeyStoreType>
    					<SSLProtocol>TLS</SSLProtocol>
    					<Algorithm>SunX509</Algorithm>
    					<CipherSuites></CipherSuites>
    					<Protocols></Protocols>
    				</SSLConfig>
    				<SocketConfiguration>
    					<ReuseAddress>true</ReuseAddress>
    					<ReceiveBufferSize>65000</ReceiveBufferSize>
    					<ReadBufferSize>65000</ReadBufferSize>
    					<SendBufferSize>65000</SendBufferSize>
    					<KeepAlive>true</KeepAlive>
    					<AcceptorBackLog>100</AcceptorBackLog>
    				</SocketConfiguration>
    				<HTTPStreamerAdapterIDs>cupertinostreaming,smoothstreaming,sanjosestreaming,dvrchunkstreaming,mpegdashstreaming</HTTPStreamerAdapterIDs>
    				<HTTPProviders>
    					<HTTPProvider>
    						<BaseClass>com.wowza.wms.http.HTTPCrossdomain</BaseClass>
    						<RequestFilters>*crossdomain.xml</RequestFilters>
    						<AuthenticationMethod>none</AuthenticationMethod>
    					</HTTPProvider>
    					<HTTPProvider>
    						<BaseClass>com.wowza.wms.http.HTTPClientAccessPolicy</BaseClass>
    						<RequestFilters>*clientaccesspolicy.xml</RequestFilters>
    						<AuthenticationMethod>none</AuthenticationMethod>
    					</HTTPProvider>
    					<HTTPProvider>
    						<BaseClass>com.wowza.wms.http.HTTPProviderMediaList</BaseClass>
    						<RequestFilters>*jwplayer.rss|*jwplayer.smil|*medialist.smil|*manifest-rtmp.f4m</RequestFilters>
    						<AuthenticationMethod>none</AuthenticationMethod>
    					</HTTPProvider>
    					<HTTPProvider>
    						<BaseClass>com.wowza.wms.http.HTTPServerVersion</BaseClass>
    						<RequestFilters>*</RequestFilters>
    						<AuthenticationMethod>none</AuthenticationMethod>
    					</HTTPProvider>
    				</HTTPProviders>
    			</HostPort>
    			-->
    			
    			<!-- Admin HostPort -->
    			<HostPort>
    				<ProcessorCount>16</ProcessorCount>
    				<IpAddress>128.104.nnn.nnn</IpAddress>
    				<Port>8086</Port>
    				<HTTPIdent2Response></HTTPIdent2Response>
    				<SocketConfiguration>
    					<ReuseAddress>true</ReuseAddress>
    					<ReceiveBufferSize>16000</ReceiveBufferSize>
    					<ReadBufferSize>16000</ReadBufferSize>
    					<SendBufferSize>16000</SendBufferSize>
    					<KeepAlive>true</KeepAlive>
    					<AcceptorBackLog>100</AcceptorBackLog>
    				</SocketConfiguration>
    				<HTTPStreamerAdapterIDs></HTTPStreamerAdapterIDs>
    				<HTTPProviders>
    					<HTTPProvider>
    						<BaseClass>com.wowza.wms.http.streammanager.HTTPStreamManager</BaseClass>
    						<RequestFilters>streammanager*</RequestFilters>
    						<AuthenticationMethod>admin-digest</AuthenticationMethod>
    					</HTTPProvider>
    					<HTTPProvider>
    						<BaseClass>com.wowza.wms.http.HTTPServerInfoXML</BaseClass>
    						<RequestFilters>serverinfo*</RequestFilters>
    						<AuthenticationMethod>admin-digest</AuthenticationMethod>
    					</HTTPProvider>
    					<HTTPProvider>
    						<BaseClass>com.wowza.wms.http.HTTPConnectionInfo</BaseClass>
    						<RequestFilters>connectioninfo*</RequestFilters>
    						<AuthenticationMethod>admin-digest</AuthenticationMethod>
    					</HTTPProvider>
    					<HTTPProvider>
    						<BaseClass>com.wowza.wms.http.HTTPConnectionCountsXML</BaseClass>
    						<RequestFilters>connectioncounts*</RequestFilters>
    						<AuthenticationMethod>admin-digest</AuthenticationMethod>
    					</HTTPProvider>
    					<HTTPProvider>
    						<BaseClass>com.wowza.wms.transcoder.httpprovider.HTTPTranscoderThumbnail</BaseClass>
    						<RequestFilters>transcoderthumbnail*</RequestFilters>
    						<AuthenticationMethod>admin-digest</AuthenticationMethod>
    					</HTTPProvider>
    					<HTTPProvider>
    						<BaseClass>com.wowza.wms.http.HTTPProviderMediaList</BaseClass>
    						<RequestFilters>medialist*</RequestFilters>
    						<AuthenticationMethod>admin-digest</AuthenticationMethod>
    					</HTTPProvider>
    					<HTTPProvider>
    						<BaseClass>com.wowza.wms.livestreamrecord.http.HTTPLiveStreamRecord</BaseClass>
    						<RequestFilters>livestreamrecord*</RequestFilters>
    						<AuthenticationMethod>admin-digest</AuthenticationMethod>
    					</HTTPProvider>
    					<HTTPProvider>
    						<BaseClass>com.wowza.wms.http.HTTPServerVersion</BaseClass>
    						<RequestFilters>*</RequestFilters>
    						<AuthenticationMethod>none</AuthenticationMethod>
    					</HTTPProvider>
    				</HTTPProviders>
    			</HostPort>
    		
    		</HostPortList>
    		
    		<HTTPStreamerAdapters>
    			<HTTPStreamerAdapter>
    				<ID>smoothstreaming</ID>
    				<Name>smoothstreaming</Name>
    				<Properties>
    				</Properties>
    			</HTTPStreamerAdapter>
    			<HTTPStreamerAdapter>
    				<ID>cupertinostreaming</ID>
    				<Name>cupertinostreaming</Name>
    				<Properties>
    				</Properties>
    			</HTTPStreamerAdapter>
    			<HTTPStreamerAdapter>
    				<ID>sanjosestreaming</ID>
    				<Name>sanjosestreaming</Name>
    				<Properties>
    				</Properties>
    			</HTTPStreamerAdapter>
    			<HTTPStreamerAdapter>
    				<ID>dvrchunkstreaming</ID>
    				<Name>dvrchunkstreaming</Name>
    				<Properties>
    				</Properties>
    			</HTTPStreamerAdapter>
    			<HTTPStreamerAdapter>
    				<ID>mpegdashstreaming</ID>
    				<Name>mpegdashstreaming</Name>
    				<Properties>
    				</Properties>
    			</HTTPStreamerAdapter>
    		</HTTPStreamerAdapters>
    
    		<HandlerThreadPool>
    			<PoolSize>480</PoolSize>
    		</HandlerThreadPool>
    		<TransportThreadPool>
    			<PoolSize>360</PoolSize>
    		</TransportThreadPool>
    		<IdleWorkers>
    			<WorkerCount>4</WorkerCount>
    			<CheckFrequency>50</CheckFrequency>
    			<MinimumWaitTime>5</MinimumWaitTime>
    		</IdleWorkers>
    		<NetConnections>
    			<ProcessorCount>16</ProcessorCount>
    			<IdleFrequency>250</IdleFrequency>
    			<SocketConfiguration>
    				<ReuseAddress>true</ReuseAddress>
    				<ReceiveBufferSize>65000</ReceiveBufferSize>
    				<ReadBufferSize>65000</ReadBufferSize>
    				<SendBufferSize>65000</SendBufferSize>
    				<KeepAlive>true</KeepAlive>
    				<!-- <TrafficClass>0</TrafficClass> -->
    				<!-- <OobInline>false</OobInline> -->
    				<!-- <SoLingerTime>-1</SoLingerTime> -->
    				<!-- <TcpNoDelay>false</TcpNoDelay> -->
    				<AcceptorBackLog>100</AcceptorBackLog>
    			</SocketConfiguration>
    		</NetConnections>
    		<HTTPTunnel>
    			<KeepAliveTimeout>2000</KeepAliveTimeout>
    		</HTTPTunnel>
    		<Client>
    			<ClientTimeout>90000</ClientTimeout>
    			<IdleFrequency>250</IdleFrequency>
    		</Client>
    		<!-- RTP/Authentication/Methods defined in Authentication.xml. Default setup includes; none, basic, digest -->
    		<RTP>
    			<IdleFrequency>75</IdleFrequency>
    			<DatagramConfiguration>
    				<Incoming>
    					<ReuseAddress>true</ReuseAddress>
    					<ReceiveBufferSize>1024000</ReceiveBufferSize>
    					<SendBufferSize>65000</SendBufferSize>
    					<!-- <MulticastBindToAddress>true</MulticastBindToAddress> -->
    					<!-- <MulticastInterfaceAddress>192.168.1.22</MulticastInterfaceAddress> -->
    					<!-- <TrafficClass>0</TrafficClass> -->
    					<MulticastTimeout>50</MulticastTimeout>
    					<DatagramMaximumPacketSize>4096</DatagramMaximumPacketSize>
    				</Incoming>
    				<Outgoing>
    					<ReuseAddress>true</ReuseAddress>
    					<ReceiveBufferSize>65000</ReceiveBufferSize>
    					<SendBufferSize>65000</SendBufferSize>
    					<!-- <MulticastBindToAddress>true</MulticastBindToAddress> -->
    					<!-- <MulticastInterfaceAddress>192.168.1.22</MulticastInterfaceAddress> -->
    					<!-- <TrafficClass>0</TrafficClass> -->
    					<MulticastTimeout>50</MulticastTimeout>
    					<DatagramMaximumPacketSize>4096</DatagramMaximumPacketSize>
    				</Outgoing>
    			</DatagramConfiguration>
    			<UnicastIncoming>
    				<ProcessorCount>8</ProcessorCount>
    			</UnicastIncoming>
    			<UnicastOutgoing>
    				<ProcessorCount>16</ProcessorCount>
    			</UnicastOutgoing>
    			<MulticastIncoming>
    				<ProcessorCount>4</ProcessorCount>
    			</MulticastIncoming>
    			<MulticastOutgoing>
    				<ProcessorCount>4</ProcessorCount>
    			</MulticastOutgoing>
    		</RTP>
    		<Application>
    			<ApplicationTimeout>60000</ApplicationTimeout>
    			<PingTimeout>12000</PingTimeout>
    			<ValidationFrequency>20000</ValidationFrequency>
    			<MaximumPendingWriteBytes>0</MaximumPendingWriteBytes>
    			<MaximumSetBufferTime>60000</MaximumSetBufferTime>
    		</Application>
    		<StartStartupStreams>true</StartStartupStreams>
    		<!-- Properties defined here will be added to the IVHost.getProperties() collection -->
    		<Properties>
    		</Properties>
    	</VHost>
    </Root>

    The two transcoders are doing this stuff, being fed a 2mbps 720p stream, either by Wirecast or Liveshell Pro appliance. None of this memory growth happens when we don't transcode, nor does it happen in the first few hours after a server restart.


    <!-- Example template for transrate, producing four new streams at different bitrates. Resultant streams can be played back individually or as a group. source, 360p and 160p encode blocks are enabled through the Enable property, other examples are not enabled. Add additional encode blocks to your template as needed. -->
    <Root>
    	<Transcode>
    		<Encodes>
    			<!-- Example Encode block for source, not required unless Member of StreamNameGroup. --> 
    			<Encode>
    				<Enable>true</Enable>
    				<Name>source</Name>
    				<StreamName>mp4:${SourceStreamName}_source</StreamName>
    				<Video>
    					<!-- H.264, PassThru, Disable -->
    					<Codec>PassThru</Codec>
    					<Bitrate>${SourceVideoBitrate}</Bitrate>
    					<Parameters>
    					</Parameters>
    				</Video>
    				<Audio>
    					<!-- AAC, PassThru, Disable -->
    					<Codec>PassThru</Codec>
    					<Bitrate>${SourceAudioBitrate}</Bitrate>
    				</Audio>
    				<Properties>
    				</Properties>
    			</Encode>
    			<!-- Setup for 720p, high bandwith, main profile for desktop or set-top box -->
    			<Encode>
    				<Enable>false</Enable>
    				<Name>720p</Name>
    				<StreamName>mp4:${SourceStreamName}_720p</StreamName>
    				<Video>
    					<!-- H.264, PassThru, Disable -->
    					<Codec>H.264</Codec>
    					<!-- default, CUDA, QuickSync -->
    					<Transcoder>default</Transcoder>
    					<GPUID>-1</GPUID>
    					<FrameSize>
    						<!-- letterbox, fit-width, fit-height, crop, stretch, match-source -->
    						<FitMode>fit-height</FitMode>
    						<Width>1280</Width>
    						<Height>720</Height>
    						<!-- <Crop>0,0,0,0</Crop> -->
    						<!-- <SourceRectangle>0,0,320,240</SourceRectangle> -->
    					</FrameSize>
    					<Profile>main</Profile>
    					<Bitrate>1300000</Bitrate>
    					<KeyFrameInterval>
    						<FollowSource>true</FollowSource>
    						<Interval>60</Interval>
    					</KeyFrameInterval>
    					<Overlays>
    						<Overlay>
    							<Enable>false</Enable>
    							<Index>0</Index>
    							<ImagePath>${com.wowza.wms.context.VHostConfigHome}/content/wowzalogo.png</ImagePath>
    							<Opacity>100</Opacity>
    							<Location>
    								<X>5</X>
    								<Y>5</Y>
    								<Width>${ImageWidth}</Width>
    								<Height>${ImageHeight}</Height>
    								<!-- horiz: left, right, hcenter - vert: top, bottom, vcenter -->
    								<Align>left,top</Align>
    							</Location>
    						</Overlay>
    					</Overlays>
    					<Parameters>
    					</Parameters>
    				</Video>
    				<Audio>
    					<!-- AAC, PassThru, Disable -->
    					<Codec>PassThru</Codec>
    					<Bitrate>${SourceAudioBitrate}</Bitrate>
    				</Audio>
    				<Properties>
    				</Properties>
    			</Encode>
    			<!-- Setup for 360p, high bandwith, main profile for desktop -->
    			<Encode>
    				<Enable>true</Enable>
    				<Name>360p</Name>
    				<StreamName>mp4:${SourceStreamName}_360p</StreamName>
    				<Video>
    					<!-- H.264, PassThru, Disable -->
    					<Codec>H.264</Codec>
    					<!-- default, CUDA, QuickSync -->
    					<Transcoder>default</Transcoder>
    					<GPUID>-1</GPUID>
    					<FrameSize>
    						<!-- letterbox, fit-width, fit-height, crop, stretch, match-source -->
    						<FitMode>fit-height</FitMode>
    						<Width>640</Width>
    						<Height>360</Height>
    						<!-- <Crop>0,0,0,0</Crop> -->
    						<!-- <SourceRectangle>0,0,320,240</SourceRectangle> -->
    					</FrameSize>
    					<Profile>main</Profile>
    					<Bitrate>850000</Bitrate>
    					<KeyFrameInterval>
    						<FollowSource>true</FollowSource>
    						<Interval>60</Interval>
    					</KeyFrameInterval>
    					<Overlays>
    						<Overlay>
    							<Enable>false</Enable>
    							<Index>0</Index>
    							<ImagePath>${com.wowza.wms.context.VHostConfigHome}/content/wowzalogo.png</ImagePath>
    							<Opacity>100</Opacity>
    							<Location>
    								<X>5</X>
    								<Y>5</Y>
    								<Width>${ImageWidth}</Width>
    								<Height>${ImageHeight}</Height>
    								<!-- horiz: left, right, hcenter - vert: top, bottom, vcenter -->
    								<Align>left,top</Align>
    							</Location>
    						</Overlay>
    					</Overlays>
    					<Parameters>
    					</Parameters>
    				</Video>
    				<Audio>
    					<!-- AAC, PassThru, Disable -->
    					<Codec>PassThru</Codec>
    					<Bitrate>${SourceAudioBitrate}</Bitrate>
    				</Audio>
    				<Properties>
    				</Properties>
    			</Encode>
    			<!-- Setup for 240p, medium bandwidth, baseline profile -->
    			<Encode>
    				<Enable>true</Enable>
    				<Name>240p</Name>
    				<StreamName>mp4:${SourceStreamName}_240p</StreamName>
    				<Video>
    					<!-- H.264, PassThru, Disable -->
    					<Codec>H.264</Codec>
    					<!-- default, CUDA, QuickSync -->
    					<Transcoder>default</Transcoder>
    					<GPUID>-1</GPUID>
    					<FrameSize>
    						<!-- letterbox, fit-width, fit-height, crop, stretch, match-source  -->
    						<FitMode>fit-height</FitMode>
    						<Width>360</Width>
    						<Height>240</Height>
    						<!-- <Crop>0,0,0,0</Crop> -->
    						<!-- <SourceRectangle>0,0,320,240</SourceRectangle> -->
    					</FrameSize>
    					<Profile>baseline</Profile>
    					<Bitrate>350000</Bitrate>
    					<KeyFrameInterval>
    						<FollowSource>true</FollowSource>
    						<Interval>60</Interval>
    					</KeyFrameInterval>
    					<Overlays>
    						<Overlay>
    							<Enable>false</Enable>
    							<Index>0</Index>
    							<ImagePath>${com.wowza.wms.context.VHostConfigHome}/content/wowzalogo.png</ImagePath>
    							<Opacity>100</Opacity>
    							<Location>
    								<X>5</X>
    								<Y>5</Y>
    								<Width>${ImageWidth}</Width>
    								<Height>${ImageHeight}</Height>
    								<!-- horiz: left, right, hcenter - vert: top, bottom, vcenter -->
    								<Align>left,top</Align>
    							</Location>
    						</Overlay>
    					</Overlays>
    					<Parameters>
    					</Parameters>
    				</Video>
    				<Audio>
    					<!-- AAC, PassThru, Disable -->
    					<Codec>PassThru</Codec>
    					<Bitrate>${SourceAudioBitrate}</Bitrate>
    				</Audio>
    				<Properties>
    				</Properties>
    			</Encode>
    			<!-- Setup for 160p, low bandwith, baseline profile for 3G mobile devices such as iOS, Android, Blackberry -->
    			<Encode>
    				<Enable>true</Enable>
    				<Name>160p</Name>
    				<StreamName>mp4:${SourceStreamName}_160p</StreamName>
    				<Video>
    					<!-- H.264, PassThru, Disable -->
    					<Codec>H.264</Codec>
    					<!-- default, CUDA, QuickSync -->
    					<Transcoder>default</Transcoder>
    					<GPUID>-1</GPUID>
    					<FrameSize>
    						<!-- letterbox, fit-width, fit-height, crop, stretch, match-source  -->
    						<FitMode>fit-height</FitMode>
    						<Width>284</Width>
    						<Height>160</Height>
    						<!-- <Crop>0,0,0,0</Crop> -->
    						<!-- <SourceRectangle>0,0,320,240</SourceRectangle> -->
    					</FrameSize>
    					<Profile>baseline</Profile>
    					<Bitrate>200000</Bitrate>
    					<KeyFrameInterval>
    						<FollowSource>true</FollowSource>
    						<Interval>60</Interval>
    					</KeyFrameInterval>
    					<Overlays>
    						<Overlay>
    							<Enable>false</Enable>
    							<Index>0</Index>
    							<ImagePath>${com.wowza.wms.context.VHostConfigHome}/content/wowzalogo.png</ImagePath>
    							<Opacity>100</Opacity>
    							<Location>
    								<X>5</X>
    								<Y>5</Y>
    								<Width>${ImageWidth}</Width>
    								<Height>${ImageHeight}</Height>
    								<!-- horiz: left, right, hcenter - vert: top, bottom, vcenter -->
    								<Align>left,top</Align>
    							</Location>
    						</Overlay>
    					</Overlays>
    					<Parameters>
    					</Parameters>
    				</Video>
    				<Audio>
    					<!-- AAC, PassThru, Disable -->
    					<Codec>PassThru</Codec>
    					<Bitrate>${SourceAudioBitrate}</Bitrate>
    				</Audio>
    				<Properties>
    				</Properties>
    			</Encode>
    		</Encodes>
    		<Decode>
    			<Video>
    				<Deinterlace>false</Deinterlace>
    				<Overlays>
    					<Overlay>
    						<Enable>false</Enable>
    						<Index>0</Index>
    						<ImagePath>${com.wowza.wms.context.VHostConfigHome}/content/wowzalogo.png</ImagePath>
    						<Opacity>100</Opacity>
    						<Location>
    							<X>5</X>
    							<Y>5</Y>
    							<Width>${ImageWidth}</Width>
    							<Height>${ImageHeight}</Height>
    							<!-- horiz: left, right, hcenter - vert: top, bottom, vcenter -->
    							<Align>left,top</Align>
    						</Location>
    					</Overlay>
    				</Overlays>
    				<Parameters>
    				</Parameters>
    			</Video>
    			<Properties>
    			</Properties>
    		</Decode>
    		<StreamNameGroups>
    			<!-- Note: Play stream using stream name ngrp:[stream-name] -->
    			<StreamNameGroup>
    				<Name>${SourceStreamName}_all</Name>
    				<Members>
    					<Member>
    						<EncodeName>source</EncodeName>
    					</Member>
    					<Member>
    						<EncodeName>360p</EncodeName>
    					</Member>
    					<Member>
    						<EncodeName>240p</EncodeName>
    					</Member>
    					<Member>
    						<EncodeName>160p</EncodeName>
    					</Member>
    				</Members>
    			</StreamNameGroup>
    			<StreamNameGroup>
    				<Name>${SourceStreamName}_mobile</Name>
    				<Members>
    					<Member>
    						<EncodeName>240p</EncodeName>
    					</Member>
    					<Member>
    						<EncodeName>160p</EncodeName>
    					</Member>
    				</Members>
    			</StreamNameGroup>
    		</StreamNameGroups>
    		<Properties>
    			<Property>
    				<Name>sourceStreamFrameRate</Name>
    				<Value>30</Value>
    				<Type>Double</Type>
    			</Property>
    		</Properties>
    	</Transcode>
    </Root>
    Environment

    Operating System:*
    Windows Server 2008 R2 6.1
    Architecture:*
    amd64
    Number of processors:*
    16
    Committed virtual memory:*
    3,591,384 kbytes
    Total physical memory:*
    25,152,600 kbytes
    Free physical memory:*
    17,560,500 kbytes
    Total swap space:*
    51,122,540 kbytes
    Free swap space:*
    43,956,992 kbytes
    
    
    using
    Virtual Machine:*
    Java HotSpot(TM) 64-Bit Server VM version 23.0-b21
    
    System CPU around 15-25% while transcoding two streams to 3 lower bitrates each
    Bascially, we see memory use grow by 1GB over 3 days. If we were to run the system for a month without restart, we fear that we will run out of heap space. Is what we are seeing normal and will this graph level off at some point? I have not seen memory use decrease over time.

  9. #9
    Join Date
    Sep 2011
    Posts
    1,933

    Default

    Hi,
    I can also see in the [Wowza-Install]/conf/VHost.xml you have configured the ProcessorCount as 16, if the server has 16 processors available this should be configured as 24 (the maximum).
    In the tuning guide you will see how the [total-core-count] should be calculated for tuning:

    Note: The HostPort/ProcessorCount field in the Admin HostPort (/Port "8086") should not be modified.
    HostPort/ProcessorCount: 2x[total-core-count] (maximum of 24)
    IdleWorkers/WorkerCount: 2x[total-core-count] (maximum of 24)
    
    NetConnections/ProcessorCount: 2x[total-core-count]  (maximum of 24)
    
    RTP/UnicastIncoming/ProcessorCount: [total-core-count]  (maximum of 12)
    RTP/UnicastOutgoing/ProcessorCount: 2x[total-core-count]  (maximum of 24)
    
    RTP/MulticastIncoming/ProcessorCount: [total-core-count]  (maximum of 12)
    RTP/MulticastOutgoing/ProcessorCount: [total-core-count]  (maximum of 12)
    
    HandlerThreadPool/PoolSize: (60x[total-core-count]) (maximum of 480)
    TransportThreadPool/PoolSize: (40x[total-core-count]) (maximum of 320)
    Jason
    thank you Jason - I will change the settings, just in time for our patch reboot for MS patches tonight.
    It looks like you didn't adjust the [Wowza-Install]/conf/VHost.xml correctly, this should be 12 or 24 which is the maximum as shown above.
    I can see you edited the Admin HostPort which should be 1 as this is not suppose to be changed.
    Note: The HostPort/ProcessorCount field in the Admin HostPort (/Port "8086") should not be modified.
    I can see you've enabled port 443 and this section should be configured the same as the other ports, currently the ProcessorCount = 4, this should be 24.
    I've linked the tuning guide below for reference,
    Tuning guide

    If you can't find any indication to the issue in the logs and the problem persists after tuning correctly can restarting then I recommend that you open a ticket by sending an email to support@wowza.com
    When opening a ticket please ensure that you give a detailed description of the issue, attach a zipped copy of the [Wowza-Install]/conf and [Wowza-Install]/logs directories for analysis and create a link to this thread for reference.

    Thanks
    Jason

Similar Threads

  1. Memory Usage increases to 100 % over time (Streaming Engine 4.1.0 on EC2 instance)
    By niemion in forum Wowza Streaming Engine in the Cloud
    Replies: 9
    Last Post: 06-20-2016, 02:50 PM
  2. Memory increase over time
    By jordanrumph in forum Performance Tuning Discussion
    Replies: 10
    Last Post: 03-20-2015, 12:59 PM
  3. Wowza Heap and Memory is getting Full over time
    By oomworld in forum General Forum
    Replies: 5
    Last Post: 07-11-2014, 03:36 AM
  4. Replies: 7
    Last Post: 06-06-2014, 09:55 PM
  5. Memory increase over time
    By sykwon in forum Performance Tuning Discussion
    Replies: 1
    Last Post: 02-13-2014, 09:38 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •