Results 1 to 10 of 10

Thread: Memory usage drastically increased in "onPublish"

  1. #1
    Join Date
    Dec 2013
    Posts
    5

    Default Memory usage drastically increased in "onPublish"

    Hello, everyone.

    I encountered a strange behavior in "onPublish" method.

    If a process in the method is forced to wait for something (like acquiring data from a database),
    JVM heap memory usage continuously rise until it is unlocked.

    It is rare that processes become waited, but if it happens, the Wowza server could raise OutOfMemory exceptions.

    Does anyone know a way around this obstacle?
    I'll leave sample codes below to reproduce this phenomena.

    Thanks.


    import com.wowza.wms.logging.WMSLogger;
    import com.wowza.wms.logging.WMSLoggerFactory;
    import com.wowza.wms.module.ModuleBase;
    import com.wowza.wms.stream.IMediaStream;
    import com.wowza.wms.stream.IMediaStreamActionNotify;
    
    public class Wowza3TestApplication extends ModuleBase {
    	public void onStreamCreate(IMediaStream stream) {
    		stream.addClientListener(new IMediaStreamActionNotify() {
    			@Override
    			public void onPublish(IMediaStream paramIMediaStream, String paramString,
    					boolean paramBoolean1, boolean paramBoolean2) {
    				WMSLoggerFactory.getLogger(null).warn("onPublish sleep start");
    				try {
    					// sleep instead of waiting for a database response
    					Thread.sleep(20000L);
    				} catch (InterruptedException e) {
    					WMSLoggerFactory.getLogger(null).warn("", e);
    				}
    				WMSLoggerFactory.getLogger(null).warn("onPublish sleep end");
    			}
    			
    			@Override
    			public void onUnPublish(IMediaStream paramIMediaStream, String paramString,
    					boolean paramBoolean1, boolean paramBoolean2) {
    			}
    			
    			@Override
    			public void onStop(IMediaStream paramIMediaStream) {
    			}
    			
    			@Override
    			public void onSeek(IMediaStream paramIMediaStream, double paramDouble) {
    			}
    			
    			@Override
    			public void onPlay(IMediaStream paramIMediaStream, String paramString,
    					double paramDouble1, double paramDouble2, int paramInt) {
    			}
    			
    			@Override
    			public void onPause(IMediaStream paramIMediaStream, boolean paramBoolean,
    					double paramDouble) {
    			}
    		});
    	}
    }
    Last edited by T.Kouno; 12-16-2013 at 12:47 AM. Reason: update java coce

  2. #2

    Default

    Hi,

    I have been testing this and can not see any more drastic run up compared to a normal publish. When GC comes along to purge old objects memory is freed. In my tests I extended your time by x100 to let it run and GC managed memory as expected.

    I would make sure you have production GC settings and your Wowza server is tuned.

    Andrew.

  3. #3
    Join Date
    Dec 2013
    Posts
    5

    Default

    Thank you for your reply.

    I've already tuned "setenv.sh" and "VHost.xml", but the heap usage continuously rises like following image when using the Java codes I posted.
    The usage amount doesn't go down even if I push the GC button.




    Here are a execution environment and contents of the two files (setenv.sh and VHost.xml) which I tested.
    Could you please give me advice?

    Thanks.


    Machine spec: m1.small of AWS EC2. (1 vCPU, 1.7GB memory)
    OS: Ubuntu 12.04 LTS (GNU/Linux 3.2.0-26-virtual x86_64)
    Java version: 1.6.0_45 (for Linux x64)
    Wowza version: 3.6.3

    setenv.sh
    #!/bin/sh
    
    _EXECJAVA=java
    JAVA_OPTS="-server -Xmx1200M"
    
    # Better garbage collection setting to avoid long pauses
    #JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:NewSize=512m"
    JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:NewSize=128m"
    
    # Uncomment to log debug garbage collection information
    #NOW=$(date +"%Y-%m-%d-%H-%M")
    #JAVA_OPTS="$JAVA_OPTS -verbose:gc -Xloggc:"/usr/local/WowzaMediaServer/logs/gc_$NOW.log" -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime"
    
    # Uncomment to write heap dump on OutOfMemoryError
    #JAVA_OPTS="$JAVA_OPTS -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/local/WowzaMediaServer"
    
    # Uncomment to force Java to use specific language settings
    #JAVA_OPTS="$JAVA_OPTS -Duser.language=en -Duser.country=US -Dfile.encoding=Cp1252"
    
    # Uncomment to fix multicast crosstalk problem when streams share multicast port
    JAVA_OPTS="$JAVA_OPTS -Djava.net.preferIPv4Stack=true"
    
    WMSAPP_HOME=/usr/local/WowzaMediaServer
    WMSCONFIG_HOME=/usr/local/WowzaMediaServer
    WMSCONFIG_URL=
    
    export WMSAPP_HOME WMSCONFIG_HOME JAVA_OPTS _EXECJAVA

    VHost.xml
    <Root>
            <VHost>
                    <HostPortList>
                            <HostPort>
                                    <ProcessorCount>2</ProcessorCount>
                                    <IpAddress>*</IpAddress>
                                    <!-- Separate multiple ports with commas -->
                                    <!-- 80: HTTP, RTMPT -->
                                    <!-- 554: RTSP -->
                                    <Port>1935</Port>
                                    <HTTPIdent2Response></HTTPIdent2Response>
                                    <SocketConfiguration>
                                            <ReuseAddress>true</ReuseAddress>
                                            <!-- suggested settings for video on demand applications -->
                                            <ReceiveBufferSize>65000</ReceiveBufferSize>
                                            <ReadBufferSize>65000</ReadBufferSize>
                                            <SendBufferSize>65000</SendBufferSize>
                                            <!-- suggest settings for low latency chat and video recording applications
                                            <ReceiveBufferSize>32000</ReceiveBufferSize>
                                            <ReadBufferSize>32000</ReadBufferSize>
                                            <SendBufferSize>32000</SendBufferSize>
                                            -->
                                            <KeepAlive>true</KeepAlive>
                                            <!-- <TrafficClass>0</TrafficClass> -->
                                            <!-- <OobInline>false</OobInline> -->
                                            <!-- <SoLingerTime>-1</SoLingerTime> -->
                                            <!-- <TcpNoDelay>false</TcpNoDelay> -->
                                            <AcceptorBackLog>100</AcceptorBackLog>
                                    </SocketConfiguration>
                                    <HTTPStreamerAdapterIDs>cupertinostreaming,smoothstreaming,sanjosestreaming,dvrchunkstreaming,mpegdashstreaming</HTTPStreamerAdapterIDs>
                                    <HTTPProviders>
                                            <HTTPProvider>
                                                    <BaseClass>com.wowza.wms.http.HTTPCrossdomain</BaseClass>
                                                    <RequestFilters>*crossdomain.xml</RequestFilters>
                                                    <AuthenticationMethod>none</AuthenticationMethod>
                                            </HTTPProvider>
                                            <HTTPProvider>
                                                    <BaseClass>com.wowza.wms.http.HTTPClientAccessPolicy</BaseClass>
                                                    <RequestFilters>*clientaccesspolicy.xml</RequestFilters>
                                                    <AuthenticationMethod>none</AuthenticationMethod>
                                            </HTTPProvider>
                                            <HTTPProvider>
                                                    <BaseClass>com.wowza.wms.http.HTTPProviderMediaList</BaseClass>
                                                    <RequestFilters>*jwplayer.rss|*jwplayer.smil|*medialist.smil|*manifest-rtmp.f4m</RequestFilters>
                                                    <AuthenticationMethod>none</AuthenticationMethod>
                                            </HTTPProvider>
                                            <HTTPProvider>
                                                    <BaseClass>com.wowza.wms.http.HTTPServerVersion</BaseClass>
                                                    <RequestFilters>*</RequestFilters>
                                                    <AuthenticationMethod>none</AuthenticationMethod>
                                            </HTTPProvider>
                                    </HTTPProviders>
                            </HostPort>
    
                            <!-- 443 with SSL -->
                            <!--
                            <HostPort>
                                    <ProcessorCount>4</ProcessorCount>
                                    <IpAddress>*</IpAddress>
                                    <Port>443</Port>
                                    <HTTPIdent2Response></HTTPIdent2Response>
                                    <SSLConfig>
                                            <KeyStorePath>${com.wowza.wms.context.VHostConfigHome}/conf/keystore.jks</KeyStorePath>
                                            <KeyStorePassword>[password]</KeyStorePassword>
                                            <KeyStoreType>JKS</KeyStoreType>
                                            <SSLProtocol>TLS</SSLProtocol>
                                            <Algorithm>SunX509</Algorithm>
                                            <CipherSuites></CipherSuites>
                                            <Protocols></Protocols>
                                    </SSLConfig>
                                    <SocketConfiguration>
                                            <ReuseAddress>true</ReuseAddress>
                                            <ReceiveBufferSize>65000</ReceiveBufferSize>
                                            <ReadBufferSize>65000</ReadBufferSize>
                                            <SendBufferSize>65000</SendBufferSize>
                                            <KeepAlive>true</KeepAlive>
                                            <AcceptorBackLog>100</AcceptorBackLog>
                                    </SocketConfiguration>
                                    <HTTPStreamerAdapterIDs>cupertinostreaming,smoothstreaming,sanjosestreaming,dvrchunkstreaming,mpegdashstreaming</HTTPStreamerAdapterIDs>
                                    <HTTPProviders>
                                            <HTTPProvider>
                                                    <BaseClass>com.wowza.wms.http.HTTPCrossdomain</BaseClass>
                                                    <RequestFilters>*crossdomain.xml</RequestFilters>
                                                    <AuthenticationMethod>none</AuthenticationMethod>
                                            </HTTPProvider>
                                            <HTTPProvider>
                                                    <BaseClass>com.wowza.wms.http.HTTPClientAccessPolicy</BaseClass>
                                                    <RequestFilters>*clientaccesspolicy.xml</RequestFilters>
                                                    <AuthenticationMethod>none</AuthenticationMethod>
                                            </HTTPProvider>
                                            <HTTPProvider>
                                                    <BaseClass>com.wowza.wms.http.HTTPProviderMediaList</BaseClass>
                                                    <RequestFilters>*jwplayer.rss|*jwplayer.smil|*medialist.smil|*manifest-rtmp.f4m</RequestFilters>
                                                    <AuthenticationMethod>none</AuthenticationMethod>
                                            </HTTPProvider>
                                            <HTTPProvider>
                                                    <BaseClass>com.wowza.wms.http.HTTPServerVersion</BaseClass>
                                                    <RequestFilters>*</RequestFilters>
                                                    <AuthenticationMethod>none</AuthenticationMethod>
                                            </HTTPProvider>
                                    </HTTPProviders>
                            </HostPort>
                            -->
    
                            <!-- Admin HostPort -->
                            <HostPort>
                                    <ProcessorCount>2</ProcessorCount>
                                    <IpAddress>*</IpAddress>
                                    <Port>8086</Port>
                                    <HTTPIdent2Response></HTTPIdent2Response>
                                    <SocketConfiguration>
                                            <ReuseAddress>true</ReuseAddress>
                                            <ReceiveBufferSize>16000</ReceiveBufferSize>
                                            <ReadBufferSize>16000</ReadBufferSize>
                                            <SendBufferSize>16000</SendBufferSize>
                                            <KeepAlive>true</KeepAlive>
                                            <AcceptorBackLog>100</AcceptorBackLog>
                                    </SocketConfiguration>
                                    <HTTPStreamerAdapterIDs></HTTPStreamerAdapterIDs>
                                    <HTTPProviders>
                                            <HTTPProvider>
                                                    <BaseClass>com.wowza.wms.http.streammanager.HTTPStreamManager</BaseClass>
                                                    <RequestFilters>streammanager*</RequestFilters>
                                                    <AuthenticationMethod>admin-digest</AuthenticationMethod>
                                            </HTTPProvider>
                                            <HTTPProvider>
                                                    <BaseClass>com.wowza.wms.http.HTTPServerInfoXML</BaseClass>
                                                    <RequestFilters>serverinfo*</RequestFilters>
                                                    <AuthenticationMethod>admin-digest</AuthenticationMethod>
                                            </HTTPProvider>
                                            <HTTPProvider>
                                                    <BaseClass>com.wowza.wms.http.HTTPConnectionInfo</BaseClass>
                                                    <RequestFilters>connectioninfo*</RequestFilters>
                                                    <AuthenticationMethod>admin-digest</AuthenticationMethod>
                                            </HTTPProvider>
                                            <HTTPProvider>
                                                    <BaseClass>com.wowza.wms.http.HTTPConnectionCountsXML</BaseClass>
                                                    <RequestFilters>connectioncounts*</RequestFilters>
                                                    <AuthenticationMethod>admin-digest</AuthenticationMethod>
                                            </HTTPProvider>
                                            <HTTPProvider>
                                                    <BaseClass>com.wowza.wms.transcoder.httpprovider.HTTPTranscoderThumbnail</BaseClass>
                                                    <RequestFilters>transcoderthumbnail*</RequestFilters>
                                                    <AuthenticationMethod>admin-digest</AuthenticationMethod>
                                            </HTTPProvider>
                                            <HTTPProvider>
                                                    <BaseClass>com.wowza.wms.http.HTTPProviderMediaList</BaseClass>
                                                    <RequestFilters>medialist*</RequestFilters>
                                                    <AuthenticationMethod>admin-digest</AuthenticationMethod>
                                            </HTTPProvider>
                                            <HTTPProvider>
                                                    <BaseClass>com.wowza.wms.livestreamrecord.http.HTTPLiveStreamRecord</BaseClass>
                                                    <RequestFilters>livestreamrecord*</RequestFilters>
                                                    <AuthenticationMethod>admin-digest</AuthenticationMethod>
                                            </HTTPProvider>
                                            <HTTPProvider>
                                                    <BaseClass>com.wowza.wms.http.HTTPServerVersion</BaseClass>
                                                    <RequestFilters>*</RequestFilters>
                                                    <AuthenticationMethod>none</AuthenticationMethod>
                                            </HTTPProvider>
                                    </HTTPProviders>
                            </HostPort>
    
                    </HostPortList>
    
                    <HTTPStreamerAdapters>
                            <HTTPStreamerAdapter>
                                    <ID>smoothstreaming</ID>
                                    <Name>smoothstreaming</Name>
                                    <Properties>
                                    </Properties>
                            </HTTPStreamerAdapter>
                            <HTTPStreamerAdapter>
                                    <ID>cupertinostreaming</ID>
                                    <Name>cupertinostreaming</Name>
                                    <Properties>
                                    </Properties>
                            </HTTPStreamerAdapter>
                            <HTTPStreamerAdapter>
                                    <ID>sanjosestreaming</ID>
                                    <Name>sanjosestreaming</Name>
                                    <Properties>
                                    </Properties>
                            </HTTPStreamerAdapter>
                            <HTTPStreamerAdapter>
                                    <ID>dvrchunkstreaming</ID>
                                    <Name>dvrchunkstreaming</Name>
                                    <Properties>
                                    </Properties>
                            </HTTPStreamerAdapter>
                            <HTTPStreamerAdapter>
                                    <ID>mpegdashstreaming</ID>
                                    <Name>mpegdashstreaming</Name>
                                    <Properties>
                                    </Properties>
                            </HTTPStreamerAdapter>
                    </HTTPStreamerAdapters>
    
                    <HandlerThreadPool>
                            <PoolSize>60</PoolSize>
                    </HandlerThreadPool>
                    <TransportThreadPool>
                            <PoolSize>40</PoolSize>
                    </TransportThreadPool>
                    <IdleWorkers>
                            <WorkerCount>4</WorkerCount>
                            <CheckFrequency>50</CheckFrequency>
                            <MinimumWaitTime>5</MinimumWaitTime>
                    </IdleWorkers>
                    <NetConnections>
                            <ProcessorCount>2</ProcessorCount>
                            <IdleFrequency>250</IdleFrequency>
                            <SocketConfiguration>
                                    <ReuseAddress>true</ReuseAddress>
                                    <ReceiveBufferSize>65000</ReceiveBufferSize>
                                    <ReadBufferSize>65000</ReadBufferSize>
                                    <SendBufferSize>65000</SendBufferSize>
                                    <KeepAlive>true</KeepAlive>
                                    <!-- <TrafficClass>0</TrafficClass> -->
                                    <!-- <OobInline>false</OobInline> -->
                                    <!-- <SoLingerTime>-1</SoLingerTime> -->
                                    <!-- <TcpNoDelay>false</TcpNoDelay> -->
                                    <AcceptorBackLog>100</AcceptorBackLog>
                            </SocketConfiguration>
                    </NetConnections>
                    <HTTPTunnel>
                            <KeepAliveTimeout>2000</KeepAliveTimeout>
                    </HTTPTunnel>
                    <Client>
                            <ClientTimeout>90000</ClientTimeout>
                            <IdleFrequency>250</IdleFrequency>
                    </Client>
                    <!-- RTP/Authentication/Methods defined in Authentication.xml. Default setup includes; none, basic, digest -->
                    <RTP>
                            <IdleFrequency>75</IdleFrequency>
                            <DatagramConfiguration>
                                    <Incoming>
                                            <ReuseAddress>true</ReuseAddress>
                                            <ReceiveBufferSize>1024000</ReceiveBufferSize>
                                            <SendBufferSize>65000</SendBufferSize>
                                            <!-- <MulticastBindToAddress>true</MulticastBindToAddress> -->
                                            <!-- <MulticastInterfaceAddress>192.168.1.22</MulticastInterfaceAddress> -->
                                            <!-- <TrafficClass>0</TrafficClass> -->
                                            <MulticastTimeout>50</MulticastTimeout>
                                            <DatagramMaximumPacketSize>4096</DatagramMaximumPacketSize>
                                    </Incoming>
                                    <Outgoing>
                                            <ReuseAddress>true</ReuseAddress>
                                            <ReceiveBufferSize>65000</ReceiveBufferSize>
                                            <SendBufferSize>65000</SendBufferSize>
                                            <!-- <MulticastBindToAddress>true</MulticastBindToAddress> -->
                                            <!-- <MulticastInterfaceAddress>192.168.1.22</MulticastInterfaceAddress> -->
                                            <!-- <TrafficClass>0</TrafficClass> -->
                                            <MulticastTimeout>50</MulticastTimeout>
                                            <DatagramMaximumPacketSize>4096</DatagramMaximumPacketSize>
                                    </Outgoing>
                            </DatagramConfiguration>
                            <UnicastIncoming>
                                    <ProcessorCount>4</ProcessorCount>
                            </UnicastIncoming>
                            <UnicastOutgoing>
                                    <ProcessorCount>8</ProcessorCount>
                            </UnicastOutgoing>
                            <MulticastIncoming>
                                    <ProcessorCount>4</ProcessorCount>
                            </MulticastIncoming>
                            <MulticastOutgoing>
                                    <ProcessorCount>4</ProcessorCount>
                            </MulticastOutgoing>
                    </RTP>
                    <Application>
                            <ApplicationTimeout>60000</ApplicationTimeout>
                            <PingTimeout>12000</PingTimeout>
                            <UnidentifiedSessionTimeout>30000</UnidentifiedSessionTimeout>
                            <ValidationFrequency>20000</ValidationFrequency>
                            <MaximumPendingWriteBytes>0</MaximumPendingWriteBytes>
                            <MaximumSetBufferTime>60000</MaximumSetBufferTime>
                    </Application>
                    <StartStartupStreams>true</StartStartupStreams>
                    <!-- Properties defined here will be added to the IVHost.getProperties() collection -->
                    <Properties>
                    </Properties>
            </VHost>
    </Root>
    Last edited by T.Kouno; 12-13-2013 at 01:55 AM.

  4. #4

    Default

    Hi,

    I would try a normal publish to see how it compares. Do remember GC is not immediate so will only happen if objects can be recovered at that time.

    From my testing a normal and a delayed publish, I extended it by x1000, did not show a massive difference and the publisher timed out waiting.

    Andrew.

  5. #5
    Join Date
    Dec 2013
    Posts
    5

    Default

    Hi.

    I tested this module on another machine, then the same phenomenon occurred.

    I would like to confirm that your server received proper amount of streaming data (500kbps or more) published from your client.

    And I also would like to confirm whether you saw a text "onPublish sleep start" in your wowzamediaserver_access.log during your test.
    If you did, please show me your configurations and environments. I'll follow them.
    If not, please register your custom module in Application.xml, and retry it.



    There are additional information about my codes and configurations.

    • Java code: I updated the former code. Import declarations were added in case.
    • protocol: RTMP
    • Application name: live
    • StreamType: live
    • Application.xml:

    <Root>
    	<Application>
    		<!-- Uncomment to set application level timeout values
    		<ApplicationTimeout>60000</ApplicationTimeout>
    		<PingTimeout>12000</PingTimeout>
    		<ValidationFrequency>8000</ValidationFrequency>
    		<MaximumPendingWriteBytes>0</MaximumPendingWriteBytes>
    		<MaximumSetBufferTime>60000</MaximumSetBufferTime>
    		<MaximumStorageDirDepth>25</MaximumStorageDirDepth>
    		-->
    		<Connections>
    			<AutoAccept>true</AutoAccept>
    			<AllowDomains></AllowDomains>
    		</Connections>
    		<!--
    			StorageDir path variables
    			
    			${com.wowza.wms.AppHome} - Application home directory
    			${com.wowza.wms.ConfigHome} - Configuration home directory
    			${com.wowza.wms.context.VHost} - Virtual host name
    			${com.wowza.wms.context.VHostConfigHome} - Virtual host config directory
    			${com.wowza.wms.context.Application} - Application name
    			${com.wowza.wms.context.ApplicationInstance} - Application instance name
    			
    		-->
    		<Streams>
    			<StreamType>live</StreamType>
    			<StorageDir>${com.wowza.wms.context.VHostConfigHome}/content</StorageDir>
    			<KeyDir>${com.wowza.wms.context.VHostConfigHome}/keys</KeyDir>
    			<!-- LiveStreamPacketizers (separate with commas): cupertinostreamingpacketizer, smoothstreamingpacketizer, sanjosestreamingpacketizer, mpegdashstreamingpacketizer, cupertinostreamingrepeater, smoothstreamingrepeater, sanjosestreamingrepeater, dvrstreamingpacketizer, dvrstreamingrepeater -->
    			<LiveStreamPacketizers></LiveStreamPacketizers>
    			<!-- Properties defined here will override any properties defined in conf/Streams.xml for any streams types loaded by this application -->
    			<Properties>
    			</Properties>
    		</Streams>
    		<Transcoder>
    			<!-- To turn on transcoder set to: transcoder -->
    			<LiveStreamTranscoder></LiveStreamTranscoder>
    			<!-- [templatename].xml or ${SourceStreamName}.xml -->
    			<Templates>${SourceStreamName}.xml,transrate.xml</Templates>
    			<ProfileDir>${com.wowza.wms.context.VHostConfigHome}/transcoder/profiles</ProfileDir>
    			<TemplateDir>${com.wowza.wms.context.VHostConfigHome}/transcoder/templates</TemplateDir>
    			<Properties>
    			</Properties>
    		</Transcoder>
    
    		<DVR>
    			<!-- As a single server or as an origin, use dvrstreamingpacketizer in LiveStreamPacketizers above -->
    			<!-- Or, in an origin-edge configuration, edges use dvrstreamingrepeater in LiveStreamPacketizers above -->
    			<!-- As an origin, also add dvrchunkstreaming to HTTPStreamers below -->
    
    			<!-- If this is a dvrstreamingrepeater, define Application/Repeater/OriginURL to point back to the origin -->
    			
    			<!-- To turn on DVR recording set Recorders to dvrrecorder.  This works with dvrstreamingpacketizer  -->
    			<Recorders></Recorders>
    
    			<!-- As a single server or as an origin, set the Store to dvrfilestorage-->
    			<!-- edges should have this empty -->
    			<Store></Store>
    
    			<!--  Window Duration is length of live DVR window in seconds.  0 means the window is never trimmed. -->
    			<WindowDuration>0</WindowDuration>
    
    			<!-- Storage Directory is top level location where dvr is stored.  e.g. c:/temp/dvr -->
    			<StorageDir>${com.wowza.wms.context.VHostConfigHome}/dvr</StorageDir>
    
    			<!-- valid ArchiveStrategy values are append, version, delete -->
    			<ArchiveStrategy>append</ArchiveStrategy>
    
    			<!-- Properties for DVR -->
    			<Properties>
    			</Properties>
    		</DVR>
    
    		<TimedText>
    			<!-- VOD caption providers (separate with commas): vodcaptionprovidermp4_3gpp, vodcaptionproviderttml, vodcaptionproviderwebvtt,  vodcaptionprovidersrt, vodcaptionproviderscc -->
    			<VODTimedTextProviders>vodcaptionprovidermp4_3gpp</VODTimedTextProviders>
    			
    			<!-- Properties for TimedText -->
    			<Properties>
    			</Properties>		
    		</TimedText>
    		
    		<!-- HTTPStreamers (separate with commas): cupertinostreaming, smoothstreaming, sanjosestreaming, mpegdashstreaming, dvrchunkstreaming -->
    		<HTTPStreamers>cupertinostreaming,smoothstreaming,sanjosestreaming</HTTPStreamers>
    		<SharedObjects>
    			<StorageDir></StorageDir>
    		</SharedObjects>
    		<Client>
    			<IdleFrequency>-1</IdleFrequency>
    			<Access>
    				<StreamReadAccess>*</StreamReadAccess>
    				<StreamWriteAccess>*</StreamWriteAccess>
    				<StreamAudioSampleAccess></StreamAudioSampleAccess>
    				<StreamVideoSampleAccess></StreamVideoSampleAccess>
    				<SharedObjectReadAccess>*</SharedObjectReadAccess>
    				<SharedObjectWriteAccess>*</SharedObjectWriteAccess>
    			</Access>
    		</Client>
    		<RTP>
    			<!-- RTP/Authentication/[type]Methods defined in Authentication.xml. Default setup includes; none, basic, digest -->
    			<Authentication>
    				<PublishMethod>digest</PublishMethod>
    				<PlayMethod>none</PlayMethod>
    			</Authentication>
    			<!-- RTP/AVSyncMethod. Valid values are: senderreport, systemclock, rtptimecode -->
    			<AVSyncMethod>senderreport</AVSyncMethod>
    			<MaxRTCPWaitTime>12000</MaxRTCPWaitTime>
    			<IdleFrequency>75</IdleFrequency>
    			<RTSPSessionTimeout>90000</RTSPSessionTimeout>
    			<RTSPMaximumPendingWriteBytes>0</RTSPMaximumPendingWriteBytes>
    			<RTSPBindIpAddress></RTSPBindIpAddress>
    			<RTSPConnectionIpAddress>0.0.0.0</RTSPConnectionIpAddress>
    			<RTSPOriginIpAddress>127.0.0.1</RTSPOriginIpAddress>
    			<IncomingDatagramPortRanges>*</IncomingDatagramPortRanges>
    			<!-- Properties defined here will override any properties defined in conf/RTP.xml for any depacketizers loaded by this application -->
    			<Properties>
    			</Properties>
    		</RTP>
    		<MediaCaster>
    			<RTP>
    				<RTSP>
    					<!-- udp, interleave -->
    					<RTPTransportMode>interleave</RTPTransportMode>
    				</RTSP>
    			</RTP>
    			<!-- Properties defined here will override any properties defined in conf/MediaCasters.xml for any MediaCasters loaded by this applications -->
    			<Properties>
    			</Properties>
    		</MediaCaster>
    		<MediaReader>
    			<!-- Properties defined here will override any properties defined in conf/MediaReaders.xml for any MediaReaders loaded by this applications -->
    			<Properties>
    			</Properties>
    		</MediaReader>
    		<MediaWriter>
    			<!-- Properties defined here will override any properties defined in conf/MediaWriter.xml for any MediaWriter loaded by this applications -->
    			<Properties>
    			</Properties>
    		</MediaWriter>
    		<LiveStreamPacketizer>
    			<!-- Properties defined here will override any properties defined in conf/LiveStreamPacketizers.xml for any LiveStreamPacketizers loaded by this applications -->
    			<Properties>
    			</Properties>
    		</LiveStreamPacketizer>
    		<HTTPStreamer>
    			<!-- Properties defined here will override any properties defined in conf/HTTPStreamers.xml for any HTTPStreamer loaded by this applications -->
    			<Properties>
    			</Properties>
    		</HTTPStreamer>
    		<Repeater>
    			<OriginURL></OriginURL>
    			<QueryString><![CDATA[]]></QueryString>
    		</Repeater> 
    		<Modules>
    			<Module>
    				<Name>base</Name>
    				<Description>Base</Description>
    				<Class>com.wowza.wms.module.ModuleCore</Class>
    			</Module>
    			<Module>
    				<Name>logging</Name>
    				<Description>Client Logging</Description>
    				<Class>com.wowza.wms.module.ModuleClientLogging</Class>
    			</Module>
    			<Module>
    				<Name>flvplayback</Name>
    				<Description>FLVPlayback</Description>
    				<Class>com.wowza.wms.module.ModuleFLVPlayback</Class>
    			</Module> 
    			<Module>
    				<Name>Wowza3TestApplication</Name>
    				<Description>Wowza3TestApplication</Description>
    				<Class>Wowza3TestApplication</Class>
    			</Module>
    		</Modules>
    		<!-- Properties defined here will be added to the IApplication.getProperties() and IApplicationInstance.getProperties() collections -->
    		<Properties>
    		</Properties>
    	</Application>
    </Root>

    I know the behavior of GC. If there are objects that are no longer in use and JVM has leeway to collect it, GC could happen. But it doesn't happen at all during the sleep in my environment. I wonder that streaming data are held by some threads until the end of the onPublish section.


    I hope your co-operation to stabilize our 80 wowza servers.

    Thanks.

  6. #6

    Default

    Quote Originally Posted by T.Kouno View Post
    Hi.

    I tested this module on another machine, then the same phenomenon occurred.

    I would like to confirm that your server received proper amount of streaming data (500kbps or more) published from your client.
    If the onPublish on the Wowza side is waiting then the publishing client should not be sending any data, so this may be the issue. If your publishing client is doing something it should not then Wowza may be accepting the data waiting for it to complete.

    All the tests I have performed are with Wirecast or similar.

    Andrew.

  7. #7
    Join Date
    Dec 2013
    Posts
    5

    Default

    I've tested with Wirecast, and couldn't observe the phenomenon. The heap memory usage didn't run up rapidly.

    But Flash clients are used mainly in our service. So I've tested this with Flash all along.
    I'd like you to check whether the run-up occurs or not with following swf which I developed for this test.
    FlashTest.swf

    And the Flex code:
    <?xml version="1.0" encoding="utf-8"?>
    <s:Application xmlns:fx="http://ns.adobe.com/mxml/2009" 
    			   xmlns:s="library://ns.adobe.com/flex/spark" 
    			   xmlns:mx="library://ns.adobe.com/flex/mx" minWidth="955" minHeight="600"
    			   creationComplete="onCreationComplete()">
    	<fx:Declarations>
    	</fx:Declarations>
    	<fx:Script>
    		<![CDATA[
    			import mx.collections.ArrayList;
    			import mx.controls.Alert;
    			import mx.controls.List;
    			import mx.utils.StringUtil;
    			
    			import spark.components.List;
    			import spark.events.IndexChangeEvent;
    			import spark.events.TextOperationEvent;
    			var camera:Camera;
    			var video:Video = new Video();
    			var nc:NetConnection;
    			var ns:NetStream;
    			
    			protected function onCreationComplete():void{
    				var arr:ArrayList = new ArrayList();
    				for ( var i:String in Camera.names ) {
    					arr.addItem(Camera.getCamera(i).name);
    				}
    				cameraList.dataProvider = arr;
    			}
    			
    			protected function start_publish_clickHandler(event:MouseEvent):void
    			{
    				start_publish.enabled = false;
    				end_publish.enabled = true;
    				
    				setVideoComponent();
    				
    				nc = new NetConnection();
    				nc.addEventListener(NetStatusEvent.NET_STATUS, onConnectStatus);
    				var clientObj:Object = new Object();
    				nc.client = clientObj;
    				nc.connect(publish_url.text);
    			}
    			
    			private function onConnectStatus(evt:NetStatusEvent):void {
    				textArea.text += evt.info.code + "\n";
    				if (evt.info.code == "NetConnection.Connect.Success") {
    					ns = new NetStream(nc);
    					ns.addEventListener(NetStatusEvent.NET_STATUS, onStreamStatus);
    					ns.attachCamera(camera);
    					ns.publish(stream_name.text);
    				}
    			}
    			
    			private function onStreamStatus(evt:NetStatusEvent):void {
    				textArea.text += evt.info.code + "\n";
    			}
    			private function setVideoComponent():void {
    				camera = Camera.getCamera(cameraList.selectedIndex + "");
    				camera.setMode(800, 600, 15);
    				video.attachCamera(camera);
    				video.width = 480;
    				video.height = 360;
    				videoComponent.addChild(video);
    			}
    			protected function end_publish_clickHandler(event:MouseEvent):void
    			{
    				start_publish.enabled = true;
    				end_publish.enabled = false;
    
    				for (var i:int = videoComponent.numChildren - 1; i >= 0; i--) {
    					videoComponent.removeChildAt(i);
    				}
    				ns.close();
    				nc.close();
    				video.attachCamera(null);
    				video = null;
    				video = new Video();
    				camera = null;
    			}
    		]]>
    		
    	</fx:Script>
    	<mx:UIComponent id="videoComponent" x="53" y="112" width="779" height="430" />
    	<s:Button id="start_publish" x="292" y="54" width="101" height="26" label="Start publish" click="start_publish_clickHandler(event)"/>
    	<s:Button id="end_publish" x="438" y="52" width="101" height="27" label="End publish" click="end_publish_clickHandler(event)"/>
    	<s:TextArea id="textArea" x="580" y="18" width="312" height="84"/>
    	<s:Label x="270" y="28" text="camera list"/>
    	<s:Label x="37" y="29" text="publish url"/>
    	<s:ComboBox id="cameraList" x="341" y="21" width="207" />
    	<s:TextInput id="publish_url" x="108" y="21"/>
    	<s:Label x="22" y="63" height="21" text="Stream name"/>
    	<s:TextInput id="stream_name" x="108" y="57"/>
    </s:Application>

  8. #8

    Default

    Hi,

    I think what is happening is that the flex app is sending video data as soon as the publish command has been completed. On Wowza, because the thread that is handling the publish command is blocked, it cannot finish setting up the publish stream completely. Because the data is already coming in, it is storing it ready to process once the publish command is completed.

    I think Wirecast is probably waiting for the NetStream.Publish.Start response from the server before starting to send the stream.

    If you make the following change to your flex code, it then appears to do the same as Wirecast.

    			private function onConnectStatus(evt:NetStatusEvent):void {
    				textArea.text += evt.info.code + "\n";
    				if (evt.info.code == "NetConnection.Connect.Success") {
    					ns = new NetStream(nc);
    					ns.addEventListener(NetStatusEvent.NET_STATUS, onStreamStatus);
    					//ns.attachCamera(camera);
    					ns.publish(stream_name.text);
    				}
    			}
    			
    			private function onStreamStatus(evt:NetStatusEvent):void {
    				textArea.text += evt.info.code + "\n";
    				if(evt.info.code == "NetStream.Publish.Start")
    				{
    					ns.attachCamera(camera);
    				}
    			}

  9. #9
    Join Date
    Dec 2013
    Posts
    5

    Default

    Hi.

    I've changed my Flash code as you said, and I can't see the run-up any more.
    I'll apply this to our Flash clients for now.

    However, I can't deal with third-party software which made by someone who doesn't know this workaround,
    moreover, Adobe says, "After attaching the video source, you must call NetStream.publish() to begin transmitting." in his reference
    Thus I think the best answer to this problem is to improve Wowza not to run up its heap memory even if it receives data during the sleep in onPublish.

    I would like your comments.
    Last edited by T.Kouno; 12-17-2013 at 09:13 PM. Reason: typo url

  10. #10

    Default

    Hi,

    We will pass this onto the engineering team to take a look and see what can be done.

    FYI, doing a database lookup where you are doing it is not good practice and it could end up stalling the server if there are too many slow lookups at the same time. Concurrent calls to your module methods run on separate VHost Handler threads so if there are more calls waiting than the size of the thread pool, it will stall Wowza.

    You really should be using a separate thread pool for db lookups.

    Roger.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •