Results 1 to 10 of 10

Thread: Very high CPU - Wowza 3.0.5

  1. #1

    Default Very high CPU - Wowza 3.0.5

    I'm load-testing Wowza 3.0.5 on EC2 m1.small instances.

    The configuration is:

    * Ubuntu 10.10 32-bit, Java -server 1.7u3
    * standard liverepeater/edge (origin is an m1.large, performance is not a problem on this machine at the moment)
    * live 128kbps mp3 streams
    * no custom modules
    * Wowza tuning guide followed (VHost.xml below)

    I'm using rtmpdump on a third server to slowly build up outgoing connections to the edge. CPU usage on the edge server increases steadily (faster than I would expect to be honest) but increases after about 300 connections. I cannot get past 400 connections without CPU reaching 100% and load average sky-rocketing.

    I would expect an m1.small to handle ~1000 outgoing 128kbps streams, taking the bandwidth of the incoming mediacaster streams into account.

    How can I address this?

    <Root>
    	<VHost>
    		<HostPortList>
    			<HostPort>
    				<ProcessorCount>2</ProcessorCount>
    				<IpAddress>*</IpAddress>
    				<!-- Separate multiple ports with commas -->
    				<!-- 80: HTTP, RTMPT -->
    				<!-- 554: RTSP -->
    				<Port>1935,80,443,554</Port>
    				<SocketConfiguration>
    					<ReuseAddress>true</ReuseAddress>
    					<!-- suggested settings for video on demand applications -->
    					<ReceiveBufferSize>24000</ReceiveBufferSize>
    					<SendBufferSize>65000</SendBufferSize>
    					<!-- suggest settings for low latency chat and video recording applications
    					<ReceiveBufferSize>16000</ReceiveBufferSize>
    					<SendBufferSize>16000</SendBufferSize>
    					-->
    					<KeepAlive>true</KeepAlive>
    					<!-- <TrafficClass>0</TrafficClass> -->
    					<!-- <OobInline>false</OobInline> -->
    					<!-- <SoLingerTime>-1</SoLingerTime> -->
    					<!-- <TcpNoDelay>false</TcpNoDelay> -->
    					<AcceptorBackLog>100</AcceptorBackLog>
    				</SocketConfiguration>
    				<HTTPStreamerAdapterIDs>cupertinostreaming,smoothstreaming,sanjosestreaming,dvrchunkstreaming</HTTPStreamerAdapterIDs>
    				<HTTPProviders>
    					<HTTPProvider>
    						<BaseClass>com.wowza.wms.http.HTTPCrossdomain</BaseClass>
    						<RequestFilters>*crossdomain.xml</RequestFilters>
    						<AuthenticationMethod>none</AuthenticationMethod>
    					</HTTPProvider>
    					<HTTPProvider>
    						<BaseClass>com.wowza.wms.http.HTTPClientAccessPolicy</BaseClass>
    						<RequestFilters>*clientaccesspolicy.xml</RequestFilters>
    						<AuthenticationMethod>none</AuthenticationMethod>
    					</HTTPProvider>
              <HTTPProvider>
                <BaseClass>com.app.wms.http.LiveServer</BaseClass>
                <RequestFilters>app_ctrl*</RequestFilters>
                <AuthenticationMethod>admin-basic</AuthenticationMethod>
              </HTTPProvider>
              <HTTPProvider>
                <BaseClass>com.app.wms.http.Stats</BaseClass>
                <RequestFilters>stat*</RequestFilters>
                <AuthenticationMethod>none</AuthenticationMethod>
              </HTTPProvider>
    					
    				</HTTPProviders>
    			</HostPort>
    			
    			<!-- 443 with SSL -->
    			<!--
    			<HostPort>
    				<ProcessorCount>4</ProcessorCount>
    				<IpAddress>*</IpAddress>
    				<Port>443</Port>
    				<SSLConfig>
    					<KeyStorePath>${com.wowza.wms.context.VHostConfigHome}/conf/keystore.cert</KeyStorePath>
    					<KeyStorePassword>[password]</KeyStorePassword>
    					<KeyStoreType>JKS</KeyStoreType>
    					<SSLProtocol>TLS</SSLProtocol>
    					<Algorithm>SunX509</Algorithm>
    					<CipherSuites></CipherSuites>
    					<Protocols></Protocols>
    				</SSLConfig>
    				<SocketConfiguration>
    					<ReuseAddress>true</ReuseAddress>
    					<ReceiveBufferSize>24000</ReceiveBufferSize>
    					<SendBufferSize>65000</SendBufferSize>
    					<KeepAlive>true</KeepAlive>
    					<AcceptorBackLog>100</AcceptorBackLog>
    				</SocketConfiguration>
    				<HTTPStreamerAdapterIDs>cupertinostreaming,smoothstreaming,sanjosestreaming,dvrchunkstreaming</HTTPStreamerAdapterIDs>
    				<HTTPProviders>
    					<HTTPProvider>
    						<BaseClass>com.wowza.wms.http.HTTPClientAccessPolicy</BaseClass>
    						<RequestFilters>*clientaccesspolicy.xml</RequestFilters>
    						<AuthenticationMethod>none</AuthenticationMethod>
    					</HTTPProvider>
    					<HTTPProvider>
    						<BaseClass>com.wowza.wms.http.HTTPCrossdomain</BaseClass>
    						<RequestFilters>*crossdomain.xml</RequestFilters>
    						<AuthenticationMethod>none</AuthenticationMethod>
    					</HTTPProvider>
    					<HTTPProvider>
    						<BaseClass>com.wowza.wms.http.HTTPServerVersion</BaseClass>
    						<RequestFilters>*</RequestFilters>
    						<AuthenticationMethod>none</AuthenticationMethod>
    					</HTTPProvider>
    				</HTTPProviders>
    			</HostPort>
    			-->
    			
    			<!-- Admin HostPort -->
    			<HostPort>
    				<ProcessorCount>1</ProcessorCount>
    				<IpAddress>*</IpAddress>
    				<Port>8086</Port>
    				<SocketConfiguration>
    					<ReuseAddress>true</ReuseAddress>
    					<ReceiveBufferSize>16000</ReceiveBufferSize>
    					<SendBufferSize>16000</SendBufferSize>
    					<KeepAlive>true</KeepAlive>
    					<AcceptorBackLog>100</AcceptorBackLog>
    				</SocketConfiguration>
    				<HTTPStreamerAdapterIDs></HTTPStreamerAdapterIDs>
    				<HTTPProviders>
    					<HTTPProvider>
    						<BaseClass>com.wowza.wms.http.streammanager.HTTPStreamManager</BaseClass>
    						<RequestFilters>streammanager*</RequestFilters>
    						<AuthenticationMethod>admin-digest</AuthenticationMethod>
    					</HTTPProvider>
    					<HTTPProvider>
    						<BaseClass>com.wowza.wms.http.HTTPServerInfoXML</BaseClass>
    						<RequestFilters>serverinfo*</RequestFilters>
    						<AuthenticationMethod>admin-digest</AuthenticationMethod>
    					</HTTPProvider>
    					<HTTPProvider>
    						<BaseClass>com.wowza.wms.http.HTTPConnectionInfo</BaseClass>
    						<RequestFilters>connectioninfo*</RequestFilters>
    						<AuthenticationMethod>admin-digest</AuthenticationMethod>
    					</HTTPProvider>
    					<HTTPProvider>
    						<BaseClass>com.wowza.wms.http.HTTPConnectionCountsXML</BaseClass>
    						<RequestFilters>connectioncounts*</RequestFilters>
    						<AuthenticationMethod>admin-digest</AuthenticationMethod>
    					</HTTPProvider>
    					<HTTPProvider>
    						<BaseClass>com.wowza.wms.transcoder.httpprovider.HTTPTranscoderThumbnail</BaseClass>
    						<RequestFilters>transcoderthumbnail*</RequestFilters>
    						<AuthenticationMethod>admin-digest</AuthenticationMethod>
    					</HTTPProvider>
    					<HTTPProvider>
    						<BaseClass>com.wowza.wms.http.HTTPProviderMediaList</BaseClass>
    						<RequestFilters>medialist*</RequestFilters>
    						<AuthenticationMethod>admin-digest</AuthenticationMethod>
    					</HTTPProvider>
    					<HTTPProvider>
    						<BaseClass>com.wowza.wms.http.HTTPServerVersion</BaseClass>
    						<RequestFilters>*</RequestFilters>
    						<AuthenticationMethod>none</AuthenticationMethod>
    					</HTTPProvider>
    				</HTTPProviders>
    			</HostPort>
    		
    		</HostPortList>
    		
    		<HTTPStreamerAdapters>
    			<HTTPStreamerAdapter>
    				<ID>smoothstreaming</ID>
    				<Name>smoothstreaming</Name>
    				<Properties>
    				</Properties>
    			</HTTPStreamerAdapter>
    			<HTTPStreamerAdapter>
    				<ID>cupertinostreaming</ID>
    				<Name>cupertinostreaming</Name>
    				<Properties>
    				</Properties>
    			</HTTPStreamerAdapter>
    			<HTTPStreamerAdapter>
    				<ID>sanjosestreaming</ID>
    				<Name>sanjosestreaming</Name>
    				<Properties>
    				</Properties>
    			</HTTPStreamerAdapter>
    			<HTTPStreamerAdapter>
    				<ID>dvrchunkstreaming</ID>
    				<Name>dvrchunkstreaming</Name>
    				<Properties>
    				</Properties>
    			</HTTPStreamerAdapter>
    		</HTTPStreamerAdapters>
    
    		<HandlerThreadPool>
    			<PoolSize>60</PoolSize>
    		</HandlerThreadPool>
    		<TransportThreadPool>
    			<PoolSize>40</PoolSize>
    		</TransportThreadPool>
    		<IdleWorkers>
    			<WorkerCount>2</WorkerCount>
    			<CheckFrequency>100</CheckFrequency>
    			<MinimumWaitTime>5</MinimumWaitTime>
    		</IdleWorkers>
    		<NetConnections>
    			<ProcessorCount>2</ProcessorCount>
    			<IdleFrequency>250</IdleFrequency>
    			<SocketConfiguration>
    				<ReuseAddress>true</ReuseAddress>
    				<ReceiveBufferSize>65000</ReceiveBufferSize>
    				<SendBufferSize>24000</SendBufferSize>
    				<KeepAlive>true</KeepAlive>
    				<!-- <TrafficClass>0</TrafficClass> -->
    				<!-- <OobInline>false</OobInline> -->
    				<!-- <SoLingerTime>-1</SoLingerTime> -->
    				<!-- <TcpNoDelay>false</TcpNoDelay> -->
    				<AcceptorBackLog>100</AcceptorBackLog>
    			</SocketConfiguration>
    		</NetConnections>
    		<HTTPTunnel>
    			<KeepAliveTimeout>2000</KeepAliveTimeout>
    		</HTTPTunnel>
    		<Client>
    			<ClientTimeout>90000</ClientTimeout>
    			<IdleFrequency>500</IdleFrequency>
    		</Client>
    		<!-- RTP/Authentication/Methods defined in Authentication.xml. Default setup includes; none, basic, digest -->
    		<RTP>
    			<IdleFrequency>75</IdleFrequency>
    			<DatagramConfiguration>
    				<Incoming>
    					<ReuseAddress>true</ReuseAddress>
    					<ReceiveBufferSize>1024000</ReceiveBufferSize>
    					<SendBufferSize>65000</SendBufferSize>
    					<!-- <MulticastBindToAddress>true</MulticastBindToAddress> -->
    					<!-- <MulticastInterfaceAddress>192.168.1.22</MulticastInterfaceAddress> -->
    					<!-- <TrafficClass>0</TrafficClass> -->
    					<MulticastTimeout>50</MulticastTimeout>
    					<DatagramMaximumPacketSize>4096</DatagramMaximumPacketSize>
    				</Incoming>
    				<Outgoing>
    					<ReuseAddress>true</ReuseAddress>
    					<ReceiveBufferSize>65000</ReceiveBufferSize>
    					<SendBufferSize>65000</SendBufferSize>
    					<!-- <MulticastBindToAddress>true</MulticastBindToAddress> -->
    					<!-- <MulticastInterfaceAddress>192.168.1.22</MulticastInterfaceAddress> -->
    					<!-- <TrafficClass>0</TrafficClass> -->
    					<MulticastTimeout>50</MulticastTimeout>
    					<DatagramMaximumPacketSize>4096</DatagramMaximumPacketSize>
    				</Outgoing>
    			</DatagramConfiguration>
    			<UnicastIncoming>
    				<ProcessorCount>1</ProcessorCount>
    			</UnicastIncoming>
    			<UnicastOutgoing>
    				<ProcessorCount>2</ProcessorCount>
    			</UnicastOutgoing>
    			<MulticastIncoming>
    				<ProcessorCount>1</ProcessorCount>
    			</MulticastIncoming>
    			<MulticastOutgoing>
    				<ProcessorCount>1</ProcessorCount>
    			</MulticastOutgoing>
    		</RTP>
    		<Application>
    			<ApplicationTimeout>60000</ApplicationTimeout>
    			<PingTimeout>12000</PingTimeout>
    			<ValidationFrequency>20000</ValidationFrequency>
    			<MaximumPendingWriteBytes>0</MaximumPendingWriteBytes>
    			<MaximumSetBufferTime>60000</MaximumSetBufferTime>
    		</Application>
    		<StartStartupStreams>true</StartStartupStreams>
    		<!-- Properties defined here will be added to the IVHost.getProperties() collection -->
    		<Properties>
    		</Properties>
    	</VHost>
    </Root>
    Last edited by robw21; 02-24-2012 at 02:47 AM.

  2. #2
    Join Date
    Sep 2011
    Posts
    1,931

    Default

    Hi
    Try tuning the server, the guide is here,
    http://www.wowza.com/forums/content....ormance-Tuning

    Jason

  3. #3

    Default

    Hi Jason,

    Thanks for the reply.

    I've already followed the tuning guide. I've posted my VHost.xml above, it's tuned for a single-core server (ProcessorCount 1-2, as per the tuning guide).

    Unfortunately performance is still bound by CPU, rather than network. What else could be a factor?

  4. #4

    Default

    It seems to me you are hitting the CPU limits of the m1.small instance.

  5. #5
    Join Date
    Dec 2007
    Posts
    21,962

    Default

    You should not tune a EC2 instance started with one of the pre-built AMIs from here because they are tuned automatically when the instance is started. I would undo the tuning changes that you made, or start a new instance. It's no problem to add custom HTTPProviders, etc but do not tune.

    Richard

  6. #6
    Join Date
    Dec 2007
    Posts
    21,962

    Default

    Also, with very low bitrate streams you are likely to hit CPU limits before network limits. In our own testing on EC2 we used 700-800Mbps test content and hit a limit at about 150Mbps.

    Richard

  7. #7

    Default

    yep that's what I figured. Thanks for the reply.

  8. #8

    Default

    (It's a custom-built EC2 instance so it did need tuning, btw).

  9. #9
    Join Date
    Nov 2011
    Posts
    14

    Default

    The lower the bitrate, the more CPU usage? It kind of makes sense because more compression cycles are going into making stream smaller but, conversely, if you do 720p streams, shouldn't the CPU be equally (or more) used?

  10. #10

    Default

    uniticon,

    Lower bitrate streams means you can stream more streams using the same overall bandwidth. Just guessing here, but I imagine more streams means increased overhead in the internal scheduler/resource manager, therefore increased CPU overhead. Also, each stream will have a small buffer, so more streams also means more RAM overhead. And for VOD, more disk I/O.

Similar Threads

  1. Wowza Transcoder Addon HIGH CPU
    By and12345 in forum AddOn: Transcoder
    Replies: 2
    Last Post: 03-11-2014, 04:07 AM
  2. High Cpu usage AMD Opteron Dual CPU Server.
    By djcenk in forum Performance Tuning Discussion
    Replies: 1
    Last Post: 11-26-2013, 08:11 AM
  3. Very high CPU usage from Wowza start
    By pr1001 in forum Performance Tuning Discussion
    Replies: 6
    Last Post: 07-05-2012, 01:29 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •