Results 1 to 6 of 6

Thread: Random CPU usage pattern

  1. #1
    Join Date
    Jun 2012
    Posts
    17

    Default Random CPU usage pattern

    After a while, without any apparent reason the CPU usage jumps from a low load (8-9%) to 98-99% and the server chokes. This was observed multiple times with the CPU (taken by the Wowza java process) staying close to 100% for variable lengths of time: 1 min, 10 mins(!) and even for more than 1 hour(!). After such an event the CPU usage falls back to normal values.
    Details: Wowza 3.1.2, jdk 1.7, JVM 64 bits, server mode, RHEL 64 bit, quad core CPU (same behavior observed with 2 quad CPUs installed), tuned exactly as suggested, Xmx 3000-3500M heap, no fancy GC.

    Today this happened twice so far for 2 mins and 1 minute.
    Latest event, aprox 280 users mixed, live & mp4 playbacks, CPU at 7-8% suddenly jumps at 98-99% for aprox 1 minute (live & streams streams choppy during this period) then falls back under 10% and everything continues.
    I have GC logging enabled and also ran 'jstat -gc', all looks good, nothing wrong with GC. Here's the GC log for the time this happened:
    56969.144: [GC [PSYoungGen: 532751K->9466K(534720K)] 582211K->59170K(604864K), 0.0137370 secs] [Times: user=0.04 sys=0.00, real=0.01 secs] 
    57006.513: [GC [PSYoungGen: 534714K->10037K(535424K)] 584418K->75931K(605568K), 0.0262200 secs] [Times: user=0.07 sys=0.01, real=0.03 secs] 
    57006.540: [Full GC [PSYoungGen: 10037K->0K(535424K)] [ParOldGen: 65894K->56010K(94336K)] 75931K->56010K(629760K) [PSPermGen: 22018K->22017K(22144K)], 0.1017750 secs] [Times: user=0.27 sys=0.00, real=0.10 secs] 
    57040.032: [GC [PSYoungGen: 525375K->2748K(503424K)] 581386K->58758K(597760K), 0.0111670 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] 
    57061.343: [GC [PSYoungGen: 503399K->3091K(480320K)] 559409K->59102K(574656K), 0.0119880 secs]
    I also took 3 jstack dumps, 2 while the CPU close to 100% and 1 after. The first 2 show most of the VHostHandler threads being 'locked' (see below) . This changes in the last jstack log. I can provide these files if needed.

    "VHostHandler._defaultVHost_.240" prio=10 tid=0x0000000018ce2000 nid=0x16f7 in Object.wait() [0x000000005c8ad000]
       java.lang.Thread.State: WAITING (on object monitor)
    	at java.lang.Object.wait(Native Method)
    	at java.lang.Object.wait(Object.java:503)
    	at edu.emory.mathcs.backport.java.util.concurrent.locks.WMSReentrantReadWriteLock$ReadLock.lock(Unknown Source)
    	- locked <0x0000000725653d60> (a edu.emory.mathcs.backport.java.util.concurrent.locks.WMSReentrantReadWriteLock$ReadLock)
    	at com.wowza.wms.vhost.VHost.getClient(Unknown Source)
    	at com.wowza.wms.vhost.VHost.getClient(Unknown Source)
    	at com.wowza.wms.request.RTMPRequestAdapter.service(Unknown Source)
    	at com.wowza.wms.server.ServerHandler.a(Unknown Source)
    	at com.wowza.wms.server.ServerHandler.a(Unknown Source)
    	at com.wowza.wms.server.ServerHandler.sessionIdle(Unknown Source)
    	at com.wowza.wms.server.ServerHandlerThreadedSession.run(Unknown Source)
    	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    	at java.lang.Thread.run(Thread.java:722)
    Wowza logs show nothing significant, no errors/warnings.
    This is critical as we're having this issues on our production system. We've been using successfully, on the same box, Wowza 1.7 but upgraded to 3 in the last few weeks. We also changed the playback files from FLV to mp4(h264). Could be this mp4 related? I'm also considering going back to 1.7 at this point.
    Any suggestion, please. Thanks!

  2. #2
    Join Date
    Jun 2012
    Posts
    17

    Default

    Forgot, memory 8G

  3. #3
    Join Date
    Jun 2012
    Posts
    17

    Default

    More info, the 'bad' jstack logs also show 24 threads as:

    "VHostHandler._defaultVHost_.208" daemon prio=10 tid=0x0000000018791000 nid=0x16b3 runnable [0x000000005aa8e000]
       java.lang.Thread.State: RUNNABLE
    	at com.wowza.util.FastTypeBuffer.getInt(Unknown Source)
    	at com.wowza.util.FastIntBuffer.get(Unknown Source)
    	at com.wowza.wms.mediareader.h264.atom.QTMediaCursor.a(Unknown Source)
    	at com.wowza.wms.mediareader.h264.atom.QTMediaCursor.b(Unknown Source)
    	at com.wowza.wms.mediareader.h264.atom.QTMediaCursor.a(Unknown Source)
    	at com.wowza.wms.mediareader.h264.atom.QTMediaCursor.moveToSample(Unknown Source)
    	at com.wowza.wms.mediareader.h264.H264Cursor.setAudioSample(Unknown Source)
    	at com.wowza.wms.mediareader.h264.H264Cursor.setCurrentPosition(Unknown Source)
    	at com.wowza.wms.mediareader.h264.MediaReaderH264.tiggerNextReadAhead(Unknown Source)
    	at com.wowza.wms.mediareader.h264.MediaReaderH264.getReadAheadPacket(Unknown Source)
    	at com.wowza.wms.mediareader.h264.MediaReaderH264.directOutput(Unknown Source)
    	at com.wowza.wms.mediareader.h264.MediaReaderH264.writePackets(Unknown Source)
    	- locked <0x0000000725d8d908> (a com.wowza.wms.stream.record.MediaStreamRecord)
    	at com.wowza.wms.mediareader.h264.MediaReaderH264.writePackets(Unknown Source)
    	at com.wowza.wms.stream.file.PlaylistPlayer.play(Unknown Source)
    	at com.wowza.wms.stream.file.MediaStreamFilePlay.play(Unknown Source)
    	at com.wowza.wms.response.ResponseStreams.output(Unknown Source)
    	at com.wowza.wms.request.RTMPRequestAdapter.service(Unknown Source)
    	- locked <0x0000000725d624a8> (a java.lang.Object)
    	at com.wowza.wms.server.ServerHandler.a(Unknown Source)
    	at com.wowza.wms.server.ServerHandler.a(Unknown Source)
    	at com.wowza.wms.server.ServerHandler.sessionIdle(Unknown Source)
    	at com.wowza.wms.server.ServerHandlerThreadedSession.run(Unknown Source)
    	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    	at java.lang.Thread.run(Thread.java:722)
    Is com.wowza.wms.stream.record.MediaStreamRecord related to real recording, cause i'm not aware of any recording happening there.

  4. #4
    Join Date
    Dec 2007
    Posts
    21,962

    Default

    Felix,

    The MediaStreamRecord is the stream implementation used when the stream type is set to “default”. It is used for both video on demand playback and recording. This stack trace is related to playback and not recording

    What is exact model and spec of the cpu? And what is your disk setup? RAID 0? Number of disks?

    Send info to support@wowza.com to open a ticket.

    You might try reverting to FLV files with the new server. MP4 files are more complicated to open and parse on initial load, which could be leading to some of the problems.

    Also, when you are looking at the cpu % in a high state, look to see whether the io wait % is high. This could point to a disk IO issue.

    Richard

  5. #5
    Join Date
    Jun 2012
    Posts
    17

    Default

    Thanks Richard,
    IO wait % looks OK, very low. I'll open a ticket and send all the info & logs.
    We switched last night back to FLVs and so far no issues, it seems to be mp4 related.

  6. #6
    Join Date
    Dec 2007
    Posts
    21,962

    Default

    Also, it might help to have AHCI enabled and properly configured to support native command queueing. It is described here:
    https://wiki.archlinux.org/index.php/AHCI

    This should help to optimize disk access.

    Richard

Similar Threads

  1. High Cpu usage AMD Opteron Dual CPU Server.
    By djcenk in forum Performance Tuning Discussion
    Replies: 1
    Last Post: 11-26-2013, 08:11 AM
  2. 50% CPU usage with 2k viewers
    By PaltopVince in forum Performance Tuning Discussion
    Replies: 1
    Last Post: 11-01-2013, 11:28 AM
  3. High CPU Usage every now and then
    By stardust2678 in forum General Forum
    Replies: 1
    Last Post: 07-09-2013, 05:06 AM
  4. CPU Usage X Memory
    By rminari in forum Wowza Media Server 3 for Amazon EC2 Discussion
    Replies: 1
    Last Post: 05-19-2012, 01:04 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •