Results 1 to 2 of 2

Thread: Our experience with garbage collection

  1. #1
    Join Date
    Jun 2009
    Posts
    7

    Default Our experience with garbage collection

    Hello,

    I write this post just to share the experience. This issue has consumed me about 2 weeks of investigation.
    Some time ago we began to have many problems with Wowza freezing, and consuming too much CPU in only one core. We did not know it, but the problem is the java process running your java garbage collection routine.

    I did some testing with the configuration recommended by Wowza (-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:newSize=512m), but after a few minutes the problem came back. So I did some tests with experimental option (-XX:+UnlockExperimentalVMOptions -XX:+UseG1GC) and it got better. But after a few hours the java process freezes again.

    Unfortunately the documentation from oracle for garbage collection is not very complete. Or if it is, is very difficult to find. I found a very usefull video[1], that explains in detail how the GC process works, and also plenty of tips for tuning GC. And also why the 10GB limit makes sense for old GC algorithms.

    But the coolest tip from video is: Use the G1 with the most updated java as possible, with as much memory you have available on the machine. So I did it. I configured the java process to use 100GB heap size (yes, one hundred gigabytes) and the jdk-7u5-x64. Now everything is working better

    For example, one of our liverepeater-origin servers is running for 20 days without any problem. This server is origin for ~80 streams, and it is the source for ~20 server farms. This server receives ~150MB/s, and send ~800MB/s signals. Our CPU peak was 24%.

    The server is a Dual Xeon 6-Core X5670@2.93GHz with 128GB of RAM.

    [1] http://vimeo.com/28761227

  2. #2
    Join Date
    Nov 2010
    Posts
    9

    Default

    Just guessing randomly here: 128 gigs is a lot of memory to GC. Do you really need that? Sure, your setup isn't small, but it doesn't sound THAT huge either.

    For one, the default new generation size is WAY too small for that amount of memory. Most Wowza objects never survive the new generation. See how much you can increase this value while still keeping the new collection pause at acceptable levels. Don't go overboard with it; I think a young GC cycle every few seconds is ok. After that see how much memory REALLY makes it into the mature collection and how much survives a full GC cycle.

    Said random guess is that you have way too much short lived memory overflowing from extremely short young GC cycles filling up your mature generations and that you really just need a fraction of that memory.

    For shits and giggles: You have 80 streams, let's assume one of the 20 server farm has 10 boxes. That accumulates to a maximum of 16000 streams served. Let's assume the insane amount of memory of 1MB per served stream, you'll still only need 16 gigs plus a little application overhead.

    I confess to being a java hater though, so take this judgement with a grain of salt. :-) As a general rule the only thing better than lots of memory is even more memory, but that doesn't apply to java. :P

Similar Threads

  1. Long Garbage Collection pauses and Choppy Streams
    By Keyston in forum Performance Tuning Discussion
    Replies: 1
    Last Post: 05-30-2012, 02:08 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •