Results 1 to 8 of 8

Thread: Wowza Crashes: Live Streams

  1. #1
    Join Date
    Nov 2008
    Posts
    18

    Default Wowza Crashes: Live Streams

    Hi Group,

    We been using Wowza for almost two years now, we have tried everything to keep our streams running smoothly for more then four days. But every four days our whole cluster crashes. Any else have this issue where wowza hit a certain number of connections and just craps out ? Below is the number that seems to kill wowza. Three weeks in a row it has done this without fail, +10 going either direction. We have five of these machines and they all drop about the same time.

    Connection Total: 333556
    Connection Current: 303
    Connection Total Accepted: 328231
    Connection Total Rejected 5325

  2. #2
    Join Date
    Nov 2008
    Posts
    18

    Default

    Running Verizon 2.2.4, Dual Xeon 3.0, 4gb of RAM, 15K SAS. So shouldn't be a system issue. I have tuned this out very aggressively.

  3. #3
    Join Date
    Dec 2007
    Posts
    21,962

    Default

    Does each Wowza server stop working, possibly with an "outOfMemory" error? Or are you just not able to reach them at that point?

    If it is the former, and the tuning is done, then that is the limit of those servers. In this case, add one or more servers or beef-up the ones you have are the solutions

    If it is the latter, you are probably hitting the limits of the servers' network connection.

    Richard

  4. #4
    Join Date
    Nov 2008
    Posts
    18

    Default

    That the strange part, i first get complaints the application are running slow, within a few minutes it then stop loading all the live streams. The only way to get everything working is to restart Wowza, then several days later it does it again. I get the OutofMemory error once in a while but when that happens wowza just stops, in this case it stays running. I cant imagine that we are hitting the max on the server. There only 300 - 400 connections per server, pushing 12 mbits on each of the 7 severs. i bumped the ulimit up to 50000 which buys me another day or so. The other option i been pushing is the open file limit I been doubling it to 128,000 from 65,000. Our system currently is split with our old FMS 2.5 and Wowza. I mabey have to reboot the FMS 2.5 servers once a year. Were confused as this has gotten worse since we upgrade to 2.2.4, when we were running 2.2.1 it would go two weeks before crashing.

  5. #5

    Default

    Hi domeger,

    So I see you are doing 12,000kbps / 300 = 40kbps. So, you are doing audio only streams?

    Try to describe your setup in more detail.

    1. Do you have 5 machines or 7 machines?.
    2. One origin?
    3. What do you mean when you say the whole cluster crashes? You can't ping/access them anymore, or Wowza stops running, or they just stop streaming? This might suggest a problem with the origin.
    4. What is the version of Wowza on each of the computers? If it is different on the origin/edge, this might be a problem.
    5. What version of Java on each computer? Upgrading to the latest Java 1.7 is critical.
    6. Do you have any custom modules running?

    You should not have to reboot your servers every two weeks, or every 4 days.

  6. #6
    Join Date
    Nov 2008
    Posts
    18

    Default

    1.) There is 6 Wowza Servers now since we have been having stability issues.
    2.) No origin server, our older system that runs off FMS 2.5 is Edge, Origin. We decided we were tired of single point of failure, so we went with all server to be Edge Servers.
    3.) By Crashing, I mean all the Wowza Server stop responding. The services is either off, or the wowza server will no longer handle anymore connections.
    4.) 2.2.4 with the patch for the IOS5
    5.) Java - 1.6.0_27-b07 - Had no clue that 1.7 was out.
    6.) Yes all of the modules are custom.

  7. #7

    Default

    It would suggest there is a memory leak or process lock issue with the custom code, lock issue more than likely, so the number of threads available maxes out and Wowza has to stop as there are no more available.

    In your custom code, add some debug logging, I suspect you have already and or look at your synchronization blocks (If you have any).

    If you can run one Wowza without the custom code this will more than likely highlight the custom code as the starting point to look.

    Shamrock

  8. #8
    Join Date
    Nov 2008
    Posts
    18

    Default

    The fix is the new GC optimization, that is recommended on the site. Never noticed you changed this, but it actually has helped 200%.

    "
    2. Experimental: Garbage-First garbage collector (G1): In the Sun Java 6 Update 14 or greater release there is a new garbage collector that is in the experimental stage. It is designed for low pause time high throughput applications such as Wowza Media Server. It is an option to explore. There is more information below. To enable this garbage collector the additional settings are:
    Code:
    -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC"

Similar Threads

  1. rtsp for vod with 3G connection crashes but not for live
    By zouzou in forum Video On Demand Streaming Discussion
    Replies: 2
    Last Post: 06-18-2013, 06:42 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •