Wowza Community

5Gbps limit intentional by Wowza?

This is something rarely addressed and it seriously bothers me. Why is it that a Wowza instance can only handle 5Gbps? Once it reaches the 5Gbps limit it begins to drop connections and greatly degrades live streams. Previously Wowza said it is a Java limitation, but I searched for over a year trying to find any mention of a java bandwidth limit. No dice. I also spoke to a competing server software company and they said they had never heard of the 5Gbps limit before, and they use Java too.

Let’s clear the air once and for all: Is this a ploy to make us purchase two licenses per server? Or is this a limitation within the Wowza server software?

And please do not respond with how a customer was able to bypass this with kernel level tuning. Unless you can provide that solution, it does zero to address this. I pay a lot of money for Wowza licenses each month and am extremely frustrated by this ambiguous 5Gbps limit nobody wants to speak about. My servers are capable of constant 10Gbps each, yet I cannot let them get near 5Gbps otherwise my Wowza instance eventually crashes and drops all connections. No amount of server and kernel level tuning has resolved this over the last year and a half.

Surely I’m not the only one experiencing this?

-Frustrated PaltopVince

EDIT: Please do not ask me to open a support ticket and send config/logs. I’ve done that countless times over this issue over the last few years. After a lot of back and forth it always ended up being a finger pointed at Java’s internal limit, which seems to only exist with Wowza and no other server software.

Hello

Thanks for contacting Wowza Forums!

Sorry about your frustration. Can you be more specific? What limitation are you referring to that is 5GB?

regards,

Jermaine

Not 5GB, but 5Gbps. Total network I/O. Bandwidth in and out.

No, that doesn’t help at all. Because no other Java based video server has a 5Gbps limit. In fact, please show me where it’s documented that Java has a 5Gbps limit per instance because I am unable to find anything about that, except for Wowza forum posts.

I setup two instances of Wowza on a single 10Gbps server. Guess what happened? Each instance would not go over 2.5Gbps each and eventually crashed. How do I know it isn’t a limitation of my server hardware? Because on the same server I installed nginx rtmp and was able to max out the 10Gbps connection without much effort. Yes I understand nginx rtmp and Wowza run on different environments, but I’ve yet to see any proof outside of this forum that there is a 5Gbps limit on Javas Virtual Machine. I’m not asking for the moon, I just want a straight up answer. That’s all.

I did my testing in a production environment to make sure I wasn’t going insane. Now I feel like I’m being ripped.

No offense but you clearly missed the part where I ran multiple instances of Wowza on a single server and still got the same result. And yes I used the guide you linked to when setting up those multiple instances.

I’m not sure I understand what you mean by “The support team is not in a position to offer the required OS level tuning”.

And again, you mention the 5Gbps Java limitation yet you do not address my original question in this thread: Where is it documented that there is a 5Gbps Java limitation. I have not been able to find a single mention of such a thing outside of Wowza.

Thanks for the detailed response! One thing I haven’t done is tinker with the GC config. Although in the past I have tried one of the default settings and it would usually end up crashing the Wowza instance after an hour. I run Linux on Dual CPU machines with 72GB RAM. I don’t do any VOD, only live streaming. I have tweaked the OS level changes you mentioned as one of my first efforts of resolving this wall I’m up against. I’ve even tried different hardware and even different distros of Linux and Java. No luck. The confusing part, as I mentioned, is I can max out the 10Gbps connection on any of my servers if I use nginx’ RTMP module. Yet I can only max out 5Gbps with Wowza on the same server. I even tried running two instances of Wowza on the same server, but each instance would max out at 2.5Gbps. Very confusing (and frustrating) to say the least.

I will dive in deeper into GC tuning, maybe that’s where the answer to all of this lies. Again I appreciate your response and I apologize for coming across frustrated.

That moment of hope was short lived. Hit 5Gbps and the Wowza instance dropped 4,000+ connections.

Hi,

The auto-tuning in the Wowza Streaming Engine 4.x software does limit the threads to 600/400 (10 CPU cores). If you have more than 10 CPU cores, then you can manually add the additional threads using the Wowza Streaming Engine Manager. To do this, navigate to the “Server > Performance Tuning > Server Thread Pools” page and click “Edit”, then add the desired values.

Notes:

  • The Handler Thread Pool Size should be 60x [total core count]
  • The Transport Thread Pool Size should be 40x [total core count]
  • The maximum values for the above are 1024 using Wowza Streaming Engine Manager

You can also run multiple instances of Wowza Streaming Engine on the same server using this guide - How to run multiple instances of Wowza Media Server on one computer

The support team is not in a position to offer the required OS level tuning to increase the 5 Gbps Java limitation. However we will be able to assist the tuning of the Wowza Streaming Engine software if needed.

Regards,

Jason Hilton

Technical Support Engineer

Wowza Media Systems

Hi,

Let me cover off some of your comments first and then lets get into some of the options for you.

You mentioned the following

I setup two instances of Wowza on a single 10Gbps server. Guess what happened? Each instance would not go over 2.5Gbps each and eventually crashed. How do I know it isn’t a limitation of my server hardware? Because on the same server I installed nginx rtmp and was able to max out the 10Gbps connection without much effort. Yes I understand nginx rtmp and Wowza run on different environments, but I’ve yet to see any proof outside of this forum that there is a 5Gbps limit on Javas Virtual Machine. I’m not asking for the moon, I just want a straight up answer. That’s all.

Each Wowza instance would not know of the other, so the limit you mention being split - so 2.5gbps - for each one suggests a shared resource somewhere. By default Wowza only changes one OS level limitation - ulimit - and does not touch anything else. You may also find the number of threads (already mentioned) needs significantly increasing.

There are many elements which cause a reduction in performance and certainly would require some investigation by yourself, for your hardware, for your environment

  • GC tuning - At very high speeds so approaching 10gbps you need to make sure GC is cycling objects as efficiently as possible. This would mean multiple threads and allocation of a high enough level of RAM to service it. There is a tonne of reading on this subject and many consider it a black art rather than a science. The G1 collector has significant improvements however the default minimum run time may be too high.

  • Thread counts - This will need changing manually (as outlined), you may even need to alter the XML files directly as the manager will limit you to 1024 and comes down to how many cores you have available.

  • IO speeds - This can cause all sorts of weird and wonderful performance issues. For live streams this is not an obvious issue, however even so the amount of logging produced can be VERY high and so certainly something to be aware of ( I am not advocating you turn off logging ! ). I have experience of many deployments of file stores, be they local or remote which certainly can not approach 10gbps for performance. If you are serving on demand content this will cause the most issues and be the hardest to troubleshoot.

The best solution to deploying 10GBps is to use MediaCache, multiple SSDs (not raided as the performance increase for raided SSDs is very little), configured with multiple stores and use the file reader option, so making the requests for content as fast and cacheable as possible.

Regarding logging, for the network speeds you are using, where are the logs being stored, is the IO wait time high/low/not 0! - then something needs to be tweaked.

4 or 5 disks, raided, even 10k RPM are not going to get you close to the performance you need - so you do need to make sure any file storage you have is good enough, speed test them, tweak them, make sure raided sets have efficient block sizes, etc etc.

  • OS level changes. If you are using Linux (not mentioned your OS ), Wowza does make one change - ulimit - this has recently been increased from 20k to 65k - you may find increasing even further - say to128k - provides further performance increases. We have seen for some OSs, some environments, this is essential.

Further OS level changes are needed at higher speeds to get more performance

rmem_default

rmem_max

wmem_default

wmem_max

optmem_max

tcp_rmem

tcp_wmem

The last two really need to be changed from default as they are WAY too low for 10gbps and will punish performance and may well be causing your low levels. There is a tonne of reading around them based , on speed, latency, memory available etc. and is not one size fits all.

Many applications do some of this tuning automatically, Wowza does not and will not. You do need some knowledge of networking to tune these properly.

  • Memory utilisation - Tune all your applications correctly. When using high speed delivery efficient use of memory means you get the best performance. The packetizers have a default of 10second chunk duration. If you are doing live make sure they are tuned to your GOP size, so if it is 2 seconds, make the duration 2 or 4 seconds, immediately reduces memory footprint, which in turn reduces GC pressure, which reduces GC time etc etc. You can also change this for on demand delivery however YOU MUST HAVE A HIGH SPEED file system. If you are not doing a delivery of a specific HTTP protocol, turn OFF the packetizer!

Andrew.

Why dont you post your OS tweaks, and the changes you have made to your GC settings, that you have tried so far to see if they can be changed further.

Andrew.

Got it.

Wowza does not have a hard limit for connections and this is based on connectivity and hardware, however Java is limited to 5 Gbps per instance without tuning at the OS level. Ensure that all the servers/instances are tuned using the Tuning guide.

Hope this helps.

regards,

Jermaine

PaltopVince,

I would also like to hear the answers to this. I assume your using a few connections to test this speed limit. I ask because in a production env I have not been able to get anywhere near these numbers. After around 350 hls/vod connections my server load is through the roof and my network traffic will not break 1Gbps. I too have built a massive server thinking I could push it hard but have done nothing but fall short. I have been trying to push a 10g fiber connection sense Wowza3 when my company first installed.

I have had to setup alot servers in LB to support my 8k or so connections per/s

-Troy Hammonds

Reflected Networks