2、When we use only 16 cpu cores, wowza can support about 1700 connectins, and when increase more connections, clients will get timeout, but different from above, VisualVM shows that nio.SocketIoProcessor$Worker.run() used the most cpu time. We looked into the GC log, and found that with 1700 connections, each Eden Area GC left object need about 50M-100M to store in survivor area (by gc log's "age 1" keyword), but with 1800 connections, each Eden Area GC left much more objects, need 500M-2G memory in survivor area.
3、When we use only 8 cpu cores, wowza can support about 1900 connectins, and when increase more connections, clients will get timeout, VisualVM and gc log is similar to 16 cpu cores.
4、Base on the results of case 1 and case 3, we start 2 wowza on this server, each using 8 cpu cores, then first start 1800 connections to the first wowza, the wowza works well, and then we start about 200 connections to the second wowza, the clients connected to the first wowza get timeout, VisualVM and gc log is similar to case 2.
Base from above test cases, something looks like strange:
1、Why with 8 cpu cores can achieve more connections than 16 cpu cores and 32 cpu cores? They have the same momory, same configrations.
2、Why when the connection increase to the limits, Eden area GC left much more alive objects? We are sure that clients create connectons and get ts evenly.
3、In case 4, why the seconds wowza influence the first one? They use different cpu cores.
With 1Mbps vod file to test, wowza can achieve 4500 connections by using 8 cpu cores, so it seem that the configration is not the problem.
Following is one of our startup command, and we using command "taskset" to limit wowza only can use some cpu cores, for example, "taskset 0x0000FFFF java " will limit the java only can use cpu core 0 to 15, can not use cpu core 16 to 31.
If it isn't your network configuration which is causing the issues then I am not sure. We don't have any way of reliably generating that amount of traffic to be able to test properly.
I'm not trying to shift blame but can you guarantee that it isn't the way that you are generating the traffic that is causing the issue or maybe some other part of the network, rather than the actual server you are testing.
What I don't understand is why people insist on wanting to use a single huge machine to do their streaming when multiple smaller machines would be more cost effective and provide automatic redundancy. If you one 20gb machine goes down, you lose 20gb of traffic. If you have 5 4gb machines and one goes down, you only lose 4gb of traffic.
An i7 or E3 based server with 16GB ram and quad 1GB nics bonded will reach its network limit very easily. A pool of these and associated switches would be a lot more cost effective than a single larger machine and 10gb switches.
We do have customers that are using hardware similar to yours with single 10gb nics but I'm not aware of anyone using multiple 10gb nics.
That being said, it may be possible but you will have to test different configurations and find one that works. As I mentioned already, we don't currently have the resources to generate the amount of traffic to test this type of configuration.
I would suggest the following. You may have already done some of this.
Confirm that it isn't the player side causing the issues. If generating test type connections, make sure the testing servers aren't getting overloaded. Remember that real connections don't all come from a handful of locations. Try to replicate real world situations.
Make sure your OS is tuned properly to work with the nics. Most OS's aren't specifically tuned to utilise these. Monitor the actual traffic at the nic level to make sure it isn't something there that is causing the issues.
On the Wowza side, you may need to look at different garbage collectors. The alternative for the Oracle JVM is the G1 garbage collector. It does improve the pausing issues seen with the concurrent garbage collector. You may also need to look at a commercial JVM such as
Azul Zing. If garbage collection pauses is the issue then the say their JVM will work a lot better.
Also look at the thread pool sizes. You should monitor the server thread pools with visualVM to make sure there are not too many threads in monitor state. If so, increasing the thread pool and processor counts may help. The handler pools are used for internal processing and the transport and processor pools are used to handle the actual network side of the streaming.
There is already a ticket open for this so if you need any further assistance want someone to look at test results, please use this ticket.