One of our customer use wowza server that has 32GB RAM and 24 core processor. They have 600 live channels on this machine - each channel consisting of 5 different bitrate streams (max 512 kbps). So in all 3000 streams published to wowza.
During popular events, they get around 50000 hits per hour. The CPU utilized is around 55%-60% at this load. However the wowza system crashes in some time and gives the outofmemory error as follows:
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
They use latest wowza 3.6.3 version. Heap memory allocated to wowza server is 28 GB. All other options are default settings.
Are they publishing too many streams to wowza? What is the upper limit considering the above machine? Should the heap memory allocated be further increased? Or could the system be running out of threads? Number of open files limit (as shown by ulimit -a) is set at 32000.
Please help out urgently as there's an event tomorrow.
What are your garbage collection settings on the system? As far as published streams, our general recommendation is no more than ~250 published streams. If you are doing hls you are likely losing a lot to the packetization process.