Wowza Community

Processor spikes running on VMWare virtual machines

We have a 3 server setup, 1 repeater - 2 edge, running under VMware ESXi on a HP ProLiant DL 120 G6, with 4 Xeon X3430 @ 2.40GHz with 12G of ram and 2 gigabit ethernet (onboard). We only have these 3 machines running there.

Config is:

repeater 1 -> 2 procs + 2G of ram

edge 1 -> 4 procs + 4G of ram

edge 2 -> 4 procs + 4G of ram

The repeater recieves 4 streams and in turn spits out 8 connections (4 per edge server if I understood correctly the repeater-edge configuration).

We only do live streaming for now, so I guess disk speed is not important.

We’re get around 700-800 connections for the 4 streams (2 are 420k, 2 are 96k).

Everything is running smooth, but we just noticed a weird trend while monitoring the servers.

At a certain point, apparently not related to number of connections, the processors spike to 90%-95% and stay like that.

The machine doesn’t die, and it keeps serving fine. I find it weird that the machine is running at 10%-20% constantly and it suddenly spikes as described before.

Any idea on what can be causing it? It happens on both the repeater and the edge servers.

Do you have VMWare tools installed on all of them?

Not sure what this would be. Patch8 is more stable than the native release of 2.2.3. I would suggest moving the latest patch and if you have problems look for errors/warnings in the log files:

http://www.wowza.com/downloads/WowzaMediaServer-2-2-3/WowzaMediaServer2.2.3-patch10.zip

Charlie

What version of Wowza are you using? Are the 3 using same version?

Have you tuned the server following the Wowza tuning guide:

https://www.wowza.com/docs/how-to-do-performance-tuning

If you have, review the guide anyway, the use of garbage collection (GC) settings has been clarified.

I suggest updating all 3 to version 2.2.3.08

https://www.wowza.com/docs/wowza-streaming-engine-software-updates

Richard

yes, patch 8 is released.

Richard

Thanks for all the answers.

We do have vmware tools installed in all of them. Actually, the 3 machines were cloned, so the setup is exactly the same, except for the config, obviously.

I will try changing the GC settings.

Is version 2.2.3.08 stable enough for a setup like this?

I still have not applied the patch (I’ll do it later today), but I can definitely say that enabling the commented out GC settings fixed the issue. Here you have a CPU monitoring chart of the past 5 days, and you can guess when I changed the settings :wink:

Mixed news here: after changing the gargage collection settings everything was running great, but after updating to patch 8 we had problems maintaining open streams for a long period of time.

Before the update to patch 8 we had a 1.2M stream running (and being watched) for over 20 hours. After the update the player would stop after 10-15 minutes, even though the stream was still there…

We had to revert to the 2.2.3 version because we couldn’t find an reason for this behavior…