Problems with CPU performance when upgrade Wowza Engine
Hi everyone,
I had upgraded my Wowza Server from 3.6.2 to Wowza Engine 4.0.4. I monitor my Wowza Engine Server, sometimes I see the CPU performance is so high while the number of connections likely have no change as the image below.
I use auto tuning for my Wowza Server.
Which problems may make the CPU increases too much?
Thanks.
I had upgraded my Wowza Server from 3.6.2 to Wowza Engine 4.0.4. I monitor my Wowza Engine Server, sometimes I see the CPU performance is so high while the number of connections likely have no change as the image below.
I use auto tuning for my Wowza Server.
Which problems may make the CPU increases too much?
Thanks.
Hi,
The images you have tried to show us are not being displayed for me.
If you're transcoding on this server then this will increase the CPU usage as transcoding is a high CPU intensive process.
Unfortunately without more information on the setup you have for Wowza I'm not able to troubleshoot this accurately.
Please give as much detail as you can on your setup and how you're using Wowza.
Alternatively you can create a ticket by sending an email to support@wowza.com
Please include a detailed description if the scenario and issue, a link to this thread for reference and a .zip file containing the following directories:
Wowza-Install]/conf/
[Wowza-Install]/logs/
[Wowza-Install]/transcoder/
[Wowza-Install]/manager/logs/
Please do make sure you provide logs which show Wowza server starting.
If you are not sure how to get this information please see this tutorial.
Regards,
Jason
The images you have tried to show us are not being displayed for me.
If you're transcoding on this server then this will increase the CPU usage as transcoding is a high CPU intensive process.
Unfortunately without more information on the setup you have for Wowza I'm not able to troubleshoot this accurately.
Please give as much detail as you can on your setup and how you're using Wowza.
Alternatively you can create a ticket by sending an email to support@wowza.com
Please include a detailed description if the scenario and issue, a link to this thread for reference and a .zip file containing the following directories:
Wowza-Install]/conf/
[Wowza-Install]/logs/
[Wowza-Install]/transcoder/
[Wowza-Install]/manager/logs/
Please do make sure you provide logs which show Wowza server starting.
If you are not sure how to get this information please see this tutorial.
Regards,
Jason
Hi Jason,
I had fixed my photos.
I don't use Wowza transcode. I didn't see CPU performance like that when I used Wowza 3.6.2. I realize that the CPU performance have problems when number of connections upper 175.
For more details, when the number of connections is about 170, CPU performance is about 10%.
When number of connections is higher than 175, CPU performance present like the images above.
I don't know what happened ?
Thank you.
I had fixed my photos.
I don't use Wowza transcode. I didn't see CPU performance like that when I used Wowza 3.6.2. I realize that the CPU performance have problems when number of connections upper 175.
For more details, when the number of connections is about 170, CPU performance is about 10%.
When number of connections is higher than 175, CPU performance present like the images above.
I don't know what happened ?
Thank you.
@hungnguyen88
We are experiencing the same problem after we upgraded yesterday. Can you try running iotop on your linux box and see if you see flush writing to the disk all the time?
We are experiencing the same problem after we upgraded yesterday. Can you try running iotop on your linux box and see if you see flush writing to the disk all the time?
Hi,
Please take a look at our Tuning guide which shows how to reduce the CPU usage on the server by increasing the Idle times for RTMP and RTSP protocols.
After editing the "Client Idle Frequency" and "RTP Idle Frequency" as per the tutorial, you will need to restart Wowza Streaming Engine for the changes to take effect.
You can also increase the IdleWorkers/CheckFrequency from 50 to 100 with a text editor in the [Wowza-install]/conf/VHost.xml file, again a restart will be required after editing this file.
You may also cant to change the garbage collection to use "G1 (Garbage First) collector" to see if this changes the behavior you're seeing.
Regards,
Jason
Please take a look at our Tuning guide which shows how to reduce the CPU usage on the server by increasing the Idle times for RTMP and RTSP protocols.
After editing the "Client Idle Frequency" and "RTP Idle Frequency" as per the tutorial, you will need to restart Wowza Streaming Engine for the changes to take effect.
You can also increase the IdleWorkers/CheckFrequency from 50 to 100 with a text editor in the [Wowza-install]/conf/VHost.xml file, again a restart will be required after editing this file.
You may also cant to change the garbage collection to use "G1 (Garbage First) collector" to see if this changes the behavior you're seeing.
Regards,
Jason
@Jason
We went ahead and tried your suggestions and this is what we got on our console:
626 be/4 root 0.00 B/s 1923.34 K/s 0.00 % 99.71 % [flush-8:0]
14116 be/4 root 0.00 B/s 2.43 M/s 0.00 % 96.57 % java -Xmx10000M -XX:+UseG1GC -XX:MaxG~m.wowza.wms.bootstrap.Bootstrap start
4096 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % java -Dcom.wowza.wms.ConfigURL="" -Dc~--directoryListings=false > /dev/null
1 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % init
2 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kth
We went ahead and tried your suggestions and this is what we got on our console:
626 be/4 root 0.00 B/s 1923.34 K/s 0.00 % 99.71 % [flush-8:0]
14116 be/4 root 0.00 B/s 2.43 M/s 0.00 % 96.57 % java -Xmx10000M -XX:+UseG1GC -XX:MaxG~m.wowza.wms.bootstrap.Bootstrap start
4096 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % java -Dcom.wowza.wms.ConfigURL="" -Dc~--directoryListings=false > /dev/null
1 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % init
2 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kth
As Jason mentioned, if you are still experiencing higher than expected CPU utilization, it would be useful for us to review your logs during this period of time. Please zip up your conf/ and logs/ directories for further review and, if possible, provide a stack trace when this escalation occurs as well. Send this to support@wowza.com.
Thanks,
Matt
Thanks,
Matt
Hi reyco1
Here is the result of iotop
Total DISK READ: 5.62 M/s | Total DISK WRITE: 37.04 M/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
13605 be/4 root 0.00 B/s 0.00 B/s 0.00 % 7.93 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
1094 be/4 root 0.00 B/s 167.91 K/s 0.00 % 7.36 % [kjournald]
13630 be/4 root 0.00 B/s 3.74 M/s 0.00 % 3.97 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13672 be/4 root 0.00 B/s 2.81 M/s 0.00 % 3.14 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13646 be/4 root 0.00 B/s 3.74 M/s 0.00 % 3.00 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13628 be/4 root 0.00 B/s 5.60 M/s 0.00 % 2.90 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13604 be/4 root 0.00 B/s 2.80 M/s 0.00 % 2.83 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13606 be/4 root 0.00 B/s 2.80 M/s 0.00 % 2.57 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13673 be/4 root 0.00 B/s 3.74 M/s 0.00 % 2.28 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13643 be/4 root 0.00 B/s 2.80 M/s 0.00 % 1.97 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13705 be/4 root 0.00 B/s 1705.25 K/s 0.00 % 1.73 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13820 be/4 root 14.48 M/s 0.00 B/s 0.00 % 1.55 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13624 be/4 root 3.73 K/s 1910.48 K/s 0.00 % 1.48 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13650 be/4 root 0.00 B/s 1914.21 K/s 0.00 % 1.40 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13703 be/4 root 0.00 B/s 1910.48 K/s 0.00 % 1.38 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13871 be/4 root 2.42 M/s 0.00 B/s 0.00 % 1.21 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13675 be/4 root 0.00 B/s 813.45 K/s 0.00 % 1.19 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13622 be/4 root 0.00 B/s 958.97 K/s 0.00 % 1.14 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13648 be/4 root 0.00 B/s 1917.94 K/s 0.00 % 0.97 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13900 be/4 root 2.59 M/s 0.00 B/s 0.00 % 0.95 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13603 be/4 root 0.00 B/s 529.86 K/s 0.00 % 0.28 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13980 be/4 root 182.84 K/s 0.00 B/s 0.00 % 0.21 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13874 be/4 root 201.50 K/s 0.00 B/s 0.00 % 0.18 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13949 be/4 root 4.17 M/s 0.00 B/s 0.00 % 0.17 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13977 be/4 root 145.52 K/s 0.00 B/s 0.00 % 0.16 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
14017 be/4 root 18.66 K/s 0.00 B/s 0.00 % 0.06 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13974 be/4 root 14.01 M/s 0.00 B/s 0.00 % 0.06 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13753 be/4 root 5.58 M/s 0.00 B/s 0.00 % 0.00 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13767 be/4 root 10.26 M/s 0.00 B/s 0.00 % 0.00 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13844 be/4 root 13.99 M/s 0.00 B/s 0.00 % 0.00 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13884 be/4 root 12.74 M/s 0.00 B/s 0.00 % 0.00 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13921 be/4 root 13.99 M/s 0.00 B/s 0.00 % 0.00 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13962 be/4 root 13.01 M/s 0.00 B/s 0.00 % 0.00 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
14336 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
Do you know my problem ?
Thank you.
Here is the result of iotop
Total DISK READ: 5.62 M/s | Total DISK WRITE: 37.04 M/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
13605 be/4 root 0.00 B/s 0.00 B/s 0.00 % 7.93 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
1094 be/4 root 0.00 B/s 167.91 K/s 0.00 % 7.36 % [kjournald]
13630 be/4 root 0.00 B/s 3.74 M/s 0.00 % 3.97 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13672 be/4 root 0.00 B/s 2.81 M/s 0.00 % 3.14 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13646 be/4 root 0.00 B/s 3.74 M/s 0.00 % 3.00 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13628 be/4 root 0.00 B/s 5.60 M/s 0.00 % 2.90 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13604 be/4 root 0.00 B/s 2.80 M/s 0.00 % 2.83 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13606 be/4 root 0.00 B/s 2.80 M/s 0.00 % 2.57 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13673 be/4 root 0.00 B/s 3.74 M/s 0.00 % 2.28 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13643 be/4 root 0.00 B/s 2.80 M/s 0.00 % 1.97 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13705 be/4 root 0.00 B/s 1705.25 K/s 0.00 % 1.73 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13820 be/4 root 14.48 M/s 0.00 B/s 0.00 % 1.55 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13624 be/4 root 3.73 K/s 1910.48 K/s 0.00 % 1.48 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13650 be/4 root 0.00 B/s 1914.21 K/s 0.00 % 1.40 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13703 be/4 root 0.00 B/s 1910.48 K/s 0.00 % 1.38 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13871 be/4 root 2.42 M/s 0.00 B/s 0.00 % 1.21 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13675 be/4 root 0.00 B/s 813.45 K/s 0.00 % 1.19 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13622 be/4 root 0.00 B/s 958.97 K/s 0.00 % 1.14 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13648 be/4 root 0.00 B/s 1917.94 K/s 0.00 % 0.97 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13900 be/4 root 2.59 M/s 0.00 B/s 0.00 % 0.95 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13603 be/4 root 0.00 B/s 529.86 K/s 0.00 % 0.28 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13980 be/4 root 182.84 K/s 0.00 B/s 0.00 % 0.21 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13874 be/4 root 201.50 K/s 0.00 B/s 0.00 % 0.18 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13949 be/4 root 4.17 M/s 0.00 B/s 0.00 % 0.17 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13977 be/4 root 145.52 K/s 0.00 B/s 0.00 % 0.16 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
14017 be/4 root 18.66 K/s 0.00 B/s 0.00 % 0.06 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13974 be/4 root 14.01 M/s 0.00 B/s 0.00 % 0.06 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13753 be/4 root 5.58 M/s 0.00 B/s 0.00 % 0.00 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13767 be/4 root 10.26 M/s 0.00 B/s 0.00 % 0.00 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13844 be/4 root 13.99 M/s 0.00 B/s 0.00 % 0.00 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13884 be/4 root 12.74 M/s 0.00 B/s 0.00 % 0.00 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13921 be/4 root 13.99 M/s 0.00 B/s 0.00 % 0.00 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
13962 be/4 root 13.01 M/s 0.00 B/s 0.00 % 0.00 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
14336 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % java -Xmx8000M -XX:+UseConcMarkSwee~.wowza.wms.bootstrap.Bootstrap start
Do you know my problem ?
Thank you.
