Old Server, New problems - Cool commands

[ajhalls]

So I am trying to troubleshoot my VOD server which isn't a bad server but has recently started acting up. Below are the stats on the server from the ./startup.sh command. I have been running this server without any trouble since October of 2010 but then for some reason problems started a couple weeks ago. My basic setup is 3 servers, users can select the one that performs the best for them.
Server 1 (Dual Xeon E5-2620 12GB RAM RAID 0) hosts the website, mysql and a copy of the media files with Wowza installed.
Server 2 and 3 (Xeon E3-1230 8GB RAM, RAID 0) are strictly media servers with Wowza and a local copy of the media files.

The real issue is the server load climbs rapidly on Server 1 as people connect.
Server 2 & 3 may have 75-125 users actively streaming and the server is still very responsive with low lag on starting streams, low server load (0.25-0.50ish)
Server 1 may only have 25-30 people connected and we start seeing excessive server load (1.50-5.50) long lag times on streaming on PC, errors frequently on ipad streaming.

In troubleshooting I have tried upgrading the java, doing the tips suggested here http://www.wowza.com/forums/content....general-tuning, Wowza to the most recent release from the Wowza 2 I was running (need to reapply tips still), and reducing the load by adding a 4th server.

As promised in the title, I have also found some cool commands to learn more about the performance of the server such as:

iostat -dx /dev/sda 5
iotop
hdparm -T /dev/sda
nmon was pretty cool but had to be installed (yum install nmon)

Since starting to write this thread I restarted the server itself as well (hadn't been done in 200+ days) and am awaiting more info as to if that made any difference. So far so good for but posting anyway in case any of this helps someone else. If I am still having trouble, I will be back

================================================== =======================

INFO server server-start Wowza Media Server 3 Perpetual Edition 3.1.2 build1624 -
INFO server comment - Server License Key: SVRP3-XXXXX-XXXXX-XXXXX-XXXXX-VrHme
INFO server comment - Maximum Connections: Unlimited
INFO server comment - Transcoder Streams Available: 0
INFO server comment - Transcoder Watermark: Yes
INFO server comment - nDVR Available: No
INFO server comment - DRM Available: No
INFO server comment - Hardware Available Processors: 16
INFO server comment - Hardware Physical Memory: 3604MB/12001MB
INFO server comment - Hardware Swap Space: 1027MB/1027MB
INFO server comment - Max File Descriptor Count: 20000
INFO server comment - Open File Descriptor Count: 47
INFO server comment - OS Name: Linux
INFO server comment - OS Version: 2.6.18-274.12.1.el5
INFO server comment - OS Architecture: amd64
INFO server comment - Java Name: OpenJDK 64-Bit Server VM
INFO server comment - Java Vendor: Sun Microsystems Inc.
INFO server comment - Java Version: 1.6.0_22
INFO server comment - Java VM Version: 20.0-b11
INFO server comment - Java Spec Version: 1.6
INFO server comment - Java Home: /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre
INFO server comment - Java Max Heap Size: 1066MB
INFO server comment - Java Architecture: 64
INFO server comment - Java Locale[user.language]: en
INFO server comment - Java Locale[user.country]: US
INFO server comment - Java Locale[file.encoding]: UTF-8
INFO server comment - Java Timezone[user.timezone]: America/Chicago
INFO server comment - Java Args[0]: -Xmx1200M
INFO server comment - Java Args[1]: -Djava.net.preferIPv4Stack=true
INFO server comment - Java Args[2]: -Dcom.sun.management.jmxremote=true
INFO server comment - Java Args[3]: -Dcom.wowza.wms.runmode=standalone
INFO server comment - Java Args[4]: -Dcom.wowza.wms.native.base=linux
INFO server comment - Java Args[5]: -Dcom.wowza.wms.AppHome=/usr/local/WowzaMediaServer
INFO server comment - Java Args[6]: -Dcom.wowza.wms.ConfigURL=
INFO server comment - Java Args[7]: -Dcom.wowza.wms.ConfigHome=/usr/local/WowzaMediaServer
INFO server comment - Server runmode: standalone
INFO server comment - Server native.platform: linux
INFO server comment - Server threads[h/t]: 10/10
INFO server comment - CMDInterface now listening: [any]:8083
INFO vhost vhost-start _defaultVHost_ -
INFO server comment - _defaultVHost_ threads[h/t]:120/80 home:/usr/local/WowzaMediaServer
INFO vhost comment _defaultVHost_ Bind attempt ([any]:1935:14)
INFO vhost comment _defaultVHost_ Bind successful ([any]:1935)
INFO vhost comment _defaultVHost_ Bind attempt ([any]:443:14)
INFO vhost comment _defaultVHost_ Bind successful ([any]:443)
INFO vhost comment _defaultVHost_ Bind attempt ([any]:8086:1)
INFO vhost comment _defaultVHost_ Bind successful ([any]:8086)
INFO server comment - Server.startShutdownHook: Start server shutdown hook
INFO server comment - Wowza Media Server is started!

[randall]

Try updating Java to the latest 1.7 version.

[ajhalls]

Ok, so first I rebooted the server like I said I would and it ran OK for a couple days and then started to be overloaded again. I tried to update to Java 1.7 and ended up just throwing more resources at the problem which provides interesting results....
Here is the new server which has 2X the hardware (New title, New server - Old problems

INFO server server-start Wowza Media Server 3 Perpetual Edition 3.1.2 build1624 -
INFO server comment - Server License Key: SVRP3-XXXXX-XXXXX-XXXXX-XXXXX-9whwN
INFO server comment - Maximum Connections: Unlimited
INFO server comment - Transcoder Streams Available: 0
INFO server comment - Transcoder Watermark: Yes
INFO server comment - nDVR Available: No
INFO server comment - DRM Available: No
INFO server comment - Hardware Available Processors: 32
INFO server comment - Hardware Physical Memory: 3304MB/32079MB
INFO server comment - Hardware Swap Space: 4047MB/4047MB
INFO server comment - Max File Descriptor Count: 20000
INFO server comment - Open File Descriptor Count: 46
INFO server comment - OS Name: Linux
INFO server comment - OS Version: 2.6.32-279.2.1.el6.x86_64
INFO server comment - OS Architecture: amd64
INFO server comment - Java Name: OpenJDK 64-Bit Server VM
INFO server comment - Java Vendor: Oracle Corporation
INFO server comment - Java Version: 1.7.0_05-icedtea
INFO server comment - Java VM Version: 23.0-b21
INFO server comment - Java Spec Version: 1.7
INFO server comment - Java Home: /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.5.x86_64/jre
INFO server comment - Java Max Heap Size: 1066MB
INFO server comment - Java Architecture: 64
INFO server comment - Java Locale[user.language]: en
INFO server comment - Java Locale[user.country]: US
INFO server comment - Java Locale[file.encoding]: UTF-8
INFO server comment - Java Timezone[user.timezone]: US/Mountain
INFO server comment - Java Args[0]: -Xmx1200M
INFO server comment - Java Args[1]: -Djava.net.preferIPv4Stack=true
INFO server comment - Java Args[2]: -Dcom.sun.management.jmxremote=true
INFO server comment - Java Args[3]: -Dcom.wowza.wms.runmode=standalone
INFO server comment - Java Args[4]: -Dcom.wowza.wms.native.base=linux
INFO server comment - Java Args[5]: -Dcom.wowza.wms.AppHome=/usr/local/WowzaMediaServer
INFO server comment - Java Args[6]: -Dcom.wowza.wms.ConfigURL=
INFO server comment - Java Args[7]: -Dcom.wowza.wms.ConfigHome=/usr/local/WowzaMediaServer
INFO server comment - Server runmode: standalone
INFO server comment - Server native.platform: linux
INFO server comment - Server threads[h/t]: 10/10
INFO server comment - CMDInterface now listening: [any]:8083
INFO vhost vhost-start _defaultVHost_ -
INFO server comment - _defaultVHost_ threads[h/t]:120/80 home:/usr/local/WowzaMediaServer
INFO vhost comment _defaultVHost_ Bind attempt ([any]:1935:4)
INFO vhost comment _defaultVHost_ Bind successful ([any]:1935)

Now the server load is not so high, yet, but could be eventually but the big problem is that I can't stream with the ipad after the server has been up for a bit.
I can stream to the ipad just after rebooting server, then later I get "The Operation Could Not Be Completed"

As another cool command I stumbled across when setting up the new server:

yum --nogpgcheck install ffmpeg

I have a harder time installing things sometimes with WHM cPanel installed on the server, before I was getting an error about the dependencies not having the public key installed, using --nogpgcheck helped.

[rrlanham]

Have you followed the tuning guide (doesn't look like it)?
http://www.wowza.com/forums/content....ormance-tuning

You could increase java heap size, maybe double it, but not more with just 3g ram available. There must be a lot else happening on that server if it has 32g ram but only 3g available ("Hardware Physical Memory: 3304MB/32079MB")

Also, there is a load test tool you might be interested in using:
http://www.wowza.com/forums/content....Load-Test-Tool

Richard

[ajhalls]

Thanks, I got the tool going today with the following setup:

<Root>
        <Tests>
                <Test>
                        <Name>vod</Name>
                        <ClassBase>com.wowza.wms.performance.remote.PerformanceRemoteClientTest</ClassBase>
                        <Properties>
                                <Property>
                                        <Name>workerCount</Name>
                                        <Value>20</Value>
                                        <Type>Integer</Type>
                                </Property>
                                <Property>
                                        <Name>fileCount</Name>
                                        <Value>0</Value>
                                        <Type>Integer</Type>
                                </Property>
                                <Property>
                                        <Name>streamName</Name>
                                        <Value>sample</Value>
                                </Property>
                                <Property>
                                        <Name>streamExt</Name>
                                        <Value>.mp4</Value>
                                </Property>
                                <Property>
                                        <Name>vhostName</Name>
                                        <Value>_defaultVHost_</Value>
                                </Property>
                                <Property>
                                        <Name>connectionString</Name>
                                        <Value>vod</Value>
                                </Property>
                                <Property>
                                        <Name>bufferTime</Name>
                                        <Value>3000</Value>
                                        <Type>Integer</Type>
                                </Property>
                                <Property>
                                        <Name>doRandom</Name>
                                        <Value>false</Value>
                                        <Type>Boolean</Type>
                                </Property>
                                <Property>
                                        <Name>doRepeat</Name>
                                        <Value>true</Value>
                                        <Type>Boolean</Type>
                                </Property>
                                <Property>
                                        <Name>ipAddress</Name>
                                        <Value>67.228.31.186</Value>
                                </Property>
                                <Property>
                                        <Name>ports</Name>
                                        <Value>1935</Value>
                                </Property>
                        </Properties>
                </Test>
        </Tests>
</Root>

I removed the <Name>live</Name> section since our site only does vod, I changed the workercount, IP address and the connectionString settings.

I got this as the result but when I look at the :8086/serverreport/index.html I am not seeing an increased number of users:

Configure logging: file:///usr/local/WowzaMediaServer/conf/log4j.properties
INFO server comment - test: **** PerformanceRemoteClientTest *****
INFO server comment - workerCount: 20
INFO server comment - fileCount: 0
INFO server comment - streamName: sample
INFO server comment - streamExt: .mp4
INFO server comment - vhostName: _defaultVHost_
INFO server comment - connectionStr: vod
INFO server comment - doRandom: false
INFO server comment - doRepeat: true
INFO server comment - ipAddress: 67.228.31.186
INFO server comment - ports: 1935
..........
INFO server comment - start [17]: 5
INFO server comment - callConnect [17]: vod
INFO server comment - connected [17]: 823969553
INFO server comment - start [18]: 5
INFO server comment - callConnect [18]: vod
INFO server comment - connected [18]: 2137534172
INFO server comment - Status running:19 currBehind:0
INFO server comment - start [19]: 5
INFO server comment - callConnect [19]: vod
INFO server comment - connected [19]: 1751980966
INFO server comment - totalBytes: 0
INFO server comment - Status running:20 currBehind:0
INFO server comment - Status running:20 currBehind:0
INFO server comment - Status running:20 currBehind:0