Results 1 to 5 of 5

Thread: Old Server, New problems - Cool commands

  1. #1

    Default Old Server, New problems - Cool commands

    So I am trying to troubleshoot my VOD server which isn't a bad server but has recently started acting up. Below are the stats on the server from the ./startup.sh command. I have been running this server without any trouble since October of 2010 but then for some reason problems started a couple weeks ago. My basic setup is 3 servers, users can select the one that performs the best for them.
    Server 1 (Dual Xeon E5-2620 12GB RAM RAID 0) hosts the website, mysql and a copy of the media files with Wowza installed.
    Server 2 and 3 (Xeon E3-1230 8GB RAM, RAID 0) are strictly media servers with Wowza and a local copy of the media files.

    The real issue is the server load climbs rapidly on Server 1 as people connect.
    Server 2 & 3 may have 75-125 users actively streaming and the server is still very responsive with low lag on starting streams, low server load (0.25-0.50ish)
    Server 1 may only have 25-30 people connected and we start seeing excessive server load (1.50-5.50) long lag times on streaming on PC, errors frequently on ipad streaming.

    In troubleshooting I have tried upgrading the java, doing the tips suggested here http://www.wowza.com/forums/content....general-tuning, Wowza to the most recent release from the Wowza 2 I was running (need to reapply tips still), and reducing the load by adding a 4th server.

    As promised in the title, I have also found some cool commands to learn more about the performance of the server such as:
    iostat -dx /dev/sda 5
    iotop
    hdparm -T /dev/sda
    nmon was pretty cool but had to be installed (yum install nmon)
    Since starting to write this thread I restarted the server itself as well (hadn't been done in 200+ days) and am awaiting more info as to if that made any difference. So far so good for but posting anyway in case any of this helps someone else. If I am still having trouble, I will be back

    =========================================================================

    INFO server server-start Wowza Media Server 3 Perpetual Edition 3.1.2 build1624 -
    INFO server comment - Server License Key: SVRP3-XXXXX-XXXXX-XXXXX-XXXXX-VrHme
    INFO server comment - Maximum Connections: Unlimited
    INFO server comment - Transcoder Streams Available: 0
    INFO server comment - Transcoder Watermark: Yes
    INFO server comment - nDVR Available: No
    INFO server comment - DRM Available: No
    INFO server comment - Hardware Available Processors: 16
    INFO server comment - Hardware Physical Memory: 3604MB/12001MB
    INFO server comment - Hardware Swap Space: 1027MB/1027MB
    INFO server comment - Max File Descriptor Count: 20000
    INFO server comment - Open File Descriptor Count: 47
    INFO server comment - OS Name: Linux
    INFO server comment - OS Version: 2.6.18-274.12.1.el5
    INFO server comment - OS Architecture: amd64
    INFO server comment - Java Name: OpenJDK 64-Bit Server VM
    INFO server comment - Java Vendor: Sun Microsystems Inc.
    INFO server comment - Java Version: 1.6.0_22
    INFO server comment - Java VM Version: 20.0-b11
    INFO server comment - Java Spec Version: 1.6
    INFO server comment - Java Home: /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre
    INFO server comment - Java Max Heap Size: 1066MB
    INFO server comment - Java Architecture: 64
    INFO server comment - Java Locale[user.language]: en
    INFO server comment - Java Locale[user.country]: US
    INFO server comment - Java Locale[file.encoding]: UTF-8
    INFO server comment - Java Timezone[user.timezone]: America/Chicago
    INFO server comment - Java Args[0]: -Xmx1200M
    INFO server comment - Java Args[1]: -Djava.net.preferIPv4Stack=true
    INFO server comment - Java Args[2]: -Dcom.sun.management.jmxremote=true
    INFO server comment - Java Args[3]: -Dcom.wowza.wms.runmode=standalone
    INFO server comment - Java Args[4]: -Dcom.wowza.wms.native.base=linux
    INFO server comment - Java Args[5]: -Dcom.wowza.wms.AppHome=/usr/local/WowzaMediaServer
    INFO server comment - Java Args[6]: -Dcom.wowza.wms.ConfigURL=
    INFO server comment - Java Args[7]: -Dcom.wowza.wms.ConfigHome=/usr/local/WowzaMediaServer
    INFO server comment - Server runmode: standalone
    INFO server comment - Server native.platform: linux
    INFO server comment - Server threads[h/t]: 10/10
    INFO server comment - CMDInterface now listening: [any]:8083
    INFO vhost vhost-start _defaultVHost_ -
    INFO server comment - _defaultVHost_ threads[h/t]:120/80 home:/usr/local/WowzaMediaServer
    INFO vhost comment _defaultVHost_ Bind attempt ([any]:1935:14)
    INFO vhost comment _defaultVHost_ Bind successful ([any]:1935)
    INFO vhost comment _defaultVHost_ Bind attempt ([any]:443:14)
    INFO vhost comment _defaultVHost_ Bind successful ([any]:443)
    INFO vhost comment _defaultVHost_ Bind attempt ([any]:8086:1)
    INFO vhost comment _defaultVHost_ Bind successful ([any]:8086)
    INFO server comment - Server.startShutdownHook: Start server shutdown hook
    INFO server comment - Wowza Media Server is started!

  2. #2

    Default

    Try updating Java to the latest 1.7 version.

  3. #3

    Default

    Ok, so first I rebooted the server like I said I would and it ran OK for a couple days and then started to be overloaded again. I tried to update to Java 1.7 and ended up just throwing more resources at the problem which provides interesting results....
    Here is the new server which has 2X the hardware (New title, New server - Old problems
    INFO server server-start Wowza Media Server 3 Perpetual Edition 3.1.2 build1624 -
    INFO server comment - Server License Key: SVRP3-XXXXX-XXXXX-XXXXX-XXXXX-9whwN
    INFO server comment - Maximum Connections: Unlimited
    INFO server comment - Transcoder Streams Available: 0
    INFO server comment - Transcoder Watermark: Yes
    INFO server comment - nDVR Available: No
    INFO server comment - DRM Available: No
    INFO server comment - Hardware Available Processors: 32
    INFO server comment - Hardware Physical Memory: 3304MB/32079MB
    INFO server comment - Hardware Swap Space: 4047MB/4047MB
    INFO server comment - Max File Descriptor Count: 20000
    INFO server comment - Open File Descriptor Count: 46
    INFO server comment - OS Name: Linux
    INFO server comment - OS Version: 2.6.32-279.2.1.el6.x86_64
    INFO server comment - OS Architecture: amd64
    INFO server comment - Java Name: OpenJDK 64-Bit Server VM
    INFO server comment - Java Vendor: Oracle Corporation
    INFO server comment - Java Version: 1.7.0_05-icedtea
    INFO server comment - Java VM Version: 23.0-b21
    INFO server comment - Java Spec Version: 1.7
    INFO server comment - Java Home: /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.5.x86_64/jre
    INFO server comment - Java Max Heap Size: 1066MB
    INFO server comment - Java Architecture: 64
    INFO server comment - Java Locale[user.language]: en
    INFO server comment - Java Locale[user.country]: US
    INFO server comment - Java Locale[file.encoding]: UTF-8
    INFO server comment - Java Timezone[user.timezone]: US/Mountain
    INFO server comment - Java Args[0]: -Xmx1200M
    INFO server comment - Java Args[1]: -Djava.net.preferIPv4Stack=true
    INFO server comment - Java Args[2]: -Dcom.sun.management.jmxremote=true
    INFO server comment - Java Args[3]: -Dcom.wowza.wms.runmode=standalone
    INFO server comment - Java Args[4]: -Dcom.wowza.wms.native.base=linux
    INFO server comment - Java Args[5]: -Dcom.wowza.wms.AppHome=/usr/local/WowzaMediaServer
    INFO server comment - Java Args[6]: -Dcom.wowza.wms.ConfigURL=
    INFO server comment - Java Args[7]: -Dcom.wowza.wms.ConfigHome=/usr/local/WowzaMediaServer
    INFO server comment - Server runmode: standalone
    INFO server comment - Server native.platform: linux
    INFO server comment - Server threads[h/t]: 10/10
    INFO server comment - CMDInterface now listening: [any]:8083
    INFO vhost vhost-start _defaultVHost_ -
    INFO server comment - _defaultVHost_ threads[h/t]:120/80 home:/usr/local/WowzaMediaServer
    INFO vhost comment _defaultVHost_ Bind attempt ([any]:1935:4)
    INFO vhost comment _defaultVHost_ Bind successful ([any]:1935)
    Now the server load is not so high, yet, but could be eventually but the big problem is that I can't stream with the ipad after the server has been up for a bit.
    I can stream to the ipad just after rebooting server, then later I get "The Operation Could Not Be Completed"

    As another cool command I stumbled across when setting up the new server:
    yum --nogpgcheck install ffmpeg
    I have a harder time installing things sometimes with WHM cPanel installed on the server, before I was getting an error about the dependencies not having the public key installed, using --nogpgcheck helped.
    Last edited by ajhalls; 07-27-2012 at 03:56 PM.

  4. #4
    Join Date
    Dec 2007
    Posts
    21,962

    Default

    Have you followed the tuning guide (doesn't look like it)?
    http://www.wowza.com/forums/content....ormance-tuning

    You could increase java heap size, maybe double it, but not more with just 3g ram available. There must be a lot else happening on that server if it has 32g ram but only 3g available ("Hardware Physical Memory: 3304MB/32079MB")

    Also, there is a load test tool you might be interested in using:
    http://www.wowza.com/forums/content....Load-Test-Tool

    Richard

  5. #5

    Default

    Thanks, I got the tool going today with the following setup:
    <Root>
            <Tests>
                    <Test>
                            <Name>vod</Name>
                            <ClassBase>com.wowza.wms.performance.remote.PerformanceRemoteClientTest</ClassBase>
                            <Properties>
                                    <Property>
                                            <Name>workerCount</Name>
                                            <Value>20</Value>
                                            <Type>Integer</Type>
                                    </Property>
                                    <Property>
                                            <Name>fileCount</Name>
                                            <Value>0</Value>
                                            <Type>Integer</Type>
                                    </Property>
                                    <Property>
                                            <Name>streamName</Name>
                                            <Value>sample</Value>
                                    </Property>
                                    <Property>
                                            <Name>streamExt</Name>
                                            <Value>.mp4</Value>
                                    </Property>
                                    <Property>
                                            <Name>vhostName</Name>
                                            <Value>_defaultVHost_</Value>
                                    </Property>
                                    <Property>
                                            <Name>connectionString</Name>
                                            <Value>vod</Value>
                                    </Property>
                                    <Property>
                                            <Name>bufferTime</Name>
                                            <Value>3000</Value>
                                            <Type>Integer</Type>
                                    </Property>
                                    <Property>
                                            <Name>doRandom</Name>
                                            <Value>false</Value>
                                            <Type>Boolean</Type>
                                    </Property>
                                    <Property>
                                            <Name>doRepeat</Name>
                                            <Value>true</Value>
                                            <Type>Boolean</Type>
                                    </Property>
                                    <Property>
                                            <Name>ipAddress</Name>
                                            <Value>67.228.31.186</Value>
                                    </Property>
                                    <Property>
                                            <Name>ports</Name>
                                            <Value>1935</Value>
                                    </Property>
                            </Properties>
                    </Test>
            </Tests>
    </Root>
    I removed the <Name>live</Name> section since our site only does vod, I changed the workercount, IP address and the connectionString settings.

    I got this as the result but when I look at the :8086/serverreport/index.html I am not seeing an increased number of users:
    Configure logging: file:///usr/local/WowzaMediaServer/conf/log4j.properties
    INFO server comment - test: **** PerformanceRemoteClientTest *****
    INFO server comment - workerCount: 20
    INFO server comment - fileCount: 0
    INFO server comment - streamName: sample
    INFO server comment - streamExt: .mp4
    INFO server comment - vhostName: _defaultVHost_
    INFO server comment - connectionStr: vod
    INFO server comment - doRandom: false
    INFO server comment - doRepeat: true
    INFO server comment - ipAddress: 67.228.31.186
    INFO server comment - ports: 1935
    ..........
    INFO server comment - start [17]: 5
    INFO server comment - callConnect [17]: vod
    INFO server comment - connected [17]: 823969553
    INFO server comment - start [18]: 5
    INFO server comment - callConnect [18]: vod
    INFO server comment - connected [18]: 2137534172
    INFO server comment - Status running:19 currBehind:0
    INFO server comment - start [19]: 5
    INFO server comment - callConnect [19]: vod
    INFO server comment - connected [19]: 1751980966
    INFO server comment - totalBytes: 0
    INFO server comment - Status running:20 currBehind:0
    INFO server comment - Status running:20 currBehind:0
    INFO server comment - Status running:20 currBehind:0
    Last edited by ajhalls; 07-31-2012 at 07:10 AM.

Similar Threads

  1. Execute FFMPEG commands when stream published
    By hussanchoudhry in forum General Forum
    Replies: 3
    Last Post: 11-27-2014, 08:18 AM
  2. Action script to send PTZ Commands
    By victor.moraes in forum Media Players and Devices
    Replies: 1
    Last Post: 10-11-2014, 12:14 AM
  3. Smooth Streaming - Script Commands
    By mcaporali in forum Live Streaming and Encoder Discussion
    Replies: 1
    Last Post: 11-07-2013, 09:40 AM
  4. Outbound telnet / socket commands
    By art.vandelay in forum Server-side Modules and Code Samples Discussion
    Replies: 1
    Last Post: 03-20-2012, 05:12 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •