Page 1 of 2 12 LastLast
Results 1 to 10 of 13

Thread: WOWZ™ protocol triggered by upgrade to all v3.5

  1. #1
    Join Date
    Dec 2010
    Posts
    24

    Default WOWZ™ protocol triggered by upgrade to all v3.5

    We've been running an origin/edge config successfully for two years, re-streaming IP network cameras with millions of views per year.

    In the last two months, we moved our edge servers to new DC's and upgraded to v3.5 . No problems. Origin still on v3.1.0 build1410 .

    Yesterday we moved our origin server to a different DC and installed v3.5.

    In old and new setups, origin also runs a local edge config.

    The new all-3.5 cluster would not reliably stream RTMP to Flash clients. We use Flowplayer v3.2.14 with their RTMP plug-in v3.2.11 ... nothing changed in that config.

    RTMP streaming was equally unreliable whether we connected via the remote edge servers or via the edge local to the origin, tending to discount the new paths between edge and origin DC's. HLS streaming to Apple devices (from the edge on the origin box) was working fine... in fact, better than before!

    Unreliable = maybe 1 in 4 connection attempts would yield video, the others would just give us a black screen, even though Flowplayer thought the stream started.

    Noticed that an all v3.5 config triggers your new WOWZ protocol ... from your 3.5 manual:
    The WOWZ(tm) protocol is a new TCP-based messaging protocol in Wowza Media Server 3.5 and is used for server-to-server communication. It's enabled by default. If one of the Wowza Media Servers in the origin/edge configuration isn't running Wowza Media Server 3.5, an RTMP connection will be established between the servers instead.

    The only error message of note is the one we reported two years ago but with no solution...

    NetConnectionConnection.connect: Failed to connect[localhost:1935]: org.apache.mina.common.RuntimeIOException: java.net.SocketException: Invalid argument: no further information

    Java+Windows socket issue....

    We still see this on our new edge servers (64-bit Windows Server 2008 + 64-bit Java 1.7.0_09) and the above on our new origin (32-bit Windows Server 2008 + 32-bit Java 1.7.0_10). First time we have seen it on a 32-bit WinOS and first time on a loopback local connection.

    Random connect failures could explain it, especially if your new WOWZ protocol is using different code paths, timing, call parameters or whatever else triggers this issue with Java.

    I'm contemplating adding a v3.1 install to the new origin box to try and remove the number of variables in this move.

    The only way we got around the SocketException errors for the last two years was to run all 32-bit WinOS + 32-bit Java ... never saw the problem again. Now we need bigger servers and greater than 4GB memory space.... hence 64-bit . Hoped that sticking with 32-bit on the new origin would dodge that bullet but apparently not.

    Any other suggestions or insights given the above symptoms?

    Thanks!

  2. #2
    Join Date
    Dec 2007
    Posts
    21,962

    Default

    Two things changed, according to your report, the origin was updated to 3.5, and it was moved.

    An origin should be the same version as each edge. Maybe the move is a factor? The log message you show is Wowza reporting a network problem.

    NetConnectionConnection.connect: Failed to connect[localhost:1935]: org.apache.mina.common.RuntimeIOException: java.net.SocketException: Invalid argument: no further information

    Richard

  3. #3
    Join Date
    Dec 2010
    Posts
    24

    Default

    Origin and edge with different versions is working fine for us. All v3.5 is not but that's probably not the core issue.

    I'm looking at this thread:
    https://issues.apache.org/jira/browse/DIRMINA-379

    ... which suggests the error message is more than just a "network problem".

    A problem with the network stack... yes. An incompatibility / bug exposure between Java and Windows TCP/IP sockets ... yes. An issue that can be resolved at the Java application level... maybe. An issue that can be resolved by tweaking TCP/IP behavior parameters... maybe.

    Please don't dismiss this issue as someone else's problem. It's a problem for one of your customers and has been for two years.

    I will try and create a reproducible environment on our new origin server which is out of production right now.

  4. #4
    Join Date
    Dec 2007
    Posts
    21,962

    Default

    Okay, that would be helpful.

    What is the two year old problem?

    Richard

  5. #5
    Join Date
    Dec 2010
    Posts
    24

    Default

    Quote Originally Posted by rrlanham View Post

    What is the two year old problem?
    http://www.wowza.com/forums/showthre...valid-argument

    Same SocketException error as above. May or may not be related to our new origin issue. Just saying that it shows in the logs and in our new edge servers which were recently installed with Win Server 2008 64-bit.

    We use StartupStreams.xml and most times, all our streams from origin to edge start okay on the edge. Sometimes they don't and then takes random number of retries, walking up the free socket list. Occasionally one or two never start and we have to reboot. We don't reset the edge servers too often so it's manageable. We run the origin to edge streams 24/7 and once started successfully... don't have a problem.

    When we reported two years ago, we took the discussion offline and both you and Charlie punted on a solution. So we switched DC's and went all 32-bit until last month.

    Our new origin should really be a 64-bit environment too but we were scared about running into this issue in a more critical place. Seems we may have hit it anyway.

    It should not be happening on any OS-Wowza combination.

  6. #6
    Join Date
    Dec 2010
    Posts
    24

    Default

    Quote Originally Posted by rainman View Post

    I will try and create a reproducible environment on our new origin server which is out of production right now.
    Very reproducible as follows using your Flash RTMP Player example (to rule out our Flowplayer config):

    Always works
    ============
    Server: rtmp://(oldOriginServer)/liveorigin
    Stream: camorigin.stream

    Server: rtmp://(newOriginServer)/liveorigin
    Stream: camorigin.stream

    Server: rtmp://(oldOriginServer)/liveedge
    Stream: cam.stream

    Unreliable (black screen more than 75% of the time)
    =======================================
    Server: rtmp://(newOriginServer)/liveedge
    Stream: cam.stream

    *** Connecting to the edge app on the new origin server is the only case that fails in these tests.

    cam.stream contains :

    rtmp://localhost:1935/liveorigin/_definst_/camorigin.stream
    camorigin.stream contains :

    rtsp://username:password@(ourCamIP):554/axis-media/media.amp?videocodec=h264&streamprofile=TEST
    Both streams started in startupstreams.xml ...

    		<!-- Cam Origin Stream -->
    		<StartupStream>
    			<Application>liveorigin/_definst_</Application>
    			<MediaCasterType>rtp</MediaCasterType>
    			<StreamName>camorigin.stream</StreamName>
    		</StartupStream>
    		<!-- Cam Edge Stream -->
    		<StartupStream>
    			<Application>liveedge/_definst_</Application>
    			<MediaCasterType>liverepeater</MediaCasterType>
    			<StreamName>cam.stream</StreamName>
    		</StartupStream>
    "INFO stream play cam.stream" is always logged to the console when attempting a client connect... whether video appears or not.

    We use ModuleHotlinkDenial which always reports first.

    No errors logged on server.

    Only one warning on startup as follows:
    LiveStreamPacketizerSmoothStreaming.handlePacket[liveorigin/_definst_/camorigin.stream]: Fragment duration greater than suggested range of 1-4 seconds. Adjust keyframe interval accordingly: Fragment durations: [5.0,5.0,5.0]
    Any ideas?

  7. #7
    Join Date
    Dec 2007
    Posts
    21,962

    Default

    Zip up and send /conf and /logs folders from the new origin and an active edge to support@wowza.com. I may follow with some GC suggestions after I take a look at what you are doing now.

    Also send live links for the tests in your previous post. What player are you testing with?

    Are your edge and origin servers located far from each other? Describe the cluster distribution in the email above.

    Include a link to this thread for reference.

    Suggestions:

    1. Make sure all drivers are up to date, especially the network drivers.

    2. In conf/MediaCaster.xml try changing the ConnectionTimeout for the liverepeater MediaCaster type from 4000 to 6000

    Richard

  8. #8
    Join Date
    Dec 2010
    Posts
    24

    Default

    Thank you Richard. I have just emailed everything you requested to support@wowza.com .

    Let me know if you need anything else.

    David.

  9. #9
    Join Date
    Dec 2010
    Posts
    24

    Default

    Quote Originally Posted by rrlanham View Post
    Suggestions:

    1. Make sure all drivers are up to date, especially the network drivers.

    2. In conf/MediaCaster.xml try changing the ConnectionTimeout for the liverepeater MediaCaster type from 4000 to 6000

    Richard
    1. Network driver is old(ish) but latest on MSFT update.

    2. Tried ConnectionTimeout of 6000 instead of 4000 ... no difference.

  10. #10

    Default

    Hi There -

    Not sure that my issue is the same but I too am experiencing a very similar issue when upgrading systems to 3.5. All of my systems are running either Centos 6.2 or Centos 6.3. Each with more than enough RAM. I have four transcoding / origin servers... 3 are running version 3.1.2. The 4th, running 3.5. I currently have 8 edge servers delivering streams. 4 are running V3.1.2 with the remaining 4 running V3.5. Each of the 8 edge servers load the same streams from all 4 transcoding/origin servers via RTMP URLs configured in the aliasmap.stream.txt file and started via StartupStreams.xml. HLS streaming as David reports works great. It seems that all IP camera streams ingested into transcoder/origin servers 1, 2 and 3 running V3.1.2 are fine and stable. The RTMP streams being pulled from Transcoder/Origin 4 running V3.5 appear to start half the time but show black. If I try to reload the stream with the player a few times, it works... I imagine that each time I try to restart the stream, it either pulls a stream from a V3.5 edge server or a V3.1.2 edge server. If V.35 edge server, I get black... ?? Since introducing V3.5, I've begun seeing these issues. Is WOWZ taking place between the V3.5 Edge server and V3.5 Origin automatically? Should the config be different?

    Thank you in advance,

    Tim

Page 1 of 2 12 LastLast

Similar Threads

  1. how is wowz protocol deffirrent rtmp
    By hehe37 in forum General Forum
    Replies: 3
    Last Post: 07-28-2014, 10:22 AM
  2. WOWZ reconnect
    By robstreamhosting in forum General Forum
    Replies: 7
    Last Post: 05-17-2013, 07:54 AM
  3. onMetaData event not triggered.
    By deshani in forum Server-side Modules and Code Samples Discussion
    Replies: 2
    Last Post: 01-25-2012, 05:26 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •