Wowza Community

Proxy issues with SMIL

Hi,

I am trying to figure out what may cause the following problem.

Several users are accessing a Wowza via a Squid proxy. The site uses jwplayer 6. Clients have the latest flash-player installed and active. Wowza listens on 1935 and 80.

Problem: only the first video plays. As long as one video is playing on gets “Error loading stream: Manifest not found or invalid” for the other videos.

I had a look with wireshark and can see that the second and following requests for the SMIL are answered by a new session id. That surprises me.

First attempt looks like this:

.
HTTP/1.0 404 Not Found
Content-Type: application/x-fcs
Server: FlashCom/3.5.7
Cache-Control: no-cache
Content-Length: 0
X-Cache: MISS from PROXY
X-Cache-Lookup: MISS from LXBM-D6.NH-Hamburg.NewHorizons.de:3128
Via: 1.0 LXBM-D6.NH-Hamburg.NewHorizons.de:3128 (squid/2.7.STABLE9)
Connection: close
[builds new connection]
POST http://stream.instantlearning.biz/open/1 HTTP/1.1
Content-Type: application/x-fcs
User-Agent: Shockwave Flash
Host: stream.instantlearning.biz
Content-Length: 1
Proxy-Connection: Keep-Alive
Pragma: no-cache
HTTP/1.0 200 OK
Content-Type: application/x-fcs
Server: FlashCom/3.5.7
Cache-Control: no-cache
Content-Length: 11
X-Cache: MISS from LXBM-D6.NH-Hamburg.NewHorizons.de
X-Cache-Lookup: MISS from LXBM-D6.NH-Hamburg.NewHorizons.de:3128
Via: 1.0 LXBM-D6.NH-Hamburg.NewHorizons.de:3128 (squid/2.7.STABLE9)
Connection: keep-alive
Proxy-Connection: keep-alive
1564440512
POST http://stream.instantlearning.biz/idle/1564440512/0 HTTP/1.1
Content-Type: application/x-fcs
User-Agent: Shockwave Flash
Host: stream.instantlearning.biz
Content-Length: 1
Proxy-Connection: Keep-Alive
Pragma: no-cache
HTTP/1.0 200 OK
Content-Type: application/x-fcs
Server: FlashCom/3.5.7
Cache-Control: no-cache
Content-Length: 1
X-Cache: MISS from PROXY
X-Cache-Lookup: MISS from PROXY:3128
Via: 1.0 PROXY:3128 (squid/2.7.STABLE9)
Connection: keep-alive
Proxy-Connection: keep-alive

So that looks as expected. The SMIL is loaded, the connection is attempted, the player realizes it has to go via the proxy, opens a new stream, gets an id and happily plays the video.

But then the second attempt to get a SMIL looks like this:

GET http://stream.instantlearning.biz/ilsgo/smil:ILS_3030020_ILS_First_Steps.smil/jwplayer.smil HTTP/1.1
Accept: */*
Accept-Language: de-DE
Referer: http://go.instantlearning.biz/fileadmin/res/js/jwplayer.flash.swf
x-flash-version: 11,7,700,169
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)
Host: stream.instantlearning.biz
Proxy-Connection: Keep-Alive
HTTP/1.0 200 OK
Content-Type: application/x-fcs
Server: FlashCom/3.5.7
Cache-Control: no-cache
Content-Length: 11
X-Cache: MISS from PROXY
X-Cache-Lookup: MISS from PROXY:3128
Via: 1.0 PROXY:3128 (squid/2.7.STABLE9)
Connection: keep-alive
Proxy-Connection: keep-alive
1948670105
POST http://stream.instantlearning.biz/idle/1564440512/105 HTTP/1.1
Content-Type: application/x-fcs
User-Agent: Shockwave Flash
Host: stream.instantlearning.biz
Content-Length: 1
Proxy-Connection: Keep-Alive
Pragma: no-cache
HTTP/1.0 200 OK
Content-Type: application/x-fcs
Server: FlashCom/3.5.7
Cache-Control: no-cache
Content-Length: 1
X-Cache: MISS from PROXY
X-Cache-Lookup: MISS from PROXY:3128
Via: 1.0 PROXY:3128 (squid/2.7.STABLE9)
Connection: keep-alive
Proxy-Connection: keep-alive

The id 1948670105 is never used anywhere later. I wasn’t expecting an id anyway, as the player sent a GET for a SMIL, not a POST to open a new stream.

What may I be doing wrong?

Best

Henrik

Henrik,

First, apply the following patch because you are using a version that had a problem with this technique. You’ll notice that the system-bitrate values are given as kbs in the jwplayer.smil rendition.

http://www.wowza.com/downloads/WowzaMediaServer-3-5-2/WowzaMediaServer3.5.2-patch8.zip

Do you have any other HTTPProviders defined in the /conf/VHost.xml /HostPort that you are using for the jwplayer.smil?

Richard

The problem is Wowza Server does not support HTTP pipelining in all cases. Once a connection is determined to be RTMPT then we cannot switch that TCP connection back to serving up the SMIL file. The best course of action is to see if there is some way to force the proxy to not pipeline requests. We need the second SMIL request to be over a different TCP connection or it will not work.

We may support pipelining in the future but at this time we have no plans to support this feature.

Charlie

We will consider for a future release.

Charlie

Henrik,

That looks okay. See if applying that patch and getting the right system-bitrate fixes this. Let me know

Richard

Cool. Thanks for the quick update

Richard

You cannot proxy RTMP traffic.

Richard

The only way to fallback to rtmpt is in the client.

Richard

Thank you, Richard,

Yes, it looks like this:

                        <HostPort>
                                <ProcessorCount>4</ProcessorCount>
                                <IpAddress>xxx</IpAddress>
                                <!-- Separate multiple ports with commas -->
                                <!-- 80: HTTP, RTMPT -->
                                <!-- 554: RTSP -->
                                <Port>1935,80</Port>
                                <SocketConfiguration>
                                        <ReuseAddress>true</ReuseAddress>
                                        <!-- suggested settings for video on demand applications -->
                                        <ReceiveBufferSize>65000</ReceiveBufferSize>
                                        <ReadBufferSize>65000</ReadBufferSize>
                                        <SendBufferSize>65000</SendBufferSize>
                                        <!-- suggest settings for low latency chat and video recording applications
                                        <ReceiveBufferSize>32000</ReceiveBufferSize>
                                        <ReadBufferSize>32000</ReadBufferSize>
                                        <SendBufferSize>32000</SendBufferSize>
                                        -->
                                        <KeepAlive>true</KeepAlive>
                                        <!-- <TrafficClass>0</TrafficClass> -->
                                        <!-- <OobInline>false</OobInline> -->
                                        <!-- <SoLingerTime>-1</SoLingerTime> -->
                                        <!-- <TcpNoDelay>false</TcpNoDelay> -->
                                        <AcceptorBackLog>100</AcceptorBackLog>
                                </SocketConfiguration>
                                <HTTPStreamerAdapterIDs>cupertinostreaming,smoothstreaming,sanjosestreaming,dvrchunkstreaming</HTTPStreamerAdapterIDs>
                                <HTTPProviders>
                                        <HTTPProvider>
                                                <BaseClass>com.wowza.wms.http.HTTPCrossdomain</BaseClass>
                                                <RequestFilters>*crossdomain.xml</RequestFilters>
                                                <AuthenticationMethod>none</AuthenticationMethod>
                                        </HTTPProvider>
                                        <HTTPProvider>
                                                <BaseClass>com.wowza.wms.http.HTTPClientAccessPolicy</BaseClass>
                                                <RequestFilters>*clientaccesspolicy.xml</RequestFilters>
                                                <AuthenticationMethod>none</AuthenticationMethod>
                                        </HTTPProvider>
                                        <HTTPProvider>
                                                <BaseClass>com.wowza.wms.http.HTTPProviderMediaList</BaseClass>
                                                <RequestFilters>*jwplayer.rss|*jwplayer.smil|*medialist.smil|*manifest-rtmp.f4m</RequestFilters>
                                                <AuthenticationMethod>none</AuthenticationMethod>
                                        </HTTPProvider>
                                        <HTTPProvider>
                                                <BaseClass>com.wowza.wms.http.HTTPServerVersion</BaseClass>
                                                <RequestFilters>*</RequestFilters>
                                                <AuthenticationMethod>none</AuthenticationMethod>
                                        </HTTPProvider>
                                </HTTPProviders>
 

If this looks clueless, it is. I just took over from somebody else.

I’ll apply the patch asap.

Thanks again

Henrik

Patch seems to have done the trick. I’ll now try to understand the bitrate issue.

Thank you very much!

Henrik

Okay, understood. Bitrates seem to be okay now as well.

Gave the thumbs up too early. I forced all traffic to go via the proxy and there it is again.

Same thing again. Sorry.

I click on the second video. Used IE developer tools this time to capture the reuest and response.

Request headers

Key	Value
Request	GET http://stream.instantlearning.biz/ilsgo/smil:ILS_215_0140_The_Ruler.smil/jwplayer.smil HTTP/1.1
Accept	*/*
Accept-Language	de-DE
Referer	http://go.instantlearning.biz/fileadmin/res/js/jwplayer.flash.swf
x-flash-version	11,7,700,169
Accept-Encoding	gzip, deflate
User-Agent	Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)
Host	stream.instantlearning.biz
Proxy-Connection	Keep-Alive

Response headers

Key	Value
Response	HTTP/1.0 200 OK
Content-Type	application/x-fcs
Server	FlashCom/3.5.7
Cache-Control	no-cache
Content-Length	9
X-Cache	MISS from PROXY
X-Cache-Lookup	MISS from PROXY:3128
Via	1.0 PROXY:3128 (squid/2.7.STABLE9)
Connection	keep-alive
Proxy-Connection	keep-alive

and the response body

54159775

I was expecting the SMIL here. So was the player, it displays “Error loading stream: Manifest not found or invalid” again.

I believe when I said it worked before, I had accidentally allowed direct internet access, so browser was able to talk to Wowza directly.

Henrik

The patch is applied “Wowza 3 Perpetual OEM Sonic Performance 3.5.2.08 build4955”.

I do not think it is the proxy returning wrong content from the cache, as I always get different numbers. See here:

GET http://stream.instantlearning.biz/ilsgo/smil:ILS_215_0140_The_Ruler.smil/jwplayer.smil HTTP/1.1
Accept: */*
Accept-Language: de-DE
Referer: http://go.instantlearning.biz/fileadmin/res/js/jwplayer.flash.swf
x-flash-version: 11,7,700,169
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)
Host: stream.instantlearning.biz
Proxy-Connection: Keep-Alive
HTTP/1.0 200 OK
Content-Type: application/x-fcs
Server: FlashCom/3.5.7
Cache-Control: no-cache
Content-Length: 11
X-Cache: MISS from PROXY
X-Cache-Lookup: MISS from PROXY:3128
Via: 1.0 PROXY:3128 (squid/2.7.STABLE9)
Connection: keep-alive
Proxy-Connection: keep-alive
1229250553
GET http://stream.instantlearning.biz/ilsgo/smil:ILS_215_0140_The_Ruler.smil/jwplayer.smil HTTP/1.1
Accept: */*
Accept-Language: de-DE
Referer: http://go.instantlearning.biz/fileadmin/res/js/jwplayer.flash.swf
x-flash-version: 11,7,700,169
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)
Host: stream.instantlearning.biz
Proxy-Connection: Keep-Alive
HTTP/1.0 200 OK
Content-Type: application/x-fcs
Server: FlashCom/3.5.7
Cache-Control: no-cache
Content-Length: 10
X-Cache: MISS from PROXY
X-Cache-Lookup: MISS from PROXY:3128
Via: 1.0 PROXY:3128 (squid/2.7.STABLE9)
Connection: keep-alive
Proxy-Connection: keep-alive
263577491
GET http://stream.instantlearning.biz/ilsgo/smil:ILS_215_0140_The_Ruler.smil/jwplayer.smil HTTP/1.1
Accept: */*
Accept-Language: de-DE
Referer: http://go.instantlearning.biz/fileadmin/res/js/jwplayer.flash.swf
x-flash-version: 11,7,700,169
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)
Host: stream.instantlearning.biz
Proxy-Connection: Keep-Alive
HTTP/1.0 200 OK
Content-Type: application/x-fcs
Server: FlashCom/3.5.7
Cache-Control: no-cache
Content-Length: 9
X-Cache: MISS from PROXY
X-Cache-Lookup: MISS from PROXY:3128
Via: 1.0 PROXY:3128 (squid/2.7.STABLE9)
Connection: keep-alive
Proxy-Connection: keep-alive
88824327

Whished I could force RTMPT without the proxy.

Regards

Henrik

I know. But I was expecting a fallback to RTMPT. And it does work for the first video, looks good. Only the next request for another SMIL while the other video is still running fails. If the GET (executed via HTTP) were answered with the SMIL content, it would work, I am sure.

I did some more testing. The situation is real. Here’s what I found so far.

Two people are behind a proxy accessing Wowza. They do not have direct internet access. The website they are accessing uses jwplayer 6.

The first user tries to watch a video. The player gets a URL like this “http://ilsdevstream.sonic-ps.de/ilsdev-hz/smil:ILS_806_0990_PivotTable_erstellen.smil/jwplayer.smil” to access a SMIL file. It downloads the file successfully using HTTP GET. Then it uses RTMPT to display the video, on can clearly see that. HTTP POST is used like this:

POST http://stream.instantlearning.biz/open/1 HTTP/1.1\r\n
POST http://stream.instantlearning.biz/idle/1593715987/0 HTTP/1.1\r\n

Works well. The video is displayed.

Now the second user tries to watch a video. Again, the SMIL is requested using HTTP GET. This time unfortunately no SMIL is returned, but a number that looks like a stream id. The user gets the aforementioned error message.

As soon as the first user stops watching the video, the second user can reload the page and watch the video.

My conclusion:

  • The fallback to RTMPT works.

  • While Wowza is delivering a stream to one IP, it won’t reply properly to HTTP GETs asking for SMILs coming from the same IP. All requests come from the same IP as they are sent via the Proxy.

    My assumption:

    If I could make Wowza return all SMILs as requested, access via the proxy would be possible, even for several users at the same time.

    Unfortunately I cannot trace the packets on the other side of the proxy. But as this is a serious issue for us, I am willing to set up a lab environment to test further.

    If anything like configuration files, logs from the server, anything else is needed, let me know.

    Thank you!

    Henrik

Dear Richard,

I know the client has to do the fallback, that’s not the issue. You must have misunderstood my observations.

What I observe is that of two or more clients connecting from behind the same proxy only the first is able to load the SMIL from the Wowza server. It then tests connection options, detects that RTMP is not possible, neither via port 1935 nor 80. It then automatically falls back to RTMPT. That works fine over the proxy, the first user can happily watch the video.

While he does so, the second client tries to do the same. But already the download of the SMIL from Wowza fails. The client is able to reach Wowza via the proxy, but instead of the expected SMIL, a number is delivered. The number suspiciously looks like the session id Wowza usually sends after a “POST http://stream.instantlearning.biz/open/1” command. The request for the SMIL is sent via HTTP (GET) to port 80. The proxy cannot be the cause, the number is different at each request, so it cannot come from the cache.

The only reason why I asked whether it were possible to change rtmp to rtmpt in the SMIL file generated, is that I wanted to test without the proxy thus eliminating one possible cause.

So to summarize:

  • SMIL generation by Wowza: works

  • Download of SMIL from Wowza using HTTP GET: works - as long as no RTMPT connection exists via the same proxy

  • RTMPT stream via proxy: works

  • Download of another SMIL via the same proxy while an RTMPT connection exists via the proxy: DOES NOT WORK

    My suspicion is that when the a client uses RTMPT to port 80 on a Wowza server, HTTP GET requests to Wowza coming from the same IP are no longer handled correctly. Maybe because they are not expected.

    Is there a way to make this an official support call? I don’t want to be a pest, it is just extremely important for us.

    Kind regards

    Henrik

Hi Charlie,

Thank you for the clarification. Now that explains the observation. So I can a) check whether the requests are really pipelined (guess they are) b) try to make Squid open a new connection for the second client and c) tell our customers that pipelinig request via proxies isn’t supported.

That’s something I can work with.

Henrik

Dear Charlie and Richard,

I would like to make this a feature request. We are a Wowza OEM partner and are delivering VOD to a growing customer base. Quite frequently we find environments where the employees can only access the internet via proxy. Though we strongly recommend bypassing the proxy for TCP 1935 and using rtmp for performance reasons, it would be fantastic if the fallback worked, even for those environments where the proxy does pipeline the requests. We do not have that much influence on our customers’ infrastructures and wouldn’t want to lose a deal just because we can’t offer our solution via proxy.

Kind regards

Henrik