live streaming architecture

[Brandon_Wilson]

Hi all, I’m working on building some architecture for a live streaming environment. I’m seeing two main issues with the setup that I have built so far. Latency and audio/video being out of sync. The video is about 17 seconds out of sync, and the audio is about another 15 seconds behind that. So I’m hoping someone might have some suggestions on what I could be doing better.

laptop cam:

using FMLE

sending out one bitrate of 1500k, of h264 720p video with aac stereo audio running at 44100hz and 192k. FMLE says this comes to a total of 1692kbps output.

That stream goes to server A which is an 8x2.66Ghz softlayer dedicated box with 12GB Ram, running Wowza on Solaris 10u9, running the ‘live’ application with the following changes made:

live-lowlatency

and

flushInterval

25

Integer

Then I have ffmpeg on server B which connects to the stream on server A. Server B is a 16x2.0Ghz softlayer.com cloud instance with 8GB ram running ubuntu 10.04 lucid. ffmpeg then reads the stream and transcodes 3 streams out, 720p @ 1500kbps, 480p @ 800kbps, and 350p @ 400kbps. All audio streams remain at 44.1/192k. These streams are all sent to server C.

(If you want to see the config/build info and the command line i’m using for ffmpeg, you can refer to the following post: http://pastebin.com/nNziQ7U1 I’d paste it here, but I don’t want to cause a whole ton of clutter in the post, so I thought I’d better reference it externally.)

Server C is an 8x2.66Ghz softlayer dedicated box with 12GB Ram, running Wowza on Solaris 10u9. It accepts the streams from ffmpeg.

Server D, which has the same specs as server C and A, is running apache and hosts a page with the jw player on it. It also hosts an xml file for the stream information. the php page with jw player loads, referencing the xml file for the stream information, and the stream is displayed on the client web browser.

So, that’s the setup.

The idea is to keep the setup and configuration easy, and required bandwidth low, for the broadcast side, yet still support multiple streams for dynamic streaming, which will give the end user the best quality and bitrate that their connection will allow. I’m using multiple servers because I’ve read that it is advisable to separate the ffmpeg transcoding process from the wowza hosting process. I’m also separating the apache hosting process as well.

But again, the issue is that the audio and video streams are not in sync, and the latency is very high. 17 seconds to 35 seconds.

Thanks to all for any advice!

[Randall_Auriemma]

So we have:

(Laptop - FMLE) -> (A: Wowza) -> (B: Cloud - ffmpeg) -> (C: Wowza) -> (D: apache)

Is server A needed in that scenario?

If you eliminate servers, you’ll at least eliminate network latency.

What is the CPU usage on your various boxes?

Can you combine functions onto one box?

Can you use VLC to check the output from each box to pin down where the latency is coming from?

Is FMLE needed? Perhaps a dedicated IP Camera/encoder will help reduce latency.

Have you tried putting everything on one server, like this:

(Laptop - FMLE) -> (A: ffmpeg -> Wowza -> apache)

[Richard_Lanham]

There is a known a/v sync issue with FMLE and some capture cards. You can debug timecodes:

https://www.wowza.com/docs/how-to-debug-aac-or-mp3-timecode-issues-with-apple-hls-packetization

FMLE works by itself for multi-bitrate streaming, but maybe (probably) not with ffmpeg involved. Your setup doesn’t seem simple to me. This is simple and all that you need:

FMLE > Wowza > Client.

Try changing StreamType to “live” instead of “live-lowlatency”, which is more useful for chat apps involving low bitrate streams.

Richard

[Ian_Beyer]

Using a publish-transcode-publish model is going to introduce a lot of extra latency just by sheer necessity, once you aggregate each server’s buffer time, as well as the time it takes to actually do the transcoding.

It’s also possible to do this with a single Wowza server - publish to server A, transcode with ffmpeg, and publish back to different stream names on the same application. If using it as a repeater origin, you could probably do it on the same box, eliminating some network latency.

With version 3.0, the transcoding will be able to happen internally to Wowza.

-Ian

[Dave_Mort]

Can’t you just send the 3 streams directly out of fmle?

[Brandon_Wilson]

Well, the reason I didn’t want to have the client send multiple streams out is so that I could ensure they had enough bandwidth. Basically, if someone has a slow upstream connection, why waste it by sending several streams?Then they may not have enough bandwidth left for sending out HD video. Instead, I think it would be a better use of resources to just send one 720p stream out, and then let the server side do the transcoding for the low, medium and high quality streams.

This makes it simple for the client, being that they only need to worry about the one stream, and it greatly reduces the client’s upstream bandwidth requirements.