1,000 viewer 720p&360p Multi Bitrate broadcast. Wowza + a bunch of EC2 instances?

[system]

I am sure load questions get asked often around here, but I couldn’t find a thread that covered my particular needs, so here goes…

I am planning on streaming this event: http://magicvalleysgottalent.com/gala-dinner-main-event/ in 720p (~2500kbit) and 360p (~600kbit). I am expecting about 1,000 viewers and would like to make the stream as compatible as possible (Wowza configured to stream in every streaming protocol it supports).

Erring on the side of a hard hit, let’s assume all 1,000 viewers tune in to the 720p stream. As this would require 2.5Gbps+ output, I’m guessing that I would need an army of EC2 instances to handle the load.

Ideally, I would be able to setup Wowza in such a way it would automatically fire up more instances when needed–who knows, maybe we’ll end up with 3,000 viewers.

Is this a road well-traveled (e.g. automated/elastic load balancing)? This will be my first time streaming with Wowza & EC2; we’ve used a free UStream account in the past and would like to step up the quality. However, the event is coming up quickly, and while technically adept, I’m not sure I’ll have enough time to pioneer such a setup in 5 days.

Thoughts?

[Ian_Beyer]

I have a number of posts on my blog about doing this: http://blog.ianbeyer.com/archives/category/streaming

You could use cloudwatch to trigger a scale operation once bandwidth or CPU on the edge servers hits a certain threshold (75-80% is probably a safe number), or you could break your 720p/360p into two separate sets of servers and scale each as needed by monitoring connection counts.

[Jed_Hurt]

Hey, thanks for the reply. I started reading through your posts. Lots of good info there. I also found a post on the LongTail Video blog wherein Jeroen leaves the comment:

Live RTMP streaming with Cloudfront is indeed not possible. It probably never will be. However, you can do HTTP adaptive streaming, where Cloudfront is used to distribute the HTTP segments from Wowza.

With the combo FMLE + Wowza EC2 + Cloudfront + JW Player, you can now do live HTTP streaming, to both iOS and Flash. It needs some work to setup though - that’s why I was referring to an additional blogpost in the “next steps” section. I will write this post, but here’s a quick overview:

*) You use FMLE / Wowza EC2 just like described here.

*) You connect your Wowza EC2 machine to Cloudfront (by setting the origin in CF).

*) You use the “adaptive” branch of JW Player to playback the Apple HTTP Live Streaming stream exported by Wowza, in both Flash and iOS.

*) Since this stream is basically a list of small TS videos, this caches great using the CloudFront HTTP caching mechanisms.

This is quite intriguing to me as it sounds like Cloudfront would be hit with the load rather than a number of EC2 instances, and would scale automatically? Unfortunately, I’m not familiar enough with Wowza or EC2 to make heads or tails of Jeroen’s bullet list. If your streaming requirements only needed to cover Flash and iOS clients, would you consider switching to this setup from the one you outline in your blog posts (i.e. a round-robin of EC2 servers)?

[Jed_Hurt]

@ianbeyer,

I read somewhere on your blog that Wowza has a config directive to resync audio/video (ours is off by about 10 frames). I looked through the Wowza user guide and couldn’t find anything on it; not sure what term to look for…

[Richard_Lanham]

Actually, I don’t think this will work. Wowza is not setup to be a cache origin for segmented streams.

Richard

[Richard_Lanham]

Karuna,

I’m not sure if Wowza works with Cloudwatch, and it would not be looking at the criteria that matters: connection counts on the edge servers. The best approach is poll the Load Balancer Listener HTTProvider:

http://[load-balancer-ip-address]:1935/loadbalancer?serverInfoXML

and start new instances based on thresholds using EC2 Tools and startup package for your edge server.

Richard

[Richard_Lanham]

We only have benchmarks with confidence for the m1.small, m1.large and m1.xlarge. You can expect about 150, 250 and 350mbs throughput (“IO Performance”) resp., which is the limiting factor. That means you could reach about 300 concurrent clients with a 500kbs stream on the m1.small. The m1.small is the best value for streaming purposes of those three.

Take a look at this table of EC2 instance types for more info:

http://aws.amazon.com/ec2/instance-types/

Note that IO Performance is approximated (low, moderate, high)

Richard

[Enrico_Kern]

The question is rather why you want to make it as portable as possible. With just 1000 viewers it is very unlikely that tons of them will have no flash and need to use as example the silverlight segmented http stream. And for iphone devices you require anyway a baseline compatible stream.

If you calculate the bandwith plus the operational costs of the ec2 instances you might get it off the shell cheaper by using a CDN right away?

However given you stil deploy a bunch of ec2 instances or other kind of servers/vps take this at estimate:

• Loadbalance with .smil files with jw player and rtmp.loadbalance as example

• Generate the output of the smil file based on your overall server usage (connectioncounts from the http provider) and switch to the least used server uppon play request

You need to know how much outgoing bandwith your instances/servers can delivery stable and how much bitrate per visitor you are using in peaks to calculate how much you can support per server.

I do it exactly this way without much problems and an average output of more then 4Gbps each night accross 16 servers. And of 1000 visitors i have, maybe only 2-4 use an apple device, so i have only one wowza with the cupertino segmenter for my iphone compatible streams.

The point is that you can stil stream in any media format wowza supports, all you need to take care of is the tracking of the connections when you initiate whatever connection to your wowzas. But for just 1000 expected viewers only a hand full of people will use something else then flash.

[Karuna_Pai]

Hi Richard,

I have configured load balancer listener and sender (edge servers). I would now like to scale the edge servers on the fly when required.

Can I use Amazon’s cloudwatch on the edge servers to trigger scaling?

If yes, which cloudwatch parameters should I use for monitoring load on Wowza edge servers?

Thanks,

Karuna.

[Karuna_Pai]

Hi Richard,

Could you please tell me the threshold values for the no. of connections for the following instances?

ami-3bab9c4f: m1.small, c1.medium

ami-25ab9c51: m1.large, m1.xlarge, m2.xlarge, m2.2xlarge, m2.4xlarge, c1.xlarge

Thanks,

Karuna.

[Karuna_Pai]

Hi,

Thanks for the reply Richard. These numbers will help us in autoscaling the instances

• Karuna.