Live and mp4 streaming schedule

[Malcolm_Taylor]

Hi krisisdnbdjmx,

On point for you: The .smil file is opened by the client in the context of a single application. So, stream names, or .stream files mentioned in the .smil will be in relation to that Wowza application. Therefore, if you have 50 .stream files referenced in one smil, all the .stream files would need to be started in stream manager under that one application.

So you are suggesting that the move from 50 applications and 50 .stream files to 1 application and 50 .stream files is the way to go?

Do I need something in the .smil file that references the live webcam .stream files or is the .smil file only used for scheduling over the top of a regular webcam stream thats startedthrough streammanager or StartupStreams.xml ??

Then the question of what http urls should be used so a user sees a specified stream with the scheduled parts added as well? Do I still use :

http://:1935/live/camera.stream/manifest.f4m

http://:1935/live/camera2.stream/manifest.f4m

http://:1935/live/camera3.stream/manifest.f4m

and it is the stream (i.e. camera.stream) which pulls in the schedule at the time specified? Is that the link between the stream name put in the .smil file and the content input camera.stream ???

[Malcolm_Taylor]

I have the stream files all set up and working fine with StartupStreams.xml so my only task is to get the scheduling of the nightly mp4 file done over the top of each one. I can access over http each webcam with no problem using urls like :

http://:1935/live/camera.stream/manifest.f4m

What is the ‘switch’ smil file for and does it need to have any specific name as that adds an extra file that hasnt been mentioned before? Do i need it or is the streamschedule.smil file all I need?

And i still dont know whether 50 applications with 50 stream files or 1 application and 50 stream files is the way to do this?

I am not using an edge server, all of this is on 1 single server so do I need to worry about liverepeater or just leave the applications set on live?

My end result is :

09.00-00.00 - 50 live webcams

00.00-09.00 - 50 streams playing an mp4 file on the same http urls so the web application does not need to do anything

[Malcolm_Taylor]

Hi Richard,

Yes I can see it will be a huge smil file for 50 streams but i can script that to generate for all 50 streams for a single day and then loop it round to generate it for 1 year and then something each year to update it to the next.

In your example what would the http url be that I would need to put in the web front end to watch Stream1 running in an application called ‘live’? Would it be :

http://:1935/live/Stream1/manifest.f4m

or is the manifest.f4m not needed now?

Are you also saying that I should go to 1 application with 50 .stream files with 1 streamschedule.smil file with the whole schedule in to keep this as simple as possible?

Sorry for all the questions but it took me a long time to create all this for 50 cameras as it stands so want to be confident im changing it to something that will work before I tear what Ive done up and work this all out hahha

[Malcolm_Taylor]

Thanks for your help. I have managed to get it working fine here now and all auto starting as well.

I hit memory problems in Java when I had my current 50 applications with 50 cameras AND my new ‘live’ application with the 50 cameras as well. In a live situation however I wont have that so once the users are happy with the new method I will remove the old 50 applications which I hope will help.

Im sure this has been asked before in a post but cant find it now, but is it possible for the streamschedule.smil file to just say ‘daily’ instead of a playlist for individual dates?

[Malcolm_Taylor]

50 incoming live streams is high. You want a late model quad core and 8 - 16 gigs of RAM to handle that many live streams.

You could have a system that set a time only in the playlist, a start date and a number of days to schedule as Property settings in Server.xml. The scheduler source code could be a starting place for that.

Richard

Hi Richard,

ok definitely got some performance issues after switching to using the schedule with 1 application and my 50 cameras. I will suggest to the customer that we need to increase memory in the server but want to be happy that im getting the most out of it as it stands first.

My GC log is showing something like this after I have been running it for a short while :

1613.218: [GC 6478469K(7115584K), 0.0053330 secs]

1614.004: [GC 6893565K->6546377K(7115584K), 0.1047760 secs]

1614.522: [GC 6763624K(7115584K), 0.0117220 secs]

1614.915: [GC 6807490K->6456016K(7115584K), 0.1012640 secs]

1615.019: [GC 6456107K(7115584K), 0.0049330 secs]

1615.847: [GC 6875472K->6528711K(7115584K), 0.1045350 secs]

1616.349: [GC 6743596K(7115584K), 0.0119390 secs]

1616.683: [GC 6840718K(7115584K), 0.0248660 secs]

1616.779: [GC 6884961K->6540091K(7115584K), 0.1049080 secs]

1617.712: [GC 6959542K->6612229K(7115584K), 0.1019890 secs]

1618.180: [GC 6832460K(7115584K), 0.0118700 secs]

1618.608: [GC 6931897K->6585314K(7115584K), 0.1053600 secs]

1618.717: [GC 6585740K(7115584K), 0.0066470 secs]

1619.525: [GC 7004755K->6660340K(7115584K), 0.1041150 secs]

1619.647: [GC 6677397K(7115584K), 0.0154620 secs]

1620.396: [GC 6670643K(7115584K), 0.0273300 secs]

1620.510: [GC 6704915K->6362038K(7115584K), 0.1045000 secs]

1621.420: [GC 6781494K->6432038K(7115584K), 0.1019040 secs]

1621.977: [GC 6654726K(7115584K), 0.0120770 secs]

1622.374: [GC 6796902K->6875726K(7115584K), 0.1208340 secs]

1622.499: [Full GC 6875726K->6348420K(7115584K), 4.1582730 secs]

1626.677: [GC 6348497K(7115584K), 0.0036760 secs]

1626.979: [GC 6767876K->6545067K(7115584K), 0.1476160 secs]

1627.616: [Full GC 6964523K->6439201K(7115584K), 4.1906110 secs]

1632.097: [GC 6858657K->6651414K(7115584K), 0.1579980 secs]

1632.447: [GC 6938309K(7115584K), 0.0356470 secs]

Unix CPU and memory utilisation :

top - 18:12:32 up 62 days, 1:07, 2 users, load average: 1.07, 2.40, 2.13

Tasks: 148 total, 1 running, 146 sleeping, 1 stopped, 0 zombie

Cpu0 : 21.3%us, 3.5%sy, 0.0%ni, 70.3%id, 0.2%wa, 0.1%hi, 4.6%si, 0.0%st

Cpu1 : 19.0%us, 1.8%sy, 0.0%ni, 79.0%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st

Cpu2 : 17.4%us, 1.5%sy, 0.0%ni, 81.0%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st

Cpu3 : 15.8%us, 1.4%sy, 0.0%ni, 82.7%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st

Mem: 8148636k total, 8003368k used, 145268k free, 85152k buffers

Swap: 4194296k total, 17656k used, 4176640k free, 140680k cached

So overall CPU across the processors is not too bad really, but because I have a heap setting of 7000 set it has eaten most of the server memory but I am not going into swap space so its not the end of the world I think.

It is the Full GC which seems to be causing the most issue with the stream moving but it is hard to 100% confirm that.

My server is a quad core with 8Gb memory running Centos 6.2

Following the performance tuning guide I set the following java options :

Heap space set at 7000M (I tried 5000 and 6000 as well but not a huge amount of difference, but maybe 7000 made the server last longer without having real problems).

The following options set :

JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:NewSize=512m" once again taken from the guide.

I also tried the default options :

JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC"

I am running the latest Java version 7 from a post you put up somewhere Richard so think I am right there? I have not tried the Experimental: Garbage-First garbage collector (G1) idea yet but do you think that may be worth me trying?

What I should also add to make this even worse is that there is barely any users on the system watching the webcams as we are still in a testing mode. So I dont really know the resources the server or java process will need when say 100-500 people are watching any one of the streams?

If you could give me some ideas then it would be appreciated!

[Malcolm_Taylor]

I think because my heap needs to be so large to start the streams then I dont even think more memory is an option.

As I am only using http streaming for the website is there maybe any options for parts of Wowza that im not using to not start which might therefore reduce the initial memory usage of the process?

[Malcolm_Taylor]

Or somehow reducing the amount of data from each stream that wowza is holding?

[Malcolm_Taylor]

I think for http streaming which is what I use that runs over sanjose yeah?

In the Application.xml file I see 2 sections for that :

live

${com.wowza.wms.context.VHostConfigHome}/content

${com.wowza.wms.context.VHostConfigHome}/keys

sanjosestreamingpacketizer <<<<<< I JUST CHANGED THIS

and

cupertinostreaming,smoothstreaming,sanjosestreaming

^^ Should I do same here as well?

[Malcolm_Taylor]

ok reducing my heap down to 6000M from 7000M and removing the cupertinostreamingpacketizer and smoothstreamingpacketizer has helped as the garbage collection is now clearing down to half of the heap roughly and it does not seem to be doing frequent Full GCs either.

297.042: [GC 5418256K->5048018K(6091584K), 0.0777310 secs]

297.982: [GC 5467430K->5092506K(6091584K), 0.0746770 secs]

298.930: [GC 5511962K->5138784K(6091584K), 0.0780830 secs]

299.837: [GC 5558232K->5186743K(6091584K), 0.0761530 secs]

300.809: [GC 5606181K->5231630K(6091584K), 0.0759890 secs]

301.703: [GC 5651085K->5275567K(6091584K), 0.0751970 secs]

302.236: [GC 5479054K(6091584K), 0.0104570 secs]

302.694: [GC 5694962K->5324287K(6091584K), 0.0802890 secs]

303.669: [GC 5743742K->5371619K(6091584K), 0.0782290 secs]

304.576: [GC 5791075K->5416885K(6091584K), 0.0790820 secs]

305.142: [GC 5637442K(6091584K), 0.0091130 secs]

305.608: [GC 5725341K->5357231K(6091584K), 0.0783970 secs]

306.563: [GC 4239960K->3866282K(6091584K), 0.0772270 secs]

307.474: [GC 3427056K->3053916K(6091584K), 0.0775210 secs]

307.957: [GC 3253223K(6091584K), 0.0112930 secs]

308.396: [GC 3473329K->3102000K(6091584K), 0.0771580 secs]

309.354: [GC 3521426K->3146849K(6091584K), 0.0771110 secs]

310.309: [GC 3566305K->3192210K(6091584K), 0.0768050 secs]

But I am 1 user watching. Is there any significant impact of 10,50,100+ users on the java memory usage or not?

Also during my ‘quiet’ period overnight where I use the schedule to play a rotating mp4 file it would be good to understand which is heavier on the server? The live camera stream or the mp4 file?

[Malcolm_Taylor]

Ok thanks.

So from a costing point of view my recommendation should be to increase memory from 8Gb to 16Gb. That option would be substantially cheaper than another server and another Wowza licence.

Are there any issues scaling Wowza to a 16Gb server apart from just increasing the heap space in java from 6000M to something like 12000M, or are there any other considerations I should make from a Wowza point of view?

[Malcolm_Taylor]

ok thats fine thanks Richard but im guessing thats a Wowza recommendation as Java suggests on 64 bit there should be no limits anywhere near that.

I have a slightly different test environment im having trouble setting up now. I dont have an IP webcam here so I am using flash media live encoder and my desktop web camera to test the same thing with scheduling. This is on a different server. I have configured the following which from the instructions, this post, and my notes look to be correct but clearly not… :

conf/live/Application.xml

live

I have copied the jar file into the lib directory

Server.xml :

com.wowza.wms.plugin.collection.serverlistener.ServerListenerStreamPublisher

and

PublishToApplication

live

streamschedule.smil :

[Malcolm_Taylor]

ok my first problem was yesterdays date in the streamschedule.smil file. Solved that now and also changed the camera.stream file to be mp4 to cope with h.264 stream (i read that somewhere else on the forum)

I forgot to paste in my camera.stream file (which I dont know if I need) which contains the url to the main stream coming from flash encoder :

http://192.168.1.67:1935/live/stream

Maybe I dont need that so in which case what do i need to reference the live stream inside the streamschedule.smil file?

[Malcolm_Taylor]

Just seen this in my access log file :

2012-07-18 20:03:49 GET comment server WARN 200 - Licensing[live/definst/stream1]: *** Hit license limit for publishing streams. Publishing stream is stopped. ***

On my test server I only have a developer licence so does this mean i cannot have a single application with the stream from flash encoder (connecting on ‘stream’) and the scheduler creating its fake ‘stream1’ ???

If you could let me know it would be appreciated. Either way I do need to know how to do this in a live environment to reference my situation above with an input stream from flash encoder.

Thanks

[Malcolm_Taylor]

issue closed although new post raised in order to reset scheduled streams if either live or mp4 content streams to a schedule stream and has problems then the stream is not monitored or reset, therefore requiring a reboot of wowza which is not ideal