Please help. I have 5 Wowza servers and can't seem to keep running smoothly even though I only do VOD and usually have no more than 50-100 customers per box.
Here is a live server monitor chart:
My old system was pretty simple, encode multiple qualities using Adobe Media Encoder, FTP to server (rsync to remote) and stream. We used to let customers select any server they want, and what stream quality. It wasn't elegant, but it worked, and when it didn't work it was easy to find out that the real reason customers couldn't stream was because they had trouble with their computer / internet.
I brought on a Wowza consultant last year to fine tune things, add extra security, adaptive bandwidth switching and such. In November we ran an update which added secure streaming over SSL and adaptive bitrate switching. Since then we started having issues. I get about 2000 customers per day, and used to only get 1-2 support emails about stream quality which were easily diagnosed as user trouble, but lately it is more like 5-6+ per day and I have no idea what to tell them.
We implemented a new system of encoding the files on the server using ffmpeg on a main server and automatically distributing them:
We were having trouble with the multiple video qualities being synched correctly, not sure if that is entirely resolved. That was causing huge problems with adaptive bitrate as it would jump back or forward in time as it was switching qualities. We disabled adaptive bitrate for now.
ffmpeg -i video.mp4 -vf "scale=320:trunc(ow/a/2)*2" -ab 128k -ar 48000 -strict -2 -g 30 -keyint_min 30 320/video.mp4
Encoding all qualities + Fastplay BASH command:
export VIDEO=test.mp4 && \ export GOP=30 && \ export AB=128k && \ ffmpeg -i $VIDEO \ -vf "scale=320:trunc(ow/a/2)*2" -ab $AB -ar 48000 -strict -2 -g $GOP -keyint_min $GOP 320/$VIDEO.mp4 \ -vf "scale=480:trunc(ow/a/2)*2" -ab $AB -ar 48000 -strict -2 -g $GOP -keyint_min $GOP 480/$VIDEO.mp4 \ -vf "scale=720:trunc(ow/a/2)*2" -ab $AB -ar 48000 -strict -2 -g $GOP -keyint_min $GOP 720/$VIDEO.mp4 \ -vf "scale=1080:trunc(ow/a/2)*2" -ab $AB -ar 48000 -strict -2 -g $GOP -keyint_min $GOP 1080/$VIDEO.mp4 \ && mv 320/$VIDEO.mp4 320/$VIDEO \ && mv 480/$VIDEO.mp4 480/$VIDEO \ && mv 720/$VIDEO.mp4 720/$VIDEO \ && mv 1080/$VIDEO.mp4 1080/$VIDEO \ && ffmpeg -i 480/$VIDEO.mp4 \ -filter_complex "[0:v]setpts=(1/1.33)*PTS[v];[0:a]atempo=1.33[a]" -map "[v]" -map "[a]" -strict -2 -ab $AB -ar 48000 -g $GOP -keyint_min $GOP fastplay/$VIDEO.mp4 \ && mv fastplay/$VIDEO.mp4 fastplay/$VIDEO
I have videos that if you seek forward, everything goes haywire with squelching, when you are trying to watch you get little flashes of video while watching the progress bar going at like 8X speed. We get problems of the screen going solid green. Videos that never seem to start playing, or stop after 15 seconds or so. Some people tell me they can watch short clips, but longer ones don't play at all.
I don't know what to do anymore. It has been 3 months of letting this consultant try fixing it, but the problems aren't getting better, only different. He keeps trying new things, has said he is working with top Wowza staff and contacts within JWplayer, but this is getting old. I have 5 servers that should be able to handle 10X the users we have, so it is hard to justify more servers, nor does it seem that would solve the issue.
I need some help, I don't know if anyone can pick out a problem with the ffmpeg command, or have a recommendation for a great company to help (Thanks anyway Charlie, I know I can email for the list but I want recommendations at this point). I tried calling Real Eyes, but their number is disconnected.