Page 1 of 2 12 LastLast
Results 1 to 10 of 20

Thread: Wowza DVR: preventing corruption and overwriting

  1. #1

    Default Wowza DVR: preventing corruption and overwriting

    Hi,

    Yesterday we upgraded from 3.0.5 to 3.1.1. This seemed to go fine, until I noticed Wowza started overwriting old DVR recordings for some (not all) streams (window size for all is 2592000; 30 days):
    May 1 00:36 0000_00_00
    May 1 00:46 0000_10_00
    May 1 00:50 0000_20_00
    Apr 12 09:18 0000_30_00
    Apr 12 09:28 0000_40_00
    Apr 12 09:38 0000_50_00

    Also the server started using 100% CPU for a couple of minutes. At one time during the 100% CPU use I restarted the server to make a small configuration change that I forgot to make earlier. Might this have something to do with the corruption?

    Also we're seeing a LOT of these in the error log:
    LiveStreamDvrRecorder.endChunk[livedvr/_definst_/oODzpwg2bQ]: Recalculating duration by estimating. Was 4153. Is now 3827.

    And also quite some of these:
    DvrPacketHandler.handleHolder(): Skipping holder that cannot be re-aligned. Holder: {DvrPacketHolder: type:V pt:2332595 utc:1335872439639 dur:1430 (KEY, ) codec:7 numPackets:0 dataLen:18448 enc.n:0 pkt:{{DvrAMFPacket: size:18448, type:9, src:1, seq:0, absTimecode: 2332595, timecode:1430, utcTc:1335872439639}}}

    In our use case the source sometimes drops out (once every few hours it drops out for 30 secs or so; it's a low bitrate IPcam without audio in this particular case; think 160 kbit/s). Some extra info about this case:
    <CanRecord>true</CanRecord>
    <CanPlay>true</CanPlay>
    <HasEncryption>false</HasEncryption>
    <ChunkGrouping>600</ChunkGrouping>
    <PurgeTime>-1</PurgeTime>
    <CurrentTime>-1</CurrentTime>

    Also, I notifed that IF one DVR-stream would get corrupted, all kinds of unpredictable things will happen and even the live feed won't play anymore. No errors are logged if this happens.

    So, my questions:
    1. Do you have any tips, do's/don'ts to prevent DVR archive corruption?
    2. And are you planning on extra measures to stabilize the way Wowza handles restarts/DVR in general? It seems to be very easy to corrupt the DVR archive with a simple restart or especially a wowza upgrade. I hope you can make the DVR addon more robust as we plan on running a couple 100 of these low bitrate streams per server.
    3. Recordings with 2592000 window size and 600 chunk grouping means Wowza will create 4320 directories per DVR per stream, or 432000 directories for 100 streams(!). Would it be wise to increase chunk grouping? What are the pros and cons?

    Hope you've got some answers and tips for us!

  2. #2
    Join Date
    May 2011
    Posts
    456

    Default

    I'm not sure if that caused the corruption. Were the streams that exhibited the problem the streams that were currently recording when the system was at 100% CPU and re-started? If you have ArchiveStrategy set to: delete and the stream is disconnected and then re-started, Wowza nDVR will delete the previous recording and record a new recording. Make sure you Server is properly tuned. The Tuning Guide is at this link.

    Those log messages indicate that there is a video/audio alignment issue. Wowza nDVR relies on the incoming live video and audio being aligned. If these messages occur a couple of times when Wowza nDVR first starts up as it is buffering its initial packets, you can ignore it. If it persists and you also see skipping chunk messages, then there is a video/audio alignment issue. When there are alignment problems, chunks cannot be recorded and degrades the quality of the recording. Alignment issues should be fixed upstream from nDVR, usually fixed in the encoding process. There are nDVR AddOn properties you can try to compensate for this problem. Note that these properties will not fix the alignment problem, but may help to work around the problem. See dvrPacketSortTime and dvrAllowableAVPacketDelta in the Wowza nDVR Advanced Configuration article.

    By default Wowza nDVR will wait 300,000 ms (5 minutes) for packets until it stops recording. This timeout is meant to account for when encoders/cameras disconnect and then re-start. The value should be a non-zero value or else your recording will stop if at any time there is no data. This should amply account for the 30 seconds your source drops.

    Take a look at the Best Practices section of the nDVR Quick Start Guide for tips on recording length, keyframe size. That large number of directories/files will likely cause an issue with the file system handling such a large number of files. Regarding recording length, the playlist will be repeatedly requested every couple of seconds (based on key frame size), therefore an extremely large playlist size will be problematic for the player and should be avoided. See the How to use Wowza nDVR AddOn Playlist Request API article for more information on how to request a portion of your recording for play back.

    -Lisa

  3. #3

    Default

    Thanks for your quick response.

    I will try to address the alignment problem later at the source. This is a separate problem.

    ArchiveStrategy is set to 'append'. And it is possible streams were recording while the server was at 100% and re-started (our application autostarts recording as soon as the input stream connects). It would be very nice if Wowza could somehow recognize it if the last chunk is corrupt and discard that and append at the last non-corrupt chunk. That by itself would probably resolve a lot of our problems. It would mean there is a visible and audible gap within the recording, but this is acceptable.

    Also the Quick Start Guide mentions that nDVR has a limit of 30 hours. I do not remember reading about this limit before. Our case requires a recording duration of 720 hours (however playback size may be much smaller, e.g. 1 hour per request).

    The server is already tuned (pretty low load) but to be honest nDVR isn't. Would it be wise in our case to increase dvrChunkGroupingSeconds to e.g. 1800? Or even 3600? This would mean it would create a maximum of 24 directories per day per stream, and 6 times less manifests, so probably quicker read times? It wouldn't fix the fact that Wowza creates a new 10 kB file every few seconds though, which may still become a problem.

    According to my (double-checked) calculations 100 streams (just 16 Mbit/s!) of 720 hours would generate a whopping 120,000,000 files. Doesn't really seem scalable. Have you got any tips/ideas/comments about this and about how we can scale? The hardware can probable handle recording 2000+ streams simultaneously, but this would mean billions(!) of files.

  4. #4
    Join Date
    May 2011
    Posts
    456

    Default

    Thank you for sharing your ideas about nDVR and details about your workflow. I will pass those on to product management as feature requests.

    The limit of 30 hours is something that was not in early preview builds, but is now set by default as the maximum recording time. It is good that your planned playback duration is much smaller (e.g. 1 hour) because a recording for 30 days worth would be too large to manage and for the player to continually read. Changing the dvrChunkGroupingSeconds would allow you to customize how you want your directory structure to be maintained, but as you already mentioned, it is the total number of files that the file system has to manage that is the bigger issue. Our advice at this time is to find hardware that can support that level of file access. You could consider distributing the recording across several drives or distribute across multiple Wowza Servers to help scale. You could change the chunk size to reduce the number files generated, but it would have other consequences which may or may not be acceptable for your workflow. Because HTTP streaming has an inherent latency of 3 chunks, a larger chunk size would result in longer latency before playback would start.

    -Lisa

  5. #5

    Default

    Ok, a quick followup question: is it possible to change the chunk size for a video or audio/video stream? The Advanced Configuration Guide only shows an example for audio only streams.

    If it is not possible to change this I would like to make a feature request, namely that it would be nice if it were possible to set a target size in seconds for Wowza to aim for as chunk size. Requested behavior:
    - If chunk size is not set, you use keyframes to determine chunk size (= probably highest performance in high bitrate streams)
    - If chunk size in application config is set (e.g. to 60), Wowza attempts to store 60 seconds of content per file before moving to the next file; if it stored 60 seconds it will start a new chunk as soon as it detects a new keyframe (so if you aim for 60, you might get 62, which is no problem). Alsof if the stream gets interrupted/restarts it can start a new chunk. If I want to start DVR at a certain startpoint Wowza should be able to start at exactly that point (which means it will have to seek within the chunk to start at the correct requested keyframe). This will greatly benefit I/O performance, at the cost of a bit of CPU performance (because it will have to seek to the next best keyframe within a chunk).

    Also it would be nice if (like I asked before) Wowza could be a little more resistant to corruption. E.g. remove the last (few) chunk(s) if they seem to be corrupt, before continuing recording. Also a fix script that can fix corruption would be nice (in our case with 30 days of recording it's rather relevant that we avoid any corruption). Wanted behavior of that script is removing all corrupt chunks and making sure the manifest is correct so that all non-corrupt material will play.

  6. #6
    Join Date
    Dec 2007
    Posts
    21,962

    Default

    You can control the chunk size for sanjosestreaming with the sanjoseChunkDurationTarget Property. Take a look at this guide:

    http://www.wowza.com/forums/content....s-(Flash-HTTP)

    It is important that key frame frequency be a factor of sanjoseChunkDurationTarget

    Richard

  7. #7

    Default

    Does this work if the incoming stream is RTMP instead of Flash HTTP?

    And what happens if the encoder drops a frame (or similar problem)? Is it possible that the key frame frequency and chunk contents will go out of sync, causing all kinds of unpredictable behavior? Or will Wowza be able to correct these kinds of problems itself?

  8. #8
    Join Date
    Dec 2007
    Posts
    21,962

    Default

    It should be okay. Let us know if you see a problem

    Richard

  9. #9
    Join Date
    Dec 2007
    Posts
    21,962

    Default

    Update/correction.

    Add this to Application.xml /DVR Properties list:
    <Property>
    <Name>dvrChunkDurationMinimum</Name>
    <Value>10000</Value>
    <Type>Integer</Type>
    </Property>
    This property will generate DVR chunks at least this size. They will still be broken on keyframes.
    Having larger chunks for DVR has a trade-off: HTTP streaming requires 3 chunks before playback starts so 10 second chunk size will have more latency than 2 seconds chunks.

    The default value is 1500 ms (1.5 seconds)

    Richard

  10. #10

    Default

    Ok, we will try that soon, thanks for the input, sounds promising!

    Are you considering a storage method that allows appending chunks to larger files? E.g. a 10 minute file that contains 2 second chunks? A manifest-type file could contain information about the contents of the larger file. Would be more efficient, especially in a use case like ours!

Page 1 of 2 12 LastLast

Similar Threads

  1. Video corruption at head of file, but rest plays fine
    By gouldtv in forum General Forum
    Replies: 0
    Last Post: 12-06-2013, 04:23 PM
  2. Preventing hotlinking of RTMP
    By stream4life in forum General Forum
    Replies: 12
    Last Post: 10-28-2013, 09:12 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •