Wowza Community

MediaCache not filling up

I’ve been running a few servers for a few days with a media cache store limit of 500Gig… but the size of the cache doesn’t fill up past about 5.6 Gig. I’ve checked all the configs and restarted the servers a few times but it simply refuses to fill up. As a result I’m pulling excessive data from my http source.

Thanks for any and all help.

Jake

The mediacache store config is:

<MediaCacheStores>
                <MediaCacheStore>
                        <Name>default</Name>
                        <Description>Default Store</Description>
                        <Path>${com.wowza.wms.context.ServerConfigHome}/mediacache</Path>
                        <MaxSize>500G</MaxSize>
                        <Level1FolderCount>24</Level1FolderCount>
                        <Level2FolderCount>24</Level2FolderCount>
                        <FileCount>1000</FileCount>
                        <WriteRate>38M</WriteRate>
                        <WriteRateMaxBucketSize>152M</WriteRateMaxBucketSize>
                        <WriteRateFillFrequency>100</WriteRateFillFrequency>
                        <Properties>
                        </Properties>
                </MediaCacheStore>
        </MediaCacheStores>

It’s possible that your source is timing out on files which expires them from cache at a rate faster than requests. You might try toggling the MediaCacheSource->MaxTimeToLive and MediaCacheSource->MinTimeToLive accordingly.

Jake,

Are you using latest MediaCache? Make sure by re-downloading if you haven’t recently. And enable mediaCacheDebugLog in Server.xml Properties

<Property>
     <Name>mediaCacheDebugLog</Name>
     <Value>true</Value>
     <Type>Boolean</Type>
    </Property>

Richard

I did forget to mention my min time to live is a week and max time to live is 2 weeks… so I don’t think we’re dealing with the timing out prematurely… especially considering I’ve got a flow of about 20 Mbps running into the wowza box from the source.

Thanks,

Jake

Applicable section below:

<MediaCacheSource>
                        <Name>http</Name>
                        <Type>HTTP</Type>
                        <Description></Description>
                        <BasePath>http://xxx.xxx.xxx.xxx:8080/</BasePath>
                        <Prefix></Prefix>
                        <BaseClass>com.wowza.wms.mediacache.impl.MediaCacheItemHTTPImpl</BaseClass>
                        <ReaderClass></ReaderClass>
                        <DefaultBlockSize>1M</DefaultBlockSize>
                        <MaxTimeToLive>1209600</MaxTimeToLive>
                        <MinTimeToLive>604800</MinTimeToLive>
                        <ReadAhead>true</ReadAhead>
                        <ReadAheadThreshold>30</ReadAheadThreshold>
                        <IsPassThru>false</IsPassThru>
                        <Properties>
                                <Property>
                                        <Name>debugHTTPRequests</Name>
                                        <Value>false</Value>
                                        <Type>Boolean</Type>
                                </Property>
                                <Property>
                                        <Name>maxPoolSize</Name>
                                        <Value>25</Value>
                                        <Type>Integer</Type>
                                </Property>
                                <Property>
                                        <Name>httpReadTimeout</Name>
                                        <Value>6000</Value>
                                        <Type>Integer</Type>
                                </Property>
                                <Property>
                                        <Name>httpConnectionTimeout</Name>
                                        <Value>6000</Value>
                                        <Type>Integer</Type>
                                </Property>
                                <Property>
                                        <Name>httpReadRetries</Name>
                                        <Value>3</Value>
                                        <Type>Integer</Type>
                                </Property>
                                <Property>
                                        <Name>httpSendBufferSize</Name>
                                        <Value>8000</Value>
                                        <Type>Integer</Type>
                                </Property>
                                <Property>
                                        <Name>httpReceiveBufferSize</Name>
                                        <Value>65000</Value>
                                        <Type>Integer</Type>
                                </Property>
                                <Property>
                                        <Name>urlDecodeStreamName</Name>
                                        <Value>false</Value>
                                        <Type>Boolean</Type>
                                </Property>
                                <Property>
                                        <Name>base64DecodeStreamName</Name>
                                        <Value>false</Value>
                                        <Type>Boolean</Type>
                                </Property>
                                <!-- Uncomment if pulling from a proxy -->
                                <!--
                                <Property>
                                        <Name>proxyHost</Name>
                                        <Value></Value>
                                </Property>
                                <Property>
                                        <Name>proxyPort</Name>
                                        <Value></Value>
                                </Property>
                                <Property>
                                        <Name>requestFullURL</Name>
                                        <Value>true</Value>
                                        <Type>Boolean</Type>
                                </Property>
                                -->
                        </Properties>
                </MediaCacheSource>

I restarted the box earlier today and watched the disk usage. It started filling normally but occasionally the media cache dir would shrink in size… then increase and shrink. It did this until it hit about 5.4 Gig and now it’s fluctuating around that size… when it should be growing and never getting smaller… especially within only 6 hours.

I’m starting to think something is fundamentally broken with the media cache. It doesn’t seem to be honoring the values in the config.

Jake

I watched the directory size over time… and here are the results with the configs mentioned above:

root@475911a:/usr/local/WowzaStreamingEngine/mediacache# du -hs
3.8G	.
root@475911a:/usr/local/WowzaStreamingEngine/mediacache# du -sh
5.4G	.
root@475911a:/usr/local/WowzaStreamingEngine/mediacache# du -sh
5.5G	.
root@475911a:/usr/local/WowzaStreamingEngine/mediacache# du -sh
5.6G	.
root@475911a:/usr/local/WowzaStreamingEngine/mediacache# du -sh
4.6G	.
root@475911a:/usr/local/WowzaStreamingEngine/mediacache# du -sh
4.8G	.

Notice the drop… and wowza has only been up for 7 hours.

Richard,

I’m on wowza 4 so it’s the latest that is packaged with that. :slight_smile:

I’ll try that debug setup and see what I can see. Where should I see the debug output?

Jake

Here is what I show when the startup happens… just want to verify all values are correct:

2014-02-19      17:33:29        CST     comment server  INFO    200     -       MediaCache[MediaCache]: Thread pool size: [writer/readahead]:16/8       -       -       -       1.192   -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -
2014-02-19      17:33:29        CST     comment server  INFO    200     -       MediaCache[MediaCache]: MAX Pending size: [writer/readahead]:500M/25M   -       -       -       1.192   -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -
2014-02-19      17:33:29        CST     comment server  INFO    200     -       MediaCache[MediaCache]: Add store: path:/usr/local/WowzaStreamingEngine/mediacache maxSize:536870912000 -       -       -       1.193   -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -
2014-02-19      17:33:29        CST     comment server  INFO    200     -       MediaCache[MediaCache]: Add source[http]: prefix: basePath:http://xxx.xxx.xxx.xxx:8080/  -       -       -       1.206   -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -
2014-02-19      17:33:29        CST     comment server  INFO    200     -       MediaCache[MediaCache]: Flushing cache: start   -       -       -       1.21    -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -
2014-02-19      17:33:30        CST     comment server  INFO    200     -       MediaCache[MediaCache]: Flushing cache: done    -       -       -       1.864   -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -
2014-02-19      17:33:30        CST     comment server  INFO    200     -       MediaCache[MediaCache]: Start MediaCache GC     -       -       -       1.864   -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -
2014-02-19      17:33:30        CST     comment server  INFO    200     -       MediaCache[MediaCache]: Started -       -       -       1.865   -

Those aren’t the debug of course… but wanted to get those in the ticket. I’ll look for more interesting items now in the access log.

Roger,

Thank you for the feedback. I’ll use jconsole to see what I can see. The media cache store is on the same partition.

The odd thing is that I see a steady increase to the disk after a restart… and then things plateau. We do not have a high rate of abandonment… these are online lectures and so people want to watch the whole thing from front to back. So the ramp up is aggressive but then it stops… so that’s why I think media cache is not working properly.

Here is a graphic of what I’m seeing… the disk should continue to fill up but it stops.

https://dl.dropboxusercontent.com/u/23700979/wowza_media_cache.png

Here is the output of df for the applicable disk… I’ve got tons of room left to fill so MediaCache shouldn’t ever feel it has to dump anything:

# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda6       732G  4.7G  690G   1% /

Thanks for all your help.

Jake

I have figured out what the problem turned out to be. With the 4.0.0 release the UI incorrectly set the timeout properties for streams in the media cache configuration file. The UI reported 7 days for the min and 14 days for the max but those values were based off of seconds rather than milliseconds whereas the media cache code reads them in milliseconds. So needless to say the max time to keep files was way too low. It ended up being around 20 minutes or something for the max.

Of course modifying the config manually to use milliseconds has cleared it up.

So considering the following incorrect code:

<MediaCacheSource>
                        <Name>http</Name>
                        <Type>HTTP</Type>
                        <Description></Description>
                        <BasePath>http://xxx.xxx.xxx.xxx:8080/</BasePath>
                        <Prefix></Prefix>
                        <BaseClass>com.wowza.wms.mediacache.impl.MediaCacheItemHTTPImpl</BaseClass>
                        <ReaderClass></ReaderClass>
                        <DefaultBlockSize>1M</DefaultBlockSize>
                        <MaxTimeToLive>1209600</MaxTimeToLive>
                        <MinTimeToLive>604800</MinTimeToLive>

The fixed config:

<MediaCacheSource>
                        <Name>http</Name>
                        <Type>HTTP</Type>
                        <Description></Description>
                        <BasePath>http://xxx.xxx.xxx.xxx:8080/</BasePath>
                        <Prefix></Prefix>
                        <BaseClass>com.wowza.wms.mediacache.impl.MediaCacheItemHTTPImpl</BaseClass>
                        <ReaderClass></ReaderClass>
                        <DefaultBlockSize>1M</DefaultBlockSize>
                        <MaxTimeToLive>1209600000</MaxTimeToLive>
                        <MinTimeToLive>604800000</MinTimeToLive>

So the MaxTimeToLive and MinTimeToLive were problematic.

The new UI does make things handy but I got bit by that bug that squeaked through.

Thanks for all the help.

Jake

Roger/Tim,

Yes I can confirm the new 4.0.1 release fixes that issue. Thank you for helping to look into this.

Jake

Jake,

Take a look at the release notes for WSE version 4.0.1, which was released yesterday. This might have been fixed with the update.

-Tim

Hi Jake,

What you may be seeing is that the OS is reporting the actual size of the content on disk but Mediacache is seeing the size of the requested content.

If you have a high abandonment rate (viewers only watch a very short period of a long video) then these two values will be very different.

When a piece of content is requested, the Mediacache reserves space in the cache for that piece of content to fit. It does so by writing an empty file to disk with the size set to the media size. Because it is empty, the OS does not see it as used space. As the blocks are pulled in, they are deposited in the file and the size on disk grows.

When the MediaCache is determining if the cache is full or needs to purge anything, it looks at it’s own reserved size and not the size on disk. If it is running full (based on what MediaCache sees) then it will try to purge items based on the Minimum TTL value. This is possibly why you are seeing the size reduce on disk.

If The mediaCache cannot purge enough old items to make space for the new content then it will not put the new content into the cache and instead, the requests will bypass the mediaCache and go directly to the storage server. This is very bad as it could overload the storage server with lots of tiny requests.

If you do have a high abandonment rate then I suggest that you use lower TTL settings so that content will get purged quicker. It will always remain in cache if it has been watched recently.

You can see what the MediaCache is doing by enabling JMX in the Manager(Server > Properties > JMX Remote Configuration) and then after the restart, connect to the server using jConsole and view the MediaCache mbeans.

Roger.

Also, if the mediacache store is on a separate partition, using the df command (rather than the du command) should report the same as what MediaCache is seeing.

Roger.

Hi Jake,

I just compared the two versions and the new release does appear to be setting the value correctly now.

Roger.