Unexpected exception.: java.lang.OutOfMemoryError: Java heap space
For the past few weeks, Wowza has been crashing with an OutofMemoryError every couple days, particularly over weekends.
This seems to have starting when we started using the Transcoder AddOn, but has persisted even though I've turned off everything except audio transrating for select streams.
There are probably about 15 streams running at a time, all of them being recorded, with users occasionally connecting to the streams as well.
Most of the time, heap usage isn't too bad, but then it spikes for unknown reasons. This doesn't match up with any memory, network, IO, or CPU profiles on the server it's on, so it doesn't appear to have to do with external programs. It also doesn't seem to have much connection to the number of connections on the server, and the streams should be coming in at the same rate all the time.
Looking at the .hprof file generated, the main culprits were Cupertino and SanJose streamers. I turned off the Cupertino, but the San Jose streamer then just expanded in memory usage to produce the same effect.
I've gone through the Performance Tuning manual, but haven't found anything to help yet in there. The heap is already set to the maximum 10G allowed.