Results 1 to 6 of 6

Thread: Transcoder errors when running 3.6.2 on low memory setting

  1. #1

    Default Transcoder errors when running 3.6.2 on low memory setting

    This may not really be a problem, but I want to rule out that this is unrelated to the memory settings of the server.

    The crashes happened with a default install Heap size of just 768M. We always had been running 8000M and now we are back to it, but I haven't started the transcoding slots up again to keep the other streams from getting dropped due to these transcoder errors crashing everything.

    We have had two crashes over the last 5 days, and each time I found transcoder errors in the log at the time the server stopped working properly (and dropping all streams, even those not using the two transcoder slots)

    from the access log the most recent crash - it always starts with something like skipping ALLFRAMES or like here "Video behind filter state change. New state: SKIP1FRAME - My Question is - could these errors be a result of memory issues, or do I have something else going on? The server has rarely more than 10 clients and there are no VOD clients, just a few live stream users.

    what exactly is going on here? To me it appears like a transcoder error followed by all streams stopping, even those that aren't transcoded? Only WPTlive and WPTliveCC are transcoded.

    2013-09-04	04:51:48	CDT	comment	server	INFO	200	-	TranscodingSession.updateBehindFilter[WPTlive/_definst_/myStream]: Video behind filter state change. New state: SKIP1FRAME	-	-	-	71311.0	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
    2013-09-04	04:52:11	CDT	comment	server	INFO	200	-	MediaStreamMap.removeLiveStreamPacketizer[WPTliveCC/_definst_/myStream_360p]: Destroy live stream packetizer: sanjosestreamingpacketizer	-	-	-	71315.736	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
    2013-09-04	04:52:06	CDT	comment	server	INFO	200	-	MediaStreamMap.removeLiveStreamPacketizer[WisconsinChannel/_definst_/mystream]: Destroy live stream packetizer: sanjosestreamingpacketizer	-	-	-	71312.291	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
    2013-09-04	04:52:20	CDT	comment	server	INFO	200	-	MediaStreamMap.removeLiveStreamPacketizer[WPTliveCC/_definst_/myStream_160p]: Destroy live stream packetizer: cupertinostreamingpacketizer	-	-	-	71320.527	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
    2013-09-04	04:52:20	CDT	comment	server	INFO	200	-	MediaStreamMap.removeLiveStreamPacketizer[WisconsinChannel/_definst_/mystream]: Destroy live stream packetizer: cupertinostreamingpacketizer	-	-	-	71321.472	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
    2013-09-04	04:52:21	CDT	comment	server	INFO	200	-	MediaStreamMap.removeLiveStreamPacketizer[WPTliveCC/_definst_/myStream_source]: Destroy live stream packetizer: smoothstreamingpacketizer	-	-	-	71321.869	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
    2013-09-04	04:52:21	CDT	comment	server	INFO	200	-	MediaStreamMap.removeLiveStreamPacketizer[WPTlive/_definst_/myStream]: Destroy live stream packetizer: cupertinostreamingpacketizer	-	-	-	71321.87	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
    2013-09-04	04:52:22	CDT	comment	server	INFO	200	-	MediaStreamMap.removeLiveStreamPacketizer[WisconsinChannelCC/_definst_/myStream]: Destroy live stream packetizer: sanjosestreamingpacketizer	-	-	-	71322.599	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
    2013-09-04	04:52:23	CDT	comment	server	INFO	200	-	MediaStreamMap.removeLiveStreamPacketizer[WPTlive/_definst_/myStream_160p]: Destroy live stream packetizer: sanjosestreamingpacketizer	-	-	-	71323.414	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
    2013-09-04	04:52:23	CDT	comment	server	INFO	200	-	MediaStreamMap.removeLiveStreamPacketizer[WPTliveCC/_definst_/myStream]: Destroy live stream packetizer: sanjosestreamingpacketizer	-	-	-	71323.414	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
    2013-09-04	04:52:24	CDT	comment	server	INFO	200	-	MediaStreamMap.removeLiveStreamPacketizer[WPTlive/_definst_/myStream_360p]: Destroy live stream packetizer: smoothstreamingpacketizer	-	-	-	71324.292	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
    2013-09-04	04:52:23	CDT	comment	server	INFO	200	-	TranscodingSession.updateBehindFilter[WPTliveCC/_definst_/myStream]: Video behind filter state change. New state: SKIP4FRAME	-	-	-	71324.207	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
    2013-09-04	04:52:25	CDT	comment	server	INFO	200	-	MediaStreamMap.removeLiveStreamPacketizer[WPTliveCC/_definst_/myStream_360p]: Destroy live stream packetizer: cupertinostreamingpacketizer	-	-	-	71325.843	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
    2013-09-04	04:52:24	CDT	comment	server	INFO	200	-	MediaStreamMap.removeLiveStreamPacketizer[WPTlive/_definst_/myStream_source]: Destroy live stream packetizer: sanjosestreamingpacketizer	-	-	-	71324.862	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
    2013-09-04	04:52:27	CDT	comment	server	INFO	200	-	MediaStreamMap.removeLiveStreamPacketizer[WPTliveCC/_definst_/myStream_360p]: Destroy live stream packetizer: smoothstreamingpacketizer	-	-	-	71327.911	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
    2013-09-04	04:52:21	CDT	comment	server	WARN	200	-	Unexpected exception.	-	-	-	71321.87	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
    java.lang.OutOfMemoryError: Java heap space
    	at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
    	at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
    	at org.apache.mina.common.SimpleByteBufferAllocator.allocate(Unknown Source)
    	at org.apache.mina.common.ByteBuffer.allocate(Unknown Source)
    	at org.apache.mina.transport.socket.nio.SocketIoProcessor.read(Unknown Source)
    	at org.apache.mina.transport.socket.nio.SocketIoProcessor.process(Unknown Source)
    	at org.apache.mina.transport.socket.nio.SocketIoProcessor.access$600(Unknown Source)
    	at org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run(Unknown Source)
    	at org.apache.mina.util.NamePreservingRunnable.run(Unknown Source)
    	at java.lang.Thread.run(Thread.java:722)
    2013-09-04	04:52:21	CDT	comment	server	WARN	200	-	Unexpected exception.	-	-	-	71321.869	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
    java.lang.OutOfMemoryError: Java heap space
    	at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
    	at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
    	at org.apache.mina.common.SimpleByteBufferAllocator.allocate(Unknown Source)
    	at org.apache.mina.common.ByteBuffer.allocate(Unknown Source)
    	at org.apache.mina.transport.socket.nio.SocketIoProcessor.read(Unknown Source)
    	at org.apache.mina.transport.socket.nio.SocketIoProcessor.process(Unknown Source)
    	at org.apache.mina.transport.socket.nio.SocketIoProcessor.access$600(Unknown Source)
    	at org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run(Unknown Source)
    	at org.apache.mina.util.NamePreservingRunnable.run(Unknown Source)
    	at java.lang.Thread.run(Thread.java:722)
    2013-09-04	04:52:21	CDT	comment	server	WARN	200	-	Unexpected exception.	-	-	-	71321.869	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
    java.lang.OutOfMemoryError: Java heap space
    	at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
    	at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
    	at org.apache.mina.common.SimpleByteBufferAllocator.allocate(Unknown Source)
    	at org.apache.mina.common.ByteBuffer.allocate(Unknown Source)
    	at org.apache.mina.transport.socket.nio.SocketIoProcessor.read(Unknown Source)
    	at org.apache.mina.transport.socket.nio.SocketIoProcessor.process(Unknown Source)
    	at org.apache.mina.transport.socket.nio.SocketIoProcessor.access$600(Unknown Source)
    	at org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run(Unknown Source)
    	at org.apache.mina.util.NamePreservingRunnable.run(Unknown Source)
    	at java.lang.Thread.run(Thread.java:722)
    It never recovered from those errors. I've searched old logs and never found any similar transcoder errors before upgrading to 3.6.2.10 (we ran 3.5.0 up to now).

  2. #2
    Join Date
    Dec 2007
    Posts
    21,962

    Default

    With heap size set to 768M this is not unusual. The SKIPFRAME messages mean that the machine is not keeping up with the demands of the Transcoder. The result is ouOfMemory, which is a crash, all other streams will be affected.

    Richard

  3. #3

    Default

    thanks for confirming. We just started one transcoded stream and watched the RAM use jump from 150MB to 500MB - clearly, with two of these at 768MB, we probably were running at the limit of what was possible.

  4. #4

    Default

    Just a follow-up question. Increasing memory has cured the problem, and 768MB clearly wasn't enough. Now almost 3 days later, I am seeing the heap size still growing, though, even though the server isn't doing anything new since I restarted it. Below the memory graph for that period. My guess is, that this will continue to grow, slowly, but steadily:



    Is this a garbage collection issue? It seems to be doing it's thing, but over time, memory use has gone from an average 700-1000MB to now 1000-1500MB. There's plenty of headroom, but the trend here doesn't look very reassuring. Am I just paranoid?

  5. #5

    Default

    This morning, the server is running between 1.7GB and 2.0GB

    Just to clarify - the system is running those 2 transcoder streams (3 bitrates transcoded from each source) and 2 single bitrate live streams, with hardly any user traffic (15-20 users max), and nothing else. No VOD. This gradual creep in memory use upwards can't be a good thing under those conditions.

  6. #6

    Default

    Monday morning - spiking at 3.5GB in use, average about 2.6GB. I ran several manual garbage collections, but the trend is a pretty linear line upwards, and at this pace, we will hit 8GB in about 5 more days.

    should I take this over to the server tuning forum? I'm pretty sure this is transcoder related, though. Definitely not traffic - since server restart 5 days ago we have seen just 772 client sessions, while the server can easily handle 2500+ concurrent connections based on prior experience.


    current status from the summary

    Current heap size:*
    2,735,660 kbytes
    Maximum heap size:*
    7,281,792 kbytes
    Committed memory:*
    2,900,672 kbytes
    Pending finalization:*
    0 objects
    Garbage collector:*
    Name = 'PS MarkSweep', Collections = 5,270, Total time spent = 28 minutes
    Garbage collector:*
    Name = 'PS Scavenge', Collections = 229,927, Total time spent = 42 minutes

Similar Threads

  1. Multiple errors with Transcoder AddOn
    By marian04 in forum AddOn: Transcoder
    Replies: 7
    Last Post: 04-15-2013, 06:58 AM
  2. Need help with tuning - crashing when low free memory
    By silver2003tl in forum Performance Tuning Discussion
    Replies: 6
    Last Post: 02-17-2012, 09:21 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •