interrupted VOD clients and stuck threads, memory leak

[proger]

When testing load with series of parallel rtmpdump invocations I've noticed that several Client objects remain 'active' in the server object graph after the connections have been interrupted:

This can be seen from the heap dump and JMX connection counters.

# jmxterm
$>bean  
WowzaMediaServerPro:name=Connections
$>get current
#mbean = WowzaMediaServerPro:name=Connections:
current = 8;

This can not be any live repeaters or such because only a single instance of the 'default' app type is running.

# ss -tp | grep java
#

# ls -lah /proc/$(pgrep java)/fd | grep mp4
lr-x------. 1 root root 64 May 31 11:30 340 -> /vod/d9aqUMgVbSc-18.mp4
lr-x------. 1 root root 64 May 31 11:30 386 -> /vod/d9aqUMgVbSc-18.mp4
lr-x------. 1 root root 64 May 31 11:30 472 -> /vod/takeE1332267049.mp4
lr-x------. 1 root root 64 May 31 11:30 473 -> /vod/takeE1332267049.mp4
lr-x------. 1 root root 64 May 31 11:30 521 -> /vod/outro.mp4
lr-x------. 1 root root 64 May 31 11:30 584 -> /vod/outro.mp4
lr-x------. 1 root root 64 May 31 11:30 692 -> /vod/takeE1332267049.mp4
lr-x------. 1 root root 64 May 31 11:30 695 -> /vod/takeE1332267049.mp4
lr-x------. 1 root root 64 May 31 11:30 719 -> /vod/youtube5/outro.mp4
lr-x------. 1 root root 64 May 31 11:30 826 -> /vod/clip.mp4
lr-x------. 1 root root 64 May 31 11:30 829 -> /vod/clip.mp4
# ls -lah /proc/$(pgrep java)/fd | grep mp4 | wc -l
11

Apart from that jstack shows 5 threads with similar stack traces which are waiting on read ahead(?) data from some files:

"VHostHandler._defaultVHost_.92" prio=10 tid=0x0000000000c40000 nid=0x6a3f in Object.wait() [0x00007f1f5fc3b000]
"VHostHandler._defaultVHost_.86" prio=10 tid=0x0000000000a84800 nid=0x6a39 in Object.wait() [0x00007f1f60241000]
"VHostHandler._defaultVHost_.52" prio=10 tid=0x0000000000b96000 nid=0x6a05 in Object.wait() [0x00007f1f63675000]
"VHostHandler._defaultVHost_.51" prio=10 tid=0x0000000000b94000 nid=0x6a04 in Object.wait() [0x00007f1f63776000]
"VHostHandler._defaultVHost_.13" daemon prio=10 tid=0x00000000008b3800 nid=0x6995 in Object.wait() [0x00007f1f6a1e0000]


"VHostHandler._defaultVHost_.13" daemon prio=10 tid=0x00000000008b3800 nid=0x6995 in Object.wait() [0x00007f1f6a1e0000]
   java.lang.Thread.State: WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 at java.lang.Object.wait(Object.java:503)
 at com.wowza.wms.mediareader.h264.H264ReadAheadRequest.waitForComplete(Unknown Source)
 - locked <0x00000000a5de51f0> (a com.wowza.wms.mediareader.h264.H264ReadAheadRequest)
 at com.wowza.wms.mediareader.h264.MediaReaderH264.getReadAheadPacket(Unknown Source)
 at com.wowza.wms.mediareader.h264.MediaReaderH264.getSampleDesc(Unknown Source)
 at com.wowza.wms.mediareader.h264.MediaReaderH264.writePackets(Unknown Source)
 - locked <0x00000000a5a50788> (a com.wowza.wms.stream.record.MediaStreamRecord)
 at com.wowza.wms.mediareader.h264.MediaReaderH264.writePackets(Unknown Source)
 at com.wowza.wms.stream.file.PlaylistPlayer.play(Unknown Source)
 at com.wowza.wms.stream.file.MediaStreamFilePlay.play(Unknown Source)
 at com.wowza.wms.response.ResponseStreams.output(Unknown Source)
 at com.wowza.wms.request.RTMPRequestAdapter.service(Unknown Source)
 - locked <0x00000000a5a49540> (a java.lang.Object)
 at com.wowza.wms.server.ServerHandler.a(Unknown Source)
 at com.wowza.wms.server.ServerHandler.a(Unknown Source)
 at com.wowza.wms.server.ServerHandler.sessionIdle(Unknown Source)
 at com.wowza.wms.server.ServerHandlerThreadedSession.run(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
 at java.lang.Thread.run(Thread.java:722)

Software versions:
Wowza Media Server 3.1.1 build1479
Java(TM) SE Runtime Environment (build 1.7.0-b147)
Java HotSpot(TM) 64-Bit Server VM (build 21.0-b17, mixed mode)
Linux 3.1.6


Increasing number of such 'ghost' users result in both increased memory consumption and dead threads in the limited pool which does not let us run WMS for long periods of time without interrupting service. Could you take a look at the issue from your side?

This can be reproduced by constantly running multiple rtmpdump instances from the other machine and randomly killing them.

[rrlanham]

Take a look at this guide re connection count problems:
http://www.wowza.com/forums/showthre...count-problems

Have you tried the Wowza Load Test tool? You can get it by writing to test@wowza.com. There is a document to sign and return.

Richard

[kutovoy]

Happened to our 100+ servers at 1st July 00:00am (different platforms/custom modules or even no modules at all/different projects).

More details in ticket #32042

we are still expiriencing big problems on many servers due to this problem.

177-277 threads, most of them are locked.

[kutovoy]

few details about our case: only x64 machines with wowza2/3, and only wowza service was affected. seems to be connected with leap second linux kernel bug.

[randall]

Rebooting the computer usually fixes the July 1st leap second Java bug.

The issue is resolved now?

[proger]

how is linux/java futex bug related to this thread at all?