Memory panic on Windows with nDVR

[realmac]

After switching on 2 features:
1. SMIL playlist with 8 streams
2. nDVR for 10 streams for 12 hours

windows 2008 R2 64bit quad-core 4gb ram ( Java wrap 4000m memory heap size ) server enter "memory panic" mode after hour of operation and with only some connected streams
with following messages:

STATUS | wrapper  | 2012/05/17 13:06:56 | --> Wrapper Started as Service
STATUS | wrapper  | 2012/05/17 13:06:56 | Launching a JVM...
ERROR  | wrapper  | 2012/05/17 14:55:14 | JVM appears hung: Timed out waiting for signal from JVM.
ERROR  | wrapper  | 2012/05/17 14:55:15 | JVM did not exit on request, terminated
STATUS | wrapper  | 2012/05/17 14:55:21 | Launching a JVM...
ERROR  | wrapper  | 2012/05/17 17:09:36 | JVM appears hung: Timed out waiting for signal from JVM.
ERROR  | wrapper  | 2012/05/17 17:09:37 | JVM did not exit on request, terminated
STATUS | wrapper  | 2012/05/17 17:09:43 | Launching a JVM...
ERROR  | wrapper  | 2012/05/17 19:33:51 | JVM appears hung: Timed out waiting for signal from JVM.
ERROR  | wrapper  | 2012/05/17 19:33:52 | JVM did not exit on request, terminated
STATUS | wrapper  | 2012/05/17 19:33:58 | Launching a JVM...
ERROR  | wrapper  | 2012/05/17 19:34:30 | Startup failed: Timed out waiting for a signal from the JVM.
ERROR  | wrapper  | 2012/05/17 19:34:30 | JVM did not exit on request, terminated
STATUS | wrapper  | 2012/05/17 19:34:35 | Launching a JVM...
STATUS | wrapper  | 2012/05/17 20:21:55 | <-- Wrapper Stopped

Are there any extra RAM requirements to run nDVR?
Could we put in production environment?

Some extra info:

Windows memory processes with nDVR:



Processes after disabling SMIL and nDVR:




Some error messages:

#Fields: x-severity	x-category	x-event	date	time	c-client-id	c-ip	c-port	cs-bytes	sc-bytes	x-duration	x-sname	x-stream-id	x-spos	sc-stream-bytes	cs-stream-bytes	x-file-size	x-file-length	x-ctx	x-comment
WARN	server	comment	2012-05-17	19:35:00	-	-	-	-	-	5.85	-	-	-	-	-	-	-	-	QTUtils.parseAtomstbl[C:\Program Files (x86)\Wowza Media Systems\Wowza Media Server 3.0.4\content\xxxx.f4v]: Error reading 'stsd' section. Structure is zero size: smhd:skip
WARN	server	comment	2012-05-17	19:35:56	-	-	-	-	-	61.714	-	-	-	-	-	-	-	-	DvrManifestContainerParser.readContainerManifest[live/_definst_]: File '0063_10_00\dvrManifest_0063_10_00.txt' does not exist.
ERROR	server	comment	2012-05-17	19:35:56	-	-	-	-	-	61.714	-	-	-	-	-	-	-	-	DvrManifestContainerParser.retrieveContainerManifest: Error reading Manifest.
WARN	server	comment	2012-05-17	19:36:19	-	-	-	-	-	0.453	-	-	-	-	-	-	-	-	HTTPStreamerAdapterCupertinoStreamer.getAppInstance: Stream not found [xxx/stream/playlist.m3u8]: stream
WARN	server	comment	2012-05-17	19:37:49	-	-	-	-	-	174.58	-	-	-	-	-	-	-	-	LiveStreamPacketizerSmoothStreaming.handlePacket[live/_definst_/xxx]: Fragment duration greater than suggested range of 1-4 seconds. Adjust keyframe interval accordingly: Fragment durations: [1.4,2.9,6.0]
WARN	server	comment	2012-05-17	19:43:41	-	-	-	-	-	526.484	-	-	-	-	-	-	-	-	LiveStreamPacketizerSmoothStreaming.handlePacket[live/_definst_/xxx]: Fragment duration greater than suggested range of 1-4 seconds. Adjust keyframe interval accordingly: Fragment durations: [10.0,10.0,5.1]
WARN	server	comment	2012-05-17	19:43:42	-	-	-	-	-	528.229	-	-	-	-	-	-	-	-	LiveStreamDvrRecorder.reportCodecInformation[live/_definst_/stream3][avc1.100.30]: H.264 (Video may not be playable on older iPhone and iPod touch devices where Baseline/Level 3.0 or lower is required) Video info: : {MediaCodecInfoVideo: codec:H264, profile:High, level:3.0, frameSize:640x480, displaySize:640x480, frameRate:25,000000}
WARN	server	comment	2012-05-17	19:43:42	-	-	-	-	-	528.229	-	-	-	-	-	-	-	-	LiveStreamDvrRecorder.reportCodecInformation[live/_definst_/stream3][No audio]: Smooth must have audio and video to play.
WARN	server	comment	2012-05-17	19:44:06	-	-	-	-	-	552.005	-	-	-	-	-	-	-	-	LiveStreamPacketizerSmoothStreaming.handlePacket[live/_definst_/xxx]: Fragment duration greater than suggested range of 1-4 seconds. Adjust keyframe interval accordingly: Fragment durations: [3.0,2.9,5.9]
WARN	server	comment	2012-05-17	19:47:40	-	-	-	-	-	765.836	-	-	-	-	-	-	-	-	LiveStreamDvrRecorder.reportCodecInformation[live/_definst_/xxx][avc1.66.31]: H.264 (Video may not be playable on older iPhone and iPod touch devices where Baseline/Level 3.0 or lower is required) Video info: : {MediaCodecInfoVideo: codec:H264, profile:Baseline, level:3.1, frameSize:640x480, displaySize:640x480, frameRate:25,000000}
WARN	server	comment	2012-05-17	19:47:40	-	-	-	-	-	765.849	-	-	-	-	-	-	-	-	LiveStreamPacketizerSmoothStreaming.handlePacket[live/_definst_/xxxx]: Fragment duration greater than suggested range of 1-4 seconds. Adjust keyframe interval accordingly: Fragment durations: [6.6,5.0,5.0]
WARN	server	comment	2012-05-17	19:47:40	-	-	-	-	-	766.19	-	-	-	-	-	-	-	-	DvrChannelManifest.remove[live/_definst_/xxx/xxxx.0] : attempt to remove manifest entry index:1 that does not exist.  range:0-136
WARN	server	comment	2012-05-17	19:48:00	-	-	-	-	-	785.957	-	-	-	-	-	-	-	-	LiveStreamDvrRecorder.reportCodecInformation[live/_definst_/xx][avc1.77.30]: H.264 (Video may not be playable on older iPhone and iPod touch devices where Baseline/Level 3.0 or lower is required) Video info: : {MediaCodecInfoVideo: codec:H264, profile:Main, level:3.0, frameSize:640x360, displaySize:640x360, frameRate:25,000000}
WARN	server	comment	2012-05-17	19:49:42	-	-	-	-	-	887.509	-	-	-	-	-	-	-	-	LiveStreamPacketizerSmoothStreaming.handlePacket[live/_definst_/xx]: Fragment duration greater than suggested range of 1-4 seconds. Adjust keyframe interval accordingly: Fragment durations: [6.2,6.0,5.9]
WARN	server	comment	2012-05-17	19:49:51	-	-	-	-	-	896.393	-	-	-	-	-	-	-	-	LiveStreamDvrRecorder.reportCodecInformation[live/_definst_/xx][avc1.66.31]: H.264 (Video may not be playable on older iPhone and iPod touch devices where Baseline/Level 3.0 or lower is required) Video info: : {MediaCodecInfoVideo: codec:H264, profile:Baseline, level:3.1, frameSize:720x404, displaySize:720x404, frameRate:29,970000}
WARN	server	comment	2012-05-17	19:50:19	-	-	-	-	-	924.983	-	-	-	-	-	-	-	-	LiveStreamDvrRecorder.reportCodecInformation[live/_definst_/xxx][avc1.66.31]: H.264 (Video may not be playable on older iPhone and iPod touch devices where Baseline/Level 3.0 or lower is required) Video info: : {MediaCodecInfoVideo: codec:H264, profile:Baseline, level:3.1, frameSize:720x404, displaySize:720x404, frameRate:29,970000}
WARN	server	comment	2012-05-17	19:50:37	-	-	-	-	-	942.324	-	-	-	-	-	-	-	-	LiveStreamDvrRecorder.reportCodecInformation[live/_definst_/xx][avc1.100.30]: H.264 (Video may not be playable on older iPhone and iPod touch devices where Baseline/Level 3.0 or lower is required) Video info: : {MediaCodecInfoVideo: codec:H264, profile:High, level:3.0, frameSize:640x480, displaySize:640x480, frameRate:25,000000}
WARN	server	comment	2012-05-17	19:50:37	-	-	-	-	-	942.371	-	-	-	-	-	-	-	-	LiveStreamPacketizerSmoothStreaming.handlePacket[live/_definst_/xxxxx]: Fragment duration greater than suggested range of 1-4 seconds. Adjust keyframe interval accordingly: Fragment durations: [12.0,12.0,12.0]
WARN	server	comment	2012-05-17	19:50:37	-	-	-	-	-	942.823	-	-	-	-	-	-	-	-	DvrChannelManifest.remove[live/_definst_/xx/xxx.0] : attempt to remove manifest entry index:1 that does not exist.  range:0-200
WARN	server	comment	2012-05-17	19:51:04	-	-	-	-	-	969.468	-	-	-	-	-	-	-	-	LiveStreamDvrRecorder.reportCodecInformation[live/_definst_xx_SV][avc1.77.30]: H.264 (Video may not be playable on older iPhone and iPod touch devices where Baseline/Level 3.0 or lower is required) Video info: : {MediaCodecInfoVideo: codec:H264, profile:Main, level:3.0, frameSize:720x404, displaySize:720x404, frameRate:25,000000}
WARN	server	comment	2012-05-17	19:51:04	-	-	-	-	-	969.492	-	-	-	-	-	-	-	-	LiveStreamPacketizerSmoothStreaming.handlePacket[live/_definst_/xx_SV]: Fragment duration greater than suggested range of 1-4 seconds. Adjust keyframe interval accordingly: Fragment durations: [6.2,6.0,5.9]
WARN	server	comment	2012-05-17	19:54:53	-	-	-	-	-	1199.18	-	-	-	-	-	-	-	-	DvrManifestContainerParser.readContainerManifest[live/_definst_]: File '0063_10_00\dvrManifest_0063_10_00.txt' does not exist.
ERROR	server	comment	2012-05-17	19:54:53	-	-	-	-	-	1199.18	-	-	-	-	-	-	-	-	DvrManifestContainerParser.retrieveContainerManifest: Error reading Manifest.
WARN	server	comment	2012-05-17	20:00:00	-	-	-	-	-	1506.052	-	-	-	-	-	-	-	-	DvrChannelManifest.remove[live/_definst_/vvvv/vvvv.0] : attempt to remove manifest entry index:1 that does not exist.  range:0-91
WARN	server	comment	2012-05-17	20:04:22	-	-	-	-	-	1767.754	-	-	-	-	-	-	-	-	LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0
WARN	server	comment	2012-05-17	20:04:22	-	-	-	-	-	1767.764	-	-	-	-	-	-	-	-	LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0
WARN	server	comment	2012-05-17	20:04:22	-	-	-	-	-	1767.767	-	-	-	-	-	-	-	-	LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0
WARN	server	comment	2012-05-17	20:04:22	-	-	-	-	-	1767.77	-	-	-	-	-	-	-	-	LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0
WARN	server	comment	2012-05-17	20:04:22	-	-	-	-	-	1767.774	-	-	-	-	-	-	-	-	LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0
WARN	server	comment	2012-05-17	20:04:22	-	-	-	-	-	1767.786	-	-	-	-	-	-	-	-	LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0
WARN	server	comment	2012-05-17	20:04:22	-	-	-	-	-	1767.795	-	-	-	-	-	-	-	-	LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0
WARN	server	comment	2012-05-17	20:04:22	-	-	-	-	-	1767.799	-	-	-	-	-	-	-	-	LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0
WARN	server	comment	2012-05-17	20:04:22	-	-	-	-	-	1767.801	-	-	-	-	-	-	-	-	LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0
WARN	server	comment	2012-05-17	20:04:22	-	-	-	-	-	1767.807	-	-	-	-	-	-	-	-	LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0
WARN	server	comment	2012-05-17	20:04:22	-	-	-	-	-	1767.812	-	-	-	-	-	-	-	-	LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0
WARN	server	comment	2012-05-17	20:04:22	-	-	-	-	-	1767.818	-	-	-	-	-	-	-	-	LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0
WARN	server	comment	2012-05-17	20:04:22	-	-	-	-	-	1767.827	-	-	-	-	-	-	-	-	LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0
WARN	server	comment	2012-05-17	20:04:22	-	-	-	-	-	1767.831	-	-	-	-	-	-	-	-	LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0
WARN	server	comment	2012-05-17	20:04:22	-	-	-	-	-	1767.832	-	-	-	-	-	-	-	-	LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0
WARN	server	comment	2012-05-17	20:04:22	-	-	-	-	-	1767.838	-	-	-	-	-	-	-	-	LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0
WARN	server	comment	2012-05-17	20:04:22	-	-	-	-	-	1767.841	-	-	-	-	-	-	-	-	LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0
WARN	server	comment	2012-05-17	20:04:22	-	-	-	-	-	1767.855	-	-	-	-	-	-	-	-	LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0
WARN	server	comment	2012-05-17	20:04:22	-	-	-	-	-	1767.861	-	-	-	-	-	-	-	-	LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0
WARN	server	comment	2012-05-17	20:04:22	-	-	-	-	-	1767.865	-	-	-	-	-	-	-	-	LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0
WARN	server	comment	2012-05-17	20:04:22	-	-	-	-	-	1767.869	-	-	-	-	-	-	-	-	LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0
WARN	server	comment	2012-05-17	20:04:22	-	-	-	-	-	1767.872	-	-	-	-	-	-	-	-	LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0
WARN	server	comment	2012-05-17	20:04:22	-	-	-	-	-	1767.874	-	-	-	-	-	-	-	-	LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0
WARN	server	comment	2012-05-17	20:04:22	-	-	-	-	-	1767.876	-	-	-	-	-	-	-	-	LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0
WARN	server	comment	2012-05-17	20:04:22	-	-	-	-	-	1767.877	-	-	-	-	-	-	-	-	LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0
WARN	server	comment	2012-05-17	20:04:22	-	-	-	-	-	1767.878	-	-	-	-	-	-	-	-	LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0
WARN	server	comment	2012-05-17	20:04:22	-	-	-	-	-	1767.88	-	-	-	-	-	-	-	-	LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0
WARN	server	comment	2012-05-17	20:08:05	-	-	-	-	-	1990.537	-	-	-	-	-	-	-	-	LiveStreamDvrRecorder.reportCodecInformation[live/_definst_/stream2][avc1.100.30]: H.264 (Video may not be playable on older iPhone and iPod touch devices where Baseline/Level 3.0 or lower is required) Video info: : {MediaCodecInfoVideo: codec:H264, profile:High, level:3.0, frameSize:640x480, displaySize:640x480, frameRate:25,000000}
WARN	server	comment	2012-05-17	20:09:48	-	-	-	-	-	2094.126	-	-	-	-	-	-	-	-	DvrStreamManagerBase.addManifestEntries[live/_definst_/xxx] :  store not found for vStream:test_signal.0
WARN	server	comment	2012-05-17	20:09:48	-	-	-	-	-	2094.138	-	-	-	-	-	-	-	-	LiveStreamDvrRecorder.reportCodecInformation[live/_definst_/xxx][avc1.77.40]: H.264 (Video may not be playable on older iPhone and iPod touch devices where Baseline/Level 3.0 or lower is required) Video info: : {MediaCodecInfoVideo: codec:H264, profile:Main, level:4.0, frameSize:1920x1080, displaySize:1920x1080, frameRate:0,000000}
WARN	server	comment	2012-05-17	20:19:58	-	-	-	-	-	2703.776	-	-	-	-	-	-	-	-	LiveStreamPacketizerSmoothStreaming.handlePacket[live/_definst_/xxx]: Fragment duration greater than suggested range of 1-4 seconds. Adjust keyframe interval accordingly: Fragment durations: [6.0,6.0,2.8]
ERROR	server	comment	2012-05-17	20:21:52	-	-	-	-	-	2818.133	-	-	-	-	-	-	-	-	execute: java.util.concurrent.RejectedExecutionException: Task com.wowza.wms.stream.live.LiveStreamPacketizerRunner@2af75e5 rejected from com.wowza.wms.vhost.ThreadPoolExecutorReinit@63ec3e7[Shutting down, pool size = 90, active threads = 2, queued tasks = 0, completed tasks = 3443884]
ERROR	server	comment	2012-05-17	20:21:52	-	-	-	-	-	2818.133	-	-	-	-	-	-	-	-	execute: java.util.concurrent.RejectedExecutionException: Task com.wowza.wms.stream.live.LiveStreamPacketizerRunner@7421432d rejected from com.wowza.wms.vhost.ThreadPoolExecutorReinit@63ec3e7[Shutting down, pool size = 90, active threads = 2, queued tasks = 0, completed tasks = 3443884]
ERROR	server	comment	2012-05-17	20:21:52	-	-	-	-	-	2818.137	-	-	-	-	-	-	-	-	execute: java.util.concurrent.RejectedExecutionException: Task com.wowza.wms.stream.live.LiveStreamPacketizerRunner@6874615b rejected from com.wowza.wms.vhost.ThreadPoolExecutorReinit@63ec3e7[Shutting down, pool size = 47, active threads = 2, queued tasks = 0, completed tasks = 3443884]
ERROR	server	comment	2012-05-17	20:21:52	-	-	-	-	-	2818.133	-	-	-	-	-	-	-	-	execute: java.util.concurrent.RejectedExecutionException: Task com.wowza.wms.stream.live.LiveStreamPacketizerRunner@4d650ec3 rejected from com.wowza.wms.vhost.ThreadPoolExecutorReinit@63ec3e7[Shutting down, pool size = 84, active threads = 2, queued tasks = 0, completed tasks = 3443884]
ERROR	server	comment	2012-05-17	20:21:52	-	-	-	-	-	2818.142	-	-	-	-	-	-	-	-	execute: java.util.concurrent.RejectedExecutionException: Task com.wowza.wms.stream.live.LiveStreamPacketizerRunner@3de39c75 rejected from com.wowza.wms.vhost.ThreadPoolExecutorReinit@63ec3e7[Shutting down, pool size = 14, active threads = 2, queued tasks = 0, completed tasks = 3443884]
ERROR	server	comment	2012-05-17	20:21:52	-	-	-	-	-	2818.144	-	-	-	-	-	-	-	-	execute: java.util.concurrent.RejectedExecutionException: Task com.wowza.wms.stream.live.LiveStreamPacketizerRunner@4974bf81 rejected from com.wowza.wms.vhost.ThreadPoolExecutorReinit@63ec3e7[Shutting down, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 3443884]
ERROR	server	comment	2012-05-17	20:21:52	-	-	-	-	-	2818.142	-	-	-	-	-	-	-	-	execute: java.util.concurrent.RejectedExecutionException: Task com.wowza.wms.stream.live.LiveStreamPacketizerRunner@8d49a62 rejected from com.wowza.wms.vhost.ThreadPoolExecutorReinit@63ec3e7[Shutting down, pool size = 14, active threads = 2, queued tasks = 0, completed tasks = 3443884]
ERROR	server	comment	2012-05-17	20:21:52	-	-	-	-	-	2818.141	-	-	-	-	-	-	-	-	execute: java.util.concurrent.RejectedExecutionException: Task com.wowza.wms.stream.live.LiveStreamDvrRunner@30a3d2f0 rejected from com.wowza.wms.vhost.ThreadPoolExecutorReinit@63ec3e7[Shutting down, pool size = 37, active threads = 2, queued tasks = 0, completed tasks = 3443884]
ERROR	server	comment	2012-05-17	20:21:52	-	-	-	-	-	2818.136	-	-	-	-	-	-	-	-	execute: java.util.concurrent.RejectedExecutionException: Task com.wowza.wms.stream.live.LiveStreamPacketizerRunner@45427824 rejected from com.wowza.wms.vhost.ThreadPoolExecutorReinit@63ec3e7[Shutting down, pool size = 63, active threads = 2, queued tasks = 0, completed tasks = 3443884]
ERROR	server	comment	2012-05-17	20:21:52	-	-	-	-	-	2818.145	-	-	-	-	-	-	-	-	execute: java.util.concurrent.RejectedExecutionException: Task com.wowza.wms.stream.live.LiveStreamDvrRunner@3158f815 rejected from com.wowza.wms.vhost.ThreadPoolExecutorReinit@63ec3e7[Shutting down, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 3443884]
ERROR	server	comment	2012-05-17	20:21:52	-	-	-	-	-	2818.144	-	-	-	-	-	-	-	-	execute: java.util.concurrent.RejectedExecutionException: Task com.wowza.wms.stream.live.LiveStreamDvrRunner@49cb0b64 rejected from com.wowza.wms.vhost.ThreadPoolExecutorReinit@63ec3e7[Shutting down, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 3443884]
ERROR	server	comment	2012-05-17	20:21:52	-	-	-	-	-	2818.144	-	-	-	-	-	-	-	-	execute: java.util.concurrent.RejectedExecutionException: Task com.wowza.wms.stream.live.LiveStreamDvrRunner@fc8d3d4 rejected from com.wowza.wms.vhost.ThreadPoolExecutorReinit@63ec3e7[Shutting down, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 3443884]
ERROR	server	comment	2012-05-17	20:21:52	-	-	-	-	-	2818.143	-	-	-	-	-	-	-	-	execute: java.util.concurrent.RejectedExecutionException: Task com.wowza.wms.stream.live.LiveStreamDvrRunner@42c2e33c rejected from com.wowza.wms.vhost.ThreadPoolExecutorReinit@63ec3e7[Shutting down, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 3443884]
ERROR	server	comment	2012-05-17	20:21:52	-	-	-	-	-	2818.143	-	-	-	-	-	-	-	-	execute: java.util.concurrent.RejectedExecutionException: Task com.wowza.wms.stream.live.LiveStreamDvrRunner@76aad4bc rejected from com.wowza.wms.vhost.ThreadPoolExecutorReinit@63ec3e7[Shutting down, pool size = 8, active threads = 2, queued tasks = 0, completed tasks = 3443884]
ERROR	server	comment	2012-05-17	20:21:52	-	-	-	-	-	2818.143	-	-	-	-	-	-	-	-	execute: java.util.concurrent.RejectedExecutionException: Task com.wowza.wms.stream.live.LiveStreamDvrRunner@52ab3e03 rejected from com.wowza.wms.vhost.ThreadPoolExecutorReinit@63ec3e7[Shutting down, pool size = 12, active threads = 2, queued tasks = 0, completed tasks = 3443884]
ERROR	server	comment	2012-05-17	20:21:52	-	-	-	-	-	2818.166	-	-	-	-	-	-	-	-	execute: java.util.concurrent.RejectedExecutionException: Task com.wowza.wms.stream.live.LiveStreamPacketizerRunner@7764efb2 rejected from com.wowza.wms.vhost.ThreadPoolExecutorReinit@63ec3e7[Shutting down, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 3443884]
ERROR	server	comment	2012-05-17	20:21:52	-	-	-	-	-	2818.225	-	-	-	-	-	-	-	-	execute: java.util.concurrent.RejectedExecutionException: Task com.wowza.wms.stream.live.LiveStreamPacketizerRunner@52d46c43 rejected from com.wowza.wms.vhost.ThreadPoolExecutorReinit@63ec3e7[Shutting down, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 3443884]
ERROR	server	comment	2012-05-17	20:21:52	-	-	-	-	-	2818.154	-	-	-	-	-	-	-	-	execute: java.util.concurrent.RejectedExecutionException: Task com.wowza.wms.stream.live.LiveStreamPacketizerRunner@5502043f rejected from com.wowza.wms.vhost.ThreadPoolExecutorReinit@63ec3e7[Shutting down, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 3443884]
ERROR	server	comment	2012-05-17	20:21:52	-	-	-	-	-	2818.145	-	-	-	-	-	-	-	-	execute: java.util.concurrent.RejectedExecutionException: Task com.wowza.wms.stream.live.LiveStreamDvrRunner@48074402 rejected from com.wowza.wms.vhost.ThreadPoolExecutorReinit@63ec3e7[Shutting down, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 3443884]
ERROR	server	comment	2012-05-17	20:21:52	-	-	-	-	-	2818.226	-	-	-	-	-	-	-	-	execute: java.util.concurrent.RejectedExecutionException: Task com.wowza.wms.stream.live.LiveStreamDvrRunner@3147f58a rejected from com.wowza.wms.vhost.ThreadPoolExecutorReinit@63ec3e7[Shutting down, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 3443884]
ERROR	server	comment	2012-05-17	20:21:52	-	-	-	-	-	2818.226	-	-	-	-	-	-	-	-	execute: java.util.concurrent.RejectedExecutionException: Task com.wowza.wms.stream.live.LiveStreamDvrRunner@556847e4 rejected from com.wowza.wms.vhost.ThreadPoolExecutorReinit@63ec3e7[Shutting down, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 3443884]
ERROR	server	comment	2012-05-17	20:21:52	-	-	-	-	-	2818.225	-	-	-	-	-	-	-	-	execute: java.util.concurrent.RejectedExecutionException: Task com.wowza.wms.stream.live.LiveStreamDvrRunner@780fcd30 rejected from com.wowza.wms.vhost.ThreadPoolExecutorReinit@63ec3e7[Shutting down, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 3443884]
#Version: 1.0

Wrapper setup:

@echo off

set _EXECJAVA=java
set JAVA_OPTS=-server -Xmx4000M

rem If running JDK, uncomment to run server environment (faster)
rem set JAVA_OPTS=%JAVA_OPTS% -server

rem Better garbage collection setting to avoid long pauses
set JAVA_OPTS=%JAVA_OPTS% -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC

rem Uncomment to fix multicast crosstalk problem when streams share multicast port
rem set JAVA_OPTS=%JAVA_OPTS% -Djava.net.preferIPv4Stack=true

rem Uncomment to force Java to use specific language settings
rem set JAVA_OPTS=%JAVA_OPTS% -Duser.language=en -Duser.country=US

set EXECCSCRIPT=cscript
if not exist "%SystemRoot%\SysWOW64\cscript.exe" goto skip64bitcscript
set EXECCSCRIPT="%SystemRoot%\SysWOW64\cscript.exe"
:skip64bitcscript

set WMSCONFIG_URL=
rem set WMSAPP_HOME=C:/wms2
rem set WMSCONFIG_HOME=C:/wms2

set WMSENVOK="false"

rem Guess WMSAPP_HOME if not defined
set CURRENT_DIR=%cd%
if not "%WMSAPP_HOME%" == "" goto gotAppHome
set WMSAPP_HOME=%CURRENT_DIR%
if exist "%WMSAPP_HOME%\bin\startup.bat" goto okAppHome
cd ..
set WMSAPP_HOME=%cd%
cd %CURRENT_DIR%
:gotAppHome
if exist "%WMSAPP_HOME%\bin\startup.bat" goto okAppHome
%EXECSCRIPT% "displaymsg.vbs" "The WMSAPP_HOME environment variable is missing or not defined correctly (%WMSAPP_HOME%). This environment variable is needed to run the server."
goto end
:okAppHome

rem Guess WMSCONFIG_HOME if not defined
set CURRENT_DIR=%cd%
if not "%WMSCONFIG_HOME%" == "" goto gotConfigHome
set WMSCONFIG_HOME=%WMSAPP_HOME%
if exist "%WMSCONFIG_HOME%\conf\Server.license" goto okConfigHome
cd ..
set WMSCONFIG_HOME=%cd%
cd %CURRENT_DIR%
:gotConfigHome
if exist "%WMSCONFIG_HOME%\conf\Server.license" goto okConfigHome
%EXECSCRIPT% "displaymsg.vbs" "The WMSCONFIG_HOME environment variable is missing or not defined correctly (%WMSCONFIG_HOME%). This environment variable is needed to run the server."
goto end
:okConfigHome

%EXECSCRIPT% "checkjavahome.vbs" //Nologo
if not %errorlevel% == 0 goto end

set WMSENVOK="true"
set _EXECJAVA=%JAVA_HOME%\bin\java.exe

:end

and

#********************************************************************
# Wrapper Properties
#********************************************************************
# Java Application
wrapper.java.command=%JAVA_HOME%\bin\java.exe

# Java Main class.  This class must implement the WrapperListener interface
#  or guarantee that the WrapperManager class is initialized.  Helper
#  classes are provided to do this for you.  See the Integration section

#  of the documentation for details.
wrapper.java.mainclass=org.tanukisoftware.wrapper.WrapperStartStopApp

# Java Classpath (include wrapper.jar)  Add class path elements as
#  needed starting from 1
wrapper.java.classpath.1=./wms-bootstrap.jar
wrapper.java.classpath.2=./wrapper.jar

# Java Library Path (location of Wrapper.DLL or libwrapper.so)
wrapper.java.library.path.1=./

# NOTE: Here you can configure the JVM's built in JMX interface.
# See the "Server Management Console and Monitoring" chapter
# of the "User's Guide" for more information on how to configure the
# remote JMX interface in the [install-dir]/conf/Server.xml file.

# Java Additional Parameters
# Make sure the uncommented items are numbered sequentially
wrapper.java.additional.1=-Xmx4000M
wrapper.java.additional.2=-Dcom.wowza.wms.native.base="win"
wrapper.java.additional.3=-Dcom.wowza.wms.runmode="service"
wrapper.java.additional.4=-Dcom.wowza.wms.ConfigURL=
wrapper.java.additional.5=-Dcom.sun.management.jmxremote=true
wrapper.java.additional.6=-Dorg.tanukisoftware.wrapper.WrapperManager.mbean=true
wrapper.java.additional.7=-Dorg.tanukisoftware.wrapper.WrapperManager.mbean.testing=false
wrapper.java.additional.8=-Djava.net.preferIPv4Stack=true
# If running JDK, uncomment to run server environment (faster)
wrapper.java.additional.9=-server
# Uncomment to force Java to use specific language settings
#wrapper.java.additional.10=-Duser.language=en
#wrapper.java.additional.11=-Duser.country=US

# Initial Java Heap Size (in MB)
#wrapper.java.initmemory=3

# Maximum Java Heap Size (in MB)
# Memory setting has been moved above to accomodate larger heap sizes
#[do not use - see wrapper.java.additional.1 property] wrapper.java.maxmemory=768

# Application parameters.  Add parameters as needed starting from 1
wrapper.app.parameter.1=com.wowza.wms.bootstrap.Bootstrap
wrapper.app.parameter.2=1
wrapper.app.parameter.3=start
wrapper.app.parameter.4=com.wowza.wms.bootstrap.Bootstrap
wrapper.app.parameter.5=true
wrapper.app.parameter.6=1
wrapper.app.parameter.7=stop

#********************************************************************
# Wrapper Logging Properties
#********************************************************************
# Format of output for the console.  (See docs for formats)
wrapper.console.format=PM

# Log Level for console output.  (See docs for log levels)
wrapper.console.loglevel=INFO

# Log file to use for wrapper output logging.
wrapper.logfile=../logs/wowzamediaserver_service.log

# Format of output for the log file.  (See docs for formats)
wrapper.logfile.format=LPTM

# Log Level for log file output.  (See docs for log levels)
wrapper.logfile.loglevel=STATUS

# Maximum size that the log file will be allowed to grow to before
#  the log is rolled. Size is specified in bytes.  The default value
#  of 0, disables log rolling.  May abbreviate with the 'k' (kb) or
#  'm' (mb) suffix.  For example: 10m = 10 megabytes.
wrapper.logfile.maxsize=0

# Maximum number of rolled log files which will be allowed before old
#  files are deleted.  The default value of 0 implies no limit.
wrapper.logfile.maxfiles=0

# Log Level for sys/event log output.  (See docs for log levels)
wrapper.syslog.loglevel=NONE

#********************************************************************
# Wrapper Windows Properties
#********************************************************************
# Title to use when running as a console
wrapper.console.title=Wowza Media Server

#********************************************************************
# Wrapper Windows NT/2000/XP Service Properties
#********************************************************************
# WARNING - Do not modify any of these properties when an application
#  using this configuration file has been installed as a service.
#  Please uninstall the service before modifying this section.  The
#  service can then be reinstalled.

# Name of the service
wrapper.ntservice.name=WowzaMediaServer

# Display name of the service
wrapper.ntservice.displayname=Wowza Media Server

# Description of the service
wrapper.ntservice.description=Wowza Media Server

# Service dependencies.  Add dependencies as needed starting from 1
#wrapper.ntservice.dependency.1=

# Mode in which the service is installed.  AUTO_START or DEMAND_START
wrapper.ntservice.starttype=DEMAND_START

# Allow the service to interact with the desktop.
wrapper.ntservice.interactive=true

[randall]

With 4GB of RAM you need to set the heap size to no more than 3GB, leaving 1G for other processes/OS.

Follow the tuning guide, and re-test. If you still have the problem follow the instructions here to submit a support request. Include a link to this thread.

[realmac]

Actually I tried before to decrease heap memory to 3000, but there was imidiate memory panic and streaming service even did not start

[randall]

Wowza should not crash on startup just because the heap is 3000. You'll want to get that working first, since that is probably the issue we're trying to fix. Try rebooting.

I notice you're using high profile on the 480p stream and main profile on the 1080p stream. Shouldn't that be the other way around?

[realmac]

Sure, Wowza service starts, but memory process go mad.
Seems like it try to initialize all streams.
Some streams from playlist starts, some not. All streams experienced huge delay or packet drops.

About profiles. Which profile settings do you mean?
If transcoder templates, we use default 2 files included in latest wowza distribution

[randall]

For H.264 there is a concept of profiles. High is for HD streams, Main is for SD streams. Wherever you are setting it, either in the transcoder template, or on your encoder, is where it should be set correctly.

Moved to ticket #29210.

[realmac]

We located the problem. Actually it is smil playlist and .mp4 file playing , not nDVR related problem.
We discovered that playing certain mp4 files in smil playlists causes overuse of memory, and if there are severel simultanious channels playing "bad" mp4 files by smil playlist server run out of memory.

We try to locate "bad" mp4 patterns.
Files recorded by wowza seems OK. h.264 and AAC

Other files are seems the same, but causing problem. We do not see any huge difference in properties using Gspot anf ffmpeg.
One of the assumptions, that Wowza do not like fragmanted mp4 files.
So we try to re-code potentially bad files using ffmpeg -i badfile.mp4 newfile.mp4 , as result we get smaller file, but still the same "memory panic" when Wowza start to play it

ANy suggestions? Do we need transcode all mp4 before putting in Wowza Playlist. Are there any tools / recommended settings?

[rrlanham]

Take a look at these suggestions

http://www.wowza.com/forums/content....ideo-on-Demand

btw, it's basically the same crew answering forum posts and support tickets.

Richard

[rrlanham]

Also, try this patch:
http://www.wowza.com/downloads/Wowza...1.1-patch8.zip

And be sure you are properly tuned

If still a problem, copy the problem file to a web server and send a link to download to support@wowza.com. Include a link to this thread for reference

Richard