I encountered the error sporadically on Wowza server. I noticed this error keeps repeating till I restart the server and blocks other clients trying to connect . Are there any specific cases when this will occur?
The error happens because of long GC pauses that cause transmission problems. So the error is due to the TCP connection becoming out of sync. Crank up the heap size. 768M is way, way, way to low for 600Mbps of transmission. Go with 64-bit OS and VM and move to 3000-4000M heap.