We see a problem on our Origin where memory is being used rapidly while 2 edges are trying to reconnect to an old stream that is defined on the Edges (using a .stream file in the /content/ directory) and that is
automatically started with a StartupStreams.xml file in /conf/ on the Edge.
As a result the Origin server with 16 Gb of memory, of which 8 Gb is reserved with -Xmx for Wowza, is consuming up to 1 Gb per hour after a restart. After about 8 hours the server runs out of memory and basically stops working.
At that time CPU usage also goes up a bit to 50% (out of 800% on a 4-core Intel E3-1270) , while it's normally just at 10%.
We see roughly 15.000 (!) "resetting connection" errors in the logs on BOTH edges since they are desperately trying to connect to the Origin.
While we understand that we can't prevent the Edges from reconnecting to the Origin (they should try that) I do think that the Origin should NOT consume an increasing amount of memory. Something is leaking and it's lethal for the server ; after 8 hours we run out of memory and it crashes hard.
Wowza on Edge and Origin us the same version:
#Software: Wowza Media Server 3.6.3 build8031
Java on Edge and Origin is the same version:
java version "1.7.0_25"
OpenJDK Runtime Environment (IcedTea 2.3.10) (7u25-2.3.10-1~deb7u1)
OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)
This is the startup command-line for Wowza on the Origin:
java -server -Xmx8000M -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/ssd/wowza_heap -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote=true -Dcom.wowza.wms.runmode=service -Dcom.wowza.wms.native.base=linux -Dcom.wowza.wms.AppHome=/usr/local/WowzaMediaServer -Dcom.wowza.wms.ConfigURL= -Dcom.wowza.wms.ConfigHome=/usr/local/WowzaMediaServer -cp /usr/local/WowzaMediaServer/bin/wms-bootstrap.jar com.wowza.wms.bootstrap.Bootstrap start
We'd love to hear some feedback on this issue.