Application shows 50% more connections than the added streams running in it?
From checking the Connection Counts (
http://[IP address]:8086/connectioncounts) I found that as an application is restarted the number listed for the Application is very close to the sum of the connection counts listed for each stream that runs in that application. As time goes by these number diverge drastically and after 15-30 minutes the count for the Application tends to be 50% (to even close to double) of that I get when I add up the totals for all streams that are listed under that Application.
I've checked this mostly on loadbalancer edge, but have also seen it on the loadbalancer origin. Funnily enough when tracking the average data rate per stream from the moment the application is started (where the Application total and added stream totals are close together) to the point where the totals are very different, the average data rate tracks the Application connections number, not the added total of individual streams. This would imply the connections are real connections and streaming, but are not linked to any of the streams running in the Application.
E.g. Just after starting the application (and as it adds connection being sent from the loadbalancer listener) I saw patterns like the following:
Before application restart: Application count: 579, total of stream counts 316 (55%)
Right after application restart: Application count: 25, total of stream counts 23 (92%)
8 minutes after application restart: Application count: 441, total of stream counts 371 (84%)
30 minutes after application restart: Application count: 525, total of stream counts 381 (73%)
15 hours after application restart: Application count: 536, total of stream counts 347 (65%)
Other servers showed a similar pattern.
What would explain the difference between the two numbers? And which one is the actually correct one? Is the difference caused by connections not being properly closed and still streaming for a while despite no longer being associated with a stream?
And, I guess most importantly, is there anything we can/should change in the set up to eliminate this discrepancy or is it part of how the server works? It seems this is reducing the capacity of our servers by 33% which would mean we'd have to run 50% more. If we can change something in the set-up to avoid that, that would of course be very welcome.
What is possibly happening here is that the player is finished playing the stream and has closed the stream but has not disconnected from the server so is still registered as a connection. It could also occur if you are running any custom code that gets a reference to the client session and then doesn't release that reference when the connection is closed. Wowza will still count it until the reference is released.
You should be able to see if the first reason is the cause by analyzing the log files and counting the following events.
connect should equal disconnect for all completed connections.
disconnect should equal destroy (if publishing, you may have connections related to the publishing so you need to subtract these)
If you see substantially more connect than disconnect events and the destroy event count is close to the connect count (once current playing streams are taken into account) then this could indicate the players are not closing the connection properly.
For the second reason, you would need to examine the custom code carefully. If possible, run a controlled test with custom code disabled so you can narrow down where the problem is coming from.