Wowza Community

ModulePushPublish - Unresolved Address Exception

Hey guys!

First off, I love this module. It’s awesome, but last night (or this morning) we experienced something rather nasty.

At approximately 10 minutes to 1AM EST this module started reporting Unresolved Address Exception 500 errors, all over the show. Now this could totally be the edgecast origin DNS being unavailable, but here’s where the weirdness begins, and why I’m posting here…

After trying to reconnect for 30 mins or so, the CPU on our origin server (the one running wowza) spiked to 100% and stayed at 100% until I restarted wowza at around 5AM EST. After which, the module connected up to the edgecast origin DNS and everything was dandy. Needless to say, 4 hours of dead air is pretty bad. :frowning:

Any idea on what could’ve happened here? I can send somebody my logs if they want/need 'em, just lemme know guys.

Thanks!

It sounds like network connectivity problems are the root of the issue.

Was there also OutOfMemory error? Something like what happened here?

Is it a MediaCaster source? What version of Wowza is the origin?

Richard

Hey Richard!

Thanks for the quick reply man :slight_smile:

“It sounds like network connectivity problems are the root of the issue.”

  • Yeah, I’m sure that’s how is started, just fishy that a reboot would save it.

“Was there also OutOfMemory error? Something like what happened here?”

  • I didn’t see one in the logs, no. I think once the CPU maxed out, the logs stopped getting written to. So can’t be 100% that it didn’t happen though.

“Is it a MediaCaster source?”

  • Nope, the streams running on this origin at the time were a combination of Icecast re-streams, Mp3s streamed from S3 as live, and writing directly to rtmp on the origin server. All those streams use the push publisher to send the stream to EdgeCast.

“What version of Wowza is the origin?”

  • Pretty sure it’s 3.5

hey guys!

Just wanted to bump this thread and let you know it happened again, it was during a critical part of our broadcast schedule so I had to quickly restart the server without getting a stack trace or heap dump, unfortunately (boooo, I know). I do however, have logs from that time if it helps at all. The exceptions happen quite frequently it seems, many times a day in fact. I was however, able to confirm that the DNS address is totally up and running when these exceptions occur. It seems that something in the push publisher is crapping out and can’t reconnect.

Any chance you think my logs could help you guys pin the issue? Hopefully I can take a heap dump next time it happens, but would LOVE to get this fixed if possible.

Thanks!

Alrighty dudes, this actually seemed to go away once I increased the Java Heap size. Super stoked about that, but ideas why we wouldn’t be getting issues with connecting to the CDN after increasing memory? Seems like they’re not connected (but what do I know)…

If anyone would like to explain that just for my sanity, that’d be dandy. Thanks!!

hey all!

Wanted to bump this thread to ask a few more questions about issues with the PushPublishing module… I haven’t been able to find any specific information on what this error message means, but if anyone can shed some light on it that’d be great. We had approximately 13 HOURS of downtime due to this issue, had to restart the servers this morning to fix.

[HTML]

NetConnectionConnection.connect: Failed to connect[fso.dca.XXXX.edgecastcdn.net:1935]: org.apache.mina.common.RuntimeIOException: Failed to get the session[ConnectException: null]

[/HTML]

We see the above error constantly in the error logs during the downtime. We also see a bunch of these as well:

[HTML]

PushPublisher.Reconnector: NOT CONNECTING!! RETRYING IN 60 seconds!!

handleResult[3242578]: can’t find resultObj: 1

handleResult[3242580]: can’t find resultObj: 1

handleResult[3242566]: can’t find resultObj: 1

handleResult[3242595]: can’t find resultObj: 1

handleResult[3242596]: can’t find resultObj: 1

handleResult[3242597]: can’t find resultObj: 1

handleResult[3242638]: can’t find resultObj: 1

[/HTML]

Before restarting the server I verified that a TCP connection could be made to our EdgeCast ingest location. Could anybody please tell me what these error messages mean? And also if there is a way we can programmatically fix it without restarting the server?

Thanks guys!

Hi Benny,

If you are using a domain name, as opposed to an ip address in your push publish configuration, that could explain the initial issue if there was a DNS problem. What has probably happened is that the dns server went away so the addresses couldn’t be resolved.

When this occurred, it sounds like Wowza may have gotten into a situation where a thread is looping continuously without sleeping which caused the cpu to spike to 100%. When this occurs, it prevents other threads from getting cpu time so everything grinds to a halt. I the meantime, the dns issue has resolved itself but the server is too busy with the looping thread to notice. When you restart, everything returns to normal.

If it does happen again, please try to get a stack trace and heap dump. This will show us what threads are being locked up. If the thread is holding a lock, it could also prevent other threads from working as they may be waiting to obtain the lock.

Use the instruction here, How to take a Java stack trace to get the stack trace. You may have to use the jstack method as JMX may be unresponsive.

See here, How to take a Java heap dump for heap dump instructions. Again, if JMX is unresponsive or not enabled, you will have to use the command line method.

These will be quite large files so please upload them to somewhere and send a link to support@wowza.com along with a description of the issue and a link to this thread. Please also zip up and include your conf & logs folders.

Roger.

hello Can I get some help with this I feel I have a similar issue:

ERROR server comment - PushPublishRTMP.internalConnect[rtmp://a.rtmp.youtube.com:1935/live2/masked]: : java.nio.channels.UnresolvedAddressException|at sun.nio.ch.Net.checkAddress(Unknown Source)|at sun.nio.ch.SocketChannelImpl.connect(Unknown Source)|at org.apache.mina.transport.socket.nio.SocketConnector.connect(SocketConnector.java:157)|at org.apache.mina.transport.socket.nio.SocketConnector.connect(SocketConnector.java:120) |at com.wowza.wms.netconnection.NetConnectionConnection.connect(NetConnectionConnection.java:138)|

ERROR server comment - PushPublishRTMP.internalConnect[rtmp://a.rtmp.youtube.com:1935/live2/masked]: : java.nio.channels.UnresolvedAddressException|at sun.nio.ch.Net.checkAddress(Unknown Source)|at sun.nio.ch.SocketChannelImpl.connect(Unknown Source)|at org.apache.mina.transport.socket.nio.SocketConnector.connect(SocketConnector.java:157)|at org.apache.mina.transport.socket.nio.SocketConnector.connect(SocketConnector.java:120) |at com.wowza.wms.netconnection.NetConnectionConnection.connect(NetConnectionConnection.java:138)|

once this happens the publishing completly stops and dies until I restart this