"Address already in use"

[repster]

Running Ubuntu 9.10.
Wowza Media Server 2, patch 9
Java 1.6.0_15

I have had this happen twice. I run /etc/init.d/WowzaMediaServer stop, followed by start and the server doesn't come back up.

When I look in the logfile, it complains about a bind exception with Address already in use. When I check netstat -atp and find that the wowza ports (8083, 8084, 8085, 8086 1935) are all listening but have no process number or name associated with them. A few of them even have active connections with data in the Recv-Q. If I telnet to 1935 it accepts the connection but there is no response. When I check ps, I find a single zombie java process (defunct).

I am not sure how you (or java or linux) have managed to get in that state, but I think it is against the rules. The only thing that seems to fix it is a reboot.

Any ideas?

[Dobrushin]

Please email the /logs and /conf folders to support@wowzamedia.com and reference this post. Do you have any other software on the machine that may be using the same ports?

[charlie]

When you run /etc/init.d/WowzaMediaServer stop you must wait until it reports that the process has been killed. This could take up to a minute. The WowzaMediaServer init.d script will kill the process if for some reason it is not responding quickly enough. If you control-C the shutdown script you could get into the state you have described. If it happens again you should be able to kill -9 the java process.

Charlie

[repster]

Dobrushin: Log and conf files have been sent.

Charlie: If you are asking if I issued a ctrl-c to the init.d script, the answer is no. It finished, and I did about 5 minutes of other work before issuing the start. The sockets are not sitting around in time-wait, they are still considered active by the kernel (listen or connected state). I can even connect to 1935 using telnet though I cannot interact once I am connected. Running 'netstat -atp' as root shows a '-' as the process owning the socket, which AFAIK should only be possible for sockets that are in TIME_WAIT.

[charlie]

It is very strange. Does this happen to you everytime? The init.d script will kill -9 the Java process if it does not stop on it's own. I am suprised that the process hangs around. It sounds like a Java VM issue.

I would suggest trying the OpenJDK for you plaform. We are hearing that the OpenJDK runs quite well (if not better) than the Sun VM on many Linux platforms including Ubuntu.

Charlie

[repster]

It is very strange. Does this happen to you everytime? The init.d script will kill -9 the Java process if it does not stop on it's own. I am suprised that the process hangs around. It sounds like a Java VM issue.

I would suggest trying the OpenJDK for you plaform. We are hearing that the OpenJDK runs quite well (if not better) than the Sun VM on many Linux platforms including Ubuntu.

Charlie

No, it has only happened twice and I was really puzzled the first time but I had to get the server up fast so I simply rebooted it. The second time I figured I would let you know about it.

I agree that this is really not your problem. I am pretty sure that this is not "legal" behavior in the sense that the linux kernel should not be able to get in a state where a zombie process holds active sockets. So, I think it is more likely to be a kernel bug than a java bug.

You might be able to work around it by trying a "more polite" shutdown first (catch kill -HUP and close sockets) before resorting to -9 if the polite version failed.