PATCH: LoadBalancerListener not found randomly on startup

[freastro]

Occasionally, when we restart our Wowza server, the ModuleLoadBalancerRedirector instances will load before the ServerListenerLoadBalancerListener instance. Here's my understanding of what's happening:
1) Wowza binds to port 1935.
2) Client connects to Wowza using the application redirect_175.41.136.177/_definst_.
3) Wowza initializes the application and creates an instance of ModuleLoadBalancerRedirector.
4) Since no ServerListenerLoadBalancerListener has been created yet, the ModuleLoastBalancerRedirector instance initializes its listener variable to null.
5) The client receives an error message and disconnects.
6) The ServerListenerLoadBalancerListener instance is created.
7) Wowza reports that the server is started.

I've selected a few lines from our log file to show exactly what's happening:

2011-10-18	00:00:28	EDT	vhost-start	vhost	INFO	200	_defaultVHost_	-	-	-	-	4.787	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2011-10-18	00:00:29	EDT	comment	server	WARN	200	-	HTTPLoadBalancerRedirector.constructor: LoadBalancerListener not found.	-	-	-	5.875	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2011-10-18	00:00:30	EDT	comment	server	WARN	200	-	HTTPLoadBalancerRedirector.constructor: LoadBalancerListener not found.	-	-	-	5.984	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2011-10-18	00:00:30	EDT	comment	server	WARN	200	-	ModuleLoadBalancerRedirector.onAppStart[redirect_175.41.136.177/_definst_]: LoadBalancerListener not found. All connections to this application will be refused.	-	-	-	6.316	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2011-10-18	00:00:30	EDT	comment	server	INFO	200	-	ModuleLoadBalancerRedirector.onAppStart: redirect_175.41.136.177/_definst_	-	-	-	6.317	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2011-10-18	00:00:30	EDT	app-start	application	INFO	200	_definst_	redirect_175.41.136.177/_definst_	-	-	-	6.42	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	
2011-10-18	00:00:30	EDT	connect-pending	session	INFO	100	71.227.178.240	-	_defaultVHost_	redirect_175.41.136.177	_definst_	0.961	[any]	443	rtmp://wowza-lb.dyyno.com:443/redirect_175.41.136.177/	71.227.178.240	rtmp	http://www.dyyno.com/assets/players/dyynoplayer-17419.swf	WIN 10,3,183,5	214170813	3518	3073	-	-	-	-	-	-	-	-	-	-	-	-	-	rtmp://wowza-lb.dyyno.com:443/redirect_175.41.136.177/	-
2011-10-18	00:00:30	EDT	connect	session	INFO	401	71.227.178.240	-	_defaultVHost_	redirect_175.41.136.177	_definst_	1.018	[any]	443	rtmp://wowza-lb.dyyno.com:443/redirect_175.41.136.177/	71.227.178.240	rtmp	http://www.dyyno.com/assets/players/dyynoplayer-17419.swf	WIN 10,3,183,5	214170813	3518	3073	-	-	-	-	-	-	-	-	-	-	-	-	-	rtmp://wowza-lb.dyyno.com:443/redirect_175.41.136.177/	-
2011-10-18	00:00:30	EDT	comment	server	INFO	200	-	ServerListenerLoadBalancerListener.onServerInit	-	-	-	6.672	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2011-10-18	00:00:30	EDT	comment	server	INFO	200	-	LoadBalancerListener.bind: 0.0.0.0/0.0.0.0:1934	-	-	-	6.7	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2011-10-18	00:00:31	EDT	comment	server	INFO	200	-	Wowza Media Server is started!	-	-	-	7.757	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-

I've noticed that HTTPLoadBalancerRedirector successfully recovers in this case, so I patched ModuleLoadBalancerRedirector.java to mirror what it's doing. Basically, the fix is to re-initialize the listener variable on every request if it's null. I haven't tried compiling this code or testing it because I assume whoever looks at it will have their own set of changes and tests they want to make.

30a31,56
>     
>     private ILoadBalancerRedirector getRedirector(IClient client)
> 	{
> 		if (this.redirector == null)
> 		{
> 			while(true)
> 			{
> 				this.listener = (LoadBalancerListener)Server.getInstance().getProperties().get(ServerListenerLoadBalancerListener.PROP_LOADBALANCERLISTENER);
> 				if (this.listener == null)
> 				{
> 					getLogger().warn("ModuleLoadBalancerRedirector.onAppStart["+getFullName(client)+"]: LoadBalancerListener not found. All connections to this application will be refused.");
>                     break;
> 				}
> 				
> 				this.redirector = this.listener.getRedirector();
> 				if (this.redirector == null)
> 				{
> 					getLogger().warn("ModuleLoadBalancerRedirector.onAppStart["+getFullName(client)+"]: ILoadBalancerRedirector not found. All connections to this application will be refused.");
> 					break;
> 				}
> 				break;
> 			}
> 		}
> 		
> 		return this.redirector;
> 	}
40a67,68
>         
>         getRedirector(appInstance);
42,58d69
< 		while(true)
< 		{
< 			this.listener = (LoadBalancerListener)Server.getInstance().getProperties().get(ServerListenerLoadBalancerListener.PROP_LOADBALANCERLISTENER);
< 			if (this.listener == null)
< 			{
< 				getLogger().warn("ModuleLoadBalancerRedirector.onAppStart["+getFullName(appInstance)+"]: LoadBalancerListener not found. All connections to this application will be refused.");
< 				break;
< 			}
< 			
< 			this.redirector = this.listener.getRedirector();
< 			if (this.redirector == null)
< 			{
< 				getLogger().warn("ModuleLoadBalancerRedirector.onAppStart["+getFullName(appInstance)+"]: ILoadBalancerRedirector not found. All connections to this application will be refused.");
< 				break;
< 			}
< 			break;
< 		}
69a81,82
>         
>         getRedirector();
101a115
>             getRedirector();

Please let me know if you have any questions. Thanks!
-Greg

[greenf]

This bug was reported 4 month ago and still not fixed by Wowza. We have same issue with Wowza 3.0.3 for AmazonEC2. Morever, the problem is more serious, then was written in the post. Since, ModuleLoadBalancerRedirector read configuration on startup, once initialized to null redirect property remains null untill next restart. So redirect application does not work at all and not only at startup.

P.S. I don't think your patch will work for several reasons:
1. In your code 'if(listener==null) { break;}'. This will break the loop, even if listener is not loaded yet. Change from '==' to '!=' will not help, since if listener is not loader, due to any exception, it will never loaded, until restart and your loo never end.
2. In addition, I don't want to edit load balancer module jar, so I need to extend module class and then I can't use redirector and listener properties of super class since it are private.

My suggestion is:

public class ModuleLoadBalancerRedirector extends
		com.wowza.wms.plugin.loadbalancer.ModuleLoadBalancerRedirector {
	
	public String getFullName( IApplicationInstance iapplicationinstance ) {
            return (new StringBuilder()).append(iapplicationinstance.getApplication().getName()).append("/").append(iapplicationinstance.getName()).toString();
        }
	
	public void onAppStart( IApplicationInstance iapplicationinstance ) {
		long timeout = iapplicationinstance.getProperties().getPropertyInt("redirectListenerLoadTimeout", 10000 );
		long ts = Calendar.getInstance().getTimeInMillis(), newTs;
		ILoadBalancerRedirector redirector = null;
		do {
			LoadBalancerListener listener = (LoadBalancerListener)Server.getInstance().getProperties().get("WowzaProLoadBalancerListener");			
    	                if( listener == null ) {
	                    getLogger().warn((new StringBuilder()).append("ModuleLoadBalancerRedirector.onAppStart[").append(getFullName(iapplicationinstance)).append("]: LoadBalancerListener not found. All connections to this application will be refused.").toString());
	                } else {
	                   redirector = listener.getRedirector();
	                }
	               newTs = Calendar.getInstance().getTimeInMillis();
		} while( redirector == null && newTs - ts < timeout );
		if( redirector == null ) {
			getLogger().warn((new StringBuilder()).append("ModuleLoadBalancerRedirector.onAppStart[").append(getFullName(iapplicationinstance)).append("]: Timeout expired and LoadBalancerRedirector still not found.").toString());
		}        
               super.onAppStart( iapplicationinstance );
          }
}

[rrlanham]

I missed the first post, somehow. Sorry about that. Maybe I read it as a problem solved.

Are you using StartUpStreams.xml? I'm wondering how the App module runs before the server listener. StartUpStreams.xml might explain it. Can you tell me how to replicate this consistently?

Have you tried with 3.0.4 AMI
http://www.wowza.com/forums/content....achine-images)

3.0.5 will AMIs will be released soon.

Richard

[rrlanham]

We can see how this might happen, perhaps because of StartUpStreams.xml or other ways possibly, and are going to look into it.

Thanks for the report,
Richard

[greenf]

- StartUpstreams? My StartUpstreams.xml is empty.
- App runs before listener? When I set loglevel to debug I see that listener starts before module, but it run in parallel, in separate threads, I guess and the issue occurs in onAppStart().
- Bug replication? I can see in production logs, that issue happened, when request to redirect app received before init() method of listener has been completed. It is hard to reproduce it in Dev, but I think you can do it with patch that delays listener before it set WowzaProLoadBalancer property or you have a load test tool that can do rtmp request to redirect every 0.1 second.
- AMI Version? WowzaMediaServer-devpay-3.0.3.07-i386 (ami-93824afa). I don't see it in change log that it was fixed in 3.0.04.

[rrlanham]

I will post back to this thread when I have more info

Thanks,
Richard

[rrlanham]

Re-download the addon, the problem should be fixed:

http://www.wowzamedia.com/downloads/...lancer_2.0.zip

Richard

[greenf]

Thanks. I'll try it in next release, and will report results.