Our stream shows no delay when we fire up our stream, but after several hours we will notice 30-60 second delay from player vs actuall stream from Adobe. Starting a new player to stream still shows delay.
The load balancer most likely has nothing to do with the delay. I would focus on the encoder. These delays are usually encoder issues where the encoder can't send the stream to the server fast enough so it hold video/audio data. What encoder are you using? If FMLE then there are option to have it drop frames to try and keep up.