Latency is also introduced into the process on the encoding side. We've found that you can achieve very low latency encoding using the Flash player's built-in encoding (Microphone and Camera). The built-in encoding system in the Flash player uses Sorenson Spark (a variant of H.263) for video and NellyMoser ASAO or Speex for audio. To achieve low latency, it's important that you configure your encoding to produce a reasonable quality at the lowest possible bitrate. The VideoChat example that ships with Wowza Media Server is a good example of how to use the built-in Flash encoder to produce a low latency stream.
Increasing the frame rate and/or decreasing the key frame interval can improve playback time in some cases.