Seeing some weird stuff with using the transcoder regarding quality at lower bitrates, similar to what the OP was experiencing, except we don't have any CPU spikes or lines in the logs that suggest an issue with either the source stream, hardware, or transcoder itself.
Some background info:
We're sending a very small stream (640 x 360) at 3 different bitrates: 1200, 500, 200. Once the audio is added they become 1392, 692, and 392. The profiles for the 692 and 392 bitrates are passthrough for framesize, keyframe interval, audio codec, etc. We're only adjusting the bitrate itself down from the input source at 1392 (1200 + 192).
At some points, usually when there is a lot of movement on the source stream, the quality of the lower bitrates gets really, really poor. Very blocky, but no stuttering or jerky movements. Only very pixelation and visual distortion, the audio however, is fine, no issues.
Video is h264 w/ AAC audio, 640 x 360 - Main profile 4.1. Serving to desktop clients, not mobile devices.
Any insight? Lemme know if you would like / need more info, I'd be happy to provide it! :o