How to reduce visual artifacts in transcoded streams when using NVIDIA NVENC accelerated encoding

When using NVIDIA NVENC hardware with older graphics drivers in your transcoding setup, you may notice that visual artifacts or reduced quality may appear every few seconds in the transcoded stream frames. This may be more noticeable on lower bitrate output renditions. This occurs because the default settings for the NVENC encoder constrain the keyframe bitrate so that the keyframe quality is reduced.

Updating the graphics driver for your NVIDIA hardware to the latest version or raising the bitrate settings of the video encoding presets in your transcoder template often resolves the problem. If that doesn't work, you can use the following workaround to reduce the effect and improve the stream quality. The workaround disables the maximum bitrate setting so that the encoder isn't constrained when creating keyframes.

  • Important: NVIDIA has addressed this issue in the latest graphics driver for NVIDIA NVENC hardware.
  • This workaround is for Wowza Streaming Engine™ 4.0.3 and later. A bug in previous versions of the server software prevents this workaround from being used.
  • The parameter settings described in the workaround can't be configured by using Wowza Streaming Engine Manager. You must edit the transcoder template files in a text editor.
  1. Open your transcoder template (for example, [install-dir]/transcoder/templates/[your-template-name].xml) in a text editor.
  2. In each encode block that's enabled for NVENC accelerated encoding, add the nvenc.peakBitRate parameter to the <Video>/<Parameters> section.
    Note: Don't add this property to the <Audio>/<Parameters> section.
  3. Save the updated template file and restart the source stream. You don't need to restart the server but the change won't take effect until the source stream is restarted.
This setting may increase the overall bitrate of the transcoded streams. You can reduce the bitrate setting in the template so that the overall bitrate is lower. The default setting for the peak bitrate is the bitrate setting + 10 percent. You can reduce the bitrate setting by 10 percent to bring the overall stream bitrate back to the expected range.
Originally Published: 03-27-2014.
Updated: 03-23-2015.

If you're having problems or want to discuss this article, post in our forum.