The Complete Guide to Live Streaming:
Video Encoding and Codecs

The second step in any live-streaming workflow is video encoding. After capturing the video with your camera(s) of choice — be it a production-quality setup, IP camera, or your end-users’ mobile devices — live video data must be digitized for efficient transportation across the internet. Video encoding is essential to live streaming, helping to ensure quick delivery and playback.

What Is Encoding?

 

Video encoding refers to the process of converting raw video into a digital format that’s compatible with many devices. Videos are often reduced from gigabytes of data down to megabytes of data. This process involves a two-part compression tool called a codec.

What Is a Codec?

 

Literally ‘coder-decoder’ or ‘compressor-decompressor,’ codecs apply algorithms to tightly compress a bulky video for delivery. The video is shrunk down for storage and transmission, and later decompressed for viewing.

 

When it comes to streaming, codecs employ lossy compression by discarding unnecessary data to create a smaller file. Two separate compression processes take place: video and audio. Video codecs act upon the visual data, whereas audio codecs act upon the recorded sound.

 

H.264, also known as AVC (Advanced Video Coding), is the most common video codec. AAC (Advanced Audio Coding) is the most common audio codec.

What Video Codec Should You Use?

 

Streaming to a variety of devices starts with supporting a variety of codecs. But to keep the encoding part of the workflow simple and fast, you can always transcode streams later when they’re ingested by the media server.

 

While industry leaders continue to refine and develop the latest compression tools, many content distributors employ older video codecs like H.264/AVC for delivery to legacy devices. H.264 is your best bet for maximizing compatibility, even though other video codecs are more technologically advanced.

 

Below are some of the most common video codecs in use today.

Video Codec Benefits Limitations
H.264/AVC Widely supported. Not the most cutting-edge compression technology.
H.265/HEVC Supports 8K resolution. Takes up to 4x longer to encode than H.264.
VP9 Royalty-free. Already made obsolete by AV1.
AV1 Open-source and very advanced. Not yet supported on a large scale.
VVC Intended to improve upon H.265. Same royalty issues as H.265.

Video Codecs Worth Noting:

H.264/AVC

The majority of encoding output today takes the form of H.264 files, also referred to as AVC (Advanced Video Coding). This widely supported codec is often incorporated with the AAC audio codec and can be packaged into .mp4, .mov, .F4V, .3GP, and .ts containers. H.264 plays on just about any device, delivers quality video streams, and comes with the least concerns surrounding royalties.

 

Required Upload Speed for 1080p60 Stream (Mbps)

H.265/HEVC

Developed as the successor to H.264, this codec supports 8K resolution. It also generates smaller files than H.264, thus decreasing the bandwidth required to view high-definition streams.

 

Uncertainties surrounding royalties initially stifled adoption, but H.265 eliminated royalty fees in 2018. Even so, the codec only accounts for about 10 percent of encoded live streaming files today.

VP9

A royalty-free alternative to H.265, VP9 offers even better quality at the same bitrate. A comprehensive list of devices, browsers, and platforms support VP9 — but Apple unfortunately does not.

What Audio Codec Should You Use?

 

AAC takes the cake when balancing quality with compatibility across audio codecs. While open-source alternatives like Opus far outperform AAC, they lack support across as many platforms and devices.

Audio Codec Benefits Limitations
AAC Most common audio codec. Higher-quality alternatives exist.
MP3 Also widely supported. Less advanced than AAC.
Opus Highest-quality lossy audio format. Yet to be widely adopted.
Vorbis Non-proprietary alternative to AAC. Less advanced than Opus.
Speex Patent-free speech codec. Also obsoleted by Opus.

Audio Codecs Worth Noting:

AAC

Defined by MPEG-4, this widely supported standard is used by YouTube, Android, iOS, and iTunes. Two extensions of AAC exist: HE-ACC for low bitrates and AAC-LC for low delay. HE-AAC works best whenever bandwidth is a concern, while AAC-LC lends itself well to two-way communication.

MP3

Nearly every audio-supported digital device in the world can play back the MP3 format, making it a viable option for live streaming. But because AAC offers superior compression, we’d recommend going with that.

Opus

Developed by the Xiph.Org Foundation, Opus provides higher-quality audio than any other lossy audio format. It’s open-source and royalty-free, but has yet to be widely adopted.

Encoding Best Practices

 

Encoding best practices go way beyond what codec you select. You’ll also want to consider frame rate, keyframe interval, and bitrate.

 

Luckily, a live stream can always be transcoded into another format once it reaches the server. This can be done using a software and your own servers, or in the cloud for professionally managed delivery.

 

In addition to the codecs used, it’s important to examine how your stream is packaged and which protocols enter the picture. Keep reading to learn more.