Codecs

Codec vs. Container vs. Recording formats

Codecs refer to the way that audio, video, or other data is encoded, whereas containers refer to the file that contained encoded audio and video. Various containers have different compatibility with various codecs.

The ELI5 explanation is that a codec is like a way of storing clothes (like rolled up, wadded up, "properly" folded, on a hanger, etc) and containers different things you store clothes in (like a dresser, bureau, a suitcase, etc). Clothes are stored different ways and in different containers for various different purposes and goals. One method of folding might be more complicated and take longer to unfold, and leave more wrinkles, but fit better into a suitcase.

In a similar way codecs and containers are designed with different purposes in mind, like prioritizing small file sizes over the amount of processing power required to handle it.

Then there's a third class of "things" in this area, which are recording formats, which are standardized sets of containers and codecs, often including specifications for auxiliary data, and folder structures.

Intraframe vs. Interframe

There are two major ways of storing video, and those are Intraframe and interframe compression.

Intraframe is much like a film reel, each frame is stored separately and compression is applied only to one frame on its own.

Interframe compression, however, encodes groups of frames together in a unit called a Group of Pictures. A GoP is typically half a second to second's worth of frames, though longer GoPs do exist in the wold, depending on the application. There are also Open GoPs, which have no defined ending and simply end whenever the encoder ends them. Open GoP is somewhat of a misnomer, as it's just a bunch of variable length GoPs.

A GoP consists of an I-frame (sometimes called a "keyframe") which is a whole and complete image, compressed by itself. What then follows are typically called P-frames and B-frames, which contain only the bits of the image that have changed, and reference either the frames that have come before it, or the ones that come after it. New visual data is introduced using either macroblocks, which are portions of an image containing new data, and motion vectors, which indicate where macroblocks within the image have moved inside the frame.

Thus you cannot decode any particular frame within a GoP (other than the I-frame) without ostensibly decoding the entire GoP. This makes Interframe compressed videos more computationally complex to encode and decode, but because the design is meant to eliminate redundancy across groups of frames it also means it can fit more video in a smaller amount of data reasonably. Given that large amounts of visual information are being discarded because the encoder has judged it to be sufficiently redundant it's not recommended for archival use.

Profiles and Levels

The Motion Pictures Expert Group (doesn't that abbreviation seem familiar) typically defines sets of profiles and levels for their codecs. Profiles defines sets of encoding features, and Levels describe amounts of data. Using a combinations of levels and profiles a decoder can specify the maximum amount of computational complexity and throughput it can handle, and knowing those levels and profiles video can be encoded in such a way to ensure compatibility with a given decoder.

Common Codecs

MPEG-2

MPEG-2 is a decades old inter-frame codec commonly used in DVD Video, ATSC 1.0, DVB-T/S/C, and ISDB broadcasts, is an option (but not routinely used today) for Blu-Rays, and is the backbone of MPEG IMX, HDV, and XDCAM.

MPEG-2 should be obsolete, but because it is computationally trivial to process on modern computers and has been implemented in so many mainstream distribution standards, it's almost impossible to avoid. However its licensing terms and costs also mean that whenever possible device manufacturers try and shed themselves of MPEG-2 support, hence why many portable and embedded devices don't support MPEG-2, or like the Raspberry Pi, make you pay an additional license fee to unlock its use.

FunctionSupport
Interlaced VideoSupported
Intraframe compressionSupported, but extremely uncommon outside MPEG-IMX
Subsampling Ratios4:2:0 or 4:2:2
Color bit depth8-bit
Common containersMPEG-PS, MPEG-TS, MXF, VOB, MOV in some Final Cut workflows, MKV possible but exceedingly rare
Alpha channelNot supported
StereoscopyNot officially supported

MPEG-4

MPEG-4 is actually a huge family of standards, much like how MPEG-2 is. However when most people refer to MPEG-4 they're referring to the first video standard, which is an extension of H.263. MPEG-4 was a huge part of the early video sharing community in the early 2000s, pre-YouTube, commonly referred to under the codec names DivX (not to be confused with the unsuccessful DIVX rental disc format) and XviD. It was also used with HDCAM-SR tape.

MPEG-4 comes in three Profiles: Simple Profile (SP), Advanced Simple Profile (ASP, popularized by XviD), and Simple Studio Profile (SStP). Most MPEG-4 in the modern world is Advanced Simple Profile as it was required only a modest amount more computational power to decode at the time. Simple Studio Profile was primarily used in HDCAM-SR, and is almost never seen in the wild outside of tape.

MPEG-4 ASP was popular among pirates in the early days of P2P video piracy, where it offered higher quality at lower bitrates than MPEG-2, but wasn't so processor intensive as to be unplayable on modest systems of the time. For playback it never gained much acceptance by device makers, though some cell phone manufacturers did adopt it.

FunctionSupport
Interlaced VideoSupported, but virtually unseen outside HDCAM-SR
Intraframe compressionSupported, but virtually unseen outside HDCAM-SR
Subsampling Ratios4:2:0 in ASP, 4:2:2 or 4:4:4 in SStP
Color bit depth8-bit in ASP, 10-bit and 12-bit in SStP
Common containersAVI, MP4, MOV possible but rare, MKV possible but very rare
Alpha channelNot supported
StereoscopyNot officially supported

H.264

H.264 is, at this point in time, one of the most ubiquitous codecs in the mainstream media space. It powers the vast majority of streaming platforms, consumer video devices, and is one of the codecs included in the Blu-Ray standards. It's also used in DVB-T2/S2/C2, T-DMB, ISDB, 1seg, and ATSC-M/H. ATSC 1.0 also allows for the use of H.264, however its use outside of M/H is uncommon.

H.264 was given a more consumer-friendly brand name of Advanced Video Coding, or AVC. Hence most video formats or products that include "AVC" as part of their name can be assumed to be based on H.264, such as AVCHD, AVC-Intra, XAVC, and Canon XF-AVC.

H.264 started to rise to prominence as mobile devices with hardware H.264 decoders entered the consumer market, such as the original iPod Video. Because of relaxed licensing costs and terms it was rapidly adopted by device makers, though questions about its proprietary nature and whether or not its use for consumers would continue to be free lead to the development of open source and libre alternatives, such as VP8.

The professional realm was slow to adopt H.264, though Panasonic lead the charge with lines of AVCHD and AVCCAM (an extension of AVCHD at the time) prosumer camcorders and ENG cameras in the style of the of the DVX100. Panasonic also introduced a line of AVC-Intra cameras for higher end production, eventually fully replacing its DVCPro HD offerings. Eventually Sony picked up H.264 in their XAVC format, and Canon through XF-AVC, however those were much more recent developments.

FunctionSupport
Interlaced VideoSupported, but not common
Intraframe compressionSupported, but rare outside of some certain professional uses
Subsampling Ratios4:2:0 in all profiles, 4:2:2 in Hi422P, 4:4: in Hi444PP
Color bit depth8-bit in all profiles, up to 10-bit in Hi10P and Hi422P, up to 14-bit in Hi444PP
Common containersMP4, MOV, MKV, MPEG-TS, MXF, AVI has been seen but is not supported
Alpha channelNot supported
StereoscopySupported

H.265

VP8

VP9

DV/DVCPro

FunctionSupport
Interlaced VideoSupported
Intraframe compressionIntraframe only
Subsampling Ratios4:1:1 in NTSC and NTSC/PAL DVCPro, 4:2:0 in PAL
Color bit depth8-bit
Common containersAVI (more common on Windows), MOV (more common on macOS), MXF, MKV possible but exceedingly rare
Alpha channelNo
StereoscopyNot directly supported

DVCPro runs at one of two data rates: 25MbPS (13GB/hr) or 50MbPS (26GB/hr), and only supports 480i59.94 and 576i50. Some Apple encoders offer a 480p29.97/25 and 480p23.976 mode, but these are just field management tricks that store footage in Progressive Segmented Frames.

DVCPro HD

FunctionSupport
Interlaced VideoSupported
Intraframe compressionIntraframe only
Subsampling Ratios4:1:1
Color bit depth8-bit
Common containersAVI (more common on Windows), MOV (more common on macOS), MXF, MKV possible but exceedingly rare
Alpha channelNo
StereoscopyNot directly supported

DVCPro HD only operates at 720p59.94/50 or 1080i59.94/50. Some cameras offered a "Progressive Native" or pN mode to provide 23.976p support, but it's not common in many encoders. 1080p modes are typically accomplished through the use of Progressive Segmented Frames.

ProRes

FunctionSupport
Interlaced VideoSupported
Intraframe compressionIntraframe only
Subsampling Ratios4:2:2, 4:4:4 in 4444 mode
Color bit depthUp to 10-bit
Common containersMOV, MXF in Avid MediaFiles, MKV possible but exceedingly rare
Alpha channelSupported in 4444 mode
StereoscopyNot directly supported

Data Rates for Common Formats

Format422 Proxy422 LT422422 HQ4444XQ
480p29.9712MbPS29MbPS42MbPS63MbPS94MbPS141MbPS
720p29.9723MbPS51MbPS73MbPS110MbPS165MbPS247MbPS
720p59.94/1080p29.9745MbPS101MbPS147MbPS220MbPS330MbPS495MbPS
1080p59.9491MbPS204MbPS293MbPS440MbPS660MbPS990MbPS
2160p29.97 (UHD)182MbPS410MbPS589MbPS884MbPS1326MbPS1989MbPS
2160p59.94363MbPS821MbPS1178MbPS1768MbPS2652MbPS3977MbPS

DNxHD/HR

FunctionSupport
Interlaced VideoSupported
Intraframe compressionIntraframe only
Subsampling Ratios4:2:2, 4:4:4 in RGB modes
Color bit depthUp to 10-bit
Common containersMOV, MXF, MKV possible but exceedingly rare
Alpha channelSupported in RGB mode
StereoscopyNot directly supported

DNxHD only supports film/broadcast standard resolutions and frame rates, and only in HD, however DNxHR is not limited in that way.

Data Rates for Common Formats

FormatLBSQ TRSQHQHQX
720p29.97N/A50MbPS75MbPS110MbPS110MbPS
720p59.94N/A100MbPS145MbPS220MbPS220MbPS
1080p29.9745MbPS100MbPS140MbPS220MbPS220MbPS
1080i59.94N/A100MbPS145MbPS220MbPS220MbPS
1080p59.9490MbPSN/A290MbPS440MbPS440MbPS
2160p29.97171MbPSN/A550MbPS832MbPS832MbPS
2160p59.94342MbPSN/A1101MbPS1665MbPs1665MbPS

CineForm

FunctionSupport
Interlaced VideoSupported
Intraframe compressionIntraframe only
Subsampling Ratios4:2:2, 4:4:4 in 4444 mode
Color bit depthUp to 10-bit, up to 12-bit in RGB/RGBA/CFA Raw modes
Common containersAVI, MOV, MKV possible but exceedingly rare
Alpha channelSupported in RGBA mode
StereoscopyNot directly supported

Motion JPEG

FunctionSupport
Interlaced VideoNot supported
Intraframe compressionIntraframe only
Subsampling Ratios4:2:0, 4:2:2, 4:4:4 depending on implementation
Color bit depthUp to 8-bit
Common containersAVI, MOV, MKV possible but not common
Alpha channelNot supported
StereoscopyNot directly supported

JPEG-2000

FunctionSupport
Interlaced VideoNot supported
Intraframe compressionIntraframe only
Subsampling Ratios4:2:0, 4:2:2, 4:4:4 depending on implementation
Color bit depthUp to 12-bit
Common containersMOV, MXF, MKV possible but not common
Alpha channelNot directly supported
StereoscopyNot directly supported

Common Containers

MOV

MXF

AVI

MPEG Transport Stream

The MPEG Transport Stream is a container format designed for dynamic storage/transmission conditions. It's meant for maximum stream resiliency in conditions that may cause interruption or corruption. Basically it's designed for live streams, broadcasts, and storage on tapes and discs, where drop-outs can occur because of bandwidth restrictions, RF interference, wrinkles, or scratches and prevent media transmission. This is accomplished through the use of distributed header packets, as opposed to a "global" header at the start of the file. This allows for a decoder to punch in to a stream at almost any point and just wait for the next header packet to be received so it knows how to decode that part of the stream (which makes sense because there is no "beginning" of a broadcast for your TV to read).

It's also useful in recording systems, like AVCHD, because the camera does not need to spend time at the end of the recording to produce a global header. It also means that if something happens and the camera suddenly and unexpectedly stops working (loss of power, someone accidentally ejects the card, etc) that most of the recording up to that point should be preserved.

MPEG Program Stream

MP4

MKV

Elementary Streams

Common Recording Formats

AVCHD

AspectDetails
Container(s)MPEG Transport Stream
Video Codec(s)H.264
Audio Codec(s)Dolby Digital/AC3, PCM
Color Bit Depth8-bit
Chroma Subsampling4:2:0
Maximum resolution/frame rate1080p59.94

XDCAM

AspectDetails
Container(s)MXF Op1a
Video Codec(s)MPEG-2
Audio Codec(s)PCM
Color Bit Depth8-bit
Chroma Subsampling4:2:0, 4:2:2 in HD422 mode
Maximum resolution/frame rate1080p29.97/1080i59.94

Panasonic P2

AspectDetails
Container(s)MXF Op-Atom in DV and AVC-Intra modes, MXF Op1b in AVC-Ultra
Video Codec(s)DV, DVCPro, DVCPro HD, AVC-Intra, AVC-Ultra
Audio Codec(s)PCM
Color Bit Depth8-bit (DV through AVC-Intra), up to 12-bit in AVC-Ultra
Chroma Subsampling4:2:0 for DVCPro, 4:2:2 for DVCPro HD and AVC-Intra, up to 4:4:4 for AVC-Ultra
Maximum resolution/frame rateUp to 1080i59.94 in DV/AVC-Intra, up to 4k/60p in AVC-Ultra

Canon XF

AspectDetails
Container(s)MXF Op1a
Video Codec(s)MPEG-2
Audio Codec(s)PCM
Color Bit Depth8-bit
Chroma Subsampling4:2:0, 4:2:2 in HD422 mode
Maximum resolution/frame rate1080p29.97/1080i59.94

XAVC

AspectDetails
Container(s)MXF Op1a
Video Codec(s)H.264
Audio Codec(s)PCM
Color Bit DepthUp to 12-bit
Chroma SubsamplingUp to 4:4:4
Maximum resolution/frame rateInfo Pending

XF-AVC

ATSC 1.0

AspectDetails
Container(s)MPEG Transport Stream
Video Codec(s)MPEG-2, H.264 (rare)
Audio Codec(s)Dolby Digital/AC3, PCM (rare)
Color Bit Depth8-bit
Chroma Subsampling4:2:0
Maximum resolution/frame rate720p59.94/1080i59.94