Video Encoding & Codecs

Everything to do with bytes-of-video. For the AV1-specific deep dive see AV1 Encoding. For the streaming protocols carrying these codecs see Streaming Protocols. For player support see Media Players. For grading-side LUTs see Color Grading. For uploaded-file storage see File Uploads.

CLI / encoder tooling

• ★ FFmpeg — LGPL/GPL OSS; the foundation of every video tool that exists. Encode, decode, transcode, mux, demux, filter, stream, capture. If a video task can be automated, FFmpeg can do it. Default for any pipeline.
• ★ HandBrake — GPL OSS; FFmpeg-backed GUI for batch transcoding to MP4/MKV; presets for every device. Default GUI for "shrink this 4K MKV to a phone-sized MP4."
• ★ MKVToolNix (mkvmerge, mkvextract, mkvinfo, MKVToolNix GUI) — GPL OSS; mux, split, append, remux to/from .mkv losslessly. The default container-surgery toolkit.
• MP4Box (GPAC) — LGPL OSS; the MP4 / fMP4 / DASH counterpart to MKVToolNix.
• Bento4 — Apache OSS; mp4info, mp4dump, HLS/DASH packaging utilities.
• Shaka Packager (Google) — see Self-host Streaming Servers; HLS + DASH packager.
• ★ mediainfo — LGPL OSS; CLI + GUI; the metadata inspector. "What's inside this file?" answer in 1 second.
• ffprobe — bundled with FFmpeg; same idea for scripts.
• VLC — GPL OSS; not strictly a tool, but the swiss-army-knife player + transcoder + RTSP server in a pinch.
• MediaCoder — closed source freemium; alternative HandBrake.

Codecs (2026 lineup)

Distribution codecs

• H.264 / AVC — ISO/MPEG; ubiquitous; mandatory floor; royalty-bearing but widely paid through OEMs. Use for max compatibility / Twitch contribution.
• H.265 / HEVC — ISO/MPEG; ~50% better than H.264; royalty-bearing (the patent mess that delayed adoption). Apple-friendly (4K HEIC / iPhone records HEVC). Use for Apple ecosystem deliveries and 4K/HDR where AV1 isn't accepted.
• ★ AV1 — Alliance for Open Media; open / royalty-free; ~50% better than H.264, ~20% better than HEVC; mainstream in 2026: Intel Arc + RTX 4000+ + Apple M3+ all hardware-encode it; YouTube, Netflix, Twitch (enhanced broadcasting), Twitch's video-on-demand all accept AV1. The new default for new-design pipelines. See AV1 Encoding.
• VP9 — Google open codec; YouTube uses heavily; AV1 is replacing it.
• H.266 / VVC — newest ISO codec; ~50% better than HEVC; software encoders only in 2026; not yet practical at scale. Watch.

Mezzanine / intermediate codecs

• ★ ProRes (Apple) — ProRes 422 / 422 HQ / 4444 / RAW; the Mac post standard; FCP/Resolve/Premiere happy. Files are big — that's the point.
• ★ DNxHR / DNxHD (Avid) — cross-platform mezzanine; Avid native; free encoders in FFmpeg.
• CineForm (GoPro) — open OSS; cross-platform; visually lossless intermediate; works in Resolve/Premiere.
• JPEG-XS — broadcast contribution mezzanine; visually lossless, ultra-low-latency; SMPTE 2110 plumbing.
• Uncompressed v210 / DPX — for finishing.

Camera RAW / specialty

• REDCODE / R3D, ARRIRAW, Blackmagic RAW (BRAW), CinemaDNG, ProRes RAW — camera-native; usually flow into the NLE through plugins.

Hardware acceleration (2026)

• ★ NVIDIA NVENC / NVDEC — RTX 4000-series and newer have AV1 encode + 8K HEVC; quality has caught up with x264 medium for streaming. The default for Win / Linux GPU encoding. RTX 5000 series strengthens further. Free with the GPU.
• Intel Quick Sync (QSV) — Arc and 12th-gen+ iGPUs; AV1 hardware encode; great power efficiency; close-second to NVENC.
• Intel VPU / Flex — datacenter Intel video processing units; AV1 / HEVC at scale.
• AMD AMF / VCN — RX 7000+ has AV1 encode; quality improved but still slightly behind NVENC.
• ★ Apple VideoToolbox — M-series chips have ProRes accelerator + HEVC + (M3+) AV1 encode; the Mac default.
• VAAPI — Linux unified video API; wraps Intel/AMD on Linux.

Picking hardware vs software encode

• Live streaming → hardware (NVENC / QSV / VideoToolbox). CPU-encode at 1080p60 is wasteful.
• VOD master encode → software x264/x265/SVT-AV1 at slower presets; hardware for proxies.
• Quality test → software at the same bitrate beats hardware modestly; gap shrinks every year.

Container formats

• MP4 — universal; web, iOS, broadcast deliveries.
• MKV — Matroska; OSS; supports anything; the archive default.
• MOV — Apple's MP4 sibling; ProRes home.
• MXF — broadcast pro; XDCAM, IMX.
• WebM — VP9 / AV1 + Opus / Vorbis; OSS, web-friendly.
• fMP4 / CMAF — fragmented MP4; HLS + DASH unified packaging.
• MPEG-TS (.ts) — broadcast / SRT contribution.

Audio in video files

• AAC — universal companion to H.264/H.265.
• Opus — open; default with VP9/AV1 in WebM.
• AC-3 / E-AC-3 — broadcast surround.
• MP3 — legacy; avoid for new masters.
• PCM — uncompressed; mezzanine.
• See Audio Editors for the audio side.

Useful FFmpeg one-liners

• Trim losslessly: ffmpeg -i in.mp4 -ss 00:00:30 -to 00:01:00 -c copy out.mp4
• Extract audio: ffmpeg -i in.mp4 -vn -c:a copy out.m4a
• Burn-in subtitles: ffmpeg -i in.mp4 -vf subtitles=in.srt out.mp4
• AV1 transcode (SVT-AV1): ffmpeg -i in.mp4 -c:v libsvtav1 -crf 30 -preset 6 -c:a libopus out.webm
• NVENC 1080p60 stream-prep: ffmpeg -i in.mp4 -c:v h264_nvenc -preset p5 -tune hq -b:v 6M -maxrate 6M -bufsize 12M -c:a aac -b:a 160k -f flv rtmp://...
• HLS package: ffmpeg -i in.mp4 -c copy -f hls -hls_time 4 -hls_playlist_type vod out.m3u8

Quality measurement

• VMAF (Netflix) — open source perceptual quality metric; the 2026 standard; FFmpeg has libvmaf filter.
• PSNR / SSIM — older metrics, still useful as quick checks.
• Subjective ABX testing — when bits really matter.

Pick this if…

• Default scriptable encode: FFmpeg.
• Default GUI batch: HandBrake.
• Default .mkv muxing: MKVToolNix.
• Inspect any file: mediainfo or ffprobe.
• Live streaming GPU: NVENC (or VideoToolbox on Mac).
• VOD master, max compression, royalty-free: SVT-AV1 — see AV1 Encoding.
• NLE proxy / mezzanine: ProRes (Mac) or DNxHR (cross-platform).
• Broadcast contribution: H.264 in MPEG-TS over SRT.
• Web playback HLS package: Shaka Packager.