Encoding transparent WebM with WebCodecs when alpha isn’t supported

Chrome’s WebCodecs VideoEncoder will not encode an alpha channel. Ask for alpha: 'keep' and it reports the configuration as unsupported, whatever the codec. A WebM file can still carry transparency. Encode the colour and the alpha as two ordinary VP9 streams, store each alpha frame in its colour frame’s BlockAdditions under BlockAddID 1, and set AlphaMode to 1 on the video track. Chrome and Firefox both play VP9 with alpha in WebM, and nothing in the pipeline needs ffmpeg.

We built this for an AI study platform. Its animated character runs as live SVG in the app and also ships as exported video clips. One pose model drives both, so the clips match the live character. The export writes seven clips at 640×640 and 30 fps, from 186 KB to 1.08 MB each. Safari can’t show VP9 alpha, so Safari users get the live SVG.

This post covers the four approaches that failed first, how WebM stores alpha, and short code for the alpha frames, the two encoders and the block layout. We checked every element ID against RFC 9559, the Matroska specification that WebM is built on.

Four approaches that failed first

We tried four things before writing a muxer. Each failure has a cause you can confirm on your own stack in a few minutes.

ffmpeg. The build we had bundled could only encode VP8. A full build with libvpx can write VP9 with alpha. Its encoder wrapper runs a second libvpx encoder for the alpha plane and attaches each result to its packet as BlockAdditional data. The rest of this post does the same job inside the browser.

WebCodecs with alpha. VideoEncoderConfig has an alpha member, and the WebCodecs spec says 'keep' should preserve alpha when the frames have it. Chrome’s implementation refuses before it looks at the codec. Its config parser returns the message “Alpha encoding is not currently supported”, and a TODO beside that code points at crbug.com/1195433 (Chromium source). In a Chromium 152 build, VideoEncoder.isConfigSupported() still returns supported: false for alpha: 'keep', and true for the same VP9 config without it.

MediaRecorder. A canvas capture track adds a frame only when the canvas is painted with new content. Browsers skip hidden documents when they update the rendering, and requestAnimationFrame pauses in background tabs. Our export page was hidden, so its canvas was never painted and the recorder received no frames. WebCodecs has no such dependency, because your code creates each frame and sets its timestamp.

webm-muxer. The package is built around WebCodecs output, but its video path has no way to take alpha frames. In version 5.1.4 the alpha option only writes the AlphaMode flag on the track. addVideoChunk() accepts no alpha data, and video frames go out as SimpleBlocks without BlockAdditions (source). The project is now deprecated in favour of Mediabunny, by the same author. Mediabunny’s source (we read version 1.46.0) splits alpha into a second encoder much as this post does, so try it before you write a muxer of your own.

How WebM stores an alpha channel

WebM carries transparency as a second VP9 stream. Each alpha frame is an ordinary VP9 frame whose luma plane holds the alpha values, and it travels in the same block as the colour frame it belongs to. FFmpeg’s wrapper builds exactly this, with the alpha copied into the Y plane and both chroma planes filled with 0x80.

In the container, each frame is written as a BlockGroup instead of the more common SimpleBlock, because only a BlockGroup can hold extra elements. The alpha frame goes in the BlockGroup’s BlockAdditions, inside a BlockMore whose BlockAddID is 1. RFC 9559 defines ID 1 as data whose meaning comes from the codec, and the track’s AlphaMode element, set to 1, tells the player that the data is alpha.

The IDs are from RFC 9559, and the EBML header elements in the full file are from RFC 8794. MaxBlockAdditionID defaults to 0, which the RFC defines as the track having no BlockAdditions at all. Chromium plays files without it, but FFmpeg’s muxer writes it when the output is seekable, and so does the sketch below. AlphaMode was added in Matroska version 3. Section 7 of the RFC requires the header’s DocTypeVersion to be at least the highest version of any element in the file, so it must be 3 or more.

Turning the canvas alpha into a VP9 frame

The alpha encoder needs a normal video frame. Build an I420 frame by hand and put the alpha bytes in its luma (Y) plane. I420 is a full-size Y plane followed by U and V planes at half the width and half the height. Fill U and V with 128, the neutral chroma value, and the encoder sees a greyscale picture whose brightness is the alpha.

One getImageData() call per frame feeds both encoders. It returns straight RGBA, where the colour has not been multiplied by the alpha (the HTML spec’s pixel manipulation section warns that converting to and from premultiplied values is lossy). The same buffer can go to the colour encoder labelled as RGBX, a format whose fourth byte is ignored.

const W = 640, H = 640, FPS = 30;
const canvas = new OffscreenCanvas(W, H);
const ctx = canvas.getContext('2d', { willReadFrequently: true });

// One readback per frame gives both encoder inputs.
function splitFrame(timestamp) {
  const rgba = ctx.getImageData(0, 0, W, H).data;

  // Colour: the same bytes labelled RGBX, so the encoder ignores the A byte.
  const colour = new VideoFrame(rgba, {
    format: 'RGBX', codedWidth: W, codedHeight: H, timestamp,
  });

  // Alpha: an I420 frame whose Y plane is the alpha channel.
  const luma = W * H;
  const i420 = new Uint8Array(luma * 1.5);   // Y, then U and V at a quarter size each
  for (let p = 0; p < luma; p++) i420[p] = rgba[p * 4 + 3];
  i420.fill(128, luma);                      // neutral chroma
  const alpha = new VideoFrame(i420, {
    format: 'I420', codedWidth: W, codedHeight: H, timestamp,
  });
  return { colour, alpha };
}

Writing the bytes into the Y plane yourself keeps the alpha out of any RGB-to-YUV conversion, which could change its values. willReadFrequently: true asks the browser for a software canvas, which suits frequent getImageData() calls. The sketch assumes even dimensions. For odd ones, round the chroma plane sizes up, as the spec’s I420 definition does.

Two encoders that must agree frame by frame

Both encoders get the same configuration. The codec string vp09.00.30.08 means VP9 profile 0, level 3, 8-bit, in the format defined by the VP9 codec binding. Level 3 is the lowest VP9 level that fits 640×640 at 30 fps. The picture has 409,600 luma samples, level 3 allows 552,960, and level 2.1 stops at 245,760.

const config = { codec: 'vp09.00.30.08', width: W, height: H, framerate: FPS };

function collect(chunks) {
  return new VideoEncoder({
    output(chunk) {
      const data = new Uint8Array(chunk.byteLength);
      chunk.copyTo(data);
      chunks.push({ data, us: chunk.timestamp, key: chunk.type === 'key' });
    },
    error(e) { console.error(e); },
  });
}

const colourChunks = [], alphaChunks = [];
const colourEncoder = collect(colourChunks);
const alphaEncoder = collect(alphaChunks);
colourEncoder.configure(config);
alphaEncoder.configure(config);

for (let i = 0; i < frameCount; i++) {
  drawPose(ctx, i / FPS);                       // your renderer: clear, then draw the pose at time t
  const timestamp = Math.round(i * 1e6 / FPS);  // microseconds
  const { colour, alpha } = splitFrame(timestamp);
  const keyFrame = i % FPS === 0;               // same key frames in both streams
  colourEncoder.encode(colour, { keyFrame });
  alphaEncoder.encode(alpha, { keyFrame });
  colour.close();
  alpha.close();
}
await Promise.all([colourEncoder.flush(), alphaEncoder.flush()]);

colourChunks.forEach((c, i) => {
  const a = alphaChunks[i];
  if (!a || a.us !== c.us || a.key !== c.key) throw new Error(`streams differ at frame ${i}`);
});

A player decodes the colour frame and the alpha frame from the same block, so the two streams must line up exactly. In the spec’s encode options, keyFrame: true means the frame must be a key frame and false leaves the choice to the browser, so request key frames on the same frames in both encoders. The check at the end refuses to mux if any pair disagrees on timestamp or type. Mediabunny goes a step further and forces an alpha key frame whenever the colour frame is one. A comment in its source warns that playback can glitch, or crash some browsers, without that.

Nothing in this loop waits for the page to paint, and each frame’s timestamp is whatever you pass in, in microseconds. That removes the dependency that stopped MediaRecorder. For long clips, watch encodeQueueSize and wait for the encoder’s dequeue event before queueing more frames.

Writing each frame as a BlockGroup

EBML is a small format. Every element is an ID, a size written as a variable-length integer, and a payload, so a few helpers can write the whole file.

const concat = (parts) => {
  const out = new Uint8Array(parts.reduce((n, p) => n + p.length, 0));
  let at = 0;
  for (const p of parts) { out.set(p, at); at += p.length; }
  return out;
};

// Element size as an EBML variable-length integer. All ones means "unknown size".
function vsize(n) {
  let len = 1;
  while (n >= 2 ** (7 * len) - 1) len++;
  const out = new Uint8Array(len);
  for (let i = len - 1, v = n; i >= 0; i--, v = Math.floor(v / 256)) out[i] = v % 256;
  out[0] |= 0x80 >> (len - 1);               // length marker bit
  return out;
}

const el = (id, ...children) => {
  const body = concat(children);
  return concat([Uint8Array.from(id), vsize(body.length), body]);
};
const uint = (n) => {
  const bytes = [];
  do { bytes.unshift(n % 256); n = Math.floor(n / 256); } while (n > 0);
  return Uint8Array.from(bytes);
};
const int16 = (n) => {
  const b = new Uint8Array(2);
  new DataView(b.buffer).setInt16(0, n);
  return b;
};

// One frame. timeMs is relative to the Cluster. refMs is the previous frame's
// time minus this frame's time, so it is negative.
function blockGroup(colour, alpha, timeMs, refMs) {
  const header = concat([Uint8Array.of(0x81), int16(timeMs), Uint8Array.of(0x00)]);
  return el([0xA0],                                     // BlockGroup
    el([0xA1], header, colour.data),                    // Block: the VP9 colour frame
    ...(colour.key ? [] : [el([0xFB], int16(refMs))]),  // ReferenceBlock, delta frames only
    el([0x75, 0xA1],                                    // BlockAdditions
      el([0xA6],                                        // BlockMore
        el([0xEE], uint(1)),                            // BlockAddID: 1
        el([0xA5], alpha.data))));                      // BlockAdditional: the VP9 alpha frame
}

const videoTrack = el([0xAE],                           // TrackEntry
  el([0xD7], uint(1)),                                  // TrackNumber
  el([0x73, 0xC5], uint(1)),                            // TrackUID
  el([0x83], uint(1)),                                  // TrackType: video
  el([0x86], new TextEncoder().encode('V_VP9')),        // CodecID
  el([0x55, 0xEE], uint(1)),                            // MaxBlockAdditionID
  el([0xE0],                                            // Video
    el([0xB0], uint(W)),                                // PixelWidth
    el([0xBA], uint(H)),                                // PixelHeight
    el([0x53, 0xC0], uint(1))));                        // AlphaMode: BlockAddID 1 is alpha

For track 1 the Block header is 4 bytes. The first is the track number as a variable-length integer (0x81), the next two hold a signed 16-bit time relative to the Cluster’s timestamp, and the last is a flags byte. A SimpleBlock’s flags byte includes a key-frame bit. A Block inside a BlockGroup has no such bit (RFC 9559, sections 10.1 and 10.2). A BlockGroup marks a delta frame by carrying a ReferenceBlock, and the RFC’s definition of ReferenceBlock says a BlockGroup without one can be decoded without any other block. Leave ReferenceBlock out and every frame claims to be a key frame. FFmpeg writes one on each delta frame with the same value as the sketch, the previous frame’s time minus this frame’s.

We checked what happens without it. In Chromium 152, a file built from the sketch with no ReferenceBlock loaded its first frame, but seeks to delta frames at 0.5 s and 1.2 s both ended in a decode error. With ReferenceBlock on the delta frames, the same seeks worked. Unity’s issue tracker lists a bug of the same kind, in which its Media Encoder mislabelled I-frames and P-frames in WebM files with alpha.

The rest of the file is short. The EBML header sets the DocType to webm. The Segment holds Info, with a TimestampScale of 1,000,000 ns so that block times are in milliseconds, then Tracks with the entry above, then the Clusters. Start a new Cluster at each key frame, as the WebM guidelines recommend. The 16-bit block time also limits a Cluster to 32.767 seconds. The guidelines ask for a Cues element that lists the key frames, and say seeking will be disabled without one.

// Start a new Cluster at each key frame. Times are in milliseconds.
const clusters = [];
let groups = [], clusterMs = 0, prevMs = 0;
colourChunks.forEach((c, i) => {
  const ms = Math.round(c.us / 1000);
  if (c.key && groups.length) {
    clusters.push(el([0x1F, 0x43, 0xB6, 0x75], el([0xE7], uint(clusterMs)), ...groups));
    groups = [];
  }
  if (c.key) clusterMs = ms;
  groups.push(blockGroup(c, alphaChunks[i], ms - clusterMs, prevMs - ms));
  prevMs = ms;
});
clusters.push(el([0x1F, 0x43, 0xB6, 0x75], el([0xE7], uint(clusterMs)), ...groups));

Why Safari gets the live SVG instead

Safari plays VP9 WebM but drops its alpha channel. Can I use marks WebM as supported in current Safari and iOS Safari, with a note that alpha transparency is not. Jake Archibald’s 2024 write-up serves VP9 with alpha to Chrome and Firefox and gives Safari HEVC with alpha, which Apple added in Safari on iOS 13 and macOS Catalina. Chrome’s WebCodecs can’t produce that format either, so it would have meant a second export path outside the browser.

Detection is awkward. The VP9 codecs string has fields for profile, level, bit depth, chroma subsampling and colour, and none for alpha, so canPlayType() cannot tell you whether transparency will survive. That leaves a user-agent rule or a runtime probe that decodes a tiny transparent clip and reads a pixel back.

The platform already had the live SVG, driven by the same pose model as the clips, so Safari users see the same character.

What we measured, and what we didn’t

The output figures cover the seven exported clips, which are 640×640 pixels at 30 fps and range from 186 KB to 1.08 MB. This post doesn’t list per-clip durations, so the sizes aren’t bitrates and don’t compare directly with other encoders’ figures. There are no numbers here for encode time, visual quality (VMAF or PSNR) or playback cost, and no HEVC-with-alpha version to compare against.

Separately, we ran the code in this post in a Chromium 152 build, on short synthetic clips, to check it end to end. Decoded frames came back as I420A. Transparent areas decoded at or near alpha 0, since the alpha stream is lossy too, and soft edges kept their colour. With AlphaMode removed, the same frames came back opaque. With ReferenceBlock removed, seeks failed as described above. Our test files had no Cues, and Chromium seeked them anyway once ReferenceBlock was in place. These checks confirm the layout and say nothing about size or speed.

A checklist for transparent WebM export

• Call VideoEncoder.isConfigSupported() with alpha: 'keep' before you design around it. In Chrome it returns supported: false.
• Read each frame back once with getImageData(). Send the bytes to one encoder as RGBX, and the alpha, copied into an I420 luma plane, to the other.
• Give both encoders the same configuration and the same key frames, and refuse to mux if any pair of chunks disagrees.
• Write each frame as a BlockGroup with the colour in the Block, a ReferenceBlock on delta frames, and the alpha in BlockAdditions under BlockAddID 1.
• Set AlphaMode and MaxBlockAdditionID to 1 on the track, and DocTypeVersion to at least 3 in the header. Start each Cluster on a key frame and write Cues.
• Test playback over a coloured background, seek into the middle of the clip, and look closely at soft edges.
• Try Mediabunny first if you’d rather not maintain a muxer.
• Plan Safari separately, with HEVC with alpha or a non-video version of the same animation.

Frequently asked questions