Burn captions and a music bed into a vertical video with one ffmpeg command

Every reel on this account is rendered by a single ffmpeg call. No editor, no timeline, no export dialog. This post walks through how that call is put together, because the pieces are reusable for any faceless vertical video.

There was no reel in the factory this week that I could write up, so this one is taken straight from the code in src/render.js rather than from a published video. Nothing below is a claim about a specific reel.

The problem

You have four still images and four voiceover clips, and you want a 9:16 video with captions that are part of the picture, because most people watch with the sound off. Doing that by hand in an editor takes a while every single day. A script has to do it.

The steps

1. Work out the timing from the audio. Each scene lasts as long as its voice clip, with a minimum of 2.2 seconds, plus 0.35 seconds of breathing room. The factory reads each clip’s duration (with ffprobe when the TTS service does not report one) and adds the scene durations up to get the total length.
2. Write the captions as an ASS subtitle file. ASS is a plain-text subtitle format that lets you style text. The factory defines three styles: a large hook line shown for the first 3 seconds, a scene headline near the top, and rolling captions near the bottom. The voiceover is split into chunks of five words and each chunk gets an equal share of its scene’s time.
3. Loop each image and give it a slow zoom. Each still is fed to ffmpeg with -loop 1 -t <duration>. The zoompan filter then zooms in on even-numbered scenes and out on odd-numbered ones, which stops the video feeling frozen. Scaling the image to double size first keeps the zoom from looking jittery.
4. Concatenate the scenes and add the logo. The concat filter joins the four video segments, and overlay places a small, slightly transparent logo in the top-right corner.
5. Burn in the captions. The subtitles filter reads the ASS file and draws it onto the frames. You need a fonts directory that actually contains the font named in the style, otherwise ffmpeg silently falls back to something else.
6. Build the audio. Each voice clip is resampled to 48 kHz and padded to its scene length with apad, then the clips are concatenated and normalised with loudnorm.
7. Duck the music. The music bed is looped with -stream_loop -1, turned down with volume=0.16, faded in and out, trimmed to the video length and mixed under the voice with amix.
8. Encode for social upload. H.264 at 30 fps, AAC at 48 kHz stereo, -movflags +faststart so the file can start playing before it has fully downloaded. A second tiny ffmpeg call grabs one frame at 1.2 seconds as the cover image.

The Node side is just execFile("ffmpeg", args) with the argument list built from arrays, which avoids shell quoting problems.

What breaks

Filter graph paths. The subtitles filter wants colons in paths escaped, and Windows-style backslashes need converting. The factory does both before it interpolates the path. If you see a cryptic “unable to open” error from the filter, look at the path first.

Braces in your captions. In ASS, curly braces start an override block. If a voiceover contains { or }, the text between can vanish or change the styling. The factory replaces braces with parentheses and turns newlines into \N before writing each line.

Missing fonts. The captions use Noto Sans. On a slim container image that font may not exist, and the render will still succeed with the wrong typeface. Install the font in the image and point fontsdir at it.

Memory. Four double-size zoompan streams in one filter graph are heavy. The service runs with a memory cap, so keep the scale-up and the clip count in mind if you copy this.

My take

If you are starting from scratch, build the subtitle file and the timing first and render with a plain background. Add the zoom and the music last. It is much easier to debug caption timing without a filter graph that takes a while to run each time.

Try it

The free workflows and the tools I use are at wrenbench.com.