Building AI-Powered Subtitles at Vimeo

Vimeo's subtitle system expects every source line to have a corresponding translated line — if the speaker talks from 0:05 to 0:08, the system needs text in that specific slot. But LLMs are designed to be smart: when they detect filler words ("um", "you know") or fragmented speech, they merge everything into one clean, grammatically correct sentence. The translation is linguistically excellent, but now three subtitle time slots are empty. The viewer sees subtitles disappear mid-speech — a "blank screen bug." This isn't a translation failure; it's a competence problem. The model is too good at its job.

Different languages don't just use different words — they organize thought in fundamentally different structures that break linear time-syncing. Vimeo identified three patterns: (1) German verb brackets (Satzklammer) place the main verb at the end of the clause, so splitting mid-sentence creates grammatically incomplete fragments the LLM resists producing. (2) Hindi SOV order reverses English structure — the end of the English thought maps to the middle of the Hindi sentence, breaking timestamp alignment. (3) Japanese compression — four lines of English filler genuinely condense into one tight Japanese sentence, leaving three empty slots. Romance languages like Spanish and Italian had ~99% clean pass rates, while Hindi had a 28.1% mismatch rate — 3-5x worse than Romance languages.

Vimeo abandoned single-prompt translation after finding that imposing format constraints on LLMs measurably degrades reasoning quality. Their solution splits the work into three phases. Phase 1 (Smart Chunking): a chunking algorithm scans for sentence boundaries across 10+ punctuation systems and groups lines into 3-5 line thought blocks — enough context to avoid hallucination but small enough to prevent drift. Phase 2 (Creative Translation): the LLM translates each chunk for meaning only — no line count enforcement. It handles German verb brackets naturally, reorders Hindi syntax correctly, and consolidates Japanese efficiently. Phase 3 (Line Mapping): a second LLM call performs purely structural work — "here are the original four English lines with timestamps, here is the translated block, break it back into four lines matching the source rhythm." By decoupling these concerns, each pass does its job without compromise.

~95% of chunks map perfectly on the first pass. For the ~5% that fail, the system enters a tiered fallback chain. Tier 1 — LLM Correction Loop: retries with explicit feedback about what went wrong ("you returned 1 line, we need 2"). The model often finds a valid synonym or slightly less natural phrasing. This resolves ~32% of failures, adding ~2-3 seconds and ~25% more tokens per retry. Tier 2 — LLM Simplified Fallback: strips all semantic instructions. The prompt becomes "split this text into exactly N lines" — no translation quality checks, no clause boundary awareness. Tier 3 — Deterministic Rule-Based Splitter: when LLMs still fail, a pure algorithm fills empty slots with the last valid content, pads insufficient lines by duplication, and truncates excess lines. Zero tokens, instant execution. Result: 100% of chunks reach the user in a valid state.

The multi-pass architecture adds 4-8% processing time and 6-10% token overhead versus single-call translation. Broken down: ~95% of chunks incur zero additional cost (first-pass success). The LLM correction loop adds ~2-3 seconds and ~25% more tokens per retry. The deterministic fallback (3-4% of chunks) is instant and free. The tradeoff: zero blank-screen bugs and approximately 20 hours of manual QA eliminated per 1,000 videos. Vimeo frames this as "the infrastructure tax of intelligence" — a dumb word-for-word translator never breaks sync, but a smart translator that merges, reorders, and condenses requires smart infrastructure around it to absorb the creative unpredictability.

Running the architecture against thousands of chunks across nine languages confirmed the design. Romance languages (Spanish, Italian) achieved ~99% clean pass rate on the line mapping phase — their structure is close enough to English that mismatches are rare. German and Hindi were the hardest: Hindi's 28.1% mismatch rate meant heavy reliance on the correction loop and fallback chain. Japanese compression was frequent but more predictable and easier for the line mapper to handle. The key insight: error handling budget should be proportional to linguistic distance from the source language, not uniform across all targets.