Captions
People can always import .srt / .vtt files or type captions. For automatic captions, give the editor a transcriber: the on-device one for English, or your own speech service for any language.
On the device (English)
npm install @video-ultra/captionsimport { createOnDeviceTranscriber } from '@video-ultra/captions';
const transcriber = createOnDeviceTranscriber();
<VideoEditor transcriber={transcriber} />;English speech becomes timed words on the device, in a worker, and nothing is uploaded. The model downloads once (tens of MB) and is cached. Silence, wind and room noise don't turn into made-up text, and captions say plainly when a browser can't read the sound.
Your own captions service (any language)
For other languages, or a speech service you already use, leave out createOnDeviceTranscriber() and connect your own endpoint with createCaptionsEndpoint from @video-ultra/ai. Your endpoint is a route on your server: it calls the speech service (its key stays there) and answers in one fixed format. The editor sends the sound in 30 s pieces, one at a time, skips silence, retries a busy service, and turns the words into styled captions in sync with edits.
import { createCaptionsEndpoint } from '@video-ultra/ai';
<VideoEditor transcriber={createCaptionsEndpoint({ endpoint: '/api/captions' })} />;What your endpoint receives and answers
A POST to /api/captions?language=hi (an ISO 639-1 code, or auto to detect). The body is the sound: 32-bit float samples, little-endian, 16 kHz, mono, at most 31 s. Answer:
{
"words": [
{ "text": "नमस्ते,", "start": 0.42, "end": 0.9 },
{ "text": "दोस्तों।", "start": 0.95, "end": 1.6 }
],
"language": "hi"
}- Times are seconds from the start of that piece. Give them when your service has them (most speech-to-text services do) — the word-by-word highlight follows them. Without times (
"words": ["नमस्ते,", "दोस्तों।"]) the words are timed from the sound itself: close on steady speech, less exact. - language is the language actually spoken. If none of the video was in the language chosen, the editor says which one it heard (“This sounds like Hindi, not French”) instead of making wrong captions.
- confidence (optional, 0–1 per word): words under 0.3 are dropped as likely made up, and captions averaging under 0.6 get a “check this” mark in the captions list. Whisper-style services report a per-segment average log-probability —
Math.exp(avg_logprob)is a good confidence. - Chinese, Japanese, Thai and other scripts written without spaces work as they come: the editor joins and wraps them without spaces and highlights them character by character.
Two optional query parameters carry hints you can pass to your service (for Whisper, as its prompt): context, the last words said before this piece, and vocabulary, names the person typed in “Names to spell right” (comma-separated), so they're spelled right.
What the editor does for you
- Cuts the sound at the quietest moment between 25 and 30 s (no word cut in two), sends up to three pieces at once (one while your service says it's busy), and retries a busy service, waiting as long as its
Retry-Afterasks. - Finds where people speak with a small speech detector on the device (0.6 MB, once): pieces with no speech — music or noise alone — aren't sent, so nothing is made up there, and a first word your service stretched over the silence before it starts where the voice does.
- Detect decides the language from the first pieces with speech and asks for it from then on (Hindi and Urdu, spoken alike, are written as the viewer's browser suggests).
- If nothing is heard clearly anywhere, it makes no captions rather than guesses.
Pass languages (e.g. ['en', 'hi', 'auto']) to offer only the ones your service hears; by default the panel offers about 100 and Detect, including Hinglish — hi-Latn, Hindi written in Latin letters the way people type it, which your service writes (Whisper does when asked for English with a Hinglish example as its prompt).
Fully custom
You can also implement Transcriber (from @video-ultra/core) yourself: the editor hands you the sound as 16 kHz mono pieces and expects { words: [{ text, start, end }], language } for the whole video.