1. The big picture
Two independent cameras don't fire their shutters at exactly the same instant. Without hardware genlock, the offset can range from a few milliseconds (consumer USB cameras) to tens of milliseconds (professional capture rigs after SDI / HDMI conversion).
The muxer attacks this in three layers, and you can verify each from the health pill on a running stream card:
- Automatic rate lock. Each input is snapped to the profile's frame slots. Bounds the in-pair L↔R offset to ≈ one frame interval (33 ms at 30 fps, 17 ms at 60 fps). Always on.
- Systematic-offset compensation. Reads the steady-state mean offset from the running stream and surfaces a "Suggested offset" hint. Apply it via
eye_left_offset_ms or eye_right_offset_ms on the profile. Drops drift to per-shutter random jitter.
- tight_pair sync mode (
cine4k_tab_30 default). Drops the per-input rate lock so vstack's framesync pairs L and R by closest capture timestamp. Tighter sync, larger demuxer buffers. Enabled by default on the demo profile.
2. The recommended workflow for a fresh camera setup
- Pick a stereo profile from the dropdown (
cine4k_tab_30 or uhd4k_tab_30 for the demo).
- Pick L / R cameras in the Stereo Sources dropdowns.
- Click Start Stream. Wait ~15 seconds for the metrics to stabilise — the warmup gate hides early values.
- Open the running stream's card. Expand Alignment. Look for the green banner: "Suggested offset (real cameras): set
eye_left_offset_ms = X in this profile."
- Apply the suggested value. Edit the profile in
config.toml (or PUT /api/v1/profiles/<name>) — the field is signed; positive values shift L later, negative values shift R later.
- Restart the stream. Watch the health pill:
drift should drop to near zero, peak stays low after warmup.
3. Reading the health pill
Every running stream card carries a one-line health summary. The sync-relevant fields:
drift X ms (peak Y) — current and worst L↔R input offset measured at showinfo@l_in / showinfo@r_in taps. Hidden when both are zero. Orange when current > 50 ms or peak > 100 ms.A/V N ms (peak M) — current and worst audio↔video offset at the encoder input. Hidden when within ITU lip-sync spec (< 22 ms). Orange > 45 ms (audible).dup N · drop M — output muxer dup/drop counters from -fps_mode cfr. Hidden when zero. Orange when > 1 % of total frames (host can't keep up — try a lighter profile).
4. Knowing when to stop tuning
- If
drift reads sub-frame (< 33 ms at 30 fps) and you can't perceive the offset visually, you're done. Per-shutter random jitter is what's left and is not software-correctable without genlock.
- If
drift reads > 1 frame consistently, the calibration loop hasn't been applied yet — re-read step 2.4–2.6 above.
- If
dup or drop climb during a stream, the encoder is the bottleneck, not sync — drop to a lighter profile (e.g. uhd4k_tab_30 instead of cine4k_tab_30) or check that the host has the right HW encoder available (NVENC on the demo box, VideoToolbox on M1 macs).
5. Fast troubleshooting
- Camera shows black / "preview timed out" → macOS Camera permission. Reset with
tccutil reset Camera, restart Terminal, relaunch the muxer, click Allow on the prompt.
- "alignment is only live-tweakable on…" → this is expected on machines without ZMQ. Edit the profile, restart the stream — the values bake in at start time.
- Stream goes
reconnecting immediately → encoder couldn't open. For 4 K stereo TAB profiles, only HEVC works (h264 caps at 4096 px in any dimension; the 4320-tall TAB image hits that ceiling).
