Existing short footage
Five seconds each. Small and clean, used to warm the model up before the full run.
Block-causal LTX-2.3 · joint audio + video
Direct the stream
while it runs.
Stream LTX rebuilds a bidirectional LTX-2.3 into a block-causal model that emits one second of picture and sound at a time. Because it never looks ahead, you can hand it a new instruction between blocks — a gesture, a spoken line, a prop to pick up, rain, nightfall — and the next second carries it out.
- Throughput
- 43 fps, DiT
- Block
- 1 second
- Hardware
- 4 × H800
- Resolution
- 512 × 768
- Continuous
- 10 min
Generation runs faster than playback, so a take can be watched as it is produced. Audio and video come out of the same model, in the same block. The 43 fps figure is steady-state DiT denoising throughput only — text encoding, VAE and audio decoding and muxing are not counted in it, and first-block latency is a separate number.
live output
block 000
Takes
Five runs, uncut
Every recording below is a screen capture of one continuous run — the console on the left, the live output on the right. Nothing is trimmed or re-timed. Pick a cue to jump to the second it lands.
Control
What one cue can change
A cue is a short piece of text. Across the five takes on this page, cues move the character's body, put words in their mouth, make them handle objects, and rewrite the weather and the light — all of it mid-stream, with the set and the character holding still.
Data
Two sources, one automatic annotation pipeline
What training needs is not a video with one caption on it, but a video cut into segments, each with its own prompt carrying that segment's action and line. That structure comes out of the pipeline below.
Sources
Constructed character performances
Fourteen seconds to thirty seconds each. Long-form acting data built for this project, used for the full training run.
Annotation pipeline
Shot detection
Finds the shot boundaries, which decide where a segment is allowed to break.
Speech timestamps
Marks where each line starts and ends, so voice lines land inside the right segment.
Per-segment captions
Describes the action and the scene of each segment; that description becomes its prompt.
Output
Video + segmented prompts
One sample is a continuous video plus a list of prompts cut along time. In training each AV block sees only the prompt of its own segment — which is exactly how the model is driven at inference, one cue per segment.
Training
Three stages, each one closer to streaming
A block is one second of picture and sound. Video and audio latents run at different rates — 3 fps and 25 fps — so a block is cut on a one-second boundary and carries both: 4 + 3k video frames beside 26 + 25k audio frames. All six attention paths, including the two cross-modal ones, carry a causal mask, so nothing inside a block can read the future.
-
Stage one
Frame-parallel teacher forcing + resample forcing
The 5-second clips warm the model up, then the 14-second clips carry the full run. The whole sequence shares one timestep and all frames denoise in parallel, which moves the bidirectional model onto a causal structure without breaking the base prior. Resample forcing is folded in: the model is rolled out without gradients and the next block is supervised on the history it actually produced rather than on ground truth alone, so training sits on the distribution inference will meet.
every block at the same noise level -
Stage two
Diffusion forcing
Every block gets its own noise level, which makes the training block-autoregressive. Inside one forward pass the earlier blocks are quiet and look like already-generated history, the later ones are noisy and look like the future still to come — the same shape as rolling forward one block at a time at inference.
one noise level per block, rising along the sequence -
Stage three
Few-step distillation
Distribution matching distillation takes sampling from 30 steps down to 4. This is the precondition for real-time throughput — without it the multi-GPU pipeline below still could not keep ahead of playback.
30-step sampling → 4-step sampling
Inference
Timestep parallelism: one noise level per GPU
Inference is split by denoising step, not by frame. Four GPUs each keep one noise level's KV resident; a block passes through the four levels in order and each finished level hands the latent to the next GPU over NCCL P2P. At steady state all four GPUs are working on four different blocks at once, and every time slice completes one block.
Why split by step and not by frame
Splitting by frame needs cross-GPU traffic inside a block, and a block-causal model's blocks are small — one second — so the communication would eat the gain. Split by denoising level, each GPU passes one latent per block boundary: fixed volume, independent of resolution.
A fifth GPU, off the critical path
One separate GPU carries prompt encoding and incremental VAE coding asynchronously. Only the DiT is left on the generation path, which is also why the 43 fps figure counts DiT alone — decoding and encoding never take a slice of the denoising pipeline.
Why a long take does not slow down
Long-context KV is split into an anchor, a content memory and a FIFO of recent blocks. Each layer scores its own keys on attention quality and temporal diversity and evicts down to a top-M budget, so a ten-minute take costs the same per block as the first minute.
What it costs to start
The first frame waits for the four-step pipeline to fill, about 2 seconds, with VAE decoding counted separately. The same model without timestep parallelism runs 30 steps at roughly 0.4 s per step, which is what real-time had to be won back from.
Notes
What this page is
The recordings
Screen captures of an internal research prototype, at the resolution and frame rate it actually ran. The first seconds of each take are the operator loading a first frame and starting the worker; the block strip begins where generation begins.
Cue timings
Timecodes are read off the console in the recording, so they mark when an instruction reached the model. The result lands a block or two later, which is what you see.
Availability
No code and no weights are released here. This page documents the work and shows what it does.