Closing the Loop: Training-Free Revisit Consistency for Autoregressive Generative Rendering
1Roblox 2The Pennsylvania State University
An autoregressive generative rendering model streams a world chunk by chunk — and forgets it just as fast. By the time the camera returns, the first visit has already left the bounded KV cache, so the model repaints the place from scratch. We put the evicted chunk back and tell attention exactly where to look inside it.
Abstract
Recent conditional video generation models have shown promising potential to transform 3D engine renderings, such as depth maps and untextured geometry, into photorealistic videos for gaming and immersive content creation. These applications require long-horizon auto-regressive generation that continuously synthesizes new frames while preserving a persistent 3D world. Auto-regressive generators synthesize video chunk by chunk with a bounded KV cache, so when the camera revisits a location after its context has been evicted, the model often regenerates inconsistent appearance, even though the conditioning renderings (e.g., depth) remain perfectly aligned with the underlying geometry.
We address this revisit inconsistency without any post-training by exploiting correspondences the 3D engine already provides: temporal correspondence retrieves pose-matched historical latent chunks into the KV cache as loop-closure memory, while spatial correspondence from camera pose and depth reprojection biases token-level attention toward geometrically corresponding regions of the retrieved chunks. We demonstrate our method on loop-closure trajectories mined from the TartanAir and TartanGround datasets to mirror complicated real-world application scenarios, where it outperforms existing training-free baselines on revisit consistency without losing overall video quality.
Bounded memory makes generators forget the world they built
Streaming generators keep memory and per-chunk compute constant with a sliding-window KV cache — which means every chunk is eventually evicted. Under a plain sliding window, the chunk depicting a revisited location is long gone when the camera returns. Adding a fixed attention sink (à la StreamingLLM) preserves the first chunk and stabilizes global appearance, but the first chunk generally is not the revisited view, so it still cannot anchor a loop closure. In both cases the model must repaint the location from scratch and may produce different textures, colors, or structures — even though the conditioning depth and camera trajectory are perfectly consistent, since they originate from the same underlying 3D geometry. We call this failure revisit inconsistency.
Two inference-time mechanisms on a frozen generator
We build on a depth-conditioned Causal Wan-VACE student distilled with Self-Forcing, and change nothing about its weights. The 3D engine already knows where the camera is and how far every pixel is — we simply hand that knowledge to the cache and to attention.
Pose-retrieved loop-closure memory
The clean-cache budget is split into a persistent anchor (first chunk), one retrieved slot, and a recent window. Evicted latents and their poses are kept outside the attention cache, so any historical chunk stays retrievable. At each step we reinstate the admissible candidate closest in pose to the current chunk — gated by a normalized return distance, a viewing-direction agreement, and a minimum temporal gap, so long-range memory is injected only at genuine revisits. If no candidate qualifies, the slot simply extends the recent window.
Geometric correspondence as an attention prior
Reinstating a chunk restores access to the earlier appearance, but attention must still match tokens across a large viewpoint change. Since the correspondence is known, we lift each query token to 3D with its metric depth, reproject it into the retrieved view, and add a Gaussian suppression bias to the retrieved-key logits around the predicted match — after an occlusion-aware visibility test. The bias narrows where the model looks; the learned attention still decides what to copy.
Logits only
Cached keys, values, and latents are never warped or blended — resampling under imperfect poses would inject blur straight into the content path.
Retrieved keys only
The bias touches only the pose-retrieved chunk's key columns in the main branch; the recent window, anchor, and control branch keep ordinary attention.
Exact fallback
Queries with an undefined, off-grid, or occluded correspondence receive a zero bias and reduce exactly to standard attention.
Video comparisons
All methods share the same Causal Wan-VACE base model, depth control, prompts, random seeds, and KV-cache size. For each scene we pair the annotated qualitative figure (first visit / return; red boxes mark the same region) with its full rollout video beside it. Both share the same columns — Reference · Depth · Self-Forcing · Deep Forcing · Infinity-RoPE · MemRoPE · Ours. Watch the return: baselines regenerate altered structures, ours re-renders what was there.
Plugged into a live game engine
We integrate the approach with an in-house game engine that streams exactly the annotations our setup assumes — camera poses and metric depth — directly to the generator. On loop-closure camera paths, the generator must dress a castle with complex architecture and a scene of untextured geometric primitives in plausible appearance. Baselines regenerate altered structures on return (e.g., an added dome); ours re-renders the same structures at both visits.
Castle: engine-streamed depth & camera poses along a loop-closure path. Columns: Reference · Depth · Self-Forcing · Deep Forcing · Infinity-RoPE · MemRoPE · Ours.
Untextured primitives: the generator invents appearance for bare geometry — and keeps its invention on return.
BibTeX
@article{ma2026closing,
title = {Closing the Loop: Training-Free Revisit Consistency for
Autoregressive Generative Rendering},
author = {Ma, Wenchao and Liu, Changran and Jiang, Haomiao and Huang, Sharon X.},
journal = {arXiv preprint arXiv:XXXX.XXXXX},
year = {2026}
}