Most 3D reconstruction systems ask you to choose: speed or accuracy. Run a fast feed-forward model and you get a rough point cloud. Run an iterative optimization pipeline and you wait — sometimes hours — for a scene that actually holds together at scale. LingBot-Map, a new open-source 3D foundation model from the Robbyant team, is built around a different premise: that a single unified architecture can deliver real-time streaming inference at ~20 FPS while maintaining state-of-the-art reconstruction quality across sequences exceeding 10,000 frames.
That's not a marginal improvement over prior work. It's the kind of gap that changes what's actually deployable on real hardware.
The Core Architecture: Geometric Context Transformer
The central innovation in LingBot-Map is the Geometric Context Transformer (GCT) — an architecture that unifies three problems that typically require separate systems: coordinate grounding, dense geometric cue extraction, and long-range drift correction. In most streaming reconstruction pipelines, these are handled by distinct modules that pass state between each other, introducing latency and error accumulation. The GCT handles all three inside a single streaming framework through three structural components:
- Anchor context — grounds coordinate predictions to stable scene anchors, reducing positional drift as sequences grow.
- Pose-reference window — maintains a sliding window of reference poses so the model can refine camera estimates without revisiting the full sequence.
- Trajectory memory — a long-range memory mechanism that catches and corrects drift across far-separated frames, enabling loop closure without a dedicated SLAM back-end.
The result is a feed-forward model that runs with paged KV cache attention — borrowing the same memory-efficient attention management popularized in LLM serving — to sustain stable inference at 518×378 resolution without memory blowup on long sequences. On benchmarks spanning KITTI, Oxford Spires, ETH3D, Tanks and Temples, and TUM-D, LingBot-Map outperforms both existing streaming approaches and iterative optimization-based methods.
Three Model Variants for Different Use Cases
Robbyant ships three checkpoints, each optimized for a different operating regime:
- lingbot-map-long — the recommended checkpoint for long sequences and large-scale scenes. Best choice for indoor walkthroughs, outdoor drives, and aerial footage exceeding a few hundred frames.
- lingbot-map — the balanced checkpoint used in the paper and all benchmark evaluations. Trades some long-sequence headroom for consistent all-around performance across short and long sequences alike.
- lingbot-map-stage1 — the Stage 1 training checkpoint. Can be loaded into the VGGT model for bidirectional inference, making it useful as a base for research fine-tuning or alternative pipeline configurations.
All three are available on both HuggingFace (robbyant/lingbot-map) and ModelScope.
Streaming Inference at Scale: How the KV Cache Strategy Works
The most practically important design decision in LingBot-Map is how it handles sequences that exceed the model's training range. The model is trained with video RoPE on 320 views; once the KV cache stores more than 320 views, performance begins to degrade. Rather than hard-capping sequence length, Robbyant ships two strategies for pushing beyond that limit without retraining:
Keyframe Interval Mode
The --keyframe_interval N flag instructs the model to cache only every N-th frame as a keyframe. Non-keyframe frames still produce per-frame predictions and contribute to the trajectory, but they don't consume KV cache slots. At --keyframe_interval 2, you effectively double the sequence length the cache can hold; at --keyframe_interval 10, you extend coverage by 10×. This is the recommended approach for sequences in the 320–3,000 frame range.
Windowed Inference Mode
For sequences exceeding ~3,000 frames — the team demonstrates a 25,000-frame, 13-minute indoor walkthrough — windowed mode (--mode windowed) engages a sliding window that resets the KV cache at each step. A key parameter here is --overlap_keyframes, which controls how many keyframes adjacent windows share, keeping cross-window pose alignment stable. At --window_size 128 and --keyframe_interval 13, each window covers roughly 1,568 actual frames of sequence (8 scale frames + 120 keyframe slots × 13 interval), while the 8-keyframe overlap provides ~104 actual frames of cross-window context.
This kind of memory-efficient attention management mirrors what's happening in LLM inference optimization — the same paged KV cache techniques that allow 70B language models to run on 4GB of VRAM are now enabling multi-hour 3D reconstruction sequences on consumer hardware.
The Offline Rendering Pipeline: Long-Sequence Batch Processing
For sequences too long for interactive visualization, LingBot-Map includes a separate offline rendering pipeline (demo_render/batch_demo.py) that runs inference and produces a headless point-cloud flythrough MP4 in a single command. The pipeline handles:
- Sky segmentation via an ONNX model (auto-downloaded from HuggingFace) — masks out sky points from outdoor scenes for cleaner reconstructions.
- GPU voxelization and frustum culling via custom CUDA extensions (
voxel_morton_ext,frustum_cull_ext). - Configurable cinematic camera paths via YAML presets — follow cam, birdeye reveal, and smooth segment transitions — without touching CLI flags.
- Per-frame NPZ prediction saving for post-hoc re-rendering with different camera or overlay settings.
Built-in presets cover indoor walkthroughs, outdoor drives, and aerial footage. A notable use case in the documentation: LingBot-Map works out of the box on footage generated by LingBot-World, the team's own world model — meaning reconstructed 3D scenes can feed back into the generative pipeline and vice versa. This kind of tight integration between world models and 3D reconstruction is exactly the kind of infrastructure stack that underpins what national physical AI programs are building toward at scale.
Installation and Hardware Requirements
LingBot-Map requires Python 3.10, PyTorch 2.8.0, and CUDA 12.8. The team recommends FlashInfer for paged KV cache attention — it's a pure-Python package that JIT-compiles CUDA kernels on first use, so a single wheel works across CUDA/PyTorch versions. If FlashInfer is unavailable, the model falls back gracefully to PyTorch's native SDPA attention via --use_sdpa.
# Create environment
conda create -n lingbot-map python=3.10 -y
conda activate lingbot-map
# Install PyTorch (CUDA 12.8)
pip install torch==2.8.0 torchvision==0.23.0 --index-url https://download.pytorch.org/whl/cu128
# Install LingBot-Map
pip install -e .
# Install FlashInfer (recommended)
pip install --index-url https://pypi.org/simple flashinfer-pythonFor users running on limited VRAM, two flags are worth knowing immediately: --offload_to_cpu (enabled by default) offloads per-frame predictions to CPU during inference, and --num_scale_frames 2 reduces the activation peak of the initial scale phase from 8 frames down to 2. The model is accessible even on an RTX 4060 with 8GB VRAM — the community has already published optimized configurations for that hardware tier. This mirrors the broader trend of open-source inference tools making large model architectures accessible on consumer hardware — something Bonsai 27B demonstrated at the LLM level.
Quick Start: Your First Reconstruction in One Command
After installation, run an example scene interactively via the browser-based viser viewer:
python demo.py --model_path /path/to/lingbot-map-long.pt \
--image_folder example/courthouse --mask_skyThis launches the viewer at http://localhost:8080. Four example scenes are included out of the box — courthouse, university, loop closure, and an outdoor Oxford scene — along with demo sequences downloadable from robbyant/lingbot-map-demo on HuggingFace.
Benchmark Coverage and Evaluation Scripts
Robbyant has released full evaluation scripts for KITTI and Oxford Spires under benchmark/, with preprocessing pipelines included. The complete benchmark suite covers nine datasets:
- KITTI (autonomous driving)
- Oxford Spires (large-scale outdoor)
- VBR, Droid-W, TUM-D (indoor/handheld)
- 7-Scenes, ETH3D (small indoor)
- Tanks and Temples (object-scale to building-scale)
- NRGBD (RGB-D indoor)
This breadth of evaluation is significant. Most open-source reconstruction models are benchmarked on one or two datasets; LingBot-Map's claim to outperform both streaming and optimization-based methods is tested across a genuinely diverse set of capture conditions, scales, and sensor types.
What's Next
The Robbyant team has flagged that a stronger model targeting longer sequences is currently in training. Given that the existing architecture already sustains quality across 25,000-frame sequences with windowed inference, the next checkpoint is likely aimed at either eliminating the window-reset overhead or extending the native RoPE training range significantly. The tight integration with LingBot-World also hints at future work connecting generative video models with metric 3D reconstruction — a combination that would be immediately relevant for robotics simulation, spatial computing, and physical-AI training pipelines. As open-source AI infrastructure matures, tools like LingBot-Map represent what's becoming possible without datacenter-scale hardware.
The repository is live at github.com/Robbyant/lingbot-map under the Apache 2.0 license, with models on HuggingFace and ModelScope.
Related Reading
- 70B LLMs on 4GB VRAM — AirLLM's Layer-by-Layer Trick That Rewrites the Rules
- Bonsai 27B: The First Frontier-Class LLM That Runs on Your Phone
- Japan Is Building a National Physical-AI System — and 27,500 Rubin GPUs Are Just the Start
- 43% of AI Code Breaks in Production — and Engineers Are Paying the Price
