~20 FPS Across 10,000 Frames: The Open 3D Foundation Model That Rewrites Streaming Reconstruction

Most 3D reconstruction systems force a choice between speed and accuracy. However, LingBot-Map changes this dynamic entirely. This open-source model achieves ~20 FPS across 10,000 frames while maintaining high quality. Therefore, it rewrites streaming reconstruction for real-world hardware.

LingBot-Map delivers real-time streaming inference without sacrificing state-of-the-art results. It outperforms both iterative optimization pipelines and existing streaming methods on nine benchmarks. This performance gap makes large-scale deployment finally practical.

Geometric Context Transformer Architecture

The Geometric Context Transformer (GCT) is the core innovation. It unifies coordinate grounding, geometric cue extraction, and drift correction. Most pipelines handle these separately, which causes latency. GCT solves all three within one streaming framework.

First, anchor context grounds predictions to stable scene anchors. This reduces positional drift as sequences grow. Second, a pose-reference window refines camera estimates efficiently. Finally, trajectory memory corrects drift across far-separated frames. This enables loop closure without a dedicated SLAM back-end.

The model uses paged KV cache attention for memory efficiency. Consequently, it sustains stable inference at 518×378 resolution. Memory usage remains stable even during long sequences.

Three Model Variants for Different Needs

Robbyant provides three optimized checkpoints for various use cases. First, lingbot-map-long suits large-scale scenes and long sequences. It is ideal for indoor walkthroughs or aerial footage exceeding hundreds of frames.

Second, lingbot-map offers balanced performance for general evaluation. It trades some long-sequence headroom for consistent results. Third, lingbot-map-stage1 serves as a base for research fine-tuning. All variants are available on HuggingFace and ModelScope.

Streaming Inference at Scale

LingBot-Map handles sequences exceeding its training range effectively. The model trains on 320 views, but performance degrades beyond that limit. Fortunately, two strategies extend coverage without retraining.

Keyframe Interval Mode

The keyframe interval flag caches only every N-th frame. Non-keyframe frames still contribute to trajectory predictions. However, they do not consume KV cache slots. This approach effectively multiplies sequence capacity for ranges up to 3,000 frames.

Windowed Inference Mode

Windowed mode engages a sliding window for longer sequences. It resets the KV cache at each step to manage memory. Adjacent windows share overlap keyframes to maintain alignment stability. This technique mirrors optimizations used in large language model serving.

Therefore, multi-hour 3D reconstruction becomes possible on consumer hardware. This efficiency democratizes access to advanced spatial computing tools.

Offline Rendering Pipeline

LingBot-Map includes a batch processing pipeline for long sequences. It produces headless point-cloud flythrough videos in a single command. The system handles sky segmentation automatically for cleaner outdoor reconstructions.

Additionally, custom CUDA extensions handle GPU voxelization and frustum culling. Users can configure cinematic camera paths via YAML presets. Per-frame predictions save for flexible post-hoc re-rendering. This integration supports workflows where reconstructed scenes feed back into generative models.

Installation and Hardware Requirements

LingBot-Map requires Python 3.10, PyTorch 2.8.0, and CUDA 12.8. The team recommends FlashInfer for paged KV cache attention. If unavailable, the model falls back to native PyTorch attention gracefully.

Users with limited VRAM can offload predictions to CPU. Reducing scale frames also lowers activation peaks significantly. Consequently, the model runs on an RTX 4060 with 8GB VRAM. This accessibility mirrors trends seen in other open-source AI infrastructure projects.

Quick Start and Benchmark Coverage

You can run example scenes interactively via a browser-based viewer. Four demo scenes are included out of the box. Additional sequences are downloadable from HuggingFace for testing.

Robbyant released full evaluation scripts for KITTI and Oxford Spires. The benchmark suite covers nine diverse datasets including indoor and outdoor environments. This breadth validates claims against varied capture conditions and sensor types.

Future Development

A stronger model targeting longer sequences is currently in training. Future checkpoints may eliminate window-reset overhead or extend native training ranges. Tight integration with world models hints at robotics and simulation applications.

Tools like LingBot-Map represent what is possible without datacenter-scale hardware. The repository is live under the Apache 2.0 license. Models are freely available for community development and research.