Turning Video into Searchable Intelligence: How VSS Skills Power Autonomous AI Agents

The Challenge of Extracting Insight from Video at Scale

Organizations are capturing more video than ever, but turning that raw footage into actionable intelligence has remained stubbornly difficult. NVIDIA's Metropolis Blueprint for Video Search and Summarization (VSS) tackles this problem head-on, converting millions of live video streams or hours of recorded footage into instantly searchable, actionable data.

VSS provides a reference architecture for building video analytics AI agents that can perceive, reason, and act in real time across massive volumes of video. It combines accelerated vision microservices, vision-language models (VLMs), large language models (LLMs), and retrievers to deliver real-time intelligence, agentic search, and automated reporting. The latest release, VSS 3, introduces a modular design, an advanced fusion search capability, and a new set of skills that let coding agents deploy and operate VSS with minimal manual setup.

VSS architecture is composed of a set of microservices, databases, and agents for analyzing video.

Building a Video AI Agent with VSS Skills and Coding Agents

Previously, developers had to manually configure and integrate VSS's rich set of microservices for video management, search, and summarization. Now, coding agents augmented with VSS skills can automate deployment, usage, and integration — all through a simple chat interface.

VSS skills are hosted on the VSS GitHub repository and follow the agent skills specification, making them compatible with a wide range of agents such as Codex, Claude Code, OpenClaw, and NemoClaw. The only prerequisite is a system set up to run VSS along with a skills-compatible agent.

These skills streamline common development tasks, including:

• Deploying VSS profiles
• Uploading and indexing video
• Running search and summarization workflows
• Configuring alert verification
• Integrating VSS outputs into custom applications

Setting Up the VSS Prerequisites

The easiest way to prepare a system for VSS is the NVIDIA Brev Launchable. From the VSS launchable documentation page, click Launch Blueprint, then Deploy Launchable. Once deployed, open the notebook, navigate to /video-search-and-summarization/scripts/deploy_vss_launchable.ipynb, paste in your NGC_CLI_API_KEY, and run the entire notebook, including the tear-down section.

After the notebook completes, install the Brev CLI on your host system, launch VSCode, and connect remotely to your Brev instance using the Brev CLI (SSH) instructions on your Launchable page. From there, you can install Codex as your coding agent through the VSCode extension marketplace.

Deploying VSS with Codex

Once Codex is installed, the next step is loading it with the VSS skills. You can prompt Codex to self-install the skills directly from the VSS GitHub repository:

Read ~/video-search-and-summarization/skills/README.md and every SKILL.md file under ~/video-search-and-summarization/skills/. For each skill in the catalog, install it for this host so I can invoke it from a shell or chat session. Use the host's standard skills directory:

Claude Code: ~/.claude/skills/<name>/
Codex: ~/.codex/skills/<name>/
Hosts that follow the agentskills.io universal path: ~/.agents/skills/<name>/

Symlink each skill folder rather than copying it so a git pull here keeps every install up to date. Skip skills that are already installed and pointing at this checkout. When you're done, list the skills you registered and which directory you used.

Once loaded, Codex can deploy VSS components and profiles directly — for example, standing up the new VSS Search profile. It plans the deployment, configures the required environment variables, and launches all the containers needed to enable search functionality.

Searching Videos with VSS and OpenClaw

With the search profile running, you can configure OpenClaw as an autonomous agent for video analysis. After installing OpenClaw and giving it context about its role, copy the VSS skills into its workspace:

mkdir ~/.openclaw/workspace/skills
cp -r ~/video-search-and-summarization/skills/* ~/.openclaw/workspace/skills

Launching the OpenClaw dashboard lets you verify skill access and start issuing natural-language tasks. For example, an operator could point OpenClaw at a folder of warehouse videos and ask it to find every instance of a worker climbing a ladder while checking for a hardhat and safety vest. Behind the scenes, OpenClaw uses VSS skills to upload the footage, generate searchable embeddings via VIOS ingestion, and run fusion search to surface the relevant clips — returning a concise report along with supporting screenshots.

Smarter Video Search: From Alerts to Agentic Reasoning

Large-scale video search is one of the hardest problems in modern information retrieval. Queries are often complex and ambiguous, and a single visual embedding rarely captures the full semantic intent of objects or events. At scale, finding one moment across millions of hours of footage becomes a genuine needle-in-a-haystack problem, where simple nearest-neighbor search over one embedding space quickly loses precision and recall.

VSS addresses this with two core capabilities:

• Multi-type embedding extraction and retrieval, paired with relevance filtering and semantic deduplication
• Agentic search orchestration that decomposes complex queries into sub-queries, applies reasoning-based retrieval at each step, and runs iterative verification and reflection loops to refine results

The pipeline begins with RTVI-CV and embedding microservices ingesting video and extracting features. The VSS agent then uses this feature data along with vision-aware tools to run a deep, iterative search — building a plan and retrieving results to pinpoint specific objects or events across the video timeline.

A Modular Architecture Built for Flexibility

VSS uses a docker-compose based modular profile system. A base agent deploys in under five minutes, with additional workflows layered on as needed:

• Base / Q&A — VLM-based question answering and report generation on short clips
• Alert Verification — CV pipeline combined with behavior analytics and VLM verification
• Real-Time VLM Alerts — continuous VLM anomaly detection on live streams
• Search — agentic multi-embedding search across video archives
• Video Summarization — chunked summarization of extended recordings

Each workflow runs across multiple GPU configurations to match different hardware and performance needs.

Performance Benchmarks

For the agentic search workflow, a single H100 GPU supports up to 33 concurrent streams with 2.24 seconds of retrieval latency, while a single RTX PRO 6000 handles up to 51 concurrent streams at 1.87 seconds of retrieval latency.

For alert verification — measured using RT-DETR as the detector and Cosmos Reason 2 as the VLM verifier on streams averaging one alert event per minute — an H100 supports 147 concurrent streams at roughly 1.01 seconds of verification latency, an RTX PRO 6000 supports 87 streams at 0.82 seconds, and a combined DGX Spark and AGX Thor setup supports 14 streams at 0.89 seconds.

The long video summarization microservice also scales well: summarization time for an hour of footage drops significantly as more GPUs are added to the topology.

Get Started with VSS Skills

VSS skills give developers a way to turn video into searchable, meaningful data using natural language, making it easier to surface insights, generate summaries, and build smarter applications on top of video data.

To go deeper, explore the official VSS documentation and the full catalog of VSS skills on GitHub. NVIDIA Metropolis also supports the broader vision AI agent lifecycle — pairing VSS with TAO skills for model fine-tuning and the Video Data Augmentation skill for synthetic data generation to boost model accuracy on limited datasets.