HFlow: An SDK for Turning Robotics Recordings into Quality-Checked Training Data

Brandon and Kingston built HFlow because they kept watching robotics teams hit the same wall. Data pipelines start as scripts—one transcodes video, another checks timestamps, a third adds labels, another copies files into a training set. It works fine until the corpus grows. Then nobody can trace which code ran on what recording, why an episode got dropped, or whether a dataset can be rebuilt six months later.

The first problem to surface is usually quality control. Frozen cameras, missing sensor topics, timestamp drift, and duplicate recordings slip into training data quietly. Brandon encountered this while training embodied AI models for two-arm industrial cleaning robots. Kingston ran into it building high-throughput infrastructure at Jane Street. When they started talking to robotics data providers, they found teams rebuilding similar processing and quality-control infrastructure from scratch repeatedly. The conclusion: processing robotics data is itself a bottleneck to improving robotics models.

What HFlow Does

HFlow is an open-source SDK that turns multimodal recordings from robots and human operators into standardized, quality-checked episodes and queryable dataset manifests. A single recording can contain synchronized video, joint states, actions, timestamps, and metadata. HFlow processes those streams together and outputs a clean episode plus a manifest that documents exactly how it was produced.

Architecture and Components

An HFlow pipeline consists of transformations, checks, labels, and enrichments. The SDK exposes them as plain Python functions that receive an episode and return measurements, artifacts, or transformed data. During development, the functions run in-process. For scheduled corpus processing, HFlow packages the same registered steps as Airflow 3 DAGs, giving teams visibility into task status, logs, retries, and reruns.

HFlow currently accepts one MCAP file per episode. MCAP is an open container format developed by Foxglove for timestamped multimodal recordings—similar in purpose to a ROS bag. It keeps video, robot state, actions, and other sensor streams synchronized in a single file. HFlow uses it because processing requires these streams together, and because the resulting recordings stay compatible with Foxglove and Rerun visualization tools. HFlow outputs a canonical MCAP with in-band H.264 video, grouped camera and state chunks, and provenance tracking that describes how the output was produced. Each step carries an explicit behavior version, and catalog records link measurements and artifacts to the source episode and pipeline run.

Quality Checks and Evidence

Quality checks store reusable evidence rather than imposing one universal definition of good data. Some failures can be measured deterministically without training a model: black frames, frozen video, missing topics, timestamp drift, and impossible joint movements. Others might require VLMs or tools like MediaPipe Hands. But their significance depends on context—a smooth trajectory could mean a successful demonstration in one setting and a stalled robot in another.

HFlow writes measurements, metadata, version stamps, and artifact locations to an append-only Parquet catalog. Teams query it with DuckDB SQL to produce a version-pinned manifest without reopening the recordings. Critical checks can quarantine an episode, but HFlow never deletes data. This design separates evidence from policy, letting teams decide how to assemble a particular dataset without losing the underlying information.

Integration and Positioning

HFlow does not try to replace existing tools. It connects MCAP for synchronized recordings, Airflow for scheduled execution, Parquet for catalog data, and DuckDB for curation. Compared with a general workflow orchestrator, it adds contracts specific to robotics episodes, processing provenance, quality evidence, quarantine, and dataset manifests. Compared with a training dataset format, it operates earlier in the pipeline and stops at curated episodes plus a manifest rather than consuming the episodes into a final format.

Target Users

Three teams stand to benefit directly. A data vendor or marketplace collecting egocentric recordings could use HFlow to detect black or frozen video, duplicate recordings, hand-object interactions, and other quality metrics before delivering data—while retaining evidence of which checks ran on every episode. A robotics team collecting teleoperated demonstrations for its own models could standardize recordings, add labels and enrichments, and produce a reproducible training manifest. A team operating robots in the field could process incoming logs, quarantine incomplete or corrupted episodes, and query the catalog for particular robot versions, environments, or failure conditions.

Availability and Roadmap

The project is pre-v1, but the core lifecycle works end to end. Teams can try it without an account, Docker, or robot hardware by cloning the repository and following the quickstart. HFlow is free under the Apache-2.0 license. The open-source deployment currently targets a single-tenant workspace, and the hosted multi-tenant control plane remains on the roadmap. The founders are considering managed workspaces and enterprise support as potential revenue paths for teams that prefer not to operate the runtime themselves.

Community Contribution

Because this processing layer is software and data, contributors do not need to own a robot. The team is specifically seeking feedback from people who have built pipelines for robotics, video, or other sensor-heavy systems. They want to know where the data model breaks down, which integrations are missing, and what would fail first on real hardware.