How we score

Finding physical events in standard video, checked against sensor measurements.

The question

How reliably can a system find physical events using only standard video?

What the tests measure

• Loading onset marks a change in measured force. It can occur during ongoing contact.
• Contact state: whether a video window contains contact that produces force in the sensor reference. Holding and release are separate events.
• RH20T timing: when a change occurs in sensor readings across three robot setups.

The video tests use recorded video or timestamped frames. Sensors provide the reference labels. Model predictions are checked against those references.

How close must the timing be?

The main test allows predictions up to 200 milliseconds early or late: ±200 ms. We also report the stricter ±100 ms test. Predictions exactly at the limit count as within it.

Each prediction can match one reference event, and each reference event can match one prediction. We choose the assignment that finds the most events, then the one with the smallest total timing error.

A prediction with no match is a false detection. A reference event with no match is a miss. Extra predictions for the same event count as false detections. Matching stays within each episode and continuous block of available input.

What the scores mean

• Precision: of all predicted events, what share matched a reference event?
• Recall: of all reference events, what share did the system find?
• F1: a score that balances precision and recall. It equals twice the matches divided by the sum of predicted and reference events.
• False events per hour: false detections divided by the hours of input used for scoring.
• Median timing error: the middle absolute timing error among matched events.

Timing error measures matched events. Recall and false-detection counts include misses and extra predictions.

Comparison checks include guesses at the middle of a time block and predictions shifted by 25%, 50% or 75% of that block, wrapping around at its end. These checks keep the same prediction count within each scoring block.

Contact answers

Contact precision measures how often a “contact” answer was correct. No-contact precision does the same for “no contact” answers. Contact recall measures the share of reference contact windows correctly identified. Coverage is the share of windows receiving either decision; an unknown answer reduces coverage.

RH20T scoring

The RH20T mean F1 gives the three robot setups equal weight. The dataset names them cfg2, cfg6 and cfg7. The Haptera entry is the fixed recorded run with the highest mean F1 across the three setups. The same run is used for every setup and metric.

Reference times mark changes in sensor readings. Reference events outside the available input remain in the recall denominator. The input-coverage table is on the RH20T results page.

Model inputs

Haptera uses a fixed retained September video configuration for each task. The loading-onset run used saved video segments with later frames available. The same run is scored at both timing tolerances. Run identities record the selected entries and RH20T run.

Gemini 3.1 Pro Preview (gemini-3.1-pro-preview) received 0.8-second video clips. Its prompt asked whether the gripper fingers touched an object or surface and when contact began. Scoring uses the sensor-defined event. Visible touch and the start of a force change can occur at different times.

Open V-JEPA2 (facebook/vjepa2-vitl-fpc16-256-ssv2) and Open DINOv2 (facebook/dinov2-large) use frozen visual features with linear prediction heads trained for each task.

The loading-onset and contact-state comparisons use the same 83 REASSEMBLE recordings. The loading reference contains 3,898 events.

The September REASSEMBLE comparison uses 83 recordings from one robot setup. The evaluated Haptera models were not trained on these recordings; the set was scored repeatedly during development.

Decision thresholds for the REASSEMBLE baselines and all RH20T entries were selected using the scored reference labels. Haptera's REASSEMBLE loading-onset thresholds were fixed on separate development recordings and applied unchanged to the 83-recording comparison.

DINOv2 used a 30/9 training/validation split instead of the planned 29/10; baseline weights are unavailable. Download the results report.

Gemini 3.1 Pro Preview was evaluated on REASSEMBLE. It has no RH20T result.

Quiet intervals are defined by the sensor reference. Fewer false detections can also result from missing more events.

The 95% intervals use resampling of recordings or episodes.

You are watching 0.8 seconds of video from a camera mounted on a robot gripper. Answer in JSON only, with exactly two keys. contact: yes if the gripper fingers are touching an object or a surface at any point in this clip, no if they are not, unsure if you cannot tell. onset_ms: if contact begins during this clip, the millisecond from the start of the clip at which it begins (an integer from 0 to 800); otherwise null.