Make physical experience usable

What connects words to our world?

Haptera
11 October 2026

How do we make physical experience as useful to AI as language?

Pick up a cup. You feel its weight and whether your grip holds. A robot needs to learn from that kind of feedback too.

Those lessons should help more than one robot. A new machine should be able to learn from interactions recorded before it existed, somewhere else, on different hardware. And the people working alongside those machines should be able to use the same evidence to understand what happened.

This is actually a question I landed on from the opposite direction: representing what happened on screen through physical feedback…

Representing an experience

When I was a kid, I rode The Right Stuff at Six Flags six times in one weekend. When we left, my Mom chuckled when I asked, “Why can’t we have that at home?”

In 2012 I began researching synchronized haptics. Vests, gloves, and other devices could convey an impact, a footstep, or an engine starting alongside the picture and sound.

The challenge was making it scale. Hardware kept changing. Different devices needed different instructions. Tying the content to each device, as haptics worked at the time, meant doing the work again whenever the hardware changed.

The breakthrough was to represent the events independently of the hardware. Describe what happened, keep it synchronized to the source, and let each device translate that representation into the feedback it could produce. The content could survive a change in hardware. That work led to patents on synchronized-haptics systems.[1]

Context mattered too. Footsteps in a horror film could call for different feedback from footsteps in a romantic comedy. We could preserve when each footstep occurred while changing how the device conveyed it.

After several years, I stepped away from haptics and spent the next chapter working in the early days of generative AI. In 2019, my cofounder and I learned, deployed, built apps on top of and landed awards from Google and contracts for our work with GPT-2.

When I revisited haptics in 2025, I wanted to know:

1. Could AI generate an event track from video?
2. And if it could, what would the model actually learn about our physical world and the context of each event?

Establishing what happened

For a robot, contact might be the goal of an insertion task or an unexpected collision during movement. The context changes what the interaction means and what the robot should do next. Either way, the system needs a reliable account of what happened.

Dr. Stevan Harnad’s work on symbol grounding helped me think about that connection. He asked how symbols could acquire meaning beyond their relationships with other symbols, and proposed grounding them in sensory representations and learned categories.[2]

For our work, that becomes a concrete question: when a model says “contact,” what physical distinction has it learned to recognize?

We can test part of the answer. Give a model observations of an interaction, then compare its predictions with synchronized measurements from an instrument that was there. Establish which event the instrument measures, whether the model recovers it, and when the available evidence cannot support an answer.

Contact detection does not settle the philosophical question of meaning. But it gives us a way to examine whether a system’s account stays connected to the physical event it claims to describe.

Giving physical events a record

A useful record might say that contact was present during an interval, identify the source recording, and show the evidence supporting that conclusion. It should also identify intervals where the evidence was insufficient.

Each event needs a defined meaning and a place in time. A camera-based estimate must remain distinguishable from a direct sensor measurement. The sensor measurement needs context too: what was measured, where, and under which conditions?

“No contact” and “we could not tell” must remain separate. Otherwise, uncertainty quietly becomes false information.

A common format makes those distinctions available to another person or system. It can connect observations from cameras and native machine signals while preserving their different origins and limits. A field named contact is easy to create. The work is deciding when the evidence supports that label.

This is the foundation of Haptera: a physical-event record that stays connected to its source and remains useful beyond the system that produced it.

Keeping the history available

Consider a battery that passes inspection and is then struck during a handoff. At the next station, its inspection result may still say “passed.” What happened afterward matters to the next decision.

The person or machine receiving it should be able to inspect that interaction and its supporting evidence. If another inspection is needed, its finding should become part of the same history, available to whoever handles the battery next.

That history could help beyond the individual incident. An engineer could compare similar handoffs. An agent could help identify a recurring problem and assess the results of an approved change. Another team facing a similar problem would have the evidence and conditions behind the finding to examine.

We’re building Haptera toward that kind of shared physical history. What happens during the work should remain useful when the work passes to someone else.

Testing what the record adds

Similarly, we built OnsetBench to independently benchmark the recovery of defined physical events against instrument references. It examines which events a system recovers, how accurately it places them in time, and what it misses or gets wrong. You can learn more at OnsetBench.com.[4]

Reusing experience across machines

Some of that experience already exists in archives. You cannot add a force sensor to last year’s recording. But the observations it preserves may still contain evidence another person or machine can use.

When I started in haptics, I wanted an experience to survive a change in hardware. Now I want what we learn from physical work to remain useful beyond the people and machines that produced it.

How much of the physical world should every new machine have to learn again?

We’re starting with contact. The goal is to make physical experience usable by every machine.

Notes and references

[1] Systems and methods for multimedia tactile augmentation, U.S. Patent 10,429,931 B2. Patent record.

[2] Dr. Stevan Harnad, The Symbol Grounding Problem, Physica D 42 (1990), pp. 335–346. Read the paper. The application to physical-event measurement is our engineering interpretation.

[3] Haptera, Onset Benchmark Review v4, 12 September 2026, reference definitions, contact-state evaluations, and serving-candidate scope. The results describe specified systems and recording conditions.

[4] Haptera, Why OnsetBench, Haptera, 11 October 2026. Haptera develops the benchmark and also supplies evaluated systems; detailed scores and evaluated configurations belong in the benchmark results.