Touch Is The Ground Truth

A camera sees the scene. It does not see the contact. The moment a hand closes on an object, the object is behind the hand, and the patch of the world that matters most is the patch the camera cannot reach. Vision supplies the whole-object layout: what the thing is, where it sits, its shape and its orientation. Touch supplies the contact detail: surface texture, compliance, grip force, and the onset of slip. Appearance lets vision guess at material. A taxel measures it. A camera cannot measure force at all, and it cannot see the millisecond in which a grip starts to fail. Biological perception splits the same way. Vision dominates spatial tasks. Touch dominates material and contact tasks. They are not two readings of one signal. They are two channels, and any system that owns both has to decide where to join them.

Early fusion projects both channels into one representation before anything is learned. A ResNet branch for the image, a small convolutional branch for the tactile frame, the features concatenated, then a classifier. It is simple, and the joint representation is learned from scratch. The cost is that a 16 by 16 tactile grid and a 2,048-pixel visual patch do not live in the same space, and forcing them into one early throws detail away. Mid fusion runs each channel through its own encoder and joins at a feature layer. A vision transformer emits 768-dimensional features, a tactile encoder emits 256, cross-attention runs between them, and the result is shared. Each modality gets the processing it needs, and the join happens where both sides are rich enough to align. Which layer to fuse is a designer's call, and cross-attention is expensive. Late fusion has each channel decide first. Vision reports a stable grasp at 80 per cent confidence, touch reports the same at 95, and a weighted average settles it. The two halves can be built independently and either can fail without taking the other down. What late fusion cannot catch is agreement between the channels: a tactile roughness that matches the visual texture, or a mismatch that should have raised a flag.

Cross-attention is where the current work sits. The visual scene is cut into patches, each patch a token. The tactile field is cut into patches or taxel groups, each its own token. Every vision token attends to the touch tokens that matter to it, and every touch token attends back. The output is a correspondence nobody annotated: this region of the image pairs with this patch of contact. The model learns the mapping between what it sees and what it feels instead of being handed it. That is what a grounded representation means in practice. The price is in the compute. Attention across two token sets grows with the product of their lengths, and a 2,048-patch visual field against a 16 by 16 tactile grid is already a large matrix, run at the rate the faster channel demands.

Vision runs at 30 to 60 frames a second. Touch runs at 100 to 1,000. A contact event that resolves in 2 milliseconds is invisible inside a 33-millisecond visual frame, and downsampling touch to meet the camera throws away exactly the events that matter. Four alignments are in use. Downsample touch to match the camera, which is fine when vision carries the task. Interpolate between visual frames, which is fine when touch carries the event. Fire the camera only when touch triggers, which is cheap in power and puts the expensive sensor in a support role. Or run both continuously and asynchronously: touch updates at its own rate and vision fills in the context around it. The last one is where general manipulation is heading, and it puts the most pressure on the join. An asynchronous update is a fusion problem every 2 milliseconds, not once per frame.

Vision guesses the material. Touch measures it.

The measured gap is bigger than most people assume. A policy trained on both channels reached 94 per cent success on cable insertion. Vision alone reached 72, and touch alone reached 68. Together they buy 22 points over the better of the two, which is the difference between a task that works in a demo and a task that works on a line. The second result is stranger and more useful. A tactile model pretrained on visual object recognition beat a tactile-only model by 15 per cent. Visual features transferred into a task with no camera in the loop. In 2024, self-supervised alignment between tactile and visual embeddings, with no labels at all, produced a shared space in which a smooth, shiny object sits in the same place whether it was seen or touched. None of that rescues a system that joins badly. The modality is not the lever. The choice of where to fuse is.

Every join needs a calibration before it means anything: where the tactile sensor sits relative to the camera, what the contact point is in camera coordinates, and whether the alignment drifts as the machine moves and the skin deforms under load. This is the grounding problem in one line. I see the cup at a point in space. I feel contact at taxels 45 to 47. Nothing links the two unless something measures it. Drift is the hard part. The skin deforms. The links flex. A calibration that holds at rest is stale the moment the arm moves, and a fusion layer trained against stale coordinates learns a correspondence that is quietly wrong. Sixteen years of field work, from rural Nigeria to post-conflict Serbia to Kazan and Fukushima, taught me where this kind of failure lands. A desk specifies a tolerance. The field supplies a deflection nobody modelled, and the readings stay plausible while the system slowly stops meaning what it says. The probability that vision and touch are ever fully interchangeable is not zero. Most people price it at zero and design accordingly, which is why so many systems bolt a force sensor onto a vision policy and call the result fusion. So the question to put to a supplier is not whether the two channels are fused. It is what the calibration costs, and who re-runs it when the skin deforms under load.