The Sensor That Cannot Be Simulated

Vision-based robotics had a decade of easy transfer because rendering got good. A ray-traced scene reproduces colour, lighting and texture closely enough that a policy trained on 100,000 synthetic frames survives the move to a real camera. Domain randomisation does the rest: jitter the light, the colour and the camera pose, and the policy stops depending on any of them. Tactile learning has no equivalent. A simulated contact signal is not a slightly worse version of a real one. It is wrong in the dimensions the task is about, which are force, friction and deformation. Vision crossed into routine transfer with colour and luminance jitter alone. Touch is still in the laboratory.

Six phenomena carry the difficulty, and each one breaks a different assumption in a standard physics engine. Friction decides slip and grip stability, and the stick-slip transition is chaotic, because the coefficient moves with micro-surface, humidity and velocity. Deformation decides force distribution and contact area, and a soft body deforms according to material parameters that are usually unknown plus a loading history that is never recorded. Adhesion at micro scale runs on surface chemistry and van der Waals forces that a rigid-body engine does not model at all. Viscoelasticity makes force depend on rate, which needs a history-dependent constitutive model and pays for it in compute. Surface roughness spans nanometres to millimetres, and no solver holds both scales at once. Contact dynamics turns impact, bounce and tapping into discontinuous events with energy loss, which produces stiff differential equations. That is why a deformable contact simulation runs 10 to 100 times slower than the rigid-body case. The gap has not closed.

The published transfer rates read like a specification sheet, and they are not flattering. Grasp stability classification transfers at 70 to 80 per cent, against 95 per cent for a policy trained on real data. Object recognition from touch lands at 60 to 75. Slip detection manages 30 to 50, which is the number that should end the discussion, because slip is the event a grip exists to prevent. Force estimation reaches 40 to 60, because contact area simulation is inaccurate. Peg-in-hole insertion reaches 50 to 70 and needs vision alongside the tactile channel to get there. Edge tracking is the strongest of the set at 60 to 80, and it is strongest because geometry carries most of that signal and physics carries least. No tactile approach has yet matched the routine 90 per cent and above that visual sim-to-real delivers.

Physics-based simulation is the first route: model the soft contact in MuJoCo, PyBullet, SOFA or Isaac Sim. It gets gross contact and approximate force magnitude right, gets fine texture and slip prediction wrong, and costs 10 to 100 times the rigid-body run. Image-based simulation is the second: render the tactile image from object geometry and a lighting model, as the TacTip, DIGIT and GelSight simulators do. That produces data which looks real enough for a convolutional network, and it fails the moment material properties change the contact geometry. Generative models are the third: train a conditional GAN or a diffusion model to turn a simulated tactile image into a convincing one. That narrows the visual domain gap and leaves the physics untouched, so the label attached to the image can still be wrong. Learned residual models are the fourth: run the simulation for approximate physics, then train on a small real dataset to correct the error. It works, and it needs real data for every new task, sensor and object. The dependency the method was meant to remove comes back.

A simulated fingertip is wrong in exactly the way that decides the grip.

Optical tactile sensors are the exception, and the reason is structural rather than clever. A GelSight-style sensor photographs a deformable gel surface from underneath, so contact becomes an image before any learning happens. Rendering that image needs geometry and a lighting model rather than a contact solver, which lets the sensor borrow the mature visual sim-to-real machinery directly. It works for the tasks where the camera inside the sensor can see what matters: shape, edge position, coarse texture. It fails where compliance is the signal, because material properties change the contact geometry and the render never sees them. The sensor succeeded by converting a physics problem into a vision problem. That trick reaches exactly as far as the physics it sidesteps.

The field needs a contact simulator that is both accurate and fast, and those two requirements currently exclude each other. Three directions are live: neural contact models trained from data, differentiable contact simulation that can be optimised end to end, and hybrid designs that run analytical methods for bulk deformation with a learned model at the interface. Until one of them holds, tactile work pays for its data in the real world, one grasp at a time. I have built companies across twelve countries and restructured a industrial group, and the same thing held in every physical technology I worked with: the published specification tells you what the machine can do, and the roadmap tells you what somebody hopes it will do. The probability that touch reaches visual transfer rates this decade is not zero, and it sits below the confidence the demonstrations carry. Read the transfer rate first.