The Half Of The Loop That Is Missing

A teleoperation rig has two halves and only one of them is finished. The finished half is a six-axis force-torque sensor at the slave end and a motor or cable drive at the master end. The sensor measures the net force vector at the gripper. The drive pushes back on the operator's hand. That channel carries push, pull, weight and the transition from contact to no contact, and it carries them across a 0 to 50 hertz band. Fifty hertz is generous. Net force is a slow signal, because loads change at the speed a human hand changes them. More than ten commercial products close that loop today. The open half carries texture, contact shape, incipient slip and the difference between a pinch and a grip. Those signals live between DC and 1,000 hertz, and vibration is the carrier. A 200 hertz signal does not survive a channel designed for 50. So the operator feels the mass of the object and not its surface. Those are two different facts about two different things, and only one of them is in the room.

The data rate decides the architecture, so start there. Rich contact data from one finger runs 1 to 100 megabits per second. Five fingers and the loop is carrying more than a compressed video call. At the slave end the sensor problem is mostly solved. A GelSight camera-based pad or a capacitive array produces a spatial map of contact, both are shipping, and both resolve contact geometry well below the finger's own spacing. At the master end nothing equivalent exists. For the operator to feel that map, a display has to press on the operator's finger with the pattern the slave sensor measured. Spatially. At the same moment. A fingertip is roughly 15 millimetres across, so the device has to fit under it while driving 100 or more points independently. Latency sets the bar for the whole loop. Under 5 milliseconds end to end reads as the operator's own hand. Above 10 milliseconds it reads as a tool. That budget covers the sensor, the network, the decode, the display and the operator's own nerve conduction, which is not free.

Seven display technologies are on the board and the table is the entire argument. Vibrotactile ERM motors are a single point below 100 hertz. Mature, cheap, and no use for texture. Piezoelectric stacks reach 1 to 5 points across 1 to 500 hertz. Better texture, still a handful of points. Electrotactile arrays reach 10 to 100 taxels from DC to 500 hertz. The sensation reads as electric rather than tactile. Pneumatic air jets reach 10 to 100 taxels at 0.1 to 10 kilopascals below 50 hertz, and the plumbing is bulky. Shape-memory alloy matches the 10 to 100 taxel count below 10 hertz. Too slow, and too hot against skin. Ultrasonic mid-air arrays reach 1 to 10 points under 10 millinewtons. It is a lovely demonstration and the force is below what a finger reports. Dielectric elastomer actuators come closest: 100 to 1,000 taxels, DC to 200 hertz, 0.1 to 10 kilopascals, in a film thin enough to wear. Three requirements have to hold at once. Resolution, refresh rate and form factor. Nothing on the list holds all three, and the dielectric film holds two of them beautifully.

The proof is in the operating theatre and it has been there for twenty years. The da Vinci Si shipped with no tactile feedback. The da Vinci Xi ships with no tactile feedback. Both are commercial standards. The surgeon watches a monitor, sees the tissue deform, and estimates the force from the deformation. That estimate is poor. Deformation tells you the tissue is soft. It does not tell you where the artery runs, it does not separate a healthy margin from a tumour, and it does not warn you that the suture is about to pull through. Research systems closed the easy half. MiroSurge from DLR and Raven II both carry a six-axis force-torque sensor. Neither carries a tactile display, because there is nothing to fit. The absence of touch is the largest unmet clinical need in the field, and tens of thousands of procedures a year run without it. The probability that a real tactile channel ships inside the next generation of surgical robots is not zero. Most procurement sheets price it at zero, which is why the camera is still the only sense in the room.

Force feedback is finished. Touch feedback has not started.

Two engineering problems are sitting on top of each other and they are not the same kind of problem. The transmission half is encoding. A hundred megabits per second per finger cannot go out raw, so the fix is event-based: report the change, not the frame, which is the same move that made event cameras useful. Compression that preserves contact geometry is signal processing with a known shape and a published literature. The display half is materials. A film that presses on a fingertip at 100 independently addressed points, 200 times a second, at 0.1 to 10 kilopascals, on a device the size of the fingertip it serves, is not an encoding problem. It is an actuator problem, and actuators are limited by strain, force density and heat, in that order. One half moves at the speed of software. The other moves at the speed of a new polymer. That asymmetry is why the demo always looks finished and the product never is.

The unglamorous part is psychophysics. Somebody has to measure which stimulus on the operator's skin matches which reading from the slave sensor, and the mapping is not obvious. Spacing, amplitude and timing all interact, and a display tuned on one hand is mistuned on the next. Then sensor and display have to be calibrated to each other in space and in time, automatically, because a manual alignment drifts the first time the rig is moved between rooms. I have built companies across twelve countries and restructured a industrial group. The pattern in hardware programmes never varies. The transmission half gets funded because it looks like software. The actuator half gets deferred because it looks like manufacturing. Then the demo ships with force feedback and a paragraph about future tactile work. A laboratory showing convincing remote touch within five years is not a zero probability. That arriving inside a commercial surgical robot on the same clock is a far smaller one, and every robot specified this year is specified with a camera and a force estimate.