Control systems for robotics.

A safe shutdown was not just another command.

The actuator documentation required a particular power-down sequence, but the controller did not enforce it. I added a configuration-gated shutdown path: an explicit state machine in the control thread, with a guard and a bounded wait on every transition.

Fault shutdown stays a separate path because disabling a brakeless arm changes the hazard: remove drive power and gravity takes over, so "disabled" is not "safe", and the fault response must not share code with the orderly sequence. Then I checked the actual frames on a bench: gtest pins the transitions, packet captures confirm the frames leave in the documented order. The useful result was not only the feature; it was making the assumptions, exception, and evidence visible.

Evidence: state-transition tests, fault-path tests, packet captures, bench handover.

One latency number hid the problem.

Early timing work treated communication delay as a single result. I instrumented the path instead: every command carries a trace, stamped from one monotonic clock at each boundary it crosses.

using steady_ns = std::chrono::steady_clock::time_point;

struct CmdTrace {      // rides with every command; no allocation on the hot path
  uint64_t  seq;       // echoed by the device; joins host trace to the capture
  steady_ns created;   // control decision taken
  steady_ns enqueued;  // handed to the transmit thread
  steady_ns sent;      // sendmsg() returned; SO_TIMESTAMPING gives the wire time
  steady_ns received;  // feedback datagram arrived
  steady_ns decoded;   // protocol translation done
  steady_ns applied;   // state estimate updated
};

The sequence number joins the host trace and the packet capture into one timeline: tshark gives the per-hop wire times, the struct gives the per-stage host times, and disagreement between the two is itself a finding. One experiment improved the median while making part of the tail worse, the signature of batching: amortised per-packet cost pulls p50 down while queue occupancy under bursts stretches p99. Seeing p50, p95, and p99 per stage stopped us declaring victory on the wrong metric.

Evidence: monotonic timestamps, repeated-run comparison, wire correlation, percentile analysis.

A twin is useful only if it respects the real boundary.

Physical hardware was shared and slow to iterate against. I connected the production controller to a physics model over the same UDP boundary used by the device path: the twin binds the same socket interface, decodes the production command frames, steps a rigid-body model at a fixed dt, and answers with observation packets on the production schema. At the protocol layer the controller cannot tell the difference, so integration and regression runs exercise serialization, timing, and state handling rather than a mocked function call. The twin deliberately does not model bus arbitration or device firmware behaviour; those claims still belong to the bench, and saying so keeps the evidence honest.

Evidence: production protocol round-trip, shared geometry, integration tests, repeatable off-bench runs.