Robot Observability and Replay

The failure that disappeared

A delivery robot brakes late once, but behaves normally when the engineer watches it. Restarting the robot removes the immediate symptom—and also destroys much of the evidence.

Retrieval pause: What would you need to record before trying the same route again?

Observability means arranging signals so a human can reconstruct what the system believed, received, decided, and commanded. Replay turns selected timestamped messages into a repeatable software experiment. Neither is a substitute for physical evidence.

The evidence loop

The order matters. Recording everything without a question can create a huge bag while missing the operator decision, parameter change, or hardware state that explains the failure.

Worked example: late braking

Suppose /scan remains healthy, but /cmd_vel stays positive 300 milliseconds after a near obstacle appears.

1. Hypothesis: perception publishes late, planning reacts late, or the actuator applies the command late.
2. Observe: log the safety state and monitor message age/period where topic statistics are enabled.
3. Record: capture /scan, /cmd_vel, /tf, /tf_static, relevant state topics, and /clock when simulation time is used.
4. Inspect: confirm those topics exist in the bag and that the failure window is present.
5. Replay: feed the recorded topics to the software under test in an isolated environment.
6. Compare: if software again emits late /cmd_vel, the evidence points into the recorded-input/software path. If it does not, investigate unrecorded state, nondeterminism, timing, middleware, or hardware.

That result narrows the investigation; it does not prove a motor physically stopped on time.

Recording and inspecting with rosbag2

ROS 2's rosbag2 records timestamped communications and can play them back. Prefer an explicit topic set tied to the hypothesis:

ros2 bag record -o late_braking /scan /cmd_vel /tf /tf_static
ros2 bag info late_braking
ros2 bag play late_braking

Use ros2 bag <verb> --help for the installed distribution's current options. Recording all topics with -a is useful for exploration, but it can increase storage and expose unrelated data. A bag is only as complete as the signals, time basis, and failure window it contains.

Logs, statistics, and bags answer different questions

• Logs explain discrete software events and decisions when messages include useful context.
• Topic statistics can summarize message age and period for enabled subscriptions; they help expose delay and irregular arrival.
• Bags preserve selected timestamped communications for later inspection and playback.
• Physical measurements show what the mechanism and environment actually did.

No single layer proves the entire causal chain.

Replay is controlled evidence, not time travel

Playback republishes recorded communications. It may differ from the original run because wall-clock load, scheduler order, random seeds, external services, unrecorded parameters, device drivers, network conditions, and physical dynamics can differ.

Three safe claims:

1. The bag contains a particular recorded sequence.
2. A consumer produced a particular output under stated replay conditions.
3. Repeated replay did or did not reproduce the software symptom.

One unsafe claim: “The replay proves the physical robot behaved identically.”

Misconception checks

“More logs always mean better observability”

Volume is not structure. Evidence needs timestamps, stable identifiers, severity, context, and a question it can answer.

“If replay succeeds once, the bug is fixed”

A successful replay may only show that one recorded case no longer fails. Test the causal hypothesis, edge cases, and live system separately.

“No message in the bag means no message was published”

It may also mean the topic was not selected, discovered, compatible, or recorded successfully. First verify the recorder's subscriptions and bag metadata.

Retrieval practice

For a robot that turns unexpectedly, name:

• two topics to record;
• one log event to add;
• one bag-inspection check;
• one conclusion replay cannot establish.

Summary

• Start with a falsifiable failure hypothesis.
• Observe and record the smallest sufficient evidence set.
• Inspect the bag before trusting it.
• Replay under documented conditions.
• Compare expected and observed evidence across software and physical layers.

Sources and further study

• rosbag2 repository and usage — official ROS 2 project documentation for recording, inspection, playback, storage, and snapshot behavior; retrieved 2026-08-19.
• ROS 2 interfaces and topic statistics — official documentation for message age and period statistics; retrieved 2026-08-19.
• ros2doctor — official package documentation for system checks and reports; retrieved 2026-08-19.
• Visualizing ROS 2 data with Foxglove — official ROS 2 guide for visualizing live and recorded data; retrieved 2026-08-19.

External sources are linked for learning; their text, diagrams, screenshots, and code are not republished.