The incident is already over when the question arrives. A surgical robot made a movement no surgeon commanded: thirty seconds of motion that ended with an instrument two millimeters from where it should have been. The patient is fine. The procedure finished by hand. Now the reconstruction begins, and the people asking are not the engineering team. The hospital safety board wants a timeline. The insurer wants to know whether the fault was in the machine or the workflow. If it goes further, a regulator or a court asks the question every investigation eventually reduces to. And before any of them, a partner asks it, because your agent's actions were inputs to their agents' decisions. When the outcome is disputed, two organizations are reconstructing the same event, and each one's record is the other's problem.
What did the agent know, and when did it know it?
Most teams believe their logging answers this. It does not.
The question your logs cannot answer
The logs are sampled, because full capture was expensive. They rotate, because storage was finite. They can be edited, because no one designed them to be evidence. And the thirty seconds that matter most were supposed to stream to a server, but the network dropped at the wrong moment, so those seconds exist nowhere at all.
The same shape repeats when a delivery robot and a pedestrian meet badly at a crossing. The city asks for the record. The company produces telemetry. Those are not the same thing. Telemetry is what engineers built to debug the system. A record is what an investigator needs to reconstruct it. As software takes control of physical systems everywhere, the demand for a real account of what that software did is not going away. The harder problem is what the record actually is, and where it has to live.
Evidence has four properties
A black box is not a log with a scarier name. For software, it is a cryptographic audit trail: a record that cannot be quietly altered and that a stranger can verify without trusting you. Four properties separate it from telemetry.
It is complete. Nothing was sampled away to save bandwidth or money. In an investigation, absence of evidence reads as evidence of concealment. When a domestic humanoid leaves a stovetop burning, the first question is whether it perceived the stovetop at all. If perception events were captured one in ten, there is no answer, and "we do not log that" sounds like an admission.
It is ordered. The sequence is the meaning. A delivery robot that stopped and then detected the pedestrian got lucky. A robot that detected and then stopped worked. Same two events, opposite conclusions. A record that cannot fix the order cannot answer the question. This is why conflict-free data types and content-addressable storage matter: they preserve order and integrity without a central clock to trust.
It is tamper-evident. Cryptographically, not procedurally. A record protected by access policy is testimony, because whoever writes the policy could rewrite the record. A record is trusted to the exact degree that no one, including its owner, could have changed it without leaving proof. That standard matters most when the reader works for the other side of the contract.
It is survivable. The record must outlive the event it describes. A failure that destroys the record of the failure has destroyed the answer along with it.
Observability tooling was never built for this. It serves engineers debugging live systems, where sampling is a cost optimization, rotation is hygiene, and mutability is a convenience. Every one of those choices is correct for debugging and disqualifying for evidence.
The record has to live with the agent
Incidents cluster exactly where connectivity, power, or time ran out. A subsea inspection vehicle damages a pipeline fitting at 1,800 meters, where the acoustic link carries kilobits on a good day. An orbital servicing craft bumps the satellite it was docking with, between ground station passes. An agricultural drone sprays the wrong field, three kilometers past the edge of coverage. These are not corner cases. They are the normal operating conditions of the physical world.
So the streamed record has a gap precisely where the answer should be. The last seconds before a failure are the most important and the most likely to be missing, because whatever caused the failure also severed the stream. The server's copy ends right before the part everyone needs to see. Aviation settled this decades ago by putting the recorder in the aircraft. Edge AI has to settle it the same way: the agent's own view of the world, in the order it formed, held locally, surviving restarts, damage, and the network being gone.
Agent memory and the black box are the same layer
A complete history of what the agent perceived, decided, and did. Strict ordering. State that can be trusted locally and proven later. A store that survives disconnection. That is not a compliance checklist. It is the requirements list for agent memory itself. An agent's decisions come out of its state, so the state layer and the evidence layer are the same layer. Build the memory right and the black box is not a second system, it is the same system held to a higher standard.
This holds even when nothing has a motor. A procurement agent commits funds it was never authorized to commit, and the supplier's fulfillment agent has already acted on the order. Now two organizations need the same reconstruction, and each needs the other to believe it. A signed, ordered, complete record of its own reasoning is the difference between an audit and an argument.
The alternative is worse than it sounds. A team that bolts a recorder onto cloud-first logging runs two systems, the state the agent acts on and the record it reports, and every gap between them is a place for doubt to live.
Whoever builds the agent's memory builds its black box. In mission-critical deployments that is not a technical detail. It is the license to operate. The deployers who can answer the question keep deploying. The ones who cannot get grounded.
At Source, we are building DefraDB, the distributed, local-first database built for the agentic edge.
