Human intent is the missing signal

Mark the Moment

A human-anchored pattern for agent debugging: pair timestamped human intent with an ordered semantic trail, then let an agent draft a reproducible scenario without guessing what mattered.

01Markersupplies intent
02Trailsupplies context
03Agentsupplies a draft—not proof

The lifecycle

From “that was wrong” to a testable hypothesis

A useful recorder does not capture everything and hope an AI finds the problem. It gives a tester a precise join point between their judgment and the machine-observed path around it.

  1. Record context

    Start a bounded session with a clear clock origin and only the metadata needed for reproduction.

  2. Mark the moment

    Drop a bug, note, or moment with a relative timestamp and short human explanation.

  3. Preserve the trail

    Retain coarse screens, semantic actions, selected failures, and errors—not raw secrets.

  4. Review and decide

    Stop capture, inspect the session, then explicitly submit or discard it.

  5. Export evidence

    Validate metadata, markers, and trail events; merge them into deterministic order.

  6. Draft a scenario

    Ask an agent for preconditions, steps, expected behavior, reported observation, and assertions.

Text equivalent: a permitted human records context, marks a significant instant, preserves the preceding trail, reviews the capture, exports a typed timeline, and gives it to an agent for a reviewable draft.

Two signals

Neither is enough alone

A marker says why this moment deserves attention. A trail says how the tester arrived there.

Marker without trail

“Something went wrong here.” Strong human intent, weak reproduction context.

Trail without marker

“Many things happened.” Strong chronology, weak understanding of what mattered.

MarkerThis mattered.
TrailThis path led to it.
Draft scenarioHere is a hypothesis to verify.

Evidence layers

Each layer has a job—and a limit

This pattern is grounded in a working implementation. The recommendations below generalize its strongest boundaries without claiming that any single signal proves causality.

Session metadata

Defines the container: platform, timing, lifecycle, build label, and optional test intent.

Does not prove: that a defect occurred.

Marker testimony

Captures what the tester considered significant and what they reported at that instant.

Does not prove: cause, correctness, or completeness.

Interaction trail

Provides bounded, semantic breadcrumbs such as screens, stable action labels, and selected failures.

Does not prove: that uncaptured events never happened.

Optional replay

Adds access-controlled visual context when available. The structured envelope still stands without it.

Does not prove: durable state or root cause.

Fictional example

A reconnect-and-save failure

All actors, screens, labels, times, and references below are placeholders. The example separates recorded evidence from inference.

Recorded session

actor: <tester-alias>
build: <build-label>
replay: <optional-replay-reference>

00:00screen · settings
00:12tap · save-control
00:13network · request-failed: timeout
00:31custom · connectivity-restored
00:33tap · save-control
00:34BUG marker
“Save looks enabled, but nothing changes after reconnect.”

Agent-drafted scenario

PreconditionsSettings is open. A first save experiences a temporary timeout. Connectivity returns.
Steps
  1. Change a non-sensitive setting.
  2. Activate save and simulate a timeout.
  3. Restore connectivity.
  4. Activate save again.
ExpectedThe retry sends one new request and reaches a confirmed saved state.
Reported observedThe control appeared enabled, but the tester reported no visible change.

Provenance: marker at 00:34; supporting trail window 00:12–00:33.

Trust boundaries

Validate at every handoff

The client captures untrusted events. The server enforces policy. The export carries reviewed evidence. The agent produces an interpretation that still needs independent validation.

  1. Humansupplies intent
  2. Clientcaptures untrusted semantic events
  3. Serverchecks capability, owner, state, and bounds
  4. Exportvalidates and orders reviewed evidence
  5. Agentdrafts a candidate interpretation
  6. Engineerverifies reproduction and assertions

Core invariants

  • A dedicated capability is checked on the server.
  • Session writes are owner-scoped and lifecycle-aware.
  • Marker testimony and trail events are append-only.
  • Notes, labels, offsets, and batches are bounded.
  • Resources and exports cross typed validation boundaries.
  • Stop leads to review; submit and discard are explicit.
  • Operational logs use safe event fields, never raw testimony.

Text equivalent: human intent flows through an untrusted client into a server policy boundary; reviewed records become a typed export, an agent drafts a hypothesis, and a person or test runner verifies it. Visual replay is an optional side reference.

Failure modes

Design for missing, noisy, and sensitive evidence

“Structured” does not automatically mean safe, ordered, complete, or true.

Privacy capture

Deny capture on sensitive screens. Never retain raw field values, request bodies, tokens, or full URLs. Redact before agent access and define retention separately.

Noisy trails

Prefer stable semantic identifiers, deduplicate repetition, cap volume, and show a bounded window around each marker.

Clock and order errors

Use monotonic offsets plus a per-session sequence and stable secondary ordering. Relative time alone cannot settle ties.

Stale sessions

Restore active state on reload, permit one live session per actor, and close abandoned sessions with an explicit reason—never silent submission.

Lost writes

Show pending versus synced state, acknowledge final flushes, use idempotency where needed, and represent known capture gaps.

Agent overreach

Label testimony, recorded facts, and inference separately. Preserve provenance. Never present a draft scenario as root-cause proof.

Adoption ladder

Start with the policy boundary, not replay

The smallest valuable version is marker-only recording with explicit review. Rich capture can arrive later.

Policy boundary

Dedicated capability, owner-scoped sessions, one active session.

Exit gate: bypassing the UI still cannot bypass authorization.

Marker recording

Timestamped bug, note, and moment testimony through an explicit lifecycle.

Exit gate: reload preserves recording or review state.

Semantic trail

Screen transitions and stable action labels without sensitive values.

Exit gate: markers show a concise ordered context window.

Reviewed envelope

Typed metadata, markers, and trail with deterministic ordering.

Exit gate: malformed or foreign-session exports fail closed.

Agent drafting

Preconditions, steps, reported observation, assertions, and provenance.

Exit gate: inference cannot be mistaken for executed proof.

Optional replay + ops

Visual context, bounded inspection, retention controls, and redaction checks.

Exit gate: the core workflow still works without replay.