Product capability monitoring for app teams.
Updated Read as Markdown ↗
Start with an outcome your users need. Connect it to the code and evidence that explain what happens. Give that capability one primary metric and a few guardrails.
Past observations establish the history. Current measurements identify the gap. The Advisor can recommend a target and a course of action; the owner decides what to pursue. An accepted recommendation does not grant release permission.
Onboarding flow
“Create my first project without getting stuck.”
Sign-up → email verification → Create project → Shared session context
Accounts activated within 24 hours
- Definition
- Activated accounts / eligible new accounts. 1,000 mature accounts per sample; exclude test and bot accounts.
- Evidence
- Funnel events + server logs + reproduced session failure
- Observation
- Session refresh loses the workspace ID.
- Advisor proposal
- Restore session continuity, then test removing one optional onboarding step. +2 percentage points. Target for the next 30 days.
- Guardrail
- Project creation p95 <2s
Follow shared issue F-17 through repair, verification and recurrence →
AI assistant
“Have the agent complete the action I asked for.”
Chat interface → Tool router → Shared session context
Successful tool calls
- Definition
- Tool calls with a validated result / eligible tool calls. 1,000 calls per sample, same tool mix.
- Evidence
- Tool traces + permission errors + shared cause confirmed in F-17
- Observation
- Tool requests lose the same workspace context.
- Advisor proposal
- Fix session-related failures and validate tool arguments before dispatch. +4 percentage points. Target for the next 30 days.
- Guardrail
- Tool response p95 <3s
Follow shared issue F-17 through repair, verification and recurrence →
Server reliability
“Finish the action without a server error.”
App requests → API services → Structured logs + traces
Requests without a server error
- Definition
- Requests without a 5xx response / eligible production requests. 100,000 requests per sample, same endpoints.
- Evidence
- Request logs + trace IDs + synthetic checks; example sample only
- Observation
- Error logs identify where requests fail. A quiet log alone is not proof of recovery.
- Advisor proposal
- Rank error clusters by affected user flow; repair the recurring timeout first. +0.4 percentage points. Target for the next 30 days.
- Guardrail
- API response p95 <500ms
Analytics dashboard
“Open a report and trust that it is current.”
Report screen → Query service → Warehouse + event pipeline
Report load time, p95
- Definition
- p95 across 1,000 report loads per sample. Same report mix and device segment.
- Evidence
- Browser timings + query traces + ingestion freshness checks
- Observation
- The event pipeline is current; slow queries still delay report loading.
- Advisor proposal
- Optimize the slowest query path and verify that cached reports stay fresh. At least 0.8s faster. Target for the next 30 days.
- Guardrail
- Data freshness <15 minutes
A passing test and a better product are separate results.
First verify the released repair independently. Then compare mature samples using the same metric definition and population. Record missing evidence or an inconclusive result. A before-and-after increase does not, by itself, establish that the repair caused the change.
Use the capability-success commands to define contracts and record measurements. See current availability for the remaining Advisor-to-work integration and full product-outcome pilot.