Map, fix, verify, watch for a return.
Updated Read as Markdown ↗
An app needs new users to finish onboarding, AI tools to complete their tasks and reports to load reliably. Factory maps those capabilities, follows their history and gives each one a measurable next target.
Want the execution details? Inspect all 12 example signals, then follow one investigation from checkpoint to verification.
The product map
A map links business capabilities to journeys, screens, agents, services, data pipelines and external dependencies. Repository history helps reconstruct what happened before. Connected runtime evidence tells you what is happening now.
Code existing in a repository does not mean it is released, enabled or being used. Missing evidence remains visible rather than becoming a reassuring green status.
Two kinds of useful work
Make it reliable
Existing capability → reported problem → underlying cause → change → release → verification.
Make it more useful
User need → opportunity → hypothesis → experiment or feature → release → outcome measurement.
The second path is part of the wider product loop under development. A new opportunity should not have to masquerade as a bug to be worth doing.
One problem, with its context
A failed sign-up might appear in feedback, a runtime error and a GitHub issue. Those can be evidence about the same problem. Grouping them avoids three agents independently trying to fix one cause.
The owner’s decision matters
The Advisor makes recommendations from available facts. Decisions are recorded durably. Accepting a recommendation and authorizing execution are separate steps; the project’s permission settings still govern what may happen.
The current decision-to-work bridge promotes an accepted canonical-problem decision through explicit checks. Capability-level recommendations do not automatically become executable work. Inspect owner-decision tools for promotion and work status, or follow the execution gates.
Verification is a separate job
A merged pull request is evidence that code changed. A release shows it was deployed. A behavioral check shows whether the fix works. A product measurement helps assess whether it was useful. Factory keeps these distinctions.
One main metric, up to two supporting measures
For sign-up, that might be completed registrations. For a conversation agent, helpful answers. Each measurement needs a definition, population, time window and enough fresh data to support the conclusion.
A small set of guardrails can protect other outcomes. “We cannot measure this yet” is a valid result.
In the public board, select a metric to change its chart, target and Advisor suggestion. Chat quality is a reply score out of 100; Cost per reply includes all model calls and retries; Tool success measures whether required actions finish correctly. Fewer tools or turns is not itself a better outcome.
Automations follows a reminder through email or in-app delivery to a completed user action. Paywall & upgrades follows the same visitor from first view to a confirmed payment, with a full conversion window. Delivery is not completion; a checkout event is not a confirmed paid subscription. Definitions, populations and missing evidence stay in the metric details.
The date range filters charts and recent activity. It never rewinds current work or hides an outstanding decision. Problem histories retain their own dates. P1, P2 and P3 describe urgency; “Needs you” is reserved for a decision or authorization that requires the owner.
Follow the same capability over a week
The homepage uses an illustrative issue, F-17. Funnel events, server logs and a reproducer show that session refresh loses the workspace ID. This blocks onboarding and AI tool calls that depend on the same context.
| When | What changes | What Factory retains |
|---|---|---|
| Day 1 | The onboarding capability is mapped. Activation is 62%. | User goal, dependencies and initial measurement |
| Day 2 | The cause is reproduced; an approved patch is released. | One issue, confirmed cause, scope, patch and release |
| Day 3 | An independent check passes on the release. | A verified repair cycle; a separate product measurement |
| Day 6 | A retry route reproduces the same missing-context cause. | F-17 reopens; the previous fix remains in its history |
| Day 7 | The retry repair is independently verified. | One issue, two verified repair cycles, one recurrence |
A similar complaint alone does not establish recurrence. The new evidence must connect it to the confirmed cause. A different cause should be tracked separately.
The example’s 62%, 63.4%, 62.7% and 64% onboarding conversion rates are illustrative, not customer results. Each cohort contains 1,000 eligible new accounts with a full 24-hour observation window. Activation means creating a first project; tests and bots are excluded. The guardrail is project creation p95 below 2 seconds. Before/after movement does not establish causation.
Past, present and the next target
Each capability keeps its metric definition, denominator, sample window, baseline and target. The homepage uses four examples:
- Onboarding: 62% to 64% activation within 24 hours. This is +2 percentage points, not a 2% relative increase.
- AI assistant: 91% to 95% successful tool calls with validated results, using the same tool mix. This measures tool reliability; answer usefulness needs its own evaluation.
- Server reliability: 99.5% to 99.9% of requests without a 5xx, with latency as a guardrail. Logs help locate the cause.
- Analytics: report load p95 from 1.8 seconds to below 1 second, while data stays less than 15 minutes old.
Targets are proposals for the next 30 days. The owner reviews the scope and hypothesis before work begins. The coding-agent tools show how to read measurements, record an expectation and compare the outcome.
Memory carries the work forward
Keep causes, decisions, attempted changes and observed outcomes with the project. Turning those lessons into automatically promoted operating rules is a later roadmap step.
How a lesson would earn the right to become a rule
Planned, not shipped. The four steps below are the review gate a candidate rule has to pass before it can guide later work.
- Repeated evidence. The same session failure caused two repair cycles. Both reproductions and both fixes stay attached to F-17. One successful patch is not a universal lesson.
- Candidate lesson. “Verify workspace context survives onboarding, tool calls and background retries.” A proposal, not a permanent instruction.
- Shadow check. Run it without expanding agent permissions. Record missed cases and false alarms before trusting it.
- Reviewed rule. Promote only after reading the evidence. Keep its version so it can be corrected or retired.