Imagine a software factory that delivers exactly what we ask for.
It turns requests into implementations, verifies the changes, releases them, and learns from their behavior. Customer complaints become fixes. Popular features receive improvements. Delivery keeps getting more efficient.
That would be a significant achievement.
But where does the next ambitious idea come from?
Who imagines a product customers have never experienced? Who challenges the workflow everyone has accepted? Who sees that a new technical capability could make an old category of work unnecessary?
A factory can become excellent at improving the current product while never questioning what product should exist.
In Path to the Dark Factory, I explored increasingly autonomous software delivery. The recent velocity articles, on teams and companies, examine what lets teams and companies make sustainable progress. The next question is how that capacity finds worthwhile direction and becomes wiser through use.
My proposed model has three connected activities: vision creates possibilities, experiments investigate them, and feedback changes what we believe and do.
Customer feedback is one essential source of evidence within that model. The factory also needs technical exploration, human imagination, domain insight, and decisions about a future that current usage cannot yet reveal.
Give the factory more than a queue of requests
Customers can explain their frustration, show us where work breaks down, and tell us what consequences matter. Their requests also reflect the products, constraints, and alternatives they know.
If we only optimize around those requests, we can inherit those constraints without examining them.
A customer might ask for a better export. Watching their work could reveal that they export data only to reconcile it manually somewhere else. A broader product idea might remove that reconciliation step entirely.
The request deserves attention. So does the possibility that fulfilling it beautifully leaves the larger problem untouched.
Ideas can begin with several different kinds of insight:
| Starting point | A question it can introduce |
|---|---|
| Customer experience | What are people struggling to accomplish? |
| Technical possibility | What has become feasible that was previously impractical? |
| Domain expertise | Which accepted assumption could we now challenge? |
| Strategic conviction | What future do we believe is worth building toward? |
| Creative exploration | What would happen if we approached this differently? |
| Internal engineering experience | Which recurring difficulty suggests a different architecture or product boundary? |
These sources can reinforce one another. An engineer sees a new capability. A designer imagines an interaction. A customer conversation reveals why it might matter. A product leader connects it to a coherent direction.
The operating model needs somewhere for that combination to happen before a detailed requirement exists.
Vision should create a testable direction
Vision gives us a reason to explore beyond what current metrics reward.
Consider a fictional company whose product helps teams build weekly status reports. Customers request richer templates and more integrations. The company could serve those requests well.
Someone proposes a different future: teams should rarely have to assemble a status report because an evidence-backed account of progress should already exist.
That idea is more than an improvement to the report editor. It changes the workflow the company wants to support.
Customer feedback did not have to originate the idea. Customers still matter enormously when evaluating it. Would the account be trustworthy? Can people correct it? Does it capture meaningful progress? Does it create an unacceptable feeling of surveillance? What should remain a human conversation?
The vision establishes a direction. Those questions identify what we need to learn.
I would express an ambitious idea through four elements: the future we want to enable, the reason it could matter, the assumptions that must hold, and the evidence that would cause us to revise it.
We do not need to pretend that an early vision is already validated. We do need to distinguish conviction from certainty.
A useful vision can survive an unsuccessful implementation. A responsible team can also abandon a vision when its central assumptions repeatedly fail.
Experiments connect imagination to evidence
The software factory can help explore an idea long before it is ready for a production roadmap.
For the reporting example, the first experiment might be a manually assembled prototype tested with a small group. Its purpose could be to learn whether people find the account useful and where they distrust it. Building a fully automated product would be premature if we have not answered that question.
A later experiment could evaluate whether the available systems contain enough evidence to produce an accurate account. Another could test a correction workflow. Only then might a limited deployment investigate sustained use.
Manual prototype with a small group → Is there enough evidence in the available systems? → Correction workflow → Limited deployment for sustained use
Each experiment should target an important uncertainty. Producing more prototypes does not guarantee that we are learning more.
AI can lower the effort of exploring alternatives. Human attention, customer access, integration work, and the consequences of a real release still impose limits. Choose experiments according to what their results can change.
At every stage, separate what we have demonstrated from what remains a belief. A compelling demo demonstrates something different from reliable operation; enthusiastic first reactions demonstrate something different from lasting value.
The factory becomes more useful when it helps us make those distinctions explicit.
What actually closes a loop?
We already have plenty of feedback mechanisms: compiler errors, reviews, test results, monitoring, support tickets, customer interviews, and retrospectives.
Their existence does not tell us whether the loop works.
Suppose a dashboard shows a rise in failed onboarding attempts. Someone notices. A report goes into a weekly meeting. Everyone agrees it deserves attention. The roadmap stays the same.
We observed something. We have not yet changed the system producing it.
I would describe a complete feedback loop through six questions:
- What did we expect to happen?
- What actually happened, and how trustworthy is the evidence?
- What does the difference suggest?
- Who can decide what to do about it?
- What changes as a result?
- How will we discover whether that change helped?
The last question matters. Acting on feedback can still make things worse. A plausible explanation is a hypothesis until we examine what follows from it.
A loop can also close with a deliberate decision to keep the current behavior. That decision should carry a reason and, where uncertainty remains, a condition for reconsideration. Otherwise, “we reviewed it” can become a polite way to ignore evidence.
The factory needs several loops
A software factory makes different kinds of decisions at different timescales. Each needs evidence appropriate to its consequences.
| Loop | Question it must answer | Evidence that can change the next decision |
|---|---|---|
| Implementation | Does this change satisfy the intended behavior? | Failed checks, independent scenarios, review findings |
| Runtime | Does the released system behave acceptably under real conditions? | Errors, latency, cost, data correctness, dependency behavior |
| Customer outcome | Can people accomplish the job we intended to help with? | Task completion, customer observation, support patterns, adoption |
| Exploration | Could a new approach create a worthwhile capability? | Prototypes, feasibility studies, observed behavior, challenged assumptions |
| Factory improvement | Why does this delivery process keep requiring correction? | Repeated interventions, escaped defects, failed attempts, recurring ambiguity |
| Company direction | Does the evidence still justify this investment? | Outcomes relative to cost, changed assumptions, competing opportunities |
These loops inform one another, but their conclusions are different.
A successful deployment tells us that a change reached production. Stable operation gives us evidence about its behavior. Neither establishes that the customer needed it.
Likewise, a useful feature does not prove that the delivery process was economical or sustainable. People may have spent days quietly repairing the automation behind it.
The factory needs enough visibility to distinguish these outcomes.
Fast feedback can still be wrong
Speed matters when it reduces the amount of work built on an incorrect assumption. But accelerating a weak evaluation can reinforce the same mistake more efficiently.
Consider an agent that interprets a requirement, implements that interpretation, and generates tests from the resulting code. Everything agrees. The original misunderstanding can remain intact across all three artifacts.
An additional reviewer agent may help, but it does not automatically create independent evidence. If it receives the same incomplete context and accepts the same assumptions, agreement tells us less than it appears to.
I would strengthen the evidence at the point where those assumptions can be challenged: examples defined with a domain owner, explicit exclusions, known failure cases, or observation of the real workflow.
The question is what could prove the implementation wrong.
A useful check needs that possibility. If every artifact is allowed to adjust itself until the pipeline turns green, the factory can lose the ability to detect a mistake while appearing increasingly successful.
Release criteria therefore need ownership and enforcement outside the component being evaluated. Changes to those criteria deserve their own review.
Follow one failure all the way back
Consider a fictional reporting service that produces monthly customer exports.
The factory ships a change. The files generate successfully. Runtime metrics look normal. A customer reports that transactions near midnight appear in the wrong month.
The implementation used UTC. The customer expected the reporting period to follow the account’s configured timezone.
There are several possible responses.
The immediate response is to correct the affected behavior and assess what happened to previously generated reports. Releasing a fix does not repair files customers have already downloaded or decisions they have already made from them.
The next response is to add a regression case covering the boundary. That helps prevent the same implementation defect.
Then comes a deeper question: where should the timezone requirement have been available before implementation began?
If it was missing from the product contract, update the contract. If it existed but the agent did not retrieve it, change context selection. If a reviewer had explained it repeatedly, investigate why that explanation never became a maintained requirement. If the test environment could not represent different account timezones, improve the environment.
Each explanation leads to a different intervention.
I would retain the failure as an evaluation case, along with the reason for the correction. Then I would check related cases the repair could break, such as daylight-saving transitions and accounts that change timezone.
Finally, verify the customer outcome: can the customer now produce and use the intended report?
That one incident can improve the product, the acceptance criteria, the context supplied to agents, and the evaluation of future factory changes. It does so only if somebody owns those changes and checks their effect.
- 01 · IncidentTransactions near midnight appear in the wrong monthThe implementation used UTC. The customer expected the account's configured timezone.
- 02 · Immediate responseCorrect the affected behaviorAssess what happened to previously generated reports.
- 03 · Next responseAdd a regression caseCover the boundary so the same implementation defect does not return.
- 04 · Deeper questionWhere should the timezone requirement have been available before implementation began?
- Missing from the product contractUpdate the contract
- It existed, but the agent did not retrieve itChange context selection
- A reviewer had explained it repeatedlyInvestigate why it never became a maintained requirement
- The test environment could not represent account timezonesImprove the environment
- 05 · RetainKeep the failure as an evaluation caseWith the reason for the correction. Check related cases: daylight-saving transitions and accounts that change timezone.
- 06 · VerifyCan the customer now produce and use the intended report?Only if somebody owns these changes and checks their effect.
Fixing the work and improving how work is produced
Every factory needs a way to correct individual outputs. It also needs a way to change the conditions that keep producing those outputs.
An engineer fixes an agent’s patch. A reviewer supplies a missing business rule. A release owner spots an unsafe assumption. The work eventually ships.
From the delivery dashboard, the process can look successful. From the engineer’s perspective, the same rescue happens every week.
I would treat repeated intervention as evidence about the factory itself.
What knowledge was missing? Which instruction was ambiguous? Which tool was unreliable? Which decision required authority the system did not have?
Record the reason in a form that supports a specific improvement. “Human review required” is usually too broad to be actionable.
Learning here does not necessarily mean retraining a model. It may mean changing a specification, a retrieval rule, a test environment, an interface, or the scope of delegated authority.
Those changes need evaluation too. A new instruction that fixes one case can damage another. Preserve earlier failures, use representative cases that were not used to design the correction, compare configurations, and keep a way to restore the previous behavior.
This is how improvement can accumulate without turning every recent complaint into a permanent new rule.
Protect exploration from the wrong scorecard
An established workflow and a speculative idea should not have to prove themselves in the same way at the same moment.
| Established workflow | Speculative idea | |
|---|---|---|
| What it brings or must show | Users, known expectations, and operational obligations | May first need to show that a technical obstacle can be overcome, or that a different interaction makes sense to anyone |
| Under the wrong scorecard | Has a structural advantage if both compete only on immediate adoption or near-term revenue | If exempt from scrutiny indefinitely, can consume capacity without producing useful understanding |
I would give exploration a bounded commitment: an owner, a question, a time or spending limit, and a review point. Judge the early work by the uncertainty it resolves and the options it opens. As evidence strengthens, move toward customer outcomes and operational requirements.
An exploration that disproves a costly assumption can be worth its budget. Calling every unsuccessful attempt “learning” without stating what changed makes that judgment impossible.
There is no universal allocation between maintaining the present and investigating the future. Leadership should choose it deliberately. Otherwise, urgent requests can absorb all available capacity, or an exciting vision can leave existing customers carrying the cost.
The factory needs permission to investigate something nobody requested. It also needs a disciplined way to decide whether that investigation deserves another investment.
Every loop needs authority and capacity
An organization can discover the right answer and still fail to act on it.
| Who sees it | What they know | Why nothing changes |
|---|---|---|
| The team | A shared API causes recurring defects | Another team owns it |
| Product | An experiment failed | The annual commitment assumes it will expand |
| Support | A workflow is confusing | Nobody has capacity to change it |
The delay sits between evidence and action.
For each consequential loop, I would name an owner and define the decisions that owner can make. What can change locally? What requires escalation? Who resolves a conflict? What happens when the owner is unavailable?
Capacity belongs in this design as well. If all engineering time is committed to new features, a factory-improvement backlog can grow indefinitely while everyone agrees that it matters.
The company must reserve some ability to respond to what it learns. The right amount depends on the work, but a plan that assumes no correction will be needed is already making a consequential bet.
The company velocity discussion extends this to funding and commitments. Learning becomes organizational progress when the people with authority can redirect resources in response.
Shorten avoidable delay; respect observation time
It is tempting to ask for every feedback loop to run faster.
Some should. Other conclusions take time.
| Should arrive promptly | Takes time to observe |
|---|---|
| A deterministic failure should reach the implementer promptly | A customer may need to complete a monthly workflow before the effect of a change becomes visible |
| A release that crosses a predefined operational limit should trigger the agreed response without waiting for a status meeting | Adoption can depend on training or rollout |
| A brief period of quiet traffic may provide little evidence about peak behavior |
Reacting to every fluctuation can make a system unstable. The factory changes something, sees a noisy early result, changes it again, and loses the ability to understand either intervention.
I would define the observation window and decision criteria before the rollout where practical. Include the conditions that justify acting early, such as a serious correctness failure.
Also distinguish silence from success. If required telemetry is missing, the result is unknown. If the relevant customer cohort has barely used the feature, confidence should reflect that exposure.
The target is the shortest time to evidence strong enough for the decision at hand.
Measure whether learning changes behavior
Counting dashboards, tests, or retrospective actions gives us an inventory of mechanisms. To understand whether those mechanisms work, follow a few important loops through to their consequences.
| Measure | What I would investigate |
|---|---|
| Time from a relevant event to detection | Whether the system notices important differences promptly |
| Time from accepted evidence to a decision | Whether ownership, disagreement, or authority creates delay |
| Time from a decision to a verified correction | Whether the organization can implement and assess its response |
| Recurrence of an understood failure | Whether learning survives into later work |
| Repeated human interventions by cause | Where delivery still depends on people repairing the same gap |
| Improvements that deliver their expected effect | Whether our explanations lead to useful changes |
Keep comparable work together. Include failed attempts and abandoned runs. Show exposure and uncertainty when failures are rare. A small number of easy changes cannot establish that a factory handles difficult ones reliably.
These measures should help identify where a loop breaks. They should not pressure people to close observations before the outcome is known.
A thoughtfully rejected proposal can represent better learning than a hurried change made to improve a closure metric.
People shape the quality of the signal
A feedback system depends on what people are willing and able to tell it.
If raising a concern damages a person’s standing, important information may arrive late. If stopping a weak experiment counts as failure, teams have a reason to keep it alive. If the only recognized contribution is shipping, the person who prevented unnecessary work can disappear from the story.
Leaders influence the signal through their response to it.
A customer revealing a problem, an engineer challenging an assumption, and a team admitting uncertainty all give the organization an opportunity to improve. That opportunity needs attention and a credible response.
This connects to AI-Free Fridays. Reflection, conversation, and recognition create space to understand what happened. They also have value to the people doing the work, beyond their effect on delivery.
We should study successes as well as failures. Which decision made a migration uneventful? What helped a customer finish a difficult task? Which collaboration prevented rework?
Understanding why something went well helps us preserve it. Taking time to appreciate it helps people experience the accomplishment together.
Build both capabilities deliberately
I would begin with two small pieces of work.
| First | Second | |
|---|---|---|
| Choose | A recurring failure or intervention | An idea that could change something meaningful but is absent from the current request queue |
| Work through | Trace where it becomes visible, who can act, what changes, and how anyone checks whether the correction worked | Write down why it matters, its most consequential uncertainty, and the smallest credible experiment that could change your view |
| Then | Repair the missing connection and observe the next comparable work | Give it an owner and a bounded commitment |
The first improves our ability to learn from what exists. The second improves our ability to discover what could exist.
Both need judgment. Both need evidence. They start from different places and may run on different timescales.
A software factory can make it easier to act on an idea. Leadership still has to create the conditions in which worthwhile ideas emerge, receive a fair investigation, and change direction when reality challenges them.
I want a factory that helps us deliver reliably, imagine ambitiously, and learn honestly.
The questions I would bring to its next review are:
What did reality teach us that changed how we work? And what possibility are we investigating that reality has not yet had the chance to show us?