Engineering
A Passing Test Is Not Production Proof
A green test proves something specific happened in a controlled environment. It does not prove a design was approved, a deployment gate passed, or production is behaving the same way.
One of the easiest ways to lie to yourself in software is with a green checkmark.
The test passed.
The build is green.
The migration ran.
The feature exists in a branch.
Those are all useful facts.
They are not the same fact.
We have been forcing that distinction very explicitly while building CIIMS, because financial software gets dangerous when evidence from one layer is promoted into a claim about another.
A passing test can prove a behavior under the conditions of that test.
It cannot prove that the design is approved.
It cannot prove that every dependency gate is closed.
It cannot prove that a draft migration is authorized for production.
And it definitely cannot prove that production is behaving correctly.
We needed more than “done” and “not done”
The useful states look more like this:
design resolved → implemented → tested → gate passed → packaged for deployment → deployed → operationally verified
Each step answers a different question.
Design resolved means the contract is specific enough to describe what should exist.
It does not mean code exists.
Implemented means somebody wrote the code or migration.
It does not mean the behavior is correct.
Tested means an executable check exercised a specific behavior in a controlled environment.
It does not mean every acceptance condition passed.
Gate passed means the complete set of required evidence for that slice has been reviewed and accepted.
It does not mean the thing is deployed.
Deployed means the authorized artifact reached the target environment.
It still does not mean the production outcome is healthy.
Operationally verified means the real system produced the expected result under real operating conditions and there is evidence to support that claim.
Those states sound obvious when written out.
They get blurred constantly in real projects.
The database test is a good example
CIIMS has a disposable database harness.
It can reset an isolated test database, apply migrations from empty, load synthetic fixtures, and run database checks.
That is exactly what we want.
If a draft schema cannot survive a clean migration or fails its security tests in an isolated database, there is no reason to talk about deploying it.
But the reverse is not true.
Passing that harness does not automatically promote the draft into production.
The CIIMS implementation baseline is explicit about this.
Draft migrations and isolated tests are allowed for dependency-complete slices.
Production Financial Core DDL and deployment can still remain blocked.
That means both of these statements can be true at the same time:
The database tests pass.
and
The production gate is still closed.
There is no contradiction.
They are claims about different layers.
A test catalog is not a test result
There is an even subtler version of the same mistake.
You can define an excellent acceptance suite.
S-01.
S-02.
S-03.
All the way through S-28.
Every important invariant has a name.
That feels like progress because it is progress.
But the existence of the checklist does not mean the checks passed.
The CIIMS baseline says this plainly: defining the acceptance registry does not make the tests pass.
That sounds almost too obvious to write down.
It was worth writing down anyway.
Because AI-assisted engineering can create an especially convincing illusion of completeness.
A model can produce:
- a detailed schema
- an impressive test catalog
- migration files
- ADRs
- fixtures
- CI configuration
- dozens of green local checks
very quickly.
The surface area grows faster than a human’s intuition for what has actually been proven.
The antidote is not less automation.
It is better evidence labeling.
“Runs” is not the same as “PASS”
Another useful distinction showed up in the CIIMS D1 work.
A lot of security and database checks now run against draft migrations and synthetic fixtures.
That is real evidence.
But the D1 acceptance record still says the named gate PASS is incomplete.
It also says that running those checks does not automatically:
- lift the parent-schema gate
- package the drafts into deployment migrations
- close unresolved infrastructure decisions
- authorize production deployment
That is the kind of sentence I want in serious engineering documentation.
Not because it is cautious for the sake of sounding cautious.
Because it makes it very hard to accidentally upgrade a local success into a production claim.
Production needs different evidence
A production claim has to answer questions the test harness cannot.
Did the authorized artifact actually deploy?
Did it run against the intended production configuration?
Were the real identity and authorization boundaries in place?
Did the data migration preserve the invariants it was supposed to preserve?
Did the system behave correctly under real concurrency and failure conditions?
Can it recover from backup?
Do the monitoring and audit records show the expected result?
A local PostgreSQL container cannot answer those questions.
Neither can a CI job.
Those tools are part of the evidence chain.
They are not the end of it.
Why this matters more with AI coding agents
AI makes implementation cheap.
That changes the bottleneck.
The hard part is increasingly not:
Can we generate the code?
It is:
What does this code actually prove?
If an agent can create five migrations, four ADRs, twelve tests, and a polished README in an afternoon, the project needs a vocabulary strong enough to say:
- proposed
- specified
- implemented
- test-only
- passing locally
- passing in CI
- blocked by dependency
- approved
- deployed
- verified in production
without collapsing those into done.
That vocabulary is governance for machine-speed development.
The larger lesson
Green is not a universal status.
It only means something in context.
A green unit test tells you the unit test passed.
A green migration test tells you the draft migrated under that harness.
A green CI run tells you the configured checks passed for that commit.
A production-ready claim requires more.
So does a production-proven claim.
The engineering habit I want to keep is simple:
Every claim should name the layer of evidence that supports it.
Not because every project needs a giant governance process.
Because the faster we can build, the easier it becomes to confuse activity with proof.
A passing test is good news.
It is just not the same thing as production proof.