The Signal · Evaluation explainer · September 29, 2026

An evaluator can only examine what it can see. But access alone does not establish independence, and independence alone does not tell a buyer which failures were tested.

On September 18, Anthropic announced a partnership with Accenture on embedded evaluation, led by Faculty, Accenture’s specialist AI business. The planned work includes red-teaming, alignment assessment, and safeguards testing. Anthropic describes access comparable to an employee’s, rather than a narrow external testing window.

The announcement also states that operating details and reporting standards remain unsettled. Anthropic will fund Accenture’s work directly, and the arrangement is non-exclusive. This is a dated development at the edge of our coverage window, not a new September 29 announcement.

For a buyer, the substantive question is what this deeper access will eventually allow an evaluator to establish and communicate.

Separate three forms of assurance

First, there is access: which models, systems, people, and development decisions the evaluator can inspect.

Second, there is independence: whether the evaluator can choose meaningful questions, challenge conclusions, and disclose material findings under its agreement.

Third, there is relevance: whether the tested behavior bears on the workflow you intend to run.

These dimensions overlap, but none substitutes for the others. An evaluator with broad access might still face disclosure constraints. A transparent report might examine risks that are different from your operational problem. A relevant test might depend on a configuration you will not use.

The right response is to ask better questions, not to discard evaluation or treat a partnership announcement as a safety certificate.

What a buyer should request

Ask for the version and configuration evaluated, the scope of the work, and the date. Ask how failures were defined and whether the evaluation included the surrounding tools and permissions.

Then ask what the report does not cover. A model-only test cannot establish that your integration correctly maps customer identities. A prompt-injection evaluation does not prove that your invoice workflow avoids duplicate writes. A strong average result can conceal a failure mode that matters disproportionately to your business.

If the full report cannot be shared, request a usable summary of methods, limitations, and unresolved findings. The absence of public detail should remain an uncertainty in your decision, not become an assumption that everything passed.

A hypothetical procurement decision

Imagine a regional service company considering an AI assistant for internal policy questions. The vendor provides impressive model evaluation results and a description of independent oversight.

The company still needs to test its own documents. Which policy version does the assistant retrieve? Does it distinguish a general guideline from an exception? Can it say that the answer is absent? Does it show the source well enough for an employee to verify it?

An evaluator’s work may improve confidence in the underlying provider. It cannot remove the need to examine these local requirements. The company should run a bounded trial with known-answer questions, ambiguous cases, and deliberately missing information.

This is a hypothetical buying process, not a finding about Anthropic, Accenture, or a Xovion customer.

Evidence needs an expiry condition

An evaluation is a snapshot. If the model, tools, instructions, or permissions change, ask which conclusions still apply.

For an internal deployment, keep a small record of the configuration and the acceptance cases you used. Re-run the important cases after material changes. Add new cases when an actual failure reveals a gap in your original sample.

Do not let the existence of an evaluation report become a reason to stop observing the system. The value of assurance is that it informs ongoing decisions; it should not make those decisions disappear.

What this announcement changes now

The immediate development is a proposed way to examine a lab more deeply, with unresolved questions openly acknowledged by the provider. The evidence does not yet establish the results of the future embedded work.

An operator can act now by improving procurement questions. Ask who evaluated the system, what they could inspect, who paid, what they could publish, and how the findings relate to your use case. Keep those answers distinct.

The practical takeaway is to buy an understandable evidence trail rather than an assurance label. External scrutiny can be valuable. Its value grows when the scope, limits, and findings are visible enough to affect a real decision.

Related: Code review needs behavioral proof and the model-switch test.

Primary source: Anthropic’s embedded evaluation announcement, checked September 29. This article analyzes the announced arrangement; it does not claim completed audit results or certify evaluator independence.