Know what the
evidence means.

ATBAE—the AI Test Battery and Assurance Examination—examines specified model and workflow behavior under declared conditions. The current Independent Agent-Control Pilot is a bounded engineering engagement on one agreed configuration. The broader examination battery is being developed and qualified; it is not presented as fully production-certified.

No. It provides scoped findings, methods and supporting evidence. Full production assurance requires the applicable scientific, execution, governance and issuance gates to close. Neither a report, a passing software test, nor the professional fee confers accreditation, regulatory acceptance or a guarantee of safe deployment.

An executive brief and technical findings, an engagement-specific methods annex, an agreed evidence inventory and supporting artifacts, a findings discussion, and one bounded correction retest. The commissioned report is based on the work performed; the short public example is an explanation of the format, not a page limit for a customer report.

Developers can investigate reproducible weaknesses; procurement and risk teams can assess a defined configuration; researchers can scrutinize methods; supervisory reviewers can inspect controlled evidence. Each audience needs a stated decision and appropriate scope. We do not imply that a regulator has endorsed ATBAE or that one result applies to all users.

A bounded examination can identify reproducible weaknesses, clarify deployment conditions and support a correction decision. Its value is evidence tied to a decision, not a universal guarantee. The report states what was observed, what remains uncertain and what further work would be needed for a stronger claim.

No. An external examination can challenge assumptions and provide another evidence source. Internal evaluation, security engineering, monitoring and incident response remain necessary. We agree where our examination complements existing work and avoid counting the same underlying observations as separate confirmations.

A supervisory reviewer may find a scoped report and its supporting evidence useful. Acceptance depends on the authority, purpose and applicable requirements. ATBAE does not claim automatic regulatory recognition or that a commercial pilot fulfills a statutory certification requirement.

Intended use connects the examination to a decision: which tasks matter, what authority the system has, what counts as useful completion, and what consequences make a failure material. Selecting a use is a scoping request, not proof that a validated specialist examination is already available.

Yes. Explain all relevant uses and their priorities. We scope their criteria separately on the same target where appropriate; a strong result for one use does not compensate for a failed required control in another. The agreed plan determines coverage, repetitions and any limits on comparisons.

Yes, a business-process scope can concern ordinary operations, such as drafting, approvals, supply-chain handoffs or internal reporting. We agree permissions, tool access and observable outcomes. A business-workflow finding is not automatically a legal-compliance assessment or a substitute for your operational controls.

A scholarly request can span those levels, but the criteria differ: age and context, citation support, learning goals, research preparation and permitted assistance all matter. The educational level and use must be specified. ATBAE does not confer academic approval, determine a student’s authorship, or replace an institution’s academic-integrity process.

Describe the intended task and the evidence a good result would require. Design, literature, pattern recognition and sector-specific workflows may need different modalities, specialist references or additional apparatus. Listing a domain in the intake form does not establish that every domain or modality is implemented or qualified.

Any such scope needs explicit authority, containment, agreed destinations and stop conditions. The pilot does not authorize activity against third-party systems or unrestricted production actions. A simulated effect, a blocked attempt and an observed real effect are reported as different outcomes.

Blanket refusal can avoid an action while failing the legitimate task. We report useful completion separately from control adherence and distinguish justified refusal from unnecessary refusal. A system that did not perform the work is not credited with completing it safely.

No transfer is assumed. Fine-tuning, system instructions, retrieval, memory, tools and safeguards can change behavior. A comparative scope identifies the configurations and holds other conditions fixed where feasible. A result about an integrated agent is not silently attributed to the underlying weights alone.

The access route is agreed before collection. An authorized customer-hosted endpoint can allow examination without transferring weights. Licensed self-hosting can be separately scoped when appropriate. Model, version and configuration properties that cannot be verified are recorded as unavailable or attested rather than silently assumed.

The result belongs to the identified model configuration and environment: prompts, decoding, tools, adapters and applicable serving details. We preserve the original and transmitted inputs when adaptation is used. Interface compatibility does not establish that different role templates or deployments are behaviorally equivalent.

Precision identifies the tested configuration; it is not a grade. A claim about BF16 weights needs evidence for those files, and a serving-precision claim needs appropriate runtime evidence. BF16 is not FP32. Weight dtype does not by itself establish every compute, accumulator or cache dtype. An opaque endpoint may support only a provider-attested statement.

No. It establishes only the interface properties actually checked, such as supported request structure, context admission or parseable output. It is preparation for the examination. Successful task performance and adherence to authority require their own observations.

The professional fee is US$15,000 pre-tax for the bounded pilot: one agreed configuration, up to four agreed control families, the specified report and evidence deliverables, and one bounded correction retest. Third-party compute/API expenditure is separately agreed and capped. Additional targets, broader coverage and specialist review require their own scope.

Collection and reporting depend on a complete, agreed scope and accepted readiness: access, permissions, compatibility, observation channels and limits. The service page describes the proposed response and delivery targets. The engagement agreement establishes the actual commitments; we do not promise unlimited access, continuous monitoring or an unconditional completion date.

We distinguish apparatus problems from observed model behavior. Failed or incomplete collection stays visible, with the reason and supported interpretation. A changed model, prompt, tool stack or environment needs a recorded configuration and agreed retest scope. A correction retest gets a new identity and preserves its baseline.

No. The agreed pilot covers one configuration and up to four agreed control families. Multiple models, new modalities, extensive domain review, continuing monitoring and larger retests require separate scope and pricing. An intake option expresses a customer need; it is not a promise of unlimited coverage.

The engagement defines authorized access, permitted actions, call limits, stopping rules and a separately agreed cap on third-party compute or API expenditure. Technical access alone does not grant permission to perform every action an endpoint supports. Scope changes must be recorded before dependent work proceeds.

Not necessarily. Repetitions, turns, variants and shared templates can be related. Reports distinguish the collection, item, family, episode, turn and request units, and state where independence is not established. A large call count is not automatically a large independent sample.

A declared policy and reference process appropriate to the construct. We distinguish constructed-rule agreement from externally grounded truth and disclose reviewer exposure and conflicts. Agreement among reviewers does not eliminate shared bias. Policy disagreements and uncertain reference labels need disposition rather than silent relabeling.

Where justified, it reports class-specific estimates with uncertainty and prior sensitivity instead of treating a point estimate as exact. The interpretation depends on the references, priors and sampling assumptions. It does not automatically validate the oracle or replace required canonical gates. Marginal intervals are not automatically a simultaneous guarantee.

They remain visible with their own counts and reasons. Reference indeterminacy, reviewer abstention, malformed responses and collection failures are not interchangeable. Missing evidence is not a clean pass, and the denominator must be stated alongside each result.

No. It describes the observations collected. An uncertainty bound requires a stated statistical model, sampling assumptions and stopping rule. Rare-event claims can require substantial valid evidence; we do not convert an absence of observed events into proof that an event is impossible.

An oracle is the process used to judge an outcome: for example, a mechanical check, a declared reference policy or an adjudication process. Its reliability must be investigated for the specific judgments it makes. Agreement on simple constructed controls does not establish general correctness on every real-world case.

Yes. An easy or cue-driven task can produce agreement without examining the intended ability. Reference quality, coverage, ambiguity and alternative explanations must be considered alongside agreement. We distinguish a software check, a measurement finding and a claim about model behavior.

No. A posterior is conditional on the model, prior, reference information and observations. We disclose those dependencies and relevant sensitivity. A narrow interval under an unsuitable model can be misleading, and a posterior probability is not automatically a frequentist coverage guarantee.

No. Useful completion and adherence to authority are distinct outcomes. A high descriptive average cannot compensate for a failed required gate. Expanded findings are not silently blended into a new overall grade; any such scoring change needs a declared and accepted rule.

Choosing when to stop after inspecting results can invalidate a fixed-sample interpretation. A plan must declare stopping and analysis rules, or use an appropriately justified sequential method. Unplanned looks and changes are recorded; an anytime-valid guarantee is claimed only when its mathematical and implementation conditions are established.

Not automatically. Separate marginal intervals and a simultaneous statement answer different questions. A report must state whether and how multiplicity is addressed. We do not turn a collection of individual uncertainty statements into an unqualified battery-wide guarantee.

A defensible comparison requires compatible task definitions, configurations, reference procedures and analysis units. Otherwise we show separate findings and explain the differences. A larger model, a different precision or a higher score in another collection is not by itself evidence of superiority in the intended use.

It can expose a measurement limit, an apparatus defect or a model behavior that matters to deployment. Removing it would distort the record. We preserve the supported interpretation and distinguish corrected analysis from a new collection, rather than treating a later favorable result as a replacement for history.

No. ATBAE does not publish or generally distribute its proprietary examination corpus or answer keys. Customer information and selected evidence are handled under agreed access, confidentiality and retention terms. Controlled inspection for an authorized regulator or assessor is a separate, scope-specific process; it is not a public release or an automatic transfer of the entire corpus.

Public questions and answer keys can become training material or targets for test-specific tuning. Restricting distribution reduces that exposure and helps protect the usefulness of future examinations. It does not prove that a model has never encountered similar material, eliminate contamination or establish validity by itself. Sandboxing is a separate control that contains test actions.

Yes. The system must receive the selected inputs to be examined. Non-publication therefore does not mean zero exposure. Hosting, provider data-use and retention terms, access controls and exposure records matter. Reuse must be interpreted accordingly; a regression test on previously seen material is not automatically a fresh, protected evaluation.

A checked hash identifies the corresponding bytes against the supplied reference. It does not, by itself, authenticate an author, establish scientific validity, prove independent review or show regulatory approval. Those require separate evidence and decisions.

We are open to appropriately authorized regulatory or independent assessment. Where necessary, restricted examination material can be considered for supervised, scope-specific inspection under an NDA or another appropriate confidentiality arrangement, subject to applicable requirements. The purpose, recipients, material and access are documented. This does not promise every requester the complete corpus or waive other customers’ confidentiality.

A successfully verified signature can bind particular bytes to a signing key under the verification scheme and trust assumptions used. It does not establish the scientific correctness of the contents, the authority of a regulator or the identity of a person without a separate trust basis.

The engagement identifies the report recipients and their access. Customers receive the agreed report and supporting deliverables; restricted operating material remains with authorized personnel. Public samples are deliberately separate. A master or operating pack is not automatically a customer entitlement or a public download.

Yes. We ask for the specific finding, configuration and supporting evidence, then distinguish factual correction, policy disagreement and a changed testing scope. Supported unfavorable findings and original records should be preserved. Corrections and retests must explain what changed and which conclusions are affected.

Funding relationships, relevant conflicts and reviewer exposure should be disclosed in the engagement. Source-code review, blinded labeling, domain expertise and independent scientific qualification are different roles. A fresh AI session alone does not establish professional competence or independence.

The service is an examination, not a purchase of a favorable conclusion. Findings must follow the agreed evidence and rules. Confidentiality and disclosure arrangements are agreed in advance; they should not be used to misstate results or conceal a material limitation from an authorized report recipient.

No. AI-assisted review can identify defects and compare interpretations, but it can also share errors or miss context. Reviewer competence, exposure, conflicts and the exact work performed must be disclosed. Technical reproduction and specialist scientific, domain or governance review are distinct forms of evidence.

This helps us prevent formatting problems from being misreported as reasoning failures. We retain the original response and distinguish compliance with the agreed response format from agreement with the reference answer and failures of the collection apparatus. An answer inside Markdown fences can fail a strict JSON contract even when its content is correct.

The original registered result remains unchanged. Any later extraction or content diagnosis is labeled as post-collection analysis; it is not a replacement score. Schema-constrained output is a separate evaluated configuration. This distinction does not waive required interface rules or establish general reasoning ability.

The purpose-built synthetic sample is clearly labeled as invented explanatory material. The retained-live excerpt is separate and tied to its historical configuration and simulated environment. Neither should be substituted for a new commissioned examination or a current production certificate.

The applicable oracle/reference process, intended-use corpus validity, confidence-process obligations, integrated execution, governance and issuance controls must be qualified for the exact claim. We report these separately. Progress in software testing or a longer report does not close an unmet scientific requirement.

It needs a defined intended use, validated measurement and implementation, accepted operating controls, a qualified execution path and a supported reporting claim. A demonstration or passing fixture is useful development evidence but is not enough on its own. Readiness must be assessed at the layer of the claim being offered.

The report should identify the affected edition, finding and evidence, explain the correction and preserve the earlier record as required by the engagement. New work receives its own identity. A correction must distinguish a changed interpretation from newly observed behavior.

It is a proposed standard preparation control for deciding how examination evidence may be used. Before a run, the intended process assesses prior exposure through session memory, retained logs, developer tuning, shared training data, related model versions and customer organizations. Known reuse and unresolved exposure must remain visible. An offline prototype has been implemented and fixture-tested; integration into the live collection path is pending. It is not yet enforced across every service.

The repeat scope should distinguish regression testing on previously used cases from a fresh examination on separately reserved, validated material. Regression asks whether a known weakness was corrected under the stated conditions. A fresh examination asks whether performance extends to appropriately assessed material beyond those disclosed cases. The report should identify both populations separately and preserve the earlier findings. This proposed admission process applies across standard catalog services and scoped pilots as integration is completed.

Yes. State which decision you need at scoping. A combined engagement should define the configuration, cases, exposure history, comparisons and reporting for each population before collection. A fresh examination is not automatically included in a bounded correction retest. The statement of work establishes supported scope and fees; this policy does not change published package entitlements.

It should not. Relevant known relationships between versions, fine-tunes, providers and customer organizations need to be considered when interpreting previous exposure. A new name or account does not establish a fresh examination. Equally, belonging to the same model family or industry does not prove that particular material was shared. Unknown relationships must be recorded without inventing either independence or contamination.

Exposure is an evidence-admission and interpretation issue, not a model behavior score. It can limit a claim, require different material, or keep a qualification decision open. Useful regression findings can still be reported with their limitations. Exposure accounting must not silently change canonical thresholds or allow a favorable score to override a required evidence gate.

No. Reworded cases may share the same structure, and isolated or confidential execution does not prevent the target from processing the inputs it receives. Fresh material needs reference and coverage validation; comparisons across forms need appropriate support. These controls reduce specified exposure routes but cannot establish that a model has never encountered related material.

The proposed reporting standard identifies the configuration and examination edition, distinguishes reused and prospectively reserved material, records relevant exposure limitations, and explains which comparisons are supported. Counts of items, variants and families must not be presented as independent observations without justification. Customer-facing explanations can provide this accountability without publishing the proprietary corpus, answer keys or protective implementation details.