Evaluation

Pre-results protocol · no public scores

This page sets the reporting rules before the numbers exist, on purpose. A good result should be hard to inflate after the fact and possible for another reviewer to understand.

Every published result will name the protocol version it used. If the method changes, I will say so in the report, and I will not rewrite this page to fit the outcome.

What one test case contains#

Each scored case needs:

• the exact question;
• the exact records available to CleverMemory;
• a decision, made in advance, about whether those records can answer it;
• the acceptable answer or answers, when one exists; and
• the machine-checkable proof and source support a valid response must provide.

The answer, proof, and source support are scored together. Finding a related passage does not count if that passage fails to establish the answer.

Ambiguous, conflicting, time-sensitive, or policy-restricted cases will be labeled before a run. The report will say how those cases were counted.

When a case expects a formatted report rather than a value, the expected structure is part of the case. A table returned as prose, a quotation returned without its attribution, or a sequence returned out of order is a failure of the case even when the underlying values are right. Form and support are reported separately so a system cannot trade one for the other.

The four scoring outcomes, all visible#

This table keeps two easy failure modes in view. A system cannot improve its score simply by refusing difficult questions, and it cannot hide unsupported answers inside a high coverage number.

Reports may also show coverage, answer error, justified-decline rate, missed-answer rate, and the tradeoff between coverage and error. Every number will include its definition and its denominator (what the number was counted out of).

How answers will be scored#

When a benchmark provides one exact answer string, exact match will be reported as a strict mechanical measure. It will not be treated as the whole result; correct answers can differ in harmless formatting.

Meaning and support will use a published review rubric anchored by people. If a judge model helps score a larger set, the report will name the model and version, publish the prompt or rubric, describe the sample checked by people, and show how disagreements were handled. A model-generated score is not independent merely because a different model produced it.

The record behind every run#

A public result will identify the state needed to understand or repeat it:

• dataset release, question set, record snapshot, and split (which cases are used for which purpose);
• expected answers, answerability labels, and review rubric;
• public pack versions, manifests, and artifact digests;
• an opaque product build ID, public interface version, and configuration;
• operating system, hardware class, locale, timezone, and clock assumptions;
• random seeds and declared outside dependencies;
• scorer version; and
• judge and human-review settings, when used.

The product build ID makes a result traceable without publishing the source code itself. Access-controlled material can be identified and hashed even when it cannot be redistributed.

Repeatability#

Repeated runs will hold that recorded state constant and compare the public outcome, answer, qualifications, and source references. A report will separate product variation from changes in an outside dependency.

Repeatability does not prove correctness. It proves that the same declared conditions can be inspected and compared.

Every check must be able to fail#

A release gate is a command with a pass condition, and each one ships with its negative control: a planted violation the gate must catch, shown to fail once before the gate counts. A check that has never fired on a deliberately broken input is not evidence of health; it is a check that has not itself been tested. The same rule applies to claimed checks of any kind: a check whose command does not exist or cannot run is a check that has not passed. Any sanctioned exception is named with its reason and its retirement condition, so an exception list cannot quietly become a skip list.

Contamination and fixes#

Answer keys will not be included in public knowledge packs. Reports will state how evaluation questions were separated from development, which people or systems could see the expected answers, and whether a source might contain benchmark material.

A fix made after a run does not change the old result. It creates a new build and a new result, with the relationship between the two recorded.

What a result will not prove#

One benchmark is bounded by its questions, records, policies, and operating conditions. It does not, on its own, establish:

• production reliability or security;
• privacy or regulatory suitability;
• performance on an organization's private data;
• language or domain coverage beyond the tested set;
• resistance to distribution change; or
• latency, capacity, resource use, or operating cost.

Those claims need their own evidence. A strong result will state its limits as clearly as its headline number.

What controlled fixtures are#

The behaviors this site marks as exercised are checked today on controlled fixtures: small, purpose-built internal document sets in which every record and every expected answer is known in advance. Behaviors marked "in validation" or "design target" are not yet covered by a passing fixture, and the status pages say which is which. Because I control both the evidence and the answer key, a fixture can exercise one behavior on demand, whether that behavior is supersession, document geometry, absence, conflict, or time. A failure can be reproduced without first untangling a real customer's data.

A pass claims less than a benchmark score and says exactly what it claims. It means that, for that fixture, wording, and configuration, the scoring table above held: the answer, its proof, and its source support satisfied the case, and declines happened only where the records could not support an answer. It also means that repeated runs under the same declared conditions produced the same outcome. Variants of a passing case then probe what survives a paraphrase, a contradiction, a missing fact, or a changed date.

Three limits matter. A small corpus does not exercise the paths a large one would, so the fixtures say nothing about scale, latency, capacity, or operating cost. They say nothing about performance on an organization's private data. And they are internal: a reader cannot inspect the answer keys or rerun the suite. Those are exactly the gaps the protocol above is designed to close.

One piece of that closure is now public. The 45-document challenge publishes a complete fictional enterprise record and its audited 55-question suite under CC0: the corpus, the expectations, and the integrity contract can be inspected and rerun by anyone. The fixtures above remain internal, and no scored engine run against the public corpus has been published; when one is, it will follow the protocol on this page.

That is the interim state, not a new kind of evidence. Until a public run exists under these rules, the site reports behaviors and the reasoning behind them rather than scores. The fixtures are how those behaviors are checked today. The protocol above is how they will be checkable by someone else.

Publication record#

When a public run is ready, this page will link to a versioned report with the four scoring counts, derived measures, exclusions, and limitations, and to the artifacts behind them: the fixture documents, the answer keys, the scored outputs, and the declared configuration, so a reader can re-score the run instead of taking the counts on trust. Anything that cannot be shared will be named, with the reason. There are no public scores today.

See the Use cases for the behaviors being exercised or Updates for the current release state.