# EXP-0002 — Fiction-CWR v0.1 directional existing-model evaluation
Project: Project Fiction, Hermes Labs | Proposed:2026-10-11  
Status: proposed; architecture documented, offline token checks completed, empirical execution not started  
Design version:0.1.1 | Updated:2026-10-12 | Epistemic label: NEW PROPOSAL  
Model runs:0 | Human runs:0 | Paid calls:0 | Spend authorization:none

## Seven required statements
1. **Hypothesis:** Pinned frontier models may show a substantive familiar-to-withheld composition exact-validity loss despite independently demonstrated primitive competence (FH-08; complementary to FH-01/FH-07).
2. **Operational definition:** Executable solutions in finite W=(S,A,T,s0,G,C,B), scored by conjunction of strict schema, supplied vocabulary/effects, legal transitions, invariants, complete goal and budget. Novelty relative to supplied demonstrations, with necessary structural withholding certified or claim narrowed.
3. **Alternative explanation:** Primitive/spec misunderstanding, generic search difficulty, memory/load, output burden, familiar surfaces, evidence imbalance, prompt/budget, mastery selection or defective graders.
4. **Control:** Independent P, finite-handbook R, E execution, primary paired F/C, independent checker/oracle, copy/random/greedy baselines, representation and higher-budget diagnostics; all task/model outcomes retained.
5. **Success criterion:** Directional pilot may justify confirmation only after measurement/matching gates; independent confirmation supports the scoped substantive loss if its prespecified lower bound reaches the frozen deficit in a qualified model and confounds are addressed.
6. **Failure criterion:** Upper bound below the chosen loss rejects that scoped size; crossing is inconclusive. Failed P prevents the primitive-understood attribution; defective matching/grading prevents intended interpretation. Strong C and null/reversed effects are preserved.
7. **Next decision:** Complete stimulus/checker/model/budget freeze, execute only authorized pilot, then narrow/reject or design independently powered confirmation. No detector/RL or runtime/reward adoption follows.

## Linked architecture and reproducibility
[Package and eight deliverables](../evaluations/fiction-cwr-v0.1/README.md) · [design](../evaluations/fiction-cwr-v0.1/DESIGN.md) · [eight task specs](../evaluations/fiction-cwr-v0.1/TASK-CATALOG.md) · [grader](../evaluations/fiction-cwr-v0.1/GRADER.md) · [model plan](../evaluations/fiction-cwr-v0.1/MODEL-PLAN.md) · [analysis](../evaluations/fiction-cwr-v0.1/ANALYSIS.md).

Inputs: eight public synthetic showcase specifications v0.1.0; one implemented token fixture/interpreter/oracle; private seeded variants and an empirically frozen matched panel not implemented. One public token F/C candidate is imported with bounded necessity/exposure audits and measured source mismatches. Conditions proposed R/E_C/F/C with independent P. Planned288 model calls=8 bases×4 conditions×3 trials×3 providers, plus unresolved P probes. Optional E_F adds72 and is separately budgeted. Repetitions are not independent task samples. Candidate δ=0.20/τ=0.90 require pre-confirmation rationale/freeze; other numeric criteria remain unresolved.

Model identities, snapshots, temperatures, reasoning/output budgets, supported seeds, prompts/token counts, official pricing, spending ceiling, timeouts/retries and empirical dates: **UNRESOLVED / not executed**. Confirmation generator seeds, independent sample count, precision/power, interval/multiplicity rule and simulator: **UNRESOLVED**. Main arm no external retrieval/tools/feedback. Internal retrieval and valid supplied-rule deduction allowed.

Runtime: Python3.10+ standard library for the token offline reference. Actual host version and executed commands are recorded in [VERIFICATION](../evaluations/fiction-cwr-v0.1/VERIFICATION.md). Raw requests/outputs/usage/hashes will be saved according to [run record template](../evaluations/fiction-cwr-v0.1/config/run-record-template.json); none exist for frontier models. Blank records are templates, not runs.

## Actual work versus proposal
Local documentation/specification and offline engineering implementation/verification are the work of this phase. [Engineering results](../evaluations/fiction-cwr-v0.1/VERIFICATION.md) record 10 passing test methods, 150 bounded variants of one recipe system, 5,370 bounded sequences, and eight fixed showcase witness checks. The saved local package was tested with Python 3.12.14. These checks neither establish a model deficit nor pass human G0–G4, GR or GW. The seven non-token full graders, private generator families, across-family/confirmation holdout certification, powered confirmation and provider runner remain unimplemented. Supported token engineering certification and one public candidate are now recorded separately.

[EXP-0001](pilot-pre-test-redesign.md) retains its human construct/label/signal scope. This benchmark is related to the broader transformation target but does not produce human/process labels. Human-to-model transfer, modality translation, practical transfer, runtime and reward usefulness remain separate unresolved questions.

## Provenance and external opportunity
Existing Hermes handover/baseline dated2026-10-06; the current task supplied a continuation brief from “PrivacyBench Analysis” and requested use of this local workspace. The underlying conversation and older raw evidence were not independently inspected. This record does not import unrelated project assumptions.

The [micro1 preparation](../evaluations/fiction-cwr-v0.1/SUBMISSION.md) records both official pages accessed2026-10-11, pre-event disclosure and the conflicting schedule. This is a Hermes/Project Fiction release opportunity; no registration/agreement, publishing, upload or submission is performed.

## 2026-10-12 — Public token audit extension
[Matching audit v0.1.1](../evaluations/fiction-cwr-v0.1/MATCHING-AUDIT-v0.1.1.md) supplies an explicit-role certificate for all 121 TW-001 sequences and two accepted plans, plus one reused EXP-0003 candidate pair. Each imported target has two valid eight-action/cost-8 plans; strict prefix/state/cost/failure comparisons certify its entire bounded grammar. [Actual results](../evaluations/fiction-cwr-v0.1/verification/token-pair-audit-v0.1.1.json) and [12 new tests](../evaluations/fiction-cwr-v0.1/verification/token-pair-tests-v0.1.1.json) are engineering records, not model runs.

The [standalone manifest](../evaluations/fiction-cwr-v0.1/config/token-pair-manifest-v0.1.1.json) whitelists F/C inputs without EXP-0003's workflow intervention. Existing quotas, independent P/R/E, numeric criteria, model/settings/spend and private distributions remain unfrozen. Reuse adds no independent base; source solution counts differ 2/13. Withholding means absence of realized multistep dependency evidence, while valid deductions from the complete rules/primitive examples remain permitted. EXP-0001 and EXP-0003 retain their distinct scopes.