Build
Read the corpus and write a persistent knowledge base. No questions are visible.
FabricBench / frozen instance v1.0
Shattered Needle
A benchmark of whether an agent can organize a large corpus before seeing the questions, then recover answers from its frozen knowledge base without the source documents.
877documents
~305kcorpus tokens
12cases
307questions
10displayed runs
Results
Weighted score is primary. Culprit synthesis receives 55% of the weight; resolution and provenance make up the remainder.
| GLM-5.3 | portable | 100% | ~$11.01 proxy | 1h 15m 28s | complete |
| Gemini 3.7 Flash | portable | 100% | $2.40 billed | 37m 18s | complete |
| Claude Opus 4.8 | historical | 100% | ~$12.56 | not captured | complete |
| GPT-5.6 Sol, clarified fabric | query-only | 100% | ~$3.26 | 6m 43s | diagnostic |
| Claude Sonnet 5 | historical | 95% | ~$5.78 | not captured | complete |
| GPT-5.6 Terra | portable | 57% | ~$7.00 | 34m 28s* | resumed |
| MiniMax M3 | portable | 52% | ~$3.36 | 2h 56m 23s* | resumed |
| GPT-5.6 Luna | portable | 11% | ~$0.78 | 30m 12s | complete |
| Claude Haiku 4.5 | historical | 2% | ~$1.01 | not captured | complete |
| DeepSeek V4 Flash | portable | no score | $3.54+ billed | 1h 57m 5s+ | QUERY failed |
Historical runs used the byte-identical corpus and frozen gold with earlier prompts. Query-only results are not end-to-end scores. API figures are estimates, not charges paid. * Resumed wall time is incomplete; DeepSeek time excludes later manual repair.
Culprit 55%
Resolution 20%
Identity provenance 10%
Ownership provenance 8%
Finding provenance 7%
Benchmark
What it measures
An agent reads scattered fictional records, resolves aliases and relationships, joins evidence across files, and preserves provenance. The held-out quiz tests what the resulting knowledge base can recover.
The knowledge representation is unrestricted. Only answers from the frozen artifact are scored.
- Instance v1.0 / run21
- Corpus 877 irregular documents
- Cases 12 interleaved investigations
- Evidence six factors per culprit
- Grading deterministic, including provenance
Protocol
The model knows the task type but not the questions during ingestion.
Remove sources
End the session, freeze the artifact, and physically remove the corpus.
Recover
A fresh session of the same model answers 307 questions from the artifact alone.
Technical notes
Full score breakdown
| Model / run | Resolution | ID provenance | Owner provenance | Finding provenance |
|---|---|---|---|---|
| GLM-5.3 | 103/103 | 103/103 | 53/53 | 36/36 |
| Gemini 3.7 Flash | 103/103 | 103/103 | 53/53 | 36/36 |
| Claude Opus 4.8 | 103/103 | 103/103 | 53/53 | 36/36 |
| Claude Sonnet 5 | 103/103 | 103/103 | 53/53 | 33/36 |
| GPT-5.6 Terra | 103/103 | 92/103 | 53/53 | 35/36 |
| MiniMax M3 | 55/103 | 55/103 | 37/53 | 14/36 |
| GPT-5.6 Luna | 18/103 | 18/103 | 22/53 | 12/36 |
| Claude Haiku 4.5 | 6/103 | 0/103 | 0/53 | 3/36 |
| GPT-5.6 Sol, clarified fabric | 102/103 | 102/103 | 53/53 | 36/36 |
Run comparability
Portable runs use the packaged harness. Historical Claude runs used pre-clarification prompts and in-session Claude Code subagents, but the byte-identical corpus and frozen gold. Terra and MiniMax resumed after interruption. The clarified-fabric Sol result isolates QUERY behavior and is not an end-to-end run.
DeepSeek and interrupted runs
DeepSeek completed coverage of all 877 documents after operator repair and deterministic consolidation, but three autonomous same-model QUERY attempts failed. A separate corpus-free, operator-assisted diagnostic recovered 289/307 answers (89% weighted, 10/12 culprits); it is not an official end-to-end score.
Pricing and token accounting
OpenRouter values are reported bills. Other values apply public standard API rates retrieved on 17 August 2026 to measured usage:
input + cache writes + cache reads + (output + separately reported reasoning)
Anthropic includes extended-thinking tokens in output_tokens. Claude estimates include five-minute cache writes and cache hits recovered from the original transcripts. GLM-5.3 uses GLM-5.2 rates as a proxy because no 5.3 pay-as-you-go tariff was listed.
Sources: OpenAI, Anthropic, MiniMax, Z.AI, and OpenRouter.
Generation and certification
A private generator starts from target propositions, lowers them through a function-free Horn derivation graph, and realizes only the leaves as natural-language documents. The forward engine verifies solvability and uniqueness. Distractors are certified not to license an alternative derivation, and all corpus files are frozen by SHA-256 manifest.
InterpretationThis is one frozen instance, not a population-level model ranking. Three perfect runs show that v1 has reached the frontier ceiling; later versions need larger and harder instances.