{
  "snapshot": "2026-09-30",
  "calculation": {
    "heading": "A detector must prove what it can detect",
    "description": "We recalculated the published control accuracy differences against high effort and retained the source's reported standard errors and token changes. The instrument's validation covered 78 items; it is not a longitudinal degradation result.",
    "sourceUrl": "https://raw.githubusercontent.com/ninjahawk/livenerf/7b8a62ee485ae96f8646d5763a9e7569e80f58c7/data/validation.json",
    "columns": [
      "Accuracy",
      "Difference vs high",
      "Reported standard error",
      "Token change"
    ],
    "rows": [
      {
        "label": "medium reasoning effort",
        "values": [
          "57.37%",
          "-4.17 pp",
          "3.91 pp",
          "-25.8%"
        ]
      },
      {
        "label": "low reasoning effort",
        "values": [
          "53.21%",
          "-8.33 pp",
          "4.49 pp",
          "-62.0%"
        ]
      },
      {
        "label": "Same-family model substitution",
        "values": [
          "57.69%",
          "-3.85 pp",
          "6.34 pp",
          "-23.4%"
        ]
      }
    ],
    "bars": [
      {
        "label": "medium effort",
        "value": -25.801622538011237
      },
      {
        "label": "low effort",
        "value": -61.97572618838605
      },
      {
        "label": "Same-family substitution",
        "value": -23.421005927701664
      }
    ],
    "barUnit": "% tokens",
    "conclusion": "The instrument responded more clearly in token usage than accuracy when effort was deliberately reduced. The same-family substitution did not meet its detection criterion. Failure to detect a difference is not proof that two services are identical; a user complaint is also insufficient to establish a provider change."
  },
  "sources": [
    {
      "url": "https://raw.githubusercontent.com/ninjahawk/livenerf/7b8a62ee485ae96f8646d5763a9e7569e80f58c7/data/validation.json",
      "metadata": {
        "sha": "7b8a62ee485ae96f8646d5763a9e7569e80f58c7",
        "path": "data/validation.json",
        "sha256": "0799080ec1c0dc174fbb7a7fe7ec4b47d957f9a0fff2b97d3d24f872b6d0fdd1"
      }
    },
    {
      "url": "https://raw.githubusercontent.com/ninjahawk/livenerf/7b8a62ee485ae96f8646d5763a9e7569e80f58c7/docs/VALIDATION.md",
      "metadata": {
        "sha": "7b8a62ee485ae96f8646d5763a9e7569e80f58c7",
        "path": "docs/VALIDATION.md",
        "sha256": "ad94fdc78af688b5bf9fe5a6b999537698f592dc9305efe0244d0b341616fd72"
      }
    },
    {
      "url": "https://raw.githubusercontent.com/ninjahawk/livenerf/7b8a62ee485ae96f8646d5763a9e7569e80f58c7/docs/DESIGN.md",
      "metadata": {
        "sha": "7b8a62ee485ae96f8646d5763a9e7569e80f58c7",
        "path": "docs/DESIGN.md",
        "sha256": "92b859a145b10d5755d1590213745014eb9204643f7f34551078ca7823ed93ab"
      }
    }
  ]
}