# Model quality needs a serving baseline: frozen protocol

Snapshot: 2026-09-30. Manifest SHA-256: 0317e543b2721b9c2fe9cb8fb797deadeb890d9e7d732d3c4a28f922f30b3a0b.

## Question
A credible model-quality audit must preserve the input and serving configuration, repeat comparable tasks, quantify variability and retain raw results. Complaints and one-day score changes cannot by themselves establish intentional model degradation.

## Sampling
- "LLM" "drift" in:description fork:false archived:false: 1085 matches, 100 downloaded; capped=true.
- "model" "degradation" in:description fork:false archived:false: 1569 matches, 100 downloaded; capped=true.
- "LLM" "monitoring" "quality" in:description fork:false archived:false: 157 matches, 100 downloaded; capped=true.
- "model" "benchmark" "provider" in:description fork:false archived:false: 158 matches, 100 downloaded; capped=true.
- "LLM" "regression" in:description fork:false archived:false: 1708 matches, 100 downloaded; capped=true.
- "LLM" "evaluation" "continuous" in:description fork:false archived:false: 109 matches, 100 downloaded; capped=true.

Searches return at most 100 non-fork, non-archived repositories per family ordered by current stars. Counts across queries overlap and are never added to estimate a market. Predeclared reference repositories ensure known measurement and serving baselines are represented; they are labelled in the CSV. Capture is resumable within this date and the exact README bytes are stored privately with their hash. Results describe this purposive sample, not the whole ecosystem.

## Screening
row => {
      if (row.attributes.queries.includes("predeclared-reference")) return true;
      const primary = `${row.title} ${row.description || ""}`;
      if (/\b(quantile regression|(?:cv|resume|candidate).{0,25}rank|ranking.{0,25}(?:cv|resume|candidate)|learning.transfer|weather forecasting|financial decision|memory poisoning defense|dynamic rule.based defense|git drift|wiki skill)\b/i.test(primary)) return false;
      return /(?:\b(?:llm|language models?|generative ai|ai (?:models?|systems?|applications?)).{0,60}(?:evaluat|benchmark|regression test|observability|quality monitor|quality gate)|(?:evaluat|benchmark|observability|quality monitor).{0,40}(?:llms?|language models?|model providers)|model (?:quality|performance).{0,30}(?:monitor|drift|degrad|eval|bench)|llm[- _](?:eval|bench)|prompt[- ]regression|livenerf)/i.test(primary);
    }

README unavailable is excluded as unknown. Curated lists are excluded. Laya game-engine projects, conventional decision tables and unrelated uses of acronyms are screened out by contextual language. Repository aliases deduplicate by numeric GitHub ID. Owner accounts are not organizations; wrappers and ports are not independent model families.

## Coding rules
- longitudinal: Repeated quality comparison; /(?:longitudinal|day[- ](?:0|zero)|daily.{0,60}(?:benchmark|eval|score)|quality.{0,35}drift|(?:regression|drift).{0,55}(?:test|detect|eval|baseline)|baseline.{0,80}(?:window|drift|over time))/i. A documented repeated-score or regression mechanism, not one launch benchmark.
- config-pinning: Configuration or revision pinning; /(?:pinned.{0,50}(?:model|cli|version|revision|prompt|harness)|(?:model|cli|prompt|dataset).{0,40}(?:hash|checksum|revision)|frozen.{0,35}(?:prompt|dataset|panel)|reproducib.{0,70}config)/i. A documented control for one or more measurement inputs; full serving identity can remain opaque.
- variance: Variance or significance treatment; /(?:confidence interval|standard error|bootstrap|permutation test|statistical significance|error bars|paired.{0,40}(?:interval|test|difference))/i. Documentation must mention an uncertainty or paired-comparison method.
- ground-truth: Grounded scoring; /(?:exact[- ](?:match|grader)|ground[- ]truth|deterministic.{0,40}(?:grad|scor)|human.{0,30}(?:label|adjudicat|annotat)|reference answer|unit[- ]test.{0,30}(?:grader|scor))/i. A reference or deterministic scoring path, not a guarantee that every label is correct.
- serving-path: Serving-path controls; /(?:fallback.{0,60}(?:model|route)|(?:model|route).{0,60}fallback|account tier|subscription tier|effort.{0,40}(?:level|control|setting)|quantization.{0,55}(?:quality|accuracy)|serving.{0,45}(?:confound|path))/i. A serving-path variable is explicitly discussed; it may still be uncontrollable.
- durable-artifacts: Durable run artifacts; /(?:append[- ]only|raw logs|run artifacts|immutable.{0,30}(?:log|result)|trace.{0,35}(?:export|persist|store)|(?:save|export).{0,35}(?:result|eval|trace))/i. A documented audit artifact is evidence of inspectability, not tamper resistance.

The quantitative screen is a documentation indicator audit, not an expert code audit. All retained README texts are screened with predeclared indicators. The explicit case table is separately analytically reviewed. A missing indicator means not documented in this captured README; it does not mean the product lacks the feature. Quoted user anecdotes and vendor performance claims are not independent measurements.

## Results
Screened 434 deduplicated candidates and retained 178 subject-relevant repositories from 176 owner accounts. 256 candidates were excluded, including 11 with unavailable README text. Owner accounts are not verified independent organizations.
Repeated quality comparison: 62/178 (34.8%); Configuration or revision pinning: 11/178 (6.2%); Variance or significance treatment: 16/178 (9%); Grounded scoring: 34/178 (19.1%); Serving-path controls: 12/178 (6.7%); Durable run artifacts: 28/178 (15.7%)

## Reproduction
Run node scripts/collect-decision-quality-research.mjs, then node scripts/publish-quality-research-2026-09-30.mjs for the dry summary. Publish requires --publish. Frozen CSV and manifest refer to the captured data; later downloads must not silently replace these published numbers. The raw GitHub objects and README copies remain in TargetedResearchItem. Search logs remain in TargetedResearchQuerySnapshot.

## Limits
No paid inference calls, private telemetry or customer interviews were run. We did not independently rerun the benchmark results in the references. Search ranking, README detail, sample selection and shared upstream dependencies bias feature coverage. Creation dates and current stars are not historic usage or adoption series. This sample provides no revenue, customer count, market share or longitudinal growth estimate.

## Pinned artifact checks
We recalculated the published control accuracy differences against high effort and retained the source's reported standard errors and token changes. The instrument's validation covered 78 items; it is not a longitudinal degradation result.

The instrument responded more clearly in token usage than accuracy when effort was deliberately reduced. The same-family substitution did not meet its detection criterion. Failure to detect a difference is not proof that two services are identical; a user complaint is also insufficient to establish a provider change.

- https://raw.githubusercontent.com/ninjahawk/livenerf/7b8a62ee485ae96f8646d5763a9e7569e80f58c7/data/validation.json; {"sha":"7b8a62ee485ae96f8646d5763a9e7569e80f58c7","path":"data/validation.json","sha256":"0799080ec1c0dc174fbb7a7fe7ec4b47d957f9a0fff2b97d3d24f872b6d0fdd1"}
- https://raw.githubusercontent.com/ninjahawk/livenerf/7b8a62ee485ae96f8646d5763a9e7569e80f58c7/docs/VALIDATION.md; {"sha":"7b8a62ee485ae96f8646d5763a9e7569e80f58c7","path":"docs/VALIDATION.md","sha256":"ad94fdc78af688b5bf9fe5a6b999537698f592dc9305efe0244d0b341616fd72"}
- https://raw.githubusercontent.com/ninjahawk/livenerf/7b8a62ee485ae96f8646d5763a9e7569e80f58c7/docs/DESIGN.md; {"sha":"7b8a62ee485ae96f8646d5763a9e7569e80f58c7","path":"docs/DESIGN.md","sha256":"92b859a145b10d5755d1590213745014eb9204643f7f34551078ca7823ed93ab"}

The artifact-audit JSON preserves the derived chart and table. Assertions reconcile source totals and deltas; they do not independently verify labels, rerun inference or certify vendor performance.