Model quality needs a serving baseline
A fresh audit of evaluation and monitoring tools asks what can distinguish a changed model from a changed prompt, runtime, account tier or serving path.
Quality monitoring exists; degradation remains unproven
Among 178 retained repositories, 34.8% document repeated quality comparisons, but only 6.2% document configuration pinning and 9.0% uncertainty treatment. Measurement tooling exists; a valid serving baseline is a separate problem from ordinary tracing. Current evidence does not establish intentional provider degradation. Configuration, routing, effort and workload must be controlled before attributing a change to the underlying model.
Market snapshot
Comparable measurements from independent market surfaces.
Retained repositories
434 screened candidates
Owner accounts
Accounts, not verified organizations
Variance or significance treatment
16/178 captured READMEs
Durable run artifacts
28/178 captured READMEs
What the sampled tools do
Mutually exclusive purpose groups within the retained snapshot. They measure visible implementation supply, not adoption or market share.
Download the frozen cohort (CSV) · Read the collection and coding protocol
What exists inside the category
Documented README indicators in 178 retained repositories. Missing mentions are unknown, not proof that a feature is absent.
Repeated quality comparison
62 · 34.8%
A documented repeated-score or regression mechanism, not one launch benchmark.
Configuration or revision pinning
11 · 6.2%
A documented control for one or more measurement inputs; full serving identity can remain opaque.
Variance or significance treatment
16 · 9%
Documentation must mention an uncertainty or paired-comparison method.
Grounded scoring
34 · 19.1%
A reference or deterministic scoring path, not a guarantee that every label is correct.
Serving-path controls
12 · 6.7%
A serving-path variable is explicitly discussed; it may still be uncontrollable.
Durable run artifacts
28 · 15.7%
A documented audit artifact is evidence of inspectability, not tamper resistance.
What the evidence supports and challenges
Implementation supply, documentation quality and measured outcomes are separate questions.
Do independent measurement tools exist?
The sample contains reusable evaluation, regression and observability tools, plus a dedicated longitudinal instrument. That establishes available measurement supply, not a quantified market for provider-quality contracts.
Do current complaints establish intentional degradation?
No. LiveNerf is collecting its baseline and documents detection limits. Stored user complaints are observations of experience, not controlled experiments. Serving-path, effort, prompt and workload differences remain competing explanations.
Can serving changes alter apparent model quality?
A separate cross-stack tool-use study describes template gates, text/native tool-call handling and denominator choices that alter measured fidelity. Its result refines the local-runtime signal: evaluation must preserve protocol failures as well as successful outputs.
Does the detector have validated sensitivity?
Published control accuracy deltas reconcile with the reported condition accuracies. Reduced effort changed token usage more strongly than accuracy; the same-family substitution remained indistinguishable under the reported criterion. This constrains what a future drift claim could mean.
| Project | Role | Documented evidence | Limit |
|---|---|---|---|
| ninjahawk/livenerf | Longitudinal instrument | Pins a CLI and harness, preserves raw results, pre-registers windows and rejects observed fallback paths. | The current series is still collecting its baseline; it cannot yet conclude degradation, and a same-family swap was not detectable in one validation. |
| promptfoo/promptfoo | Regression testing | Provides configuration-driven evaluations, assertions and comparisons across model providers. | A fixed application test does not reveal undisclosed internal provider routing. |
| confident-ai/deepeval | Evaluation | Exposes evaluation metrics and test-case workflows for LLM applications. | Model-judged quality requires its own judge and dataset validation. |
| EleutherAI/lm-evaluation-harness | Benchmark harness | Provides common model/task interfaces and reproducible evaluation settings. | An evaluation run is not inherently a continuous audit of a hosted account. |
| UKGovernmentBEIS/inspect_ai | Evaluation infrastructure | Separates datasets, solvers and scorers and stores inspectable evaluation logs. | A framework provides measurement machinery; the study designer must supply a valid causal comparison. |
| langfuse/langfuse | Observability | Combines tracing, evaluation and experiments around application execution. | Production traces alone do not separate workload change from provider change. |
| vllm-project/vllm | Serving reference | Exposes a configurable inference engine and documented serving features. | A named model can behave differently under different templates, precision or tool-call paths. |
| sgl-project/sglang | Serving reference | Separates inference execution from application-level evaluation. | Engine support and task correctness must be tested together. |
A detector must prove what it can detect
We recalculated the published control accuracy differences against high effort and retained the source's reported standard errors and token changes. The instrument's validation covered 78 items; it is not a longitudinal degradation result.
| Measure | Accuracy | Difference vs high | Reported standard error | Token change |
|---|---|---|---|---|
| medium reasoning effort | 57.37% | -4.17 pp | 3.91 pp | -25.8% |
| low reasoning effort | 53.21% | -8.33 pp | 4.49 pp | -62.0% |
| Same-family model substitution | 57.69% | -3.85 pp | 6.34 pp | -23.4% |
The instrument responded more clearly in token usage than accuracy when effort was deliberately reduced. The same-family substitution did not meet its detection criterion. Failure to detect a difference is not proof that two services are identical; a user complaint is also insufficient to establish a provider change.
Download the frozen arithmetic audit (JSON)
Representative projects
Signal context and research questions
Related stored publications explain the research question. They are not additional numeric cohort members or independent replications of the measured results.