Decision models need more than a compatible API
A fresh repository and documentation audit separates the supply of typed decision systems from the evidence needed to trust their probabilities, thresholds and local ports.
A shared API still needs workload-specific assurance
Among 368 retained repositories, 68.5% document API or SDK compatibility, but only 22.8% document a calibration procedure and 7.9% an explicit parity indicator. The strongest conclusion is architectural: a shared decision API is becoming reusable, but confidence semantics are not interchangeable. The Laya confidence definition and local-port caveats make workload-specific assurance plausible. These are documentation indicators, not reproduced reliability scores; customer demand and ROI remain unmeasured.
Market snapshot
Comparable measurements from independent market surfaces.
Retained repositories
506 screened candidates
Owner accounts
Accounts, not verified organizations
Calibration procedure
84/368 captured READMEs
Explicit port or protocol parity
29/368 captured READMEs
What the sampled tools do
Mutually exclusive purpose groups within the retained snapshot. They measure visible implementation supply, not adoption or market share.
Download the frozen cohort (CSV) · Read the collection and coding protocol
What exists inside the category
Documented README indicators in 368 retained repositories. Missing mentions are unknown, not proof that a feature is absent.
Typed decision contract
277 · 75.3%
An explicit question or wire contract, rather than a free-form answer.
API or SDK compatibility
252 · 68.5%
Compatibility is a documented claim; it does not establish equal probabilities.
Calibration procedure
84 · 22.8%
A fitting or held-out procedure, not the word calibrated alone.
Numerical calibration results
78 · 21.2%
Reported numbers are author claims until independently reproduced.
Abstention or escalation
95 · 25.8%
A documented mechanism or example; correctness under distribution shift is untested.
Explicit port or protocol parity
29 · 7.9%
A parity indicator is narrower than general benchmark or unit-test wording.
What the evidence supports and challenges
Implementation supply, documentation quality and measured outcomes are separate questions.
Is this more than one model launch?
The retained cohort spans models, serving layers, applications and evaluation. Repositories are distinct artifacts; ports and wrappers are not counted as independent model families or paying customers.
Does API compatibility imply interchangeable decisions?
The documented Laya/Jev confidence mismatch and the local port's calibration caveats show why response shapes are insufficient. Workload-specific labels, thresholds and cross-backend parity checks are needed.
Have calibrated decisions been proved generally reliable?
Numerical calibration results, an independent byte-identical benchmark and Sys1Cal supply testable evidence. They also expose domain, annotation and probability-semantics limitations. We did not rerun model inference or validate commercial ROI.
Do released benchmark artifacts reconcile?
All 27 model-by-suite accuracy values agree between the figure data and report. Each aggregate uses exactly 1,240 decisions. The sensitivity calculation removes the multilingual suite without changing the ranking. We did not independently validate labels or execute the models.
| Project | Role | Documented evidence | Limit |
|---|---|---|---|
| NandhaKishorM/laya | Open model | Documents an entropy-based confidence value that differs from Jev's confidence definition, and advises held-out threshold fitting. | Wire compatibility does not make an existing threshold portable. |
| jaredpalmer/kev | Open model | Reports fitted temperatures and separates development, test, trained-source and new-source evaluation. | A good released temperature does not establish calibration for a customer's domain. |
| nokia-applied-research/AnyJev | Decision adapter | Reports a before/after calibration experiment, ablations and a small labelled evaluation cohort. | One reported setup cannot establish general decision reliability. |
| mizorewww/laya-mlx | Local port | Publishes port-parity fixtures and warns about temperature clamps and precision-dependent probabilities. | Fixture parity covers the reported configurations, not every input or backend. |
| instax-dutta/sysone-bench | Independent benchmark | Uses common input bytes, pinned model identities, an evaluation split and separate raw/fitted calibration metrics. | Ground truth has one human reviewer assisting an AI draft; there is no independent second annotation. |
| rawwerks/one-system | Compatibility layer | Provides one TypeSafe-shaped surface for local and hosted decision implementations. | A shared transport cannot guarantee shared statistical meaning. |
| browser-use/jev-ultrafast | Application | Separates offline tests from paid live calls and qualifies a small paired browser-task timing experiment. | One task and a few repeats are not general reliability evidence. |
One common-input benchmark, three different outcomes
AILANTA recomputed weighted accuracy from nine published suite counts and checked every figure value against the pinned report. These are reused experiment results, not new model runs.
| Measure | Jev 1.13.0 | Laya 0.3.11 | Qwen2.5 1.5B PCD |
|---|---|---|---|
| All nine suites (1,240 decisions) | 90.65% (1124/1240) | 68.63% (851/1240) | 60.48% (750/1240) |
| Excluding multilingual intent (1,120 decisions) | 89.64% | 71.16% | 59.64% |
| agnews | 98.75% | 85.00% | 91.25% |
| banking77_12 | 94.79% | 81.25% | 58.33% |
| emotion | 84.38% | 65.63% | 41.15% |
| guardrails | 100.00% | 76.04% | 29.17% |
| mnli | 86.67% | 55.83% | 34.17% |
| moderation | 94.44% | 75.69% | 72.92% |
| multilingual_intent | 100.00% | 45.00% | 68.33% |
| sst5 | 65.00% | 33.33% | 52.50% |
| triage | 93.23% | 87.50% | 78.13% |
The arithmetic agrees across the source tables. Removing the multilingual suite does not reverse the ranking. The labels came from a single human reviewer correcting an AI draft, with no second independent reviewer; this comparison cannot establish broad deployment reliability or customer-domain calibration.
Download the frozen arithmetic audit (JSON)
Representative projects
Signal context and research questions
Related stored publications explain the research question. They are not additional numeric cohort members or independent replications of the measured results.