AI Research Taste Gap

The evaluation problem is becoming concrete enough to cross from watch into the published feed. Amazon researchers show that LLM judges can distort conclusions about research agents unless their failure modes are measured, while agent benchmarks increasingly evaluate strategy, efficiency and executable outcomes rather than polished answers. The market implication is demand for evaluation systems that test scientific judgment and failure behavior, not only benchmark scores.

A new practitioner warning says authors are submitting papers they may not understand, while independent evaluation discussion continues to show that frontier systems can exploit shallow or outdated tests. This is an early watch-line continuation, not proof of a broad failure in science. The next check is whether journals, benchmark maintainers or labs introduce stronger provenance and understanding tests.

Astra produced technically correct but uninteresting entrepreneurship studies when asked to choose and preregister original hypotheses, while ARC-AGI-3 results show a large gap between the model and a human baseline despite gains on conventional evaluations. The emerging limitation may be problem selection and judgment rather than execution quality. Two observations are not enough to define a market, but they point to a distinct evaluation and tooling need around research taste.