Can we trust LLMs to judge whether a research idea is novel?

Ai2 released "Old Ideas, Novel Problems," a study with a benchmark, data, code, and an automated pipeline for testing LLM novelty judges, built from ICLR 2026 peer reviews to form Human-Only and Human+Generated evaluation sets. Across six judges (GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, Claude Opus 4.6), small prompt changes shifted accuracy by more than 50 percentage points on AI-generated ideas, retrieval and extra reasoning did not consistently help, and two purpose-built novelty systems scored below a simple prompted Claude Opus 4.6 baseline.