Can we trust LLMs to judge whether a research idea is novel?

A new study by Ai2 evaluates six large language models as novelty judges for scientific research ideas and finds substantial inconsistency and unreliability in their assessments, impacted by prompt wording and evaluation setup. The study introduces a benchmark, data, and tools to better measure idea novelty using extracted signals from ICLR 2026 peer reviews.

This Wire brief sits within Fusion42's coverage of AI Frontier Models and NLP & Speech.