Agentic IT: Can LLMs Design a Benchmark

Twelve large language models were tasked with designing and running benchmarks for text-to-SQL and tool-calling tasks, but none passed all the rubric's criteria and several key quality checks were universally missed, highlighting significant shortcomings in their benchmark design abilities.

This Wire brief sits within Fusion42's coverage of AI & ML.