A handful of dominant AI corporations have been quietly manipulating one of the crucial influential public leaderboards for chatbot fashions, probably distorting perceptions of mannequin efficiency and undermining open competitors, in line with a brand new research.
The analysis, titled “The Leaderboard Phantasm,” was printed by a crew of consultants from Cohere Labs, Stanford College, Princeton College, and different establishments. It scrutinized the operations of Chatbot Area, a extensively used public platform that permits customers to check generative AI fashions by pairwise voting on mannequin responses to person prompts.
The research revealed that main tech corporations — together with Meta, Google, and OpenAI — got privileged entry to check a number of variations of their AI fashions privately on Chatbot Area. By selectively publishing solely the highest-performing variations, these corporations had been in a position to increase their rankings, the research discovered.
