Meta's Llama 4 Launch Marred by Benchmark Cheating Scandal

In a dramatic turn for the AI industry, Meta's April 2025 release of its Llama 4 models—Scout and Maverick—has been overshadowed by allegations of unethical benchmarking practices. The controversy, erupting just days after the launch, has sparked heated debates across tech communities, casting a shadow over Meta's ambitious push into advanced AI.

The saga began with a rumor, first posted on a Chinese social media platform and amplified on X and Reddit, claiming Meta artificially boosted Llama 4’s benchmark scores by training on test sets—a practice widely condemned as cheating in AI research. Further fueling the fire, it emerged that Meta submitted an experimental version of Llama 4 Maverick, dubbed "Llama-4-Maverick-03-26-Experimental," to the LMArena benchmarking platform. This version secured an impressive number-two ranking, but users soon noticed discrepancies with the publicly released model, which delivered less verbose responses and lacked the experimental version’s flair. Critics labeled this a "bait-and-switch," accusing Meta of optimizing the experimental model to inflate rankings.

Meta’s VP of generative AI, Ahmad Al-Dahle, swiftly denied the allegations in an X post, asserting that no test set training occurred and attributing performance gaps to implementation stabilization. Meta clarified that the experimental version was a chat-optimized variant, part of routine testing. Meanwhile, LMArena responded by releasing over 2,000 battle results for transparency and updating leaderboard policies to ensure fairness.

The scandal has ignited discussions about ethics in AI benchmarking, with mixed reactions from users. Some report underwhelming results with Llama 4, while others debate Meta’s transparency. As the tech world watches, this controversy underscores the challenges of maintaining trust in an increasingly competitive AI landscape, raising questions about how companies balance innovation with integrity.

Sources
The Register
Tech Crunch
ZDNet