Are you still smarter than an AI? There’s a way to keep track

Visitors can see how each model performs on specific benchmarks, as well as what its average score is overall. No model has yet achieved a perfect score of 100 points on any benchmark. Smaug-72B, a new AI model created by the San Francisco-based startup Abacus.AI, recently became the first to break past an average score of 80.

Many of the LLMs are already surpassing the human baseline level of performance on such tests, indicating what researchers call “saturation.” Thomas Wolf, a co-founder and the chief science officer of Hugging Face, said that usually happens when models improve their capabilities to the point where they outgrow specific benchmark tests — much like when a student moves from middle school to high school — or when models have memorized how to answer certain test questions, a concept called “overfitting.”

When that happens, models do well on previously performed tasks but struggle in new situations or on variations of the old task.

“Saturation does not mean that we are getting ‘better than humans’ overall,” Wolf wrote in an email. “It means that on specific benchmarks, models have now reached a point where the current benchmarks are not evaluating their capabilities correctly, so we need to design new ones.”

Some benchmarks have been around for years, and it becomes easy for developers of new LLMs to train their models on those test sets to guarantee high scores upon release. Chatbot Arena, a leaderboard founded by an intercollegiate open research group called the Large Model Systems Organization, aims to combat that by using human input to evaluate AI models.

Parli said that is also one way researchers hope to get creative in how they test language models: by judging them more holistically, rather than by looking at one metric at a time. 

“Especially because we’re seeing more traditional benchmarks get saturated, bringing in human evaluation lets us get at certain aspects that computers and more code-based evaluations cannot,” she said.

Chatbot Arena allows visitors to ask any question they want to two anonymous AI models and then vote on which chatbot gives the better response.

Its leaderboard ranks around 60 models based on more than 300,000 human votes so far. Traffic to the site has increased so much since the rankings launched less than a year ago that the Arena is now getting thousands of votes per day, according to its creators, and the platform is receiving so many requests to add new models that it cannot accommodate them all.

Chatbot Arena co-creator Wei-Lin Chiang, a doctoral student in computer science at the University of California-Berkeley, said the team conducted studies that showed crowdsourced votes produced results nearly as high-quality as if they had hired human experts to test the chatbots. There will inevitably be outliers, he said, but the team is working on creating algorithms to detect malicious behavior from anonymous voters.

As useful as benchmarks are, researchers also acknowledge they are not all-encompassing. Even if a model scores well on reasoning benchmarks, it may still underperform when it comes to specific use cases like analyzing legal documents, wrote Wolf, the Hugging Face co-founder.

That’s why some hobbyists like to conduct “vibe checks” on AI models by observing how they perform in different contexts, he added, thus evaluating how successfully those models manage to engage with users, retain good memory and maintain consistent personalities.

Despite the imperfections of benchmarking, researchers say the tests and leaderboards still encourage innovation among AI developers who must constantly raise the bar to keep up with the latest evaluations.

Leave a Reply

Your email address will not be published. Required fields are marked *

You May Also Like

Past tweets resurface for California Dem who ousted GOP leader

Past Tweets Resurface for California Democrat Who Ousted GOP Leader

Hipolito Cerros, a little-known 28-year-old Democrat whose past posts included jokes about…
Murder suspect in 'Purge' mask identified -- after student killed on Halloween

Purge-Mask Murder Suspect Identified After Student Killed on Halloween

Colorado authorities have identified everyone shown in security footage released after a…
Columbia's gates will remain closed to NYC locals, judge rules

Judge Rules Columbia University Gates Will Remain Closed to NYC Residents

Columbia University can continue keeping its gates closed to the public after…
Virginia parents charged after house party draws hundreds of teens, fireworks and alleged underage drinking

Virginia Parents Charged After Teen House Party Draws Hundreds, Fireworks and Alleged Underage Drinking

A Virginia couple who hoped to provide a safe place for their…
Georgia student’s first words revealed after she survived 8-story drunken fall from dorm while listening to Miley Cyrus

Georgia Student’s First Words Revealed After Surviving Eight-Story Dorm Fall While Listening to Miley Cyrus

A Georgia college student who miraculously survived an eight-story fall had been…
Florida man sends letter filled with peanut butter to allergic ex-wife's friend: cops

Florida Man Accused of Peanut Butter Letter to Ex’s Allergic Friend

He’s nuts! A Florida man allegedly filled a card with peanut butter…
FAA router in middle-of-nowhere Arkansas identified as cause of nationwide flight chaos

Arkansas FAA Router Identified as Cause of Nationwide Flight Chaos

A single faulty FAA router in rural Arkansas appears to have triggered…
Deer seen rock-rocking on NYC's Rockaway Beach in middle of mating season

Deer Spotted Rock-Rocking on Rockaway Beach During Mating Season

This beach visitor certainly wasn’t afraid to make waves. A white-tailed deer…
Fire outside MetLife Stadium sends smoke billowing over empty field after Jets' fifth straight loss

Fire Outside MetLife Stadium Sends Smoke Over Field After Jets’ Fifth Straight Loss

A fire erupted in an empty tent outside New Jersey’s MetLife Stadium…
Sen. Bill Cassidy slams RFK Jr. for

Sen. Bill Cassidy Slams RFK Jr. in New Criticism

Washington — Louisiana Republican Sen. Bill Cassidy on Sunday sharply criticized Health…
SpaceX's Starship returns from orbit and is captured by launch pad arms in a dramatic first

SpaceX Catches Starship Booster With Launch Tower Arms in Historic First

CAPE CANAVERAL, Fla. — SpaceX’s Starship returned safely to Earth Saturday, completing…
Two-thirds of NY voters want to lift cap and expand charter schools: poll

Poll: Two-Thirds of New York Voters Back Lifting Charter School Cap

Nearly two-thirds of New York voters support lifting the state’s cap on…