AI model benchmarking is shifting from academic datasets to more practical, business-relevant evaluation systems. According to TechCrunch, startup Vals, founded in 2024, is aiming to capitalize on this trend. The company recently closed a $40 million Series A round led by Andreessen Horowitz.
No longer testing only general knowledge
Val's assessment is that traditional public benchmarks are increasingly struggling to reflect the true capabilities of next-generation models. On one hand, models are being updated rapidly; on the other hand, public test questions may be repeatedly targeted by model developers, leading to discrepancies between scores and actual performance.
The company has not focused on general knowledge Q&A or exam-style tests, but rather emphasizes whether the model can accomplish specific industry tasks. Reports indicate that Vals primarily evaluates complex tasks in areas such as law, finance, and programming, and attempts to compare the quality of model outputs with human-generated results.
Closed testing covers more scenarios.
Unlike many public benchmarks, Vals does not disclose specific test materials. The goal is to minimize interference from test-specific preparation, allowing evaluation results to better reflect the model's performance in real-world scenarios.
In addition to common industry tasks, Vals is also expanding its testing scope. The company states that current evaluations now extend to recursive self-improvement, mental health, cybersecurity, biosecurity, and the laws of armed conflict, while also monitoring potential negative outcomes the model may produce.
Revenue growth and expansion of government clients
Vals' business model involves charging model developers for evaluation services. The report notes that although such services may appear to be "paying to find issues," continuous measurement helps companies identify weaknesses and improve their models, which also influences purchasers' decisions when selecting models.
- Current income is approximately eight times that of the same period last year.
- The team size has grown from 8 at the beginning of the year to 25.
- The plan is to continue hiring 10 to 15 more people.
In addition, Vals has recently launched a model evaluation program for U.S. federal agencies, aiming to expand its business from tech companies to government clients. TechCrunch believes that as more AI companies enter the capital markets, model evaluation results may gradually become part of public disclosures, procurement decisions, and investor communications.
