OpenRouter Launches Ori Eval to Help Developers Choose AI Models

According to Beating Monitor, OpenRouter has launched Ori Eval to help developers determine which model to use for their AI applications. It scans code repositories to locate model invocation points and then runs multiple candidate models against real tasks from the project. Developers can specify whether they prioritize performance, speed, or cost. During evaluation, Ori Eval locks down the testing framework, model configuration, and inference intensity, then delivers a direct ranking to ensure fair comparisons between models under identical conditions. In addition to assessing answer quality, it checks whether the Agent invokes the correct tools and avoids unauthorized operations. Developers can describe a bug in natural language, and Ori Eval will generate a test case to reproduce it. After fixing the issue, integrating with GitHub Actions enables automatic checks whenever models, prompts, or code are changed, ensuring the bug does not reappear. Public leaderboards only compare general model capabilities and cannot directly answer which model to choose for customer service, search, or coding products. Ori Eval transforms manual, one-by-one model testing into an automated selection process based on real-world business use cases.