OpenJEV Splits Into Four Approaches: Accuracy and Probability Calibration Remain Key Challenges

ME AI News, Dongcha Beating AI Bulletin: Less than a week after Jev’s release, the open-source community has already begun replicating this “don’t write answers, just give probabilities” model. Everyone is trying to do the same thing: skip token-by-token generation and directly output options with probabilities. But how to achieve it has already split into several entirely different approaches. 1. SemIf is the simplest—it doesn’t even require retraining the model. It takes an existing Qwen model, intercepts it just before it generates A, B, or C, prevents further text generation, and directly reads the scores for each option, converting them into probabilities. 2. Simple Jev also uses off-the-shelf large models but further eliminates redundant computation: the same input is processed only once, and subsequent multiple questions share this result before independently evaluating answers. The author explicitly states that this version replicates Jev’s usage pattern but does not match its accuracy, speed, or probability calibration. 3. Laya abandons text-generating large models entirely and instead uses encoding models more akin to traditional classifiers. The benefit is smaller size and faster inference, but the drawback is clear: when choosing among 77 options, Laya’s accuracy is only 42.5%, compared to Jev’s 87.0%. 4. Von also follows the dedicated decision-model route. With only about 400 million parameters, it does not generate text; instead, it completes Choice, Noul, and Score in a single forward pass—functioning more like a next-generation classifier that reads natural language. 5. Verdict is even smaller, with only about 150 million parameters. The author focused on solving two key issues: overconfidence in wrong answers and instability when answer order is shuffled. Its priority is stabilizing both predictions and probability estimates. 6. Kev chooses to retain large models. Built on Qwen as a foundation and augmented with a dedicated decision structure, it processes each input only once and computes individual probabilities for multiple questions. Third parties previously reverse-engineered Jev and speculated it might use a similar design. In Kev’s self-tests using unseen questions, the 8B version achieves ~78% accuracy versus Jev’s ~86%. The more significant gap lies in confidence calibration: Kev still occasionally assigns over 90% confidence to incorrect answers. 7. Nimble focuses on training. Using Qwen3.5-9B, it fine-tunes on a dataset where changing just one fact flips the correct answer. On 324 unseen test samples, Nimble selected the reference answer 90.1% of the time, compared to Jev’s 93.2%. However, the authors caution that the model still cannot guarantee that a 90% confidence rating corresponds to actual 90% accuracy. 8. OpenJev goes furthest—replacing the architecture entirely with DiffusionGemma. It leaves blank slots for possible answers and fills them in one go, like a diffusion model completing an image. On an RTX PRO 6000, the median latency per request is ~94ms. In just a few days, OpenJev has already branched into four paths: directly extracting scores from existing models, training dedicated decision models, converting large models into decision models, and using diffusion models to directly fill in answers. Jev’s approach seems easy to replicate—but the real differentiators remain accuracy, generalization, and probability calibration. The hardest challenge is ensuring that when a model claims 90% confidence, that 90% is truly trustworthy. (Source: BlockBeats)