Meta researchers have developed a method to detect distilled AI models, identifying 79% of tested copied models with a 0.1% false positive rate without examining the internal structure of the suspicious models. The technique, named WM-Basin, targets models trained using outputs from watermarked "teacher" models; the related paper was published on October 27, 2026.
Key points
- WM-Basin analyzes API-generated text, not model weights or logits.
- Meta evaluated this technique on 1,200 student models.
- This method maintains its detection capability under various model modifications, but its effectiveness diminishes after further fine-tuning and rewriting with clean data.
How does WM-Basin detect distilled AI models?
WM-Basin uses a separate, clean probe model to analyze text generated by suspected distilled models. Researchers from Meta FAIR and Meta Superintelligence Labs introduced this method in their paper titled "Watermark Basins: Black-Box Detection of Distilled LLMs," arXiv:2610.37266.
The paper investigates a specific replication scenario: someone collects responses from a watermarked teacher model via an API and uses those responses to train a student model. WM-Basin uses only the text generated by the student model to find evidence of this training relationship.
The detector feeds these texts into a fixed probe model and converts them into gradient representations—i.e., it observes how the model’s parameters would move if trained on these texts. It then identifies the alignment between standard fine-tuning directions and the "watermark basins" in parameter space associated with a specific key.
This makes the technology a black-box analysis method: there is no need to access the weights or logits of the suspicious model—only the ability to obtain text responses via an API is sufficient for detection.
79% detection rate across 1,200 student models
Researchers report that, with n-gram=1, WM-Basin achieves a true positive rate of 79% while maintaining a false positive rate of just 0.1%; under the same conditions, the strongest previous black-box token-level detector achieved a true positive rate of only 29%.
The author also notes that as the detection threshold becomes stricter and the n-gram window size increases, the advantage of WM-Basin over competing methods further expands.
The evaluation covered 1,200 student models, ranging in scale from 135 million to 8.2 billion parameters. Researchers trained these models using mathematical reasoning traces from OpenR1-Math, encompassing six student architectures, multiple teacher models, and various watermarking methods.
These results support the paper’s conclusion that distilled AI models exhibit improved detection capabilities in mathematical reasoning evaluations.
Which modifications preserve or weaken detection effectiveness?
Researchers found that the detection signal of WM-Basin persists even after quantization, pruning, and reinforcement learning. Quantization compresses the model to reduce operational costs, while pruning removes certain parameters.
The method remains more reliable even when token-level signals weaken, including during low-temperature sampling.
However, other changes can reduce its effectiveness: fine-tuning on large-scale, clean, unwatermarked data weakens detection capabilities, and rewriting the teacher model’s inference trajectories offers another potential evasion pathway. The authors specifically note that continuing supervised fine-tuning on unwatermarked materials is one such method.
Watermark Evidence for AI Developers
Higher detection rates and lower false positive rates may strengthen the credibility of watermark-based evidence in legal or contractual claims of model copying. For such uses, the false positive rate is particularly important, as it measures misidentifications rather than successful detections.
The comparison in the WM-Basin report achieves a 79% true positive rate at the same 0.1% false positive rate, whereas previous token-level methods achieved 29%. The associated tests were conducted using a student model trained on OpenR1-Math mathematical reasoning traces.
This article was generated with the assistance of artificial intelligence and reviewed by the editorial team.
Francesco Antonio Russo
A Web 3.0 entrepreneur with over four years of experience, specializing in cryptocurrency and artificial intelligence. He applies cross-disciplinary skills to functional and trend-driven social media management.