Noise Can Enhance Model Performance in Semi-Supervised Learning

In transfer learning, "source data" does not have to be limited to images, text, or audio.

Noise randomly sampled from a Gaussian distribution, with no semantic meaning, can also help the model learn better with limited annotations.

This work is called Semi-Supervised Noise Adaptation (SSNA), and the paper has been published at ICML 2026.

The SSNA team further proposes a Noise Adaptation Framework (NAF), which constructs discriminative class structures using randomly generated noise and transfers these structures to real target data, thereby enhancing the model's learning performance in scenarios with limited annotations.

△NAF projects the noise domain and target domain into a shared representation space and aligns them at the category level. Image courtesy of the paper.

With only four labeled samples per class, NAF, based on the ResNet-18 backbone, achieves accuracy improvements of 12.35, 7.61, 4.38, and 2.74 percentage points over the standard supervised learning baseline ERM (Empirical Risk Minimization) on CIFAR-10, CIFAR-100, DTD-47, and Caltech-101, respectively. The source code for this paper is now open-sourced; details can be found at the end of the article.

Traditional transfer learning typically requires a source domain with abundant labels. However, real source data is not always readily available; privacy, confidentiality, and copyright restrictions may prevent sharing of source domain data. SSNA aims to eliminate this assumption: instead of real samples, the source domain is replaced with noise generated from simple probability distributions.

The specific approach is not complex: assuming the target task includes C categories, the research team first randomly samples a mean vector in a 1024-dimensional space for each category, using the identity matrix as the covariance, thereby constructing C Gaussian distributions. Then, 50 noise vectors are sampled from each distribution. The noise domain uses the same set of category indices as the target domain, with each noise category pre-mapped to a target category index; however, this mapping itself carries no semantic information.

The numbering here has no inherent meaning. Class 0 as noise does not naturally represent "cat"; it was simply fixed by researchers to correspond to a specific class in the target domain before training began. What is truly transferred is not the visual knowledge of cats or dogs, but the discriminative structure formed in the noise domain.

Noise from the same category is compressed together, while noise from different categories is pulled apart. As long as this structure aligns with the target domain, it provides clearer classification boundaries for real target samples.

A few labels are still necessary.

Noise can substitute for source data but cannot fully replace target domain labels. The reason is straightforward: noise originates from a different space, while class labels are artificially assigned; the model must rely on a small number of labeled target samples to determine which true class noise class 0 should be aligned with.

When the number of labeled samples per class was reduced to zero on CIFAR-100, the accuracy of ERM and NAF was only 0.97% and 1.34%, respectively, both nearing random levels. As soon as a small number of labeled samples were provided, NAF consistently outperformed ERM.

Therefore, the conclusion of this work is that in a semi-supervised classification setting, the true source domain is not necessarily a requirement for positive transfer.

Based on the generalization bound provided in the paper, NAF splits the training objective into three parts.

First, learn a small number of real labels.

The target domain encoder maps real samples into a shared representation space, and the classifier computes cross-entropy loss using a small number of labeled samples. This part is consistent with standard supervised learning and serves to establish a connection between noisy classes and true categories.

Let the noise form clear category structures.

The noise projector maps random vectors into the same representation space, while the classifier identifies these noises according to predefined category labels. After training, noises of the same class gradually cluster together, while noises of different classes become separated.

Finally, align both sides.

NAF also computes the distribution discrepancy between the source and target domains. The distribution alignment module is not restricted to any specific implementation; the paper compares multiple approaches and empirically adopts Negative Domain Similarity (NDS) as the default mechanism in experiments. NDS simultaneously compares the global means and class-wise means of both domains, using cosine similarity to bring them closer. The class-wise means of unlabeled target samples are iteratively estimated using pseudo-labels generated by the model.

The entire set of objectives can be written as

Among them,

Responsible for classifying a small number of real, labeled target samples,

Responsible for noise classification,

Responsible for aligning the distributions of the target domain and the noise domain. The generalization bound provided in the paper corresponds to these three terms: empirical error on the target domain, empirical error on the noise domain, and the distribution discrepancy between the two domains in the shared representation space.

The main experiment covers eight visual datasets and one text classification dataset. For visual tasks, except for ImageNet-1K, four labeled samples per class are used, with the remaining training samples treated as unlabeled data.

On ResNet-18, NAF achieves Top-1 accuracy improvements of +12.35 percentage points over ERM on CIFAR-10, +7.61 on CIFAR-100, +4.38 on DTD-47, and +2.74 on Caltech-101.

△

Fine-grained classification is equally effective. With ResNet-18, NAF improves by 8.94, 5.51, and 7.74 percentage points over ERM on CUB-200, Oxford Flowers-102, and Stanford Cars-196, respectively.

On the larger ImageNet-1K dataset, the research team retained 100 labeled samples per class. NAF achieved an accuracy of 37.10%, 0.99 percentage points higher than ERM. On the text task AG News-4, NAF with BERT achieved 82.82%, 4.18 percentage points higher than ERM’s 78.64%.

NAF can also be directly integrated into existing semi-supervised methods. The paper applies NAF to UDA, FixMatch, FlexMatch, DebiasMatch, DST, LERM, and SA-FixMatch, achieving consistent improvements across all. For example, with 20 rounds of training on CIFAR-10, incorporating NAF into UDA and FixMatch increased accuracy by 20.83 and 9.91 percentage points, respectively.

Why does noise help? The ablation study in the paper points to the same factor: whether there is a separable structure between classes.

The team first compressed all categories of noise into a single point. As a result, the noise domain completely lost its discriminative structure, causing NAF to not only gain no benefit but also exhibit significant negative transfer. The CIFAR-10 accuracy dropped from 58.15% with ERM to 33.34%, and CIFAR-100 fell from 42.24% to 6.79%.

In contrast, as the distance between noise class centers gradually increases, the CIFAR-100 accuracy improves from 43.80% to 49.78%. This indicates that the more distinguishable the noise classes are, the stronger the structural guidance they typically provide.

The number of noise samples is not as critical. When the number of noise samples per class increased from 10 to 100, accuracy remained stable at approximately 50%; it slightly decreased after increasing to 200. The paper concludes that, as long as a separable pattern can be formed, a small amount of noise is sufficient to be effective.

△

The research team also simplified the noise domain to a single center point per class. Regardless of whether the centers were fixed or learnable, the results surpassed ERM; learnable centers performed better than fixed centers but still fell short of the full NAF.

In the migration experiment from Amazon to Caltech-10, the real source data still performs slightly better overall, but the noisy source domain has already improved accuracy from ERM’s 83.51% to approximately 88%–89%. This provides a more realistic perspective: when real source data is unavailable, synthetic noise can serve as a low-cost alternative.

It also differs from common data augmentation, which typically involves rotations, cropping, interpolation, or generation near real samples; SSNA first constructs an independent noise domain and then achieves cross-domain alignment in the representation space.

This work offers a counterintuitive new perspective on transfer learning: source data can aid the target task not necessarily by relying on its true semantic meaning, but also through the category structure formed in the representation space, which may itself constitute transferable knowledge.

NAF leverages this by allowing random noise to form discriminative structures within a shared representation space, then transferring these structures to real data using only a small number of labeled samples. Experimental results show that even if the source domain contains no real images, text, or audio, as long as the appropriate class structure is preserved, it can still positively influence the target task.

In other words, what transfer learning transfers may not only be “what the data conveys,” but also “how the data is organized in the representation space.” This offers a new research direction for scenarios where privacy constraints, copyright sensitivity, or difficulty in accessing real source data are concerns.

Paper Title: Semi-Supervised Noise Adaptation: Transferring Knowledge from the Noise Domain

Paper URL: https://arxiv.org/pdf/2606.00558

Source code address: https://github.com/AIResearch-Group/SSNA

Video interpretation: https://www.bilibili.com/video/BV1UV7h61EvW/

This article is from the WeChat public account "Quantum Bit," authored by the SSNA team.