Nous Research's TST Training Method Sparks Controversy Over Similarity to Previous Work

ME News reports that on May 14 (UTC+8), according to monitoring by Beating, Nous Research has unveiled a new pretraining method for large models called Token Stacking Training (TST). This approach compresses adjacent tokens into bundles during the early stages of training, reducing pretraining time by 2 to 3 times under the same computational budget. TST consists of two phases: during the first 20% to 40% of training, the model no longer processes tokens individually; instead, it averages adjacent tokens into bundles and predicts which tokens will appear in the next bundle (without preserving internal order). Subsequently, the model reverts to conventional next-token prediction. Since the underlying architecture remains unchanged, the resulting model behaves identically to standard models during inference. The method has been validated on MoE models with up to 10 billion parameters. At its core, this technique trades data for compute—accelerating training by consuming training data more rapidly. However, if high-quality text corpora become scarce in the future, this data-intensive nature could become a limitation. Additionally, hours after the paper’s release, readers noted that TST’s mechanism bears strong resemblance to a 2024 publication titled “Beyond Next Token Prediction.” The research team subsequently acknowledged on Hugging Face that this was an “unfortunate case of convergent research” and pledged to update the paper with proper citations. (Source: BlockBeats)