The most absurd part is: I have no background in machine learning, don’t know the standard model training process, and am unfamiliar with many technical details. What I did was essentially just keep giving Sol requirements and feedback on results, letting it figure out the problems, design experiments, and continuously iterate on its own.Author: Anshu
Article compiled, source: ME News
This is the first time I’ve truly felt like “AGI has already arrived.”
I trained my own automatic correction model using GPT-5.6 Sol. In the end, this local model with only 1.7 billion parameters performed slightly better on the test set than GPT-5.6 Sol.
The most absurd part is: I have no background in machine learning, don’t know the standard model training process, and am unfamiliar with many technical details. What I did was essentially just keep giving Sol requirements and feedback on results, letting it figure out the problems, design experiments, and continuously iterate on its own.
The entire process costs nothing.
It all started with an increasingly serious "typing problem"
After chatting with AI for a long time, I’ve noticed my typing skills getting worse.
I’ve become accustomed to typing quickly and no longer carefully check for spelling, letter order, or missing characters. Instead of retraining my typing skills, I’ve decided to adopt a solution better suited to the AI era: using even more AI to solve the problem.
Traditional auto-correction continuously modifies text during input, which can easily disrupt your train of thought. My idea is to let users type quickly without interference, even if their input contains many errors, and then let AI clean it up all at once after they finish typing.
At the same time, I want this model to be as small as possible.
Smaller models run faster, consume less power, and are better suited for fully local operation. Whether for efficiency, battery life, or simply out of curiosity, I wanted to see: to what extent can a small enough local model achieve automatic error correction?
So, I decided to train one myself.
Make Sol an automated research scientist
This project was inspired by Andrej Karpathy's "autoresearch" experiment.
I used Codex's /goal mode to design a cyclical workflow for Sol:
Choose an experiment, carry it out, and record the results in the document; if it fails, abandon this path; then plan the next experiment while avoiding previously verified mistakes.
I provided only a few required input examples, strict latency targets, and the desired final outcome, then let Sol run on its own.
What happened next exceeded my expectations.
Sol first retrieved and compared multiple candidate base models, including Qwen 3.5, Gemma 4, and Liquid LFM 2.5. It then found a dataset on Hugging Face related to real typing text.
But the real data is still not enough.
To generate spelling errors closer to actual user input, Sol wrote a simulator that mimics "fingertips typing on a Mac keyboard." It uses a Gaussian distribution to model fingertip landing positions based on the keyboard's physical layout and generates various common errors, such as:
- Press adjacent keys;
- Letters reversed;
- Enter again;
- Missing character;
- Fingers touching multiple keys at the same time.
With the base model, text data, and keyboard error simulator in place, Sol fine-tuned directly on my MacBook using MLX.
In less than an hour, it produced a working prototype.
The problem is that the accuracy of the first version was not ideal.
First bottleneck: The tokenizer cannot understand spelling errors.
Sol read the relevant papers and designed a series of tests, ultimately concluding that the model’s main bottleneck was not in the training data, but in the Tokenizer, or tokenizer.
Large language models typically do not understand text letter by letter; instead, they first split the text into tokens. Normal words can be broken down into stable semantic units, but spelling errors often disrupt the original token structure.
This means that a letter error that is obvious to humans may appear to the model as a completely unfamiliar set of tokens.
The model struggles to truly "understand" errors and can only mechanically memorize mappings between misspellings and correct spellings. This approach results in poor generalization and fails to leverage the model’s existing language knowledge effectively.
Sol first tried Google's ByT5.
ByT5 is a model that processes byte sequences directly without relying on traditional tokenizers. This approach yielded noticeable improvements, but since ByT5 was released earlier and has limited language knowledge, its final performance still falls short of GPT-5.6 Sol.
After further research, Sol realized that the issue didn't necessarily require "completely canceling the Tokenizer" to be resolved.
It instead chose T5Gemma, a model with an Encoder-Decoder architecture.
Unlike models that simply predict the next token, an Encoder-Decoder model first fully understands the input through the encoder, then generates corrected text via the decoder. More importantly, Sol can further fine-tune the encoder to help the model better recognize inputs containing spelling errors.
This route significantly raises the model's performance ceiling.
Second bottleneck: Traditional loss functions encourage the model to "not modify"
After changing the model architecture, a new issue has emerged.
The model has been able to accurately correct some errors, but often overlooks other obvious spelling issues. Even when errors are present in the input, it tends to copy them verbatim.
Sol eventually discovered that the issue stemmed from the most common cross-entropy loss function.
In the auto-correction data, most characters are already correct, and only a very small proportion actually require modification. If trained directly with standard cross-entropy, the model’s safest strategy will be to “change as little as possible.”
Because copying the original text yields the correct answer in most cases, while actively modifying it may introduce errors.
In other words, traditional training objectives are rewarding models for remaining unchanged.
To solve this problem, Sol wrote a set of custom loss functions.
It first aligns the original text with the target text at the byte level, then uses a dynamic programming algorithm to compute the minimum edit path between the two texts, identifying which positions are copies and which are actual insertions, deletions, or substitutions.
On this basis, Sol significantly increased the training weight for "correct modifications" while reducing the reward for merely copying characters.
After multiple rounds of parameter tuning, the model's correction accuracy has significantly improved.
Third bottleneck: Once the model goes off track, it cannot backtrack.
The final major issue stems from the autoregressive generation mechanism.
When generating text, the model can only predict the next token based on what has already been generated. If an error occurs at any earlier step, subsequent generations will be built upon that incorrect result, and the model cannot truly go back to correct it.
In theory, a model can be trained to "think" before answering, like a reasoning model, but this would significantly increase latency and is unsuitable for auto-correction scenarios requiring immediate responses.
Sol eventually found a more elegant solution: Beam Search.
The model no longer selects only the single highest-probability path at each step; instead, it simultaneously retains multiple possible generation branches and explores different correction outcomes in parallel. After the search is complete, the full path with the highest cumulative log probability is selected.
This is equivalent to replacing single-threaded reasoning with parallel search.
Beam Search significantly improves the final result, but introduces an experience issue: users see no output until the entire search is complete.
Sol then made a very insightful observation.
After each round of search, all remaining branches can be compared. As long as these branches share the same beginning, this “longest common prefix” will inevitably appear in the final result.
Therefore, the system can immediately display this content to the user.
As the search continues, weaker paths are gradually eliminated, and the common prefix of the remaining branches becomes increasingly longer. Ultimately, users see not a result that appears all at once, but continuously generated corrective text.
Sol has turned the entire process into a custom MLX inference pipeline, leveraging the MacBook GPU for parallel decoding.
Ultimately, the output latency for the first token was only about 40 milliseconds—fast enough, and the entire process was completed locally.
Final result: 1.7 billion parameter model outperforms GPT-5.6 Sol
The final evaluation uses "error reduction rate" as the metric, where a higher value indicates that the model has corrected more input errors.
The review results are as follows:
- Apple autocorrect: 49.66%
- GPT-5.6 Luna: 82.47%
- GPT-5.6 Terra: 87.64%
- GPT-5.6 Sol: 90.56%
- Our trained model with 1.7 billion parameters: 91.02%
This local small model ultimately surpassed GPT-5.6 Sol by a very narrow margin.
I also specifically checked for data leakage or model "cheating." During testing, we actively exclude words that appear in the training data to verify that the model is not merely memorizing mappings between errors and answers.
The final cost of the entire project is:
A model quota reset and $0 cash outlay.
What truly impressed me wasn't just the final score
During the project, numerous experiments were not fully explored, including directions such as contrastive learning, GRPO, DPO, and dynamic masking.
Not all attempts succeeded, but Sol proactively read materials, identified issues, formulated hypotheses, designed experiments, analyzed results, and used lessons from failures to plan the next round of attempts.
For me, what’s truly astonishing isn’t the fact that “a 1.7 billion parameter model outperformed GPT-5.6 Sol.”
More importantly, someone with no background in machine learning can now use AI to carry out experimental workflows that previously required a professional research team.
I didn’t have a complete understanding of all the underlying knowledge or a pre-designed technical roadmap. I simply knew what problem I wanted to solve, and I kept pushing Sol to keep finding answers.
It doesn't just write code—it also takes on the roles of researcher, engineer, and experimental designer.
This may have been the first time I truly “felt” AGI.
Don't let inexperience stop you from getting started with experiments.
When AI can help ordinary people overcome professional barriers, many technology projects that once seemed out of reach may no longer be so far away.