You think you're using AI, but it's actually AI using you.

Article author and source: 36Kr

Grok CLI 5.10 GiB upload event

In a real-world test, security researchers set up a 12 GB local code repository. During this task, the model’s interactive requests amounted to only 192 KB, but Grok silently uploaded a full 5.10 GiB compressed archive of the entire repository in the background.

The ratio is 27,800:1. Auxiliary programming? This is more like data migration.

On July 13, security researchers analyzing network traffic for the official xAI Grok CLI (npm package @xai-official/grok version 0.2.93) discovered that, before and after each task, Grok packages the current working directory into before_codebase.tar.gz and after_codebase.tar.gz, then silently uploads them via a separate side channel to xAI’s Google Cloud storage bucket. The uploaded archives contain .env keys, the complete .git history, ~/.claude.json outside the repository, and over 30 Skill files.

Security researchers have empirically verified that, even when system prompts explicitly restrict "no access to any local files," the full-package upload logic still triggers. The upload behavior is entirely independent of the model's task and is a hardcoded, mandatory process at the CLI's core.

xAI responded quickly. On the morning of July 13, they remotely disabled the default upload behavior via server-side settings. However, there was no announcement, no email notification to installed users, and no explanation for why this design existed in the first place.

Legally, it is difficult to define this as "theft"; a more accurate description is "uninformed, default bulk data collection," existing in a gray area between user agreements and product design. But gray does not mean acceptable. When product behavior exceeds users’ default expectations, trust can collapse with just one packet capture.

It's easy to vent emotions, but difficult to ask structural questions.

Everyone online is criticizing Musk for being hypocritical. He constantly denounces other AI companies as untrustworthy on X, yet his own tool has done the most untrustworthy thing. No matter how harshly he criticizes, it remains at the moral level.

The issue with Grok CLI stems from its architectural design. Version 0.2.93 embedded the upload logic into the default workflow, introducing a side channel, two sets of packaging files, and no user prompts throughout. The simultaneous presence of these elements makes it highly unlikely to be explained as "an engineer accidentally typoed a line of code."

A comparison reveals the difference. Claude Code Personal and the free version of GitHub Copilot default to an "incremental collection" logic, uploading only code snippets, modification histories, and error feedback that have been interacted with by the AI; users can manually disable the permission to "allow use for model improvement" in settings. The enterprise version supports data remaining within the local environment and not participating in training. This provides a buffer.

xAI is different. Full packaging, side-channel upload, no prompts at all, no toggle. Buffer zone? Removed outright.

Other companies are “taking quietly,” while xAI won’t even bother stepping through the door—they just take the entire wall. The difference isn’t about moral superiority, but about compliance granularity and technological maturity.

You think you're calling on AI, but it's actually AI calling on your workflow.

Here's a more intriguing detail.

What is Musk’s public stance in the tech community? He is the one who claimed that "the excessive concentration of AI power is dangerous"—suing OpenAI, opposing Microsoft’s monopoly, and advocating for open source and transparency. Yet xAI’s official tool secretly bundled and uploaded users’ code repositories, keys, and Git histories without their knowledge. The fix? A silent remote shutdown—with no announcement, no user notification, and no explanation.

A company claiming to combat AI centralization is using precisely the same architectural logic it opposes: enabled by default, invisible to users, one-way data transparency, and no post-hoc explanations.

This reveals a deeper industry reality: when model companies face dual pressures from compute costs and training data, "user control" is the first variable to be sacrificed. Training code generation capabilities with code is one of the lowest-cost paths with the highest data quality. Rather than spending money on datasets, it’s more efficient to directly feed the model with repositories from real developers. Musk isn’t unaware that this approach has issues—he simply chose to prioritize getting the business model up and running first.

But this is not merely a personal contradiction in Musk’s actions—it’s a collective paradox of the entire AI industry. Companies all promote the narrative that “AI empowers individuals and enterprises,” yet the underlying business model of every major player is built on the logic of “using user data to feed back into models.” The louder the proclaimed values, the more the business model needs to be masked by moral framing. xAI simply makes it more explicit, directly embedding the industry’s unspoken rule into its product code.

This aggressive data collection is no accident. When general internet data has already been fully consumed by large models, high-quality industrial-grade code and real-world enterprise business logic have become the core fuel for the next generation of model iterations. Whoever acquires more real engineering data will gain a generational advantage in coding capability.

Many people think: “Isn’t it enough just to delete the sensitive fields? Clear the keys from the .env file, anonymize customer data, and remove hardcoded passwords.”

That idea is too naive.

What the model extracts from your code is not the plain-text keys themselves, but rather the architectural concepts, debugging experience, business logic, and engineering patterns. A SaaS company using AI to write core code for a customer management system—even after removing all customer data and plain-text keys—will still have its concurrency handling logic, role-based access architecture, fallback error handling strategies, and database indexing design absorbed by the model.

When competitors in the same space use the same AI tool, they indirectly gain access to the costly lessons you’ve learned over hundreds of thousands of dollars. The model freely teaches your architectural insights to your rivals, and you may never even know it’s happening.

What's masked is your face, but the model observes your gait.

After this incident, every company using AI programming tools must reassess: How much is your code asset really worth?

It is recommended to divide the code into three levels.

Level 1: Non-core code (general-purpose tools). Open-source components, general scripts, frontend pages, and internal utility code; leakage does not affect core competitiveness.

Level 2: Core business code (requires on-premises deployment). The product architecture, business logic, proprietary algorithms, and permission system must use an on-premises deployment model, ensuring data never leaves the enterprise network.

Level 3: Confidential Code (External AI Prohibited). Encryption protocols, risk control models, core patented algorithms, and unpublished underlying architecture are strictly prohibited from integration with any external AI tools and must be developed and audited manually.

The gap in AI coding capabilities among future enterprises isn't just about model strength—it's about whether core code has flowed into the training pool of general-purpose models.

This misstep by xAI won’t kill AI programming tools, but it will kill the “upload by default” business model. Regulation will inevitably follow, but regulation always lags behind technology. The real driver of change is the market: when companies realize the true cost of “free trials” is the loss of their code assets, they will vote with their feet.

An counterintuitive truth: Many believe open-source AI tools are safer, but in reality, many open-source coding tools also contain hidden data reporting logic. Security boundaries have nothing to do with whether software is open or closed source—only whether your data leaves your local environment.

Following this incident, the privacy policies of all AI coding tools will be forced into reassessment. Regulation did not kill this model—the market put a price on it.

Your code taught Grok how to write better code, but Grok won't tell you what it learned.

This is probably the most honest definition of a free tool.