ByteDance reportedly plans 10 trillion total-parameter model with 30,000 GPUs

ByteDance is reportedly gearing up to pre-train an AI model with roughly 10 trillion parameters, a scale that would dwarf every known Chinese AI system and put the company in direct competition with the most advanced Western labs. The effort would require approximately 30,000 GPUs and an estimated 3 to 6 months of continuous pre-training. To put 10 trillion parameters in perspective, that is more than three times the size of Moonshot AI’s Kimi K3, which sits at 2.8 trillion parameters and currently ranks among the largest models produced in China. Parameters are essentially the knobs an AI model tunes during training to learn patterns in data. What ByteDance is actually building The model in question uses a Mixture of Experts (MoE) architecture. Rather than activating every parameter for every query, MoE models route each input to a subset of specialized “expert” sub-networks. This makes them far more efficient to run at inference time than a dense model of the same total size. ByteDance’s Seed AI team, which reportedly consists of around 2,000 staff members, is leading the effort. The team has prior experience scaling training runs, having previously trained models up to 175 billi...