Olmo-core 3: Open Framework for Training Trillion-Parameter MoE

👁 114 reads

Allen AI released Olmo-core 3 on October 1, 2026 - a thoroughly redesigned, fully open framework for training large language models in Mixture-of-Experts (MoE) architecture that scales beyond a trillion parameters.

MoE is an architecture where a model contains many specialized components - experts - but activates only a subset of them for each input fragment. This allows you to build models with enormous parameter capacity without proportionally increasing computational costs for each forward pass. The problem is that as the number of experts grows, so does the cost of coordination and communication between GPU cluster nodes - and this overhead can eliminate most of the gains from selective activation.

New Engine Under the Hood

The key change in Olmo-core 3 is abandoning fully sharded data parallelism (FSDP) for MoE in favor of distributed data parallelism (DDP). In the previous approach, the framework collected and redistributed model weights with every small batch of training data. The new solution keeps experts permanently on assigned GPUs and routes the appropriate data to them - rather than the other way around. The effect is measurable: on eight NVIDIA B300 cards, a 47-billion-parameter model achieved 52,000 tokens per second per GPU compared to 19,400 in the previous implementation - 2.7 times more.

In a practical scalability test, the expert pool grew from 8 to 128, while still activating only four of them for each token. The number of active parameters per token remained similar - around 3.2 billion - while the total model capacity jumped from 4.6 billion to 47 billion parameters. Training throughput dropped by less than 5%. The same infrastructure was tested at over a trillion total parameters.

Openness as a Differentiator

Allen AI emphasizes that Olmo-core 3 is one of the key components of the next generation Olmo model and that all the code is publicly available. In an environment where large labs typically train their MoE models on closed infrastructure stacks - like NVIDIA Megatron-Core - an open framework with this level of performance can genuinely lower the barrier to entry for smaller academic centers and independent labs.

If the benchmark numbers hold up in production training runs, Olmo-core 3 will be one of the few truly open tools that keeps pace with closed solutions at this scale.

// read moreLiquid AI d1: Open Decision Model, AA Benchmark, and PerformanceNewsWhy Telecom Operators Are Choosing Open AI ModelsMagazineCompoWorld: Auto-Generating Training Environments for AI AgentsState of AI

Kinetyra