Google DeepMind’s Decoupled DiLoCo Technology
Treating Global Data Centers as a Single Unit
Advancing Google’s Expertise in Computer Fault Tolerance

Editor's NoteThis series aims to explain the essential yet often unfamiliar technologies that shape our everyday lives, from AI and semiconductors to communications and biotechnology, in an easy-to-understand way.

The greatest challenge in artificial intelligence (AI) development is the data center. Big Tech companies are investing in data centers to secure more computer chips than their competitors. However, if it became possible to combine the countless data centers spread across the globe to train AI, the chip bottleneck problem plaguing tech companies could be dramatically alleviated. This is why Google’s advanced AI research lab, DeepMind, is drawing attention with the release of its “Decoupled DiLoCo” technology.


Google DeepMind’s Next-Generation AI Training Technology ‘Decoupled DiLoCo’


An image illustrating the principle of decoupled diloco. By treating AI data centers distributed worldwide as a single computer, it is possible to realize a "Global Data Center." Google DeepMind

An image illustrating the principle of decoupled diloco. By treating AI data centers distributed worldwide as a single computer, it is possible to realize a "Global Data Center." Google DeepMind

View original image

Decoupled DiLoCo is a technology unveiled by DeepMind on April 24 (local time) via its website and the preprint platform Arxiv. The name stands for “Decoupled Distributed Low-Communication” AI training. While the terminology may sound daunting, breaking it down makes the concept much more accessible.


“Decoupled” refers to the loose connection between computer chips for AI. Currently, tens of thousands of GPUs installed in AI data centers can transmit and receive data at ultra-high speeds, thanks to chiplet technology (the process of combining diverse semiconductors via high-speed interfaces) and optical cables. This allows the data center to be managed as if it were a giant single computer—a process known as “synchronization.” DeepMind’s breakthrough is that it can manage tens of thousands of AI chips as a single unit even without sophisticated synchronization technology.


“Distributed low-communication” refers to communication over geographically distant, relatively slow networks. With Decoupled DiLoCo, DeepMind succeeded in integrating four data centers spread across North America into one, using only ordinary internet speeds of 2–5 gigabytes (GB) per second.


Utilizing Computer Chips Worldwide Simultaneously


In other words, Decoupled DiLoCo is a technology that enables the simultaneous control of AI graphics processing units (GPUs), tensor processing units (TPUs), and more, even if they are located far apart, and without expensive, cutting-edge equipment. This means you can borrow computing power from any data center around the world for AI training and inference. Essentially, it is a technology with the potential to realize a “global-scale data center.”


Decoupled DiLoCo overcomes the limitations of conventional AI training. As AI training relies heavily on data exchange between chips and memory, a drop in synchronization can cause a sharp decline in AI performance or, in severe cases, cause models to fail completely.


Fault Tolerance Technology by Further Dividing GPU Bundles


Difference between AI training divided by GPU slices (above) and AI training divided by islands. Even if the error frequency is the same, there is a difference in the number of GPUs that must stop (marked in red), causing a large gap in overall training performance and training time. Google DeepMind

Difference between AI training divided by GPU slices (above) and AI training divided by islands. Even if the error frequency is the same, there is a difference in the number of GPUs that must stop (marked in red), causing a large gap in overall training performance and training time. Google DeepMind

View original image

The secret to DeepMind’s breakthrough lies in a computer operation technique known as “fault tolerance.” Fault tolerance refers to all measures that allow a computer network to continue operating even if some components fail. For example, if certain devices in a data center are damaged, immediately blocking errors and activating backup systems are part of fault tolerance.


Fault tolerance is essential in AI data centers. With tens of thousands of GPUs operating simultaneously, some components are bound to malfunction. Rapid detection and isolation of faulty components are critical to prevent errors from spreading throughout the system. Until now, Big Tech companies have achieved fault tolerance by grouping GPUs into “slices” of hundreds to thousands of units. Decoupled DiLoCo takes this further by subdividing slices into even smaller “islands.” Previously, entire GPU slices had to be halted in response to errors, but now, problems can be contained at the island level, minimizing the impact.


DeepMind explains, “By separating hardware at the island level rather than by slices, the frequency of errors remains the same, but the impact of each error is reduced. As a result, operations can continue with little to no interruption, and data center bandwidth (communication speed) requirements are drastically reduced, enabling AI training to proceed even over long distances.”


Saving Chip Costs Through Large-Scale Distributed Training


Google Data Center. Google

Google Data Center. Google

View original image

If all GPUs can be used for AI tasks regardless of data center location, the cost of purchasing chips can be significantly reduced. Chips that were previously idle due to distance or logistical reasons can be fully utilized for training.



The advantages of Decoupled DiLoCo do not end there. Because the level of synchronization between chips is low, chips from different generations can be mixed and used together. For example, DeepMind was able to train AI models using both fifth- and sixth-generation TPUs, thanks to Decoupled DiLoCo. DeepMind emphasized, “Next-generation hardware is not released simultaneously across all data centers. By using multiple generations of chips together, physical logistics bottlenecks can be avoided, and even older hardware can meaningfully improve AI training speed.”


This content was produced with the assistance of AI translation services.

© The Asia Business Daily. All rights reserved. Unauthorized AI training and use prohibited.

Today’s Briefing