If All the World's GPUs Could Be Used at Once... Google's Ambition [Tech Talk]
Google DeepMind’s Decoupled DiLoCo Technology
Treating Global Data Centers as a Single Unit
Advancing Google’s Expertise in Computer Fault Tolerance
The greatest challenge in artificial intelligence (AI) development is the data center. Big Tech companies are investing in data centers to secure more computer chips than their competitors. However, if it became possible to combine the countless data centers spread across the globe to train AI, the chip bottleneck problem plaguing tech companies could be dramatically alleviated. This is why Google’s advanced AI research lab, DeepMind, is drawing attention with the release of its “Decoupled DiLoCo” technology.
Google DeepMind’s Next-Generation AI Training Technology ‘Decoupled DiLoCo’
An image illustrating the principle of decoupled diloco. By treating AI data centers distributed worldwide as a single computer, it is possible to realize a "Global Data Center." Google DeepMind
View original imageDecoupled DiLoCo is a technology unveiled by DeepMind on April 24 (local time) via its website and the preprint platform Arxiv. The name stands for “Decoupled Distributed Low-Communication” AI training. While the terminology may sound daunting, breaking it down makes the concept much more accessible.
“Decoupled” refers to the loose connection between computer chips for AI. Currently, tens of thousands of GPUs installed in AI data centers can transmit and receive data at ultra-high speeds, thanks to chiplet technology (the process of combining diverse semiconductors via high-speed interfaces) and optical cables. This allows the data center to be managed as if it were a giant single computer—a process known as “synchronization.” DeepMind’s breakthrough is that it can manage tens of thousands of AI chips as a single unit even without sophisticated synchronization technology.
“Distributed low-communication” refers to communication over geographically distant, relatively slow networks. With Decoupled DiLoCo, DeepMind succeeded in integrating four data centers spread across North America into one, using only ordinary internet speeds of 2–5 gigabytes (GB) per second.
Utilizing Computer Chips Worldwide Simultaneously
In other words, Decoupled DiLoCo is a technology that enables the simultaneous control of AI graphics processing units (GPUs), tensor processing units (TPUs), and more, even if they are located far apart, and without expensive, cutting-edge equipment. This means you can borrow computing power from any data center around the world for AI training and inference. Essentially, it is a technology with the potential to realize a “global-scale data center.”
Decoupled DiLoCo overcomes the limitations of conventional AI training. As AI training relies heavily on data exchange between chips and memory, a drop in synchronization can cause a sharp decline in AI performance or, in severe cases, cause models to fail completely.
Fault Tolerance Technology by Further Dividing GPU Bundles
Difference between AI training divided by GPU slices (above) and AI training divided by islands. Even if the error frequency is the same, there is a difference in the number of GPUs that must stop (marked in red), causing a large gap in overall training performance and training time. Google DeepMind
View original imageThe secret to DeepMind’s breakthrough lies in a computer operation technique known as “fault tolerance.” Fault tolerance refers to all measures that allow a computer network to continue operating even if some components fail. For example, if certain devices in a data center are damaged, immediately blocking errors and activating backup systems are part of fault tolerance.
Fault tolerance is essential in AI data centers. With tens of thousands of GPUs operating simultaneously, some components are bound to malfunction. Rapid detection and isolation of faulty components are critical to prevent errors from spreading throughout the system. Until now, Big Tech companies have achieved fault tolerance by grouping GPUs into “slices” of hundreds to thousands of units. Decoupled DiLoCo takes this further by subdividing slices into even smaller “islands.” Previously, entire GPU slices had to be halted in response to errors, but now, problems can be contained at the island level, minimizing the impact.
DeepMind explains, “By separating hardware at the island level rather than by slices, the frequency of errors remains the same, but the impact of each error is reduced. As a result, operations can continue with little to no interruption, and data center bandwidth (communication speed) requirements are drastically reduced, enabling AI training to proceed even over long distances.”
Saving Chip Costs Through Large-Scale Distributed Training
If all GPUs can be used for AI tasks regardless of data center location, the cost of purchasing chips can be significantly reduced. Chips that were previously idle due to distance or logistical reasons can be fully utilized for training.
Hot Picks Today
"Two Spend 500,000 Won in a Flash, Yet Wait Two Hours?" ... 'Dreamland' Shunned by Kids and Foreign Tourists
- The Era of Getting 200 Million Won After 30 Years Is Over... A Japanese Bank's "Perpetual Department Head" Who Just Clocked In Cries Out
- Spain Hit by Its Worst Tragedy... Lee Kangin, Moreno and Real Madrid Join Soccer World in Mourning
- They Said, “Make It Your Home for Life” - Then Suddenly Kicked 200 Passengers Off... What Happened to the Around-the-World Cruise?
- "130 Rice Bowls Sold Out in a Day"... 'Unfiltered Face' Adds 8 Million Won to Night Market Sales: China's Bungeunwoo
The advantages of Decoupled DiLoCo do not end there. Because the level of synchronization between chips is low, chips from different generations can be mixed and used together. For example, DeepMind was able to train AI models using both fifth- and sixth-generation TPUs, thanks to Decoupled DiLoCo. DeepMind emphasized, “Next-generation hardware is not released simultaneously across all data centers. By using multiple generations of chips together, physical logistics bottlenecks can be avoided, and even older hardware can meaningfully improve AI training speed.”
© The Asia Business Daily. All rights reserved. Unauthorized AI training and use prohibited.