While “GPU cloud” is now common language in AI, its architectural implications are often blurred.
That distinction matters. Applying traditional cloud assumptions to GPU infrastructure can result in systems that struggle to scale reliably in production, limiting both technical performance and business impact.
Our new series dives into the architectural decisions required to build reliable, robust, and governable cloud infrastructure for AI. Starting from the very beginning.
Our first piece defines what constitutes a true cloud, explains why GPU environments change the design constraints, and outlines the systems-level considerations required for robust AI at scale.
What actually is a cloud? (vs what people think it is)
At its most basic, cloud computing is the abstraction of infrastructure and resources – making compute, storage, and networking available on demand, self-service, and at scale, accessible via API and shared across multiple tenants. It’s the shift from managing physical infrastructure to consuming it as a flexible, elastic resource.
But the term has since become so ubiquitous that its meaning has blurred. Many assume any remote compute resource accessed via API qualifies as cloud infrastructure.
Real cloud infrastructure requires specific architectural foundations, whereby:
- Redundancy is built in everywhere as core design, because scheduling maintenance windows doesn’t scale when serving thousands of customers.
- Isolation serves not only as a guarantee of security and privacy between tenants, but also as a quality-of-service assurance, preventing “noisy neighbour” scenarios where one customer’s workloads degrade another’s performance.
- Standardisation allows workloads to move seamlessly between physical machines and enables efficient capital allocation at scale by procuring only necessary components across large deployments. Standardised rack-scale products can be deployed uniformly and reallocated across customers as demands shift.
- Minimised state throughout the system maintains resilience rather than fragility.
These operational necessities are just some of the considerations that determine whether infrastructure can scale reliably, remain available during failures, and support diverse workloads simultaneously.
However, what often gets labelled as “cloud” in practice falls short of these requirements. In some cases, providers purchase servers, configure them, and provide direct access to the hardware, which is essentially managed infrastructure, not cloud architecture. Or in others, providers take cluster management tools originally designed for single-tenant research computing (shared within one organisation, not across isolated tenants) and add APIs to approximate cloud-like access. Neither approach delivers the architectural foundations that make infrastructure actually function as a cloud.
Understanding this distinction is incredibly important because the operational characteristics of each service differ at their core. Managed hardware shifts the risk and complexity to customers, while retrofitted cluster managers inherit the brittleness and operational overhead they were never designed to eliminate. Actual cloud infrastructure is architected from the ground up to handle these challenges.
What is a GPU cloud, and why is it different?
A GPU cloud applies these same cloud principles – on-demand access, multi-tenancy, elasticity, API-driven infrastructure – but built around accelerated compute rather than general-purpose servers. For AI workloads, this largely means GPUs, which are uniquely suited to the parallel processing demands of training and running AI models at scale.
Intuitively, it can easily be assumed that a GPU cloud simply means adding GPUs to existing cloud infrastructure. This is an assumption we hear a lot, even from experienced engineers. In practice, however, this assumption underestimates the challenges GPU workloads bring.
GPU workloads fundamentally change what infrastructure must do and how it must be designed. The physical footprint alone is dramatically different: power consumption, cooling requirements, and network demands operate at a different order of magnitude than traditional cloud infrastructure. Take networking, for example: requirements shift both physically – with interconnected nodes requiring high-speed fabrics – and in traffic patterns, generating massive east-west traffic that exhausts traditional network architectures designed for 50Gbps and below. On top of this, the software and configuration requirements of AI workloads mean that multi-tenant isolation, ensuring one customer’s environment doesn’t interfere with another’s, becomes significantly more complex to achieve.
Beyond scale, GPU infrastructure also requires thinking differently about trade-offs. The default instinct is often to maximise performance on individual components: the fastest storage, the highest network throughput, the most capable models. But production infrastructure doesn’t succeed by optimising components in isolation. What matters is how the entire system performs together under real operational conditions.
Strategically, the implication is significant. GPU infrastructure isn’t an incremental evolution of traditional cloud computing. It requires fundamentally reconsidering how each layer of the stack is designed, what trade-offs actually matter in production, and how to think about the system as a whole.
Beyond individual components
Building cloud infrastructure that performs reliably for AI workloads isn’t primarily a hardware or software problem alone, or indeed a problem of any one component. It’s a systems challenge.
Every individual piece of the puzzle contributes collectively to reliable performance at scale, as each decision made at one layer of the stack implicates every other layer in turn.
This matters strategically because the decisions that determine whether AI workloads are run successfully at scale aren’t always the most visible ones. The most expensive options don’t guarantee the best outcomes, and the most well-known platforms don’t always translate to production-grade reliability. And critically, the gap between systems that work in controlled conditions and systems that perform under real operational demand – at scale, under load, across diverse workloads – is almost always determined by how well the system was designed as a whole.
The foundation for what follows
Understanding this systems-level perspective is the foundation for everything that follows in this series, as we explore each layer that comes together to comprise robust, reliable cloud infrastructure for AI – from hardware selection through to virtualisation – and how each operates as part of one interconnected whole to deliver high-value business outcomes and strong ROI.
