What is AI Infrastructure?
Hosting AI workloads isn’t like running traditional cloud apps. The infrastructure looks more like a high-performance computing (HPC) cluster than a standard cloud setup.
At its core, AI infrastructure is about: 
- Compute power – GPU-heavy clusters, often built with Nvidia’s A100s, H100s, or TPUs.
- Storage – High-speed, distributed storage systems capable of moving petabytes of training data.
- Networking – The glue that holds AI clusters together, ensuring GPUs communicate at ultra-low latencies.
And this last part,networking, is where the real battle is happening.
InfiniBand: The High-Performance Standard
InfiniBand has long been the go-to networking fabric for HPC and AI.
- Low and stable latency – Crucial for AI training at scale.
- Lossless network fabric – Uses credit-based flow control instead of best-effort packet delivery.
- Adaptive Routing – Dynamically selects the best path to avoid congestion.
- Tightly controlled ecosystem – Nvidia now owns InfiniBand (via Mellanox), meaning it’s deeply integrated into Nvidia’s AI solutions.
InfiniBand is essentially the high-performance networking option with no compromises, but it locks users into a closed ecosystem.
RoCE: The Ethernet-Based Challenger
RoCE (RDMA-over-Converged-Ethernet) is an alternative to InfiniBand that allows RDMA over standard Ethernet.
- Runs on Ethernet fabric – making it easier to adopt than InfiniBand.
- Cost-effective – compared to InfiniBand.
- Hyperscalers are leading adoption –Meta, Tesla, and others are deploying massive RoCE-based GPU clusters.
However, RoCE isn’t perfect. For example, Meta have published a whitepaper detailing their journey to getting congestion control and priority flow control working to a high standard in large-scale deployments. This is where the Ultra Ethernet Consortium (UEC) comes in who will (hopefully) implement new standards in a new RoCE version.
Who’s Backing What?
| Company | Backing |
|---|---|
| Nvidia (Mellanox) | InfiniBand |
| Meta, Tesla, other ‘Hyperscalers’, UEC | RoCE |
| Stelia | RoCE |
Where is This All Going?
Stelia, among other UEC members, believes InfiniBand is on borrowed time—and that the future of AI networking lies in open standards, and have planted the flag in the ground. We believe in innovating in the open – and that is what we will push internally and externally as part of that mission.
InfiniBand has dominated for decades, but Ethernet-based solutions like RoCE are improving rapidly. The Ultra Ethernet Consortium (UEC) is aggressively working on implementing new standards for RoCE specific to AI, and if they succeed, InfiniBand could go the way of other once-dominant, closed networking standards.
The future of AI networking is still being written. Will RoCE fully take over? Will Nvidia adjust its strategy? Nvidia’s Spectrum-X product line is Ethernet based and looks to be a compelling choice.
One thing’s for sure—this is a space worth watching.
What’s Next?
Want to dig deeper into open networking for AI? Stay tuned for our next piece on SONiC (Software-for-Open-Networking-in-the-Cloud), the next frontier in AI data center networking.
Join Us at Nvidia GTC
If you’re looking to avoid the fate of companies that clung to siloed architectures and missed the hyperscale boat, don’t repeat the past. Instead, discover how to build on the lessons learned by hyperscalers and apply them to the AI revolution.
NVIDIA #GTC2025 Conference Session Catalog
Attend our session, “Beyond Silos: Unlocking AI’s Full Potential with Petabit-Scale Data Mobility,” Tuesday, Mar 18 4:20 PM – 4:35 PM PDT and learn how interconnected, elastic infrastructures are transforming AI at every level. We’ll dissect:
- Why traditional cloud computing creates bottlenecks for AI
- How a petabit-scale platform accelerates data mobility
- The blueprint for building an interconnected compute model
Ready to break free from the siloed past?
Join us at Nvidia GTC for a 15-minute live presentation and Q&A that could change the way you think about AI infrastructure forever.
