The first two instalments of this series have explored why infrastructure strategy demands boardroom attention and what happens when the pressure to move fast overrides the discipline to build well. But while understanding where organisations can go wrong is an important starting point, the harder and more important question is what follows.
What does getting it right actually look like?
In the latest instalment of our ‘In Conversation With’ series with Cao Hoang, Stelia’s VP of Platform engineering, we move from problem to real-world benchmarks – defining what truly production-grade AI infrastructure enables for an organisation, and what the teams who will win at production AI over the next few years are doing to ensure that right now.
What does good actually look like and what does truly production-grade AI infrastructure deliver for an organisation?
Good looks remarkably boring – because the chaos has been engineered out. It acts as an invisible, intelligent enabler for product teams. At a technical level it boils down to a few core areas.
You need to start with reliability – graceful degradation and self-healing when hardware inevitably fails under heavy loads. Then there’s elasticity and utilisation. It’s not just about scaling up. The underlying platform needs to maximise the utilisation of expensive compute assets, dynamically provisioning resources based on inference queue depth, utilising continuous batching and KV cache sharing via serving engines like vLLM or TensorRT-LLM, and scaling down to zero to protect profit margins when demand drops. As investment in compute infrastructure accelerates, this is becoming absolutely key.
Beyond that, observability has to go deeper than standard CPU and RAM metrics. You need hardware-level tracing to track time-to-first-token, time-between-tokens, tokens-per-second, and GPU memory fragmentation – so you know exactly why a response was slow. And finally, it needs to bake in zero-trust network isolation and strict role-based access control directly at the orchestration layer.
When all of that is in place, infrastructure stops being something you manage around and starts to become the thing that makes everything else possible.
It is a high bar to account for all of these different layers within one system, and one that most organisations are still some way from clearing. But the competitive implications of doing so are significant – and as Cao explains, the distance between the teams making smart choices and those trailing behind is already beginning to show.
What will the teams that lead with production-grade AI have got right that most organisations are currently getting wrong?
The teams that win treat AI as a rigorous software and distributed systems engineering discipline, rather than a sprawling, isolated data science experiment.
Right now, many organisations build AI in silos, treating it as a separate data science function rather than part of the standard software lifecycle. The winners are bridging this gap with dedicated developer platforms tailored for AI, like Stelia’s, which abstract away the infrastructure complexity so machine learning engineers can iterate quickly, while subjecting models to the same rigour of version control, automated testing, and CI/CD as any traditional microservice.
They aren’t just betting on having the smartest model. They are building the most resilient and scalable factory floor to safely deliver any model to their users.
It is the kind of rigour that we have built our engineering practice around – working with some of the world’s most complex organisations, with some of the most complex challenges, to close the gap between AI experimentation and production at scale.
What we have consistently witnessed is that the teams that get there fastest are not the ones that move carelessly in the early stages. They are the ones who made fewer decisions they later had to unwind, built platforms their engineers could trust, and treated infrastructure not as a cost to be minimised but as a foundation to be invested in.
