• by Admin

Cloud or On Premises? 94% of Companies Regret Their AI Decisions

There is a question that may sound simple but has turned out to be one of the most expensive decisions in a company's AI strategy: where should the AI workload actually live?

It may sound like a purely technical infrastructure issue. In reality, the answer connects to almost everything, including operating costs, data security, the level of control an organization holds, power requirements, and latency. Getting it wrong at the start can have consequences that last for years.

To understand how organizations are actually making this decision, a survey of more than 1,200 infrastructure leaders was conducted together with Omdia. The results turned out to be rather surprising.

Almost Everyone Has Regrets

The most striking figure from this research is that 94 percent of the organizations surveyed admitted to regretting their initial AI infrastructure strategy. That is not a small fraction or a minority. It is nearly everyone.

However, this is not a story about AI technology failing. The real issue has more to do with speed. Many teams made decisions faster than they were able to fully understand the economics, security requirements, and operational realities of the workloads they were running. Interestingly, 96 percent of organizations have already adopted a hybrid approach, combining cloud, on premises data centers, and edge infrastructure. The problem does not lie in the concept itself, but in the execution. Finding the right composition has proven to be far more difficult than expected.

The Real Mistake: Jumping In Without Workload Analysis

When asked what they would do differently given the chance to start over, infrastructure leaders did not simply say they should have switched platforms. Instead, their answers pointed to strategic gaps they only recognized in hindsight.

• 34% said they should have conducted more rigorous cost analysis before their first deployment. 

• 33% regretted not investing in their own infrastructure earlier. 

• 28% felt they should have pushed back harder against the overly rapid pace of AI adoption. 

A clear pattern emerges here. The mistake was not choosing cloud or on premises incorrectly but rushing ahead before truly understanding what the workload actually required.

So Where Should AI Workloads Actually Live?

Once AI goals become more specific, the cloud versus on premises debate stops being an endless argument and turns into a practical question of optimization. Fifty four percent of organizations cited workload specific optimization as the main reason they adopted a hybrid strategy.

A clear pattern is starting to emerge.

On premises tends to suit organizations that need full control. Custom models built on proprietary data, such as financial algorithms or manufacturing specifications, tend to remain, or are increasingly moved, on premises. The reasons include preserving data sovereignty, achieving tighter security, and maintaining a more predictable cost structure.

Cloud tends to suit organizations that need scale. AI functions that are more standardized and less sensitive are often better suited to cloud or hybrid environments, particularly when organizations need faster deployment and flexible scalability.

One manufacturing CIO who took part in the survey summarized the philosophy simply: "Anything that requires a real time decision stays on premises. Everything else can go to the cloud."

It sounds simple, but that is exactly the point. A clear and deliberate split, reviewed regularly, is what separates organizations that avoid regret from those that do not.

The Next Bottleneck Is Not Compute, It Is the Network

This is a part that is often overlooked. Many organizations are busy expanding server capacity for AI, yet they forget that the network is actually becoming a more urgent bottleneck. 67% of respondents expect their network to hit its capacity limits within the next 12 months due to the surge in AI traffic.

This is not a minor issue. If the network drops packets, expensive GPUs simply sit idle waiting for data. Idle GPUs mean delayed job completion times, which ultimately means wasted investment.

The problem becomes even more complex as organizations move into distributed AI and edge computing, where coordination between systems grows significantly more complicated. Ultimately, decisions about where to place workloads only pay off if the underlying infrastructure, including compute, network, and security, operates as a single integrated system, supported by unified observability and a consistent operational model.

AI Strategy Is Not a One-Time Bet

For anyone currently reevaluating their company's AI strategy, three principles are worth keeping in mind.

1. Audit your workload placement. Optimize based on each specific use case, weighing total cost, control, performance, and deployment speed across cloud, on premises, and edge environments.

2. Treat network capacity as a prerequisite rather than an afterthought. Do not wait until the network hits its limit before thinking about scale. The network is the foundation of AI performance.

3. Build your strategy as a single system. Data center and edge strategies should move in lockstep rather than being addressed one after another.

Ultimately, AI infrastructure decisions are not a one-time bet that can be made and then forgotten. What separates organizations that continue to move forward from those trapped in regret is a willingness to treat workload placement as an ongoing process, one that is continuously reviewed as the workloads themselves mature and evolve.

Author: Ghea Devita

Marketing Communication PT Perkom Indah Murni