The world is far more complex that people imagine. Often when reading arguments for or against using AI locally the author reveals their own circumstances rather than judgements about the world. It’s very hard to comprehend the true scale and complexity of the global economy and all of the actors in it. We just look at the world through a tiny pin prick and don’t see all of the dynamics and circumstances which even large sections of the population are subject to.
At the macro scale we know that cloud compute dominates, but it is actually not clear to what extent and why. The simplest explanation is that it’s most efficient to share compute. Almost anywhere in the world you can access a slice of the best compute just for the period of time that you need it.
Some influencers have gone so far as to suggest that anyone trying to run LLMs locally are insane. This of course will be true in some cases, but the reality is that insanity is relative. Quite often what appears to be insane in one set of circumstances actually appears perfectly rationale in another. A good exercise is to imagine how things would have gone for some very successful people if events outside of their control had gone slightly differently.
However let’s get down to some of the reasons that organisations and individuals choose to run AI locally.
Network
Unimaginable to some people who live in cities is that the internet could be seen as a liability. However for some organisations dealing with an unreliable connection to the internet is a daily concern.
For remote monitoring stations and facilities the only source of a network connection is satellite or terrestrial radio. Even with Starlink, both have coverage and latency issues. Both have energy requirements.
Even for people inside cities, any downtime caused by an internet outage must be avoided. Electricity can be provided locally via generators or storage, but the only precaution against an internet outage is multiple connections. If you have a wireless backup to a wired connection then this can help, but the latency and bandwidth of the backup may not suffice.
Internet outages sometimes effect an entire region as well. All providers go down because a switch is damaged or there is a cyber attack. In the later case especially it may not be that the internet altogether ceases to function, but that you can’t establish a connection to the cloud provider for some model.
The internet outage may not even effect you directly, but if one of the cloud providers regions is knocked offline, the other regions can become overloaded. Likewise if something happens like an undersea cable being cut, then it may not cut the connection between you and a provider, but it may overload the remaining parts of the network causing a cascade of failures.
This particularly matters for high-bandwidth use-cases such as video analysis. If a very high bandwidth connection is required for service to continue, then even the slightest disruption could cause disaster. The same goes for if low latency is required. In fact not even low latency, but just reliable latency that doesn’t spike to a few seconds.
Presently I am sat on a train using the onboard WiFi. Latency varies and spikes, sometimes the connection dros altogether. However I am still able to login via SSH to a remote server over a VPN and type commands at the terminal. I am able to fetch code and browse the internet. What is really struggling though is Claude Code. In theory a connection that intermittently drops shouldn’t be an issue for an LLM, but in practice Claude has timeouts and retry limits that prevent it from operating in an acceptable manner.
When you send a request to an LLM on an unreliable channel, then both the request and the response need buffering. The buffering needs to happen in fast memory so that the requests can be processed as soon as it is fully received and the result has to be kept in a readily accessible place for the client to receive it as soon as they are able.
There are all sorts of mitigations for these issues, but the problem goes away altogether when the computation is done locally. In a lot of cases it doesn’t need to be done absolutely locally, but just relatively locally. Having a data-center carriage as part of the train would be local enough. This seems somewhat absurd for a commuter train, but for the military or industrial transport it doesn’t seem so ridiculess. In fact even for commuter trains, if there is an AI safety system monitoring passengers, the track and the train’s engines; then it would make sense to have it onboard or else still close the track.
Any distance you put between the physical application and the compute introduces communication challenges. Ironically the reason why data-centers are so efficient is that they minimize physical distance between components within the compute cluster itself. A distributed super computer is relatively inefficient because of communication. Plus you can do things like put power generation onsite or put the datacenter close to a cheap source of power.
A distributed super computer is very robust however. One node in the cluster can be destroyed without effecting the others due to the physical distance between them. The source of power to one node could be cut off, but the same root cause is unlikely to effect other nodes that are some distance away.
These arguments all cut both ways, the main point I have though is that in particular situations running compute locally is the best trade-off.
Hedging and arbitrage
Let’s say you make a voice agent that takes orders and provides status updates. You sell this to businesses as a service. So essentially you are a wrapper company.
You charge a flat fee because your clients want to be able to budget. They have usage limits so they can’t exceed their usages. However you are charged for your usage of the upstream APIs you use.
If you are using open weight models then you can use a service like OpenRouter or switch between a few different providers depending on price. This provides some hedging for incase one provider goes down or hikes their prices.
However let’s say that one day another wrapper company releases a killer voice app that suddenly everyone is using. There is a huge surge in demand for these models and only a few providers have the inference software to run them reliably.
This causes a rapid increase in the spot price. This doesn’t effect you too much to begin with though because you bought inference contracts with some of the providers. This guaranteed you a upper limit on the price you pay.
As it turns out though not all of the providers are that reliable or upstanding. One eventually decides they could make more money by reneging on your contract to take a better price in the market. Another goes down due to a cyber attack, possibly orchestrated by someone with a financial interest in one of the other providers.
Suddenly your price is not simply going up, you don’t have the compute to service all of your clients, so now you are the unreliable party letting people down.
There are some more ways you could have hedged against this situation. One is to rent some GPUs directly and run your own software on them. You are still renting so you don’t have full control, but GPUs are more of a commodity than a particular set of models. Especially if we are not talking about LLMs but more specialized ones.
So for the same situation to occur the whole GPU market has to be hit with something. However this is not as unlikely as one might like, because things such as a new Chinese model release have correlated with a spike in GPU prices.
If a major cloud GPU provider went down, this could have a cascade effect which pushes up prices of everything. It doesn’t have to be the whole provider even, just one region of a hyperscaler’s cloud could trigger a sudden scramble for compute.
Now let’s say that you don’t actually use the very largest LLMs that much. So the bulk of your usage is midscale models that can run without a water cooled rack. Instead you can cover X amount of your usage, using a regular air cooled server that you can plug into an electricity supply that most commercial properties use.
So you can purchase or lease a server that then covers some of your usage, so that in the event that prices suddenly increase, not all of your usage is exposed to those price rises.
You are exposed to electricity prices in your area, but this can cut both ways. Global prices can increase while your local one decreases. If there is a spike in local prices, you could turn your machines off and switch over to remote providers exclusively.
For businesses that aren’t software companies and have large electricity demands anyway. The electricity used by compute could have a marginal cost close to zero. You may even have excess energy at times. Which if you have the luxury of quickly switching from local to remote, then you can choose the cheapest option at any given time.
Autarky and control
In hedging and arbitrage we already hinted at the fact that having some local compute makes you more robust or even able to take advantage of shocks to the system.
There is a type of person and groups of people who take this to the extreme and attempt to be self-sufficient in sofar as they can possibly manage to do so.
Complex technology essentially makes this impossible. Creating computer chips is a global effort with companies spanning every continent involved in creating a frontier CPU or GPU.
In some cases there is just one company, in one location that produces a component. Without them, we would regress 5-10 years.
However this doesn’t stop people trying to be as autarkic as possible. There is not necessarily any logic to it, but the end result can be an increased level of robustness.
It can also have the opposite effect as well because as we saw in the previous section having the option between both local and remote communication provides the best coverage.
Even if you are establishing a colony on Mars, you may want to be Autarkic in fact, but not in principal, because being able to draw on Earth’s resources when they are available will improve your quality of life.
However for some people there is just the pleasure of being as self-sufficient as possible, even if it’s an illusion or doesn’t make sense from a survival point of view.
If you really want to be Autarkic then probably you need to limit yourself to computer tech prior to the 1970s give or take a few decades. Depending on the size of the group trying to be Autarkic of course.
It has to be said that buying a computer, even if you can’t realistically power it without an external electricity provider, is materially different from paying for tokens or even renting a GPU.
The cloud is another step down from ownership to serfdom. Where essentially everyone just rents the tools they need to survive. This ignores the fact that you can buy shares in tech companies, but buying a tiny fraction of a company doesn’t feel like ownership to most people. Ownership implies control of some sort beyond being able to buy or sell an inconsequential amount.
How much control you really have of a computer you own, running an AI model you don’t understand is debatable, but it is clearly more than if it is in the cloud.
To see why, think how you could verify that you turned off your cloud machine and deleted the data. With a local machine you can take the drives out, pull the plug etc.
Alternatively, if your cloud machine stops responding. How can you confirm or rule out that it was unplugged? For a local machine you can just look at it.