Now that software is essentially free to write, the only limit is your imagination. Does that mean your computer now works exactly how you want?
Was the last time you saw a bug weeks ago? Are there no security vulnerabilities anymore? Does every app you encounter have perfect UX?
Have you stopped paying for every app except an LLM which codes everything for you before you have even asked for it? When you make a bank transaction is it generating that all on the fly?
Everything works perfectly and is correct now is it? Nobody is hiring software devs anymore, they are extinct?
Of course I am being facetious, there are more bugs than ever and earth shattering security vulnerabilities are a weekly occurrence. Open software developer roles have been increasing steadily since the great post-COVID layoffs.
The layoffs themselves could be explained by the vast hiring spree that occurred in COVID as companies raced to take advantage of the move to digital communication.
That’s not to mention product manager roles who are tracked separately and would be the people replacing software developers if LLMs abstracted the whole coding process. In such a case we’d expect to see them increase and software devs decrease. Instead we see an increase in both.
If software were dead altogether then we’d also expect to see PMs die with it, because they are only there to speak to users, identify their problems and fit a solution to those problems that is compatible with a working business model.
LLM powered agents can speak to users, identify solutions and implement them. So why do any of these roles still exist? I’m not joking about this, LLMs can do it in some capacity.
Admittedly the types of software development roles that are open have changed. Layoffs have been strong ever since the pandemic with AI increasingly being cited as the cause.
Meaning churn is very high and it is not clear if there has been a net positive or negative increase in tech jobs since the pandemic layoffs.
Everyone is asking for senior engineers and React skills are a lot less important than they used to be.
In fact particular knowledge of any sort of API is now far less important. It doesn’t really matter how well you know a given language even. The people who write job adverts haven’t absorbed this yet, but it’s always been true to some extent and only a subset of employers ever got the message.
If we look at startups in the YC incubator and take this as a leading indicator then there is a shift to AI native wrapper companies and hardware startups. Which in turn means a shift to AI engineers, systems developers and hardware related roles.
Anyway, I digress, the main point here is that right now, today, software development is continuing. It’s changed in dramatic ways, but also there is something LLMs can’t do that is lost amongst the awe inspiring progress.
What LLMs can do, extremely well, is copy things that already exist. Recently someone produced clones of Adobe’s product suite along with some Microsoft Office clones for good measure.
These clones are free open source software, so no subscription required. Meanwhile Adobe had their best quarter, so when these clones have had time to take some market share we should see a reversal.
If, that is, the developers of these clones are motivated to keep developing them because tokens and the time spent prompting LLMs is not free. Meanwhile one of the great motivators for developing open source was to display your prowess as a developer to get a job or attract clients to your enterprise offerings.
Being an expert at using LLMs to copy other people’s software is not a highly sought after skill judging by looking at the jobs market. I have used it to great effect to convert PyTorch projects to C++/GGML, but it’s difficult to justify keep doing it unless I have a direct need for the AI model implementation.
The business model for the company behind the clones is to sell model inference through the apps. Which is viable if people are nice to them and don’t just use a different provider or can achieve what they want with a local model.
Now, this is the current state of things, so the declaration that software development is dead, apps are dead, developers are dead, open source is simultaneously both dead and de facto the only type of software there is now (more on that later), and there is no software moat anymore, must therefor be a forward looking statement.
It must be said that most of the population are unaware of just how good LLMs are at producing code and controlling a computer to perform routine tasks. So it’s possible that just not enough people have discovered coding agents yet.
There is also the fact that LLMs keep getting better, but I’m going to argue that this is deceptive.
LLMs have been getting better, repeatedly breaking into new areas, but there are dimensions on which they have not improved at all.
A fact brought to my attention on X by Andras Gerlits making statements like “they can’t design”. This at first seems patently wrong because they create nice looking designs, but that’s not what the word “design” means in this case.
Most critically they can not invent a new abstraction at inference time and then use it consistently. If they can create a new abstraction at all at inference (unlikely), it is represented internally as a recombination of things seen during training.
It’s likely that the new abstraction will quickly be blurred into concepts that were present in training. So that further inferences based on that abstraction become subtly wrong.
This is not how the humans work, we learn constantly. There is no training or inference distinction. We can begin using something as we learn it. The new thing takes root in our mind as a new object in its own right.
We also learn by recombining things we have previously seen, but our neurons form new structures on the fly.
LLMs also have no sense of direction, no will of their own. You can provide one in writing, but it’s not a living object that evolves over time. So either you have to keep providing it or let the LLM update its own goal.
If you do that latter, then the LLM sooner or later goes off on a random walk through its probability distribution. It has no nervous system to provide motivational signals never mind whatever it is that gives humans intellectual motivation.
Nonetheless improvements have been made on long-horizon tasks and LLMs get better at producing temporarily impressive results when vibe coding games, apps or operating systems kernels.
This is because they are the ultimate plagiarism machine. Any possible thing you could want that has been done before, it can go grab a piece of that, then combine it with a bit of something else. Thus producing a unique combination that is impressive. At least until you have seen enough of these results to start to see a pattern which makes them all seem the same.
I digress again though, the point isn’t that LLMs don’t produce original stuff. First of all they do, it’s just that it’s interpolation rather than extrapolation. This is how OpenAI managed to defeat so many open maths problems. (just as I finished writing this they announced progress on P=NP!)
The problem with LLMs is that not everything can be accurately represented as a recombination of previously seen things. In fact I will assert almost nothing can be. What we have is an illusion created by a very good approximation combined with a form of brute force search.
Calling what LLMs produce slop is a very intuitive way of expressing this. Essentially the output and the computation to get it, is a jumble of different concepts that looks like what you or the situation called for.
What you tend to notice after a while though is that this output contains repeating patterns that betray the “scaffold” or “templates” that an LLM or even LLMs in general rely on most.
It appears however that despite these limitations that producing code with LLMs is extremely effective. In particular if we run the LLM in a loop until it reaches some verifiable goal. That is we setup a hill climbing exercise.
Because they are so effective at this we have to ask the question whether it matters that they can’t design or accurately use systems of thinking.
Indeed ML models themselves are trained with a hill climbing exercise and the results are astounding.
Perhaps the challenge is now setting the goals and creating a framework to efficiently climb towards it. The LLM itself can even help with goal setting and the harness. Giving us a recursive improvement.
For types of software with clear quantifiable goals and operational bounds I can’t think of a reason this won’t work. That is if we have accurate performance, security and functionality goals that are limited in scope, then the code can be iteratively improved until we reach those goals.
This can be done so fast and so cheaply that anyone with the motivation and token budget can do it. So that there is no technical moat in the creation of the code itself.
Discovering the goals and bounds under which we need to operate is then the only potential moat, but if we can discover those goals via reverse engineering, then that essentially disappears too.
If you imagine that the goals you set are a jug, then the LLM sprays the opening of the jug with slop until enough of it gets in to fill the jug. If we imagine the slop has chunks in it then the slop approximately fits the shape of the jug.
So where in software could there still be a moat? Some people have suggested AI models themselves and specifically the training data, rather than architecture, are a moat.
AI models are software in case it has been forgotten. What I find interesting is the similarity between training an ML model and creating code with agents.
In both cases we set goals and then hill climb towards fulfilling those goals. We can almost view writing an app with agents as a training run. Where we are training the code to perform the goals by setting a loss function that is interpreted by an LLM.
This is important because if there is a moat for AI models based on the training data, then there is also one for regular code defined by the goals.
If that is, there is some software where the goals are as obscure as the training data for an ML model.
I’d argue that for some software this is the case. Because even if you have the binary for some software, the full range of behaviours of that software is not clear until you run it. At least from this low level representation.
In fact even if you have the source code, due to the halting problem, the actual runtime behaviour is fundamentally not reproducible from looking at the code.
Further to that running the code to observe every possible behaviour is impossible, because it’s easy to create code that not only never halts, but produces unique sequences after a long period.
For a lot of software, this is likely entirely academic. However for something like a distributed database or word processing app. In order to make sure these behave for all of your users, across vastly different use-cases and runtime environments, is a concrete problem.
Now, one way to ensure that software behaves across all possible scenarios that you care about, is to abstract away the details and create a collection of high level objects that can be reasoned about in finite time.
Which is exactly what LLMs can’t do with novel abstractions. They also have difficulty with abstract objects in general because they think in tokens which have multiple meanings given the context and these other meanings can seep into the abstraction.
This brings me onto the core of my argument, which is if there is a pure software moat left. One that survives reverse engineering and agentic coding in general, it is correctness.
I will say now that I do not think this moat will apply to a lot of software due to the “good enough principal”. Basically some sloppy solution is good enough. Pushing beyond that won’t create a market advantage.
The two questions left for me are how much software requires very high levels of correctness and what is the software developer’s role in this?
I believe the security events in the last week provide insights into both. Firstly PHP was “popped” using AI, Brotli also and OpenCode.
Clearly these software require high levels of correctness in regards to security at least. It’s also with AI assistance that these vulnerabilities came to light.
The vast increase in bugs being found is with LLM assistance in general. So I would say that on the face of things, LLMs are a core component of achieving correctness.
Even if LLMs are sloppy and are fundamentally limited by the halting problem. They can look through more code than humans can and have a broader knowledge of computing than any cyber security team.
To my knowledge the PHP and Brotli bugs were found with LLMs and the vulnerable code was written by humans. OpenCode on the other hand is written with LLM assistance, so LLMs don’t write perfectly secure code.
In fact it is safe to say that LLMs write highly insecure code. No doubt OpenCode use an iterative process when writing code as well, so it’s unlikely that the vulnerable code was never reviewed by an LLM.
Now, using good abstractions is a way of making code more secure for many reasons. Including that if you have clear security boundaries, where you can perform checking, then you have clear data flows in your system that are either trusted or not.
This isolates where you need to put the most intense focus of your security audits and where correctness is most important for security.
In my personal experience, this is something that some humans are better at, but LLMs still have a role to play in the design of these abstractions.
When you very specifically ask LLMs to find inconsistencies, they often do a good job of just that. They may just be approximating your abstractions, internally, but it is good enough to find real issues.
They are also good at suggesting improvements, which can also bring your system more in line with standard practices. They are good at searching the internet and existing code for details that are too much for the human brain.
Implementing the core interfaces and data structures is also something where I think humans, at least some of them, are better when absolute correctness is at stake.
The LLM again will be able to find inconsistencies and suggest improvements. Most likely with a level of completeness and with a range of knowledge that would be impossible to match even by a team of developers.
I think the only question here, is how much should the human be doing before consulting the magic answer machine.
In order for said human to be producing a very good direction and overall structure, they need skills. One of the reasons I like writing code, is that you get feedback, first from the compiler and then from runtime tests.
If you use formal verification, you can also get rapid feedback from whether your theorems are correct. However what you don’t get are solutions. You have to figure those out and if you are careful in your thought, then you’ll reach them a lot faster.
LLMs often just give you an answer, so writing your designs in
natural language doesn’t seem like the best path to maintaining your
skills. It is however a lot faster. Even just on a mechanical level
it is easier to type out “function that returns one”, than
int fun(){ return 1; } (it literally might not be for
you, but read between the lines a bit).
Beyond simply maintaining skill, there is also the formulation of abstractions. While a high level object removes the details, it’s important to know the details when forming the abstraction.
If you don’t know the details, then you can’t actually abstract them away. If you treat a function as being side effect free and it actually does have side-effects. You need to know if those side-effects are observable by the consumer of an abstraction or you whether you can just ignore them.
Remember that LLMs are not good with ad-hoc abstractions. They might find issues with one you have created or false positives because they are approximating it. So it is really you, the human(s), that are providing the gold here.
Designing software on this level is time consuming and requires expertise. My feeling is that to get ultimate correctness in the shortest time posssibe, a very large token budget is not required, but LLM usage is a must in some capacity.
Writing code by hand or at least being very exact about what the LLM should write is a must. It may not be the final code you ship, but it is what provides the correctness guarantees. In fact shipping partially LLM generated slop, but having a secret behavioral model seems like a great idea if reverse engineering is a concern.
So what we can see here is that human time is needed and it requires judgement to know how much. Over use of LLMs will cause more harm than good, but wasting time on mechanical tasks will have the same result.
So it appears that expert human time is needed and that right there is a moat. It is purely a temporal one because there are lots of smart humans on a global scale, but what’s new?
Attempts to clone Microsoft Windows have failed because it would take a lot of expert human time, not because there is some secret magic in the kernel code that can’t be reproduced.
You may find that even now, with LLMs, reproducing the correct behaviour of the Microsoft kernel proves to be infeasible, unless you are willing to fund a whole bunch of experts, working with clients, to recover the correct behaviour.
It may appear at first that a large functional clone can be reproduced quickly, but it’s likely only an approximation that falls apart under prolonged or wider use.
I’ve mainly been using security as an example, but performance, business logic and user experience are all effected by correctness. They all benefit from well structured abstractions that at the very least, will save on token budget and at most will provide a solid moat.
You may wonder that I am just trying to protect my job or my craft. I think that the probability that my blog will impact the trajectory of technological development is zero. However I do worry that my arguments are what I want to be true, rather than an objective assessment.
What I feel though is that since LLMs started cloning large apps and doing math proofs, things have gotten a bit extreme on the “software is dead”, narrative. It seems that any sort of investment in non-AI software is being impacted.
Possibly this could lead to an opportunity for those who stubbornly refuse to go all in on LLM based development.
Especially where people can identify areas in software where a sprinkling of well thought out LLM usage provides a massive boost. Meanwhile throwing an agentic cluster at it creates a false feeling of progress and ultimately results in failure.
I also want to see how I can weather this storm. For quite some time this year I was furiously prompting agents into creating all sorts of things. I think that while this had some very good results, it also created downsides. My skills to write code manually were not impacted as much as you might expect, but I feel like too much time was spent just nudging the LLM.
Regardless of whether my startup, freelancing or employment work out. I need to come out with intellectual skills that are useful in a post software world. Nudging an LLM based on skills from the pre-LLM days is not sustainable.
I need something in software where real skill and judgement is required. Where I think it still lives is in correctness and quality in general. Especially when combined with novelty because there is nothing for the LLM to copy.
Moving closer towards the hardware also provides some security because whether something works in the physical world is not easily verifiable. Making the same hill climbing exercises that are so well suited in ideal software settings more difficult to perform albeit not impossible.
So my final conclusion is that there is still a moat in software. It centers around abstraction of complexity and the fundamental workings of LLMs. Taking advantage of it is difficult and requires carefully picking the type of software you work on. Ensuring the use-case calls for high levels of correctness and that the correct behaviour is resistant to discovery.
Refusing to use LLMs altogether will maintain your skills, but the resulting software will not be able to compete. Leaving the implementation entirely to LLMs meanwhile is likely to deskill you unless you deliberately practice critical thinking in some other way.
For those who decide to stay in software there is a potential opportunity, but also a risk that the planet can essentially run on a small core of highly reliable code maintained by a small exclusive group of people.
In any case though maintaining critical thought, developing algorithms and abstractions, is not going to be wasted effort. Even if the AI gains these abilities, it may not be able to translate them into real world gains efficiently.
The purely mechanistic parts of programming may be about as valuable to the market as my hand driven drill or digging a trench by hand, but identifying and creating abstractions that make problems easier to solve is most likely still going to be a high value skill.
Ultimately though my claims are stronger than this. It’s that there is still a software moat based on the difficulty of reproducing the code. It’s unclear how difficult it is to take advantage of it, but it is still there and it will continue to exist.