LLMs and Software Margins

A few months back, I came across this article that talked about margins in the software industry. Long ago, computer software was well known to be an insanely high gross margin industry. However, it is not the case any more.

If you look at SaaS (software as a service) companies, a lot of them barely make much profits any more. So what changed?

The answer is infrastructure. In the olden days, when all hardware was “on premise”, software would be a bunch of lines of code that would get sold, and then run on the client’s on-premise hardware. Thus, once the code had been written and tested and perfected, the only cost that the vendor faced was to install the code on the client’s hardware (including the cost of engineers involved in the installation). And the margins soared.

Then (I’m still paraphrasing the article that I had read, and now can’t find), the cloud happened. Hardware wasn’t all on-premise any more. People figured out that software could be sold “as a service” (hence SaaS). Which means, instead of charging for installing some code on a computer, you could charge for API hits, or function calls. Everything became smooth.

The catch, though, was that the software would now have to be hosted on hardware maintained (in the cloud) by the vendor. Which meant now the marginal cost of delivery suddenly became non-zero. Rather, it went from (one time installation) to (costing each time it gets hit, or the time for which it is maintained). And this had a material impact on software margins.

I’m thinking of this now in the wake of new-fangled open source LLMs that keep getting announced every day. Every new LLM that comes out gets compared with ChatGPT, and people tell you that this new LLM is “open source”. And you get excited that you can get for free what you would have to pay for with ChatGPT.

Of course, the catch here is that ChatGPT is like SaaS – not only does it provide you the “LLM service” it also hosts the service for you and answers your questions, for a fee.

These open source models are like the traditional “on-premise” computer software industry – they have good code but the issue of course is that you need to supply your own hardware. Add in the cost of maintaining the said hardware, and you see where you might spend with the open source LLMs.

That said, Free != Open Source. The Open Source LLMs are not only free, but also open source – and so, the real value in them is that you can actually build on the existing algorithms and not have to pay a fee (except for your own infrastructure).

And from that perspective, it’s exciting that so many new tools are coming along.