I think most of this is correct, in spite of potentially being built on a bad assumption.
The assumption is that LLMs should be writing the code and human engineers reviewing and verifying the LLM output. And that this pushes the cost of producing down. And I fundamentally disagree with that.
Every time I ask LLMs to write code, even with Opus 4.8 (haven't tried it with Opus 5 yet), what I get ends up being totally rewritten. LLMs still aren't good at writing maintainable code. Can they write plausibly functional code? Yes. But it won't survive the long term. People using LLMs to write all their code are gambling on them eventually getting to a point where the LLMs can fix their own code. It's possible, but I wouldn't necessarily bet on it.
Where I have found immense value from LLMs is in code review. Repeated review by LLMs catches an amazing amount of potential issues. They really shine on security review, but are very effective with any kind of review.
The other thing that the "LLMs write code camp" misunderstands is that writing was never the bottleneck. Understanding was. And understanding the code is still the bottleneck. But understanding is truly gained during the writing loop. The understanding you gain from pure reading or code review is marginal compared to the understanding you gain while writing.
Most of the time previously spent writing was actually spent updating and deepening our understanding of the system under development. There's no replacement for that understanding in a world where LLMs are doing the writing.
But if you flip it: humans write, LLMs review, then you still get a major gain -- not in speed, but in quality. And you keep the understanding loop intact. I would propose that this might be the best way to deploy LLMs.
You're right about understanding. Where we might disagree is the conclusion. I think it's possible to build the understanding without having to type out syntax by hand (debugging, writing tests, etc). Maybe we're missing much better verification tools? LLMs will likely play a big part in those too (explain the codebase, walk me through A, B, C etc)
I write code in Go in an established codebase. 90% of the time the LLM gets a basic api/feature right and fully e2e tested as long as I give it enough business context in the prompt. The other 10% of the time I have to do some follow up prompts to either change the behaviour or change the approach of a given step in the flow or just to point out that the wrong pattern was used and please rather use our codebase standard. I haven’t actually hand-written code in well over a year. Is this way faster? Absolutely. Does it lead to better code? Yes, because I have time to write all the regression tests that keep the behaviour as I wanted it when others come bungling around.
I do think the comment is right about understanding. I'm still feeling out what I think is the right level of understanding to invest in now. I think it is probably not the deep line by line understanding I used to have of the systems I worked on. But I also think it's easy to remain too aloof from how the system is being built, and that this is very bad. I think the right answer is somewhere in between, but I'm still working through my own process and forcing functions to strike that balance properly.
" But understanding is truly gained during the writing loop. The understanding you gain from pure reading or code review is marginal compared to the understanding you gain while writing."
Debugging code step by step is how I understand complicated code.
Code was very rarely the bottleneck in the first place.
If programmer productivity was something we actively optimized for, we wouldn't have crammed programmers like sardines in warm and noisy open floor offices with 2000 ppm CO2 levels and then further constantly interrupt them with emails and slack pings and meetings all day long, Jira rigmarole wouldn't make up a significant portion of what they did, programmers would have instead mostly been thinking and programming.
We've always had the ability to 2X if not 10X the output of each and every one of those poor souls. You don't end with this sort of programming purgatory because it's a productivity optimum, it very clearly isn't, but because it's a billable hours optimum and/or an org chart clout optimum and/or because of Jevons paradox got hands even in business management and the IT department was allocated too many dollars.
> Code was very rarely the bottleneck in the first place.
I disagree. Kinda
What AI has made much simpler is that you don't have to waste time checking docs and have the best autocomplete system by a long shot - this was a bottleneck unless you were doing Java or some other language with "perfect" AC
What AI made "kinda easier": solving for usual problems. The stuff you would search Stack Overflow, or think a couple of minutes for an optimized solution - not a bottleneck but not 100% smooth neither
You still have to test and validate your code. AI made this easier-ish but this is still where I see manual work being needed (even if you are automating tests - you still have to think on what you want the code to do)
I disagree. Coordinating 100 folks to align on a problems statement with "agile" and sprints was always the biggest bottleneck. Difference in opinion, internal politics is always a bottleneck.
I think now, code is the bottleneck. Just because you can generate million lines of code, people with different skill level think they are accomplishing the task, testing, merge conflicts, trust has become the bottleneck.
The “old way” would be lots of debate (both bike shedding and useful) among engineers during design phase, and then you’d implement.
Now it’s shifted so there are no design docs and there is only the generated prototype. People trying to do their design review while there’s already a functional-ish prototype and it goes nowhere. There’s an anchoring effect in place because the first thing already exists and management says “this seems to work, just use it and move on”. The result is that useful debates about substantive issues don’t happen and bikeshedding is all way get to do
This just says that we can output code faster. I'm saying that the rate at which we output code wasn't the thing that was slowing down development. We've always been able to increase that even without AI by adjusting the working environment and removing obstacles to programming.
In larger organizations, quite often it's the business that is holding back development. They can only handle so much change and speed needs direction to be velocity. Drafting requirements is generally much slower than implementing them.
Like the number one complaint from programmers has been that they don't get to do programming. They want to write code, not update jiras or spend hours in meetings.
What is the right, best software organization in the current era of AI coding? This question is critical and wholly unanswered in comprehensive research along the same axis as Accelerate (2018, Forsgren, Humble, Kim).
There are a lot of (excruciatingly) long-form posts about what folks are pioneering but not a whole lot of follow up about what failed. Where are the short posts on the negative space? How did halving your staff work out? Flattening your org? All those dark factories, what haven't they produced? How about all the other things tried, failed, and unceremoniously scrapped?
We need to explore and communicate the negative space more efficiently. Don't repeat the same mistakes, and don't make me read 2653 words when 300 do it better.
A tech comm course I took in college was graded on two 20-page papers and accompanying 5-minute presentations. That was like 10k written words in a single semester. It was a challenging class, and gave me substantial sense of accomplishment, just to hand in completed work.
Similar to a functioning side project in the 5-10k LOC range. Announcing something that worked a year ago, was laudable, even if not profitable.
I vibe coded 15k LOC this morning and read 20k words of AI generated text while doing so. No longer are either noteworthy or valuable public contributions just by virtue of having been done. I don't think that's widely recognized yet.
AI reduces the cost of writing code, but it makes adding things that nobody uses even cheaper. The bottleneck then becomes deciding what deserves to exist—and having the discipline to remove the rest. So this will imply more time spent in code reviews that lead to more iterations in PR's.
AI is doing a good job on writting code these days! Nothing against it; I use it every day, but the context switching is costing us a lot!
I don't think it changes much for good managers. It should always be able setting people and processes up so the team can land durable measurable impact. The managers that thought the job of software engineers was to write code were bad managers. PRs or LoC were never good metrics.
In my experience, management is mostly pissing away the gains made by AI by either:
1. Pursuing polish and quality beyond previous norms
2. Replacing $100/mo/seat SAAS with something coded by a junior costing $200/day to develop over months.
The cost of code approaches zero, but the cost of communication, having accountability, and hosting remains the same, and so individuals need to only coordinate to the extent that those things remain finite resources. Management needs to stop insisting that their directs adopt each others vibe coded tooling.
If you wrote an initial draft you should consider using a better model or just posting what you wrote.
The smaller models can be sufficient for coding but for document writing not highly specific I've yet to be satisfied with AI output. I certainly wouldn't expect gemma to produce good outputs.
Makes sense, perhaps I'll post the raw notes + more material with the next article, prior to any edit. I like to use local models, even if they are not that powerful precisely because of that, I'm forced to put in more thought and it's my signed off article in the end.
You don't build credibility by putting out slop and correcting it every time someone points out that it's wrong. You think it's our job to proofread and fact-check?
A genuine mistake. I make these posts for myself, mainly to structure & share my thoughts. Let me know if you find other errors, I appreciate the feedback.
I'm really open to feedback. I checked your past comments and you posted:
> AI is good at coding if there's an oracle. If the system is ancient, unreadable, untestable, that's exactly the opposite. It won't get the exact set of corner cases.
I sort of mention this in the article, so I'm sure we're somewhat aligned on the core. How would you have worded things?
I did, the problem is that you need a sort of standardized feature change to compare similar repos. So the metric is relative only to same projects lacking that same feature. So no, its a useless thing. No architecture score comparisson between apples and oranges.
Software development has always evolved. Sometimes slowly, sometimes quicker.
LLMs have brought a different unlock, and for everything we're seeing become easier, it allows people learn to use the tools to take on solving problems that couldn't be approached before.
Code was always expensive. Entire industries of design tools cropped up simply to avoid writing the wrong code since it is so expensive. There are entire journals dedicated to this. Organizations are fixated on making sure that the right code is written since the cost of getting that wrong is so great.
The assumption is that LLMs should be writing the code and human engineers reviewing and verifying the LLM output. And that this pushes the cost of producing down. And I fundamentally disagree with that.
Every time I ask LLMs to write code, even with Opus 4.8 (haven't tried it with Opus 5 yet), what I get ends up being totally rewritten. LLMs still aren't good at writing maintainable code. Can they write plausibly functional code? Yes. But it won't survive the long term. People using LLMs to write all their code are gambling on them eventually getting to a point where the LLMs can fix their own code. It's possible, but I wouldn't necessarily bet on it.
Where I have found immense value from LLMs is in code review. Repeated review by LLMs catches an amazing amount of potential issues. They really shine on security review, but are very effective with any kind of review.
The other thing that the "LLMs write code camp" misunderstands is that writing was never the bottleneck. Understanding was. And understanding the code is still the bottleneck. But understanding is truly gained during the writing loop. The understanding you gain from pure reading or code review is marginal compared to the understanding you gain while writing.
Most of the time previously spent writing was actually spent updating and deepening our understanding of the system under development. There's no replacement for that understanding in a world where LLMs are doing the writing.
But if you flip it: humans write, LLMs review, then you still get a major gain -- not in speed, but in quality. And you keep the understanding loop intact. I would propose that this might be the best way to deploy LLMs.
Debugging code step by step is how I understand complicated code.
I think perhaps the assumption that engineering managers should have any employees may be outdated.
If programmer productivity was something we actively optimized for, we wouldn't have crammed programmers like sardines in warm and noisy open floor offices with 2000 ppm CO2 levels and then further constantly interrupt them with emails and slack pings and meetings all day long, Jira rigmarole wouldn't make up a significant portion of what they did, programmers would have instead mostly been thinking and programming.
We've always had the ability to 2X if not 10X the output of each and every one of those poor souls. You don't end with this sort of programming purgatory because it's a productivity optimum, it very clearly isn't, but because it's a billable hours optimum and/or an org chart clout optimum and/or because of Jevons paradox got hands even in business management and the IT department was allocated too many dollars.
I disagree. Kinda
What AI has made much simpler is that you don't have to waste time checking docs and have the best autocomplete system by a long shot - this was a bottleneck unless you were doing Java or some other language with "perfect" AC
What AI made "kinda easier": solving for usual problems. The stuff you would search Stack Overflow, or think a couple of minutes for an optimized solution - not a bottleneck but not 100% smooth neither
You still have to test and validate your code. AI made this easier-ish but this is still where I see manual work being needed (even if you are automating tests - you still have to think on what you want the code to do)
I think now, code is the bottleneck. Just because you can generate million lines of code, people with different skill level think they are accomplishing the task, testing, merge conflicts, trust has become the bottleneck.
The “old way” would be lots of debate (both bike shedding and useful) among engineers during design phase, and then you’d implement.
Now it’s shifted so there are no design docs and there is only the generated prototype. People trying to do their design review while there’s already a functional-ish prototype and it goes nowhere. There’s an anchoring effect in place because the first thing already exists and management says “this seems to work, just use it and move on”. The result is that useful debates about substantive issues don’t happen and bikeshedding is all way get to do
In larger organizations, quite often it's the business that is holding back development. They can only handle so much change and speed needs direction to be velocity. Drafting requirements is generally much slower than implementing them.
Like the number one complaint from programmers has been that they don't get to do programming. They want to write code, not update jiras or spend hours in meetings.
There are a lot of (excruciatingly) long-form posts about what folks are pioneering but not a whole lot of follow up about what failed. Where are the short posts on the negative space? How did halving your staff work out? Flattening your org? All those dark factories, what haven't they produced? How about all the other things tried, failed, and unceremoniously scrapped?
We need to explore and communicate the negative space more efficiently. Don't repeat the same mistakes, and don't make me read 2653 words when 300 do it better.
Similar to a functioning side project in the 5-10k LOC range. Announcing something that worked a year ago, was laudable, even if not profitable.
I vibe coded 15k LOC this morning and read 20k words of AI generated text while doing so. No longer are either noteworthy or valuable public contributions just by virtue of having been done. I don't think that's widely recognized yet.
AI is doing a good job on writting code these days! Nothing against it; I use it every day, but the context switching is costing us a lot!
1. Pursuing polish and quality beyond previous norms
2. Replacing $100/mo/seat SAAS with something coded by a junior costing $200/day to develop over months.
The cost of code approaches zero, but the cost of communication, having accountability, and hosting remains the same, and so individuals need to only coordinate to the extent that those things remain finite resources. Management needs to stop insisting that their directs adopt each others vibe coded tooling.
Does this guy have access to Gemini 4 already?
I'm guessing Gemma 4 was happy to be mistaken for Gemini and didn't catch this mistake.
The smaller models can be sufficient for coding but for document writing not highly specific I've yet to be satisfied with AI output. I certainly wouldn't expect gemma to produce good outputs.
Thank you for the suggestion.
I work at a company where the biggest problems are not 'writing code', they are:
- Organising teams
- Designing the system
- Prioritisation of work
The fuckups that we make on a daily bases are not 'code errors' they are failures in THOSE three things. I'll go into detail if anyone cares.
> AI is good at coding if there's an oracle. If the system is ancient, unreadable, untestable, that's exactly the opposite. It won't get the exact set of corner cases.
I sort of mention this in the article, so I'm sure we're somewhat aligned on the core. How would you have worded things?
LLMs have brought a different unlock, and for everything we're seeing become easier, it allows people learn to use the tools to take on solving problems that couldn't be approached before.
Not everything has to be written as though it’s a middle manager’s idea of what makes for a good TED talk.
> What follows is a cleaned-up version of notes I accumulated over the past year. Gemini 4 helped with the editing.
The image is made by an AI image generator on fal.ai. It's better I spare you all my design skills :)
> Gemini 4 helped with the editing.
There is no Gemini 4, unless the author is writing from the future.