People often assume that code is knowledge. They want "self explanatory" or "well documented code". Companies and managers often treat developers as interchangeable. But therein lies the mistake.
Knowledge exists in mental models and team structures. Small components and systems can be understood by a person, but team structure also embodies knowledge of larger systems. People will need complementary mental models to understand a large system together.
That's why adding manpower to a late project makes it later. That's why maintenance is hard and handover is harder. That's why systems devolve into big balls of mud. Because companies and managers do not respect the fact that you need people and teams who have good mental models and that mental models take time to build and share.
No amount or quality of code can make up for this fact. And simulacra of ownership - having "product owners" or whatever - won't cut it. You need people to own their systems, understand them deeply. Moving people around, churn, treating developers as interchangeable, substituting rituals for deep work, accumulating 'technical debt' (deferred work as in ship now and think later) etc are all detrimental to building and sharing sound mental models.
I've handed off a few services I built with minimal oversight or documentation. The receiving teams have been able to make changes without my involvement and everyone is happy.
I believe the only reason I've been successful in this is because I agonize over simplicity. There are times during the development of any project where one might be tempted to hack around an issue, or commit the ugly code that seems to work. These are the rough edges that inheritors of a codebase use as evidence that a blank slate would be preferable. They're also the bits where the underlying business logic becomes murky. My goal is for the code to be so clear that documentation would feel redundant.
This approach of course takes more time and requires that your management trusts you and is willing to compromise on timelines. It's extremely rewarding if you can sell it and deliver.
This is a well-referenced essay, drawing the on writing of David Parnas [1], Peter Naur [2], and Zach Tellman [3].
As software developers we’re intimately familiar with these ideas. But the industry still treats it as “folk knowledge”, despite decades of academic work and systemization attempts like the original Agile.
We really need more connective work, relating the theoretical ideas to the observed behavior of real-life software projects, and to the subsequent damage and dysfunction. I liked this essay because it scratches that itch for me. But we need this work to go beyond personal blogs/newsletters/dev.to articles. It needs to be recognized & accepted as formal “scientific” knowledge, and to be seen and grokked by industry and corporate leadership.
[1] https://dl.acm.org/doi/pdf/10.5555/257734.257788
> the mental model that allows the designer to map a subset of the world (the domain) to and from the system, and not the system itself, is the primary product of the software design activity
This is spot on. I was never able to put it in such precise words.
This theory has Brook's Law as a corollary:
"Adding manpower to a late software project makes it later."
Because the developers need time to develop this mental model before they can meaningfully contribute to the codebase.
Most of our white collar jobs are about knowledge sharing and synchronization between people.
And surprisingly this is an aspect in which I see very very little progress.
The most we have are tools like confluence or Jira that are actually quite bad in my opinion.
The bad part is how knowledge is shared. At the moment is just formatted text with a questionable search.
LLMs I believe can help in synthesize what knowledge is there and what is missing.
Moreover it would be possible to ask what is missing or what could be improved. And it would be possible to continuously test the knowledge base, asking the model question about the topic and checking the answer.
I am working on a prototype and it is looking great. If someone is interested, please let me know.
I feel that the real problem is a lot of people don’t care about collecting requirements.
It’s one of my favorite parts of the process.
People just want to build the app that they want to build. I’ve talked to engineers who just say “I don’t really care until we can start coding.”
I got into engineering because I like building things that are useful.
I have another suggestion which I'm sure played a large part in this.
SVC was an unwanted child. It wasn't their "product". One employee was tasked to write it to save paying money to a "seemingly innocuous middleware SaaS".
To anyone in ORG working on it, it was a dead end. No one wanted to own it and perhaps no one did. A team was asked to add features to it.
Doing the ground work of actually understanding SVC had many negative consequences:
* It would take a very long time, making managers not happy. It would be largely a wasted effort, since no further work was then needed on SVG.
* If you became an expert on SVG, it would be yours to keep and no one wanted that.
This is an interesting argument for the (definitely common) pattern the author describes in the intro.
A more cynical take (that I'm inclined towards is): the median software developer is simply not very good. X10 was a good developer; the people on TEAM and TEAM++ were not.
This is really why AI is going to hit s/w development so hard. It's not merely going to make code easier to write, it's going to be a massive knowledge repository that takes a team from initial conception, through product design and finally engineering design and coding.
I find this article spot on and resonates with what I've experienced.
The article mentions Zach Tellman's newsletter "Explaining Software Design" (https://explaining.software/) which I highly recommend reading. I have found his works to provide deep insight into the process of software design.
The focus of this article is on the big political project that fails (the SVC), but the part that resonates most with me is the small forgotten project that lives (the SaaS).
Over the years, I've built a number of contraptions under a similar set of circumstances: a technical problem usually created by an organizational issue suddenly appears that is both severe enough to threaten a project yet falls outside the core business, so it needs to be fixed both yesterday and on the cheap.
Inevitably, I get saddled with it and produce a kludge that is equally effective and cursed before going back to business as usual. More than once I've learned to my horror that years later the thing is not only unmaintained yet still in place, but its usage expanded to the point where it became load-bearing, because the underlying organizational issue was never solved.
In a manner of speaking, it is the opposite situation as described in the article: a complete lack of software design that somehow manages to survive in spite of a lack of knowledge building.
Even if the organistion is fully signed up to the knowledge building philosphy, it can still be derailed if the staff aren't up to scratch.
Around 12 years ago, my employer tasked me with building a quote engine for a new product they wanted to sell online. The engine needed to produce 4 additional quotes (2 lower, 2 higher) to either prevent potential walk-aways or to offer upsells to capture potential additional revenue.
And it struck me at the time that this sort of hand-wavey selling tactic would be just the sort of thing that was likely to change so I put all this logic into a pure function - pass in an original quote request and the function returns a collection of alternative quote requests.
And, sure enough, a couple of years later, the business decides to change the approach and offer 1 lower quote and 3 upsells. And they give the job of implementing this to another developer.
I was still at the company and was known as the original developer (my name was in a comment at the top of the file, for a start) so I was asked to review the code changes.
I was surprised to learn that the other dev had left the pure function untouched and had, instead, written a bunch of new logic to generate the alternative quotes. Not based on the original quote request but on the collection of alternative quotes returned from the original function. Furthermore, this new logic was placed in main procedure - mama's finest spaghetti in the making, right there.
So I rejected the change and told the dev where to put the actual logic. Then I waited for the re-review request to come in.
What happened instead is that the code went live anyway - the dev had simply re-raised the review request and assigned it to another dev who rubber stamped it.
Looking back, I don't think all the documentation in the world would have prevented this behaviour. A better approach would be for the company to pass the changes to the original developer and to pair with another dev - like Fred Brooks' Chief Programmer plus Assistant recommendation.
I was never approached for an end of year review for this developer and I left the company before them. It's not personal but I'll resign on the spot if a company I work for employs this developer in future.
Also related: https://blog.codinghorror.com/commandos-infantry-and-police/
I work a lot in the transition area between commando and infantry aka X_10 and TEAM. I’ve also found myself on TEAM++ coming in to replace TEAM.
It is difficult to explain to customers that SVC was built on a set of assumptions which I turn informed the design. Once the assumptions changed then design typically needs to change as well.
Yes, artifacts (PRs, tickets, commit messages) should have good context (documentation) associated with them, and yes, simplicity is very important, and yes, some tech debt is always going to be incurred, but most importantly, not understanding a "legacy" system that is already running in production, and then modifying it significantly, is not going to lead to good results. I think it's as simple as that.
There's some really interesting stuff in here, but I think, given the example, it is burying the lede: organizations systemically underestimate the total cost of ownership of a service. By, like, orders of magnitude.
In this example, it isn't entirely clear if this service ('saas middleware') is deeply integrated with org's core competency or value. But I'll assume it isn't.
They do not understand the service domain well enough and cannot staff or motivate the people to build and maintain it correctly. This is exactly why SaaS exists and is so ubiquitous. You're just not going to be able to build something as good or better for less in the long run.
If they properly understood the cost of building and maintaining this, compared to the (probably) insignificant increase in enterprise value to the org, they would have probably would have just RIF'd these spare engineers and just pay the SaaS provider.
I deal with the internals of many engineering teams across companies of all sizes, and sure as shit, every. single. one. of them has multiple of these internal failed creations. I just don't think people truly understand how much of a liability these systems are orgs.
But yeah, once they made the first mistake, the rest of the blog pretty much hits the nail on the head.
Totally resonated with me re software lifecycle etc. Great article.
I know this is not what the article is about. But perhaps exec should have spent resources and time trying to increase revenue instead of cutting cost marginally, and creating an expensive system down the road. Derailing team focus.
Build vs buy… Build is almost always not cheaper. Many other reasons to build though.
> A dead program may continue to be used for execution in a computer and to produce useful results. The actual state of death becomes visible when demands for modifications of the program cannot be intelligently answered.
I totally believe this. I see zombie programs at my employer, including the ones that are currently making a profit, that just don't know yet that they are dead. The Jira model of non-ownership software development being a leading cause in my opinion...
I see star, lone-wolf ICs get too out in front of their teams all the time. It usually doesn’t end well. The star IC could objectively be building the right thing (like a state of the art recommendation system). But, like the article says, requirements change, bugs need to be fixed, the team needs to adjust the implementation and eventually the team reimplements the thing to within their capabilities.
It’s more than the usual software maintenance too-It’s the entire operation of a piece of software. Scaling it out, being on call for it, adding monitoring, alerting and logging. Inter-operating with other software in the company. Developing libraries and services for other developers to consume. Security. Understanding and deploying the dependencies of the software. And more.
The clever recsys in my example is only the tiny kernel of the actual challenge of delivering this to users. Its the complex care and feeding of a live service that matters.
> Knowing that [program] revival [i.e. bringing a new development team “up to speed” by having them learn the model] is a plausible future need has powerful consequences for our work.
I’m not sure that most developers are willing to revive software, based on my observation that very few read anything at all, especially the source code. Instead I see a lot of adjusting the input and output of the existing program by adding a new layer. This new code is totally understood by the new dev, and they can modify/maintain it easily without worrying that they broke the existing system. It also usually duplicates something that already exists inside the system. As the process repeats more and more layers are added.
I think a few lucky teams have developed a culture that encourages learning the existing code. (Popular web frameworks comes to mind as an example.)
> The problem is that TEAM members don’t have enough elements to build a satisfactory mental model of SVC. They need to go by a mix of the client’s interpretation of what the system should be, and what they can tell from the code that the system actually is. These views can be disconnected and contradictory. The code may tell the what and the how, but it doesn’t tell the why. Only X10 could say what was a functional requirement, what a technical necessity, what a whim, what an accident. The team has to resort to reverse engineering, extrapolating, and guessing.
Hence, write down your thought process, mental model, and assumptions alongside the code. Tip: Call the process "writing documentation" instead of "commenting code".
Part of the story that is mentioned but not discussed is that X_10 had a fixed and tight schedule. Under that constraint you expect some compromises for writing the mental model down in a way others can learn, because that model presumably lived in her head.
It might have been possible to say, great job you delivered on time, now you have as much time as you need to write your mental model down. On full pay of course, and we won't count that against you as non-technical work in the next performance review.
That increases the upfront cost of SVC but is an investment that pays back interest as soon as someone else has to fix anything.
This is a general problem. Why's and how's. Or in project management, and what should be associated documentation, process and procedures. In this case code is procedure or how. The steps to do the thing. Which is great but it's hard to make meaningful and useful changes without understanding the Why, process, does the thing needing to be done and why, process, are the procedures or code doing the things they are in the way they are.
Presumably for code you would get enough why/process via comments but that seems unlikely. Perhaps coding needs to take some other tools from project management or something? Knowledge sharing/transfer is hard.
I can't resist to point out that, in theory, there are at least two other options avoid wasting many resources via the failing new teams:
1. Get the original dev to explain his theories (keep employees longer or engage them as consultants) 2. Make and get a "diary" of the original devs theory building.
In this story, and in probably many places, the business environment however supports the explained outcome.
Kind of a roundabout way of saying that you can’t evolve a program without fully understanding the codebase
I imagine that experienced cross-disciplinary designers would concur that all complex design is knowledge building, which is why documenting design decisions is important. You tend to learn this lesson when maintaining projects of nontrivial complexity over a longer period.
The problem I have is that only people who "know" already tend to accept this insight.
i.e. explaining it to those who make decisions doesn't work that well because it doesn't fit their mental model very well - they don't understand that a large part of their asset is sitting in people's heads.
That guy wants 5k more.....? Pay it. The cost of hiring someone new and training them up and relearning it all will be far higher. Don't force people back to the office, be relaxed about everything and keep the knowledge.
At the same time make sure other people are learning it so you do have replacements.
At the same time make sure you have proper tests so you can work with code you don't understand yet if someone leaves.
At the same time try to gather all the correct and uptodate documentation and explanations somewhere.
Good article, both in spirit and factually.
One thing to add: the author talks about reviving a system as “slow and difficult process”, and it is. However, the concrete example described is not worthy of hand-wringing of this kind: a system that could have been built by a single competent engineer in 6 months (inevitably of alpha quality, at best), could be resurrected by a competent team of several programmers and brought to, say, beta quality, while keeping the lights on their alpha system on in how long? Let’s say, 9-12 months. No biggie, really.
Most companies routinely discard man-years of programmer’s effort, so those 9-12 months are likely just a blip in the lifetime of that firm.
> The program should preferably be discarded, and the new team should be given the opportunity to resolve the problem from scratch.
Yeah right. "We don't know how it works so we're going to ditch it and create it again."
It works in some cases but by no means should that be the default.
Great article, +1 Insightful.
> the ultimate goal of software design should be (organizational) knowledge building