« BackBuild systems, not heroesvitonsky.netSubmitted by thunderbong 6 hours ago
  • layla5alive 3 hours ago

    This article represents the death of excellence and craftsmanship. Its the McDonald's-ization of software development. McDonald's does make the most consistent "food" in the world. But nobody would say it makes good food. No chef would ever want to work preparing food at McDonald's.

    The author pointed out many good ideas - the problem is the extremism - the author clearly hates software developers and the complexity of software development (a common trait among management), and wants to eliminate the complexity and variability inherent to dynamic human creation. Once you've taken all the creativity and joy out of it, and distilled it down to something any monkey or even ChatGPT can do, yes you've achieved such a system, and it's a quite dreadful place to work.

    A less extreme version of what the author proposes can enhance creativity and net velocity - but these systems need to be in balance, supporting and enabling the creative process, not eliminating it.

    • solatic 3 hours ago

      But the world has enough room for both McDonald's and the farm-to-table restaurant. They serve different markets and fulfill different needs. There is room for both Big Agriculture monocultures that have industrialized agriculture and the Polyface Farms intensive farmers who are both happier and more productive and profitable per-acre than the monocultures. But large enterprises, for all their cookie-cutter, cog-in-the-machine dehumanizing process, are great places for many folks: for the early career folks who are just getting started on the experience ladder learning professional development and get lots of benefit from all the guardrails, for mid-career professionals starting families who need to tread water in their career while they focus on their young children at home, for late-career elders who prioritize both stability and the late-career paycheck to help get them into a comfortable retirement.

      The macro goal though isn't to attempt to deny the benefits of process in large enterprises, it's to promote more productive small business. There are natural benefits to scale in large enterprises, but in software, many of those benefits to scale are not inevitable. Outfits like WhatsApp succeeded at building immense scale with very few engineers. It may be a bit of survivor bias, but small businesses can outmaneuver large enterprises when they have more productive talent that is not hemmed in by all the safeguards that larger enterprises are required to have. But this is still separate from the fact that each have their place.

      • insensible 2 hours ago

        I’m a permaculture designer and have run a small vegetable farm. I have visited Polyface twice and spoken personally with Joel Salatin both times. I’ve read several of his books. I think that Polyface shows up in this discussion very much on the side of the systemic safeguards the article recommends. His farm is full of systems and safeguards a great many things by design. He plans for every risk he can think of and adapts his system in response. He got the farm from his father and has largely transferred it to his son, and they have built a business training both interns and the interested public in their methods.

        Safety measures ≠ world-dominating industrial scale.

      • Enk1du 3 hours ago

        The best advice I've seen on the issue is "Don't scar on the first cut", as in you shouldn't try to add a new rule every time you have an outage.

        That being said, I absolutely hate heroes.

        I've worked with a couple that get their thrills from the adrenaline buzz of swooping in and fixing the big problem ... and walking away. They don't put the work into documentation or making systems resilient because that's boring. I like boring. Boring means I can clock off at the regular time and not think about work until the next day.

        • drewcoo 2 hours ago

          > I absolutely hate heroes

          I dislike the heroes' bosses. They're to blame for the situation. Managing a team of engineers is not like being a dungeon master.

          • satisfice 2 hours ago

            You don’t know what heroism is. Read the literature of heroism before you decide you hate your fellow humans. Each one of us is capable of heroism.

            • Angostura 2 hours ago

              And a system that requires it if it’s employees to function is fundamentally abusive

              • Jolter an hour ago

                The article is not about real heroism though. It’s about workplace “heroes” in the sense of diligent, creative, productive workers who have to take personal responsibility for the success of the business despite lacking support from said business.

                • Enk1du an hour ago

                  Ah, now the Natural Born Heroes heroism and the virtue of Xenia (ξενία) I can get behind.

                  It's the ego-inflating ""heroes" in scare quotes" that crave validation that I find so exasperating.

              • jonathanlydall 3 hours ago

                I agree that the author’s proposal is too extreme, but some of their advice is good.

                I think any company wanting to scale to a significant size will need to learn to implement effective systems rather than rely too heavily on the efforts heroic individuals.

                However, what the author is proposing is the death of evolution in systems, they seem to think that an optimally efficient and effective system already exists and everyone should just be using it.

                This is not true because your competitors are probably always looking to change their own systems such that they can out manoeuvre you, unless you respond by adapting your systems, you will likely be out competed.

                My company’s approach is to progressively systematise everything reasonably possible, but it also has the philosophy that any existing system can be challenged for change or removal.

                We regularly have retrospectives primarily to try identify how we should change our systems. We never try to change too much at once, but over long periods of time our systems have tended to towards being highly effective.

                But understanding and identifying highly effective changes (beyond the low hanging fruit) requires exceptional individuals.

                We will always need heroes prepared to challenge existing thinking in case there is a better way.

                • bjornsing 2 hours ago

                  I agree wholeheartedly. I’ve seen this philosophy take over a software organization (Ericsson way back) and the results are dreadful. But it’s not only that the org turns into a dreadful place to work. You also lose in the market to competitors with a more creative culture. Ericsson could easily have been the AWS of the world and the creators of Android. Now it’s like… a shadow of its former self.

                  (There are other reasons too of course, but I think this “process over people” philosophy was a major contributing factor.)

                  • baldeagle 3 hours ago

                    This comment combined with a recent reflection on the ‘irony of automation’ makes me worried that the product I’m building to make life easier will instead lead to loss of craft, pride, and ingenuity.

                    Honestly, still processing through it to make sure that all the people in my products and systems are seen.

                    • pell 9 minutes ago

                      It might sound redundant but I do think there can be an achievable balance. I worked at an organization that was allergic to documentation and safeguards of any kind and relied fully on certain talents to resolve issues. This was not just incredibly inefficient but also caused many problems when some of these people left the company. I think some safeguards and consistent documentation would have avoided the majority of this. And the talented individuals would also have had more time to work on other things, learn new skills and get even better at crafting in a more broad sense instead of only playing fire department.

                      And finally, I can reassure you that there will always be room for “individuals” and “heroes” to do their craft and also save the day. Most systems don’t run like clockwork, there are always bugs, weird failures, strange oddities, most systems will need saving at some point. So, creativity and craft will generally have a place and value.

                    • havkom 2 hours ago

                      Yes. Focus on building all your team members to heros instead.

                      The other way around is possible, but you will end up with a team of 1000 people doing the same work as 10.

                      • havkom 2 hours ago

                        BTW, I am currently working in an enterprise with a small team mixed with experienced developers (heros - but still always learning because of new complexities) and new developers (heros in becoming).

                        Absolutely fantastic and we create wonders, but it requires management to acknowledge skill and exceptionalism.

                      • voidr an hour ago

                        If creativity means allowing people to be creative with dangerous language features that they themselves probably don't fully understand and write code in a way that nobody else understands, then I would want none of it.

                        Having working and maintainable software is a lot more important than having a team member showing off what they are able to do with eval, they can put that into a demo side project.

                        • jongjong 3 hours ago

                          Spot on. The comparison between a skilled software engineer and a skilled assembly line worker is a flawed one.

                          The game of the assembly line worker is mass production; the impact of their skill is limited to a subset of all units and their contribution to each unit is also limited.

                          A software engineer is more like a civil engineer or a surgeon. Their work affects the entirety of whatever they're working on at a foundational level. The value of the unit they work on is extremely high. Also, a software engineer doesn't assemble things; they plan and design things. Their impact outlasts their tenure by many years. Just like a surgeon's impact extends far beyond the operating table. Once the software engineer has quit the company, the code which they wrote is still crunching numbers and still earning income for their ex-employers while everybody sleeps.

                          It's not just their code which lasts, it's the entire philosophy which they breathed into the product and their team which lasts as well.

                          • drewcoo 2 hours ago

                            > The comparison between a skilled software engineer and a skilled assembly line worker is a flawed one.

                            And the difference is that we engineers design and maintain the assembly lines, as opposed to being line workers.

                        • atomicnature 3 hours ago

                          I think the author gets it utterly wrong.

                          My argument would be - greater the complexity, higher level of personal initiative and responsibility is needed.

                          This is not an arm-chair assertion. For example - read the works of Admiral Rickover - https://govleaders.org/rickover.htm - in buidling and managing nuclear facilities

                          You'll see the enormous focus on individual initiative and personal responsibility.

                          Heavily regimented processes are too brittle for dealing with complex problem-solving end of the day.

                          In fact, one way I differentiate sophisticated leader from a less sophisticated one is via the understanding of this fact: do they value individual initiative or not?

                          If they do not, they're not in a great position to lead complex initiatives.

                          • layla5alive 2 hours ago

                            Spot on! It seems to be one of those things where the left and right sides of the bell curve both value individual initiative, creativity and responsibility, while the middle appears to believe systems are a panacea.

                            Once you've built enough systems and seen the tradeoffs and understood the raw high dimensionality of the problems, one may come full circle and realize yes, good systems are great, and some are table stakes, but only insofar as they add rather than subtract value relative to humans who are invested with great personal responsibility, mastery and creativity.

                            Rather than turning everything into a dreadful machine where humans are but a cog (ala the matrix), build systems which are more like force multiplying exoskeletons (edge of tomorrow)

                          • 000ooo000 2 hours ago

                            >However, we had a long discussion with one programmer who insisted it was a bad idea because of "premature optimization", the fact that React docs does not have recommendations about it, and "nobody does it that way". In my opinion, this was a clear case of a programmer resisting the system approach because he wanted to spend time fixing the same problems over and over and pretend to be working hard.

                            This article and advice is awful. The fact that the author ignores completely reasonable arguments made by the dissenting programmer, instead attributing their 'resistance' to some dubious, made-up notion that they would prefer to do meaningless rework, speaks volumes.

                            • grncdr an hour ago

                              I think it’s worth also quoting the rule that the dissenting programmer objected to.

                              > One of the solutions was to introduce a rule to memoize everything returned by hooks, to automatically prevent future problems with memoization.

                              … which introduces a systemic performance penalty to prevent outlier performance problems.

                              The article argues against “heroes”, but I think the more appropriate term is “experts”.

                            • kstenerud 3 hours ago

                              The author's being a bit hyperbolic, but the point stands that systems should be built to prevent or at the very least mitigate damage due to human error, because human error will ALWAYS happen (even to the heroes).

                              No, this isn't about bringing on cheaper, less skilled labor; it's about making a system accessible so that people can work on it without fear.

                              You STILL need those heroes, those masters of the system, because they're the only ones who can not only keep the innate and unavoidable complexity of the system in their heads, but can also expertly manage and improve the human interface simplifications and safeguards that prevent catastrophe.

                              We've done this in other industries, and yet there's still so much pushback when it comes to software engineering...

                              • devdude1337 3 hours ago

                                The author seems to try to enforce programming constraints in order to fix one problem they identified, to fix all related problems that may occur in future.

                                It is obvious that he doesn’t have experience with long living systems, behause this approach leads to inflexible software. For example: sometimes memoization of react hook returns is sometimes good, but never always. He also values processes much over individuals - well we had a manifesto against such views…

                                Future proofing, assembly-line mentality, knows-it-all thinking - I feel sorry for those he manages and the company when they’ll become unable to add features at all.

                                • rafftre 3 hours ago

                                  Enterprise programming is the management of labour costs, aka lowering software developer wages. Prioritizing systems is a way of pursueing this objective, while introducing less prepared people to work trying to limit their consequential mistakes.

                                  • mtkd an hour ago

                                    Keeping enterprise software behemoths on better life-support is no longer a viable strategy for most ... much of it is vulnerable to generative and predictive AI powered incarnations of whatever it was originally built to do ... retro-fitting will, in most cases, not be as competitive as rebuilding using a completely different architecture

                                    This is the early 1700s for enterprise software and we are witnessing the equivalent of discovering the steam engine -- almost all will need to be rebuilt in next 15 years and that requires the inverse of a cookie-cutter engineering approach as optimal patterns are yet to be established

                                    • gabrieledarrigo 2 hours ago

                                      I agree 100%. And in fact it's one of the principles of the devops culture. Automation, trust, light processes, early feedbacks.

                                      Companies that relies on heroes and constant firefighting to keep running are doing it wrong. Very wrong.

                                      • krig 3 hours ago

                                        Surely, this article is lampooning the idea that with enough rules and regulations, skill and experience don’t matter.

                                        For a contrasting argument, I recommend reading Programming as Theory Building by Peter Naur.

                                        • andycowley an hour ago

                                          I find it quite telling that the author quotes the "fundamental attribution error" early in the article, and then spends the rest of the article making textbook examples of exactly that.

                                          • bbojan an hour ago

                                            Ralph Waldo Emerson is as relevant as always: "A foolish consistency is the hobgoblin of little minds, adored by little statesmen and philosophers and divines."

                                            • sabbaticaldev 2 hours ago

                                              the example the author gives in the end is such a low level bad programming advice that it’s not possible to take the rest seriously

                                              • shehjar 4 hours ago

                                                The degree of specialists that current tech stacks encouraged is not helping prevent creation of such heroes.

                                                It’s no wonder that we now have software developers working under the assumption that one person can run the whole show.

                                                • micahcc 4 hours ago

                                                  > If you have strong enough rules, you don’t need to worry about the skills of the programmers.

                                                  lol. lmao, even.

                                                  • pedrosorio 3 hours ago

                                                    Followed by

                                                    > In my opinion, this was a clear case of a programmer resisting the system approach because he wanted to spend time fixing the same problems over and over and pretend to be working hard

                                                    Which is funny when the first bullet point under "skills" in the author's resume [0] is:

                                                    > Team player mentality

                                                    I love working with "team players" like this.

                                                    [0] The "About me" page in website: https://latexonline.cc/compile?git=https://github.com/vitons...

                                                    • tpoacher 2 hours ago

                                                      I didn't know about latexonline.cc. TIL. Thank you :)

                                                      • therein an hour ago

                                                        If I were the author of this blog post, I'd be embarrassed. This is plain embarrassing.

                                                      • yxhuvud 3 hours ago

                                                        Indeed. That strong rule at the end where he want to memoize everything made me laugh out loud. What could ever go wrong with such a rule..

                                                        • stonethrowaway 4 hours ago

                                                          [laughs in job security]

                                                          • bigs 3 hours ago

                                                            Sounds like magic

                                                          • yard2010 3 hours ago

                                                            Alternatively - if you're that 10x oil rig worker, make sure no one ever does this, because then you're turning from a hero to garbage

                                                            • therein an hour ago

                                                              > However, we had a long discussion with one programmer who insisted it was a bad idea because of "premature optimization", the fact that React docs does not have recommendations about it, and "nobody does it that way". In my opinion, this was a clear case of a programmer resisting the system approach because he wanted to spend time fixing the same problems over and over and pretend to be working hard.

                                                              Author has some serious issues he needs to work on. Also probably should delete this post because it looks embarrasing.

                                                              He wanted to memoize everything returned by React hooks as a kneejerk response. I don't even have anything to compare it to. It is plain silly.

                                                              • jackhodkinson 2 hours ago

                                                                When the engineering department of a company is not the major lever in the company's growth this philosophy might be fine. That's the case of many large enterprise and probably many startups too.

                                                                When the competitive advantage is not the technology, it is probably smart to make the engineering department dumb.

                                                                Clearly that leaves the company exposed to challengers who will outcompete them with technology. But not all markets work like that.

                                                                • satisfice 2 hours ago

                                                                  No, build heroes. Then the heroes will build systems as they see fit.

                                                                  Build societies, and then the societies will build heroes.

                                                                  I am a humanist. I will agitate and undermine any initiative or attempt to dehumanize our systems or systemitize dehumanization.

                                                                  A hero is not some extraordinary person. A hero is an ordinary person who, in the ordinary way humans do, transcends mere established procedure and solves an ambiguous problem— a “wicked” problem— for the benefit of others. That’s a hero. A hero is not a six sigma person among other people, but rather unique among any set of non-human systems