My perspective is that your typical business does not incentivize, or care to allow their engineers to do "proper" software engineering, which is effectively a proper abstraction of the requirements and its implementation from scratch (to a great extent), in order to maximize the extensibility and control of the software. Most orgs love to use some kind of existing framework or library that comes with heavy constraints on how you need to implement requirements (typically everything above the database layer).
If business domains are properly abstracted, most of the time extending the software with changes is something close to adding some configuration or using existing functions with different parameters. But the hesitation to create things from scratch comes at the cost of training and also transparency, two of the highest priorities for managers and stakeholders who not only want to hire people with minimal overhead to begin contributing, but also want to understand as much as possible what the engineer is doing and how they are doing it.
Software is incredibly maleable and has unlimited potential, yet somehow most places i've worked at get constantly bottle-necked by recurring problems that are very easy to solve with proper abstractions.
Figuring out the proper abstractions is much, much harder than writing the software itself. Software changes requirements... When you do something new and useful for a customer, they will extend use of the software till it breaks again. You will always run into a new bottleneck.
The proper abstraction is that which can easily be thrown away. I've not found that to be too difficult to figure out. Maybe when I was new to development – that was too long ago to remember – but once you have experience, at least.
It can become extremely difficult to figure out if you mar yourself with someone else's framework. That is quite true. But is also what the parent said.
It's hard to generalize as domains vary greatly, but in general abstractions can account for more than enough extensibility unless the requirements are extremely vague and not well defined. This tends to be the case of why so many changes are constantly needed, because usually Product and Design are not defining requirements with abstractions in mind.
Domain Driven Design basically aimed at tackling this issue, by trying to synchronize abstractions across departments, so I get that there are no easy solutions, especially in large orgs.
Especially in startups clawing there way to MVP release and increasing users/income. Everything is write code to solve specific need now without time to see if it can be better used/implemented. Gotta be first. If we're not first, gotta get traction to overtake first.
Even knowing that and attempting to build something with best principles is still going to be a pain to maintain as it's always a moving target with no proper specs.
There's best principles and then there's over engineering. Business domains can't be properly abstracted when the business itself is in the process of being defined, so if you're the kind of software dev that craves building to a spec that won't ever change, one possibility is that a fast-moving 12-person startup just might not be your cup of tea. Nothing wrong with that, just different.
And if you are the type that does well with band-aid code strung together with duct tape and bailing wire and even bubble gum, then don't get upset when your company gets bought and the entire code base gets rewritten into something that teams can manage
Fully agree. People are generally incentivised to solve their domain problems not to develop something that solves theirs - and potentially - problems of their adjacent team. If you can solve for this then you should be encouraging the right behaviour for abstractions which (as others have said) is a hard but worthy problem to spend time on.
Well put. I think building software systems can be done with proper engineering techniques. But more often, many businesses don't want that! I've seen some go out of their way to chose technology and processes that eliminate any need for making engineering decisions. The goal is often to "standardize" and explicitly eliminate all novelty. And then they wonder why innovation is stalled.
It's way easier to tell your developers to sprint every two weeks in some random direction, because at least then the business has control. If they were to use the scientific method, the direction would be chosen based on experiments and empirical data that they don't understand or control. Can't have that.
Which proper engineering techniques are those?
In general I agree with you, but the devil is in the details and a licensure body is surely the wrong way to do this.
Engineering (IMO) has nothing to do with licensing, and everything with how you approach the problem - Do you use gut feeling to make decisions, or do you follow the scientific method?
In most so called "engineering" orgs, the scientific method is explicitly banned - decisions are made by weird scrum rituals and fabricated storypoint metrics.
> Well put. I think building software systems can be done with proper engineering techniques. But more often, many businesses don't want that! I've seen some go out of their way to chose technology and processes that eliminate any need for making engineering decisions. The goal is often to "standardize" and explicitly eliminate all novelty. And then they wonder why innovation is stalled.
That's not necessarily bad. The company is derisking by relying on building blocks developed by established companies with deep talent. The vast majority of projects where local developers wrote everything from scratch are nightmarish to maintain, secure and upgrade.
It's not a bad thing if you're doing pure applied technology (IT) work. That should be standardized, there's really no innovation to be had, just optimization. It's when you develop novel products that innovation is required and where rigid standardization becomes a constraint.
Part of the challenge is that most developers are not engineers. In proper engineering training, there are best practices and safety guidelines taught across the board by accredited schools. In software any cowboy with a keyboard can produce code. Unsurprisingly, most of it is shite.
I feel like we managed to get much closer to proper engineering in our own little corner: version control, CI/CD, code reviews, don't-write-your-own-crypto, standardization, etc.
The problem is that it's still hard for software engineers to go to business people and say "no, we can't build a bridge that crosses the Atlantic Ocean".
With the introduction of non-technical Product Managers it became even harder. Now we have to convince two layers of people that some idea is unsafe or that it's gonna be more trouble than it's worth.
> With the introduction of non-technical Product Managers it became even harder. Now we have to convince two layers of people that some idea is unsafe or that it's gonna be more trouble than it's worth.
It speaks again to the lack of rigor in software. In professional engineering practice, the project manager is always a P.E. who takes responsibility for the final design and signs off on the product before it goes out the door.
> My perspective is that your typical business does not incentivize, or care to allow their engineers to do "proper" software engineering
Which fields do you think are allowed to do "proper" ________ in a typical business?
If you give me the choice between working with 2 groups of people where one says, "Let's get this done the right way!" and one that says, "Let's get this done!", I'll join the second one every time.
It's not so much that there are wrong engineering techniques as much as engineers being heavily constrained and not allowed to exercise their full expertise. An example of where this is not so much the case would be areas like civil engineering because it has serious consequences if the engineering isn't done right, i.e. a bridge falling. With software there is a lot more room for interpretation as to what engineering really is or should accomplish.
> It's not so much that there are wrong engineering techniques as much as engineers being heavily constrained and not allowed to exercise their full expertise.
What other fields do you believe are not heavily constrained and are allowed to exercise their full expertise at a typical business?
My point is that nearly everybody feels constrained at work because of decisions from higher-ups or whatever. It’s not like software developers are some unique set of people who face challenges nobody else faces.
Other engineering disciplines, for one.
And I'm not saying it's perfect, I'm just saying it's done properly.
Boeing practices would disagree with you.
Boeing was charged with fraud, to the tune of $2.5 billion. Crime and outliers are not a counter-example.
I think I’ll agree that some licensed fields (medicine, engineering) are more likely than not going to be allowed to do things the “Right Way.”
But at the same time I’d bet if you asked people in those professions they’d likely say, “Our bosses don’t let us do things the proper way.”
It’s more likely everybody thinks they know better than their boss regardless of the field.
I worked as an electrical engineer in the past, and there was definitely a lot of corner cutting, less-than-ideal parts, less-than-ideal suppliers, low-quality materials, and simple day-to-day things at work that just make the life of engineers bad.
But that doesn't mean you throw the baby with the bathwater. I was still liable for things like fire hazard or RF interference, among other things.
Professional Engineer licensure to be able to actually charge for software (this carves out exception for FOSS and freeware) could raise the costs enough to make this kind of software engineering viable. The problem today is competitors will iterate with piles of temporary solutions and eat your lunch while you're still designing your abstractions. If they got sued into oblivion for shipping broken crap, they wouldn't be so quick to iterate on junk!
PE licensure could also foster a security culture in software like none other, as no PE is going to put his fortunes on the line to save a buck skipping formal verifications and avoiding inconvenient best practices!
> Professional Engineer licensure to be able to actually charge for software (this carves out exception for FOSS and freeware) could raise the costs enough to make this kind of software engineering viable.
You would almost certainly see an increase of in-house software developers and consultants where the code is never charged for, but is developed as it always has been. A few big companies like Microsoft and IBM would get certified and smaller software companies would be driven out of existence entirely.
There would definitely not be a golden age of rigorous software.
I would love for the U.S. to standardize software engineering like electrical engineers. It wouldn't change that much, as most companies would end up switching to "developers", but it would make it clear when a business wants some serious problem solving and not just churning out half-baked features as fast as possible.
Law may be part of the solution, but overall we need a culture of demanding tailored demos. If you ask an average SaaS, "show me how this can produce value in my environment before I buy in", they may be incredulous and expose that it will take months/years of effort to get it properly integrated.
OTOH there are many genuinely great products that don't require massive upfront effort to start producing value. Demanding tailored demos can help you separate the wheat from the chaff.
Marginally related. I wish I knew the context any more, but I encountered something like this.
App keeps updating. Version N understands version N-1's saved data, maybe N-2, N-3, but not indefinitely. Still you're lulled into a sense of security; your data will always be retrievable. Goes without saying that the app modifies the data store to the latest version.
But then you don't use it for a while and you it's now at N+10 and you still have data from N, and it's slid out of the "backwards compatibility window" and is no longer readable. And the intermediate versions of the App are not available. Now you're screwed. One such scenario, for sure, is updating from version N to, say, N+5 of Android and trying to migrate your data, but I think it was something else.
I thought about this a fair bit, and thought that the way to fix it is to do versioned everything which is an interface to your application or library. What I mean is, as best as one can, one should never break convention and API without including a mechanism to automagic it to how it was before. Making changes which break downstream should be considered the gravest of sins, following at a close second place behind leaving security vulnerabilities unpatched.
This is where good abstraction, while expensive to develop, is important: if the abstractions are well-formed then fixing code or adding features should not cause downstream breakage. We always want downstream to focus on using our projects to make cool shit, we don't want downstream spending most of his time fixing his abstractions because we changed ours for the fifth time in seven years.
Version 10 should have the figurative similarity matrices to project any version of the format from v1 to v9 into v10 format.
Most software hardly moves at all. Consumer hardware moves backwards though, I can no longer get a phone with all the features I had in 2012, removable battery, headphone jack, and an sd-card slot.
Even though I agree with you, if adding features meant keeping all existing features we'd never progress anywhere.
Like in medicine?
You can absolutely buy a phone with all of those features.
There are at least 45 such devices meeting those exact criteria released in the last four years: https://www.gsmarena.com/results.php3?nYearMin=2020&chk35mm=...
This is why it's important to say dumb things :)
Thank you!
A better title is "Once you start keeping financial state, there are no rollbacks". The current title might as well be click bait.
I was going to say... Someone didn't seem to have experience with restoring from backup.
“You'll find finance moving it to spreadsheets and other weird programs, data integrations putting it in warehouses and BI tools, and it's going out in emails and 3rd party payment processors and their bank and your bank”
How did you miss something so close to the beginning of the article? I’d love to hear how you “restore from backup” all the external systems that data propagates to.
You just deal with it. Manually if need be, sending someone a spreadsheet. Not that hard, just a right pita.
Tbh the article's actual example is stupid, if I desperately need to rollback that change I'd just take a backup of that table and change the new types back to the main payment method, 'Stripe'. When we've got it fixed and redeploy the changes, change them back.
Problem solved in two SQL queries.
Not sure if the author's never had to deal with a problem like this before, but his 'impossible' problem's got a bloody obvious solution.
SELECT * INTO _paymentsbackup FROM payments WHERE PaymentType in (3, 4)
UPDATE payments SET PayemntType = 1 WHERE PaymentType in (3, 4)I take a different lesson from this - there are parts of software that you can change easily and parts that are 100x more difficult, and you should design your system so that you know which regime you're in at all times.
They all look the same in the codebase unless you have deep prior context.
my framing of this is "Data outlasts Code, yet Code keeps winning" - i think data/schema design should be much more emphasized in bootcamps and college classes but sadly it seems most of us have to learn these painful lessons on the job.
Your link did not outlast your comment. I'd love to read about schema design if you have an article you liked...
i really liked the schema design exercises in https://www.dynamodbbook.com/ and ofc there's the classic Kleppmann DDIA book
I've never bought this distinction between code and data. They are two distinct things but they are inseparable. Neither outlasts the other.
Software only moves forward in the sense that time only moves forward. Not sure there's much more to it than that.
This highlights why a design process is so important. It feels great to start with a clean slate and “just code”, and many seem to get rewarded for that in a “move fast and break things” culture.
However there always needs to be a plan on how to migrate data that uses the old way to the new way. It’s not as sexy, but it needs to be done and takes time away from building new features. Skipping the design process is tempting, but it’s a direct path to this type of technical debt that’s easy to foresee.
Yes, but more than a plan: there needs to be a well-tested schema migration process. Schemas can be upgraded. Sometimes you write to both the old table and the new table for a while. The best way to make this less painful is to get good at it, investing in the tools to make it easier.
You can’t anticipate all design changes, but you can get better at making them so they happen faster. This means getting practice at doing them.
This is why it is important to consider what kind of software or software components are surviving the tests of time and using them as a guide for future software.
As a software dev my gut feeling is that this requires the polar opposite of move fast and break things — instead of going for the obvious solution it requires you to think a bit more how data flows (or ought to) flow through the organizations using the software.
You can always refactor. Sure it takes time and effort but it is possible. And it serves as a good painful experience to invest more on design first, to your point :)
Moving software "backwards" is hard, but not impossible. There are ways to structure both data and code that make it easier. A change being "rollback-safe" can be an important question in designs and code reviews.
Event store architecture seems to solve this rather eloquently.
Store a single event on the new architecture, and you can't go back anymore either.
For your service, sure.
And I mean, I could quibble, but let's roll with it.
Good luck getting finance to work with "event store-based spreadsheets", finding an "event store data lake" for your analysts, getting that other engineering team to also switch to an "event store"-based view of the world when they just want to query the system about a point-in-time about whether a user is up-to-date and aren't going to make another one for another month no matter what you do to your data store, so you'd better be right....
Event stores are an internal implementation detail, like whether or not you use Postgres or MySQL. They don't save you from the effects of changes and the parts of your state that leave your control. The example is well-chosen... billing state is going to leave your system, and the business forces causing it to leave your system are going to be powerful to the point that if you try to stand in front of them and yell "STOP! You all need to use an event store architecture if you want billing data!" that you will be fired and replaced.
Which is not to say that it's all hopeless. You can take care of the data well inside of your own domain. But as I know from experience, you're not freed from these forces just because you've got a good architecture. You really, really want to careful how you proceed forward, the moreso because nobody else working with billing data is going to be.
I'm responsible for the "usage based billing" portion of the billing system. I've often thought my primary contribution to the billing system hasn't been the usage based billing, but sitting in on the meetings as a senior engineer with a bunch of other people who are not engineers and whose programming teams are also busy with other non-engineering tasks, asking the right questions, and making sure that we get down to the true crux of the matter and find the exact right groups to do the right tasks with the right data coming from the right place, and making sure we don't get a whole bunch of workarounds from the wrong people sending the wrong data to do the wrong tasks, and then someone somewhere "just" "fixing" it later. Even when, and perhaps even especially when, the final task results in no action items for me or my team and all I did was ask the right questions, it is just so much easier when you don't make architectural mistakes in the first place in these cross-team decisions. Basically, using those requirements elicitation skills more than anything else.
My system basically is an event store. That doesn't so much solve the problems on its own, as keeps me flexible and able to respond to the other much less flexible systems as we work through problems.
This article brings up some very simple software engineering problems and the most obvious/widely used approach to dealing with them, but this is by no means the only approach that works best at scale. Also not necessarily the best approach at scale because sometimes you do need to roll back.
If you're serious about writing robust and reliable systems, think about V-1 and V+1 not just the version you are writing and deploying. The point about your code interacting with multiple versions in production is a good one, but also upgrades and downgrades should be considered.
I thought _everything_ only moved forward... ?!