Doing great product engineering

I’ve been lucky in my life to observe some great engineering teams, and to work with great people.

This post is an attempt to put together the core threads of my learnings on what it takes to do great product engineering especially when you’re a smaller team. I’ve written a bit in a similar vein before, but I want to go higher-level and describe why these details matter, and what sort of culture it creates.

Going 0-to-1 is a unique skillset and something most engineers never learn how to do. For most engineers, they’ll work at an organization where someone else has already made a lot of those fundamental technical decisions, and they have to work with whatever is already there. As a result, they don’t have a deep understanding of how much those early decisions affect the success of the team (and the overall organization as a whole). And to add a more nuanced point to that, they often don’t know which problems need to be solved now and which ones can be put off till later.

People love to say that they want to build great products with small teams (an unconventional outcome) but make all the most conventional, mediocre (or below-average) decisions thinking they will get there. Unfortunately, making mediocre decisions that compound is not how you get to an unconventional outcome. In a world where great engineers can work anywhere they want, why would anyone want to work with you? Especially if you’re a smaller team with limited resources?

My general framework for thinking about great product engineering is that the following things matter the most:

1. Choosing great (i.e. high-leverage) tools.
2. Hiring great people, which using great tools gives you one of few things you can differentiate on especially as a smaller team (most startups can’t win on compensation or really much of anything else). People who are obsessed with their craft and want to build real solutions to problems can move mountains and bring success into the realm of possibility through sheer willpower.
3. Iterating quickly on not just the product (as most half-decent teams already know they’re supposed to do) but on the codebase itself, which having great tools and great people allow you to do.
4. Solving problems early on in the development lifecycle, and not waiting till things end up in production, which having great tools and great people allow you to do.

Each of these engineering/product decisions are deeply related to each other. You are unlikely to attract the best product engineers over time with a mediocre product and/or a mediocre codebase.

Notice how everything follows great tools? And ultimately great people? You’ll see that pattern in the remainder of this post.

The most important decision to care about in the early days are the fundamental tools that you choose. It’s fundamental to how engineers express themselves. The teams that (often quietly) ship great products with small teams are principled first and foremost about the tools they choose, and playing to their strengths. This comes in many forms. For some teams, it’s using an obscure programming language no one has heard of (which tends to self-select for more hardcore/passionate programmers, aside from the unfair advantage the tooling itself may provide when driven by the right person). For others, this means pushing parts of a “conventional” stack much farther than people think you can (relying on a database like postgres to also hold a lot of business logic). For some , it means using a stack that’s fully open-source and self-hosted with complete control and deep understanding of every tool that’s being used. Using high-leverage tools allow you to build a leaner team while shipping higher-quality product faster. Not all tools are created equal (and certainly not created equal for all types of problems). One of the simplest tells that an engineer hasn’t internalized the importance of tools is when they claim that all tools are somehow equal, and that it all depends on the player skill to be able to turn something good or bad (i.e. it’s all relative). I’ve seen this incorrect train of thought expressed by both individual contributors and engineering leaders alike. The best engineers tend to learn (typically the hard and painful way) that tools are not equivalent commodities (which is the common and incorrect wisdom) and success is not entirely dependent on the player skill alone.

So how do you choose the right tools? I like to describe a codebase as a living, dynamical system. A human being is an example of such a system - our bodies are constantly going through different processes to keep us alive, and we have to consciously do work to perform at an optimal level (go to the gym, sleep properly, eat well, etc.). Similar to how your body evolves when you go to the gym, the only thing that’s constant about a great codebase being built from the ground up is a need to evolve. This is probably a heretical statement, because I’ve seen many engineers praise great codebases on how little work needs to be done to them after things work. However, the reality of a small team or organization is that your work is just getting started, and more likely than not as your organization grows, you will need to evolve the codebase. The thing is, surprisingly often, engineers don’t seem to have the foresight to think about what the consequences are of choosing tools that are not suitable for growing teams, evolving problems, and evolving products. The tools you choose should allow you to confidently change any part of the system knowing that things are not going to break, and to iterate quickly. The truth about an early product is that you usually can’t (and shouldn’t) be solving scale problems at the beginning. But what you do want is a setup that’s capable of being evolved/improved. You’re not going to get every decision right, but it should be fixable over time instead of feeling like it’s going to fall apart at any second. High-leverage tools that let you punch well above your weight are probably the biggest unlock an engineering team can have - the most incredible thing about choosing high-leverage tools is that they also have a much greater likelihood of attracting great engineers (ask your favorite designer if they would choose Microsoft Paint over Figma/Photoshop). My favorite high-leverage tool is a language with a powerful type system like TypeScript (which also happens to come with a ton of additional tooling that I also use, such as linters, formatters, monorepos, unique libraries, etc.).

One potentially counterintuitive thing I’ve learned from working with great engineers is that good/great engineers can make fundamentally different decisions than mediocre engineers. Sometimes, mediocre engineers will judge a great engineer’s decision as a risky or bad decision, because they don’t have the skill to actually drive some of those decisions. For example, improving/rewriting things is commonplace with the right engineering team. Larger rewrites might still be less common, but deciding to improve something should just be part of how you operate. If an engineer wants to improve something in the codebase, they should be able to demonstrate that something can make things better and quickly get feedback from the team on whether to proceed. If it’s a larger change, they should be able to validate it within days/weeks, and not spend months making major changes that people can’t see the viability of. This iteration loop becomes even faster once your team aligns on your engineering principles and why you choose certain tools over others. On the first team I worked with, if someone could demonstrate improvement using a certain tool and got the team bought in, we would start moving to it (quickly weighing the tradeoff between time to migrate and resulting improvement). I would often do the initial slate of work myself so that I could build that muscle of constant, iterative improvement. The worst place to be is on a team where even minor decisions take forever - you’ll have multiple meetings to decide on things that should take all of five minutes for a good team to move forward on with minimal discussion. A constant desire to improve and iterate gets baked into your culture when engineers actually see things get better. Great engineers tend to their garden without being asked and can still make amazing progress on the overall objectives.

In a lot of poor codebases, issues are often caught only by users in production. In any decent codebase, the feedback to the engineer on what they might be doing wrong or need to change is as real-time as possible (i.e. seeing the error in your editor/compiler as your working on it, or being able to visually see changes). As many issues as possible should be solved at this first layer of defense. The next typical layers of defense are tests, then CI checks, then preview environments, then staging, QA, and so on and so forth. The thing is that most small teams don’t have time or the need to set up all of these layers of defense, so if you start from the first layers of defense on day 1, you’ll have a much easier time adding the remaining ones as needed (and have fewer issues to fix). I’ve also found that with poor tooling, it becomes painful to iterate on your product, the system grows increasingly brittle (hard to change and easily broken by any perturbation from the real world, such as users or external services), and user dissatisfaction increases over time.

It turns out when you get these details right, great things happen. It’s a positive domino effect. Great people want to come work with you, which results in more (and higher-quality) work getting done, more progress, more great people wanting to join… and before you know it you’ve increased the odds of success dramatically. Obviously it’s more complicated than this in practice, but those early decisions can completely change the trajectory of your organization. And surprisingly often people make all the opposite (suboptimal) decisions in the early days and forever kill the potential of their product/organization. Something many (especially non-technical) people don’t realize is a lot of those early decisions, when made poorly/sloppily to achieve short-term goals, typically have a negative domino effect. Those technical or hiring decisions rarely have a neutral or minor effect. In those scenarios, you need capable people to come in as soon as possible and stop/reverse the negative domino effect, otherwise you likely doom the potential of your product.

What great engineers want is really the same as most self-actualizing people - an opportunity to do their best work, work with great people, and get better at their craft. Most organizations do not offer this at a level that is sufficient for the best of the best. If you’re a large organization, you might be lucky enough to offer things like great compensation and benefits (which may not be what certain types of people are looking for anyway). It’s not surprising that many of the larger startups and companies suck talent out of the ecosystem by offering them compensation that’s basically impossible for a smaller organization to beat. You have to give great people something that allows them to make the most of themselves and to also eventually enable greatness in others.

Addendum

• Ok this all sounds great, but what if your organization made all these bad decisions to get things started? It’s not too late to fix it if you have the decision-making power. It’s really important that you have the decision-making power to fix it otherwise the antibodies of the existing organization will kick in and probably drive you out for “changing too much”. The longer you wait, you’re unlikely to fundamentally change the DNA of an organization (which is what’s needed to build a great product) and fix all the fundamental issues which only get worse over time. No great engineer wants to be the person left holding the bag. It’s a gargantuan (maybe improbable) effort to “fix” a non-existent product culture once a company has already grown. Assuming your company lasts long enough to actually fix that in the first place. Do not delay on this decision. The longer you wait, you significantly decrease the likelihood of not just success, but surviving altogether. It’s going to be a hard thing to fix with a lot of work required even if you try to correct it fairly early on, but it’s work that needs to be done if you want to significantly increase your likelihood of success (e.g. iterate more quickly, attract good people).

• Most people have never worked with great engineers, and I’ve found this easier to tease out over time in conversation. I can’t tell you the number of times in recent past I’ve heard “oh we just need to get things done, we’ll write better code when we raise our series B, I’ve seen other teams write garbage they fixed later on”. And guess what? To no one’s surprise, all of those teams tend to build low to mediocre quality products and either hire way more people than they need to (who are often mediocre), struggle to hire altogether, or just die - I wonder why this happens? 🙃 The common refrain is “oh we need to focus on getting distribution and growing” - yeah… like every other business ever created? Building a great product is a philosophical choice, something you have to decide matters to your organization. Every successful business needs to grow - it’s not mutually exclusive with building a great product. In fact, great products that solve a real problem often grow faster than ones that don’t. The speed vs. quality tradeoff is a myth on great teams. Engineers can work at a lot of successful companies - a much smaller number can say they truly worked on a great product.