Everybody wants to be more innovative, but it’s rare to see someone actually succeed. Despite talk about moving to the cloud, the rise of AI, or whatever digital transformation is, most orgs are still stuck with legacy systems and manual processes. Some try to tackle the problem culturally, telling people to be more agile, bold, and creative. But this rarely achieves much more than everyone getting new t-shirts. It’s not that technology and culture aren’t important. They are. It’s that the root causes for these problems run much deeper. To solve them we first have to understand why innovation, especially in the public sector, is actually really really hard.

First you have to work with people who don’t want anything. When we started building FormSG, we talked to dozens of agencies, ministries, and stat boards. Most of them said the same thing: “Who are you?”, “Why are you here?”, “Please go away.”. The only person who wanted help was someone at NEA whose job it was to collect pigeon complaints. Singapore has a lot of pigeons and Singaporeans like to complain. So of course there was a form for people to complain about pigeons. When he heard we could help automate his job he was very happy.

Then you have to work with people who want too many things. When Parking.sg was getting close to launch the three partner agencies (HDB, LTA, URA) all suddenly had long lists of features they needed from the app. They wanted it to manage season parking, pay fines, support family sharing, and literally a hundred other ideas. At that point we couldn’t even do refunds yet. We only managed to launch because I convinced them to let us do our alpha testing first, and we’d try to get to those features later. Thankfully once it was clear people liked the app, they stopped asking for more.

Perhaps the worst problem is when you actually do launch something, a lot of people get very upset. Our team helped with the new ActiveSG app last year. The old system was very expensive and unmaintainable, and there were some challenges working with the vendor. So we knew we had to replace it. We worked very hard over 3 months to hit the handover date. It wasn’t perfect but we had a solid system. Except if you’re a parent wrangling three kids showing up for swim class and suddenly you’re told to use a new app, it doesn’t matter how good it is, you’re going to be annoyed.

So it is hard. Oddly enough, it is rarely the technology that is difficult. And it’s not enough to tell people to try harder. You have to understand the underlying systems that allow some teams to be innovative while others can only talk about it. There’s a lot to unpack, but for me it comes down to 3 things: outcomes, structure, and iteration.

Innovation needs clear outcomes

A common problem in innovation programs is that we don’t know what we’re innovating for. Are we trying to reduce costs? Improve usability? Save time? Or are we just trying to do something “new”? Without a clear goal your only reference point is what you’re already doing. And your only source of feedback is whether anyone is unhappy about change, which someone always is. And so you get stuck, wanting to innovate but not able to move.

Conversely, when you have a clear goal you can be very flexible about how to get there. In the private sector, it might be profit. In F1, it’s lap time. In AI, it’s quality benchmark scores. Once you know what you’re trying to achieve, you can stop obsessing over how you achieve it. Good metrics tell you what to care about, but also what not to care about. Practically, in the public sector even when a team manages to overcome the bureaucracy, technical challenges, and operations to build something really good and present it to leadership, it often gets shot down with a simple “That’s not how we do things”. It’s not really anyone’s fault. It’s hard to make something happen when your job is to make sure nothing happens.

As an example, AskGov is our Q&A system for the government. Originally, when we did product reviews we would spend a lot of time diving into bug fixes, new buttons, and various design details. While these are all good details for the team to care about, it was not productive for them to walk me through every detail. That changed when we aligned on just two metrics I would care about: Reducing call volume and increasing citizen satisfaction. If calls went down while satisfaction went up, then I did not need to care exactly how the team went about it. On the other hand, if call volume stayed the same and citizens were not happier, then I also did not care what the team had built. Clear outcomes gave the team more freedom to innovate because it gave me a better way to know if it was working.

This also means that you need to adjust your performance management to match. Innovation awards and recognition are great, but the moment reviews come around and people see they’re still being judged on something else, all that motivation disappears. This is especially important because, by default, performance ratings are based on your manager’s impression of you. If you don’t deliberately tie performance to the outcomes, then you’ll get a lot of innovation theatre: Projects that get attention and hype, but ultimately don’t deliver anything. Outcomes are only real when anchored in accountability.

Innovation needs structural space

Traditional orgs are based on a military structure: top down, central decision making, optimized for command and control. They’re designed for a small group of people to make decisions, and for everyone else to align quickly and consistently. This is great when you already know what to do and just need to execute. It’s not so great if you need to figure out what needs to be done.

Just telling people to innovate doesn’t work. You need to build structures that give them the space to do it. Innovation, almost by definition, requires creativity and autonomy. This is the opposite of command and control and so you need to design the org in the opposite direction.

At OGP we designed the team to be a support structure rather than a command structure. The job of management is to provide the space, resources, and tools for the teams to work as effectively as possible, while still holding the team accountable for their outcomes. The team decides what to build, how to build it, and how to prioritize. That’s where innovation happens.

A Rube Goldberg machine is one of those things where a marble knocks over some dominoes, which flips a switch, which pushes a broom, and eventually a light turns on. If you removed even a single domino the whole thing would stop working. With traditional structures, we tend to form human Rube Goldberg machines. The CEO tells the director, who tells the team lead, who tells the procurement manager, who tells the vendor, who tells the project manager, who tells the developer. If any part of that chain changes, the whole company might stall. It’s no surprise then when nobody innovates.

At Singpass we previously had a lot of Rube Goldberg problems. Even minor launches required coordination from at least six different groups and dozens of people. To address this, we restructured to group people around each product with a clear goal, and explicitly mapped out the main ways the different teams depended on each other. For example, the authentication team had the goal “Maximize scale and reliability of logins”, while the customer experience team had “Minimize user frustration.”. Each new team was cross-functional (design, frontend, backend, operations). This meant that the vast majority of the work could happen within each team, only occasionally requiring cross team collaboration.

To be clear, every org structure makes some things easy and other things hard. The key is to make the most common and important things easy, and accept that rare and less important things will be hard. So if you want to be innovative, you need to structure for it deliberately. This means providing support, reducing dependencies, and creating space for creativity and autonomy.

Innovation needs iteration

Even with clear goals and a good structure, innovation fails most of the time. At OGP, we run a month long hackathon at the start of each year. The team explores different ideas, builds quick prototypes, and takes the best ones to launch. We start with around 100 ideas. About 25 make it to prototypes. Fewer than 5 become full products. The other ideas aren’t bad. There are just a lot of reasons they may not work. Maybe the operations were too complex. Maybe the integrations weren’t ready. Maybe the users didn’t think there was a problem in the first place.

This means that even if a very smart leader has an idea, the likelihood that it fails is 95%. Even if a committee thoroughly gathers requirements and debates scope for months, only 1 in 20 will succeed. It’s not that they did their job poorly. These are just probabilities you have to work with.

To get good ideas, you have to start with a lot of ideas, then experiment to find out what can actually work. What makes an idea valuable is not the idea itself, but whether it’s been refined through iteration.

One example is Armory, an app we built for SCDF to help firemen track their equipment. After a year of development and iteration we got something really good rolled out to all the fire stations. By moving off paper, it made equipment checks easier, tracked necessary maintenance, and improved operational readiness. Since the firemen loved it, we thought the paramedics could use it too.

They hated it. Ambulances carry hundreds of small items and medications. Armory forced them to check every item, every shift, even ones they hadn’t used. This added up to over 400 different checks every day. It was slow, frustrating, and not worth it. It’s surprisingly hard to anticipate why an idea might not work. Even good ideas can fail when subtle differences between seemingly similar users turn out to matter more than expected.

So we kept iterating. We cut unnecessary checks. We added item groups, so teams could clear items in bulk. We introduced scheduling, so some items only needed to be checked weekly or monthly. This cut the number of daily checks from over 400 to less than 70. It took lots of feedback and testing, but eventually it became something paramedics actually wanted to use. With so many ways for ideas to fail, iteration is the only way to successfully innovate.

Why we need to be innovative

We’ve talked a lot about how we can innovate, but I think it’s worth closing off by asking why we should? After all, the Singapore system is pretty good. Why bother with innovation and just focus on keeping things running instead?

For me it’s because, while we’re doing relatively well, there are still a lot of people who need help. People who just live with medical problems because getting treatment means taking too much time off work. People who get scammed out of their life savings because it was too hard to protect yourself online. People who could have avoided life changing issues if they only got the right support at the right time just a little bit earlier. If we say “this is good enough” what we’re saying to these people is “this is what you have to live with”.

Innovation isn’t about being cool, or getting attention, or building excitement. It’s about being honest that we still have real problems to solve and hard work to do in solving it. We’re not here to admire how well the system works for most people. We’re here to make sure it works for everyone.

This article was originally written for and published by Civil Service College Singapore

What I ended up doing was talking to anyone in tech who was willing to share their responsibilities and compensation. This gave me a baseline from which I could create competitive roles with similar standards, then adjusted them to our needs. For example, we use rapid prototypes to get buy-in from agencies, so our engineers need more fluency in being full-stack than they do in scaling infrastructure. Similarly, our product managers don’t need to do much marketing, but they need to be very good at navigating stakeholders. It took a while, but to get this right I had to let go of setting standards philosophically and focus instead on the practical implications.

When I set standards too high, it was relatively straightforward to diagnose. People got frustrated with their performance ratings. My best officers left for other roles with greater responsibility and compensation. And good candidates consistently got past our interviews only to reject our offers. This was painful, but at least it was easy to fix.

When I set standards too low, the problem was more insidious. In the short run, everyone was happier. Bigger performance bonuses. Promotions were celebrated. Candidates were happily accepting generous offers. It felt like we had done the right thing.

Where it bit us later was when teams failed to deliver. Quality dropped and deadlines slipped. People lost trust as colleagues dropped the ball. Yet despite these problems, people stuck around and teams got bigger while delivering worse results. I spent a lot of time chasing these issues before identifying the core problem: people were in roles beyond their capability.

This can get very complicated in practice. You might expect too much of junior officers while being too lax with senior ones. Mismatched standards meant I had amazing infrastructure engineers struggling because I really needed webdevs. Because our roles are very different from other orgs, I had to accept that new hires wouldn’t be able to hit the ground running, and we needed to double down on training to get them up to speed. You never get it perfect, but what’s important is you continually adjust as your needs change.

At OGP, our mission is to get government technology working with the same efficiency and innovation that the tech industry is known for. I’ve found that getting standards right has been much more important than any strategic direction or product decision.

Practical steps for how I set roles and levels for OGP

The broad approach is to:

• Decide org structure

• Design roles and levels for it

• Map them to the market

• See where you’re wrong and iterate

Concretely:

1. Figure out what your org is trying to do. OGP’s goal is to experiment with new tech ideas across the public sector. So our strategy is to have lots of independent product teams who are good at rapid prototyping.

2. Figure out your ideal long term org structure. Our basic product team is 1 PM 1 Designer 3 Engineers. A division has 5 teams to allow managers to keep context. Division leadership has functional leads to keep expertise in management. This is a bit “draw the rest of the owl” but it doesn’t have to be perfect.

3. Identify stereotypical roles you have in this structure. Write a job description for each of these roles so you’re clear on what you expect them to do. Actual people will differ from these stereotypes, but they are a useful reference point for expectations.

4. Group similar roles and label them with an approximate seniority needed. In reality you might need someone more senior/junior there depending on project needs, but it serves as a good reference point. Some roles are progressions, where others are different jobs entirely.

5. Go talk to anyone who is in a similar space and is willing to share. Show them what you’ve mapped out and ask them what they think. You will learn a lot about how reasonable your expectations are. Or if you’re even talking to the right industry to begin with.

6. Map competitive roles in other companies. Your roles will never be an exact match. What matters is not whether they are philosophically the same, but practically what you are competing with for the people you need.

7. Try to hire & retain and adjust as you fail. All this was just to get an approximation. As candidates decline offers and people leave for other jobs, ask them if they are okay sharing what the other offer is like. You’ll quickly get a sense of how off you are. Sometimes it’s just a small adjustment to comp to make it competitive. Sometimes you’ll need to rethink your entire structure. For example, with OGP we found that good cross-functional leaders are exceedingly rare, so I’m shifting to teams of leaders from different functions instead.

When I first started building a team, learning about different job titles and career frameworks all felt very arcane. Taking this practical approach really helped me demystify things. There’s no universal “software engineer,” or “senior,” or even “critical skills”. There’s just the org you want, the roles to run that org, and the competitive market for those roles. Everything else is a contrivance to get to that.OGP is looking for experienced engineers and product managers!

They are all working very hard, are very skilled, and are working together extremely well. You’ve looked at every single job and you really can’t remove anyone without the whole team collapsing. Yet costs are high, output is low, and the org as whole is stuck.

You’ve probably seen a Rube Goldberg machine before. It’s those things where a marble knocks over some dominoes that pushes a broom etc etc and long time later the TV is turned on or something. If you removed even a single domino the whole thing would stop working. I think the problem is that we form human Rube Goldberg machines.

Rube Goldberg machines show you can take an arbitrarily simple task and make it arbitrarily complex. And you can do so while every component of the machine remains critical. The CEO tells the director who tells the team lead who tells the procurement manager who tells the vendor who tells the project manager who tells the developer. It happens horizontally as well, the app team gives requirements to the backend team who puts requests to the infrastructure team who has to clear the security team who has to coordinate with the launch calendar. Working between teams is always necessary, but if you’re not careful it can quickly become degenerate.

I don’t think anyone sets out to build human Rube Goldberg machines. Ironically, I think they’re the result of optimization efforts at a geological scale. As organizations grow and shrink, any roles that aren’t on a critical path get cut while jobs attached to critical paths get kept. Overtime, this cycle of deposition and erosion leaves the organizations made up mostly of very long critical chains.

To fix this, you cannot look at things piece by piece. You will correctly conclude that everything is critical but incorrectly conclude that nothing can be done. You have to know what the objectives of the org are, what the actual efficient flows looks like, and do a larger scale restructuring to get there. Practically, I’ve found the most effective way to fix a critical chain is not to try and restructure it live, but instead to build a bypass flow and ramp that up while winding down the inefficient one.

At Open Government Products I try to let the product teams run as independently as possible specifically to minimize Rube Goldberg problems. To be clear, every org structure is going to make some things easy and other things hard. What structure you choose depends on what you’re trying to do as an organization. The trick is to make the common important things easy, and accept clunkiness on things which are uncommon and less important, but that’s something I’ll write more about another time.

If you give resources to Team B you get 2 projects delivered instead of 1. Leadership! But now your weaker team is bigger and whatever skills/communication/structural issues there were are made worse. Repeat this a few times and suddenly most of your organization is doing poorly. Even with the best of intentions, you can see how easy it is for organizations to slide into inefficacy

I think the right response is to give resources to Team A, but also move some of Team B’s responsibilities over. It seems more complicated, but this grows your strongest teams while giving weaker teams a chance to focus and perform on a smaller scope.

In the real world it’s rarely so clear though. Instead Team A will be handling things so smoothly that you don’t even think about them. While Team B will always be meeting with you and explaining how their challenges are so big it’s impossible without more resources. Seeing the earnest efforts and plight of Team B, you go take some resources from a quiet team that doesn’t seem to be doing much, and pass them to Team B.

To be clear, they’re not lying. To a lower skilled team with structural issues it really does seem impossible without more resources. Just as I would find it impossible to run a marathon, they’re being completely honest when they report problems and say more resources are needed.

This is why I think leaders need domain expertise. Even with completely honest feedback, if you just address the problems as reported you end up growing your lowest performing teams. You need to be able to judge which problems are actually hard, empower your best performers while helping your low performers focus and succeed.

I think this is probably the much harder problem than choosing which projects to pursue. Get it wrong and your hardest working teams get frustrated you’re not supporting them. Even if you get it right, people are likely still going to be frustrated with you. But you have to do it otherwise you’ll face a surreptitious decline of the organization.

Bad software is one of the few things in the world you cannot solve with money. Billion dollar airlines have flight search apps that are often inferior to those built by groups of students. Established taxi companies the world over have terrible booking apps despite the threat they face from ride-sharing services. And painful corporate IT systems are usually projects with massive budgets, built over the course of many years. Whatever the cause of bad software is, it does not seem to be a lack of funding.

Surprisingly, the root cause of bad software has less to do with specific engineering choices, and more to do with how development projects are managed. The worst software projects often proceed in a very particular way: The project owners start out wanting to build a specific solution and never explicitly identify the problem they are trying to solve. They then gather a long list of requirements from a large group of stakeholders. This list is then handed off to a correspondingly large external development team, who get to work building this highly customised piece of software from scratch. Once all the requirements are met, everyone celebrates as the system is launched and the project is declared complete.

However, though the system technically meets specifications, severe issues are found when it is put in the hands of actual users. It is slow, confusing, and filled with subtle bugs that make using it an exercise in frustration. Unfortunately, by this time the external development team has been dismissed and there are no resources left over to make the necessary fixes. By the time a new project can be initiated years later, all knowledge of what caused these problems has left the organisation and the cycle starts over again.

The right coding language, system architecture, or interface design will vary wildly from project to project. But there are characteristics particular to software that consistently cause traditional management practices to fail, while allowing small startups to succeed with a shoestring budget: • Reusing good software is easy; it is what allows you to build good things quickly; • Software is limited not by the amount of resources put into building it, but by how complex it can get before it breaks down; and • The main value in software is not the code produced, but the knowledge accumulated by the people who produced it.

Understanding these characteristics may not guarantee good outcomes, but it does help clarify why so many projects produce bad outcomes. Furthermore, these lead to some core operating principles that can dramatically improve the chances of success:

1. Start as simple as possible;

2. Seek out problems and iterate; and

3. Hire the best engineers you can.

While there are many subtler factors to consider, these principles form a foundation that lets you get started building good software.

Reusing Software Lets You Build Good Things Quickly

Software is easy to copy. At a mechanical level, lines of code can literally be copied and pasted onto another computer. More generally, the internet is full of tutorials on how to build different kinds of systems using ready-made code modules that are available online. Modern software is almost never developed from scratch. Even the most innovative applications are built using existing software that has been combined and modified to achieve a new result.

The biggest source of reusable code modules is the open source community. Open source software is software in which code is freely published for anyone to see and use. Many of the largest contributors to the open source community are giant tech companies. If you want to use a state-of-the-art planet scalable database as Facebook does, just download the code for Cassandra that they open sourced in 2008. If you want to try out Google’s cutting-edge machine learning for yourself, download the TensorFlow system published in 2015. Using open source code does not just make your application development faster, it gives you access to technology that is far more sophisticated than anything you could have developed yourself. For the most popular open source code, it is even more secure as there are many more people paying attention and fixing vulnerabilities. This is the reason digital technology has made such rapid progress: even the newest engineers can build upon the most advanced tools our profession has to offer.

The advent of cloud services has taken reusability even further, offering the full use of even proprietary systems for just a subscription fee. Need a simple website? Just configure one in a few clicks using a website building service like Squarespace or Wix. A database? Subscribe to a virtual one from Amazon Web Services or Microsoft Azure. Cloud services allow developers to benefit from specialisation; the service provider handles the setup, maintenance, and continued development of a reliable, high-quality piece of software that is used by all its subscribers. This allows software developers to stop wasting time on solved problems and instead focus on delivering actual value.

You cannot make technological progress if all your time is spent on rebuilding existing technology. Software engineering is about building automated systems, and one of the first things that gets automated away is routine software engineering work. The point is to understand what the right systems to reuse are, how to customise them to fit your unique requirements, and fixing novel problems discovered along the way.

Software Is Limited by Complexity

How useful a piece of software can be is usually limited by its complexity rather than the amount of resources invested in building it.

IT systems are often full of features but are still hated by users because of how confusing they become. In contrast, highly ranked mobile apps tend to be lauded for their simplicity and intuitiveness. Learning to use software is hard. Beyond a point, new features actually make things worse for users because the accumulated complexity starts to become overwhelming. For example, after serving as the hub of Apple’s media ecosystem for almost 20 years, iTunes was split into three different apps (for music, podcasts, and TV shows) this year, as its features had grown too complex for one app to handle. From a usability perspective, the limit is not how many features can be implemented, but rather what can fit into a simple intuitive interface.

Even ignoring usability, engineering progress slows to a halt once a project becomes too complex. Each new line of code added to an application has a chance of interacting with every other line. The bigger an application’s codebase, the more bugs are introduced whenever a new feature is built. Eventually, the rate of work created from new bugs cancels out the rate of work done from feature development. This is known as “technical debt” and is the main challenge in professional software development. It is the reason why many large IT systems have issues that go unfixed for years. Adding more engineers to the project just adds to the chaos: they start running faster in place as the codebase keels over from its own weight.

In such cases, the only way forward is to take a step back to rationalise and simplify the codebase. The system architecture can be redesigned to limit unexpected interactions. Non-critical features can be removed even if they have already been built. Automated tools can be deployed to check for bugs and badly written code. Bill Gates once said “Measuring programming progress by lines of code is like measuring aircraft building progress by weight”. Human minds can only handle a finite amount of complexity, so how sophisticated a software system can get depends on how efficiently this complexity budget is used.

Building good software involves alternating cycles of expanding and reducing complexity. As new features are developed, disorder naturally accumulates in the system. When this messiness starts to cause problems, progress is suspended to spend time cleaning up. This two-step process is necessary because there is no such thing as platonically good engineering: it depends on your needs and the practical problems you encounter. Even a simple user interface such as Google’s search bar contains a massive amount of complexity under the surface that cannot be perfected in a single iteration. The challenge is managing this cycle, letting it get messy enough to make meaningful progress, but not letting it get so complicated that it becomes overwhelming.

Software Is about Developing Knowledge More than Writing Code

In software development, most ideas are bad; this is not anyone’s fault. It is just that the number of possible ideas is so large that any particular idea is probably not going to work, even if it was chosen very carefully and intelligently. To make progress, you need to start with a bunch of bad ideas, discard the worst, and evolve the most promising ones. Apple, a paragon of visionary design, goes through dozens of prototypes before landing on a final product. The final product may be deceptively simple; it is the intricate knowledge of why this particular solution was chosen over its alternatives that allows it to be good.

This knowledge continues to be important even after the product is built. If a new team takes over the code for an unfamiliar piece of software, the software will soon start to degrade. Operating systems will update, business requirements will change, and security problems will be discovered that need to be fixed. Handling these subtle errors is often harder than building the software in the first place, since it requires intimate knowledge of the system’s architecture and design principles.

In the short term, an unfamiliar development team can address these problems with stopgap fixes. Over time though, new bugs accumulate due to the makeshift nature of the additional code. User interfaces become confusing due to mismatched design paradigms, and system complexity increases as a whole. Software should be treated not as a static product, but as a living manifestation of the development team’s collective understanding.

This is why relying on external vendors for your core software development is difficult. You may get a running system and its code, but the invaluable knowledge of how it is built and what design choices were made leaves your organisation. This is also why handing a system over to new vendors for “maintenance” often causes problems. Even if the system is very well documented, some knowledge is lost every time a new team takes over. Over the years, the system becomes a patchwork of code from many different authors. It becomes harder and harder to keep running; eventually, there is no one left who truly understands how it works.

For your software to keep working well in the long term, it is important to have your staff learning alongside the external help to retain critical engineering knowledge in your organisation.

3 Principles for Good Software Development

Start as Simple as Possible

Projects that set out to be a “one-stop shop” for a particular domain are often doomed. The reasoning seems sensible enough: What better way to ensure your app solves people’s problems than by having it address as many as possible? After all, this works for physical stores such as supermarkets. The difference is that while it is relatively easy to add a new item for sale once a physical store is set up, an app with twice as many features is more than twice as hard to build and much harder to use.

Building good software requires focus: starting with the simplest solution that could solve the problem. A well-made but simplistic app never has problems adding necessary features. But a big IT system that does a lot of things poorly is usually impossible to simplify and fix. Even successful “do it all” apps like WeChat, Grab, and Facebook started out with very specific functionality and only expanded after they had secured their place. Software projects rarely fail because they are too small; they fail because they get too big.

Unfortunately, keeping a project focused is very hard in practice: just gathering the requirements from all stakeholders already creates a huge list of features.

One way to manage this bloat is by using a priority list. Requirements are all still gathered, but each are tagged according to whether they are absolutely critical features, high-value additions, or nice-to-haves. This creates a much lower-tension planning process because features no longer need to be explicitly excluded. Stakeholders can then more sanely discuss which features are the most important, without worrying about something being left out of the project. This approach also makes explicit the trade-offs of having more features. Stakeholders who want to increase the priority for a feature have to also consider what features they are willing to deprioritise. Teams can start on the most critical objectives, working their way down the list as time and resources allow.

We followed a similar process for all our most successful apps. Form.gov.sg started out as a manual Outlook Macro that took us six hours to set up for our first user but today has processed about a million public submissions. Data.gov.sg started out as a direct copy of an open source project and has since grown to over 300,000 monthly visits. Parking.sg had a massive list of almost 200 possible features that we never got around to building but still has over 1.1 million users today. These systems are well received not in spite of their simplicity but because of it.

Seek Out Problems and Iterate

In truth, modern software is so complicated and changes so rapidly that no amount of planning will eliminate all shortcomings. Like writing a good paper, awkward early drafts are necessary to get a feel of what the final paper should be. To build good software, you need to first build bad software, then actively seek out problems to improve on your solution.

This starts with something as simple as talking to the actual people you are trying to help. The goal is to understand the root problem you want to solve and avoid jumping to a solution based just on preconceived biases. When we first started on Parking.sg, our hypothesis was that enforcement officers found it frustrating to have to keep doing the mental calculations regarding paper coupons. However, after spending just one afternoon with an experienced officer, we discovered that doing these calculations was actually quite simple for someone doing it professionally. That single conversation saved us months of potentially wasted effort and let us refocus our project on helping drivers instead.

Beware of bureaucratic goals masquerading as problem statements. “Drivers feel frustrated when dealing with parking coupons” is a problem. “We need to build an app for drivers as part of our Ministry Family Digitisation Plans” is not. “Users are annoyed at how hard it is to find information on government websites” is a problem. “As part of the Digital Government Blueprint, we need to rebuild our websites to conform to the new design service standards” is not. If our end goal is to make citizens’ lives better, we need to explicitly acknowledge the things that are making their lives worse.

Having a clear problem statement lets you experimentally test the viability of different solutions that are too hard to determine theoretically. Talking to a chatbot may not be any easier than navigating a website, and users may not want to install yet another app on their phones no matter how secure it makes the country. With software, apparently obvious solutions often have fatal flaws that do not show up until they are put to use. The aim is not yet to build the final product, but to first identify these problems as quickly and as cheaply as possible. Non-functional mock-ups to test interface designs. Semi-functional mock-ups to try different features. Prototype code, written hastily, could help garner feedback more quickly. Anything created at this stage should be treated as disposable. The desired output of this process is not the code written, but a clearer understanding of what the right thing to build is.

With a good understanding of the right solution, you can start work on building the actual product. You stop exploring new ideas and narrow down to identifying problems with your particular implementation. Begin with a small number of testers who will quickly spot the obvious bugs that need to be fixed. As problems are addressed, you can increasingly open up to a larger pool who will find more esoteric issues.

Most people only give feedback once. If you start by launching to a large audience, everyone will give you the same obvious feedback and you’ll have nowhere to go from there. Even the best product ideas built by the best engineers will start out with significant issues. The aim is to repeatedly refine the output, sanding down rough edges until a good product emerges. Even after all this iteration, after launch is when problems with a product matter the most. A problem that happens only 0.1% of the time may not get noticed during testing. But once you have a million users, every day the problem goes unresolved is a thousand more angry people you have to deal with. You need to fix problems caused by new mobile devices, network outages, or security attacks before they cause substantial harm to your users. With Parking.sg we built a series of secondary systems that continuously check the main system for any discrepancies in payments, duplicate parking sessions, and application crashes. Building up an “immune system” over time lets you avoid being overwhelmed as new issues inevitably come up.

Overall, the approach is to use these different feedback loops to efficiently identify problems. Small feedback loops allow for quick and easy correction but miss out on broader issues. Large feedback loops catch broader issues but are slow and expensive. You want to use both, resolving as much as possible with tight loops while still having wide loops to catch unexpected errors. Building software is not about avoiding failure; it is about strategically failing as fast as possible to get the information you need to build something good.

Hire the Best Engineers You Can

The key to having good engineering is having good engineers. Google, Facebook, Amazon, Netflix, and Microsoft all run a dizzying number of the largest technology systems in the world, yet, they famously have some of the most selective interview processes while still competing fiercely to recruit the strongest candidates. There is a reason that the salaries for even fresh graduates have gone up so much as these companies have grown, and it is not because they enjoy giving away money.

Both Steve Jobs and Mark Zuckerberg have said that the best engineers are at least 10 times more productive than an average engineer. This is not because good engineers write code 10 times faster. It is because they make better decisions that save 10 times the work.

A good engineer has a better grasp of existing software they can reuse, thus minimising the parts of the system they have to build from scratch. They have a better grasp of engineering tools, automating away most of the routine aspects of their own job. Automation also means freeing up humans to work on solving unexpected errors, which the best engineers are disproportionately better at. Good engineers themselves design systems that are more robust and easier to understand by others. This has a multiplier effect, letting their colleagues build upon their work much more quickly and reliably. Overall, good engineers are so much more effective not because they produce a lot more code, but because the decisions they make save you from work you did not know could be avoided.

This also means that small teams of the best engineers can often build things faster than even very large teams of average engineers. They make good use of available open source code and sophisticated cloud services, and offload mundane tasks onto automated testing and other tools, so they can focus on the creative problem-solving aspects of the job. They rapidly test different ideas with users by prioritising key features and cutting out unimportant work. This is the central thesis of the classic book “The Mythical Man-Month”1: in general, adding more software engineers does not make a project go faster, it only makes it grow bigger.

Smaller teams of good engineers will also create fewer bugs and security problems than larger teams of average engineers. Similar to writing an essay, the more authors there are, the more coding styles, assumptions, and quirks there are to reconcile in the final composite product, exposing a greater surface area for potential issues to arise. In contrast, a system built by a smaller team of good engineers will be more concise, coherent, and better understood by its creators. You cannot have security without simplicity, and simplicity is rarely the result of large-scale collaborations.

The more collaborative an engineering effort, the better the engineers need to be. Problems in an engineer’s code affect not just his work but that of his colleagues as well. In large projects, bad engineers end up creating more work for one another, as errors and poor design choices snowball to create massive issues. Big projects need to be built on solid reliable code modules in an efficient design with very clear assumptions laid out. The better your engineers, the bigger your system can get before it collapses under its own weight. This is why the most successful tech companies insist on the best talent despite their massive size. The hard limit to system complexity is not the quantity of engineering effort, but its quality.

Conclusion

Good software development starts with building a clear understanding of the problem you want to solve. This lets you test many possible solutions and converge on a good approach. Development is accelerated by reusing the right open source code and cloud services, granting immediate access to established software systems and sophisticated new technology. The development cycle alternates between exploration and consolidation, quickly and messily progressing on new ideas, then focusing and simplifying to keep the complexity manageable. As the project moves forward, it gets tested with successively larger groups of people to eliminate increasingly uncommon problems. Launching is when the real work ramps up for a good development team: layers of automated systems should be built to handle issues quickly and prevent harm to actual users. Ultimately, while there are infinite intricacies to software development, understanding this process provides a basis to tackle the complexities of how to build good software.

This article was originally written for and published by Civil Service College Singapore