The biggest mess in my professional career
Those Who Do Not Learn History Are Doomed To Repeat It.
Whenever I'm interviewing candidates for a role in my team, one of my favourite interview questions is the following:
What has been your biggest career mistake? The major mess? The massive fckup?*
I love this question for the reasons I explained in another article.
Unfortunately I have had few opportunities to answer the same question in an interview myself. Either the question is not really relevant; or more people should read my newsletter!
Mistakes are a great way to learn, and today I want to share with you the story of the biggest mess that I’ve been responsible for.
It all started with good intentions, but the results were quite dramatic.
The reason why I failed is that I violated Rule 0 of any major software project:
Never bundle a major technical refactoring and a complete product revamp in a single big bang launch.
What's sneaky about this rule is that you can get away with violating it a few times, mostly out of luck.
But then one day you en up failing spectacularly.
Like many of us, I've been lucky a few times, until I wasn't any longer.
Today's article is the first of a mini-series on what happened, how it was handled and the biggest lessons I've drawn from it.
💪🏼 Go big or go home!
The story I'm about to tell you took place during the best part of 2016.
Back then I was the CTO for an online marketplace in Mexico. It was my first experience as CTO, and I had to go through the process of learning the job the hard way.
The business was on a decent growth trajectory, but we were limited by some internal factors:
We were operating a quite antiquated platform that could have benefitted from a more modern approach. This was especially true on the frontend side, where we used a combination of PHP and a custom built templating language.
The general UX of the site had a lot of gaps or unintuitive features, which also had a lot to do with the limitations in the frontend stack.
The brand had been around for more than two decades - originally as a printed classified newspaper - and was struggling to attract a younger audience. It was getting “old” from a marketing perspective.
Until then, the engineering team had been experimenting with a new frontend stack that had been deployed on a couple of new and low traffic pages. Our original idea was to progressively and steadily migrate all the existing pages to the new stack, one by one.
That initiative though was progressing slowly as it was competing with a gazillion of other priorities on the roadmap.
The external pressure on the market was mounting, and shareholders were becoming increasingly impatient. Wed had to do something to accelerate our growth trajectory.
This is when we decided to launch a big initiative with the intention of addressing the 3 main issues mentioned above, all at once. We assumed we could kill 2 - actually 3 - birds with one stone, and decided to rally the troops with the following goals:
A complete rebuild of the web stack
A major overhaul of the UI and main user flows
A new brand, colour scheme and tone of voice
Funnily enough we decided to call this project Phoenix, a reference to the mythological beast that would set itself on fire, and from the ashes a newborn Phoenix would rise.
Attentive readers might have gotten the irony of the project name.
For the unaware I'd just like to remind you that The Phoenix Project is the title of a seminal book published in 2013. This books tells the story of an e-commerce company struggling with their technical platform, and going through a massive initiative of rebuilding it called The Phoenix Project.
The parallels between the fictional story and what we went through are numerous. If you haven't read the book, I recommend you go and do it. That's what I did, just a couple of years too late.
Don't be me, don't be late!
🎯 What the web stack rebuild entailed
It's important to note that the team I was responsible for had inherited the stack and codebase from the group owning our platform and business.
The pattern used until then was that each new marketplace launched in a new country would inherit a copy of the original codebase, and then evolve it independently based on local needs.
The reason why this is important is the following: though the team was very familiar with the codebase, they had not created it from scratch.
As often happens in these cases, we ended up taking many things from granted, and underestimated the importance of some optimisations or design choices as we never took part to those.
This is a common pattern in software engineering projects: underestimating the inherent complexity of something built by someone else, and naïvely assume you can build something that solves the same problem, but better and in a relatively short amount of time.
I call it the innate hubris of software engineers.
When managed properly it can lead to great results. When mismanaged, it leads to massive frustration and failure.
What we set out to achieve as part of the Phoenix project on the engineering side was a combination of the following:
✅ A complete rebuild of the Frontend stack
We were planning to abandon the original stack built on PHP, vanilla JS with some JQuery and the custom built templating system.
One thing to note here is that many developers were complaining about limitations to their productivity due to the template system which used a somewhat unintuitive syntax.
This language was transpiled into C, and then built into a binary as part of our build system.
The thing was slow to develop, but extremely fast to run in production. We deliberately decided it was time to focus more on engineering productivity rather than pure runtime performance.
As part of the previous experiments, we had settled on using VueJS as the foundation for the new stack.
✅ Move away from having 2 website codebases
Like many web platforms in the early 2010s, we had built 2 versions of the website. One for the traditional “Desktop” access, and one for Mobile access, commonly dubbed as m-site.
By 2016 though the responsive approach had become a standard that offered both better user experience and increased engineering productivity.
For these reasons we planned to get rid of the old m-site and implement a single responsible website with the new technological stack.
✅ Expand and evolve the mobile API layer
In addition to our 2 web platforms, we also had native apps for iOS and Android. These leveraged an API layer that would abstract away the underlying legacy platform and allow the native apps to operate seamlessly.
The API was another area where a lot of code duplication was happening. Whenever we wanted to introduce a new feature in our platform we had to implement it in the 2 web stacks mentioned above, as well as in the API layer to make the feature available to our mobile apps.
As we wanted to move towards client side rendering on the web platform, we decided to leverage the existing API layer to this purpose. This way all our apps, be it web or native, would be using the same API, significantly reducing the amount of engineering work require to introduce new changes.
The main goal was ultimately to increase Engineering Productivity and happiness.
Given the breadth of changes required, this would have represented in an of itself an already ambitious target to achieve. Rebuilding a tech stack is a journey full of unknowns, and things always end up being significantly more complicated than the original estimates.
Not to mention that we had to still maintain and make minor improvements in the existing platform while doing all of the above.
We could have limited our efforts to this, but instead we bundled this already ambitious set of goals with a revamp of both the brand and the product.
This is how we though we would set ourselves up for success. Did we?
➡️ In the next episode…
In the next article of this series I'll tell you the story of how the project actually unfolded, and what have been the key learnings I've taken with me in the next steps of my career.
I'm planning to publish it within a week of this post going out, so stay tuned!
Do you think you know what happened next?
Have you gone through similar experiences in your career?
Please share your thoughts in the comments section below while waiting for the next episode to be published!
I've become quite familiar with this experience, and I view it as one of the significant mistakes in my career. Thanks for sharing Sergio!