How I managed a product that sent 1 Billion Messages

We sent 1 Billion Messages in the last 9 months of the public launch of our product(Truepush.com), The next 1Bn is going to come in 2 – 2.5 months and I believe this growth will have the same curve. We got incredible feedback on product design, product and customer support over the last few months.

While everyone in our team did a great job. I have seen the whole team doing a lot of mistakes, owning them and learning from them. I have my share. There are a lot of things that worked great. But I am sharing my experiences from my Atomic Failures. This is one of the most honest articles I can put about myself and my work especially because I am writing about what failed and how we fixed than boosting about what worked and giving a how-to guide. If you are managing or owning a product, I hope you can relate to it.

I learned more about our customer when we shipped the product

I did customer development before starting to build the product. I spoke to people I know who are using similar services. I spoke to potential clients, etc. But that’s only a good start. Most of the RoadMap I wrote Initially is only 50% done. The remaining 50% is still relevant(I feel) but the reason it isn’t touched is that our customers taught us what is more important to them while they used our product. This feedback is far more relevant than the ones we got from customer development. I put that 50% aside and went with the customer’s needs(read it as listening to the market).

Learning to prioritize

There are tons of requests that come from customers. The outreach/Growth team tries their best to accommodate the maximum number of clients. From a product point of view, I thought — okay, if we build this feature, another major customer is going to add to our list.

It worked until it broke. For a couple of months, we broke the Product company DNA and we became a service company. We had so much heat between the two teams that we had to take a big pause and all the leadership team had to sit and find out what is going on.

As a Product Manager in our team, It was my responsibility to set expectations and had to learn it the hard way. We have a simple rule now. Any live issues found will be fixed immediately/asap. Any feature requests have to wait until the next cycle. And at every new cycle, we sit with the relevant stakeholders and prioritize.

Communicating clearly

When I got a new team, I know my intention of saying anything but in the Initial months, there is a communication and expectation gap. I feel when you get into a new team, the first thing to build is trust, the next thing is to convey your intentions clearly(over communication helps here). It initially looked like I was pushing them on unrealistic deadlines(which is true) and did not care if they can manage it(which is not true). I had to learn how they work, think and how they manage/prioritize the work.

I had to over-communicate my Intentions from then on so much that my team in one of the meetings told me to stop repeating myself and that they got what I mean whatever I mean along with my Intention. It was worth over-communicating.

Trusting myself, all the time, every time

It’s easy to lose myself with self-doubt, especially when I am learning many things on the job with a scaling product, team and everyone’s learning levels. There were few silly screw-ups and few bad screw-ups. They happened because either I didn’t know if I should be looking at them or they happen because I overlooked them and decided to deal it with later.

These kinds of things are not fine. But at the end of the day, I realize, I am imperfect and have to deal with it. I try to learn, analyze and change. Few are ingrained as a personality and need some rewiring of your brain which comes with practice over time. When I screw up or things go bad. I try now try to fix it as soon as possible. Also, see what to do for it to not happen again.

Trusting the team but asking questions in detail (Details Matter)

Initially, I was getting into so many details with our tech team but at one point I realized I should not be managing them with too many details. Then I stopped being so deep and only started looking at deadlines.

Something happened. A lot of things that looked minor appeared to be annoying and they weren’t minor as one would think. For example, once we experienced a low view % of our daily average for our notifications. I asked my tech team to check and everything was perfect. Then we checked with the server team and everything seems to work fine too. We thought it was from the sudden surge of increase in push notifications(it happened to us once in the past and I thought it was a pattern but this time, it wasn’t).

The view % of notification started going down. At surface level, everything worked but when dug further, the server team found that the issue was at OS level (file descriptors wasn’t supporting the scale of incoming requests). Sure, that’s something that never happens in most cases but happened in our case. But it was my job to ask questions like — what are the dependencies other than what we check? how to measure them, etc. Not that any of the members would have guessed the issue but at least, that may have brought down the resolution time a little earlier.

I realized that asking a lot of questions and with details will help. It will help the person you are working with to think deeper. The more specific questions and scenarios you pose the more they think.

Many times, they may not know the answer but at least now they can go back and work on it.

Fix as soon as you know what got screwed up

In the Initial months, our tech team would try to complete the deadlines. If we get any Issues from clients, I would report/convey it to my team and ask them to work on it. But it used to take a day or two generally to complete while our clients heat us up with chats and emails. It wasn’t so damaging when we hardly had few clients. But as we grew, stakes started to grow up.

The issue for one customer would be the same Issue for many customers. It was tough for us to make a transition for an immediate reaction to live issues. But then now we try to fix the issues Immediately/asap. It not only helps one customer but also other customers and future customers. Every time we fixed something, our product quality is upgraded.

Upgrading my skill

Being a Product Manager is being a generalist. Meaning, as we grow it’s my job to learn about scaling, being updated about the market, learning to work with growing clients, growing team, growing expectations of growth itself, growing vision, growing product. Every stage is different. This is one thing that runs in the background all the time(mostly on the job).

Plan for scale

We Initially did not optimize for image requests(optimizing size, etc,.) and a lot of other things. When we hardly had few clients, it wasn’t worth it to work on these details. What the heck right? Our system seems to handle growth for the next few months. It seemed logical to roll out more features and attract more customers or build features for customers and try to impress them so they would start using us.

We didn’t even see any issues at a 1Mn/day scale which we felt was huge at that time(we were hardly seeing a few thousand messages sent every day in the first two months). Everything worked well until it did not. After a certain scale, we started seeing real scale issues. Now we are trying to be 10X capable of the expected scale. Now we are building an easily scalable product for future growth and scope.

All this is on the client-side for handling scale. With growth comes a need to understand our customer behavior better. One of the things I am learning to do it to help our outreach/growth team to give correct data/analytics about the user behavior so they can stop doing a lot of things manually + get great insights so they can be on their feet to get much better growth. With a limited team, I am learning to balance to build for scale both Internally(our customer analytics to growth/outreach team) and Externally(for clients).

I seem to have a 3-year road map(direction)

What Initially we thought will take 6 months, now looks like 3 years for two reasons 1) we accounted only for features but did not count the issues we would find, requirements of customers we would build and amount of research and time that is needed to build the product for scale 2) Now the scope of the tool has expended much beyond our Initial planning a much bigger vision. Actually, I only have 2 years’ direction in mind but let’s add another year anyways.

No, this 2 – 3-year road map is not a road map on the paper but in my head. Based on my previous experience. I put down the features/new tools and layers that would come in the coming months and came up with an estimate in my head. The direction for this stays in my head and other leadership team members.

I know nothing

(Did you ever watch GoT? 🙂 ) As Aristotle said, ‘the more you know, the less you know’. The more I am trying to learn. The less I feel I know. As Charlie Munger said more long term advantage people tend to be less stupid for as long as possible. Though he might have said in the context of Investments. The more I learn on Product Building, I feel the less I know. One of the main things I try to do is be less stupid by asking a lot of questions and building processes so we can avoid the same mistakes and hopefully guess the new break downs and be on a lookout for it.

If you are building a product. Remember Murphey’s Law — ‘Everything that can go wrong will go wrong’. Think hard on most scenarios, build dashboards and processes to monitor what’s going right and to analyze and anticipate if anything goes wrong.

And that’s how I ‘Managed’ a product that sent 1Bn Messages.

Our Team

This is our current team while we hit our 1st 1Bn Messages

This article originally published here: https://medium.com/@manojsurya3110/how-i-managed-a-product-that-sent-1bn-messages-8b1382cd03f2

Comments

comments