June 3, 2021

What nobody tells you about being on call

Charity Majors is the co-founder and CTO of Honeycomb. Honeycomb brings modern observability tooling to software engineers in an era of distributed systems. She is the co-author of the book Database Reliability Engineering. 

We invited Charity on our productivity and engineering podcast and had a great fireside chat on infrastructure, engineering productivity and in the between.

Her life and background :

Charity Majors made a career out of being the first infrastructure engineer at a startup. She loved that position and the fast pace and adrenaline that it brought. But she would run into unsolvable problems. It was because many people were using the same pool of hardware. Everything was so complicated. And none of the tools helped her solve the problem. The problem was that she didn’t know what she was looking for. 

She and her team finally got some datasets into this tool, which was not user friendly. But the tool lets you slice and dice in near real time on dimensions of very high cardinality. With that tool, they went from taking guesses about what was not working to breaking down the app and being able to precisely figure out what the problem was the time it took them to answer the questions and solve the problem got reduced significantly.

When she left her job, she realized that she didn’t know how to engineer without those tools anymore. And that is why she started the company. 

Charity Majors says that she never wanted to build a company. She never liked the mystique of the founder. There is a whole industry devoted to making it seem as if it was something only special people can do.  But she transformed the anger and fear she had into good fuel. She founded her company out of that fuel. At first she thought they were going to fail. 

What has been exciting in this journey is hearing the customers have the same thing happen to them. The same realization that they needed that tool to engineer. Once they start using it, they can’t go back. They can’t operate without their product anymore. Honeycomb helps you find what you are looking for.

Developers self-report that they spend about forty to fifty percent of their time looking for something. But That does not move the business forward, and that is self-reported so that is gonna be optimistic.

At Veamly, we want people to care about their time the way they care about their health. Value your own time the way you value the time of people sitting next to you.

Challenges she faced :

Charity Majors got thrown into the tech world at a very young age. She was surrounded by men. She struggled with internalized misogyny since the beginning of her career. But she has been overcoming that in the past decades. She also grew up in a very fundamentalist home where women were supposed to have babies and do the dishes. She knew that was not going to be her. She got into operations because she likes emergencies and fires. She also didn’t want to have to plan for the next few months. 

What is interesting is that now everyone’s job is operations. But also everyone in tech is a software engineer, everyone has to write code and everyone has to run it. Many people hear this and think that you have to know everything and do everything. But that is not true. You need to know and understand and operate over the full lifecycle of your code. If you try to chop that up you're not going to have context. That leads to silos. That leaves all these people burned out and frazzled because they're trying to be on call for something that they don't understand.

The reunification project of the engineering profession is a good thing. In exchange for anything to cover more ground roles, you have to know less in depth. The shape of what you have to know is changing. So it does not mean that knowing more is more difficult.

Ownership is something that people shy away from because they associate it with the bad parts. But its intent is a very empowering thing. We all want autonomy and mastery. To have that, we need to have ownership over our work. Although ownership has some downsides, like being responsible when the site goes down. It can also be part of something that enriches your life and makes it more meaningful.

Changes in the work culture :

When it comes to writing a book, Charity Majors says that you need something that triggers you. You need something that you know is going to have an impact on people. You need to believe that the book can change things.

She and her co-author friend come from different backgrounds. They wanted to write this book together to compliment each other. With the rise of site reliability engineering and the merging of disciplines, some of the last to join the party have been the databases. That is because they are a bit more conservative. Because the mistakes they make are more impactful. So what motivated them to write the book was this desire to bring database people into the modern world.

Charity Majors says that our work culture should be outcome oriented. Once you break the link between doing work and being at a certain place at a certain time, you open yourself up to amazing people. It helps create opportunities for people who can’t live in a specific place. She thinks it is the future. It is best if companies start it early.

Charity Majors believes that almost everyone should try management at some point in their career. This will help demystify what management is. But we should encourage people to go back and forth between engineering and management. Going back from management to engineering is not a demotion. Management should also be more to help managers and take something off their plate.

When measuring productivity, Charity Majors says that she prefers to manage the managers of the teams. She believes that when you tell people that you are going to measure their productivity, they change the way they work. She prefers to check people based on the judgement of their managers.

She uses the four Dora metrics for evaluating how well a team performs. Those metrics are good to track how your team is performing. They get right to the heart of what makes you effective.

She has always been more of a writer than a speaker. She has always advocated for written communication. But, for the last few years, she has changed her point of view a little bit. She noticed that there is so much information that is lost when you are reading someone's writing. She thinks that synchronous and asynchronous communication are different. They have different strikes. They both need to exist.

Some final advice :

To help her focus, Charity has different playlists that give her two different types of work. It is some kind of shortcut for her to get into the right headspace. She has been listening to this playlist for these types of work for a long time. 

The good interruption is something you want and you are grateful to have had. It can propel you forward. The bad interruption is a cost. At the same time, you don’t want to shut people down. As a workaholic, she has always felt a lot of joy from working. But she cannot expect  anyone else to be the same. She says that most of us have about four or five hours of concentration time everyday. We can spend more time doing things and sometimes it's valuable, but we can't concentrate more than that. Coming to terms with that was a real eye-opener for her.

Listen to the full episode with our CEO Emna Ghariani on your favorite podcast platform.