Brainiac Corner with Twitter's Senior MySQL Database Administrator

by Kyle Redinger on 13 May 2013

At the Brainiac Corner, we meet with some of the sharpest minds in the system, database, devops, and IT world. If you'd like to share your thoughts on pirates, ninjas, the future of system administration, or any other relevant topic, please don't hesitate to contact us.

headshot

At Percona Live, we asked Will Gunty, Senior MySQL Database Administrator at Twitter a few questions. Here's what we asked, and here's what he answered:

How did you get from stork to brainiac (i.e. what do you do today, and how did you get there)?

I've always had an aptitude for technical stuff. I was doing “web design” back in Junior high in the mid 90′s using Claris Homepage (okay, so WYSIWYG mostly). It wasn't really until college when I started working for a large hosting company that I really started to get into the systems side of things. That was about 8 years ago. It was only about 2 years ago while working at Box that I started to focus on the database/MySQL side of things. This focus started out as a necessity, due to the extremely small size of the TechOps team at the time, but really became a huge passion of mine.

At Box, I learned not only the ins and outs of running a database, but how to do so at scale and how to grow things quickly enough to stay ahead of aggressive growth instead of simply trying to keep up with it. This, of course, led to where I am now, working on the Sharded Database team at Twitter, running some of the largest MySQL Database deployments in the world.

What's in your group's technology stack?

The core technology in the @TwitterDBA stack itself is, of course, our own build of MySQL. Our sharded clusters utilize gizzard and flockdb to handle sharding and snowflake for generation of some id's. Most of these are written in scala, java, and/or ruby.

On the management side, we are heavy users of puppet for configuration management. A custom implementation of fact like attributes allows us to fully configure and bootstrap MySQL hosts without any human interaction. For monitoring, we use a combination of Nagios and a Twitter specific TSDB implementation. Most of our management tools are written in Ruby.

Who would win in a fight between ninjas and pirates, and why?

Ninjas. Their superior stealth would allow them to easily infiltrate any pirate ship and take them down from the inside out. Not to mention that peg legs are rather vunlerable to being swept.

What's a more accurate state of the world: #monitoringsucks or #monitoringlove?

I believe I'd have to say #MonitoringSucks, but #VisibilityLove. Hard threshold monitoring is often painful. For example, one of the alerts that we have is for MySQL uptime that goes off when the uptime is < 300 seconds. It would be much better to have alerting based on the cause of that. Was the host manually restarted, did mysql crash, how many times has it crashed? There are often places with that specific alert that don't really matter.

As for #VisibilityLove, though, I feel that you can never have too much visibility into your systems. The more information you can get the better. Obviously the limit there is the impact that polling has on the systems. Using things like OpenTSDB, statsd, or some other kind of time series store, measure all you want! If you are already polling global status for one variable, why not track them all. There are so many times that I've experienced where something that we never thought would be useful actually has become critical in diagnosing an issue.

In six words or fewer, what's the biggest challenge your organization faces?

Getting the most from MySQL

What's the best piece of advice you've ever received?

I'm going to list two. The first I was told when I was at the O'Reilly MySQL conference in 2011, when I was just starting to really get into focusing on MySQL. “If you want to be a DBA, call yourself one and the rest will follow.” This really pushed me to grow and be the DBA that I wanted to instead of just a SysAdmin who happened to know a bit about databases.

The second is actually a quotation from Christopher Robin to Pooh, “Promise me you'll always remember you're braver than you believe, and stronger than you seem, and smarter than you think.” Anytime I'm having a rough time or I know I need to push myself, I think of this and it helps me drive myself to do something that I didn't think was possible.

What principles guide your expertise in your given domain?

  1. Take the time to do it right. If you have the time, do things the right way the first time around. Doing something halfway only means that you will need to come back and spend more time doing it again later.
  2. Automate where you can. If you can automate something, do it. It WILL save you time down the road.
  3. Avoid band-aid fixes. Often in a rush situation, things are put together as a fix with bubble gum and a couple of toothpicks. Frequently, those fixes, while meant to be temporary, become permanent. This goes hand in hand with “Take the time to do it right”.
  4. Always be learning! The more you learn, the better you can be.
  5. Think for the future. If you are thinking only about the short term, you will always be in reactive to growth and issues. If you are able to focus months, or even years in advance you will be proactive and be able to anticipate both growth and issues before they come up.

What is your vision for system administration in 5 years?

Open source automation. One thing that I am faced with extremely frequently is that each MySQL shop has their own suite of tools and management methods that they use. Dozens of different solutions to the same problem. Twitter is a huge member of the open source community. Percona has done a great job with their Percona Toolkit. I would love to see other companies open sourcing their tools and information. This would be especially helpful to smaller shops.

One thing that has really been enabling a number of DBA's to start sharing their knowledge in this manner is DBHangops. Every other week, a group of DBA's from Twitter, Box, Yelp, and a number of other companies run a google Hangout to talk about their issues and experiences. The goal of these is to share our knowledge with others. Anyone is welcome!

comments powered by Disqus