At the Brainiac Corner, we meet with some of the sharpest minds in the system, database, devops, and IT world. If you'd like to share your thoughts on pirates, ninjas, the future of system administration, or any other relevant topic, please don't hesitate to [contact us*]1. *
We slipped away from last week's Percona Live MySQL Conference & Expo to talk to Harrison Fisk, MySQL luminary and Performance Engineer at Facebook. Here's what we asked, and here's what he answered:
How did you get from stork to brainiac (i.e. what do you do today, and how did you get there)?
I'm from Walton, New York, which is the middle of nowhere. My dad, who was an accountant, was one of the first people in the region to computerize his business. So I used to get his old computers and play various stupid computer games on them, and try to program them. We had a modem very early on—I remember I used it once, and we got a gigantic long distance bill. I didn't use a modem again until college.
After a brief stint in chemistry, I went on to do a degree in computer science at SUNY Buffalo. While I was there, I ran a file-sharing site. Everyone used to use Windows Share, internally, so I created a search engine that used PHP MySQL—I was using 3.23 beta—to walk through the network and find all the files, legal and illegal, that everyone was sharing. There was another guy there who also had a search engine, and the two of us competed on features for our internal search engines at Buffalo. It was very exciting.
That led me to do lots of MySQL stuff, and I started to get involved in the open source MySQL community. I met Jeremy Cole online, and it turned out that he was teaching a MySQL class in Toronto. To save him money, he flew into Buffalo and I drove him to Toronto—and he let me attend his MySQL class for free. After I graduated he asked me if I wanted to do training for MySQL, which I did. It was kind of a trip at the time, because I was living at home at the time with my parents…and when I explained to my mom that I'd been hired to do training, she was pretty sure it was a scam. She was really confused over the MySQL stuff. She was like, ‘but I thought it was *your *SQL!'
I did training and consulting for about four years, and then I didn't want to travel as much because I was getting married. I transitioned to support in MySQL, and I basically became the head technical support guy for MySQL at MySQL AB for a number of years. Then Sun acquired us, then Oracle acquired us, and then I left and joined Facebook. Very recently I've become manager of our team, and that's where I am now.
What's in your group's technology stack?
It's interesting. Our group—the team I joined originally—was called Database Performance, and was originally focused purely on MySQL. So there was MySQL, Memcached, tao, in our entire technology stack. Our new group, Dataperf, actually additionally has HBase and Hadoop. We're branching out to cover all of these technologies. There are actually a lot of similarities between them. Even though they're completely different technologies that work completely differently, they have the same problems, like users do stupid things—and critically, they don't know what they're using. They don't know the resources, so a lot of it is being able to tell them.
And now for the most important question: Who would win in a fight between ninjas and pirates, and why?
Ninjas, because pirates wouldn't see them coming. They'd be cut down in the middle of the night, and it'd be game over. If pirates saw them, they might be able to use their technological advantages. Maybe.
What's a more accurate state of the world: #monitoringsucks or #monitoringlove?
I don't know hashtags—I work for Facebook. :-)
But generally speaking, at Facebook our monitoring is awesome. But most other people I've seen…their monitoring sucks. We have something called ODS, which is our own internal monitoring system, built on HBase, and it's ridiculously powerful. You can basically do ad hoc reporting: I can do things like, I want to find out what the 75th percentile is of this stat over the last 30 days, and it can just spit that data out.
In six words or fewer, what's the biggest challenge your organization faces?
Technical debt, faddish technologies, taking risks.
What's the best piece of advice you've ever received?
What would you do if you weren't afraid? That's a completely Facebook slogan thing that's all over our walls, but it's really good. It's actually why I'm a manager today.
What principles guide your expertise in your given domain?
The big thing to me is analysis of data, gathering data. So particularly within the Dataperf team we spend a lot of time figuring out what data needs to be gathered, first. But a lot of people don't realize what metrics are missing, whether it's response time or any number of other things. So we gather the necessary data, and then basically we do our analysis. Generally, once you figure out what information needs to be gathered, it's a pretty straightforward analysis. Almost anyone could come to the conclusions we do. The hardest part is figuring out what they need to gather. That's what I think a lot of engineers miss. They'll look at some metrics and be like ‘oh, we get so many errors,' but how does that actually translate to user-facing errors? So there's a translation layer that many people miss. So I'd say the big thing—the hardest part—is analysis of data, based on appropriate data.
Okay, last question. What's your vision for system administration in five years?
To me, automation is king. This is probably more true at Facebook than other places, but it's everywhere. Everywhere, eventually, you'll have enough servers that everything will need to be automated. To me, system administration is primarily two things: first, creating automation, and second, troubleshooting where automation goes wrong.