Intro music by Rod Johnson: Prelude in C# minor, commonly known as The Bells of Moscow.
01:07 – Welcome to “Anarcho-Suyndicalist Tech!” …we mean, “Greater Than Code!”
02:03 – Writing Blog Posts: “Blogging is shipping.”
07:17 – How to Ask Good Questions
20:26 – Operations (Ops); Testing in Ops
— Jessica Kerr (@jessitron) January 18, 2017
Ryan Kennedy: Fear Driven Development @ OSB 2015
Effective DevOps: Building a Culture of Collaboration, Affinity, and Tooling at Scale by Jennifer Davis and Katherine Daniels
38:42 – Zines & Drawings
Sam: Having concrete strategies for asking question more effectively.
Julia: If something is painful, then do it more often.
Jessica: If asking questions is scary, put some work into the question and then you can ask it with confidence and know you’re not wasting someone’s time.
We are (currently) listener supported! Support us via Patreon!
Please leave us a review on iTunes!
JESSICA: Good morning. Thank you to Rod Johnson at the Atomist HQ for the intro music and I am super happy to be here. I’m Jessica Kerr and this ‘Anarcho-Syndicalist Tech!’
SAM: Yay! Jessica, I love how enthusiastic you are and I love you but I think that we’re looking for something that appeals to maybe a few more people and maybe just… I don’t know, has something a little bit more. What else have we got?
JESSICA: Everybody like [inaudible].
SAM: That is true but I was talking about the show name.
JESSICA: Okay, okay. Let’s see. Greater Than Code!
SAM: Yay! I’m Sam Livingston-Gray and I am here to welcome the amazingly talented and scary-smart Julia Evans. Julia is a developer who works on infrastructure at Stripe and she likes making programs go fast and learning how to debug weird problems. She thinks you too can be a wizard programmer. Julia welcome to the show.
JULIA: Hello. I’m delighted to be here.
JESSICA: Julia, you’re a fabulous infrastructure developer at Stripe. I know this from experience but you also write some really good blog posts.
JULIA: Yeah. I write, I think an abnormal amount of blog posts.
SAM: Why would you do such a thing?
JULIA: I started to write maybe a weird reason. I don’t tell people this but I guess I’ll tell you now. I was at the Recurse Center, which is like [inaudible] program and I was like, “I want to get a job at the end. Well, how do I get an awesome job?” So I decided I was going to have a media strategy and my media strategy was I wrote a blog post of what I learned every day.
SAM: That’s fantastic.
JESSICA: Every day, wow.
JULIA: That’s for 12 weeks. Then after I did that, I was like, “Oh, blogging is really awesome. People seem to like these weird blog posts I write every day. Maybe I’ll keep writing,” so I did and now it’s three years later.
JESSICA: I guess, writing blog posts every day for 12 weeks takes the fear out of writing a blog post.
JULIA: I think so, a little bit. I was like, “No one is going to read this, except maybe my friend, Tavish and Tavish would read them and it would be like, “Great blog post, Julia,” and he was trying to maintain that attitude and I was like, “Oh, no one’s going to read this except maybe Tavish,” even if it’s not true.
JESSICA: That’s awesome. Yeah, write for your friends and if anybody else happens to get benefit out of it, it’s a bonus.
JESSICA: I guess that’s part of your origin story. We always ask people for their origin story. It’s now a tradition. How did you come to be at the Recurse Center?
JULIA: My partner convinced me to go. Basically, I had a job and then I was like, “I don’t think I want to have this job anymore,” and my partner was like, “Why don’t you quit your job and go to the Recurse Center for three months and do fun programming things.”
SAM: Was the job not in tech?
JULIA: It was in tech. I did work as a programmer. I worked as a programmer for two years before going to RC and before that, I did two computer science degrees.
JESSICA: You’re not starting from nothing here.
JULIA: I was not starting from nothing. No, but I did still feel there are so many awesome things that I could learn and still are true today so I was like, “I’m just going to spend 12 weeks learning awesome stuff about computers and I did.
SAM: That sounds so amazing.
JESSICA: It’s like you have a second developer life.
JULIA: Right. Yeah, I think so. I think that was the start of my second life.
SAM: Does this mean that you have an alter ego, like a superhero?
SAM: We won’t tell anyone, I promise.
JESSICA: Yeah, and it’s like you develop that alter ego by learning awesome things about computers for 12 weeks and blogging about it.
JULIA: Right. I think with my blogging, I was like, “I can be the person who writes awesome things on the internet,” because I would read all kinds of awesome things on the internet for my whole life before that like I’ve been programming for ten years and I have read so many awesome things on the internet but I was like, “I could write them too.”
JESSICA: Yes, that’s an interesting transformation. Not everybody can quit their job and go spend three weeks at the Recurse Center but everybody can learn an awesome thing and blog about it, maybe at a lower frequency.
JULIA: Probably at a lower frequency. It’s true and I read so many awesome things written by people. I read this guy’s name is Adam Perry who wrote a blog post about how he was rewriting the libc in Rust. I think that was it. His project was called ‘musl’ and he was like, “I’m going to rewrite these functions from C to Rust, one at a time and then test whether or not they work. This project is 20% done,” and it was really cool. I was like, “This is an amazing blog post.”
JESSICA: Yes, he was like totally irrational, “What would you ever use this for activities?” What you do is you learn something and then you blog about it. This is something that we’re pushing it out of us, blogging is shipping because it’s something works but nobody knows about it. It’s not useful but helping people know about it is that’s actually —
JULIA: That’s right. I think [inaudible] did it myself as a developer add to kit like on the side but not for things that I make or for my company makes. I’m like, “I’m going to sell you on [inaudible]. Here’s ‘/proc’.” You should deal about it, right? [inaudible] try to sell you stuff that you probably already have on your computer.
JESSICA: That was fabulous.
JULIA: Then I have no [inaudible], except that I think it’s wonderful. Like I talk about strace and I’m like, “It’s kind of like I wrote strace,” even though I didn’t because there were always people who would know that strace would work for me.
JESSICA: You made the stickers. You can have the stickers. You are totally a developer advocate for strace.
JULIA: I am. All of those who think I created strace for those people, right?
JESSICA: Exactly. They didn’t have it before and now it’s useful for them.
JULIA: Yeah and I didn’t write any code at all.
SAM: But you wrote English, which is harder than code.
JULIA: English is hard. It’s definitely less [inaudible] on strace though.
JESSICA: Using code is hard. It turns out and when you teach people to use code, then you have value to it. That’s sweet.
SAM: Like knowing that you can do a thing, that’s huge.
JESSICA: Just of speaking of blog post, you wrote one about asking questions.
JULIA: Yeah, asking questions is one of my favorite things. I realized that I knew something about it that I should maybe tell people. I heard this blog post that was just like, “How do you ask a good question?” and I was kind of inspired by this blog post that I hate by ESR called ‘In public, how not ask stupid questions’. I think it’s called ‘How To Ask Questions The Smart Way’ and he spends most of the time insulting you. Then somehow, there’s a good advice and I was like, “I want to write it closer to how to ask good questions that has good advice but that does not mostly insult you.”
SAM: Thank you. Everybody appreciates that, I’m sure.
JULIA: I was thinking about how do you ask a good question? What are the parts of that? Should I tell you what the parts of it and figure that are?
JULIA: Okay. Part one that I love to do is to say what you already know. Maybe you’re asking a question about, “How the Linux kernel works?” You could be like, “I don’t know a lot but I know that there is code running on my computer and some of it is reading in C and some that is written in assembly,” and the first could be like, “Great. Good start,” or you can actually frame your question in the context of what you know.
Actually, sitting on what you know is pretty hard and I just want to know several things about Linux here so I’m asking you about Linux and I need to figure out like, “What are the relevant facts that I know right now?” It helps me to reflect to what I know. Then the other person knows what kind of question I’m asking.
SAM: This also a great way to find out if you have any incorrect assumptions early on in the process, which is really important.
JULIA: Because frequently the things that I say I know are actually wrong. I’m like, “I know these five things,” and they were like, “Four of those are true.”
SAM: And you’re like, “Yay! I’m learning something before I even ask the question I want to ask.”
JESSICA: Oh, yeah. One, it helps you organize where you are now and forces you to put that into words but it helps. It’s amazing how much English can clarify our understanding of code. Two, you might find an assumption that [inaudible] and three, it lets the person answering the question as a teacher, it lets them put themselves in your contacts.
JESSICA: I know that before asking a question, you should state briefly what you know. What else can I do by asking a good question?
JULIA: The way I’m asking questions is like there’s a person who has information that I want and they might not have that talk prepared about the information. They have the information in their head somewhere but it’s may not be organized in a way, where they can just tell me it. I might be like, “How do SQL queries work?” and they’ll be like, “Well, I know a lot about that but how do SQL query work is a pretty broad question,” and if they haven’t already create an answer to that, they might not be able to actually answer it.
What I like to do is try to ask something that is really specific and in particular maybe a fact-based. One example I come up with in this post was if we’re talking about like joins, I’d be like, “I know that we’d be using the thing called the hash joint,” which is where you have a hash table of all the primary keys and you just look them up. You can’t keep a hash table in memory and then you go through your other cable and you just look it all up. It’s a joining strategy that postgres uses, if I know the person knows a lot of postgres and they could be like yes or no.
SAM: Or sometimes.
JULIA: Or sometimes. The question is a lot closer to a fact than, “How do joins work in postgres.” Then maybe that’ll bring out something related. That reminds me of something else a lot more useful to get really specific.
JESSICA: Yeah, that makes sense, because you need to give people a direction to go or in that case, you’re giving them a starting point. It’s almost a yes or no but it’s never going to be like, “Yes! No!”
JULIA: Right. If it’s a yes or no question, then the answer will probably be sort of complicated.
JESSICA: Right because you’re asking for the subtleties and the alternatives and things like that so you give them a starting point.
SAM: This is like when somebody comes up and asks you, “What’s the best way to fit this thing into that thing,” and you sort of like it can go well, “I guess you could do this.” Then you find out that the reason they’re trying to fit this thing into that thing is because they want to like water their yard and maybe the best answer is not to try to fit this thing into that thing. It’s to have your kid do it or something, which is a horrible non-tech analogy but that’s what I had on the top of my head.
JULIA: Right, that’s really good. It’s like the person like some contact to what you’re trying to do.
JESSICA: Yeah, like a direction to go.
SAM: As somebody who maybe is being asked that question, I can be like, “If you needed to do this, here’s some strategies that might work. But since I know what you’re actually trying to accomplish, let’s go and look at this whole other new thing that I get to teach you about.”
JESSICA: That’s true because you need some combination of specific and open-ended.
JULIA: Right. I think I often use this at weird specific questions like sometimes I’m trying to extract information from someone but I don’t know exactly what information that is yet. Does that makes sense? Like I haven’t figured it out.
JESSICA: If we did, we just Google it.
JULIA: Right. They’re like these weird probes.
SAM: They’re kind of like nerd-sniping, just to get the person’s attention?
JULIA: I think so, yeah. This is kind of a weird setting, where sometimes I asked people questions just to increase my knowledge about something, like when I’m working on somebody that knows a lot Scala and I need to know more about this but I don’t. Sometimes, I don’t have a specific test but I just want to have one information. It’s like, “Please tell me more information about Scala.” It’s probably will not result to a useful answer so that was asking for some weird specific question that I think might have an interesting answer and see if it works or not.
JESSICA: If nothing else, it’s fun to get people talking about something they like.
JULIA: Yeah, exactly.
JESSICA: Also, it builds relationships.
JULIA: Someone pointed that I left that originally because I forgot about it but someone was like, “Julia, you have to be willing to say when you don’t understand something,” and I was like, “Oh, it’s true.” You have to be willing to say what you don’t understand.
SAM: Just like you have to be willing to ask a question in the first place and maybe appear like you don’t know everything and that it is okay.
JULIA: It is okay and that’s something that I practice that I’ve become really good at it, like to keep alive… I don’t know, what does that mean? And I’ve started to think of being willing to say that you don’t understand is like something that senior developers do. I’m like, “This is a property of someone who’s comfortable on what they’re doing. Because I’ve noticed that when I talk to people who really know what they’re doing, they’re often like, “I don’t understand this one thing,” and it’s not a big deal for them to say that.
JESSICA: Right because when you do understand how join queries work in [inaudible], it’s not so painful to see that you don’t know how they work in postgres and this is ahead of confidence. Also, as a senior person, I might hear somebody say a sentence about joins in postgres and I might hear one or two terms that I don’t know and those I can ask about. It’s different when I hear eight terms that I don’t know.
JULIA: Exactly and I think that is something that happens really often to junior people. It’ll be like, “I don’t know any of those 30 words mean, like it feels too much to ask about all of them.”
JESSICA: Yeah, exactly.
SAM: For seniors who are not even in an explicit leadership role but if you’re in a role where you’re around more junior developers and you’re trying to model behavior, this is a really useful strategy as well. I will often deliberately ask a question that I know the answer to just so that I make it okay to ask a question.
JULIA: Yeah, that’s really important.
JESSICA: Sometimes, I’ll do that just to make it okay for a woman to ask the question because it’s been all men up to that point.
JULIA: Yeah. Do you want to say like, I’ve definitely been in situations where there are 30 terms I didn’t understands and I did ask about all 30 of them but I was just like, “Can we [inaudible] for 2 hours?”
JESSICA: Right. When you have —
JULIA: I have to ask like these 40 words mean.
JESSICA: It’s like following all the links in a Wikipedia article, except more interesting.
JULIA: Yeah, until it’s okay to do that but it’s different to ask because you’re just not [inaudible]. I have a very large number of questions.
JESSICA: But when you explore it like that, you really get the information and then the other thing is, I am totally willing to spend two hours explaining something to Julia because she’s going to go and write a blog post about it and then anybody else who ask, I could just point them to that.
JULIA: That’s right. If you’re asking a lot of questions, I’m like writing down what you learn that towards can be a good way to give back.
SAM: Then to solidify the information in your own brain too.
JESSICA: I think this episode is going to motivate a lot of people to blog.
SAM: I hope so.
JESSICA: Yeah, so if you hear this episode and you write a blog post, please ping us on Twitter @GreaterThanCode and we will be tweeted.
SAM: Yes, please. Oh that reminds me, there was another thing that I’ve heard a lot of people talk about who write blogs is that they will blog something, even if it’s just like some stupid technical thing like I couldn’t figure out how to get library X to do Y. Then they find like a year and a half later that they’re trying to do that again and they Google and they find their own blog post so it benefits you too.
JESSICA: Oh, I’ve done that.
JULIA: This [inaudible] all the time. The other day, I was having a conversation by a friend Amy and she was saying this really interesting thing and I was like, “Can I just write this in a blog post so I remember.” It really works. I think if I don’t write things on my public blog, I don’t remember them.
JESSICA: Oh, yeah. It’s an extension of our brains, right?
SAM: There’s a cognitive bias for that.
JESSICA: There’s a cognitive bias for that?
SAM: Yeah, there’s a cognitive bias called the Google Effect, which is where we actually find it easier to forget information if we know we can find it again later. I’m just quoting that from memory but let me go look that up, actually.
SAM: I didn’t even mean it that way. That’s so great. Google Effect in Wikipedia, “The Google Effect also called digital amnesia is the tendency to forget information that can be found readily online by using search engines, such as Google.” Boom!
JESSICA: I know, there’s not so much I can hold in my brain.
JULIA: I have one more question asking tip, which I kind of came up while I’m writing this post and I love it. I think there’s a tendency to be like who should I ask a question about this thing, and you are like, “I’ll find the person who knows the most about it and I’ll ask them because they’ll know.” That’s pretty effective if you ask the person who knows the best about a thing, they probably know the answer.
But it’s also kind of a problem because that person is kind of overloaded so I try to be better at being like, “Can I find the person who knows with the least knowledge, who probably knows the answer to my question?” Then ask them. I think this is kind of cool because it distributes question-answering power over more people and it’s cool for the person because they’re like, “Oh, I am important.” Is that everyone always asks the person who knows the most, then everyone was like, “No one asked me. I guess I don’t matter.”
JESSICA: Yeah that’s a great thing.
SAM: It’s an opportunity to realized, as the person being asked that you do know something too so it’s a great confidence builder.
JESSICA: But if they don’t, then you go together to the person who knows the next most and then you both learn the thing.
JULIA: Yes, right. That’s like the cache miss.
SAM: It’s the cache miss that’s benefits everybody.
JESSICA: I like how it strikes you have a person on the team on run, who’s in charge of answering questions and they’re not in charge of knowing everything. They’re just in charge of receiving the question, providing a prompt response of, “Oh, hey I care about you,” and then they go find out the answer and then they know the answer so knowledge really gets spread around the team that way.
JULIA: Yeah, that’s really good but I think it’s hard to remember to do. I think it’s hard to not just go to that person with all the answers but I think it really pays off.
JESSICA: Because there’s another reason too and that’s because the person you know is a little bit more than you is going to have a much easier time explaining it to you in terms of you already know. If you don’t want to take two hours to track down the eight words in that sentence that the expert just used that you don’t understand, you’re going to get a useful answer faster than the person who just learned it.
JULIA: Yeah, I think we know everything about asking question now.
SAM: Or at least we know enough to get started.
JULIA: Or at least we know enough to get started.
JESSICA: And how to ask about how to get better.
SAM: Okay. I noticed one of the next things you wanted to talk about was ops, which is great because I can’t even ops.
JULIA: I think I’m trying to be the person who just learns about ops —
JULIA: — So that I should explain it. I read a lot of [inaudible] blog. I think of her as being like the person who knows all the things and I’m trying to be like, “I know three things that I just learned it.”
JESSICA: Oh, yeah that’s a good point. The best time to write a blog post about something is when you just learn about it because then it’s fresh and it’s not too involved. It’s bit-size because [inaudible] a thing.
SAM: Julia, what do you know about ops that you can help us learn?
JESSICA: What is ops?
JULIA: You write software and then at some point, you need to run your software on a computer or 200 computers or whatever and there’s this exciting thing that happens when you run software, which is a lot of stuff will go wrong in unexpected ways.
SAM: Stuff that is not even in your software.
JULIA: Which is stuff that is not even in your software goes wrong in unexpected ways and I was trying to articulate the cases I went through and learning about operations. I started out as my software will work, what are you talking about, which was like I would work on these relatively small webapps, which would mostly work and their databases would work and everything was fine. Not that many people tried to use them and they’re generally just kind of up. I was like, “What’s operations? Everything is fine but I can set up Apache, what’s the problem?” This is kind of, I think appropriate at that point. It was like a workable strategy.
Then at some point, I could do work at Stripe, which has many more computers and also with higher reliability requirements and many more components and I was like, “Oh, my God. There are so many possible things that can break. How does anyone run software? This is impossible,” so I kind of like, “This is not even a problem. How can you do this? It doesn’t make sense.” Maybe you just hide under the bed because there’s a lot of things that can happen, like you can expect some traffic which maybe you didn’t expect or maybe some piece of the software or a service just like somebody stops working for a minute or for ten minutes or for an hour —
JESSICA: Which triggers an error case in one piece of your software, which triggers some error code down later that was never tested because it’s error cases that are hard to test and then all kinds of things happened.
JULIA: Right, yeah. It’s like all these unexpected things happen and I was like, “How am I supposed to write software that works under these conditions?” Then I kind of learn to be scared. I think it’s like learning to be kind of a little scared of software learning introduction. It’s like in knowledge you’re scared but just being like something could go wrong or this might break and moving away from this that I’m [inaudible] confident that everything will be fine.
SAM: Shutting your done-in-career syndrome.
JULIA: Yeah, exactly. That has sort of helpful but it’s not like not a good place to stop because I was never [inaudible] with my code because I’m scared of what will go wrong. that’s like an extremely bad place to be, probably worse than everything is fine on what could go wrong that you just never make any changes because you’re too scared to make them happen.
SAM: Which is absolutely not a metaphor for anything else in life.
JESSICA: Yeah. As a perfectionist, you either delude yourself that everything’s going to be fine or you run away so you have to kind of get past that and you can. What’s the third phase?
JULIA: The third phase I think is a [inaudible] blog phase and it’s like maybe an entire career.
JESSICA: Did you call it ‘a happily ever after?’
JULIA: No, I think it might be called operations engineering.
SAM: This is what I have to look forward to.
JULIA: It doesn’t actually stop but it’s not like, “Oh, you learn how to do operations and then there are no more problems. You solved the problems and sets you forever.
JESSICA: I think you just said that development culminates in operations engineering and eventually we get there.
JULIA: I guess what I was trying to solidify in my head is I think I had this idea that you can just write software and if you’re really good at writing software, you can write software that was correct and might would work. It turns out that this is not true like what you do is you write software and then you try to write software that can eventually evolve into something that works.
You put your software in production and you design it in a reasonable way. Then it gets hit by all these external service that doesn’t work and you kind of learn over time what goes wrong with your service and then you learn how to make it resilience to those things. It’s more of a process of designing something in a hopeful and reasonable way. Then teaching it about all of things that get wrong as they happen. I think the idea is if enough bad things happen and you deal with them, then you’ll end up with something that mostly works.
SAM: That’s really interesting. Can I try and recast this in a slightly different way?
SAM: What I found in my own journey as a software developer is that my sort of overarching career goal has been to get better and better at writing software. I started out writing software that I thought was decent and then I found out that it was crap. Then I figured out how to, maybe possibly someday make it less crap.
At this point in my career, I’ve gotten to the point where I will start something and I’ll put it out there and then I’ll see what else happens and I will trust in my ability to not necessarily start with the best design from the beginning but to evolve, to refactor to get there. Are you describing a similar process for learning how ops works where you eventually you don’t know all the answers but you trust in your ability to fix things as they come up?
JULIA: Yeah, that’s right. I think you have to accept that it’s impossible to know all the answer about how to operate your software when you write it. But then as you operate it, you could learn what’s necessary to operate that piece of software.
JESSICA: Yeah because there are infinitely many problems that can go wrong so you just have to make it so you can notice what does and then handle the ones that actually happen.
SAM: And be able to ask questions to find out what went wrong when you don’t know.
JESSICA: Yes so you need that visibility into your software. I think that’s one piece of the software that can be evolved into something that works.
JULIA: Right and this is where you get all of these awesome things like monitoring where you’d be like, “Oh, I can see what myself [inaudible] because I’m really a good monitor [inaudible].”
JESSICA: Then you have to be able to change it. I think a lot of what we just said is that software has changed. The development process is continually changing our software, evolving it into what we need and therefore, change has to be easy. Once you have the need visibility to find out what is needed and then you need the ability to make those changes without fear. I guess I found my favorite phrase today — software that can be evolved into something that works.
JULIA: There’s a really good [inaudible] talk by Ryan Kennedy called Fear Driven Development, which is about fear in deploying code and how you can end up in the state where you are too scared to deploy your code but not for any real reason. You can be like, “I’m scared.” But why are you scared and you’ll be like, “I don’t know. I’m just scared. That’s a really bad place to be.
SAM: Because of Pavlovian, the association, right? I’ve learned that when I deploy a code, I get an electric shock.
JULIA: Yeah, but of course it’s good to be scared but those things has to be associated with reasons. You can be like, “I’m scared because I know that this code path is untested so I’m worried that when I put it to production it will work but it will break,” which is the very reasonable but that’s something you can do something about that like maybe untest.
Then you need to have this like, “Why am I scared?” Then associate it with better reason and then deal about reason. Then be like, “I’m not allowed to be scared anymore,” because like what you dealt with all the reasons you used to go to play it and not just like, “I’d still [inaudible].”
JESSICA: Yeah and if you make an automated test, then you never have to be scared again, at that particular thing. That is a lot of work, though.
SAM: What I know now is that I am pretty good at testing and I feel like the things that are reasonable to test, I can test pretty well. The things that are unreasonable to test, I can usually still figure out. But I feel like at this point in my devops career, which hasn’t even really started yet, I feel like I don’t have even any idea what’s going on so I can address all of those fears that have to do with untested code paths, like I got that. But I don’t know how to deal with fears about unknown things that could go wrong with ops or if there is something that I know could go wrong with ops, I don’t know how to test it.
JESSICA: Yeah, Like how do you test file systems as full?
SAM: Yeah, are there strategies we can use their ops-specific to help mitigate those fears?
JULIA: Great question. The [inaudible] is something [inaudible] when it’s interesting. Maybe this is wrong but I guess I try to separate the fears and the things that I think will definitely happen and things that I don’t know that will happen or not. To some extent, with things where you’re not sure if they’re going to happen or not, like maybe there’s something that was never happened before and you’ll like, “What if that happens?”
I feel like what you often do is you just let go of it and you’re like, “When it happens, we’ll deal with it,” with the things which don’t really happen. If you spend too much time worrying about them, it’ll slow you down. Like your service isn’t going to be up 100% of the time. This is like one thing, I think we’ve learned that we’ve seen a lot, it’s this idea like an error budget and you’re like, “Things are going to go wrong.” As long as you think it won’t be a catastrophe, if there’s an unlucky thing happens, then maybe you just let it happen.
JESSICA: Also, I like to think about it as in the terms of what’s the worst that can happen? Then if you deal with making it just not the end of the world, the worst that can happen is we have to restart from backup but we’re going to find out that that happened in this way and then we’re going to take this action. Then you’ve covered ‘file system not found’, you covered the servers exploded. If you can make one ‘not the end of the world’ scenario around a bunch of errors, then you can be nervous and might get paged but not scared.
JULIA: Yeah and then through things that you know will happen and that are happening regularly, like you have some experience of them, like you’ve seen them happen before probably. Then maybe simulate the error cases around those.
JESSICA: Right and then you can do frustration-driven development like, “I’m tired of getting paged at three in the morning. I’m going to make this problem be taken care of itself.” I think you can justify that because you can say, “Look, it happens. We measured it twice and now I’m going to fix this and now I expect those to be zero,” which gets back into the monitoring. It’s so hard to make software that’s [inaudible].
JULIA: It is worth it.
JESSICA: We were talking about operation and how scary it is but how you can evolve into dealing with things as they come up?
JULIA: I came up with a concrete thing, actually. We’re going to talk about knowing how your services and one thing I did when I was kind of getting started with [inaudible] 100% operations is I made a dashboard for a service I have, that showed the current status. That was useful to me when there was a problem. I could look at the dashboard and be like, “That’s on fire,” or I could look at it and be like, “Oh, that’s fine.” I feel like it was a really useful task for me to do and it was also helpful to other people.
JESSICA: And it looks pretty.
JULIA: And it looks pretty.
JESSICA: Making dashboard is shipping.
JULIA: I learned recently with devops [inaudible] by reading Effective DevOps, which is by the amazing @beerops and @sigje. What I learned was I thought devops was like you use Chef and you run servers and stuff. But it turns out that devops is actually about collaboration and that the point is not like you run servers and stuff. It’s like when you have people who are running services and people who are operating those services, those people need to work well together. I think this especially comes up in larger organizations because in very small organizations, you’re just like, “I’m going to write the service and I’m going to do all the operations for it and I’m going to do everything. This is great.” That’s actually fine and people do that and it works when you’re in a small company.
JESSICA: I thought about it as devops. Devops means you write it, you run it. Then, now I feel all these pressure to understand all the things.
JULIA: Right, but then in a larger organization, this is not I think what happens because what happens is that you have people who specialize and there are some people who are very good at operations engineering and they learn a lot of services. They know a lot about how to make services runs and there are people who know, maybe a lot about frontend development and our amazing frontend developers. You probably don’t need your frontend developers to be your database engineers. It’s reasonable for these for two different people.
SAM: Right but in a lot of organizations, there are really strong forces that push people into their own silos.
JULIA: I think devops is not about making those literally the same person but it’s more of having people work well together. Maybe an example of this is I work on infrastructure and as a security team, and I’m not on the security team and I don’t necessarily know as much about security as I would like. But I just try to maintain a good relationship with the security team and also to tell them about concerns I have really early and involve them in discussions.
I can be like, “I’m doing this thing. Should we do it like X or Y,” and they’ll be definitely X and I’m like, “Good. It’s great that we talked about that really early because now I know what to do.”
SAM: For me, devops has a lot in common with, I guess it’s an old idea now. The old idea from extreme programming of having a whole team, which is XP Dragon 4. Have everybody who works on a project actually work on a project together as part of the same team, rather than forcing people to go to a different floor of the building and schedule meetings and make it easy to have low-latency, high-bandwidth communication across all of the disciplines that you need to effectively ship your product.
JESSICA: And shared objectives.
JESSICA: Like if the security team’s only objective is to prevent a security breach, then their job has nothing to do with getting software out and is in direct conflict with yours.
SAM: Right so there’s no code like no codes.
JULIA: Right, like you want to be like our goal is to ship secure software.
JESSICA: Yes so the security person shares your objective of shipping this thing along with and not making it terribly insecure, then you can collaborate. If devops is collaboration, what is Chef?
JULIA: I don’t know. Provision and automation?
JESSICA: Yeah, I guess it’s an automation part of operations engineering at that point.
JULIA: Yeah and it’s an important part of operation engineering.
SAM: It’s a tool and it’s an important tool but it’s what Skype is to devops for a distributed team. It’s something that you need to do your job but the point is not really the tool.
JULIA: Yeah, and another thing that I learned is considered to be a part of the devops movement is continuous integration because I’ve always done the idea that you deploy your site 10 times a day or 20 times a day.
JESSICA: Like continuous deployment?
JULIA: Continuous deployment, maybe both. Continues integration is having small features and you embed it to master frequently, I think like you don’t develop on a branch for three months.
JESSICA: Then, if as soon as those tests passed, the software gets deployed without you pushing a button, then you have continuous deployment.
JULIA: Yeah, I think for continuous deployment will also just mean that you deploy your site very frequently and it’s not the weekly deploy so it doesn’t necessarily have to instantly when you merge your code. It just happens a lot. But I think what I didn’t realize is deploying your site 20 times a day is a radical idea or was a radical idea.
SAM: Still is in a lot of places.
JULIA: And it still is in a lot of places and I was like, “Of course, it makes sense that it’d be a radical idea. That’s something that you were doing before.
JESSICA: I remember working in the enterprise and they had four deployments a year: April, July and September, and occasionally an emergency in October, which let them say four.
SAM: Right. I think that too has its roots in the XP in [inaudible] communities where there’s this recurring idea that if there is some part of the development process that is causing you pain, you should do it much more frequently so that you’re forced to confront that pain on a daily basis and develop ways of making that thing a lot easier to do.
JESSICA: Great, and then you get the automation and you want to make it not scary to do, which in the end, being software developers, we solve with automation and I guess this is where some of the confusion comes in. As devops, you might think of it as bringing development to operations as in automating your operations like with Chef. But it also about bringing awareness of operations into development with that collaboration.
JULIA: Yeah and I think this is like a lot of, especially they have an idea that people in operations are trying to push because they’re like, “We automated operations but you developers still need to know how to operate your services. You can’t just say we wrote puppet and never done.” I think there’s maybe more awareness that operations people need to do some programming, than there is that developers need to know about operations.
JESSICA: And so you write blog post about it.
JULIA: And so you write blog post about it.
JESSICA: I mean, you specifically, and zines, we didn’t talk about zines at all.
JULIA: Okay, Let’s talk about zines.
JESSICA: Because you’re known for conveying information about things like a strace, not just through your enthusiasm in English words but also with cute drawings.
JULIA: Yeah. This has been kind of a weird thing. The only reason I started writing drawings is that in the middle of 2015, my wrist hurt a lot and I didn’t want to type on my computer. I still have a lot of ideas that I want to post on the internet so I would sometimes draw them on a piece of paper and take a picture and post them on Twitter. This was surprisingly successful despite my complete lack of drawing skills so I started doing it more. Now, apparently this is something that people know that I do.
JESSICA: Typing hurt so you did something else more and it turned out that was like a surprise win.
JULIA: Yeah, it’s like surprisingly amazing. As I keep on doing it because it works unreasonably well, like why does it work so well? I don’t know if I know why it works so well —
JESSICA: Does that even matter?
JULIA: I think it really doesn’t matter. For example, I wanted to write about stuff and someone was like, “Julia, write about ‘/proc’,” and I was like, “Okay.” Everyone knows about ‘/proc’ but I’ll write about it anyway. I should have known this by now but every time I think… Oh, no everyone knows about this already. No one knows about /proc —
SAM: I don’t even know what it is right now as we speak.
JULIA: Exactly, so /proc is a file system in Linux, it’s a directory in Linux that has all kinds of cool information about your computer. It’ll show you all of the files that every process has opened. In particular, if you accidentally delete a file but it’s still open, like if a process is still open, you can recover it using /proc, which is really cool. It’s kind of like a wizard thing that exists on Linux, that is really useful. It also has all the environment variables for every process so if you want to know what environment variables you process, you can just read it out of /proc.
JESSICA: Because Linux uses the file descriptor abstraction for like everything.
JULIA: Right. It’s like everything is a file. Then the list of environment variables for your process is a file? I don’t know… but it is and you can —
JESSICA: There’s got to somewhere.
JULIA: It’s somewhere.
JESSICA: In Linux, everything is going to be a file descriptor.
JULIA: Yeah. I wrote like a little comic about this and I posted on Twitter and everyone was like, “What is this? This is amazing,” and I was like, “Oh, yeah. It is amazing.”
JESSICA: Awesome. We’ll have a link to some of your drawings in the show notes. Is it time for reflections?
JULIA: I have one more thought about drawings —
JESSICA: One more thought about drawings. Go on.
JULIA: — Which I gave a bunch of talks and I realized that I got really mad that I couldn’t force people to remember the stuff I said in the talk so I wanted to be able to give them something physical, which is kind of why I started writing zines. For example, I know people can’t see this in real life but I’m currently holding up a small zine called Production Machine Learning, which I wrote last night —
JESSICA: It’s so cute.
JULIA: — And it has a bunch of comics, about things I learned about doing machine learning and production because I’m giving a talk about it. I’ll just print out 30 of these little tiny booklets with comics in them and give them to people at the talk and then maybe they’ll remember what I said.
SAM: That’s like a little eight-page thing with tube sheets of paper stapled together in the middle and folded.
JULIA: Yeah, exactly. It’s literally two-pieces of paper, stapled together and folded.
JESSICA: Which sounds totally simple. All you have to do is take eight drawings and put them together. But I’ve seen Julia work on these and she does layouts and she does plans, then she gets feedback on each page so print out her zines and appreciate them. Absolutely, try to make your own but don’t think it’s going to be as easy as she makes it look.
JULIA: This is what I wrote last night. It took me two hours.
SAM: Yeah but how many did you write before that?
JULIA: Only three but the other three, I spent a lot more time on. But I’m kind of experimenting with the idea of writing this in two hours and then print [inaudible] here. This is a cool tidy thing, instead of edit it for a hundred hours, which is way —
JESSICA: Making a zine is shipping.
JULIA: Yeah because shipping things is fun also.
JESSICA: Yes, it’s like someone could try that for their next project meeting and that would be memorable.
JULIA: Yeah and then people can read it on the bus. This is my vision of zines.
JESSICA: Yeah, maybe totally start a conversation on the bus with somebody next to you who is like, “What are you learning about? That looks amazing. What is this /proc? I need this in my life.”
SAM: Listeners, if this happens to you, please tell us.
JESSICA: Okay, awesome. The one thing we do at the end of the show is be to ask each of us what something that you will take away from this that you learned and maybe continue thinking about.
SAM: Well, I can start. One thing that I am going to take away from this is this idea that even at this point in my career, I still ask a lot of questions and I’ve just sort of gotten in the habit of saying whatever comes into my head. But it’s really useful to have a bunch of little concrete strategies for asking questions more effectively. I think I might actually have to try a couple of these explicitly and deliberately and see what happens. Thank you very much for that.
JULIA: The thing I’m going to think of that is you said about if something is painful and do it more often, like it’s something that I’ve heard before but I feel it’s very hard to do because it’s such a human instinct is something that is painful to do it less often and be like, “Let’s just not do that, okay?”
SAM: But didn’t you do that when you wrote a blog post every day?
JULIA: That wasn’t actually painful.
SAM: Oh. Yay!
JULIA: But it was very effective to do it more often and actually, yeah. It’s a good point. I think it is the same thing like no matter how well I know that thing, it’s still important for me to think about it because there are always new things that are painful. But I need to do more often.
JESSICA: Typing is not one of them, though. If it hurts to type, do something else.
JULIA: I think that’s right.
JESSICA: Or get a brace or work on your ergonomics or something.
SAM: We’re talking about existential pain here, not physical pain.
JESSICA: Yeah, we’re talking about fear and that has been a theme today of how do you deal with fear, how do you confront your fear. In the case of writing a blog post, if that’s scary to you, just do it anyway because you’re going to find that those fears are unfounded. If asking questions is scary, put some work into the question and then you can ask it with confidence and you’re not wasting someone’s time because you put thought into this. Confront your fear of development and of operations in a couple of ways.
First of all, if you’re not scared, you’re doing it wrong. you don’t know enough and once you are scared, you can consciously think about, “Is this a fear that I can solve with automation, such as testing,” or the other one is when you have a whole bunch of fears, “I have put into production anyway and see which ones happen and then deal with those.” But make sure you can see which ones are happening by adding visibility because we fear the unknown and if we can make what’s happening a little more known, then we can be nervous but not too afraid and then we can ship.
Those were awesome. Julia, thank you so much for joining us.
JULIA: Thank you so much for having me. This was extremely fun.
SAM: Yes, it was. Thank you. Thank you.
JESSICA: Yes and also, Greater Than Code is a community supported podcast. Mandy, who is our podcast manager and editor does a fantastic job and this is made possible by our Patreons so if you want to support this podcast, go to Patreon.com/GreaterThanCode and then you can get access to our little community Slack, in which we have nice, supportive, happy conversations and sometimes abusing rants.
SAM: Oh, and stickers.
JESSICA: I will both send you stickers. Mandy will send you stickers.
SAM: All right, listeners. Thank you and we’ll see you next time.
This episode was brought to you by the panelists and Patrons of >Code. To pledge your support and to join our awesome Slack community, visit patreon.com/greaterthancode. Managed and produced by @therubyrep of DevReps, LLC.