Janelle Klein: @janellekz | Idea Flow: How to Measure the Pain in Software Development | Open Mastery
00:16 – Welcome to “Goats On Podcasts” …we mean, “Greater Than Code!”
01:19 – Origin Story
04:36 – The Development of Development
06:58 – Automated Tests and Mistake Detection
09:21 – Designing Releases and Best Practices
20:13 – “The Code is Better”
— Jessica Kerr (@jessitron) April 12, 2017
15:08 – Measuring Effort, #CollaborativePain, and The Error Handling Process
— Jessica Kerr (@jessitron) April 12, 2017
33:24 – Discovery and Documentation
37:44 – Agile Fluency
40:42 – Building a Conceptual Model of our Brains with Code
51:56 – Identifying Project Pain: Slicing and Dicing
57:23 – Change Sizing
Rein: Gerald M. Weinberg’s Quality Software Management Series
Sam: The pain that we experience in software development is really cognitive dissonance.
Jessica: Programming is like summoning a demon.
Please leave us a review on iTunes!
JESSICA: Hello and welcome to Episode 28 of ‘Goats on Podcast’. I’m Jessica Kerr and I am happy to be here today with Rein Henrichs.
REIN: And I am happy to be here today with my friend, Sam Livingston-Gray.
SAM: Hi, everybody. I’m pretty sure this is Greater Than Code but let’s just go with it and we’re joined today by Janelle Klein. Janelle is a NameNoFluffJustStuff Tour Speaker and author of the book, Idea Flow, a technique for visualizing the friction and flow between a developer and the software, like an EKG that monitors your pain and creates a feedback loop that helps you improve.
After researching Idea Flow for years, Janelle turned into a hobbyist cognitive scientist, obsessed with the challenge of building a brain and code like you do. After a 17-year career as a developer, consultant and CTO specialized in data-intensive analytics, she’s now a full time entrepreneur on a crazy mission to bring free mastery level education to every human in the world by conquering generalized AI. Welcome to the show, Janelle.
JANELLE: Thank you. Happy to be here.
JESSICA: We’d like to start the show with your origin story. How did you get to be a superhero?
JANELLE: I don’t know about superhero but when I was growing up, my mom always told me to just follow your dreams and find your passion and to do whatever that thing is that you’re drawn to in life. When I grew up, I was really into music. I play flute and guitar all through high school. Music was my entire life. Then when I went to college, I went to college to be a professional songwriter. I got in class though and I started to realize what a career in music would be like and I was like, “I don’t think I really want to do this.” It sort of takes the passion in and fun out of it.
I was talking to my boyfriend at the time and I was all upset and he was trying to make me feel better. He’s like, “Well, let’s take a class together. That would be fun.” He’s looking through the class catalog and he’s like, “Oh, assembly programming. That sounds like fun — X86 assemblies,” and you got to keep in mind that my only experience with computers at this point was playing King’s Quest in high school. Since here I am, in this assembly programming class and it’s like programming my TI-85 in math class. I’ve got a few basic calculations, I can move some registers around them and I’m like, “I got this.”
Then I started thumbing through the textbook and then I found Interrupts and I was so excited like I could switch that computer into graphics mode and make the PC speaker beep and all these awesome things that I could do. I started just reading through the whole book and then making the game BrickOut with the color blocks, a little paddle that moves around with the mouse and completely in assembly, not knowing any other languages out there even existed.
I showed my teacher what I was working on and my teacher is like, “Why don’t you just keep showing me what you’re doing and as long as you do that, you get an A.” I’m like, “I like this class. This is pretty awesome.” But that’s when it really hit me of just I can create anything that I can dream. It was like this incredible art form where I could take ideas in my head and dreams that I’ve always had and I could bring these things to life. One of my new friends I made — I am trying to remember her last name, Denise — and she said, “Software is the closest thing in the world to magic,” and I thought that was so awesome to just think about yourself as one of the magicians in the world. How awesome is that? We bring ideas to life. It’s pretty cool.
SAM: One of my computer science instructors like to use the metaphor of summoning a demon where he talked about writing a program as being like constructing a summoning circle and if you did not get every detail exactly right, the thing was going to break out of it and kill you.
JANELLE: Yeah. I thought of those break out and attack before. It hasn’t been good. I generally emerged victorious, though.
JESSICA: Yeah, this is totally why I think any story of how you got into development is a superhero origin story.
SAM: Pretty much.
JESSICA: In your case, you have even more because you started studying the development of development.
JANELLE: Yeah, I just got fascinated with the problems and we got all of these ideas in our industry about the stuff that’s supposed to work and following best practices. As long as we follow the right recipe, everything will end up turning out okay and I believe that for a long time. I was like super gung-ho, TDD zealot-type. Then I found myself on this project where we were doing all the right things: CI, unit testing, design reviews, code reviews. We had a religious theme by today’s standards. This was the days of rational where Agile is this cutting edge thing where we were taking all of our heavy weight processes and heavy discipline and just getting into this new XP thing and I was really into that.
Then when I was on this project and we ended up basically driving off the rails anyway and our project brought on production three times in a row, it went completely off the rails and it took us a couple years to be able to turn this project around after all of that. In that process of figuring out, what was actually going wrong and what was causing our pain and the data we started measuring running totally counter to a lot of the best practice ideas. It really made me start rethinking how much of what we were doing in the way that we’re thinking about our problems has to do more with just how we’ve learned to see goodness in a certain kind of design and just our own cognitive bias influencing how we see the world.
SAM: No, as programmers, we are perfectly rational creatures. That’s our superpower, right?
JANELLE: Yeah, I guess probably the main thing that I’ve learned is that our brains don’t actually operate on rational principles at all. Our entire wiring, powering of our brain is based on emotion. You really see that in the world going on around you, how much emotional [inaudible] affects people.
JESSICA: Totally. Janelle, in your book you said that one of the things you learned and one of the things that was holding your team back from releasing and releasing reliably was automated tests?
JANELLE: Yeah, that’s kind of funny. We had this mountain of automated tests that we had that everybody is sort of experienced the maintenance burden associated with lots of automation. But the other thing that happens is once we started getting a high degree of noise in the system, it has all these counterintuitive effects and because your brain’s kind of tune out when you’re doing stuff that it’s tedious and we can’t really help it, it’s just kind of how we’re wired. When we start doing these tedious things, we start making assumptions and just greenwashing lots of tests. It isn’t really feel like you’re doing that. It’s just stuff that you start seeing in hindsight of like, “Oh, I made that bar green and I probably shouldn’t have,” and —
JESSICA: By noise you mean, test failures?
JANELLE: Yes, tests that are failing and nothing’s actually broken that they just need repair. I call those quote false alarms.
SAM: Brittle tests.
JANELLE: But it’s not necessarily brittle tests exclusively. If you look at the testing pyramid suggestion that we ought to write lots of unit tests and if you work component test and only a few UI tests at the top and then you start looking at what kind of tests that ends up generating, what we tend to do when we think about our tests is what kind of code coverage are we getting and how do we sort of maximize that. The way I think we need to be thinking about it, as our code moves from one state into another state, what are the mistakes that I can potentially introduce during that transition and how do I build a mistake detector that’s going to help and tell me that when certain types of mistakes are made? It’s like trying to get the right information to the human being is the challenge.
You’ve got the noise thing but you’ve also got really poor signals where we aren’t thinking about that flow from one state to another and thinking about the application as validating that works in a static state and then you change it and validate it again, as opposed to how do I detect mistakes in the context of flow. Does that make sense?
JESSICA: That’s really is fascinating. You used a beautiful phrase in your book that you started to design your releases.
JANELLE: Yeah. One of the things that we started doing is designing our releases in a way to reduce risk. As we looked at any particular release as what are the changes that we’re going to make in the context of that release and what types of functionality you’re going to be at risk and then we came up with a strategy for mitigating that risk, sometimes you have a particular change in your release that just drives up the risk too much. It could be even that one line of change that is going to take you a month to properly validate. One option is to change the functionality or potentially split it out into a release of its own and designing the releases around your risk profile so it’s easier to control.
JESSICA: Michelle Brush did a talk at O’Reilly Software Architecture last week where she mentioned the same thing where they started every release, they would make a grid of what’s changing and how likely is this to cause a problem and how big is that problem going to be, if it happens. Then building mitigation for any sort of a risk that was either likely to happen or likely to be big, if it does.
I love your focus on change as opposed to just snapshots as if the code really has meaning in a specific state. The point is that we’re building a system and therefore, it’s constantly in flux.
JANELLE: Yes, like the only constant is change, right?
SAM: Right. The entire focus of XP and Agile in general was this idea of making change easier to deal with.
JESSICA: Stop imagining but there is such a thing as right. No code is ever correct. It’s just on the road toward better.
JANELLE: Yeah, I think the other thing that you start to see a lot is once you start seeing how the change that you write affects other people, you start to realize how biased we are just in believing our code is good because it looks good to us. Then once you see how other people suffer from the design decisions you make, it totally changes your perspective on things.
SAM: You hope that it does. I’ve worked with enough people who never got to that stage.
JESSICA: There’s a beautiful, little illustration in your book again, which I totally spent a couple of hours last night reading because as soon as I saw it, I was super fascinated. There’s a picture of one person with a thought bubble that says, “Better.” Next to it, another person with that thought bubble that says, “Better,” and a big ‘not equal to’ in the middle.
JANELLE: Yeah, I found that most of the time because our definition of better has been so fuzzy and hard to define in our industry that we’ve tended to drift toward proxies, which is where best practices come in. If you talk to people about what they believed better to be, it is often better implementing best practices as opposed to better solving the problems. We really have to get in that habit of asking, “What problem am I trying to solve?” And just making that a mantra to get out of this solution-focused anchoring. The problem is we don’t have enough test automation. It’s a really common thing I hear.
REIN: It’s funny because if you go back to the original introduction of this best practices idea into the software industry, which is largely due to Kent Beck, he was very clear that a best practice lives in a context and you have to apply it in the context for it to be the correct thing to do. I think somehow we’ve lost that.
SAM: We grow up focusing on tools — that’s what we learn about for the first several years of our career, at least — and it’s so easy for us to get into that trap of everything is about the tool and about the solution and it’s all too easy for us to forget that what matters is what does the user of our system want to do with it.
JESSICA: User experience and then it turns out that we can maximize that by maximizing our experience. Janelle has a particular definition of ‘better,’ right? The code is better if we can flow our ideas into it with less friction. Did I get that right?
JANELLE: Yeah. I think the only difference, I would say is from the very beginning of your statement of the code is ‘better when’ and the problem I think I have is that I think we need to abandon the code is better as an object or a statement and start shifting to the experience is better and all of our experiences are better, as opposed to assuming that the code has any innate quality of betterness out of context. I think trying to define that into an absolute is so anchored in our head because it’s what we’ve done over the last history of software development, that trying to shift out of that paradigm, initially requires a deliberate awareness of those type of anchoring.
JESSICA: That’s beautiful. There is no, “The code is better.” There’s only, “Our experience is better.”
JESSICA: It’s kind of like when I was in college, I would spend money on things because I didn’t have very much money so if I spent it, I wanted to have something that I got to keep. After I got out of college and start working as a developer, I accumulated too many things that I decided that what matters is actually experience and started spending my money on that and I’m much happier ever since.
JANELLE: That’s awesome. Things are fun if they give you good experience.
SAM: Right. My experience with things is often that I just have to manage them.
JESSICA: Oh, great which gets back into the code is not an asset, it’s a liability.
REIN: We’re talking about how we’re not very good at the subjective measurement of time. I agree but I also think that there’s a problem with the way that we measure effort and that we don’t really understand where it’s going so we’re talking about, let’s say fixing a bug. How long does it take to fix a bug? Some teams track that. They have an issue tracker. They can tell how long it took from a bug to go from open to resolve. But what does that mean? Where does that time really go? What are the actual steps involved in fixing a bug?
It’s more than just reading the bug report and then writing code. There’s a lot more. There is discovery. There’s a whole series of steps that go into solving a bug and which one of those are getting the time? What if the most amount of time you spend solving a bug is figuring out which person should take the bug report?
SAM: Right. For me the most valuable step is often that 10 seconds where I’m talking to somebody else and they say some random thing and I’m like, “Wait. That’s it.”
JESSICA: Janelle, with your idea, do you have a particular way of dividing up that time?
JANELLE: Yeah. I’ve been focusing on creating a model of the problem solving process itself and then measuring the amount of friction in each stage of the problem solving process. With the example of debugging some bug that you have and the first part being having to go and figure out who the bug belongs to, in the Idea Flow map, that period of time would be marked as strategy and the hashtag with collaborate team because it’s an issue with having to figure out who’s knowledge and effort this belongs to.
We’re basically got tools that integrate with IDE and standardized model that works across development because it’s measuring things that are just part of the problem solving process itself, like cognitive dissonance for example. In general, what I associate with the word pain now is I’ve done to deliberately to rewire my brain is measuring the duration of cognitive dissonance as pain. You can think of it as diagnostic difficulty. If I’ve got some unexpected thing that comes up on the screen, how long does it take me from the time that unexpected observation occurs to the time that the problem is resolved.
Then one of the things you start seeing is the skyrocketing cost of diagnostic difficulty having to do with the 90% of our software that isn’t our code but everybody else’s code, in terms of third party libraries as that one line of magic code that we’ve optimized for brevity and cyclomatic complexity and whatnot is this abstraction that’s 100 levels high and it’s all well and good when that abstraction works fine. But as soon as something breaks, it’s like an egg that’s cracking and all of its guts spill out and suddenly you have to understand this conflict picture between cause and effect and that’s where we’re losing all of our time now is in diagnostic difficulty.
REIN: I want to unpack a little bit, specifically this error handling process because I think it might be a good example for us to talk about for a while. In a book by Gerald Weinberg called Why Software Gets In Trouble, he describes the process that goes into resolving a failure. It starts with detection — detecting that there was a failure where failure is defined as a difference between what you experience and what you expected to experience, specifically one that you don’t like then the fault of the underlying cause of that failure within the system or causes.
He breaks this down into detection, location or isolation, which is matching the failure with the underlying faults that caused it. Resolution is the process that ensures that a fault no longer exists, which could be done by making a change to the code. Classifying the bug as won’t fix. There are a variety of ways to resolve a bug. Distribution, which is the process of getting the bug report into the hands of the person that is going to eventually fix it. Looking at that as the framework for evaluating your error resolving process, how would you model that within your system?
JANELLE: Interesting. As you’re going through this list of things, I kept thinking of this is a set of goals to achieve as opposed to a process, by which I would arrive at the result, as sort of like the set of results. If I think about how I go about troubleshooting a particular issue, I think about it as a series of experiments I’m going to run. I imagine that my code is like a sandwich that I’ve got electricity going through. I’ve got some set of behavior that’s changing and then I’ve got things that make it difficult to observe what’s going on and things that make it difficult to kind of manipulate the inputs. I’m thinking about how do I run an experiment that’s going to give me a clue that’s going to help me narrow down the relationship between cause and effect.
I think about what are the scope of possibilities and what are the potential clues that I might come up with. I realized there’s a step of isolation but I see those as a set of goals to achieve and not a process, by which I would arrive at a solution to those things. I don’t see how I could do those things step by step in that way in my actual work flow.
REIN: Yeah, that’s fair. I don’t think it’s a process so much as just a way of classifying the time spent in the process of fixing a bug.
JANELLE: Sure and maybe that’s it. It’s just different than the troubleshooting model I have in my head because the way I break out my metrics is by experimentation cycles. As I measuring haystack size in terms of from the moment you stop staring at the code and decide to start moving into doing stuff, I measure that time from initial changing things and tell the time you run your first executable thing like validating that the thing works to some degree. Then measuring the time as haystack size and then I’m looking at the characteristics of the experimentation cycles that go into isolating any bugs and problems within that haystack. Then what is the correlation between the inputs and outputs of the experiments you’re running relative to the nature of the code that you’ve changed inside this box.
Then when we look at diagnostic difficulty, we can look at patterns and the characteristics of experiments and how we can improve observability of what’s going in the box or make it easier to manipulate inputs, relative to the types of things that seem like we’re slowing down that process of experimentation but it gives you visibility into things, for example ambiguous clues when you get an output on the screen and there’s multiple possibilities for how behavior can occur and you make a bad assumption and go down some rabbit hole, whether it’s non-deterministic behavior or something that’s just abstracted in a way that creates ambiguity, you start looking at the system in terms of its inputs and outputs from an experimentation perspective, as opposed to the design of the code itself.
Often, when you look at the data, a lot of times we’ve had cases where we’ve refactored stuff to make the code better and then it looked all neat and modular but the experimentation characteristics ended up adding all this time to troubleshooting because sometimes having with unit test a little part but find ourselves having to integrate in order to find the majority of the mistakes anyway because they were in that integration space which increasing the scope and the difficulty of isolation. You start looking at all the set of problems under a different model.
REIN: It’s interesting. We agree on a larger concept here which is that you can only know what you choose to observe. If you only measure how long it takes to resolve a bug, that’s all you know. You don’t know where you spent the time. If you, in addition choose to measure, how long does it take to get a test running then you know something else, you know something more. Instead of just how long does it take to resolve the bug, if you break it down into detect, locate and resolve, then you might learn something surprising which is that it’s taking your team longer to locate the source of the bug that is to actually write the code to fix it. We both seemed to agree that depends what you measure informs your outcome in some sense.
JESSICA: Janelle has as another point there. Janelle, tell me if I got this right. When you’re looking at breaking down a problem, you’re not looking to break it down into steps that are goals. You’re looking to break it down into smaller problems. You can’t solve a problem until you can do an experiment to find out whether it worked. Whereas, in the bug process, if you break it down into distribute, locate — these little sub-goals — you don’t know whether you succeeded at the first one until you get to the very end of all of that and see if you’re able to fix the bug.
JANELLE: That’s an excellent point like based on the problem breakdown thing, you’re right. It’s not a breakdown of the problem into smaller problems. It would violate that rule.
SAM: It seems more like a retroactive description of what you did.
SAM: And maybe if you go through the experimental process enough times, you start to recognize, “I think I might have just crossed over from this phase into that phase,” but it doesn’t necessarily guide you in the moment.
JANELLE: Do you think you could carve up those things after the fact of how much time you spent in each of those buckets?
REIN: Yeah, that’s exactly what was done. There are studies where people were asked to track these things and they were given clear instructions as to where the boundaries are.
JANELLE: I see, so the use of it was retroactively looking at your experience that you had about how much time did you spend in each of these buckets estimate.
REIN: Yes. Correct, so you do the work that you would normally do solving a bug but you marked down times where you spent in these different buckets. When you’re done, you can go back and look at the breakdown of where you spent the time and it was often surprising where the time was being spent. But it is different from explicitly doing experiments in the way that you’re describing.
JANELLE: I think the thing is that I found that a lot of these things that we do in hindsight bias, we basically look at the problems after the facts and we think, “This was clearly a problem. I should do better next time,” and then we believe that because we thought about it that will mean that we automatically do better next time. In reality, when we’re in the moment of that decision, in order for the past experiences to come to mind, that situational recognition has to trigger the memories of our past experiences for those things to actually come to mind.
If everything we’re doing is in hindsight bias, what happens is based on hypotheses, based on the predictions that we’re making in hindsight bias as opposed to reflecting on the experiences and stuff that happened in the moment and then linking those things situationally to past experience. For example, if I just categorize those things and went, “That really took us a long time to isolate those bugs.” What do I do with that information? How do I know that the things that I’m doing are going to actually make the situation any better? How do I even know if, after I go and see those metrics and make improvements to make the bugs easier to isolate that the things that I did even did any good?
REIN: That would be where this process of experimentation would come in. The thing here is knowing that there is such a thing as the time it takes to look at a bug on, which you can spend the attention to improve it. If you only measure the time it takes to resolve a bug as one thing, that process is actually composed of multiple other smaller things you do and you may actually be good at one of them and then trying to improve that thing would be a waste of time, compared to trying to improve some other thing. But then the process of improving that thing you’ve decided to spend your time on does benefit from experimentation and I think the other things that you were talking about.
JESSICA: Like the part where you learned that we actually don’t spend much time waiting for test so optimizing for test execution speed is not nearly as useful as optimizing for what was at the clarity of the tests.
REIN: This is definitely analogous to that. It actually, for a lot of organizations, takes longer to locate the source of a failure, what part of the code or what part of the infrastructure is causing the failure than it does to actually fix that thing once you find it so if you’re spending your time —
JESSICA: — a haystack size.
REIN: Yeah. If you’re spending your time improving the time it takes you to fix a bug in the code once you’ve found it, you’re optimizing for the wrong variable.
JANELLE: From your experience, do you see people like developers not being aware that there’s these multiple aspects of things? For example that thinking time dominates troubleshooting more than fixing the bug. Seems like a thing that managers might not understand but developers, I would think would be a generally understood thing.
REIN: In my experience is that there’s been a poor distribution of this information throughout the organization. For example, developers might not know how long it took the bugs to land on their desk from when it was reported.
JANELLE: I could see that for sure.
REIN: They may have some insight into where they’re spending their time when they fix the bug, but it wouldn’t be measured. It would just be intuition.
JANELLE: Well, it sounds like if you’re doing hindsight estimates that it’s going to be intuition based, anyway.
REIN: I’m not sure. This is live. You do it as you’re working it. You report the time so you say, “I’m going to start doing location now,” and then you said, “Okay, I stop location and I’m moving on to resolving.”
JANELLE: Okay. I’m sorry, I didn’t realize that. I was thinking —
REIN: This is just time tracking but breaking it down into more discrete and sort of task oriented reporting.
JANELLE: I apologize. I misunderstood that thing.
REIN: That’s all right.
JESSICA: Yeah, I got the same but it’s fine. Now we know.
REIN: It seems similar in that way, in terms of you need to know if you want to figure out why your testing process has a lot of friction. You need to know which part has the friction, specifically and the more specific you can get, the better you can do in targeting your improvements.
JESSICA: Yeah, Janelle has different accesses. You could read the book. To our listeners, the Idea Flow book is super fascinating, the way she breaks it down into smaller problem solving cycles. A lot of it, I noticed came down to building a conceptual model of how the software works like your team having a model of how the software works… Go on.
JANELLE: I was just going to say, one of the things that we noticed was how much time and how many mistakes that we’re making because people had misunderstandings of how the software worked so we spent all this time working on test automation and trying to get the test to somehow catch the bugs. Ultimately, ended up solving a lot of our quality problems by building a data dictionary and sitting around a room with a whiteboard and just hashing out vocabulary and stuff, like the effect that it had on improving the code in the conceptual models to the team which just mind-blowing and it took a day, as opposed to the year of test automation that we did to solve a lot of our quality problems.
JESSICA: A day? Wow! That sounds so practical.
JESSICA: What if for hackathon day, we instead hack on it — universal language.
SAM: Understanding the domain, what use of that could possibly have?
JANELLE: It’s one of my skills that need to get taught. I think it’s largely gone out the window with the shift from heavyweight processes to, “Agile [inaudible]. We don’t need to draw architecture diagrams anymore,” and all the stuff that kind of gets thrown out. I think of all the most practical things, it’s understanding how to build an ubiquitous language, glossary, ER diagram, just those basic skills of understanding how to hash that out with your team and in the importance of it.
SAM: I just want to go on a real quick rant here which is that you talked about throwing away all this heavy process and going to Agile [inaudible]. But the only reason that Agile works at all is that it’s about this idea of gathering information and using it to create meaningful feedback and then acting on that feedback to change your process. So many teams adopt scrum and the idea of making your reporting visible to your upper management structure but don’t ever actually focus on this idea of using feedback and they just keep repeating the same mistakes over and over and over again.
JANELLE: I think it is worse than that because if you think about what we do in retrospectives, we sit around and we start with a question like, “What are the biggest problems that we need to solve?” And then we started brainstorming and coming up with a big lists and then once we’ve got this big list of improvements, we just start chipping away our improvements for months. What happens in this meeting, brainstorming about the most complex question ever, what are the biggest, most important problems that we ought to improve? What comes to mind, again cognitive bias is what problems do I feel the most intensely about? Then all of these different biases come to mind. There’s those unit tests that you feel like you should have written and you feel really guilty about so that stuff comes to mind. Sunk cost bias of us having all these tests and things that we’ve invested all this energy and has problems with it that once we invest in that, we want to fix it.
There’s just so many things that make all the stuff come to mind that isn’t necessarily in correlation with the most important things and then we go and say, “We’re doing feedback and science and Agile and improving,” and we end up spending tons of time improving all of the stuff that doesn’t make any difference and we just get caught in this cycle of ritual. Like all these things that we’re doing in Agile have just lost their meaning. It just makes me miss the days of XP when the things that we’re doing about we’re about hardcore discipline and learning and discovery and these words used to have meaning.
JESSICA: I wonder if back in the day when it was about hardcore discipline and discovery, that was the particular things we needed in that context and now, we need to discover something different.
JANELLE: Yeah. Well, I’ve seen this happened in a lot of different areas where there’s this pendulum that swings multiple ways for a while and then every time it crosses over the middle again, it’s like, “Wait a minute. We need to have a paradigm shift and shift to a mode of principles.” There’s all these systems that seem to have that same quality as an oscillating pendulum that eventually shifts into something different, then eventually we go, “Oh, there’s principles.” I think at some level, we need some kind of concrete system to anchor around principle, as opposed to getting stuck on best practices at one end of the pendulum or the other.
JESSICA: We need unit tests for practices.
JANELLE: Yes, the solution is more unit tests like the hammer for every problem.
JESSICA: Well, it’s an experiment. Anything that we choose to do in our retro that we need to fix this, how are we going to know that work? How are we going to know that help?
JANELLE: Yeah, but we have no feedback loop. That’s been a challenge.
JESSICA: Right a feedback loop at its best is, “What’s our bias next week?”
JESSICA: And the documentation is an example of the pendulum swing. We clearly had way too much internal documentation and now, we’ve swung to none and can we get back toward the middle already?
JESSICA: So the question isn’t, “Should we have documentation or not?” It’s, “Which documentation is useful in our context right now?”
JANELLE: I’d love that. In this moment, what does I mean right now, what is the question I need to ask so I can make the right decision this moment?
JESSICA: For me, it’s usually what yak do I need to shave so that right now, I don’t have to experience this pain again.
REIN: I wonder if every organization can adopt these changes or if it’s more successful for a certain organizations, depending on their development in terms of incorporating other sorts of feedback into their process.
JANELLE: So that some organizations would be more mature than others? Or —
REIN: There are organizations that don’t even incorporate feedback loops into the process at all. There are organizations that are pretty advanced in the way that they incorporate feedback loops but they haven’t gotten over these biases that you’re talking about. From my experience, there is something of a linear track that organizations get on in terms of increased levels of competence. You have to start by incorporating feedback cycles about the product you’re building and then you can incorporate feedback cycles about the process you’re using to build it. But my experience is that trying to jump ahead isn’t always very successful.
JANELLE: It makes sense so you can only go meta one level at a time.
REIN: Yeah, that’s I think a much more succinct way of saying what I was saying. I guess my question is where does an organization have to be culturally — in terms of their progress in these other areas — to be ready to adopt to these techniques?
JANELLE: That’s a really, really awesome question. I’d say probably the biggest handicap to adoption has nothing to do with the ability to use the techniques themselves so much as the vulnerability issue. There are organizations where developers are in a lot of pain from just constant, not use the word abuse but in some cases, I would definitely call it abusive. But sometimes a lot of the abusive environments when you find out what’s going on, it’s more ignorance than anything else. But that said, being vulnerable and feel like, if you collect data that is going to be used against you, is a tough challenge.
REIN: I’m so glad that you’re talking about this. One of the things that characterizes these sort of lower function in organizations is the way they use blame.
JESSICA: Right. I shudder whenever somebody says the phrase ‘root cause’, ‘we need to find a cause of this’.
SAM: Where root cause means, “Where were we when we decided that we had spent enough time trying to figure this out?”
SAM: Rein and possibly our listeners, one interesting way of framing this organizational question that I have seen and it seems really useful is to say the idea that Jim Shore and Diana Larsen have been working on for a couple of years, which is this idea of Agile Fluency. They have this model of different phases that an organization will go through in its adoption of Agile and the characteristics of the work that they do and the way that they do it. You can look it up, just search for Agile Fluency, Jim Shore, Diana Larsen and you’ll find some of the stuff they’ve written about it. It’s quite useful.
REIN: It’s interesting to me that this concept keeps reappearing back in, I think the 70s or maybe the early-80s, Philip B Crosby created a concept of the quality management maturity grid and he rank organizations on a maturity scale that started with these different cultural patterns that they work under, starting with one that he called ‘variable’, which is basically the point at which developers realized that they can’t go at it alone and that they need managers but they still hate managers.
SAM: [inaudible] on that?
REIN: This concept of what are the levels of maturity or the levels of capability, I’m not a fan of maturity necessarily because I think that if you were writing a shell script to manage your whatever on your machine, you don’t need this big Agile process. You can have a relatively immature process and still be fine. I think it’s more about competence or how challenging are the problems you want to solve. But anyway, this idea of these different levels has been around for a long time.
SAM: You’re saying is that maturity has a context just like everything else?
REIN: Yeah, maturity is sort of a loaded term but I don’t think it is.
SAM: Yes, it’s very judgmental.
JESSICA: Speaking of context, do we have a listener shout out today?
SAM: Wait! How does that have anything to do with context?
JESSICA: This is the context of our listener shout out.
SAM: Very well.
REIN: Well played.
SAM: Listeners, we are a 100% listener-supported show. If you like what we’re doing and you want to do more of it, you can donate to us at Patreon.com/GreaterThanCode. A donation at any level will get you into our Slack community and if you donate at the $10 a month level, you will get a shout out at least once on the show. Today’s shout out goes to Paula Gearon. She’s @Quoll on Twitter and she describes herself there as Miffy to her friends, a semantic web programmer, Clojurer, physics enthusiast, martial artist, triathlete, musician, spouse and parent but not necessarily in that order. I spoke with Paula a few times in our Slack community and she’s got some really interesting things to say. If you want somebody else cool to follow on Twitter, go check her out. Now, back to our show.
JESSICA: Thank you. We talked about the conceptual model that we need in our brains but Janelle is building a conceptual model of our brains in code.
JANELLE: Yeah, I’ve got a long way to go on this project. We’re just getting started but I’ve been working on the architecture of how to do it for several years now of basically synthesizing a bunch of research, cross discipline and cognitive science and tangent arenas and figuring out how to model the recursive metal loop structure of consciousness.
JESSICA: How many levels of meta do you have to get to from that one?
SAM: One more.
JANELLE: It’s like the invention of metaprogramming for [inaudible]. Once we invent the meta, it becomes this recursive meta thing in the same way of abstraction works in code. You can keep doing it over and over again and each innovated level of abstraction, it takes some thinking to figure out how to implement in another layer but once you have the ability to go meta, you can just go meta-meta.
SAM: Now, I want to frame cognitive biases as leaky abstractions.
JESSICA: They totally are. We’ve got a category in our brain, we’ve got an abstraction somewhere in there and it’s not accurate as no model is.
SAM: Well, I was thinking about this morning. Some of them like the unit completion bias, this is the idea that for example, we are strongly biased to finish the food that is on our plate even if it’s more food than we actually need right now. I mean, that one has a clear evolutionary basis in that. If you are inclined to stop eating just as you get full, then you might not survive until you find your next trove of food. But there are other ones that I don’t know where the heck they come from.
JESSICA: Now, you mentioned hindsight bias earlier, would you define that?
JANELLE: Sure. It’s when something looks preventable in hindsight that in the context of the moment of the decision is actually not preventable.
JESSICA: Ah, yeah. We tend to judge decisions based on their outcomes as opposed to the information available at the time. The latter takes a lot more empathy.
SAM: Yeah, more so when we’re talking about other people than versus ourselves, which is another bias.
SAM: How much do you hate brains after thinking about all of this? Or rather how much do you hate your own brain after thinking about all this?
JANELLE: How can anyone hate brain? The most fascinating thing in the world is ourselves, right? You just like to think about how cool brains are.
SAM: That’s true but as somebody with ADHD, there are times where I wish my brain worked a little bit differently than it does.
JANELLE: Yeah. I found that I’ve just sort of learned to roll with it, even just in the context of this conversation, I’ve had times where I get excited on a thread that my brain goes somewhere else and I’m like, “Oh, lost my thought.” That happens to me a lot. What I found is that there are some things that I’m really good at because my brain is sort of wired weird. I’m really good at connecting metaphors and being able to synthesize across totally different domains. Yet at the same time, there are some basics functioning stuff, I really, really struggle with so I’ve had to find friends and people to fill in my gaps in life for things that I’m not good at and then I kind of roll with what my brain wants to do and be good with it. We all get dealt a different hand of cards and the body we live in is a constraint so might as well enjoy yourself.
JESSICA: Yeah, it’s important to recognize your own constraints and they’re not the same as everybody else is. Also, let yourself be bad at the things you’re bad at. I figured I need to sort out my weaknesses exactly far enough to hand it off to someone else.
REIN: Can I ask you a question?
JANELLE: Of course.
REIN: Yay! Once you come up with a model for this system that you’re working with, you understand the pain points or friction points where you have a good model of the system, how do you decide what to change when you’re trying to improve it?
JANELLE: In addition to the picture of pain, which I basically found wasn’t enough and when you start measuring the pain, you still have the same problem of how do I improve? This ultimately turned into a bunch of principles about what are the likely causal factors of pain in these circumstances. This is where I started talking about the thickness of the code sandwiched as the cause of diagnostic difficulty and if you can attribute something to that model like in observability problem in your code sandwich, then you’ve got a more tractable thing to do in an experiment to run then to see if improving observability in this way reduce your diagnostic cycles.
I also focus on feedback that’s risk associated. Part of it is you try stuff and learn. But a lot of it ends up, after you start to start to see the patterns of what works which are largely informed by the body of knowledge that we have in our industry around best practices but kind of getting back to first principles of those things of like continuous integration and problem breakdown and agility and usability and all these things that we’ve largely learned, you start seeing and learning the situational context and all the nuance to how all that applies and then have a feedback loop so you can tell if the things that you’re doing actually made things better or worse. Then that kind of helps you to clarify the situational context roles around that.
REIN: I like that you mentioned risk, for a number of reasons but one is that it’s a good example of the importance of my most basic heuristic for making changes, which is you have to do better than random chance.
JANELLE: I like that.
SAM: You’d like to do better than random chance?
REIN: You might think that would be easy but I’ll give you a good example of when it’s not. Let’s talk about deciding which risks to take on in the project in terms of some features or bugs that will be harder or more risky to solve or could have bigger downside impact. We have a natural tendency as human beings, I think most of us to tackle the easier stuff first — the low-hanging fruit — get that out of the way. But if you do that what you’re actually doing is making the project more risky over time, where if you were just to allot those decisions randomly and randomly picked a thing to do, you would spread that risk out over time. A literal example, where doing something randomly would be a better choice than the actual choice that you’re making. Not you, obviously but the general.
JESSICA: This is a property of complex systems. There’s simple systems that you can see very clearly the cause and effect and then there’s complicated systems that are so random like molecules in a room that don’t affect each other. Then there’s something about complex organized systems — someone on Twitter, please remind me of this phrase — that there’s a zillion parts and they affect each other. Like Rein said, you’re not going to get complex adaptive. I don’t now but you’re not going to get random chance. You’re likely to get something a lot less predictable than that.
REIN: Well, you’re sort of talking about our chaotic systems, I think maybe, where actions have unexpected effects.
JANELLE: Yeah, chaotic path dependent ones as opposed to actual random ones like the molecules.
REIN: Anyway, when I’m deciding what do I want to do to change the system to improve it, one of the questions I ask is what if I made a random decision? Could that work? Sometimes it actually can and the next question is if I’m going to make an intentional decision, it has to do better than that.
SAM: That’s interesting that you chose that example of sorting for low risk versus randomly selecting risk. One of the things that I’ve slowly been learning over the past 20 years or so is this idea of that I will be better off if I try to identify the highest risk part of the system first and try to work on that so I can mitigate that risk, and only then will I know if it’s worth pursuing the rest of the thing. I don’t always remember to apply it. Sometimes, I just have habits or I have too much fear and I go back and I work on the little things that I feel like I’m safe dealing with. But more and more, I’m starting to recognize that I have to start with the greatest risk.
REIN: It’s interesting, front loading risk is actually a good strategy for a number of reasons. One is that the overall risk goes down because the most risky things are no longer there so the remaining things are less risky. The other is that if things do go bad, you have more time to correct them.
SAM: Or you can abort more quickly without spending as much money and resources on it.
REIN: Yeah, so it turns out that the strategy we choose intuitively is actually the pessimal strategy.
JESSICA: Did you said, “The pessimal?”
SAM: As opposed to optimal.
SAM: As like optimist / pessimist.
JESSICA: As opposed to whack-a-mole?
REIN: Yes. That will do.
JESSICA: Which is also how we usually work. Janelle, you said something about, going back to the first principles of best practices which there’s a lot of buzz words in there, does that mean getting back to why the best practices are good in the first place?
JANELLE: Well, I was going to say yes but it’s the why behind why we came up with the best practices. In the context of when these things were applied, why did they work and how do we replicate the why? Even if we don’t replicate the best practices.
REIN: Yeah, I again, have to bring up Kent Beck because what he described patterns was as a resolution to a specific set of tensions.
SAM: Yeah and it’s interesting. Many years ago, I was listening to a really interesting interview with Kent about his writing the book Smalltalk Best Practice Patterns and he said that the way he wrote the book was essentially he would sit down and he would write a line of code. Then he would think about why he wrote that line of code that way and sometimes spend three or four hours teasing out all of the various factors that went into that and the result, essentially was this book.
REIN: There is a certain level of… I don’t know if you’ve used the right word there but think that whatever decision I happen to make in the moment is the best way to do something is interesting. I think in the book, I love the patterns. I just love that was his concept which was whatever I did was the best thing to do.
SAM: Or if it wasn’t, what would be better, right? but I think it would take a lot of humility and honesty with oneself to go through that process, to have that sort of level of meta awareness of first off why did they do that and then yes to address the question that you raise, Rein of was that the right thing to do, and if not, why not?
REIN: I don’t think that Kent Beck has a problem with —
SAM: Sure. Yes, that’s true.
REIN: I feel like, maybe we could drill down a bit deeper into — maybe the answer here is to read the book — what is this process of identifying the pain in a project like? What do you do? What are the actual steps?
JANELLE: The tooling integrates with your IDE environment and measures your pain in the context of an experience. Then process-wise, before the test starts, spend 15 minutes going for a walk, one of the people on the team is doing the task, the other person is the inquisitor that ask questions and helps to dig into risk and help thinks through the risks. Then come up with kind of a rough initial strategy: what you’re planning to do and a plan to mitigate the risk and then do the task and measure your experience.
Then after the task, do the same thing going on a 15-minute reflection walk and look at the initial risk that you set out to mitigate versus what actually happened and where the pain points were coming from in that context. Then tagging the particular incidences with hashtags and note that describe like what the causes were, what could be better, what suggested refactorings might help with improving the development experience for that, question that you want to ask yourself next time to try and think of things that you didn’t think of this time. It’s trying to get out of that kind of hindsight bias by putting actual actionable things forward.
Then if you have for example, an improvement idea, say for example you want to refactor the code because you believe it’s going to have this sort of effect on usability for an upcoming story, how can we run a what I call, prefactoring experiment where you basically has front load the little refactoring part to be able to see what that experience is going to be like in the context of that actual change so you can learn a bit about it. After you start tagging all these things, you can add up the different problems by tag type and get a feel for where your problems are and then use that as guidance for driving improvement. You kind of slice and dice and aggregate things in different ways.
There’s definitely situational context involved with things but the nice thing is since all of the metrics are measuring a deviation from a norm, from how much risk you’re accumulating in terms of frequency and magnitude, it’s really easy to aggregate the data and then slice and dice by primary dimensions, which all written up in the book about what’s slice and dice-able model-wise but we’ve got it boiled down so that you can actually make statistical models and do statistical process control on developer experience data, which is what we’re building the tools around to be able to do.
JESSICA: This is super fascinating. When you slice and dice, do you compare it to more data from your own team or developers in general?
JANELLE: It’s interesting because it’s definitely in general within your own team, to an extent but there’s a lot of things that are fairly generalizable that we’re looking at aggregating data in much larger scale for. For example, we share a whole lot of the same infrastructure and libraries and cross what is the learning curve of Angular 2. I’m learning that right now. It would be nice to have, for example as opposed to having a problem and your only option out there being trying to find something on Stack Overflow, you’re trying to evaluate learning curves or the problems that people have run into in the past and having all of the experiences and problems and bugs that people ran to, all cataloged in a big database so you can just search for Angular 2 and get an overview of learning curve and amount of friction involved.
JESSICA: That’s really fascinating.
REIN: Or we could have a heat map of API documentation where people were like, “This is annoying. This doesn’t make sense,” and then I could look at, it’ll just show up in red where the bad documentation is.
SAM: I’m sorry, I know you said ‘heat map’ but it kind of sounded like ‘hate map’ but I like that too.
REIN: Oh, let’s go with that. That’s good. Maybe I could trademark that.
JESSICA: That’s interesting that we could compare developer experience across libraries, across tools. You can also do it across changing techniques within your team or try to retro.
JANELLE: Yeah. There’s definitely some statistical significance questions in there on how you kind of slice and dice and break up the data but it seems like with the type of things that you need scale for, you be able to get statistical significance in scale without having so much noise in there that you can’t interpret the results.
For example, if you were looking at learning curves, you have to factor in a lot of other kinds of contextual things to be able to, for example how long have you been working with this framework already. How significant impact on what your experiences with it at that moment. You’re working on something unfamiliar or familiar, you’re going to see something totally different.
JESSICA: Yeah, there’s those different definitions of better again.
REIN: Do you have anything on change sizing? What I mean by that is how big of a change or how many things should you change at once?
JESSICA: Sounds like haystacks.
JANELLE: I think it’s one of those things that is a gray-relative to the kind of thing that you’re doing. In general, we want to keep haystack small. I think it’s a pretty good principle. That said, I think there’s a matter of practicality to that and that batch size creates efficiency to some degree. If I’m changing something and there’s a whole bunch of other effects that I’m not going to lose that ability to see cause and effect, I’d say that’s probably the best guidance I could give on haystack sizes: how it’s going to affect your ability to experiment and track down isolate —
REIN: I’m always looking for better mental models for this because one of the things that I’ve noticed is that the teams that need the most help, you often have to make the smallest changes because their systems are so chaotic that it’s hard to predict or evaluate a change. That’s all I had about that.
JESSICA: As a temporary reflections?
REIN: Yeah. Okay, I’m going to do my reflection. I mentioned earlier in the show a book by Gerald Weinberg and what originally happened was he wrote this giant tome on what he called Quality Software Management and his publisher was like, “That’s 1500 pages or something. We can’t publish that,” so he ended up splitting it up into, I think six or seven books, all under the heading of Quality Software Management and there were about systems thinking, understanding feedback mechanisms, the topic of resulting bugs that I mentioned it’s just one part of a book on understanding why software fails.
All the books are great but what they mostly point out for me is that this problem has been around as long as there has been software and it keeps showing up. People write these books and I don’t know what the problem is. Maybe we need to actually read the books, maybe that’s the problem. But this problem has been around for a long time. I’m really excited to see new work in this area. I think that the concepts in your book are fascinating and are quite different.
In some ways, they’re very complimentary but in some ways quite different from the concepts in this book. I think you could read your book and then also read these books and get a good, broad overview of how we’ve been thinking about these problems for a long time.
JANELLE: On that theme of plugging other people’s books, there was one book that, again was written 25 years ago by Peter Senge, The Fifth Discipline: The Art & Practice of The Learning Organization. It’s another whole area that is so rich with incredible knowledge that is very applicable to software development of the discipline itself of learning and codifying knowledge and sharing it with others and being able to think together as a team.
It’s one of those books that fundamentally got me obsessed with this whole learning discipline dynamic of figuring out how to pull off what sounds like organizational utopia. Yet, when I talk to developers at conferences, almost nobody has read this book. The Fifth Discipline needs to be in the realm of software stuff on everybody’s reading list.
REIN: Yeah, that’s exactly what Gerald Weinberg books were for me when I discovered them 20 years after they had been written.
SAM: It’s so funny, I was actually just about to jump in with a recommendation for The Fifth Discipline. I should probably go re-read it. It’s been long enough that I’ve lost a fair bit of it but for our listeners who may not have heard of it, this was described to me as the Systems Thinking 101 textbook. What I remember getting out of it, the first time I read it was it has this really interesting visual language for describing complex systems with multiple different feedback loops so that you can get a handle on what’s happening and maybe use that understanding to figure out what might be the most effective thing to change next.
But going back to the conversation, I think my takeaway is I’m going to have to listen to it about three or four more times to make sure I get everything. But one thing that really struck me as we were going through was this idea that the pain that we experience in software developments is really cognitive dissonance. That really rings true to me and it’s really useful reminder for me to, maybe stop and think the next time I’m going, “God damn it,” and stop and try to unpack exactly what it is that I’m feeling because as humans, we have this tendency to resolve cognitive dissonance in the fastest way possible, which is almost never the most effective way but that’s just what we do. That was a really good reminder to me. Thank you.
JESSICA: One thing that came out today and was important in Janelle’s book was about how important it is to have an accurate model of the system we’re building in your head. I’d like Sam’s description of programming as like summoning a demon, “If you don’t get the incantation just right, it’s going to jump out the circle and eat you like a goat,” and that conceptual model is like in order to summon it even correctly, you have to hold it in your mind just right. Yet, to summon a really big demon like a system that is big and complicated and does all the stuff that we want our software to do these days, you have to have a whole team of people holding that demon in our minds and it has to be sufficiently accurate in all of them and that’s really hard to do.
The only way we can know whether we are all holding the same demon in our mind before we activate the summoning and it eats all of us is to do this little experiments. We have to devise ways of knowing, whether what we’re doing is actually having the effect we want. One fascinating thing about Janelle’s book is that we can ask those questions, we can do those experiments on ourselves, not just on the code. As we get better at it, well as we do more these experiments, then we get the demon more cohesive in all of our minds and then we really get faster without the scary danger.
REIN: I’m glad that Sam and I were both muted earlier when we were talking because we’re laughing because you should now be awarded for your goat-shenanigans.
JESSICA: Baaa —
REIN: Okay, fine. That’s funny.
JANELLE: The goat thing, it wasn’t expected today but there it is.
JESSICA: It’s really important that we give our listeners a way to find out more about your work, Janelle.
JANELLE: All right. My book is available on Leanpub.com/IdeaFlow, website: OpenMastery.org, Twitter: @JanelleKZ. I think all of those will provide links to more stuff that people can read if their interested in finding out more.
JESSICA: Fantastic. Thank you to all of our listeners and we will see you on the next episode of Greater Than Code. Maybe also on Patreon and on our Patreon-only Slack.
This episode was brought to you by the panelists and Patrons of >Code. To pledge your support and to join our awesome Slack community, visit patreon.com/greaterthancode. Managed and produced by @therubyrep of DevReps, LLC.