#begin

 

Debugging

The next section of the book is all about debugging. We have talked about debugging before in earlier episodes. As Uncle Bob puts it: “Debugging is not a skill to be desired!”. And I totally agree, however, you must be able to do it. And in this section Andrew and David will talk about the psychology of debugging and the mindset you need to have to do it effectively. They will also explain, and coin the now world famous term “Rubber Ducking”, at least in the software world that is. So let’s not waste any more time and dive into it.

They start this chapter off with a little history lesson about where the word bug in a software context came from. It was of course Rear Admiral Dr. Grace Hopper! Back in the early days of computing, a computer was the size of a room. And Grace Hopper observed that the computer was misbehaving and found that the cause of that behaviour was a Bug being stuck in the electronics. So there was a literal “bug in the system”. By the way, Grace Hopper is also the inventor of the COBOL programming language but maybe more importantly, the inventor of compilers. She had the idea that we as humans don’t need to write actual binary or assembler. We can write at an higher level of abstraction and turn that into machine code by means of automation aka compiling. This lead to her inventing the COBOL programming language in the first place. Cool right!? She’s a true legend!

In modern days, we still have bugs in our systems, but they’re some other kind of bugs. They manifest themselves in different ways like misunderstood requirements, flaws like critical coding errors or mistakes in form of bugs. Even now, compilers, linters or static analysis tools are rather limited at finding them. Well, maybe ChatGTP is able to help you hunt down the bugs. I haven’t tried that yet but it sounds rather interesting just thinking about it. We also have to accept the fact that most, if not all software suffers from bugs, however strict you process might be. Even dong TDD, bugs can slip through.

 

Psychology of Debugging

Andrew and David start the next section about the psychology of debugging off with a very nice little paragraph:”Debugging itself is a sensitive, emotional subject for many developers. Instead of attacking it as a puzzle to be solved, you may encounter denial, finger pointing, lame excuses, or just plain apathy”. Apathy means lack of emotion or interest by the way. And they are right, aren’t they!? I mean how many times have you been tracking down bugs and when you finally found it you check the git logs who put the damn thing in there? I know I did! And often not for nefarious reasons. Sometimes I just ask the person who wrote the bug into the code for some context so we can make a proper fix instead of just treating the symptoms. So not necessarily finger pointing. But I can remember situations where that did happen.

You should just embrace the fact that debugging is another form of problem solving. This is something we as developers are really good at, and thus we should attack it as such. When you find a bug, you should not blame the person! You could however ask him or her for some additional information about the piece of code so you can fix it. So focus on the problem, not the blame! Especially when dealing with devs more junior than you. Don’t scare them off by telling their code is shit. Maybe even pair with them to fix the problem so they can see how you fix it and they can enjoy some extra mentoring.

But let’s talk a bit about the debugging mindset next, as it is the next section.

 

A Debugging Mindset.

According to Andrew and David, part of the debugging mindset is to turn off, or at least lower your defenses that protect your ego and tune out any pressures you are under. And of course get yourself comfortable because this could take a while. Above all, remember the first code of debugging: Don’t panic! Haha, yeah.

Sometimes other people will breath down your neck while you are trying to find a fix for a bug. This can be very stressful and cause people to panic. Haha, this reminds me of this meme that says: “How not to write bugs into software”, with an image of North Korea’s leader watching over the shoulder of some developer! Unfortunately, I guess there is some truth in that picture.

But it is important that you do a step back, breath slowly and actually think about the problem in your own pace. Next they have a really funny little paragraph: “If your first reaction on witnessing a bug or seeing a bug report is “That’s impossible”, you are plainly wrong. Don’t waste a single neuron on the train of thought that begins “but that can’t happen” because it quite clearly can, and has”.

Haha, recognize this!? I certainly do. “I call bullshit!” haha. Yeah, we’ve all been here, right? Just remember that QA is just doing their job, they’re not out to get you.

 

Where to Start

Next they give us some tips on how to start debugging: they say that before you start, make sure that you are working on code that compiled cleanly, without warnings. HAHA, compiling without warning in a Unity3D context is the utopia you will never reach! But, I guess, this is true for ALL modern software since we heavily depend on 3th party code. But my advice for a very first step in debugging is: make sure you got the correct branch checked out, and are connected to the right back-end or database! How many times have you debugged some code for a long time without any luck of reproducing the error, only to find out you are on the wrong branch!? I know I have. You could also find yourself in the situation where you got the correct branch, but are targeting the wrong environment like dev vs. QA vs. Live. This still happens to the best of us sometimes. It sucks, but when we are not precise about these things, they will happen.

The authors also mention that they often set compiler levels as high as possible. Well I double dare them on that one in the current day and age. Maybe back in 1999 this was an option, but nowadays with all of this open source software, however great it might be don’t get me wrong, you simply cannot set the compilers warnings to the highest level because you will be overwhelmed and spammed by them. You can impossibly fix all of these things!

But on the other hand, if you are developing some kind of library yourself, without any, or many external dependencies you can pull this off. But when you are building a product, you often cannot.

The reason why they tune up the compiler warning as high as possible is because they don’t to spend time on problems, the compiler already found while debugging. David and Andrew say they want to focus on the really hard problems. The easy ones, are the ones found by your compiler through static analysis.

 

Debugging Strategies

The next section is about debugging strategies. I think it’s great to have a section dedicated to this topic by the way. They start of by saying the following: “Once you think you know what is going on, it’s time to find out what the program thinks is going on.” Haha, that’s funny right!?

Their first advice to visualize your data. Back in the old days, the values would simply be printed to some console. And people still use this approach. I do too. I prefer the debugger much more than the console, but if you need real time information, bound to the frame rate or maybe physics, Debug.Logging something to the console in unity comes in really handy. Just don’t forget to remove the logs before checking your code in and merging it to production. We don’t want to see all those logs.

A couple of years ago when I was building some Unity3D application that had to 24/7 in an amusement park we couldn’t allow anything to be logged to the console. Why!? Well because the logfile became so large that it would slow down and crash the system. So when the app was running for x-number of days it would crash all of the sudden. If you restart your game often, this might not be a problem for you. But if you truly need it to run 24/7, you need to reduce logging as much as you can in production, even your external assets btw.

But let’s get back to the book. Through a debugger you will gain much deeper insight into the data than simply logging parts of it, you think you need. How often have you debugged something, set a breakpoint somewhere and then inspected the data, only to find the problem lies somewhere else lower on the stack. I bet it more often is, than isn’t. Especially when you combine it with TDD, the debugger can become a really powerful tool. You won’t need it as often, and when you do, you will find issues quickly since all the rest of covered in tests.

Another strategy is to add tracing. Tracing are those little messages you might add into your code that say, “reached step 1” or “health = 100%”. Then while running the code you can see the trance, and know how “far” the code got during executing before crashing. Tracing can be really helpful in some situations but it’s still a primitive approach compared to modern IDE’s capabilities.

And next up comes the, in the software world, world famous advice on the practice of Rubber Ducking. Ow yeah. If you don’t know about this, listen well because this is really simple, highly effective and above all, pretty silly. So… have you ever found yourself in a situation where you have been debugging for hours, then ask a colleague for help, and while explaining the problem to him the solution to the bug just simply clicks in your mind, even before you are done speaking? I bet you have experienced this.

So Andrew and David came up with this cool approach called Rubber Ducking. Which is exactly the same as I described just now, yet you talk to your silly rubber duck on your desk. Yes, that’s one of the reasons why many programmers have random toys on their desk. They help us debugging. It’s true. I’ve got a little Flying Spaghetti Monster on my desk to talk to. All you have to do is to surrender yourself to his noodliness and all the answers you will ever need will appear right in front of your meatbally eyes.

Having to articulate and verbalize your assumptions will help you to gain new insights into the problem you are trying to solve. If you have never tried this approach, I strongly encourage you to do so. But I highly doubt anyone in the game or software industry for that matter haven’t heard about Rubber Ducking. It’s such a meme nowadays that you can’t possibly miss it.

 

Process of Elimination

Next is a section about the process of elimination. I think this is a great philosophy to tackle many problems. Just by ruling things out empirically, you can narrow down on the problem. This is also done a lot in healthcare, most notably nutrition. Sometimes, in order to find out what is causing someone inflammation or some other chronic condition, one might try an elimination diet to see what’s bugging them. No pun intended. So first they rule out dairy, then maybe nuts or seeds, next garlic and unions, next red meat or even veggies or high fat foods. In the end people might settle down in some Weston A. Price or Mediterranean diet or maybe even some forms of keto-genic, vegan or maybe carnivore diets. Whatever makes the person feel best.

You can take a similar approach to debugging. If you are debugging why some JSON serialization explodes, you can’t simply blame your JSON library because it’s probably battle tested. It’s probably the garbage you throwing at it that makes it explode. So don’t blame the generic software, find the blame within your own code because it’s just most likely there.

I do however remember working in University with this meta-programming language called Rascal. I’ve talked about this in an earlier episode as well. The thing with that language though, was that the language itself was still heavily in development. I remember that we wrote some list-comprehension that just did not behave as expected. In that case, me and my team members were convinced there was something broken in the language. But that doesn’t happen often, trust me. Maybe when you use a gazzilion NPM packages you will find a shit ton of broken code. Just remember that the Flying Spaghetti Monster is one ramen away.

Andrew and David have a funny sentence about this as well: “Remember, if you see hoof prints, think horses – not zebras. The OS is probably not broken. And the database is probably just fine.” I guess in the majority of cases this is indeed true. But there are exceptions to this rule indeed. Maybe you are working with the Unity3D beta program. Well trust me, you will most definitely find some weird shit in there at some point. Maybe in the IL2CPP compilation phase to just iOS 64 bit or something. I’ve encountered many obscure problems in beta versions of unity. So I’ve learned my lesson well, not to use beta in production haha. Only if there is truly no other way you might want to but, you should definitely way the pro’s and con’s.

Another thing while trying to fix a bug is that new problems might arise. Have you ever fixed a bug, only to create 3 new ones? I bet you have, I most certainly have haha. This happens sometimes when code is tangled up like a freshly cooked spaghetti monster. You fix one thing, and another thing falls over. Even worse, is when you fix something over here, but something totally unrelated over there breaks. That’s what Uncle Bob would call, a mess! You need to maintain proper coding disciplines to create clean code. We’ve talked extensively about this topic, so listen to previous episodes to find out how to write clean systems.

But there is another thing you really want to keep track of, and that’s breaking changes that could be the result of a bug fix. This has happened to me on numerous cases as well. Especially when writing back-end API’s! Haha! When you fix a bug, you find out you have to change some input parameter to a function, or maybe refactor the JSON schema. But at that point, it’s already too late since other stakeholders are already consuming that API. So you need to introduce a breaking change in order to fix it.

The trick here is to phase out the incorrect parameter. Maybe throw an exception when someone uses the parameter. Introduce the new one and later once consumers have had the time to correct for the old parameter delete it. You can add very nice [Obsolete] attributes to these parameters, or find some other clear way to communicate that the parameter should not be used.

But if you create breaking change as a result of a bug fix, and there is no other way around it. Make sure you postpone it till the next version. Don’t just patch it quickly. Make it a proper version so people will actually take notice.

 

The Element of Surprise

In the next section of this chapter, Andrew and David talk about the element of surprise. By this they mean the reaction you sometimes have that goes like this: “That’s impossible!”. Well, it’s not and the fact that it happened proves it is true. Never assume that code that has been in the system for a long time, is free of bugs. Because if you do, you could be missing it entirely. Maybe the old code that you assumes works, just never got triggered in this particular execution path. Remember that, especially in Object Oriented code, state explosion is a real thing and can cause lots of problems. Don’t underestimate this!

I remember a story by one of my professors in University while studying for my masters. He taught classes on model checking, testing and system verification. This was really interesting stuff, and I’ve learned so darn much from this single course. It made me look at software from a totally different view. But that’s maybe a story for another time. Anyhow, he was telling us how he got brought in to model check the software on large hadron particle collider, or maybe some part of it, at CERN in Switzerland. There was a bug in that system which they just could not find. They ended up model checking the entire thing resulting in millions, if not billions of states. Only to find out that events, did not properly bubble up to the surface, so adding the missing event-handler fixed the problem. Such a simple fix, for such a large system. I think he and a colleague also model checked the Dutch Delta werken, the famous dams in The Netherlands. Great stuff.

But to get back to the book; Never assume, it makes an ass of you and me! Well, jokes aside, don’t think that old code, that’s battle tested cannot have bugs. It has a lower chance maybe, but there still can be issues. So be vigilant, don’t assume but prove it. Find where the data is coming from. Maybe the data is corrupt, maybe the transformations that data goes through propagates problems that result in an error. Prove it!

And when you find the bug, make sure you know when and how it happens so you can write a proper fix; not merely something to treat the symptoms. So you know what to do, and how the code behaves the next time this particular execution path is triggered. This also goes for situations when the bug is simply a result of someone misunderstanding the requirements. Talk to him or her and explain how it should go. Don’t be too aggressive or confronting too. Remember, they were coding with best intentions. When people come to work in the morning, no one thinks; Let’s add some bugs into the code today. Game development is a team sport, so let’s behave like team mates!

 

Debugging Checklist

They finish this chapter with a short checklist of how to debug. It’s very useful and I’ll simply quote the entire thing since I think they describe it very well. So here we go:

“Is the problem being reported a direct result of the underlying bug, or merely a symptom?”

 

“Is the bug really in the compiler? Is it in the OS? Or is it in your code?”

 

“If you explained this problem in detail to a coworker, what would you say?”

 

“If the suspect code passes its unit tests, are the tests complete enough? What happens if you run the unit test with this data?”

 

“Do the conditions that caused this bug exist anywhere else in the system?”

And that’s all they have to say about debugging and yeah, that’s a boat load of information. I really like the way how David and Andrew go to such lengths to talk about this subject in depth. Great stuff!

#end

01010010 01110101 01100010 01100101 01101110