#begin

Code That’s Easy to Test

David and Andrew start this new section of with a description of people comparing software to an integrated circuit. Software components should be combined just as such circuit and such components must be known to be reliable.

This analogy is taken from the hardware world. Chips are designed to be tested, both in the factory, when they’re installed and even when they are deployed. More advanced chips often have a Build-in Self Test (BIST) feature that runs some base-level tests and diagnostics internally. Now isn’t that awesome!?

How great would it be that you could start your game in self-test mode to run some scenario and observe its results. Wouldn’t that be great!? Even your end-users could run this tests to validate their optimal graphics settings etc.

Unit Testing

David and Andrew say that unit testing is akin to chip-level testing for the hardware. Testing done for each module in isolation to verify its behavior. When modules are tested in isolation the tests will give you an understanding of how it will behave in the big wide world. Its very difficult to cover all scenarios and edge-cases but it will shape your understanding.

I don’t really want to get into the discussion of what a unit exactly is, but in the book they refer to a unit as a single module. So a module could be a class or a collection of classes, but not the covering an entire component. Yeah, excuse me for being vague. I think the best description of what I mean can be found in the ‘Clean Craftsmanship’ book by Uncle Bob. In there he writes tests for a I think its a movie rent store. The ‘Store’ class starts out simple. But as the requirements increase all kinds of classes get created which are hidden by the ‘Store’ facade class. The unit tests all go through the Store class as that’s the public API but the functionality is delegate to these internal classes. So the unit being tested is the Store class, yet the code under test also includes all of these internal classes. 

The deal with Unit Testing is that we test against expected outputs or against outputs from previous runs aka. regression testing. Unit testing is sometimes also referred to as expectation testing. I’m not sure why some people decided this was a better name but it might have to do with the discussion about a unit being difficult to define. But I do believe that by changing the name to expectation testing you loose the granularity of it. I can do integration tests on a very large scale and still call it an expectation test. But I guess you could do the same vice verse… But lets get back to the book.

And next up comes what I think is one of the more interesting topics on this chapter and that’s about what should be unit tested. And remember, TDD did not exist in its final form yet. David and Andrew say that programmers typically test random bits of the code and call it tested.

Testing Against Contract

David and Andrew think that code should be tested against their contract. Which makes total sense if you ask me. We need to write code, and tests that honor and strengthen the contract. Tests should tell us two things: wether the code meets the contract and wether the contract means what it thinks it means. We want to test that a module delivers on its promises over a wide range of test cases and edge cases.

Using this strategy we need to test subcomponents before we start testing the greater parts. Once the subcomponents are tested and are proven to uphold their contract we can move on to testing on the module level.

In regards to testing, the design effort of your code should thus be focused on two things; you must design its contract and the code to test that contract. Now this sounds a whole lot like TDD to me! By designing code to pass their tests and fulfill their contract a lot of things are necessary. You have to consider the happy flow in your system, but also all the edge cases that might occur. If you have a clear contract for your code this becomes easier since you have something to fall back on. Naysayers of TDD often tell you that TDD is worthless since you can’t test all edge cases anyway. And I partly agree since in essence they are right, but… If I have a large suite of tests, and some weird edge case slips through, I can add a new test to the suite that covers the bug and now for the entirety of the project, that bug will never occur again. Over time, your test suite will strengthen more an more and result in a robust code base.

And the last line in this chapter is a very cool one and I quote:”in fact, building the tests before you implement the code, you get to try out the interface before you commit to it.” So they also recognized the benefits of what we now call TDD. Pretty neat.

Writing Unit Tests

So first of all; writing unit tests needn’t be difficult. Modern off springs of the Junit test package have made testing very easy and also very clear. Fluid assertions have also made its way to mainstream which further increases readability. When you you such a framework, and for us in Unity3D it’s most likely Nunit, test become first class citizens in your project. They are located in specific folders and even your IDE has a special window or view for them.

David and Andrew say that test should not be shoved in some far-away corner of the source tree. They need to be conveniently located. And I agree, plus using modern test frameworks will help you organize your code properly. Also note that, since we mostly use the same test frameworks it becomes common knowledge where the tests are located and how they are structured, which further increases the readability and understandability.

They also say that its often convenient, but not practical that each class has its test counterpart. This is a very important concept to understand so lets discuses this. If you design your test suite so that each production class has its test counterpart, your test suite will suffer from the fragile test problem. You must not design your test suite like this because, if the production code gets refactored, your tests are directly affected by it. This will lead to many, many broken tests after each refactoring. Uncle Bob talks about this a lot to and I will refer you to the example I gave earlier about the video rent store. I will also refer you to one of the greatest talks about TDD by Ian Cooper called: “TDD where did it all go wrong”. Here he explains this concept. You should test for behaviour, not functionality. Kent Beck also talks about this in his book about TDD yet most of us, me included, misunderstood the concept of TDD, especially when you just get started. Being good at TDD requires a lot of practice. Once you master the skill you will recognize the fact that behavior is more important than functionality testing.

I can also give you an example. When I started to learn TDD I used to test each class with a testing counterpart. This was mistake number one. But mistake number two was far worse and that was that I would reach into the production code using reflection to access private members and functions in order to test internal logic. This made refactoring really difficult since there were no compile errors, only run-time errors. So, never, ever do this. Or take Ian approach of deleting tests that cover internals before you merge the code to master. So you have these dirty tests during the development of your feature, but once you check the code in to master you delete them because they are really fragile.

So now that we covered that, lets return to the book.

David and Andrew say that you should be able to test a portion of your tests and also with varying input. Fortunately this is covered by the frameworks we have now. But in the book they are talking about using different shell scripts to trigger portions of tests with certain input. So you’ll have this collection of scripts and data files to run your test suite. Yeah, I’m glad we don’t have to do this stuff.

Using Test Harnesses

Next up they talk about what they call a “Test Harness”. They say that a test harness is there to handle common operations like logging status, analyzing input and output of expected results, selecting and running unit tests. Yeah, they just described Junit and all its relatives to us. And I couldn’t agree more. They even refer to Kent’ Becks work here of early work on xUnit. Xunit is the family name for all the test unit packages like Junit, Sunit and Nunit. Andrew and David say that Kent Beck and Erich Gamma have done most of the heavy lifting by developing xUnit packages.

There are a couple of things such frameworks provide like; a standard way to specify setup and cleanup, methods for selecting individual tests or all tests, means of analyzing the in and output and the results that come from the tests, and lastly a standardized form of failure reporting. These frameworks are also built in a composable manner. You can add as many tests as you want. Partition them in any way you want and each group of tests can be tested separately, even in parallel.

But since most of the test harness concept is covered by our frameworks anyway, let’s move on to the next section.

Build a Test Window

Next we will investigate a section called build a test window. This boils down to having some sort of monitoring built in to your system. In modern systems this is pretty main stream. I mean we have this movement called open telemetry which is basically leading the way in the observability space. Ever since the micro-services trend, observability has taken a foreground position when it comes to quality attributes. More or less everyone is talking about it, at least in the cloud space.

But what about the gaming space? Well as a matter of fact, Unity3D has been hammering down on their analytics services a lot as well. There’s so much to integrate to get real-time information about your users. This can be really helpful when trying to optimize gameplay etc.

An example would be to add certain analytics events in your game at specific points. When you see that 90% of the players never come past one of them, its probably too difficult. So you turn down the difficulty on that point a bit to allow more people to stream through. I’ve never really used this in practice; mainly because I don’t build games and on another aspect is that the systems I’ve build in unity rely on proprietary cloud services due to privacy and security concerns. So we’ve had to build most of this stuff ourselves. Which is fine, and it gives you a whole new level of appreciation for it.

The second aspect of building a test window is that most software hides some kind of secret diagnostics window. Like in unity, on mobile, when you do a three-finger double tap it will show the URP debug window. Such windows cane give really valuable insights in production system. And, in lots of systems I’ve build we added such a window. We basically exposed the Unity3D editor console in a production build. This way we could see all logs, warnings and errors while running a production app. We also added custom commands to connect to another back-end or to clear the Application.persistentDataPath of assets that might be corrupted. This was really useful and I strongly encourage you to add something similar in what-ever software you are writing.

A Culture of Testing

The last paragraph of this section is about building a culture of testing. This is what Uncle Bob talks a lot about too. We want all engineers on the team writing tests, because is some of us don’t it will leave holes in the test suite. And when there are holes in the test-suite, we can’t really trust it… right?

So what you might do is add static analysis to your repo. When a pull-request is created, you run the static analysis and when the new code is not covered in tests with some acceptable percentage you fail the ci pipeline. We have this setup at work as well and it can be harsh sometimes but it will keep the quality of the product high. No body is able to merge untested code in to main line. Which is great. Because of this, we don’t experience hot-fix scenario’s often. Meaning we don’t have to quickly patch main line and being slowed down by our static-analysis complaining about test coverage.

#end

01010010 01110101 01100010 01100101 01101110