#begin

Hi there again! Today we are going to continue with a blog about another chapters in the book called A Philosophy of Software Design by John R. Ousterhout. We are going to dive into my favorite chapter of this book so without further ado, let’s check it out.

Modules should be deep

John says that one of the most important techniques for managing software is modular design. It allows engineers to manage complexity by only focusing on a small fraction of it. The goal of modular design is to decompose a system into a collection of relatively independent modules. And a module in a C# or OO sense could be as small as a class, but also components or entire services or sub systems. The goal of such modules is that programmers or engineers can work with these independently of other modules and thus do not have to know the other modules. This way you are able to manage complexity of game development or software engineering overall. There will however be some dependencies between those modules, yet they will be abstracted out and you as a developer do not need to worry about those that much.

He then continues with the bold observation that a module consists of two parts; an interface, so an API to talk to and an implementation which is hidden behind the interface. This is what Uncle Bob talks a lot about as well. We always want to make nice interfaces and hide internal implementation by proper encapsulation. As a consumer of a module, you are often not interested in the internals. The interface described what the module does, not how it does it. This is a nice way of hiding complexity of things.

John also describes two interesting elements of an interface; there are formal and informal parts of an interface. Where the formal part of the interface relates to the specific API, or function signatures. These specify names, types and parameters, return types and maybe even information about exceptions. These are what I would call the obvious parts on an interface. But there are also the informal elements of an interface, which I would call the hidden properties, or in programming terms; functions with side effects. He says that there are some elements that are not specified in a way that are understood, or enforced by programming languages. These functions often target high level behaviour. The example he gives in the book is one where a function “DeleteFile()” would delete the actual file from the file system. This is a side effect which will impact the subsequent calls to that function. But there are also other kinds of side effects, like when functions need to be called in order.

I think we have probably ran into this issue once in our careers. I bet, that you at some point fixed a bug, by just changing the order of function calls. Sometimes it can be really difficult to express these kinds of things in an interface and you might need some knowledge of the domain you are working in, in order to understand how the interface is supposed to work. For example, in your game, you might not be able to enter the character creation screen, or any other part of the game, before you have confirmed your email address. This is pretty hard to express merely by an API. You need some kind of documentation for this.

And I think he made some really nice observation here; the fact that the informal, or unknown aspects of an interface are important to communicate somehow as well is very cool. After reading this again now, I’m definitely going to try and communicate such things in a better way. I’m not entirely sure how to do this properly, but in a use-case driven approach it would be fairly easy to express I think. If we go back to the previous example where the user needs to confirm his email before he can enter the character creation screen, we could embed this logic in the account registration usecase. Only after the registration use case is fully done and the email is confirmed will the user be able to continue to actual in-game content. He might of course in the mean time access the menu’s and stuff but no actual content. And this would also be pretty easy to restart when the user closes the game. When he comes back, check his status, and if he’s not confirmed his email, run the confirmation usecase again. Note that the usecase objects embeds, the informal elements of the interface now.

John then continues with an interesting definition of abstraction and he says: “An abstraction is a simplified view of an entity, which omits unimportant details”. Notice how he specifically says “unimportant details”. This is really nice and practical information. How often have you as a developer been bothered with “unimportant” details of some interface. I know I have. But I unable to come up with decent example from the top of my head.

He also highlights the fact that “unimportant” is crucial in his definition of abstraction. He says that abstraction can go wrong in two ways; 1) it can include details that are not really important. When this happens, the abstraction is over-complicated which increases the cognitive load on the developer. And 2) when an abstraction tells you not enough about it, so the abstraction creates obscurity and developers looking at the abstraction will not have the information they need to use it correctly. And the perfect example I know of this is the WWW routines in Unity3D. They are supposed to be HTTP requests, but you can abuse them for request to the file system as well. And as a matter of fact, you are required to use them when you need to get assets from the streaming assets folder on mobile platforms because they are compiled into binaries for optimization and distribution purposes. So you can use these multi-media requests for multiple usecases, which makes them flexible or versatile but also confusing when you are starting out with Unity3D. John says that: “the key to designing abstractions is to understand what is important, and to look for designs that minimize the amount of information.”

He then goes on to describe on what I think are one of the most important concepts in this book; and that is the notion of deep and shallow modules. He starts of with describing deep modules. John describes such models that provide a nice, concise interface but do some really heavy lifting under the hood. He also puts a really nice visualization of deep and shallow modules in the book. Where the deep module is very narrow, but well, deep, and the shallow module is very wide yet shallow. These boxes also have their interface visualized by some tiny little top header. Which in the deep module is very narrow, yet in the shallow module is very wide and thus the shallow module provides a relatively large interface compared to the functionality it provides. The deep module on the other hand provides a relatively small interface compared to the functionality, the heavy lifting it provides. If this does not make sense, go check that youtube video out I referenced earlier. These visuals are in there as well.

And next is something I really wanted to put in the blog and that is john’s all time favorite interface. I just could not leave it out, he speaks so passionate about this in any of his talks and that is the file I/O mechanism provided by the Unix operating system it only has 5 functions, Open, read, write, seek and close. But this interface does a lot of heavy lifting like how files are represented on disk, how are directories stored and what paths do they refer to, how are permissions enforced, how can file accesses be implemented like interrupt handlers, how are scheduling policies implemented to allow for concurrent access, how are caches implemented to reduce overhead when accessing files multiple times, how are secondary storage devices like external disks and flash drives incorporated into a single file system.

And the list goes on and on. I definitely encourage you to check him explain it himself because you can see how passionate his is about this stuff. It’s really cool.

So this interface, providing these 5 functions is very, very deep. It is a simple easy to understand interface and yet it provides you with all of this functionality. This is the very best description of a deep module.

So let’s then have a look at what he calls shallow modules. And these are modules whose interface is relatively complex in comparison to the functionality that they provide. He gives an example about how a class that implements linked lists is shallow since it does not take much code to manipulate a linked list like inserting or deleting elements takes only a few lines. So a linked list abstraction does not hide much of the details. He also raises the first red flag in the book which is to look out for shallow modules. And that’s because shallow modules do not just do not provide enough functionality to justify their existence. It might be even that you need to spend time learning its interface before you can use them. And, well. I heavily disagree with him on this one. Although some modules are very, very shallow does not mean you should remove them. He gave the perfect example himself, would you remove the linked list class from the language just because it’s shallow? I wouldn’t is a very nice utility to have. And I can give you more examples. A very simple one would be, in C# we have this function called string.IsNullOrWhiteSpace. This is very very shallow, yet very useful. Are you really going to write string == null || string == string.empty? Really? Or maybe another example, the trygetvalue function on a dictionary. It’s really shallow, you can easily write it yourself but don’t you like this utility?

When you create functions for this you can also cover it with unit tests to make sure it works properly. So cover the specific function with some tests, to be sure it works every single time. Instead of writing the same unit tests for everything where you inlined this functionality. And then another reason why I heavily disagree, and this comes from the clean code book directly and that is, functions serve as documentation. So a function called IsNullOrWhiteSpace documents what the line is doing. This is polite remember? When you read this, you don’t need to figure out whatever optical illusion conditional logic is behind it.

But, OK, He’s right that many shallow modules are simply badly designed objects with complicated interfaces. But there’s a difference between, useful utility functions and objects that serve as a remedy against complexity through abstraction and documentation and simply a badly designed class. But take this advise with a grain of salt, remember there are often trade-offs or nuances! But let’s continue I think I ranted long enough haha.

He then continues with a concept he calls Classitis which happens when people break things up in far to many little classes than is actually needed. He says this comes from the advice that classes should be small, and functions too. So they should be divided into multiple classes and multiple functions. And he makes the perfect observation here and that is he says this stems from the view that “classes are good, so more classes are better.” and thus all these classes contribute to the complexity of a game ore system. This really contradicts the advise in clean code, right? Well, to be honest, I’ not so sure. The feeling I got when I read this part is that some people follow the “classes are good, so more is better” religion to heavily. They misunderstand the Single Responsibility Principle and abstract far to much than is actually needed to comply with the SRP. They naively implement the SRP in their systems and thus complexity accumulates. But the SRP is pretty straightforward, extract things out that answer to a different stakeholder. And a stakeholder can be many things like a webservice, database or some other system in your game. When you separate them you are less depended on them and thus take less risk when something needs to change.

But, he also gives an example of this and that is the Java FileStream API. And I think he has a point here haha. This is a horrible API, and it seems to me, that the SRP has not been implemented properly here. In the example he explains how you can create filestreams, but if you want to make it buffered filestream, you must feed that stream into a buffered stream and if you want to read and write to it, you must feed that again into an objectstream. But, this is the common usecase so it should be the default. He has a point here. But my comment would be; this is where so called “shallow” modules could help you with some very simple extension functions. And on the other hand, I think this is also just the nature of the Java stream API. I think their design goal was to make a pluggable stream system, where you can hook different kinds of streams together to create like a pipeline. But He’s right, it’s a horrible API and since Object streams are of then the one you want they should be default.

But that’s a wrap again! I think this chapter was really nice and contained really valuable information. There are some contradictions with clean code in here which I tried to explain, and give you my opinion on. Let me know how you think about this I’m pretty curious about that!

#end

01010010 01110101 01100010 01100101 01101110.