SjwagMeister

SjwagMeister is a library made for Unity3D to generate Open API (version 3.0.0 and up) clients. The goal of this project was to simplify and mostly increase the joy of implementing WebClients. There are both official and unofficial tools that generate clients for OpenApi but they use CLI tooling and often generate code that’s incompatible with Unity3D. So I decided to write my own generator for Unity3D, which has become SjwagMeister.

DevLog

Before I started development of Sjwagmeister I did some research about the options for code generation in C# (mono) in Unity3D. I didn’t want to use predefined .txt files for example because then I would need to do a lot of string replace magic to get things working. So for example a string like “public class {0}{}” should be formatted to “public class  MyClass{}”.

So I really wanted another solution. The coolest way to solve this would be the use of .Net’s source code generators. The good news is that Unity3D does support it according to their website. But for this tool to be any useful I really needed the installation part to be simple. If I was only going to use SjwagMeister in one project this would be fine. But I wanted to make a generic package that should be pluggable in many projects. If you need to do all this config every time people wont use it.

The next thing that came to mind were T4 Templates, which I vaguely remember helping an intern with once. He used these templates, in a Unity3D setting, for generating some boilerplate code. In fact, he used to to generate DTO files for communication where he generated the properties in public partial classes and allowed methods to be added in other pieces of the partial class(es). So since I knew the T4 templates would work, I chose that solution.

There is also 1 MAJOR difference between source code generators and T4 Templates, which is of course that source code generators are a run-time solution and T4 Templates are a design time solution. And since I really wanted to have something tangible at design time so T4 was just the logical way to go. I didn’t fall for the new shiny object this time :D.

 

T4 Templates

So T4 templates stand for Text Template Transformation Toolkit. T4 templates work with a domain specific language (DSL) to express certain constructs to, in this case generate text. Not that I mentioned ‘text’ and not code since you can generate what-ever you want with such T4 templates. You can also generate plain text, for configuration files for example, or JSON, but also C# source-files or any other language you want. T4 templates are pretty flexible but, they are a pain in the ass to work with. Part of the problem is the sub-optimal integration in your IDE but let’s explain it a bit.

T4 templates have little features but are pretty powerful if applied correctly. They offer the following functionality:

• • Directives: elements that control how the template is processed.
  • Text blocks: content that is copied directly to the output.
  • Control blocks: program code that inserts variable values into the text, and controls conditional or repeated parts of the text.

Directives

Directives are statements about how processing of the template is done (Note the angle brackets and hashtags!). A simple example is the output extension:

<#@ output extension=”.txt” #>

This tells the T4 Generator to set the .txt extension to the file when generated. You can change this into what-ever you want so if you are generating C# or JSON files you would change it to ‘.cs’ or ‘.json’.

Another directive you will most certainly use is the Import directive:

<#@ import namespace=”namespace” #>

This will import a namespace, just like with regular ‘using’ statements, to be able to be used as design-time, not run-time. So these import statements will not be set as usings in your generated file, but are simply accessible during generation! Really cool!

 

Text Blocks

Text blocks are the simplest kind of blocks. They simply directly output what-ever text is in there. A very simple example would be the following:

<#@ output extension=”.txt” #>
Hello

The above snippet will output a `.txt` file with a string “hello” in there. So it takes the raw literal string and simply outputs it. This includes any syntax of a programming language, like C#. So if you write “public class Foo {}” in a text block it simply returns that.

 

Control Blocks

Control blocks on the other hand are far mote interesting. These blocks are by far the most important to understand. With control blocks you can manipulate the way, and what exactly is generated. Let’s again take a quick look for an example:

<#@ output extension=”.txt” #>
<#
for(int i = 0; i < 4; i++)
{
#>
This is hello number <#= i+1 #>: Hello!
<#
}
#>

So within the <##> tags, you can actually write code that executes during the process of generation. The example above will generate a file that simply says “This is hello number i: Hello!” for 4 times. And the index i will be replaced by 0-1-2-3. by the expression block marked <#=#>. Cool right!? There are some more control blocks like feature class control blocks in which you can define functions specifically oriented towards the generation process. But I’ll leave these out for this log since I think I got the point across how powerful these T4 templates are.

However, this is exactly where the struggle with T4 Templates begins; the formatting is weird and off. This might not seem like a problem for now, but once your template starts to grow, the formatting becomes a real b*tch.

 

T4 in Unity3D

Now, for all this magic to work properly we need support for T4 in Unity3D. The cool part is that Unity3D supports T4 templates when the API Compatibility level is set to .NetFramework. I also noticed that for getting t4 to work I had to assign the mono compiler in my IDE.

To run these T4 templates in Unity3D you simply pre-process them so you can manipulate them by code. In a normal .Net project you wouldn’t necessarily need to do this, but for Unity3D I found this makes life a bit easier. If you pre-process them, you access them by code, and simply push data through the generation function “TransformText” and get the output. So if you write any Unity3D Editor extensions you can use that code to run the template.

So I now have a way to generate these T4 templates on the NetFramework API compatibility front, but I didn’t have a way to solve the same problem for .NetStandard. But There will be a way to solve this, which I will get back to later.

 

OpenAPI 3.x.x Spec

What the heck is OpenAPI spec!? Well I bet you have heard from Swagger, or Swagger docs, right? Well these swagger docs can be generated because the documentation of such API comply with the OpenAPI specs.

The cool part of any such swagger docs is that you can download the entire spec to your device since it’s just a JSON file:

In the image you can see the “/swagger/v1/swagger.json” hyperlink. When you click that link it shows you the raw JSON of the API spec. This is how you also can easily import such spec into clients like PostMan for example. If you have never checked this out, just go to your favorite API and click that link to inspect it. It’s really nice to just copy that JSON and importing it in PostMan so it generates all the endpoints for you. Check this open source API, which I used for testing purposes as well 🙂

Parsing the OpenAPI Spec

Since the OpenAPI spec is simply JSON, it means we can parse it like any other JSON file. Especially with more advances JSON tooling like Newtonsoft JSON. With Newtonsoft we are able to parse every like in a JSON string individually if we really wanted to. This gives us a lot of flexibility.

The idea is to parse the spec and gather the meta data needed to feed it to the T4 templates. We can do that because we have these control and expression blocks. You simply pre-process the template, which will result in a CSharp partial class file. So I extend that partial class with my parsed data, and now I have easy access to it. Then I can simply use that data to fill in the empty holes of the template and generate it.

Code Generation

My first goal was to start with generating the DTO, or “schemas” as they referred to in OpenAPI terminology. In such openAPI spec there is a field called “components.schemas” which is where the DTO descriptions reside. In the example below there is a schema called “Book” and it’s properties.

{
  "openapi": "3.0.1",
  "info": {
    ...
  },
  "paths": {
    ...
  },
  "components": {
    "schemas": {
      "Book": {
        "type": "object",
        "properties": {
          "bookId": {
            "type": "integer",
            "format": "int32"
          },
          "title": {
            "type": "string",
            "nullable": true
          },
          "author": {
            "type": "string",
            "nullable": true
          },
          "category": {
            "type": "string",
            "nullable": true
          },
          "price": {
            "type": "number",
            "format": "double"
          },
          "coverFileName": {
            "type": "string",
            "nullable": true
          }
        },
        "additionalProperties": false
      },
      
      ...
    }
  }
}

Fortunately not just get a property’s name but we also get all kinds of meta-data associated with such schema like the type, if its nullable, format and even if it’s some collection type like an array (, which is not the case in the example above).

The end result of the “Book” schema above looks like this:

using System;
using System.Collections.Generic;

namespace BookCartAPI_v1.Schemas
{

    /// <summary>
    /// Book
    /// </summary>
    public partial class Book
    {

        /// <summary>
        /// BookId
        /// </summary>
        public int BookId { get; set; }

        /// <summary>
        /// Title
        /// </summary>
        public string Title { get; set; }

        /// <summary>
        /// Author
        /// </summary>
        public string Author { get; set; }

        /// <summary>
        /// Category
        /// </summary>
        public string Category { get; set; }

        /// <summary>
        /// Price
        /// </summary>
        public float Price { get; set; }

        /// <summary>
        /// CoverFileName
        /// </summary>
        public string CoverFileName { get; set; }

    }

}

The generation process

So how does it work exactly!? Well, first I need to pull in and download the JSON that describes the OpenAPI spec of the API. As I mentioned before, this url of often publicly accessible. Currently, there is no support to access an authorized OpenAPI spec, but it’s low effort to add this functionality.

Once the downloaded JSON file is loaded into memory I will feed it into NewtonSoft JSON and parse it to a JObject. From that moment on I have full control and freedom to do what-ever parsing of the JSON I want. NewtonSoft JSON provides the user with a very complete API to dissect JSON objects, however, it can get messy.

To abstract the dirtiness of JObjects a bit I added custom JSON parsers for each element in the OpenAPI spec that are interesting to parse. Some of the objects are not needed now but I might add them later. The result of the parsing step is an/are object(s) called BluePrints. These objects contain all the data needed to have a clean abstract interface for my T4 templates. I could have parsed the JSON straight in the T4 templates but that would probably have lead to being promoted by the Spaghetti Monster HIMself to exalted high priest in the church of Pastafism. T4 templates are a b*tch to write already, let alone do the parsing in there.

So I really wanted an abstraction that simply provides me with getters and utility functions for the data I need to feed into the T4 template. The Blueprint for a Schema looks as follows:

using System.Collections.Generic;
using Editor.Generation.CodeGen.BluePrints.Abstract;

namespace Editor.Generation.CodeGen.BluePrints
{
    public class SchemaProperty
    {
        public string Name { get; set; }
        public string Type { get; set; }
        public bool IsCollectionType { get; set; }
        public bool IsReference { get; set; }
        public string Description { get; set; }
    }

    public class SchemaBluePrint : AbstractBluePrint
    {
        public string Name { get; set; }
        public string Description { get; set; }
        public List<SchemaProperty> Properties { get; private set; }

        public SchemaBluePrint()
        {
            Properties = new List<SchemaProperty>();
        }
    }
}

As you can see, it simply reflects the data provided in the openAPI spec. Nothing special here. Let’s also take a look at the T4 template. Then you will see how it all fits together:

<#@ template language=”C#” #>
<#@ assembly name=”System.Core” #>
<#@ import namespace=”System.Text” #>
<#@ import namespace=”System.IO” #>
<#@ import namespace=”System.Collections.Generic” #>
///////////////////////////////////////
////// GENERATED BY SJWAGMEISTER //////
// DO NOT EDIT, CHANGES WILL BE LOST //
///////////////////////////////////////

using System;
using System.Collections.Generic;

namespace <#= _nameSpace #>.Schemas
{
/// <summary>
/// <#= _bluePrint.Description #>
/// </summary>
public partial class <#= _bluePrint.Name #>
{
<#
foreach (var prop in _bluePrint.Properties)
{
#>
/// <summary>
/// <#= prop.Description #>
/// </summary>
<#
this.WriteLine(prop.IsCollectionType
? $” public List<{prop.Type}> {prop.Name} {{ get; set; }}”
: $” public {prop.Type} {prop.Name} {{ get; set; }}”);
#>

<#
}
#>
}
}

The template uses the BluePrint object to get it’s data. (Yes, there is some additional data like the namespace, which comes from another source since it’s more of a “global” notion for generation). As you can probably see just from this tiny template; things can get messy. Note that there is no IDE support what-so-ever for formatting and checks for correctness. So if you have larger templates, things can become really tricky to work with.

With this code I was able to parse all schemas and write them out as C# class files. And at this point my mission was somewhat complete, because this is what I set out to do. But of course, now, I could see much more potential in this little library. And thus I expanded my goals to parse the routes/paths to be able to generate a strongly typed front-end client.

What does that mean exactly!? Well, when I parse the routes I can generate actual C# functions that send requests to these routes including their function parameters like, schemas. So this way, you will never have to write front-end web-code ever again. Cool huh!?

NetStandard support

But first I needed to add some logic that made it possible to run on NetStandard. The pre-processed T4 templates require namespaces that’s only available in the NetFramework compatibility level. And as I mentioned before, one does not simply… But alright, I needed some way to support NetStandard but I couldn’t really find anything. So as any geek, I wrote some custom code.

I ended up writing a couple of objects that form what I call the TemplateBuilder. This is a collection of objects that implement a very simple Fluent interface/Builder pattern to define templates. It also provides a high degree of testability.

An example of such template builder is the following “FieldBuilder”:

using System;
using System.ComponentModel;

namespace Editor.Generation.CodeGen.TemplateBuilder
{
    public class FieldBuilder : AbstractTemplateBuilder
    {
        public string Name { get; }
        private readonly AccessModifier _accessModifier;
        private readonly string _type;
        private AccessModifier _getModifier;
        private AccessModifier _setModifier;

        public FieldBuilder(AccessModifier accessModifier, string type, string name)
        {
            _accessModifier = accessModifier;
            if (string.IsNullOrWhiteSpace(type))
                throw new ArgumentException("Must define a type when creating a field!");
            _type = type;
            if (string.IsNullOrWhiteSpace(name))
                throw new ArgumentException("Must define a name when creating a field!");
            Name = name;
            if (accessModifier == AccessModifier.None)
                throw new InvalidEnumArgumentException("Field must have an access modifier! (none is invalid!)");
        }

        public override string Build()
        {
            var summary = HasSummary() ? $"{GetSummary()}{Environment.NewLine}" : "";
            return
                $@"{Environment.NewLine}{summary}{Indentation}{_accessModifier.ToString().ToLower()} {_type} {Name};{Environment.NewLine}";
        }
    }
}

When used it looks a bit like this:

_fieldBuilder = new FieldBuilder(AccessModifier.Protected, "string", "myField");

which in turn “Builds” into this:

protected string myField;

I can then chain all these template builders together to create classes.

 

Tying it all together:

Last, I needed to connect all loose ends to make a simple Unity3D editor plugin. I created some ScriptableObject(SO) and added a small custom editor class for it. This way you can simply run the code straight from the SO.

And now what!? Well, at this point, I can see even more unlocked potential of this awesome little plugin I wrote. I will not get into it here but I’m definitely going to expand on this asset in the future to make it more feature-complete in my view. Stay tuned to see what that means exactly.

But for now you can check the code for SjwagMeister here.