Howl

Howl is a library to serve as a core module for embedded scripting languages in Unity3D. It’s main goal is to take care of interopability, script loading and dependency injection. Currently the focus in on LUA, but it would be really interesting to add support for LISP or Clojure in the future. Combining Unity3D and LISP would be the greatest thing ever 😛

DevLog

The basic idea for Howl boils down to providing a flexible run-time extension model. In the games industry this is often called modding. Modding is mostly done through (interpreted) scripting languages that are evaluated and executed at run-time. There are also modding options that support C# or other compiled languages but our Big Tech App Store overlords don’t allow for proper DLL injection since they cannot review and validate the code when you download and plug in a DLL at run-time.

As a response to this, most companies will circumvent it by using Lua. Using an interpreted and sandboxed language falls in a gray area of the user agreements. Both Google as well as Apple don’t allow for “executable” code to be injected at run-time. Yet interpreted code is fine, until they notice or find the developers doing something malicious. Also, when you inject code at run-time the code must be readable by its users. This basically means that it, the dynamic code, has to be open sourced. One could in theory deploy a simple ToDo but change it into a crypto miner at run-time by using Lua. The overlords will ban your app when they get a hold of such activities.

Apple says the following concerning dynamic code execution:

“2.5.2 Apps should be self-contained in their bundles, and may not read or write data outside the designated container area, nor may they download, install, or execute code which introduces or changes features or functionality of the app, including other apps. Educational apps designed to teach, develop, or allow students to test executable code may, in limited circumstances, download code provided that such code is not used for other purposes. Such apps must make the source code provided by the app completely viewable and editable by the user.”

Whereas Google has this to say:

“An app distributed via Google Play may not modify, replace, or update itself using any method other than Google Play’s update mechanism. Likewise, an app may not download executable code (e.g., dex, JAR, .so files) from a source other than Google Play. This restriction does not apply to code that runs in a virtual machine or an interpreter where either provides indirect access to Android APIs (such as JavaScript in a webview or browser).  

Apps or third-party code (e.g., SDKs) with interpreted languages (JavaScript, Python, Lua, etc.) loaded at run time (e.g., not packaged with the app) must not allow potential violations of Google Play policies.”

So.. as long as we provide a proper sandboxed environment and protect native Android and iOS API’s we should be allowed to extend our apps into oblivion.

Sandboxing

The first thing I needed to take care of was creating a sandbox-like environment. For Howl, there are three core responsibilities a sandbox will have:

1. 1. Making sure script imports work automatically and effortlessly.
   2. Serve as a boundary for interop between C# and the target scripting language.
   3. Act as a gatekeeper for unauthorized API’s like IO.

The library we are using for Lua support called MoonSharp has an abstraction called a Script. Through the script class you can run arbitrary Lua code or load entire files. So the implementation of Howl’s Sandbox uses the MoonSharp script to provide the necessary interop. Another interesting point about MoonSharp is the fact that the developers already thought about a sandboxed environment. But I suppose that’s pretty default with such embedded scripting languages.

Point three of my list above is covered through using the sandbox capabilities provided by MoonSharp. They support different kinds of (enum) flags to allow for access to Core Modules. Examples of core modules are Math, String or Coroutine. But also OS and IO, which might be prohibited in specific use-cases like code injected by end-users. Thus, by setting the Core Module flags correctly, we can constrain our sandbox to only allow what MoonSharp allows.

I think this is a nice way of abstracting things. But I do want to point out that Howl, will most likely, in the fure provide C# proxies for loading files or specific unity3D assets. These will not be covered automatically by MoonSharp’s Core Module flags. This must be implemented manually.

But how do we handle interop between C# and the scripting language, in this case Lua. Let’s discuss that next 🙂

 

Ineropability

Since Interopability is a very important puzzle piece of what we are trying to achieve here, I think it’s crucial to have a simple and effective API. There is however a massive constraint I took upon myself and that is; avoiding run-time reflection at all costs. Reflection is very powerful, yet very slow. We don’t want our mobile apps to slow down because of script interop. I can imagine scenario’s where interop is used a lot. So I suppose a fine balance needs to be found. I’ve seen other implementations of Lua interopability in C# and Unity3D specifically and they relied heavily on annotations placed on certain classes, fields, properties and methods. I want to avoid this and only allow interop when specifically asked for. And, yes, I know that MoonSharp has support for reflection based interop but I still want to avoid it.

So I came up with a simple and nice API for a sandbox so let’s take a look at the ISandBox iterface:

namespace HamerSoft.Howl.Core
{
    public interface ISandbox : IDisposable
    {
        public bool IsActive { get; }
        public string Name { get; }

        public void Start(params AbstractScriptHandler[] args);
        public void Restart(params AbstractScriptHandler[] args);
        public void Stop();

        public bool ScriptExists(string name);
        public void AppendScript(string scriptName, params AbstractScriptHandler[] args);
        public void AppendCode(string luaCode, params AbstractScriptHandler[] args);
        
        public void Invoke(string functionName, string selfRef = null, params object[] args);
        public bool TryInvoke(string functionName, string selfRef = null, params object[] args);
        public T Invoke<T>(string functionName, string selfRef = null, params object[] args);
        public bool TryInvoke<T>(string functionName, out T returnValue, string selfRef = null, params object[] args);
        
        public T GetValue<T>(string luaBinding);
        public bool TryGetValue<T>(string luaBinding, out T value);
        
        public void SetValue(string luaBinding, object value);
        public bool TrySetValue(string luaBinding, object value);
    }
}

Even though the interface exposes lot’s of methods I still think it adheres to the Single Responsibility Principle (SRP). Because, well it’s a scripting sandbox and it should provide everything nessecary to accomplish it’s goal. Nothing in this interface has reasons to change other than being responsible for the points noted in the previous section.

So the ISandbox interface requires implementations to implement:

 

• • life-cycle functions (Start, Restart, Stop).
  • script loading (ScriptExists, AppendScript, AppendCode).
  • Invoke (lua function) logic (Invoke, Invoke<T> and TryInvoke alternatives).
  • Get (Lua) Value and TryGet alternative.
  • Set (Lua) Value and TrySet alternative.

I added the Try(Invoke, Get, Set) versions since I really like this way of ‘defining errors out of existence’. This way you can hide exceptions by simply returning a boolean, how simply awesome is that!? Whenever I see such an API I know that the writer has ut some thought into it.

But to get back to the interface: There’s one caveat to this interface and that is, it’s string based. And, it’s on purpose. I wanted users to have full control over interop even to the point where the C# code needs to interact to Lua. By using strings (or maybe a higher abstraction in the future, I could parse the strings into special kind of interop handlers or something) you can do what-ever you want. But let me give you some examples of the interop to Lua:

Imagine the following Lua code

function f1()
    print 'f1'
end

function f2(a)
    print(a)
end

function f3(x, y)
    return x + y
end

-- The logic below demonstrates an OO approach in Lua

ObjectA = {}
ObjectA.__index = ObjectA

function ObjectA.new()
    local instance = setmetatable({}, ObjectA)
    instance.f4 = function(v)
        return v * v
    end
    return instance
end

function ObjectA:f5(b)
    return self.f4(b)
end

InstanceA = ObjectA.new()

ObjectB = { }
ObjectB.__index = ObjectB

function ObjectB.new()
    local instance = setmetatable({}, ObjectB)
    instance.a = ObjectA.new()
    return instance
end

InstanceB = ObjectB.new()

We can then interact with this code through the C# (Try)Invoke Api provided by the Sandbox like so:

            var api = new Api();
            var sb = api.CreateSandBox(new SandboxConfig("mySB"));
            sb.Start();
            sb.AppendCode(_invokeLua);
            // f1
            sb.Invoke("f1");
            // f2
            sb.Invoke("f2", null, "Hello, World");
            // f3
            var resultF3 = sb.Invoke<int>("f3", null, 3, 3); // result will be 6 
            // f4
            var resultF4 = sb.Invoke<int>("InstanceA.f4", null, 3); // result will be 9
            // f4, through objectB
            var resultF4a = sb.Invoke<int>("InstanceB.a.f4", null, 2); // result will be 4
            // f5
            var resultF5 = sb.Invoke<int>("ObjectA:f5", "InstanceA", 5); // result will be 25
            // TryInvoke Example
            if (sb.TryInvoke<int>("InstanceB.a.f4", out var resultF4Instance, null, 2)) // result will be 4
                Debug.Log($"Succeeded Invoke with result {resultF4Instance}");

What it does internally is simply traversing the hierarchy of Tables to invoke the correct functions on the correct ‘object’ with the number of arguments given.

If you are new to Lua: Lua is NOT an OO language. Every object’ish construct is basically a associative array (key value). So your Player ‘object’ is just a table (think hashtable/dictionary in C#) with KeyValuePairs like <“Name”, “HamerSoft”> and <“Health”, 1337>.

The (Try)Get/SetValue Api’s work similar to the invoke API’s so I won’t cover them here. You can check them in the README of the project.

So now that we have covered interop let’s take a look at the last point of our list: script loading i.e. the ‘require’ statement in Lua.

 

Proxies

To interact from the scripting language (Lua in this case) to C# you should use so called proxy objects. These object form the architectural boundary between them. When you use proxies for interop, you can limit the exposure of C# types to the scripting environment. This will help you a lot with dependency management.

Howl supports three kinds of proxies, Singleton, Transient and Static.

The SingletonProxy is as the name suggests a singleton, in the context of the scripting language. It will have the same reference anywhere you access it in your SandBox.

The TransientProxy differs from the singleton proxy because it generates a new reference each time it is accessed.

The StaticProxy is a special kind of proxy to register (C#) static functions as functions to call in Lua.

In the context of MoonSharp, proxies are Global. This means you register them in the API and then Howl will make sure they will be available inside your sandbox and scripts. Note that scoping might disallow the usage of certain proxies as they expose privileged API’s.

 

ScriptLoading

The last thing we need covered to implement our SandBox is finding scripts on disk or where-ever they reside. Lots, of not all programming languages have the option to load or import code from other locations. In C# we have the ‘using’ statement, in Clojure we ave the ‘require’, ‘use’ or ‘import’ statements and in Lua we simply get ‘require‘.

However, we do not want our end-users (scripters) ‘requiring’ core logic or modules that they are not supposed to. So we need to make the script locators smart enough to handle this kind of constraint.

For Howl I implemented 4 different ScriptLocators:

 

1. 1. CoreDependencyLocator – The core locator that uses the Unity3D Resources API to locate built-in modules by Howl.
   2. ResourcesLocator – A locator that searches the Unity3D resources relative to the path you provided.
   3. FileSystemLocator – A locator that uses the specified root path to search for script files.
   4. StreamingAssetsLocator – A locator that searches the Unity3D StreamingAssets relative to the path you provided.

When you create a new SandBox through the API the CoreDependencyLocator is always injected. This locator will serve as a loader for Howl Built-in modules. None there yet, but we imagine modules to interop with GameObjects, Audio, UI, FileSystem and many other API’s Unity3D provides.

Depending on the RootDirectory you set in the SandBoxConfig you provided one, and only one other locator is instantiated. The StreamingAssetsLocator is chosen when the path is rooted and started with Application.StreamingAssetsPath, if it’s rooted but not starts with the streaming assets path, then the FileSystemLocator is instantiated. If the path is not rooted, the ResourcesLocator is chosen.

When you do not provide a RootDirectory, you will only have access to the core locator by default.

 

Future Improvements

Future improvements will orient towards two main goals:

1. 1.  Providing core modules for Lua interop to Unity3D API’s like GameObject, Transform and VisualElement
   2. Hot code reload to improve the developer experience

The core functionality for this is done and thus the stakeholders I have for this specific project can move forward. I’ll implement the points above asap but without a hurry. I’ll keep this dev log updated :).