Early iOS Export 28 Dec 2017

Taking a break from the posts outlining the inner workings of the compiler to report some exciting news: late last night, for the first time, Clojure code compiled with MAGIC successfully ran on an iPhone!

The TL;DR is that we’ve confirmed that MAGIC can achieve its original goal of bringing Clojure to more restrictive environments. In particular, the code in this prototype survived Unity’s IL2CPP translation into native code suited for iOS. We expected this to be possible, but seeing it running on a physical device puts to rest any lingering doubts.

Code

The demo is an implementation of the classic boids flocking algorithm. It is available at this gist and for brevity I will not list it inline here.

boid.core is the first prototype I wrote using Clojure/MAGIC, and is most representative of what a namespace will look like when MAGIC is more complete and can stand on its own. For reasons that I get into below, a namespace like this does not “Just Work” yet.

boids.build is the namespace I used to produce the iOS-ready assemblies. You can see the bodies of functions and macros copied from boid.core, but there’s a lot of compiler-specific machinery surrounding it. It should be reproducible on the latest MAGIC commit (73531fc2 as of the time of this writing). I built the assemblies from within Unity using the Arcadia REPL.

BoidBehaviour.cs is the Unity component that invokes the Clojure code. It is attached to each of the 330 boids and passes the local transform into the Boids.Core.update method.

Strategy

Additional work had to be done to make the demo function given the current state of MAGIC. None of this reflects what a real-world workflow would be like given a finished compiler, but rather documents my process in getting this to work on MAGIC as it exists today.

Limitations

This demo is restricted to the parts of Clojure that require no runtime support, namely:

MAGIC supports more than this, namely invoking Clojure vars, but looking up vars requires the support of the Clojure runtime, which requires clojure.core to be loaded, which MAGIC cannot compile yet. In more usual uses of MAGIC this is not a problem, as MAGIC is usually run from within the stock compiler and leans on it to gain access to the Clojure runtime. In an exported context, this is not possible, because we have no way of getting clojure.core to run on an iPhone without MAGIC being able to compile it.

Even given those limitations, MAGIC code is fairly expressive. Namespaces like arcadia.linear are completely useable, and a lot of call sites that would normally be vars get turned into inlined bytecode by magic.intrinsics, including aget, aset, <, int, inc and more. MAGIC’s macroexpander is working, so macros like boids.build/transforms are also fair game.

Static Methods

Without runtime support, we cannot compile a proper Clojure namespace, so instead this demo compiles C# classes with static methods that contain the bodies of our functions. The default analysis passes are modified to generate static methods instead of instance methods, and the boids.build/static-methods macro is designed to generate a CLR type with a named static method for each function.

Assembly Juggling

There are some shenanigans involved in preparing the assemblies. First, note that we actually emit three assemblies in the comment at the end of boids.build. This is to get around our inability to invoke vars. For a function to call another function, it must treat it as interop. The flock function invokes the separation, cohesion, and alignment functions as static C# methods. This requires the Boids.Rules assembly to be compiled first and loaded into memory. The same is true when update invokes flock.

This is a bit tedious, but it works. I uncomment each of the forms in turn and emit each assembly to disk separately. The resulting assemblies (rule.dll, flocking.dll, boids.dll) can be merged into a single assembly using ilrepack. Once dropped into a Unity project and referenced from your IDE, they are completely useable from C# and even participate in intellisense!

From here, BoidBehaviour.cs can call our update method, passing in its transform. The BoidBehaviour component is attached to the boid GameObjects, which are positioned n a three dimensional grid in the scene. This works in the editor and survives IL2CPP export, because as far as Unity is concerned all it’s dealing with is C# code and one assembly full of IL it knows how to convert. I needed to add the make-array intrinsic, but everything else converted without issue.

Next Steps

The goal is for something that looks like boid.core to “Just Work” without additional intervention. To get there, MAGIC needs to be able to compile whole namespaces, closures, dynamic call sites, and the remaining special forms in a way that IL2CPP can convert. Compiling clojure.core is the real hurdle. MAGIC’s approach to maximizing the flexibility of the bytecode emission means we can make our IL look how ever it needs to look to survive translation onto platforms like iOS. This early demo is an extremely promising milestone, and an indication that we’re on the right track.

🎩✨