Wednesday, 25 January 2012

A Year Of Clojure

Bloody hell, more than a year evaporated.

What happened in that time? Mostly, a lot of Clojure, at least with respect to home stuff. It finally clicked something like nine months ago, in that I suddenly felt much happier writing in this language than pretty much any other. It was also marked by the crushing realisation that I didn't really know how to elegantly structure a program the size of the game I was making, in the new-ish language I was making it in. I still don't having restarted in a new and exciting language, but starting from scratch means the pressure has been relieved. Temporarily.

I've dithered with the old Scala project a bit, but to be honest it has too many ugly bits and not enough game to be worth salvaging. I'll likely rip out the tech that worked and give it a whirl in the new language du jour; the deferred rendering was fun, if simple, and the transparency hack surprisingly effective.

2011 apparently having been the year of Clojure and non-blogging, figured I'd start the renewed attempt at updates with a few notes and statements concerning the language.

Records don't behave like functions of keys, unlike maps which they otherwise resemble.

I can see why this is the case, sorta. I'm sure you don't want records to implement too many interfaces by default. But I like to be able to get the value of a key using both (map :key) and (:key map) syntax, in addition to (get map :key) etc. It's also useful to be able to pass a record/map to higher order functions at times.

To enable the above, you just need to make your record type implement clojure.lang.IFn and a trivial invoke method:
(defrecord record-thing [id biscuit]
  (invoke [this k]
    (get this k)))

Now instances of record-thing will support both (:biscuit a-record-thing) and (a-record-thing :biscuit) syntax. Easy.

Handles, handles for everyone!

This is more of a general functional thing. Any language which encourages the use of immutable data structures obviously invalidates a common method by which objects refer to each other, namely by holding a pointer or reference to the other object(s). Take for example a simple case where a monster wants to keep a short list of relevant enemies, for use as input state to some AI functions. Our monster is stupidly simple, and simply stupid:
(def fred-the-goblin
  {:type :goblin
   :health 10
   :position [0 0]
   :targets []})
What goes in the :targets list? Clearly it can't be equally simple representations of target entities. They'd have to be updated as the world evolves, and may refer to Fred in their own target lists and thereby create recursive doom. In clojure, we could use atoms or refs everywhere, but that's hideous and rather belies our intent to be pure. What I've been tending to do is stick an :id field on everything, using this as a convenient handle for any object that should be uniquely identifiable (within a collection of similar objects at any rate, I make no attempt at global uniqueness. Quite possibly this will redound to my detriment).

Of course, in order to actually get at the entity referred to by the handle, we really want some nice associative structure.
(defn auto-map [entities] (into {} (map vector (map :id entities) entities)))
This seems like the most reasonable approach. So far it feels like spatial organisation structures, turn ordering, other such ways of organising the set of active entities seem more naturally expressed as collections of smaller chunks of relevant data with an ID attached.

I'm still not convinced this is the best approach though. It seems somewhat brittle. How does a more determinedly pure language handle things?

I don't read enough documentation

I had a semi-rant here about the lack of a promise-delivered? function, equivalent to future-done?... but whilst writing it, I discovered that this function exists. It's just called realized? (and it also works for futures, delays, and lazy sequences). I can't believe I missed this for so long, and it will help simplify some code that currently combines the two.

Promises are nice

Continuing the above: don't get me wrong, I love futures. They're handy little things. But there's a great reason to use promises for games: controlling the thread upon which you do work. One very common case when dealing with things around the level of OpenGL 2.0/DX9 is rendering resource creation, which generally takes place on the rendering thread. In my case, I have a queue of pending jobs on the rendering thread in the form of promise/closure pairs. Each frame, the rendering thread pops a number of these jobs off the queue, executes the closures and delivers the result to the associated promise. Other threads can block or skip executing code that depends on those GPU resources/tasks, as they choose.

Dealing with the (lack of) speed

No real getting around it: Clojure is pretty slow. Not agonisingly slow, but if you're used to decently optimised C++ it can be striking. Now, many things act against this, not least the fact that it's so very joyful to write in the language (In my opinion, YMMV etc). Concurrency is orders of magnitude more tractable. Concision, dynamic typing and a functional style frequently makes iteration on algorithmic optimisations much easier than it would be for a huge lump of C++. Transient collections speed up some common operations without sacrificing referential transparency, and so on.


Still slow. Especially when writing somewhat idiomatic, pure code, which is what I want to be doing.

Thankfully GPUs are fast, and much as with the Scala project my aim is to use relatively few draw calls per frame and make each as expensive as possible to get the results I want. State from the CPU should be small and cuddly. This just leaves physics, AI, sound, IO and general game logic to be worried about when it comes to performance. For a turn based game the remaining problems are greatly reduced in potential for menace, but I'm sure I'll have fun with them when I do something real-time and action-y.

Vectors as vectors

I've lost track of the number of little linear algebra libraries I've written and used. For my Clojure experiments, I wrote yet another, but kept it deliberately stupid.

I have minimal requirements from such a library. In fact after reducing my initial bulletpoints a bit, I have but one: Be like HLSL/GLSL.

That means adding a scalar to a vector is legal. Multiplying two vectors together, also legal (it does this componentwise, dot and cross products are handled as separate ops). Preferably, don't do the GLSL thing and conflate transforming a vector using a matrix with your vector-vector and vector-scalar multiplications.

These are very much not based around sensible mathematics, but instead represent the kind of things you actually do with vector quantities on a regular basis.

Oh, and I like having decent quaternion support, because they're approximately infinitely nicer than trying to keep rotation matrices/Euler angles sane. Swizzling is optional, as it doesn't come up that often in practice.

The end result is that I use Clojure's vector collection type for all vector-like objects. I has nice literal syntax in the form [x y z] and supports unpacking directly: (let [[x y z] v] ...). Components are not named members of a structure. All the basic operations are based around a version of map which accepts a mixture of scalars and collections in all positions.

Because this needs more attention, however negligible the increment may be

A discussion of the concept of "wat" in software.

It's just a few things that Ruby and JavaScript do which... may be surprising. I'm sure there are more than a few of those lurking closer to home, possibly only unremarked upon due to my overlong exposure to the madness.

In closing...

I shouldn't have let this blog rot. I should get some pretty screenshots for next time!


swannodette said...

Screenshots please! And more detailed blog posts! I think game development is a really challenging subject area for functional programming and we need more experience reports like this :)

Snut said...

Thanks, I hope to provide both screenshots and more posts in the future!

One area I want to expand upon is why Scala and F# code feels more awkward to write than Clojure, despite their support for functional styles.

Thomas K said...

Hey there, why not use the jMonkeyEngine for games development in clojure?
I can see how doing everything (i.e. OpenGL, Physics, Sound, etc.) yourself can be fun and very rewarding in a way - and if that is what you're after, then this will be the right way. But personally I have yet to finish up any game I started to create by going down that path. (and I've been working with OpenGL in science projects for 3 years now). I think if you want to finish up at some point, you'd be better of going for a library that has already most of these things figured out. Otherwise you may end up writing your rendering pipeline again and again, simply because you don't like the way it is structured or you ran into a problem that is not easily solvable and give up entirely.

Snut said...

Hi Thomas, thanks for the comment.

There are three big reasons for my skipping gleefully past extant game engines:

1. They're largely based on Java idioms. Lots of destructive updates, large interfaces, deep inheritance hierarchies, object-centric models... lots of stuff I don't really care for. Working out how to interact with OpenGL (I use LWJGL as a lightweight wrapper) in a way that is pleasing to me is a large part of this project and blog's purpose.

2. I'm intending to stick with turn-based games, by and large, so physics isn't a concern. I'm a graphics programmer at my day job, so barring the oddities of OpenGL vs DirectX, the rendering framework is actually straightforward. That only leaves audio among the big engine features I'd be interested in, and the structure of the code itself. The former is scary, but the latter is definitely something I want to become better at doing.

3. Related to both previous points, my stumbling block previously has always been wither game design or implementation of game logic in a functional manner. Some game engines might have a decent entity framework or skeleton game loop, but it's unlikely to conform to the latter or help with the former.

And, yes, it is satisfying to do things from scratch. Certainly less productive, and if I find myself yearning to make that action-RPG or open world exploration game, I'll probably revisit the monolithic Java game engines. Who knows, by then one may have been written in idiomatic Clojure with an API I love at first sight!

Anonymous said...

If you want to revist the Game engines, have a look at the jMonkeyEngine. Since it is Java, it's working from scratch (without any wrapper), which is fantastic. Look in Google and <a href=">here</a> for examples.

Anonymous said...

link corrected