A Year Of Clojure

Bloody hell, more than a year evaporated.

What happened in that time? Mostly, a lot of Clojure, at least with respect to home stuff. It finally clicked something like nine months ago, in that I suddenly felt much happier writing in this language than pretty much any other. It was also marked by the crushing realisation that I didn't really know how to elegantly structure a program the size of the game I was making, in the new-ish language I was making it in. I still don't having restarted in a new and exciting language, but starting from scratch means the pressure has been relieved. Temporarily.

I've dithered with the old Scala project a bit, but to be honest it has too many ugly bits and not enough game to be worth salvaging. I'll likely rip out the tech that worked and give it a whirl in the new language du jour; the deferred rendering was fun, if simple, and the transparency hack surprisingly effective.

2011 apparently having been the year of Clojure and non-blogging, figured I'd start the renewed attempt at updates with a few notes and statements concerning the language.

---

Records don't behave like functions of keys, unlike maps which they otherwise resemble.

I can see why this is the case, sorta. I'm sure you don't want records to implement too many interfaces by default. But I like to be able to get the value of a key using both (map :key) and (:key map) syntax, in addition to (get map :key) etc. It's also useful to be able to pass a record/map to higher order functions at times.

To enable the above, you just need to make your record type implement clojure.lang.IFn and a trivial invoke method:

(defrecord record-thing [id biscuit]
  clojure.lang.IFn
  (invoke [this k]
    (get this k)))

Now instances of record-thing will support both (:biscuit a-record-thing) and (a-record-thing :biscuit) syntax. Easy.


Handles, handles for everyone!

This is more of a general functional thing. Any language which encourages the use of immutable data structures obviously invalidates a common method by which objects refer to each other, namely by holding a pointer or reference to the other object(s). Take for example a simple case where a monster wants to keep a short list of relevant enemies, for use as input state to some AI functions. Our monster is stupidly simple, and simply stupid:

(def fred-the-goblin
  {:type :goblin
   :health 10
   :position [0 0]
   :targets []})

What goes in the :targets list? Clearly it can't be equally simple representations of target entities. They'd have to be updated as the world evolves, and may refer to Fred in their own target lists and thereby create recursive doom. In clojure, we could use atoms or refs everywhere, but that's hideous and rather belies our intent to be pure. What I've been tending to do is stick an :id field on everything, using this as a convenient handle for any object that should be uniquely identifiable (within a collection of similar objects at any rate, I make no attempt at global uniqueness. Quite possibly this will redound to my detriment).

Of course, in order to actually get at the entity referred to by the handle, we really want some nice associative structure.

(defn auto-map [entities] (into {} (map vector (map :id entities) entities)))

This seems like the most reasonable approach. So far it feels like spatial organisation structures, turn ordering, other such ways of organising the set of active entities seem more naturally expressed as collections of smaller chunks of relevant data with an ID attached.

I'm still not convinced this is the best approach though. It seems somewhat brittle. How does a more determinedly pure language handle things?


I don't read enough documentation

I had a semi-rant here about the lack of a promise-delivered? function, equivalent to future-done?... but whilst writing it, I discovered that this function exists. It's just called realized? (and it also works for futures, delays, and lazy sequences). I can't believe I missed this for so long, and it will help simplify some code that currently combines the two.


Promises are nice

Continuing the above: don't get me wrong, I love futures. They're handy little things. But there's a great reason to use promises for games: controlling the thread upon which you do work. One very common case when dealing with things around the level of OpenGL 2.0/DX9 is rendering resource creation, which generally takes place on the rendering thread. In my case, I have a queue of pending jobs on the rendering thread in the form of promise/closure pairs. Each frame, the rendering thread pops a number of these jobs off the queue, executes the closures and delivers the result to the associated promise. Other threads can block or skip executing code that depends on those GPU resources/tasks, as they choose.


Dealing with the (lack of) speed

No real getting around it: Clojure is pretty slow. Not agonisingly slow, but if you're used to decently optimised C++ it can be striking. Now, many things act against this, not least the fact that it's so very joyful to write in the language (In my opinion, YMMV etc). Concurrency is orders of magnitude more tractable. Concision, dynamic typing and a functional style frequently makes iteration on algorithmic optimisations much easier than it would be for a huge lump of C++. Transient collections speed up some common operations without sacrificing referential transparency, and so on.

But.

Still slow. Especially when writing somewhat idiomatic, pure code, which is what I want to be doing.

Thankfully GPUs are fast, and much as with the Scala project my aim is to use relatively few draw calls per frame and make each as expensive as possible to get the results I want. State from the CPU should be small and cuddly. This just leaves physics, AI, sound, IO and general game logic to be worried about when it comes to performance. For a turn based game the remaining problems are greatly reduced in potential for menace, but I'm sure I'll have fun with them when I do something real-time and action-y.


Vectors as vectors

I've lost track of the number of little linear algebra libraries I've written and used. For my Clojure experiments, I wrote yet another, but kept it deliberately stupid.

I have minimal requirements from such a library. In fact after reducing my initial bulletpoints a bit, I have but one: Be like HLSL/GLSL.

That means adding a scalar to a vector is legal. Multiplying two vectors together, also legal (it does this componentwise, dot and cross products are handled as separate ops). Preferably, don't do the GLSL thing and conflate transforming a vector using a matrix with your vector-vector and vector-scalar multiplications.

These are very much not based around sensible mathematics, but instead represent the kind of things you actually do with vector quantities on a regular basis.

Oh, and I like having decent quaternion support, because they're approximately infinitely nicer than trying to keep rotation matrices/Euler angles sane. Swizzling is optional, as it doesn't come up that often in practice.

The end result is that I use Clojure's vector collection type for all vector-like objects. I has nice literal syntax in the form [x y z] and supports unpacking directly: (let [[x y z] v] ...). Components are not named members of a structure. All the basic operations are based around a version of map which accepts a mixture of scalars and collections in all positions.


Because this needs more attention, however negligible the increment may be

A discussion of the concept of "wat" in software.

It's just a few things that Ruby and JavaScript do which... may be surprising. I'm sure there are more than a few of those lurking closer to home, possibly only unremarked upon due to my overlong exposure to the madness.



In closing...

I shouldn't have let this blog rot. I should get some pretty screenshots for next time!

More references

I sometimes (often) feel the Internet is too damn big to be useful.

But it is possible to find some really good stuff...

Like these fascinating posts dealing with writing retro games in functional languages. I linked the last part in the mini-series, because it posits a solution to one of my many current problems. Rather than allowing game logic to modify game state directly, emit a list of 'actions' which can be dealt with in a more or less atomic fashion at the end of a frame.

Initially this smacks of just moving hard work elsewhere, but I think it does help more than that. Your world state/list of entities becomes static for the bulk of a frame, so the order in which you process things is irrelevant, and consequently dereferencing entity handles becomes slightly more tractable. Of course there's a real danger that by the end of a frame, you may have accumulated mutually incompatible actions. However, imposing an ordering over actions and allowing them to be tolerant of faults seems like a better solution than threading the complete game state through everything related to an update - which essentially allows these supposedly transparent areas to side effect by fiddling with the world state directly.

Hmm.

Another fascinating post calls out the lack of a simple dictionary type as a major implementation headache for games in the chosen functional language (Erlang). That seems to be tacit support for the 'property bag' model of game entities in a functional context. I wonder how many other game programmers are lurking out there in the wilds of the internet, ready and willing to provide useful feedback from the experience of using these patterns in functional languages, if only my Google-fu were strong enough to find them.

Meanderings

The festive season never seems to be a good time to get any coding done...

With that out of the way, I'm going to press on with my crazy schemes. Over the past few days, I've been trying to recreate the skeleton of my game in Haskell using Nanocurses (this was mostly an excuse to learn Haskell, having tried and failed many, many times already).

I still think in altogether too imperative a fashion for Haskell to look on approvingly, especially when doing things with Curses involves slipping into 'do' notation all the time. Ah well, it'll be interesting to see if rendering my problem in a more pure language will help me spot any solutions.

Functional Update Musings

Recently I've been focussing a lot on the rendering part of this project. Quite fruitfully at times, but it certainly wasn't my intention.

The reason for this is simply when I sat down to work on some gameplay stuff, I found a hard problem, and the solutions I've arrived at so far have not been... nice.

As fair warning, this is a brain dump, and a long, boring one at that. It contains much partly-functional waffle because my brain is full of partially functional waffle.

Background: Mostly for my own illumination, I've been trying to use more functional idioms in this game than is strictly sensible. I've mentioned before that I wish to have a (mostly) immutable world, for example. The world update step therefore creates a new world state every 'turn'. The current world state is stored in a mutable variable at the end of each update, but other than that the entire update should be immutable and cuddly from a gameplay viewpoint.

Problem: Creating a suitable update step for my basic case is very easy. The only entities that can affect the world are Agents. Each agent has a (poorly named) 'energy' integer value. When an agent has zero energy, it can act. Actions themselves are the things that change the world state, they are essentially partial functions of type Action :: WorldState -> WorldState, and almost all have the secondary effect of reducing the current agent's energy value when they're executed. After the current agent has acted, the agent list is sorted, some simple transformations are applied to the entity list (dead creature agents are replaced with corpse entities, for example) and time is advanced to bring another agent into relevancy.

This works after a fashion, but certain things are more sensibly represented as continuous functions of time than as agents that act with some (possibly inconsistent) frequency. A better way might be to allow these functions of time to run after the current agent acts, as conveniently we implicitly have the delta time before the next agent is ready to act. This will allow us to damage creatures in lava/acid, or heal creatures that are regenerating, for example. Easy enough, we can make something like TimeFunction :: Int -> WorldState -> WorldState. This is very ugly in practise, especially if there is no ordering defined over the set of active TimeFunctions (consider the joyous difference in observed state if a timefunction spawns a monster and then another damages/kills it, versus damaging monsters then spawning them). There are also potentially huge problems if the list of active TimeFunctions is part of the world state, subject to being updated by TimeFunctions as they run. Blurgh!

There's another big source of ugly too: edge detection. In general, we're quite interested in observing certain changes in world state. An example which Andrew Doull writes about here is a quest to scare seven goblins, which I think can be done by detecting changes in the afraid state of goblin entities. If we perform a comparison of world state before and after the player-controlled agent acts, we can in theory unambiguously attribute any newly-scared goblins to the player and update quest progress. Again we can create something like: EdgeDetector :: WorldState -> WorldState -> WorldState, but it's even nastier now than the case for TimeFunctions. We also have three transitions that may be of interest and could have associated detectors: comparing the original world state to that after the current agent acts; post-agent-action world state to post-timefunction world state; and original state to post-timefunction world state. This would allow us to differentiate between the player killing a goblin with a mace and the goblin burning to death because it stupidly fell into some lava, even in the absence of creatures storing their last attacker or whatever.

It may help to constrain these ugly bits somewhat. For example mandating that TimeFunctions must define a strict ordering and only affect the list of entities and their own internal state, whilst EdgeDetectors can only update their internal state. This in turn means that some other object has to dig through the EdgeDetectors to extract useful data, but I suppose that's not too evil. Both edge detectors and time functions are created and destroyed only by agents, which is closer to the simplicity of the initial design.

But even with these slightly draconian measures in place, a single update is a far nastier affair than I feel it has any right to be, with multiple intermediate world states, and I'm unwilling to set this down in code outside my messy prototype framework.

I've been wrestling with the few papers I've found talking extensively about reactive/game programming in functional languages (by which I mean Haskell) but it's dense, dry, nearly unintelligible stuff as far as I'm concerned. Admittedly they're generally solving the far harder problem of realtime games, whereas I currently go the cheatsy route of using a great deal of mutable state for all realtime code paths such as rendering.

Anyway, brain dump complete. I hope I'll find a better solution and can come back and mock my doubtless shocking ignorance in this post, but right now it just makes me uneasy. The net result of which is my experimented with deferred rendering pipelines and funky post process effects, so its not all bad.