A Year Of Clojure

Bloody hell, more than a year evaporated.

What happened in that time? Mostly, a lot of Clojure, at least with respect to home stuff. It finally clicked something like nine months ago, in that I suddenly felt much happier writing in this language than pretty much any other. It was also marked by the crushing realisation that I didn't really know how to elegantly structure a program the size of the game I was making, in the new-ish language I was making it in. I still don't having restarted in a new and exciting language, but starting from scratch means the pressure has been relieved. Temporarily.

I've dithered with the old Scala project a bit, but to be honest it has too many ugly bits and not enough game to be worth salvaging. I'll likely rip out the tech that worked and give it a whirl in the new language du jour; the deferred rendering was fun, if simple, and the transparency hack surprisingly effective.

2011 apparently having been the year of Clojure and non-blogging, figured I'd start the renewed attempt at updates with a few notes and statements concerning the language.

---

Records don't behave like functions of keys, unlike maps which they otherwise resemble.

I can see why this is the case, sorta. I'm sure you don't want records to implement too many interfaces by default. But I like to be able to get the value of a key using both (map :key) and (:key map) syntax, in addition to (get map :key) etc. It's also useful to be able to pass a record/map to higher order functions at times.

To enable the above, you just need to make your record type implement clojure.lang.IFn and a trivial invoke method:

(defrecord record-thing [id biscuit]
  clojure.lang.IFn
  (invoke [this k]
    (get this k)))

Now instances of record-thing will support both (:biscuit a-record-thing) and (a-record-thing :biscuit) syntax. Easy.


Handles, handles for everyone!

This is more of a general functional thing. Any language which encourages the use of immutable data structures obviously invalidates a common method by which objects refer to each other, namely by holding a pointer or reference to the other object(s). Take for example a simple case where a monster wants to keep a short list of relevant enemies, for use as input state to some AI functions. Our monster is stupidly simple, and simply stupid:

(def fred-the-goblin
  {:type :goblin
   :health 10
   :position [0 0]
   :targets []})

What goes in the :targets list? Clearly it can't be equally simple representations of target entities. They'd have to be updated as the world evolves, and may refer to Fred in their own target lists and thereby create recursive doom. In clojure, we could use atoms or refs everywhere, but that's hideous and rather belies our intent to be pure. What I've been tending to do is stick an :id field on everything, using this as a convenient handle for any object that should be uniquely identifiable (within a collection of similar objects at any rate, I make no attempt at global uniqueness. Quite possibly this will redound to my detriment).

Of course, in order to actually get at the entity referred to by the handle, we really want some nice associative structure.

(defn auto-map [entities] (into {} (map vector (map :id entities) entities)))

This seems like the most reasonable approach. So far it feels like spatial organisation structures, turn ordering, other such ways of organising the set of active entities seem more naturally expressed as collections of smaller chunks of relevant data with an ID attached.

I'm still not convinced this is the best approach though. It seems somewhat brittle. How does a more determinedly pure language handle things?


I don't read enough documentation

I had a semi-rant here about the lack of a promise-delivered? function, equivalent to future-done?... but whilst writing it, I discovered that this function exists. It's just called realized? (and it also works for futures, delays, and lazy sequences). I can't believe I missed this for so long, and it will help simplify some code that currently combines the two.


Promises are nice

Continuing the above: don't get me wrong, I love futures. They're handy little things. But there's a great reason to use promises for games: controlling the thread upon which you do work. One very common case when dealing with things around the level of OpenGL 2.0/DX9 is rendering resource creation, which generally takes place on the rendering thread. In my case, I have a queue of pending jobs on the rendering thread in the form of promise/closure pairs. Each frame, the rendering thread pops a number of these jobs off the queue, executes the closures and delivers the result to the associated promise. Other threads can block or skip executing code that depends on those GPU resources/tasks, as they choose.


Dealing with the (lack of) speed

No real getting around it: Clojure is pretty slow. Not agonisingly slow, but if you're used to decently optimised C++ it can be striking. Now, many things act against this, not least the fact that it's so very joyful to write in the language (In my opinion, YMMV etc). Concurrency is orders of magnitude more tractable. Concision, dynamic typing and a functional style frequently makes iteration on algorithmic optimisations much easier than it would be for a huge lump of C++. Transient collections speed up some common operations without sacrificing referential transparency, and so on.

But.

Still slow. Especially when writing somewhat idiomatic, pure code, which is what I want to be doing.

Thankfully GPUs are fast, and much as with the Scala project my aim is to use relatively few draw calls per frame and make each as expensive as possible to get the results I want. State from the CPU should be small and cuddly. This just leaves physics, AI, sound, IO and general game logic to be worried about when it comes to performance. For a turn based game the remaining problems are greatly reduced in potential for menace, but I'm sure I'll have fun with them when I do something real-time and action-y.


Vectors as vectors

I've lost track of the number of little linear algebra libraries I've written and used. For my Clojure experiments, I wrote yet another, but kept it deliberately stupid.

I have minimal requirements from such a library. In fact after reducing my initial bulletpoints a bit, I have but one: Be like HLSL/GLSL.

That means adding a scalar to a vector is legal. Multiplying two vectors together, also legal (it does this componentwise, dot and cross products are handled as separate ops). Preferably, don't do the GLSL thing and conflate transforming a vector using a matrix with your vector-vector and vector-scalar multiplications.

These are very much not based around sensible mathematics, but instead represent the kind of things you actually do with vector quantities on a regular basis.

Oh, and I like having decent quaternion support, because they're approximately infinitely nicer than trying to keep rotation matrices/Euler angles sane. Swizzling is optional, as it doesn't come up that often in practice.

The end result is that I use Clojure's vector collection type for all vector-like objects. I has nice literal syntax in the form [x y z] and supports unpacking directly: (let [[x y z] v] ...). Components are not named members of a structure. All the basic operations are based around a version of map which accepts a mixture of scalars and collections in all positions.


Because this needs more attention, however negligible the increment may be

A discussion of the concept of "wat" in software.

It's just a few things that Ruby and JavaScript do which... may be surprising. I'm sure there are more than a few of those lurking closer to home, possibly only unremarked upon due to my overlong exposure to the madness.



In closing...

I shouldn't have let this blog rot. I should get some pretty screenshots for next time!

More references

I sometimes (often) feel the Internet is too damn big to be useful.

But it is possible to find some really good stuff...

Like these fascinating posts dealing with writing retro games in functional languages. I linked the last part in the mini-series, because it posits a solution to one of my many current problems. Rather than allowing game logic to modify game state directly, emit a list of 'actions' which can be dealt with in a more or less atomic fashion at the end of a frame.

Initially this smacks of just moving hard work elsewhere, but I think it does help more than that. Your world state/list of entities becomes static for the bulk of a frame, so the order in which you process things is irrelevant, and consequently dereferencing entity handles becomes slightly more tractable. Of course there's a real danger that by the end of a frame, you may have accumulated mutually incompatible actions. However, imposing an ordering over actions and allowing them to be tolerant of faults seems like a better solution than threading the complete game state through everything related to an update - which essentially allows these supposedly transparent areas to side effect by fiddling with the world state directly.

Hmm.

Another fascinating post calls out the lack of a simple dictionary type as a major implementation headache for games in the chosen functional language (Erlang). That seems to be tacit support for the 'property bag' model of game entities in a functional context. I wonder how many other game programmers are lurking out there in the wilds of the internet, ready and willing to provide useful feedback from the experience of using these patterns in functional languages, if only my Google-fu were strong enough to find them.

More Entity Broken-ness

Haskell is weird.

Having dispensed with that, I got some great comments about ways to approach entity composition after my Doing it Wrong post a couple of days ago. Something which I'd given little attention was brought up, and I'm thinking I should give it a second chance: representing entities as name/value pairs.

A bit of stumbling around later, and I found this huge post by Steve Yegge talking about the same thing, with extra bonus links at the end for additional reading. I'm still digesting it.

There do seem to be some unique challenges attached, though, in addition to the inherent oddity that are types in such a scheme (I'd also like to take this opportunity to bitch about type erasure under the JVM: bitch bitch bitch whine hate hate).

For starters, entities want to be able to refer to each other. At the very least, many will want a reference to a 'parent' so I can implement a simple scheme for inheriting properties. That's all fine, apart from the fact that I can't actually store references to objects because they're immutable, constantly subject to being deleted and recreated with different data. So there must be some kind of handle or ID system built in - no big problem, it's a useful thing to have in the scheme of things. Except then you need to provide a some kind of reference to the list of current objects, somewhere, for the handle/ID to search within and produce an entity object when really needed. Urgh, OK, we can do that... a simple function which returns the current world state's entity list will do, and provided entities and their clients behave that should be kindof safe.

Except it's not, of course. When a given entity acts, several other entities may be changed in an indeterminate order (invalidating the old list), and the world itself might pass through several intermediate steps during the update process - I can't currently see a way to ensure that entities always deal with the correct list of entities. It's possible an implicit variable might be a way ahead in Scala, providing a kind of weakly dynamic scope for the entity list. That doesn't sound like a good solution though, it sounds like a hack.

Of course, there's also some unrelated bookkeeping attached to ensuring that entity handles/IDs are always valid, unique, and easily generated. Eww.

On the plus side, if I vaccillate back to wanting Lua scripting, it would probably integrate very nicely with Lua's solitary data structure (last I checked, anyway, everything was an associative array in Lua). But then, ensuring or even pretending immutability in the presence of Lua could be a whole different kettle of fish...

Do other roguelikes ever use a scripting language? Choosing to have one is a pretty big step for any game, even if you don't roll your own and produce a hideous abomination (I'm currently using some middleware at work that provides it's own scripting language, somehow combining the concision of VB with all the elegance of COBOL, speed of Ruby, rapid iteration capabilties of C++ and expressiveness of a wheelbarrow of dead badgers) and most roguelikes appear to be more amenable to being data- rather than script-driven.

Pfft. I should go write some code. This is all getting a bit abstract.

Functional Update Musings

Recently I've been focussing a lot on the rendering part of this project. Quite fruitfully at times, but it certainly wasn't my intention.

The reason for this is simply when I sat down to work on some gameplay stuff, I found a hard problem, and the solutions I've arrived at so far have not been... nice.

As fair warning, this is a brain dump, and a long, boring one at that. It contains much partly-functional waffle because my brain is full of partially functional waffle.

Background: Mostly for my own illumination, I've been trying to use more functional idioms in this game than is strictly sensible. I've mentioned before that I wish to have a (mostly) immutable world, for example. The world update step therefore creates a new world state every 'turn'. The current world state is stored in a mutable variable at the end of each update, but other than that the entire update should be immutable and cuddly from a gameplay viewpoint.

Problem: Creating a suitable update step for my basic case is very easy. The only entities that can affect the world are Agents. Each agent has a (poorly named) 'energy' integer value. When an agent has zero energy, it can act. Actions themselves are the things that change the world state, they are essentially partial functions of type Action :: WorldState -> WorldState, and almost all have the secondary effect of reducing the current agent's energy value when they're executed. After the current agent has acted, the agent list is sorted, some simple transformations are applied to the entity list (dead creature agents are replaced with corpse entities, for example) and time is advanced to bring another agent into relevancy.

This works after a fashion, but certain things are more sensibly represented as continuous functions of time than as agents that act with some (possibly inconsistent) frequency. A better way might be to allow these functions of time to run after the current agent acts, as conveniently we implicitly have the delta time before the next agent is ready to act. This will allow us to damage creatures in lava/acid, or heal creatures that are regenerating, for example. Easy enough, we can make something like TimeFunction :: Int -> WorldState -> WorldState. This is very ugly in practise, especially if there is no ordering defined over the set of active TimeFunctions (consider the joyous difference in observed state if a timefunction spawns a monster and then another damages/kills it, versus damaging monsters then spawning them). There are also potentially huge problems if the list of active TimeFunctions is part of the world state, subject to being updated by TimeFunctions as they run. Blurgh!

There's another big source of ugly too: edge detection. In general, we're quite interested in observing certain changes in world state. An example which Andrew Doull writes about here is a quest to scare seven goblins, which I think can be done by detecting changes in the afraid state of goblin entities. If we perform a comparison of world state before and after the player-controlled agent acts, we can in theory unambiguously attribute any newly-scared goblins to the player and update quest progress. Again we can create something like: EdgeDetector :: WorldState -> WorldState -> WorldState, but it's even nastier now than the case for TimeFunctions. We also have three transitions that may be of interest and could have associated detectors: comparing the original world state to that after the current agent acts; post-agent-action world state to post-timefunction world state; and original state to post-timefunction world state. This would allow us to differentiate between the player killing a goblin with a mace and the goblin burning to death because it stupidly fell into some lava, even in the absence of creatures storing their last attacker or whatever.

It may help to constrain these ugly bits somewhat. For example mandating that TimeFunctions must define a strict ordering and only affect the list of entities and their own internal state, whilst EdgeDetectors can only update their internal state. This in turn means that some other object has to dig through the EdgeDetectors to extract useful data, but I suppose that's not too evil. Both edge detectors and time functions are created and destroyed only by agents, which is closer to the simplicity of the initial design.

But even with these slightly draconian measures in place, a single update is a far nastier affair than I feel it has any right to be, with multiple intermediate world states, and I'm unwilling to set this down in code outside my messy prototype framework.

I've been wrestling with the few papers I've found talking extensively about reactive/game programming in functional languages (by which I mean Haskell) but it's dense, dry, nearly unintelligible stuff as far as I'm concerned. Admittedly they're generally solving the far harder problem of realtime games, whereas I currently go the cheatsy route of using a great deal of mutable state for all realtime code paths such as rendering.

Anyway, brain dump complete. I hope I'll find a better solution and can come back and mock my doubtless shocking ignorance in this post, but right now it just makes me uneasy. The net result of which is my experimented with deferred rendering pipelines and funky post process effects, so its not all bad.

GLState Snippet

Nothing particularly big and clever, just something that seems to work quite well for slightly wrapping up OpenGL state in Scala (complete with manual and probably incomplete syntax highlighting):

trait Bound {
  def unbind: Unit
}
trait Bind {
  protected def bind: Bound
  def boundWith( p: =>Unit ) = {
    val b = bind
    p
    b.unbind
  }
  def &&&(other: Bind) = new Bind {
     class CompositeBound( a:Bound, b:Bound ) extends Bound {
       def unbind = {
        a.unbind
        b.unbind
      }
    }
    protected def bind = {
      new CompositeBound( Bind.this.bind, other.bind )
    }
  }
}

And an implementation for those wonderful glEnable/glDisable chains:

object GLUnchanged extends Bound{ def unbind = () }
object GLStateUtil {
  def glSet( s:Int, v:Boolean ) = if(v) GL11.glEnable(s) else GL11.glDisable(s)
}
case class GLEnabledBound( state:Int, value:Boolean ) extends Bound {
  def unbind = GLStateUtil.glSet(state,value)
}
case class GLSetEnabled( state:Int, value:Boolean ) extends Bind {
  protected def bind = if( GL11.glIsEnabled(state) == value ) {
    GLUnchanged
  } else {
    GLStateUtil.glSet( state, value )
    GLEnabledBound( state, !value )
  }
}

And finally a stupid example of usage (I'm refactoring a lot of the rendering code, so I don't have a real example to hand right now):

val playerState = GLSetEnabled( GL13.GL_TEXTURE_CUBE_MAP, false ) &&&
    GLSetEnabled( GL11.GL_TEXTURE_2D, false ) &&&
    GLSetEnabled( GL11.GL_LIGHTING, false )

playerState boundWith {
  playerModel.render
}

I suspect there's a better way to handle the Bound class returned such that it's not sat in the open, and the terminology may change quite a bit (bind/unbind came from various C++ implementations of similar stuff, none this nice to use). I quite like the way state changes can be composed, although it doesn't even attempt to perform any optimisation on the combined set.

I think it could also be made less verbose to good effect, and possibly a way to skip the redundant state checks would also be nice in case the state of the GL changes due to things outside the scope of these classes - especially for debugging.

At some point I think I'll combine all the resources into implementations of this class.

Anyway, back to playing with code...

Sub-textures

I apologise for the lack of lighting, but as I haven't gotten around to making the normal map equivalents it seemed best. This is just a simple way to add extra variation to the dungeon, by doing a crazy version of texture splatting basically lifted wholesale from here (the post on the 10^th April, can't find a direct link to the appropriate post).

At the moment, the blend weightings (violent primary colours above) are simply normalised fBm output, but adding in some stuff based on local environment should give far nicer results... some moss growth around water, for example, and sand/mud under the water level. Anyway, these are used to control the placement of four textures (other half of the above image). You can see the red corresponds to my original ground texture, green is a mossy muddy thing, and blue is some dry earth stuff. I'll hopefully get a better picture with normal maps and generally nicer textures up soon.

To diverge into a rant briefly, OpenGL support under Vista (ugly ugly ugly Vista) is shocking, at least for my nasty Intel chipset. To add insult to injury, there's no tex2Dgrad equivalent under GLSL even on my shiny desktop hardware, which will make some of my planned shaders a complete PITA to implement. So expect to see fewer pretty screenshots and more waxing lyrical about things like dungeon generation in the near future...

Right, I'm off to a weekend of Dwarf Fortress and beer.

Chasm++

A very very small change, but a nice one. Although there are nashty, nashty evils going on where the higher-res terrain hits the lower-res chasm geometry, they're not so visible form this PoV.

Hooray!

Bridges and chasms

Not a huge amount of change recently. Binding keys to console commands now works relatively well, and I've fiddled with the map to allow for chasms (tiles with no floor at all), and with that bridges, spurs and ledges. There are a couple of niggles with it (witness the raised lips
around chasm tiles, also the strange shelf on the pillar) but it's a start.

Bridges and things are implicit at the moment; if a floor tile has chasm spaces flanking it in any direction it's counted as a bridge or ledge tile, with no solid matter holding it up. This does mean an isolated floor tile will in fact be floating in space rather than appearing the top of a supporting pillar, but this is an easy exception case. Generally it works well, and I hope for many dramatic battles over bridges spanning apparently infinite depths. Erm, not that they'll really be infinite of course, the plan is that falling into a chasm will drop you onto a deeper dungeon level with much pain and suffering.

I should really put some more coarse geometry under the terrain to give some impression of depth there... and remove the nasty water level hack... and add water... gah. So much to do.

As an aside I also refactored the terrain mesh generation code, and as a result it's now a lot faster. Not fast enough by a long shot, but it was a nice easy start. Easy apart from the hideous annoying stupid bug that ate my entire lunch break of course, but that's to be expected.

Minor improvements

A few additional tweaks to things.

• Items! They don't do anything beyond have a name, bulk, owner and position. But you can pick them up and drop them. They're currently all rendered as little purple spheres, making the game look like an overengineered, underfeatured turn-based pacman clone.
• Console commands! Swapped away from hard-coded key associations to binding keys to console strings. This isn't fully working yet, but it's looking positive.
• Better font page generation! I found some blatant stupidity in the generator which was causing different characters to be offset stupidly, causing the ugly bumpy text. Fixed it, and text now looks better.
• Coloured text! This was trivial, but it's nice to have. I also adjusted the rendering so text is somewhat softer. Although this makes it look a little blurry currently, it removes some aliasing artifacts.
  

More screenies o'doom

So, because we all know how fascinating empty caves are, here's some more screenshots. Of course, I got no modelling done at all over the weekend, electing instead to laze about and play games. D'oh. Just pretend that green sphere is some neat player model, and the yellow ones are hideous monsters. Do it.

Anyway, specular! It's overdone for testing purposes, but it does make the surfaces more interesting (if less realistic).


Next, a more Roguelike view of the environment. The top of the cavern has been lopped off, and a hacky bit of shader code put in place to blend the tops of walls/pillars to black. Eww. Still, this is the main option for RL gameplay, and it'd be quite easy to do something less hideous.


All 'round, there's a real need for shadows and a lot to be done to reduce the ambient lighting etc. plus the top-down view obviously needs the texture scaling fiddled with. There are many interesting questions still with the rendering, such as how to handle different terrain types, important features, environmental lighting, water, terrain destruction... oooh, it makes me all tingly and afeared.

Cavern discoveries

One thing I love about procedural content in a project, is that you discover little things that just look... interesting. Of course, making them actually interesting, or even consistently aesthetically pleasing, is a huge challenge.




I discovered this little shelf in the cavern ceiling and thought it was interesting enough to screenshot. Maybe I'm weird.

Regardless, it's a bit easier to see the normal mapping in this shot... might even consider parallax mapping for close bits like this... and it also displays some of the ugly stretching artefacts characteristic of triplanar texture mapping. The floor is currently very prone to such horrors, and I suspect they'll plague the project from now on. I suspect I can massage my normal generation to minimise them, but we'll see.

This weekend, it's my aim to venture into the very scary world of Blender to make some clutter. Rocks, mushrooms, bizarrely verdant foliage... the world needs Stuff!

What I'd really like to do is get some placeholder monsters, items and the like in there. I fear such is waaay beyond my negligible talent though.

The stage is set...

printf( "Hello, world" );

Let's set the tone; this will (hopefully) track the course of my attempts at writing home-grown games. As a professional codemonkey, sometimes the very last thing I want to do of an evening is stare at an IDE and write yet more code, so I'm thinking this will be sporadically updated.

Still, the goad of updating this journal may help push along development of ideas when the going gets rough. Writing code to do shiny things is easy... finishing even a little pet game is stupidly hard. There's so much to do that is painful, or dull, or otherwise requires more commitment than can be scavenged from the damp recesses of my soul.

My current, hopefully not-doomed project is a simple 3D Roguelike. For a quick rundown of the Roguelike concept and some common elements, Wikipedia is helpful as always. Mine own effort is intended to have traditional 2D movement and such, but representing the world with 3D eyecandy. The hope is to end up with a visually rich and immersive environment for the same kind of thoughtful play evoked by a good Roguelike.

Other games have done similar shiny things to the Roguelike formula. Diablo (and all its offspring, legitimate and by-blows both) springs to mind, with a special mention to Fate as I played it relatively recently. These sacrifice a lot of the complexity of the originals in updating the gameplay to be real-time, with an interface suitable for the quick reactions required by such. And very fun they are, but I miss turn-based combat occasionally.

Anyway, the high-level design is rough, and for the moment I'm focusing on getting the base technology up and running; I'd like to get the crucial elements affording roguelike gameplay implemented using 3D elements rather than a flat ASCII map and see where that leads.

For completeness, a small screenshot of the progress so far. Not much, but its only been in development for a few days... and no asset pipeline means nasty test meshes and textures. You can just about make out the solid black 'shadows' showing the field of view.