The Fool on the Hill: The Game Engine Problem

[this note was first written on the 18th April 2024 and substantially revised on the 25th of May and 24th of June]

I'm at the stage with The Great Game that I need to make a choice of game engine. This doesn't mean that I believe that the game can ever be completed (at least not by me, in my lifetime), or that it will ever be in any real sense playable. I don't. I think that's really unlikely.

Of course a game can be built from scratch, from the ground up. But a game engine provides a lot of useful services — for example asset management, renderer, physics engine — each of which would be vastly time consuming either to build from scratch, or to integrate from unrelated third party libraries, as a single coherent whole with a single API. That's a pretty big win; especially so if you're building a fairly vanilla game which can be built largely with these pre-prepared components, less so if you're breaking a lot of new ground. But what I am planning here breaks so much new ground that I really need as much help as I can get.

But, all my exploratory work thus far on the game has been in Clojure, and Clojure (or another Lisp dialect, but in my opinion Clojure is the best Lisp dialect currently in common use) is what I find most productive to use: it's most efficient of my time. But there is currently no serious game engine which seriously supports Clojure.

Any game engine is a large and complex software system, which requires a very considerable investment in time before it can be used well. So this is an important decision.

So the question this note sets out to address is, which is the right choice of game engine, now, for me?

A digression: NightMod

There is one Clojure-native game engine, NightMod (sources here). It works and it's delightful, but it seems aimed primarily as a teaching tool, and at developing simple games like 2D platformers. There are no fully 3D examples given; and, sadly, it's no longer under active development.

The Candidates

So what are the game engines I could use? The candidates are, as I see it,

In this table I've assigned jMonkeyEngine '3' (out of 5) for 'Documentation' and for 'Community', to give a base for comparison. Greater than 3 means 'better than jMonkeyEngine'; less than 3, worse. This isn't to sat "jMonkeyEngine is average." I'm using this scaling because jMonkeyEngine is my probable choice. Its documentation is OK-ish — there's a lot of it but it is of varying quality and some parts are very out of date; its community seems to have a small number of active members, but those active members are both knowledgeable and responsive.

Working with Clojure or Lisp

As I've said, none of the game engines I'm considering directly supports Clojure.

But Clojure is a language which compiles down to the Java Virtual Machine, and consequently it's fairly easy for pieces of code written in Java and in Clojure to efficiently exchange data. A hybrid Clojure/Java program can exist in a single process. That's score one to jMonkeyEngine. There's also a really good open source library, jme-clj which greatly helps integrating Clojure code with jMonkeyEngine-based games. I'll discuss this in more detail later.

There was at one stage considerable support for a source-compatible variant of Clojure targeting Microsoft's 'Common Language Runtime' (CLR) virtual machine, which would make a hybrid Clojure/C# program equally viable. The project still exists, but it's unclear how well supported it is. If it works well, then Godot and Unity would also be able to integrate Clojure code into a single process.

Clojure cannot be interfaced directly in a single project with C++, but Common Lisp can via the CFFI library. And while Common Lisp is not my first choice of language (and I don't know of any current Common Lisp tooling which I'm comfortable with), the mutability of Common Lisp data structures, while being a feature of which I generally disapprove, is likely to have some performance benefits in game code. So from an interoperability point of view, a Common Lisp/C++ single-process game would also be feasible.

The separation of concerns

However, it isn't necessary for all the code of a game to run in a single process, and the things I'm doing in Lisp (currently Clojure) are neatly discrete from rendering high quality animated models on a screen. So a system in which a 'client' process, handling user interaction and visual display, and written in C++ or C#, communicated over a socket pair with a 'server' process which handled character knowledge, conversation planning, maintenance of world state, economy models and so on, is entirely feasible. The Aleph library, for example, would support this.

What I am doing in the layer that's currently in Clojure (for this discussion let's call it 'the server') is primarily knowledge, needs, planning, intention, conversation: things which are fundamentally symbolic. They percolate up to the visible and audible layers written in the game engine's native code (let's call this 'the client') because a character's intention, based on their knowledge and their needs, will cause them to plan to go to places in the world and to do things there, and consequently the player may see them moving or see them acting. The player may interact with them by speaking to them, in which case again the interaction is symbolic: the player creates a speech act, which is a stream of words, and the character responds by finding appropriate items in their knowledge base and formulating those into idiomatic speech to be spoken to the player.

So assuming that I can do real-time JALI-like lip and jaw animation and text-to-speech generation at the client side of the bridge, and speech-to-text interpretation of the player's speech acts also at the client side of the bridge, all that needs to be passed across the bridge are

1. Location information, in the form of xyz tuples;
2. Movement information, in the form of a tuple comprising a character identifier and a location, with possibly some sort of animation modifier (running, walking, marching, sneaking);
3. Animation-to-play information, in the form of a tuple comprising a character identifier and the symbolic identifier of the animation;
4. Speech act information, in the form of a tuple comprising a character identifier of the speaker, an (optional) character identifier of the character addressed, a string of text to speak, an identifier of a voice profile to speak it in, an (optional) emotional modifier;

In other words, these can be fairly simple and constrained packets of information which can be passed across the bridge in the form of packets whose payload is either EDN or JSON. This cannot be fire-and-forget: the server layer needs to know when an action which was requested has been completed, or whether for some reason it failed.

But in practice this isn't hard to conceptualise, and although it's easier if the client and server are both running in a single JVM process (which is only possible if the client is running in jMonkeyEngine), it isn't hugely difficult even if the two processes are separate. It's frankly not even difficult if they're running on physically separate machines.

Open Source and The Great Game

So let's consider the candidates in reverse order; but first, let's talk about open source, and where I consider the boundary lies between my game as a particular game, and the technology which I create to support my game but which others could use to create their own games.

I am intellectually committed to open source; it's deeply embedded in my anarchist view of the world. It's my opinion that a world in which everything that is made is freely gifted to those who need it or wish to make use of it, in which all knowledge is freely shared, is just a better world.

The reason an open source game engine matters is that a lot of what a game engine provides is in the form of library functions, which means that a lot of game engine code exists in any working implementation of every game built with it. Of course one could distribute the code of one's own game open source, and require those who wished to play it to download their own copy of the game engine and compile the game themselves; but, firstly, that's a pretty high bar anyone would have to jump through to play the game, and secondly, game engines in active use are all being actively developed and updated, so any code I released would have to be continually maintained to keep in step with the game engine it was written for, or it would become unusable.

The alternative is to release the game as a conventional, proprietary, closed product. That's what the makers of the commercial game engines expect you to do and want you to do: they make their money not on selling their game engine to you, but taking a cut of the price at which you sell your game to others.

Of course, I could release the code parts of my game as a stand-alone module, communicating over a socket pair, which other people who were interested in using it in other games could integrate with the game engine of their choice.

In any case, I see the code that makes the game work, and the actual world and story of the game, as separate things. I intend to give both away for free, yes; but I'll give the code away with the intention that others should be able to freely copy and modify it as suits them. I'm not yet certain how happy I am with people modifying the story elements; I certainly wouldn't want modified versions being distributed as instances of my game.

But that's a problem for if I ever get anything sufficiently finished that there is a game that other people can actually play. And realistically, that isn't very likely. If I ever have a finished game, it would be much better if it was built on an open source engine; but if I'm never going to, then it really doesn't matter.

Wicked Engine

Wicked Engine is an open source game engine, in C++ but with support for Lua scripting. Demonstrations are impressive, with high quality, fairly naturalistic graphics. It appears to be a one-person project, without substantial community support. The code looks good, and the build process works; however, I have not yet manage to create a build that runs. If there is documentation, I cannot find it from the website. I'm therefore not evaluating further now.

Unreal Engine

Unreal Engine is almost certainly the best game engine out there today. It's certainly the best supported and has the most features which would make building the sort of world I want to build easy. It's an engine designed, built and used to build very large, very cinematic, game worlds, and is used by many large game studios to produce very high production value games. It's probably the engine in which I could most quickly produce a playable game, and it's certainly the engine in which I could most easily create a very good looking game.

It also has a lot of the tooling I need, and an active marketplace of components made by third parties, many of which would be useful (but these are also generally not open source, and not free).

It's written in C++, and encourages the writing of game code mainly in C++, but as I say that's not an insuperable problem. What feels for me more of a problem is that the game I produced (if it ever were finished) could not be given away. I probably never will produce anything playable -- I'm too old, it's too big a project, and I have too many other things I must do. If I give up the idea of ever finishing, then Unreal Engine would be a good choice. But if I then, to my surprise, did finish, that would be sad.

Unity

Unity is also a big commercial game engine, with all of the disadvantages for me that that brings; and somewhat worse than Unreal, because the company that makes it is seeking to take a bigger cut of game revenue. Against that, it has no significant benefits vis-a-vis Unreal; it can be used to build very high quality games, but its support for large open worlds seems to be poorer and its ecosystem of third-party add-ons seems to be poorer. The company which produces Unity has also laid off a significant number of staff in the past year, and may be in trouble.

I see no benefits of Unity over Unreal. Compared to the open source engines it's better supported and has a better portfolio of high quality games produced with it, but if you feel that the support and portfolio offered by proprietary engines justifies making the compromise of using them, then it seems to me Unreal is just better.

Trial

I confess I discovered Trial late into this process and have not yet investigated it as much as I'd like. There is a huge amount of work behind it, and in fact it appears quite feature-rich; however, it is also very clearly a work in very active progress and so far I have only got about half of the example projects to build and run. The upside is that it is in Lisp (albeit Common Lisp, against which I have to swallow quite a lot of prejudice), and that the principle author is someone I have mutuals in common with and seems very approachable.

There are commercially available games built with Trial, but they appear both to be 2D.

In summary this is definitely worthy of further investigation, and may become my preferred choice.

O3DE

O3DE (Open Three Dimensional Engine) started life as a fork of CryEgine, a big commercial game engine, was bought by Amazon, and was then open sourced by them. They (and several other corporates) still seem to be supporting development, however. The GitHub activity page shows that it is under very active development, with several contributors making frequent contributions. It's clearly aimed at high production value, AAA style, games. Documentation is extremely extensive; how up to date it is I can't yet say.

CryEngine itself was developed to support realistic looking fairly open world games, and games made with it include Kingdom Come Deliverance, which is entirely up to the highest production standards I might aspire to.

A .deb package for O3DE is available from the project website (a snap package is also available but I have not tried that); the documentation suggests Ubuntu, but in practice this installs and seems to run satisfactorily on Debian stable. The source pulled from GitHub also builds without difficulty and runs.

In everything except Clojure integration. this is a very good candidate.

jMonkeyEngine

jMonkeyEngine has been in continuous development for fifteen years and seems to have a reasonable community and a dedicated development team. It's much less sophisticated than Unreal Engine. It has far fewer features and there is far less third party support. There are no very high production value games that have been produced using jMonkeyEngine; this doesn't really matter because there's no way I could really produce such a game anyway. The production quality of the best games produced with it is adequate.

However, it's open source, it targets the Java virtual machine which makes it play reasonably nicely with Clojure, and it's good enough. The tooling is OK; the jMonkeyEngine SDK is essentially a heavily modified version of NetBeans, but is none the worse for that. And there's lots of experience out there of using it in concert with other open source tools, such as Blender.

Of relevance to me is that it is primarily a 3D engine, with good terrain and physics support; and while no-one has yet tried to use it to build as large a world as I envisage, the architecture should in principle support large open worlds.

jme-clj

There's also an excellent library, jme-clj, that not only makes it easy to integrate Clojure code with jMonkeyEngine code, but in practice allows you to do a great deal of jMonkeyEngine development from the Clojure REPL, which I find much more congenial. However, at present, jme-clj is lagging considerably behind the core jMomkeyEngine libraries, and doesn't seem to be currently actively maintained. So if I use it, which if I commit to jMomkeyEngine I shall certainly want to do, I'm going to have to take on at least a share of the maintenance of jme-clj myself. That doesn't feel unduly onerous; it's a well written library.

Godot Engine

Godot engine is more than a decade old, and has again been under continuous development since then, with a good development team and plenty of community support; and it's also open source. While its 'home' language is C#, and its default target is the CLR virtual machine, it does have support for a number of other languages. And the very best games written with Godot have massively better production values than any games produced with jMonkeyEngine. It is also said to have a much bigger community and much better community support, although I haven't yet personally explored this.

However, it seems a poorer fit for me. It's primarily a 2D engine, and doesn't seem to support large 3D games well. There is no native terrain support, although there is a third party terrain plugin available. Also, while a number of languages are directly supported, neither Clojure nor Java are.

Fyrox

Fyrox, like Wicked Engine, seems to me to be a one-man project. As such it's pretty impressive, and the choice of Rust as an implementation is definitely forward looking. The documentation is also good. Having said that, it doesn't appear to have much adoption yet and I can see no evidence of an active community.

Flax

Again, a one man project, although in this case it's someone who I'm told has a good reputation, and there is something of an active community. There is good documentation. Although it's source-available it's not open source. It is apparently quite impressive, but although a Linux build is available from the project website, it does not run on Debian.

Character Editors

The first thing I'm tackling with integrating my code into a game engine is creating characters using simulated genetics. For that, I don't strictly need a character editor, but I need a lot of the things which underpin a character editor, and this is the problem I'm currently butting my head against.

• I can't find an official Godot character editor, but there's at least one impressive-looking open source third party one, here (there's also a video presentation of this);
• jMonkeyEngine doesn't have a built-in character editor component, but one of the core members of the team has built a character editor called Maud. This is more of a technology demonstrator than anything else — it works, but the user interface is anything but intuitive — however, it gives me enough to build on;
• O3DE apparently has a character editor, but the tutorial page has not yet been written;
• Unreal Engine has an extremely impressive character editor;

Procedural Terrain and Vegetation

One thing The Great Game needs is a huge amount of procedural terrain — it has to be procedural, as the world is much too big for it all to be designed. That terrain needs to be procedurally vegetated, and that vegetation needs to vary with biome.

• There are a number of tutorials for producing procedural terrain for Godot, and it's reasonably straightforward;
• There's at least one procedural vegetation library for Godot (video here), and the quality is adequate;
• There are many procedural terrain tutorials for jMonkeyEngine, too, although it has to be said none of them are outstanding;
• The best jMonkeyEngine procedural vegetation demo I've seen is this student project, which is adequate;
• There's an official presentation of O3DE procedural terrain and vegetation generation, and to be honest it's embarrassingly unimpressive; but I'm reasonably confident something better can be done with it!
• Needless to say, Unreal Engine has hugely impressive terrain and vegetation tools.

Tentative Conclusion

It seems to me that of the commercial engines, Unreal is the clear winner. Of the open source engines, O3DE seems technically the best, but at present jMonkeyEngine (or possibly Trial) feels like the better fit for me. The choice for open source comes down to two issues:

1. Ideologically, open source matters very much to me;
2. If I actually do ever finish anything, then if it's built with open source libraries I can give it away; if it's not, I can't.

The choice between open source game engines (and indeed to use a game engine at all rather than building from scratch) comes down to a group of other issues

1. Ease of integration with what I'm currently doing; jMonkeyEngine and Trial clearly win here;
2. Stability and maturity of the platform; again, jMonkeyEngine scores highly here (although Godot is also worth considering);
3. Informed and helpful community is another of jMonkeyEngine's strong points;
4. Polish, and support for large game worlds, rather points to O3DE;
5. Commercial support and speed of development should be on the side of O3DE, but the quality of their public presentations really isn't good and it doesn't look like a team with clear focus or good morale.

I think that means that, although I think I could create a better-looking game with Unreal or O3DE, and that I'd be more likely to get something finished with Unreal, jMonkeyEngine (by which I mean, largely, jme-clj) is the correct choice for me. However, I have thus far had much more trouble getting things working with jMonkeyEngine than I anticipated, and it might be worth running a parallel investigation of Trial.

Appendix: Glossary of the languages mentioned here

(This whole section is an opinionated digression; feel free to ignore it.)

Lisp

Lisp is how you write software properly. Rather than being a bunch of hacks which just happened to be convenient to implement, it is a systematic implementation of the lambda calculus, which is the fundamental model of computation. This makes it enormously powerful, flexible and expressive. It's also, by computing standards, old, dating back to the 1950s. So it's gone through a lot of development over the years, and has developed a number of dialects, of which currently the most influential is Common Lisp. For a number of aesthetic and technical reasons, Common Lisp is not a dialect I much admire.

C++

Once upon a very long time ago, Cambridge University was building one 'high level' computer language, and London University was building another. The UK government, unwilling to fund development of two competing languages, persuaded them to collaborate on a thing called the Combined Programming Language, or CPL. But the research interests of the two teams did not align, and it seems that egos got in the way; in any case, the project collapsed. Out of the wreckage came BCPL, a one person project by Martin Richards, a member of the Cambridge Team.

BCPL was a very simple, very low level, untyped, block structured imperative language, with the particularly interesting feature that the (very small) compiler compiled down not to native code for a particular machine, but for what was called 'CINT code' (compact interpreted code), which was in effect code for what we would now call a virtual machine. The compiler itself was written in BCPL; it was thus a self hosted language. As such, it was extremely easy to get BCPL working on new hardware architectures — all you had to do was port the CINT code interpreter, which was a small and simple program, and all existing BCPL programs, including the BCPL compiler, immediately became available on the new architecture. Consequently, BCPL became quite influential among people working on experimental new machines in the late 1960s and early 1970s.

Ken Thompson and Dennis Richie of Bell Labs in the US took BCPL, made a few minor syntactic changes, and, more significantly, stripped out the CINT code layer altogether in favour of having the compiler generate code for the actual (PDP) hardware they were using. They called this 'B'. Richie then want on to add a macro preprocessor as a standard part of the compile pipeline; he called the resulting construct 'C'. Thompson and Richie then used C to write UNIX, itself also the scaled down successor to a failed collaborative megaproject.

It's worth pointing out here that C's macro preprocessor is a very simple text macro processor, operating simply on the source code as a stream of characters, rather than operating on the semantic structure of the language as Lisp macros do.

UNIX became extraordinarily influential, partly because it is a very good operating system, but partly because it was given away to universities for free and consequently widely used in computer science teaching; with the consequence that its native language, C, also became very influential.

In the late 70s, Bjarne Stroustrup, then also working at Bell Labs, became interested in object oriented programming, but found that the object oriented languages available to him ran too slowly on the machines he had. So he used C's macro preprocessor to write a set of extensions to C which enabled him to write object oriented programs. He called this confection C++.

C++ has become an extremely influential language, and is very widely used. It can produce fast code — but it can because at the bottom of this tottering stack of intellectual endeavour there's still the spiritual successor to that tiny little non-type-safe, non-bounds-checked compiler that Martin Edwards hacked together as a side project in the computer lab at Cambridge back in 1967. This isn't to say anything bad about Edwards; he was undoubtedly a very able man. His objective was to write a very small compiler, to run on very small machines. He succeeded admirably.

Java

Back in the 1990s, streaming video seemed to be a potential product that consumers were going to want, and in order to deliver it to people's then-analogue televisions, some sort of device — a 'set top box' — was going to be needed to decode the digital signal (and, in a commercial market, also descramble it, because it was expected to be encrypted so that providers could prevent people who hadn't paid from watching). This device didn't need to be very high powered but was expected to be produced in very large quantities, and Sun Microsystems wanted a share of this market.

So James Gosling and Patrick Naughton, at Sun, wrote a special purpose language to run on small, low-power devices with limited memory. This was a then-modern language: it was type safe and bounds checked, it had automatic memory management. To make it easily portable they adopted the same strategy that Martin Richards had twenty five years before: they had their compiler compile down not to the actual machine instructions of a particular hardware architecture, to those of a small, easily portable CINT-code interpreter, or, as they called it, a virtual machine. They called this language 'Oak'.

To make it feel familiar to the programmers whom they hoped would adopt it, who had mostly been taught C and/or C++ at university, they gave Oak a syntax which was extremely like that of C (and, consequently, a direct linear descendant of BCPL's syntax).

The marketing department at Sun disliked the name 'Oak', so they renamed the language to 'Java'.

Because Java programs were compiled for a virtual machine, which was small and intentionally portable, the same, unchanged program could run on any machine architecture to which the virtual machine had been ported; because Java had been designed to target very small, low powered machines, the compiled programs were relatively small. These two features made it very easy to pass compiled Java executables over the network without worrying too much about what the machine on the other end was. This made Java very popular; it became the most influential computer language of the early years of this century.

Lua

Lua is a simple lightweight interpreted scripting language which is extremely easy to integrate into other software, and is consequently widely used as a scripting language by game engines.

C#

Sun Microsystems, as we've discussed, wrote a language called Java. Microsoft copied it, and then 'embraced and extended' it. Sun sued, and the judge told Microsoft to stop messing around. So Microsoft changed just enough about their version of Java to claim that it was a 'different language', and called it C#. It was compiled for a virtual machine which Microsoft called 'the Common Language Runtime', which is very like, but of course incompatible with, the Java virtual machine. The consequence is, C# and Java are now incompatible in all sorts of subtle ways, but they're technically and syntactically very, very similar. C# is, in summary, little more than a fit if pique as software.

Clojure

Clojure is a modern dialect of Lisp written to compile for the Java virtual machine, and consequently to integrate well with the Java ecosystem. It also has a feature which I value highly, which is that it treats data as immutable except in very special circumstances. This makes it excellent for making efficient use of machines with significant numbers of processors, which is to say, most machines we use these days.

Share on social media

I'm no longer offering links to kleptocrat owned social media. If you're still using it, please think about whether you should be.