The Fool on the Hill: Post Scarcity state of play, April 2026

I write up the state of play of the Post Scarcity Software Environment in a developer's log on a fairly regular basis; but it only gets published (here) when I publish a new release, which is to say, rarely.

I assume that no one else at all is interested in the project, that it is simply my lone obsession; but, my blog is also mostly my lone obsession, and, on the off-chance that someone out there is sufficiently mad to be following along, I thought I would publish soft-of-monthly updates on my blog. The discipline of doing so may help me stay focused on the project, too.

So here's the first such.

20260424

To have c_ functions or not to have c_ functions, revisited

Right, I was hugely pleased with my 'make everything a Lisp function, and then call it from C' idea. I wrote things like:

        print( make_frame( 2, base_of_stack,
                 eval( make_frame( 1, base_of_stack,
                         read( make_frame( 1, base_of_stack, input_stream ) ) ) ),
                 output_stream ) );

Isn't it beautiful? Isn't it elegant? Isn't it clear? Yes, it is. Does it work? Yes, actually, it does. Is it a total crock? Unfortunately, dear reader, it is. In this pattern, we don't have a handle on any of the stack frames made with make_frame, so we can't dec_ref them, so they don't get garbage collected. And while during bootstrap it's inevitable that there's a little crud left over because it was created before we have enough infrastructure set up, what I'm seeing at present from a 'start up and shut down run' is

The 452 unfreed objects in size class two are cons cells and string fragments, and they mostly represent the metadata on the streams *in*, *out*, *log* and *sink*, all of which are deliberately protected from garbage collection because, frankly, you don't want those things going away under you; so that's kind of OK. The one in size class three is an exception, and I'm quite pleased I'm only throwing one exception during bootstrap (although it would be nice it it got cleaned up).

But the 45 unfreed objects in size class four are stackframes, and the reason they're unfreed is the coding pattern you see above.

So, how to get around this?

The code snippet above could be rewritten:

		struct pso_pointer next = inc_ref( make_frame(1, base_of_stack, input_stream));
		struct pso_pointer read_value = inc_ref(read(next));
		dec_ref( next);
		
		next = inc_ref( make_frame(1, base_of_stack, read_value));
		struct pso_pointer eval_value = inc_ref( eval( next));
		dec_ref( next);
		dec_ref( read_value);
		
		next = inc_ref( make_frame(2, base_of_stack, eval_value, output_stream));
		print( next);
		dec_ref( next);
		dec_ref( eval_value);

This is much more prolix and, to me, less elegant; but it does get the garbage collected. In each stanza we're first setting up a frame with the arguments for the function we're about to call, then calling that function with the frame we've set up, and then dec_refing the frame. We shouldn't need to dec_ref the value returned by print, since we don't use it and the only thing holding a reference to it is the frame in which it was created, which we do dec_ref.

I could dec_ref read_value, for instance, as soon as I've put it into the frame for eval rather than after eval has actually been invoked, since the frame is now protecting it from garbage collection; but I've delayed doing so until afterwards out of caution.

Once we have eval/apply working, we won't need to do all this bureaucratic incrementing and decrementing of reference counts explicitly, since eval/apply should take care of it automatically.

I'm still not 100% confident I can make the reference counting garbage collector work reliably, irrespective of whether it's actually efficient.

To recode or not to recode?

There are 55 calls to make_frame in existing C code, and they're almost all written in the 'elegant but insanitary' pattern. Could they be rewritten more cleanly? Yes, they could. But my hope is most of this code will be replaced with code written in Lisp, once we have Lisp sufficiently bootstrapped to make that possible.

So I think I'm going to put up with the uncollected garbage until we get to that point, at which point I'll audit the C code to see what is actually still in use, sanitise that, and delete the rest.

However, any new C code (and there is going to have to be some) must be written in the sanitary but bureaucratic pattern.

21:24

Well, at the end of the day I think the git log says it all:

commit 63906fe817d509adb6171a72d16c045c2793ebed (HEAD -> feature/reengineering-17-21)
Author: Simon Brooke <simon@journeyman.cc>
Date:   Fri Apr 24 21:20:23 2026 +0100

    Print is less badly broken. Read is less badly broken. GC is too aggressive.

commit 22b0160a266999c939c9a21df150542f8b2f0b25 (origin/feature/reengineering-17-21)
Author: Simon Brooke <simon@journeyman.cc>
Date:   Fri Apr 24 09:22:06 2026 +0100

    Builds and runs, but print is badly broken. Need some rethink.

I could just disable the garbage collector until I've got eval/apply working. I believe that with eval/apply I'll be able to automate all the garbage collection bookkeeping work. I hope so. Mark and sweep, or even my preferred mark but don't sweep, on a massively parallel machine, just doesn't bear thinking on.

20260420

Still on side projects, but those side-projects are giving me thinking time; and over the past few days I've logged four issues that I've tagged Architecture change.

These are:

• 17: Add readtables; implement quote and keyword through readtables.
• 18: Consider converting from wchar_t to char32_t, everywhere.
• 20: Environment in stack frame.
• 21: Temporary objects in a function must be bound to a locals slot in the stack frame

These, especially the last, mean a fundamental change not only to the Lisp calling convention, but also to everything which may create objects — even if they're never expected to be called directly from Lisp. Generally, every such thing must be called with the standard Lisp calling convention (and so potentially could be called directly from Lisp), except for those very rare things where calling them with the standard calling convention would cause a runaway infinite recursion (the obvious ones are the constructors for stack_frame and cons, but there may be others); and the Lisp calling convention has to change. Which means a lot of things which have already been written for 0.1.0 have to change.

So I have this morning started a new feature branch, feature/reengineering-17-21, to work on these four issues together; and I think the first thing to do is to audit the existing code for functions that are affected by these changes (I mean: every Lisp-callable function is affected by 20, but apart from that). This may also resolve the [MANAGED_POINTER_ONLY](https://git.journeyman.cc/simon/post-scarcity/src/commit/812a1be7d9eb97c25aa07477eb71605b1af93397/src/c/payloads/function.h#L16) issue (see 20260415). I may leave that in as a compile time switch because passing the unmanaged pointer is certainly a performance optimisation, but it will make writing the compiler a bit harder.

I'm not ignoring the fact that a lot of stuff in 0.1.0 is still fundamentally broken, and the REPL still doesn't work; but getting the calling convention right at this point is still the right thing to do, and won't make any of those problems worse. Indeed, it may resolve some of them.

I think this week is going to be mostly a thinking week — partly because the weather forecast is unusually benign, and it would be sensible get some outdoor work done.

21:30

Right, I have spent a lot of time hauling timber out of the wood today, but I've also done a substantial amount of coding, doing a sort of hybrid not-quite-standard-lisp calling convention; and I'm now convinced all this work is wrong and needs to be backed out, and I need to go for full on Lisp calling convention.

So where I'm now calling make_cons as in this sample:

struct pso_pointer c_reverse( struct pso4* frame, struct pso_pointer sequence ) {
    struct pso_pointer result = nil;

    for ( struct pso_pointer cursor = sequence; !nilp( sequence );
          cursor = c_cdr( cursor ) ) {
        result = make_cons( frame, c_car( cursor ), result );
    }
    
    return result;
}

we would instead be doing this:

struct pso_pointer reverse( struct pso_pointer frame) {
    struct pso_pointer sequence = fetch_arg( frame, 0);
    struct pso_pointer result = nil;
    
    for ( struct pso_pointer cursor = sequence; !nilp( sequence );
          cursor = cdr( make_frame( 1, frame, cursor ) ) {
        result = cons( make_frame( 2, frame, 
                                  car( make_frame( 1, frame, cursor )),
                                  result);
    }

    return result;
}

Note that instead of c_reverse, c_cdr, c_car this is using reverse, cdr, car. That's because these are actual Lisp functions, callable from Lisp, which don't have to be duplicated or wrapped in Lisp-compatible wrappers.

This has to be the right way to go.

20260415

OK, I have been diverted down a side-project on a side-project. I decided that since Post Scarcity definitely needs a compiler, I should learn to write a compiler, and so I should start by writing one for a simpler Lisp than Post Scarcity. So I started to write one in Guile Scheme for Beowulf. This is started but a long way from finished. I'm also not very enamoured of Guile Scheme, and am starting to wonder whether in fact I should be writing if in Beowulf for Beowulf.

I do believe I can complete the Naegling/Beowulf compiler, and that having written it, I can write a Post Scarcity compiler in Post Scarcity. But to do that I still need to have to have at least all of

• apply

• assoc

• bind! (or put! or set!, but I think I prefer bind!)

• car

• cdr

• cons

• cond

• eq? (or identical?)

• equal?

• eval

• λ

• nil

• print

• read

• t

  and, essentially, have all the parts of a working REPL.

  My brain is not working very well at present; I can't do more than a very few hours of focussed work a day, and jumping between Naegling and Post Scarcity is probably not a good plan; but in periods when I need to do thinking about where I'm going with Naegling I may switch to Post Scarcity (and vice versa).

Standard signature for compiled functions

While I'm on this, I'm wondering whether I've got the standard signature for compiled functions right. What we've inherited from the 0.0.X branch is documented as:

    /** 
     * pointer to a function which takes a cons pointer (representing
     * its argument list) and a cons pointer (representing its environment) and a
     * stack frame (representing the previous stack frame) as arguments and returns
     * a cons pointer (representing its result).
     * \todo check this documentation is current!
     */
    struct cons_pointer ( *executable ) ( struct stack_frame *,
                                          struct cons_pointer,
                                          struct cons_pointer );

But actually the documentation here is wrong, because what we actually pass is a C pointer to a stack frame object (which in 0.0.X is in vector space), a cons pointer to the cons space object which is the vector pointer to that stack frame, and a cons pointer to the environment.

We definitely don't need to pass a pointer to the argument list (and in fact we didn't before, the documentation is wrong); we also don't need to pass both a C pointer and a cons pointer to the frame, since the frame is now in paged space, so passing our managed pointer is enough.

It might be that passing both an unmanaged and a managed pointer is worth doing, since recovering the managed pointer from the unmanaged pointer is very expensive, and while recovering the unmanaged pointer from the managed pointer is cheap, it isn't free.

But it's worth thinking about.

20260331

Substrate layer print is written; all the building blocks for substrate layer read is in place. This will read far less than the 0.0.6, but it will be extensible with read macros written in Lisp, so much more flexible, and will gradually grow to read more than the non-extensible 0.0.6 reader was. Pleased with myself.

The new print may grow to be extensible in Lisp, as well. In fact, it will have to!

20260326

Most of the memory architecture of the new prototype is now roughed out, but in C, not in a more modern language. It doesn't compile yet.

My C is getting better... but it needed to!