\chapter{State of Play}
\hypertarget{md_workspace_2post-scarcity_2docs_2state-of-play}{}\label{md_workspace_2post-scarcity_2docs_2state-of-play}\index{State of Play@{State of Play}}
\label{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md127}%
\Hypertarget{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md127}%
 \hypertarget{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md128}{}\doxysection{\texorpdfstring{20260215}{20260215}}\label{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md128}
Both of yesterday\textquotesingle{}s regressions are fixed. Memory problem still in much the same state.

\begin{quote}
Allocation summary\+: allocated 1210; deallocated 10; not deallocated 1200. \end{quote}
That left the add ratios problem which was deeper. I had unintended unterminated recursion happening there. \+:-\/(

It burned through 74 cons pages each of 1,024 cons cells, total 76,800 cells, and 19,153 stack frames. before it got there; and then threw the exception back up through each of those 19,153 stack frames. But the actual exception message was {\ttfamily Unrecognised tag value 0 ( )}, which is not enormously helpful.

However, once I had recognised what the problem was, it was quickly fixed, with the added bonus that the new solution will automatically work for bignum fractions once bignums are working.

So we\textquotesingle{}re down to eight unit tests failing\+: the memory leak, one unimplemented feature, and the bignum problem.

At the end of the day I decided to chew up some memory by doing a series of moderately large computations, to see how much memory is being successfully deallocated.


\begin{DoxyCode}{0}
\DoxyCodeLine{::\ (mapcar\ fact\ '(1\ 2\ 3\ 4\ 5\ 6\ 7\ 8\ 9\ 10\ 11\ 12\ 13\ 14\ 15\ 16\ 17\ 18\ 19\ 20))}
\DoxyCodeLine{}
\DoxyCodeLine{(1\ 2\ 6\ 24\ 120\ 720\ 5,040\ 40,320\ 362,880\ 3,628,800\ 39,916,800\ 479,001,600\ }
\DoxyCodeLine{1,932,053,504\ 1,278,945,280\ 2,004,310,016\ 2,004,189,184\ 4,006,445,056\ }
\DoxyCodeLine{3,396,534,272\ 109,641,728\ 2,192,834,560)}
\DoxyCodeLine{::\ }
\DoxyCodeLine{}
\DoxyCodeLine{Allocation\ summary:\ allocated\ 10136;\ deallocated\ 548;\ not\ deallocated\ 9588.}

\end{DoxyCode}


So, about 5\%. This is still a major problem, and is making me doubt my reference counting strategy. Must do better!

Note that the reason that the numbers become eratic past about two billion is the bignum arithmetic bug.\hypertarget{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md129}{}\doxysection{\texorpdfstring{20260214}{20260214}}\label{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md129}
\hypertarget{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md130}{}\doxysubsection{\texorpdfstring{Memory leaks}{Memory leaks}}\label{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md130}
The amount I\textquotesingle{}m leaking memory is now down by an order of magnitude, but the problem is not fixed. Better, not good enough. And although I\textquotesingle{}m aware of the amount to which Lisp objects are not being reclaimed, there may also be transient C objects --- cheifly strings --- which are also not being freed. This is an ongoing process.

But you\textquotesingle{}ll remember that a week ago my base case was\+:

\begin{quote}
Allocation summary\+: allocated 19986; deallocated 245; not deallocated 19741. \end{quote}
Now it\textquotesingle{}s

\begin{quote}
Allocation summary\+: allocated 1188; deallocated 10; not deallocated 1178. \end{quote}
That is better.\hypertarget{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md131}{}\doxysubsection{\texorpdfstring{Unit tests}{Unit tests}}\label{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md131}
The unit test system got into a mess because the bignum tests are failing. But because I know some tests are failing, and the bignum problem feels so intractable that I don\textquotesingle{}t want to tackle it, I\textquotesingle{}ve been ignoring the fact that tests are failing; which means I\textquotesingle{}ve missed regressions --- until I started to get an \textquotesingle{}Attempt to take value of unbound symbol\textquotesingle{} exception for {\ttfamily nil}, which is extremely serious and broke a lot of things.

That arose out of work on the \textquotesingle{}generalised key/value stores\textquotesingle{} feature, logged under \mbox{[}\#20260203\mbox{]}(20260203), below. However, because I wasn\textquotesingle{}t paying attention to failing tests, it took me a week to find and fix it.

But I\textquotesingle{}ve fixed that one. And I\textquotesingle{}ve put a lot of work into \href{https://git.journeyman.cc/simon/post-scarcity/commit/222368bf640a0b79d57322878dee42ed58b47bd6}{\texttt{ cleaning up the unit tests}}. There is more work to do on this.\hypertarget{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md132}{}\doxysubsection{\texorpdfstring{Documentation}{Documentation}}\label{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md132}
I\textquotesingle{}m also gradually working through cleaning up documentation.\hypertarget{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md133}{}\doxysubsection{\texorpdfstring{Regressions}{Regressions}}\label{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md133}
Meantime we have some regressions which are serious, and must be resolved.\hypertarget{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md134}{}\doxysubsubsection{\texorpdfstring{equals}{equals}}\label{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md134}
The core function {\ttfamily equals} is now failing, at least for integers. Also.


\begin{DoxyCode}{0}
\DoxyCodeLine{(=\ 0.75\ 3/4)}

\end{DoxyCode}


fails because I\textquotesingle{}ve never implemented a method for it, which I ought.\hypertarget{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md135}{}\doxysubsubsection{\texorpdfstring{cond}{cond}}\label{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md135}
The current unit test for {\ttfamily cond} and that for {\ttfamily recursion} both fail but {\itshape I think} this is because {\ttfamily equals} is failing.\hypertarget{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md136}{}\doxysubsubsection{\texorpdfstring{rational arithmetic}{rational arithmetic}}\label{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md136}
I have a horrible new regression in rational arithmetic which looks as though something is being freed when it shouldn\textquotesingle{}t be.\hypertarget{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md137}{}\doxysubsubsection{\texorpdfstring{All tests failing as at 20260214}{All tests failing as at 20260214}}\label{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md137}
As follows\+:


\begin{DoxyEnumerate}
\item unit-\/tests/bignum-\/expt.\+sh =\texorpdfstring{$>$}{>} (expt 2 61)\+: Fail\+: expected \textquotesingle{}2305843009213693952\textquotesingle{}, got \textquotesingle{}\textquotesingle{}
\item unit-\/tests/bignum-\/expt.\+sh =\texorpdfstring{$>$}{>} (expt 2 64)\+: Fail\+: expected \textquotesingle{}18446744073709551616\textquotesingle{}, got \textquotesingle{}\textquotesingle{}
\item unit-\/tests/bignum-\/expt.\+sh =\texorpdfstring{$>$}{>} (expt 2 65)\+: Fail\+: expected \textquotesingle{}36893488147419103232\textquotesingle{}, got \textquotesingle{}\textquotesingle{}
\item unit-\/tests/bignum-\/print.\+sh =\texorpdfstring{$>$}{>} unit-\/tests/bignum-\/print.\+sh =\texorpdfstring{$>$}{>} printing 576460752303423488\+: Fail\+: expected \textquotesingle{}576460752303423488\textquotesingle{}, got \textquotesingle{}0\textquotesingle{}
\item unit-\/tests/bignum-\/print.\+sh =\texorpdfstring{$>$}{>} printing 1152921504606846976\+: Fail\+: expected \textquotesingle{}1152921504606846976\textquotesingle{}, got \textquotesingle{}0\textquotesingle{}
\item unit-\/tests/bignum-\/print.\+sh =\texorpdfstring{$>$}{>} printing 1152921504606846977\+: Fail\+: expected \textquotesingle{}1152921504606846977\textquotesingle{}, got \textquotesingle{}1\textquotesingle{}
\item unit-\/tests/bignum-\/print.\+sh =\texorpdfstring{$>$}{>} printing 1329227995784915872903807060280344576\+: Fail\+: expected \textquotesingle{}1329227995784915872903807060280344576\textquotesingle{}, ~\newline
 got \textquotesingle{}0\textquotesingle{}
\item unit-\/tests/bignum.\+sh =\texorpdfstring{$>$}{>} unit-\/tests/bignum.\+sh =\texorpdfstring{$>$}{>} Fail\+: expected \textquotesingle{}1,152,921,504,606,846,976\textquotesingle{}, got \textquotesingle{}0\textquotesingle{}
\item unit-\/tests/bignum-\/subtract.\+sh =\texorpdfstring{$>$}{>} unit-\/tests/bignum-\/subtract.\+sh =\texorpdfstring{$>$}{>} subtracting 1 from 1152921504606846976\+: Fail\+: expected \textquotesingle{}1152921504606846975\textquotesingle{}, got \textquotesingle{}4294967295\textquotesingle{}
\item unit-\/tests/bignum-\/subtract.\+sh =\texorpdfstring{$>$}{>} subtracting 1 from 1152921504606846977\+: Fail\+: expected \textquotesingle{}1152921504606846976\textquotesingle{}, got \textquotesingle{}0\textquotesingle{}
\item unit-\/tests/bignum-\/subtract.\+sh =\texorpdfstring{$>$}{>} subtracting 1 from 1152921504606846978\+: Fail\+: expected \textquotesingle{}1152921504606846977\textquotesingle{}, got \textquotesingle{}1\textquotesingle{}
\item unit-\/tests/bignum-\/subtract.\+sh =\texorpdfstring{$>$}{>} subtracting 1152921504606846977 from 1\+: Fail\+: expected \textquotesingle{}-\/1152921504606846976\textquotesingle{}, got \textquotesingle{}0\textquotesingle{}
\item unit-\/tests/bignum-\/subtract.\+sh =\texorpdfstring{$>$}{>} subtracting 10000000000000000000 from 20000000000000000000\+: Fail\+: expected \textquotesingle{}10000000000000000000\textquotesingle{}, got \textquotesingle{}2313682944\textquotesingle{}
\item unit-\/tests/cond.\+sh =\texorpdfstring{$>$}{>} unit-\/tests/cond.\+sh\+: cond with one clause... Fail\+: expected \textquotesingle{}5\textquotesingle{}, got \textquotesingle{}nil\textquotesingle{}
\item unit-\/tests/memory.\+sh =\texorpdfstring{$>$}{>} Fail\+: expected \textquotesingle{}1188\textquotesingle{}, got \textquotesingle{}10\textquotesingle{}
\item unit-\/tests/ratio-\/addition.\+sh =\texorpdfstring{$>$}{>} Fail\+: expected \textquotesingle{}1/4\textquotesingle{}, got \textquotesingle{}Error\+: Unrecognised tag value 4539730 ( REE)\textquotesingle{}
\item unit-\/tests/recursion.\+sh =\texorpdfstring{$>$}{>} Fail\+: expected \textquotesingle{}nil 3,628,800\textquotesingle{}, got \textquotesingle{}\textquotesingle{}
\end{DoxyEnumerate}\hypertarget{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md138}{}\doxysubsection{\texorpdfstring{New master version}{New master version}}\label{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md138}
I haven\textquotesingle{}t done a \textquotesingle{}release\textquotesingle{} of Post Scarcity since September 2021, because I\textquotesingle{}ve been so despondent about the bignum problem. But actually a lot of this {\itshape is} usable, and it\textquotesingle{}s at least sufficiently intereting that other people might want to play with it, and possibly even might fix some bugs.

So I\textquotesingle{}m currently planning to release a new master before the end of this month, and publicise it.\hypertarget{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md139}{}\doxysection{\texorpdfstring{20260204}{20260204}}\label{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md139}
\hypertarget{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md140}{}\doxysubsection{\texorpdfstring{Testing what is leaking memory}{Testing what is leaking memory}}\label{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md140}
\hypertarget{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md141}{}\doxysubsubsection{\texorpdfstring{Analysis}{Analysis}}\label{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md141}
If you just start up and immediately abort the current build of psse, you get\+:

\begin{quote}
Allocation summary\+: allocated 19986; deallocated 245; not deallocated 19741. \end{quote}
Allocation summaries from the current unit tests give the following ranges of values\+:

\tabulinesep=1mm
\begin{longtabu}spread 0pt [c]{*{4}{|X[-1]}|}
\hline
\PBS\centering \cellcolor{\tableheadbgcolor}\textbf{ }&\PBS\centering \cellcolor{\tableheadbgcolor}\textbf{ Min   }&\PBS\centering \cellcolor{\tableheadbgcolor}\textbf{ Max   }&\PBS\centering \cellcolor{\tableheadbgcolor}\textbf{ }\\\cline{1-4}
\endfirsthead
\hline
\endfoot
\hline
\PBS\centering \cellcolor{\tableheadbgcolor}\textbf{ }&\PBS\centering \cellcolor{\tableheadbgcolor}\textbf{ Min   }&\PBS\centering \cellcolor{\tableheadbgcolor}\textbf{ Max   }&\PBS\centering \cellcolor{\tableheadbgcolor}\textbf{ }\\\cline{1-4}
\endhead
Allocated   &19991   &39009   &\\\cline{1-4}
Deallocated   &238   &1952   &\\\cline{1-4}
Not deallocated   &19741   &37057   &\\\cline{1-4}
\end{longtabu}


The numbers go up broadly in sinc with one another --- that is to say, broadly, as the number allocated rises, so do both the numbers deallocated and the numbers not deallocated. But not exactly.\hypertarget{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md142}{}\doxysubsubsection{\texorpdfstring{Strategy\+: what doesn\textquotesingle{}t get cleaned up?}{Strategy: what doesn't get cleaned up?}}\label{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md142}
Write a test wrapper which reads a file of forms, one per line, from standard input, and passes each in turn to a fresh invocation of psse, reporting the form and the allocation summary.


\begin{DoxyCode}{0}
\DoxyCodeLine{\#1/bin/bash}
\DoxyCodeLine{}
\DoxyCodeLine{while\ IFS=\ read\ -\/r\ form;\ do}
\DoxyCodeLine{\ \ \ \ allocation=`echo\ \$\{form\}\ |\ ../../target/psse\ 2>\&1\ |\ grep\ Allocation`}
\DoxyCodeLine{\ \ \ \ echo\ "{}*\ \$\{allocation\}:\ \$\{form\}"{}}
\DoxyCodeLine{done}

\end{DoxyCode}


So, from this\+:


\begin{DoxyItemize}
\item Allocation summary\+: allocated 19986; deallocated 245; not deallocated 19741.\+:
\item Allocation summary\+: allocated 19990; deallocated 249; not deallocated 19741.\+: ()
\item Allocation summary\+: allocated 20019; deallocated 253; not deallocated 19766.\+: nil
\end{DoxyItemize}

Allocating an empty list allocates four additional cells, all of which are deallocated. Allocating \textquotesingle{}nil\textquotesingle{} allocates a further {\bfseries{29}} cells, 25 of which are not deallocated. WTF?

Following further work I have this, showing the difference added to the base case of cells allocated, cells deallocated, and, most critically, cells not deallocated.

From this we see that reading and printing {\ttfamily nil} allocates an additional 33 cells, of which eight are not cleaned up. That\textquotesingle{}s startling, and worrying.

But the next row shows us that reading and printing an empty list costs only four cells, each of which is cleaned up. Further down the table we see that an empty map is also correctly cleaned up. Where we\textquotesingle{}re leaking memory is in reading (or printing, although I doubt this) symbols, either atoms, numbers, or keywords (I haven\textquotesingle{}t yet tried strings, but I expect they\textquotesingle{}re similar.)

\tabulinesep=1mm
\begin{longtabu}spread 0pt [c]{*{4}{|X[-1]}|}
\hline
\PBS\centering \cellcolor{\tableheadbgcolor}\textbf{ {\bfseries{Case}}   }&\PBS\centering \cellcolor{\tableheadbgcolor}\textbf{ {\bfseries{Delta Allocated}}   }&\PBS\centering \cellcolor{\tableheadbgcolor}\textbf{ {\bfseries{Delta Deallocated}}   }&\PBS\centering \cellcolor{\tableheadbgcolor}\textbf{ {\bfseries{Delta Not Deallocated}}    }\\\cline{1-4}
\endfirsthead
\hline
\endfoot
\hline
\PBS\centering \cellcolor{\tableheadbgcolor}\textbf{ {\bfseries{Case}}   }&\PBS\centering \cellcolor{\tableheadbgcolor}\textbf{ {\bfseries{Delta Allocated}}   }&\PBS\centering \cellcolor{\tableheadbgcolor}\textbf{ {\bfseries{Delta Deallocated}}   }&\PBS\centering \cellcolor{\tableheadbgcolor}\textbf{ {\bfseries{Delta Not Deallocated}}    }\\\cline{1-4}
\endhead
{\bfseries{Basecase}}   &0   &0   &0    \\\cline{1-4}
{\bfseries{nil}}   &33   &8   &25    \\\cline{1-4}
\texorpdfstring{$\ast$}{*}\texorpdfstring{$\ast$}{*}()\texorpdfstring{$\ast$}{*}\texorpdfstring{$\ast$}{*}   &4   &4   &0    \\\cline{1-4}
\texorpdfstring{$\ast$}{*}\texorpdfstring{$\ast$}{*}(quote ())\texorpdfstring{$\ast$}{*}\texorpdfstring{$\ast$}{*}   &39   &2   &37    \\\cline{1-4}
\texorpdfstring{$\ast$}{*}\texorpdfstring{$\ast$}{*}(list )\texorpdfstring{$\ast$}{*}\texorpdfstring{$\ast$}{*}   &37   &12   &25    \\\cline{1-4}
\texorpdfstring{$\ast$}{*}\texorpdfstring{$\ast$}{*}(list 1)\texorpdfstring{$\ast$}{*}\texorpdfstring{$\ast$}{*}   &47   &14   &33    \\\cline{1-4}
\texorpdfstring{$\ast$}{*}\texorpdfstring{$\ast$}{*}(list 1 1)\texorpdfstring{$\ast$}{*}\texorpdfstring{$\ast$}{*}   &57   &16   &41    \\\cline{1-4}
\texorpdfstring{$\ast$}{*}\texorpdfstring{$\ast$}{*}(list 1 1 1)\texorpdfstring{$\ast$}{*}\texorpdfstring{$\ast$}{*}   &67   &18   &49    \\\cline{1-4}
\texorpdfstring{$\ast$}{*}\texorpdfstring{$\ast$}{*}(list 1 2 3)\texorpdfstring{$\ast$}{*}\texorpdfstring{$\ast$}{*}   &67   &18   &49    \\\cline{1-4}
\texorpdfstring{$\ast$}{*}\texorpdfstring{$\ast$}{*}(+)\texorpdfstring{$\ast$}{*}\texorpdfstring{$\ast$}{*}   &36   &10   &26    \\\cline{1-4}
\texorpdfstring{$\ast$}{*}\texorpdfstring{$\ast$}{*}(+ 1)\texorpdfstring{$\ast$}{*}\texorpdfstring{$\ast$}{*}   &44   &12   &32    \\\cline{1-4}
\texorpdfstring{$\ast$}{*}\texorpdfstring{$\ast$}{*}(+ 1 1)\texorpdfstring{$\ast$}{*}\texorpdfstring{$\ast$}{*}   &53   &14   &39    \\\cline{1-4}
\texorpdfstring{$\ast$}{*}\texorpdfstring{$\ast$}{*}(+ 1 1 1)\texorpdfstring{$\ast$}{*}\texorpdfstring{$\ast$}{*}   &62   &16   &46    \\\cline{1-4}
\texorpdfstring{$\ast$}{*}\texorpdfstring{$\ast$}{*}(+ 1 2 3)\texorpdfstring{$\ast$}{*}\texorpdfstring{$\ast$}{*}   &62   &16   &46    \\\cline{1-4}
\texorpdfstring{$\ast$}{*}\texorpdfstring{$\ast$}{*}(list \textquotesingle{}a \textquotesingle{}a \textquotesingle{}a)\texorpdfstring{$\ast$}{*}\texorpdfstring{$\ast$}{*}   &151   &33   &118    \\\cline{1-4}
\texorpdfstring{$\ast$}{*}\texorpdfstring{$\ast$}{*}(list \textquotesingle{}a \textquotesingle{}b \textquotesingle{}c)\texorpdfstring{$\ast$}{*}\texorpdfstring{$\ast$}{*}   &151   &33   &118    \\\cline{1-4}
\texorpdfstring{$\ast$}{*}\texorpdfstring{$\ast$}{*}(list \+:a \+:b \+:c)\texorpdfstring{$\ast$}{*}\texorpdfstring{$\ast$}{*}   &121   &15   &106    \\\cline{1-4}
\texorpdfstring{$\ast$}{*}\texorpdfstring{$\ast$}{*}(list \+:alpha \+:bravo \+:charlie)\texorpdfstring{$\ast$}{*}\texorpdfstring{$\ast$}{*}   &485   &15   &470    \\\cline{1-4}
\texorpdfstring{$\ast$}{*}\texorpdfstring{$\ast$}{*}\{\}\texorpdfstring{$\ast$}{*}\texorpdfstring{$\ast$}{*}   &6   &6   &0    \\\cline{1-4}
\texorpdfstring{$\ast$}{*}\texorpdfstring{$\ast$}{*}\{\+:z 0\}\texorpdfstring{$\ast$}{*}\texorpdfstring{$\ast$}{*}   &43   &10   &33    \\\cline{1-4}
\texorpdfstring{$\ast$}{*}\texorpdfstring{$\ast$}{*}\{\+:zero 0\}\texorpdfstring{$\ast$}{*}\texorpdfstring{$\ast$}{*}   &121   &10   &111    \\\cline{1-4}
\texorpdfstring{$\ast$}{*}\texorpdfstring{$\ast$}{*}\{\+:z 0 \+:o 1\}\texorpdfstring{$\ast$}{*}\texorpdfstring{$\ast$}{*}   &80   &11   &69    \\\cline{1-4}
\texorpdfstring{$\ast$}{*}\texorpdfstring{$\ast$}{*}\{\+:zero 0 \+:one 1\}\texorpdfstring{$\ast$}{*}\texorpdfstring{$\ast$}{*}   &210   &14   &196    \\\cline{1-4}
\texorpdfstring{$\ast$}{*}\texorpdfstring{$\ast$}{*}\{\+:z 0 \+:o 1 \+:t 2\}\texorpdfstring{$\ast$}{*}\texorpdfstring{$\ast$}{*}   &117   &12   &105   \\\cline{1-4}
\end{longtabu}


Looking at the entries, we see that


\begin{DoxyEnumerate}
\item each number read costs ten allocations, of which only two are successfully deallocated;
\item the symbol {\ttfamily list} costs 33 cells, of which 25 are not deallocated, whereas the symbol {\ttfamily +} costs only one cell fewer, and an additional cell is not deallocated. So it doesn\textquotesingle{}t seem that cell allocation scales with the length of the symbol;
\item Keyword allocation does scale with the length of the keyword, apparently, since {\ttfamily (list \+:a \+:b \+:c)} allocates 121 and deallocates 15, while {\ttfamily (list \+:alpha \+:bravo \+:charlie)} allocates 485 and deallocates the same 15;
\item The fact that both those two deallocate 15, and a addition of three numbers {\ttfamily (+ 1 2 3)} or {\ttfamily (+ 1 1 1)} deallocates 16 suggest to me that the list structure is being fully reclaimed but atoms are not being.
\item The atom {\ttfamily \textquotesingle{}a} costs more to read than the keyword {\ttfamily \+:a} because the reader macro is expanding {\ttfamily \textquotesingle{}a} to {\ttfamily (quote a)} behind the scenes.
\end{DoxyEnumerate}\hypertarget{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md143}{}\doxysubsection{\texorpdfstring{The integer allocation bug}{The integer allocation bug}}\label{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md143}
Looking at what happens when we read a single digit number, we get the following\+:


\begin{DoxyCode}{0}
\DoxyCodeLine{2}
\DoxyCodeLine{Entering\ make\_integer}
\DoxyCodeLine{Allocated\ cell\ of\ type\ 'INTR'\ at\ 19,\ 507\ }
\DoxyCodeLine{make\_integer:\ returning}
\DoxyCodeLine{\ \ \ \ \ \ \ \ INTR\ (1381256777)\ at\ page\ 19,\ offset\ 507\ count\ 0}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Integer\ cell:\ value\ 0,\ count\ 0}
\DoxyCodeLine{Entering\ make\_integer}
\DoxyCodeLine{Allocated\ cell\ of\ type\ 'INTR'\ at\ 19,\ 508\ }
\DoxyCodeLine{make\_integer:\ returning}
\DoxyCodeLine{\ \ \ \ \ \ \ \ INTR\ (1381256777)\ at\ page\ 19,\ offset\ 508\ count\ 0}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Integer\ cell:\ value\ 10,\ count\ 0}
\DoxyCodeLine{Entering\ make\_integer}
\DoxyCodeLine{Allocated\ cell\ of\ type\ 'INTR'\ at\ 19,\ 509\ }
\DoxyCodeLine{make\_integer:\ returning}
\DoxyCodeLine{\ \ \ \ \ \ \ \ INTR\ (1381256777)\ at\ page\ 19,\ offset\ 509\ count\ 0}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Integer\ cell:\ value\ 2,\ count\ 0}
\DoxyCodeLine{Entering\ make\_integer}
\DoxyCodeLine{Allocated\ cell\ of\ type\ 'INTR'\ at\ 19,\ 510\ }
\DoxyCodeLine{make\_integer:\ returning}
\DoxyCodeLine{\ \ \ \ \ \ \ \ INTR\ (1381256777)\ at\ page\ 19,\ offset\ 510\ count\ 0}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Integer\ cell:\ value\ 0,\ count\ 0}
\DoxyCodeLine{Entering\ make\_integer}
\DoxyCodeLine{Allocated\ cell\ of\ type\ 'INTR'\ at\ 19,\ 506\ }
\DoxyCodeLine{make\_integer:\ returning}
\DoxyCodeLine{\ \ \ \ \ \ \ \ INTR\ (1381256777)\ at\ page\ 19,\ offset\ 506\ count\ 0}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Integer\ cell:\ value\ 0,\ count\ 0}
\DoxyCodeLine{Entering\ make\_integer}
\DoxyCodeLine{Allocated\ cell\ of\ type\ 'INTR'\ at\ 19,\ 505\ }
\DoxyCodeLine{make\_integer:\ returning}
\DoxyCodeLine{\ \ \ \ \ \ \ \ INTR\ (1381256777)\ at\ page\ 19,\ offset\ 505\ count\ 0}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Integer\ cell:\ value\ 0,\ count\ 0}
\DoxyCodeLine{Entering\ make\_integer}
\DoxyCodeLine{Allocated\ cell\ of\ type\ 'INTR'\ at\ 19,\ 504\ }
\DoxyCodeLine{make\_integer:\ returning}
\DoxyCodeLine{\ \ \ \ \ \ \ \ INTR\ (1381256777)\ at\ page\ 19,\ offset\ 504\ count\ 0}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Integer\ cell:\ value\ 0,\ count\ 0}
\DoxyCodeLine{}
\DoxyCodeLine{Allocated\ cell\ of\ type\ 'STRG'\ at\ 19,\ 503\ }
\DoxyCodeLine{Freeing\ cell\ \ \ \ STRG\ (1196577875)\ at\ page\ 19,\ offset\ 503\ count\ 0}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ String\ cell:\ character\ '2'\ (50)\ with\ hash\ 0;\ next\ at\ page\ 0\ offset\ 0,\ count\ 0}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ value:\ "{}2"{}}
\DoxyCodeLine{Freeing\ cell\ \ \ \ INTR\ (1381256777)\ at\ page\ 19,\ offset\ 504\ count\ 0}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Integer\ cell:\ value\ 2,\ count\ 0}
\DoxyCodeLine{2}
\DoxyCodeLine{Allocated\ cell\ of\ type\ 'SYMB'\ at\ 19,\ 504\ }
\DoxyCodeLine{Allocated\ cell\ of\ type\ 'SYMB'\ at\ 19,\ 503\ }
\DoxyCodeLine{Allocated\ cell\ of\ type\ 'SYMB'\ at\ 19,\ 502\ }
\DoxyCodeLine{Allocated\ cell\ of\ type\ 'SYMB'\ at\ 19,\ 501\ }
\DoxyCodeLine{Freeing\ cell\ \ \ \ SYMB\ (1112365395)\ at\ page\ 19,\ offset\ 501\ count\ 0}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Symbol\ cell:\ character\ '*'\ (42)\ with\ hash\ 485100;\ next\ at\ page\ 19\ offset\ 502,\ count\ 0}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ value:\ *in*}
\DoxyCodeLine{Freeing\ cell\ \ \ \ SYMB\ (1112365395)\ at\ page\ 19,\ offset\ 502\ count\ 0}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Symbol\ cell:\ character\ 'i'\ (105)\ with\ hash\ 11550;\ next\ at\ page\ 19\ offset\ 503,\ count\ 0}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ value:\ in*}
\DoxyCodeLine{Freeing\ cell\ \ \ \ SYMB\ (1112365395)\ at\ page\ 19,\ offset\ 503\ count\ 0}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Symbol\ cell:\ character\ 'n'\ (110)\ with\ hash\ 110;\ next\ at\ page\ 19\ offset\ 504,\ count\ 0}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ value:\ n*}
\DoxyCodeLine{Freeing\ cell\ \ \ \ SYMB\ (1112365395)\ at\ page\ 19,\ offset\ 504\ count\ 0}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Symbol\ cell:\ character\ '*'\ (42)\ with\ hash\ 0;\ next\ at\ page\ 0\ offset\ 0,\ count\ 0}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ value:\ *}

\end{DoxyCode}


Many things are worrying here.


\begin{DoxyEnumerate}
\item The only thing being freed here is the symbol to which the read stream is bound --- and I didn\textquotesingle{}t see where that got allocated, but we shouldn\textquotesingle{}t be allocating and tearing down a symbol for every read! This implies that when I create a string with {\ttfamily c\+\_\+string\+\_\+to\+\_\+lisp\+\_\+string}, I need to make damn sure that that string is deallocated as soon as I\textquotesingle{}m done with it --- and wherever I\textquotesingle{}m dealing with symbols which will be referred to repeatedly in {\ttfamily C} code, I need either
\begin{DoxyEnumerate}
\item to bind a global on the C side of the world, which will become messy;
\item or else write a hash function which returns, for a {\ttfamily C} string, the same value that the standard hashing function will return for the lexically equivalent {\ttfamily Lisp} string, so that I can search hashmap structures from C without having to allocate and deallocate a fresh copy of the {\ttfamily Lisp} string;
\item In reading numbers, I\textquotesingle{}m generating a fresh instance of {\ttfamily Lisp zero} and {\ttfamily Lisp ten}, each time {\ttfamily read\+\_\+integer} is called, and I\textquotesingle{}m not deallocating them.
\item I am correctly deallocating the number I did read, though!
\end{DoxyEnumerate}
\end{DoxyEnumerate}\hypertarget{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md144}{}\doxysection{\texorpdfstring{20260203}{20260203}}\label{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md144}
I\textquotesingle{}m consciously avoiding the bignum issue for now. My current thinking is that if the C code only handles 64 bit integers, and bignums have to be done in Lisp code, that\textquotesingle{}s perfectly fine with me.\hypertarget{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md145}{}\doxysubsection{\texorpdfstring{Hashmaps, assoc lists, and generalised key/value stores}{Hashmaps, assoc lists, and generalised key/value stores}}\label{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md145}
I now have the oblist working as a hashmap, and also hybrid assoc lists which incorporate hashmaps working. I don\textquotesingle{}t 100\% have consistent methods for reading stores which may be plain old assoc lists, new hybrid assoc lists, or hashmaps working but it isn\textquotesingle{}t far off. This also takes me streets further towards doing hierarchies of hashmaps, allowing my namespace idea to work --- and hybrid assoc lists provide a very sound basis for building environment structures.

Currently all hashmaps are mutable, and my doctrine is that that is fixable when access control lists are actually implemented.\hypertarget{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md146}{}\doxysubsubsection{\texorpdfstring{assoc}{assoc}}\label{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md146}
The function {\ttfamily (assoc store key) =\texorpdfstring{$>$}{>} value} should be the standard way of getting a value out of a store. ~\newline
\hypertarget{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md147}{}\doxysubsubsection{\texorpdfstring{put!}{put!}}\label{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md147}
The function {\ttfamily (put! store key value) =\texorpdfstring{$>$}{>} store} should become the standard way of setting a value in a store (of course, if the store is an assoc list or an immutable map, a new store will be returned which holds the additional key/value binding).\hypertarget{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md148}{}\doxysubsection{\texorpdfstring{State of unit tests}{State of unit tests}}\label{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md148}
Currently\+:

\begin{quote}
Tested 45, passed 39, failed 6 \end{quote}
But the failures are as follows\+: 
\begin{DoxyCode}{0}
\DoxyCodeLine{unit-\/tests/bignum-\/add.sh\ =>\ checking\ a\ bignum\ was\ created:\ Fail}
\DoxyCodeLine{unit-\/tests/bignum-\/add.sh\ =>\ adding\ 1152921504606846977\ to\ 1:\ Fail:\ expected\ 't',\ got\ 'nil'}
\DoxyCodeLine{unit-\/tests/bignum-\/add.sh\ =>\ adding\ 1\ to\ 1152921504606846977:\ Fail:\ expected\ 't',\ got\ 'nil'}
\DoxyCodeLine{unit-\/tests/bignum-\/add.sh\ =>\ adding\ 1152921504606846977\ to\ 1152921504606846977:\ Fail:\ expected\ 't',\ got\ 'nil'}
\DoxyCodeLine{unit-\/tests/bignum-\/add.sh\ =>\ adding\ 10000000000000000000\ to\ 10000000000000000000:\ Fail:\ expected\ 't',\ got\ 'nil'}
\DoxyCodeLine{unit-\/tests/bignum-\/add.sh\ =>\ adding\ 1\ to\ 1329227995784915872903807060280344576:\ Fail:\ expected\ 't',\ got\ 'nil'}
\DoxyCodeLine{unit-\/tests/bignum-\/add.sh\ =>\ adding\ 1\ to\ 3064991081731777716716694054300618367237478244367204352:\ Fail:\ expected\ 't',\ got\ 'nil'}
\DoxyCodeLine{unit-\/tests/bignum-\/expt.sh\ =>\ (expt\ 2\ 60):\ Fail:\ expected\ '1152921504606846976',\ got\ '1'}
\DoxyCodeLine{unit-\/tests/bignum-\/expt.sh\ =>\ (expt\ 2\ 61):\ Fail:\ expected\ '2305843009213693952',\ got\ '2'}
\DoxyCodeLine{unit-\/tests/bignum-\/expt.sh\ =>\ (expt\ 2\ 64):\ Fail:\ expected\ '18446744073709551616',\ got\ '16'}
\DoxyCodeLine{unit-\/tests/bignum-\/expt.sh\ =>\ (expt\ 2\ 65):\ Fail:\ expected\ '36893488147419103232',\ got\ '32'}
\DoxyCodeLine{unit-\/tests/bignum-\/print.sh\ =>\ printing\ 1152921504606846976:\ Fail:\ expected\ '1152921504606846976',\ got\ '1'}
\DoxyCodeLine{unit-\/tests/bignum-\/print.sh\ =>\ printing\ 1152921504606846977:\ Fail:\ expected\ '1152921504606846977',\ got\ '2'}
\DoxyCodeLine{unit-\/tests/bignum-\/print.sh\ =>\ printing\ 1329227995784915872903807060280344576:\ Fail:\ expected\ '1329227995784915872903807060280344576',\ \(\backslash\)n\ \ \ \ \ \ \ \ \ \ \ got\ '1151321504605245376'}
\DoxyCodeLine{unit-\/tests/bignum.sh\ =>\ unit-\/tests/bignum.sh\ =>\ Fail:\ expected\ '1,152,921,504,606,846,976',\ got\ '1'}
\DoxyCodeLine{unit-\/tests/bignum-\/subtract.sh\ =>\ unit-\/tests/bignum-\/subtract.sh\ =>\ subtracting\ 1\ from\ 1152921504606846976:\ Fail:\ expected\ '1152921504606846975',\ got\ '0'}
\DoxyCodeLine{unit-\/tests/bignum-\/subtract.sh\ =>\ subtracting\ 1\ from\ 1152921504606846977:\ Fail:\ expected\ '1152921504606846976',\ got\ '1'}
\DoxyCodeLine{unit-\/tests/bignum-\/subtract.sh\ =>\ subtracting\ 1\ from\ 1152921504606846978:\ Fail:\ expected\ '1152921504606846977',\ got\ '2'}
\DoxyCodeLine{unit-\/tests/bignum-\/subtract.sh\ =>\ subtracting\ 1152921504606846977\ from\ 1:\ Fail:\ expected\ '-\/1152921504606846976',\ got\ '1'}
\DoxyCodeLine{unit-\/tests/bignum-\/subtract.sh\ =>\ subtracting\ 10000000000000000000\ from\ 20000000000000000000:\ Fail:\ expected\ '10000000000000000000',\ got\ '-\/376293541461622793'}
\DoxyCodeLine{unit-\/tests/memory.sh}

\end{DoxyCode}


In other words, all failures are in bignum arithmetic {\bfseries{except}} that I still have a major memory leak due to not decrefing somewhere where I ought to.\hypertarget{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md149}{}\doxysubsection{\texorpdfstring{Zig}{Zig}}\label{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md149}
I\textquotesingle{}ve also experimented with autotranslating my C into Zig, but this failed. Although I don\textquotesingle{}t think C is the right language for implementing my base Lisp in, it\textquotesingle{}s what I\textquotesingle{}ve got; and until I can get some form of autotranslate to bootstrap me into some more modern systems language, I think I need to stick with it.\hypertarget{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md150}{}\doxysection{\texorpdfstring{20250704}{20250704}}\label{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md150}
Right, I\textquotesingle{}m getting second and subsequent integer cells with negative values, which should not happen. This is probably the cause of (at least some of) the bignum problems. I need to find out why. This is (probably) fixable.


\begin{DoxyCode}{0}
\DoxyCodeLine{::\ (inspect\ 10000000000000000000)}
\DoxyCodeLine{}
\DoxyCodeLine{\ \ \ \ \ \ \ \ INTR\ (1381256777)\ at\ page\ 3,\ offset\ 873\ count\ 2}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Integer\ cell:\ value\ 776627963145224192,\ count\ 2}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ BIGNUM!\ More\ at:}
\DoxyCodeLine{\ \ \ \ \ \ \ \ INTR\ (1381256777)\ at\ page\ 3,\ offset\ 872\ count\ 1}
\DoxyCodeLine{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Integer\ cell:\ value\ -\/8,\ count\ 1}

\end{DoxyCode}


Also, {\ttfamily print} is printing bignums wrong on ploughwright, but less wrong on mason, which implies a code difference. Investigate.\hypertarget{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md151}{}\doxysection{\texorpdfstring{20250314}{20250314}}\label{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md151}
Thinking further about this, I think at least part of the problem is that I\textquotesingle{}m storing bignums as cons-\/space objects, which means that the integer representation I can store has to fit into the size of a cons pointer, which is 64 bits. Which means that to store integers larger than 64 bits I need chains of these objects.

If I stored bignums in vector space, this problem would go away (especially as I have not implemented vector space yet).

However, having bignums in vector space would cause a churn of non-\/standard-\/sized objects in vector space, which would mean much more frequent garbage collection, which has to be mark-\/and-\/sweep because unequal-\/sized objects, otherwise you get heap fragmentation.

So maybe I just have to put more work into debugging my cons-\/space bignums.

Bother, bother.

There are no perfect solutions.

However however, it\textquotesingle{}s only the node that\textquotesingle{}s short on vector space which has to pause to do a mark and sweep. It doesn\textquotesingle{}t interrupt any other node, because their reference to the object will remain the same, even if it is the \textquotesingle{}home node\textquotesingle{} of the object which is sweeping. So all the node has to do is set its busy flag, do GC, and clear its busy flag, The rest of the system can just be carrying on as normal.

So... maybe mark and sweep isn\textquotesingle{}t the big deal I think it is?\hypertarget{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md152}{}\doxysection{\texorpdfstring{20250313}{20250313}}\label{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md152}
OK, the 60 bit integer cell happens in {\ttfamily int128\+\_\+to\+\_\+integer} in {\ttfamily \doxylink{integer_8c}{arith/integer.\+c}}. It seems to be being done consistently; but there is no obvious reason. {\ttfamily MAX\+\_\+\+INTEGER} is defined in {\ttfamily \doxylink{peano_8h}{arith/peano.\+h}}. I\textquotesingle{}ve changed both to use 63 bits, and this makes no change to the number of unit tests that fail.

With this change, {\ttfamily (fact 21)}, which was previously printing nothing, now prints a value, {\ttfamily 11,891,611,015,076,642,816}. However, this value is definitively wrong, should be {\ttfamily 51,090,942,171,709,440,000}. But, I hadn\textquotesingle{}t fixed the shift in {\ttfamily integer\+\_\+to\+\_\+string}; have now... still no change in number of failed tests...

But {\ttfamily (fact 21)} gives a different wrong value, {\ttfamily 4,974,081,987,435,560,960}. Factorial values returned by {\ttfamily fact} are correct (agree with SBCL running the same code) up to {\ttfamily (fact 20)}, with both 60 bit integer cells and 63 bit integer cells giving correct values.

Uhhhmmm... but I\textquotesingle{}d missed two other places where I\textquotesingle{}d had the number of significant bits as a numeric literal. Fixed those and now {\ttfamily (fact 21)} does not return a printable answer at all, although the internal representation is definitely wrong. So we may be seeing why I chose 60 bits.

Bother.\hypertarget{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md153}{}\doxysection{\texorpdfstring{20250312}{20250312}}\label{md_workspace_2post-scarcity_2docs_2state-of-play_autotoc_md153}
Printing of bignums definitely doesn\textquotesingle{}t work; I\textquotesingle{}m not persuaded that reading of bignums works right either, and there are probably problems with bignum arithmetic too.

The internal memory representation of a number rolls over from one cell to two cells at 1152921504606846976, and I\textquotesingle{}m not at all certain why it does because this is neither 2\textsuperscript{63} nor 2\textsuperscript{64}.

\tabulinesep=1mm
\begin{longtabu}spread 0pt [c]{*{3}{|X[-1]}|}
\hline
\PBS\centering \cellcolor{\tableheadbgcolor}\textbf{ }&\PBS\centering \cellcolor{\tableheadbgcolor}\textbf{ }&\PBS\centering \cellcolor{\tableheadbgcolor}\textbf{ }\\\cline{1-3}
\endfirsthead
\hline
\endfoot
\hline
\PBS\centering \cellcolor{\tableheadbgcolor}\textbf{ }&\PBS\centering \cellcolor{\tableheadbgcolor}\textbf{ }&\PBS\centering \cellcolor{\tableheadbgcolor}\textbf{ }\\\cline{1-3}
\endhead
2\textsuperscript{62}   &4611686018427387904   &\\\cline{1-3}
2\textsuperscript{63}   &9223372036854775808   &\\\cline{1-3}
2\textsuperscript{64}   &18446744073709551616   &\\\cline{1-3}
Mystery number   &1152921504606846976   &\\\cline{1-3}
\end{longtabu}


In fact, our mystery number turns out (by inspection) to be 2\textsuperscript{60}. But {\bfseries{why}}? 