I ended my last post noting that I had seemingly successfully overcome the disaster that was the Atari’s wide-character support.  There were a few additional hiccups along the way of finalizing the wide-character support.  After having replaced almost every wide-character source file in the MiNTLib, I was still having issues with paths, for example.  It turned out that I had failed to replace the function mbstowcs() since it lived in a file I hadn’t expected.  Finding that gem led to implementing most of musl’s multibyte functionality as well.  With mbstowcs() implemented properly, my Python interpreter was finally reading in actual Python code from its standard library.

The MiNTLib was also missing two minor definitions.  The first was sched_yield(), which appeared defined in a header file, but kept popping up as undefined during the linking phase.   Searching the library didn’t reveal any actual function, so I dropped in a simple stand-in that basically asked the program to sleep for a very brief moment.  The second was nl_langinfo(), a simple function for retrieving a specific aspect of the locale.  Since the Atari’s support of locale’s was pretty minimal in most respects, a basic implementation would suffice.  After these last two bugs, I was able to compile.

I finally found myself faced with a nasty little error.  The interpreter would complain about an ImportError of some sort upon startup.  The most annoying aspect of this particular ImportError was that it was reported by the interpreter as “Could not import runpy module.”  Doesn’t seem particularly useful, right?  Having never heard of such a module, I looked in the standard library for the offending code.  There was a runpy module, but there was no obvious reason why it couldn’t be imported.  I next tried a search to hunt down where the error was issued from.  It turns out that the error originated in the interpreter startup code in Modules/main.c.  I was annoyed to find that, if the import of this module failed for whatever reason, it would spit out this short, vague message and exit.  An error was available, and I still don’t know why the developer who authored this code didn’t spit out the resulting exception.  To fix the issue, I decided to dump the Python error to stdin.

The source of the error appeared to be a failure to import the imp module, which provides much of the necessary infrastructure for, well, importing modules.  The source of the failure was a line where some functions from the _imp module were imported.  The _imp module, in turn, is a C module compiled into Python itself.  I had actually encountered it earlier when trying to figure out the wide-character mess, mistakenly assuming the C _imp module was actually the imp module, which was failing to load thanks to the MiNTLib (in fact, the _imp module internally thinks it is named the imp module…).

The error occurred because the imp module was attempting to load the function load_dynamic from the _imp module.  This function is linked to a pure-C function, imp_load_dynamic(), that is only compiled on platforms where shared libraries are supported.  Because Atari computers effectively have no true shared objects, I disabled all shared libraries during the build, meaning imp_load_dynamic() was never defined.  Becuase it wasn’t defined, the imp module was throwing an error during import.

This bug was reported on the core Python bug tracker after revealing itself.  The simple fix is to either not import load_dynamic at all, which works dandy for the Atari, or to set load_dynamic to None if not present, which is the suggested fix.  By fixing this minor issue, the interpreter was able to spring to life, presenting me finally with a fully function Python 3.3 prompt:

In my previous post about bringing Python 3.3 to the Atari, I mentioned that the end result was yet another completely nonfunctional Python 3.3 executable. In an effort to find out what had gone terribly wrong, I fired up GDB on the ARAnyM virtual machine to sort out what had occurred.

I started with letting the interpretter run to see if an error that I could understand had occurred.  The problem with starting this way is that the Python interpretter is anything but simple.  I don’t mean this description in a bad way, just that blindly walking into what is occurring is somewhat daunting.  In the end, I couldn’t figure a thing out.

The interpretter was crashing with two major symptoms.   First, it complained about the environment and ‘<prefix>’ directories not being defined properly.  This crash seemed particularly odd since it shouldn’t be looking for environment settings; the crash was occurring as  a step in the Python build process after the executable was built but before the system was installed.  Second, it threw an import error complaining it couldn’t import the imp module.  Not loading the imp module concerned me greatly as this was necessary to import anything else.

I started with trying to figure out the failure to import imp. Looking through the builtin C modules, I noticed the import module was trying to call itself  imp. However, a bit further down the road, it was actually assigned the name _imp. In the Lib directory, however, there was an imp.py, which was actually what was necessary.  I spent some time making sure it actually wanted to load the pure-Python file via a combination of GDB and some printf statements.

If this basic module was failing to be pulled in, I though that perhaps something was wrong with the bootstrap process.  At startup, most of the Python fun kicks off with importlib._bootstrap.  I noticed that this Python file existed in the Lib directory, and I decided to throw a few clever print() statements into the code to see where things were going wrong.  Changing this file, however, changed nothing.

Back in the Python C code, I added a few more printf statements to try and track down the issue.  When I rebuilt the executable, though, I was met with an entirely new build error.  An executable named freeze_importlib had caused a nasty error.  This error had obviously never occurred before because I always arrived at a complete, but not working, executable.  Examining the command calling this executable, it was clearly reading in importlib._bootstrap and writing out a header file.

Firing up GDB, I ran freeze_importlib with the same arguments as the Makefile provided.  After a bit of tracing, I found that the failure came when it attempted to compile the Python to bytecode.  I was at a loss.

There were a number of points, though, that I had to keep in mind:

1. The build didn’t work on the Atari platform – it worked just fine almost everywhere else, though
2. Python 2.7 did work on the Atari platform – there was some massaging, but nothing extreme
3. Python 3.x uses Unicode for almost everything – or that was my understanding at least, while Python 2.7 did not

Looking at these three points, I suspected the issue stemmed from mintlib’s embarrassingly incomplete support for wide character functions.  Recall that I had grafted on three functions, wcschr, wcsrchr, and wcstok, from FreeBSD just to get Python to compile.  I had to do a similar procedure with Python 2.7, but I never tested any Unicode functions under that interpretter.  I decided to go out looking for some proper wide character support.

Obviously I wasn’t going to bother looking at GNU libc’s implementations.  I was concerned that those might be a bit confusing.  Using Wikipedia,  I searched for some C standard library implementations that weren’t GNU libc.  I ended up downloading a handful of them and examining their support for wide characters.  The musl implementation, though, seemed the smallest, simplest, and most straightforward.  I especially liked that it was, “Designed from the ground up for static linking,” since there’d be no dynamic linking under FreeMiNT on the Atari.

I didn’t want to replace all of mintlib with musl, just every bit of wide character support.  In the mintlib build tree, I deleted every mintlib mention of wide character functionality, and I created some dynamic links back to the musl library’s directory tree.  After a few speedbumps and editing a list of C files in mintlib, I successfully built a new mintlib with every wide character function replaced.

Back in the Python directory, I decided to reattempt building the Python executable against my new mintlib.  I cleaned and restarted the build process (slow even with ARAnyM on a relatively quick Athlon-based system) and waited expectantly.  I had not un-modified the print() additions in importlib._bootstrap, however, meaning that the freeze_importlib executable would be run again.  When the Makefile reached the freeze step, I smiled as it progressed happily past building the necessary header file, continuing on with the build process.

My smile didn’t last long, though, as the Python executable was launched by the make process.  Once again, an ImportError popped up saying the interpretter couldn’t import imp.  Something was still amiss…

Back in 2011, I successfully attempted to get Python 2.7 running on the Atari platform, culminating in a PyOhio presentation demonstrating the port. I pointed out during that presentation that my ultimate goal was to bootstrap Python 3.x on the Atari, but my successful compilation of the interpreter yielded an executable that didn”t seem to do anything. After having a subsequent talk rejected from PyCon (twice, actually, and I”m starting to think nobody cares about Python on the Atari…), I let the project simply fade.

Over the last few weeks, however, I decided to give it a shot once again for fun. I recompiled the latest ARAnyM emulator for two of my desktops and set out again on this adventure in retrocomputing.

About the Atari

My goal was to compile Python 3.3 for the FreeMiNT operating system running on the Atari ST/TT/Falcon line of computers. The last Atari-built computer was probably manufactured in the early to mid nineties, making the platform somewhat dated. The machines ran on the Motorola 68k series of CPUs. The FreeMiNT operating system, originally known as MiNT, then MultiTOS, then MiNT again, and finally FreeMiNT, is an odd concept to people unfamiliar with the Atari platform. Basically, the operating system provides a BSD kernel with full preemptive multitasking for the Atari machines. The odd part is that FreeMiNT doesn”t actually replace the entirety of the Atari”s operating system in ROM, TOS; rather, it simply replaces the parts that need replacing to allow multitasking. It”d be as if the Linux kernel could boot from DOS, but allow DOS sound drivers to manage sound cards or something equally as odd.

FreeMiNT is normally beefed up with a significant number of add-ons. Basically, my current install on the ARAnyM virtual machine boots into a Atari TOS-compatible GUI comprised of a replacement Virtual Device Interface (fVDI) and Application Environment Services (XaAES). Additionally, using TosWin2, I can open a bash prompt on the machine, where it behaves similar to most UNIX-y free operating systems, the big difference being that it continues to maintain full compatibility with the bare Atari TOS (more or less, but let”s not stroll down that path).

Since I can get to a bash prompt, one might think that the only step involved in building Python on the Atari is a pair of commands: configure and make. There is, of course, far more involved.

Compiling on the Atari

Under FreeMiNT, the de facto compiler continues to be GCC 2.95.3. There are more modern versions available for the platform, but only 2.95.3 is packaged for the SpareMiNT distribution, an RPM-based FreeMiNT installation and arguably the only one still maintained. Using 2.95.3 from SpareMiNT also has the added bonus of allowing prebuilt binary libraries to be used rather than having to manually build Python”s meager dependencies.

There are ports of the GCC 4 series to compile Atari software. However, most ports are cross compilers, requiring the software to be built on a PC. Anyone familiar with building CPython should know that cross compiling Python is basically never suggested. Additionally, the Atari line of computers are desktop systems, not embedded devices; having to cross compile software for them feels dirty and pointless.

To build Python, in this case, I decided on sticking with GCC 2.95.3 as planned. So now I can just type configure and make, right? You can, but you”ll immediately get some errors. Actually, using the stock Python distribution, you”ll get relatively far before encountering an error in a Python header file itself. Before we dig into what”s wrong with Python, though, we first need to discuss the problems surrounding mintlib.

Although FreeMiNT mostly relies on GCC as its compiler, it does not use the GNU Libc standard library. Rather, it uses a C runtime library called mintlib. This C runtime library is a mishmash of standard C functions as one would expect with some of the historically lesser used functions grafted on from various other implementations. Different portions are governed by different licenses, including old and new BSD and the GNU GPL. I”m not sure mintlib is actually conforming to some of the licenses, but it”s obscure enough that probably nobody will care.

During my work porting Python 2.7, I encountered some missing standard library calls related to wide characters. The Atari runtime library did not contain wcschr() and wcsrchr() functions. I threw together a little library containing these function from their FreeBSD implementations, allowing me to build Python 2.7 to completion. I also needed to do this when I attempted to build Python 3.2 during that same period, but, as I said, the resulting executable was completely nonfunctional.

I decided to do the same thing here to get Python 3.3 booting. Basically, I built a little library containing the two functions above and wcstok(), another missing function, to get the build to progress. Although I was able to get the build to complete (with some other changes I'll explain in a later post), I ended up with a completely nonfunctional executable once again. My first reaction was to give up, but I decided to continue my quest for Python 3.3 on an Atari machine.