Taking the NeXT step (mach kernel upgrades)

Started by NCommander, March 14, 2017, 04:51:02 pm

Previous topic - Next topic


(I'll show myself out for that pun)

As a proof-of-concept photo, NeXTstep 3.3 running on OPENSTEP's kernel with the VBE driver present: http://imgur.com/a/R7fzM


So in what is possibly a fit of insanity, I was inspired by the posting of the Darwin 0.1 sourcecode to see if it's possible to actually upgrade the underpinnings of the original NeXT system to something slightly more modern. As a first step of this, using a NeXTstep 3.3 install in VirtualBox, I used the OPENSTEP upgrade kit, and the boot floppies to jury-rig 3.3 into using a newer mach and the VBE driver, and got a fairly decent understanding on how most of the stack fits together.

DriverKit basically operates on a back-plane of buses, specifically ISA/EISA, and PCI, and those buses have to be present in mach for the kernel to successfully boot. There's a tight binding between these bus drivers and mach itself, *however*, drivers that attach to the bus appear to be relatively kernel independent; I can successfully load some NeXTstep 3.3 drivers that I've tried. This is a good thing given that the existing mach/driverkit sources don't have the individual drivers themselves. It might be possible to backport IOKit drivers to DriverKit as the API mostly stayed the same from a conceptional level, it just had a change in programming language.

A couple things I've been able to tell from the ASPL darwin 0.1 source is specifically how the kernel loads coming out of boot, and why there are certain limitations such as maxing out at ~500 MiB of RAM. I'm kinda fried at the moment, but I think I could reasonably remove this restriction fairly easily by doing some low level magic within the kernel to properly recalculate the max amount of RAM available up to at least the 1 GiB mark if not more; I have to take apart the MMU driver to know more for sure. x86 gets a bit weird(er) when you start approaching the theoretical max supported by the architecture.

The 8 GiB drive limit also is likely due to using older forms of LBA addressing and may be surpass-able if the ATA driver can be reasonably rewritten and replaced. I need to look more how the storage classes work, but this might simply be a matter of writing a VirtIO driver.

I'm not sure if anyone is actually interested in this, and as a free-time project, I really don't know how much time I can dedicate to it, but I may poke at it here and there. I need to coax Darwin 0.1 to build with NeXTstep's toolchain, or failing that, get Darwin 1.4.1/x86 running, and then go backwards. Not sure if there's ObjC ABI issues with that route though ...

Figured it was worth a post to gauge actual interest though.


Rob Blessin Black Hole

All of the above, is awesome an upgrade to NeXTStep and Openstep will be amazing.  I will support this endeavor and look forward to testing it all out.
Rob Blessin President computerpowwow ebay  sales@blackholeinc.com http://www.blackholeinc.com
303-741-9998 Serving the NeXT Community  since 2/9/93


Well, well!  Congratulations!!!  I am now REALLY tempted to try that on m68k!

I have been hunting for old sources for years.  As you may have seen in some of the posts we even wrote several letters to Apple senior management and they were forwarded through some internal connections.  Unfortunately no one ever responded...

The other stuff rumbling in my head was to try back-porting XNU to m68k with a compatibility layer allowing all the BSD4.3 stuff to run on it.  But this is way beyond my scope and simply a daydream.  For me anyhow.  Although from my successful experiments in porting OS X compiler tools onto NEXTSTEP 3.3 it is clear how close the heritage really is.

You will notice that somewhere between Darwin 0.1 and 1.0 the kernel sources started morphing from what was resembling the original CMU mk-*** organization in kernel-1 under Darwin 0.1, to xnu-*** organization under Darwin 1.0.

It would be so awesome to have access to rhapsody kernel sources!  All we need really in addition to driver development is to fix the UNIX time bug, add support for large disks and add support for modern libraries with POSIX...

Why someone may ask?  I don't know.  Maybe because it's cool.

Rob Blessin Black Hole

Rob Blessin President computerpowwow ebay  sales@blackholeinc.com http://www.blackholeinc.com
303-741-9998 Serving the NeXT Community  since 2/9/93


Apparently my post I made before I went to bed didn't go through. Oops.

So a couple of notes: this is only going to really work for NS/OS i386 as that's what the source code to darwin has (plus PPC, but we don't care about that right now). It would theoretically be possible to port Darwin back to SPARC/HP-RA/m68k but it would require a lot of work to do so, and I think those platforms would not benefit as much as better HDD support as they should be relatively less hamstrung by the inherent issues with the x86. Unfortunately, while we have the sources for drivekit, it doesn't look like the drivers in and of itself survived, but we do have some support for backwards compatibility.

For VBE, the boot sector for NeXTstep has to be updated mostly because mach runs in protected mode; boot0/1 does some memory tests, loads mach, and sets up a kernel boot structure (very similar to ATAGS on Linux/ARM) which passes the memory and such. As such the system starts up in a 1:1 memory map in protected mode and then it does a bunch of house keeping to start up the machine. mach uses /private/Drivers/x/System.config to determine the modules it needs for the DriverKit backplane, and loads all those. The backplane is pretty tied to mach but the individual drivers themselves seem relatively independent of mach version as the 3.3 drivers do load and appear to work; the NIC driver from 3.3 at least initializies but I don't get network. Then again, I also didn't get network under the standard NS 3.3 kernel idea so ... yeah ...

From some the experimentation I did way back in the age, the base ABI of NeXTstep was retained; basically if you have a statically built POSIX app, it should "just work" between NeXtstep and Mac OS X. When I screwed with this ages ago, if you put a non-x86 NeXT bundle, Mac OS X would actually say "This is an unsupported architecture" just like NeXT did so a lot of base stuff survived at least up to 10.4 or so.


This is really interesting. Back when we "hacked" at Rhapsody DR2 we had the same idea to replace the mach_kernel with some of the earlier OS X versions - however we never had any success (nor access to Darwin 0.1 as far as I remember).

But it's fantastic to see this. It opens a lot of fun possibilities.

Since you fear for the amount of time, that you can put into this project - and since you're the first to ever present such a progress with a screenshot; could I encourage you to make a detailed step by step on how you actually succeeded with this mach_upgrade? That'll first of all document this for the future but it might also help getting more people to help with the project.

Also I can would highly recommend developing a script and use Github to store this as a project.



My NeXT VM doesn't have working network (or won't until I punt OPENSTEP'S AMD driver into it. My OS VM did manage to talk to the world, then a badly time crashed wrecked the filesystem so I just finished re-installing), so I can't copy off the files, but basically, this is what I did.

This was honestly just a few hours of hacking, I never ran NeXTstep before in my life so I'm a bit surprised no one else managed a kernel transplant. The DR2 kernel should have been able to go backwards in time as well by roughly the same process, and once I have Darwin building again, I'll know what it will take to get this actually working. I have however worked on Mach based kernels before and know how they're put together and built Darwin from source years and years ago so that helped.

I didn't take practically good notes since this was just a quick check to see if it was even viable; I was expecting it to take a lot more effort to transplant the kernel. Darwin 0.1's sources were available when Mac OS X Server 1.0 (which is essentially NeXTstep 5.x) was up, and then vanished when 10.0 shipped.

Based on how DriverKit is connected to mach, it MIGHT be possible to swap out new XNUs with DriverKit, or write a IOKit->DriverKit adapter. It mostly depends how much WindowServer talks to it; if it just wants IOGraphics, it's not super hard as that interface is pretty much identical between the two, just needs a new ObjC binding.

Basically this is what I did:
1. Install NeXTstep 3.3

2. Put the OPENSTEP 4.2 Y2K disk in, you need the MachUpdate4.pkg, or the files in it.

3. Take apart the package using installer_bigtar (its in /NextApps/Installer.app/Contents). This will get you a bunch of updated files.

4. Snapshot the VM

5. Replace /mach_kernel with OpenStep's one

6. Replace /usr/standalone/i386/boot with the one from the update. This is necessary because VBE initialization has to do be done in real mode and thus has to be done

7. Update the boot sector with boot /dev/rhd0a /usr/standalone/i386/boot (has the side effect the system says OPENSTEP at startup).

8. Move /private/Drivers/i386 out of the way and recreate it.

9. Copy System.config from the old drivers folder to the new

10. Put in the OpenStep floppies, you need both the boot one, and drivers one. Dump the files into /private/Drivers/i386. Copy the VBE driver from the OS upgrade kit into this folder.

11. Copy any NeXTstep 3.3 drivers you need (DO NOT COPY ANY OF THE PCI OR BUS DRIVERS, YOU'LL HOSE IT) back into /private/Drivers/i386.

12. Run configure to select the VBE display driver.

13. Edit the Instace0.table and System.config files in the System.config directory to correct the DriverKit buses. My NeXT VM is refusing to boot at the moment since I think I hosed the network on it, but I have it snapshotted so I'll fish out the exact config you need later. The System.config on OS boot is how I figured out how the file should look. It's fairly obvious when you get it wrong, you'll get a kernel panic with "missing symbols" due to the bus not being correctly initialized in order.

14.  Cross your fingers and reboot. If you did it right, you'll get a full color startup of NeXTstep 3.3 with the updated mach like I did.


Thanks for the update - really helpful.

It's my personal feeling that a lot of people in this forum are power users of the system itself, but as soon as you dive into the more complicated stuff with compiling and replacing system components it becomes more of a blur for most and in general it's difficult to figure out where to start properly.

I believe that with the step by step instructions you just provided, you might actually be able to spark the interest and get more people to hack away on their setup.

With a properly created project on Github and hereby a master branch starting with the pure Darwin 0.1 code, and instructions for compiling it, the community would be good to go forward with new branches where people could work on different features like increasing the RAM amount.

In general what I'm talking about is a NeXT community project where the aim is to update the MACH kernel where it makes sense.

Thanks for all your efforts and willingness to document it for future usage.



I'm currently trying to get Darwin to build, I've got the devkit setup, and a NFS mount to my desktop which helps, but I slammed into a brick wall. I need GNUmake to build this, and GNUmake needs a shell slightly less AT&T SVR4 for autoconf to actually work with it.

It seems a lot of the FOSS builds for the platform dropped off the face of the earth, so I'm trying to find a suitably old ksh I can convince NeXT's tools to build, and then bootstrap from there.


So a follow up, after some teeth pulling, I managed to build both bash and gmake and dug somewhat deeper into this. There's some good and bad here; let's start with the good.

As far as I can tell, these sources are complete, and I also have boot which has the graphical start stuff, and the system identifies itself as Apple Rhaspody in the x86; I was also able to build some parts of it but nothing notable to report. That's where the good ends.

The bad is that these sources need newer compilers than what's in the OpenStep compiler which is surprising as its not THAT much older. The compiler falls over trying to parse some of the headers which is irritating to say the least. As far as the code goes, its roughly half way through the filesystem changes so there's a lot of hardcoded paths for /System/Frameworks and such. Furthermore, the ASPL has a license upgrade clause (in 1.1 anyway) that lets any code I use be licensed under the APSL 2.0 which is free software compatible so yay.

In the case of xnu however, it doesn't look like Apple intended for this to be built standalone. Darwin was built as a whole by a master build system which set up some build flags and such and could do what's called a current build, or a release candidate build (where the source is RO). xnu's Makefile looks like it needs to be invoked from that environment and contains references to utilities that are MIA. I can probably fix this, but I need to figure out what sorta environment this actually wants to be built on since in several places such as in boot, it looks like it wants headers that are not normally distributed as part of the installed system (they're in the kernel source directory). Compounding the problem is the C compiler is falling over on them. I may need to see if I can get cctools to actually do anything.

Also, NFS is freaking flaky. Not sure if that's NeXTstep specific, NIC stuff, or whatever but I feel like I'm using a m68k :/. I may run a NFS server from NeXT instead of reverse just so I can use my preferred editor (and git) instead of building on NFS. Tips on setting that up appreicated.


There's been a few updates om the compilers; I don't know if it's enough at all, but if you take a look in this forum, you'll find a few links to these for OpenStep.
They've been uploaded to some of the ftp servers.

Might be good enough to get a bit further?



Quote from: "Daxziz"There's been a few updates om the compilers; I don't know if it's enough at all, but if you take a look in this forum, you'll find a few links to these for OpenStep.
They've been uploaded to some of the ftp servers.

Might be good enough to get a bit further?


I saw that there were GCC updates, but I thought the GCC 3 was for m68k but I'm not sure what exactly I need at the moment. I'm fairly fried at the moment so I probably going to leave this for today and pick it up tomorrow.


Sounds like a wise decision. I'll look through the posts and find the ftp link and post it here.
Assuming I'm not wrong about it being for x86 hardware.



All of my compiler efforts were primarily focused on m68k since this is where my interest was.  I also focused on NEXTSTEP as opposed to OPENSTEP.  I had issues with i386 compiler builds and I abandoned them quite early in the process.

One important thing is that from what I know you will need an Apple compiler to compile XNU and Darwin.  Generic GCC does not have the required customizations.

I think that looking through OpenDarwin archives, whatever is still available, may be helpful in determining how to build Darwin on i386.  I know that it is not trivial.

Some potentially useful links (but you probably know those already):