Taking the NeXT step (mach kernel upgrades)

[NCommander]

Well, I made some more headway.

Something I knew but didn't remember until this "morning" was that NeXT/Apple's GNUmake had non-standard extensions in it, which is the references XNU wanted. I didn't realize gnumake was installed under that name on OpenStep, but once I found it, I got somewhere.

Course the build falls over pretty quick because some of the pre-dependancies are MIA. With some effort, I got bootstrap_cmds built, and files copied into the right places. This was a headache because it uses ProjectBuilder makefiles + internal makefiles that weren't shipped as part of the base system. Some hacking got those installed, and OS's pb.makefiles now work. The trick is you need the ability to generate the versions objects, which is done in next-sgs.make (in pb-makefiles for Darwin/OSX 10.0). They also want newer headers than what was shipped out of the box which requires some compiler controtions. The new mach headers seem to break some of the other ones in the system (or I installed them wrong), so I couldn't just do a blanket upgrade. I had to setup a specific folder with handpicked headers in them and then lay them out in place so the compiler could find what it wanted.

I also had to build updated yacc and flex from Darwin 0.1, but I've got a decent hand on how these go together, and what the CoreOS Makefiles are supposed to do. At some point its probably possible to fix the sources to point to the correct locations on NeXTstep so a mass build could (theorically) be done. As a note to the wise: anything that uses next-sgs has to have its build-directory renamed to name-ver, or the build will fail in weird ways.

I'm now currently stuck dealing with mig, the Mach Interface Generator. mig is pretty standard fair on mach based systems, but Apple really butchered it. /usr/bin/mig on NeXT is a shell script that calls out to /usr/lib/migcom; which is the actual message compiler. It seems MiG grew multiple variants, and those variants live in bootstrap commands so now I have migcom, migcom_typd, and migcom_utyped. What I don't have is the wrapper driver that runs the hot mess, and it doesn't look like its in any of the darwin 0.1 sources. It's likely in the Darwin 1.x ones but I'll be damned if I can divine which package its in.

EDIT: I'm a turkey. mig.sh was right there and I missed it in bootstrap_cmds. After some bricks to fix the path issues, gnumake got much further. Still need to work out what else is missing though. It falls over generating the MD and MI headers.

[ardi]

While it is very interesting (and useful IMHO) to get rid of some of the hardware limits of NS, however it is important to notice that if you make NS able of achieving current hardware limits, you extend the lifetime of NS, and the 2038 year is not that far from today. So, any attempt at modernizing NS should deal with approaches for overcoming the 2038 year problem, IMHO.

[NCommander]

So, a final update before I stop banging my head onto the desk more.

After a LOT of teeth pulling, I managed to get most of mach to build, but there are headers missing, specifically to those relating to the DriverKit IO buses. For mach itself, it isn't super problematic because its mostly related to EISA device initialization, nothing super critical.

Then I ran into a problem it couldn't find "/usr/local/lib/libDriver_kern.so". As far as I can tell, this is a non-distributed build of the driverkit package, which we have the source to (yay!), *BUT* it talks very heavily to the kernbus backplane I talked about before, who's API is defined in the missing headers.

I managed to stub out most of the functions which would get me a broken kernel but see what would actually be necessary beyond this point. I tripped into objc which is the ObjC runtime and has to be statically linked in such a way the PB makefiles can't handle. I'll have to merge the code from the old to the new to try and fix this. That's going to be an epic headache.

[t-rexky]

This very much reminds me of building OS X cctools for NEXTSTEP that I discussed in another thread.  I had to mix-and-match headers between OS X and NEXTSTEP in order to satisfy all the pre-processor definitions and dependencies.

Although the difference is that you actually know what you are doing, I was just playing  

[NCommander]

While it is very interesting (and useful IMHO) to get rid of some of the hardware limits of NS, however it is important to notice that if you make NS able of achieving current hardware limits, you extend the lifetime of NS, and the 2038 year is not that far from today. So, any attempt at modernizing NS should deal with approaches for overcoming the 2038 year problem, IMHO.

Unfortunately, that's not technically possible without the full source code to NeXTstep *and* breaking compatibility with pre-existing applications. The best that can be done is resetting the epoch to something later than 1970 but  would have to have bad side-effects. Notably, 32-bit Linux has the same problem, as does NetBSD (which changed to 64-bit time_t, but has a compat layer for old binaries)

The underlying 2038 value is due to the fact that time_t is a signed integer on many platforms, and unlike Y2K, is a problem for fundamental to the design of the system vs. a formatting/spacing issue. Two very different things.

This very much reminds me of building OS X cctools for NEXTSTEP that I discussed in another thread. I had to mix-and-match headers between OS X and NEXTSTEP in order to satisfy all the pre-processor definitions and dependencies.

Although the difference is that you actually know what you are doing, I was just playing  Razz

At least you had a full set of files . I have no idea if the EISA/ISA/PCI driver source code was ever released (which has the bits I need to fix this properly vs. guessing my way through) which makes my job considerably more difficult. I can at least get an idea of the interfaces requires by examining the existing binaries, but the struct part of it is going to be considerably more difficult. I don't even have a great idea of how much is missing vs. what has to be there and I've done very little ObjC work overall.

[Rob Blessin Black Hole]

Hello All: I know there were differences between NeXTStep and Openstep headers , also I'm wondering if anything in the FreeBSD port to black hardware would help or better yet gnustep.org . So I'm positive many more NeXT developers will join in soon. I'll contact a few and see if they have further insite. Also would having a copy of Rhapsody help this endeavor? I know I have one . I heard it was possible to copy the NeXT libraries over to Rhapsody and they actually run. I also have the Openstep developer tools for windows nt. Also at one point they even had a Linux port to NeXT hardware going . Andreas and Simon would probably have some helpful ideas as well.

[NCommander]

Right now, the biggest hangup is mostly due to the fact that this is an undocumented interface; something internal that connects the mach kernel to driver kit. Annoying/Fortunately, the powerpc port has a full set of files so I have something to reference.

I had to take yesterday off as a mental health day but I'll be poking this more w/ a stick today once I get my bearing straight.

[Rob Blessin Black Hole]

Hello Ncommander: The documentation install packages may have some clues and may be located under / NeXT libraries bookshelves directory , there is also a set of NeXT Developer books including  writing loadable kernel servers , the complete set is 30 lbs which has syntax . Manuals Like in This video https://youtu.be/mk7jlh40qLI of a Turbo Cube system I'm currently integrating!

[NCommander]

Right now, I doubt the necessary header files are on Rhaspody, but it might be useful. If anyone has a boot log for Rhaspody would be handy, as well as the boot floppies. Now that I've rested and had some time to think about it, I've managed to get the kernel linked and built; I had to implement some missing symbols with stubs (specifically, nodev(), which I can't even figure out where it's defined).

Anyway, here's some interesting screenshots:




It scrolls out of sight, but the kernel does report a build date here in 2017 . I might have more success swapping out the OpenStep drivers for Rhaspody ones, though I'm probably going to fix the devicekit build to be more in-line w/ reality.

One major problem is the active_u structure doesn't exist between the original NS kernel and this one which appears to be used for device probing; the ATA probe address changes from reboot to reboot. Notably, the PCI detection stuff actually works properly which is really nice which means we're getting fairly far into boot, and displayServer is at least partially working as in non-verbose boot I get a brief flash of it before it panics.

EDIT screenshot of the kernel boot string:

[Daxziz]

Wow... simply speechless here. Seeing the Rhapsody 5.3 on x86 is huge and I can see the memory amount is fairly high as well.
It does indeed bring some good possibilities if it all succeeds.


-Daxziz

[NCommander]

Wow... simply speechless here. Seeing the Rhapsody 5.3 on x86 is huge and I can see the memory amount is fairly high as well.
It does indeed bring some good possibilities if it all succeeds.


-Daxziz

512 MiB is the maximum supported by the OPENSTEP bootloader. We have the source to the Rhapsody bootloader which has the same hard-coded limitation, but I know enough about how it goes together to bump it at least to the 1 GiB mark if not higher. I have not, as of writing, managed to get it to build mostly because of the mess of symbols in the base system. I can fix it, just haven't done it yet.

Given a closer examination of how it works, I can understand why Apple dumped DriverKit for IOKit; there's a massive performance overhead for ObjectiveC in the kernel space since you have to deal with both the very slow mach messaging system, and the objective C runtime overhead which makes device calls painfully slow (NeXTstep feels damn laggy compared to Darwin). It's broadly similar to the performance characteristics of GNU HURD of all things.

Notably, the kernel identifies itself as "Mac OS X Server Operating System" on PowerPC, and as "Rhapsody Operating System" on x86. DR2, based on screenshots, used "Rhapsody Mach Operating System". Right now, my best bet moving forward is to determine the structures of what it wants and see if I can make it happy enough to keep booting then fix it properly. As far as I can tell, the stock mach kernel is built with symbols (or at least nm can read them); if I could managed to attach a kernel debugger, I could read them out and dump out the structures in RAM getting a better idea of what is and is not missing. Unfortunately, I haven't figured out how to this as of yet on OPENSTEP, and its possible the gdb stub is MIA.

[Daxziz]

The reason for why I said what I did before is because DR2 is crippled on x86, so the RAM is limited to around 192.
Seen from a Rhapsody perspective it would be fantastic to get just 512.


-Daxziz

[NCommander]

Well, after a fair bit more of hacking, I was able to reconstruct enough of a guestimate of the missing components to get DriverKit considerably more happy. Aside from one minor code change, driverKit builds out of the box without code modifications now. Now it looks like it just doesn't like OPENSTEP's drivers, though I haven't run down the exact error:



Still, considering its managing to enumerate the harddrive is an impressive bit.

EDIT: Implemented a stub to fix the old syscall and convert it to the new (kinda a dodgy fix but it works for now). I also had to teach the kernel to ignore a duplicate call to biodone() (which was a panic. Now it just silently ignores the second call) Now it gets further.



The failure to mount root filesystem is concerning because its POSSIBLE the UFS format changed between OpenStep/DR2/Mac OS X 1.0 Server; probably to add better support for Mac stuff. I have no good way to test that theory though.

When I popped in the OS4.2 CD, and told it to use the root filesystem from there, it actually got into single user mode, though is very unhappy.



Looks like something broke with mach-o architecture detection; it dislikes the fact the binaries are FAT (i386/SPARC). I can seemingly load anything that's i386-only but other then that, I get bangs.

[Daxziz]

You might be spot on about the file system. DR2's disks, even the CDs as far as I remember, were not readable on much else than DR2.
However I wouldn't have guessed that it had that much impact on the kernel itself.

Once again I'm amazed with your progress in just 3 days. I've never seen the like here or anywhere else.


-Daxziz

[NCommander]

I'm a little shocked Apple changed the UFS ondisk format, especially since HFS support *is* present, and to my knowledge, Server 1.0 used HFS, and not UFS. Theoretically, I could just reformat / to HFS, and copy the files over though who knows if the extended ACLs survived intact. What's really annoying is there are about ten different variants of UFS all across the UNIX family tree, and I'm not aware of anything beside NeXTstep that can mount its own filesystems.

Then again, it did mount the CD. I need to check if the CD is ISO9660 or UFS formatted (my guess its the former, but I'm not sitting in front of the desktop to check). it can also *load* some binaries, sh and ls worked, file didn't; when I boot it back into OS4.2, I need to examine those binaries to see whats different. There look like some odd changes in the mach-o loader, though a dyld failure suggests we're blowing up in userspace and not kernelspace which doesn't make me optimistic on a quick fix.

Since both the HDD and the CD-ROM run through the EIDE driver, at least it can TALK to the ATA interface, and I can browse successfully with "ls" and such. The floppy driver from OpenStep won't load due to duplicated symbols. The DR2 drivers are likely more compatible overall though.

The saving grace here is that the format might be well enough documented in the NextHeaders folder to fix it and get the kernel to agree to NeXTstep's variant of UFS. Might just be as simple as a filesystem magic check, vs. an actual on disk change. From as far as I can tell, UFS in NeXT/Rhaspody is directly descended from the BSD variants, which means it has a very simple on-disk format. If I can find what broke and line the bits back up, it should get things back up and working. I might also try a netbooting solution if I can figure out how to do it though it's not loading the PCnet driver; that might just require a boot to the head in Instance0 to load it in preboot though. I don't get a symbol load error or anything on it though; it might be trying to come fully up before loading it since as far as I can tell, when you netboot NeXT, it downloads mach_kernel, and the necessary configuration files over TFTP; it never (officially) supported disk-assisted netboot.

NeXT built filesystem support directly into mach and boot. This is in contrast to  most other operating systems, but similar to Mac OS X. I always thought it was a holdover from NeXTstep basically unchanged from there. The only time UFS ever had a use was if you needed case sensitivity in the early Mac OS X's before 10.4(?) added HFSX for case sensitivity.