Taking the NeXT step (mach kernel upgrades)

[NCommander]

So, quick update before I call it for the night.

Traced through mach's filesystem code, what it calls "UFS" is actually the Berkeley Fast Filesystem which was basically what BSD 4.4 used with some NeXTstep modifications. Comparing this to what's in Mac OS X 10.0 shows that the only real change that happened here was that HFS replaced UFS as the first filesytem tried for mounting /.

I found a few Apple specific changes that are enabled in the UFS code and some reference docs talking about NeXTstep/OS filesystem changes. I'm hoping this will be straightforward to fix ...

[Rob Blessin Black Hole]

Hello Ncommander : Way to burn rubber man, Jedi coding going down, if this is redundant stuff apologies but it may help...
so NeXTSTEP 3.3 had 2  variants of the user CD NeXTSTEP 3.3  68K/Intel  and NeXTSTEP 3.3  HPPA Risc/Sparc  .  So you would be able to compile quad fat NeXT apps with Developer tools.  
Then with Openstep the idea was to write the app using Openstep for Mach, Sparc or NT then with a recompile to Run on  Mach , NT and Sparc for crossplatform compatability.
Also I have a weird Openstep 4.0 alpha , I can drop box it to you if you like?  It appear as NeXTSTEP 4.0 in verbose boot with different filesystem folders on login , its buggy , they dropped it but it may have transition stuff from the NeXTSTEP 3.3 release to gold master Openstep 4.0 , may be there is something there you can use.
There were 2 versions of Rhapsody  for Intel which the kept under wraps  until MAC OS X on Apple Hardware switched from Power PC to Intel processors so the processor error may be related to the version of Rhapsody you are running if its PPC instead Intel.  
Apple had Rhapsody set to autoeject on Black Hardware to prevent us from tinkering LOL> so it would be fun to see if that could be bypassed to start a Rhapsody install on NeXT 68K hardware.
How much 68K stuff was still intact from NeXTSTEP / Openstep  for a port of Rhapsody back to NeXT hardware is an unknown.  
The NeXT app Daydream now  Darkmatter boots up a 68K  NeXT as a Macintosh , I often wondered if that was possible as NeXT computer architecture was very similar to Mac Quadras  to modify NeXTSTEP 68K to boot a Mac ... Ncommander you would be the one to ask.
Also check out our software archives in case you haven't seen this http://www.nextcomputers.org/NeXTfiles/Software/    the Peanuts disks haves loads of NeXT software also tons of developer apps !
Another cool upgrade for NeXT 68K Hardware would be either a 68060 Processor or powerpc chip 603E upgrade for Black Hardware   he he he like this one.
http://lowendmac.com/2016/powerpc-601-upgrades-for-68040-pds-macs/
and/or support for 30pin 16Mb simms or 72pin 64Mb simms not sure if this would be possible possibly even in software .
, I've heard rumors it existed not sure if it did or if even possible quadras had a daughter card ..... the NRW was going going to be NeXT's new risc based hardware but they shut down hardware production before completion
http://www.nextcomputers.org/NeXTfiles/Images/Rare_NeXT_Hardware/NRW-NeXT_RISC_Workstation/
, I also heard the had NeXTSTEP running on IBM 6000 and DEC Alpha but never released.

[bobo68]

Seems to be hard to find info about NeXT's/Apple's UFS implementation.

There seems to be a problem with block sizes between NeXT's and Apple's UFS: https://lists.apple.com/archives/darwinos-users/2001/Apr/msg00423.html
Some general info: http://www.softpanorama.org/Internals/Unix_filesystems/ufs.shtml
https://web.cs.wpi.edu/~rek/DCS/D04/UnixFileSystems.html
Linux could or still can mount NeXT's UFS volumes read-only: ftp://linuxmafia.com/kb/VALinux-kb/nextstep-ufs-support.html and https://kernel.googlesource.com/pub/scm/linux/kernel/git/stable/linux-stable/+/linux-2.6.19.y/Documentation/filesystems/ufs.txt
Current Ubuntu doc say they still can: http://manpages.ubuntu.com/manpages/zesty/man8/mount.8.html
Linux source for UFS support: https://github.com/torvalds/linux/tree/master/fs/ufs
MacFUSE UFS support: http://osxbook.com/blog/2008/12/16/more-user-space-file-system-goodies/

Maybe you can learn something from this info.

HTH, bobo68

[NCommander]

Just a brief update. I did find some of those UFS sources but unfortunately, the names in the UFS driver don't match each other so I can't tell by quick examination what changed and what didn't. Compounding the matter is the keyboad on the desktop decided that today is a lovely day to kill itself so I'm stuck until I can replace it.

[NCommander]

Here's an update folks. Decided to put the UFS issue aside since I was getting a migrane from it, and dealt with the mach-o loading issue. The primary issue came down to the fact that *something* broke with subarchitecture detection. the MachO binary filesystem can actually contain variants of the same architecture (i386/i486/etc).

What was happening is that code built for i386 (that is an actual 386 processor) would work, but i486 (which are most of the binaries) won't. After a LOT of fiddling, I tracked down the root issue to the architecture detection code; it looks like Apple made some breaking changes on how mach handles both architecture detection and pentium detection, and some left over test code. I haven't decoded the rats nest yet, so I cheated, and forced the kernel to always believe it's running on a Pentium/586 filesystem. I can now run all the apps on the CD successfully w/o issue.

Needless to say, that's a victory. I need to track down the mounting root filesystem issue which probably means adding a ton of debug code to this, but we've now crossed the "it's running console apps" milestone of progress.

[Venture37]

also I'm wondering if anything in the FreeBSD port to black hardware would help or better yet gnustep.org

NetBSD/next68x
http://wiki.netbsd.org/ports/next68k/

[Daxziz]

Needless to say, that's a victory. I need to track down the mounting root filesystem issue which probably means adding a ton of debug code to this, but we've now crossed the "it's running console apps" milestone of progress.

That's amazing! Good work. I do hope you track changes in Git ;)

[NCommander]

So, after more head-disk banging, I think the fundamental problem is an issue with disk label vs. filesystem. I can (seemingly) read and write the HDD so we're talking to the disk controller properly but it's not getting any partitions or anything. It also looks like raw disk access is present on the "h" slice of the partition map. Might be a bit late, but that's probably help for dumping NeXTstep MO disks since that seems to have been an issue for Previous.

I need to find where mach loads disklabels and such and work from there.

[t-rexky]

I cannot offer much of value, unfortunately.

I do however have two shots in the dark for you, based on my previous tinkering:  (1) is the system page size correct? I ran across some changes Apple did in cctools that modified the i386 architecture page size.  (2) perhaps this has something to do with big-endian / little-endian conversion of the file system?  I know there was something funky going on on NS file system between m68k and i386...

Just some random ramblings.

[ardi]

the MachO binary filesystem can actually contain variants of the same architecture (i386/i486/etc).

Yes, although I've never seen this documented enough. For example, in theory it should be possible to have an OS X fat binary that has both G4 and G5 code, but I've never seen it, and I really don't know if it can be done.

it looks like Apple made some breaking changes on how mach handles both architecture detection and pentium detection, and some left over test code.

Do you mean changes between NeXTSTEP/OpenSTEP and Darwin 0.1, or between Darwin 0.1 and later OS X releases? I suppose you refer to the former. It's a pity that Apple dropped 68K support in Darwin 0.1, it could have helped us to maintain NeXTSTEP in the future.

[NCommander]

I cannot offer much of value, unfortunately.

I do however have two shots in the dark for you, based on my previous tinkering:  (1) is the system page size correct? I ran across some changes Apple did in cctools that modified the i386 architecture page size.  (2) perhaps this has something to do with big-endian / little-endian conversion of the file system?  I know there was something funky going on on NS file system between m68k and i386...

Just some random ramblings.

I'm going to assume it's correct right now, since if the system page size was wrong, I would expect considerable amounts of confetti trying to load and access dynamic libraries. I'm tied up at work at the moment but as soon as I have a brether, I'm going to check the OPENSTEP ISOs and see what file system formatting its using; if its UFS based (which I suspect), then I'm almost certain I'm dealing with a disk label problem

the MachO binary filesystem can actually contain variants of the same architecture (i386/i486/etc).

Yes, although I've never seen this documented enough. For example, in theory it should be possible to have an OS X fat binary that has both G4 and G5 code, but I've never seen it, and I really don't know if it can be done.

it looks like Apple made some breaking changes on how mach handles both architecture detection and pentium detection, and some left over test code.

Do you mean changes between NeXTSTEP/OpenSTEP and Darwin 0.1, or between Darwin 0.1 and later OS X releases? I suppose you refer to the former. It's a pity that Apple dropped 68K support in Darwin 0.1, it could have helped us to maintain NeXTSTEP in the future.

Its not documented at all; I didn't even realize this support existed until I dug into mach and I've played in MachO land before. Basically, when dealing with fat binaries, you have multiple mach-o headers sandwiched together; with the m68k header being first (likely due to backwards compatiblility so old NeXTsteps can load FAT binaries).

Past the architecture bit is the subtype field which defines what type of processors can be used or not. This is a bitfield, and the values in the bitfield changed between Darwin and NeXTstep; specifically dealing with Pentium support. When the kernel tries to launch a mach-o binary file, it tries to assign a grade to each mach-o image it can run; highest grade wins. So if you have a hypothetical mach-o binary with 386/486/586 code, it will run the best possible code on the platform you're currently on. What broke here is that Apple was in the process of redoing this to be more "generic" as far as I can tell.

Mach runs two sets of CPU detection tests; the first one is at very early boot and makes sure that you're running on a 486SX or greater. If it fails, kernel panic. This one works as is out of the box.

The second one (which was broken) was it tries to do cpuid and gets confused, and believes its running on a i386. This breaks any binary that is tagged for running on the 486/586, which is why I kept getting all those CPU type incorrect load errors. There were some hardcoded strings being tested and comments saying engineering test that *shouldn't* be there. I couldn't figure out how to make the test happy so I just forced it to 586 compatibility. Anyone who's running this on a legit 486 isn't going to get much from a new mach kernel . This allows the subtype ID check to successfully pass.

Reporting Darwin to the 680x0 is likely possible; I've proven that the core components for at least the kernel are in place. It's just a massive headache but a lot of the support for for the 680x0 is still in place; i.e., mach IDs and such. It's more they deleted anything with "m68k/sparc/hppa" out of the file folder. Assuming the kernel bus architecture is similar between x86->other, getting it all working is in the realm of possible. If you were persistent enough, you could probably even add support for some IOKit device drivers and bridge them to DriverKit ones. Pity that attempts to get the older NeXT kernels out of Apple failed though :/

[Rob Blessin Black Hole]

You may check etc/fstab  for disk tab entries , you can also write custom disk tab entries or mount additional partitions. Also when NeXTSTEP runs check disk  by default, I've had a complete new clean install  on a larger hard drive even after partitioning them and mounting the partions in etc / fstab only to have the install completely wiped out on a hard shutdown and reboot , where it goes to the default check disk app , everything goes haywire , it may be because it doesn't understand the disk drive geometry beyond 2gb partitions for NeXTSTEP , Openstep handles it better,
the man pages should be located under /nextlibrary/bookshelves  should help
http://polarhome.com/service/man/?qf=disktab&tf=2&of=NeXTSTEP&sf=5
Hope this helps! Great work

[NCommander]

I haven't had that much time to work on this as of late. I'll try and hack on it this weekend.

[t-rexky]

I had another, likely very stupid thought.  Is it possible that the partition layout or structure has changed somehow?

I don't really see that happening since in i386 world everything pretty much had to be MBR until UEFI was adopted.  But maybe they did not follow or even support MBR at that point because this was so early in the life of Darwin?

This is likely not the case because, of course, they had everything running on i386 with Rhapsody and with Darwin DP2...

[NCommander]

I had another, likely very stupid thought.  Is it possible that the partition layout or structure has changed somehow?

I don't really see that happening since in i386 world everything pretty much had to be MBR until UEFI was adopted.  But maybe they did not follow or even support MBR at that point because this was so early in the life of Darwin?

This is likely not the case because, of course, they had everything running on i386 with Rhapsody and with Darwin DP2...

That's actually what I think happened. Without having the old kernel source available as reference, I've had to infer basically how it works. On old school MBR partitioning, you'd have a set of physical partitions, and then NeXtstep would install into one of thoses, and write the disk label in it.

On MBR disks, you basically have a small 1k set of data at the start to load boot code (which is boot0 and boot1), and then each partition has an additional header for a second stage bootloader (which is just plain boot). OPENSTEP/NEXTSTEP's second stage bootloader switches to protected mode opens the filesystem up, grabs mach_kernel, the System.config file, and then loads the "Boot" driver set from Instance0.table. Configuration is passed into mach via the kernel parameters structure and control passes off to that. Mach loads its brain and the in-RAM drivers, and then re-initializes all devices off the each of the three buses, including the physical partition of where stuff lives.

Within that physical partition, you have a logical partition, which NeXTstep calls the disklabel and is how that space is subdivided into smaller parts. This scheme was extremely common on BSD derived systems. As far as I can tell, those DOS partitions are invisible in the running system, only the bootloader actually knows where they are. There were some extensive changes made to support PowerPC partition maps including a Mac OS boot partition, OpenFirmware partitions, etc (if you've ever used old Mac OS X, this is why / is something like disk0p9). Unless there's some difference on the DVD image, the image itself is formatted UFS. NeXTstep identifies the filesystem in mount as "type 4.3".

The kernel source has this to say on the subject

Code Select Expand

#if COMPAT_43
       /*
        * Historically filesystem types were identified by number. If we
        * get an integer for the filesystem type instead of a string, we
        * check to see if it matches one of the historic filesystem types.
        */
       fstypenum = (u_long)uap->type;
       if (fstypenum < maxvfsconf) {
               for (vfsp = vfsconf; vfsp; vfsp = vfsp->vfc_next)
                       if (vfsp->vfc_typenum == fstypenum)
                               break;
               if (vfsp == NULL) {
                       vput(vp);
                       return (ENODEV);
               }
               strncpy(fstypename, vfsp->vfc_name, MFSNAMELEN);
       } else
#endif /* COMPAT_43 */


COMPAT_43 is noted as "BSD 4.3" compatibity. The new kernel can also mount that partition. I notably also can't get FDISK to show partions even when started with the old kernel on the CD so I need to look at the installer and figure out how it does it's voodoo.

(compat 43 is defined)

EDIT: Just had a brainway. I *think* the device major/minor numbers might have changed between Rhaspody and NeXTstep. I need a DR2 CD to check though :/