Taking the NeXT step (mach kernel upgrades)

Started by NCommander, March 14, 2017, 04:51:02 pm

Previous topic - Next topic

t-rexky

Are you reigniting your enthusiasm for additional tweaking of the Darwin sources?

:wink:

I wish I could be more active but my only free time nowadays is consumed by redesigning my home network including the rack, wiring, firewall, UTM, NAS, etc.  I am drowning...

neozeed

I have been stepping back through the process...

I need to document it more, it really is a mountain of "special" to be standing on...

That and pull in a whole lot of the rest of Darwin to get a better footing, along with leaning a lot more on Rhapsody......

It's been a year since I posted anything significant, and I'm dying to change that a little......

I also want to check out Visual Studio 2017 and it's "cross Linux" tools and see if it'll just ssh to any other OpenSSL host...
# include <wittycomment.h>

neozeed



Also I kinda worked around the weird installer thing and got Darwin 0.3 to install!

You need to install OS 9, I used the G4 Cube version, onto a disk.  Then get the Darwin 1.0 CD image, and copy the disk imaging tool to the hard disk.  Then using that tool, restore the image on the Darwin 0.3 CD onto a empty disk.

And then I used Steve Troughton Smith's BootX loader https://github.com/steventroughtonsmith/BootX


qemu-system-ppc.exe -L pc-bios -drive file=..\darwin03.qcow2,index=0,format=qcow2,media=disk -drive file=BootX_custom.dmg,index=2,format=raw,media=disk -prom-env "boot-device=ide1:2,\BootX" -prom-env "boot-args=-v rd=hd0 debug=0xffe kdp=2" -prom-env "boot-file=ide0:8,\mach_kernel" -M g3beige

Yes it does take a while to boot up, but yay now I can test feature parity between x86 and PowerPC

Oh yeah download link:

https://sourceforge.net/projects/aapl-darwin/files/qemu-images/Darwin03-PowerPC_qemu-2.11_04_22_2018.7z/download

Also here is Darwin 1.0

https://sourceforge.net/projects/aapl-darwin/files/qemu-images/Darwin1.00PowerPC_qemu-2.11_04_22_2018.7z/download

At least on Qemu Darwin 1.0 is significantly faster than 0.3 ..
# include <wittycomment.h>

neozeed

Q. But I heard Darwin OS runs on Intel PCs...

Most of Darwin code is platform-independant, and some of the platform-dependant code includes some support for Intel processor-based systems. There are, however, some parts of the system (eg. HFS+ and AppleTalk) which have not been ported to the Intel platform. The current Darwin release is compiled for PowerPC only, but it should be possible to bring up an Intel version with some additional work.

http://web.archive.org/web/19990821004420/http://publicsource.apple.com:80/projects/darwin/faq.html

Well that certainly explains a lot.

And the original 0.1 certainly was just some source files:

Q. What do I need to compile this code?
Presently, you need to buy and install a copy of Mac OS X Server with the developer tools. We are actively working on putting together a self-hosting binary distribution of Darwin so that you can install that as your development platform. This will include all of the needed development tools.

http://web.archive.org/web/19990508075706/http://www.publicsource.apple.com:80/projects/darwin/faq.html
# include <wittycomment.h>

neozeed

Something I should have done years ago, but I guess no time like the present...


https://opensource.apple.com//tarballs/

mirror this directory.

# du -hs opensource.apple.com
26G     opensource.apple.com


Ever since the 'great reset' of Darwin with the release of 10.0 it really isn't that much to be honest..

https://opensource.apple.com/release/mac-os-x-100.html
# include <wittycomment.h>

t-rexky

I pulled all the tarballs for 10.0 through 10.5 back in the days when I was fiddling with cctools.  I have them stored locally on my Synology NAS.

neozeed

Quote from: "t-rexky"I pulled all the tarballs for 10.0 through 10.5 back in the days when I was fiddling with cctools.  I have them stored locally on my Synology NAS.


I wonder if anything went missing during this time..?
# include <wittycomment.h>

t-rexky

Quote from: "neozeed"I wonder if anything went missing during this time..?


I will make a file list when I have a chance...

t-rexky


evolver56k

Excellent work!

I'm not sure how you guys managed to compile the kernel without it panicking! I've applied every patch on the forums and on neozeed's site, it compiles fine. I just can't get it to boot.

The freshly compiled kernel will boot to single user, but will panic as soon as you exit to multiuser after rc starts /etc/startup/0300_Devices (which uses driverLoader to load the Active Drivers in the Instance table)

Any hints on what I could be missing? FWIW, I'm using the darwin-0.3 dpkg-based compile scripts, shoehorning in dpkg/config files to all the source packages. They're pretty complete on the build dependencies, only a few packages fail to build.

EDIT: to clarify, I'm trying to build Darwin 0.1 sources (kernel 5.3)

neozeed

Quote from: "evolver56k"Excellent work!

I'm not sure how you guys managed to compile the kernel without it panicking! I've applied every patch on the forums and on neozeed's site, it compiles fine. I just can't get it to boot.

The freshly compiled kernel will boot to single user, but will panic as soon as you exit to multiuser after rc starts /etc/startup/0300_Devices (which uses driverLoader to load the Active Drivers in the Instance table)

Any hints on what I could be missing? FWIW, I'm using the darwin-0.3 dpkg-based compile scripts, shoehorning in dpkg/config files to all the source packages. They're pretty complete on the build dependencies, only a few packages fail to build.

EDIT: to clarify, I'm trying to build Darwin 0.1 sources (kernel 5.3)


Sorry I've been locked out for a few months.

Can you show the steps you have done, and where it bombed?
# include <wittycomment.h>

evolver56k

Eventually figured it out, it was the patch to modify the PCI bus lookup in IOPCIDirectDevice.m in driverkit. I'm now running my own build of 5.3, and (ancient) openssh/openssl:

RhapsodyVM01:Administrator root# uname -a ; ssh -V
Rhapsody RhapsodyVM01 5.3 Kernel Release 5.3: Thu Jan 31 13:53:26 EST 2019; root(rcbuilder):kernel/BUILD/RELEASE_I386 Copyright (c) 1988-1995,1997-1999 Apple Computer, Inc. All Rights Reserved.  i386
OpenSSH_3.8p1, SSH protocols 1.5/2.0, OpenSSL 0.9.7i 14 Oct 2005
RhapsodyVM01:Administrator root#

neozeed

Quote from: "evolver56k"Eventually figured it out, it was the patch to modify the PCI bus lookup in IOPCIDirectDevice.m in driverkit. I'm now running my own build of 5.3, and (ancient) openssh/openssl:

RhapsodyVM01:Administrator root# uname -a ; ssh -V
Rhapsody RhapsodyVM01 5.3 Kernel Release 5.3: Thu Jan 31 13:53:26 EST 2019; root(rcbuilder):kernel/BUILD/RELEASE_I386 Copyright (c) 1988-1995,1997-1999 Apple Computer, Inc. All Rights Reserved.  i386
OpenSSH_3.8p1, SSH protocols 1.5/2.0, OpenSSL 0.9.7i 14 Oct 2005
RhapsodyVM01:Administrator root#


Good job!

It's a bit of a rough ride to 5.3, but once you get it running it's great.

I should have kept better notes on the whole thing.

I had partial notes on using CVS to download and bootstrap from DR2, although I keep on getting distracted.  :cry:
# include <wittycomment.h>

neozeed

So in the exciting world of Mach, I was given the disk images to the Mt. Xinu 386 Mach, and I compiled the source to Mach 2.5 both MK83 and some later version.  The catch is that the setup to protected mode & paging is wrong and crashes the processor.  And then I wanted to go back to Darwin to check out how some stuff was done there, and I totally forgot how to use the Debian build process.


This is more so for me, although anyone following on from the future will probably want to know.  Sorry I thought I kept good notes, but apparently I do not. :(

Using the Qemu image on the sourceforge page Darwin03_qemu090_26_5_2017.7z (because the networking works!... I forget how to put the modules into place and have them activate to fix the newer one. I think it's all in this thread somewhere, but it turns out that it's easier to update this image).

Mount the CD-ROM Darwin03_debs_source_31_5_2017.iso


mount_cd9660 /dev/sd0a /mnt
cd /tmp
ar x /mnt/debs/buildtools_0.1-2_i386-apple-rhapsody.deb
cd /
tar -zxvf /tmp/data.tar.gz
cd /tmp/
tar -zxvf /mnt/source/dpkg_scriptlib-1.tar.gz
cp dpkg_scriptlib-1/perl5/Dpkg/Package/*.pm /usr/lib/perl5/Dpkg/Package/

I'm not sure if dpkg_1.4.1.0.2-3_i386-apple-rhapsody.deb is needed to be extracted as well.  It's late and I don't want to re-build my VM again.

This should be enough to get the build packages working.  However out of the gate, it's going to try to build for the PowerPC.  And this is an Intel world.  What we are going to do here is shatter the fairy tale of 'one apple employee working from home kept the OS X on Intel dream alive' nonsense by showing that OS X was always buildable on Intel.

We edit the file /usr/lib/perl5/Dpkg/Package/Builder.pm and change the PowerPC crap to i386.

bash-2.02# diff -ruN Builder.pm Builder.pm_i386
--- Builder.pm  Wed Aug  4 17:37:35 1999
+++ Builder.pm_i386    Thu May 25 23:47:16 2017
@@ -60,7 +60,7 @@
  'DSTROOT' => undef,
  'SUBLIBROOTS' => undef,
  'RC_JASPER' => 'YES',
- 'RC_ARCHS' => 'ppc',
+ 'RC_ARCHS' => 'i386',
  'RC_CFLAGS' => '',
  'RC_hppa' => '',
  'RC_i386' => '',
@@ -122,15 +122,15 @@
  }

  if ($target eq 'installhdrs') {
-    $flags->{'RC_CFLAGS'} = '-arch i386 -arch ppc ' . &liststring (@cflags);
-    $flags->{'RC_ARCHS'} = 'i386 ppc';
+    $flags->{'RC_CFLAGS'} = '-arch i386 ' . &liststring (@cflags);
+    $flags->{'RC_ARCHS'} = 'i386';
    $flags->{'RC_i386'} = 'YES';
-    $flags->{'RC_ppc'} = 'YES';
+    $flags->{'RC_ppc'} = '';
  } else {
-    $flags->{'RC_CFLAGS'} = '-arch ppc ' . &liststring (@cflags);
-    $flags->{'RC_ARCHS'} = 'ppc';
-    $flags->{'RC_i386'} = '';
-    $flags->{'RC_ppc'} = 'YES';
+    $flags->{'RC_CFLAGS'} = '-arch i386 ' . &liststring (@cflags);
+    $flags->{'RC_ARCHS'} = 'i386';
+    $flags->{'RC_i386'} = 'YES';
+    $flags->{'RC_ppc'} = '';
  }

  return $flags;
@@ -292,7 +292,7 @@

  $package->{'package'} = $pname;
  $package->{'version'} = "0";
-  $package->{'architecture'} = 'powerpc-apple-darwin';
+  $package->{'architecture'} = 'i386-apple-darwin';
  $package->{'source'} = $package->{'package'};
  $package->{'description'} = "No description available.";
  $package->{'maintainer'} = "Anonymous <darwin-development\@public.lists.apple.com>";
@@ -335,7 +335,7 @@
      $package->{'maintainer'} = "Anonymous <darwin-development\@public.lists.apple.com>";
  }

-  $package->{'architecture'} = 'powerpc-apple-darwin';
+  $package->{'architecture'} = 'i386-apple-darwin';
  $package->{'source'} = $package->{'package'};

  return $package;

Now that we have purged "powerpc-apple-darwin" from the build program, we can get started building stuff.

This simple script will build through all the source on the CD-ROM, and will compile stuff to the /compiled directory.  It tries to keep things 'neat' although it will hammer the hell out of your disk... Not that it matters on modern setups.


cat /build_all.sh
#!/bin/bash

# This is a simple script to compile all of the sources on the CD-ROM
#
# This will use the existing debs on the CD-ROM.  The build script
# from the debian folks will let new debs take precident over the old ones.
# This is basically to prove the build system works

#is the CD mounted? it should be on /mnt
# The CD I'm using is called:
# Darwin03_debs_source_31_5_2017.iso
mount_cd9660 /dev/sd0a /mnt


# Make directories to store the source and the compiled debs
rm -rf /sources
rm -rf /tmp/roots
mkdir /sources
mkdir /compiled
# for each of the source files on the CD
for filename in /mnt/source/*gz; do
cd /sources
# unpack the source
tar -zxf $filename
  for dirname in /sources/* ; do
# run the buildpackage script
      echo "darwin-buildpackage --dir --target all $dirname /mnt/debs /compiled"
      darwin-buildpackage --dir --target all $dirname /mnt/debs /compiled
      rm -rf $dirname
  done
rm -rf /tmp/roots
done

The magic comes from the darwin-buildpackage script.

Anyways Mach back in MK83 days was expecting either COFF or A.OUT not O-MACH.  I don't know if it even really existed yet.  Seems yet another dead end. :(
# include <wittycomment.h>

evolver56k

You can create a 'manifest file' from the extracted tarballs from something like this where you extracted them (if using a bourne shell):
for i in $(ls -1) ; do echo "dir $i all" >> buildmanifest.txt
and using:
darwin-buildall buildmanifest.txt /path/to/debrepo /path/to/debdestination
Will build all the packages it can, and automatically deleting the temp files under /tmp/roots/ on a successful build.
The third field of the manifest is the same as the --target flag for darwin-buildpackage, so it can be any of all|headers|objs|local

There's a Manifest file on the Darwin 0.3 disc (Darwin OS Install CD/Binary Packages/Manifest ... it has mac line endings, so you'll have to change those to unix line endings first before using it with buildall, and change ^cvs to dir) that I believe shows how they built the deb packages for the release