Crazy idea... openstep on darwin?

[gsyoungblood]

Without looking, I'd say this would be near impossible as my guess is that the binary format for darwin is significantly different than os 4.2.

But, on the slimmest of chances, has anyone looked at the possibility of trying to replace the BSD under the hood of OS with the newer Darwin? If, by some miracle, it was possible, then it seems that it would open up support for much newer hardware, except for maybe the video cards.

[cuby]

The binary formats (Mach-O) are almost identical - even the universal binary stuff works (NS/OS fat binaries are the same as Universal Binaries on OS X). Most of the development tools on OS X even support old NeXT binaries - you can still disassemble a m68k NS binary using otool. Try starting an x86 OpenStep binary on a current x86 OS X Mac. It will start, but crash soon in the initialisation.

One still more interesting thing would be possible: creating a wrapper that runs OpenStep binaries on OS X/x86 and translates system and library/framework calls to their OS X equivalents or reimplements them. I'm actually working on this in my spare time.

One problem is supporting NS as well as OS binaries, since the mechanisms for loading shared libraries have changed between NS and OS. Another is decoding and converting the nib files, since this format has changed significantly (there are still more problems around, but these are two of the most obvious).

But running FrameMaker and similar NeXT software on OS X would be great. Actually, I have some m68k NeXT command line binaries running in emulation (CPU emulation combined with system call translation) on Linux and OS X at the moment, but this is highly experimental.

[gsyoungblood]

I'm pretty surprised how close you say they are.

If the difference between NS and OS are that great, but from OS to MacOSX are not as great, then perhaps that's the starting point. Get OS gui, etc. working on top of Darwin (open source) first. Once that's done, try to add NS support second.

At the very least, it would open up new hardware options to keep OS running longer, whether on power pc or intel.

[nextchef]

I'm pretty surprised how close you say they are.

If the difference between NS and OS are that great, but from OS to MacOSX are not as great, then perhaps that's the starting point. Get OS gui, etc. working on top of Darwin (open source) first. Once that's done, try to add NS support second.

At the very least, it would open up new hardware options to keep OS running longer, whether on power pc or intel.

Is this not what the openstep people are trying to do?  Emulate / reverse-engineer that which is necessary to run OS / OSX apps?

Chef

[ash]

My understanding of GNUStep is that they are replicating the class libraries that are required to build and run programs that once ran on OPENSTEP at a minimum.  This also includes the necessary build tools (Gorm ==> Interface Builder, ProjectCenter ==> Project Builder).

So, in theory, if you compile your application against GNUStep, you should be able to compile it against NEXTSTEP/OPENSTEP.  I've had success working with Xcode on OS X and compiling against the base libraries there and then transitioning to GNUStep.  However, the application I was using was not a GUI one.  There are a lot of caveats when doing this:  http://www.kernelthread.com/mac/osx/programming.html

The Application Binary Interface (ABI) for the GNUStep libraries is probably not the same, particularly since GNUStep is probably not sensitive to Mach-O issues, so running pure OPENSTEP apps on this hybrid will probably not work.  It's more likely that you would achieve success in OS X, but even if you could overcome the ABI differences in OS X (what is probably causing the OPENSTEP apps to crash immediately) programatically using Mach-O binary tools, it's unlikely that you would get something that worked as well as your old OPENSTEP application.

The disgusting thing about trying to use the darwin kernel, bsd environment, and GNUStep is that to get anything on the screen, you're going to have to use X11, because that's the backend gnustep-gui will hook into.  In the end, it's just another UNIX with some weird driver interface (IOKit) running X11.

Finally, applications for NEXSTEP 3.3 and below are m86k, so the likelihood of those working properly is zero.  All the applications would probably need to be Quad-FAT to insure i386 code availability.

Oh, and OpenDarwin is now defunct, increasing the difficulty of getting this to work.

(I hope I got all that correct....)

I wouldn't say that the idea is impossible, but it's too much work for too little payoff.

[cuby]

I'm pretty surprised how close you say they are.

Well, the heritage still shows :-)

If the difference between NS and OS are that great, but from OS to MacOSX are not as great, then perhaps that's the starting point. Get OS gui, etc. working on top of Darwin (open source) first. Once that's done, try to add NS support second.

Putting a new Mach-based kernel below the original userland shouldn't be too hard - and support NS as well as OS binaries. The major difference is (apart from differences in the ObjC class libraries) that shared libraries use the old SunOS-like style of being linked to fixed virtual addresses in NS whereas OS supports relocatable shared libs like almost every curreny Unix. I think some of the Mach message semantics may have changed in the transition from 2.5 (NS/OS) to 3.0 (Darwin), but that should be easy to fix.

At the very least, it would open up new hardware options to keep OS running longer, whether on power pc or intel.

I think much of the same could be achieved using DriverKit-based drivers - at least as long as your hardware is still PCI-based. But there are not too many people left who know (and are willing to) write drivers for NS/OS. I still haven't started working on the hme driver for NS/OS-Sparc...

[pTeK]

One still more interesting thing would be possible: creating a wrapper that runs OpenStep binaries on OS X/x86 and translates system and library/framework calls to their OS X equivalents or reimplements them. I'm actually working on this in my spare time.

One problem is supporting NS as well as OS binaries, since the mechanisms for loading shared libraries have changed between NS and OS. Another is decoding and converting the nib files, since this format has changed significantly (there are still more problems around, but these are two of the most obvious).

But running FrameMaker and similar NeXT software on OS X would be great. Actually, I have some m68k NeXT command line binaries running in emulation (CPU emulation combined with system call translation) on Linux and OS X at the moment, but this is highly experimental.

How is this coming along? Would it be possible to convert 68K to x86 assembler and redirect the system calls to the NeXTSTEP x86?

[cuby]

How is this coming along?

I managed to run some command-line binaries, but back then a lot of information on shared library loading, internals of the frameworks, etc. was missing. This could probably be extracted from the available source code now.

Would it be possible to convert 68K to x86 assembler and redirect the system calls to the NeXTSTEP x86?

There's a number of different approaches to translate software:

• Runtime interpretation (the approach I used by utilizing the Musashi 68k emulation library)
• JIT binary translation - requires quite a bit more effort, some versions of Basilisk II (MacOS emulation) used this
• Static translation - this is usually not possible without support by the other methods
  

Apple uses a combination of all three approaches in Rosetta 2 to translate x86-64 binaries to ARM Aarch64. In an ahead-of-time compilation step, as much as possible of the existing binary code is translated to ARM. However, there are some things which cannot be statically translated. Apart from (rather) obvious problems like self-modifying machine code, it's hard to statically decide for a variable-length ISA such as x86 at which byte an instruction starts (since you can have dynamic jump targets like function pointers which are not known at compile time). For these and other corner cases (such as a program reading from its code/.text segment), interpretation is used and the original x86 executable is preserved. If profiling during the execution of interpreted code indicates that the code is used more often (depending on some heuristics), Rosetta 2 then employs a JIT compiler.

If you're interested in details, there are a lot of interesting papers [1-4] and a book I can highly recommend [5] available. It's quite a deep rabbit hole...

See dougallj's blog article for more details. I also wrote a bit about this on my own blog.

And if someone is really interested in these topics (and has an MSc degree in CS), I have an open, fully funded PhD position (~50k€ gross/year) in my research group in Bamberg, Germany .

[1] R. Nigel Horspool and Nenad Marovac. An approach to the problem of detranslation of computer programs. Computer Journal, 23(3):223–229, 1980.
[2] Bernard Amanieu d'Antras, Low Overhead Dynamic Binary Translation for ARM, PhD thesis, University of Manchester, 2016
https://apt.cs.manchester.ac.uk/ftp/pub/apt/theses/Dantras17_phd.pdf
[3] Jiunn-Yeu Chen et al., Effective code discovery for ARM/Thumb mixed ISA binaries in a static binary translator, Proceedings of CASES 2013
http://dx.doi.org/10.1109/CASES.2013.6662525
[4] Bor-Yeh Shen et al., A Retargetable Static Binary Translator for the ARM Architecture, ACM TACO Vol. 11, No. 2, 2014
https://dl.acm.org/doi/10.1145/2629335
[5] James E. Smith and Ravi Nair, Virtual Machines - Versatile Platforms for Systems and Processes, 2005, ISBN 978-1-55860-910-5
https://www.sciencedirect.com/book/9781558609105/virtual-machines

[spitfire]

I for one would love to run Framemaker and Lotus Improv on either OSX, or GNUStep/Linux.
What's the most direct path for this? Do you still have your interpreter running command line apps around? Start there and move slowly on?

Note there are some really good 68K JITs freely available now. Given that the fastest NeXTs were 33mhz and ran just fine, I would settle for a JIT that gives me a ~330mhz CPU to an application.[0]

As you mentioned NS was statically linked, whereas OS was dynamically linked. Given all the apps people care about are NS, that would make sense as the target.

Do we restart work getting command line apps going again?

• Things like emu68 give you *far* more than ~330mhz equivilant these days.

[gtnicol]

As you mentioned NS was statically linked, whereas OS was dynamically linked. Given all the apps people care about are NS, that would make sense as the target.

Do we restart work getting command line apps going again?

That seems like a fruitful path. It would be interesting to see how much it'd take to get graphic apps running. I'd expect display postscript to be needed.

[cuby]

That seems like a fruitful path. It would be interesting to see how much it'd take to get graphic apps running. I'd expect display postscript to be needed.

I think that this depends quite a bit on the approach chosen. Command line apps can be mostly emulated by wrapping the libc calls in libSystem to native libc calls.

For graphic apps, we need support for emulating Mach ports, as there is communication between the AppKit and WindowServer (and more) going on. Some details can probably be extracted from the NetBSD compat_darwin(8) code - they tried to get OS X 10.2 WindowServer to work (on a PPC NetBSD machine, no binary translation involved).

This is probably easier if we "only" want to run m68k binaries on the identical x86 version of NeXT/OpenStep (and maybe also on Rhapsody x86/PPC?). I would expect that we could get quite far by wrapping the common frameworks.

Running m68k NeXT binaries on other platforms either requires a recreation of the NeXT frameworks (based on e.g. GNUstep, cocotron or some of the other efforts - a lot of work) or using the original m68k NeXTstep binary frameworks and a combination of system call/Mach port emulation and - I think – at least a recreation of WindowServer.

AFAIK, most of the communication between AppKit and WindowServer is not documented. But I would love to hear everyone's opinion on the right way to approach m68k NeXT user mode emulation...

Update: I remember I read somewhere that the 1.0 protocol was documented in a file "WindowServerProtocol.wn", see this old usenet posting.

Does anyone have a copy of that file?

[gtnicol]

For graphic apps, we need support for emulating Mach ports, as there is communication between the AppKit and WindowServer (and more) going on. Some details can probably be extracted from the NetBSD compat_darwin( code - they tried to get OS X 10.2 WindowServer to work (on a PPC NetBSD machine, no binary translation involved).

I was thinking display postscript might be needed under the assumption that at least some of those messages might contain postscript. Mach port emulation should be straightforward. I'd love to get at least a few of the apps running... things like Frame, Improv etc.

[spitfire]

AFAIK, most of the communication between AppKit and WindowServer is not documented. But I would love to hear everyone's opinion on the right way to approach m68k NeXT user mode emulation...

Update: I remember I read somewhere that the 1.0 protocol was documented in a file "WindowServerProtocol.wn", see this old usenet posting.

Does anyone have a copy of that file?

I can do you one better. We have the source code for the display postscript server.
In the NeXT source coe thread there's a link to NeXTDPS repo, that's it.

I'm not sure if GNUStep or OSX are the right path to go down. GNUStep emulates the openstep API so might be a little closer (has postscript, etc), but OSX will have mach binary support and tooling around m68K still somewhat intact.

I have to admit I'd get a thrill running framemaker and quantrix/improv on OSX. I've been missing those things for *ages*.

[gtnicol]

I can do you one better. We have the source code for the display postscript server.
In the NeXT source coe thread there's a link to NeXTDPS repo, that's it.

I'm not sure if GNUStep or OSX are the right path to go down. GNUStep emulates the openstep API so might be a little closer (has postscript, etc), but OSX will have mach binary support and tooling around m68K still somewhat intact.

I have to admit I'd get a thrill running framemaker and quantrix/improv on OSX. I've been missing those things for *ages*.

I'll look for the WindowServerProtocol.wn file - I probably have it here somewhere. The Dimension uses similar messages from memory.

On another thread I pondered about porting XNU to something like a RISC-V and then bringing up DPS, WindowServer, Workspace, etc. etc. I always thought that DPS was in many ways more elegant than X11 et al. Nowadays you'd probably want to secure it a bit better of course.

[cuby]

I'll look for the WindowServerProtocol.wn file - I probably have it here somewhere. The Dimension uses similar messages from memory.

I tried to find out some more information on the protocol. When NXHost'ing an application to a remote machine, messages are relayed via netmsgserver on TCP port 2453. The communication seems to use mostly binary encoded (D)PS messages. So it might be interesting to create an NXHost display server first (without emulating m68k code) - this could then also be a basis to implement a local WindowServer and might also be a useful addition to Previous.