Oooh, I LOVE this! Especially the ability to "Overriding emulated code with C# code" I had a similar idea years ago (https://gabrielgambetta.com/remakes.html), not in the context of a debugger or reverse engineering per se, but in the context of remakes and "special edition" games. Not entirely surprised that this is a byproduct of OpenRakis. Amazing work!
I tried doing something like this about 15 years ago but specifically for audio by routing NES NSF rom audio data (square, triangle, PWM, etc) to virtual midi cables attached to VSTs so you could play any old school Nintendo game with modern instrumentation. Was a pretty fun project.
The closest thing I can think of for graphical rehauls is probably shader pack type stuff - Minecraft is a great example of this.
Reverse engineering old games is like digital archaeology—except instead of digging up fossils, you’re unearthing spaghetti code and DRM nightmares. Spice86 seems like an exciting new shovel for the job!
Forty years ago I had a Sinclair QL with an 8086 emulator. Because the Sinclair QL had preemptive multitasking, I could easily search memory for patterns, monitor locations, stop and start the emulation, or change memory programmatically and easily from the QDOS side. It was worlds easier than using a debugger, particularly since I didn't own an 8086 system.
I always thought it was a clever way to get insights in to software while it was running that wasn't available to people with 8086 systems, and it's interesting to see this idea so many years later.
Bochs and MAME both have superb and widely-used debuggers, while Qemu is more limited but still has some debugging capabilities in its monitor, as well as a gdb integration. (Can’t say anything about PCem/86Box.) It seems that developers of emulators targeting good coverage of old stuff simply can’t not build a debugger, because it’s an integral part of their task to figure out what the hell the devs of the latest failing thing did to make it fail. Bochs is (was?) also quite popular in the OSDev scene as a debugging tool.
DOSBox can be configured to include a debugger. The feature is not enabled in the official binary but the enhanced derivative projects probably have it (DOSBox-X definitely does):
Hey because I don't feel like making a phone call to my BSD expert, is that a flag? Like in Gentoo it would be like
USE="debugger" emerge -vaD bochs #portage reads env and will set flags this way or in /etc/portage/packages.use/bochs (folder name packages.use is arbitrary and I'm old school.)
Curious if BSD is like that too and I am way to tired to attempt to search for it with correct words...
Oh yeah, that is right. thank you! i didn't want to have to install a bsd just to check, i don't have one handy right now!
to clarify: USE="debugger" would be a flag that the package has, when you call the package manager to build it from source it just enables that in the make or whatever. package.use is if you want to make sure that some other package doesn't uninstall/modify your package. I am doing a poor job of explaining.
package.use is useful for maintaining your flags during upgrades. USE="debugger" or USE="-debugger" are one off, and i should have specified by putting emerge -va1D bochs (or whatever)
gentoo is a source-based OS, in a similar way to at least the bsds i've used.
Gentoo does have "system wide USE flags" they go in /etc/portage/make.conf, where you would set like USE="-gtk -alsa -X" or whatever, and portage will balk if some package tries to pull in masked packages. "Package has been masked by USE -X"
make.conf makes sure that specific tech stacks won't be pulled in as dependencies unless you specifically unmask the flag for that package in package.use or USE="" emerge[...].
package.use is where you keep your "this is how i want this software built when you do it for me" {this part is wrong, but i don't feel like explaining all of the portage package dot whatever weirdness, because anyone who started using gentoo in the last decade or so might strongly disagree with my assertions about naming and other convention - Ed.}
USE="" is if i need to install some package with a specific feature that isn't enabled by default. Such as debugging real mode or whatever. I generally only USE="" if it's something i need for a few moments and then gets uninstalled. On a source based OS keeping oddball archive support packages or whatever adds time and heat to your updates.
In general, compared to gentoo, things like buildflags, for CPU(-features/levels/generations), general compilerflags like -02 or -O3, and linker-flags would be set for the build of the kernel, or additionally the 'world'.
IF you are building from source at all. 'World' meaning the complete basic userland, in sync with the kernel, as one cohesive thing, (or in linux-speak the base image of the 'Distro/Distribution')with individual choices of 'ports' runnig atop of that.
That's less granular than gentoo. OTOH gentoo can be used with binaries too, or not?
Anyways, the fine granularity of gentoos use-flags isn't replicated anywhere else, afaik.
(Yah well, Paludis on Exherbo possibly, but too much hassle for me)
If you ask nixers (Nix-Os) about it, they don't get it, because of reproducible builds.
And on FBSD, I had the feeling that too much 'ricing' is being frowned upon.
Though I did that, a looong time ago, with both Free- and NetBSD.
Nowadays I'm just relying on others doing the 'tuning' and compiling for me(mostly), and in case of CachyOS that works really well for me, atm.
From my brief experience, it seems that reversing old games is one of those disciplines where there is no good step-by-step course. One can start learning some theory (I did so by reading “The Art of Assembly Language Programming” – if I had more time, I'd try “Reverse Engineering for Beginners”) but then one has to get his hands dirty. Real-life games are typically not simple, but one usually just needs to reverse some small parts to produce new interesting modifications.
But surely reading tutorials is useful to learn techniques and tricks. I recommend this article to start: https://www.lodsb.com/reversing-lz91-from-commander-keen (not totally for beginners, but very friendly, and it can help to get used with the jargon). I am also publishing war stories on this topic on my blog (marnetto.net).
In this case one hard part was trying to get code to compile to identical byte for byte output. Which meant working out which compiler options were used, and which specific compiler too. That gives you a hint of what kinda things are involved.
Have you mentioned this before? I saw a similar context comment maybe 3 days ago on here, down to "byte for byte output [...W]hich compiler options were used"
Why can't ghidra (or any other reverse engineering tool) be used directly on the .exe? Why do you have to go through this emulator? Is it because the thing you want to debug only runs in x86 realmode?
Very roughly spoken, the older the platform, the "weirder" stuff that happens at runtime gets in order to make the best of the measly hardware.
I'm looking at a many decades old C64 game right now, and the way I'm doing it is by having taken a memory snapshot when the game was in the state that I'm most interested in at first.
Starting with just the "binary" on disk would be much harder, since there's so much code that just loads, decompresses, initializes overlays etc. which isn't particularly interesting from an actual game logic point of view (though may be very interesting for other reasons), and only exists because the whole game just doesn't fit into all of memory.
There's also a bunch of loading and overlay magic at runtime of the game, but by looking at a snapshot of the game in the state that interests me, I don't have to dive into that (yet).
-there is sometimes not a single statical exe (that means all code inside) but overlays(DOS like DLLs) or serveral other ways of loading code at runtime (example for sound/gfx-drivers) - DOS allows technicaly nearly everything so everything is done in games :)
-many game loaders combine code/data parts of a game in memory - for keeping floppy releases smaller
-self modifying code, also hard to disassemble statically with Gidrah/IDA
-good old segment/offset 16bit realmode games - a complete different beast compare to 32bit linear DOS games (Ghidra isn't very good at this, IDA is much much better)
some examples:
the Stunts loader combines several (in itself non valid) files in memory to create a exe (the single files are packed and the result in exe in memory is also packed) - not that easy to static disassemble something like that
Alpha Waves also got an loader and self modifying code that is not easy to reverse statical
its good to have the best disassemblers available and the best (or better dedicated) debuggers around to keep your reversing project shorter then decades :)
I believe part of the problem is the fact that Aa.exe fil B is created BY packaging multiple library files And or graphics , arrays ETC. and there is no default order into which part of the EXE file they land.
there are some Tools ... hex editors come to mind. I seem to recall NOPING out A jump or two in my younger days
edit: the these days that probably wouldn't work due to CRC checks... but there was a time... Then again that may be just the perfect place to start riverus engineering;) smile I have some good memories of playing a Medal of honor in which I changed all the door Textures to transparent window textures and having to work around CRC protection... good times smiley :)
Approximately how long does it take to collide a CRC naively? I'm guessing there's a trick that makes it faster, these days?
It takes my computer on a single core about 7 minutes to find a nonce for an arbitrary files sha256 to prefix the left side with 4 or 5 zeros (like bitcoin difficulty doubling). Obviously the heat death of the universe would occur trying to collide sha256 on a single core, but CRC - Gemini says it depends on the algorithm, but crc 32 should take about an hour to collide, but it didn't specify "any" or "arbitrary" collisions, but mentioned "any" right before that. So if the most probable sentence after "any collision" is a time estimate, with the logic of LLM implies that's the easier case of any collision.
I'm surprised brute force is even needed. As far as I know, CRC has absolutely no intent to be a cryptographically secure one-way function. It is purely used against unintentional corruption of data. With that in light, does it really take an hour to find a collision? Can't you construct one much quicker?
893 byte source file, python can single-threaded do ~1 million crc32 per second. It's about "10%" done with the AI's version which is nonce = 0 while true nonce +=1 crc32 (data+nonce.to_bytes) which i don't buy will actually cover the whole field, so i'll just repurpose my sha256sum bash script to use cksum and run it again - that puts arbitrary bytes, not arbitrary integers as bytes into the file.
nonces tried: 515000000
nonces tried: 516000000
Collision found with nonce: 1327202703
walltime ~12 minutes. So whoever i replied to (sorry my vision is going blurry so i don't want to breadcrumb back) was correct, modern CPUs just blow through this.
you'll note the nonce is > 2^32. Cute.
This is what AI does. This sort of crap is going to make everything feel slower. except now instead of inefficient humans making inefficient code because "hey, just scale" or "my desktop has 32 threads, why do i care about a 50ms hot loop? one of the other 31 threads can pick up slack" - now it's ... this.
i absolutely cannot believe it chose the absolute most naive way to accomplish this monumentally trivial task.
x86 segmentation makes it very hard to statically analyze anything. In real mode, any byte can be referenced in 4096 different ways. It is even messier in protected mode, since now every selector is an entry in a table, so its value itself is meaningless. So, without runtime analysis, there is no way to tell if 04:1234 is or is not the same byte as fa:1204
> It is even messier in protected mode, since now every selector is an entry in a table, so its value itself is meaningless.
Actually, my experience is that things are much easier in protected mode. Since selector values are chosen by the OS, that means you rely a lot more on internal relocations. And the use of segment selectors is a strong indicator that you have a pointer in the first place.
Unfortunately, ghidra itself struggles to apply these techniques, especially in the decompiler, which seems completely unable to cope with the concept of far pointers.
My guess is portability, then obviously performance.
edit: actually there is a specific answer for this particular project - "We had to rewrite the project in C# to add automated code generation (java doesn't have the goto keyword, making automated ASM translation challenging)". There you are.
I mean, that's more or less the reason why it isn't Java, not why it's ultimately C#. My guess is that Java is just what they're most comfortable with, with C# being similar enough but avoiding specific limitations in that case.
Wasn't C# essentially microsoft throwing their hat in the ring against Oracle and to show off how cool this .net stuff is?
I dabbled in both at around the same time a long time ago for console apps and visual studio's autocomplete / assist / library fetch etc made it easier than Java to get working in but...
Its been so long I forget the origin stories sometimes.
The closest thing I can think of for graphical rehauls is probably shader pack type stuff - Minecraft is a great example of this.
https://www.sonicether.com/seus
I always thought it was a clever way to get insights in to software while it was running that wasn't available to people with 8086 systems, and it's interesting to see this idea so many years later.
- https://www.vogons.org/viewtopic.php?t=3944
- https://github.com/joncampbell123/dosbox-x/wiki/DOSBox%E2%80...
https://github.com/joncampbell123/dosbox-x/blob/master/READM...
USE="debugger" emerge -vaD bochs #portage reads env and will set flags this way or in /etc/portage/packages.use/bochs (folder name packages.use is arbitrary and I'm old school.)
Curious if BSD is like that too and I am way to tired to attempt to search for it with correct words...
See https://www.freshports.org/emulators/dosbox-x/
to clarify: USE="debugger" would be a flag that the package has, when you call the package manager to build it from source it just enables that in the make or whatever. package.use is if you want to make sure that some other package doesn't uninstall/modify your package. I am doing a poor job of explaining.
package.use is useful for maintaining your flags during upgrades. USE="debugger" or USE="-debugger" are one off, and i should have specified by putting emerge -va1D bochs (or whatever)
gentoo is a source-based OS, in a similar way to at least the bsds i've used.
Gentoo does have "system wide USE flags" they go in /etc/portage/make.conf, where you would set like USE="-gtk -alsa -X" or whatever, and portage will balk if some package tries to pull in masked packages. "Package has been masked by USE -X"
make.conf makes sure that specific tech stacks won't be pulled in as dependencies unless you specifically unmask the flag for that package in package.use or USE="" emerge[...].
package.use is where you keep your "this is how i want this software built when you do it for me" {this part is wrong, but i don't feel like explaining all of the portage package dot whatever weirdness, because anyone who started using gentoo in the last decade or so might strongly disagree with my assertions about naming and other convention - Ed.}
USE="" is if i need to install some package with a specific feature that isn't enabled by default. Such as debugging real mode or whatever. I generally only USE="" if it's something i need for a few moments and then gets uninstalled. On a source based OS keeping oddball archive support packages or whatever adds time and heat to your updates.
In general, compared to gentoo, things like buildflags, for CPU(-features/levels/generations), general compilerflags like -02 or -O3, and linker-flags would be set for the build of the kernel, or additionally the 'world'.
IF you are building from source at all. 'World' meaning the complete basic userland, in sync with the kernel, as one cohesive thing, (or in linux-speak the base image of the 'Distro/Distribution')with individual choices of 'ports' runnig atop of that.
That's less granular than gentoo. OTOH gentoo can be used with binaries too, or not?
Anyways, the fine granularity of gentoos use-flags isn't replicated anywhere else, afaik.
(Yah well, Paludis on Exherbo possibly, but too much hassle for me)
If you ask nixers (Nix-Os) about it, they don't get it, because of reproducible builds.
And on FBSD, I had the feeling that too much 'ricing' is being frowned upon.
Though I did that, a looong time ago, with both Free- and NetBSD.
Nowadays I'm just relying on others doing the 'tuning' and compiling for me(mostly), and in case of CachyOS that works really well for me, atm.
But surely reading tutorials is useful to learn techniques and tricks. I recommend this article to start: https://www.lodsb.com/reversing-lz91-from-commander-keen (not totally for beginners, but very friendly, and it can help to get used with the jargon). I am also publishing war stories on this topic on my blog (marnetto.net).
https://neuviemeporte.github.io/category/f15-se2
In this case one hard part was trying to get code to compile to identical byte for byte output. Which meant working out which compiler options were used, and which specific compiler too. That gives you a hint of what kinda things are involved.
Why can't ghidra (or any other reverse engineering tool) be used directly on the .exe? Why do you have to go through this emulator? Is it because the thing you want to debug only runs in x86 realmode?
Very roughly spoken, the older the platform, the "weirder" stuff that happens at runtime gets in order to make the best of the measly hardware.
I'm looking at a many decades old C64 game right now, and the way I'm doing it is by having taken a memory snapshot when the game was in the state that I'm most interested in at first.
Starting with just the "binary" on disk would be much harder, since there's so much code that just loads, decompresses, initializes overlays etc. which isn't particularly interesting from an actual game logic point of view (though may be very interesting for other reasons), and only exists because the whole game just doesn't fit into all of memory.
There's also a bunch of loading and overlay magic at runtime of the game, but by looking at a snapshot of the game in the state that interests me, I don't have to dive into that (yet).
-there is sometimes not a single statical exe (that means all code inside) but overlays(DOS like DLLs) or serveral other ways of loading code at runtime (example for sound/gfx-drivers) - DOS allows technicaly nearly everything so everything is done in games :)
-many game loaders combine code/data parts of a game in memory - for keeping floppy releases smaller
-self modifying code, also hard to disassemble statically with Gidrah/IDA
-good old segment/offset 16bit realmode games - a complete different beast compare to 32bit linear DOS games (Ghidra isn't very good at this, IDA is much much better)
some examples:
the Stunts loader combines several (in itself non valid) files in memory to create a exe (the single files are packed and the result in exe in memory is also packed) - not that easy to static disassemble something like that
Alpha Waves also got an loader and self modifying code that is not easy to reverse statical
its good to have the best disassemblers available and the best (or better dedicated) debuggers around to keep your reversing project shorter then decades :)
It takes my computer on a single core about 7 minutes to find a nonce for an arbitrary files sha256 to prefix the left side with 4 or 5 zeros (like bitcoin difficulty doubling). Obviously the heat death of the universe would occur trying to collide sha256 on a single core, but CRC - Gemini says it depends on the algorithm, but crc 32 should take about an hour to collide, but it didn't specify "any" or "arbitrary" collisions, but mentioned "any" right before that. So if the most probable sentence after "any collision" is a time estimate, with the logic of LLM implies that's the easier case of any collision.
nonces tried: 515000000 nonces tried: 516000000 Collision found with nonce: 1327202703
walltime ~12 minutes. So whoever i replied to (sorry my vision is going blurry so i don't want to breadcrumb back) was correct, modern CPUs just blow through this.
you'll note the nonce is > 2^32. Cute.
This is what AI does. This sort of crap is going to make everything feel slower. except now instead of inefficient humans making inefficient code because "hey, just scale" or "my desktop has 32 threads, why do i care about a 50ms hot loop? one of the other 31 threads can pick up slack" - now it's ... this.
i absolutely cannot believe it chose the absolute most naive way to accomplish this monumentally trivial task.
Oddly, this kind of topic doesn’t get a lot of attention
Actually, my experience is that things are much easier in protected mode. Since selector values are chosen by the OS, that means you rely a lot more on internal relocations. And the use of segment selectors is a strong indicator that you have a pointer in the first place.
Unfortunately, ghidra itself struggles to apply these techniques, especially in the decompiler, which seems completely unable to cope with the concept of far pointers.
so indeed you'd know it is a far pointer, but may not know what to :D
edit: actually there is a specific answer for this particular project - "We had to rewrite the project in C# to add automated code generation (java doesn't have the goto keyword, making automated ASM translation challenging)". There you are.
I dabbled in both at around the same time a long time ago for console apps and visual studio's autocomplete / assist / library fetch etc made it easier than Java to get working in but...
Its been so long I forget the origin stories sometimes.