Duke4.net Forums: Restoration of a few games' EXEs versions - Duke4.net Forums

Jump to content

Hide message Show message
Welcome to the Duke4.net Forums!

Register an account now to get access to all board features. After you've registered and logged in, you'll be able to create topics, post replies, send and receive private messages, disable the viewing of ads and more!

Page 1 of 1
  • You cannot start a new topic
  • You cannot reply to this topic

Restoration of a few games' EXEs versions  "Let's call it software archaeology?"

User is offline   NY00123 

  • 102

#1

Hey there,

I think it's a good (albeit belated) time to introduce you to the following, at least if you aren't yet aware of this: https://bitbucket.or...ver-recreation/

To summarize, the above repository covers modifications of open source games, with attempts to recreate original old EXEs byte-by-byte. Sometimes it's more successful, while other times it's less. A few examples:
- For most Wolfenstein 3D and Spear of Destiny EXEs, with the right tools, you may now faithfully recreate the original EXEs from the 90s, byte-by-byte. There may be this or that EXE for which only the very first few bytes (in the EXE header) differ.
- For Rise of the Triad, I had less luck, and there are also technical limitations due to the usage of Watcom 10.0b (from the 10.0(x) series), but I believe it's *very* close.

Now, let's get to the following. Note that I consider this code recreation *partial*, hence you'll find it in a "bonus" subdirectory. In particular:
- This covers game code only.
- This does *not* cover engine code, audio library code* or closed-source libraries. Original binary OBJ and LIB files are used for these.

Basically, I'm talking about the game code (and no other code piece) of Duke Nukem 3D!

These EXEs are now covered in the "duke3d" directory (bonus/duke3d) within my repository:
- Duke Nukem 3D: Atomic Edition v1.5, up to differences described below (possibly hidden under "Spoiler" tags).
- The following EXEs originally built by Matt (recreated byte-by-byte): NAM/NAPALM Full Version 1.0, WW2GI Full Version 1.0 and EDuke 2.00.23 (the original binary EDuke release from the year 2000 mostly known as "EDuke 2.0").

Note that for any comparison you might be making, there should *not* be any (originally) unofficial patch applied, like the removal of copy protection. Digital releases (e.g., Steam and GOG.COM) may be impacted, too.

For Duke3D v1.5, use Watcom C 10.0 and *not* any other version; Not even 10.0a or 10.0b.
For any of Matt's EXEs, use Watcom C 10.6a (and no other version).

For a little background behind the beginning of all of this, I'll quote myself (with any possible minor diff) from the RGB Classic Games forums (https://www.classicd....php?f=3&t=1410):

Spoiler


Some more details regarding Duke Nukem 3D, quoting from a recent RGB forums post with a few changes (https://www.classicd...hp?p=8406#p8406):

Spoiler


* In fact, this isn't entirely accurate. I've modified the Apogee Sound System sources so you can more-or-less recreate v1.1. However, you'll still prefer to use the existing AUDIO_WF.LIB file, if only due to unused gaps between C string literals and more being filled with contents differing from the original. As in the case of the game code, this might be related to the Watcom C v10.0(x) series.


This post has been edited by NY00123: 11 May 2018 - 12:50 AM

7

User is offline   MrFlibble 

  • 499

#2

Probably not the focus of your endeavour, but is it possible to create a shareware v1.5 version of Duke3D using your findings?
0

User is offline   NY00123 

  • 102

#3

I should probably begin with a "tl;dr" summary.

So, with the modified Duke3D codebase of mine, you may now create a Duke3D EXE that (with very few exceptions) behaves *exactly* like v1.5.
Among other things, this means:
- Identical file sizes (up to the presence/lack of the DOS4GW extender).
- Identical binary EXE file contents (up to minor diffs that don't matter + DOS4GW).
- Compatibility with demos recorded with v1.5 (at least the original 3 demos seem to be OK).
- Compatibility with v1.5 saved games (although surprisingly enough, these may work in v1.4 after a simple hex-edit of BYTEVERSION in the SAV files; Your mileage may vary, though).

3 more EXEs may further be recreated, and even perfectly. (NAM v1.0, WW2GI v1.0 and Enhanced Duke (EDuke) v2.00.23.)


View PostMrFlibble, on 12 May 2018 - 04:47 AM, said:

Probably not the focus of your endeavour, but is it possible to create a shareware v1.5 version of Duke3D using your findings?


I suppose that a "new" shareware EXE can be made. Out of curiously, I've even tried to do this on my own.
For reference, see the attached source code patch and EXE (which requires DOS4GW or compatible).

A few notes about my "shareware v1.5" code changes:
- VOLUMEALL and PLUTOPAK shall *not* be defined.
- On the other hand, VOLUMEONE should be. A definition of NOCOPYPROTECT is probably desired, too.
- There are a few instances in which the Atomic Edition logo (PLUTOPAKSPRITE and other few sprites) is shown. While there's code to exclude this, only the UK macro is checked for some reason, not PLUTOPAK. So, better fix these as well.

As for the data, I've used shareware v1.3d's. However, it looks like the new EXE is incompatible with v1.3d's CON, so I've used v1.5.

There may still be this and that issue. I've got this message of a missing sound file in-game. This might be related to at least one of the CON files.

Attached File  duke3d_v15_recreated_shareware_patch_20180512.7z (395.11K)
Number of downloads: 3


This post has been edited by NY00123: 12 May 2018 - 06:14 AM

1

User is offline   MrFlibble 

  • 499

#4

View PostNY00123, on 12 May 2018 - 06:04 AM, said:

As for the data, I've used shareware v1.3d's. However, it looks like the new EXE is incompatible with v1.3d's CON, so I've used v1.5.

The GRP from the Macintosh demo of Duke3D includes the missing Atomic logos for the title screen & menu, and the CONs might be v1.4/1.5 compatible although I haven't checked this yet. I have always thought that the Mac demo GRP could actually be a stripped-down version of the post 1.3d shareware that never went public.
1

User is offline   NY00123 

  • 102

#5

View PostMrFlibble, on 12 May 2018 - 12:39 PM, said:

The GRP from the Macintosh demo of Duke3D includes the missing Atomic logos for the title screen & menu, and the CONs might be v1.4/1.5 compatible although I haven't checked this yet. I have always thought that the Mac demo GRP could actually be a stripped-down version of the post 1.3d shareware that never went public.


This is a good idea. I've checked not just duke68kdemo.bin, but also duke3d.sit.hqx. I've used "The Unarchiver" (unar) for unpacking these.
These two demo archives seem to have the same CON and Group files. The CON files are exactly the same as in DOS v1.5, up to the differing new line formatting (CR for the old Macs, CR+LF for DOS).

While this seems to more-or-less work, there are still a few differences to mention. For instance:
- The Mac demo(s) lack MIDI files.
- There are less ART graphics when compared to DOS version. The missing graphics include the Atomic logos, too.

There are chances the Atomic logos are stored in a separate file. This may also apply to the graphics used for misc. Mac-exclusive cheat codes, like DN1984.

To conclude, though, then yeah, the Mac demo art is clearly based on DOS v1.5's, just with less of it.
As for comparison to shareware version 1.3d, all I'll say for now is the following:
- Shareware v1.3d and the Mac demo both have TILES000.ART-TILES012.ART and no more.
- The same non-shareware tiles were stripped from both versions i.e., there's no tile present in one of them and not in the other.
- Unsurprisingly, though, there are tens of tiles differing. Note that a few tiles didn't change between *registered* DOS versions 1.3d and 1.5, while there are earlier revisions in *shareware* version 1.3d.


This post has been edited by NY00123: 12 May 2018 - 02:43 PM

0

User is offline   TerminX 

  • el fundador
  • 5,207

  #6

View PostNY00123, on 12 May 2018 - 02:36 PM, said:

Note that a few tiles didn't change between *registered* DOS versions 1.3d and 1.5, while there are earlier revisions in *shareware* version 1.3d.

That's correct. The art was not updated between shareware releases, so shareware 1.3d still contains things like the angled rocket sprites from 1.0, a switch sprite that exists in LameDuke, etc.

EDuke32 wiki svn builds bugs
Join us in #eduke32 on irc.freenode.net!
1

User is offline   0815Jack 

  • 9

#7

sorry for asking this dumb question, but what is the point of recreating the exe byte-by-byte ?
1

User is offline   Mark. 

  • Honored Donor
  • 2,299

#8

I wondered the same thing. I kept quiet thinking I was the only dumb one who doesn't know. ;)
1

User is offline   NY00123 

  • 102

#9

This is actually a very good question! I'll give here two ways in which recreating original code may assist:

- Heard of the "Chocolate Doom" source port? It's the one that aims to accurately reproduce the experience of playing the original Doom games from the 90s using any of the original DOS EXEs, at least as of version 1.9.
Among other things, this includes the general feeling, as well as compatibility with original saved games, demo files and maps (while *not* supporting contents which is *not* compatible with any of the original DOS EXEs).

Thus, recreating an EXE byte-by-byte, even if just partially, can *very greatly* assist with such an endeavour.

Note that in the case of Chocolate Doom, they managed to do so without this so far - The Linux Doom sources were used, and there are some different behaviors from 1.9, so more work had to be done.
There may still be certain technically differing behaviors as a consequence.

- Secondly, and this mostly applies to code in my repository for other games, like Wolfenstein 3D and Blake Stone (sorry, not ETA for Duke Nukem 3D or any other game):
If anybody has ever been curious to know how do two versions of a game differ, not just in data but also in the code - this is more-or-less what recreated code allows you.
Note that there are a few possible pitfalls:
* You may assume that the various #if/#ifdef/#ifndef statements I've added are sufficient for locating *all* technical differences, but it isn't exactly accurate. In fact...
* Even if you change the compilation optimizations for just ONE compilation unit (e.g., a C source file), good chances are the EXE will behave somewhat different.
This can be anything, from speed-sensitive diffs that you won't notice, to undefined/unspecified behaviors that you *will* notice (even if not rarely).
* Finally, even if you change *just* the code, the compiler may add other technical differences you haven't caught (again related to undefined/unspecified behaviors).


This post has been edited by NY00123: 13 May 2018 - 11:36 AM

0

User is offline   Mark. 

  • Honored Donor
  • 2,299

#10

I'm not familiar with Doom ports. But maybe my question has been answered. My uneducated guess was that maybe this new exe file might be an alternative to having to use Dosbox in windows to run games as they were originally meant to run in dos. In other words, playing a dos game without having to mount drives and configure this and that. But it doesn't look like it.

This post has been edited by Mark.: 13 May 2018 - 12:31 PM

1

User is online   oasiz 

  • 658

#11

Nah, this is more about accurately reconstructing the original DOS exe that you got back in the days.
Source releases that we usually get are not 1:1 of what they had when they pressed "compile". Like in D3D case it was said above that it's somewhere before 1.5.
Often they remove bits and pieces that they don't want to share like audio libraries, etc..

Cool aspect about this is actually getting the original files working and in this case we can finally re-create stuff like the shareware 1.5.
Sure it's a bit of a novelty for most, but it's really useful for comparison purposes and preservation.

Not to mention any source code studies and "chocolate" ports.
2

User is offline   0815Jack 

  • 9

#12

thx for the detailed explanation :)
0

User is offline   NY00123 

  • 102

#13

oasiz is surely spot-on with his explanation, thanks!

View PostMark., on 13 May 2018 - 12:29 PM, said:

My uneducated guess was that maybe this new exe file might be an alternative to having to use Dosbox in windows to run games as they were originally meant to run in dos. In other words, playing a dos game without having to mount drives and configure this and that.


The modified codebase for DOS may still be a basis for some individual eventually adapting it to modern platforms, thus letting you do exactly that! (Basically the "Chocolate Doom" idea.)
1

User is offline   Mark. 

  • Honored Donor
  • 2,299

#14

Thanks for explaining. I couldn't think of anything else other than it being a "nerd experiment". And I don't mean that in a bad way.
1

User is offline   NY00123 

  • 102

#15

In fact, how could have I forgotten this!

The results of my work to recreate the NAM and WW2GI EXEs (EDUKE.EXE had been the first) were used to fix a few inaccuracies with the NAM and WW2GI support in EDuke32.


This post has been edited by NY00123: 14 May 2018 - 11:09 AM

1

User is offline   leilei 

  • 438

#16

Different lineage, there's also some functionality loss in the source releases for Quake and Quake III Arena that existed in prior versions that are hard to recreate and have been omitted from the source release:

Quake:
- Menu fade tint (easily recreateable with the dead gelmap code scrap)
- Sound attenuation calculation is different
- DJGPP makefile is missing from source release

Quake III Arena:
- Dynamic lights updating the lightmap (Q3Test 1.01 to 1.07)
- Null sound effect sawtooth buzz (Q3Test 1.01 to 1.08)
- Fast 8-bit sample mixing (Q3Test 1.01 to Q3 1.27g - s_loadas8bit and s_compression)
- Earlier protocols/traps before the big mod regression (1.11 to 1.17)

Just mentioning it here because the q scene has 0 interest in anything that's not fucking promode aerowalk / brutal phase / bad hd packs. I don't really see interest in anything historical elsewhere.

Posted Image

This post has been edited by leilei: 14 May 2018 - 07:19 PM

0

User is offline   Little Tijn 

  • 20

#17

View PostNY00123, on 14 May 2018 - 10:57 AM, said:

In fact, how could have I forgotten this!

The results of my work to recreate the NAM and WW2GI EXEs (EDUKE.EXE had been the first) were used to fix a few inaccuracies with the NAM and WW2GI support in EDuke32.


I saw your name in the changelog of Eduke32 related to this. I must say, nice done!

This project has been proved to be useful already... ;)

There still is a problem with the automatic weapons in WW2GI in Eduke32. When quickly pressing the fire button, the firing animation is played, although no shot is fired. If you have time and are willing to check why this is, that will be great! :)

Details are here:
https://forums.duke4...post__p__300746

This post has been edited by Little Tijn: 15 May 2018 - 11:14 AM

1

User is offline   Little Tijn 

  • 20

#18

Oops, hijacked this thread a bit with my request above, sorry about that.

More on-topic:
I would have expected that the released sources of games should be the latest code available, seeing that this code is used before to build the latest version. They had this code, to build the game. So why isn't it used, but instead some in-between version? Isn't it easier to release the latest code available? That are questions that I got from this thread.

I know that one of the most important fixed in the day for Duke Nukem 3D was a memory leak fixed in 1.5. Because with computer at that time with 8MB RAM etc, it would make a big different. Wondering is this is also included in the source release. Where there some other significant changes?

Speaking of released versions, we have some cases where the code to a later version that the releases binary is available. Off the top of my head, the code of Eduke 2.1 and the released code in dubious legal state of Witchaven II. If I recall correctly, the compressed file with the source of Witchaven II even included a unreleased patch for the offical release as a installable EXE that patched the binaries etc. Fixed a annoying bug with the strafing buttons not working. I guess the code also includes the changes of this patch.

BTW. How is the source of Shadow Warrior in comparison with the latest binary?
0

User is offline   NY00123 

  • 102

#19

View PostLittle Tijn, on 15 May 2018 - 10:46 AM, said:

I saw your name in the changelog of Eduke32 related to this. I must say, nice done!

This project has been proved to be useful already... ;)


Thanks!
As previously stated by me, it was used for fixing NAM & WW2GI behaviors EDuke32. Guess it's the first time such a thing has occurred, at least with regards to the Duke code.

View PostLittle Tijn, on 19 May 2018 - 11:27 PM, said:

I would have expected that the released sources of games should be the latest code available, seeing that this code is used before to build the latest version. They had this code, to build the game. So why isn't it used, but instead some in-between version? Isn't it easier to release the latest code available? That are questions that I got from this thread.


It's not necessarily as simple as may initially seem. The sources were released on April 1st of 2003, about 6.5 years after Duke3D v1.5 was out.
So while what you've just stated makes sense, chances are the people who were in the offices by that time weren't entirely sure how were the files organized. Maybe some of the files moved between different locations and/or got lost for good. At the least, the binary libraries' OBJ and LIB files were clearly the ones used in v1.5 (and NAM+WW2GI+EDuke).

Quote

I know that one of the most important fixed in the day for Duke Nukem 3D was a memory leak fixed in 1.5. Because with computer at that time with 8MB RAM etc, it would make a big different. Wondering is this is also included in the source release. Where there some other significant changes?


If you're comparing to v1.4, I don't know for sure. Don't think I saw a lot of differences in the Duke3D code between 3DR's released sources and v1.5, although there's that fixed music bug in MENUES.C.

Quote

BTW. How is the source of Shadow Warrior in comparison with the latest binary?


I don't know for sure about this. What I can tell, though, is that we're much less lucky with the SW sources, when compared to DN3D (in terms of recreating any original EXE). There are multiple reasons for this, but here are just a few examples I could spot.

Regarding Duke Nukem 3D:
- We have the EDuke 2.0 sources which more-or-less cover the v1.5 code. The MAKEFILE from the same EDuke archive also appears to have the original optimizations passed to Watcom C, for the creation of v1.5.
- We also have the correct libraries' OBJ and LIB files: Build Engine, Apogee Sound System, MACT and Total Entertainment Network (TEN).
- In the Duke3D sources prepared by Charlie Wiederhood for release by 3DR, actual changes to the code are clearly marked with "CTW" comments. There are just 22 pairs of these, and they aren't present in the EDuke sources at all.

As for Shadow Warrior:
- We have the sources released by 3D Realms and prepared by Charlie Wiederhood; No additional source archive like EDuke is available.
- When Charlie Wiederhood originally got the files, it wasn't as organized and ready for building as the Duke3D sources were. Certain files were even missing, so they were borrowed from the Duke3D sources.
- There are *way* more CTW changes; I think there are more than 100.
- We also don't have the exact binary library OBJ/LIB files, at least for certain libraries. While it might be possible to recreate the Build Engine and Apogee Sound System code, the TEN lib is closed-source and appears to have at least a few differences from the code in DUKE3D.EXE (from BUTWCD4.LIB). I'm also not sure about the closed-source MACT.
- On the contrary to Duke3D, it looks like the SW code has some more TEN-specific code integrated. Certain code pieces may even be missing. Duke3D didn't use as much, and we have for the latter the relevant code either way.
0

User is offline   Hendricks266 

  • EDuke32 Senior Developer
  • 5,943

  #20

Fortunately, one of the things I did receive from the 3DR archives were full backups of the hard drives from Jim Norwood and Frank Maddin's workstations.
1

User is offline   Little Tijn 

  • 20

#21

That's some interesting information about Duke and SW. Thanks for that!

It's a bit pity that the source code of Shadow Warrior isn't in such a great state. Although we have some nice ports for it even with the code that was given.

Any idea what those changes by Charlie Wiederhood could have been and why they where needed?

View PostNY00123, on 20 May 2018 - 07:56 AM, said:

It's not necessarily as simple as may initially seem. The sources were released on April 1st of 2003, about 6.5 years after Duke3D v1.5 was out.
So while what you've just stated makes sense, chances are the people who were in the offices by that time weren't entirely sure how were the files organized. Maybe some of the files moved between different locations and/or got lost for good. At the least, the binary libraries' OBJ and LIB files were clearly the ones used in v1.5 (and NAM+WW2GI+EDuke).


That are good reasons why sources aren't always the latest. Seeing that source code has been lost for games before already. It reminds me of stories of lost sources of games that I read on the Internet, like System Shock 2.

View PostHendricks266, on 20 May 2018 - 09:14 AM, said:

Fortunately, one of the things I did receive from the 3DR archives were full backups of the hard drives from Jim Norwood and Frank Maddin's workstations.


That's interesting! Are the sources of Bio Menace on it by any change? ;) (Just kidding). But on a serious note, it might provide some very good insight of some things, or even some missing part.

This post has been edited by Little Tijn: Yesterday, 11:48 AM

0

User is offline   NY00123 

  • 102

#22

Guess what? (Not exactly my find, btw.)

Turns out the "extras" directory from the Duke3D sources, as released by 3DR, includes the code for v1.5!
I suppose that I've suspected this beforehand. It's just a bit weird this hasn't been confirmed until now. Then again, maybe this isn't very surprising.
- The libraries' OBJ and LIB files are missing, and should be brought from the "source" directory of 3DR's sources (or from the EDuke sources).
- BUILD.H is also missing. You shall copy the one from the EDuke sources, *not* from 3DR's Duke3D sources.
- MAKEFILE is more-or-less the same as in the EDuke sources, while MAKEFILE.LNK is based on the one from my repository.

As a side-note, there's a bit of concerning news regarding BUILD.H:
When I attempt to use BUILD.H from 3DR's sources (rather than the EDuke sources),
the output EXE's layout becomes somewhat different, even if not by much. Maybe certain global variables are positioned differently.
The only real difference in BUILD.H code is the addition of "extern" declarations of unused variables ("validmode*" vars),
as well as minor changes in comments (the addition of support for masked/translucent ceilings/floors via ceilingstat/floorstat bits).

View PostLittle Tijn, on 20 May 2018 - 11:17 AM, said:

Any idea what those changes by Charlie Wiederhood could have been and why they where needed?


It's probably the best to see the CTW comments in the code for yourself; In some cases, there's an explanation, but when there's not, chances are it's related to TEN.
Also, without some of his changes, it wouldn't even be possible to build an EXE at all. There were also a few obstacles on the way (like some issue with CD Audio playback).

View PostHendricks266, on 20 May 2018 - 09:14 AM, said:

Fortunately, one of the things I did receive from the 3DR archives were full backups of the hard drives from Jim Norwood and Frank Maddin's workstations.

View PostLittle Tijn, on 20 May 2018 - 11:17 AM, said:

That's interesting! Are the sources of Bio Menace on it by any change? ;) (Just kidding). But on a serious note, it might provide some very good insight of some things, or even some missing part.


I should probably note here that I've referred only to sources which were already made available to the public in various forms.
0

Share this topic:


Page 1 of 1
  • You cannot start a new topic
  • You cannot reply to this topic


All copyrights and trademarks are property of their respective owners. Instead of reading this text, you could be playing Ion Maiden! ;) © 2018 Voidpoint, LLC

Enter your sign in name and password


Sign in options