[NY00123]

Update (Feb 29 2020): The repository has been converted to git, split to submodules and moved to a new location: https://bitbucket.or...ver-recreation/

The rest of this post remains unchanged as of writing this.

Original post contents:

Hey there,

I think it's a good (albeit belated) time to introduce you to the following, at least if you aren't yet aware of this: https://bitbucket.or...ver-recreation/

To summarize, the above repository covers modifications of open source games, with attempts to recreate original old EXEs byte-by-byte. Sometimes it's more successful, while other times it's less. A few examples:
- For most Wolfenstein 3D and Spear of Destiny EXEs, with the right tools, you may now faithfully recreate the original EXEs from the 90s, byte-by-byte. There may be this or that EXE for which only the very first few bytes (in the EXE header) differ.
- For Rise of the Triad, I had less luck, and there are also technical limitations due to the usage of Watcom 10.0b (from the 10.0(x) series), but I believe it's *very* close.

Now, let's get to the following. Note that I consider this code recreation *partial*, hence you'll find it in a "bonus" subdirectory. In particular:
- This covers game code only.
- This does *not* cover engine code, audio library code* or closed-source libraries. Original binary OBJ and LIB files are used for these.

Basically, I'm talking about the game code (and no other code piece) of Duke Nukem 3D!

These EXEs are now covered in the "duke3d" directory (bonus/duke3d) within my repository:
- Duke Nukem 3D: Atomic Edition v1.5, up to differences described below (possibly hidden under "Spoiler" tags).
- The following EXEs originally built by Matt (recreated byte-by-byte): NAM/NAPALM Full Version 1.0, WW2GI Full Version 1.0 and EDuke 2.00.23 (the original binary EDuke release from the year 2000 mostly known as "EDuke 2.0").

Note that for any comparison you might be making, there should *not* be any (originally) unofficial patch applied, like the removal of copy protection. Digital releases (e.g., Steam and GOG.COM) may be impacted, too.

For Duke3D v1.5, use Watcom C 10.0 and *not* any other version; Not even 10.0a or 10.0b.
For any of Matt's EXEs, use Watcom C 10.6a (and no other version).

For a little background behind the beginning of all of this, I'll quote myself (with any possible minor diff) from the RGB Classic Games forums (https://www.classicd....php?f=3&t=1410):

Spoiler

Let's begin with telling how did I get to all of that.

Back on September of 2014, following a campaign to acquire the rights to publish the "lost" episode of Commander Keen known as "Keen Dreams", original sources for this episode were made available. These include sources not just for one, but apparently all versions of Keen Dreams originally made in the 90s for DOS, with the possible exception of 1.93 (which isn't that different from 1.92 anyway and could also be reconstructed).

It looked to me like the sources were in a relatively good state, and that the released forms were not greatly different from the originals. It's also stated that Borland C++ 2.0 was originally used to build the EXEs, and it also looks clear that a lot of the EXEs were originally packed using LZEXE 0.91.

Therefore, I decided to have a little attempt, using Borland C++ 2.0 and LZEXE 0.91 to build and pack version 1.13 (Shareware release). Turns out, I got the exact same EXE as created in 1992, byte-by-byte. I also got such results with at least a few other versions.

This looks like the main motivation behind starting the "gamesrc-ver-recreation" repository above (although originally under a different location). I think that initially, I wanted to try and recreate the Catacomb Abyss v1.13 EXE. Eventually, this followed with the Catacomb 3-D v1.0 EXE and the Catacomb Abyss v1.24 EXE. To compare, the Catacombs sources as released on June of 2014 can be used to build the Catacomb 3-D v1.22 EXE, as well as an EXE *very* close to Catacomb Abyss v1.24. However, for some reason, in order to get exactly the original v1.24 EXE, STARTTILE8 has to be edited in GFXE_ABS.EQU, but not in GFXE_ABS.H (there's some function in ID_VW_AE.ASM which shouldn't be called but is compiled into the EXE, for which this has an effect). The exact same edit is also required for recreation of the v1.13 EXE.

Some more details regarding Duke Nukem 3D, quoting from a recent RGB forums post with a few changes (https://www.classicd...hp?p=8406#p8406):

Spoiler

Now, here's the thing. The Duke Nukem 3D sources as released by 3D Realms on 2003 are of a revision somewhere in-between versions 1.4 and 1.5. However, Hendricks266 gave me the (very great) hint of looking into the Enhanced Duke (i.e., EDuke) 2.0 sources, released by Matt Saettler later that same year. The assumption was that they should have all of the 1.5 code with very few differences, provided that macros like NAM, WW2 and EDUKE are *not* defined.

As it turned out, he was right. In fact, it's not just about the code, but also the make file. Matt's MAKEFILE as originally released must be much closer to what was originally used for v1.5 than MAKEFILE from 3DR's sources. This covers the optimizations toggled for the Watcom C compiler, as well as the EXE originally being named GAME.EXE. The latter applies for Duke3D v1.5 as well as the EXEs Matt made using the code.

Matt's sources are of EDuke 2.0 build 21 i.e., EDuke 2.00.21, though, while the original EXE is 2.00.23. I used the EDuke 2.1 sources (one of the archives) as a reference for the changes.

Additionally, I still had to copy the MACT library's headers from 3DR's sources, as these were missing from Matt's original release of the EDuke 2.00.21 sources. (I know there was a second release, but I eventually started from the first.)
I further had to copy MAKEFILE.LNK from there and adapt it. Figuring out that the EXE was originally named GAME.EXE, rather than DUKE3D.EXE or EDUKE.EXE (or anything else) was one thing, but I were also left with minor diffs of 3 bytes from the original EXEs of Matt. Eventually (thanks Watcom), adding "segment type code lo" and "segment type data lo" to MAKEFILE.LNK turned out to be the solution.
Another thing that was figured out earlier, is the usage of "file" instead of "library" for the LIB files in the LNK file. This leaded to some changes, like a different (original) order of AUDIO_WF.LIB symbols, as well as a forced presence of BUTWCD4.LIB in Matt's EXEs, even though it's unused.

As stated above, Matt's EXEs can be recreated byte-by-byte (with Watcom C 10.6a).

Regarding Duke3D v1.5, well, there are a few differences. I've been a bit stuck with this for a while, initially assuming that a version of Watcom in-between 10.0b (as used for ROTT v1.3 and the Apogee Sound System v1.04) and 10.6a (Matt's EXEs) was originally used for the game code. Sadly, though, it didn't seem like Watcom 10.5 assisted here. Not being entirely sure which wcc386 parameters were originally used, and/or if I'd missed anything in the code, didn't really feel helpful.

Surprisingly, though, it turned out the README file from 3DR's release of the code was right. Watcom C 10.0 was used. Yes, this is an EARLIER version than what was reportedly used for ROTT. As for the optimizations, they're the same as in Matt's MAKEFILE.

Note that v1.5 won't be *exactly* the same as the original. There are two reasons for this:
- The original EXE has a copy of DOS/4GW Professional. The tools can be used in a similar manner for the newly recreated EXE, but there are good chances you'll still get (another) minor difference from the original, say of 6 bytes.
- Additionally, as previously occurring with ROTT and the Apogee Sound System, some unused data gaps in the binary EXE/LE data section/segment (mostly between C strings) differ from the originals. The exact contents seem to depend on the environment in which the compiler was ran, or at least on the *exact* contents of the compiled source files. I suspect this is related to the v10.0(x) series of Watcom C, since this was not reproduced with Matt's EXEs (built with 10.6a).

Further note that some binary object and library files were used. These were made available with the sources as released by 3DR and Matt altogether. I also have *guessed* versions of Watcom originally used for some objects, based on modification dates of Watcom headers. Here's a list:
- A.OBJ, CACHE1D.OBJ, ENGINE.OBJ, MMULTI.OBJ: The Build Engine, probably built with some variant of Watcom C 10.5.
- AUDIO_WF.LIB: Apogee Sound System v1.1. I'm more-or-less convinced it was built with Watcom C 10.0b.
- MACT386.LIB: The MACT lib., with another mention of 10.5.
- BUTWCD4.LIB: The Total Entertainment Network (i.e., TEN) lib., also presumably built with 10.5.

* In fact, this isn't entirely accurate. I've modified the Apogee Sound System sources so you can more-or-less recreate v1.1. However, you'll still prefer to use the existing AUDIO_WF.LIB file, if only due to unused gaps between C string literals and more being filled with contents differing from the original. As in the case of the game code, this might be related to the Watcom C v10.0(x) series.

[MrFlibble]

Probably not the focus of your endeavour, but is it possible to create a shareware v1.5 version of Duke3D using your findings?

[NY00123]

I should probably begin with a "tl;dr" summary.

So, with the modified Duke3D codebase of mine, you may now create a Duke3D EXE that (with very few exceptions) behaves *exactly* like v1.5.
Among other things, this means:
- Identical file sizes (up to the presence/lack of the DOS4GW extender).
- Identical binary EXE file contents (up to minor diffs that don't matter + DOS4GW).
- Compatibility with demos recorded with v1.5 (at least the original 3 demos seem to be OK).
- Compatibility with v1.5 saved games (although surprisingly enough, these may work in v1.4 after a simple hex-edit of BYTEVERSION in the SAV files; Your mileage may vary, though).

3 more EXEs may further be recreated, and even perfectly. (NAM v1.0, WW2GI v1.0 and Enhanced Duke (EDuke) v2.00.23.)

MrFlibble, on 12 May 2018 - 04:47 AM, said:

Probably not the focus of your endeavour, but is it possible to create a shareware v1.5 version of Duke3D using your findings?

I suppose that a "new" shareware EXE can be made. Out of curiously, I've even tried to do this on my own.
For reference, see the attached source code patch and EXE (which requires DOS4GW or compatible).

A few notes about my "shareware v1.5" code changes:
- VOLUMEALL and PLUTOPAK shall *not* be defined.
- On the other hand, VOLUMEONE should be. A definition of NOCOPYPROTECT is probably desired, too.
- There are a few instances in which the Atomic Edition logo (PLUTOPAKSPRITE and other few sprites) is shown. While there's code to exclude this, only the UK macro is checked for some reason, not PLUTOPAK. So, better fix these as well.

As for the data, I've used shareware v1.3d's. However, it looks like the new EXE is incompatible with v1.3d's CON, so I've used v1.5.

There may still be this and that issue. I've got this message of a missing sound file in-game. This might be related to at least one of the CON files.

 duke3d_v15_recreated_shareware_patch_20180512.7z (395.11K)
Number of downloads: 6

[MrFlibble]

NY00123, on 12 May 2018 - 06:04 AM, said:

As for the data, I've used shareware v1.3d's. However, it looks like the new EXE is incompatible with v1.3d's CON, so I've used v1.5.

The GRP from the Macintosh demo of Duke3D includes the missing Atomic logos for the title screen & menu, and the CONs might be v1.4/1.5 compatible although I haven't checked this yet. I have always thought that the Mac demo GRP could actually be a stripped-down version of the post 1.3d shareware that never went public.

[NY00123]

MrFlibble, on 12 May 2018 - 12:39 PM, said:

The GRP from the Macintosh demo of Duke3D includes the missing Atomic logos for the title screen & menu, and the CONs might be v1.4/1.5 compatible although I haven't checked this yet. I have always thought that the Mac demo GRP could actually be a stripped-down version of the post 1.3d shareware that never went public.

This is a good idea. I've checked not just duke68kdemo.bin, but also duke3d.sit.hqx. I've used "The Unarchiver" (unar) for unpacking these.
These two demo archives seem to have the same CON and Group files. The CON files are exactly the same as in DOS v1.5, up to the differing new line formatting (CR for the old Macs, CR+LF for DOS).

While this seems to more-or-less work, there are still a few differences to mention. For instance:
- The Mac demo(s) lack MIDI files.
- There are less ART graphics when compared to DOS version. The missing graphics include the Atomic logos, too.

There are chances the Atomic logos are stored in a separate file. This may also apply to the graphics used for misc. Mac-exclusive cheat codes, like DN1984.

To conclude, though, then yeah, the Mac demo art is clearly based on DOS v1.5's, just with less of it.
As for comparison to shareware version 1.3d, all I'll say for now is the following:
- Shareware v1.3d and the Mac demo both have TILES000.ART-TILES012.ART and no more.
- The same non-shareware tiles were stripped from both versions i.e., there's no tile present in one of them and not in the other.
- Unsurprisingly, though, there are tens of tiles differing. Note that a few tiles didn't change between *registered* DOS versions 1.3d and 1.5, while there are earlier revisions in *shareware* version 1.3d.

[TerminX]

NY00123, on 12 May 2018 - 02:36 PM, said:

Note that a few tiles didn't change between *registered* DOS versions 1.3d and 1.5, while there are earlier revisions in *shareware* version 1.3d.

That's correct. The art was not updated between shareware releases, so shareware 1.3d still contains things like the angled rocket sprites from 1.0, a switch sprite that exists in LameDuke, etc.

[0815Jack]

sorry for asking this dumb question, but what is the point of recreating the exe byte-by-byte ?

[Mark]

I wondered the same thing. I kept quiet thinking I was the only dumb one who doesn't know.

[NY00123]

This is actually a very good question! I'll give here two ways in which recreating original code may assist:

- Heard of the "Chocolate Doom" source port? It's the one that aims to accurately reproduce the experience of playing the original Doom games from the 90s using any of the original DOS EXEs, at least as of version 1.9.
Among other things, this includes the general feeling, as well as compatibility with original saved games, demo files and maps (while *not* supporting contents which is *not* compatible with any of the original DOS EXEs).

Thus, recreating an EXE byte-by-byte, even if just partially, can *very greatly* assist with such an endeavour.

Note that in the case of Chocolate Doom, they managed to do so without this so far - The Linux Doom sources were used, and there are some different behaviors from 1.9, so more work had to be done.
There may still be certain technically differing behaviors as a consequence.

- Secondly, and this mostly applies to code in my repository for other games, like Wolfenstein 3D and Blake Stone (sorry, not ETA for Duke Nukem 3D or any other game):
If anybody has ever been curious to know how do two versions of a game differ, not just in data but also in the code - this is more-or-less what recreated code allows you.
Note that there are a few possible pitfalls:
* You may assume that the various #if/#ifdef/#ifndef statements I've added are sufficient for locating *all* technical differences, but it isn't exactly accurate. In fact...
* Even if you change the compilation optimizations for just ONE compilation unit (e.g., a C source file), good chances are the EXE will behave somewhat different.
This can be anything, from speed-sensitive diffs that you won't notice, to undefined/unspecified behaviors that you *will* notice (even if not rarely).
* Finally, even if you change *just* the code, the compiler may add other technical differences you haven't caught (again related to undefined/unspecified behaviors).

[Mark]

I'm not familiar with Doom ports. But maybe my question has been answered. My uneducated guess was that maybe this new exe file might be an alternative to having to use Dosbox in windows to run games as they were originally meant to run in dos. In other words, playing a dos game without having to mount drives and configure this and that. But it doesn't look like it.

[oasiz]

Nah, this is more about accurately reconstructing the original DOS exe that you got back in the days.
Source releases that we usually get are not 1:1 of what they had when they pressed "compile". Like in D3D case it was said above that it's somewhere before 1.5.
Often they remove bits and pieces that they don't want to share like audio libraries, etc..

Cool aspect about this is actually getting the original files working and in this case we can finally re-create stuff like the shareware 1.5.
Sure it's a bit of a novelty for most, but it's really useful for comparison purposes and preservation.

Not to mention any source code studies and "chocolate" ports.

[0815Jack]

thx for the detailed explanation

[NY00123]

oasiz is surely spot-on with his explanation, thanks!

Mark., on 13 May 2018 - 12:29 PM, said:

My uneducated guess was that maybe this new exe file might be an alternative to having to use Dosbox in windows to run games as they were originally meant to run in dos. In other words, playing a dos game without having to mount drives and configure this and that.

The modified codebase for DOS may still be a basis for some individual eventually adapting it to modern platforms, thus letting you do exactly that! (Basically the "Chocolate Doom" idea.)

[Mark]

Thanks for explaining. I couldn't think of anything else other than it being a "nerd experiment". And I don't mean that in a bad way.

[NY00123]

In fact, how could have I forgotten this!

The results of my work to recreate the NAM and WW2GI EXEs (EDUKE.EXE had been the first) were used to fix a few inaccuracies with the NAM and WW2GI support in EDuke32.

[leilei]

Different lineage, there's also some functionality loss in the source releases for Quake and Quake III Arena that existed in prior versions that are hard to recreate and have been omitted from the source release:

Quake:
- Menu fade tint (easily recreateable with the dead gelmap code scrap)
- Sound attenuation calculation is different
- DJGPP makefile is missing from source release

Quake III Arena:
- Dynamic lights updating the lightmap (Q3Test 1.01 to 1.07)
- Null sound effect sawtooth buzz (Q3Test 1.01 to 1.08)
- Fast 8-bit sample mixing (Q3Test 1.01 to Q3 1.27g - s_loadas8bit and s_compression)
- Earlier protocols/traps before the big mod regression (1.11 to 1.17)

Just mentioning it here because the q scene has 0 interest in anything that's not fucking promode aerowalk / brutal phase / bad hd packs. I don't really see interest in anything historical elsewhere.

[Little Tijn]

NY00123, on 14 May 2018 - 10:57 AM, said:

In fact, how could have I forgotten this!

The results of my work to recreate the NAM and WW2GI EXEs (EDUKE.EXE had been the first) were used to fix a few inaccuracies with the NAM and WW2GI support in EDuke32.

I saw your name in the changelog of Eduke32 related to this. I must say, nice done!

This project has been proved to be useful already...

There still is a problem with the automatic weapons in WW2GI in Eduke32. When quickly pressing the fire button, the firing animation is played, although no shot is fired. If you have time and are willing to check why this is, that will be great!

Details are here:
https://forums.duke4...post__p__300746

[Little Tijn]

Oops, hijacked this thread a bit with my request above, sorry about that.

More on-topic:
I would have expected that the released sources of games should be the latest code available, seeing that this code is used before to build the latest version. They had this code, to build the game. So why isn't it used, but instead some in-between version? Isn't it easier to release the latest code available? That are questions that I got from this thread.

I know that one of the most important fixed in the day for Duke Nukem 3D was a memory leak fixed in 1.5. Because with computer at that time with 8MB RAM etc, it would make a big different. Wondering is this is also included in the source release. Where there some other significant changes?

Speaking of released versions, we have some cases where the code to a later version that the releases binary is available. Off the top of my head, the code of Eduke 2.1 and the released code in dubious legal state of Witchaven II. If I recall correctly, the compressed file with the source of Witchaven II even included a unreleased patch for the offical release as a installable EXE that patched the binaries etc. Fixed a annoying bug with the strafing buttons not working. I guess the code also includes the changes of this patch.

BTW. How is the source of Shadow Warrior in comparison with the latest binary?

[NY00123]

Little Tijn, on 15 May 2018 - 10:46 AM, said:

I saw your name in the changelog of Eduke32 related to this. I must say, nice done!

This project has been proved to be useful already...

Thanks!
As previously stated by me, it was used for fixing NAM & WW2GI behaviors EDuke32. Guess it's the first time such a thing has occurred, at least with regards to the Duke code.

Little Tijn, on 19 May 2018 - 11:27 PM, said:

I would have expected that the released sources of games should be the latest code available, seeing that this code is used before to build the latest version. They had this code, to build the game. So why isn't it used, but instead some in-between version? Isn't it easier to release the latest code available? That are questions that I got from this thread.

It's not necessarily as simple as may initially seem. The sources were released on April 1st of 2003, about 6.5 years after Duke3D v1.5 was out.
So while what you've just stated makes sense, chances are the people who were in the offices by that time weren't entirely sure how were the files organized. Maybe some of the files moved between different locations and/or got lost for good. At the least, the binary libraries' OBJ and LIB files were clearly the ones used in v1.5 (and NAM+WW2GI+EDuke).

Quote

I know that one of the most important fixed in the day for Duke Nukem 3D was a memory leak fixed in 1.5. Because with computer at that time with 8MB RAM etc, it would make a big different. Wondering is this is also included in the source release. Where there some other significant changes?

If you're comparing to v1.4, I don't know for sure. Don't think I saw a lot of differences in the Duke3D code between 3DR's released sources and v1.5, although there's that fixed music bug in MENUES.C.

Quote

BTW. How is the source of Shadow Warrior in comparison with the latest binary?

I don't know for sure about this. What I can tell, though, is that we're much less lucky with the SW sources, when compared to DN3D (in terms of recreating any original EXE). There are multiple reasons for this, but here are just a few examples I could spot.

Regarding Duke Nukem 3D:
- We have the EDuke 2.0 sources which more-or-less cover the v1.5 code. The MAKEFILE from the same EDuke archive also appears to have the original optimizations passed to Watcom C, for the creation of v1.5.
- We also have the correct libraries' OBJ and LIB files: Build Engine, Apogee Sound System, MACT and Total Entertainment Network (TEN).
- In the Duke3D sources prepared by Charlie Wiederhold for release by 3DR, actual changes to the code are clearly marked with "CTW" comments. There are just 22 pairs of these, and they aren't present in the EDuke sources at all.

As for Shadow Warrior:
- We have the sources released by 3D Realms and prepared by Charlie Wiederhold; No additional source archive like EDuke is available.
- When Charlie Wiederhold originally got the files, it wasn't as organized and ready for building as the Duke3D sources were. Certain files were even missing, so they were borrowed from the Duke3D sources.
- There are *way* more CTW changes; I think there are more than 100.
- We also don't have the exact binary library OBJ/LIB files, at least for certain libraries. While it might be possible to recreate the Build Engine and Apogee Sound System code, the TEN lib is closed-source and appears to have at least a few differences from the code in DUKE3D.EXE (from BUTWCD4.LIB). I'm also not sure about the closed-source MACT.
- On the contrary to Duke3D, it looks like the SW code has some more TEN-specific code integrated. Certain code pieces may even be missing. Duke3D didn't use as much, and we have for the latter the relevant code either way.

[Hendricks266]

Fortunately, one of the things I did receive from the 3DR archives were full backups of the hard drives from Jim Norwood and Frank Maddin's workstations.

[Little Tijn]

That's some interesting information about Duke and SW. Thanks for that!

It's a bit pity that the source code of Shadow Warrior isn't in such a great state. Although we have some nice ports for it even with the code that was given.

Any idea what those changes by Charlie Wiederhood could have been and why they where needed?

NY00123, on 20 May 2018 - 07:56 AM, said:

It's not necessarily as simple as may initially seem. The sources were released on April 1st of 2003, about 6.5 years after Duke3D v1.5 was out.
So while what you've just stated makes sense, chances are the people who were in the offices by that time weren't entirely sure how were the files organized. Maybe some of the files moved between different locations and/or got lost for good. At the least, the binary libraries' OBJ and LIB files were clearly the ones used in v1.5 (and NAM+WW2GI+EDuke).

That are good reasons why sources aren't always the latest. Seeing that source code has been lost for games before already. It reminds me of stories of lost sources of games that I read on the Internet, like System Shock 2.

Hendricks266, on 20 May 2018 - 09:14 AM, said:

Fortunately, one of the things I did receive from the 3DR archives were full backups of the hard drives from Jim Norwood and Frank Maddin's workstations.

That's interesting! Are the sources of Bio Menace on it by any change? (Just kidding). But on a serious note, it might provide some very good insight of some things, or even some missing part.

[NY00123]

Guess what? (Thanks to NukeYKT for the hint!)

Turns out the "extras" directory from the Duke3D sources, as released by 3DR, includes the code for v1.5!
I suppose that I've suspected this beforehand. It's just a bit weird this hasn't been confirmed until now. Then again, maybe this isn't very surprising.
- The libraries' OBJ and LIB files are missing, and should be brought from the "source" directory of 3DR's sources (or from the EDuke sources).
- BUILD.H is also missing. You shall copy the one from the EDuke sources, *not* from 3DR's Duke3D sources.
- MAKEFILE is more-or-less the same as in the EDuke sources, while MAKEFILE.LNK is based on the one from my repository.

As a side-note, there's a bit of concerning news regarding BUILD.H:
When I attempt to use BUILD.H from 3DR's sources (rather than the EDuke sources),
the output EXE's layout becomes somewhat different, even if not by much. Maybe certain global variables are positioned differently.
The only real difference in BUILD.H code is the addition of "extern" declarations of unused variables ("validmode*" vars),
as well as minor changes in comments (the addition of support for masked/translucent ceilings/floors via ceilingstat/floorstat bits).

Little Tijn, on 20 May 2018 - 11:17 AM, said:

Any idea what those changes by Charlie Wiederhood could have been and why they where needed?

It's probably the best to see the CTW comments in the code for yourself; In some cases, there's an explanation, but when there's not, chances are it's related to TEN.
Also, without some of his changes, it wouldn't even be possible to build an EXE at all. There were also a few obstacles on the way (like some issue with CD Audio playback).

Hendricks266, on 20 May 2018 - 09:14 AM, said:

Fortunately, one of the things I did receive from the 3DR archives were full backups of the hard drives from Jim Norwood and Frank Maddin's workstations.

Little Tijn, on 20 May 2018 - 11:17 AM, said:

That's interesting! Are the sources of Bio Menace on it by any change? (Just kidding). But on a serious note, it might provide some very good insight of some things, or even some missing part.

I should probably note here that I've referred only to sources which were already made available to the public in various forms.

[MrFlibble]

NY00123, on 12 May 2018 - 02:36 PM, said:

While this seems to more-or-less work, there are still a few differences to mention. For instance:
- The Mac demo(s) lack MIDI files.
- There are less ART graphics when compared to DOS version. The missing graphics include the Atomic logos, too.

There are chances the Atomic logos are stored in a separate file. This may also apply to the graphics used for misc. Mac-exclusive cheat codes, like DN1984.

Uhh, concerning the Atomic logo, I remembered that wrong, sorry. There's no Atomic logo in the Mac demo. I ran it through Executor and took some screenshots, here's the menu:

As for MIDIs, they're useless on Mac so were cut out, I guess if there is any music (Executor doesn't handle it so I can't tell) it's probably stored in the data part of the binary or whatever that's called on Macs.

NY00123, on 12 May 2018 - 02:36 PM, said:

To conclude, though, then yeah, the Mac demo art is clearly based on DOS v1.5's, just with less of it.

Thanks for confirming that.

NY00123, on 12 May 2018 - 02:36 PM, said:

- The same non-shareware tiles were stripped from both versions i.e., there's no tile present in one of them and not in the other.

That's what made me think that this GRP for the demo was based on a v1.5 shareware release with further cuts of MIDI files and levels beyond E1L3.

NY00123, on 12 May 2018 - 02:36 PM, said:

- Unsurprisingly, though, there are tens of tiles differing. Note that a few tiles didn't change between *registered* DOS versions 1.3d and 1.5, while there are earlier revisions in *shareware* version 1.3d.

Not counting the Atomic Edition menu font, I only remember the yellow switch w/ red/greed lights which looks more like the LameDuke version in shareware

TerminX, on 12 May 2018 - 03:36 PM, said:

a switch sprite that exists in LameDuke, etc.

IIRC it's slightly different although the design is clearly more LameDuke-ish than the registered version.

[PokeThePower]

Little Tijn, on 20 May 2018 - 11:17 AM, said:

Any idea what those changes by Charlie Wiederhood could have been and why they where needed?

The vast majority are split into three categories:

• Moving initializations around due to mismatches between SW and Duke 3D assumptions.
  
• Removing TEN dependencies/references.
  
• Updating to current (at the time) compiler expectations in terms of variable definitions.

And if I recall correctly one item... just because.

[Little Tijn]

JumpJuneJump, on 02 June 2018 - 07:20 PM, said:

The vast majority are split into three categories:

• Moving initializations around due to mismatches between SW and Duke 3D assumptions.
  
• Removing TEN dependencies/references.
  
• Updating to current (at the time) compiler expectations in terms of variable definitions.

And if I recall correctly one item... just because.

Thanks for the interesting information!

Now that the code has been recreated code from 1.2, we can know for use what the changes where.

With the release of NBlood and BloodGDX I started to check my original version of Blood 1.00 and reading the patch details with the patches that came after it.
I think it would be very interesting to restore (or reverse-engineer) the code of those version as well. Especially considering the changes (nerfing) to the weapons with the 1.10 patch.

From 1.00 till 1.02 the life leech replenish your health with use and the voodoo-doll alt fire was way more powerful. Also in the patch notes there is a reference that Tchernobog is not so weak anymore and Cerberus could be killed a bit too easy with the Tesla cannon. The bobbing of the screen is much more visible on 1.00 and 1.01. From 1.02 it is heavily reduced especially when crouching.

Maybe it is a nice challenge to try to recreate (or reverse-engineer) the code for these version as well. It would be very interesting to see what has been been changed exactly.

Also, it would be nice to have the original working of the Life leech and the voodoo-doll as a option in the Blood ports. Playing the game without the nerfed weapons is quite a bit of fun. Although I can understand the nerfing. The life leech is a bit unbalanced, not to speak of the voodoo-doll.

[NY00123]

I think that it's a good time for a new post

Turning out to be quite lengthy (of course), I've decided to hide most of the details under spoiler tags, and keep a short/medium summary on the top.

I should probably begin by saying that the bonus/duke3d directory was collapsed into duke3d. You should see why soon. A few build scripts were also changed again.

Before getting to actual updates in the gamesrc-ver-recreation repository itself, I'll first link to the following threads, in case you aren't aware of at least one of the covered topics:

https://forums.duke4...de-re-released/ - Thanks to Devolver Digital, last February had a re-release of Shadow Warrior sources, this time confirmed to match versions 1.0, 1.1 and 1.2 in behaviors (some work will still be required for saved games compatibility); Additionally, thanks to Ken Silverman, the sources for the ENGINE.C/ENGINE.OBJ file as used in SW 1.1 and 1.2 are also covered, along with a renamed CACHE1D.C file from 1995 named CACHE1D.TXT.

https://forums.duke4...es-and-sources/ - Again thanks to Ken, this covers historical Build Engine revisions, mostly in the forms of object and header files. There's also a 95 revision of ENGINE.C which covers the abandoned groudraw feature.

It is now the time to list some additions to gamesrc-ver-recreation. As usual, I cannot guarantee that I'll do more than what's listed here in the future.

A summary is following:

- The repository has a new "kenbuild" directory. Among other things (a portion of which to be mentioned soon), you can use it to recreate the version of Ken-Build's GAME.EXE as originally released by Ken on June 2000, using Watcom C 11.0. Note that this should *not* be confused with minor edits done by him on Nov 2002.
- Using the ENGINE.C code which is matching SW 1.1/1.2, SW 1.0's revision could be recreated. Turns out it differs from 1.1/1.2 just by a few constants in drawmapview, clippoly and fillpolygon (unless I've missed something).
- Nuke.YKT came in and used the above SW 1.1-1.2 revision of ENGINE.C in order to recreate Duke3D 1.5's. In fact, originally he wanted to do something similar for Redneck Rampage, but eventually it turned out to pave the way for the Duke.
- A bit later, I came in and took the Ken-Build 2000 revision of A.ASM, in order to recreate the one common to Duke3D 1.5 and SW 1.0-1.2. Unlike the 2000 revision which should better be built with WASM, the older revision requires MASM 5.10.
- As for CACHE1D.C, both Nuke.YKT and me confirmed that the Duke3D 1.5 revision isn't that different from the 2000 one. The former is also the same revision as in SW 1.0-1.2.
- No change was required for MMULTI.C: It's the same as in the Ken-Build release, and also in a source archive of Ken's 2DRAW engine.
- Later, I got to take care of Duke Nukem 3D: Atomic Edition 1.4. This covers the game code and Build Engine code altogether, with a few bits of very important help from Nuke.YKT. I think that most of the changes were in the Build Engine. 1.4 and 1.5 don't differ as much, and the GPLed release's "source" dir is closer to 1.4 than 1.5.
- Finally, following the Shadow Warrior sources re-release, I faithfully recreated almost all of the Apogee Sound System revision used in SW 1.0-1.2 i.e., version 1.09 of AUDIO_WF.LIB. Only actual code difference after compiling, is that the recreated FX_SetupCard code has a bit different register read, although the behaviors should be equivalent.

More details for the interested:

Spoiler

- As stated above, the repository has a new "kenbuild" directory. Initially, it had the open-sourced Ken-Build release, as uploaded by Ken on June 2000 and later edited on Nov 2002 (just minor edits). I confirmed that the June 2000 GAME.EXE file can be recreated with Watcom C 11.0. I'm not sure about the Nov 2002 revision, but I *guess* that it comes to a Watcom C 11.0c beta, or alternatively another early Open Watcom revision preceding 1.0.
- The SW 1.1/1.2 revision of ENGINE.C should be covered as well. Evidence shows that their OBJ files were originally made with Watcom 10.6, rather than 10.5 as used for the rest of the non-ASM objects; For reference, all objects other than ENGINE.OBJ are the same as in Duke3D 1.5. A.OBJ and MMULTI.OBJ are further the same as in Duke3D 1.4.
- The SW 1.0 revision of ENGINE.C is also in. Turns out it differs from 1.1/1.2 just by a few constants in drawmapview, clippoly and fillpolygon (unless I've missed something).

* The following is the very first significant contribution to the repository from Nuke.YKT:
- According to him, initially, he tried to reconstruct Redneck Rampage's revision of the engine, just by comparing function sizes. He then compared RR's engine to Duke3D 1.5's and figured out that the sizes matched. In fact, the only remaining differences were things that he missed. Afterwards, the Duke3D 1.5 / RR revision of the engine was perfectly reconstructed by him.
- As expected, most of the work was on ENGINE.OBJ. Both of us confirmed that 1.5's CACHE1D.OBJ, also used in SW 1.0-1.2, didn't differ a lot from the open-sourced release of 2000. As stated above, MMULTI.C was the same code as in Ken-Build and 2DRAW.
- I took some looks at A.OBJ. While the Ken-Build revision should be built with WASM, it turned out that the revision from Duke3D 1.5 and SW 1.0-1.2 would better be built with MASM 5.10. Ken's released A.ASM file even has hints about this. I further had a bit of important assistance from Nuke.YKT at some point.

* The next step was recreating the game and Build Engine code used in Duke Nukem 3D: Atomic Edition 1.4. On the game-side of things, this more-or-less started using the GPLed release's "source" dir, after reverting changes applied for the release (like making it compatible with the bundled Build Engine files, which turned out to match 1.5).

Regarding the Build Engine, initially an October 96 revision of ENGINE.OBJ was used. It's really close to 1.5's revision. Later, a few portions of the aforementioned 95 revision of the ENGINE.C code were used. Towards the end of the Build Engine recreation for 1.4, there were what were presumably minor issues, for which I got very important help from Nuke.YKT, mostly to do with the way global variables are ordered in ENGINE.OBJ's layout (and eventually the EXE layout).

As previously stated, Duke3D 1.4 isn't that far from 1.5 in terms of game code, while there are clearly more differences in the Build Engine. Also, again, he GPLed release's "source" dir is closer to 1.4 than 1.5.

Currently identified e.g., by grepping AV_DR_DN3DGPLSRC in the sources, you can spot differences related to sound management, as well as file handles. The most interesting, though, is that 1.4 doesn't have some SE40 code which is found in 1.5.

Wait, SE40 code? Isn't that some unused code found under that extras/se40.c file? Well, this is true, but it's essentially a bit different revision of code in version 1.5 of game.c. As figured out by Nuke.YKT later, Duke3D 1.5 does have quite partial and incomplete SE40 ROR support!

Regarding the libraries, only the Build Engine changed between 1.4 and 1.5. The Apogee Sound System, MACT and TEN remained unchanged.

Some examples of changes in Build Engine code, not just between versions 1.4 and 1.5 of Duke3D:
- Some revision preceding the public Ken-Build revision from 2000 had CACHE1D.C:MAXCACHEOBJECTS increased from 6144 to 9216. According to BUILD2.TXT, the Oct 4 97 revision had BUILD.H:MAXTILES increased in a similar manner for Xatrix. Both were presumably done for Redneck Rampage Rides Again.
- CACHE1D.C:allocache had a replacement of a call to copybuf with copybufbyte, fixed for Plutonium Pak as described in BUILD2.TXT.
- ENGINE.C:drawvox was still under development before the release of Duke3D 1.5.
- Red-blue stereo mode (this info isn't actually based on inspecting the code): Duke3D 1.4 still has the functionality at a level similar to 1.3d's, although you should choose mode 6 instead of 7, as in 1.5.

* Finally, following the Shadow Warrior sources re-release, I faithfully recreated almost all of the version of the Apogee Sound System used in SW 1.0-1.2 i.e., version 1.09 of AUDIO_WF.LIB. The recreated FX_SetupCard code still has a bit different register read after compiling, although the behaviors should be equivalent. This also applies to the mostly complete recreation of version 1.04 (used in ROTT 1.3), btw. There wasn't that much code to recreate, anyway, since 1.09 is in-between 1.04, which was recreated for most, and 1.1, which was completely recreated.

Interestingly, according to strings in the EXEs, it turns out that SW 1.0-1.2's revision of AUDIO_WF.LIB, 1.09, is actually a bit *older* than the one used in all of Duke3D 1.1+. Apparently, Duke3D 1.0 is also using 1.09, while the leaked 19960104 build is using 1.08. When it comes to SW, sw960611 and later use 1.09, while sw950115 and sw950210 use 1.00 (the /ASSVER parameter is also functional for certain older builds).

[NY00123]

Hey there,

I hope to get some feedback on the following details about a plan to migrate to git.

I don't have much to add in terms of contents, although I've recently been doing work on the BUILD editor in a separate "branch", which is actually a separate hosted copy of the repository.

As some of you should know by now, gamesrc-ver-recreation is currently a Mercurial repository, which is hosted on Bitbucket. Thing is, Mercurial support is being deprecated in Bitbucket. You can't create new Mercurial repositories over there as of this month, and existing repositories are expected to be removed by June: https://bitbucket.or...rt-in-bitbucket

Personally, I'll probably convert the repository to git, mostly since I'm way more used to git nowadays (and it generally seems to be way more supported).

Thing is, since the repository's history (including all commit hashes) will essentially be recreated, this is a chance to make one-time changes to its history.

So, it can assist if anybody has some feedback on any of the following points:

- Before creating gamesrc-ver-recreation as a Mercurial repository hosted on Bitbucket, I briefly had this GitHub-hosted git repository titled game-srccode-ver-recreation, covering just Catacomb Abyss shareware v1.13. After about a week of use, I removed the repository from GitHub and created gamesrc-ver-recreation, currently a Bitbucket-hosted Mercurial repository. At some later point, I re-uploaded the original git repository to Bitbucket, currently titled game-srccode-ver-recreation-git-legacy. So I think that I can basically kind-of merge the two, re-applying the Mercurial changesets as git commits on top of the original repository, possibly after adding a new transitional git commit with a short explanation.

- What about removing the commit/changeset renaming the wolf3d directory to w3d_plus? Originally I did this after adding recreated Super 3-D Noah's Ark DOS code to the Wolf3D/SOD tree. It still makes sense, given that it's a different game made by a different company; Not entirely sure about this as of now. Of course, if I do rename the directory, then other commits referencing w3d_plus (even if just in commit messages) should be updated accordingly.

- Alternatively, instead of changing the repository's history by removing the above commit, I can also add 1-2 new commits reverting the change.

- Rename gamesrc-ver-recreation? I know, this might seem a bit unexpected. Reason for suggesting this, is that this repository isn't exactly limited just to game code anymore; There are libraries, like the Build Engine, and I've also been working on a tool, the BUILD editor. Also, I occasionally think of this as a "reconstruction" rather than a "recreation", but the latter term is shorter and possibly considered more accurate. It's also the case that this repository is already known as "gamesrc-ver-recreation", so maybe its name should indeed remain unchanged.

- What to do with the older Mercurial repository, including Mercurial changeset IDs? Obviously, existing URLs directly linking to data in specific revisions of the repository will become broken, so I at least want to archive the changeset IDs in some way. I can re-upload the Mercurial repository's files (including hg-specific files) as a git repository, and/or upload a TXT file with a list of logs and IDs of the changesets.

Thanks again for any interest in a portion of the contents found in this repository.

[NY00123]

Hi all,

I've gotten an important technical update:
- Following Bitbucket's planned removal of Mercurial support, I've converted the repository to git.
- Additionally, I've then split it into separate submodules, and set up a Bitbucket team for all of them (thanks to Blzut3 for the suggestions).

As a consequence, gamesrc-ver-recreation has just gotten a new URL: https://bitbucket.or...ver-recreation/

We'll see if the above migration has worked well, and especially if there's a difference in the convenience of using it, when compared to working with just one repository.

Notes for anybody trying to clone any of the submodules:

Spoiler

Currently, if you want just one git repository, then you may simply clone it, but otherwise you may recursively clone the gamesrc-ver-recreation and all of its submodules. Any instance of ame code which depends on a library currently assumes this is the way it's done (which is similar to the situation before splitting into submodules). For instances, if you're in the "duke3d" dir and it needs files built in the "build" dir, then it currently assumes that they're in "../duke3d".

================

As for any added contents, there's now the addition of a tool, which is still gaming-related, but isn't exactly a game on its own. Basically, this is the Build Editor for Duke Nukem 3D.

This covers the editor as distributed with registered version 1.3d of Duke Nukem 3D, as well as the editor bundled with Duke Nukem 3D: Atomic Edition v1.4-1.5 (1.4's editor was reused for 1.5).
As a small bonus, the Build Engine objects as used in the editor for Shadow Warrior v1.2 are also covered.

================

What can I say about the code recreation process, for the two Duke3D editor versions? Well, it was quite different for each of them.

For the ones not aware, like the rest of the Build Engine, most of the Build editor code was originally offered to developers in a binary BUILD.OBJ file, with no source code.
However, they had the option to extend its functionality, with the assistance of hooks present in a file named BSTUB.C.

For Duke Nukem 3D, this file was renamed ASTUB.C. The Shadow Warrior sources have FMSTUB.C and JNSTUB.C, although it was eventually the latter which was used for the editor as distributed with SW 1.2.

So basically, there were two components to take care of: ASTUB.C, and the Build Engine code.
After the Duke3D sources were GPLed, a version of ASTUB.C was released under the same terms. I suspected that it matched v1.3d, but I first wanted to recreate 1.4/1.5's editor. As expected, there was a bunch of code to add, rather than delete (dealing with additional enemies introduced in 1.4 was just one example).

- At this point, only the ASTUB.C:Keys3d function is not 100% matching the original in terms of layout, but it should be equivalent in behaviors from what I recall.
- Fun fact: It actually turned out that certain bits of Duke3D's ASTUB.C code were later used in SW's JNSTUB.C. This clearly helped me, especially with the Keys3d function.
- In terms of Build Engine code, I initially assumed that it matched Duke3D v1.4. While A.OBJ and CACHE1D.OBJ are matching, it actually turned out that the editor uses a somewhat earlier revision of ENGINE.OBJ. I also had to update the code for BUILD.OBJ, which was expected (it was originally matching just Ken-Build as released in 2000, including a minor update from 2002).

I then went on to take care of v1.3d's editor. I'm reminding about the guess that the GPLed ASTUB.C file is matching it. As it eventually turned out, except for the version string being "DUKE NUKEM BUILD: V032696" instead of "DUKE NUKEM BUILD: V041996", as well as changes I assume that Charlie did for compatibility with Duke3D v1.5's engine for most, it was indeed perfectly matching v1.3d.

- Unlike v1.4, almost all work done for Duke3D v1.3d's editor was in the engine code, with very little done in ASTUB.C itself.
- Portions of code from a 1995 revision of ENGINE.C released on May 2019 were used (https://forums.duke4...es-and-sources/).
- Thanks again to Nuke.YKT, he already got a great deal (in fact, most?) of the required work on v1.3d's engine done. He was originally doing this for Redneck Rampage, with the initial assumption that its engine was at least earlier than Duke3D v1.4's in some way, before figuring out that it's actually matching 1.5 (also used by me in the same repository later).
- A few ENGINE.OBJ functions still don't perfectly match, but it should be quite close now, and the file mostly matches anyway (with a bit of hacks covering assembly pragmas, so a couple of functions are at least a bit closer in layout, and match in size).

Also, this time, there were instances in which I used a combination of a few files, like a script and a (variant of a) makefile. What for? Well, instead of manually guess, say, what should be the order in which the local variables of a certain function are defined, I could kinda brute-force this via scripting. A few more details:

Spoiler

It's possible to automatically build some C code for DOS (like ENGINE.C) with wcc386.exe, either the DOS or the Windows (NT) version of the latter; While doing this, wcc386 can be told to define certain C processor macros, with the exact definitions being chosen via the scripting, and changing for each invocation of wcc386. At some point, I even made these choices pseudo-random, rather than being ordered in a specific clear deterministic way.

Before this is forgotten, what about Shadow Warrior v1.2's Build Editor again? Well:
- All of the Build Engine objects are covered. In particular, A.OBJ is the same as used in Duke3D v1.4-1.5 and their editor, CACHE1D.OBJ is the same as in Duke3D v1.5, and BUILD.OBJ is currently identified as the 19961213 revision. The version found in the Ken-Build sources as released on 2000 seems to be virtually identical. It think that the couple of changes from the 10/4/97 entry of BUILD2.TXT are the only differences. This comes to increasing MAXTILES from 6144 to 9216 for Xatrix, as well as adding "#pragma push" and "#pragma pop" to BUILD.H, reason being the default struct packing was changed from 1 to 8 with the move to Watcom 11.0.
- For the SW-specific additions, see the JNSTUB.C file in the SW sources as re-released on Feb 2019 (REGCODE subdir): https://gitlab.com/N...EGCODE/JNSTUB.C

[MrFlibble]

I just had a random thought. A few days ago I learned about a project called FastDoom, a DOS source port of Doom that aims to optimise the game for maximum speed.

Without getting carried away too much, how realistic would it be to make optimisations to the restored code of later games, e.g. Hexen or Duke3D, to attain better performance on low-end machines/emulators (for example, browser implementation of DOSBox) without compromising quality in the same was as switching to low detail/decreasing screen size does? Is it at all possible?

[K1n9_Duk3]

DOSBox is a bad example in this context, since it is in no way authentic when it comes to the timing of the CPU instructions. Code that's fast on real 386 or 486 CPUs might be slower in DOSBox, while code that's fast in DOSBox might be slower on real CPUs.