[Phredreeke]

This is speculation on my part, but it wouldn't surprise me if the assembly optimisations have the opposite effect when using DOSBox's dynarec (forcing it to translate the same code over and over again as it's modified). Has anyone tried building Duke or SW without such optimisations and running in DosBox for comparison?

[MrFlibble]

Phredreeke, on 26 June 2020 - 01:17 PM, said:

it wouldn't surprise me if the assembly optimisations have the opposite effect when using DOSBox's dynarec (forcing it to translate the same code over and over again as it's modified). Has anyone tried building Duke or SW without such optimisations and running in DosBox for comparison?

How simple would it be to tweak the code to turn those off?

My experience in this field is limited to compiling MBF in DOSBox to support HACX.WAD, and that was some six years ago.

[MrFlibble]

In the meantime, I ran some tests. I'm using vanilla DOSBox 0.74-3 configured to go no higher than 11600 cycles, and normal core only. This is roughly what JS-DOS does as implemented here (I used FastDoom's built-in FPS counter to get an approximation). I ran Duke3D shareware v1.1, v1.3D, registered Atomic 1.5 (GOG.com version) and Shadow Warrior shareware v1.2. I used FRAPS for the FPS counter, and ran each game with the DOS/32A extender.

For shareware Duke3D, I tracked the FPS counter during demo playback. The demos are different but feature the same E1L2 locations and combat situations so I think that's fair.

First the expected results:

• reducing screen size ups the speed (I settled with the size that is three steps down from the default setting)
  
• switching from OPL3 to MIDI music yields an increase of some 2 frames per second
  
• reducing the number of voices, mixing rate and sampling rate does contribute to faster performance (I compared 4 vs. 8 voices, 8-bit vs. 16-bit mixing and 11kHz vs. 22kHz -- but not separately; all settings were either set to high or to low, low gives some 1 frame per second increase or so)

Somewhat interestingly, I found no notable effect of turning off shadows or ambient sounds.

Now for somewhat less obvious results:

• Duke3D shareware v1.1 is quite faster than v1.3D. FRAPS' FPS counter would go over 20 FPS in some scenes with moving enemies, whereas v1.3D stuck at 11-14 FPS most of the time.
  
• Atomic 1.5 is completely unplayable with the above setup, demo playback is literally a slideshow at 1 FPS and controls are almost unresponsive (I'm talking of the setup with similarly reduced screen size, MIDI music and low-quality sound mixing and sampling). I expected roughly the same results with Shadow Warrior but
  
• SW actually performs better than Atomic, but worse than Duke3D shareware. Relatively calm scenes give you around 12-13 FPS, combat drops down to 6 and below. It's technically playable, probably even more so with lower screen size.

[K1n9_Duk3]

I'm sorry to say this, but based on my experience, FRAPS won't always give you reliable results for games running in DOSBox. I think what happens is that FRAPS registers any update in the DOSBox "screen" as one frame. If the game draws the exact same frame twice in a row (i.e. the graphics on the "screen" don't change, then FRAPS doesn't count that. On the other hand, if there are any refresh issues like some flickering areas or screen tearing on the DOSBox "screen", FRAPS will count those as a frame, even though the game hasn't actually rendered a new frame. At least that's what I remember from my experiments with FRAPS and DOSBox. Maybe you're using a more up-to-date version and these problems have been fixed by now, but I thought I'd let you know anyway.

[MrFlibble]

I probably should have stressed that I'm taking the FRAPS counter output with a pinch of salt, for the lack of a better tool at this moment. I should get one if I'm going to do more serious research I guess.

But FRAPS output does agree with what I observe while playing/watching.

I ran Blood shareware with the same DOSBox settings, versions 0.99 (the first public release) and 1.11. The latter seems slightly faster. Combat situations slow down very notably anyway, and dynamite explosions result in a drop to around 1-2 FPS. But Blood loads relatively fast, possibly faster than SW. The game also does not have the pixellated Low Detail mode, instead there's a detail slider that I frankly haven't figured out what it does. But it moved it to the lowest settings for tests.

[NY00123]

Hey there,

I guess that I'll also use this thread for non-Build games which didn't directly involve Apogee. As MrFlibble briefly hinted earlier, there's also recreated code for Hexen. Heretic was further added later on.

The open-source release of Hexen turned out to match one revision identified as version 1.1, out of two. I basically found out that an earlier 1.1 revision differs just in A_SoAExplode, the VERSION_ID macro and the date in the VERSIONTEXT macro. It wasn't initially obvious that, in fact, it wasn't just one of these that was actually released as "v1.1".

The A_SoAExplode fix involves checking the "nomonsters" option, as well as the lack or presence of an mobjinfo's MF_COUNTKILL flag, before spawning a monster after breaking a suit of armor.

I later recreated the behaviors of Hexen v1.0, for which there turned out to be two virtually-identical revisions again. They differ just in a CD music check within P_SetupLevel.

At some later point, I started to work on a 4-level demo of Hexen from Oct 18 1995.
Following this, I continued with Heretic, covering versions 1.0 and 1.2. It turned out that the released sources match 1.3, and as confirmed by PVS of Doomworld, that 1.2 is usable with maps from all episodes within the Shadow of the Serpent Riders' IWAD.

As of the last day, I've gotten another addition for the Hexen repository. To make this short, it should now cover code which is more-or-less fully equivalent in behaviors to the 4-level beta (Oct 2 1995).

There's probably too much to write about the code itself, so I'll simply mention the following examples of information:
- This revision has code for the removed fly creature (https://doomwiki.org/wiki/Fly).
- The 4-level demo from Oct 18 1995, previously named HEXDMO10 in my repository, was renamed HEXDM10B, while the earlier demo was named HEXDM10A.
- I considered renaming HEXDM10B back to HEXDMO10, while HEXDM10A would be renamed using (a subset of) the EXE's original modification date; Reason being, the latter is identified as a beta in-game, and I already did a similar thing with a Wolf3D proto. beforehand.
- However, both versions are referred to as demos in the README.TXT files from 1995. I also don't currently recall any mention of a version number, like 1.0. For now, I just keep using the names of HEXDM10A and HEXDM10B.
- I originally started to inspect the 4-level beta as a possibility after finishing with Heretic 1.0-1.2. I eventually returned to the beta more recently. What's clear is that it required more work to recreate the code than the previously covered versions of Hexen; Maybe even more than all previously covered versions of Heretic and Hexen, combined.
- In addition to the previously known issue of global variables not being fully ordered as in the original EXE, there are also a few functions that I couldn't get their compiler-generated layouts to fully match. Unless I missed anything, they should still match in behaviors. The functions in question are A_Quake, P_XYMovement and P_ZMovement. The latter's C code was actually not changed by me at all.

[dwtietz]

K1n9_Duk3, on 26 June 2020 - 12:29 PM, said:

DOSBox is a bad example in this context, since it is in no way authentic when it comes to the timing of the CPU instructions. Code that's fast on real 386 or 486 CPUs might be slower in DOSBox, while code that's fast in DOSBox might be slower on real CPUs.

Responding to an old comment here, but I'd suggest that PCem should be used as a superior form of platform emulation as it does a very decent job of emulating period specific hardware configurations along with their timings. Only the hardware is emulated and a true DOS operating system needs to be installed. I've been running this for a short while now and it's a million times better at everything than DOSBox ever was.

[Phredreeke]

Yes, DOSBox CPU timing is inaccurate, but for a correctly programmed game this shouldn't matter because of the inherent variance in PC hardware.

Though I'm curious, is PCem capable of running Chasm the Rift without hacking the executable? (this is an example of a not correctly programmed game, which terminates in DOSBox's dynamic core due to a division by zero in a system benchmark run on launch)

[dwtietz]

Phredreeke, on 09 May 2021 - 05:21 PM, said:

Though I'm curious, is PCem capable of running Chasm the Rift without hacking the executable? (this is an example of a not correctly programmed game, which terminates in DOSBox's dynamic core due to a division by zero in a system benchmark run on launch)

Sorry, I don't know. If I had a copy of the game I'd try it out to let you know, but I don't.

I've only built two machines with PCem so far. One is a 486 DX4 100 running MS DOS 6.22 and the other is a Pentium II 450 running Win98SE. The DOS system has been absolutely amazing with it's performance with everything so far. The Windows 98 system has been working really well although I find the sound in Windows to be really choppy; this is almost entirely cleared up when I launch a DOS based game from Windows 98 (for those that will run from within Windows anyway. The early Tekwar demos run okay, but the retail versions don't). Even with the choppy sound in Win98, this is still the best VM I've ever used for setting up an older system environment.

[NY00123]

Phredreeke, on 26 June 2020 - 01:17 PM, said:

This is speculation on my part, but it wouldn't surprise me if the assembly optimisations have the opposite effect when using DOSBox's dynarec (forcing it to translate the same code over and over again as it's modified). Has anyone tried building Duke or SW without such optimisations and running in DosBox for comparison?

MrFlibble, on 04 July 2020 - 06:24 AM, said:

How simple would it be to tweak the code to turn those off?My experience in this field is limited to compiling MBF in DOSBox to support HACX.WAD, and that was some six years ago.

You can begin by trying to build your own working Duke3D or SW executable, using Watcom C.
Afterwards, repeat but with the Build engine, then make a Duke3D/SW exe using the generated Build OBJ files instead of the ones provided with the game sources.
Finally, locate A.C (look for buildc.zip), edit Build's makefile to use it instead of A.ASM, make a new A.OBJ file and then use it to make a new Duke3D/SW exe. Note that the choice of compiler options (e.g., optimizations) might have a great impact on the performance.

The most important thing here is that, ignoring A.ASM/A.C, you should otherwise use the exact same revision of the Build engine. Otherwise, you might unintentionally introduce additional differences in performance.

Thus, while the released Duke3D and SW sources may come with their own Build Engine OBJ files, you should exclusively use your own constructed files for the purpose of benchmarking.

To simplify things, you can use the version of Ken-Build available from Ken's website. It should be (at least mostly) compatible with Duke3D and SW as open-sourced in 2003-2005.

[MrFlibble]

NY00123, on 12 May 2021 - 09:20 AM, said:

You can begin by trying to build your own working Duke3D or SW executable, using Watcom C.
Afterwards, repeat but with the Build engine, then make a Duke3D/SW exe using the generated Build OBJ files instead of the ones provided with the game sources.
Finally, locate A.C (look for buildc.zip), edit Build's makefile to use it instead of A.ASM, make a new A.OBJ file and then use it to make a new Duke3D/SW exe. Note that the choice of compiler options (e.g., optimizations) might have a great impact on the performance.

Thanks, I'll try to figure it out when I got some time.

On an unrelated note, would it be possible for you to produce similar reconstructions for Blood binaries? Or do you need the complete source code for at least one release to do this?

[NY00123]

MrFlibble, on 13 May 2021 - 02:19 AM, said:

On an unrelated note, would it be possible for you to produce similar reconstructions for Blood binaries? Or do you need the complete source code for at least one release to do this?

While it's not impossible to do so without the source code, I'd rather have access to original Blood sources under an appropriate license, in case I were to inspect Blood.

[Phredreeke]

Not to mention, we already have two reverse engineered ports of Blood. IMO efforts would be better spent on games that doesn't currently have a port, so the reconstructed source could be used as a basis for that

[NY00123]

So, it's possible that anybody who tries to cover all versions from a single EXE might find it difficult to do so at this point.
That is, unless you go for what I did in Reflection Wolfenstein 3D, as it more-or-less covers separate builds of different game versions in a single exe.

As of this post, the following additions are now in:
- 3-level shareware beta of Heretic (Dec 20 1994).
- Retail store beta of Hexen (Sep 26 1995).

These will probably be the last additions related to Heretic and Hexen. As usual, there's no promise regarding any future endeavor.
To summarize, we ended with 5 distinct DOS builds of Heretic, and 7 of Hexen.

As I suspected, it still took significantly more time to work on the beta release of the 4-level Hexen demo (given preceding work on v1.0 and the demo re-release), compared to the work on the aforementioned two betas of Heretic and Hexen.

In certain technical manners, the two Hexen betas might be close, even if obviously not identical. The aforementioned removed fly code is the same, just for one example.

I did have issues with the generated layouts of the following functions in the Hexen betas: A_Quake, P_XYMovement and P_ZMovement. The latter's C code was actually not changed by me at all.

In the case of Heretic, I suddenly spotted a different layout for M_FindResponseFile in a recent build of 1.2. I did it see beforehand, also in Hexen, but I thought that it was not reproduced after I finished working on each of the differing Heretic versions, excluding the beta. At the least, the code size, including padding for alignment, remained the same, so the other functions did match in layouts and locations.

[Nuke.YKT]

Hi,

As you know Doom source code as released in 1997 was not for original DOS version, but rather was Linux version cleaned up by Bernd Kreimeier. During clean up, code specific for DOS version was stripped out. Another big change was reorganizing header files, specifically doomdef.h, p_local.h and r_local.h were split to smaller header files. This is where Heretic sources comes in handy. Raven released their code as is, without any sort of clean up. Specifically it has majority of Doom's DOS specific code and header files organization is very close to Doom's original form.

The goal of this project is to combine both these sources and get codebase that is close to the Doom's original codebase as possible. Actually I had this idea for a pretty long time and had previous attempts in the form of PCDoom and PCDoom-v2. This time I decided to start mostly from Heretic sources, gradually reverting Raven's code changes by comparing it to Linux Doom sources. I targeted id Anthology Final Doom EXE as it was closest to linuxdoom sources. Eventually I got compiling EXE, but as expected it had lots of differences to original EXE files. Then I gradually fixed all the differences and got EXE that identical to original EXE by behaviours. Once I covered both revisions of Final Doom, NY00123 volunteered and also covered a lot more Doom revisions down to prototype v1.666. This also includes Chex Quest.

Meanwhile I decided to try to do the same for Strife executable using restored Doom code. As Strife source code was never released, I had to rely on reverse engineering of original EXE file. After finishing initial reversing resulted EXE had some differences to original EXE. Eventually after lots of trial and error and not without some help from NY00123 we eventually got EXE that identical to original EXE (up to usual garbage data between string literals and differences due to __LINE__ macro).

As original release of Doom this does not include proprietary DMX code. You can use APODMX replacement library instead to get complete GPLv2 compatible codebase.



TL;DR These repositories recreate different revisions of Doom and Strife codebase. Compiled EXE files are identical to original EXE in term of behaviours. In total both repositories cover 15 different revisions of Doom, Strife and Chex Quest.



List of covered Doom revisions:

Doom II prototype v1.666
Doom Shareware / Doom II early v1.666
Doom v1.666
Doom II v1.7
Doom II v1.7a
Doom II French v1.8
Doom / Doom II v1.8
Doom / Doom II v1.9
Doom v1.9 Special Edition prototype exe
The Ultimate Doom
Final Doom
Final Doom, later id Anthology revision
Chex Quest

List of covered Strife revisions:

Strife v1.3
Strife v1.31

Doom repository
Strife repository

[MrFlibble]

As usual, this is entirely awesome

[NY00123]

MrFlibble, on 29 January 2022 - 12:40 PM, said:

As usual, this is entirely awesome

Thanks for showing your interest (also as usual)!

For the sake of completeness, I'll mention additional (and partially less Doom-related) updates to gamesrc-ver-recreation, available before or during the same day the Doom and Strife trees were published. Additional notes specific to the Doom and Strife restorations (probably more for Doom) might be left for a separate post.

- Nuke.YKT updated the apodmx repository, i.e., his DMX wrapper that uses the Apogee Sound System. This technically impacts not just Doom or Strife, but also Heretic and Hexen.
- Ken Silverman figured out how to recreate Ken-Build's GAME.EXE as uploaded in November 2002. So, I added it as an option.
- For a few exes that can be built from the duke3d tree, 3 bytes would differ before adding these linker directives: "segment type code lo", "segment type data lo". As it recently turned out, an alternative fix was the simple removal of the directive "system dos4g".

Additionally, Nuke.YKT is now a part of gamesrc-ver-recreation.

There might still be restrictions on what's uploaded to gamesrc-ver-recreation. For instance, a reversed engineered game is generally not covered. Exceptions are still a possibility. For instance, after Blake Stone: Planet Strike was open-sourced by Apogee, it was stated that the Aliens of Gold sources were assumed to be lost, thus explaining their lack. Therefore, I didn't mind building upon Blzut3's earlier reverse-engineering efforts, and later uploading reconstructed sources for the game.

[Nuke.YKT]

Update to the Strife restoration

A couple more revisions of the Strife executable are covered now: registered v1.1(aka v1.0) and registered v1.2. Both reconstructed EXEs are identical to the original EXE files (up to garbage data between string literals and differences due to the __LINE__ macro). Thus gamesrc-ver-recreation now covers all known registered versions of Strife.


The next obvious step is to try to cover the demo versions of the Strife, but I expect much more differences because both demo versions use much earlier revisions of the executable, so I guess I'll leave this for later.

[MrFlibble]

Had a thought the other day. I wonder if the same restoration process could be applied to Tyrian/Tyrian 2000, considering that the original Pascal code is not available but only the C rewrite of OpenTyrian?

[Frenkel]

What about this September 1993 version of WOLF3D.EXE?

[Nuke.YKT]

gamesrc-ver-recreation now covers original shareware v0.99/v1.0 release of doom
https://bitbucket.or...recreation/doom

[NY00123]

And so, Nuke.YKT decided to go even further than version 1.2, all the way back to the very first proper shareware release! On a side-note, so far, outside of Chex Quest, all covered versions included clear support for Doom II, to different extents. The sprite and mobj definitions were also the same. With shareware v0.99/1.0, the situation is obviously different now.

By the way, I've had really minor updates to batch scripts in the Doom, Strife, Build and Duke3D sub-modules. Earlier in this November, it was discovered that the lack of use of the command-line switch /S for the program CHOICE was apparently a mistake. It might not be observed as a problem in DOSBox v0.74-3, but it otherwise was. Let's hope this is fixed now.

Frenkel, on 06 June 2023 - 03:16 AM, said:

What about this September 1993 version of WOLF3D.EXE?

That's a good question! The usual answer (at least for work of mine) is that what one sees from me is what you (can) get.

On a related note, I did see it mentioned by a New Blood person in one public Discord server that there were plans to release the nowadays leaked prototypes. There are (or at least were?) still plans to do this, but that's what I know for now.

[NY00123]

For a small update, after bringing up the topic of running a reconstructed Doom v0.99/1.0 binary that uses APODMX, Nuke.YKT realized that APODMX was incompatible with v0.99/1.0. That was correct, due to API changes in the functions SFX_PlayPatch and SFX_SetOrigin: https://doomwiki.org...d_pitch_removed

Thus, I decided to update the APODMX repository, so it should let you build two versions using the different APIs in separate sub-folders now. I also updated all of the Doom, Heretic, Hexen and Strife repositories accordingly. I further applied technical modifications to the Heretic and Hexen repositories related to the DOBUILD.BAT and DOCLEAN.BAT scripts, making them and related files closer to the ones for Doom and Strife (and partially also Build and Duke3D).

[MrFlibble]

NY00123, on 05 December 2023 - 03:12 PM, said:

Thus, I decided to update the APODMX repository, so it should let you build two versions using the different APIs in separate sub-folders now. I also updated all of the Doom, Heretic, Hexen and Strife repositories accordingly.

Just to clarify, does this mean that now any version can be built with random pitch working properly?

[NY00123]

MrFlibble, on 13 December 2023 - 09:26 AM, said:

Just to clarify, does this mean that now any version can be built with random pitch working properly?

That was also the case before the changes if you manually fixed the code by swapping the relevant function arguments.

What's different is that you can use APODMX with the reconstructed Doom sources made to match v0.99. Previously, you couldn't do this, because APODMX used just the new API, expecting more function arguments than what's passed in Doom v0.99.

[MrFlibble]

Alright, thanks!

[Phredreeke]

Btw on the subject of source code reconstruction, NukeYKT posted this earlier this year https://github.com/nukeykt/Blood-RE

I didn't see it brought up elsewhere on the forum

[NY00123]

Phredreeke, on 17 December 2023 - 12:01 PM, said:

Btw on the subject of source code reconstruction, NukeYKT posted this earlier this year https://github.com/nukeykt/Blood-RE

I didn't see it brought up elsewhere on the forum

Also not found by me are the reconstructed RR sources. These were originally added to the NBlood repository by the beginning of 2020, before eventually moving into a separate repository of Nuke.YKT earlier this year. They currently cover a more-or-less accurate RR reconstruction, while it is still not the case for RR:RA. Work on Blood-RE was halted before the release, due to technical difficulties (say with optimized functions), but they should be demo-accurate for the stock demos as far as I know.

We'll see if Nuke.YKT decides to create a new thread for at least one of the repositories.

As I've maybe written elsewhere, I tend to be a bit careful with what enters gamesrc-ver-recreation. This is a reason for the reconstructed RR and Blood sources remaining in separate locations, while Strife is a part of gamesrc-ver-recreation, just for a few examples. To explain the reasoning for Strife, it already had separately reconstructed sources approved for a GPLed release (as found in Chocolate Strife, and Strife: Veteran Edition).

P.S. On the subject of topics brought up on Discord but not in these forums, I've also found only one forum post mentioning NotBlood (outside of this post of mine).

[MrFlibble]

NY00123, on 17 December 2023 - 11:14 PM, said:

NotBlood

For a second there I hoped that this would be the libre data set project for NBlood.

[NY00123]

I'll start with a demonstration of Nuke.YKT's recent addition of Doom shareware v0.99/1.0. Basically, the day preceding the 30th anniversary, Nuke.YKT and I were going through this version in a cooperative session. We both used executables built from the reconstructed sources, albeit not exactly the same one.

Nuke.YKT's video (using DMX):

Spoiler

My video (using the Apogee Sound System via APODMX):

Spoiler

MrFlibble, on 18 December 2023 - 04:10 AM, said:

For a second there I hoped that this would be the libre data set project for NBlood.

This looks like a good proof of what may happen when topics are discussed outside of websites or forums like this one. The existence of NotBlood can indeed be discovered by going through multiplayer videos on at least one YouTube channel (related to Build Bros), and the list of GitHub forks of the NBlood repositories (accessible via the "Insights" page) still shows NotBlood as of now: https://github.com/n...kt/NBlood/forks

Generally speaking, though, outside of checking something like relevant Discord servers during the right time periods, a discovery of NotBlood's existence might be coincidental; One additional example for the latter, other than this thread, is the ZDoom forums' Raze sub-forums having a topic about NotBlood.