[Jan Satcitananda]

Helixhorned, on 26 February 2014 - 11:39 AM, said:

Edit: I can start LNGA2 fine around here. Just to make sure we're speaking of the same LOOKUP.DAT: is it 36480 bytes in length?

He is talking about ExtCLUT mod, not the LNGA. I can confirm that it doesn't launch in new builds because it uses altered pal #0:

Quote

ERROR: attempt to load lookup at pal 0

Now it's mandatory to have a legacy support of pal #0 remapping like it was before.

[Jan Satcitananda]

Helixhorned, on 26 February 2014 - 12:00 PM, said:

CraigFatman, why? Lookup 0 is the first 256 bytes of the base shade table after all, which you provide with PALETTE.DAT.

I was trying to remove repeated colors from the ordinary color set to use them for special effects (like monochrome, invert, darken, brighten etc.).

[Helixhorned]

Yeah, I'll revert it. I see now that it's with almost certainty used for Lezing's crazy color index hackery. The lookup comes last and is not the identity map. It has 247 unique indices. Interesting.
EDIT: ninja'd.
EDIT2: committed.

[Jan Satcitananda]

That was fast, lol.

Btw, there's a problem I forgot to mention. I've noticed the way the files are handled by writearraytofile and readarrayfromfile commands in 64-bit builds: whilst the gamevars are 32-bit integers, these commands apparently treat them as having 8 bytes per entry, what makes binary files saved with the 32-bit version incompatible with the 64-bit version. This bugs me because my current mod uses LOADS of precalculated binaries to work.

[Helixhorned]

Yes, that's a known issue in C-CON. LunaCON has this right: it stores gamearrays as a sequence of 32-bit. little-endian integers. (The important part on endianness is only that we know how to reconstruct it from disk. It could be fixed to big-endian, or be native-endian with the file additionally telling which one is stored in it.)

Fixing it in C-CON can be in one of two ways: making gamearrays always int32 in memory (harder), or only fixing the read/write routines (easier, only needs some temp. allocation). I'm not very keen on fixing any but the most critical C-CON bugs though.

[Jan Satcitananda]

Helixhorned, on 26 February 2014 - 01:12 PM, said:

Fixing it in C-CON can be in one of two ways: making gamearrays always int32 in memory (harder), or only fixing the read/write routines (easier, only needs some temp. allocation). I'm not very keen on fixing any but the most critical C-CON bugs though.

I would state that both versions should handle integer overflows the same way to avoid possible compatibility problems, so it's better to have them strictly 32-bit. I also thought that simple replacing of all "int" types with "signed __int32" does the job, doesn't it?

[Helixhorned]

I generally agree about "avoiding possible compatibility problems", and ideally gamearrays and the like would be int32_t throughout in C-CON. However, the issue of integer overflow is yet a different one. C-CON implements arithmetic on its numbers using the C operators (*, -, +, /) directly, without any checking. In C, overflow of a signed integer is undefined behavior. Thus, you must not rely on any particular observed behavior. While this is sort of a bug (for a well-defined program, it should not be possible to trigger undefined behavior by any input data), this also allows some simplifications. In particular, LunaCON does its arithmetic on Lua numbers, which are double-precision floating point (with approriate rounding for division). It also sports an option -ftrapv to error on overflow, or -fwrapv to provide wrapping, which however incurs some overhead.

[Hendricks266]

Helixhorned, on 27 February 2014 - 02:02 AM, said:

In C, overflow of a signed integer is undefined behavior.

Undefined by the language, but won't the hardware overflow as you expect? A full-adder bitstrip or any optimization of that design will.

[Helixhorned]

The x86 and x86_64 architectures define instructions like ADD to do exactly that. But the point here is that you must not reason about C as "high-level assembler". Nothing could be further from the truth. Undefined behaviors enable the C compiler to do a broad range of optimizations. I won't reiterate what others have already written about, but this quote from the first link may be insightful:

Quote

It is also worth pointing out that both Clang and GCC nail down a few behaviors that the C standard leaves undefined. The things I'll describe are both undefined according to the standard and treated as undefined behavior by both of these compilers in their default modes.

(...)

Signed integer overflow: If arithmetic on an 'int' type (for example) overflows, the result is undefined. One example is that "INT_MAX+1" is not guaranteed to be INT_MIN. This behavior enables certain classes of optimizations that are important for some code. For example, knowing that INT_MAX+1 is undefined allows optimizing "X+1 > X" to "true". Knowing the multiplication "cannot" overflow (because doing so would be undefined) allows optimizing "X*2/2" to "X". While these may seem trivial, these sorts of things are commonly exposed by inlining and macro expansion. A more important optimization that this allows is for "<=" loops like this:

for (i = 0; i <= N; ++i) { ... }


In this loop, the compiler can assume that the loop will iterate exactly N+1 times if "i" is undefined on overflow, which allows a broad range of loop optimizations to kick in. On the other hand, if the variable is defined to wrap around on overflow, then the compiler must assume that the loop is possibly infinite (which happens if N is INT_MAX) - which then disables these important loop optimizations. This particularly affects 64-bit platforms since so much code uses "int" as induction variables.

You may object that in the case of CON additions, the compiler doesn't know anything about the addends and should just emit an ADD without any optimization magic kicking in. Arguably, this argument doesn't really stand on a solid footing though. Who are we as C users to say what the compiler does or not? Compilers are very complex pieces of software. The language sets forth (most of the time) clear rules, and that's what one should follow.

Anyhow, in our particular case, the second reason why overflow is "undefined" is due to what Lezing mentioned: on 64-bit archs, we simply get values outside of [INT_MIN .. INT_MAX].

[Stabs]

helix, any chance you can include a script in a.m32 that finds the average the ceiling or floors heights of selected sectors and flattens them to a uniform height and removes any sloping on them?
like a "do state uniformceil" or "do state uniformfloor"

this would be so great for TROR

[Helixhorned]

DanM, on 28 February 2014 - 05:16 AM, said:

helix, any chance you can include a script in a.m32 that finds the average the ceiling or floors heights of selected sectors and flattens them to a uniform height and removes any sloping on them?
like a "do state uniformceil" or "do state uniformfloor"

Sure, see r4357. Contrary to what the log says, it's "state uniformceil" as you proposed.

[Stabs]

much thanks dude

[Fox]

Anyone know what algorithm was used to create the shade table from Duke 3D?

[Jan Satcitananda]

Fox, on 01 March 2014 - 04:53 AM, said:

Anyone know what algorithm was used to create the shade table from Duke 3D?

I'm pretty sure that the original shade table was done mostly by hand, with little aid of transpal utility or something.

[Stabs]

so i noticed i have command lines for cache option, would it be beneficial for me to increase that via command line for my new DNE release?

[TerminX]

Not unless you got really stoned and replaced everything with 8-bit assets, no.

[Stabs]

so can i get a command line option that ignores autoexec.cfg in favor or your own defined settings, for example dnelowsettings.cfg which would only contain the startup commands i really want, atm you have to configure controls for each gfx setting if you set a custom cfg

[Helixhorned]

Fox, on 01 March 2014 - 04:53 AM, said:

Anyone know what algorithm was used to create the shade table from Duke 3D?

This is as far as I got, from earlier in this thread:

Helixhorned, on 03 January 2014 - 06:59 AM, said:

I think it's been generated similar to this: Assume that the base palette contains all color ramps in consecutive blocks (like in the rearranged base palette colors plot on the wiki) and for a color index i, let b(i) denote the least color index belonging to the same ramp as i (the "base" color index of that ramp). Then, for a shade s and a color index i, palookup[s][i] = b(i) + round((s/32)*(i - b(i))).

Note that I still got the formula wrong even after correcting it once. It should read something like (give/take off-by-one or rounding errors):
palookup[s][i] = b(i) + round(((32-s)/32)*(i - b(i)))

This second attempt at recreating it is implemented in shadexfog.lua's create_base_shtab_2(). No guarantees that I didn't write something entirely nonsensical though. Analyzing the resulting shade table still shows differences that aren't (as far as I can see) easily explained. So, that question might be something to ask Ken one day.

[Fox]

CraigFatman seems to be partially correct. The shades of gray, blue, red and green have the exact same table, which requires some artistic touch. Also there is a second table used for the colors indigo and brown, which makes it more darker.

[Jblade]

Any ETA on blend support for HUD stuff? I really want to pretty up these muzzleflashes

[Fox]

The shade table really doesn't match a regular algorithm for shading multiplication...

[Stabs]

well it appears any swinging door with a custom speed bombs the FPS in polymer now, like 1-2 fps once the cache gets big enough, it's not as bad with a clear cache.

I tested this on DNE01 without any mods too, the rear doors to the cinema and in the theater can really display this problem really well, whats worse is overall fps gets worse and worse the more times you swing doors, even just opening and closing the same one will produce this effect.

[Tea Monster]

When are we going to bite the bullet and work out some way to get Polymer optimized?

[Jblade]

'we' he says, it's pretty much down to Plagman as far as I'm aware and I imagine he's probably very busy with life to dedicate the time to start coding in the theoretical performance enhancements.

[Stabs]

its running pretty much fine now and better than ever, i think it could be something to do with the DNE01 map itself, the hole on the side of the building that the commander makes when he burst into the cinemas office will not fog, the 3/4 sectors there will just not fog at all, i have tried pasting from tiles that do work and nothing seems to make these sectors want to accept fog.

edit : fixed that by copying the exploded wall sectors out, saving into a new map, loading DNE01 deleting the old walls and placing the newly copied section in its place, also notice substantial increases in FPS with every floor with a translucency on it stamped out, must of been why DNE bombed hard aswell in the FPS department, you could in the old days press T on floors and they would do a subtle weird effect but now they really show up and taking them out really ups the FPS.

just need a script that finds floors or ceilings that are NOT tror with translucency and turns them normal again and i should nail that one.

[Helixhorned]

Fox, on 05 March 2014 - 05:08 PM, said:

The shade table really doesn't match a regular algorithm for shading multiplication...

Recall that the base palette -- that is, all colors that are ever displayed in a single scene -- has 256 colors in total. One particular "ramp" has less colors at its disposal. In the case of the gray ramp, there are 32 colors of those. So, the left picture looks a lot like the right one quantized to 32 levels.

Spoiler

Some quick Octave code demonstrates this.

a = repmat((0:31)/32, 32, 1) .* repmat((0:31)'/32, 1, 32);
imagesc(a); colormap(gray); colorbar; set(gca,'YDir','normal')
% ^ looks like the right picture

b = floor(32*a)/32;
imagesc(:blink:; colormap(gray); colorbar; set(gca,'YDir','normal')
% ^ looks like the left one

I'm attaching a ZIP with four images showing the colors of the whole shade table, for the original one, my two attempts, and the one generated by 'transpal'.

Summarizing, my formula ("attempt 2") produces a similar result to the original one for the five ramps of 32 colors each (gray, brown, blue, green, red), but doesn't work well for the remaining three ramps of 16 colors each. Using color indices whose colors are close to linearly-blended ones ("attempt 1", "transpal") produces visible artefacts, most noticeably the fact that colors "leak" into foreign ramps, such as with the brown and red ones. Note that I'm using a color index rearrangement, as documented on the wiki.

Edit: also, just in case you have missed that, I should point out that my formula makes no reference to the actual color components of particular color indices.

Attached File(s)

•  shtab_comparison.zip (92.95K)
  Number of downloads: 356

[Tea Monster]

James, on 06 March 2014 - 07:58 AM, said:

'we' he says, it's pretty much down to Plagman as far as I'm aware and I imagine he's probably very busy with life to dedicate the time to start coding in the theoretical performance enhancements.

I meant we. Plagman would be doing all the work, but In the past, I've suggested we have a kickstarter or similar so that he has more incentive to do it.

Glad its running better. I know it was a big bottleneck when the mod first came out.

[Fox]

Since we are talking about shading, why is the shading of HUD sprites different in OpenGL? Map structures are relatively similar to 8-bit, while HUD sprites are not. Most notable when you are throwing a Pipebomb, which changes from the HUD to a spawned sprite.

[Helixhorned]

DanM, on 06 March 2014 - 05:33 AM, said:

well it appears any swinging door with a custom speed bombs the FPS in polymer now, like 1-2 fps once the cache gets big enough, it's not as bad with a clear cache.

DanM, on 06 March 2014 - 08:49 AM, said:

its running pretty much fine now and better than ever, i think it could be something to do with the DNE01 map itself, (...).

edit : fixed that (...), also notice substantial increases in FPS with every floor with a translucency on it stamped out, (...).

Wait, you mean you have worked around the original issue so that it won't show up in the latest DNE any more? I was interested into looking into it, especially because the suggested causal relationship sounds so incredible. Why should a property of a game-side object such as a door have an influence on the performance of a renderer, which is part of the engine? Also, when is the cache (assuming you mean the hightile one, residing in 'textures' and 'textures.cache') "big enough"? When it approaches the gigabyte range?
Your solution seems to imply that the swinging door speed had nothing to do with it though. My guess is that when Polymer encounters a translucent ceiling or floor that does not have another sector at the other side, it sort of thinks that the rest of the map is visible through this ceiling/floor and starts loading all textures that are "visible" this way.

Fox, on 06 March 2014 - 11:09 AM, said:

Since we are talking about shading, why is the shading of HUD sprites different in OpenGL? Map structures are relatively similar to 8-bit, while HUD sprites are not. Most notable when you are throwing a Pipebomb, which changes from the HUD to a spawned sprite.

Test case + specifics? I quickly looked at the pipebomb HUD in E1L1, once while on the outside and once in a shaded sector, and didn't find any difference between either the HUD and in-world sprite, or classic and Polymost.

[Micky C]

I was thinking, maybe making floors and ceilings transparent without TROR should be restricted to expert mode? I've come across transparent floors in maps such as the Metropolitan Mayhem episode which must have accidentally had their transparency set, and which only made a difference in polymer.