[Danukem]

I get this error when trying to start EDuke32 with the latest build:

ERROR loading lookup.dat: failed reading enough data

All builds after 4331 give me this error, 4331 and earlier work fine.

EDIT: I am using Lezing's CLUT pack, which has a modified LOOKUP.DAT

[Helixhorned]

That either means what it reads, or that LOOKUP.DAT is trying to load a lookup table at a reserved palette, in which case this message should immediately precede it:

ERROR: attempt to load lookup at reserved pal <number>

In case kread() really doesn't read enough data, it could mean two things. First, the file is really too short and we expect bytes that are not there. Second, the actual reading functions return a less-than-expected byte count. For the file I/O read() function, it is legal for that to happen. If that's a normal case on the OS you're using, we need to dig into it. If it's kzread(), Ken't ZIP implementation... well, that would need to be looked into as well, but it's not as urgent. (Personally, I'm not very interested in looking at that code.)
Note that in any case, the error only notifies you of an issue that was previously there, too. The earlier code simply didn't check the return value of kread(), which is bad style at the very least.

What's the LOOKUP.DAT you're loading and from where? (File system, ZIP, or GRP?)
Edit: I can start LNGA2 fine around here. Just to make sure we're speaking of the same LOOKUP.DAT: is it 36480 bytes in length?

[Danukem]

The one I'm using is 36,737 bytes.

I can reproduce the error by starting a vanilla (i.e. no mods) game with the addition of the attached PALETTE.DAT and LOOKUP.DAT files.

The game will start fine in r4331 and earlier.

Attached File(s)

•  CLUT.zip (44.12K)
  Number of downloads: 499

[Helixhorned]

Oh, OK, it's this:

ERROR: attempt to load lookup at reserved pal 0

CraigFatman, why? Lookup 0 is the first 256 bytes of the base shade table after all, which you provide with PALETTE.DAT.

[Danukem]

So is that lookup table easy to fix? I really want to keep using those extra pals for my Attrition mod.

[Jan Satcitananda]

Helixhorned, on 26 February 2014 - 11:39 AM, said:

Edit: I can start LNGA2 fine around here. Just to make sure we're speaking of the same LOOKUP.DAT: is it 36480 bytes in length?

He is talking about ExtCLUT mod, not the LNGA. I can confirm that it doesn't launch in new builds because it uses altered pal #0:

Quote

ERROR: attempt to load lookup at pal 0

Now it's mandatory to have a legacy support of pal #0 remapping like it was before.

[Jan Satcitananda]

Helixhorned, on 26 February 2014 - 12:00 PM, said:

CraigFatman, why? Lookup 0 is the first 256 bytes of the base shade table after all, which you provide with PALETTE.DAT.

I was trying to remove repeated colors from the ordinary color set to use them for special effects (like monochrome, invert, darken, brighten etc.).

[Helixhorned]

Yeah, I'll revert it. I see now that it's with almost certainty used for Lezing's crazy color index hackery. The lookup comes last and is not the identity map. It has 247 unique indices. Interesting.
EDIT: ninja'd.
EDIT2: committed.

[Jan Satcitananda]

That was fast, lol.

Btw, there's a problem I forgot to mention. I've noticed the way the files are handled by writearraytofile and readarrayfromfile commands in 64-bit builds: whilst the gamevars are 32-bit integers, these commands apparently treat them as having 8 bytes per entry, what makes binary files saved with the 32-bit version incompatible with the 64-bit version. This bugs me because my current mod uses LOADS of precalculated binaries to work.

[Helixhorned]

Yes, that's a known issue in C-CON. LunaCON has this right: it stores gamearrays as a sequence of 32-bit. little-endian integers. (The important part on endianness is only that we know how to reconstruct it from disk. It could be fixed to big-endian, or be native-endian with the file additionally telling which one is stored in it.)

Fixing it in C-CON can be in one of two ways: making gamearrays always int32 in memory (harder), or only fixing the read/write routines (easier, only needs some temp. allocation). I'm not very keen on fixing any but the most critical C-CON bugs though.

[Jan Satcitananda]

Helixhorned, on 26 February 2014 - 01:12 PM, said:

Fixing it in C-CON can be in one of two ways: making gamearrays always int32 in memory (harder), or only fixing the read/write routines (easier, only needs some temp. allocation). I'm not very keen on fixing any but the most critical C-CON bugs though.

I would state that both versions should handle integer overflows the same way to avoid possible compatibility problems, so it's better to have them strictly 32-bit. I also thought that simple replacing of all "int" types with "signed __int32" does the job, doesn't it?

[Helixhorned]

I generally agree about "avoiding possible compatibility problems", and ideally gamearrays and the like would be int32_t throughout in C-CON. However, the issue of integer overflow is yet a different one. C-CON implements arithmetic on its numbers using the C operators (*, -, +, /) directly, without any checking. In C, overflow of a signed integer is undefined behavior. Thus, you must not rely on any particular observed behavior. While this is sort of a bug (for a well-defined program, it should not be possible to trigger undefined behavior by any input data), this also allows some simplifications. In particular, LunaCON does its arithmetic on Lua numbers, which are double-precision floating point (with approriate rounding for division). It also sports an option -ftrapv to error on overflow, or -fwrapv to provide wrapping, which however incurs some overhead.

[Hendricks266]

Helixhorned, on 27 February 2014 - 02:02 AM, said:

In C, overflow of a signed integer is undefined behavior.

Undefined by the language, but won't the hardware overflow as you expect? A full-adder bitstrip or any optimization of that design will.

[Helixhorned]

The x86 and x86_64 architectures define instructions like ADD to do exactly that. But the point here is that you must not reason about C as "high-level assembler". Nothing could be further from the truth. Undefined behaviors enable the C compiler to do a broad range of optimizations. I won't reiterate what others have already written about, but this quote from the first link may be insightful:

Quote

It is also worth pointing out that both Clang and GCC nail down a few behaviors that the C standard leaves undefined. The things I'll describe are both undefined according to the standard and treated as undefined behavior by both of these compilers in their default modes.

(...)

Signed integer overflow: If arithmetic on an 'int' type (for example) overflows, the result is undefined. One example is that "INT_MAX+1" is not guaranteed to be INT_MIN. This behavior enables certain classes of optimizations that are important for some code. For example, knowing that INT_MAX+1 is undefined allows optimizing "X+1 > X" to "true". Knowing the multiplication "cannot" overflow (because doing so would be undefined) allows optimizing "X*2/2" to "X". While these may seem trivial, these sorts of things are commonly exposed by inlining and macro expansion. A more important optimization that this allows is for "<=" loops like this:

for (i = 0; i <= N; ++i) { ... }


In this loop, the compiler can assume that the loop will iterate exactly N+1 times if "i" is undefined on overflow, which allows a broad range of loop optimizations to kick in. On the other hand, if the variable is defined to wrap around on overflow, then the compiler must assume that the loop is possibly infinite (which happens if N is INT_MAX) - which then disables these important loop optimizations. This particularly affects 64-bit platforms since so much code uses "int" as induction variables.

You may object that in the case of CON additions, the compiler doesn't know anything about the addends and should just emit an ADD without any optimization magic kicking in. Arguably, this argument doesn't really stand on a solid footing though. Who are we as C users to say what the compiler does or not? Compilers are very complex pieces of software. The language sets forth (most of the time) clear rules, and that's what one should follow.

Anyhow, in our particular case, the second reason why overflow is "undefined" is due to what Lezing mentioned: on 64-bit archs, we simply get values outside of [INT_MIN .. INT_MAX].

[Stabs]

helix, any chance you can include a script in a.m32 that finds the average the ceiling or floors heights of selected sectors and flattens them to a uniform height and removes any sloping on them?
like a "do state uniformceil" or "do state uniformfloor"

this would be so great for TROR

[Helixhorned]

DanM, on 28 February 2014 - 05:16 AM, said:

helix, any chance you can include a script in a.m32 that finds the average the ceiling or floors heights of selected sectors and flattens them to a uniform height and removes any sloping on them?
like a "do state uniformceil" or "do state uniformfloor"

Sure, see r4357. Contrary to what the log says, it's "state uniformceil" as you proposed.

[Stabs]

much thanks dude

[Fox]

Anyone know what algorithm was used to create the shade table from Duke 3D?

[Jan Satcitananda]

Fox, on 01 March 2014 - 04:53 AM, said:

Anyone know what algorithm was used to create the shade table from Duke 3D?

I'm pretty sure that the original shade table was done mostly by hand, with little aid of transpal utility or something.

[Stabs]

so i noticed i have command lines for cache option, would it be beneficial for me to increase that via command line for my new DNE release?

[TerminX]

Not unless you got really stoned and replaced everything with 8-bit assets, no.

[Stabs]

so can i get a command line option that ignores autoexec.cfg in favor or your own defined settings, for example dnelowsettings.cfg which would only contain the startup commands i really want, atm you have to configure controls for each gfx setting if you set a custom cfg

[Helixhorned]

Fox, on 01 March 2014 - 04:53 AM, said:

Anyone know what algorithm was used to create the shade table from Duke 3D?

This is as far as I got, from earlier in this thread:

Helixhorned, on 03 January 2014 - 06:59 AM, said:

I think it's been generated similar to this: Assume that the base palette contains all color ramps in consecutive blocks (like in the rearranged base palette colors plot on the wiki) and for a color index i, let b(i) denote the least color index belonging to the same ramp as i (the "base" color index of that ramp). Then, for a shade s and a color index i, palookup[s][i] = b(i) + round((s/32)*(i - b(i))).

Note that I still got the formula wrong even after correcting it once. It should read something like (give/take off-by-one or rounding errors):
palookup[s][i] = b(i) + round(((32-s)/32)*(i - b(i)))

This second attempt at recreating it is implemented in shadexfog.lua's create_base_shtab_2(). No guarantees that I didn't write something entirely nonsensical though. Analyzing the resulting shade table still shows differences that aren't (as far as I can see) easily explained. So, that question might be something to ask Ken one day.

[Fox]

CraigFatman seems to be partially correct. The shades of gray, blue, red and green have the exact same table, which requires some artistic touch. Also there is a second table used for the colors indigo and brown, which makes it more darker.

[Jblade]

Any ETA on blend support for HUD stuff? I really want to pretty up these muzzleflashes

[Fox]

The shade table really doesn't match a regular algorithm for shading multiplication...

[Stabs]

well it appears any swinging door with a custom speed bombs the FPS in polymer now, like 1-2 fps once the cache gets big enough, it's not as bad with a clear cache.

I tested this on DNE01 without any mods too, the rear doors to the cinema and in the theater can really display this problem really well, whats worse is overall fps gets worse and worse the more times you swing doors, even just opening and closing the same one will produce this effect.

[Tea Monster]

When are we going to bite the bullet and work out some way to get Polymer optimized?

[Jblade]

'we' he says, it's pretty much down to Plagman as far as I'm aware and I imagine he's probably very busy with life to dedicate the time to start coding in the theoretical performance enhancements.

[Stabs]

its running pretty much fine now and better than ever, i think it could be something to do with the DNE01 map itself, the hole on the side of the building that the commander makes when he burst into the cinemas office will not fog, the 3/4 sectors there will just not fog at all, i have tried pasting from tiles that do work and nothing seems to make these sectors want to accept fog.

edit : fixed that by copying the exploded wall sectors out, saving into a new map, loading DNE01 deleting the old walls and placing the newly copied section in its place, also notice substantial increases in FPS with every floor with a translucency on it stamped out, must of been why DNE bombed hard aswell in the FPS department, you could in the old days press T on floors and they would do a subtle weird effect but now they really show up and taking them out really ups the FPS.

just need a script that finds floors or ceilings that are NOT tror with translucency and turns them normal again and i should nail that one.