[Plagman]

This is a followup to the discussion in this thread.

Tinting is very much known to not be able to replace all the subtleties of 8-bit palette swaps. For example in the thread above, all you could get when trying to replicate pal 20 was something that looked vaguely grayish-blue on the whole surface. This new system is a lot more flexible as it allows people to define highpalookups that map every high-res color to a new one. There's no memory cost except for the initial overhead of the highpalookup map (one per palette, and all textures benefit from it) and a very slight additional per-texel cost when it's in use.

The downside of all this flexibility is that it's not all doable through the DEF language. You have to generate a highpalookup map offline and feed it to EDuke32 to DEF. The good news is that I wrote a sample Python script that does exactly that, though it only handles a couple of palettes right now (and they still need tweaking). The intent here is that a community trial-and-error effort will make this thing converge to maximal fidelity for the original Duke3D palettes and that modders will be able to do all kinds of insane stuff for their own purposes. See here for the script.

Here's a shot of how my initial pal 20 (on the texture that Norvak was talking about in the above thread) implementation looks:



The script gives you access to the RGB and HSV value of every possible color and lets you make arbitrary decisions as to how you want to map it in the colorspace. It conveniently slaps a TGA header in front of its output file, so you can manually edit it in your favorite image editor to tweak it even more (please resave it as TGA without compression if you still want EDuke32 to be able to read the file).

Once you have generated your file, you use the new DEF token, eg.:

highpalookup { pal 6 file "highpalookup_7_7_7_6" }
highpalookup { pal 20 file "highpalookup_7_7_7_20" }

Of course manual palette replacements still take precedence over this (but it overrides tinting) as this is even more flexible if you have the time and skill to do it. You can use r_pr_highpalookups in-game to toggle it in-game (don't forget to restartvid).

Recommended reading:

http://en.wikipedia....iki/HSL_and_HSV
http://docs.python.org/tutorial/

The GIMP color picker is also great to get a grasp of what H/S/V/R/G/B correspond to in reality.

Full credit goes to helixhorned for doing all the hard work on this.

[Danukem]

Can this be made to work with the 8-bit art? It would be nice to have a unified system, and it would be easier than editing palette.dat and lookup.dat if you want a mod with non-vanilla palettes.

[Plagman]

I think it already does, and I'm not sure why you'd be using 8-bit art exclusively with Polymer. You're still going to need to edit PALETTE.DAT for the software renderer.

[Jblade]

Cool stuff, I can't run polymer but it's cool that maps that make extensive use of the later pals will look much better when seen with the HRP now

[Jimmy]

Plagman, on Dec 30 2010, 05:21 AM, said:

I'm not sure why you'd be using 8-bit art exclusively with Polymer.

Why not?

[Plagman]

Because you don't get anything from it except the hassle of having to set up the ART files and palette data. The 8-bit tile gets converted to high-color and you lose all of the benefits of the shade offsets and the palookups.

[Hendricks266]

Awesome. Thanks, Plagman and Helixhorned.

Captain Awesome, on Dec 30 2010, 11:44 AM, said:

Why not?

For normalmaps, specmaps, etc., you need to feed the art through the DEFs, even if it's 8-bit styled art.

[Danukem]

Hendricks266, on Dec 30 2010, 11:46 AM, said:

Awesome. Thanks, Plagman and Helixhorned.

Are you going to work on making the lookup maps?

[Plagman]

BTW, helixhorned came up with a new script that's a superset of mine:

http://eduke32.svn.sourceforge.net/viewvc/...amp;view=markup

[The Commander]

Plagman, on Jan 6 2011, 12:11 PM, said:

BTW, helixhorned came up with a new script that's a superset of mine:

http://eduke32.svn.sourceforge.net/viewvc/...amp;view=markup

I got really good performance increases with the build, obviously due to the "This gives a gig more private virtual memory on XP"

[Plagman]

Doubtful, since that should only matter if you boot Windows with /3G on x86. BTW, I really don't recommend that you do that in general since this can cause severe resource constriction issues on the display driver side. Do yourself a favor and use an x64 OS if you can help it. :-)

[Hank]

DeeperThought, on Dec 30 2010, 03:58 AM, said:

Can this be made to work with the 8-bit art? It would be nice to have a unified system, and it would be easier than editing palette.dat and lookup.dat if you want a mod with non-vanilla palettes.

Since this came alive again
What is a non-vanilla palette? And if it is what I think it is, are there any plans to expand the default palette?

[Danukem]

Hank, on Jan 6 2011, 12:08 AM, said:

Since this came alive again
What is a non-vanilla palette? And if it is what I think it is, are there any plans to expand the default palette?

That would be any palettes other than the ones used in Duke 3D (i.e. any modified palette).

Check this out: http://lzg.duke4.net/extclut.htm

I use it in Attrition for the extra monster colors.

[bioman]

Hello,

Is possible to include the scripts in the synthesis, 'build/src/util' seem to be a good place.

[Jan Satcitananda]

Great idea to have palette conversions looking similar to good ol' lookup pals for truecolor textures in Polymer.
However, imho you guys are misleaded by difference between indexed and non-indexed color models.

First of all, original Duke uses an 8-bit indexed palette, and Polymer employs de facto 24-bit truecolor for both textures and rendering. That is not just usage of different number of colors, but something that requires a completely another approach to color handling. In indexed color, look-up tables are indispensable, because it takes too much operations to find the nearest color in the palette after we perform some operations over the colors. In high/true color, the RGB scale is uniform and searching nearest colors reduces to simple memory addressing, so we can apply various realtime effects with more ease (as long as they don't involve complex calculations).

If we represent all colors of the true color model in the fashion of indexed colors, we'll end up with an enormous amount of data, orders of magnitude greater than that with 256-color palette. If each of 16777216 colors is mapped by a 3-byte substitutional color, it's 48 MB per pal. As I can see from the script (I didn't run it, just looked over), it undersamples the color space to 7 bits per channel, leading to eight times fewer colors and thus 6 MB per palette, but this is still too much data to address randomly.

Since the shader model allows to incorporate various instructions inside a pixel shader, all computations related to color remapping can be done in realtime, and I would prefer this way. It should be noted that since we're dealing with large number of colors and only some of them are to be remapped, there may be issues with banding artifacts, when a texture contains a large area of pixels that lie near the edge of a range selected for filtering what leads to a weird pattern. So if we don't want filtered textures to contain sharp transitions and have a crappy look, we might have to improve the quality of such palette filters with fuzzy logic (meaning that there should be "soft" selection of colors to convert).

[Plagman]

Running the required logic once per pixel in realtime is the first approach that I had considered and prototyped and it turned out to be way more expensive and not practical for this purpose. The highpalookup maps do not contain too much data to be addressed randomly; try it for yourself ingame. After all, they're the same size as an average texture in today's games. I agree about soft transitions, especially when dealing with badly compressed or filtered textures, and I intend to provide a set of helper functions to make that easier in the generation script for people that aren't that experienced.

[Plagman]

bioman, on Jan 6 2011, 12:47 AM, said:

Hello,

Is possible to include the scripts in the synthesis, 'build/src/util' seem to be a good place.

Done with revision 1753. It does make more sense that way, thanks.

[Helixhorned]

You guys probably want to know what's up with the PNG files generated by the script. It's simply an image containing all possible RGB values for 7 bits arranged in a 128x(128x128) pattern. Basically, you can take that and edit it with your favourite image editor but because it's likely to be a pain in the ass, our approach right now is to manipulate it programmatically.

I've attached some test highpals to this post so you can judge the quality for yourself. Pals 1..8 of those are the current PHRP's tint equivalents and the rest are rough prototypes.

BTW, one of the goals of the system's development was to do away with largely unnecessary hightile images which have only a portion of some area colored. Duke's jeans and boot skin would be a good example.

Attached File(s)

•  hipal_images_test1.zip (741.39K)
  Number of downloads: 365

[Danukem]

Helixhorned, on Jan 9 2011, 11:32 AM, said:

BTW, one of the goals of the system's development was to do away with largely unnecessary hightile images which have only a portion of some area colored. Duke's jeans and boot skin would be a good example.

Yeah, that could greatly decrease the size of the HRP, and the size of other mods as well.

[bullerbullerseven]

That would be great. But i don't see why this is so neccessary.
I see many other things that is far more important. But may be impossible.

[Danukem]

Jhect, on Jan 9 2011, 12:14 PM, said:

That would be great. But i don't see why this is so neccessary.
I see many other things that is far more important. But may be impossible.

Helixhorned has added a very useful feature, and it's not taking away from anything else that's being worked on, so what is your problem?

In my opinion, by far the most important thing is getting Polymer optimized so that it runs better. But optimizing Polymer is clearly a job for Plagman, since he wrote it.

[The Commander]

Jhect, on Jan 10 2011, 09:14 AM, said:

That would be great. But i don't see why this is so neccessary.

*durp durp* To save bandwidth and file space.

Jhect, on Jan 10 2011, 09:14 AM, said:

I see many other things that is far more important. But may be impossible.

Anyone is welcome to send there programming code to TX, just like Helix did which is now why Mapster and other parts are working even better than before.

Your post was really dumb Jhect.

[bullerbullerseven]

Yea that is all very good. but i would rather see polymer being optimized just as DT said. That is not dumb lol.

[Jan Satcitananda]

I've just written an Evaldraw script from scratch that employs some kind of statistical computing over palette look-up tables and I'm getting interesting results. The generated color map tries to mimic original palette colors as close as possible. However it can't process correctly colors that lie outside palette's gamut (to leave original colors there, the "dry" signal is introduced into the model); also some colors that remain unchanged during conversion may cause distortion (like in pals #10 and over). I hope I'll manage to improve my model soon that it could produce suitable color maps.

[Plagman]

Cool! We've talked about using that as a base, but we thought it would require some hand-tweaking in the end because of highres stuff that doesn't exactly match the color distribution of ART, little details, etc (hence going full programmable with a script instead). I wish the two approaches could be combined somehow. Does your evaldraw script already output the whole RGB space to a file that the renderer can use as is? The format is pretty trivial, it's just a 3D volume of dimensions 128x128x128 and RGB coordinates.

[Helixhorned]

Looks promising! I'm especially interested in how well it would behave with a representative sample of the HRP, which is not always color-consistent with the original tiles. Here's the sample I've been using for my tests btw:

./sprites/monsters/1680_trooper_gear_0.png
./sprites/monsters/2000_pigcop_dead.png
./sprites/monsters/2000_pigcop.png
./sprites/characters/0753_geisha.png
./sprites/characters/1312_stripper.png
./sprites/characters/4510_pirate1a.png
./sprites/characters/4874_woman.png
./sprites/characters/1405_duke.png
./sprites/characters/3450_cheerleader.png
./sprites/props/4400_bear.png
./sprites/props/4386_cable.jpg
./sprites/props/0680.png
./sprites/props/0568.png
./sprites/pickups/1348_holoduke.png
./sprites/pickups/0060_keycard.png
./sprites/switches/0130_accessswitch.jpg
./sprites/switches/0171_accessswitch2.png
./sprites/firstperson/2563_tripkeyhand.png
./sprites/firstperson/2521_mightyboot.jpg
./textures/0832.png
./textures/0251.png
./textures/5084.jpg
./textures/0207_d.png
./textures/4108.png
./textures/0428.png
./textures/3407.png
./textures/4215.png
./textures/0390.png
./textures/4222.png
./textures/0399.jpg
./textures/0387.png
./textures/1215_d.png
./textures/0857.jpg
./textures/0332.jpg
./textures/1074.png
./textures/0156.png
./textures/4097.png

[Jan Satcitananda]

Plagman, on Jan 10 2011, 03:13 AM, said:

Cool! We've talked about using that as a base, but we thought it would require some hand-tweaking in the end because of highres stuff that doesn't exactly match the color distribution of ART, little details, etc (hence going full programmable with a script instead). I wish the two approaches could be combined somehow. Does your evaldraw script already output the whole RGB space to a file that the renderer can use as is? The format is pretty trivial, it's just a 3D volume of dimensions 128x128x128 and RGB coordinates.

Evaldraw still lacks consistent functions for binary output, so it's easiest to plot all RB planes onto the screen, then capture a couple of screenshots and compile a highpalookup map compatible with Polymer. Otherwise, this app gets really handy in doing stuff like this. So far the script doesn't produce any useful data, but I continue developing it.

Btw, I can't download the Python script anymore by the link in the first post. Has it been removed?

Helixhorned, on Jan 10 2011, 03:23 AM, said:

Here's the sample I've been using for my tests btw:

Thx, I'll try them.

[Scott_AW]

CraigFatman, on Jan 10 2011, 01:02 AM, said:

Evaldraw still lacks consistent functions for binary output, so it's easiest to plot all RB planes onto the screen, then capture a couple of screenshots and compile a highpalookup map compatible with Polymer. Otherwise, this app gets really handy in doing stuff like this. So far the script doesn't produce any useful data, but I continue developing it.

Btw, I can't download the Python script anymore by the link in the first post. Has it been removed?


Thx, I'll try them.

Isn't Evaldraw fun? I think there's a work around for that, I'll get back to you if I remember. I think Ken had added something that I think can be used to create an output.

[Hendricks266]

CraigFatman, on Jan 10 2011, 03:02 AM, said:

Evaldraw still lacks consistent functions for binary output, so it's easiest to plot all RB planes onto the screen, then capture a couple of screenshots and compile a highpalookup map compatible with Polymer. Otherwise, this app gets really handy in doing stuff like this. So far the script doesn't produce any useful data, but I continue developing it.

Btw, I can't download the Python script anymore by the link in the first post. Has it been removed?

This palette stuff is really awesome. As for the Python script, it was moved. See here for both of them.

[Jan Satcitananda]

Scott_AW, on Jan 10 2011, 12:35 PM, said:

Isn't Evaldraw fun? I think there's a work around for that, I'll get back to you if I remember. I think Ken had added something that I think can be used to create an output.

Oh yeah, I've used the fputc command to code TGA output today. I didn't know about it.

http://img593.images...3/2899/pal2.png

This is the truecolor version of palette #2 generated by an incomplete version of my script (that is, no manual editing). I've resaved it to PNG to reduce file size. It seems to fit HRP textures quite well. Other palettes do as well, but I'm not satisfied with ones which preserve most of the colors unchanged. I'll try to refine my color mixing algorithm to get something like a fuzzy Voronoi diagram.