[MrFlibble]

Mark, on 03 February 2019 - 12:07 PM, said:

You could try adding the scale look back in with a texture filter. Heres a quick example.

Actually that's a very good idea, thank you Sir!

What programme exactly did you use and what texture? Is there a way to limit it to only the skin parts while leaving the Orc's armour alone? (or did you already do that?)

MusicallyInspired, on 03 February 2019 - 11:44 AM, said:

Why do you keep doing that??

In the case of Ken's Labyrinth image, to reduce the file size because Imgur auto-converts large PNGs (over 1 MiB in size) to JPEG.

But generally I dislike the blurry look of the results, whereas the indexed palette gives a sharper image without any loss of detail. Or, to put it the other way, I don't think that there is any advantage in having a true colour image in any of the experiments above.

[Mark]

I use Paint Shop Pro 7.04
I adjusted the selection tool to highlight only the green. Then from dropdown menus went to Effects/TextureEffects/Textures and chose a very bumpy texture that looks kind of like cooled lava rock. I applied it using the very lightest setting. You could do the same for other parts like a brush aluminum finish or scratched metal for the armor.

The only drawback I see, and it might not even be noticible in the game, is each frame of the enemy sprite will have the texture applied slightly different because the size and shape of the selected area has changed. But I doubt it would stand out enough to scrap the whole idea.

[MrFlibble]

Thanks!

Mark, on 04 February 2019 - 06:50 AM, said:

The only drawback I see, and it might not even be noticible in the game, is each frame of the enemy sprite will have the texture applied slightly different because the size and shape of the selected area has changed

Yeah, that's what MasonFace from Daggerfall Workshop forums also suggested. But considering that the characters are almost constantly moving it shouldn't be very noticeable anyway.

[Mark]

He stole my reply word for word.

Duh, I scrolled upwards and saw your earlier post where credit was given. Never mind.

[MrFlibble]

Looks like there's yet another GAN called ProSRGAN:
https://github.com/fperazzi/proSR

Haven't checked it out yet.

[MrFlibble]

For a test, I upscaled some character sprites from Daggerfall with ESRGAN using the procedure described here.

[MusicallyInspired]

Interesting. I never gave any thought to adding a soft glow to the edges. That would solve the hard jagged edges of some images. It would take a lot of time and specific hands-on fine-tuning pre-upscale to various images, but the same technique could be applied to the Sierra backgrounds I was making.

It's too bad my computer went kaput. Anything I try to do now would take forever on my old laptop here. No accelerated GPU to aid.

[MrFlibble]

Sorry to hear that!

It seems that ESRGAN doesn't really do a particularly splendid job when it comes to hand-drawn images even with Manga-base interpolations:


I tried adding a little HSV noise (in GIMP, default settings) in addition to the anti-aliasing steps described earlier, the result gets somewhat better but not by a lot:

The noise seems to somewhat accentuate the detail, but it may also break some of the existing patterns in the image because it's random.

This time I didn't try to convert the stuff back to the original indexed palette

[MusicallyInspired]

By "hand-drawn" you mean digital art, yes? Because it does wonders for actual hand-drawn/painted art.

I also got a hold of some original scans and images of some Sierra backgrounds. I don't know how it works specifically, but I had intended to experiment training a new model based on these scans as compared to the in-game art.

[Phredreeke]

This is my attempt at upscaling that scene (the contrast was a bit off in the screenshot I found that matched the scene, tried to correct it afterwards):

[MrFlibble]

MusicallyInspired, on 28 March 2019 - 08:24 AM, said:

By "hand-drawn" you mean digital art, yes? Because it does wonders for actual hand-drawn/painted art.

Scanned, yes. But for Kyrandia backgrounds for example, which I gather were drawn by hand on a computer, the results are generally not as good as with pre-rendered graphics.

Don't get me wrong, this is miles better than waifu2x, but once the novelty wears off the limitations of ESRGAN become more apparent. Presumably a model specifically trained for upscaling low-res video game graphics would do better?

MusicallyInspired, on 28 March 2019 - 08:24 AM, said:

I also got a hold of some original scans and images of some Sierra backgrounds. I don't know how it works specifically, but I had intended to experiment training a new model based on these scans as compared to the in-game art.

Well, I suppose you would need to match the dimensions of each pair of images exactly for this to work. That might entail some trial-and-error if the full-res origginals were cropped before being scaled down.

BTW, you mentioned that some other people intended to train a model based on manually edited upscales, did anything come out of it?

Phredreeke, on 28 March 2019 - 08:34 AM, said:

This is my attempt at upscaling that scene

Kinda blurry, and the tree bough on the top right clearly retains that staircase pixel effect (also some other areas too but less noticeably).

[Phredreeke]

That's because I used my lightest antialiasing script. Although it's true that results are better when the image is sourced from scans/photography (Blood) or 3D models (Duke)

[MusicallyInspired]

MrFlibble, on 28 March 2019 - 01:00 PM, said:

Scanned, yes. But for Kyrandia backgrounds for example, which I gather were drawn by hand on a computer, the results are generally not as good as with pre-rendered graphics.

Definitely doesn't do well with digital hand-made art, I agree. Stuff like Indy3 VGA, Monkey 1 VGA, Loom VGA, Fate of Atlantis, Kyrandia, etc all produce less than ideal results.

Quote

BTW, you mentioned that some other people intended to train a model based on manually edited upscales, did anything come out of it?

I haven't seen anybody accomplish it yet, no. Which is surprising. I wish I hadn't waited to do it myself like I had planned. A bunch of IRL stuff just came up suddenly and then my PC died.

[MrFlibble]

I just realised one thing. With the method of scaling the source image up with xBR/xBRZ before ESRGAN, you get smooth edges and no pixel staircases exactly where there are none in the respective xBR/xBRZ upscales. Take a took:

Left is an image that was scaled up with xBRZ and then down to the original size with Bicubic before being fed to ESRGAN, and to the right the same image but scaled with xBR.

It is obviously no coincidence that the flag and the back part of the roof are smooth in the xBRZ image and jaggy in the xBR one:


This actually makes sense, because come to think of it, ESRGAN is not a tool to upscale low-resolution images, it's a tool to restore downscaled images to their original size. Or at least, the models are trained to work that way. So when an image is scaled up with xBRZ prior to processing in ESRGAN, it's not to "anti-aliase" or "soften" the input to supposedly make it more similar to a photograph; it actually creates a "mould" for the prospective ESRGAN upscale. I should have guessed this earlier when I noted that sprite outlines produced by waifu2x are very similar to those produced by xBRZ.

This means that the results of ESRGAN upscales are unlikely to be better than what xBR(Z) can do (save for finer lines and such), e.g. xBRZ doesn't do a good job with straight lines at oblique angles, and ESRGAN does not improve that on its own in any way.

My preliminary conclusion from this is that if we want some kind of improvement of ESRGAN results, we need to manipulate an upscaled image and then scale it down with Bicubic interpolation, not try to alter the low-resolution image.

[leilei]

Jedi Knight had a ESRGAN treatment
https://kotaku.com/n...n-ai-1833651613

A lot of the textures are rather noisy and don't look good for the character textures (rather Waifu2x-ish) and have JPEG artifacts coming through possibly from using Jedi Outcast's large amounts of JPEG textures as a scaling model. The better results are on the grainier environment textures.

[Phredreeke]

A few sprites upscaled from Powerslave/Exhumed

[MrFlibble]

Very good! Once again the results look very much like what real 1996 sprites could've been if created at that resolution.

[Phredreeke]

MasonFace just posted the first version of the PVE upscale pack for Daggerfall, which I contributed masks for

https://forums.dfwor...php?f=27&t=1956

[Zaxx]

This Heretic pack looks real nice:
https://www.nexusmod...ods/1?tab=files

[NightFright]

Actually all the neural texture packs look nice:
Doom - Latest version: 0.95
Heretic - Latest version: 0.9
Hexen - Latest version: 1.0.3a

Hexen seems to be the most complete so far. There are a few minor drawbacks (e.g. sprite fixes, widescreen sprites or custom HUDs screw things up), but brightmaps are still working at least. It's the first time I am actually accepting highres content since it keeps the authentic look. Very nice that this is finally possible!

[thelastdisciple]

I think this method was done with Max Payne and the results.... I'm still not sure about it.

Gave off a vibe where much of the textures just ended up looking smudged or something, like it was all done using the smudge tool in photoshop if you get what i mean.

[Phredreeke]

Would there be any interest in a brief guide on how to tackle palswaps?

[MrFlibble]

Yes please.

[Phredreeke]

1. The simplest but least efficient method is to simply apply palswaps to each image before upscaling. You can still save some processing time though by only generating the mask for the sprite once (this obviously does not apply to opaque tiles) and reusing that for each palswap. What I do is put the masks in one folder and the palswapped version in another and use a batch file like this

for %%f in (*.png) do ("C:\Program Files\ImageMagick-7.0.7-Q8\Magick.exe" montage %%f "..\proc-downscalepalette2\%%f" -tile x1 -geometry +0+0 png32:composite\%%f)

Then I use a PSP script which makes a duplicate layer, shifts it by half the image width, and applies that as a mask to the image, and then crop out the other half of the image.

Spoiler

from JascApp import *

def ScriptProperties():
    return {
        'Author': '',
        'Copyright': '',
        'Description': '',
        'Host': 'Paint Shop Pro',
        'Host Version': '8.00'
        }

def Do(Environment):
    App.Do( Environment, 'LayerDuplicate', {
            'GeneralSettings': {
                'ExecutionMode': App.Constants.ExecutionMode.Default, 
                'AutoActionMode': App.Constants.AutoActionMode.Match
                }
            })

    App.Do( Environment, 'SelectLayer', {
            'Path': (0,-1,[],App.Constants.Boolean.false), 
            'GeneralSettings': {
                'ExecutionMode': App.Constants.ExecutionMode.Silent, 
                'AutoActionMode': App.Constants.AutoActionMode.Default
                }
            })

    App.Do( Environment, 'Offset', {
            'FillColor': (0,0,0), 
            'EdgeMode': App.Constants.EdgeMode.Wrap, 
            'HorizontalOffset': App.TargetDocument.Width/2, 
            'VerticalOffset': 0, 
            'Center': App.Constants.Boolean.false, 
            'GeneralSettings': {
                'ExecutionMode': App.Constants.ExecutionMode.Default, 
                'AutoActionMode': App.Constants.AutoActionMode.Match
                }
            })

    App.Do( Environment, 'MaskFromImage', {
            'CreateMaskFrom': App.Constants.CreateMaskFrom.Luminance, 
            'InvertMaskData': App.Constants.Boolean.false, 
            'SourceImage': 0, 
            'GeneralSettings': {
                'ExecutionMode': App.Constants.ExecutionMode.Default, 
                'AutoActionMode': App.Constants.AutoActionMode.Match
                }
            })

    App.Do( Environment, 'SelectLayer', {
            'Path': (1,1,[],App.Constants.Boolean.false), 
            'GeneralSettings': {
                'ExecutionMode': App.Constants.ExecutionMode.Silent, 
                'AutoActionMode': App.Constants.AutoActionMode.Default
                }
            })

    App.Do( Environment, 'DeleteLayer', {
            'Path': None, 
            'MergeMask': App.Constants.Boolean.true, 
            'GeneralSettings': {
                'ExecutionMode': App.Constants.ExecutionMode.Default, 
                'AutoActionMode': App.Constants.AutoActionMode.Match
                }
            })


    App.Do( Environment, 'ResizeCanvas', {
            'AspectRatio': 2, 
            'FillColor': (188,0,160), 
            'HoriPlace': App.Constants.HorizontalType.Left, 
            'MaintainAspect': App.Constants.Boolean.false, 
            'NewDimUnits': App.Constants.UnitsOfMeasure.Pixels, 
            'NewHeight': App.TargetDocument.Height, 
            'NewWidth': App.TargetDocument.Width/2, 
            'VertPlace': App.Constants.VerticalType.Center, 
            'GeneralSettings': {
                'ExecutionMode': App.Constants.ExecutionMode.Default, 
                'AutoActionMode': App.Constants.AutoActionMode.Match
                }
            })

You're still going to need to generate the separate palettes for the palswaps though, which will vary depending on the game. For Duke Nukem 3D (also Redneck Rampage), Shadow Warrior and Blood I'll point you towards Fox's thread where he extracted the raw palette data for other games you're on your own.

2. If you want to make "PLU compatible" upscales though, you're gonna have to start editing palette files. Make two copies of the base palette, in the first one you black out (you could go for other colors depending on how your script works) everything but the section affected. Do the same thing for the other where just the affected section is blacked out or replaced with other colors in the palette. If you're using JASC palette format this can be easily done in a text editor.

Now, make a copy of the image with the first palette (with only the affected parts) applied (while maintaining indexes). Upscale this (for this method you want transparent/unused parts of the image to be black) and apply the palette again (this time with nearest color matching) to the upscale.
Do a regular upscale of the image (it's not necessary to apply a different palette at this point) and post-upscale apply the second palette you made (excluding the affected colors)

Combine the two images using ImageMagick, then make a script that makes a duplicate layer, shifted half the image width, make another duplicate of that layer, apply a threshold effect (you may have to experiment with threshold value as too low and you may get dark fringing or too high and part of the new layer will be missing)

Spoiler

from JascApp import *

def ScriptProperties():
    return {
        'Author': '',
        'Copyright': '',
        'Description': '',
        'Host': 'Paint Shop Pro',
        'Host Version': '8.00'
        }

def Do(Environment):
    App.Do( Environment, 'LayerDuplicate', {
            'GeneralSettings': {
                'ExecutionMode': App.Constants.ExecutionMode.Default, 
                'AutoActionMode': App.Constants.AutoActionMode.Match
                }
            })

    App.Do( Environment, 'Offset', {
            'FillColor': (0,0,0), 
            'EdgeMode': App.Constants.EdgeMode.Wrap, 
            'HorizontalOffset': App.TargetDocument.Width/2, 
            'VerticalOffset': 0, 
            'Center': App.Constants.Boolean.false, 
            'GeneralSettings': {
                'ExecutionMode': App.Constants.ExecutionMode.Default, 
                'AutoActionMode': App.Constants.AutoActionMode.Match
                }
            })

    App.Do( Environment, 'LayerDuplicate', {
            'GeneralSettings': {
                'ExecutionMode': App.Constants.ExecutionMode.Default, 
                'AutoActionMode': App.Constants.AutoActionMode.Match
                }
            })

    App.Do( Environment, 'ColorAdjustThreshold', {
            'Threshold': {
                'Threshold': 1
                }, 
            'GeneralSettings': {
                'ExecutionMode': App.Constants.ExecutionMode.Default, 
                'AutoActionMode': App.Constants.AutoActionMode.Match
                }
            })

    App.Do( Environment, 'SelectLayer', {
            'Path': (0,-1,[],App.Constants.Boolean.false), 
            'GeneralSettings': {
                'ExecutionMode': App.Constants.ExecutionMode.Silent, 
                'AutoActionMode': App.Constants.AutoActionMode.Default
                }
            })

    App.Do( Environment, 'MaskFromImage', {
            'CreateMaskFrom': App.Constants.CreateMaskFrom.Luminance, 
            'InvertMaskData': App.Constants.Boolean.false, 
            'SourceImage': 0, 
            'GeneralSettings': {
                'ExecutionMode': App.Constants.ExecutionMode.Default, 
                'AutoActionMode': App.Constants.AutoActionMode.Match
                }
            })

    App.Do( Environment, 'SelectLayer', {
            'Path': (1,1,[],App.Constants.Boolean.false), 
            'GeneralSettings': {
                'ExecutionMode': App.Constants.ExecutionMode.Silent, 
                'AutoActionMode': App.Constants.AutoActionMode.Default
                }
            })

    App.Do( Environment, 'DeleteLayer', {
            'Path': None, 
            'MergeMask': App.Constants.Boolean.true, 
            'GeneralSettings': {
                'ExecutionMode': App.Constants.ExecutionMode.Default, 
                'AutoActionMode': App.Constants.AutoActionMode.Match
                }
            })

    App.Do( Environment, 'LayerMergeDown', {
            'GeneralSettings': {
                'ExecutionMode': App.Constants.ExecutionMode.Default, 
                'AutoActionMode': App.Constants.AutoActionMode.Match
                }
            })

    App.Do( Environment, 'ResizeCanvas', {
            'AspectRatio': 2, 
            'FillColor': (188,0,160), 
            'HoriPlace': App.Constants.HorizontalType.Left, 
            'MaintainAspect': App.Constants.Boolean.false, 
            'NewDimUnits': App.Constants.UnitsOfMeasure.Pixels, 
            'NewHeight': App.TargetDocument.Height, 
            'NewWidth': App.TargetDocument.Width/2, 
            'VertPlace': App.Constants.VerticalType.Center, 
            'GeneralSettings': {
                'ExecutionMode': App.Constants.ExecutionMode.Default, 
                'AutoActionMode': App.Constants.AutoActionMode.Match
                }
            })
			

3. You can take it one step further, by only using the first upscale above as a mask, while making a separate version of the second upscale with only the first palette applied



My compositing script for the above

Spoiler

afrom JascApp import *

def ScriptProperties():
    return {
        'Author': '',
        'Copyright': '',
        'Description': '',
        'Host': 'Paint Shop Pro',
        'Host Version': '8.00'
        }

def Do(Environment):
    App.Do( Environment, 'LayerDuplicate', {
            'GeneralSettings': {
                'ExecutionMode': App.Constants.ExecutionMode.Default, 
                'AutoActionMode': App.Constants.AutoActionMode.Match
                }
            })

    App.Do( Environment, 'Offset', {
            'FillColor': (0,0,0), 
            'EdgeMode': App.Constants.EdgeMode.Wrap, 
            'HorizontalOffset': -App.TargetDocument.Width/3, 
            'VerticalOffset': 0, 
            'Center': App.Constants.Boolean.false, 
            'GeneralSettings': {
                'ExecutionMode': App.Constants.ExecutionMode.Default, 
                'AutoActionMode': App.Constants.AutoActionMode.Match
                }
            })

    App.Do( Environment, 'LayerDuplicate', {
            'GeneralSettings': {
                'ExecutionMode': App.Constants.ExecutionMode.Default, 
                'AutoActionMode': App.Constants.AutoActionMode.Match
                }
            })

    App.Do( Environment, 'Offset', {
            'FillColor': (0,0,0), 
            'EdgeMode': App.Constants.EdgeMode.Wrap, 
            'HorizontalOffset': -App.TargetDocument.Width/3, 
            'VerticalOffset': 0, 
            'Center': App.Constants.Boolean.false, 
            'GeneralSettings': {
                'ExecutionMode': App.Constants.ExecutionMode.Default, 
                'AutoActionMode': App.Constants.AutoActionMode.Match
                }
            })

    App.Do( Environment, 'SelectLayer', {
            'Path': (0,-1,[],App.Constants.Boolean.false), 
            'GeneralSettings': {
                'ExecutionMode': App.Constants.ExecutionMode.Silent, 
                'AutoActionMode': App.Constants.AutoActionMode.Default
                }
            })



    App.Do( Environment, 'MaskFromImage', {
            'CreateMaskFrom': App.Constants.CreateMaskFrom.Luminance, 
            'InvertMaskData': App.Constants.Boolean.false, 
            'SourceImage': 0, 
            'GeneralSettings': {
                'ExecutionMode': App.Constants.ExecutionMode.Default, 
                'AutoActionMode': App.Constants.AutoActionMode.Match
                }
            })

    App.Do( Environment, 'SelectLayer', {
            'Path': (1,1,[],App.Constants.Boolean.false), 
            'GeneralSettings': {
                'ExecutionMode': App.Constants.ExecutionMode.Silent, 
                'AutoActionMode': App.Constants.AutoActionMode.Default
                }
            })

    App.Do( Environment, 'DeleteLayer', {
            'Path': None, 
            'MergeMask': App.Constants.Boolean.true, 
            'GeneralSettings': {
                'ExecutionMode': App.Constants.ExecutionMode.Default, 
                'AutoActionMode': App.Constants.AutoActionMode.Match
                }
            })

    App.Do( Environment, 'LayerMergeDown', {
            'GeneralSettings': {
                'ExecutionMode': App.Constants.ExecutionMode.Default, 
                'AutoActionMode': App.Constants.AutoActionMode.Match
                }
            })


    App.Do( Environment, 'ResizeCanvas', {
            'AspectRatio': 2, 
            'FillColor': (188,0,160), 
            'HoriPlace': App.Constants.HorizontalType.Left, 
            'MaintainAspect': App.Constants.Boolean.false, 
            'NewDimUnits': App.Constants.UnitsOfMeasure.Pixels, 
            'NewHeight': App.TargetDocument.Height, 
            'NewWidth': App.TargetDocument.Width/3, 
            'VertPlace': App.Constants.VerticalType.Center, 
            'GeneralSettings': {
                'ExecutionMode': App.Constants.ExecutionMode.Default, 
                'AutoActionMode': App.Constants.AutoActionMode.Match
                }
            })
			

This avoids the dark fringes you might get with a too low threshold value in the prior method, while at the same time providing more coverage.



As of right now, "PLU compatible" upscales are just a theoretical exercise as there is currently no way to use them in game. However it has the potential to solve a lot of issues with current upscales.

[MrFlibble]

A quick test with Strife sprites:


It came out half as horrible as I expected it to be, but not from the first try of course!

First off, I blended the neural upscale result wit a straight 4BRZ upscale. The image I fed to ESRGAN was slightly sharpened and then pre-processed with the two-pass xBR/xBRZ method (it blurs the image so prior sharpening improves the result a bit):

• source image loaded in GIMP, scaled up to 4x with nearest neighbour and then back down to original resolution with Sinc. This has the effect of slight sharpening the image - the following steps will introduce noticeable blur
  
• load the image in xBRZ Scaler Test and scale to 4x with xBR. Save the result, open in GIMP, apply Gaussian blur at 1.0 pixel radius and scale it down to the original size with Sinc
  
• scale up the result to 4x with xBRZ this time. In GIMP, apply Gaussian blur at 1.0 pixel radius and scale back down to the original size with Sinc

The image pre-processed in such way is then scaled up with ESRGAN using an interpolation of the Manga109Attempt model and the PSNR model at alpha = 0.8. (To create this model open the net_interp.py script that comes with ESRGAN and replace the default ESRGAN model with Manga109Attempt).

The ESRGAN output image is then blended with a simple (no two-pass) xBRZ scale of the source (unsharpened) image using G'MIC's Layer -> Blend [median] function in GIMP.

To get the mask, I apply Kuwahara-Nagao blur at 2 pixel radius (no detail protection) to the ESRGAN upscale, then convert to the original palette. The purple background in the resulting image is the mask which is pasted on the ESRGAN/xBRZ blend.

[MusicallyInspired]

Aside from that one dude's nose turning into a gaping hole and erasing his actual mouth, that's not too bad at all!

[MrFlibble]

MusicallyInspired, on 19 April 2019 - 08:58 AM, said:

Aside from that one dude's nose turning into a gaping hole and erasing his actual mouth, that's not too bad at all!

The current result is way better than what I had achieved previously, which had the faces mangled to an even greater extent.

BTW, I forgot to mention that during the two-pass xBR/xBRZ procedure the scaled-up image should be blurred wit Gaussian blur at 1.0 pixel radius before scaling it back down each time. I edited the previous post to insert this info.

[Phredreeke]

Upscale of Doom II's Icon of Sin I made some days ago



Also here's an upscaled Cacodemon I made using my combined XBRZ-ESRGAN hybrid method

[MrFlibble]

Testing the multipass/blend approach described above, this time with three passes (one xBR and two xBRZ):



Comparison with original low-res images (corrected to 640x480) here.

[leilei]

XBRZ is old and can be done real-time in shaders pretty fast though. It's nowhere near as interesting as results with neural scalers (if there is any esrganning in those last 3 pictures, it's eaten off by the brz too much to be visible).