[Hendricks266]

icecoldduke, on 05 May 2016 - 03:27 PM, said:

Would there be a good reason to want to load PNG's over precompiled assets during level load?

If you want to give your users the option to run without texture compression. Or, if you have textures that must not be compressed because they become illegible in some way. (This is the reason for the existence of the nocompress and nodownsize def tokens.)

icecoldduke, on 05 May 2016 - 03:27 PM, said:

The good thing with precompiled assets is they are memory ready. You just grab the data blob, decompress, copy to vram. No processing necessary.

Unless you're running on a big-endian architecture. (Which still isn't a lot of processing.)

icecoldduke, on 05 May 2016 - 03:27 PM, said:

I'm thinking developers that want seemless level generation they can have some kind of sector that would kick off the loading of the next map. In old maps that have episodes setup, during the score screen at the end of the map, we can start loading the next map, and have it ready by the time the user skips through the score screen.

Yes, those are some of the "several things" I alluded to. They apply to PNGs in addition to precompiled stuff.

icecoldduke, on 05 May 2016 - 03:27 PM, said:

Unfortunately both of these ideas would mean we have to clean up all the nasty ass global variables that is used by the map loading code, and throw all of that into a class.

Unless PolymerNG does stuff specific to an individual level's geometry, not necessarily. We could always do a pass through the structures to pick out tilenums without reading the entire maps into RAM. The assets could be loaded in the background (in a separate thread even, except for uploading the textures to GL) as long as the map loading code is smart enough not to throw away everything from the previous map and start over.

[icecoldduke]

Hendricks266, on 05 May 2016 - 03:37 PM, said:

If you want to give your users the option to run without texture compression. Or, if you have textures that must not be compressed because they become illegible in some way. (This is the reason for the existence of the nocompress and nodownsize def tokens.)

Do we have a example of textures that get trashed by compression? Are these artifacts bad enough to warrant the need to use uncompressed data? I also want to say I'm using DevIL to load and convert the images to DXT(this means you guys basically use whatever image format you want). I'm not using Ken's code. I'm also thinking of implementing YCoCg using the source code id released awhile back(non gpl version).

Hendricks266, on 05 May 2016 - 03:37 PM, said:

Unless you're running on a big-endian architecture. (Which still isn't a lot of processing.)

I thought x64 CPU's were little endian by default. Are there big endian x64 chips?

Hendricks266, on 05 May 2016 - 03:37 PM, said:

Unless PolymerNG does stuff specific to an individual level's geometry, not necessarily. We could always do a pass through the structures to pick out tilenums without reading the entire maps into RAM. The assets could be loaded in the background (in a separate thread even, except for uploading the textures to GL) as long as the map loading code is smart enough not to throw away everything from the previous map and start over.

That's what I was thinking we could just preload all the assets for the next map, then let the engine do the rest of its work.

[Hendricks266]

icecoldduke, on 05 May 2016 - 03:57 PM, said:

Do we have a example of textures that get trashed by compression? Are these artifacts bad enough to warrant the need to use uncompressed data?

I don't have an example, but such cases absolutely exist. Also, pixel art.

icecoldduke, on 05 May 2016 - 03:57 PM, said:

I also want to say I'm using DevIL to load and convert the png's to DXT. I was thinking of using id's YCoCg conversion code they posted online awhile back, and support YCoCg DXT compressed diffuse images. That should fix most if not all the DXT compression issues.

kplib should be sufficient for decoding PNGs. I can't speak to DXT compression.

icecoldduke, on 05 May 2016 - 03:57 PM, said:

I thought x64 CPU's were little endian by default. Are there big endian x64 chips?

Nothing x86 is big endian. Most ARM in the wild is configured to be little endian. Not much uses big endian any more, but that's not a reason to let our support rot.

[icecoldduke]

Hendricks266, on 05 May 2016 - 04:02 PM, said:

I don't have an example, but such cases absolutely exist. Also, pixel art.

In my opinion the artifacts have to be pretty bad to warrant usage of non compressed assets. I'm aware that DXT compression issues exist but I'm also aware that people abuse that flag for no good reason. Look at the guy that had 4gb of loaded assets at once and was complaining cause he was running out memory .

I think his answer to the problem was to increase his virtual memory, rather then downsize his content. :/ If I enable uncompressed images, and the mod goes over a certain amount of ram, those are the first images that will be magenta'ed.

Hendricks266, on 05 May 2016 - 04:02 PM, said:

Nothing x86 is big endian. Most ARM in the wild is configured to be little endian. Not much uses big endian any more, but that's not a reason to let our support rot.

Sure; as far I'm concerned though this is a down the road issue, and something easy to implement before it goes back to main.

[Hendricks266]

icecoldduke, on 05 May 2016 - 04:18 PM, said:

Sure; as far I'm concerned though this is a down the road issue, and something easy to implement before it goes back to main.

Famous last words.

It can't be too hard to throw B_LITTLE32 in all the right places.

[icecoldduke]

Hendricks266, on 05 May 2016 - 04:21 PM, said:

Famous last words.

It can't be too hard to throw B_LITTLE32 in all the right places.

It's added as a P2 bug in my bug database. I can add B_LITTLE32 to the game.payloads load code easily enough, since I would not have to swap endian on the payloads themselves, only the headers at the start of the file that describe each payload. On the other hand the "mega-mesh" file we would have to iterate over every piece of data to swap its endianess, which means it will be slower. I'd like to profile this before I implement it.

If I'm going to support something, I want it to be fast .

[Hendricks266]

icecoldduke, on 05 May 2016 - 04:25 PM, said:

On the other hand the "mega-mesh" file we would have to iterate over every piece of data to swap its endianess, which means it will be slower.

In a case like that you would check #if B_BIG_ENDIAN == 1 before performing the looping.

[icecoldduke]

Hendricks266, on 05 May 2016 - 04:27 PM, said:

In a case like that you would check #if B_BIG_ENDIAN == 1 before performing the looping.

It will be in in the next PolymerNG milestone .

[Mblackwell]

I've definitely had issues with compression in the past causing things like illegible text or smooth fades turning into gradients. I don't have any ready examples however.

[icecoldduke]

Mblackwell, on 05 May 2016 - 05:52 PM, said:

I've definitely had issues with compression in the past causing things like illegible text or smooth fades turning into gradients. I don't have any ready examples however.

Text I could understand. I should have said we talking about world textures only.

[icecoldduke]

Is there a way to set the FOV for the software renderer?

[Hendricks266]

icecoldduke, on 05 May 2016 - 05:59 PM, said:

Text I could understand. I should have said we talking about world textures only.

Text on world textures.

icecoldduke, on 05 May 2016 - 06:07 PM, said:

Is there a way to set the FOV for the software renderer?

All I can think of is the world scale, which I don't know how to change. It probably isn't possible to change the FOV.

[icecoldduke]

Hendricks266, on 05 May 2016 - 06:10 PM, said:

Text on world textures.

There could be errors on text in the world, but is it going to be bad enough for the average player to notice as he is playing through the map? Players don't usually get ass up on images and start looking for errors .

Hendricks266, on 05 May 2016 - 06:10 PM, said:

All I can think of is the world scale, which I don't know how to change. It probably isn't possible to change the FOV.

Damn.

[Hendricks266]

icecoldduke, on 05 May 2016 - 06:17 PM, said:

There could be errors on text in the world, but is it going to be bad enough for the average player to notice as he is playing through the map? Players don't usually get ass up on images and start looking for errors .

They might in a game where fine details on textures is important to the gameplay.

[icecoldduke]

Hendricks266, on 05 May 2016 - 06:18 PM, said:

They might in a game where fine details on textures is important to the gameplay.

Damn. I forgot a valid use case in build is putting letters on walls. Fuck. Then how do we prevent uncompressed textures from being abused?

[Hendricks266]

Document the matter well.

[icecoldduke]

Hendricks266, on 05 May 2016 - 06:22 PM, said:

Document the matter well.

Maybe ill limit the amount of uncompressed textures that can visible at once? You exceed the max limit, your shit goes magenta .

[Hendricks266]

Nothing like that is acceptable in mainline.

Suggestion: Detect when foreseeable problems are the cause of actual slowdown, and report them to the log file. We can augment this feature to use stricter thresholds through a cvar to simulate running on a weaker machine than a dev may have. And if a dev is lazy about performance, then when users complain about slowdown we can tell them to paste their log just like in the case of crashes, which has worked out very well.

[icecoldduke]

Hendricks266, on 05 May 2016 - 06:27 PM, said:

Nothing like that is acceptable in mainline.

Suggestion: Detect when foreseeable problems are the cause of actual slowdown, and report them to the log file. We can augment this feature to use stricter thresholds through a cvar to simulate running on a weaker machine than a dev may have. And if a dev is lazy about performance, then when users complain about slowdown we can tell them to paste their log just like in the case of crashes, which has worked out very well.

This is a longer conversation to be had and I feel I need to be less tired to have it properly . In my experience its easier to force content creators to do things properly from the start of the project; and the best way to do that is to make your shit render a nice magenta, when the engine detects stupid assets. Then give them proper feedback on why I choose to render there hard work a nice magenta. The plan is to come up with proper metrics to ensure your mod runs on the min spec configuration(in this case a Xbox One), then when I detect your game will cause framerate drops on minspec, you shit renders a magenta until you fix the problem.

A lot of content developers will stop maintaining there mods after they release. If we choose to go the log only route, content developers will intentionally ignore warnings, and try to push there renderer as much as they can on there own box and release a fucking mess. You can argue in principle it will be the content developers fault they made fucked up content. What happens after the author stops maintaining the mod? There will be cries for engineering to go in and fix the framerate issues. If I choose not too, people will talk shit about the renderer because they don't know any better. Then if that type of thing happens enough, I for see a cascade of nasty shit happening from then on out.

I really want to force the magenta texture when I detect performance issues/bad assets. This would only be in development mode. I would really like you and Richard to be on board with this. I'm not asking anyone to make a decision now, but could we post pone this conversation for a later date?

[TerminX]

It's silly to see you guys argue about this again. What ICD proposes makes perfect sense (hence the reason you've personally expended so much effort moving to this setup for HTTKC), but I think loading .pngs from disk is still a necessary feature for backward compatibility. It's not optimal for shipped content, but being able to just throw a map together and add your own assets without any kind of pre-processing has long been one of the mainstay features of the engine. I don't think it's too much for people to ask that the code used to build asset packages offline also be included in the main binary--there is no reason the engine can't regenerate missing precompiled assets from source materials itself if the materials are available.

If we had static maps (.bsp, etc) and game code that sat in a .dll that needed to be compiled separately, I would agree that loading .png directly isn't strictly necessary. However, this isn't the case and I don't believe there is a good reason to disrupt existing workflows, nor do I want to field support requests from people who have trouble figuring out the asset compilation when all they want to do is play some old mods.

[icecoldduke]

TerminX, on 05 May 2016 - 06:54 PM, said:

It's silly to see you guys argue about this again. What ICD proposes makes perfect sense (hence the reason you've personally expended so much effort moving to this setup for HTTKC), but I think loading .pngs from disk is still a necessary feature for backward compatibility. It's not optimal for shipped content, but being able to just throw a map together and add your own assets without any kind of pre-processing has long been one of the mainstay features of the engine. I don't think it's too much for people to ask that the code used to build asset packages offline also be included in the main binary--there is no reason the engine can't regenerate missing precompiled assets from source materials itself if the materials are available.

If we had static maps (.bsp, etc) and game code that sat in a .dll that needed to be compiled separately, I would agree that loading .png directly isn't strictly necessary. However, this isn't the case and I don't believe there is a good reason to disrupt existing workflows, nor do I want to field support requests from people who have trouble figuring out the asset compilation when all they want to do is play some old mods.

Trooper Dan brought up a good idea a few posts up. Old mods would load in the "development" mode. The engine would detect the assets haven't been built, and would build the assets for the mod without any user intervention. This would be similar behavior to how the engine works now, except the engine would launch the offline tool to build assets, and this would happen on startup, rather then as the level loads. Would this type of thing be acceptable? The asset builder will only build modified assets, between builds you will not have to recompile everything.

[Mblackwell]

About rendering magenta:
Not everyone is going to target Xbox One hardware. Most games that run poorly on Xbox One run quite well on an i3 and a GTX 750Ti.

Which is why I advocate performance metrics over just making shit break.

[TerminX]

I'm in favor of performance metrics instead of making shit break too. Implement various stats and counters similar to the things Unreal keeps track of and display an embedded icon representing "slow" if any of them exceed what we think are sane limits.

[Danukem]

Mblackwell, on 05 May 2016 - 08:03 PM, said:

About rendering magenta:
Not everyone is going to target Xbox One hardware. Most games that run poorly on Xbox One run quite well on an i3 and a GTX 750Ti.

Which is why I advocate performance metrics over just making shit break.

Sure, and the content maker with that nice hardware will release his project to users who don't have nice hardware, the project will run like crap for those users and the content maker and users will all blame the engineers. At least that is what I'm sure icecoldduke would say (since he has already said it).

Maybe a compromise: Render magenta in problem cases by default, but have a cvar that can disable it.

[icecoldduke]

Trooper Dan, on 05 May 2016 - 08:15 PM, said:

Maybe a compromise: Render magenta in problem cases by default, but have a cvar that can disable it.

Then developers will just run with it off, I know how you guys are . It has to be forced on in order for people deal with it. You guys will be able to make some awesome stuff with the renderer, but I don't want you guys doing bad things that cause problems. Anyway this is a down the road issue, I hope to revisit at a later time with TX.

TerminX, on 05 May 2016 - 08:14 PM, said:

I'm in favor of performance metrics instead of making shit break too. Implement various stats and counters similar to the things Unreal keeps track of and display an embedded icon representing "slow" if any of them exceed what we think are sane limits.

Its possible I can live with a icon, well see. I have more work to do before we get at this point anyway.

[Mblackwell]

Trooper Dan, on 05 May 2016 - 08:15 PM, said:

Sure, and the content maker with that nice hardware will release his project to users who don't have nice hardware, the project will run like crap for those users

That's why when you release something you tell people what the minimum and recommended specifications are. And if they fall below that, caveat emptor.

[Hendricks266]

The number of silly worst-case hypotheticals in this thread is too damn high.

Trust developers.

Don't break stuff arbitrarily.

Share plenty of information in the log.

Lossless formats exist for a reason.

[Tea Monster]

With all due respect to the fact that people have lives and are doing this for free - very little progress has been made on optimizing Polymer or fixing any of the existing bugs and shading issues in the renderer.
Most problems with EDuke get fixed with glacial slowness.

Again, I understand that people have lives and are doing this for free - but if after 6 or so years your way hasn't shown any progress - let someone else have a try.

At the very least - stop publicly sniping at him if he has stepped forward and offered his time and knowledge to help the project.

[Mblackwell]

This thread is for discussing what might be done on a new renderer and engine features to create the best possible outcome for everyone.

Icecoldduke has asked that meta discussion be taken to PMs.

[MusicallyInspired]

I tried compression on textures once....turned it off and never looked back. Made everything look like trash. Noticeably.