[Helixhorned]

Esteemed fellow Dukers,


I'm pretty happy to announce a preview release of Lunatic, a Lua-based scripting system for EDuke32 and more. Lunatic has been in development since September 2011, being conceived in an IRC chat about how certain internal aspects of CON may be improved. But given the starting point of embedding LuaJIT into EDuke32 made these ideas outgrow in scope, so that pretty much every time I said something along the lines of "feature X is for the future", this was the reason. In fact, I alluded to Lunatic as far back as August 2011!
I'll return to a bit of history, but for now, onwards to the matter!


What the heck is Lunatic?

From a modder's perscpective, Lunatic is an interface to EDuke32 that allows pretty much what currently CON is for -- customizing actor behavior or changing the behavior at certain events -- but in a fashion that flows naturally with the Lua way of doing things. So, the emphasis is on redesign of these interfaces, not merely one-to-one duplication, which would be rather fruitless. For example, functions dealing with positions of game objects usually accept it as a single argument, expecting it to be indexable with "x", "y" and "z". This makes the code look somewhat more readable than passing the components separately.

From our (the EDuke32 developers') perspective, Lunatic significantly eases development of exposing EDuke32 features to scripting, but also adds new ways of extending EDuke32 itself. As an example, the Lunatic build comes with a new map format, or rather, two of them. As represented in memory, it has been extended with separate fields for the TROR members, so the collisions with the members these data were previously shoehorned into are gone. On disk, any map containing TROR is now saved in a completely new, text-based format ("map-text"; loading V9 maps is still supported). The genius of this is that external and internal representations are completely decoupled. We may add features to the map format and trivially keep backward compatibility, just by providing sensible default values for new members. Also, in maps of the new format, proper drawing of walls with non-power-of-two ysize tiles is enabled unconditionally. It was previously not possible to enable it because existing maps, among others the original Duke3D ones, relied on them being drawn as they are.

Modularity

Lunatic is conceived with modularity in mind from the start. A Lua file that is loaded into EDuke32 is effectively an encapsulated entity: game variables are defined per-module (and may even be invisible to the outside), so that there is no potential for name clashes among them. It may "require" other modules using a mechanism that is similar to Lua 5.1's system, but also allows arguments being passed to modules. So for instance, you could create badass actor code, but give the user of your module various customization options, or allow to set its tile number to a user-defined value. The Lunatic build of EDuke32 allows to load several root Lua files, enabling to write "mutators" that change only certain aspects of the game. (This is not unlike CON mutators loadable with -mx, but in effect, those are still merely textually appended to the main CON code.)

The focus on writing separate Lua files for independent functionality means that modders can adopt a workflow typically found in free software / open source projects. Instead of taking and adapting existing code for one's needs, a module always has an "upstream" maintainer, to which users can contribute features and bugfixes. Thus, improvements to that module propagate back to every user, and modders have more time caring about the artistic side of their mod. In the long run, it is desirable to have a repository of projects that are useful on their own, preferably backed by a revision control system.


About Lua

Lua is an extremely likeable dynamically typed, general purpose language and doesn't really need an introduction, having been used in various games such as The Witcher and a well-known MMORPG. It lends primarily to an imperative programming style, but borrows concepts from other paradigms, such as object-oriented or functional programming. One of its virtues is being light-weight: rather than prividing a full-blown array of features or data structures, it gives a few constructions with which many concepts may be realized if desired. An outstanding feature of Lua is that it comes with only one composite data type -- the so-called _table_, which is really an associative container type. In other words, tables can be indexed and can contain pretty much everything one wishes, and can be used to implement common data structures such as stacks or queues.
Lua uses a single numeric data type throughout, double-precision floating point. Thus, you have both the ability of writing clear code involving geometrical calculations, as well as the full range of a 32-bit signed integer (and more).

While Lua is the language visible to a modification author using Lunatic, under the hood EDuke32 uses LuaJIT, which is nothing short of Lua on a wild mix of steroids. LuaJIT is on one hand a tracing just-in-time compiler for the Lua language. This means that code (which is first present as bytecode getting interpreted) which is run frequently and contributes to a significant portion of the run time ("hot code") is compiled to _native machine code_ on-the-fly. But beyond that, LuaJIT gives us (the devs) a foreign function interface (FFI) to access data and call functions on the C side from Lua, which is the primary reason for the aforementioned easing of the scripting system development.


LunaCON -- a reworked CON implementation

Lunatic completely reimplements the CON run-time as well as translation facilities. It parses CON with a Lua module written completely from scratch that uses the invaluable LPeg library. All CON code is translated to Lua code that could in principle be run like a stand-alone Lunatic module, with the exception that it uses private interfaces, and for this and other reasons must not be used as a base for new Lunatic code.

The main aim of LunaCON (the name given collectively to the CON to Lua translator as well as the underlying facilities) is correctness, in a broad sense. First, because the translator has a clear syntactic model of the CON language instead of being processed by mostly hand-written C code, it lets only well-formed CON code pass: glitches such as the parser wrongly accepting an unfinished directive at the end of a file (like "gamevar 0") are a thing of the past. It is delightfully strict in the semantics department, too; many constructs that would previously pass but have no sensible meaning (such as using getactorvar on a per-player gamevar) give errors. Occasionally, (unintentional) misuse of the CON system was found out to be too common when testing various released mods. For these cases, compatibility switches have been added that e.g. turn errors into warnings, and generating code that may or may not make sense.

On the run-time side, since the LunaCON implementation is completely unrelated to C-CON, one can expect many quirks^[1] of the latter to be non-existent. In some places, the syntax or behavior may be augmented: for instance, LunaCON permits nested array expressions and allows redefinequote to be issued with a previously unallocated quote number (one for which no definequote was given).

^[1] See for example: http://forums.duke4....post__p__143260 and http://forums.duke4....post__p__148019 . [The forums didn't render these correctly as hyperlink.]

Compatibility with existing mods

The LunaCON system would be pointless if it wasn't able to run a good portion of CON code found "in the wild". But while compatibility is one of the main aims, the emphasis on correctness means that LunaCON is also unfogiving with errors in the CON code -- whenever Lunatic encounters an erroneous condition (such as an attempt to access an array beyond its bounds), it will display an error message persistently on the screen; furthermore the code following the error will not be executed. This means that mods and TCs will need to undergo reviews and the respective bugs fixed -- making the code much more cleaner in the process, a most desired and undervalued thing!


Documentation

An essential part of every software project is accurate and useful documentation. One of its main purposes is to set a scope of features that users can rely on -- namely everything described in it. By implication, this means that everything the documentation does not mention must be considered nonexistent when assuming the role of a user. This is really the only way to achieve robust, future-proof software.

Lunatic comes with documentation of Application User Interface (API) as well as an overview of LunaCON. The goal of the former is an accurate description of everything that the system provides. But as so often, it seriously lags behind the development. It is critical (and I probably sound like a broken CD player already) that only that which is written down in the documentation is assumed to be ready for use.^[2] Accuracy means that the important implications of using an interface should be inferrable from reading its documentation, as opposed to resorting to trial and error. Many parts should be familiar -- after all, in a sense the same cake (EDuke32) wrapped in a different icing.

Currently, the documentation can be downloaded in HTML form at the end this post.

^[2]For example, there's no documentation on player[].visibility yet, although you might find out that a member with such a name exists. But maybe I'll want to rename it to "visfactor" for easier grepping, and because it has different semantics (the see-to-infinity value is 0, not 240).

Undocumented features

Besides the lack of documentation for features that expose game functionality, some helper modules have not yet been described either. This is partially due to uncertainty whether they're very useful, although sometimes they have been inspired by forum discussion; I'd be glad to hear any feedback. They are the following:

• A module implementing objects that return pseudo-random numbers using a generator with better statistical properties than the engine's krand(). They can be initialized either to a fixed initial state (for reproducibility) or to a pseudo-random one by using the CRC32 of values returned from the high-precision timer.
  
• A module that effectively provides arrays of boolean values with a storage requirement of one bit per value (plus constant overhead).
  
• A module implementing objects collecting simple running statistics, largely useful for profiling. "Running" means that the statistics (mean, variance) are updated each time a new value is added, without the need to store all values.

Preview version and compatibility considerations

This release is a preview release. As such, it is not targeted toward the general audience, but rather at modders wishing to get familiar with Lunatic or people interested in the development of EDuke32 itself. Currently, Lunatic does not have the breadth of interface features that could replace CON, although this is the planned direction. The decision to release a not fully functional version has various reasons.

First, everyone interested can start getting aquainted with Lua and the interfaces provided by Lunatic right now. Some features like setting actor behavior properties can be considered finalized. Others are implemented but lack documentation (forbidding any careful programmer to use them). Many features or interface exposures have not even been started, but that can be seen from the positive side: everyone is welcome to contribute ideas. The goal is nothing less than to have the best possible interface to EDuke32 features, be it engine, sound, video or game ones. But good interfaces have to be justified with actual use cases, which is where your feedback is valued.

But for the time being, I suggest the primary focus be the maintenance of existing and work-in-progress CON modifications. You'd be surprised at how many things can go wrong if you take them under close scrutiny. I have tested various CON mods of mainly modern times while developing LunaCON, and many have accumulated patches that fix translation with LunaCON. Sometimes, but not to the same extent, I also played these mods to find out what kind of errors they contain (but also to fix bugs in LunaCON and drive its development forward, of course). If you're a mod author, feel free to request these patches -- I'll give them away without asking. But if you want to get the most information gain, you might consider trying fixing the issues on your own (and discussing it on the board, which is highly encouraged).

A good point to start would be to play around with the stand-alone LunaCON translator. I have compiled a Windows version of LuaJIT and LPeg, linked from my signature, that can be used with the "lunacon.lua" script obtainable from the EDuke32 SVN repository. Further directions are given in the LunaCON documentation.

As yet, there is no synthesis build of Lunatic, but brief instructions for compiling one can be found in source/lunatic/doc/how_to_build_lunatic.txt in the EDuke32 SVN repository.


A bit of history and acknowlegdements

The idea to use LuaJIT was put forward by Plagman in an IRC session. I'm sure he'll chime in to tell about his point of view. One of his visions was that development using the scripting system should be more like programming a Build game from scratch, rather than bending into shape hard-coded behavior to achieve one's goals. It's an ideal I agree with, but realistically, it's a huge undertaking to "exorcise" the hard-wired code to the Lua side. In that IRC session, Plagman figuratively described adding more features to CON as "putting lipstick on a dead horse", and this basically was the igniting point for Lunatic -- I simply couldn't agree more.

The idea to have a textual format for map representation is due to TerminX. He got through my thick skull that this does not exclude having an in-memory representation. I was initially pretty confused about that point.

A person I'm unsure wants to be called by name contacted me in the course of Lunatic development and was eager to send patches. However, I had to reject those because I had a different direction in mind. So, contribution is welcome, but please coordinate it with me.

Finally, the Lunatic project would not have been possible without the work of various people on the systems that it builds on.

• The Lua maintainers at PUC-Rio deserve credit for creating, and importantly, keeping alive and evolving, a very well-thought out language.
  
• Without LuaJIT, Lunatic could not have been implemented in such a convenient way. Mike Pall, the author and maintainer of LuaJIT, has engineered an extraordinary piece of software. He was also very responsive with bug reports.
  
• The LPeg module for Lua, by Roberto Ierusalimschy, made it possible to write a CON parser in a natural way. While it's thinkable to cope with such a task using Lua's string functionality, the result would have been rather ad hoc.

Closing remarks

This brief introduction gave an overview of what Lunatic will have to offer. In a fairy world, the limit would be imagination. Realistically, it is time. I can't say for sure when it will reach a state that could be called "release", but I hope that your feedback will guide its development to the right direction. So, take a look and discuss!

---

UPDATE 2012-08-23: The documentation is now available at http://lunatic.eduke32.com/ .

[Micky C]

Great work Helix! Even as a non-coder who only semi understands what you wrote, this sounds like exciting stuff. I know I keep saying this but for various reasons hit a roadblock, I'm still going to try learning scripting for Duke, and I'm taking a second coding course this semester at uni which will hopefully help.

The new map format also sounds fantastic.

Quote

the Lunatic build comes with a new map format, or rather, two of them. As represented in memory, it has been extended with separate fields for the TROR members, so the collisions with the members these data were previously shoehorned into are gone.

I take this to mean that you can now fully pan the textures on TROR surfaces?

Quote

We may add features to the map format and trivially keep backward compatibility, just by providing sensible default values for new members.

I know you're going to be focusing your time on the development of lunatic and lunaCON, but does this mean down the road it will be easy to add those 1:n redwalls for TROR? If it does what I think it does, it will make TROR vastly more flexible and efficient under certain circumstances.

[DavoX]

Freaking awesome Helix, Plagman and TerminX!! I'm no coder but I'm sure this will be really useful for the CON modders

[Helixhorned]

Mikko C, on 21 July 2013 - 03:52 PM, said:

I take this to mean that you can now fully pan the textures on TROR surfaces?

Yes, exactly.

Quote

I know you're going to be focusing your time on the development of lunatic and lunaCON, but does this mean down the road it will be easy to add those 1:n redwalls for TROR? If it does what I think it does, it will make TROR vastly more flexible and efficient under certain circumstances.

There will be a road, but it will not be easy.

[TerminX]

This is certainly one of the most important of all EDuke32 advancements. I look forward to the day when Lunatic builds are the normal ones and the crusty old CON code is in the #ifdef instead.

[Tea Monster]

Drek and I had some crazy ideas that you could maybe create a map exporter for Blender, which would allow you to create much more visually appealing (read: less blocky) maps in a 3D application, then export them as ready-to-play map files. Would this new text-based map format and the addition of LUA scripting somehow make this possible?

Before you think I'm too crazy, one already exists for exporting Quake maps.

[Jblade]

No that won't ever be possible, sectors is pretty much tied to the core of Duke's code and Build engine or whatever. There's a fundamentally huge difference between Quake and Duke which is why you see things like that for Quake engine games and not 2.5D ones.

[Micky C]

I don't really see what the point of it is either, even if the polygons were somehow converted to sectors. Mapster32 is already pretty fast to work in, and is pretty easy to learn once you've got the key combinations sorted out more or less. It's not like mapster32 mappers are going to learn to use blender, and for newcomers, they might as well just pick up mapster. Given a perfect conversion there'd still be texturing/gameplay/sector tagging to do as well.

The only thing it might be remotely good for is terrain generation, however that gets very complicated very fast seeing as sectors cannot have arbitrary slopes but must slope perpendicular to a given wall. Remotely complex smooth terrains for example would use a megaton of walls given the number of triangles needed (even then I'm not sure if all terrain is possible). Given the effort it's best done by hand anyway.

[Tea Monster]

The point would be that you can't make really complex map geometry in Mapster. Any attempt to make anything like a convincing rockscape or terrain is doomed to fail.

Making anything like this:



in mapster is impossible. Yes, you can make it in max and import it, but keeping track of all the different bits and having to texture them separately and keep track of the assets and their textures is a nightmare. If you are having a new map format, why not try to do something that we can use in the future? Why do we have to stick to the old map format? Both Polymer and Polymost use a 'true' 3D space, even if the maps are based on a 2.5D setup. Why not have an option to get rid of the old restrictions for the new map formats?
It would certainly make TROR a lot simpler to implement - make your 3d geometry, then render it.

Its not as if we are going to get anything else soon. Unless someone hacks the editor of Duke Nukem Forever, we are going to be using BUILD for a loooonnnnnngggg time. Might as well make it as future proof as possible.

[Jblade]

Quote

Why do we have to stick to the old map format?

Because, as I said, it's intrinsicly part of how the engine and the game works. You'd have to rewrite the game logic to not deal with sectors and walls - I don't know how much work that is but someone who has worked with the source code can probably explain it in more detail.

Maybe with enough effort, you COULD re-write it completely (probably the same amount of effort it would take to remove the sector/wall/sprite limit as well, too much for any of the Eduke32 to take with the limited time they have) - but who could actually use that, here in this community? You, maybe a couple of others? That's a ridiculous amount of work for something only a handful of people will use. This is a huge problem with trying to compete with modern games, that they require a huge amount of time and dedication to do which unpaid people simply can't offer on a reasonable timescale. I can appreciate wanting a modern grahical duke but we got that in DNF, like it or not. Trying to convert a fundanmentally 20 year old engine to accept 3D maps is just not worth the effort.

tl;dr version - you're not going to get modern graphical fidelity out of Build, no matter how many shaders or polygons you use. Even if you could do that, there's a million other milestones before it even approaches acceptable levels of a modern 3d engine. proper AI, ragdolls, skeletons, actual model clipping, a proper sound engine, hitboxes, .etc .etc .etc .etc The ONLY reason you want to stick to build is so you can use Duke Nukem as a character, because for what you want it would be far better to use UDK or Unity.

[DavoX]

As for terrain it seems you didn't check Kaiser land 3 Tea Monster

[Radar]

Importing ground like what TM posted into Mapster32 looks like a great way to get 0 frame rate.

[Jimmy]

I'm all for advancements in technology to the BUILD engine, but there is seriously a point where people need to just use something else.

[Tea Monster]

I'd love to use something else. Maybe you are forgetting that 3dr/GBX sent out C&D orders to every group who tried. The one group who did get an official go-ahead were later told that, although they could make the game, they could never release it.

[Jimmy]

Then do something original.

[Mark]

Instead of the complex shape in TM's post, I'd be happy enough just to have easily made smooth rolling hills and valleys.

[Hank]

I thought we are excited moving towards Lua?
I am using this with ConEdit,

http://luaedit.sourceforge.net/

Thanks Helixhorned, you are the best

and of course, it would be nice if we had a CryEngine/Unity 3D/Unreal 3 - plug-in

[Mark]

The only way I picked up on what little con coding I know was to read many, many examples from wiki and other people's code. I have a feeling it will be the same with Lua. I'll have to see real world examples staring me in the face before I understand any of it. So I'm hoping there will be some dedicated hard working pioneers that will post sections of con and lua code side by side to show how to do certain basic things. I'll be happy to contribute more donations for those that do it.

[Helixhorned]

A synthesis build of Lunatic is available at last with r4023. Its EDuke32 executable is named leduke32_PREVIEW, deliberately to look a little dangerous. You may want to start by trying out to run existing (or your own in-development) CON mods with it, or take a look at how Lua scripting looks like by examining and running the files packaged under the lunatic/ directory. Quick start: issue

leduke32_PREVIEW -jlunatic test.lua -v1 -l1

for a slightly psychedelic experience.

Have fun, but note that as with generated LunaCON code, you should not use the provided Lua files as a base for own code without verifying that you're only using features that are documented in the User Manual.

[Hank]

I can't open the exe file, it simply does not open or initiate the game

[m210®]

What is a "Failure setting up Lunatic"?

---- [0] Translating file "GAME.CON"
[1] Translating file "DEFS.CON"
[1] Translating file "USER.CON"
GAME.CON 476:11: error: label "BLIMPRESPAWNTIME" is not an `action' value
GAME.CON 1305:27: warning: label "randomangle" is not a `move' value, assuming 0
GAME.CON 3968:2: warning: found `else' with no `if'
GAME.CON 4022:6: warning: action "ATROOPSUFFERDEAD" has incval different from -1, 0 or 1
GAME.CON 5891:2: warning: found `else' with no `if'
[0] Matched whole contents (1 errors).
runtime error: Failure compiling CON code, exiting.
stack traceback:
	[C]: at 0x00483a10
	[C]: in function 'error'
	/var/www/synthesis/eduke32/eduke32/source/lunatic/defs.ilua:2203: in main chunk
Lunatic: Error preparing global ELua state (code 4)
{
 {
  -- event times, [event]={ total calls, total time [ms], mean time/call [us] }
 },

 {
  -- actor times, [tile]={ total calls, total time [ms], {min,mean,max} time/call [us] }
 },
}
Failure setting up Lunatic!
Uninitializing DirectDraw...
Switching keyboard layout from 00000419 to 

[m210®]

I'm found and deleted action BLIMPRESPAWNTIME and now it's works. Look's like interesting

[Helixhorned]

M210, on 19 August 2013 - 12:24 AM, said:

What is a "Failure setting up Lunatic"?

(...)
GAME.CON 476:11: error: label "BLIMPRESPAWNTIME" is not an `action' value
(...)
runtime error: Failure compiling CON code, exiting.

Well, that's LunaCON's strictness in action here. Nice, huh?

Edit: To be more exact, this is a case of a type error. You're not supposed to pass defined numeric values to commands expecting a composite (like e.g. an action). An exception are the special values 0 and 1, which denote composites that have all members set to zero, but having the respective ID (with which comparisons are made in e.g. ifaction). This is usually used in initialization code in CON, like "ifaction 0 { action AMYENEMYWALK; init-code... }".

[Helixhorned]

Hank, on 18 August 2013 - 04:28 PM, said:

I can't open the exe file, it simply does not open or initiate the game

The two most likely causes are either 1) a crash or 2) a CON compilation error like for M210. Does the log show anything noteworthy? Hendricks266 already reported a crash, so you may have the same issue. What's the OS?

[Hendricks266]

Try deleting nedmalloc.dll.

[TerminX]

Helixhorned, on 19 August 2013 - 01:19 AM, said:

Well, that's LunaCON's strictness in action here. Nice, huh?

Edit: To be more exact, this is a case of a type error. You're not supposed to pass defined numeric values to commands expecting a composite (like e.g. an action). An exception are the special values 0 and 1, which denote composites that have all members set to zero, but having the respective ID (with which comparisons are made in e.g. ifaction). This is usually used in initialization code in CON, like "ifaction 0 { action AMYENEMYWALK; init-code... }".

I think that error is present in the 1.3D or 1.4 (not 1.5) CONs. It will probably require the strictness be relaxed a bit since we really need to be able to run all the versions of Duke that shipped on discs as-is...

[Hank]

Hendricks266, on 19 August 2013 - 01:35 AM, said:

Try deleting nedmalloc.dll.

@ Helixhorned

[m210®]

I tried a new lunacon release and have a question:
Can I add an increment to coordinates equal less then 1.
I tried add increment equal 0.5 to X coordinate (x=x+0.5), but sprite didn't move and, of course, it was moved then I added an increment equal 1 (x=x+1)?

[m210®]

And second:
analog of RETURN variable in Lunacon is g_RETURN. Can I see somewhere a list of those variables?
And I don't understood, why I must write gv.g_RETURN = -1 instead g_RETURN = -1. That is a gv?

Also gv.weaponcount or may be con.weaponcount doesn't works, but this is a system variable of CON

[Helixhorned]

M210, on 21 August 2013 - 01:46 AM, said:

I tried a new lunacon release and have a question:
Can I add an increment to coordinates equal less then 1.
I tried add increment equal 0.5 to X coordinate (x=x+0.5), but sprite didn't move and, of course, it was moved then I added an increment equal 1 (x=x+1)?

The in-memory BUILD map format still represents coordinates as signed 32-bit integers. It would mean a huge rewrite to have it otherwise. Maybe I should add that bit of information to the Manual. The only reason I didn't mention a type for e.g. sprite[].x is that I didn't want to suggest that all int32 values are permitted.

M210, on 21 August 2013 - 01:50 AM, said:

And second:
analog of RETURN variable in Lunacon is g_RETURN. Can I see somewhere a list of those variables?

gv.g_RETURN isn't documented in the User Manual. Don't use it, I may still change anything not explicitly listed there.

Quote

Also gv.weaponcount or may be con.weaponcount doesn't works, but this is a system variable of CON

It's called "hudweap.count". In general, you can't assume that names are taken over 1:1. I made various data accessible under different names from CON for consistency and to group them in a hierarchical fashion where possible.