Lua in Grim Fandango
Published on Wednesday, May 26, 1999 by Bret Mogilefsky on Usenet
I was the lead programmer on Grim Fandango. I’d like to tell you guys how Lua was used in the game, but I feel a little awkward doing this without LucasArts’ permission.
If possible, *please* remove this message from the web log of the mailing list… I’ll feel a lot better. I’m reminded of the artist who worked here years ago who used an early version of 3D Studio. Asked for a quote by Autodesk for use in marketing materials, he replied, “It’s like having ILM in a box!” Needless to say, his employment here didn’t last long. Please don’t use my comments in any way other than for your own edification. They are entirely unofficial.
I started the project thinking I was going to try to get 3D characters going in the legacy LucasArts adventure game engine which had it’s own embedded scripting language called SCUMM. I quickly found out that both the engine and SCUMM were much more special-purpose and hard-coded than I’d hoped, and decided to rewrite with 3D in mind from the beginning. Getting all of the base stuff replaced was easy… Setting up framebuffers, loading up resources, even displaying 3D characters thanks to Jedi Knight’s rendering library. The big question mark was the scripting language… I had no desire to go all the way down to lexing and parsing Yet Another Language Invented Here. About that time, a friend pointed me to the Dr. Dobb’s article on Lua. SCUMM had very forgiving syntax and fairly simple semantics, and Lua’s elegant design compared very favorably… Our project lead had programmed for years in SCUMM and I wanted him to be comfortable in this “new” language, whatever it was.
He wasn’t convinced that this was a good idea, but I took a single afternoon to embed Lua 2.5 in our engine and had functions throwing up text on the screen the next day. There was still something missing that we’d enjoyed in SCUMM however, and that was cooperative multitasking. In an adventure game, it’s very nice to have a separate script running that makes pigeons fly around, makes another character follow you, or that looks for opportunities to change the camera angle. We could do all that via callbacks, but that means that the engine has to know about pretty much everything that could happen in the scripting environment, and we wanted it to be more open-ended than that. Or, maybe we just weren’t too rigid in our design. =) I spent about two weeks sorting through the interpreter and figuring out how to add some primitive functions like “start_script”, “stop_script”, “wait_for_script”, “sleep”, etc… Basically everything you need for tasks to communicate with each other cooperatively. I didn’t want a separate C thread per top-level function because it was overkill and not too efficient, and Lua 2.5 wasn’t set up for that to happen easily anyway. Also, we still wanted functions to have a notion of when a frame actually gets drawn since we still had lots of 2D stuff animating in the game. So real multithreading was out and cooperative tasking in the scripting environment only was in. I also added some handles and meta-functions for tasks so that you could iterate over them just as with next() and nextvar().
It worked pretty well, and we really liked using it, so that pretty much cemented it. I wrote a bunch of system scripts to handle basic object interaction, dialogs with other characters, and control handling. I tried to be as object-oriented as possible, but the engine was in C and I wasn’t much of an object-oriented programmer, and as a result I tended to make things overly procedural. The project leader, who laid out most of the scripts for those who came on to do scripting later, followed my examples and I regret to say we could have made scripting a lot tighter if we’d gone full OO from the beginning… We ended up with some problems in debugging. If we’d had the full tag method fallbacks of Lua 3.1 from the beginning, we would have been able to do a lot more to track variable access, etc. But by the time Lua 3.1 was here we had written too much to really get that in underneath.
So how does the engine actually use Lua? It starts up by initializing it’s internal state and registering around 150 functions with Lua. Then it loads up a Lua script called _system.lua and calls just one function: BOOT(). BOOT() is expected to load up other script objects, fire off new tasks, and tell the engine to load up the initial room for display. After that function returns, the engine enters it’s main loop: Render the world, then call lua_updatetasks(). This was the API function I added that iterates over all the tasks and runs them until they return out of their top-level function, go to sleep, or run far too long (via a counter in the VM as it crunched opcodes… this was so we could detect bugs where we ended up in a nasty loop and still have enough control over the engine to figure out what’s happening). The engine does not itself do *anything* related to the game’s content. It provides primitives for loading up sets and costumes, for displaying text and some simple graphics, and for querying and setting the state of other forms of data. It plots waypoints for walking and and plays animations. It only calls explicit Lua functions for a few things, notably changes in the state of the controls and when actors collide. It doesn’t know anything about adventure games, or talking, or puzzles, or anything else that makes Grim Fandango the game it is. It just knows how to render a set from data that it’s loaded and draw characters in that set (and other stuff about loading up resources and playing movies and internationalization and 3D and colliding points with boundaries, but those are all just the details. =) )
The real heroes in the development of Grim Fandango were the scripters. They wrote everything from how to respond to the controls to dialogs to camera scripts to door scripts to the in-game menus and options screens. Oh yeah, and carefully tuned room and puzzle code, plus around 8,000 lines of sterling dialog. A TREMENDOUS amount of this game is written in Lua. The engine, including the Lua interpreter, is really just a small part of the finished product. In reality, there are parts of the engine that might have been buffed up a lot more to take some of the burden off of the scripters… They ended up coding a lot of state machines by hand in Lua for individual animations and dialogs, when these things should really have been coded more rigidly in the engine and exposed to the scripts via a higher-level abstraction.
That said, there are other ways we used Lua that benefited us greatly. It is incredibly valuable to be able compile and run code on the fly. We can pop up a dialog box, enter some arbitrary bit of Lua code, and submit it to the running game to be compiled and run (of course this functionality is locked in the final game). We were also able to patch the scripting environment on the fly… You could find a function with a bug, copy it to a separate file, then have the engine source the patch file time and again as you edit and iterate toward a fix, all without restarting the game. We used this in testing, too… Burning whole sets of CDs to get around a single critical bug that stops forward progress in the game is a big pain in the ass. So we’d send down a patch for the testers containing just the crucial fix to get them going again, and they’d continue to happily test the game… It’s normal to give testers a new executable, but being able to patch scripts without sending them a whole new set (thereby invalidating the testing process) is something special. This was critical in the patch for the game as well.
We had a choreography tool that ran in a window alongside the engine in a .DLL, used for rehearsing animations in the game world and linking them together with logical transitions. It was also used for adding sound effects, flipping through 2D bitmaps synchronized to 3D movement, and as an added wrinkle, changing the value of Lua variables in the gamestate. This allowed us to do complex stuff like play footstep sounds that changed based on the surface the character was walking on, choreograph game events that depended on knowing when a certain frame in the middle of an animation was hit, etc. The tool actually exposed it’s own set of functions to Lua, and since Lua was also in a .DLL, the script environment was shared between the two. This made it really easy to add features to the choreography tool such as macros for repetetive tasks, moving large blocks of keys, etc. It also made it possible to encapsulate a bug from the engine… You could isolate a section of code that was generating an animation glitch and bind it to a menu item from Lua, then trigger it repeatedly from the tool so you could debug it in relative isolation from the game logic.
Finally, one of our scripters took it upon himself to write a pretty cool graphical debugger that took into account all of the tasking changes and had nice features like watches and such. However, a little while after it was written, we found some bugs in my tasking implementation and also figured it was time to move to Lua 3.1 to incorporate any fixes that might have happened. I ended up rewriting all of my tasking functionality in a MUCH cleaner way, but around this time we got into crunch mode and out of sync with the debugger, and no one had time to really go fix it for the rest of the project. The other mechanisms above made it possible to debug without it, but it was sorely missed… Since then it’s been fixed and is very useful once again. I haven’t had a chance to look at the debugger that’s on the Lua projects page to compare the two.
There are other small changes… We modified the binary form somewhat to make it easier to stream Lua objects off of a CD-ROM without blowing the CD drive’s cache (critical for speeding up the game startup, long though it still was), and obscured a lot of strings to make it non-trivial to see how the game was put together. We also handled saving and loading the game by saving out the entire Lua script environment, functions and all. This wasn’t the best way to do it, but it worked pretty well and hey, you have to ship someday!
That pretty much covers it. Hope you enjoyed reading as much as I enjoyed sharing… But geez, hope I don’t get in trouble for this. (Bosses reading: Uh, please don’t fire me.)