Jump to content
Double Fine Action Forums
Sign in to follow this  
DF Oliver

Programming Update 9: All Hail the Bug Overlords II

Recommended Posts

A warm welcome to all connoisseurs of funny-looking bugs!

It’s been a while since the last post about some of the more hilarious bugs we ran into. The programming team has been super busy and of course we ran into some more crazy issues since last time. And so: the current selection of funny bugs.

As-Sorted Confusion

programming9_01.png

Everybody knew that the Lumberjack was performing strange experiments in the woods, but nobody could have guessed how hideous his creations really were.

Awhile ago we improved the runtime performance of the animation system and while working on the optimization the sort-order somehow got reversed for parts of the body. Geometry that was supposed to show up behind stuff was drawing on top of everything, creating the egg-man clones.

The same issue also made Jesse’s eye a bit bigger than they are supposed to be.

programming9_02.png

Off-by-One

programming9_03.png

Even the egg-men clones don’t dare to venture into the forest during a night, because this is the time when really horrifying creatures roam around.

Even though our engine is implemented in C++ the game-play logic is written in Lua. One unfortunate difference between the two languages is that the indexing scheme. Lua is one-based, which means that the first element of an array is located at index 1, whereas in C++ the same array element has the index 0. This can at times lead to very frustrating bugs. In this case the outcome was pretty funny though.

The same problem also caused this animation of Vella to look off… by one.

[vimeo]70425933[/vimeo]

Finish Him!

[vimeo]70426119[/vimeo]

Performing all of these crazy experiments is pretty exhausting, so the Lumberjack practices martial arts after work to wind down. His biggest inspiration is Baraka [http://en.wikipedia.org/wiki/Baraka_(Mortal_Kombat)] so he created his Knifejack outfit.

The attachment system makes it possible for the characters to pick things up. While writing the code that makes one object follow another the scale for the attached knife somehow got messed up leading to this cool bug.

Smooth Operators… in SPACE!

programming9_04.png

One of Shay’s favorite pastimes is synchronized dancing and I think we can all agree he is pretty good at it!

We recently needed a new feature that lets us rotate the character procedurally (on top of an animation). While Shay was supposed to be tilted like that the doors in the background should have been unaffected. The problem turned out to be a ‘leaked’ transformation state influencing all skinned characters in the scene.

I hope you guys enjoyed the second forum post about funny bugs in the game. Let me know if you have questions about this stuff.

Share this post


Link to post
Share on other sites

Shay totally needs a spin-off game where his special power is to dance in sync with all the enemies! Oh wait ... that was the Michael Jackson Moonwalker video game.

I need to remember to take screenshots of my own bugs! These are fun. :-)

Seth B.

Share this post


Link to post
Share on other sites

"Leaked" transformations? How does that happen? I've been coding business software for desktop/web for a decade now, even with some complex UI stuff and I don't see bugs like that. I want to do some game programming but for all the bugs I see from day to day it's all simple cause and effect and it's localized to whatever is behaving incorrectly. It seems like game programming in general has a lot of issues where a bug in one part can cause bugs elsewhere. Finding those must be really difficult. Was that bug in the C++ code you modified in MOAI or was that in the LUA code? How do you debug for something like that? Is it still as easy as breakpoints and inspecting objects or is it more complicated than that?

Share this post


Link to post
Share on other sites
"Leaked" transformations? How does that happen? I've been coding business software for desktop/web for a decade now, even with some complex UI stuff and I don't see bugs like that. I want to do some game programming but for all the bugs I see from day to day it's all simple cause and effect and it's localized to whatever is behaving incorrectly. It seems like game programming in general has a lot of issues where a bug in one part can cause bugs elsewhere. Finding those must be really difficult. Was that bug in the C++ code you modified in MOAI or was that in the LUA code? How do you debug for something like that? Is it still as easy as breakpoints and inspecting objects or is it more complicated than that?

I'm not a graphics programmer at all so I can't really speak to the engine-level C++ stuff here but 2HB, our MOAI IDE, does have really great debug tools for setting breakpoints and inspecting objects and stuff in Lua. I love working with it.

Share this post


Link to post
Share on other sites

"Leaked" transformations? The leaker must be found and prosecuted to the fullest extent of the law!

Pretty soon it'll be all over the blogs with sensationalist headlines: "Broken Age Rotated 45 Degrees To The Left, Backers Get Their Panties In A Twist"

Share this post


Link to post
Share on other sites
"Leaked" transformations? How does that happen? I've been coding business software for desktop/web for a decade now, even with some complex UI stuff and I don't see bugs like that. I want to do some game programming but for all the bugs I see from day to day it's all simple cause and effect and it's localized to whatever is behaving incorrectly. It seems like game programming in general has a lot of issues where a bug in one part can cause bugs elsewhere. Finding those must be really difficult. Was that bug in the C++ code you modified in MOAI or was that in the LUA code? How do you debug for something like that? Is it still as easy as breakpoints and inspecting objects or is it more complicated than that?

Based on my limited knowledge of basic OpenGL (which I assume MOAI uses since it's cross-platform), when you add objects to a scene to draw, you tell OpenGL how to apply transformations and stuff before you add the next object, and it retains the information provided until changed so multiple objects can be added with the same or similar properties if desired. It's more complex than that, of course, but that's the general idea.

I'm guessing what happened is that the rotate transform was applied to Shay, but wasn't reset before adding in the other objects. Hence the "leak".

Feel free to correct me if I'm wrong.

Share this post


Link to post
Share on other sites

It's the Julian Assange of Transformations, leaking all over the place ;-)

Nice post Oliver :-)

Share this post


Link to post
Share on other sites

Great post!

What was the most frustrating / time-consuming bug that you have encountered so far?

also what is the hand icon inside the red circle in the first picture?

thanks!

Share this post


Link to post
Share on other sites
"Leaked" transformations? How does that happen?

I'm a graphics programmer/researcher and while I don't think there is any one official thing that is a 'transformation leak' this kind of thing happens pretty easily with the transformation stack.

OpenGL and D3D both maintain a stack of transformation matrices (several in fact). The one on top is the currently active transformation. Why a stack? Well, it's really handy for hierarchical objects. If an arm transforms then the hand that's attached to it should as well, but the hand might want to rotate separate from the arm. Transform for the arm, draw the arm, push the transformation stack, transform for the hand, draw the hand, pop to get back to just the arm. In order for this to work, these stacks always have a different sort of 'push' than a normal stack. Instead of pushing a specific new matrix, when you push a transformation stack it copies the top of the stack and pushes that matrix.

Because the push always copies the top of the stack (rather than pushing something specific) it's easy to have a transformation affect things you don't intend. If you forget to clear the previous transformation (by setting it to the identity matrix) or worse, you forget to 'pop' after a push then these sorts of things happen. A transformation 'leaks' into the next object, whatever it may be.

Forgetting to pop is usually a game breaking bug as the stack will quickly exceed capacity and the system will start throwing errors and drawing nothing. So that's probably not what happened here. Maybe it was just forgetting to reset to identity after a push or pop. He talked about this in the context of synchronization (which is kind of like setting up a temporary hierarchy between objects so they share a transformation). I'm guessing something just wasn't right with this part of the code that caused things to be sharing a transformation that shouldn't be. Perhaps it was transformation stack related, perhaps not.

Just my buck fifty here. :-)

Seth B.

Share this post


Link to post
Share on other sites

BTW, don't you mean "Smooth Criminal" instead of "Smooth Operator"? IF you're referencing the Michael Jackson lean in the "Smooth Criminal" music video.

Share this post


Link to post
Share on other sites
BTW, don't you mean "Smooth Criminal" instead of "Smooth Operator"? IF you're referencing the Michael Jackson lean in the "Smooth Criminal" music video.

My inference was that it was a reference to "Smooth Criminal," making usage of the programming term "operator"—I'm not sure how that specifically applies in this case though! And of course I can't speak for Oliver!

Share this post


Link to post
Share on other sites

so is the lumberjack playable at some point in the game? I'm so confused why he is still used, and those scenes look nice

Share this post


Link to post
Share on other sites
so is the lumberjack playable at some point in the game? I'm so confused why he is still used, and those scenes look nice
Ooh - maybe it'll be a little mini-game before Broken Age starts, like those shorts that play before Pixar films.

Share this post


Link to post
Share on other sites
so is the lumberjack playable at some point in the game? I'm so confused why he is still used, and those scenes look nice
Ooh - maybe it'll be a little mini-game before Broken Age starts, like those shorts that play before Pixar films.

From what I've seen of his evil experiments in the forest, and I'm calling it now, he's going to eventually turn into the monster that threatens Sacrifice Girl's village.

Share this post


Link to post
Share on other sites
"Leaked" transformations? How does that happen? I've been coding business software for desktop/web for a decade now, even with some complex UI stuff and I don't see bugs like that. (...)

I work on business code, and when I started reading up on Xna I got a headache.

The approach is drastically different than what you're used to. For one, there's a rendering loop that does things continuously and the thing pretty much never stops. I found that bit already confusing given the approach to writing business solutions.

Ironically, a friend of mine who DOES do some game coding envies me my programming skills while I envy him his.

What I'm trying to say is... yeah, things are weird in game development for people who come from other programming areas. ;)

Share this post


Link to post
Share on other sites

Shay appears to be performing a trust fall in the space ship. Trust falls are not single person things...

Seriously though, as a QA Engineer I find these updates enjoyable. Once beta testing is up and running, are we going to have a method for reporting bugs should we find them? I fully plan on giving the game a play through first and then hitting it heavy with my QA routine.

But man, the doors rotating like that is fun.

Share this post


Link to post
Share on other sites

Shay pulling a Michael Jackson is the funniest thing I've seen all morning, thank you. Although Knifejack is very close behind.

I'm glad you guys are all working as hard as you can to fix the bugs. Best of luck, I know it'll turn out great.

Share this post


Link to post
Share on other sites

That MOAI C++ indexing, vs Lua indexing scheme has caught me once or twice, particularly if you happen to be travelling regularly between the two code-bases. What's so wrong with zero based indexing Lua? HUH? You're written in C after all!

Share this post


Link to post
Share on other sites
so is the lumberjack playable at some point in the game? I'm so confused why he is still used, and those scenes look nice

From my experience of developing a system there are typically two separate places for development of a software solution: the prototype application and the actual application. Without going into detail, I was working on a managed directx map component for a piece of software, I had a prototype application which allowed me to add icons and such to the map and stop and start animation. The actual application was an entire software suite of which the map was only a small part of it.

I'm guessing that DF are doing a similar thing, the lumberjack "game" is being used as a testbed for new features and once they are solid and seemingly bug-free the features are then added to the version of the engine running in the actual Broken Age build. Doing it this way means that even if you mess something up in the test application, the actual game build is unaffected and minimises the risk of breaking the game.

Of course through the use of a repository software such as Perforce or Subversion (of which I'm sure DF is using the former), you can revert any game breaking changes like this anyway, but its always good to have a separate place to test components of software, in my opinion.

Again, all of this is just a guess but it looks to be that way.

*back on topic*

Interesting bugs! Though nothing is going to top the bug which made the lumberjack have long stretchy legs... :P

Share this post


Link to post
Share on other sites

also what is the hand icon inside the red circle in the first picture?

It's an old piece of UI art and has been replaced since taking the screenshot. :-)

What was the most frustrating / time-consuming bug that you have encountered so far?

While the bugs I talked about above are fun most of them are pretty easy to find and fix.

The most time consuming bugs are usually related to multi-threading (e.g. race conditions) since they can be very elusive. The worst bugs of this kind are called 'Heisenbugs' because they seem to disappear when you are trying to find them. We had a few race conditions when we were parallelizing a lot of our low-level systems.

Another cause for frustration is the lag of debugging facilities on some platforms (yes I'm looking at you Android). While every programmer should be able to find and fix bugs w/o a debugger it certainly saves you a lot of time having one.

Share this post


Link to post
Share on other sites
That MOAI C++ indexing, vs Lua indexing scheme has caught me once or twice, particularly if you happen to be travelling regularly between the two code-bases. What's so wrong with zero based indexing Lua? HUH? You're written in C after all!

Yep. In my opinion this is the major weakness of Lua.

Share this post


Link to post
Share on other sites

I'm guessing that DF are doing a similar thing, the lumberjack "game" is being used as a testbed for new features and once they are solid and seemingly bug-free the features are then added to the version of the engine running in the actual Broken Age build. Doing it this way means that even if you mess something up in the test application, the actual game build is unaffected and minimises the risk of breaking the game.

Of course through the use of a repository software such as Perforce or Subversion (of which I'm sure DF is using the former), you can revert any game breaking changes like this anyway, but its always good to have a separate place to test components of software, in my opinion.

We tend to create branches only for big refactoring jobs, because there are a lot of steps involved to create a fully working branch (mostly because of shared/non-shared code, the build server setup and stuff like that).

You are right though we use these old scenes as testbeds for new features. So far having the main game and the tests in the same branch hasn't been a problem. For bigger projects (CoD, Halo, ...) this is a completely different story and you probably want to keep things a bit more separated.

Share this post


Link to post
Share on other sites

You guys been spot-on explaining the 'leaked transformations' by the way. Well done dudes! :-)

Share this post


Link to post
Share on other sites

Thanks for sharing those bugs! Other people's bugs are fascinating. Game developers have the disadvantage most of their problems have to be found by carbon based lifeforms, 3D graphics requires skill to understand, but there are some very complicated business systems out there.

Share this post


Link to post
Share on other sites

Everybody knew that the Lumberjack was performing strange experiments in the woods, but nobody could have guessed how hideous his creations really were.

Haha, priceless. Thanks for fun read, I should have read Programming updates sooner.

Share this post


Link to post
Share on other sites
Sign in to follow this  

×
×
  • Create New...