Want to see a man torture a video game?

"Super Mario Spacetime Organ" is a short video by Chris Novello. It depicts the NES classic Super Mario Bros. piped through two devices, an illucia patchbay, which Novello invented, and the multitouch Soundplane created by Madrona labs. Using the Open Sound Control protocol (a modern MIDI alternative) Novello uses the hardware to directly manipulate the game's state in memory.

It starts off simply enough. Novello plays the game for a bit before stroking the Soundplane to make Mario hover in mid air. He begins physically patching parts of the memory into other parts and the game goes wild. Then the mallets come out…

"I was exploring the RAM state of Super Mario Bros., and I discovered that the game could be controlled in amazing ways by directly manipulating the memory," says Novello. "It is a game we all know super well, so it was really fascinating to experience it in an entirely different way."

The game in the video is running on an emulator that runs Lua scripts. By writing OSC hooks into the scripts, software from outside the game can read and write to the game's memory directly. By mapping the OSC addresses in the scripts on to the illucia's OSC addresses, Novello can change settings and variables by fiddling knobs and flicking switches.

The process isn't that different from how the Game Genie alters the ways games are played, except that Novello isn't in it to win. "I never used Game Genie for extra lives," he says. "I always played as the bosses in Mortal Kombat for the 30 seconds or so before the game crashed!"

Novello calls this technique codebending, a nod to circuitbending which involves creatively short-circuiting electronics to create experimental sound instruments. "For me, all of this is about asking: 'what else can a videogame do?'" he says. "I deeply believe in the artistic, expressive, communicative potential of games and software. I want to see more people push the boundaries of the medium — to see that it is possible to do something different."

Novello says he's a big fan of modular synthesizers, and he wondered what a videogame equivalent would look like. "I wrote small games, but I made them speak Open Sound Control so that they could communicate with each other," he says. "Basically I made video games that could play other video games."

"I discovered that games go berserk when you modulate their interior parameters with things like sine waves and step sequencers. They burst into generative patterns. I started thinking of games like cellular automata — they're just simple sets of rules that erupt with amazing generative behavior when pushed."

When he got bored of pushing the games around with a mouse and keyboard, Novello went back to his synth-loving roots and built a physical controller, the illucia. The one seen in the video is an illucia dtr, designed to be more portable and durable than its predecessor. "I made it Arduino code compatible and buildable using commonly available parts," says Novello, who is working on releasing is as Open Source Hardware.

There's something magical about turning the abstract values of game memory into something that you can manipulate with a hardware dial. At some level, we shouldn't be surprised by this, as it's not different from how a physical controller communicates information to the game. But watching Novello in action, it feels like he's reaching into a brain and making a body dance. The illusion of the inviolate game world is shattered and Novello describes himself as an evil demon in Mario's universe.

"Isn't it crazy?" says Novello. "What I'm doing with illucia is exactly the same as what one does with a controller…until it isn't. So weird. I love realizing how wrong I am about a game — how little I know about a system — how a black box can spill alien surprises."

Novello says that he hopes his efforts will begin to push out the bounds of what it means to think of games as cultural artifacts. "I wonder a lot about how systems themselves can be treated as art," he says. "Not just the output of a system, but a system's design as an expressive, aesthetic, or ethical statement."

"Interactive digital mediums (like videogames) have such expansive potential for human expression, communication, and knowledge," says Novello. "Much more vast than the culture surrounding the medium at this moment in history. I think that questioning our expectations of things like videogames is one important step toward enabling a richer relationship with digital culture. "