Over the course of 2023, I worked on evaluations for Gemini, Google DeepMind's recently-launched family of multimodal (vision, audio, language) AI models. The scientific challenge of evaluating this evolving cohort of multilingual, multimodal, multipurpose models was a formidable and unprecedented challenge. My work facilitated getting better, more rigorous insights, faster, from state-of-the-art AI systems.

It's all under wraps for now, but I'm hopeful there will be a venue soon to share some of the proprietary tech and researcher UX we built for this.

This was an exercise in updating myself in the latest workflow of old friends React, NPM, webpack, and, of course, the Chrome extension API. As an optional, flashy feature, the extension can type over existing text in a ChatGPT-like live-typing animation. The extension is extensively customizable -- more of a playground than a specific tool -- and I built support for AI21, OpenAI, and co:here's language model APIs.

This approach requires you offer a caption for the image you're strating with -- say, "Captain Picard in Star Trek" and change it to reflect an edit you'd like, such as, "Bryan Cranston in Star Trek". The model can be induced to produce a "mask" that determines where the image should best be changed to accommodate the new prompt. With the original image, the mask, and the new prompt, the diffusion model can then apply those edits, resulting in Bryan Cranston -- or whoever -- on the bridge of the Enterprise!

My exploration successfully implemented this approach and explored some alternatives to deriving the mask. This project was inspired by work in a recent fast.ai course.

This paper describes our first steps to train an agent capable of playing many different games without needing human interaction data. We created a vast game environment we call XLand, which includes many multiplayer games within consistent, human-relatable 3D worlds. This environment makes it possible to formulate new learning algorithms, which dynamically control how an agent trains and the games on which it trains.

The Playhouse is a humanistic environment well-suited for simulating familiar domestic tasks. This environment was used in service of the research and development of the the Multimodal Interactive Agent. This cutting-edge AI agent blends visual perception, language comprehension and production, navigation, and manipulation to engage in extended and often surprising physical and linguistic interactions with humans.

This ambitious project sought to explore meta-learning -- the capability of learning to learn, typically including the sub-skills of planning, hypothesis formation, and exploration.

This research required an environment inspired by a chemistry (or, rather, alchemy) lab that had strategic structure at multiple levels and timescales.

The DeepMind Memory Task Suite is a set of 13 diverse machine-learning tasks that require memory to solve. They are constructed to let us evaluate generalization performance on a memory-specific holdout set.

As was the case for all these research projects, I collaborated with a diverse team of researchers and engineers, iterating on designs for environemnts and experimental scenarios to interrogate the capabilities of a novel memory-focused reinforcement learning agent.

expressionPad began as a 10-touch matrix controller in the first wave of iPad apps in 2012. In 2017, I re-conceived the app in Swift and used AudioKit for the synth and MIDI capability.

The unique thing about expressionPad is that each individual touch can control expression, such that individual fingers or hands can smoothly slide from pitch to pitch--imagine a fretless piano or a continuous Ableton controller.

Meanwhile, the complete re-configurability of the key layout means you can discover the best layout for your own abilities. I wonder: if you can thumb-type proficiently, how fluently can you play expressionPad?

Emerging out of a prototype for the 2017 Seven Day Roguelike game jam, Slashcards: Learn Japanese is an action adventure that challenges the player to level-up their Japanese mastery to complete their quest. Featuring beautiful low-poly design, quirky animations, procedural levels and up-to-four-player co-op, players can compete together, adventure together, and learn to read and write Japanese;

Slashcards: Learn Japanese includes Japanese content covering both syllabaries, hiragana, katakana and over 100 kanji. I also built a variety of competitive versus modes for 1-4 players--optionally with AI opponents--including Battle Bingo and Slash Racer.

The game features a novel text input scheme for gamepads. I wrote a detailed post describing the novel dual-stick input approach I used for both English and Japanese. I also wrote some Unity tips for the facial animation approach I took.

In April 2016, I created a VR prototype in preparation for a gamejam. Gifs of that prototype went viral and soon my mini-game was featured on the #1 gaming YouTube channel, PewDiePie's.

A successful Kickstarter followed and the game was launched later that year on Steam and Oculus, right on-time (a notorious Kickstarter rarity!) This casual, affirming action game was a lot of fun to put together.

I wrote an article detailing the dynamics of throwing in VR in 2016 -- an exploration from first principles.

The concept of Word Fireworks was born from a passion for particle systems and a new-found interest in learning Korean. The scope of the game quickly grew from a mini-game restricted to the Korean alphabet to a full-blown language learning experience across four different languages. Featuring non-Latin languages, Word Fireworks challenges students to learn not just how to pronounce and read non-Latin characters, the game has prompts requiring handwriting.

Male and female recordings play for each word, letter, and syllable in the game. Players can either progress through the game as structured or skip ahead if they already have some experience. Word Fireworks initiates players into these languages with quick, quirky lessons and gorgeous fireworks. Sprint levels, listening stages, common-mistake challenges lead to launching your very own fireworks show!

Fireworks Hero! takes my educational game Word Fireworks and amps up the action. In lieu of Word Fireworks' language learning content, Fireworks Hero! challenges players' reflexes with fireworks that need to be tapped, plucked, sliced, and charged.

Featuring a wide range of game modes, ranging from action-packed Arcade to creative modes to two-player versus modes. Design your own fireworks, or challenge a friend to Spark Pong. Take on the Challenge stages, or fire your own fireworks in Launcher mode. Travel around the world, from New York City to Johannesburg, from Seoul to Rio de Janiero, from Mumbai to Cairo, see the Eiffel Tower and Jerusalem's Dome of the Rock...launch beautiful fireworks over stunning skylines -- and see if you can be the next Fireworks Hero!

The Grading Game was conceived during a the Seven Day FPS gamejam. In lieu of yet another violent shooting game, I conceived a game where the player would "shoot" grammar and spelling errors. I called that prototype "First Person Tutor", and after the positive response to the prototype, I built a full game experience around the idea. The Grading Game is the result!

The Grading Game puts you in the role of a woeful grad student who, at the bidding of an unhappy tenured professor, has the unhappy task of grading the oft-plagiarized papers of his undergraduate students. As it happens, these papers seem to be cribbed from Wikipedia -- so you find yourself learning more than ever as you sift through the grammer and spelling errors!

Moreover, the procedural "level" generation injects spelling and grammar errors drawn from Wikipedia's own vast corpus of historical edit errors. The result is a plausible simulacrum of copyediting real-world text. Upgrade your copyediting skills even as you absorb quirky Wikipedia articles, ranging from the boy scout who built a nuclear reactor, to Korean fan death, a widely-held conspiracy theory!

In 2012, I tackled making an infinite procedural voxel terrain engine that ran on an iPhone 3GS. I built my own engine with a mixture of multi-threaded C and Objective-C. The result was tens of thousands of positive reviews and millions of downloads. In its time, Lunacraft was a best-in-class experience on mobile.

Featuring varied crafting options, a variety of xenobiota, Newtonian physics, a day-night cycle, dynamic lighting, enemy astronauts, and a creative mode, Lunacraft offered a straightforward (and modestly novel) take on the burgeoning voxel exploration genre.

In hindsight, further development would no doubt have been lucrative. Whatever the case, I'd moved on to other, fresher challenges once Lunacraft was out in the world.

As a serious musician myself, I was (in 2010) disappointed with the offerings available for music performance and creation on mobile devices. Beginning as an experiment to see just how low latency could go on the iPhone, I dove into the unlit, unhappy labyrinth of iOS CoreAudio at that era.

The result was a cutting-edge (and popular!) drumming app that was featured regularly on Apple's App Store. With expandable drum sets, a vast sound library, realism-enhancing generative synthesis, and a novel simulacrum of pressure sensitivity via the accelerometer, Ratatap Drums made a compelling product for mobile musicians interested in exploring their rhythmic options.

Noteplex was an exploration into an alternative representation of music. In lieu of noteheads, stems, and flags on the five-lines of Western sheet music, Noteplex presents a hexagonal matrix where differently-shaped "nodes" emit sounds and--crucially--pulses that can set off other nodes.

Placing nodes of various types (and pitch and timbre) leads to music that sets itself in motion. Moreover, the creation of music itself in Noteplex can be a musical performance. Initially created in Flash, Noteplex on iOS uses a custom 2d engine to display hundreds of independent nodes within view -- a non-trivial engineering challenge in 2011!

My first retail offering was a flashcard service called Flash My Brain, launched in 2007. I bootstrapped this software-as-a-service into sustainability in less than six months. A compelling offering at the time, the app featured printing onto cardstock templates, spaced repetition study, several memory games, and a rich editor supporting, among other content, RTL languages. My users created millions of flashcards, and the desktop app was eventually joined by a mobile app when Apple's App Store first launched.

Being one of the first published app creators led to lucrative consulting work, which in turn supported continued experimental work. So, I designed and published a family of musical instrument apps for the iPhone. These apps contrasted with the toy-like apps of the time in their true-to-life control model: Boehm fingerings for Clarinet, valves-and-partials for Trumpet, and so on.

Music composition tools followed in 2010, including a matrix sequencer called moxMatrix and the initial Flash-based implementation of the above-described visual composition environment, Noteplex. I created the first incarnation of my experimental MIDI controller, expressionPad, for the original iPad. Meanwhile, I started to get involved with game development by participating in gamejams.

I also continued to make creative experiments using new web technologies, such as WebGL and the various social APIs that were sprouting up like weeds at the time.

From there, you can find my favorite work above. Myriad experiments, game jam prototypes, and explorations have been omitted. Feel free to get in touch if you have questions or inquiries.