Current role. I'm a designer and engineer on Google DeepMind’s AI experience team, where I build tools, prototype, and survey research for transformative AI.
I work across engineering and science teams to align research and tech strategy to accelerate DeepMind’s research agenda. I also work on cultivating the inclusive and collaborative working environment necessary for such an ambitious mission.
Past work. Pre-DeepMind, I reached millions of users as I independently created a variety of products across mobile, web, and desktop platforms.
I chased interesting problems, tackling novel UX and design challenges. I created every aspect of the resulting products.The result was a diverse corpus consisting of creative apps, off-beat games, and interactive art pieces. Some are commercial releases; others are experiments.
generative future history (2023, personal blog)
Let’s peer into the future encoded in an AI’s training data. With a bit of prompting, we can bootstrap a journey from today to next year to decades to millenia in the “future”–as a powerful LLM trained on most of the Internet would predict.
Explain with AI (2023, personal) Rephrase text in a webpage in-situ with simpler explanatory language, using a language model.Expand for more info.
This was an exercise in updating myself in the latest workflow of old friends React, NPM, webpack, and, of course, the Chrome extension API. As an optional, flashy feature, the extension can type over existing text in a ChatGPT-like live-typing animation. The extension is extensively customizable -- more of a playground than a specific tool -- and I built support for AI21, OpenAI, and co:here's language model APIs.Twitter thread
Semantic image editing. (2022, personal) I explored implementations of editing images with natural language with Stable Diffusion -- inspired by Couairon, et al's DiffEdit paper.Expand for more info.
This approach requires you offer a caption for the image you're strating with -- say, "Captain Picard in Star Trek" and change it to reflect an edit you'd like, such as, "Bryan Cranston in Star Trek". The model can be induced to produce a "mask" that determines where the image should best be changed to accommodate the new prompt. With the original image, the mask, and the new prompt, the diffusion model can then apply those edits, resulting in Bryan Cranston -- or whoever -- on the bridge of the Enterprise!
My exploration successfully implemented this approach and explored some alternatives to deriving the mask. This project was inspired by work in a recent fast.ai course.Twitter thread
Xland (2021, preprint) Open Ended Learning Team, et al. “Open-Ended Learning Leads to Generally Capable Agents.” ArXiv:2107.12808 [Cs], July 2021. arXiv.org.Expand for more info.
This paper describes our first steps to train an agent capable of playing many different games without needing human interaction data. We created a vast game environment we call XLand, which includes many multiplayer games within consistent, human-relatable 3D worlds. This environment makes it possible to formulate new learning algorithms, which dynamically control how an agent trains and the games on which it trains.DeepMind blog | Two-Minute Papers Feature [YouTube] | arxiv pre-print
Playhouse Environment. (2021, NeurIPS) DeepMind Interactive Agents Team, et al. “Creating Multimodal Interactive Agents with Imitation and Self-Supervised Learning.” ArXiv:2112.03763 [Cs], Feb. 2022. arXiv.org.Expand for more info.
The Playhouse is a humanistic environment well-suited for simulating familiar domestic tasks. This environment was used in service of the research and development of the the Multimodal Interactive Agent. This cutting-edge AI agent blends visual perception, language comprehension and production, navigation, and manipulation to engage in extended and often surprising physical and linguistic interactions with humans.Video demo [YouTube] | DeepMind: author's note | arxiv paper
Alchemy Task Suite. (2021, NeurIPS) Wang, J.X., et al. (2021). Alchemy: A structured task distribution for meta-reinforcement learning. NeurIPS.Expand for more info.
This ambitious project sought to explore meta-learning -- the capability of learning to learn, typically including the sub-skills of planning, hypothesis formation, and exploration.
This research required an environment inspired by a chemistry (or, rather, alchemy) lab that had strategic structure at multiple levels and timescales.GitHub | DeepMind: author's note | arxiv paper
DeepMind Memory Task Suite. (2019, NeurIPS) Fortunato, M., Tan, M., Faulkner, R., Hansen, S.S., Badia, A.P., Buttimore, G., Deck, C., Leibo, J.Z., & Blundell, C. (2019). Generalization of Reinforcement Learners with Working and Episodic Memory. NeurIPS.Expand for more info.
The DeepMind Memory Task Suite is a set of 13 diverse machine-learning tasks that require memory to solve. They are constructed to let us evaluate generalization performance on a memory-specific holdout set.
As was the case for all these research projects, I collaborated with a diverse team of researchers and engineers, iterating on designs for environemnts and experimental scenarios to interrogate the capabilities of a novel memory-focused reinforcement learning agent.GitHub | arxiv paper
expressionPad. (2017, App Store) Unique, highly-configurable music controller with built-in additive synthesizer. Full 10-touch expression with individual continuous control.Expand for more info.
expressionPad began as a 10-touch matrix controller in the first wave of iPad apps in 2012. In 2017, I re-conceived the app in Swift and used AudioKit for the synth and MIDI capability.
The unique thing about expressionPad is that each individual touch can control expression, such that individual fingers or hands can smoothly slide from pitch to pitch--imagine a fretless piano or a continuous Ableton controller.
Meanwhile, the complete re-configurability of the key layout means you can discover the best layout for your own abilities. I wonder: if you can thumb-type proficiently, how fluently can you play expressionPad?Microsite | App Store
roguetronic (2019) A casual roguelike game with RC racing-like controls. Procedural levels, some flashy visual effects, and a 80's hacking theme. Made for the Seven Day Roguelike (7DRL) gamejam.itch.io microsite.
Slashcards: Learn Japanese (2017, alpha) A Zelda-like co-op language learning action RPG. Features some novel UX for multi-lingual text input--in an action game. Playable alpha.Expand for more info.
Emerging out of a prototype for the 2017 Seven Day Roguelike game jam, Slashcards: Learn Japanese is an action adventure that challenges the player to level-up their Japanese mastery to complete their quest. Featuring beautiful low-poly design, quirky animations, procedural levels and up-to-four-player co-op, players can compete together, adventure together, and learn to read and write Japanese;
Slashcards: Learn Japanese includes Japanese content covering both syllabaries, hiragana, katakana and over 100 kanji. I also built a variety of competitive versus modes for 1-4 players--optionally with AI opponents--including Battle Bingo and Slash Racer.
The game features a novel text input scheme for gamepads. I wrote a detailed post describing the novel dual-stick input approach I used for both English and Japanese. I also wrote some Unity tips for the facial animation approach I took.Microsite | itch page
Rescuties! VR. (2016, Steam/Oculus Store) Shock and aww! Rescue babies and cute animals from ten crisis situations in this room-scale VR experience. An Oculus Touch launch title.Expand for more info.
In April 2016, I created a VR prototype in preparation for a gamejam. Gifs of that prototype went viral and soon my mini-game was featured on the #1 gaming YouTube channel, PewDiePie's.
A successful Kickstarter followed and the game was launched later that year on Steam and Oculus, right on-time (a notorious Kickstarter rarity!) This casual, affirming action game was a lot of fun to put together.
I wrote an article detailing the dynamics of throwing in VR in 2016 -- an exploration from first principles.Microsite | Trailer
bigblueboo gifs. (2013-2016) 800+ code sketches and renders, many going viral on Reddit and Imgur. Explorations in perception, generative art, algorithm visualization, photorealistic rendering, and much more.bigblueboo.tumblr.com.
Word Fireworks: Learn Korean! (et al) (2015, App Store) Learn to read and write a new language with arcade action! Available in Korean, Chinese, Japanese, and Hebrew.Expand for more info.
The concept of Word Fireworks was born from a passion for particle systems and a new-found interest in learning Korean. The scope of the game quickly grew from a mini-game restricted to the Korean alphabet to a full-blown language learning experience across four different languages. Featuring non-Latin languages, Word Fireworks challenges students to learn not just how to pronounce and read non-Latin characters, the game has prompts requiring handwriting.
Male and female recordings play for each word, letter, and syllable in the game. Players can either progress through the game as structured or skip ahead if they already have some experience. Word Fireworks initiates players into these languages with quick, quirky lessons and gorgeous fireworks. Sprint levels, listening stages, common-mistake challenges lead to launching your very own fireworks show!Microsite | App Store
primes.io. (2014) Number theory explorer that visualizes Ulam spirals, the phenomenon whereby prime numbers appear with suspicious frequency when the number line is wound in a spiral. Led to a talk at Google I/O.Try primes.io.
fractals.cc. (2014) Real-time Mandelbrot and Julia fractal explorer. Fractals are calculated on the GPU; an experiment in pixel shaders.Try fractals.cc.
Fireworks Hero! (2015, App Store) Test your gaming skills to unlock all 10 heroes. A dozen game modes with more than 100 firework types.Expand for more info.
Fireworks Hero! takes my educational game Word Fireworks and amps up the action. In lieu of Word Fireworks' language learning content, Fireworks Hero! challenges players' reflexes with fireworks that need to be tapped, plucked, sliced, and charged.
Featuring a wide range of game modes, ranging from action-packed Arcade to creative modes to two-player versus modes. Design your own fireworks, or challenge a friend to Spark Pong. Take on the Challenge stages, or fire your own fireworks in Launcher mode. Travel around the world, from New York City to Johannesburg, from Seoul to Rio de Janiero, from Mumbai to Cairo, see the Eiffel Tower and Jerusalem's Dome of the Rock...launch beautiful fireworks over stunning skylines -- and see if you can be the next Fireworks Hero!Microsite | App Store
attractors.io. (2014) Real-time, interactive visualization of strange attractors: dynamical systems with a fractal internal structure. A haunting beauty reveals itself.Try attractors.io.
dyson modes. (2013) Visual-acoustic performance piece. A series of pitch-responsive abstract mechanisms are controlled via saxophone augmented with a TouchOSC interface.dyson modes excerpt on Vimeo.com.
The Grading Game. (2013, App Store) Race against time to find spelling and grammar errors--and pay down your crippling student debt!Expand for more info.
The Grading Game was conceived during a the Seven Day FPS gamejam. In lieu of yet another violent shooting game, I conceived a game where the player would "shoot" grammar and spelling errors. I called that prototype "First Person Tutor", and after the positive response to the prototype, I built a full game experience around the idea. The Grading Game is the result!
The Grading Game puts you in the role of a woeful grad student who, at the bidding of an unhappy tenured professor, has the unhappy task of grading the oft-plagiarized papers of his undergraduate students. As it happens, these papers seem to be cribbed from Wikipedia -- so you find yourself learning more than ever as you sift through the grammer and spelling errors!
Moreover, the procedural "level" generation injects spelling and grammar errors drawn from Wikipedia's own vast corpus of historical edit errors. The result is a plausible simulacrum of copyediting real-world text. Upgrade your copyediting skills even as you absorb quirky Wikipedia articles, ranging from the boy scout who built a nuclear reactor, to Korean fan death, a widely-held conspiracy theory!Microsite | App Store
cubes.io. (2013) Explore shiny 3D cellular automata in this interactive explorer powered by WebGL. Discover beautiful, complex 3D tapestries arise from simple rules.Try cubes.io.
Lunacraft. (2012, App Store) Explore a sci-fi procedural voxel world. Infinite terrain, crafting. (Impressive stuff for the time!)Expand for more info.
In 2012, I tackled making an infinite procedural voxel terrain engine that ran on an iPhone 3GS. I built my own engine with a mixture of multi-threaded C and Objective-C. The result was tens of thousands of positive reviews and millions of downloads. In its time, Lunacraft was a best-in-class experience on mobile.
Featuring varied crafting options, a variety of xenobiota, Newtonian physics, a day-night cycle, dynamic lighting, enemy astronauts, and a creative mode, Lunacraft offered a straightforward (and modestly novel) take on the burgeoning voxel exploration genre.
In hindsight, further development would no doubt have been lucrative. Whatever the case, I'd moved on to other, fresher challenges once Lunacraft was out in the world.Microsite | App Store
Ratatap Drums. (2011, App Store) The drumming app with the fastest response and richest expression. 15+ built-in drumkits, and unlimited customization.Expand for more info.
As a serious musician myself, I was (in 2010) disappointed with the offerings available for music performance and creation on mobile devices. Beginning as an experiment to see just how low latency could go on the iPhone, I dove into the unlit, unhappy labyrinth of iOS CoreAudio at that era.
The result was a cutting-edge (and popular!) drumming app that was featured regularly on Apple's App Store. With expandable drum sets, a vast sound library, realism-enhancing generative synthesis, and a novel simulacrum of pressure sensitivity via the accelerometer, Ratatap Drums made a compelling product for mobile musicians interested in exploring their rhythmic options.Microsite | App Store | Youtube demo
noteplex. (2011, App Store) A visual composition toy where notes are nodes lighting up a hexagonal grid. Make music with chain reactions!Expand for more info.
Noteplex was an exploration into an alternative representation of music. In lieu of noteheads, stems, and flags on the five-lines of Western sheet music, Noteplex presents a hexagonal matrix where differently-shaped "nodes" emit sounds and--crucially--pulses that can set off other nodes.
Placing nodes of various types (and pitch and timbre) leads to music that sets itself in motion. Moreover, the creation of music itself in Noteplex can be a musical performance. Initially created in Flash, Noteplex on iOS uses a custom 2d engine to display hundreds of independent nodes within view -- a non-trivial engineering challenge in 2011!Microsite | Youtube Demo
Parameter Ghosts. (2014) A short animated movie generated by wandering around the parameter space of one particular strange attractor system.HD video hosted on Vimeo.com.
NESTFALL. (2015) Immersive VR total-body flying game. Mashing up body tracking and head-mounted display so players can spread their arms to fly down a virtual chasm.Clip of NESTFALL gameplay on Vimeo.com.
United Nanolytics Knowledge Vault. (2014, LD30) Generative game made in 72 hours for the Ludum Dare 30 gamejam.Try UNKV (Flash).
space boids. (2014) Avoid/align/approach 3d flocking, inspired by Craig Reynolds' "boids". My first Unity experiment.Video on Vimeo.com.
emotweet. (2010) Visualize Twitter reactions. Sentiment analysis. Final project for Harvard CS171 Data Visualization. (API deprecated.)Microsite (defunct).
Wow Such Web. (2012) Doge-ify the web. Chrome extension that swaps and alters content, such media. Wow.Chrome Store link.
Ellsworth. (2013) Abstract puzzler inspired by the 20th century Abstractionist, Ellsworth Kelly.Try Ellsworth (WebGL, desktop).
My Kingdom for a Line Block! (2012) Two-player asymmetric version of the famous tetromino game. One player chooses the worst-possible block for the other. Built for Ludum Dare 25.Try My Kingdom For A Line Block! (Flash).
Polyglot. (2012) Inspired by browser extensions like Rikai, Polyglot was an iPhone web browser that enabled you to discover, accumulate, and practice new vocab as you encountered it while reading.
Caucus. (2012) Choose your candidate, listen to voters' preferences and transmit your positions on the issues -- but don't let the voters discover you're a flip-flopper! A real-time political strategy game about lying. Built for the Global Game Jam 2012.Try Caucus (Flash).
Space Opera Singer. (2011) Sing the correct notes to defeat the aliens -- a sight-singing space shooter.Try Space Opera Singer (Flash).
Legacy Work. (2007-) I created many products before those described above. The many lessons those experiences taught me informed subsequent work.Expand for more info.
My first retail offering was a flashcard service called Flash My Brain, launched in 2007. I bootstrapped this software-as-a-service into sustainability in less than six months. A compelling offering at the time, the app featured printing onto cardstock templates, spaced repetition study, several memory games, and a rich editor supporting, among other content, RTL languages. My users created millions of flashcards, and the desktop app was eventually joined by a mobile app when Apple's App Store first launched.
Being one of the first published app creators led to lucrative consulting work, which in turn supported continued experimental work. So, I designed and published a family of musical instrument apps for the iPhone. These apps contrasted with the toy-like apps of the time in their true-to-life control model: Boehm fingerings for Clarinet, valves-and-partials for Trumpet, and so on.
Music composition tools followed in 2010, including a matrix sequencer called moxMatrix and the initial Flash-based implementation of the above-described visual composition environment, Noteplex. I created the first incarnation of my experimental MIDI controller, expressionPad, for the original iPad. Meanwhile, I started to get involved with game development by participating in gamejams.
I also continued to make creative experiments using new web technologies, such as WebGL and the various social APIs that were sprouting up like weeds at the time.
From there, you can find my favorite work above. Myriad experiments, game jam prototypes, and explorations have been omitted. Feel free to get in touch if you have questions or inquiries.