charlie deck // @bigblueboo

Hi. My name is Charlie Deck. I am a digital designer, engineer and researcher.

I've created tools, games, apps, and creative toys ranging from iPad synthesizers to VR games.

My particular passion is exploring how technology can amplify our creativity.

I'm in London. Reach out if you want to chat.

skipping robot gif
negative space gif

Current role. I'm a designer and engineer on Google DeepMind’s AI experience team, where I build tools, prototype, and survey research for transformative AI.

I'm currently leading demos across engineering and research teams working on generative AI, aligning research and tech strategy to accelerate DeepMind’s research agenda. I'm also an active advocate for the inclusive and collaborative working environment necessary for our ambitious mission.

Past work. Pre-DeepMind, I reached millions of users as I independently created a variety of products across mobile, web, and desktop platforms.

I chased interesting problems, tackling novel UX and design challenges. I created every aspect of the resulting products.

The result was a diverse corpus consisting of creative apps, off-beat games, and interactive art pieces. Some are commercial releases; others are experiments.

reverse voxelized space filling algorithm gif
recent // blogs, ai, deepmind (2018-)
card image top

generative future history (2023, personal blog)

Let’s peer into the future encoded in an AI’s training data. With a bit of prompting, we can bootstrap a journey from today to next year to decades to millenia in the “future”–as a powerful LLM trained on most of the Internet would predict.

Full blog entry.

card image top

infinite storyteller (2023, personal blog)

“Infinite Storyteller” is a generative choose-your-own-adventure, told with state-of-the-art speech synthesis, generative imagery, and presented as a Twitch broadcast in a real-time, 3d game engine.

Full blog entry.

Gemini (2023, Google DeepMind)

Gemini is Google DeepMind's state-of-the-art large multimodal (vision, audio, multilingual text) model powering diverse Google products. More details in the Gemini tech report.

Over the course of 2023, I worked on evaluations for Gemini, Google DeepMind's recently-launched family of multimodal (vision, audio, language) AI models. The scientific challenge of evaluating this evolving cohort of multilingual, multimodal, multipurpose models was a formidable and unprecedented challenge. My work facilitated getting better, more rigorous insights, faster, from state-of-the-art AI systems.

It's all under wraps for now, but I'm hopeful there will be a venue soon to share some of the proprietary tech and researcher UX we built for this.

DeepMind Gemini site | Gemini: All you need to know in 90 seconds [YouTube] | Two-Minute Papers feature on Gemini

Explain with AI (2023, personal) Rephrase text in a webpage in-situ with simpler explanatory language, using a language model.

This was an exercise in updating myself in the latest workflow of old friends React, NPM, webpack, and, of course, the Chrome extension API. As an optional, flashy feature, the extension can type over existing text in a ChatGPT-like live-typing animation. The extension is extensively customizable -- more of a playground than a specific tool -- and I built support for AI21, OpenAI, and co:here's language model APIs.

Twitter thread

Semantic image editing. (2022, personal) I explored implementations of editing images with natural language with Stable Diffusion -- inspired by Couairon, et al's DiffEdit paper.

This approach requires you offer a caption for the image you're strating with -- say, "Captain Picard in Star Trek" and change it to reflect an edit you'd like, such as, "Bryan Cranston in Star Trek". The model can be induced to produce a "mask" that determines where the image should best be changed to accommodate the new prompt. With the original image, the mask, and the new prompt, the diffusion model can then apply those edits, resulting in Bryan Cranston -- or whoever -- on the bridge of the Enterprise!

My exploration successfully implemented this approach and explored some alternatives to deriving the mask. This project was inspired by work in a recent fast.ai course.

Twitter thread

Xland (2021, preprint) Open Ended Learning Team, et al. “Open-Ended Learning Leads to Generally Capable Agents.” ArXiv:2107.12808 [Cs], July 2021. arXiv.org.

This paper describes our first steps to train an agent capable of playing many different games without needing human interaction data. We created a vast game environment we call XLand, which includes many multiplayer games within consistent, human-relatable 3D worlds. This environment makes it possible to formulate new learning algorithms, which dynamically control how an agent trains and the games on which it trains.

DeepMind blog | Two-Minute Papers Feature [YouTube] | arxiv pre-print

Playhouse Environment. (2021, NeurIPS) DeepMind Interactive Agents Team, et al. “Creating Multimodal Interactive Agents with Imitation and Self-Supervised Learning.” ArXiv:2112.03763 [Cs], Feb. 2022. arXiv.org.

The Playhouse is a humanistic environment well-suited for simulating familiar domestic tasks. This environment was used in service of the research and development of the the Multimodal Interactive Agent. This cutting-edge AI agent blends visual perception, language comprehension and production, navigation, and manipulation to engage in extended and often surprising physical and linguistic interactions with humans.

Video demo [YouTube] | DeepMind: author's note | arxiv paper

Alchemy Task Suite. (2021, NeurIPS) Wang, J.X., et al. (2021). Alchemy: A structured task distribution for meta-reinforcement learning. NeurIPS.

This ambitious project sought to explore meta-learning -- the capability of learning to learn, typically including the sub-skills of planning, hypothesis formation, and exploration.

This research required an environment inspired by a chemistry (or, rather, alchemy) lab that had strategic structure at multiple levels and timescales.

GitHub | DeepMind: author's note | arxiv paper

DeepMind Memory Task Suite. (2019, NeurIPS) Fortunato, M., Tan, M., Faulkner, R., Hansen, S.S., Badia, A.P., Buttimore, G., Deck, C., Leibo, J.Z., & Blundell, C. (2019). Generalization of Reinforcement Learners with Working and Episodic Memory. NeurIPS.

The DeepMind Memory Task Suite is a set of 13 diverse machine-learning tasks that require memory to solve. They are constructed to let us evaluate generalization performance on a memory-specific holdout set.

As was the case for all these research projects, I collaborated with a diverse team of researchers and engineers, iterating on designs for environemnts and experimental scenarios to interrogate the capabilities of a novel memory-focused reinforcement learning agent.

GitHub | arxiv paper
past work // solo (2007-2017)

expressionPad. (2017, App Store) Unique, highly-configurable music controller with built-in additive synthesizer. Full 10-touch expression with individual continuous control.

expressionPad began as a 10-touch matrix controller in the first wave of iPad apps in 2012. In 2017, I re-conceived the app in Swift and used AudioKit for the synth and MIDI capability.

expressionpad gif

The unique thing about expressionPad is that each individual touch can control expression, such that individual fingers or hands can smoothly slide from pitch to pitch--imagine a fretless piano or a continuous Ableton controller.

Meanwhile, the complete re-configurability of the key layout means you can discover the best layout for your own abilities. I wonder: if you can thumb-type proficiently, how fluently can you play expressionPad?

Microsite | App Store
roguetronic image

roguetronic (2019) A casual roguelike game with RC racing-like controls. Procedural levels, some flashy visual effects, and a 80's hacking theme. Made for the Seven Day Roguelike (7DRL) gamejam.

itch.io microsite.

Slashcards: Learn Japanese (2017, alpha) A Zelda-like co-op language learning action RPG. Features some novel UX for multi-lingual text input--in an action game. Playable alpha.

Emerging out of a prototype for the 2017 Seven Day Roguelike game jam, Slashcards: Learn Japanese is an action adventure that challenges the player to level-up their Japanese mastery to complete their quest. Featuring beautiful low-poly design, quirky animations, procedural levels and up-to-four-player co-op, players can compete together, adventure together, and learn to read and write Japanese;

Slashcards screenshot

Slashcards: Learn Japanese includes Japanese content covering both syllabaries, hiragana, katakana and over 100 kanji. I also built a variety of competitive versus modes for 1-4 players--optionally with AI opponents--including Battle Bingo and Slash Racer.

Slashcards screenshot

The game features a novel text input scheme for gamepads. I wrote a detailed post describing the novel dual-stick input approach I used for both English and Japanese. I also wrote some Unity tips for the facial animation approach I took.

Microsite | itch page

Rescuties! VR. (2016, Steam/Oculus Store) Shock and aww! Rescue babies and cute animals from ten crisis situations in this room-scale VR experience. An Oculus Touch launch title.

In April 2016, I created a VR prototype in preparation for a gamejam. Gifs of that prototype went viral and soon my mini-game was featured on the #1 gaming YouTube channel, PewDiePie's.

Rescuties VR gif

A successful Kickstarter followed and the game was launched later that year on Steam and Oculus, right on-time (a notorious Kickstarter rarity!) This casual, affirming action game was a lot of fun to put together.

Rescuties VR gif

I wrote an article detailing the dynamics of throwing in VR in 2016 -- an exploration from first principles.

Microsite | Trailer
recursive 3d maze gif

bigblueboo gifs. (2013-2016) 800+ code sketches and renders, many going viral on Reddit and Imgur. Explorations in perception, generative art, algorithm visualization, photorealistic rendering, and much more.

bigblueboo.tumblr.com.

Word Fireworks: Learn Korean! (et al) (2015, App Store) Learn to read and write a new language with arcade action! Available in Korean, Chinese, Japanese, and Hebrew.

The concept of Word Fireworks was born from a passion for particle systems and a new-found interest in learning Korean. The scope of the game quickly grew from a mini-game restricted to the Korean alphabet to a full-blown language learning experience across four different languages. Featuring non-Latin languages, Word Fireworks challenges students to learn not just how to pronounce and read non-Latin characters, the game has prompts requiring handwriting.

Word Fireworks gif

Male and female recordings play for each word, letter, and syllable in the game. Players can either progress through the game as structured or skip ahead if they already have some experience. Word Fireworks initiates players into these languages with quick, quirky lessons and gorgeous fireworks. Sprint levels, listening stages, common-mistake challenges lead to launching your very own fireworks show!

Microsite | App Store
card image top

primes.io. (2014) Number theory explorer that visualizes Ulam spirals, the phenomenon whereby prime numbers appear with suspicious frequency when the number line is wound in a spiral. Led to a talk at Google I/O.

Try primes.io.
card image top

fractals.cc. (2014) Real-time Mandelbrot and Julia fractal explorer. Fractals are calculated on the GPU; an experiment in pixel shaders.

Try fractals.cc.

Fireworks Hero! (2015, App Store) Test your gaming skills to unlock all 10 heroes. A dozen game modes with more than 100 firework types.

Fireworks Hero! takes my educational game Word Fireworks and amps up the action. In lieu of Word Fireworks' language learning content, Fireworks Hero! challenges players' reflexes with fireworks that need to be tapped, plucked, sliced, and charged.

Fireworks Hero gif

Featuring a wide range of game modes, ranging from action-packed Arcade to creative modes to two-player versus modes. Design your own fireworks, or challenge a friend to Spark Pong. Take on the Challenge stages, or fire your own fireworks in Launcher mode. Travel around the world, from New York City to Johannesburg, from Seoul to Rio de Janiero, from Mumbai to Cairo, see the Eiffel Tower and Jerusalem's Dome of the Rock...launch beautiful fireworks over stunning skylines -- and see if you can be the next Fireworks Hero!

Microsite | App Store
card image top

attractors.io. (2014) Real-time, interactive visualization of strange attractors: dynamical systems with a fractal internal structure. A haunting beauty reveals itself.

Try attractors.io.
card image top

dyson modes. (2013) Visual-acoustic performance piece. A series of pitch-responsive abstract mechanisms are controlled via saxophone augmented with a TouchOSC interface.

dyson modes excerpt on Vimeo.com.

The Grading Game. (2013, App Store) Race against time to find spelling and grammar errors--and pay down your crippling student debt!

The Grading Game was conceived during a the Seven Day FPS gamejam. In lieu of yet another violent shooting game, I conceived a game where the player would "shoot" grammar and spelling errors. I called that prototype "First Person Tutor", and after the positive response to the prototype, I built a full game experience around the idea. The Grading Game is the result!

Grading Game gif

The Grading Game puts you in the role of a woeful grad student who, at the bidding of an unhappy tenured professor, has the unhappy task of grading the oft-plagiarized papers of his undergraduate students. As it happens, these papers seem to be cribbed from Wikipedia -- so you find yourself learning more than ever as you sift through the grammer and spelling errors!

Moreover, the procedural "level" generation injects spelling and grammar errors drawn from Wikipedia's own vast corpus of historical edit errors. The result is a plausible simulacrum of copyediting real-world text. Upgrade your copyediting skills even as you absorb quirky Wikipedia articles, ranging from the boy scout who built a nuclear reactor, to Korean fan death, a widely-held conspiracy theory!

Microsite | App Store
card image top

cubes.io. (2013) Explore shiny 3D cellular automata in this interactive explorer powered by WebGL. Discover beautiful, complex 3D tapestries arise from simple rules.

Try cubes.io.
card image top

Argument Champion. (2012) Rhetorical combat with semiotic gameplay dynamics.

Here's some fun coverage at Rock, Paper, Shotgun.

Try Argument Champion (Flash).

Lunacraft. (2012, App Store) Explore a sci-fi procedural voxel world. Infinite terrain, crafting. (Impressive stuff for the time!)

In 2012, I tackled making an infinite procedural voxel terrain engine that ran on an iPhone 3GS. I built my own engine with a mixture of multi-threaded C and Objective-C. The result was tens of thousands of positive reviews and millions of downloads. In its time, Lunacraft was a best-in-class experience on mobile.

lunacraft screenshot

Featuring varied crafting options, a variety of xenobiota, Newtonian physics, a day-night cycle, dynamic lighting, enemy astronauts, and a creative mode, Lunacraft offered a straightforward (and modestly novel) take on the burgeoning voxel exploration genre.

lunacraft screenshot

In hindsight, further development would no doubt have been lucrative. Whatever the case, I'd moved on to other, fresher challenges once Lunacraft was out in the world.

Microsite | App Store

Ratatap Drums. (2011, App Store) The drumming app with the fastest response and richest expression. 15+ built-in drumkits, and unlimited customization.

As a serious musician myself, I was (in 2010) disappointed with the offerings available for music performance and creation on mobile devices. Beginning as an experiment to see just how low latency could go on the iPhone, I dove into the unlit, unhappy labyrinth of iOS CoreAudio at that era.

ratatap image

The result was a cutting-edge (and popular!) drumming app that was featured regularly on Apple's App Store. With expandable drum sets, a vast sound library, realism-enhancing generative synthesis, and a novel simulacrum of pressure sensitivity via the accelerometer, Ratatap Drums made a compelling product for mobile musicians interested in exploring their rhythmic options.

Microsite | App Store | Youtube demo

noteplex. (2011, App Store) A visual composition toy where notes are nodes lighting up a hexagonal grid. Make music with chain reactions!

Noteplex was an exploration into an alternative representation of music. In lieu of noteheads, stems, and flags on the five-lines of Western sheet music, Noteplex presents a hexagonal matrix where differently-shaped "nodes" emit sounds and--crucially--pulses that can set off other nodes.

noteplex gif

Placing nodes of various types (and pitch and timbre) leads to music that sets itself in motion. Moreover, the creation of music itself in Noteplex can be a musical performance. Initially created in Flash, Noteplex on iOS uses a custom 2d engine to display hundreds of independent nodes within view -- a non-trivial engineering challenge in 2011!

Microsite | Youtube Demo
card image top

Parameter Ghosts. (2014) A short animated movie generated by wandering around the parameter space of one particular strange attractor system.

HD video hosted on Vimeo.com.
card image top

NESTFALL. (2015) Immersive VR total-body flying game. Mashing up body tracking and head-mounted display so players can spread their arms to fly down a virtual chasm.

Clip of NESTFALL gameplay on Vimeo.com.
card image top

VSpacewar. (2015) Local multiplayer 6DOF space arena. Gamejam prototype.

Video clip on vine.co.
card image top

United Nanolytics Knowledge Vault. (2014, LD30) Generative game made in 72 hours for the Ludum Dare 30 gamejam.

Try UNKV (Flash).
card image top

space boids. (2014) Avoid/align/approach 3d flocking, inspired by Craig Reynolds' "boids". My first Unity experiment.

Video on Vimeo.com.
card image top

emotweet. (2010) Visualize Twitter reactions. Sentiment analysis. Final project for Harvard CS171 Data Visualization. (API deprecated.)

Microsite (defunct).
card image top

Wow Such Web. (2012) Doge-ify the web. Chrome extension that swaps and alters content, such media. Wow.

Chrome Store link.
card image top

Ellsworth. (2013) Abstract puzzler inspired by the 20th century Abstractionist, Ellsworth Kelly.

Try Ellsworth (WebGL, desktop).
card image top

evophonia. (2012) Evolving rhythm game for Ludum Dare 24.

Try evophonia (Flash).
card image top

My Kingdom for a Line Block! (2012) Two-player asymmetric version of the famous tetromino game. One player chooses the worst-possible block for the other. Built for Ludum Dare 25.

Try My Kingdom For A Line Block! (Flash).
card image top

Polyglot. (2012) Inspired by browser extensions like Rikai, Polyglot was an iPhone web browser that enabled you to discover, accumulate, and practice new vocab as you encountered it while reading.

card image top

Caucus. (2012) Choose your candidate, listen to voters' preferences and transmit your positions on the issues -- but don't let the voters discover you're a flip-flopper! A real-time political strategy game about lying. Built for the Global Game Jam 2012.

Try Caucus (Flash).
card image top

Space Opera Singer. (2011) Sing the correct notes to defeat the aliens -- a sight-singing space shooter.

Try Space Opera Singer (Flash).

Legacy Work. (2007-) I created many products before those described above. The many lessons those experiences taught me informed subsequent work.

My first retail offering was a flashcard service called Flash My Brain, launched in 2007. I bootstrapped this software-as-a-service into sustainability in less than six months. A compelling offering at the time, the app featured printing onto cardstock templates, spaced repetition study, several memory games, and a rich editor supporting, among other content, RTL languages. My users created millions of flashcards, and the desktop app was eventually joined by a mobile app when Apple's App Store first launched.

flash my brain screenshot

Being one of the first published app creators led to lucrative consulting work, which in turn supported continued experimental work. So, I designed and published a family of musical instrument apps for the iPhone. These apps contrasted with the toy-like apps of the time in their true-to-life control model: Boehm fingerings for Clarinet, valves-and-partials for Trumpet, and so on.

musicofx screenshot

Music composition tools followed in 2010, including a matrix sequencer called moxMatrix and the initial Flash-based implementation of the above-described visual composition environment, Noteplex. I created the first incarnation of my experimental MIDI controller, expressionPad, for the original iPad. Meanwhile, I started to get involved with game development by participating in gamejams.

polyglot screenshot

I also continued to make creative experiments using new web technologies, such as WebGL and the various social APIs that were sprouting up like weeds at the time.

From there, you can find my favorite work above. Myriad experiments, game jam prototypes, and explorations have been omitted. Feel free to get in touch if you have questions or inquiries.

I used Jekyll and Bootstrap 4 to build this.
Last updated in 2023.