Enter generative AI. As someone who has been in the AI space for my entire career, including a PhD in large neural networks obtained back in 2010, I am generally skeptical of new AI fads; however, generative AI (forward-models more generally) are extremely powerful and I am convinced there is significant value that will be unlocked by these models over the next five-ten years.

For much of our civilization, we have operated in information scarcity. The ability to have the most popular books of the past 500 years available at the press of a button was a pipe dream for academics. It was so out of reach that it wasn’t even a dream for most. Now it’s available for all, but we now have an attention scarcity. Alice in Wonderland simply can’t compete with Hawk Tuah girl, who has a team of marketing professionals and an army of psychological weapons, not the least of which is online advertising: a tool that has minted at least two of the fortune 5 companies. Can we use generative AI to level the playing field? Can we made the information-rich books of our history pop up and capture us the way a well-curated youtube thumbnail can?

This is the hypothesis that Scott Wilcox ( https://hereandtomorrow.com/ ) and I set out to prove, and it was an incredible journey that cemented a new friendship and resulted in the hands-free illustration of over 30 books.

Getting started

Fortunately, epub files are actually just zip files with a specification that guarantees the presences and schema of some files inside the zip, namely the OEBPS/content.opf file which is an XML file. After getting some public domain books from Project Gutenberg (PG), we hacked the epub to strip out the PG headers and replace them with a uuid that allowed us to have multiple copies of the same book in our epub readers without conflict.

    with zipfile.ZipFile(output_book_path, 'w') as new_book_zip:
        with zipfile.ZipFile(input_book_path) as original_book_zip:
            manifest = original_book_zip.read("OEBPS/content.opf").decode('utf-8')

            manifest = manifest.replace("""</dc:identifier>""", f"""/{model_details}/{run_id}</dc:identifier>""")

            manifest = manifest.replace("""</dc:title>""",f""" (Version {model_details}/{run_id})</dc:title>""")

            filenames = original_book_zip.namelist()
            print(filenames)
            for file in filenames:
                if file == "OEBPS/content.opf":
                    # Already handled
                    continue
                if file.endswith('.xhtml'):
                    print("Processing",file)
                    with original_book_zip.open(file) as myfile:
                        xhtml_contents = myfile.read()

                        manifest, new_xhtml = process_xhtml(new_book_zip, xhtml_contents, manifest, text_model, image_model)
                        new_book_zip.writestr(file, new_xhtml)
                else:
                    with original_book_zip.open(file) as myfile:
                        new_book_zip.writestr(file, myfile.read())

            new_book_zip.writestr("OEBPS/content.opf", manifest)

With that out of the way, we set about creating classes for text-to-image and chat, a caching layer so we can rerun the same book without waiting for the same images to render, and html parsing to extract the content from the book and inject the images:

One major challenge that we found up-front was that, sometimes, the images were garish or completely out of place. Here’s an example of a kitten with a foot for a head:

Here’s a man with two extra pairs of legs:

The way we ended up solving this was through rejection sampling. We passed the newly-created image to a vision-language model (VLM) and asked a series of questions: Is there a person with the wrong number of legs? Is there a person with an animal head? Is there excessive blood or gore? Was this image inspired by Cthulu or some other Deep One? You get the idea. Images that didn’t pass the inspection are thrown out and regenerated with a different random seed until we either get a passing image or give up.

Another problem was the theme. Here are some pictures from the same print of 80 Days Around the World:

The protagonist changed from an 1800s Gentleman (first) to a 1950s businessman (second) to a disheveled dock worker (third). The protagonist’s assistant changed races and genders throughout the book. The general art style and theme changed massively from image to image. This was completely immersion-breaking.

Our solution was to recursively summarize the book and characters. We accumulate summaries of each section and then summarize those into shorter summaries until we build a summarization pyramid. The top of this pyramid is included with each chunk of text to ground the characters and the theme.

For some items, such as the era, we kept track of the era as we discovered it. Thus, even if a chunk did not give away the era of time that it’s based in, we could infer it from previous chunks of text.

The covers were created from the pyramid of summaries and had their own set of VLM validators.

I could write for hours about all the technically clever things we did, but let’s put a bookmark in it and move to: Reading!

Making the classics fun again

Personally, the additional illustrations and covers made these books an entirely new experience. I’m really intrigued to see what the AI imagines from the text and how it enhances my own imagination of the story. Just building this has renewed my interest in reading fiction books permanently, making it a personally extremely satisfying project.

Is it a great business? No. Sadly, the total addressable market of classic books simply isn’t there to justify this as a sustainable business. But I look at it as a community project, similar to taking a few months worth of weekends to clean up one’s local river or park.

Here’s some GIFs showing the evolution of our system over time on the same area of the book:

Conclusion

The Good: The books are delightful and the epub format was relatively simple to hack

The Bad: After buying an expensive GPU, I still paid over $800 out of pocket to do all the image generation in parallel. Even a powerful GPU is too slow compared to letting someone with a server farm go wild.

The Ugly: Many of the misshapen people and chimeras haunt me when I get migraines. You can’t un-see Alice in Wonderland with two horse butts instead of a head.

My first programmable computer was an ATARI 400. It came with a cartridge called BASIC and a manual full of with cool things that you could program. It was an amazing, mesmerizing machine. One of my favorite programs was “Delta Drawing”, a turtle drawing application. The 400 was great, but one time I lost a very elaborate drawing because someone turned off the machine and was heartbroken. I wanted a computer that could store things so I wouldn’t lose my work.

My mom searched the For Sale section of the newspaper for a few months until she came across a person selling a Commodore 64. It was an older gentleman who had decided to upgrade to the C-128 and was happy to spend some time to teach me how the C64 worked as part of the sale. I remember him visiting our house and setting it up. I remember panicking and running out of the room when he said that I can’t have powerful magnets near the computer because I had left a snap bracelet downstairs.

After spending a day with me to show me how the commodore worked, the old gentleman gave us all of his software that he had been collecting: two large caches full of disks and a box full of manuals. Either he didn’t realize that the C-128 could run C-64 software (doubtful), or he gave it to us out of generosity. It was better than Christmas.

The Commodore-64 was an amazing machine. The computer was built into the keyboard, and it included a monitor and disk drive. Optionally, one could buy a tape drive for significantly more storage (but with atrocious I/O speeds), joysticks, and other peripherals. It also came with an awesome manual that had great explanations and many programming exercises.

When you turned on the C-64, you were greeted with a BASIC interpreter prompt:

The previous owner had purchased many interactive fiction games, such as “Essex”, and “Amnesia”. Essex in particular had a grip on me. The manual was 121 pages, the majority of which was the opening chapters of a novel explaining the characters with beautifully designed, hand-drawn artwork every few pages. Many of the words were unfamiliar to me, so my mother relied on random people in the phonebook to help me out. English isn’t my parents’ native language, and my mom did not want me to have an Italian accent. Showing astounding resourcefulness, she decided to cold-call random phone numbers in the phone book and ask them to pronounce words or define them for me. Several people were more than happy to help and became regulars. With the help of these strangers I was able to win the Ontario provincial spelling bee later that year.

With the manual understood, I booted up the game. Picking up exactly where the manual left off, but then being able to enter commands and change the story, was an incredible feeling. The game seemed to understand everything I could type. The tour guide, Tiny Ed, Kaxalgazmurithar the Fraxulian Alien, they were all there! I could ask them questions about what they said in the novel. At one point I told Kaxal that he was a great friend and he thanked me for the compliment. This wasn’t part of the game plot, but I genuinely found Kaxal to be kind and thought he deserved a compliment. The tour guide led us through the ship and, near the end of the tour, the ship exploded and we all died, game over.

I played Essex many times (you can play the DOS version here https://www.myabandonware.com/game/essex-5t/play-5t using the manual from here: https://archive.org/details/Essex/page/n57/mode/2up). I could never figure out how to keep the ship from blowing up, or even how to advance the plot an inch, but this didn’t matter back then. I was captivated by the character and scenery of the ship. I was occasionally trying to figure out who was blowing up the ship, but I wasn’t frustrated that I had made no progress. This is a sharp contrast to today, where I gave up after about sixty minutes of playtime and resorted to reading a walkthrough online, and then gave up on the walkthrough because the game is too random for the walkthrough to be consistent. We get frustrated at our vintage games because we forgot what it’s like to play them as a kid. As a kid, I loved the metaphors and would spend ages thinking about them, “Tiny Ed is underfoot”, or “In the gym, you can hear people injuring themselves in the name of good health”. Playing Essex today with the memories of my original playthrough reminds me to stop and smell the roses. Sometimes I get frustrated at my kids because I forgot what it’s like to be a kid.

Beyond playing the games and running the programs that came with my Commodore, I also enjoyed writing programs and saving them to the giant floppy disks. I made text adventures, a maze game, a program that drew your name all over the screen, and my most popular game: Commodore Arm Wrestling.

Commodore Arm Wrestling was a simple game where you had to hit the spacebar X times per second. If you were able to hit it at least X times, you started winning the match, as shown by an ASCII-art drawing that refreshed every second. After enough seconds of winning, you won the entire match and entered a new match with the next opponent, who required X+1 times. This kept going until you couldn’t press the space bar fast enough. I played it a few times, but it got old quickly.

What I didn’t realize was that other people would play the game expecting to reach “the end” and get some reward. Or that my cousins would compete to get to the strongest opponent. This gave them incentive to get better and better. At some point after I got my C64, my cousin Kevin also got one. I brought my Arm Wrestling game to his house and he asked to keep it. A few weeks later, I found out that him and his brother had broken the space bar on the keyboard, putting his commodore into the infirmary. By this time, I think we were upgrading to IBMs anyways. So we can call Commodore Arm Wrestling a smashing success.

The Good

Like so many others, these early machines and the people who created content for them, captivated me and shaped my life in so many profound ways. I’ll be eternally grateful for their dedication.

The Bad

I remember my Uncle Joe coming into Kevin’s room multiple times and yelling at us that we were going to break the keyboard by pounding on the space bar. We thought that there was no way the space bar would break: it’s the biggest key on the keyboard! Being a parent is hard.

The Ugly

Many years later, my dad threw my Atari in the garbage. My children won’t be able to inherit that machine with 32 kb RAM and the ketchup smell. R.I.P. ATARI 400.

The Takeaway

It’s difficult to make something that provides more hours of entertainment than it cost to build. Take movies for example, which require thousands of hours of labor to produce 90 minutes of content. Of course a single movie entertains many people. But still, Arm Wrestling took a about an hour to program and provided my cousins over ten times as many hours of entertainment. There’s something magical about that, and I am certain that the mechanical leverage of fun is made so much more higher with computers.

There are many efforts to create languages that transpiled to JavaScript including CoffeeScript, TypeScript, Google Web Toolkit (GWT), and Dart. Currently, TypeScript from Microsoft appears to be the clear winner. TypeScript has nice integrations with webpack and the ability to slowly and painlessly migrate files from JavaScript. But, this story begins long before TypeScript rose to dominate the JavaScript transpiler wars.

It was 2011 and Google was pushing Dart really hard. The brand new Google TV frontend was set to be entirely written in Dart, and many people who were writing Java for the web using GWT were excited to see a java-like syntax for the browser with nicer Javascript interop. As part of promoting Dart, the product managers for Dart organized a hackathon, where small teams of engineers work together over a 24 hour period to produce cool projects that show off the power of Dart.

I decided to enter the hackathon with the goal of converting a Java Nintendo emulator called vNES to Dart. Because the languages are similar, it was a matter of porting the logic to Dart and then porting the UI from Java2D and Java audio to the corresponding Dart interfaces. To help with the graphics and audio work, we also looked at jsnes, a javascript nes emulator.

This all seems pretty straight-forward, except for two challenges:

1. Dart was a very immature language at the time, and many HTML5 features were not supported, so we knew we were going to have to hack a lot of it together with the help of the Dartlang developers

2. The entire hackathon, from idea to completion, has to be done in 24 hours.

Let’s pause right here. The proposal is to port/write 15,000 lines of a language that almost no one had seen before, to build an emulator for a machine that we didn’t have the specification for, in 24 hours of wall-time, to (maybe) win a plastic trophy. It’s absurd on its face, and yet I was shocked to find more than a few people who were as excited as I was to build this. What motivated people was the vision of the finished product (super mario running in a browser window), the challenge of actually finishing the product in a day, the opportunity to work on cutting-edge technology built by people we all respected (dartlang), and the fact that anyone who joined this project also shared these values and/or was equally insane. All the talk of product-market-fit, minimum viable products, network effects, etc. doesn’t apply here. Dartendo is a maximally unviable product. Dartendo is art for art’s sake.

In 2011, building an emulator for the web was extremely difficult. One of the key challenges was that fixed integer support in Javascript was very poor and things like WebAssembly didn’t exist. Thus, we needed to apply many bitmasks to the java code in places where they expected an overflow, handle integer division correctly, etc.. There were many places where we could introduce errors so we needed to have plenty of time to do tired debugging. To get this all done efficiently, we broke down the project into several tasks:

1. Porting the core logic from java (vNES) to dart.

2. Porting the mappers from java to dart.

3. Porting the audio/video interfaces from javascript (jsnes) to dart

4. Setting up AppEngine, the frontend html/css and input handling

5. Handling ROM loading into html5 localstorage, and deserialization

Everything after that was kind of a blur. I can’t remember who worked on what but we collectively wrote over 13000 lines of code in a single evening. I remember stepping through the dart and java debuggers line-by-line to catch many numerical issues. Unlike normal game development where you can cut features, with an emulator it either works or it doesn’t. I remember being sure that there was no way this was going to work by the deadline. I remember when we saw the opening curtain of Mario 3 in the wrong shade of red and everyone huddled around the laptop and cheered while memory leaks ground the laptop to a halt. It was magical. It was epic. It was the memory leak that caused me to hard reboot my laptop and lose hours of work.

It all came together in the last few hours. We ended up winning the hackathon and each got one of these plastic trophies. The base is hollow but it holds up a ton of sentimental value.

Here we are, looking very tired:

We made Dartendo available on the Internet and posted it on Hacker News ( https://news.ycombinator.com/item?id=3689299 ). At the time I didn’t realize that it required a bleeding-edge version of chrome and a very fast computer. Most people couldn’t run it (as you can see from the comments).

And this is where the story would end, except that I was rummaging through some old boxes yesterday and found the dart trophy, which made me search for “Dartendo” on the web, which led me to the aforementioned hackernews post, which made me a little sad. It was such a cool project, but it was built on such an early version of dart that no one outside a few people in Google could use it. So I set out to bring Dartendo back to life.

Fortunately, the repo was still available on googlecode (a github-like website that went into archive mode years ago). Upon opening the project with the latest dart environment, I was met with hundreds of compiler errors. Fortunately, many of them could be fixed with some regular expressions applied across all the files. The biggest change was that modern browsers block websites from playing audio before user interaction with the site. As a result, I had to add a “click to play” feature. The vast majority of the fixes came from googling answers and applying them.

After about a half-day of hacking, Dartendo was alive again! https://mistertea.github.io/Dartendo/

The project is now hosted on github so anyone can easily browse at the source: https://github.com/MisterTea/Dartendo . Maybe someone will add on-screen buttons for mobile, or joystick support one day.

The Good

Dartendo was an amazing project. I still keep in contact with several of the folks I met that day. Working on emulators is incredibly satisfying, and I went on to build MAMEHub, which lets people around the World share an arcade together over the Internet.

The Bad

Although Dartendo runs flawlessly now, at the time the technology and hardware wasn’t where it needed to be and the game stuttered too much to be playable.

The Ugly

At one point, I had to meet some sketchy guy in a Walmart parking lot because some video game company (which will go unnamed) served me with a cease-and-desist letter for working on emulators.

The Takeaway

No project dies, it just goes into hibernation. Also, keep backups.

While I was at Google Research, I became super interested in data mining and unsupervised learning. Manually labelling data to train a machine learning model or to do some natural language processes was a lot of work. Even if there were companies such as Crowdflower to help with it, I’ve never liked the idea of someone sitting down with scores of mundane work in service of some hungry Artificial Intelligence algorithm.

Wikipedia is often used as a source of data mining and unsupervised learning for these reasons:

1. All of the data can be downloaded in bulk and there are several formats for different methods.

2. The view counts for every Wikipedia page, aggregated by day, can also be readily downloaded in bulk, meaning that it’s easy to focus on the top content and crowd-source other behavioral information.

3. Because pages reference each other and there are categories, Wikipedia contains a nice graph of information that is extremely useful.

So the idea behind Trivipedia was to make the world’s largest trivia game by scraping popular facts from Wikipedia and turning them into multiple-choice questions where players had to guess the right answer for points.

The algorithm to generate the questions works like this:

1. Fetch the most popular X pages on Wikipedia based on views.
2. For each page, get the 3 nearest neighboring pages based on the page graph.
3. Get sentences inside the page containing the title of the page.
4. Replace the page title with blanks
5. Create possible answers using the page title and the titles of the nearest neighbors.

This approach worked well! Some questions are extremely difficult (e.g. can you figure out Japan is Japan just from the amount of landmass?) but overall it was stellar. With some player data, the nonsense questions could be thrown out over time as more people played the game.

Surprisingly, the most viewed pages on Wikipedia are about porn stars. Even on Wikipedia! How many times does one read about the Early Life of Sasha Grey? Fortunately, all of the porn star pages belong to a porn star category so it was easy to remove them. Someone reading this will make a Pornpedia app and become a millionaire.

The Good

Trivipedia was incredibly fun and one of these days I need to bring it back from the dead. I loved the graph of people’s performance in your geographic location:

The Bad

With no error handling or reporting of any kind, it wasn’t clear how to grow the app’s popularity and it faded into disrepair.

The Ugly

Despite the server being down for years, I don’t know how to remove the app from the Amazon app store. People are still buying it and demanding a refund.

The Takeaway

I learned a ton about mobile development and about data mining. The data mining was wildly successful and it showed me how incredibly important semi-supervised methods are. I ended up adding semi-supervised support to Sibyl during my tenure at Google Research, which helped many teams leverage unsupervised models to improve performance of their supervised models. I also remember a bunch of random factoids such as the cup size of Japan and the number of square miles in Sasha Grey.

There were a few programs available online to solve Pre-Calc and AP Calculus problems and students with the PC-calculator cable downloaded and shared them with the class. One example was QUAD2, which displayed the simplified quadratic formula given the coefficients. I loved QUAD2 and the idea of letting my calculator take the calculus tests for me was really exciting. Early in the semester, I decided to learn TI-BASIC, jump ahead in the pre-calc textbook, and code up solvers to many of the questions I would answer that school year. This also involved merging all of the code from QUAD2 and others into one massive TI-BASIC program called JGMATH.

Needless to say, my classmates loved these programs, but most of the teachers did not. The exception was my AP Calculus teacher: Dr Phingstag. Dr Phingstag was an extremely witty and empathetic man. I don’t know if he had a doctorate or we called him doctor because he was very smart. He day-traded in between classes and hosted the chess club after school. One time at some stuff-on-the-walls restaurant (I think it was TGIFridays) I happened to be sitting a couple of tables away from Dr and Mrs Phingstag. It was inspiring to see the two of them having a pleasant conversation. This may not make sense to a lot of readers, but seeing a contented older couple had such a powerful impact on me.

Dr Phingstag gave nicknames to many of the students in his class. Most of them got numbers, because “prisoners don’t have names, only numbers.” My nickname was JGMATH. Other students began calling me JG or JGMATH outside of class. This nickname stuck and became part of the school folklore after I graduated. When I went back to visit Dr Phingstag years later, students were still using JGMATH. The class did not believe that JGMATH was a real person. Dr Phingstag deduced that I must be a descendant of Pythagoras because I copied QUAD2 into JGMATH, similar to how Pythagoras stole the Pythagorean Theorem from “real” mathematicians.

Joe Brum was another student in my Pre-Calc class who was great at TI BASIC. Joe and I worked together on an procedurally-generated video game (we’ll talk about that in another post) and some joke apps that spread with virality through the school. One day, Joe figured out how to dial the school PA system from an outside line and broadcast the cancellation of school two hours early. Joe and I had a lot in common and got along well.

About half-way through Junior Year, disaster struck. Someone was caught storing written answers to chemistry questions on their calculator and the teacher was furious. The school staff decided on a new policy to wipe our calculators before the start of each exam. People who depended on JGMATH to pass calculus (not my intention when I wrote it) panicked. I couldn’t afford a computer link or another calculator, so I was going to lose hundreds of hours of work. Joe and I put our heads together.

The way teachers wiped the calculators was by having all students go to the “10: ARE YOU SURE YOU WANT TO WIPE ALL DATA?” screen when class started. Then the teachers would walk down the rows pressing the OK button, see the calculator begin to factory reset, and move down the line, similar to how a warden checks the cells of their inmates. When the teachers got to our calculators, they were also displaying “10: ARE YOU SURE YOU WANT TO WIPE ALL DATA?”. Upon pressing OK, JGMATH would count by ones to a billion and then return, data intact. The latest and greatest tool in the JGMATH tool belt: a fake factory reset app.

I still remember Joe’s hysterical laughter after we pulled it off. It’s by far the most euphoric I’d ever seen him. After the fake factory reset trick went mainstream, the school decided to mandate regular calculators for most subjects and graphing calculators for advanced math. Dr Phingstag never asked anyone to wipe their calculators before a test.

The good

Writing solvers for math tests was really fun and the first program I wrote that was useful to many people. During my senior year, I was hired by the Orange County School District to help kickstart an online learning program called the “Florida Virtual School". I worked there for five years, which helped pay for college expenses and provided education to thousands of sick, disabled, or otherwise home-bound children.

The bad

I don’t know what ever happened to the JGMATH program. By the time I could get a TI Computer link, my calculator had died and I lost those programs. I figure by now people use their smart phones or have an iPad in the classroom so the JGMATH program is in digital Valhalla.

The Ugly

Later that Junior year, Joe dropped out of high school. I don’t know this for sure, but I believe there were problems at home and with drugs. The last time I spoke to him in class, he said he would join the navy on a nuclear submarine. To this day I don’t know if that was a lucid plan or not. Joe Brum, if you are still alive out there, I hope you joined the military like you planned. You were always super smart.

The Takeaway

It may seem crazy to lose hundreds of hours of programming, and it is definitely a bummer. But there are so many projects, so many ideas, that it didn’t seem significant at the time. Fast forwarding to today: work has it’s ups and downs. When the downs come, we can remember ourselves as children, coding for free and for the fun of it. That’s what can keep us in the game when times are tough.

We always underestimate how much things change in the future. Your “permanent record” isn’t permanent. In fact, very few bad things are permanent. So, it’s important to take a step back and laugh things off. Laughter is the best medicine. Unless you have diabetes, then insulin is the best medicine.

“Whatever you do, you must give it 100%. You can’t be any garbage man. But you can be the best garbage man if that’s what you want to do.”

My mother is an endless source of advice and inspiration. My Father’s advice was slightly different:

“You are on the computer all day. You talk on the computer; you play games on the computer; you read on the computer. Now you want to get a degree in computers. College should be for doctors and lawyers, not TV or computer repairmen”

Through luck, gifts, and the grace of God, I scored in the top 1% on the PSAT and earned the National Merit Scholarship. I would love to talk about all the hours at home studying (that never happened) or my GPA (that sucked), but I can’t be proud of any of that. What I did have was an obsession with programming computers. I had an obsession with reading books. I also was lucky enough to be part of a magnet program.

Only a few states have “magnet” high school programs. The idea with magnet schools is that students who show proficiency in a subject can be bussed to a faraway school that specializes in that subject. As someone who grew up surrounded by failing schools, this was a life-saver. Through the magnet program, I could meet parents and teachers who went to college and learn from their experiences. One teacher suggested that I take an SAT Prep class since my freshman PSAT score was 1220/1600, which was high for my school.

SAT Prep class was amazing. The SAT is a multiple-choice test full of patterns. The way the wrong answers ladder up to the correct one is a pattern. The way they try to screw you by having the answer that is one careless mistake away from correct is a pattern. It turns out that memorizing these patterns is something I can be really good at. After taking SAT Prep my score went from 1220 to 1560/1600, meaning that I could go to College just about anywhere in the state without having to pay the $75 application fee. Winning the National Merit scholarship meant that I was going to do something that no one else among my family or friends had done: go to college.

Choosing a college was pretty easy. My mother told me that if I lived in a dorm, I would get bacterial meningitis and lose my limbs. That narrowed down my choices to Valencia Community College and the University of Central Florida. In hindsight, it doesn’t make a lot of sense to optimize for bacterial meningitis. I’m extremely lucky to have a top-100 computer science university within driving distance. Many companies refuse to open offices in SLC, Austin, Boulder, or other cities because “there isn’t enough talent”. I wish they knew that the country is peppered with boy- and girl-geniuses who will be staying local for college because it’s their home, because of some bad advice, or because of bacterial meningitis.

The Good

College was an amazing experience. I enjoyed it so much I went back two more times, thanks to the Presidential Fellowship. Because these scholarships guarantee attendance, I have never actually applied to college despite getting three degrees.

The Bad

Tenure, nepotism, and in-group behavior has made college an echo chamber and much less fun. The academic industrial complex has become Frankenstein’s Monster, and is ripe for disruption.

The Ugly

There was a vaccine for bacterial meningitis widely available even when I was in high school.

The Takeaway

On the surface this reads as an academic rags-to-riches story, with the failing public school system and the low GPA and the irrational fear of meningitis. But what I have discovered after years of reflection are all the small-but-important efforts of countless Good Samaritans along the way. I remember my AP Physics teacher driving me to a chess tournament. I remember my English teacher helping me submit poetry to a magazine for review. My parents didn’t give always the best advice but they cared deeply about me and obsessed over my education. It used to take a village to raise a child, but now it takes a whole nation. I hope that we can continue to make the next generation even better than the ones that came before.

Enter UHC: Underworld Hockey Club. I integrated the bullet 3d physics engine with the Open Source Graphics Rendering Engine (OGRE). I didn’t know anything about databases, art pipelines, or making games. Deciding when to turn a player into a ragdoll was a tough math problem that I enjoyed. The game was really fun for a few minutes, but without any progression or story (what can you say about skeleton armies playing hockey in hell?), it gives away the farm and has no redeemable gameplay value. Still, it’s pretty fun to smash skeletons into walls and maybe someone can bring it back from the dead.

My first PhD advisor was an angry man with a handlebar moustache. He was less than amused with Underworld Hockey Club. About half way through development of UHC, my PhD advisor spontaneously kicked me out of the lab and gave me an F on a “research credits” course that was supposed to be non-graded. I assumed this was because I accomplished almost no research of value, but then I found out later that this happened to several other students in the lab, even some productive ones. If you ever wanted to know What happens to professors with tenure when a regular employee would have gotten fired?, I know the answer. They end up receiving random titles & responsibilities. Is your PhD advisor “Dean of Student Library Studies”? Run away. In hindsight, getting kicked out of the Graphics lab was one of the best things to happen to me, although I was really upset about it at the time.

After a couple of years of part-time work on UHC, I submitted a video and a copy of the game to the Indie Awards. It received terrible reviews. I’m still really proud of being able to cobble this together with no prior knowledge of game development. The feeling of checking another player through the glass is euphoric. Being able to body check the Devil is satisfying. Having a vision up-front ensures that you don’t rely solely on other people to define your success.

Here’s a very silly pitch video for Underworld Hockey Club:

The Good

The moments where the skater-to-ragdoll worked *just* right is so incredibly satisfying. It’s like a birdwatcher getting that perfect picture, except you are watching a demon get thrown through a sheet of glass or a skeleton get body slammed.

The Bad

There were no unit tests in the code and I built many things by hand (e.g. my own database) so bugs grinded the production of the game to a halt.

The Ugly

Because there aren’t many graded courses in a PhD program, I carry an abysmal PhD GPA. I still remember the fear struck through me when Google asked for my PhD GPA as part of the hiring process.

The Takeaway

I learned almost everything in UHC the “hard way”. Reading books/blogs or watching videos on game design would have saved countless hours and resulted in a much better game. Building UHC taught me that I need to do my research before starting a project. In that sense, it was an extremely valuable research project. I think I deserve an A for that. Ok maybe not.

After UHC, I became more methodical about reading papers and discussing ideas before diving into the code.

It was 2018 and I found myself in a robotics + AI Meetup. Talking to real mechanical/electrical engineers gave me maker fever. All of these years of engineering and I had never actually built anything.

My wife expressed a reasonable amount of concern. This sounded expensive and dangerous, until I found that 3-D printer kits could be purchased from China for $150. Then it only sounded dangerous. After months of discussion, I decided to take the leap and order a kit.

A month later, the kit arrives in a cardboard box. One of the things I unpacked was a cord with a wall plug on one end and three bare wires on the other. What??? A feeling of fear and regret washed over. I was most worried that I would be out $150 and/or burn the house down. I didn’t like the idea that, even small mistakes, could cost something material. Working in the realm of bits means that you can only lose time, but working in the realm of atoms means that you can lose a circuit board or a finger.

In hindsight, I should have taken some classes or read a book on electrical engineering. Fortunately, there were some very detailed videos explaining how to build the printer step-by-step narrated in Mandarin. After a few evenings of assembly, it was ready to rock. Without a multimeter to guide me, it was a leap of faith to plug the cord into the wall. I hold my breath and plug it in.

Nothing happened.

Then the screen turned on.

Then it went off again.

Then I smelled burning plastic, and saw smoke rising out of the circuit board.

I unplugged the cable and my heart sank. All of that back-and-forth to get approval, all of those days waiting for the package to arrive. I looked at where the burning came from, and noticed that some copper strands from the wires that went into the circuit board were touching. This caused the plastic cap to melt, but the board itself looked fine. With some electrical tape as shielding, the system booted fine and worked! It came with an schematic to print, a small cylinder with a Chinese symbol on it.

The Good

The printer lasted for several years and is a part of a few successful and many failed projects. Building a 3d printer from scratch was an amazing experience and a cost-effective way to get into 3d printing. Being able to make almost anything is extremely satisfying.

The Bad

Cheap printers require constant maintenance because the prints stick to the cheap levelling plate, the parts are janky and weak, and the amount of assembly means that there will be things that aren’t aligned. In hindsight, I should have upgraded to a better printer much sooner. Because I didn’t:

The Ugly

Years later, while I was asleep, the heat finally melted away the tape I had forgotten about and the circuit board was completely annihilated by 120 volts of raw, sustained energy. This caused the entire garage to fill with toxic smoke and an acrid smell that never fully went away. This is probably what the fifth circle of Dante’s Inferno smells like.

The Takeaway

Beyond the satisfaction of building an electrical appliance from parts, I still have this awesome set of screw-puzzles.

This is an autobiography of all the countless projects, finished and unfinished, that I’ve at least started to build. Along the way, readers get a shotgun seat into a rollercoaster ride through the pursuits of grandiose visions and delusions. Although each post is a separate project, these narratives span decades and, when combined, they weave the fabric of my mind. Every project spins a thread that oscillates between dormancy and mania. Many people have a false stereotype of the inventor as a Thomas Edison, solely focused on trying lightbulb filaments for years. The life of an inventor, particularly one creating software, is one of extraordinary serendipity. It’s both trial-by-error and trial-by-fire.

This is me. As of writing this I’m a senior manager at Facebook AI leading a project named ReAgent that I founded in 2016. ReAgent lets computers and people work together to make better decisions. The ReAgent AI makes about 200,000,000,000 decisions every day for Facebook, and probably even more than that across all the other companies that use it (it’s open-source at https://www.reagent.ai/ ). Although its been six years since the project started, I can still remember working 24-hour days during the most exciting phase of version 1.0. There’s something surreal about getting to your desk, seeing the morning cleaning crew twice over a span of 24 hours, then leaving with a mind full of new ideas that will have to wait until I could come back in. Before Facebook I worked in research labs across silicon valley, but we’ll get to all that later.

I co-host the Programming Throwdown podcast with Patrick Wheeler. We’ve introduced programming and engineering to hundreds of thousands of people around the world. Anyone can grab a tutorial on how to write code, but we let people experience what it’s like to do the job. Our goal is to take “Google” from some ephemeral product to an actual building you can walk into, swipe your badge, sit at your desk, and work at.

The best project to date has been the Gauci Family project. I’m incredibly blessed to have such an understanding wife and amazing family. Even more important than inventing some new project is to take the time to appreciate all of the gifts life has to offer, and be mentally ready for all the challenge life throws in the way.

Let’s talk about some projects!