Tech Blog on ✰Vicki Boykis✰

Build yourself flowers

Mon, 20 Apr 2026 00:00:00 +0000

This is an edited transcript of the keynote I gave at the Applied Machine Learning Conference in Charlottesville, VA in April 2026.

I first wrote a draft of this talk by hand. This part took 2 months.

I then recorded myself giving a version of this talk with MacWhisper, and transcribed it with Whisper locally. This part took 45 minutes (the total time of my practice run.)

Then, I ran it through Gemini Flash 2.5 running in Pi to break into paragraphs. I also had Gemini break up my slide deck from a PDF I generated from Google Slides, into individual images to insert into the blog post and optimize the image format to webp for blog rendering. This part took about 10 minutes.

Then I went through the text paragraph by paragraph manually to make corrections, remove redundant phrasing and pauses, and added clarifications to make it more legible than a talk. This part took 3 hours.

Before all that, I generated the content for this talk. This part took 13 years.

I’m Vicki, and I build machine learning systems.

I debated for a long time how to introduce myself. Am I a data scientist? Am I still a machine learning engineer? Am I an AI engineer now? I’m not really sure. I think, like a lot of people over the past six months in the industry, I’ve been having existential angst. So, I’ll go with “I build machine learning systems.”

I’ve built and broken systems at Tumblr, at Automattic, at Duo, at Mozilla.ai. Now, I build realtime personalization and search systems at Malachyte.

We’ve had a lot of different conversations in all of the corners of the internet about how we should incorporate LLMs into machine learning workflows. And the question that came to me is, if all we’re doing is generative AI, where does traditional machine learning even fit in here? Are we still doing machine learning at all? So, the larger question behind my existential angst is, what is the state of machine learning engineering as an industry today? That is - is it still worth doing machine learning engineering?

And the second question that came to me was, not only is it still worth doing ML, but, in an era where we’re having LLMs generate a lot of code, when the most important thing is for us to ship quickly, to get to a prototype quickly, is it still worth doing machine learning well?

So hopefully we’ll be able to definitively solve all of my existential crises in 45 minutes, but until then, I wanted to take a step back and talk about flowers.

In 2024, the lovely folks at PyData Amsterdam, invited me to give a keynote. I called it Build and Keep Your Context Window. ChatGPT was still fairly new, out for just about a year and a half at the time, and everyone was freaking out. I talked about how important it is to build and keep your own context window - the historical understanding of software tools and concepts, so that we can use the new tools and understand where they fit into the context of existing software engineering.

The analogy I used was the ChatGPT UI. At the time, everybody was using the text interface, where you have a sidebar and you have the big text box. All your chat sessions live in the sidebar, and the sessions have titles. And I talked about that as being part of a context window.

The context window is the amount of text that a model can recall at any given time, as measured in tokens. If the text you input overflows the context window, the model loses the thread of the conversation and can’t reply coherently anymore. What determines the context window in reply is the attention mechanism in the Transformers model. And the attention mechanism is really a cache, which is really a hash map. So if you understand all these four things and how they fit together as concepts that naturally emerge over time successively, it’s not perhaps as surprising that we get the emergence of LLMs.

Before I gave this talk, I got to spend a few days in Amsterdam (I was told they were the only two sunny days in the Netherlands in September) and particularly in the Rijksmuseum, which is the national Dutch museum of art.

It’s absolutely enormous and beautiful. You could spend days there. But I only had a few hours, so of course I went to see the headline piece, which is the Night Watch by Rembrandt. When I went to see it, it was covered by this big plastic window and there was machinery set up all around it. I was thinking, what is this? Why is this blocking the painting?

When I went later to PyData Amsterdam, Robert Erdmann, who was working at the Rijksmuseum, gave an incredible talk about how he was using deep learning-based ink-removal, especially 3D imaging, pixel detection, and high resolution photography to see details in art at the museum that nobody was able to see before.

So for example, for The Night Watch, they set up all this equipment to be able to photograph it many times in many resolutions. And as a result, they were able to get high resolution pictures like the figure on the left, who is believed to be actually Rembrandt, the artist, in a cameo. And if you really zoom in, further, further, you can see in his eye there’s a little white sliver.

And so the question is, how do you become good enough that you know if you put just a little bit of a white dot, it will render as an iris of an eye to somebody? And so those are the kinds of questions, around the mystery of mastery, that he was now able to explore as a result of this photography.

Rachel Ruysch

One of the artists whose paintings are also on display at the Rijksmuseum is Rachel Ruysch, one of the premier flower painters during the Dutch Golden Age, and that’s who I really want to talk about. Being a premier painter of the Dutch Golden Age was a huge accomplishment, and I want to understand more about how she works, because maybe we can use her as a guide to navigate this crazy time where we’re doing a lot of code generation with LLMs.

So who was she? Rachel lived in Amsterdam, and all she did was paint flowers. Lots and lots of different compositions of flowers in all sorts of shapes and sizes, starting from when she was 15, well into her 80s.

Let’s take a step back and talk about the Dutch Golden Age for a minute. What made this era of creativity possible? In the late 1500s, the Dutch won independence from the Spanish government. The political stability was followed by economic stability and the rise of the Dutch empire. As Dutch life got significantly easier and more people joined the growing middle class, they wanted to buy art. In response, art, became seen as an important aspect of the Dutch national character, and art which had previously been mostly religious in theme, became secular. People started painting scenes from real life, science, and nature. All of this resulted in millions of paintings, many of them scientific or representative of everyday life.

What people wanted to see were pictures of everyday life that were accurate, that were true to what was going on around them, of local scenes, of scientific discovery. Of themes of religion and mortality, but conveyed in a different way.

So they wanted to paint stuff like gentlemen smoking and playing backgammon in a tavern. They wanted to paint stuff like people skating in the winter.

They wanted to paint stuff like people getting rowdy in their house while the mistress was asleep.

And if you look at all these paintings, you can see something come through, which is that the colors are down to earth. They’re very normcore.

Another thing people wanted was a lot of different paintings of flowers, hundreds and thousands of paintings of flowers.

Out of the five to ten million paintings that we think were created during the Dutch Golden Age, thousands of them were flowers. Why flowers? People wanted flowers in their homes that wouldn’t wilt over the long Dutch winters, and that showcased how worldly and interesting the homeowners were. Exotic and unusual flowers in particular (such as tulips of the famous Dutch tulip mania) were prized, and flower painters sought to accommodate this trend. There were many very famous Dutch flower painters and hundreds to thousands more, all producing and focusing on flowers. For artists, painting flowers accurately and lifelike was an important test of mastery.

And so Rachel Ruysch came into her skills at a time when people demanded this. And she was really good at it. And so the question I had, was that, if you have all these people painting flowers, why is it important to paint good flowers? Why do you want to stand out?

And the same question of, in a world where it’s easy and fast to write code, why is technical excellence still important?

As I was researching this, I came across a report from NASA

NASA has been on my mind lately with the Artemis II mission and their orbit of the moon and their triumphant return home. Because spaceflight is so hard and so enormously error-prone NASA has been continuously on a loop to improve quality, because it’s already very dangerous to be an astronaut at NASA. I’d seen somewhere that the Artemis crew knew they had a 1 in 30 chance of dying.

But NASA is always looking at risk, and they did an investigation in 2012 and came out with this report of what happens when things fail. Why do they fail? And they came up with these five problems. And the one that really struck out to me was the first one, shifting engineering excellence to insight and oversight.

So there were periods when NASA outsourced a lot of their engineering to subcontractors. And what happened was that the engineers who understood the design and the system design based on actual experience left. And the shift that resulted eliminated independent analysis and testing and understanding of the systems. And this was what led to degradation of engineering quality.

And if it sounds familiar, this is because it’s a very easy analogy for what happens when we outsource our work as engineers and data scientists and machine learning engineers entirely to AI.

So, striving for technical excellence in industry saves us from engineering, product, and cost problems. Quality is important.

The second reason is that software engineering is a craft, just like painting.

And it takes time to get good at painting. It takes time to understand where to put the little white iris. It takes time to get good at painting flowers. And so it’s important to understand that because it takes time to get to quality.

And finally, technical excellence is important because striving for mastery is just about what it means to be human.

We like when people do things well. We appreciate people who are competent. We like it when systems are designed and built well. It feels good for us to be part of systems that work. One tangential idea that I’ve seen going around the internet is that you shouldn’t talk down to LLMs, or call them “stupid”. Not because they understand, they’re just models, but because it’s bad for us as humans. In that same way, it’s important for us to strive for good things as humans.

Okay, so if we want to paint good flowers, how do we become good at painting good flowers? What made Rachel technically excellent? What might we be able to replicate?

The first one is that she was in an artistic family. So she was in one of the premier families of Amsterdam where all her great uncles were painters. Her grandfather was involved in art. Her father was a great artist and botanist, Frederick Ruysch. You can see him here with this creepy skull. He collected a lot of really creepy dead things, which is where she learned about scientific accuracy, embalming, preserving flowers, etc.

It was here that she also learned discipline: how to capture minute details, She was known for her technical skill, and for gathering flowers in botanic gardens and pressing them so she could have different compositions for different seasons, surprising the Dutch, who were generally orderly and used to seeing only seasonal bouquets. During her life, her paintings sold for 3x Rembrandt’s.

Then when she was 15, she was apprenticed to Willem van Aelst, who was one of the premier painters in Amsterdam at the time. It was tradition at the time for masters to take on apprentices who would live with them and start with stuff like washing their paintbrushes and work all the way up to painting, and then finally produce a work of art which, if the master accepted, the apprentice could become a master in their own right. So she had a very big support system and she was steeped in a culture of mentorship being extremely important.

Second, she painted her whole life. She was keenly interested in botany, in science, and made sure that her art reflected the true appearance of nature. She spent a long time - over 60 years, honing and working on her career, not even slowing down to have ten children. For her, her career was a vocation. She got inspiration from many different places, and she would do the same thing, again and again, but differently.

And finally, she remixed and experimented. So you can see in the background of this particular painting, there’s a cactus. And it’s not easy to get a cactus in Amsterdam in the summer or the winter. So she got this from the New World because she had connections to Amsterdam botanical societies and she knew people who were coming and going all the time. She bartered for and preserved exotic flowers. And she also remixed flowers from different seasons by drying them. She collected them all the time. She was always changing something about how she was working.

So, given all of this, how can we, as engineers, be like Rachel?

Building Rijksearch

I thought about all this and decided to see how I could apply these ideas to a concrete side project that I was working on. And the side project is called Rijksearch, a semantic search engine.

A semantic search engine searches by intent rather than keyword. So what you do is you type in “chill dude” and it’ll return you this portrait of, for example, a man with a dancing dog. Which I think is actually a really good example result. Or like a “confidential chat” or “man smoking pipe”, the last one for sure being a chill dude. So it’s a search engine that works by intent.

Or, you type in “pensive” and it will return a philosopher in his study, which is definitely pensive, a boy with a golf club probably not, you don’t want toddlers to start thinking too hard about what they can do with a golf club. But like, ok, these results are not bad!

Or “flower party”, which didn’t work that well because you can see it’s doing more keyword matching for “flower” than semantic matching for “party”, but this is where the craft of tuning semantic search comes into play.

There are already a lot of wonderful tools that the Rijksmuseum itself has that you can build with on public data and that they’ve built with like Art Explorer.

But I decided to build my own because I wanted to remix concepts. Rachel learned how to remix as an apprentice and from the artists she talked to. I learned it from working at Tumblr where the reblog feature was one of the original places on the social web that implemented the functionality of you posting something and somebody else riffing on top of it.

So what was I thinking about remixing? Oh, I had a lot of stuff going in my head. First, over the past couple years, I’ve been thinking about embeddings a lot. In fact, I’m thinking about embeddings right now.

In search and recommendations, embeddings - vector representations of text or images (or other media) - matter a lot. They also matter within LLMs themselves because positional embeddings are how we decide what concepts are related to each other when we train the models. I wrote a really long text about them, examining their place in ML systems, called What Are Embeddings?, and I spent a lot of time understanding how they fit into the context of our workflows.

Second, I’ve also been working with Redis for a while now. It’s one of my favorite pieces of software in the world because it just works. It’s extremely cleanly designed. It’s really fast. It’s extremely reliable. It is just very elegantly put together. So I had worked with Redis for search before, and I wanted to see if I could use it again. At the time, about a year back, Salvatore Sanfilippo, antirez, who initially wrote Redis, picked up working on it again and created a new data structure called vector sets.

Vector sets in Redis work much like sorted sets, which are collections of unique, non-repeating strings sorted by a score. In the case of vector sets, the string elements are associated with a vector. For anyone working with embeddings, this unlocks a huge amount of very fast, efficient vector operations, and they allow for vector similarity lookups using HNSW, a popular and efficient approximate nearest neighbor search algorithm.

Third, I’ve also been hacking around AtProto, which is the protocol that powers Bluesky, a social network that has become popular in the past couple years. It turns out that a lot of the Bluesky codebases are open, and when I started looking around, it turned out that a lot of Bluesky services were written in Go. (And TypeScript, but I scoped myself.)

Fourth, I’ve worked in Python (and Java and Scala, unfortunately), before, and I was curious to explore a new language that was meant for high concurrency and throughput and was the spiritual successor to Java. I did a couple projects with atproto where I created a feed of content in Go, and wanted to see what else it could do.

Of course, all the other stuff swirling around in my head was LLMs.

It’s impossible not to hear about them. I wish I could block the news out because it does feel like it distracts me enormously in my day-to-day life. What I’m particularly interested in with LLMs is the power of small, local models. So when llama.cpp came out and you could use models directly, or with apps like LM Studio to serve GGUF artifacts, I became interested in that.

I also became interested in hosted embedding APIs because before I’d been working with hosted models that you either train or pre-train models that you have to serve. And OpenAI made those accessible as API endpoints, and Gemini started serving new multimodal embedding models, so I became interested in those.

And then finally, the humanities. A lot of my projects revolve partially around the humanities. A lot of the stuff that I write ties back in some way to the humanities because I don’t think you can really have a balanced view of what developing a good product looks like if you’re just looking at tech. And, I like art. So I’m always down to look at and use art.

Build the same thing again and again

All of this stuff percolated for a year or so in my mind, and the result was that, at some point, I realized I wanted to re-implement Viberary, which is a semantic search engine.

A quick note on search engines. They come in several different forms. There’s keyword search, which is if you type in Apple, you get back results for Apple varieties. And there’s semantic search, which is you type in Apple and you get back iPhone, and maybe also Android, depending on what you’re looking for, which is why the “chill dude” query works.

And there are many different ways to implement semantic search. The way that I had implemented it before, one of the ways, was in Viberary.

Viberary was a side project that I’d done in 2023 that was based on text-only representation of books from Goodreads and all their reviews. And the idea was that you type in a phrase like sci-fi, funny sci-fi, or moody drama, and you get back a list of books that are recommended based on vibe rather than the books themselves, which is something that I always wanted.

I’m not always in the mood for a particular title, but I’m sometimes in the mood for a particular type of book. And so you can see for this one, funny Sci-Fi Space with Feeling, one of the results you get back is Asimov, which I guess, yes, kind of, and Haruki Murakami, which kind of also makes sense? So these vibes kind of fit.

What allows us to do this is that we’re not comparing just text or images. We project them into the embedding space by creating dense vectors out of them and comparing the vectors, the numerical representations.

That way, we can compare “dog” and “bird” and “fly” and see that “bird” and “fly” are closer to each other because they’re semantically similar. And this is also essentially the way the transformer attention mechanism also works on a much, much, much larger scale.

It used to be the case that people only learned or fine-tune their own embeddings, but now many of the major AI providers have their own hosted sets of embeddings. For example, Google recently released embeddings for Gemini, with both API and local versions being available.

Two Tower Architectures

Building semantic search involves implementing a two tower architecture, a really common paradigm in both search and recommendations, and one that I’ve reached for again and again. The two tower architecture and semantic search are for me what flowers were for Rachel.

There are generally three steps to creating a two-tower model.

Embed the items once, in batch.
Embed your query in realtime at inference.
Compare the distance between the two.

So when I search for “chill dude” it generates an embedding for that phrase and then we do a similarity computation, usually using cosine similarity or dot product, or there are other distance measures you can look at, that says “this embedding is most similar to the embeddings and then it surfaces those results just as we do here.

But there are just so many decisions to make in this process that only become apparent once you start looking at the constraints of the data and the engineering space you’re working on.

This is a very simple structure but it’s very elegant because it allows for fast indexing and fast lookup on a large, coarse dataset. When we do any kind of information retrieval whether it’s for recommendations or for search, our goal is to get from tens of millions of items the entire internet a whole catalog worth of 800, 000 paintings like the Rijksmuseum has down to tens that you can look at in a carousel or in a screen or in a list of results.

Two towers is one way that allows us to pre-compute the embedding for our entire catalog and then it allows us to filter them quickly using approximate nearest neighbor search, which is a fast algorithm which says this thing is closer to this group of things but not this other group of things.

That allows us to do a first pass from tens of millions to tens of thousands or hundreds, and then we can use more expensive models to rank the smaller result sets and really think about how we would present those last, smaller result sets, to the user.

Many large companies then expand on this architecture to create search and recommendations data sources for different product surfaces.

Here’s a couple examples. For example, the one on the left is YouTube’s recommender system where they take millions of videos that they could recommend you and narrow them down to a few hundred and then rank the ones that are actually on display.

The input features for their custom trained model include “videos you’ve watched before” and then “videos you’ve searched for”, “geography, " and then metadata about you. And the query is the video in question. And then the model similarity lookup generates a list of candidates to that video based on all this input data.

Facebook Search also uses embeddings. It contains the search text plus the query context, like geography or language. The document side includes Facebook names or page titles, and graph-derived signals like relationships to other people.

So, there are many variants of these systems in production. How does this look in practice for Rijksearch in particular? We have our input query, which comes from the front end as happy flowers. The embeddings are from the Gemini API. And then we also already have all our document metadata that we’ve embedded before. The indexing app is in Go, writing into Redis vector sets, and the serving web app is written in pure Go. Redis is served as a docker container that connects to the web app with everything served in docker-compose.

It’s not a particularly original idea: there are now a lot of semantic image/vibe search engines out there today. But it’s the one that I personally wanted to do because I’d developed a connection to the Rijksmuseum, and because the amount of data they generously make public made it lovely to work with.

Getting data for Rijksearch

Something that you’ll notice is missing here if you’ve worked in data before is the actual source data. Where do we actually get this image data? How do we collect it? And that’s its own hard problem that we’ll talk about in a minute.

But okay, so how do we build this, given that we’ve already built Viberary and other versions of this system before? The way that I thought about it is that every company gets about three innovation tokens. This is from an amazing and often-cited blog post by Dan McKinley, which says that every company gets three of these tokens.

You can spend those however you want, but then the supply is fixed. That’s also true for any project, whether it’s at work or a side project. For a specific project, a way to think about innovation tokens is that you can try new things in several dimensions:

The domain area
The language stack
The app architecture
How/where you deploy it

If you do three new things that you can’t do or don’t know how to do, you’re just going to get held up with your product. So where should we spend our innovation tokens?

We should spend them on what helps us grow, because humans want to reach mastery. So we should spend them on what helps us reach mastery.

Taking a look at the same steps for Viberary and Rijksearch, you can see what I implemented differently. So there’s a couple steps. First, we have to collect the data. Then we have to think about what embedding model we want to use and how we want to embed it. Then we need to index our embeddings. This is an important step. And then we need to serve our embeddings and create a web server so we can actually stand up the front end and return results.

So the first time I created Viberary by using text-only data from Goodreads, a locally-hosted embeddings model artifact, indexing using FT.SEARCH from Redis’s search , and Python’s Flask to serve the app.

So I lied earlier about using 3 innovation tokens: I cheated a bit and used four here.

For Rijksearch data, I used multimodal text and image data because I was interested in creating text and image embeddings. This made processing the data trickier because I needed to associate a given set of text metadata to a link of an image, as well as process the images in an efficient way so they’d download quickly and non-disruptively.

For the actual embedding inference, I changed from a hosted model to an API, which was easier in some ways, but now required cost and latency considerations. Additionally, test and image processing together was trickier because you have to think about text and image aggregation together..

For indexing, I switched to vector sets, though since vector sets don’t require the construction of a search index and are a native structure. This makes it easier in that you don’t need to update fields, but vector sets also don’t support metadata updates in the same way that search indices do because vector sets are schema-free (though you can add JSON through VSETATTR ), so there is a tradeoff.

For serving, I changed from Flask to Go, which was a pretty big change because I had to relearn web development in a new language. Go has a lot of tooling for web development out of the box, as well as support for concurrency, and for processing web data. This was probably the innovation token that took me the longest time to overcome.

And, interestingly throughout this process, I also tracked the steps where AI helped. The yellow parts are where I included AI or I had AI assistance.

I consider myself a fairly normcore AI user. I’ll usually wait a couple months for something to come on the market to try it. If something has been on the market for 6 months and has lasted, it probably makes sense to keep. And so in 2023, what was on the market was MidJourney for generating some of my logos, ChatGPT for answering some questions, and then some of the open models. Mistral was making good open models at the time and I was using it locally via llama-cpp-python, though getting limited results beyond summarization and tagging.

When I did Rijksearch, I found several new use cases for AI. For example, I used Claude, as well as GPT-OSS 20B locally to help me figure out some of the paradigms for data collection from the Rijksmuseum, as well as write some quick data analysis scripts. I used Gemini APIs for embeddings and Perplexity to help me decide which embeddings to use. And I used Claude Code to generate some of the front-end code, and Qwen Coder and GPT-OSS and Perplexity to help me learn about Go.

Data Sense

For Rijksearch, data collection was also a lot of work, even more than scaffolding the app. But as ML and data professionals, what helps us here is the same thing that helped Rachel. She had help and mentorship from her family. As data practitioners, what we have from doing this for a while, and being surrounded by data-intensive environments, is data sense. We have been in the trenches of collecting data for a long time. We understand the pain points. We understand NaNs and what loopholes to look for in non-deterministic data flows, such as the ones generated by LLMs. We understand what it means to “look at the data.”

And this is something that Rob Pike, one of the creators of Go, said. If you’ve chosen the right data structures and organized things well, the algorithms will almost always become self-evident around the data. Data structures, not algorithms, are essential to programming. And this was really important for me because good similarity matching between images and text requires good data for both.

Luckily, the Rijksmuseum has a lot of data, and it’s all open and accessible to the public. It’s an amazing resource.

They have metadata for 800, 000 objects, high-resolution photographs. But something that was new to me in working with this data was working with OAI-PMH, the data format they offer their artifacts in. It’s available through an API, but the API is very different from what I’ve generally seen. OAI-PMH is an XML-based protocol over HTTP, which is different from most of the JSON-based REST APIs I’ve worked with.

LLMs helped me quite a bit here because I needed to understand, for example, ListSet records I needed. And, since this was open data it was easy to traverse the API results and get an understanding of what lives in that data set.

And it turns out there’s a lot of data to figure out.

First, I pulled all the paintings we needed from the record files. The data’s stored across multiple collections, so I chose to use the “Dutch Paintings of the Seventeenth Century in the Rijksmuseum” set because that collection had the most visually striking paintings. From there, I extracted each entry, making sure to keep the full metadata (title, artist, date, medium, etc.) instead of just a short snippet. This gave me a clean, complete list to work with for the rest of the project.

How do we actually collect that data into our app? Here’s where Go really helps. Go structs are typed collection fields, much like Pydantic models, and they really allow you to do this well. If you have an XML snippet, you can either run it through an LLM to get results, or use a site like XML to Go, no code generation needed. But, of course, LLMs do really well with Go because it is so strongly typed.

It really takes a while to wrangle all the data into the correct format, then, from collecting it as XML, to writing it to disk in Redis.

Then the fun starts, because you collect these records and you start to see that not every piece of data is labeled correctly. This is true of any given dataset, but particularly true here, because this data has been collected over lots and lots of time in many different languages, across historical collections, and who amongst us has not had the wrong language for a title or duplicate data in any of our data. This is where data sense and really looking at and being hands-on with the data helps.

Picking Embedding Models

The second part of data is picking an embedding model. For Viberary, I usedMiniLM just because it was small, fast, and it was standard. Now we have larger models that are hosted as APIs. Generally, depending on your use-case, it may make sense to use a domain-specific model, or fine-tune. But for a side project, a general model was just fine.

There are a lot of other factors we care about when selecting an embedding model, such as inference costs, storage costs, and the overhead of setting up a new account. For me, multimodality was also important (whether the model was trained on both images and text) because that becomes relevant for retrieval.

To help narrow down my options, I used Perplexity to generate a comparison grid and verify the information. One thing I like about Perplexity is that it cites its sources, so you can click through and confirm that the model didn’t hallucinate anything.

And so I came to three choices based on price, storage costs. What was interesting is that what really unblocked me was I already had a Google account with access to Google AI. And so I could just use and set up an account key easier than I could for Cohere and Mongo, although I’ve heard good things about both.

And then you have to pick which vector index you’d like to use. The index is the data structure that helps you quickly find related items.

Vector sets already implement HNSW by default. That makes the application really fast at retrieval time, which is what I care about. I didn’t really care as much as how long batch indexing takes, because at the end I only had something like 2, 000 paintings that I needed to index at a time. If I had a large selection, the choice between HNSW and IVF, which is great for data sets with large memory constraints, would have become less clear.

So then you embed everything, and then you start to get your results. And of course, because this is a data-intensive project, you get this thing where you query for “boat”, and you get the same thing for every single result, which doesn’t look like a boat at all. But there’s a pyramid, and you’re like, okay, why is this happening?

So then you have to go through and debug step by step, and it turns out the issue is not in the data structure, and not in the ingest, and not in the Gemini embeddings, and not in the indexing, and not in the web server, but in the fact that you picked an art category that only has 100 artifacts and your query is similar to all of them. And that’s all part of the process of looking at your data, too.

Where AI helped and where it didn’t

Rachel painted her cactus. This was her remixing and tastefully adding small, exotic elements to the larger whole. For me, this was AI and really evaluating where it might help or hurt my process to create good things and get better at what I do. So, where did I end up?

I used AI to build the front end, which turned out nice. It wasn’t anything out of the ordinary, and I am not sure how solid the actual code is, and whether it’s production-ready, but it was nice to get going since I’m not a front-end person at all and we always want to get to a good demo. Last time, I used a bootstrap template for this that I tweaked slightly, which I guess is not that much different.

LLMs also helped me with Go, but they didn’t help me by writing all of the code. How I’ve been learning Go is step by step: first, by reading dead tree books, then by trying out Go by Example, and finally, by asking about different Go concepts from the LLMs, or working with a very small, method-level snippet at a time.

Because what happened when I was having the model write Go directly is I found, due to cognitive offloading, that I couldn’t understand what “I’d” written, at all, and when I got up from the computer, I couldn’t replicate it. But it helped me answer questions like how structs work, what you should put in a struct, what a channel does. The kind of stuff that you would have to constantly bug a senior person about, because Go is a new language for me, the LLM will just answer. And particularly in some cases where LLM systems have RAG results, it will lead you to the source documentation because Golang has really, really excellent documentation, and that’s really what we want out of learning by reinforcement.

LLMs also helped with data formatting and exploration. So writing all of the matplotlib methods, the pandas methods, all that stuff where you need to constantly look stuff up or you need to get data from the API. A lot of that glue stuff, the models, any of them really, are good with.

So where did it not help? Largely with architecture changes and decisions. Understanding that I needed a two-tower model, what two-tower models were, how they’d been used previously in search and recommendations, understanding my latency constraints, knowing that serving this app has to be fast and that the indexing job’s speed didn’t matter as much: all of that came from my head and past experience rather than AI.

Picking vector sets as a storage and retrieval primitive was similar. I only learned they existed because I pay close attention to what’s happening in the vector space. (haha) You could probably build a bot to synthesize that kind of knowledge for you, but it was much easier to verify for myself.

More complex concepts like Go channels , which I haven’t gotten to, were also a weak spot. I couldn’t verify whether what it gave me in Go was actually correct, and had a lot of non-idiomatic code until I got an actual experienced Go developer to take a look and review “my” code. And of course, AI didn’t help in writing this talk. It took a very long time, and none of it was written with AI. It all came from ideas that had been slowly combining in my head. Which brings me to the final point: it takes time to build good things.

It takes time to build good things

It took Rachel 60 years to get good at painting.

You can see one of her first paintings on the left, and then one of her later, much, much later paintings on the right. Take a look at how much more complex the composition is, how the flowers are behind each other, how much more delicate the background is, the movement of the flowers in the space. It takes so much time to get good at that stuff. And we have to remind ourselves in the world of speed that that’s true for programming too.

Peter Norvig wrote this post a long time ago about teaching yourself programming in 10 years. There is no shortcut for deliberative practice, which is what we’re trying to do here when we build ourselves flowers.

For me, I found it takes time because I was going through my tweet archive to see what I could find for this talk. In 2013, I wrote about how I was learning Python and counting words and just really struggling through it. And it’s like, it’s not very optimal and I don’t really understand what’s going on and what RGV is, etc. And then 10 years later I was able to do Viberary just through deliberate practice. And three years later, I did it again, in Go, with multimodal embeddings.

And the other thing is, I didn’t work on my craft alone. I’ve written about how important it is to write with your alphabet radio on. This is a concept from Bird by Bird by Anne Lamott, which is a book on how to write, but also on how to be creative.

Anne Lamott talks about this alphabet radio that plays in your head. Of all the past people that have either criticized or praised you and how you have to tune some of them out selectively when you work. For me, it’s all the people that have reviewed my code and that I’ve worked with and the ones that I respect are always echoing in my head. That kind of feedback comes from good mentorship. That’s the apprenticeship that Rachel went through as well right so it takes time to get good at that.

It also takes time to teach other machines. Here’s another small example from another side project I did before called Gandinsky, where I tried to train a generative adversarial network to paint pictures in the style of Vassily Kandinsky. At the time, this didn’t work well. This was like eight years ago, and it didn’t work because you need thousands of samples to train a GAN and Kandinsky unfortunately in all his artistic prolificacy only painted 200 paintings.

Nowadays, you can just get Nano Banana to do style transfer for almost anything for free in an API call because we’ve been working on these models for such a long time. But it’s taken years and years for the models to get this good.

And finally, it takes time to own your own vision, as in this quote is from antirez and his recent thoughts on agentic programming.

Programming as an automatic vision is not yet. That only comes from doing the work over and over and over again. So building a semantic search engine really took me about a month once I buckled down and did it (though I’m still not entirely done). But it also took me ten years. And I think that’s important to keep in mind.

Okay, so now we’re at the end of the talk, and the question is, does this still matter? Yes, I think so. I think data sense matters, which we all inherit from having worked in machine learning and data science for years. I think mastery matters, which we get by working with others and building our own intuition for mechanical sympathy. I think craft still matters.

So let’s try to build ourselves flowers. Thank you.

Mechanical sympathy

Mon, 13 Apr 2026 00:00:00 +0000

Weaver, seen from the Front, Vincent van Gogh, 1884

Something that’s been floating around in my head lately is the idea that I don’t know any truly good engineers who are also not good at at product design.

Product design can roughly be designed as the contract between the creator and the user, where the contract is designed by a set of affordances, or actions that the product allows the user to take. This is all cribbed from Don Norman and The Beauty of Everyday Things.

For example, an affordance of a chair is that it allows you to sit. An affordance of most social media feeds is that they allow you to pull to refresh. A search bar, in theory, affords you the ability to look for results. Recently this has changed because we’ve been vibe spaced.

What makes good engineers good at product design is the same thing that makes them good at engineering. They feel for the boundaries of what the code and the product allows them to do and stop at those boundaries.

Another name for being able to understand and plan for affordances, either through good product intuition, or experience, or both, in the real world is mechanical sympathy.

Mechanical sympathy was initially coined in referring to Formula 1 racecar handling, and in software, has been adapted to mean “writing the kind of code that makes sense with the underlying stack so we can get good performance from our systems.”

Mechanical sympathy for both developers and end-users means understanding when asyncio is and is not helpful. It means using the right language, the right build system, the right font. It means using the least amount of tooling possible. Allowing for local development. It means reading code inside out rather than top to bottom. Using uv. Removing code where not necessary. Respecting boundaries.

I was watching this really great talk by Mario on how he built an agentic coding harness, and was reminded that agentic coding tools, just like many cloud services, don’t have mechanical sympathy.

Things that have happened to me over the past few weeks, off the top of my head, across various projects, languages, and providers:

A test failed, producing a 500 error. I asked the agent to fix the test. The first time, it tried to fix the code instead of the test. The second time, it rewrote the test to assert a 500 instead of a 200.
Even with a spec file, agents (and most chatbots, still) will use Python’s List and Dict, which have been deprecated for years.
Agents won’t use vectorized numerical operations unless prompted.
When writing a test, the agent fails numerous times, and instead of suggesting that the code should be refactored to be simpler and easier to test, just keeps trying, endlessly, to work agains the grain of the code flow and fix the failing thing.

In new Python projects, uv is skipped entirely as a first-class citizen across many providers as a suggestion of how to package projects.

Here’s more stuff:

Maybe agents.md / gastown / mcp / ralph / subagents / using a different provider that was nerfed last week but is cracked this week, and switching from the provider that was cracked last week but is cooked this week, solve this. It’s possible that by the time I publish the post, or three microseconds from now, a new model or harness that will come out that renders these cases obsolete.

But the larger point is that sympathy, and good product design, is made up of hundreds and thousands of moments like this, where missing one is not critical, but, when looked at in its entirely, result in an app that feels “hard” to use and develop.

Mechanical sympathy, just like real sympathy, comes from an enormous context window learned over a human lifetime. People, over time, learn to get good at mechanical sympathy. Coding agents can’t. It’s (maybe) possible they will someday, until then we need to provide the intuition.

On Programming Joy and Octocat

Mon, 06 Apr 2026 00:00:00 +0000

While GitHub has been busy losing its last nine of availablility, I’ve been thinking about how the internet used to be.

Not the internet people talk about from the 90s, but the internet that we used to have even 10-15 years ago. This was the heyday of startups like GitHub, Twitter, Airbnb, and, Google was in its prime (though likely slightly past it at that point - Linus’s git tech talk there was in 2007.)

I’ve specifically been thinking about the Octocat Builder. GitHub created it back in those years, hard to say when, but it dates back to at least 2018. The Octocat mascot itself was created in 2006.

The builder is so much fun - you can select different colors of cat, hairstyles, accessories, and send to friends.

There is no reason for this to exist. Perhaps there is some branding cache that GitHub got from this, but really, someone in the company just wanted to exist, and now it does.

Something like this would have a hard time being born in today’s tech industry, I think, with its cycle of promo packets, now layoffs, anxiety about AI, and now older, sclerotic tech companies thinking about how to stay technically and socially relevant. In a world with one nine left and the rest of the app dedicated to the growth of Copilot, there is no room for a small purple Octocat.

There’s some hope: there is a little whimsy in products like Claude Code’s spinner verbs and animations, but mostly it feels like whimsy ended with ZIRP, and that makes me a bit sad. I’m happy that Octocat is still here.

NASA Elements of Engineering Excellence

Sun, 05 Apr 2026 00:00:00 +0000

I stumbled across this report from NASA, “Elements of Engineering Excellence”, published in 2012,

The inspiration for this paper originated in discussions with the director of MSFC Engineering in 2006 who asked that we investigate the question: “How do you achieve excellence in aerospace engineering?” The authors’ approach to answering this question was a short course on Excellence in Engineering which is documented in this report.

The report talks about five areas of an organization that led to failures at NASA:

Shifting from engineering “hands-on” and “excellence” to “insight/oversight”. Lack of ownership.
“Normalization of the deviances”. Not questioning anomalies.
Lack of critical thinking. Over-reliance on procedures and computer codes.
Decentralization of authority.
Organizational and technical complexity.

And I found this first point particularly relevant in the age of agents:

The five root causes are not listed in any priority order. The first root cause listed deals with a shift in the NASA culture, where the organization moved from a hands-on engineering approach to an insight/oversight approach. In the early days the heritage was basically the arsenal approach where you designed, built and verified the system before contracting it out for production. The engineers really understood the design, the hardware/software and the system based on actual experience. In the early culture much of the technology development was an in-house, hands-on activity. The shift has resulted in the elimination of much independent analysis and test and experience based understanding of the systems required to catch and prevent problems. Howard E. McCurdy in his book Inside NASA says, “NASA officials from the original cultures believed they needed to provide their engineers and scientists with hands-on experience in order to maintain the technical side of the house. It was the only way to keep them technically sharp. By keeping their own engineers and scientists sharp, they could penetrate the work of the contractor. During the first decade of space flight, a strong technical culture guided the work of NASA employees. The norms typical of that period required NASA to maintain a corps of professional employees deeply involved in the details of space flight and aeronautics. The technical culture counterbalanced many organizational forces that rose up to challenge it. It overpowered the usual bureaucratic tendencies present in government operations. It provided a counterweight to the centralizing and organizational necessities of the Apollo mission.” [McCurdy, H. 1993] The loss of the technical excellence based on hands-on experience has led to many of the problems and therefore is one of the root causes of problems. To prevent problems, NASA needs to re-establish the culture of technical excellence based on hands-on work.

Antidote

Wed, 04 Mar 2026 00:00:00 +0000

If you love building things, and the process of building is just as important to you as the result itself, it’s not unreasonable that you’re in a slump these days.

The world is telling you that your thinking process is extraneous, unnecessary, and must be commoditized and compressed. But you are multidimensional, you need room to touch the code, to explore, to rise above the local minima. In engineering, the journey is the destination. The working system in production is our reward, and we always move towards that.

Unstick yourself. Do stuff that is not the most efficient. Build something from scratch with your bare hands and your brain. Read programming books written on dead trees. Reason through something on the back of an envelope. Scribble and daydream. Join Discords where people are actively building, reviewing pull requests, building community. Write a blog post. Mentor a colleague. Answer an email from a budding engineer. Create a private repo and merge to main without a PR. Take a library or tool and look underneath the hood. Go to an in-person meetup. Pick apart the living, breathing beast that is the transformer model that governs most of our daily conversations these days, down to the attention mechanism, down to the RoPE embeddings. Explain it back to yourself.

Having agency is one of the most important things a creator can have. Give the gift of it to yourself. It’s still there, waiting for you.

Querying 3 billion vectors

Sat, 21 Feb 2026 00:00:00 +0000

Recently, I got nerd-sniped by this exchange between Jeff Dean and someone trying to query 3 billion vectors. I was curious to see if I could implement the optimal map-reduce solution he alludes to in his reply.

A vector is a list/array of floating point numbers of n dimensions, where n is the length of the list. The reason you might perform vector search is to find words or items that are semantically similar to each other, a common pattern in search, recommendations, and generative retrieval applications like Cursor which heavily leverage embeddings.

I started by writing an extremely naive implementation which made the following assumptions:

we have 3 billion searchable (document) vectors and ~1k query vectors (a number I made up)
Both of the vector sets are stored on disk in .npy format (simple format for storing numpy arrays
We’d like to compare each of the query vectors against the larger pool of document vectors and return the resulting similarity (dot product) for each of the vector combinations.
3k total reference vectors (to see if we could intially run this amount before scaling)
The vectors are of dimensionality (n) 768, a common dimensionality for many models that allow for similarity-based embedding queries

import numpy as np
from loguru import logger
import time
import os

# start with 3_000 vectors to keep things small 
total_vectors_num = 3_000_000_000
query_vectors_num = 1_000

def generate_random_vectors(num_vectors:int)-> np.array:
 logger.info(f"Generating {num_vectors} vectors...")
 rng = np.random.default_rng() 
 vectors = rng.random((num_vectors, 768)) 
 return vectors

def get_dot_products(vectors_file:np.array, query_vectors:np.array) -> list[np.array]:
 total_products_computed = 0

 dot_products = []
 for v in vectors_file:
 for qv in query_vectors:
 dot_product = v @ qv
 dot_products.append(dot_product)
 total_products_computed += 1
 if total_products_computed % 100000 == 0:
 logger.info(f"Total vectors processed:{total_products_computed}")

 return dot_products


# Generate initial vectors and query vectors and write to disk
doc_vectors = generate_random_vectors(total_vectors_num)
query_vectors = generate_random_vectors(query_vectors_num)
np.save('vectors.npy', doc_vectors)

# Load vectors from disk 
logger.info("Loading file from disk...")
vectors_file = np.load('vectors.npy')

start_time = time.time()
logger.info("Getting dot products...")
results = get_dot_products(vectors_file, query_vectors)
end_time = time.time()

logger.info(f"Execution time: {end_time - start_time:.4f} seconds")
logger.info(f"Number of dot products computed: {len(results)}")

This, predictably, didn’t do so great, even on my M2 Macbook, even at 3,000 vectors, one million times less than 3 billion embeddings, taking 2 seconds.

Results:

2025-12-13 17:53:25.675 | INFO | __main__:generate_random_vectors:9 - Generating 3000 vectors...
2025-12-13 17:53:25.691 | INFO | __main__:generate_random_vectors:9 - Generating 1000 vectors...
2025-12-13 17:53:25.698 | INFO | __main__:<module>:39 - Loading file from disk...
2025-12-13 17:53:25.700 | INFO | __main__:<module>:43 - Getting dot products...
2025-12-13 17:53:27.688 | INFO | __main__:get_dot_products:24 - Total vectors processed:3000000
2025-12-13 17:53:27.688 | INFO | __main__:<module>:47 - Execution time: 1.9877 seconds
2025-12-13 17:53:27.688 | INFO | __main__:<module>:48 - Number of dot products computed: 3000000

So I vectorized the numpy operation, which made things much faster.

import numpy as np
from loguru import logger
import time

total_vectors_num = 3_000
query_vectors_num = 1_000

def generate_random_vectors(num_vectors:int)-> np.array:
 logger.info(f"Generating {num_vectors} vectors...")
 rng = np.random.default_rng() 
 vectors = rng.random((num_vectors, 768))
 return vectors

def get_dot_products_vectorized(vectors_file:np.array, query_vectors:np.array):
 dot_products = vectors_file @ query_vectors.T
 return dot_products.flatten() # collapse into single dim

# Generate initial vectors and query vectors and write to disk
ram_vectors = generate_random_vectors(total_vectors_num)
query_vectors = generate_random_vectors(query_vectors_num)
np.save('vectors.npy', ram_vectors)

# Load vectors from disk 
logger.info("Loading file from disk...")
vectors_file = np.load('vectors.npy')

start_time = time.time()
logger.info("Getting dot products...")
results = get_dot_products_vectorized(vectors_file, query_vectors)
end_time = time.time()

logger.info(f"Execution time: {end_time - start_time:.4f} seconds")
logger.info(f"Number of dot products computed: {len(results)}")

At .017 seconds, this was a big improvement!

Results:

2025-12-13 17:52:52.810 | INFO | __main__:generate_random_vectors:9 - Generating 3000 vectors...
2025-12-13 17:52:52.831 | INFO | __main__:generate_random_vectors:9 - Generating 1000 vectors...
2025-12-13 17:52:52.874 | INFO | __main__:<module>:39 - Loading file from disk...
2025-12-13 17:52:52.876 | INFO | __main__:<module>:43 - Getting dot products...
2025-12-13 17:52:52.887 | INFO | __main__:<module>:47 - Execution time: 0.0107 seconds
2025-12-13 17:52:52.887 | INFO | __main__:<module>:48 - Number of dot products computed: 3000000

I tried a 3 million sample size with this improvement. This took 12 seconds.

2025-12-13 19:39:43.830 | INFO | __main__:generate_random_vectors:12 - Generating 3000000 vectors...
2025-12-13 19:39:57.509 | INFO | __main__:generate_random_vectors:12 - Generating 1000 vectors...
2025-12-13 19:39:58.978 | INFO | __main__:<module>:57 - Loading file from disk...
2025-12-13 19:40:00.131 | INFO | __main__:<module>:61 - Getting dot products...
2025-12-13 19:40:12.984 | INFO | __main__:<module>:65 - Execution time: 12.8491 seconds
2025-12-13 19:40:12.992 | INFO | __main__:<module>:66 - Number of dot products computed: 3000000000

We could also reduce even further by converting the data to float32:

doc_vectors = generate_random_vectors(total_vectors_num).astype(np.float32)
query_vectors = generate_random_vectors(query_vectors_num).astype(np.float32)

2025-12-13 18:13:52.152 | INFO | __main__:generate_random_vectors:10 - Generating 3000 vectors...
2025-12-13 18:13:52.168 | INFO | __main__:generate_random_vectors:10 - Generating 1000 vectors...
2025-12-13 18:13:52.176 | INFO | __main__:<module>:55 - Loading file from disk...
2025-12-13 18:13:52.178 | INFO | __main__:<module>:59 - Getting dot products...
2025-12-13 18:13:52.182 | INFO | __main__:<module>:63 - Execution time: 0.0045 seconds
2025-12-13 18:13:52.182 | INFO | __main__:<module>:64 - Number of dot products computed: 3000000

With these small improvements, we’ve already sped up inference to ~13 seconds for 3 million vectors, which means for 3 billion, it would take 1000x longer, or ~3216 minutes.

|approach | query_vectors | doc_vectors | time |
|----------- |---------------|---------------|----------|
| Naive | 1,000 | 3,000 | 1.9877s |
| Vectorized | 1,000 | 3,000 | 0.0107s |
| Vectorized | 1,000 | 3,000,000 | 12.8491s |
| Np.Float32 | 1,000 | 3,0000 | 0.0045s |

When we start to run it to test, however, we run into a different problem: OOM. Why? The amount of memory needed to process 3 billion objects, each as float32 object that’s 4 bytes in size, would be 8 million GB.


vectors = rng.random((1, 768)).astype(np.float32)
print(vectors.nbytes)
3072
print(vectors.itemsize)
4

bytes_per_float32 = 4
memory_gb = (3000000000 * 1000 * 768 * bytes_per_float32) / (1024**3)
8583068.84765625 = 8.6 TB

In order to improve this, we would need to do some heavy lifting of the kind Jeff Dean prescribed. First, we could to change the code to use generators and batch the comparison operations. We could write every n operations to disk, either directly or through memory mapping. Or, we could use system-level optimized code calls - we could rewrite the code in Rust or C, or use a library like SimSIMD explicitly made for similarity comparisons between vectors at scale.

Before I started on any further optimizations, upon further inspection, there were some things about the problem that I realized weren’t clear to me: 3 billion vector embeddings queried a few thousand times could mean:

I have a single query vector, and I query all 3 billion vectors once, get the dot product, and get all results
I have a single query vector, I query all 3 billion vectors once, get the dot product, and return top-k results, which is easier because we can do ANN search
- In this case, do I need to return the two initial vectors also? Or just the result?
- Do I need to re-rank the results by similarity in any way?
I have 1,000 query vectors, and I query all 3 billion vectors once, and get the dot product of all results
Are these vectors already in-memory when we intially start working with them or will they always be on-disk? Are we reading them one at a time, or streaming them?
What kind of machine are we assuming: Are we running this locally? What are the specs of the machine? Are we assuming the vectors come to us in a specific, optimized format?
- Do we have GPUs and are we allowed to use them?
How big are our embeddings? - this is extremely important and could significantly impact our representation, input vector size and output results
Are we assuming we can compress their representation at all, i.e. is compressiong from float64 to float32 tolerable wrt to accuracy?
How much time do we have to generate this one-off project? Are we sure it’s really a one-off?

All of these dictate the additional time and resources spent on the solution. What I realized is the same thing I’ve seen so many of these problems over the years, that the technical solution is no longer the hardest one to achieve: the hardest one is nailing down the requirements.

2025 in review

Mon, 22 Dec 2025 00:00:00 +0000

Jeune fille lisant une lettre à la bougie, Jean-Baptiste Santerre, 1700

Machine learning engineers spend their lives alternating between two states: staring at tqdm progress bars during model training and staring at error logs during model inference.

A third category now involves staring at coding agent CLI progress bars, but using too much AI assistance during coding makes me feel like I’m losing my own context window.

I started a new job as a founding MLE in March and, as is true for engineers in any small and young team, I’ve been staring at error logs all across the stack.

The good news is that if you are at a point where you care a lot about error logs, it means you have users who care a lot about the answer to “What if this breaks?” If you have people other than you who care about your software, congratulations, you are in production. Being in the state of production is the best possible outcome for software engineers because it means the work we’re doing is useful.

Standing up a new thing in production, though, is scary, because production means you are responsible - to the end-user, to the other developers on your team, and, finally, to yourself.

In “The Tombs of Atuan”, Ursula K. Le Guin writes of a girl, Tenar, who is taken at a very young age from her family and goes to live in a holy city. In a ritual ceremony, her name is changed and she loses her identity to become Arha, the Eaten One. Arha is the High Priestess of the Place of the Tombs, a vast series of caverns under the earth, silent, and in complete darkness, “full of gold and the swords of old heroes, and old crowns, and bones, and years, and silence.”

In her first trip to the tombs with an elder priestess, they descend into the dark without a torch.

“Light is forbidden here.” Kossil’s whisper was sharp. Even as she said it, Arha knew it must be so. This was the very home of darkness, the inmost center of the night. Three times her fingers swept across a gap in the complex, rocky blackness. The fourth time she felt for the height and width of the opening, and entered it. Kossil came behind. In this tunnel, which went upward again at a slight slant, they passed an opening on the left, and then at a branching way took the right: all by feel, by groping, in the blindness of the underearth and the silence inside the ground. In such a passageway as this, one must reach out almost constantly to touch both sides of the tunnel, lest one of the openings that must be counted be missed, or the forking of the way go unnoticed.

This is what building a new production system is like. You go into it, like Arha, scared, groping entirely in the darkness.

If you need a cache, you can use a Go map, lru cache, or Redis. If you need a vector store, you can use np.array, or Postgres, or Pinecone/Chroma/Turbopuffer/ or Elasticsearch. You could run your whole stack on Docker Compose, or a single bare-metal server in your basement, or Kubernetes in the cloud, or on a vape.

Forget large-scale architecture choices. How will you name things? Should you name your method get_results or get_query_results? Will you ever get results that are more than just the answer to a query? Will you ever use this method in a more abstract way? Does it need to be part of a class? What exceptions should you write for it? We want to write code that must never lie, but what if, at the time, we are telling the truth to ourselves?

How soon do we need to ship get_query_results? How many other methods will rely on it? How many of our teammates need this method for their work as well? Will they understand it? Code is meant, largely still and hopefully for the forseeable future, for humans to read first and machines to compile second.

In “The Bell Jar”, Sylvia Plath laments about the endless myriad combinations of a possible life, “I saw my life branching out before me like the green fig tree in the story. From the tip of every branch, like a fat purple fig, a wonderful future beckoned and winked.” By the time she’s done thinking through all these futures,she hasn’t made a decision, and they all slip away from her.

The forking branches of a decision tree in a codebase, are likewise boundless, and the neat part is that there is no right answer. You are constrained by your business requirements, but the choice of implementation of those requirements is of an endless variety. It will depend on: the stack you already have, the budget for the rest of the stack, your own past experience and engineering values, the social norms and expectations of your team and their collective experience, the industry’s vocabulary, the language you’re writing in, its conventions and affordances, and how much time you have to actually think about, finish, and merge this current PR before you need to deploy. That’s why staff engineers make money mostly by saying “it depends” for years on end in different ways.

Let’s say your goal is to build a scalable, distributed, microservice architecture that processes inputs and streams machine learning inference outputs to users with minimal downtime, and that the system is resilient to failures via container orchestration. I’ve just described every machine learning systems since the beginning of time, this is the machine learning system they show you in Plato’s ML engineering cave, in every single Medium post on the planet, this is the set of boxes connected by dotted lines that arrives to unbidden to every engineer in their dreams. This is the light at the end of your labyrinth.

But, in building such a system, you never start by building such a system. You have to start with a single keystroke in the inky blackness.This is one of the reasons it’s so hard to write code with LLMs and evaluate their output, by the way: humans reason from the inside out - “When you write this code, you don’t go from top to bottom, you go from chunks and you grow the chunks…” - and LLMs write code from top down.

So, instead, you take a deep breath and write a single method to tokenize a string.

def tokenize(string:str) -> list[str]:
 tokens = string.split()
 return tokens

Tokenizing a string is the process of splitting it into subsections and converting each of them into a number so they can be processed by a transformer-style model. Tokenization is not specific to LLMs. We’ve been doing tokenization for a long time. Here’s a paper on how important it is for preprocessing from 1992!

The first part of tokenizing any text is splitting it into those sub-parts. Those sub-parts can be words, syllables, or characters. We start with words to make it easier.

sentence = "The truest power is the power to choose. - Ursula K. Le Guin"

def tokenize(string:str) -> list[str]:
 tokens = string.split()
 return tokens

print(tokenize(sentence))

['The', 'truest', 'power', 'is', 'the', 'power', 'to', 'choose.', '-', 'Ursula', 'K.', 'Le Guin']

Easy, and we have something concrete. Do we count “K.” as a word? How about the punctuation, do we generally want to include that for tokenization in a transformer-style model?

What do we do with input text that has numbers in it? Do we include them or strip them out?

sentence = "The truest 1 power is the power to choose. - Ursula K. Le Guin"

print(tokenize(sentence))

['The', 'truest', '1', 'power', 'is', 'the', 'power', 'to', 'choose', 'Ursula', 'K.', 'Le Guin']

What happens when we instead get malformed inputs from a webpage, or if we have delimiters?

sentence = "The&lt;html>truest\npower\nis\nthe\npower\nto\nchoose\nUrsula\nK\nLe\nGuin"
print(tokenize(sentence))

['The&lt;html>truest', 'power', 'is', 'the', 'power', 'to', 'choose', 'Ursula', 'K.', 'Le', 'Guin']

What if the input is an empty string? Do we want to send that empty string downstream?

What if we don’t have a string to tokenize? We now get an exception that we have to think about how to handle if we’re in production.

string = None
tokenize(sentence)

Traceback (most recent call last):
 File "/Users/vicki/arha/tokenize.py", line 9, in &lt; module>
 print(tokenize(sentence))
 File "/Users/vicki/arha/tokenize.py", line 5, in tokenize
 tokens = string.split()
AttributeError: 'NoneType' object has no attribute 'split'

What if we receive emojis? This string representation returns only one token, which may not be correct in representing these two different emotions. And how do we know the model accepts emoji codepoints as input anyway? Many GPT-style models do, but many BERT models don’t.

sentence = "🤪🥰"
print(tokenize(sentence))

['🤪🥰']

What if we send an extremely large string because we don’t have upstream error handling for our tokenization queries? This code block will take a long time to return because you are creating a large object in memory. Since we don’t have logging yet, we won’t know in the rest of our application where that large object creation takes place. We will just have a web service that hangs silently, indefinitely.

num_tokens = 500_000_000
pattern = "The truest power is the power to choose. - Ursula K. Le Guin"
large_string = pattern * num_tokens

print(tokenize(large_string))
hanging forever...

What if our input includes languages other than English? How do we account for Unicode?

Sometimes, you’re able to come up with most of these edge cases at once. Mostly, though, you end up building out these edge cases over countless iterations, because software engineering is programming integrated over time.

Eventually, you build up to a codebase that has a level of thought and care around tokenization, punctuation, and input sanitization. You’ve now written a codebase at the level of tokenizers, or spacy or tiktoken.

In fact, you should probably use the libraries that people have already put collectively dozens of combined years of thought into - those corridors are closed, safe, and tested, in the case of open-source libraries, by a much larger community than just you. You import tokenizers.

In doing so, you close off a number of paths further into the labyrinth behind you, the doors darkened and abandoned. You light a single flickering torch to mark your path. On top of that code, you start adding the things that make that path more resilient (but also, then harder to refactor if you decide to pivot): precise exception handling, traces, clear human-legible logging, and unit tests. A single corridor in your application starts to shine. With the path in front fo you illuminated, you move forward.

Now that tokenization works, since you need this method throughout your application, you’ll want to create a tokenizer inference service.

Within the logic itself, if you’re using an external tokenizer, you’ll need to make sure it matches the model you’re using, because the model depends on a given tokenization scheme, In this case, we are better off relying on AutoTokenizer, which handles the import and comparison logic for us:

from transformers import AutoTokenizer

name_or_path = "arha-based-uncased"
tokenizer = AutoTokenizer.from_pretrained(name_or_path)

If you are downloading the tokenizer from HuggingFace, as AutoTokenizer does unless you specify a custom tokenizer path, you add a network call and an external dependency to the complications of your initial service call.

A new branch blooms on your decision tree. If you make this call in your service, how much latency does it add? On cold-start, can you assume the artifact will always be there? How long does it take to load the tokenizer? To stave off that branch of questions, you could instead download the artifact locally, or write your own tokenizer. Now, you’re in the business of object storage and artifact management.

You decide to tackle another question: if you’re calling the tokenizer again and again, you’ll need to create a method, or better yet, a class, because it’s highly likely you’ll want other functionality attached to the tokenizer.

And now, if you have a service, you need to Dockerize the code. If you have a Docker image, you need to pin dependencies for reproducibility. If you are pinning dependencies and have Docker, you need a build process for your deployable artifact.

The closer and closer to production you get, the more questions around “what happens if this breaks for other people who include my team and my users” start to form.

You write the service and add logs. Lots of logs. So many logs. Logs are your first line of defense in production. They are the candle in the darkness. If they’re good enough for Brian Kernighan, who wrote, “The most effective debugging tool is still careful thought, coupled with judiciously placed print statements,” they’re good enough for you.

You need to register the service into your observability platform. You add exception handling. So many Exceptions. What if the tokenizer doesn’t load? What if it loads, but to the wrong directory? What if HuggingFace is down? What if the tokenizer service can’t talk to other services? What happens if the new container doesn’t launch? What happens if the tokenizer artifact is corrupted? What happens if you’re serving the tokenizer, but the intra-service network fails? Exceptions and logging, logging and exceptions, unit tests and integration tests, layer upon layer of defenses.

Beej, who previously wrote an incredible guide to git, recently released a guide to learning computer science, in which he says that in order to be a good computer scientist, you need to think like a villain.

Expect the unexpected in terms of data that your code will receive. Expect malicious actors to feed in data in an attempt to gain unauthorized access or manipulate the system in undesirable ways. Test for that stuff in your code.

You think like a villain, because the villain in the production environment is usually yourself. With each new door, comes the possibility of falling through into a bottomless pit.

For me, this usually happens either immediately after I deploy, or at 2:17 in the morning a week from now. The tokenization method I wrote or imported fails. Or there’s a memory leak. Or there’s a network outage that causes a cascading failure.

I add more logs. I debug. I run tests. I develop locally. I add more tests. I consult and I fix, and, finally, I am back on the golden path. The alerts subside. I add more torches. The system becomes stronger, more legible from me having struggled with it. My reasoning about the system becomes stronger and clearer, too. We grow together, the system in production and me.

There is one comforting thought about being with the system: when you enter the Tombs, you are not actually truly alone. Software development is a team sport, and when we build a system as a team, we built it together, PR by PR.

And, we are also building with others: we build on the libraries and tools others before us have hardened, with the tools that others have built. In spite of the fact that an increasing amount of code is machine-assisted, software development is still mostly humans talking to each other in language that computers also understand. As we build, alone in the dark with our torches in the Tombs, we hear the echoes of others in other hallways, each carefully closing off dead ends, placing their own torches.

Once you illuminate enough of the system by reasoning through failure points, architectural decisions, the hallway is no longer unlimited darkness. It is an illuminated system in production. You keep walking around the app and harden the system. You start to get more sleep.

At the end of the year, I made it out of the Tombs.

Just in time to go back to staring at tqdm.

I want to see the claw

Mon, 20 Oct 2025 00:00:00 +0000

I respect quality software and the people who write it. And, I’ve invested years of my life in working on becoming one of these people (even if the journey has been long and hard and has involved lots of YAML). I have seen and used code written by people who care about software correctness, who get pleasure out of defining correct interfaces, who have walked the app, who have spent years working towards mastery. And, I have seen and written the code the rest of us write.

Selfishly, I want more people to aspire to mastery, because wanting to be good at writing code means you care about the code, but more importantly, about the people on the other end of the code. And, I want a world where developers care about other developers and users. I want a world of teams of seniors shipping good code and mentoring juniors and working with product managers and open-source committers. I don’t want a world where three agents in a trenchcoat from whichever AI lab is better this week quietly churn out CRUD slop that I approve out of the corner of my brain every thirty minutes.

Perhaps the first time I understood the value of mastery and care was my first job out of college, where I, a baby analyst, handed in a report to my senior analyst to check. “Did you double-check that SQL to make sure it’s correct, ” she asked. I had not, and when she ran my data, she got different results. I learned to always check my work.

A bit later, I worked with a senior architect trying to extract data into a relational database from a schemaless store. I watched him setting up for days, first preparing a staging environment and backups, then running and re-running again and again, carefully timestamping the files so he could roll back to an earlier run. Often, when I watched, his chair was empty - he was in meetings with the team who owned the database, making sure that when he hit their service to extract the data it didn’t fall over and impact them. Finally, when he was done, he wasn’t done. He checked it against the downstream reporting we were doing to make sure the numbers matched.

There is no single element that comprises this quality of care, the striving towards excellence and mastery. But, as Bernoulli said when he received an anonymous solution to the brachistochrone problem that turned out to be Isaac Newton, “I recognize the lion by its claw.”

There is a great recent post that posits that most industry programmers are impure engineers: we will always be up against business constraints and reality, and do not have the time and luxury that pure engineers, who care about only the heart of the technical solution, to care about quality.

But over and over in my career, as I tuned my alphabet radio, I came to consistently understand that intent matters. Even if we are, almost all of us, impure engineers and mere mortals and the sprint is ending in two days but I haven’t finished my t-shirt size medium task yet, the closer impure engineers strove to work at the level of purity (and did their second job of being easy to work with), the more I wanted to both work with them and use their software.

I recognize the claw of the lion in software like Redis, cURL, uv, Ghostty, sqlite, llama.cpp - software that is elegant, well-built, considered and thoughtful. Software that is joyful to use. Software that helps me. I want to write software like this, and I want to use software like this, and I want us as programming people to be incentivized to value the process that creates software like this.

It has, with generative code, become harder and harder to strive towards the lions because the models produce code that is, quite literally, mid, the compressed and weighted average of every excellent Stack Overflow answer, but also questions like, “What happens if the explanatory and response variables are sorted independently before regression?” It is the average of all publicly available software, updated at some cadence and mixed into training data soup and then RLHFed according to some arbitrary metrics, and as such can only offer a ghost of quality.

We are being overrun by mediocrity and sloppiness, we are trying to fight against it, and yet, no matter how good the models and ecosystems around them get, I find myself more and more wanting software that I know is written with humans at the wheel - we are still better at reasoning, at aesthetic judgment, at architecture.

The best code is no code, programming still sucks and always will, and yet, I find myself still searching for the claw, the mark of mastery. Because that mark comes from people who want to reach other people directly. I want to see the claw, because if there is a claw, it means there is a living, breathing lion on the other side of the screen building the software that elevates us and binds us together as a community of software engineers.

Walking around the app

Tue, 09 Sep 2025 00:00:00 +0000

There is a very vigorous debate happening online right now around what shape evaluation for LLM-based products should take. I don’t want to rehash all of it, other than saying that if you are building any applications with with non-deterministic components (most AI-powered apps), or applications that are data-intensive (also most AI powered apps!), and you are serious about these features in your application, you should at a very minimum be testing your online outputs in production on a daily basis. If offline testing (testing your given model outputs internally before it goes live in production) is a luxury you can also afford based on your engineering capacity and tooling maturity, even better!

More importantly: if you are building an app, you need to also constantly be testing and touching different parts of it to see if the app flow makes sense. Look at the UI dropdowns, try the search bar. Toggle the toggles. How long does it take to load a result? What do the results look like? What colors are the buttons? How do they work on different devices? At different internet speeds? What about your payment gateway? How does onboarding look for a brand new user in North America? In New Zealand? Could you onboard from a fresh device in under three minutes? Do images load from your CDN?

I’ve heard this kind of testing called different things in different product cultures, but the phrase that stuck with me the most was when Dan called it “walking around the app”. Walking around the app is exactly what it sounds like - checking out the sidewalks, picking up idle pieces of trash, letting the store across the street know their street lamp is out. You could call this QA, although it’s more than that. QA is scoped transactionally and takes place at the end of a PR. Walking around the app is a mindset, a process something you do every day, broadly, habitually, passionately, with interest and curiosity, because the app is the neighborhood you’re building for the people you’re building for.

Because what happens if you don’t walk around the app regularly is that you start to get broken windows. The broken window theory posits that any given signs of decay in a town make it seem permissive to create more: littering begets more littering. People understand the impact of this: For example, in real life, in the Italian town of Brescia, there is an anonymous guy who goes around at night painting over malignant graffiti.

In the applications we build, builders are both the graffiti creators and the graffiti erasers, and we have the responsibility to make our applications as habitable as possible for both our users and ourselves. What does it mean to clean up broken windows for users? Buttons that go nowhere, results pages that don’t render, links that are missing results, misspellings, cache misses, tiemouts, misaligned CSS. Deprecated routes. Latency.

How about for ourselves? Apps are easier to walk around if they are consumer-facing apps that have front-ends, but APIs and backends can be walked around, too. Naming inconsistencies. Broken local builds. A nested class inheritance hierarchy that’s multiple levels deep is a broken window because it’s hard to trace in our mind - humans can only really keep 2+5 things in memory. A hard-coded environment variable that creates extra mental load. A method that relies on an untestable internal method that results in having to use a mock. A build process so long it causes you to lose interest and tab away to Reddit. Anything that creates cognitive overhead, that Grug doesn’t like, can be a broken window.

There is not one thing that, once fixed, makes the app work well, but it becomes intuitively obvious to end-users what a well-loved app feels like, because it’s constantly being walked around and being fixed in a million small ways, every day.

We are mere mortals and will always create bugs, especially when it comes to data work. Walking the app will not prevent this - to write code is to generate bugs and the best code is the code never written.

But, if we are walking around our app every day, we’ll catch at least as many things as we can create. And if we have that in place, evaluation of any kind becomes infinitely easier to establish as a habit, too.

How big are our embeddings now and why?

Mon, 01 Sep 2025 00:00:00 +0000

A few years ago, I wrote a paper on embeddings. At the time, I wrote that 200-300 dimension embeddings were fairly common in industry, and that adding more dimensions during training would create diminishing returns for the effectiveness of your downstream tasks (classification, recommendation, semantic search, topic modeling, etc.)

I wrote the paper to be resilient to changes in the industry since it focuses on fundamentals and historical context rather than libraries or bleeding edge architectures, but this assumption about embedding size is now out of date and worth revisiting in the context of growing embedding dimensionality and embedding access patterns.

As a quick review, embeddings are compressed numerical representations of a variety of features (text, images, audio) that we can use for machine learning tasks like search, recommendations, RAG, and classification. The size of the embedding is how many features our item has.

For example, let’s say we have two butterflies. We can compare them among many dimensions, including wingspan, wing color, number of antennae. Let’s say we have a 3-dimensional feature for a butterfly, it might look like this.

Doing a visual vibe scan, we can eyeball the data and see that butterfly_1 and butterfly_3 are more similar to each other than to butterfly_2 because their features are closer together.

But butterflies are 3-dimensional animals and some of these features are numerical. When we talk about embeddings in industry these days, we generally mean trying to understand the properties of text, so how does this concept work with words? We can’t directly compare words like “bird” and “butterfly” or “fly” in a given text but we can compare their numerical representations if we map them into the same shared space. We can see that “bird” and “flying” are more similar to each other than to “dog” through their numerical representations.

Intuitively, we know this is true, but how do we artificially create these relationships?

There are different ways of creating text embeddings from a given word, all of which rely on analyzing that word in relation to the words around it in a given corpus of data. We can use traditional count-based approaches like TF-IDF based on term frequency in documents, or statistical approaches like PCA or LSA.

With the advent of deep learning models, we started learning representations generated from models like Word2Vec that maximized the probability that a left-out word would be next to other given words in the training dataset.

When we learn the embedding representation of a given word using probabilistic models, we are comparing how similar these words are to other words. Each feature is not an explicit feature like “wing color”, but rather a vibe-based latent representation in the latent space that doesn’t have a clear explanation.

For example, one dimension might be “this word is an action word” or maybe “this word is related to other words about food”, but we generally don’t know exactly what the model thinks each feature represents. In fact, this is a fascinating area of study we are just starting to understand how these latent representations work through ideas like control vectors, a concept that Anthropic explored in the famous Golden Gate Claude paper.

When we train a model, embedding size is initialized as a hyperparameter before model training, and we iterate on the size depending on our downstream evaluations after training. Picking the right hyperparameter is (alchemy) a combination of art and science and depends on optimizing training throughput, final embedding storage size, and the performance of the embedding on your downstream task both qualitatively and wrt to latency of performing search on embeddings of different sizes.

Since previous generations of models were smaller and trained in-house, hyperparameters were usually not published by companies, and as such, there was no standard agreement on embedding size. We generally, as an industry, understood that somewhere around 300 dimensions for a given embedding model might be enough to compress all the nuance of a given textual dataset. 300 was the number of dimensions typically used by earlier models like Word2Vec and GloVE.

After the publication of the attention paper BERT was released in 2018. This model’s architecture introduced embeddings of 768 dimensions. Although previous RNN and LSTM models had been trained on GPUs, BERT was one of the first larger embedding models to be trained on GPUs (and TPUs), which meant that GPU optimization now became increasingly important.

The key behind training Transformer models efficiently is the ability to efficiently move data onto GPU and parallelize their matrix multiplication operations between several pipelines, aka attention heads, where each attention head can focus on understanding and defining a different part of the embedding feature space. As such, the embedding needs to be able to be partitioned evenly between the number of attention heads.

BERT has 12 attention heads, so 768 dimensions was selected from a combination of trial and error and efficiently parallelizing computation to “attend” to different parts of the feature space, meaning that, in model training, each head operates on a 64-dimensional subspace of the original 768-dimensional input embedding. This itself comes from the Transformer paper, where each sub-embedding size per head is commonly chosen as 64.

As a result, many BERT-style models, and related model families, used 768 as a baseline for embedding dimensions. Although it had a much larger training dataset, GPT-2 also implemented 768. And, although CLIP uses an embedding size of 768 as derived from the Vision Transformer architecture that CLIP uses for its image encoder, and for consistency, the text encoder also uses this dimension size[^1].

Even though training cycles for BERT were fairly small (4 days for the original BERT model) compared to the months-long pretraining processes that LLMs require these days, it was still hard computationally to infer these embedding sizes, even with GPU optimizations. For BERT, for example,

Finding the most similar pair in a collection of 10,000 sentences requires about 50 million inference computations (~65 hours).

So in 2019, UKP created and optimized a model, SBERT, focusing on sentence-level representations, which unlocked faster inference, and Minilm became the standard baseline model for document-level chunk embeddings at 384 dimensions and still performs extremely well for a number of tasks.

What else changed was the rise of the open-source HuggingFace platform where model artifacts could be shared. Previously, they were either hosted in undiscoverable repos in GitHub or locked away in scientific repositories missing metadata. The rise of HuggingFace as a platform and the transformers library as a centralized API into training and inference for PyTorch-based models unlocked a level of standardization: now, many more people could simply download and replicate the existing models instead of rebuilding arcane research code from scratch. This led to much more standard embedding and architecture sizes used both in academia and industry.

Although 768 held for a fairly long time, with upwards pressure from market competition on model sizes as the result of the release of GPT-2 (the backbone of ChatGPT), we started seeing standard embedding sizes increase.

Part of this was the fact that companies no longer had to infer their own embeddings. Previously, embeddings had been custom-learned in labs or in companies whose core competency was processing large amounts of information that could be culled in retrieval through search or recommendation.

But now, with the advent of ChatGPT and API-based model availability, embeddings became a commodity available with a GET request, and the most popular embeddings became OpenAI’s, which used 1536 dimensions, in line with GPT-3, which used much more training data than any previous model. (570GB versus GPT-2 which was 40GB, and 96 attention heads.)

It used to be the case that people only learned or fine-tune their own embeddings, but now all of the major AI providers have their own hosted sets of embeddings. Particularly common ones are OpenAI, with Cohere and Nomic close behind. Google also recently released embeddings for Gemini, with both API and local versions being available.

In addition to standardization via HuggingFace and APIs, MTEB, which benchmarks embeddings publicly, has grown as a resource where you can compare embedding models.

If we look at MTEB, the embedding benchmark today, we can see embedding sizes anywhere from our classic 768 to 4096 and beyond (note all of them are neatly divisible by 2, in line with architectural constraints)

Finally, another consideration has been the change in the landscape with vector databases, which used to be huge, are now becoming more commoditized features in software and platforms like Postgres, S3, and Elasticsearch, leading to less out of the box necessity in vector storage and performance tuning.

With the constant upward pressure on embedding sizes not limited by having to train models in-house, it’s not clear where we’ll slow down: Qwen-3, along with many others is already at 4096. It does appear that we are beginning to trim down on growth. OpenAI implemented a concept called matryoshka representation learning that trains embeddings with the “most important” concepts first, and additional embeddings adding incremental gains, meaning that an embedding learned in 1024 dimensions might be just as useful in 64, as long as the first 64 dimensions compress most of the information efficiently, and also making sure we re-normalize them. There are also research that indicates that not all embeddings are necessary in retrieval + search tasks, and that we can in fact, in some cases, truncate 50% of them anyway.

It’s been fascinating to watch the rise of dimensionality and the constant struggle in tradeoffs between creating ever-larger models and then the need for those embedding sizes to perform at inference and storage time, aka continuously coming up against the age-old machine learning tradeoff of recall versus precision, and the engineering tradeoff of hardware limitations versus software versus business considerations.

As these architectures have matured and internal models have become inference points of public-facing paid APIs, embeddings have gone from a mysterious byproduct of internal machine learning systems in companies with lots of data to a commodity used across many AI-powered applications across numerous application stacks.

All in all, it’s been so much fun watching this space evolve, even if it means it looks like I’ll be having to update my paper over and over again.

Enabling Hugo static site search with Lunr.js

Fri, 08 Aug 2025 00:00:00 +0000

There comes a time in every woman’s life when she only wants one thing: for her mininmal static site to finally have some of the same features that dynamic blogging platforms do, namely search.

So now I’ve implemented search on this blog, you should see it in the top right and the results render in-line in the drop-down.

I’m going to tweak the results because the boosting is not quite where I’d like it to be yet, but try it out!

Some implmenetation details:

My blog runs on Hugo, so I looked for solutions that work with Jekyll/Hugo. I picked very fun and lightweight lunr.js, which is a single file that implements BM-25 search based on Solr primitives (it’s called Lunr because it’s smaller and “less bright” than Solr, which I love.)

To create the index that Lunr uses for search, I had Claude write a quick Python script that traverses my post directories and outputs a single JSON file of all indexes posts.

The entries look like this:

{
 "id": "2018/07/23/good-small-datasets/",
 "title": "Good small datasets",
 "ref": "2018/07/23/good-small-datasets/",
 "content": "John Lavery, The Chess Players (1929) I've been working on a project that, like most projects, requires testing with a dataset. My personal criteria are: + Relatively small size (Less than 100 KB, or 100ish rows) + At least 5-6 features (columns) + Should have both numerical and text-based features + Ideally a range of different kinds of numbers + Has good documentation + Is open and available to the public + Relatively available for both R and as individual CSV files or Python imports (APIs and",
 "summary": "John Lavery, The Chess Players (1929) I've been working on a project that, like most projects, requires testing with a dataset. My personal criteria are: + Rela...",
 "date": "2018-07-23T00:00:00Z",
 "creator": "",
 "site": "",
 "url": "/blog/2018-07-23-small-datasets"
 }

following this logic:

return {
 'id': slug,
 'title': frontmatter.get('title', 'Untitled'),
 'ref': f"{slug}", 
 'content': text[:500], 
 'summary': text[:160] + '...' if len(text) > 160 else text,
 'date': frontmatter.get('date', ''),
 'creator': frontmatter.get('creator', ''),
 'site': frontmatter.get('site', '')
 }

I keep that code separate from my Hugo site, in a search directory that’s initialized as a uv project:

├── README.md
├── create_index.py
├── pyproject.toml
└── uv.lock

and generates the lunr.json index into my static repo.

This code is run at build time in my GitHub Actions before the hugo build step:

- name: Install uv
 uses: astral-sh/setup-uv@v4
- name: Generate search index
run: |
 echo "Generating search index..."
 uv run search/create_index.py lunr-index.json
 echo "Search index generated successfully"

Generating the script correctly took several passes because Claude missed some data cleaning nuances (I don’t want any HTML output, I wanted slightly longer summarization outputs, I didn’t want to include base pages, etc.), but now it regenerates the index in <5 sec every time.

Then, all of that is rendered in my navbar via Hugo templates, namely by including the logic to serve the search bar in the header partial. I’m abstracting away a ton of “draw the owl” features here, such as doing passes to make sure the CSS for the search bar matches my main blog theme, and rendering the dropdown results.

LLMs helped move this feature along a fair bit here because:

It’s an extremely small feature where I can clearly test the output and see the generated code
The context window for changes was fairly small
The biggest lift was in automating index creation and getting the right syntax for the Python script and the data was easy to check.
It has a tight local testing loop: hugo build and hugo serve are extremely fast locally and offer 95% parity to the served site in GH pages.

This has been a fun addition and I’m excited to do more such as potentially add typeahead suggestions and tune the query.

My favorite use-case for AI is writing logs

Wed, 16 Jul 2025 00:00:00 +0000

One of my favorite AI dev products today is Full Line Code Completion in PyCharm (bundled with the IDE since late 2023). It’s extremely well-thought out, unintrusive, and makes me a more effective developer. Most importantly, it still keeps me mostly in control of my code. I’ve now used it in GoLand as well. I’ve been a happy JetBrains customer for a long time now, and it’s because they ship features like this.

I frequently work with code that involves sequential data processing, computations, and async API calls across multiple services. I also deal with a lot of precise vector operations in PyTorch that shape suffixes don’t always illuminate. So, print statement debugging and writing good logs has been a critical part of my workflows for years.

As Kerningan and Pike say in The Practice of Programming about preferring print to debugging,

…[W]e find stepping through a program less productive than thinking harder and adding output statements and self-checking code at critical places. Clicking over statements takes longer than scanning the output of judiciously-placed displays. It takes less time to decide where to put print statements than to single-step to the critical section of code, even assuming we know where that is.

One thing that is annoying about logging is that f-strings are great but become repetitive to write if you have to write them over and over, particularly if you’re formatting values or accessing elements of data frames, lists, and nested structures, and particularly if you have to scan your codebase to find those variables. Writing good logs is important but also breaks up a debugging flow.


from loguru import logger

logger.info(f'Adding a log for {your_variable} and {len(my_list)} and {df.head(0)}')

The amount of cognitive overhead in this deceptively simple log is several levels deep: you have to first stop to type logger.info (or is it logging.info? I use both loguru and logger depending on the codebase and end up always getting the two confused.) Then, the parentheses, the f-string itself, and then the variables in brackets. Now, was it your_variable or your_variable_with_edits from five lines up? And what’s the syntax for accessing a subset of df.head again?

With full-line-code completion, JetBrains’ model auto-infers the log completion from the surrounding text, with a limit of 384 characters. Inference starts by taking the file extension as input, combined with the filepath, and then the part of the code above the input cursor, so that all of the tokens in the file extension, plus path, plus code above the caret, fit. Everything is combined and sent to the model in the prompt.

The constrained output good enough most of the time that it speeds up my workflow a lot. An added bonus is that it often writes a much clearer log than I, a lazy human, would write, logs. Because they’re so concise, I often don’t even remove when I’m done debugging because they’re now valuable in prod.

Here’s an example from a side project I’m working on. In the first case, the is autocomplete inferring that I actually want to check the Redis URL, a logical conclusion here.

redis = aioredis.from_url(settings.redis_url, decode_responses=True)

In this second case, it assumes I’d like the shape of the dataframe, also a logical conclusion because the profiling dataframes is a very popular use-case for logs.

from pandas import DataFrame

data = [['apples', '2.5'], ['oranges', '3.0'], ['bananas', '4.0']]
column_names = ['Fruit', 'Quantity']
df = pd.DataFrame(data, columns=column_names)

Implementation

The coolest part of this feature is that the inference model is entirely local to your machine.

This enforces a few very important requirements on the development team, namely compression and speed.

The model has to be small enough to bundle with the IDE for desktop memory footprints (already coming in at around ~1GB for the MacOS binary), which eliminates 99% of current LLMs
And yet, the model has to be smart enough to interpolate lines of code from its small context window
The local requirement eliminates any model inference engines like vLLM, SGLM, or Ray which implement KV cache optimization like PagedAttention
It has to be a model that’s fast enough to produce its first token (and all subsequent tokens) extremely quickly,
Finally, it has to be optimized for Python specifically since this model is only available in PyCharm

This is drastically different from the current assumptions around how we build and ship LLMs: that they need to be extremely large, general-purpose models served over proprietary APIs. we We find ourselves in a very constrained solution space because we no longer have to do all this other stuff that generalized LLMs have to do: write poetry, reason through math problems, act as OCR, offer code canvas templating, write marketing emails, and generate Studio Ghibli memes.

All we have to do is train a model to complete a single line of code with a context of 384 characters! And then compress the crap out of that model so that it can fit on-device and perform inference.

So how did they do it? Luckily, JetBrains published a paper on this, and there are a bunch of interesting notes. The work is split into two parts, model training, and then the integration of the plugin itself.

The model is trained is done in PyTorch and then quantized.

First, they train a GPT-2 style Transformer decoder-only model of 100 million parameters, including a tokenizer (aka autoregressive text completion like you’d get from Claude, OpenAI, Gemini, and friends these days). They later changed this architecture to Llama2 after the success of the growing llama.cpp and GGUF community, as well as the better performance of the newer architecture.
The original dataset they used to train the model was a subset of The Stack, a code dataset across permissive licenses with 6TB of code in 30 programming languages
- The initial training set was “just” 45 GB and in preparing the data for training, in data cleaning, for space constraints, they remove all code comments in the training data specifically to focus on code generation
- They do a neat trick for tokenizing Python (using a BPE-style tokenizer optimized for character pairs rather than bytes, since code is made up of smaller snippets and idioms than natural language text) which is indentation-sensitive, by converting spaces and tabs to start-end <SCOPE_IN><SCOPE_OUT> tokens, to remove tokens that might be different only because they have different whitespacing. They ended up going with a tokenizer vocab size of 16,384.
- They do another very cool step in training which is to remove imports because they find that developers usually add imports in after writing the actual code, a fact that the model needs to anticipate
They then split into train/test for evaluation and trained for several days on 8 NVidia A100 GPU with a cross-entropy loss objective function

Because they were able to so clearly focus on the domain and understanding of how code inference works, focus on a single programming languages with its own nuances, they were able to make the training data set smaller, make the output more exact, and spend much less time and money training the model.

The actual plugin that’s included in PyCharm “is implemented in Kotlin, however, it utilizes an additional native server that is run locally and is implemented in C++” for serving the inference tokens.

In order to prepare the model for serving, they:

Quantized it from FP32 to INT8 which compressed the model from 400 MB to 100 MB
Prepared as a served ONNX RT artifact, which allowed them to use CPU inference, which removed the CUDA overhead tax(later, they switched to using llama.cpp to serve the llama model architecture for the server.
Finally, in order to perform inference on a sequence of tokens, they use beam search. Generally, Transformer-decoders are trained on predicting the next token in any given sequence so any individual step will give you a list of tokens along with their ranked probabilities (cementing my long-running theory that everything is a search problem).

Since this is computationally impossible at large numbers of tokens, a number of solutions exist to solve the problem of decoding optimally. Beam search creates a graph of all possible returned token sequences and expands at each node with the highest potential probability, limiting to k possible beams. In FLCC, the max number of beams, k, is 20, and they chose to limit generation to collect only those hypotheses that end with a newline character.

Additionally they made use of a number of caching strategies, including initializing the model at 50% of total context - i.e. it starts by preloading ~192 characters of previous code, to give you space to either go back and edit old code, which now no longer has to be put into context, or to add new code, which is then added to the context. That way, if your cursor clicks on code you’ve already written, the model doesn’t need to re-infer.

There are a number of other very cool architecture and model decisions from the paper that are very worth reading and that show the level of care put into the input data, the modeling, and the inference architecture.

The bottom line is that, for me as a user, this experience is extremely thoughtful. It has saved me countless times both in print log debugging and in the logs I ship to prod.

In LLM land, there’s both a place for large, generalist models, and there’s a place for small models, and while much of the rest of the world writes about the former, I’m excited to also find more applications built with the latter.

20 years of YC

Mon, 17 Mar 2025 00:00:00 +0000

I saw recently that YCombinator celebrated its 20th anniversary.

Hacker News is slightly younger, but to me the two go hand in hand.

As far as I can tell, I actively started reading Hacker News around 2011. I don’t remember how I heard about it. It was probably on Reddit or Digg. Once I found it, I started reading every day, mostly because the comment sections were so full of smart people in tech.

At the time I was working as a data analyst, mostly with SQL and Excel. I understood that I would need to learn much more to move into engineering, which I was getting excited about, but didn’t quite understand how to bridge the gap.

When I first started reading HN, I didn’t understand 99% of the linked content, the jargon in the discussions, or the companies mentioned, but I was determined to learn. My general approach was to skim the top headlines and headlines that I could understand, read the post, and then read the HackerNews discussion.

As I read the discussion, I would come upon terms I had no idea about: Big O notation, collaborative filtering, caching, Hindley-Milner type systems, lambda architectures, CI/CD pipelines, cryptography, generics, build or buy, B-trees, Bloom filters, trunk-based development, red/green deploys. I would look all of them up and go down countless rabbit holes.

HN is a bit bigger these days, but, thanks to contributors and the tireless efforts of dang, still so perfectly ambiently captures what tech is thinking about in the current moment. If you don’t have a supportive community at work － like many of us data scientists － HN was the perfect ambient watercooler to be near senior technically excellent people in the industry.

In the first few years, I read maybe 2 links out of the 30 ever on the front page. But, because I read HN 4-5 times a day, after a few years, things started falling into place.

Other than constantly studying, Hacker News was one of the main things that pulled me up out of my bootstraps from a non-technical major afraid of SSHing into a server, to a person confidently deploying code to millions of users. There is only one other thing that has brought as much value to my career in tech, and that was the connections I made on old Twitter.

Over time, I gained enough confidence to start blogging on technical topics, HN gave me something new: being aware that HN might pick apart that content forced me to learn to write clearly and precisely for a highly technical, educated audience.

My favorite feeling is when, via a link, something I have written or created has made an impact on someone out there in the ether. Or, from the reader side, when I read something that makes me go “I’m not alone. This person has also thought about these problems,” and I can’t wait to see if there are any good discussions on the topic.

YC is proud of the companies it launched: Reddit, Airbnb, Instacart, Doordash, Stripe. Ironically, the biggest multiples it’s generated for the industry have been from a low-dazzle text forum written in Arc Lisp.

Thank you for everything, HN.

You can just hack on ATProto

Thu, 23 Jan 2025 00:00:00 +0000

Icon by iconixar

Since I signed up for Bluesky last year, I’ve been wanting to make something using the AT Protocol that the platform is built on top of.

I finally had a chance to do it over the holiday break and built GitFeed, a small Go app that filters the Bluesky network firehose by posts that have GitHub links and renders them into a refreshable, ephemeral feed.

You can see GitFeed here, but it might not actually be running since I didn’t build it for scale, and it’s running on a tiny DigitalOcean Droplet that’s specced out at 1 GB Memory / 1 Intel vCPU / 35 GB Disk , is entirely unloadbalanced, un-load-tested and has zero observability or alerting when my very scientific process of running nohup fails.

The cool part of an open protocol is that you can also just clone the repo and run a hosted version of it yourself.

ATProto

Bluesky is both a decentralized protocol, called AtProto and a social media company, called Bluesky plc that develops both the protocol and one of the Apps running on the protocol, Bluesky.

There is a lot more in the AT Protocol Paper, but the basics are this:

Data for each user is hosted individually in a PDS - a personal data store - which is a database storing a collection of records that are cryptographically signed and encoded in DAG-CBOR format. The record schema is defined by a “lexicon”, which is dependent on the type of data being transferred. The records themselves are stored in a merkle search tree structure which makes it easy to rebalance records efficiently both on read and write. Its default storage engine is SQLite.
Each user has a PDS, exposed as a web service to network indexers. There is an indexer, the relay, which “scrapes” but really hits all the PDSes in the network for updates. Right now there is only one true relay, run by Bluesky the company, and there is a lot of debate around what that means for a decentralized network and efforts to diversify and decentralize. To better get a sense for how the data model works, you can play around with this tool, which is what I spent a lot of time doing during this project.

The TL; DR is that you can think of the At Proto Atmosphere as a collection of databases, or, really, websites, that the relay indexes and turns into the firehose. Data is then filtered on the firehose side for CSAM and other logic, before it’s turned into an AppView. The AppView is what you see if you sign into bsky.app.

If this sounds familiar, it’s because it’s how web crawlers, including Google work, with the exception that their crawled results are not available to everyone for access.

Steve has a very nice write-up of all of this, with a beautiful ascii diagram.

Al(most) all of the data streaming through each person’s PDS is public, and enables the creation of projects like the Bluesky firehose as a screensaver, or goodfeeds, surfacing feeds across the network., or TikTok and Instagram-like apps. As you can imagine, the protocol then lends itself to a lot of nice experimentation (make sure to check the TOS/Developer guidelines before you do so).

Let’s find all the gists

Initially, I wanted to create a custom feed on Bluesky. Generally, people create these by filtering the network to include all feeds about cats, or feeds from only mutual follows, or one of my recent favorites, gift articles, which includes links from gift articles that you can click through to read.

My idea was: collect all the posts that have a link to github gists, because people put really cool stuff in gists, so I could find and expose to other users some really cool code snippets of what people are hacking on around the platform.

Initially I thought I might be able to create a lightweight recommendation feed based on aggregate likes. Or, I could create a trending links feed. But, if you want to do any machine learning, you need to start consuming the firehose at scale, collecting the data, and setting up storage, and I wanted to learn Go, not implement distributed systems - don’t use N computers when you can use one.

Moreover, in order to implement a feed, generally, you need to also implement:

pagination
a cursor in case you lose your place in consuming the feed
latency considerations
Thinking about how you render feed objects (eventually they become large and need to be hydrated)
potentially ranking that feed’s content
retries and handling for if you skip feed elements

Additionally, specific to atproto, your feed is published at your own PDS - Personal Data Server, a LOT more about this here. You can see this at the link on the Gift Articles Feed:

https://bsky.app/profile/did:plc:o4s55v3tsfph6whswxccpsia/feed/aaaixbb5liqbu

Given that the feed is published and linked to your own data store, it made me hesitant to experiment in case I messed something up and lost all of my data.

There are lots of clients to create and publish feeds, but as a nerd, I didn’t want to have a third party handle this out of principle. Moreover, for gists only, there was not enough data to be interesting. The Bluesky network is growing, having surpassed 29 million users. But, at this scale, interesting content when you filter at this level of granularity is sparse.

After messing around with the (excellent) Python client a bit in this repo, I narrowed down to the problem I actually wanted to solve:

consuming all links with "github.com" in the link name, and consuming them via jetstream rather than the firehose.

That’s how GitFeed was born.

Why GitFeed? Why Go?

McFunley says you only have so many innovation tokens. This also applies to side projects. The other way I’ve seen this explained, is that you can either pick a new language, a new stack, a new business problem to solve, or new people to work with, but not all four at the same time, otherwise you will never ship.

I always want to get to a demo quickly. But, the goal of side projects is to do stuff you wouldn’t get to explore otherwise, so I chose to spend my innovation tokens on:

learning a new backend language, Go: this thing had to serve code fast as a binary and have a very simple deployment story. I have a background in Java/Scala and initially thought about Java, but unfortunately there is no small light-weight Java server that I’m aware of (Spring doesn’t count) and Go has steadily been growing in popularity since its inception and especially after the language added features like generics and I wanted to check it out. Just as importantly, the backend of Bluesky and the protocol were specced out and written in Typescript and Go, and I thought it might be easier to navigate the Atmosphere in those languages.
getting slightly better at front-end dev: I’d dabbled a bit with front-end design for Viberary but used almost no Javascript, which I’d need to render a feed.
understanding the At Proto data model

I didn’t need to:

Understand the problem of working with a feed of streamable content - I did a lot of feeds work when I worked on recommendations at Tumblr and I find it to be one of the most fun and satisfying problems in engineering to work on.
Understand how to run/deploy on DigitalOcean using GitHub Actions - I already did this for Viberary and the path was pretty smooth
Understand SQLite
Finally, I wanted to track and catalogue my usage of LLMs for this project. I’m constantly evaluating their use-cases.

The Jetstream

I didn’t want to spend time implementing pagination, hydration, latency mitigation, and a data storage strategy. I wanted to get going quickly, so I decided to use the Jetstream. Jetstream is a relatively new content source for Bluesky content (relatively because everything is brand-new and being built on the fly.)

Working with the Bluesky firehose has a set of complications other than pagination: it also has its own data formats (CBOR and CAR for all the full merkle trees in the git repos) that take time in learning how to parse.

Moreover, the sheer volume of firehose events has grown to the point where folks consuming it need to invest heavily in scaling strategies for downstream application consumers.

Jetstream instead streams content in JSON, with reduced bandwidth and costs.

As a tradeoff, Jetstream is less stable, doesn’t contain content that needs API verification, and don’t offer pagination, activity offets, or uptime guarantees. As the docs say, it’s good for low-stakes side projects that don’t require heavy authentication or veracity, aka gitfeed.

Serving GitFeed

We’re building a simple web app with two components: that are two separate go processes:

An ingest go module that will ingest posts and writes them to a SQLite DB that we instantiate as part of the application
A serve go module that serves the app via API.

The serving module is just a few Javascript files with static HTML pages. I was very overwhelmed by all the choices in the front-end ecosystem (although the developer docs helped a ton!), so I just ended up going with plain old Vanilla JS.

When we refresh the site, we make a call to the DB (via the posts API endpoint) to surface the posts in reverse chronological order.

export async function fetchPosts() {
 const container = document.getElementById('postContainer');
 container.innerHTML = '<div class="loading">Loading posts...</div>';
 try {
 console.log('Fetching new posts...');
 const response = await fetch('/api/v1/posts');
 if (!response.ok) {
 throw new Error(`HTTP error! status: ${response.status}`);
 }
 const posts = await response.json();
 container.innerHTML = '';

And here’s the actual call site:

import {
 fetchPosts,
 updateTimestamp
} from './feed.js';

console.log('Main.js loaded');

document.addEventListener('DOMContentLoaded', async () => {
 console.log('DOM Content Loaded');
 try {
 await fetchPosts();
 await updateTimestamp();
 } catch (error) {
 console.error('Error in main initialization:', error);
 }
});

I was surprised at how much about JS was already familiar to me from Python and PHP, but where I really got stuck was in understanding how the DOM and Javascript work together, what a Javascript app structure looks like, and the Javascript ecosystem.

Using LLMs to build GitFeed:

I’ve done a lot of work with llamafile, and recently, I’ve also been enjoying the local LLM stack of: Ollama for the model backend and OpenWebUI for the front-end. Ollama serves versioned GGUF model weights wrapped in a Docker-like paradigm that hits an API wrapping llama.cpp in an (of course) Go interface. For this project, I used mistral:latest and qwen2.5-coder:latest , the best code model at the time (in the ancient space of 3 months ago, Deepseek3 wasn’t out). I did reasonably well between the two of them, with only 5% of requests that I had to bypass and send to Claude. I did find myself getting frustrated because I couldn’t clearly articulate the unknown unknowns I had about Javascript, though, and eventually I just gave up and bought Eloquent Javascript which I’m hoping to dig into this year to better understand what Qwen and I wrote together and how I can improve it.

Ingest

Consuming the Jetstream is easy. There are four public instances of Jetstream, hosted by Bluesky the company, so we need to connect to one of these, consume the content, process and filter the data, and save it to our database for serving.

You can check what’s up in the Jetstream with this nifty command line tool:

websocat wss://jetstream2.us-west.bsky.network/subscribe\?wantedCollections=app.bsky.feed.post | jq .

This is everything people are posting on Bluesky! (It sometimes gets very, very weird so be careful if you don’t want to look at NSWF texts.)

And now let’s look at GitHub posts. There aren’t a lot of them, so you might not get an even for at least a few minutes:

websocat wss://jetstream2.us-west.bsky.network/subscribe\?wantedCollections=app.bsky.feed.post | grep "github" | jq .

Jetstream is implemented as a websocket connection to the source.

wsManager := NewWebSocketManager(
 "wss://jetstream2.us-west.bsky.network/subscribe?wantedCollections=app.bsky.feed.post",
 pr,
)

A websocket is a protocol (like HTTP) that enables client-server communication over TCP but works best for streaming data without the need for continuous polling or webhooks (unlike HTTP).

A connection is instantiated with a handshake between the client and the server in HTTP first, and when the request is processed, both switch to websockets for communication.

I used the gorilla/websocket implementation of the websocket procool. for handling the core websocket logic.

We need to be able to read and write to/from the websocket:

The main thing to internalize about working with WebSockets in Go is that each client connection should get at least two goroutines: one that continuously processes messages coming from the client (i.e., a “read pump”), and one that continuously processes message going out to the client (i.e., a “write pump”).

However, this becomes easier since for GitFeed, we’re only reading from and not writing to the websocket, we need to implement logic to readPump

func (w *WebSocketManager) readPump(ctx context.Context) {

	w.Connect(ctx)
	counter := 0
	for {
		select {
		case <-ctx.Done():
			log.Printf("Exiting readPump: got kill signal\n")
			return
		default:
			var post db.ATPost
			if err := w.conn.ReadJSON(&post); err != nil {
				w.Connect(ctx)
				continue
			}
			counter++
			if counter%100 == 0 {
				log.Printf("Read %d posts\n", counter)
			}

			// Process the post
			dbPost := ProcessPost(post)

			if err := w.postRepo.WritePost(dbPost); err != nil {
				w.errorHandler(fmt.Errorf("failed to write post: %v", err))
				continue
			}
			log.Printf("Wrote Post %v", dbPost.Did)
		}
	}
}

However, unlike an HTTP call, websocksets are open persistently and don’t offer any guarantees of retries, so we have to implement this logic ourselves. Fortunately, gorilla has a lot of good examples.

You’ll notice a couple key points here: first, we log and handle the case where the web socket disconnects. Then, we do some ultra-fancy print logging to keep track of how many posts we’ve actually processed. And finally, we now get to the actual data, an var post db.ATPost that we process, parse, and write to the database, which we initialize as a post repository, a fancy word for “database with dependency injection”.

Golang tooling:

Go just works out of the box. Unlike my beloved Python, it doesn’t need uv, formatting, linting, or special build processes. At least, for a fairly small project, everything is batteries included. In fact, its boringness and rigidity allowed me to move really quickly. What surprised me is that I thought that VSCode would work really well with go, but actually didn’t as code I imported wouldn’t get loaded automatically, and there were a couple bugs that made me switch to Goland, which works extremely smoothly, without fail, and its local autocomplete at the line level is much better than PyCharm’s equivalent, likely because Go is much smaller, and statically-typed.

Jetbrains LLMs:

Jetbrains’ local LLMs are extremely well-done and I’d encourage anyone interested to check out the paper.

What is an AtProto Post

Now we get to the heart of the matter: once our websocket is open, we are ingesting a stream of JSON objects. AtProto has its own data model, defined using schemas called “Lexicons”. For posts and actions, they look like this.

{
 "did": "did:plc:eabmaihciaxprqvxpfvl6flk",
 "time_us": 1725911162329308,
 "kind": "commit",
 "commit": {
 "rev": "3l3qo2vutsw2b",
 "operation": "create",
 "collection": "app.bsky.feed.like",
 "rkey": "3l3qo2vuowo2b",
 "record": {
 "$type": "app.bsky.feed.like",
 "createdAt": "2024-09-09T19:46:02.102Z",
 "subject": {
 "cid": "bafyreidc6abdkkbchcyg62v77wbhzvb2mvytlmsychqgwf2xojjtirmzj4",
 "uri": "at://did:plc:ab7b35aakoll7hugkrjtf3xf/app.bsky.feed.post/3l3pte3p2e325"
 }
 },
 "cid": "abfyreidwaivazkwu67xztlmuobx35hs2lnfh3kolmgfmucldvhd3sgzcqi"
 }
}

The DID is the ID of the PDS (user repository) where the action happened, the record collection type of app.bsky.feed.post is what we care about, and each record has both a text entry, which truncates the text, and a facet, which has all the contained links and rich text elements in the post.

{
 "did": "did:plc:",
 "time_us": 1735494134541,
 "type": "com",
 "kind": "commit",
 "commit": {
 "rev": "",
 "type": "c",
 "operation": "create",
 "collection": "app.bsky.feed.post",
 "rkey": "",
 "record": {
 "$type": "app.bsky.feed.post",
 "createdAt": "2024-12-29T17:42:14.541Z",
 "embed": {
 "$type": "app.bsky.embed.external",
 "external": {
 "description": "",
 "thumb": {
 "$type": "blob",
 "ref": {
 "$link": ""
 },
 "mimeType": "image/jpeg",
 "size": 
 },
 "title": "",
 "uri": ""
 }
 },
 "facets": [
 {
 "features": [
 {
 "$type": "app.bsky.richtext.facet#link",
 "uri": ""
 }
 ],
 "index": {
 "byteEnd": 85,
 "byteStart": 54
 }
 }
 ],
 "langs": [
 "en"
 ],
 "text": "..."
 },
 "cid": ""
 }
}

So we parse these JSON objects and store them as Go structs. Go has a super handy tool where you can paste a JSON object and get back the Go struct.

And then we write that struct to a DB with a posts table that we’ve already instantiated for this purpose.

if err := w.postRepo.WritePost(dbPost); err != nil {
 w.errorHandler(fmt.Errorf("failed to write post: %v", err))
 continue
}

SQLite

Enough has been written about why SQLite is awesome and amazing for smaller, and even larger projects so I’ll skip that here and say that I didn’t even consider using anything else for GitFeed.

There are a ton of optimizations you can perform on SQlite to really juice performance, and I set a few of them in anticipation of many users hitting the server, and also easing read-write contention on the db.

Serving

After working extensively with FastAPI and Flask, I was really excited to learn just how batteries-included net/http module was in Go. I didn’t need to install anything extra - I was immediately writing routes and handlers.

We set up several routes to deal with our saved posts here.

// Start post service
	fmt.Println("Connect to post service...")
	pr := db.NewPostRepository(database)
	postService := &handlers.PostService{PostRepository: pr}

	// Create web routes
	routes.CreateRoutes(postService)

	log.Printf("Starting gitfeed server...")
	log.Fatal(http.ListenAndServe(":80", nil))

And then we’re up and running as soon as we build and run our Go executable.

Building and Running Go Artifacts

So easy. Just a small Makefile and we’re rebuilding and testing, and then serving the binaries.

Data Considerations

I don’t want to keep any data or manage it, this is meant to be an ephemeral snapshot, so a goroutine deletes the oldest data once there are more than 10 posts in the database.

GitHub

One last piece we need, which early GitFeed testers suggested, was a way to, after you load the posts from the database, render the associated GitHub metadata with the repo, so we also, after we load the posts from the DB, hit the GitHUB API to enrich the post.

Ops Considerations

I run all of this on a small DigitalOcean droplet, and redeploy to the droplet with new code code merged to main via GitHub actions. There’s no monitoring or alerting, something I’d like to add for the future.

Final Reflections

This app was so much fun to develop and I learned an enormous amount of stuff. Small, self-contained apps are a joy, and especially when there’s a front-end component where you have a self-reinforcing feedback loop. Since us machine learning engineers work at what a friend called “the back-end of the backend”, we don’t often get to experience UI changes, and seeing through something end-to-end was a joy.

There were points of friction: It was definitely frustrating getting up and going with a whole new language and tech stack, but once I got back into the flow, it was great.

As always, the hardest part of this project, as with any project, was understanding the data model and the business logic, and parsing out those objects correctly. The second-hardest was aligning elements in CSS.

I’d love to get to a point where the app can surface Trending GitHub repos.. And maybe add some unit tests.

Upon writing this, I realized I have like three other posts I want to write about this process, so I’ll leave this as-is for now.

How FastAPI path operations work

Tue, 14 Jan 2025 00:00:00 +0000

If you’re building a new Python web app these days, there’s a good chance you’re using FastAPI. There are a lot of features that make FastAPI easy to get started with. There are also a lot of nuances that take a while to understand. One feature I’ve been untangling is the way FastAPI manages calls to API routes via decorated path parameters. The new year is a perfect time to take a deeper dive.

What happens in a web server

When we build a web app, one of the critical components is a web server, a program that listens for incoming requests from the network. It then translates those requests into methods that are called in the backend.

To better understand what’s going on under the covers, we can first implement a simple web server using the http.server module included in Python’s standard library.

We need to write a program that listens on a port and accepts HTTP requests. It accepts the request, parses the path route, and parses any data attached to the HTTP call. Or, “All I want is to cURL and parse a JSON object”.

import json
from http.server import BaseHTTPRequestHandler, HTTPServer
from urllib.parse import urlparse, parse_qs

class RequestHandler(BaseHTTPRequestHandler):

 def parse_path(self, request_path: str)-> dict:
 """
 Parse request path
 """
 parsed = urlparse(request_path)
 params_dict = parse_qs(parsed.query)
 return params_dict

 def store_urls(self, request_path: str)-> None:
 """
 Parse URLs and store them
 """
 params = self.parse_path(request_path)
 for key, val in params.items():
 self.data_store.put_data(val[0])

 def return_k_json(self, k:dict)-> BinaryIO:
 """
 Return json response
 """
 self.send_response(200)
 self.send_header("Content-type", "application/json")
 self.end_headers()
 # Contains the output stream for writing a response back to the client. 
 # BufferedIOBase that writes to a stream
 # See https://docs.python.org/3/library/io.html#io.BufferedIOBase.write
 self.wfile.write(json.dumps(k).encode('utf-8'))

 def bad_request(self):
 """
 Handle bad request
 """
 self.send_response(400)
 self.send_header("Content-type", "application/json")
 self.end_headers()

 def do_GET(self):
 
 request_path = self.path

 if self.path == "/":
 self.return_k_json({"ciao": "mondo"})
 if request_path.startswith("/get"):
 key = self.parse_path(request_path)
 self.return_k_json({"jars": key["key"]})
 self.send_response(200)
 else:
 self.bad_request()
 self.end_headers()

 def do_POST(self):
 request_path = self.path

 if request_path.startswith("/set"):
 self.store_urls(request_path)
 self.send_response(200)
 else:
 self.bad_request()

if __name__ == "__main__":
 host = "localhost"
 port = 8000

 server = HTTPServer((host, port), RequestHandler)
 print("Server started http://%s:%s" % (host, port))
 server.serve_forever()

What’s going on here?

Let’s say that we produce Nulltella, an artisinal hazlenut spread for statisticians, and are looking to build a web app that keeps track of all of our Nulltella jars so we can later stand up a prediction service.

We would start by designing a super simple API: As users,

we want to test the server and get back a simple response
we’d like to add jars to our inventory, and
to see the jars we added.

We translate these actions to GET and PUT requests so we can write HTTP calls for them. For simplicity’s sake, we won’t actually store them server-side but we will write them so we can can very simply see how to send data to our app:

We want to test the server:

> python serve.py
> curl -X POST http://localhost:8000/
> {"ciao": "mondo"}

We want to store items:

> curl -X POST http://localhost:8000/set\?key\=8
200 OK

And get back the stored items:

> curl -X GET http://localhost:8000/get\?key\=8 
> {"jars": ["8"]}

Our server needs a way to parse the key pieces of information it receives:

They type of request. do_GET and do_POST handle this implicitly in the HTTP implementation.
The parameters we pass to the path request so that we can do something with them
A route to a method inside our application itself that processes the data

In our simple server, the heart of the routing happens at the method level. If we send a base path, we return {"ciao": "mondo"} . Otherwise, we return the amount of jars we’ve passed in via the request path by parsing the parameters in the path.

def do_GET(self) -> None:
 request_path = self.path

 if self.path == "/":
 self.return_k_json({"ciao": "mondo"})
 if request_path.startswith("/get"):
 key = self.parse_path(request_path)
 # action performed within the web app here
 self.return_k_json({"jars": key["key"]})
 self.send_response(200)
 else:
 self.bad_request()
 self.end_headers()

We can see how this can become complicated quickly. For example, what if we have multiple operations we perform during a GET : what if we get data from a database, or a cache, or we retrieve assets? We’ll have different methods that we process depending on how the path is parsed. What if we also have PUT/DELETE verbs? What if we need authentication? To write to a database? Static pages? Our code complexity relative to our starting point starts to grow, and we now need a framework.

Starlette

Early Python web dev frameworks include juggernauts Django and Flask. More recently, since Python’s async story has grown stronger, frameworks like Starlette have come onto the scene to include async functionality out of the box.

Starlette was built by the creator of Django Rest Framework and includes lightweight operations for the core functionality of HTTP calls and additional operations like web sockets, with the added bonus of being async by default.

To manage an HTTP call the same way we would with our simple server, we can do the following with Starlette:

from starlette.applications import Starlette
from starlette.responses import JSONResponse
from starlette.routing import Route

async def homepage(request):
 return JSONResponse({'ciao': 'mondo'})

app = Starlette(debug=True, routes=[Route('/', homepage),])

We start an instance of a Starlette application, which has processes routes. Each route is linked, at the path level, to the actual method it calls. If Starlette sees that specific route, it calls the method, taking into account logic for parsing and reading HTTP request headers and bodies.

What if we want to add a second method call based on a different route, getting our jar count again?

from starlette.applications import Starlette
from starlette.responses import JSONResponse
from starlette.routing import Route

async def homepage(request):
 return JSONResponse({'ciao': 'mondo'})

async def get_jars(request):
 return JSONResponse({'jars': ['8']})

app = Starlette(debug=True, routes=[
 Route('/', homepage),
 Route('/get_jars', get_jars)
])

We see that we are also passing and processing params, and there is logic that processes the path params based on the method as they come in from the request.

FastAPI’s implementation

FastAPI wraps Starlette - “as it is basically Starlette on steroids” per the docs - and includes Pydantic type validation at the logical boundaries of the application.

Under the covers, when we instantiate a FastAPI application, it’s really “just” an instance of a Starlette application with properties that we override at the application level.

from fastapi import FastAPI

app = FastAPI()

@app.get("/")
async def root():
 return {"ciao": "mondo"}

@app.get("/jars/{id}")
async def get_jars(id):
 return {"message": f"jars: {id}"}

In development, FastAPI uses uvicorn , an ASGI server to listen for incoming requests and handle them according to the routes defined in your application.

Uvicorn initializes the ASGI server, binds it to a socket connections on port 8000 , and starts listening for incoming connections. So, when we send a GET request to the main route hosted by default on port 8000, we expect to get back ciao mondo as a response.

Like our previous applications, FastAPI is still delegating path operations and methods to a router that processes them and parses parameters, but it wraps these in a Python decorator. This is easier to write, but adds a level of complexity at the layer of understanding how the path processing actually happens.

When we perform a path operation in FastAPI, we’re performing the equivalent work of routing that we do with our simple method, but with a lot more rigor and nested definitions.

Within our simple server, we:

Start the server
Listen on port 8000 for incoming requests
When we receive a request, we route it to the do_GET method
Depending on the path of the request, we route it to "/"
We return the results to the client via a 200 status

In FastAPI, we:

Start the uvicorn web server (if in development mode, if production we have to choose gunicorn using the compatible worker class)
Listen on port 8000 for incoming requests
We instantiate an instance of the FastAPI application
This in turn instantiates an instance of Starlette
When we receive a GET request, it’s routed to the application’s self.get method
This in turn calls self.router.get with the path operation
The router is an instance of routing.APIRouter
The .get method on APIRouter takes the path and retunrs return self.api_route. This is the point where the decorater is actually called - we can see the decorator in that method takes a DecoratedCallable function as input and returns a decorated add_api_route, which actually appends the route to the list of routes.

This is purely the set of steps that happens for correct routing - and we didn’t yet address how the path parameters in the path are processed.

Path Parameter Routing

Path parameter routing happens in Starlette, where path parameters are parsed out of the request into a dictionary (just like we do in our simple web application), via the magic of Jinja Templating.

TL; DR

When we write a route in FastAPI that accepts path parameters, we are creating a lengthy callstack that goes through several levels of logic in FastAPI using decorators as input into an application that routes requests and appends methods using decorators to a group of route methods; those requests are then passed onto Starlette which does the work of parsing the path variables, using Jinja templates, into dictionaries which the application can then work with and return data to you!

Everything I did in 2024

Fri, 03 Jan 2025 00:00:00 +0000

I want to get back into writing more regularly this year, so in light of that, here’s my last year in review.

Evaluating LLMs

Like many of us in tech, I spent a large portion of 2024 thinking about and working with LLMs, but I was lucky enough to do it for work. I spent the year designing, building, open-sourcing, (and naming! 🐊) an application to evaluate LLMs, Lumigator.

In support of that work, I did open-source work in the LLM ecosystemand learned a ton of stuff along the way. Just when I thought I had reached machine learning enlightenment, I learned a ton about the weird nondeterministic properties of LLMs, their evaluation methods, Ray + Ray Serve, vLLM, Llamafile, GGUF, FastAPI, OpenAPI-compatible APIs, and much, much, much more. I am hoping to translate some of these into blog posts as well.

There is always more to learn in machine learning, particularly in the fast-moving world at the bleeding edge. Lucikly, the fundamentals always remain the same.

My biggest takeaways from this year are that:

Machine learning in industry is mostly still made up of good engineering, and in order to work with LLMs, you need to have both machine learning context and engineering discipline. There’s some debate whether the “old” machine learning skills (gradient boosted trees anyone? 👵) still apply in this Brave New World of NonTabular Scraped Internet, and I am of the opinion that these skills are more valuable than ever, particularly as engineers without ML context enter the frustrating world of building with non-deterministic experimental loops.

But what’s even more valuable is being able to ship things that hold engineering rigor. Even if you are deep in academic research, taking an engineering approach will make your life significantly easier.

As part of this, I spent a significant amount of the year widening my own context window about engineering best practices: Over the summer, I learned C and this fall, I learned JavaScript and Go and built my first working project, Gitfeed. I am particularly enamored with Go: it’s so boring, so clean, and so easy to build things that are simple and move quickly out of the box compared to the Python ecosystem.

This is not to say that the Python world, which is my first love, is not changing and evolving. I moved to uv from pyenv for both work and personal projects in 2024 and have never looked back, particularly as uv continues to work to gain support for PyTorch and all of its cuda-related install issues.

There is too much going on in the model landscape: too many models, too much choice, too many platforms, too much money, too much drama. My main strategy is to follow what’s going on on a daily basis to keep my finger on the pulse and then basically ignore it. If people mention it again over the course of 2-3 months, continue to pay attention.

In light of this, some contradictary advice if you’re in the space is that you should also always be playing around with stuff, trying it out, building it and breakign it. There are some classic papers and books for understanding the context and theory, but most of the stuff I read is still in r/LocalLLaMA or in various tweets/skeets and blog posts. We are not at the level yet for this stuff where cannon exists, although I have noticed that there are technical books being published about AI engineering this year, which means we are starting to solidify.

Some LLM tools I tried out and loved this year were llamafile, for getting started with GGUF transformer and embedding models with a server extremely quickly, and ollama + openwebui for an experience that is nearly identical to using Claude. In fact, I’ve switched over from Claude to using mistral:latest locally for most of my LLM usage, which is basically code search. Mistral is a model that has consistently worked well for me both at work and for my own purposes, and I’ve enjoyed the chance to pit it against newer models like Qwen2.5 Coder, Llama2, and phi.

I am still having a hard time wrapping my mind around what LLM models are useful for, in general, and for me personally. I spend a lot of time reading blog posts about how other (smarter) people use LLMS and against academic papers that study the success of these models at machin learning tasks stemming in their roots as NLP models focusing on problems like text completion, classification, translation, and summarization.

A very cool and fun related thing I did while reviewing the “classics” was edit this wonderful series of posts by Katharine about model memorization which you should read if you’re interested in their internals.

I also spent a significant part of the year working on web development. I wrote earlier in the year that there is a bifurcation in how we consume LLMs:

The LLM ecosystem is currently bifurcated between HuggingFace and OpenAI compatibility: An interesting pattern has developed in my development work on open-source in LLMs. It’s become clear to me that, in this new space of developer tooling around transformer-style language models at an industrial scale, you are generally conforming to be downstream of one of two interfaces: models that are trained and hosted using HuggingFace libraries and particularly the HuggingFace hub as infrastructure - in practicality, this means dependence on PyTorch’s programming paradigm, which HuggingFace tools wrap (although they now also provide interop between Tensorflow and JAX)

Models that are available via API endpoints, particularly as hosted by OpenAI. Given that OpenAI was a first mover in the product LLM space, they currently have the API advantage, and many tools that have developed have developed OpenAI-compatible endpoints which don’t always mean using OpenAI, but conform to the same set of patterns that the Chat Completions API v1/chat/completions offers. For example, adding OpenAI-style interop chat completions allowed us to stand up our own vLLM OpenAI-compatible server that works against models we’ve started with on HuggingFace and fine-tuned locally.

If you want to be successful in this space today and you’d like to cater to a broad audience, you as a library or service provider have to be able to interface with both of these.

I still believe this is true and it will be interesting to see how this develops over the course of the next year as APIs and their model ecosystems become even broader.

These models will integrate with, not replace traditional machine learning systems. In the beginning, everyone thought that we could replace whole classes of engineering and machine learning problems with LLMs. It’s becoming increasingly clearer that one big-ass model is not as a good as many smaller models for specific tasks performed in industry, which makes complete sense because LLMs as a concept rose out of academic labs whose goals are to generalize on out-of-sample problems, whereas the task of industry is to specialize in healthcare or OCR document scanning for legal, or sentiment analysis for social media. Even if fine-tuning did decline as a practice this year, people are making models specialized in a lot of deifferent ways, which I think the rise of agents at the end of this year really makes clear.

An area that has particularly interested me is the integration of LLMs into recommender systems, my favorite area of applied machine learning. Earlier last year, I, along with James and Ravi, started a position paper on what will change in recommender systems in the light of LLMs based on a joke tweet I had made,

personalized recommendations based on implicit matrix factorization from data acquired through large log streaming architectures were a low interest rate phenomenon

My hypothesis was that companies used to collect a lot of streaming user data, particularly in social and other B2C settings, that were then used for personalization. Now, all the data collection happens publicly at the level of the interet, where it is re-compressed into LLMs. What does this mean for personalization, now?

Unfortunately we didn’t finish the post (maybe someday?), but our main hypothesis was that traditional recsys will not be replaced with, but instead augmented with LLMs, that LLMs could help fill the cold-start gap because they’re so good at zero-shot retrieval and general topic classification and can serve as places to augment cold-start problems and in onboarding, and that LLMs could offer more explainability in recommendations. This has so far proven to be true.

Italy + Learning Italian

I had the extreme pleasure of being invited to give a keynote at PyCon Italia in Florence in May. I took all these learnings that had been running on crazed hamster wheels in my brain and threw them into a talk based loosely on the Decameron.

In addition to subjecting my poor audience to jokes about the Medicis, I also gave the intro in Italian to an audience of +100, which was extremely nervewracking. I’ve been wanting to study Italian since I was 18 and the stars aligned two years ago for me to seriously start. and I’ve been learning ever singe. This year, I also managed to read my first (Level A1/A2) books in Italian.

My family also came to Italy. It was the kids’ first trip to Europe. Everyone had an amazing time (at least after my talk was over), ate a lot of gelato, and experienced Rome and Florence in the spring. One of my favorite moments of the trip (and of my year) was listening to a Ricchi e Poveri cover band playing at a disco outside our hotel at 11:30 at night in Florence.

Inspired by all of this, and to take a break from LLMs, I also started learning how to mix music, and hopefully will have something more concrete to write/share about DJing this year.

I also managed to read some books.

As I spent the year trying my best to keep up with the wild, unruly, and growing LLM landscape, another part of my ecosystem was wilting. It’s clear for all intents and purposes that Twitter as we knew her is dead. It doesn’t have the juice anymore, as the kids say. I’m surprised that it died from human intervention rather than engineering failure, but as anyone who’s read my newsletter knows, this is by far the most likely outcome for techno-social systems. I have been wanting to write a long eulogy on my personal experinece and what Twitter meant to me: it was a place where I made friends, learned to be a serious programmer, worked out my best ideas, got leads for jobs, started a newsletter, organized Normconf, and most importantly, had fun. But in the end, it all died with a whimper, and I, like many others, just quietly left.

I’ve always been a firm believer in owning your own internet space, and I continued to do that by blogging, and moving from Substack to Buttondown, but I also need micro social interaction because I get energy and ideas from it. So, I engaged more on Bluesky, where I’ve been since last year, and was pleasantly surprised to see the machine learning community start to migrate there. Amost immediately as soon as I came back to the platform, I had a conversation that led to an idea for a post.

A lot of people have (rightfully so) sworn off agglomerated social in favor of group chats. This makes complete sense to me, yet as a creator and someone who thrives on community, I was sorely missing this space until I came back to Bluesky. There is a lot of conversation around protocol versus platform and what it means for Bluesky the app versus Bluesky the protocol to succeed.

In particular, I encourage anyone interested in the space to read this series of exchanges on the question but I remain hopeful that that space grows and remains a place where users can choose and experiment. It allowed me, at the end of the year, to hack on gitfeed, which is very, very very cool.

2025

What next? Who knows, who can even predict the future? Not machine learning models, and certainly not me. I’m just excited to keep learning and building and making and dreaming, and, hopefully writing more about it all.

Write code with your Alphabet Radio on

Mon, 16 Dec 2024 00:00:00 +0000

There is a lot of debate in the software community around whether LLMs can replace developers. Part of the reason is the way we formulate the problem of what it means to write software. In industry, we still give outsize cultural deference to software developers as lone wizards who come into the room, put on their hoodie, crank up the techno, and write the application on their own.

When we sit down at our keyboard, we assume that we sit down alone, with our precious treasure trove of programming knowledge that we have built up over the years, hoarding the nuggets about byte size and data types like a dragon guarding its gold. It is us and our hard-earned brain patterns from years of algorithm classes and hundreds of pages spent trawling StackOverflow versus the machine, working alone, against each other.

We sit down, we crack our knuckles and deserialize the knowledge from our brain into clear streams of beautiful code, in solitude and blessed silence. These days, we are spending less time with books and search results and more time with ChatGPT or Claude, and it’s this prompting process, we assume, that will ultimately replace us. But that’s only true if we consider the developer as an individual unit of work separate from anything else, without our own context window.

Machine learning is compression and LLMs doubly so. When we sit with an LLM, we are not only asking the average of the internet but the average, as Karpathy puts it, of every human data labeler that has contributed to moving LLMs forward.

When we ask LLMs to write code, we get back compressed representations of the entire field, optimized via RLHF, sorted by descending probability, and limited by the size of the model’s context window. So we are not alone, but we are also often don’t ascend outside a local minimum of average code, sometimes not even outside the local minima of suckiness. Not to mention, these models have no access to hundreds and thousands and millions of lines of elegant code that has never seen the public internet because it sits quietly, undiscoverable in corporate git servers. When we ask the model, almost any model (with the exception of finetunes on our own data), we are peeking into a very small window of what good code could be.

When I sit down at the keyboard, I also sit down with the average of the local minima, but I also have a secret superpower. I’ve had the enormous good fortune to work with and learn from very good developers, and it’s the average habits of all of these developers that lift me out of the depths of compressed scraped GitHub.

These developers’ good habits stream from my memories into my consciousness. As I write code, they enrich my decision-making process, like a radio station of good advice that never turns off.

A insists that I need to learn fundamental algorithms and data structures inside and out because they are the basis for what makes good code fast. “Even if you never implement a LinkedList in your own code, you’ll be able to more clearly reason through the decisions others have made and use the libraries that make the most efficient use of them.”

B tells me, through hundreds of PR comments over years and years, that it’s not good enough to have a PR that just passes tests. The code needs to be clean, reasonable, and legible for others because we read code more than we write it.

C tells me to write code that is elegant. C says it will take longer to write this kind of code than something that just ships, but that it is our professional responsibility as developers to carve out that time, to demand it. “Write your code, then write a second time. Make it work, make it right, make it fast.”

D tells me not to mess with fancy tools. A simple print debug statement goes a long way. A 2-line unit test fixes 100 lines of obtuse code. Code completion editors don’t work when you don’t know what you want to autocomplete, yet.

E tells me to master my tools, to dig into problems I don’t understand, to get to a reproducible example, to play with things in the terminal, take them apart, write smaller and smaller pieces of code until they make sense.

F tells me to take time to analyze the data correctly, and to be crisp and concrete about my analysis.

G and H tell me to start without machine learning if I can, and if I have to use it, to use a simple model first of all.

I tells me to get to a local development loop as soon as possible.

J tells me to take my craft seriously, but never myself.

K tells me to be patient and have grace, both for others, and for myself. Code is hard, but humans are harder.

Above all of them is Grug, constantly whispering “complexity bad” with every line I add.

Over and over my Alphabet Radio plays my greatest hits, fixing a variable name, digging through a class hierarchy, adding a test, using dependency injection, avoiding unnecessary libraries, deleting lines of code until the number of lines I add is less than the number I subtract, refactoring, refactoring, refactoring.

I am so lucky to have my Alphabet Radio. Engineers like me who come from data land sometimes don’t get the privilege of building a playlist like this because we don’t work in teams, and for a long time at the beginning of my career, I was one of these people.

It was very lonely and I felt wrong a lot of the time without guardrails or intuition as to why what I was doing was inefficient and incorrect. LLMs don’t and can’t help with this part. Repeated exposure to the best practices of good developers does.

When I am at the keyboard, my joyous, cacophonous radio station keeps streaming the greatest hits on an infinite loop and allows me to rise above the local minimum, encouraging me to work harder to produce code that is not just functional, but easy for others to read, fast to modify, and doesn’t break for longer.

Nothing is black and white. Code is not precious, nor the be-all end-all. The end goal is a functioning product. All code is eventually thrown away. LLMs help with some tasks, if you already know what you want to do and give you shortcuts. But they can’t help with this part. They can’t turn on the radio. We have to build our own context window and make our own playlist.

When LLMs can stream advice as clearly and well as my Alphabet Radio, then, I’ll worry. Until then, I build with my radio on.

Why are we using LLMs as calculators?

Sat, 09 Nov 2024 00:00:00 +0000

We keep trying to get LLMs to do math. We want them to count the number of “rs” in strawberry, to perform algebraic reasoning, do multiplication, and to solve math theorems.

A recent experiment particularly piqued my interest. Researchers used OpenAI’s new 4o model to solve multiplication problems by using the prompt:

Calculate the product of x and y. Please provide the final answer in the format: 
Final Answer: [result]

These models are generally trained for natural language tasks, particularly text completions and chat.

So why are we trying to get these enormous models, good for natural text completion tasks like summarization, translation, and writing poems, to multiply three-digit numbers and, what’s more, attempt to return the results as a number?

Two reasons:

Humans always try to use any new software/hardware we invent to do calculation
We don’t actually want them to do math for the sake of replacing calculators, we want to understand if they can reason their way to AGI.

Computers and counting in history

In the history of human relationships with computers, we’ve always wanted to count large groups of things because we’re terrible at it. Initially we used our hands - or others’ - in the Roman empire, administrators known as calculatores and slaves known as calculones performed household accounting manually.

Then, we started inventing calculation lookup tables. After the French Revolution, the French Republican government switched to the metric system in order to collect property taxes. In order to perform these calculations, it hired human computers to do the conversions by creating large tables of logarithms for decimal division of angles, Tables du Cadastre. This system was never completed and eventually scrapped, but it inspired Charles Babbage to do his work on machiens for calculation along with Ada Lovelace, which in turn kicked off the modern era of computing.

UNIVAC, one of the first modern computers, was used by the Census Bureau in population counting.

The nascent field of artificial intelligence developed jointly in line with the expectation that machines should be able to replace humans in computation through historical developments like the Turing Test and Turing’s chess program, the Dartmouth Artificial Intelligence Conference and Arthur Samuel’s checkers demo.

Humans have been inventing machines to mostly do math for milennia, and it’s only recently that computing tasks have moved up the stack from calculations to higher human endeavors like writing, searching for information, and shitposting. So naturally, we want to use LLMs to do the thing we’ve been doing with computers and software all these years.

Making computers think

Second, we want to understand if LLMs can “think.” There is no one definition of what “thinking” means, but for these models in particular, we are interested to see if they can work through a chain of steps to come to an answer about logical things that are easy for humans, as an example:

all whales are mammals, all mammals have kidneys; therefore, all whales have kidneys

One way humans reason is through performing different kinds of math: arithmetic, solving proofs, and reasoning through symbolic logic. The underlying question in artificial intelligence is whether machines can reason outside of the original task we gave them. For large language models, the ask is whether they can move from summarizing first a book if they were trained for books, to a movie script plot, to finally, summarizing what you did all day if you pass it a bunch of documents about your activity. So, it stands to reason that if LLMs can “solve” math problems, they can achieve AGI.

There are approximately seven hundred million benchmarks to see if LLMs can reason. Here’s an example, and here’s another one. Even since I started this draft yesterday, a new one came out.

Since it’s hard to define what “reasoning” or “thinking” means, the benchmarks try to proxy to see if models can answer the same questions we give to humans in settings such as university tests and compare the answers between human annotators generating ground truth and inference run on the model.

These types of tasks make up a large number of LLM benchmarks that are popular on LLM leaderboards.

How calculators work

However, evaluating how good LLMs are at calculation doesn’t take into account a critical component: the way that calculators arrive at their answer is radically different from how these models work. A calculator records the button you pressed and converts it to a binary representation of those digits. Then, it stores those number in memory registers until you press an operation key. For basic hardware calculators, the machine has built-in operations that perform variations of addition on the binary representation of the number stored in-memory:

 + addition is addition, 
 + subtraction is performed via two's complement operations, 
 + multiplication is just addition, and 
 + division is subtraction

In software calculators, the software takes user keyboard input, generates a scan code for that key press, encodes the signal, converts it to character data, and uses an encoding standard to convert the key press to a binary representation. That binary representation is sent to the application level, which now starts to work with the variable in the programming language the calculator uses, and performs operations on those variables based on internally-defined methods for addition, subtraction, multiplication, and division.

Software calculators can grow to be fairly complicated with the addition of graphing operations and calculus, but usually have a standard collected set of methods to follow to perform the actual calcuation. As a fun aside, here’s a great piece on what it was like to build a calculator app Back In The Day.

The hardest part of the calculator is writing the logic for representing numbers correctly and creating manual classes of operations that cover all of math’s weird corner cases.

However, to get an LLM to add “2+2”, we have a much more complex level of operations. Instead of a binary calculation machine that uses small, simple math business logic to derive an answer based on addition, we create an enormous model of the entire universe of human public thought and try to reason our way into the correct mathematical answer based on how many times the model has “seen” or been exposed to the text “2+2” in written form.

We first train a large language model to answer questions.

Source

This includes:

Gathering and deduplicating an enormous amount of large-scale, clean internet text
We then train the model by feeding it the data and asking it, at a very simplified level, to predict the next word in a given sentence. We then compare that prediction to the baseline sentence and adjust a loss function. An attention mechanism helps guide the prediction by keeping a context map of all the words of our vocabulary (our large-scale clean internet text.)
Once the model is trained initially to perform the task of text completion, we perform instruction fine-tuning, to more closely align the model with the task of performing a summarization task or following instructions.
The model is aligned with human preferences with RLHF. This process involves collecting a set of questions with human responses, and having human annotators rank the response of the model, and then feeding those ranks back into the model for tuning.
Finally, we stand up that artifact (or have it accessable as a service.) The artifact is a file or a collection of files that contain the model architecture and weights and biases of the model generated from steps 2 and 3.

Then, when we’re ready to query our model. This is the step that most people take to get an answer from an LLM when they hit a service or run a local model, equivalent to opening up the calculator app.

We write “What’s 2 + 2” into the text box.
This natural-language query is tokenized. Tokenization is the process of first converting our query into a string of words that the model uses as the first step in performing numerical lookups.
That text is then embedded in the context of the model’s vocabulary by converting each word to an embedding and then creating an embedding vector of the input query.
We then passing the vector to the model’s encoder, which stores the relative position of embeddings to each other in the model’s vocabulary
Passing those results to the attention mechanism for lookup, which compares the similarity using various metrics of each token and position with every other token in the reference text (the model). This happens many times in multi-head attention architectures.
Getting results back from the decoder. A set of tokens and the probability of those tokens is returned from the decoder. We need to generate the first token that all the other tokens are conditioned upon. However, afterwards, returning probablities takes many forms: namely search strategies like greedy search and and sampling, most frequently top-k sampling, the method originally used by GPT-2. Depending on which strategy you pick and what tradeoffs you’d like to make, you will get slightly different answers of resulting tokens selected from the model’s vocabulary.

Finally, even after this part, to ensure that what the model outputs is an actual number, we could do a number of different guided generation strategies to ensure we get ints or longs as output from multiplication, addition, etc.

So this entire process, in order to add “what is 2+2”, we do a non-deterministic a lookup from an enormous hashtable that contains the sum of public human knowledge we’ve seen fit to collect for our dataset, then we squeeze it through the tiny, nondeterministic funnels of decoding strategies and guided generation to get to an answer from a sampled probability distribution.

These steps include a large amount of actual humans in the loop guiding the model throughout its various stages.

And, all of this, only to get an answer that’s right only some percent of the time, not consistent across all model architectures and platforms and in many cases has to be coaxed out of the model using techniques like chain of thought.

As an example, here’s an aswer I’ve tried on OpenAI, Claude, Gemini, and locally using Mistral via llamafile and ollama:

Claude Sonnet 3.5

Gemini 1.5 Flash

OpenAI ChatGPT GPT-4 Turbo

Llamafile Mistral 7-B Instruct 2

Ollama Mistral

If you ask any given calculator what 2+2 is, you’ll always get 4. This doesn’t work with LLMs, even when it’s variations of the same model, much less different models hosted across different service providers and in different levels of quantization, different sampling strategies, mix of input data, and more.

Why are we even doing this?

From a user perspective, this is absolutely a disastrous violation of Jakob’s Law of UX, which states that people expect the same kind of output from the same kind of interface.

However, when you realize that the goal is, as Terrence Tao notes, to get models to solve mathematical theorems, it makes more sense, although all these models are still very far from actual reasoning.

I’d love to see us spend time more understanding and working on the practical uses he discusses: drafts of documents, as ways to check understanding of a codebase, and of course, generating boilerplate Pydantic models for me personally.

But, this is the core tradeoff between practicality and research: do we spend time on Pydantic now because it’s what’s useful to us at the moment, or do we try to get the model to write the code itself to the point where we don’t even need Pydantic, or Python, or programming languages, and can write natural language code, backed by mathematical reasoning?

If we didn’t spend time on the second, we never would have gotten even to GPT-2, but the question is, how much further can we get? I’m not sure, but I personally am still not using LLMs for tasks that can’t be verified or for reasoning, or for counting Rs.

Dead Internet Souls

Thu, 19 Sep 2024 00:00:00 +0000

In the 1800s, before serfdom was abolished in the Russian empire, landowners paid taxes based on how many serfs they had. A census was conducted every few years by government employees traveling across the empire and doing counts; a manual map-reduce of epic proportions. If a person was dead, it would often be years before the government cleared the cache, so to speak, and landowners continued to pay taxes on these dead souls.

Alexandr Pushkin, the greatest living Russian-language author at the time, heard a story about how landowners took advantage of this by buying up dead souls from landowners, and passed this story onto fellow writer, Nikolai Gogol as an idea for a book or play, which resulted in Gogol’s seminal satirical work, “Dead Souls.” Gogol meant dead souls on two levels: both the serfs, and the banality and falsity of Russian landowning society at the time.

The internet today is filled with dead souls. Or, more accurately, souls who were never alive in any sense of the word: text copypasta from LLMS, slop artwork generated from generative art tools, and bots on aging social networks that are quickly emptying of real content as real people migrate to group chats.

It used to be different. In the beginning, the internet was made of people. I came online in the mid 1990s, along with the rest of America, and for the first few years of my internet experience, my primary threat model was someone on AOL finding out that I was 12 (A/S/L , and also where I live. After all, no one on the internet knew if I was a dog.)

Based on my priors, I thought that this would be the world my kids would grow up into as well, and as they got older, I started preparing them against the dangers of strangers on the internet.

But the explosion of generative AI over the past two years has brought to light something even worse: the problems we encounter on the internet today are not because of people, but because of the lack of them, and now the danger is not that you will find people who want to do you harm, but that the amount of slop will overwhelm us all: Not 1984, but Brave New World.

It continues to get worse, and will get much worse before it gets better.

But the good news is that, when there is a lot of noise, it is even easier for people who have original thoughts to stand out. As Justine writes in this important and thought-provoking piece,

In a world of infinite automation and infinite surveillance, survival is going to depend on being the least boring person.

If you are a human out there in the vast wilds of the new internet of dead souls, speak up. We are still out there, on the small web, being ingested into indie search engine indexes, on blogs, on websites.

The machines are thinking, but so are people, and they’re still much better. I’m still out here. Come be out here with me, too.

Don't worry about LLMs

Sat, 25 May 2024 00:00:00 +0000

This is a near-transcript of the talk I gave at PyCon Italia 2024 in May in Florence.

Introduction

Buongiorno PyconIt, grazie per avermi invitata a parlare! Avrei voluta fare tutto il discorso in italiano, ma lo sto ancora imparando.

Per adesso posso parlare soltanto di gelato o colori. Perché non so ancora dire, “don’t worry about LLMs”, il resto sarà in inglese.

I’m Vicki and I work as a machine learning engineer at Mozilla.ai, building a platform to enable developers to evaluate and select between different LLMs. Before working on LLMs, I’ve built large-scale ML systems in security, social recommenders, and tv content recommendations

After working with LLMs for the past year, what I’ve found is that the new engineering systems we’re building around these LLMs are a lot like the old ones. Once we cut away the hype, what we’re usually left with are plain engineering and machine learning problems.

But how do we as practitioners cut away the hype? Since we are in Firenze, I want to tell a story I recently heard while talking to other tech folks that took place here that might help us to navigate this. Around town, there is a company called Medici Corp.

They are a very large organization, spread out into lots of different industries in banking, pharma, fine arts, and philanthropy. They were very successful, but recently, their CEO was worried about being left behind in AI.

And she wanted to see if there was some way they could incorporate a chatbot or similar into their offerings.

The CEO created a small R&D task force of developers and machine learning engineers and tasked them with investigating what it would take to add AI to their product over the course of a sprint.

Now, the developers, machine learning engineers, and PMs, were all very experienced in industry, but new to LLMs, and when they started looking at all the different model choices, platforms, and the buzz around LLMs, they became very distraught.

There were too many open-ended options , a lot of people who were loud online. So, the developers decided to rent an Airbnb outside the city for a week so they could really focus, isolate, and ship some code. When they got together around a whiteboard, they frantically started researching what tools to build an LLM with, and what they saw was this chart.

“What do we do, ” they asked. How were they going to compete in this insanely crowded market? More importantly, how could they as engineers even understand this landscape?

As they worried their team lead stepped forward and said, “In times like these, I turn to the wisdom of the foundational thinkers. There is one who said,

The developers turned to the team lead and said, “This man says the truth, but how can we possibly turn this into actionable advice that we can write features for with feature flags and unit tests in a two-week sprint? And deliver a product that has a magic emoji on it?” In other words, “How do we deliver AI?”

The team lead said, “Here’s what we’re going to do. We’re going to get in small groups and research what other people have done in the situation, and then we’re going to present what other people did.”

The engineers groaned, because group work is the worst. But the team lead said, “Reading about what people in the past have done is the only way to build and keep our context window, which is our knowledge base of classical architecture patterns and historical engineering context that allows us to make good, grounded engineering decisions.”

The Singular Machine Learning Task

So the engineers agreed and went off to do research. All day the engineers researched and read, and the next day, they all gathered in a group. One stepped forward, and began to tell her story.

She said, once upon a time in Barcelona, they were building a grand cathedral, called Santa Maria Del Mar. At that time, many Gothic cathedrals were built in periods that took a long amount of time, usually fifty to one hundred years. Santa Maria del Mar was finished in only fifty five, even notwithstanding a fire and the plague that started the neighborhood where it was being built, and is the only church surviving in the Catalan Gothic style.

It’s unique because it was one of the only churches built with backing from commoners as opposed to the nobility of the city of Barcelona. The rich would pay with their money, and the poor would pay with their labor, for the neighborhood to build the cathedral together.

A key force on the project were the bastaixos, or porters. They had an organized guild, and these men were traditionally in charge of loading and unloading ships. They were already extremely good at one thing: carrying heavy things. When they heard about the cathedral, the guild volunteered to take the stones from the royal quarry at Montjuïc, at a high elevation above the city, to the cathedral.

Right now, to get from here to the Cathedral, it takes about 50 minutes to walk, downward. In those days, the bastaixos would have walked past the Port of Barcelona to get there, taking longer, over an hour, carrying a stone that weighs over 40kg on his back alone.

So a bastaix would first put the stone on his back, and then move it all the way to the cathedral. Then, he would go back and move another stone. They did this all day, day-in and day out. The stones weighed 40 kg and the only protection they had was the turban on their head, called a capcana that rolled up above the neck.

The machine learning engineer paused and said, you know, now that I’m telling this story, it reminds me of something, and that something is gradient descent.

Gradient descent is a key algorithm that many machine learning models, including neural networks in the transformer family that powers GPT-style models, use for training the model.

Gradient descent minimizes the loss function by iteratively adjusting the model’s parameters (weights and biases). The process involves calculating the gradient of the loss function with respect to the model parameters and then updating these parameters in the direction that reduces the loss.

For example, let’s say that we produce artisanal hazelnut spread for statisticians, Nulltella. We want to predict how many jars of Nulltella we’ll produce on any given day. Let’s say we have some data available to us, and that is, how many hours of sunshine we have per day, and how many jars of Nulltella we’ve been able to produce every day.

It turns out that we feel more inspired to produce hazelnut spread when it’s sunny out. We can clearly see this relationship between input and output in our data (we do not produce Nulltella Friday-Sunday because we prefer to spend those days talking about Python.)

Now that we have our data, we need to write our prediction algorithm, where we know, based on our current values, what future values could potentially be. The equation to predict output Y from inputs X for linear regression is outlined here.

Our task is to continuously adjust our weights and biases for all of our features to optimally solve this equation for the difference between our actual as presented by our data and a prediction based on the algorithm to find the smallest sum of squared differences, between each point and the line.

In other words, we’d like to minimize epsilon, because it will mean that, at each point, our predicted Y is as close to our actual Y as we can get, given the other points.

The equation we use for this is RMSE (Root mean squared error).

Let’s say we initialize the function with some x values and weights. How do we optimize it? Using gradient descent. We start with either zeros or randomly-initialized values for the weights and continue recalculating both the weights and error term until we come to an optimal stopping point. We’ll know we’re succeeding because our loss, as calculated by RMSE should incrementally decrease in every training iteration.

For our particular model, we can see that the loss curve reaches a local minimum after ten iterations. If you look at the curve for the elevation the bastaixos walked from Montjuic to Santa Maria del Mar, you’ll see that it follows the same pattern.

That’s because these two things lay out a fundamental pattern: working on optimizing one thing at a time.

The bastaixos had a single goal: the completion of the cathedral, and the single functionality of carrying stones to that goal. They didn’t work on carving the stone, nor on the stained glass, nor on architecture. They just moved rocks. Everything else was peripheral, and the focus allowed them to get just one great thing done. The engineers realized that they also needed to learn to be bastaixos and focus on a single machine learning task.

But what they realized was that LLMs in and of themselves, while also operating on the principle of gradient descent, are set up to solve an unbounded number of open-ended tasks: they can write poems, complete recipes, classify text, summarize text, evaluate other models, act as chatbots, and much, much more.

When the engineers went back to the drawing board, it was clear what they needed: the focus on a single use-case for their customers. What’s the best way to figure out what you need LLMs for? List all the problems your business is facing and see if machine learning will be the right way to address them.

Machine learning generally, and LLMs more specifically, are good for where the number of heuristics you develop starts to outweigh their maintenance cost. I’ve heard this also called the 1000-intern heuristic: if it’s a task that can be simplified by a thousand people entirely new to the task doing it, then do that.

The simplest tasks for large language models to do are summarization, classification, translation, named entity recognition, and similar. If your problems fall in that space, you’ll have an easier time than with open-ended tasks like reasoning. This is also the reason why, often, simpler models perform better for specific tasks than general LLMs that are meant to generalize to out-of-distribution tasks.

When the team thought about all the numerous things their company did, what they realized is that one of their biggest problems was trending topic detection: they have a lot of documents constantly floating around, particularly around the patronages they’re performing, and they perform a lot of art patronage.

They wanted to be able to get a sense of the types of art and types of artists in the hands of the corporation at any given time so they could further explore those trends and make sure their art funding was even.

Armed with this information, The Medici team decided that they wanted to use it internally to look at all the documents they had related to the large trove of artwork they had acquired. It would be easier to do internal topic detection than other outward-facing use-cases because they had subject-matter experts, and the categories of art were fairly established since they had in-house art experts. They presented this plan to the CEO, who was pleased. Now, they were at their next task, but they were still overwhelmed.

The Measurable Goal

On the second day, another developer had a story. She said, I’ve been reading about medieval monks.

Jamie Keiner, at the Department of Classics in the University of Georgia, wrote a book called “The Wandering Mind”, about how medieval monks used to harness attention to make their prayers more effective. Every day, their goal was to get higher on the spiritual plane away from earthly matters. But they kept getting distracted by the daily minutiae of life - legal disputes, farming and livestock, gossip, banality of everyday life that overloads your attention.

From the writings of one monk, John Cassian, a Christian monk in the Roman empire, for example, who traveled to Egypt, it turns out that distractions are not new: In the 420s, he wrote that “the mind gets pushed around by random distractions.” Like him, many monks and nuns saw distraction as a “primordial struggle”, and something they felt obligated to continually fight

What monks and nuns recognized is that, if they wanted to get closer to true understanding, they would have to separate from distractions that bound them to society, because it was impossible to focus on one thing at a time, and they only wanted to focus on prayer. As a monk called Hildemar of Civate put, “it is impossible to focus on two things at once - “in uno homine duas intentiones esse non posse.”

Much later, scientists confirmed that human brains have very poor capabilities for multitasking, as well. The most items a human can hold in memory is about 7, plus or minus two, and we definitely cannot do two things simultaneously. After 7 items our cache is cleared and we start getting confused and lost.

That means when we as engineers try to build large systems with multiple components, with large amounts of decisions to be made, the brain will shut down. This is the case in the LLM land today. The average LLM system might look something like this.

Each of these components is a million decisions to be made. Here is a list I can tick off just by looking at this diagram.

Should we use our own model or an open-source base-model or an API vendor? Which API vendor?
How should we go about testing our prompts? How should we constrain the outputs of the model? + Do we want to generate plain text, or JSON, or binary responses?
What kind of UI do we need? Is text-based chat enough?
Where do we store the model artifact - do we store it locally
Are we using cloud vendors, are we using their implementations of cloud artifacts?
Are we doing model fine-tuning, do we need separate data-collection mechanisms?
How do we evaluate this model?

The monks told us that, even though we will have a lot of distractions and environmental variables, we need to focus on developing one good piece of software at a time as we build.

The developer telling the story said this reminds me of another established traditional development pattern: the philosophy of Unix, which says, tools that we build should be minimal and modular and do one thing at a time so we can combine them together later.

It’s true that when we build machine learning systems, we have no choice: we need vector databases, and APIs, and data stores, and, most importantly, models. But, when we build an entire machine learning system at once, for example, connecting five different APIs rather than one, or trying five models rather than one, five use-cases, we see a deterioration at each stage of the process rather than focusing on streamlining a single use-case and task end-to-end.

How do we get to programs that do one thing well in machine learning, i.e. perform our task of topic detection? The best way is to understand first that machine learning, unlike developing product features like “the ability to click a button” or “sending data to the database”, is a cyclical process that involves a lot of iteration before we get something working we’re happy with, and the way to get to happiness is to pick an evaluatable baseline.

Machine learning-based systems are typically also services in the backend of web applications. They are integrated into production workflows. But, they process data much differently. In these systems, we don’t start with business logic. We start with input data that we use to build a model that will suggest the business logic for us.

This requires thinking about application development slightly differently - we need to be able to loop through a machine learning inference cycle quickly and examine the results: are they right? If yes we keep the model, if no we go back and change one thing and move on.

This means there’s a lot of trial and error. Not many people realize it, but machine learning is more like alchemy than even software engineering. The top people in the field even said so, as Ali Rahmi did in his NeurIPS keynote in 2017.

This is doubly true for LLMs, where our main medium of input and output is freeform text. We have some model that we add text requirements to, and hope to get back logical text, or instructions, or code out. Because the inputs of natural language are so varied and so are the outputs, the process becomes three times harder to control and evaluate.

What this means for focus is that the developer, like the monks, has to create a center of focus, and that center of focus is the first use-case for evaluation for your model. That means, if you’re a bank, you can’t evaluate a model on how well it writes poetry. It needs to be on how well a model creates top-level topics for all of your documents.

This means creating both what we call offline and online evaluation metrics. Online evaluation metrics are those that can be assessed by people using your product or platform, or simply doing what we call a vibe check, and offline evaluation is more scientific and offers us a grounded reference against academic benchmarks to see if our model generalizes well. In order to have a well-functioning machine

So the developers decided to look at two metrics:

The vibe check
The offline eval

The vibe check is as simple as: creating a list of documents and manually labeling topics for these documents, for a number of different documents. Here’s an example from the wikipedia page for Michelangelo.

We can see that we added some human topics based on simply scanning the text and our knowledge of categories, as experts in art.

The comparison is now to run it against our LLM and see what topics it generates. We use llamafile locally, with a local quantized model for quick iteration between prompts and responses without having to use external models.

We can now compare the results we generated ourselves and what the model generates.

We can see the results are ok, but maybe not as low-level as we’d like? For example, “Renaissance” doesn’t help us at all since all of our artists are from the Renaissance. And literature is not a category in art. So our next step would be to modify the prompt.

We can do this kind of prompt tuning manually for a bit, or use automatic prompt-tuning libraries, but then we might find that our model itself doesn’t have enough information about art, in which case we might want to fine-tune it with samples of text specifically related to Renaissance art, and try again.

Once we go through this manual cycle, we can also perform offline metric evaluation, which allows us to more systematically evaluate the model based on agreed-up academic benchmarks. Topic alignment is an imprecise science because it’s based on what humans think are good topics for categories, but we can use metrics like cosine similarity to look at how well given topic pairs match.

Once the developers did this and tuned the model, the CEO was once again happy. But, now the team had a different problem, the curse of success. They needed to deploy this model to production.

Now, the team had a use-case for trending topic detection, and they had an evaluation metric: manual vibes and rescaled dot product for similarity between human-assigned and machine-assigned topics.

They had problems, though. The systems they were building were big and complicated. They now involved the model, and something horrible called LLM ops that now involved updating, monitoring the system, storing prompts, monitoring model output, monitoring latency, security, package updates. They had built a product with an LLM producing summarization inputs, but now, whenever something broke, they didn’t know where to look.

The Reproducible Example

So, on the final day, the final developer shared a story. He said, “This story is about Ellen Ullman. She was a software engineer who worked on complex systems starting in the late 1970s including at startups and large companies, wrote a number of essays about the art of practicing computer science. Her latest book is called “Life in Code”, and in it, she describes the mind of a programmer as they are writing a program. And this ties together everything we’ve been talking about so far.

She writes that keeping track of translation between human and code logic is hard. “When you are writing code, your mind is full of details, millions of bits of knowledge. This knowledge is in human form, which is to say rather chaotic. For example, try to think of everything you know about something as simple as an invoice. Now try to tell an alien how to prepare one. That is programming… To program is to translate between the chaos of human life and the line-by-line world of computer language.”

She says that, in order to do this, a developer must not “lose attention. As the human world-knowledge tumbles about in your mind, you must keep typing, typing. You must not be interrupted.”

In this kind of environment, you are information-constrained, but focused. It is harder to get through a mistake, but easier to find it and test possible solutions because the feedback cycle is fairly short. The reason this was the case was because, in earlier times, developers worked closer to the machine, closer to the metal, closer to the source of the computation, and even though there were interruptions and blockers, they were different.

Most programs were written and compiled locally in languages like C. There were much fewer third-party libraries. Implementations were written from scratch, and the main source of information were books, man pages, and Usenet. The approximate user experience would be similar to writing python by only being able to access python.org.

Ullman describes this phenomenon in another part of the book where she describes working with a developer named Frank. Frank previously had worked as a hardcore technical contributor, but when he was working with Ullman, he had moved to financial reports, and he personally was miserable, and he also hated Ullman because she was “close to the metal.”

These days, it is extremely hard to be close to the metal, because when we work with distributed systems, and machine learning, and the cloud, each of these have been built on top of the levels of turtles that previous developers have built, and it is easier to get distracted. When you throw non-deterministic LLMs and the distributed systems used to train and serve them into the mix, you come up with a special kind of hell that makes it impossible to have a good developer experience.

What we can do is what people have always done: create reprex. A reprex is a reproducible example, an idea that comes from the R community. In R, it’s fairly easy because it’s a self-contained piece of software, but we can also strive for the same thing.

Here’s an example of a reprex in Python, the RMSE code we just reviewed in the first section. We can know it will run the same way on our reviewer’s computer as ours, we can troubleshoot and check.

def root_mean_squared_error(y_true:float, y_predicted:float) -> float:
 cost = math.sqrt(np.sum((y_true-y_predicted)**2) / len(y_true))
 return cost

Reprex comes in handy when you’re dealing with complex distributed systems. For example, here’s an actual problem the developers were dealing with while trying to serve their model: trying to troubleshooting Ray.

Ray is a Python and C++-based distributed framework for training and serving machine learning models, used by companies training LLMs. Its predecessors include Hadoop and Spark. It also takes inspiration from other distributed libraries. The cool thing about Ray is that it is very easy to run locally, and spinning up a single instance is something that comes out of the box, meaning we are already closer to the metal.

However, if you have an issue, it can take a while to get to the bottom of it because of how complex the architecture is. Ray has several orthogonal patterns working together. First, there are tasks that you can execute on a remote cluster and actors, which are stateful tasks. This comes from the actor pattern in computer science which receives messages from its environment and can send messages back.

We also have the cluster-level communication patterns, with the global control store managing transactions and if you are running this on top of Kubernetes, also the Kubernetes primitives there is also the Task execution graph, the various modules: the dashboard, ray train, and Ray serve. The amount of patterns you have to understand in order to wrap your mind around it is truly astounding.

But don’t forget that humans can only keep several things in memory when they trace through complexity!

In building their trending topic application, the team had an issue where they were looking to test a topic detection pipeline of their model using Ray serve. Ray Serve takes your local deployable code and ships it to your Ray cluster using the Ray client, which is an API that connects a Python script to a remote Ray cluster.

Ray Serve allows you the ability to serve a model with code that gets sent using bundled Ray actors known as deployments. A Deployment is served usually on top of Kubernetes, on top of Ray, within some sort of cloud cluster. It’s made up of several Ray Actors, which are stateful services run and managed by the Ray control plane. The Controller acts as the entrypoint for the deployment , tied to a proxy on the head node of a Ray cluster, and forwards it to replicas which serve a request with an instantiated model.

In order to serve a deployment, you can use the pattern of specifying a YAML-base config.

One critical piece here is the working dir, which specifies the where code to download to the cluster at runtime: This is part of how ray specifies the RuntimeEnv.

A copy of the working_dir will be downloaded to the cluster at runtime, and the current working directory of each remote Ray worker will be changed to that working_dir copy

In a production-grade deployment, it’s recommended that the working_dir comes from a served zip executable, which is usually pinned to a hash in GitHub. So the team had their config file set up like this:

However, there are several points of failure in the YAML file, which looks to simplify but in fact hides complexity from us in the ways config is read and implemented in the code itself. But this model didn’t launch or run. Why? If you look at all the failure points, there are at least three or four that could trip you up. First, the way Ray handles import paths could be different behavior than the way we usually assume uvicorn routes them. Then, there is the question of the runtime environment: what happens when our deployable asset is hosted on github. Then, we have the issue of Python dependencies: when you’re working with fast-moving libraries like transformers and torch, it’s always guaranteed that you’ll have conflicts, sometimes even if you pin them to specific versions. Finally, there is the Ray deployment logic itself: what happens when we specify CPU and GPU options, and how do those work with our Kubernetes cluster?

Whew. Time to start debugging.

The way to start here is to start reading logs and stack traces. When you have a very large, distributed system, here’s a sample of the stacktrace you might get, and it can be very discouraging because it doesn’t point you to where the issue is.

There’s a lot going on here, and it seems to be coming from importlib rather than Ray itself, but Ray uses importlib as a dependency. So, in order to isolate it, the team went into the module in Ray that calls importlib, which lives in python/ray/_private/utils.py .

What the team then did was to find the method that opens the import path, and created a reprex for the issue, where they took that exact method calling importlib.import_module, put it into a notebook, and called the code directly.

This directly didn’t work, and they proved it without using any external dependencies or Ray itself - they found the source of the issue. Which meant that, in their case, they couldn’t use the nested path to build their model deployment, but for simplicity’s sake, they could serve it from the root level. It turns out that the issue was in the import_attr, which meant that they had to include the config file and the script to launch the cluster in the same root-level of the repository.

They weren’t done yet, though. When they tried to deploy this, they also got a 404 error when they tried to download the deployable from the Git hash. This is because they were trying to enable asset downloads, aka our Topic trend detection code, from a private GitHub repo onto the head node in the cluster for running the code. Initially, they tried to use a fine-grained PAT: This didn’t work, so they had to use a legacy GitHUB token. Along the way, they learned that Ray uses a library called smart_open for streaming files from S3/GitHub, which helped them troubleshoot further issues.

All of this is abstracted away from us by the YAML, quickly-changing documentation, and multiple parts of the system. To get from that to the piece of information you need, the team needed to ruthlessly strip out all areas of detail and focus on the problem, and, hopefully, get to a reproducible example. This is getting closer to the metal, this is attention.

Finally, the team had a prototype running in production. With this issue resolved, the team moved on, and was successfully able to deploy the topic detection app, much to the delight of the CEO. Everyone got raises and lived happily ever after in their own castles in Florence.

Until they started to develop more products.

Conclusion

The developers celebrated. They had launched something into production, managed to cut away from the hype of the LLM space, and focused on a use-case that was important to their company.

They now came together to do a retro on what they had accomplished, and why they were able to do so. They started by picking a single use-case for generative AI in their application and evaluating the models available for that use-case. Then, they picked online and offline evaluation metrics to continue the evaluation loop. Finally, when it came down to building the system, they focused on the smallest composable parts possible instead of starting with the top, and broke down the code until they understood it.

All of this helped them concentrate, focus on what was most important and stay close to the metal.

Thanks to Davide Eynard, Guenia Izquierdo, James Kirk, and Ravi Mody for patiently reading versions of this talk.

Credits

Icons: Freepik
Bronzino, Portrait of Eleonora of Toledo, c. 1539
The Decameron, Franz Xaver Winterhalter 1837
View of Barcelona, by Anton van den Wyngaerde, 1563
Gradient Descent
Gradient Descent