Tech Blog on ✰Vicki Boykis✰

Running local models is good now

Mon, 15 Jun 2026 08:00:00 -0500

I’ve been working with local models since they came out, and finally, they’re surprisingly good now.

I have a 2022 M2 Mac with 64 GB RAM and 1TB storage and I’ve used

Mistral 7B
Gemma 3
OpenAI OSS-20B
Qwen 3 MOE, as well as a number of other Qwen variants like Qwen 2.5 Coder

across a lot of different system setups like

raw llama.cpp with Open WebUI
llama-cpp-python
Ollama
llamafiles and
LM Studio

Where are local models now?

Early on, models were slow, hard to use, and just not that accurate for most programming tasks. The idea that local models were severely lagging behind was largely true until, for me, the release of GPT-OSS. I have no concrete scientific evidence of this - my own personal vibe metric of “is a model good enough” is, “do I have to double-check it against an API model”, and GPT-OSS was the first one where I started doing that a lot less often.

As a result, I’ve mostly been using local models as fast, personalized Google for development questions that don’t require recency.

But with the most recent releases from Google in the Gemma 4, family, I’ve finally been able to do agentic coding locally and have loops work at about ~75% the accuracy/speed of frontier models, which is incredible.

I’ve so far been using gemma-4-26b-a4b LM Studio implementation as my default local model. I’ve used the local setup so far to: Refactor a Python script that was a notebook into a repo of 5-6 modules, lint that module to use correct type hints for generics (most frontier models now do this automatically, but not always).

I’ve also used it to proofread some blog posts, write unit tests, and to bootstrap a repo that stands up a two-tower model for recommendations just to see what the agent would do with a blank slate. Here’s what it generated, which was pretty basic but still beyond the scope of anything I would have thought possible last year:

Note that the environment is restricted because I run all my agentic workflows in a Docker container with limited access to execution.

I’m also building an app that surfaces trending topics from Arxiv papers. Out of curiosity, I had Pi go through my past LM Studio session logs and figure out what I was using LM Studio for:

Unsurprisingly, since I’ve been working on Rijksearch,

None of these are groundbreaking tasks (again, a lot of personalized Google/docs lookups), and working on them does give my GPUs and RAM a workout and the K-V cache grows to 64 GB RAM.

But, the larger story for me is that these kinds of tasks, even as simple as they are, used to be impossible for local models as recently as 6 months ago.

Gemma-4-12b-qat just came out but I’ve already also really been impressed with its performance relative to its size. The model architecture itself is really interesting and proposes a bunch of interesting questions like, “if we are constrained by performance and price, what architectural tradeoffs do we need to make?” a question that so far has not really been asked in the mad token gold rush.

Running agentic models locally today

But don’t take my word for any of this, try it out for yourself! You’ll need a local model inference engine, an agentic harness, and the local model artifact if you want to try to run local agentic flows. You’ll need to set up the harness to point at your local inference endpoint, the downloaded model artifact served via the inference engine.

For my local setup, I’m currently using Pi as the agent harness and LM Studio as the inference server, although it would likely be faster if I just used llama.cpp directly - a potential direction for a future experiment.

This post was very easy to follow to set up agentic coding with Pi and LM Studio, although I did make a few tweaks to the post’s setup.

Model: The post recommends Gemma 26B A4B , but gemma-4-12b-qat is more recent and smaller and faster, without much sacrifice in accuracy.
Security: I run every Pi session in a Docker container and give it permissions only to bash so that it can’t run Python code or do web browsing, although I do plan to allow curl in a different image for some research work I’m doing.
Agent Harness Config: Since I run everything in Docker, I edited Pi’s models.json in order to get Pi to talk to the model.

"lmstudio": {
 "baseUrl": "http://host.docker.internal:1234/v1",
 "api": "openai-completions",
 "apiKey": "not-needed",
 "models": [
 {
 "id": "google/gemma-4-12b-qat",
 "input": [
 "text",
 "image"
 ]
 }
 ]
 }

Here’s my Docker Compose config:

services:
 pi:
 build:
 context: .
 dockerfile: Dockerfile
 image: pi-agent:0.74.0
 init: true
 stdin_open: true
 tty: true
 extra_hosts:
 - "host.docker.internal:host-gateway"
 environment:
 ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY:-}
 OPENAI_API_KEY: ${OPENAI_API_KEY:-not-needed}
 GEMINI_API_KEY: ${GEMINI_API_KEY:-}
 OPENAI_API_BASE: ${OPENAI_API_BASE:-http://host.docker.internal:1234/v1} # note that you'll need to specify a base if you also use OpenAI to access OpenAI's actual completions endpoint
 WHATEVER_API_KEY: ${WHATEVER_API_KEY:-}
 volumes:
 - ${HOME}/.pi/agent/models.json:/config/models.json
 - ${WORKSPACE:-.}:/workspace
 - pi-config:/config
 - pi-sessions:/sessions
 working_dir: /workspace

volumes:
 pi-config:
 pi-sessions:

and here’s the bash script that runs pi .

#!/usr/bin/env bash

# Pi — Start the containerized Pi agent.

# Directory containing this script and the compose files.
SCRIPT_DIR="$(cd -- "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

# Workspace to mount into the container. 
WORKSPACE_DIR="${WORKSPACE:-$(pwd)}"
case "$WORKSPACE_DIR" in
 /*) ;; 
 *) WORKSPACE_DIR="$(cd -- "$WORKSPACE_DIR" && pwd)" ;; 
esac
export WORKSPACE="$WORKSPACE_DIR"

sandbox="${PI_SANDBOX:-0}"
pi_args=()

while (($#)); do
 case "$1" in

 --sandbox) sandbox=1 ;;
 --no-sandbox) sandbox=0 ;;
 *) pi_args+=("$1") ;;

 esac
 shift
done

compose_files=( -f "$SCRIPT_DIR/docker-compose.yml" )
if [[ "$sandbox" == "1" ]]; then
 # an even more secure sandbox
 compose_files+=( -f "$SCRIPT_DIR/docker-compose.sandbox.yml" )
fi

# Derive a container name from the workspace directory's basename.
# Sanitize to characters Docker accepts: [a-zA-Z0-9][a-zA-Z0-9_.-]*
repo_slug="$(basename -- "$WORKSPACE_DIR" | tr -c 'a-zA-Z0-9_.-' '-' | sed 's/^-*//')"
[[ -z "$repo_slug" ]] && repo_slug="workspace"
container_name="pi-${repo_slug}-$$"

api_key_args=(
 -e OPENAI_API_KEY
 -e DEEPSEEK_API_KEY
 -e ANTHROPIC_API_KEY
 -e GEMINI_API_KEY
)

cmd=(
 docker compose
 --project-directory "$SCRIPT_DIR"
 "${compose_files[@]}"
 run --rm
 --name "$container_name"
 "${api_key_args[@]}"
 pi
)

if ((${#pi_args[@]})); then
 cmd+=("${pi_args[@]}")
fi

exec "${cmd[@]}"

I build the Docker container and make changes to the files in its own repo. Then, I run Pi in the repo I’m working in, which spins up Docker so that Pi can’t wipe files or directories by acting on my physical hard drive. This also enables Pi running in the container to see my custom model json config by shipping it into the container. All of this has been working fairly well for my experiments.

There are still issues with local models: inference can be slow, context windows are small and limited to your own hardware, and the ecosystem, although it’s made a ton easier by tooling like LM Studio and HuggingFace’s Use This Model button. Early releases suffer from prompt template mismatches. But, these are usually patched extremely quickly. Needless to say, I’m not sure this is ready for production software development quite yet.

The benefits, though, are numerous and the ecosystem critical to invest in, particularly now. One of the very cool parts of local models is you can introspect almost everything, like watching the token inference process live,

and watching tokens in/out.

You can do things like change the local context window and watch performance improve or degrade, and really dig into how your tokens are processed on the GPU. You can change the system prompt, the quantizations. You can pit models against each other. You can also change and introspect the harness side.

The possibilities are endless, and the tools only keep getting better.

We should be more tired than the model

Thu, 28 May 2026 00:00:00 +0000

Lately, I’ve been feeling like I’m losing control over the code I write when I work with agentic code generation.

When I finish an agentic session, I get all the outward signs of having written code, but none of the internal processes that happen when we write code by hand.

As a quick primer, the human brain has several types of memory, short-term, working, and long-term. Short-term memory gathers information temporarily and processes it quickly, like RAM. Long-term memory includes things you’ve learned previously and tucked away, like database storage. Working memory takes the information from short-term memory and long-term memory and combines them to synthesize, or process the information and come up with a solution.

When we’re working on code, (and by working on, we mean most often reading someone else’s code) all of these processes are going on in our brain simultaneously to try to help us make sense of the programming environment.

It’s not surprising. Code generation, in its default mode, is antithetical to skill retention, particularly because its UX affordances are reminiscent of a slot machine’s: you pull the lever, you get a reward (a solution to your coding problem.) In some ways, we’ve replaced the social media feed with a stream of tokens, and I look forward to reading those papers in ten years.

It really does take extra concerted effort to move from just generating answers to using the tool deliberately. One thing Oz suggested when I posted on X that I felt like I came away from an agentic session with a brain fog was rewriting portions of code myself.

Inspired by that advice, the paper, thoughts on slowing down, and using AI to write better code more slowly, and Mitchell’s adoption journey, I’ve been working on using the tool more deliberately and adding friction back into development.

Here’s what’s worked for me so far:

Writing the initial implementation myself and asking the agent to review the code, then going through comment by comment and manually making the changes
Using the agent to keep asking questions about pieces of the code I don’t understand instead and pull up relevant documentation and PRs.
Asking the agent to think about implementing two approaches and choosing between them and then critiquing the other approach
Discussing an agent’s proposed implementation with another person instead
Starting to use the agent only after I’ve spent 20 minutes on the problem
Going back and reading books and academic papers
Reimplementing fundamental data structures

All of these negate the supposed speed up effects of LLM-generated code in the short-term by adding friction, and yet, in the longer term, make me better at using the tool, because they solidify my own foundation instead of the foundation models'.

We should be more tired than the model.

Tagging my blog posts with BERTopic and LLMs

Mon, 18 May 2026 00:00:00 +0000

I recently added tags to my blog using BERTopic and a mix of LLMs. You can see the tags in the sidebar to the right (or in the footer on mobile). I’ve done this before in 2023, with GGUF Mistral using llama-cpp, but never finished the project. Now, because the models have been getting so good, and my project was small, relatively well-defined, and easy to evaluate, the project took me about 6-10 hours over a month, using BERTopic, Claude Code, and Pi with Deepseek.

Why so many different AI tools? Mostly to evaluate their different ways of working. Much of that time was spent noodling on the UX experience of the tags rather than iterating on the tags themselves.

One of the genuinely useful use-cases of LLMs these days is for finishing personal projects that don’t touch production and have a small surface area that’s personalized for you. In other words, as Robin Sloan wrote, an app can be a home-cooked meal.

I love having a static site because of how easy it is to write and publish content and how fast it loads, but sometimes I wish it was slightly more fully-featured. LLMs have allowed me to add features like search. The theme I use, Hugo Bear Blog, already has support for tags, but I’d never added them to posts, and I also wanted a slightly different way to visualize them.

We consumers generally use LLMs for text or image or, if we’re developers, for code generation. But, one of the most underrated features of LLMs is the ability to compress rather than generate. This is really unsurprising: LLMs are, after all, natural language models.

Since they were trained and fine-tuned originally on language modeling tasks they also perform really well at all the tasks that language models are meant for, such as summarization, information retrieval, question answering. LLMs are really good at labelling things. That is, they’re good at topic modeling, the machine learning task behind tagging, especially in a zero-shot context (where they have no previous training data from you specifically).

What were tags?

In the early days of online blogging, tags were important for facilitating content discovery. People initially started tagging their blog posts on their individual blogs. Eventually, site aggregators like Delicious, surfaced top links with tags by aggregating tags across top links shared by users.

Pinboard was another prominent platform where finding content through tags and looking at aggregated tags was an important feature of the platform. Coincidentally, Delicious was later acquired by Pinboard.

Early on, the best way to find things that you liked was to manually participate in the curation of these folksonomies. Twitter and Tumblr developed some of the most creative folksonomic tagging systems. On Tumblr, tags became a way to not only discover posts, but to have conversations with other people about your post. On Twitter, hashtags became a way to signal communal discovery of people with shared interests before the implementation of Twitter’s SimClusters algorithm.

Tagging and hashtags served as an implicit contract of content discovery across platforms for over a decade, across Twitter, Instagram, TikTok, and many other services. However, big social has been in big decline for a while now. Group chats have arisen as a medium for exchange and discovery, as well as Discord groups that rely less on traditional tagging mechanisms. Bluesky, a social platform that was founded more recently, has the ability to add hashtags to posts, but most folks don’t do so. Discovery happens, like with many platforms today, through starter packs and custom algorithmic feeds.

The rise of LLMs led to an even greater decrease in the power of individual websites to add signal. An increase in AI overview features in search results that offer either summarization or RAG-assisted source synthesis has meant that visits to actual websites are dropping faster than ever. With LLMs, the rise of semantic and blended agentic-style search as a discovery mechanism means tags are not as important.

Within blogging, content surfaces like (oh God) LinkedIn native posts and X’s longform posts are contributing to platform-specific lock-in. All of our public blog content is being scraped as training data anyway, and agentic search and RAG mean people access content through an LLM’s interpretation of it rather than going directly to a page. RSS still exists, but who is going to syndicate a site when they could write an article on X or LinkedIn? (Me, sure.) You can subscribe to my feed!. Blog tags as a mechanism for understanding what my blog is about realistically still probably matter only to me. But I still want them!

Historical approaches to tagging with LDA

Generally, synthesizing and detecting topics across a body of text is an unsupervised learning machine learning problem. We don’t know ahead of time what the categories are. We have to infer them from the corpus.

Traditionally, this was a really hard statistical problem in machine learning solved with approaches like latent dirichlet allocation (LDA). I really recommend Ted’s post if you’re interested in more of the core implementation details. The key idea is that, given a corpus of text documents, each document is modeled as a collection of topics. Each topic is a distribution over words, which means that each word in our corpus has a stated probability for being a member of a topic.

There were a number of unsolved problems with this approach, and with all statistical approaches. The biggest one was that the underlying assumption of topic modeling is that documents are bags of words. That is, words that are independent of each other. But, because words lose their context when they’re broken up from their constituent sentences (as we did when we created the tokenized input text for the model), the model wouldn’t recognize that the word “bank” in these two sentences, with “I went to the bank” and “The bank of the river overflowed with water” are different from one another.

One of the most trite and overused quotes in NLP is “You shall know a word by the company it keeps”, and this turned out to be one of the core successes of evolving neural approaches for language processing.

From embeddings to tagging with early LLMs in 2023

The evolving use of embeddings made it much easier to represent items that were semantically close together mathematically, by projecting into the same multi-dimensional vector space. The following generation of contextual embeddings calculated vectors based on the position or representation of the word, which eventually led to models that fully modeled the context - or the relationship of every word to every other word in a given text at the same time, creating weights of how relevant a word is to any other given word, aka the attention mechanism.

The first part of any topic modeling approach is picking your seed topics that the model can extrapolate from. You can do this in a number of ways: by bootstrapping from a couple of selected ones and creating more detailed ones, by picking plausible topics across your document set through tf-idf or LDA, by going through all the content and having an LLM zero-shot them.

Seeding topics with Mistral

This is what I tried to do the first time I tried to do this exercise. When local models like Mistral 7B-Instruct became available in 2023, I started playing around with them and realized that they did well at tagging. I decided I wanted to see if I could tag my blog with topics.

The first part was figuring out a set of tags from the given documents. I did some searching and found a paper called TopicGPT which was already exploring ways to use LLMs to do tasks that were previously done by BERT encoder models, and prior to that, latent dirichlet allocation.

I adapted the template in the paper with GGUF quants of Mistral 7B using llama-cpp-python (ancient history by now!).

Prompt templates were really important, less so than now, and the code looked like this:

system = f"""
You will receive a document and a set of top-level topics from a topic hierarchy.
Your task is to identify generalizable topics within the document that can act as top-level topics from the hierarchy.
Output the existing top-level topics as identified in the document. If the document has new topics,
suggest those.

{topics}

[Examples]

{examples}

[Instructions]

Step 1: Determine topics mentioned in the document.
- The topic labels must be as GENERALIZABLE as possible. They must not be document-specific.
- The topics must reflect a SINGLE topic instead of a combination of topics.
- The new topics must have a level number, a short general label, and a topic description.
- The topics must be broad enough to accommodate future subtopics.
Step 2: Perform ONE of the following operations:
1. If there are already duplicates or relevant topics in the hierarchy, output those topics and stop here.
2. If the document contains no topic, return "None".
3. Otherwise, add your topic as a top-level topic. Stop here and output the added topic(s). DO NOT add any additional levels.

Output:
existing_topics: ["Preprocessing", "LLMs"]
new_topics: ["Python"]
"""

Although both llama-cpp-python and the model worked well (for what was available at the time), I was constrained by a couple things:

A small context window - Mistral 7B GGUF had 32k tokens which meant that I had to make decisions about how to fit both my prompt and the longer documents entirely into the context window. Generally compacting to fit into a context window involves either cutting off text or creating a summary of your text and then summarizing that summary, neither of which are ideal when you’re looking for specific terms. You can see the token counting that has to happen in the process.
It was annoying to deal with reprompting the model to run in a loop for every document and keeping track of the results of those documents, especially as the smaller, local model ran out of memory.
Constraining guaranteed outputs for categories using a combination of the prompt and Outlines was finicky, even with the help of Outlines, which made it easier. Side note: Read the paper, it’s very cool.
I couldn’t decide on the proper method to create an initial pass at my first-level taxonomy topics and ended up doing it with this prompting approach, but still had to seed it manually, which didn’t create good categories. This problem isn’t specific to LLMs and is a known problem that’s discussed in the original topic paper, way back from 2005:

The underlying factor behind this variation may be that basic levels vary in specificity to the degree that such specificity makes a difference in the lives of the individual. A dog expert has not only the skill but also the need to differentiate beagles from poodles, for example. Like variation in expertise, variations in other social or cultural categories likely yield variations in basic levels.

For the purposes of tagging systems, however, conflicting basic levels can prove disastrous, as documents tagged perl and javascript may be too specific for some users, while a document tagged programming may be too general for others.

Tagging is fundamentally about sensemaking. Sensemaking is a process in which information is categorized and labeled and, critically, through which meaning emerges (Weick, Sutcliffe & Obstfeld forthcoming). Recall that “basic levels” are related to the way in which humans interact with the items at those levels (Tanaka & Taylor 1991); when one interacts with the outside world, one makes sense of the things one encounters by categorizing them and ascribing meaning to them. However, in practice, categories are often not well defined and their boundaries exhibit vagueness (Labov 1973). Items often lie between categories or equally well in multiple categories. The lines one ultimately draws for oneself reflect one’s own experiences, daily practices, needs and concerns.

Moving to LLMs + Agentic Harnesses in 2026

A couple of things have improved over the past several years that made me revisit the project.

Model context windows, even for open models, have increased considerably over the past three years. Anywhere from 200k to a million tokens is considered a fair baseline for a given leading model, although it’s unclear how much of that context is actually used effectively.
The emergence of agentic harnesses as a pattern for LLM access - hi Pi, the one I prefer to use! - into the frothy pot of LLM land means that you can now not only use models to tag individual documents, but also make your harness and underlying model clean up an entire site for less than 10 bucks, depending on the size of your blog and with much less manual wrangling of files.
As more LLM systems come online, it’s become increasingly clear that the most effective LLM systems are ones that incorporate blended approaches to machine learning. That is, they are composed partly of generative parts where they make sense, and classical deep learning or tabular approaches where these make sense. I blended older transformer models with newer approaches, and this ended up working really well.

So, we can simplify the work with LLMs by just starting to cluster with BERTopic.

Using BERTopic

BERTopic is pretty great and free in terms of cost (though not compute), so I decided to start there.

BERTopic accepts a given set of documents. For our purposes, each of my blog posts is a document. For each document, BERTopic creates an embedding of 384 dimensions. Then, BERTopic performs dimensionality reduction using UMAP, which preserves locality, to reduce the embeddings into 5-10 dimensions, but in dimensions that still make sense relative to each other.

Then, it uses HDBSCAN to actually cluster the points together. Then, it produces a bag-of-words representation from the clusters, similar to what we produced with LDA, which combines all the documents in a given cluster into a single document, and counts frequency per cluster. Finally, the algorithm performs tf-idf on the words in a cluster per topic.

Throughout the process, we are constantly performing compression. But, unlike LDA, which has to re-assign every word in the corpus across many iterations, BERTopic compresses at the document level. With BERTopic, we generate one embedding per document, then cluster those, which speeds up computation quite a bit.

It’s very elegant and extremely fast on (even) a Mac M2 since it uses a lightweight sentence-transformers model, and my blog’s document base is not that big: 176k words.

printf 'Lines: %d\nWords: %d\nBytes: %d\n' \
 $(wc -l -w -c bertopic_input.txt | tr -s ' ')
Lines: 4115
Words: 176282
Bytes: 1063606

The “context window” (really, max_seq_length) for all-mini-lm is 256 tokens, which means that I could break my posts down into, essentially, paragraphs, and have each paragraph be an individual document so that I didn’t need to summarize or fold into a context window.

I fed that into BERTopic and got an initial first pass at the topics.

Refining the topics

The thing about clustering algorithms is that they will cluster things into groups, but won’t label the groups. For example, here are some of the clusters that BERTopic produced.

You can see that the attempted titles loosely fit with the tags, but that it would be better for the “software” cluster to come up with something slightly better. Software is a really generic tag in and of itself - my whole blog is about software.

So, what concretely do I want to say about software? I like to talk about best practices, things we should aspire to. A better word for that might be craft? But it’s not just any craft, it’s software craft.

As the Delicious paper alludes to, it takes a while to get to a good tag, because you’re creating a contract not only for yourself, but what you think your audience understands to be true. This was so important that many early taxonomy systems had experts: aka librarians! That created the first taxonomies. Clay Shirky wrote about just how hard this can be, particularly when you are creating a taxonomy in a digital space that has no boundary constraints.

This was such a complex problem that companies like Tumblr, for example, had a meme librarian that explicitly spent time cataloging and thinking about trends specifically on the platform.

Since these particular blog tags are just mine, I can formulate them however I want. Once I got the original output from BERTopic, I created a list. I then spent a bit of time in the Gemini chatbot window tuning them. Why Gemini? I’d already used it to write a small script cleaning up my markdown posts into documents, but Claude or ChatGPT would have done a similar job.

For example, here’s some iteration I did to get to the “shipping” tag, which you’d think would be extremely easy, but it turns out was actually hard to get to. When people talk about “taste” with relation to LLMs, this is what I think they mean - it takes time to get it to generate something that is like something you yourself would generate, versus what it (or the average of the internet) thinks.

I then decided to get some more help from GPT-OSS running locally on LM Studio in thinking about what, if any, other top-level categories I should have, and refining the final list.

Now, I had a taxonomy, and I was ready to classify.

Batch-tagging with Claude Code

I took the initial list of topics and spun up Claude Code in my blog repo. I asked it to go through all the posts that I had, in batches, and tag them, using this prompt:

selecting from this list of tags, go through all the posts in the `blog` repo and select the appropriate tags based on the content of the posts. If no tags fit, skip. do not suggest related tags. Select between 5-7 appropriate tags for each post. Start with the latest post and work backwards chronologically.
cloud
engineering
data science
embeddings
machine learning
collaboration
diffusion
LLMs
.... [other tags from my list, up to 30 or so]

Claude worked for about 7 minutes, and the result was that each post had tags.

I then went through manually and made some adjustments as necessary. For example, one of the tags I’d jokingly added was “leading thoughts”, but when I saw it in a list of serious tags, it didn’t make sense and I didn’t feel it was helpful, so I took it out.

I had an end-result, which I put at the bottom of the page, and sat with it for a few weeks.

Generating the UI with Pi

I had my tags, but I was unsatisfied both with the amount of tags (too many, too many mixed levels) and with how things looked, because they were too cluttered. In my first pass, I added some more fine-grained categories like “Java”, which I ended up removing, because it became too cluttered.

It was around this time that Claude Code started having reliability issues that were partially obscured from users. After doing some research, I switched to Pi as my harness for a number of reasons, but mostly after watching this incredible talk by Mario. The engineering philosophy he describes really resonates with how I think about these tools, and software in general. That is, they should be introspectable, small, and light, plug and play, and open source.

I switched to Pi and have since tried out a number of models, including the latest from Gemini, OpenAI, and DeepSeek 4. Many of the models are largely interchangeable for this task with slight differences in personality, and I’m also excited to try hooking up Pi to Gemma 4 to see if I can get the same quality of local inference.

The UX piece took just as much, or even more, than the machine learning. Initially I represented the tags in the bottom bar, but decided to move them to the side for discoverability. Another issue was that initially, I had too many tags and collapsed them in the design.

In a second pass, I reduced the number of tags, but the collapsible menu still wasn’t exactly what I wanted.

After moving the tags to the sidebar, I did a bunch of front-end wrangling with Pi using a number of different models to both reduce the number of tags and get them in the correct format on the sidebar, which led to the results you see now.

Final thoughts

It only took me 6-10 hours (over the course of a month) to complete this project, but it truly is incredible how much work LLMs, as they are accessed via a harness in 2026, compressed for me. Outside of how great Pi is generally, most of the models and harnesses these days feel fairly interchangeable, but still allow a lot of productivity for general development tasks that are 1. small, 2. well-scoped, 3. within the developer’s area of expertise so they can assess it.

It’s worth noting that using pre-LLM transformer BERT models did a fair amount of the heavy lifting here, and they did it better and cheaper than LLMs did. I suspect as more vendors raise per-token prices and gate unlimited plans, people encountering rising token costs will at least partly come back to blending LLMs with traditional ML approaches.

Other than that, it was just much easier to work with models in a harness like Claude Code and Pi than the previous TopicGPT-style approach.

I am currently happy with the results. The posts were tagged fairly correctly, I was able to get to a finished result, and I’ve used the tags a few times afterwards.

Will the remaining seven real people on the internet find it helpful? Guess we’ll see!

Build yourself flowers

Mon, 20 Apr 2026 00:00:00 +0000

This is an edited transcript of the keynote I gave at the Applied Machine Learning Conference in Charlottesville, VA in April 2026.

I first wrote a draft of this talk by hand. This part took 2 months.

I then recorded myself giving a version of this talk with MacWhisper, and transcribed it with Whisper locally. This part took 45 minutes (the total time of my practice run.)

Then, I ran it through Gemini Flash 2.5 running in Pi to break into paragraphs. I also had Gemini break up my slide deck from a PDF I generated from Google Slides, into individual images to insert into the blog post and optimize the image format to webp for blog rendering. This part took about 10 minutes.

Then I went through the text paragraph by paragraph manually to make corrections, remove redundant phrasing and pauses, and added clarifications to make it more legible than a talk. This part took 3 hours.

Before all that, I generated the content for this talk. This part took 13 years.

I’m Vicki, and I build machine learning systems.

I debated for a long time how to introduce myself. Am I a data scientist? Am I still a machine learning engineer? Am I an AI engineer now? I’m not really sure. I think, like a lot of people over the past six months in the industry, I’ve been having existential angst. So, I’ll go with “I build machine learning systems.”

I’ve built and broken systems at Tumblr, at Automattic, at Duo, at Mozilla.ai. Now, I build realtime personalization and search systems at Malachyte.

We’ve had a lot of different conversations in all of the corners of the internet about how we should incorporate LLMs into machine learning workflows. And the question that came to me is, if all we’re doing is generative AI, where does traditional machine learning even fit in here? Are we still doing machine learning at all? So, the larger question behind my existential angst is, what is the state of machine learning engineering as an industry today? That is - is it still worth doing machine learning engineering?

And the second question that came to me was, not only is it still worth doing ML, but, in an era where we’re having LLMs generate a lot of code, when the most important thing is for us to ship quickly, to get to a prototype quickly, is it still worth doing machine learning well?

So hopefully we’ll be able to definitively solve all of my existential crises in 45 minutes, but until then, I wanted to take a step back and talk about flowers.

In 2024, the lovely folks at PyData Amsterdam, invited me to give a keynote. I called it Build and Keep Your Context Window. ChatGPT was still fairly new, out for just about a year and a half at the time, and everyone was freaking out. I talked about how important it is to build and keep your own context window - the historical understanding of software tools and concepts, so that we can use the new tools and understand where they fit into the context of existing software engineering.

The analogy I used was the ChatGPT UI. At the time, everybody was using the text interface, where you have a sidebar and you have the big text box. All your chat sessions live in the sidebar, and the sessions have titles. And I talked about that as being part of a context window.

The context window is the amount of text that a model can recall at any given time, as measured in tokens. If the text you input overflows the context window, the model loses the thread of the conversation and can’t reply coherently anymore. What determines the context window in reply is the attention mechanism in the Transformers model. And the attention mechanism is really a cache, which is really a hash map. So if you understand all these four things and how they fit together as concepts that naturally emerge over time successively, it’s not perhaps as surprising that we get the emergence of LLMs.

Before I gave this talk, I got to spend a few days in Amsterdam (I was told they were the only two sunny days in the Netherlands in September) and particularly in the Rijksmuseum, which is the national Dutch museum of art.

It’s absolutely enormous and beautiful. You could spend days there. But I only had a few hours, so of course I went to see the headline piece, which is the Night Watch by Rembrandt. When I went to see it, it was covered by this big plastic window and there was machinery set up all around it. I was thinking, what is this? Why is this blocking the painting?

When I went later to PyData Amsterdam, Robert Erdmann, who was working at the Rijksmuseum, gave an incredible talk about how he was using deep learning-based ink-removal, especially 3D imaging, pixel detection, and high resolution photography to see details in art at the museum that nobody was able to see before.

So for example, for The Night Watch, they set up all this equipment to be able to photograph it many times in many resolutions. And as a result, they were able to get high resolution pictures like the figure on the left, who is believed to be actually Rembrandt, the artist, in a cameo. And if you really zoom in, further, further, you can see in his eye there’s a little white sliver.

And so the question is, how do you become good enough that you know if you put just a little bit of a white dot, it will render as an iris of an eye to somebody? And so those are the kinds of questions, around the mystery of mastery, that he was now able to explore as a result of this photography.

Rachel Ruysch

One of the artists whose paintings are also on display at the Rijksmuseum is Rachel Ruysch, one of the premier flower painters during the Dutch Golden Age, and that’s who I really want to talk about. Being a premier painter of the Dutch Golden Age was a huge accomplishment, and I want to understand more about how she works, because maybe we can use her as a guide to navigate this crazy time where we’re doing a lot of code generation with LLMs.

So who was she? Rachel lived in Amsterdam, and all she did was paint flowers. Lots and lots of different compositions of flowers in all sorts of shapes and sizes, starting from when she was 15, well into her 80s.

Let’s take a step back and talk about the Dutch Golden Age for a minute. What made this era of creativity possible? In the late 1500s, the Dutch won independence from the Spanish government. The political stability was followed by economic stability and the rise of the Dutch empire. As Dutch life got significantly easier and more people joined the growing middle class, they wanted to buy art. In response, art, became seen as an important aspect of the Dutch national character, and art which had previously been mostly religious in theme, became secular. People started painting scenes from real life, science, and nature. All of this resulted in millions of paintings, many of them scientific or representative of everyday life.

What people wanted to see were pictures of everyday life that were accurate, that were true to what was going on around them, of local scenes, of scientific discovery. Of themes of religion and mortality, but conveyed in a different way.

So they wanted to paint stuff like gentlemen smoking and playing backgammon in a tavern. They wanted to paint stuff like people skating in the winter.

They wanted to paint stuff like people getting rowdy in their house while the mistress was asleep.

And if you look at all these paintings, you can see something come through, which is that the colors are down to earth. They’re very normcore.

Another thing people wanted was a lot of different paintings of flowers, hundreds and thousands of paintings of flowers.

Out of the five to ten million paintings that we think were created during the Dutch Golden Age, thousands of them were flowers. Why flowers? People wanted flowers in their homes that wouldn’t wilt over the long Dutch winters, and that showcased how worldly and interesting the homeowners were. Exotic and unusual flowers in particular (such as tulips of the famous Dutch tulip mania) were prized, and flower painters sought to accommodate this trend. There were many very famous Dutch flower painters and hundreds to thousands more, all producing and focusing on flowers. For artists, painting flowers accurately and lifelike was an important test of mastery.

And so Rachel Ruysch came into her skills at a time when people demanded this. And she was really good at it. And so the question I had, was that, if you have all these people painting flowers, why is it important to paint good flowers? Why do you want to stand out?

And the same question of, in a world where it’s easy and fast to write code, why is technical excellence still important?

As I was researching this, I came across a report from NASA

NASA has been on my mind lately with the Artemis II mission and their orbit of the moon and their triumphant return home. Because spaceflight is so hard and so enormously error-prone NASA has been continuously on a loop to improve quality, because it’s already very dangerous to be an astronaut at NASA. I’d seen somewhere that the Artemis crew knew they had a 1 in 30 chance of dying.

But NASA is always looking at risk, and they did an investigation in 2012 and came out with this report of what happens when things fail. Why do they fail? And they came up with these five problems. And the one that really struck out to me was the first one, shifting engineering excellence to insight and oversight.

So there were periods when NASA outsourced a lot of their engineering to subcontractors. And what happened was that the engineers who understood the design and the system design based on actual experience left. And the shift that resulted eliminated independent analysis and testing and understanding of the systems. And this was what led to degradation of engineering quality.

And if it sounds familiar, this is because it’s a very easy analogy for what happens when we outsource our work as engineers and data scientists and machine learning engineers entirely to AI.

So, striving for technical excellence in industry saves us from engineering, product, and cost problems. Quality is important.

The second reason is that software engineering is a craft, just like painting.

And it takes time to get good at painting. It takes time to understand where to put the little white iris. It takes time to get good at painting flowers. And so it’s important to understand that because it takes time to get to quality.

And finally, technical excellence is important because striving for mastery is just about what it means to be human.

We like when people do things well. We appreciate people who are competent. We like it when systems are designed and built well. It feels good for us to be part of systems that work. One tangential idea that I’ve seen going around the internet is that you shouldn’t talk down to LLMs, or call them “stupid”. Not because they understand, they’re just models, but because it’s bad for us as humans. In that same way, it’s important for us to strive for good things as humans.

Okay, so if we want to paint good flowers, how do we become good at painting good flowers? What made Rachel technically excellent? What might we be able to replicate?

The first one is that she was in an artistic family. So she was in one of the premier families of Amsterdam where all her great uncles were painters. Her grandfather was involved in art. Her father was a great artist and botanist, Frederick Ruysch. You can see him here with this creepy skull. He collected a lot of really creepy dead things, which is where she learned about scientific accuracy, embalming, preserving flowers, etc.

It was here that she also learned discipline: how to capture minute details, She was known for her technical skill, and for gathering flowers in botanic gardens and pressing them so she could have different compositions for different seasons, surprising the Dutch, who were generally orderly and used to seeing only seasonal bouquets. During her life, her paintings sold for 3x Rembrandt’s.

Then when she was 15, she was apprenticed to Willem van Aelst, who was one of the premier painters in Amsterdam at the time. It was tradition at the time for masters to take on apprentices who would live with them and start with stuff like washing their paintbrushes and work all the way up to painting, and then finally produce a work of art which, if the master accepted, the apprentice could become a master in their own right. So she had a very big support system and she was steeped in a culture of mentorship being extremely important.

Second, she painted her whole life. She was keenly interested in botany, in science, and made sure that her art reflected the true appearance of nature. She spent a long time - over 60 years, honing and working on her career, not even slowing down to have ten children. For her, her career was a vocation. She got inspiration from many different places, and she would do the same thing, again and again, but differently.

And finally, she remixed and experimented. So you can see in the background of this particular painting, there’s a cactus. And it’s not easy to get a cactus in Amsterdam in the summer or the winter. So she got this from the New World because she had connections to Amsterdam botanical societies and she knew people who were coming and going all the time. She bartered for and preserved exotic flowers. And she also remixed flowers from different seasons by drying them. She collected them all the time. She was always changing something about how she was working.

So, given all of this, how can we, as engineers, be like Rachel?

Building Rijksearch

I thought about all this and decided to see how I could apply these ideas to a concrete side project that I was working on. And the side project is called Rijksearch, a semantic search engine.

A semantic search engine searches by intent rather than keyword. So what you do is you type in “chill dude” and it’ll return you this portrait of, for example, a man with a dancing dog. Which I think is actually a really good example result. Or like a “confidential chat” or “man smoking pipe”, the last one for sure being a chill dude. So it’s a search engine that works by intent.

Or, you type in “pensive” and it will return a philosopher in his study, which is definitely pensive, a boy with a golf club probably not, you don’t want toddlers to start thinking too hard about what they can do with a golf club. But like, ok, these results are not bad!

Or “flower party”, which didn’t work that well because you can see it’s doing more keyword matching for “flower” than semantic matching for “party”, but this is where the craft of tuning semantic search comes into play.

There are already a lot of wonderful tools that the Rijksmuseum itself has that you can build with on public data and that they’ve built with like Art Explorer.

But I decided to build my own because I wanted to remix concepts. Rachel learned how to remix as an apprentice and from the artists she talked to. I learned it from working at Tumblr where the reblog feature was one of the original places on the social web that implemented the functionality of you posting something and somebody else riffing on top of it.

So what was I thinking about remixing? Oh, I had a lot of stuff going in my head. First, over the past couple years, I’ve been thinking about embeddings a lot. In fact, I’m thinking about embeddings right now.

In search and recommendations, embeddings - vector representations of text or images (or other media) - matter a lot. They also matter within LLMs themselves because positional embeddings are how we decide what concepts are related to each other when we train the models. I wrote a really long text about them, examining their place in ML systems, called What Are Embeddings?, and I spent a lot of time understanding how they fit into the context of our workflows.

Second, I’ve also been working with Redis for a while now. It’s one of my favorite pieces of software in the world because it just works. It’s extremely cleanly designed. It’s really fast. It’s extremely reliable. It is just very elegantly put together. So I had worked with Redis for search before, and I wanted to see if I could use it again. At the time, about a year back, Salvatore Sanfilippo, antirez, who initially wrote Redis, picked up working on it again and created a new data structure called vector sets.

Vector sets in Redis work much like sorted sets, which are collections of unique, non-repeating strings sorted by a score. In the case of vector sets, the string elements are associated with a vector. For anyone working with embeddings, this unlocks a huge amount of very fast, efficient vector operations, and they allow for vector similarity lookups using HNSW, a popular and efficient approximate nearest neighbor search algorithm.

Third, I’ve also been hacking around AtProto, which is the protocol that powers Bluesky, a social network that has become popular in the past couple years. It turns out that a lot of the Bluesky codebases are open, and when I started looking around, it turned out that a lot of Bluesky services were written in Go. (And TypeScript, but I scoped myself.)

Fourth, I’ve worked in Python (and Java and Scala, unfortunately), before, and I was curious to explore a new language that was meant for high concurrency and throughput and was the spiritual successor to Java. I did a couple projects with atproto where I created a feed of content in Go, and wanted to see what else it could do.

Of course, all the other stuff swirling around in my head was LLMs.

It’s impossible not to hear about them. I wish I could block the news out because it does feel like it distracts me enormously in my day-to-day life. What I’m particularly interested in with LLMs is the power of small, local models. So when llama.cpp came out and you could use models directly, or with apps like LM Studio to serve GGUF artifacts, I became interested in that.

I also became interested in hosted embedding APIs because before I’d been working with hosted models that you either train or pre-train models that you have to serve. And OpenAI made those accessible as API endpoints, and Gemini started serving new multimodal embedding models, so I became interested in those.

And then finally, the humanities. A lot of my projects revolve partially around the humanities. A lot of the stuff that I write ties back in some way to the humanities because I don’t think you can really have a balanced view of what developing a good product looks like if you’re just looking at tech. And, I like art. So I’m always down to look at and use art.

Build the same thing again and again

All of this stuff percolated for a year or so in my mind, and the result was that, at some point, I realized I wanted to re-implement Viberary, which is a semantic search engine.

A quick note on search engines. They come in several different forms. There’s keyword search, which is if you type in Apple, you get back results for Apple varieties. And there’s semantic search, which is you type in Apple and you get back iPhone, and maybe also Android, depending on what you’re looking for, which is why the “chill dude” query works.

And there are many different ways to implement semantic search. The way that I had implemented it before, one of the ways, was in Viberary.

Viberary was a side project that I’d done in 2023 that was based on text-only representation of books from Goodreads and all their reviews. And the idea was that you type in a phrase like sci-fi, funny sci-fi, or moody drama, and you get back a list of books that are recommended based on vibe rather than the books themselves, which is something that I always wanted.

I’m not always in the mood for a particular title, but I’m sometimes in the mood for a particular type of book. And so you can see for this one, funny Sci-Fi Space with Feeling, one of the results you get back is Asimov, which I guess, yes, kind of, and Haruki Murakami, which kind of also makes sense? So these vibes kind of fit.

What allows us to do this is that we’re not comparing just text or images. We project them into the embedding space by creating dense vectors out of them and comparing the vectors, the numerical representations.

That way, we can compare “dog” and “bird” and “fly” and see that “bird” and “fly” are closer to each other because they’re semantically similar. And this is also essentially the way the transformer attention mechanism also works on a much, much, much larger scale.

It used to be the case that people only learned or fine-tune their own embeddings, but now many of the major AI providers have their own hosted sets of embeddings. For example, Google recently released embeddings for Gemini, with both API and local versions being available.

Two Tower Architectures

Building semantic search involves implementing a two tower architecture, a really common paradigm in both search and recommendations, and one that I’ve reached for again and again. The two tower architecture and semantic search are for me what flowers were for Rachel.

There are generally three steps to creating a two-tower model.

Embed the items once, in batch.
Embed your query in realtime at inference.
Compare the distance between the two.

So when I search for “chill dude” it generates an embedding for that phrase and then we do a similarity computation, usually using cosine similarity or dot product, or there are other distance measures you can look at, that says “this embedding is most similar to the embeddings and then it surfaces those results just as we do here.

But there are just so many decisions to make in this process that only become apparent once you start looking at the constraints of the data and the engineering space you’re working on.

This is a very simple structure but it’s very elegant because it allows for fast indexing and fast lookup on a large, coarse dataset. When we do any kind of information retrieval whether it’s for recommendations or for search, our goal is to get from tens of millions of items the entire internet a whole catalog worth of 800, 000 paintings like the Rijksmuseum has down to tens that you can look at in a carousel or in a screen or in a list of results.

Two towers is one way that allows us to pre-compute the embedding for our entire catalog and then it allows us to filter them quickly using approximate nearest neighbor search, which is a fast algorithm which says this thing is closer to this group of things but not this other group of things.

That allows us to do a first pass from tens of millions to tens of thousands or hundreds, and then we can use more expensive models to rank the smaller result sets and really think about how we would present those last, smaller result sets, to the user.

Many large companies then expand on this architecture to create search and recommendations data sources for different product surfaces.

Here’s a couple examples. For example, the one on the left is YouTube’s recommender system where they take millions of videos that they could recommend you and narrow them down to a few hundred and then rank the ones that are actually on display.

The input features for their custom trained model include “videos you’ve watched before” and then “videos you’ve searched for”, “geography, " and then metadata about you. And the query is the video in question. And then the model similarity lookup generates a list of candidates to that video based on all this input data.

Facebook Search also uses embeddings. It contains the search text plus the query context, like geography or language. The document side includes Facebook names or page titles, and graph-derived signals like relationships to other people.

So, there are many variants of these systems in production. How does this look in practice for Rijksearch in particular? We have our input query, which comes from the front end as happy flowers. The embeddings are from the Gemini API. And then we also already have all our document metadata that we’ve embedded before. The indexing app is in Go, writing into Redis vector sets, and the serving web app is written in pure Go. Redis is served as a docker container that connects to the web app with everything served in docker-compose.

It’s not a particularly original idea: there are now a lot of semantic image/vibe search engines out there today. But it’s the one that I personally wanted to do because I’d developed a connection to the Rijksmuseum, and because the amount of data they generously make public made it lovely to work with.

Getting data for Rijksearch

Something that you’ll notice is missing here if you’ve worked in data before is the actual source data. Where do we actually get this image data? How do we collect it? And that’s its own hard problem that we’ll talk about in a minute.

But okay, so how do we build this, given that we’ve already built Viberary and other versions of this system before? The way that I thought about it is that every company gets about three innovation tokens. This is from an amazing and often-cited blog post by Dan McKinley, which says that every company gets three of these tokens.

You can spend those however you want, but then the supply is fixed. That’s also true for any project, whether it’s at work or a side project. For a specific project, a way to think about innovation tokens is that you can try new things in several dimensions:

The domain area
The language stack
The app architecture
How/where you deploy it

If you do three new things that you can’t do or don’t know how to do, you’re just going to get held up with your product. So where should we spend our innovation tokens?

We should spend them on what helps us grow, because humans want to reach mastery. So we should spend them on what helps us reach mastery.

Taking a look at the same steps for Viberary and Rijksearch, you can see what I implemented differently. So there’s a couple steps. First, we have to collect the data. Then we have to think about what embedding model we want to use and how we want to embed it. Then we need to index our embeddings. This is an important step. And then we need to serve our embeddings and create a web server so we can actually stand up the front end and return results.

So the first time I created Viberary by using text-only data from Goodreads, a locally-hosted embeddings model artifact, indexing using FT.SEARCH from Redis’s search , and Python’s Flask to serve the app.

So I lied earlier about using 3 innovation tokens: I cheated a bit and used four here.

For Rijksearch data, I used multimodal text and image data because I was interested in creating text and image embeddings. This made processing the data trickier because I needed to associate a given set of text metadata to a link of an image, as well as process the images in an efficient way so they’d download quickly and non-disruptively.

For the actual embedding inference, I changed from a hosted model to an API, which was easier in some ways, but now required cost and latency considerations. Additionally, test and image processing together was trickier because you have to think about text and image aggregation together..

For indexing, I switched to vector sets, though since vector sets don’t require the construction of a search index and are a native structure. This makes it easier in that you don’t need to update fields, but vector sets also don’t support metadata updates in the same way that search indices do because vector sets are schema-free (though you can add JSON through VSETATTR ), so there is a tradeoff.

For serving, I changed from Flask to Go, which was a pretty big change because I had to relearn web development in a new language. Go has a lot of tooling for web development out of the box, as well as support for concurrency, and for processing web data. This was probably the innovation token that took me the longest time to overcome.

And, interestingly throughout this process, I also tracked the steps where AI helped. The yellow parts are where I included AI or I had AI assistance.

I consider myself a fairly normcore AI user. I’ll usually wait a couple months for something to come on the market to try it. If something has been on the market for 6 months and has lasted, it probably makes sense to keep. And so in 2023, what was on the market was MidJourney for generating some of my logos, ChatGPT for answering some questions, and then some of the open models. Mistral was making good open models at the time and I was using it locally via llama-cpp-python, though getting limited results beyond summarization and tagging.

When I did Rijksearch, I found several new use cases for AI. For example, I used Claude, as well as GPT-OSS 20B locally to help me figure out some of the paradigms for data collection from the Rijksmuseum, as well as write some quick data analysis scripts. I used Gemini APIs for embeddings and Perplexity to help me decide which embeddings to use. And I used Claude Code to generate some of the front-end code, and Qwen Coder and GPT-OSS and Perplexity to help me learn about Go.

Data Sense

For Rijksearch, data collection was also a lot of work, even more than scaffolding the app. But as ML and data professionals, what helps us here is the same thing that helped Rachel. She had help and mentorship from her family. As data practitioners, what we have from doing this for a while, and being surrounded by data-intensive environments, is data sense. We have been in the trenches of collecting data for a long time. We understand the pain points. We understand NaNs and what loopholes to look for in non-deterministic data flows, such as the ones generated by LLMs. We understand what it means to “look at the data.”

And this is something that Rob Pike, one of the creators of Go, said. If you’ve chosen the right data structures and organized things well, the algorithms will almost always become self-evident around the data. Data structures, not algorithms, are essential to programming. And this was really important for me because good similarity matching between images and text requires good data for both.

Luckily, the Rijksmuseum has a lot of data, and it’s all open and accessible to the public. It’s an amazing resource.

They have metadata for 800, 000 objects, high-resolution photographs. But something that was new to me in working with this data was working with OAI-PMH, the data format they offer their artifacts in. It’s available through an API, but the API is very different from what I’ve generally seen. OAI-PMH is an XML-based protocol over HTTP, which is different from most of the JSON-based REST APIs I’ve worked with.

LLMs helped me quite a bit here because I needed to understand, for example, ListSet records I needed. And, since this was open data it was easy to traverse the API results and get an understanding of what lives in that data set.

And it turns out there’s a lot of data to figure out.

First, I pulled all the paintings we needed from the record files. The data’s stored across multiple collections, so I chose to use the “Dutch Paintings of the Seventeenth Century in the Rijksmuseum” set because that collection had the most visually striking paintings. From there, I extracted each entry, making sure to keep the full metadata (title, artist, date, medium, etc.) instead of just a short snippet. This gave me a clean, complete list to work with for the rest of the project.

How do we actually collect that data into our app? Here’s where Go really helps. Go structs are typed collection fields, much like Pydantic models, and they really allow you to do this well. If you have an XML snippet, you can either run it through an LLM to get results, or use a site like XML to Go, no code generation needed. But, of course, LLMs do really well with Go because it is so strongly typed.

It really takes a while to wrangle all the data into the correct format, then, from collecting it as XML, to writing it to disk in Redis.

Then the fun starts, because you collect these records and you start to see that not every piece of data is labeled correctly. This is true of any given dataset, but particularly true here, because this data has been collected over lots and lots of time in many different languages, across historical collections, and who amongst us has not had the wrong language for a title or duplicate data in any of our data. This is where data sense and really looking at and being hands-on with the data helps.

Picking Embedding Models

The second part of data is picking an embedding model. For Viberary, I usedMiniLM just because it was small, fast, and it was standard. Now we have larger models that are hosted as APIs. Generally, depending on your use-case, it may make sense to use a domain-specific model, or fine-tune. But for a side project, a general model was just fine.

There are a lot of other factors we care about when selecting an embedding model, such as inference costs, storage costs, and the overhead of setting up a new account. For me, multimodality was also important (whether the model was trained on both images and text) because that becomes relevant for retrieval.

To help narrow down my options, I used Perplexity to generate a comparison grid and verify the information. One thing I like about Perplexity is that it cites its sources, so you can click through and confirm that the model didn’t hallucinate anything.

And so I came to three choices based on price, storage costs. What was interesting is that what really unblocked me was I already had a Google account with access to Google AI. And so I could just use and set up an account key easier than I could for Cohere and Mongo, although I’ve heard good things about both.

And then you have to pick which vector index you’d like to use. The index is the data structure that helps you quickly find related items.

Vector sets already implement HNSW by default. That makes the application really fast at retrieval time, which is what I care about. I didn’t really care as much as how long batch indexing takes, because at the end I only had something like 2, 000 paintings that I needed to index at a time. If I had a large selection, the choice between HNSW and IVF, which is great for data sets with large memory constraints, would have become less clear.

So then you embed everything, and then you start to get your results. And of course, because this is a data-intensive project, you get this thing where you query for “boat”, and you get the same thing for every single result, which doesn’t look like a boat at all. But there’s a pyramid, and you’re like, okay, why is this happening?

So then you have to go through and debug step by step, and it turns out the issue is not in the data structure, and not in the ingest, and not in the Gemini embeddings, and not in the indexing, and not in the web server, but in the fact that you picked an art category that only has 100 artifacts and your query is similar to all of them. And that’s all part of the process of looking at your data, too.

Where AI helped and where it didn’t

Rachel painted her cactus. This was her remixing and tastefully adding small, exotic elements to the larger whole. For me, this was AI and really evaluating where it might help or hurt my process to create good things and get better at what I do. So, where did I end up?

I used AI to build the front end, which turned out nice. It wasn’t anything out of the ordinary, and I am not sure how solid the actual code is, and whether it’s production-ready, but it was nice to get going since I’m not a front-end person at all and we always want to get to a good demo. Last time, I used a bootstrap template for this that I tweaked slightly, which I guess is not that much different.

LLMs also helped me with Go, but they didn’t help me by writing all of the code. How I’ve been learning Go is step by step: first, by reading dead tree books, then by trying out Go by Example, and finally, by asking about different Go concepts from the LLMs, or working with a very small, method-level snippet at a time.

Because what happened when I was having the model write Go directly is I found, due to cognitive offloading, that I couldn’t understand what “I’d” written, at all, and when I got up from the computer, I couldn’t replicate it. But it helped me answer questions like how structs work, what you should put in a struct, what a channel does. The kind of stuff that you would have to constantly bug a senior person about, because Go is a new language for me, the LLM will just answer. And particularly in some cases where LLM systems have RAG results, it will lead you to the source documentation because Golang has really, really excellent documentation, and that’s really what we want out of learning by reinforcement.

LLMs also helped with data formatting and exploration. So writing all of the matplotlib methods, the pandas methods, all that stuff where you need to constantly look stuff up or you need to get data from the API. A lot of that glue stuff, the models, any of them really, are good with.

So where did it not help? Largely with architecture changes and decisions. Understanding that I needed a two-tower model, what two-tower models were, how they’d been used previously in search and recommendations, understanding my latency constraints, knowing that serving this app has to be fast and that the indexing job’s speed didn’t matter as much: all of that came from my head and past experience rather than AI.

Picking vector sets as a storage and retrieval primitive was similar. I only learned they existed because I pay close attention to what’s happening in the vector space. (haha) You could probably build a bot to synthesize that kind of knowledge for you, but it was much easier to verify for myself.

More complex concepts like Go channels , which I haven’t gotten to, were also a weak spot. I couldn’t verify whether what it gave me in Go was actually correct, and had a lot of non-idiomatic code until I got an actual experienced Go developer to take a look and review “my” code. And of course, AI didn’t help in writing this talk. It took a very long time, and none of it was written with AI. It all came from ideas that had been slowly combining in my head. Which brings me to the final point: it takes time to build good things.

It takes time to build good things

It took Rachel 60 years to get good at painting.

You can see one of her first paintings on the left, and then one of her later, much, much later paintings on the right. Take a look at how much more complex the composition is, how the flowers are behind each other, how much more delicate the background is, the movement of the flowers in the space. It takes so much time to get good at that stuff. And we have to remind ourselves in the world of speed that that’s true for programming too.

Peter Norvig wrote this post a long time ago about teaching yourself programming in 10 years. There is no shortcut for deliberative practice, which is what we’re trying to do here when we build ourselves flowers.

For me, I found it takes time because I was going through my tweet archive to see what I could find for this talk. In 2013, I wrote about how I was learning Python and counting words and just really struggling through it. And it’s like, it’s not very optimal and I don’t really understand what’s going on and what RGV is, etc. And then 10 years later I was able to do Viberary just through deliberate practice. And three years later, I did it again, in Go, with multimodal embeddings.

And the other thing is, I didn’t work on my craft alone. I’ve written about how important it is to write with your alphabet radio on. This is a concept from Bird by Bird by Anne Lamott, which is a book on how to write, but also on how to be creative.

Anne Lamott talks about this alphabet radio that plays in your head. Of all the past people that have either criticized or praised you and how you have to tune some of them out selectively when you work. For me, it’s all the people that have reviewed my code and that I’ve worked with and the ones that I respect are always echoing in my head. That kind of feedback comes from good mentorship. That’s the apprenticeship that Rachel went through as well right so it takes time to get good at that.

It also takes time to teach other machines. Here’s another small example from another side project I did before called Gandinsky, where I tried to train a generative adversarial network to paint pictures in the style of Vassily Kandinsky. At the time, this didn’t work well. This was like eight years ago, and it didn’t work because you need thousands of samples to train a GAN and Kandinsky unfortunately in all his artistic prolificacy only painted 200 paintings.

Nowadays, you can just get Nano Banana to do style transfer for almost anything for free in an API call because we’ve been working on these models for such a long time. But it’s taken years and years for the models to get this good.

And finally, it takes time to own your own vision, as in this quote is from antirez and his recent thoughts on agentic programming.

Programming as an automatic vision is not yet. That only comes from doing the work over and over and over again. So building a semantic search engine really took me about a month once I buckled down and did it (though I’m still not entirely done). But it also took me ten years. And I think that’s important to keep in mind.

Okay, so now we’re at the end of the talk, and the question is, does this still matter? Yes, I think so. I think data sense matters, which we all inherit from having worked in machine learning and data science for years. I think mastery matters, which we get by working with others and building our own intuition for mechanical sympathy. I think craft still matters.

So let’s try to build ourselves flowers. Thank you.

Mechanical sympathy

Mon, 13 Apr 2026 00:00:00 +0000

Weaver, seen from the Front, Vincent van Gogh, 1884

Something that’s been floating around in my head lately is the idea that I don’t know any truly good engineers who are also not good at at product design.

Product design can roughly be designed as the contract between the creator and the user, where the contract is designed by a set of affordances, or actions that the product allows the user to take. This is all cribbed from Don Norman and The Beauty of Everyday Things.

For example, an affordance of a chair is that it allows you to sit. An affordance of most social media feeds is that they allow you to pull to refresh. A search bar, in theory, affords you the ability to look for results. Recently this has changed because we’ve been vibe spaced.

What makes good engineers good at product design is the same thing that makes them good at engineering. They feel for the boundaries of what the code and the product allows them to do and stop at those boundaries.

Another name for being able to understand and plan for affordances, either through good product intuition, or experience, or both, in the real world is mechanical sympathy.

Mechanical sympathy was initially coined in referring to Formula 1 racecar handling, and in software, has been adapted to mean “writing the kind of code that makes sense with the underlying stack so we can get good performance from our systems.”

Mechanical sympathy for both developers and end-users means understanding when asyncio is and is not helpful. It means using the right language, the right build system, the right font. It means using the least amount of tooling possible. Allowing for local development. It means reading code inside out rather than top to bottom. Using uv. Removing code where not necessary. Respecting boundaries.

I was watching this really great talk by Mario on how he built an agentic coding harness, and was reminded that agentic coding tools, just like many cloud services, don’t have mechanical sympathy.

Things that have happened to me over the past few weeks, off the top of my head, across various projects, languages, and providers:

A test failed, producing a 500 error. I asked the agent to fix the test. The first time, it tried to fix the code instead of the test. The second time, it rewrote the test to assert a 500 instead of a 200.
Even with a spec file, agents (and most chatbots, still) will use Python’s List and Dict, which have been deprecated for years.
Agents won’t use vectorized numerical operations unless prompted.
When writing a test, the agent fails numerous times, and instead of suggesting that the code should be refactored to be simpler and easier to test, just keeps trying, endlessly, to work agains the grain of the code flow and fix the failing thing.

In new Python projects, uv is skipped entirely as a first-class citizen across many providers as a suggestion of how to package projects.

Here’s more stuff:

Maybe agents.md / gastown / mcp / ralph / subagents / using a different provider that was nerfed last week but is cracked this week, and switching from the provider that was cracked last week but is cooked this week, solve this. It’s possible that by the time I publish the post, or three microseconds from now, a new model or harness that will come out that renders these cases obsolete.

But the larger point is that sympathy, and good product design, is made up of hundreds and thousands of moments like this, where missing one is not critical, but, when looked at in its entirely, result in an app that feels “hard” to use and develop.

Mechanical sympathy, just like real sympathy, comes from an enormous context window learned over a human lifetime. People, over time, learn to get good at mechanical sympathy. Coding agents can’t. It’s (maybe) possible they will someday, until then we need to provide the intuition.

On Programming Joy and Octocat

Mon, 06 Apr 2026 00:00:00 +0000

While GitHub has been busy losing its last nine of availablility, I’ve been thinking about how the internet used to be.

Not the internet people talk about from the 90s, but the internet that we used to have even 10-15 years ago. This was the heyday of startups like GitHub, Twitter, Airbnb, and, Google was in its prime (though likely slightly past it at that point - Linus’s git tech talk there was in 2007.)

I’ve specifically been thinking about the Octocat Builder. GitHub created it back in those years, hard to say when, but it dates back to at least 2018. The Octocat mascot itself was created in 2006.

The builder is so much fun - you can select different colors of cat, hairstyles, accessories, and send to friends.

There is no reason for this to exist. Perhaps there is some branding cache that GitHub got from this, but really, someone in the company just wanted to exist, and now it does.

Something like this would have a hard time being born in today’s tech industry, I think, with its cycle of promo packets, now layoffs, anxiety about AI, and now older, sclerotic tech companies thinking about how to stay technically and socially relevant. In a world with one nine left and the rest of the app dedicated to the growth of Copilot, there is no room for a small purple Octocat.

There’s some hope: there is a little whimsy in products like Claude Code’s spinner verbs and animations, but mostly it feels like whimsy ended with ZIRP, and that makes me a bit sad. I’m happy that Octocat is still here.

NASA Elements of Engineering Excellence

Sun, 05 Apr 2026 00:00:00 +0000

I stumbled across this report from NASA, “Elements of Engineering Excellence”, published in 2012,

The inspiration for this paper originated in discussions with the director of MSFC Engineering in 2006 who asked that we investigate the question: “How do you achieve excellence in aerospace engineering?” The authors’ approach to answering this question was a short course on Excellence in Engineering which is documented in this report.

The report talks about five areas of an organization that led to failures at NASA:

Shifting from engineering “hands-on” and “excellence” to “insight/oversight”. Lack of ownership.
“Normalization of the deviances”. Not questioning anomalies.
Lack of critical thinking. Over-reliance on procedures and computer codes.
Decentralization of authority.
Organizational and technical complexity.

And I found this first point particularly relevant in the age of agents:

The five root causes are not listed in any priority order. The first root cause listed deals with a shift in the NASA culture, where the organization moved from a hands-on engineering approach to an insight/oversight approach. In the early days the heritage was basically the arsenal approach where you designed, built and verified the system before contracting it out for production. The engineers really understood the design, the hardware/software and the system based on actual experience. In the early culture much of the technology development was an in-house, hands-on activity. The shift has resulted in the elimination of much independent analysis and test and experience based understanding of the systems required to catch and prevent problems. Howard E. McCurdy in his book Inside NASA says, “NASA officials from the original cultures believed they needed to provide their engineers and scientists with hands-on experience in order to maintain the technical side of the house. It was the only way to keep them technically sharp. By keeping their own engineers and scientists sharp, they could penetrate the work of the contractor. During the first decade of space flight, a strong technical culture guided the work of NASA employees. The norms typical of that period required NASA to maintain a corps of professional employees deeply involved in the details of space flight and aeronautics. The technical culture counterbalanced many organizational forces that rose up to challenge it. It overpowered the usual bureaucratic tendencies present in government operations. It provided a counterweight to the centralizing and organizational necessities of the Apollo mission.” [McCurdy, H. 1993] The loss of the technical excellence based on hands-on experience has led to many of the problems and therefore is one of the root causes of problems. To prevent problems, NASA needs to re-establish the culture of technical excellence based on hands-on work.

Antidote

Wed, 04 Mar 2026 00:00:00 +0000

If you love building things, and the process of building is just as important to you as the result itself, it’s not unreasonable that you’re in a slump these days.

The world is telling you that your thinking process is extraneous, unnecessary, and must be commoditized and compressed. But you are multidimensional, you need room to touch the code, to explore, to rise above the local minima. In engineering, the journey is the destination. The working system in production is our reward, and we always move towards that.

Unstick yourself. Do stuff that is not the most efficient. Build something from scratch with your bare hands and your brain. Read programming books written on dead trees. Reason through something on the back of an envelope. Scribble and daydream. Join Discords where people are actively building, reviewing pull requests, building community. Write a blog post. Mentor a colleague. Answer an email from a budding engineer. Create a private repo and merge to main without a PR. Take a library or tool and look underneath the hood. Go to an in-person meetup. Pick apart the living, breathing beast that is the transformer model that governs most of our daily conversations these days, down to the attention mechanism, down to the RoPE embeddings. Explain it back to yourself.

Having agency is one of the most important things a creator can have. Give the gift of it to yourself. It’s still there, waiting for you.

Querying 3 billion vectors

Sat, 21 Feb 2026 00:00:00 +0000

Recently, I got nerd-sniped by this exchange between Jeff Dean and someone trying to query 3 billion vectors. I was curious to see if I could implement the optimal map-reduce solution he alludes to in his reply.

A vector is a list/array of floating point numbers of n dimensions, where n is the length of the list. The reason you might perform vector search is to find words or items that are semantically similar to each other, a common pattern in search, recommendations, and generative retrieval applications like Cursor which heavily leverage embeddings.

I started by writing an extremely naive implementation which made the following assumptions:

we have 3 billion searchable (document) vectors and ~1k query vectors (a number I made up)
Both of the vector sets are stored on disk in .npy format (simple format for storing numpy arrays
We’d like to compare each of the query vectors against the larger pool of document vectors and return the resulting similarity (dot product) for each of the vector combinations.
3k total reference vectors (to see if we could intially run this amount before scaling)
The vectors are of dimensionality (n) 768, a common dimensionality for many models that allow for similarity-based embedding queries

import numpy as np
from loguru import logger
import time
import os

# start with 3_000 vectors to keep things small 
total_vectors_num = 3_000_000_000
query_vectors_num = 1_000

def generate_random_vectors(num_vectors:int)-> np.array:
 logger.info(f"Generating {num_vectors} vectors...")
 rng = np.random.default_rng() 
 vectors = rng.random((num_vectors, 768)) 
 return vectors

def get_dot_products(vectors_file:np.array, query_vectors:np.array) -> list[np.array]:
 total_products_computed = 0

 dot_products = []
 for v in vectors_file:
 for qv in query_vectors:
 dot_product = v @ qv
 dot_products.append(dot_product)
 total_products_computed += 1
 if total_products_computed % 100000 == 0:
 logger.info(f"Total vectors processed:{total_products_computed}")

 return dot_products


# Generate initial vectors and query vectors and write to disk
doc_vectors = generate_random_vectors(total_vectors_num)
query_vectors = generate_random_vectors(query_vectors_num)
np.save('vectors.npy', doc_vectors)

# Load vectors from disk 
logger.info("Loading file from disk...")
vectors_file = np.load('vectors.npy')

start_time = time.time()
logger.info("Getting dot products...")
results = get_dot_products(vectors_file, query_vectors)
end_time = time.time()

logger.info(f"Execution time: {end_time - start_time:.4f} seconds")
logger.info(f"Number of dot products computed: {len(results)}")

This, predictably, didn’t do so great, even on my M2 Macbook, even at 3,000 vectors, one million times less than 3 billion embeddings, taking 2 seconds.

Results:

2025-12-13 17:53:25.675 | INFO | __main__:generate_random_vectors:9 - Generating 3000 vectors...
2025-12-13 17:53:25.691 | INFO | __main__:generate_random_vectors:9 - Generating 1000 vectors...
2025-12-13 17:53:25.698 | INFO | __main__:<module>:39 - Loading file from disk...
2025-12-13 17:53:25.700 | INFO | __main__:<module>:43 - Getting dot products...
2025-12-13 17:53:27.688 | INFO | __main__:get_dot_products:24 - Total vectors processed:3000000
2025-12-13 17:53:27.688 | INFO | __main__:<module>:47 - Execution time: 1.9877 seconds
2025-12-13 17:53:27.688 | INFO | __main__:<module>:48 - Number of dot products computed: 3000000

So I vectorized the numpy operation, which made things much faster.

import numpy as np
from loguru import logger
import time

total_vectors_num = 3_000
query_vectors_num = 1_000

def generate_random_vectors(num_vectors:int)-> np.array:
 logger.info(f"Generating {num_vectors} vectors...")
 rng = np.random.default_rng() 
 vectors = rng.random((num_vectors, 768))
 return vectors

def get_dot_products_vectorized(vectors_file:np.array, query_vectors:np.array):
 dot_products = vectors_file @ query_vectors.T
 return dot_products.flatten() # collapse into single dim

# Generate initial vectors and query vectors and write to disk
ram_vectors = generate_random_vectors(total_vectors_num)
query_vectors = generate_random_vectors(query_vectors_num)
np.save('vectors.npy', ram_vectors)

# Load vectors from disk 
logger.info("Loading file from disk...")
vectors_file = np.load('vectors.npy')

start_time = time.time()
logger.info("Getting dot products...")
results = get_dot_products_vectorized(vectors_file, query_vectors)
end_time = time.time()

logger.info(f"Execution time: {end_time - start_time:.4f} seconds")
logger.info(f"Number of dot products computed: {len(results)}")

At .017 seconds, this was a big improvement!

Results:

2025-12-13 17:52:52.810 | INFO | __main__:generate_random_vectors:9 - Generating 3000 vectors...
2025-12-13 17:52:52.831 | INFO | __main__:generate_random_vectors:9 - Generating 1000 vectors...
2025-12-13 17:52:52.874 | INFO | __main__:<module>:39 - Loading file from disk...
2025-12-13 17:52:52.876 | INFO | __main__:<module>:43 - Getting dot products...
2025-12-13 17:52:52.887 | INFO | __main__:<module>:47 - Execution time: 0.0107 seconds
2025-12-13 17:52:52.887 | INFO | __main__:<module>:48 - Number of dot products computed: 3000000

I tried a 3 million sample size with this improvement. This took 12 seconds.

2025-12-13 19:39:43.830 | INFO | __main__:generate_random_vectors:12 - Generating 3000000 vectors...
2025-12-13 19:39:57.509 | INFO | __main__:generate_random_vectors:12 - Generating 1000 vectors...
2025-12-13 19:39:58.978 | INFO | __main__:<module>:57 - Loading file from disk...
2025-12-13 19:40:00.131 | INFO | __main__:<module>:61 - Getting dot products...
2025-12-13 19:40:12.984 | INFO | __main__:<module>:65 - Execution time: 12.8491 seconds
2025-12-13 19:40:12.992 | INFO | __main__:<module>:66 - Number of dot products computed: 3000000000

We could also reduce even further by converting the data to float32:

doc_vectors = generate_random_vectors(total_vectors_num).astype(np.float32)
query_vectors = generate_random_vectors(query_vectors_num).astype(np.float32)

2025-12-13 18:13:52.152 | INFO | __main__:generate_random_vectors:10 - Generating 3000 vectors...
2025-12-13 18:13:52.168 | INFO | __main__:generate_random_vectors:10 - Generating 1000 vectors...
2025-12-13 18:13:52.176 | INFO | __main__:<module>:55 - Loading file from disk...
2025-12-13 18:13:52.178 | INFO | __main__:<module>:59 - Getting dot products...
2025-12-13 18:13:52.182 | INFO | __main__:<module>:63 - Execution time: 0.0045 seconds
2025-12-13 18:13:52.182 | INFO | __main__:<module>:64 - Number of dot products computed: 3000000

With these small improvements, we’ve already sped up inference to ~13 seconds for 3 million vectors, which means for 3 billion, it would take 1000x longer, or ~3216 minutes.

|approach | query_vectors | doc_vectors | time |
|----------- |---------------|---------------|----------|
| Naive | 1,000 | 3,000 | 1.9877s |
| Vectorized | 1,000 | 3,000 | 0.0107s |
| Vectorized | 1,000 | 3,000,000 | 12.8491s |
| Np.Float32 | 1,000 | 3,0000 | 0.0045s |

When we start to run it to test, however, we run into a different problem: OOM. Why? The amount of memory needed to process 3 billion objects, each as float32 object that’s 4 bytes in size, would be 8 million GB.


vectors = rng.random((1, 768)).astype(np.float32)
print(vectors.nbytes)
3072
print(vectors.itemsize)
4

bytes_per_float32 = 4
memory_gb = (3000000000 * 1000 * 768 * bytes_per_float32) / (1024**3)
8583068.84765625 = 8.6 TB

In order to improve this, we would need to do some heavy lifting of the kind Jeff Dean prescribed. First, we could to change the code to use generators and batch the comparison operations. We could write every n operations to disk, either directly or through memory mapping. Or, we could use system-level optimized code calls - we could rewrite the code in Rust or C, or use a library like SimSIMD explicitly made for similarity comparisons between vectors at scale.

Before I started on any further optimizations, upon further inspection, there were some things about the problem that I realized weren’t clear to me: 3 billion vector embeddings queried a few thousand times could mean:

I have a single query vector, and I query all 3 billion vectors once, get the dot product, and get all results
I have a single query vector, I query all 3 billion vectors once, get the dot product, and return top-k results, which is easier because we can do ANN search
- In this case, do I need to return the two initial vectors also? Or just the result?
- Do I need to re-rank the results by similarity in any way?
I have 1,000 query vectors, and I query all 3 billion vectors once, and get the dot product of all results
Are these vectors already in-memory when we intially start working with them or will they always be on-disk? Are we reading them one at a time, or streaming them?
What kind of machine are we assuming: Are we running this locally? What are the specs of the machine? Are we assuming the vectors come to us in a specific, optimized format?
- Do we have GPUs and are we allowed to use them?
How big are our embeddings? - this is extremely important and could significantly impact our representation, input vector size and output results
Are we assuming we can compress their representation at all, i.e. is compressiong from float64 to float32 tolerable wrt to accuracy?
How much time do we have to generate this one-off project? Are we sure it’s really a one-off?

All of these dictate the additional time and resources spent on the solution. What I realized is the same thing I’ve seen so many of these problems over the years, that the technical solution is no longer the hardest one to achieve: the hardest one is nailing down the requirements.

2025 in review

Mon, 22 Dec 2025 00:00:00 +0000

Jeune fille lisant une lettre à la bougie, Jean-Baptiste Santerre, 1700

Machine learning engineers spend their lives alternating between two states: staring at tqdm progress bars during model training and staring at error logs during model inference.

A third category now involves staring at coding agent CLI progress bars, but using too much AI assistance during coding makes me feel like I’m losing my own context window.

I started a new job as a founding MLE in March and, as is true for engineers in any small and young team, I’ve been staring at error logs all across the stack.

The good news is that if you are at a point where you care a lot about error logs, it means you have users who care a lot about the answer to “What if this breaks?” If you have people other than you who care about your software, congratulations, you are in production. Being in the state of production is the best possible outcome for software engineers because it means the work we’re doing is useful.

Standing up a new thing in production, though, is scary, because production means you are responsible - to the end-user, to the other developers on your team, and, finally, to yourself.

In “The Tombs of Atuan”, Ursula K. Le Guin writes of a girl, Tenar, who is taken at a very young age from her family and goes to live in a holy city. In a ritual ceremony, her name is changed and she loses her identity to become Arha, the Eaten One. Arha is the High Priestess of the Place of the Tombs, a vast series of caverns under the earth, silent, and in complete darkness, “full of gold and the swords of old heroes, and old crowns, and bones, and years, and silence.”

In her first trip to the tombs with an elder priestess, they descend into the dark without a torch.

“Light is forbidden here.” Kossil’s whisper was sharp. Even as she said it, Arha knew it must be so. This was the very home of darkness, the inmost center of the night. Three times her fingers swept across a gap in the complex, rocky blackness. The fourth time she felt for the height and width of the opening, and entered it. Kossil came behind. In this tunnel, which went upward again at a slight slant, they passed an opening on the left, and then at a branching way took the right: all by feel, by groping, in the blindness of the underearth and the silence inside the ground. In such a passageway as this, one must reach out almost constantly to touch both sides of the tunnel, lest one of the openings that must be counted be missed, or the forking of the way go unnoticed.

This is what building a new production system is like. You go into it, like Arha, scared, groping entirely in the darkness.

If you need a cache, you can use a Go map, lru cache, or Redis. If you need a vector store, you can use np.array, or Postgres, or Pinecone/Chroma/Turbopuffer/ or Elasticsearch. You could run your whole stack on Docker Compose, or a single bare-metal server in your basement, or Kubernetes in the cloud, or on a vape.

Forget large-scale architecture choices. How will you name things? Should you name your method get_results or get_query_results? Will you ever get results that are more than just the answer to a query? Will you ever use this method in a more abstract way? Does it need to be part of a class? What exceptions should you write for it? We want to write code that must never lie, but what if, at the time, we are telling the truth to ourselves?

How soon do we need to ship get_query_results? How many other methods will rely on it? How many of our teammates need this method for their work as well? Will they understand it? Code is meant, largely still and hopefully for the forseeable future, for humans to read first and machines to compile second.

In “The Bell Jar”, Sylvia Plath laments about the endless myriad combinations of a possible life, “I saw my life branching out before me like the green fig tree in the story. From the tip of every branch, like a fat purple fig, a wonderful future beckoned and winked.” By the time she’s done thinking through all these futures,she hasn’t made a decision, and they all slip away from her.

The forking branches of a decision tree in a codebase, are likewise boundless, and the neat part is that there is no right answer. You are constrained by your business requirements, but the choice of implementation of those requirements is of an endless variety. It will depend on: the stack you already have, the budget for the rest of the stack, your own past experience and engineering values, the social norms and expectations of your team and their collective experience, the industry’s vocabulary, the language you’re writing in, its conventions and affordances, and how much time you have to actually think about, finish, and merge this current PR before you need to deploy. That’s why staff engineers make money mostly by saying “it depends” for years on end in different ways.

Let’s say your goal is to build a scalable, distributed, microservice architecture that processes inputs and streams machine learning inference outputs to users with minimal downtime, and that the system is resilient to failures via container orchestration. I’ve just described every machine learning systems since the beginning of time, this is the machine learning system they show you in Plato’s ML engineering cave, in every single Medium post on the planet, this is the set of boxes connected by dotted lines that arrives to unbidden to every engineer in their dreams. This is the light at the end of your labyrinth.

But, in building such a system, you never start by building such a system. You have to start with a single keystroke in the inky blackness.This is one of the reasons it’s so hard to write code with LLMs and evaluate their output, by the way: humans reason from the inside out - “When you write this code, you don’t go from top to bottom, you go from chunks and you grow the chunks…” - and LLMs write code from top down.

So, instead, you take a deep breath and write a single method to tokenize a string.

def tokenize(string:str) -> list[str]:
 tokens = string.split()
 return tokens

Tokenizing a string is the process of splitting it into subsections and converting each of them into a number so they can be processed by a transformer-style model. Tokenization is not specific to LLMs. We’ve been doing tokenization for a long time. Here’s a paper on how important it is for preprocessing from 1992!

The first part of tokenizing any text is splitting it into those sub-parts. Those sub-parts can be words, syllables, or characters. We start with words to make it easier.

sentence = "The truest power is the power to choose. - Ursula K. Le Guin"

def tokenize(string:str) -> list[str]:
 tokens = string.split()
 return tokens

print(tokenize(sentence))

['The', 'truest', 'power', 'is', 'the', 'power', 'to', 'choose.', '-', 'Ursula', 'K.', 'Le Guin']

Easy, and we have something concrete. Do we count “K.” as a word? How about the punctuation, do we generally want to include that for tokenization in a transformer-style model?

What do we do with input text that has numbers in it? Do we include them or strip them out?

sentence = "The truest 1 power is the power to choose. - Ursula K. Le Guin"

print(tokenize(sentence))

['The', 'truest', '1', 'power', 'is', 'the', 'power', 'to', 'choose', 'Ursula', 'K.', 'Le Guin']

What happens when we instead get malformed inputs from a webpage, or if we have delimiters?

sentence = "The&lt;html>truest\npower\nis\nthe\npower\nto\nchoose\nUrsula\nK\nLe\nGuin"
print(tokenize(sentence))

['The&lt;html>truest', 'power', 'is', 'the', 'power', 'to', 'choose', 'Ursula', 'K.', 'Le', 'Guin']

What if the input is an empty string? Do we want to send that empty string downstream?

What if we don’t have a string to tokenize? We now get an exception that we have to think about how to handle if we’re in production.

string = None
tokenize(sentence)

Traceback (most recent call last):
 File "/Users/vicki/arha/tokenize.py", line 9, in &lt; module>
 print(tokenize(sentence))
 File "/Users/vicki/arha/tokenize.py", line 5, in tokenize
 tokens = string.split()
AttributeError: 'NoneType' object has no attribute 'split'

What if we receive emojis? This string representation returns only one token, which may not be correct in representing these two different emotions. And how do we know the model accepts emoji codepoints as input anyway? Many GPT-style models do, but many BERT models don’t.

sentence = "🤪🥰"
print(tokenize(sentence))

['🤪🥰']

What if we send an extremely large string because we don’t have upstream error handling for our tokenization queries? This code block will take a long time to return because you are creating a large object in memory. Since we don’t have logging yet, we won’t know in the rest of our application where that large object creation takes place. We will just have a web service that hangs silently, indefinitely.

num_tokens = 500_000_000
pattern = "The truest power is the power to choose. - Ursula K. Le Guin"
large_string = pattern * num_tokens

print(tokenize(large_string))
hanging forever...

What if our input includes languages other than English? How do we account for Unicode?

Sometimes, you’re able to come up with most of these edge cases at once. Mostly, though, you end up building out these edge cases over countless iterations, because software engineering is programming integrated over time.

Eventually, you build up to a codebase that has a level of thought and care around tokenization, punctuation, and input sanitization. You’ve now written a codebase at the level of tokenizers, or spacy or tiktoken.

In fact, you should probably use the libraries that people have already put collectively dozens of combined years of thought into - those corridors are closed, safe, and tested, in the case of open-source libraries, by a much larger community than just you. You import tokenizers.

In doing so, you close off a number of paths further into the labyrinth behind you, the doors darkened and abandoned. You light a single flickering torch to mark your path. On top of that code, you start adding the things that make that path more resilient (but also, then harder to refactor if you decide to pivot): precise exception handling, traces, clear human-legible logging, and unit tests. A single corridor in your application starts to shine. With the path in front fo you illuminated, you move forward.

Now that tokenization works, since you need this method throughout your application, you’ll want to create a tokenizer inference service.

Within the logic itself, if you’re using an external tokenizer, you’ll need to make sure it matches the model you’re using, because the model depends on a given tokenization scheme, In this case, we are better off relying on AutoTokenizer, which handles the import and comparison logic for us:

from transformers import AutoTokenizer

name_or_path = "arha-based-uncased"
tokenizer = AutoTokenizer.from_pretrained(name_or_path)

If you are downloading the tokenizer from HuggingFace, as AutoTokenizer does unless you specify a custom tokenizer path, you add a network call and an external dependency to the complications of your initial service call.

A new branch blooms on your decision tree. If you make this call in your service, how much latency does it add? On cold-start, can you assume the artifact will always be there? How long does it take to load the tokenizer? To stave off that branch of questions, you could instead download the artifact locally, or write your own tokenizer. Now, you’re in the business of object storage and artifact management.

You decide to tackle another question: if you’re calling the tokenizer again and again, you’ll need to create a method, or better yet, a class, because it’s highly likely you’ll want other functionality attached to the tokenizer.

And now, if you have a service, you need to Dockerize the code. If you have a Docker image, you need to pin dependencies for reproducibility. If you are pinning dependencies and have Docker, you need a build process for your deployable artifact.

The closer and closer to production you get, the more questions around “what happens if this breaks for other people who include my team and my users” start to form.

You write the service and add logs. Lots of logs. So many logs. Logs are your first line of defense in production. They are the candle in the darkness. If they’re good enough for Brian Kernighan, who wrote, “The most effective debugging tool is still careful thought, coupled with judiciously placed print statements,” they’re good enough for you.

You need to register the service into your observability platform. You add exception handling. So many Exceptions. What if the tokenizer doesn’t load? What if it loads, but to the wrong directory? What if HuggingFace is down? What if the tokenizer service can’t talk to other services? What happens if the new container doesn’t launch? What happens if the tokenizer artifact is corrupted? What happens if you’re serving the tokenizer, but the intra-service network fails? Exceptions and logging, logging and exceptions, unit tests and integration tests, layer upon layer of defenses.

Beej, who previously wrote an incredible guide to git, recently released a guide to learning computer science, in which he says that in order to be a good computer scientist, you need to think like a villain.

Expect the unexpected in terms of data that your code will receive. Expect malicious actors to feed in data in an attempt to gain unauthorized access or manipulate the system in undesirable ways. Test for that stuff in your code.

You think like a villain, because the villain in the production environment is usually yourself. With each new door, comes the possibility of falling through into a bottomless pit.

For me, this usually happens either immediately after I deploy, or at 2:17 in the morning a week from now. The tokenization method I wrote or imported fails. Or there’s a memory leak. Or there’s a network outage that causes a cascading failure.

I add more logs. I debug. I run tests. I develop locally. I add more tests. I consult and I fix, and, finally, I am back on the golden path. The alerts subside. I add more torches. The system becomes stronger, more legible from me having struggled with it. My reasoning about the system becomes stronger and clearer, too. We grow together, the system in production and me.

There is one comforting thought about being with the system: when you enter the Tombs, you are not actually truly alone. Software development is a team sport, and when we build a system as a team, we built it together, PR by PR.

And, we are also building with others: we build on the libraries and tools others before us have hardened, with the tools that others have built. In spite of the fact that an increasing amount of code is machine-assisted, software development is still mostly humans talking to each other in language that computers also understand. As we build, alone in the dark with our torches in the Tombs, we hear the echoes of others in other hallways, each carefully closing off dead ends, placing their own torches.

Once you illuminate enough of the system by reasoning through failure points, architectural decisions, the hallway is no longer unlimited darkness. It is an illuminated system in production. You keep walking around the app and harden the system. You start to get more sleep.

At the end of the year, I made it out of the Tombs.

Just in time to go back to staring at tqdm.

I want to see the claw

Mon, 20 Oct 2025 00:00:00 +0000

I respect quality software and the people who write it. And, I’ve invested years of my life in working on becoming one of these people (even if the journey has been long and hard and has involved lots of YAML). I have seen and used code written by people who care about software correctness, who get pleasure out of defining correct interfaces, who have walked the app, who have spent years working towards mastery. And, I have seen and written the code the rest of us write.

Selfishly, I want more people to aspire to mastery, because wanting to be good at writing code means you care about the code, but more importantly, about the people on the other end of the code. And, I want a world where developers care about other developers and users. I want a world of teams of seniors shipping good code and mentoring juniors and working with product managers and open-source committers. I don’t want a world where three agents in a trenchcoat from whichever AI lab is better this week quietly churn out CRUD slop that I approve out of the corner of my brain every thirty minutes.

Perhaps the first time I understood the value of mastery and care was my first job out of college, where I, a baby analyst, handed in a report to my senior analyst to check. “Did you double-check that SQL to make sure it’s correct, ” she asked. I had not, and when she ran my data, she got different results. I learned to always check my work.

A bit later, I worked with a senior architect trying to extract data into a relational database from a schemaless store. I watched him setting up for days, first preparing a staging environment and backups, then running and re-running again and again, carefully timestamping the files so he could roll back to an earlier run. Often, when I watched, his chair was empty - he was in meetings with the team who owned the database, making sure that when he hit their service to extract the data it didn’t fall over and impact them. Finally, when he was done, he wasn’t done. He checked it against the downstream reporting we were doing to make sure the numbers matched.

There is no single element that comprises this quality of care, the striving towards excellence and mastery. But, as Bernoulli said when he received an anonymous solution to the brachistochrone problem that turned out to be Isaac Newton, “I recognize the lion by its claw.”

There is a great recent post that posits that most industry programmers are impure engineers: we will always be up against business constraints and reality, and do not have the time and luxury that pure engineers, who care about only the heart of the technical solution, to care about quality.

But over and over in my career, as I tuned my alphabet radio, I came to consistently understand that intent matters. Even if we are, almost all of us, impure engineers and mere mortals and the sprint is ending in two days but I haven’t finished my t-shirt size medium task yet, the closer impure engineers strove to work at the level of purity (and did their second job of being easy to work with), the more I wanted to both work with them and use their software.

I recognize the claw of the lion in software like Redis, cURL, uv, Ghostty, sqlite, llama.cpp - software that is elegant, well-built, considered and thoughtful. Software that is joyful to use. Software that helps me. I want to write software like this, and I want to use software like this, and I want us as programming people to be incentivized to value the process that creates software like this.

It has, with generative code, become harder and harder to strive towards the lions because the models produce code that is, quite literally, mid, the compressed and weighted average of every excellent Stack Overflow answer, but also questions like, “What happens if the explanatory and response variables are sorted independently before regression?” It is the average of all publicly available software, updated at some cadence and mixed into training data soup and then RLHFed according to some arbitrary metrics, and as such can only offer a ghost of quality.

We are being overrun by mediocrity and sloppiness, we are trying to fight against it, and yet, no matter how good the models and ecosystems around them get, I find myself more and more wanting software that I know is written with humans at the wheel - we are still better at reasoning, at aesthetic judgment, at architecture.

The best code is no code, programming still sucks and always will, and yet, I find myself still searching for the claw, the mark of mastery. Because that mark comes from people who want to reach other people directly. I want to see the claw, because if there is a claw, it means there is a living, breathing lion on the other side of the screen building the software that elevates us and binds us together as a community of software engineers.

Walking around the app

Tue, 09 Sep 2025 00:00:00 +0000

There is a very vigorous debate happening online right now around what shape evaluation for LLM-based products should take. I don’t want to rehash all of it, other than saying that if you are building any applications with with non-deterministic components (most AI-powered apps), or applications that are data-intensive (also most AI powered apps!), and you are serious about these features in your application, you should at a very minimum be testing your online outputs in production on a daily basis. If offline testing (testing your given model outputs internally before it goes live in production) is a luxury you can also afford based on your engineering capacity and tooling maturity, even better!

More importantly: if you are building an app, you need to also constantly be testing and touching different parts of it to see if the app flow makes sense. Look at the UI dropdowns, try the search bar. Toggle the toggles. How long does it take to load a result? What do the results look like? What colors are the buttons? How do they work on different devices? At different internet speeds? What about your payment gateway? How does onboarding look for a brand new user in North America? In New Zealand? Could you onboard from a fresh device in under three minutes? Do images load from your CDN?

I’ve heard this kind of testing called different things in different product cultures, but the phrase that stuck with me the most was when Dan called it “walking around the app”. Walking around the app is exactly what it sounds like - checking out the sidewalks, picking up idle pieces of trash, letting the store across the street know their street lamp is out. You could call this QA, although it’s more than that. QA is scoped transactionally and takes place at the end of a PR. Walking around the app is a mindset, a process something you do every day, broadly, habitually, passionately, with interest and curiosity, because the app is the neighborhood you’re building for the people you’re building for.

Because what happens if you don’t walk around the app regularly is that you start to get broken windows. The broken window theory posits that any given signs of decay in a town make it seem permissive to create more: littering begets more littering. People understand the impact of this: For example, in real life, in the Italian town of Brescia, there is an anonymous guy who goes around at night painting over malignant graffiti.

In the applications we build, builders are both the graffiti creators and the graffiti erasers, and we have the responsibility to make our applications as habitable as possible for both our users and ourselves. What does it mean to clean up broken windows for users? Buttons that go nowhere, results pages that don’t render, links that are missing results, misspellings, cache misses, tiemouts, misaligned CSS. Deprecated routes. Latency.

How about for ourselves? Apps are easier to walk around if they are consumer-facing apps that have front-ends, but APIs and backends can be walked around, too. Naming inconsistencies. Broken local builds. A nested class inheritance hierarchy that’s multiple levels deep is a broken window because it’s hard to trace in our mind - humans can only really keep 2+5 things in memory. A hard-coded environment variable that creates extra mental load. A method that relies on an untestable internal method that results in having to use a mock. A build process so long it causes you to lose interest and tab away to Reddit. Anything that creates cognitive overhead, that Grug doesn’t like, can be a broken window.

There is not one thing that, once fixed, makes the app work well, but it becomes intuitively obvious to end-users what a well-loved app feels like, because it’s constantly being walked around and being fixed in a million small ways, every day.

We are mere mortals and will always create bugs, especially when it comes to data work. Walking the app will not prevent this - to write code is to generate bugs and the best code is the code never written.

But, if we are walking around our app every day, we’ll catch at least as many things as we can create. And if we have that in place, evaluation of any kind becomes infinitely easier to establish as a habit, too.

How big are our embeddings now and why?

Mon, 01 Sep 2025 00:00:00 +0000

A few years ago, I wrote a paper on embeddings. At the time, I wrote that 200-300 dimension embeddings were fairly common in industry, and that adding more dimensions during training would create diminishing returns for the effectiveness of your downstream tasks (classification, recommendation, semantic search, topic modeling, etc.)

I wrote the paper to be resilient to changes in the industry since it focuses on fundamentals and historical context rather than libraries or bleeding edge architectures, but this assumption about embedding size is now out of date and worth revisiting in the context of growing embedding dimensionality and embedding access patterns.

As a quick review, embeddings are compressed numerical representations of a variety of features (text, images, audio) that we can use for machine learning tasks like search, recommendations, RAG, and classification. The size of the embedding is how many features our item has.

For example, let’s say we have two butterflies. We can compare them among many dimensions, including wingspan, wing color, number of antennae. Let’s say we have a 3-dimensional feature for a butterfly, it might look like this.

Doing a visual vibe scan, we can eyeball the data and see that butterfly_1 and butterfly_3 are more similar to each other than to butterfly_2 because their features are closer together.

But butterflies are 3-dimensional animals and some of these features are numerical. When we talk about embeddings in industry these days, we generally mean trying to understand the properties of text, so how does this concept work with words? We can’t directly compare words like “bird” and “butterfly” or “fly” in a given text but we can compare their numerical representations if we map them into the same shared space. We can see that “bird” and “flying” are more similar to each other than to “dog” through their numerical representations.

Intuitively, we know this is true, but how do we artificially create these relationships?

There are different ways of creating text embeddings from a given word, all of which rely on analyzing that word in relation to the words around it in a given corpus of data. We can use traditional count-based approaches like TF-IDF based on term frequency in documents, or statistical approaches like PCA or LSA.

With the advent of deep learning models, we started learning representations generated from models like Word2Vec that maximized the probability that a left-out word would be next to other given words in the training dataset.

When we learn the embedding representation of a given word using probabilistic models, we are comparing how similar these words are to other words. Each feature is not an explicit feature like “wing color”, but rather a vibe-based latent representation in the latent space that doesn’t have a clear explanation.

For example, one dimension might be “this word is an action word” or maybe “this word is related to other words about food”, but we generally don’t know exactly what the model thinks each feature represents. In fact, this is a fascinating area of study we are just starting to understand how these latent representations work through ideas like control vectors, a concept that Anthropic explored in the famous Golden Gate Claude paper.

When we train a model, embedding size is initialized as a hyperparameter before model training, and we iterate on the size depending on our downstream evaluations after training. Picking the right hyperparameter is (alchemy) a combination of art and science and depends on optimizing training throughput, final embedding storage size, and the performance of the embedding on your downstream task both qualitatively and wrt to latency of performing search on embeddings of different sizes.

Since previous generations of models were smaller and trained in-house, hyperparameters were usually not published by companies, and as such, there was no standard agreement on embedding size. We generally, as an industry, understood that somewhere around 300 dimensions for a given embedding model might be enough to compress all the nuance of a given textual dataset. 300 was the number of dimensions typically used by earlier models like Word2Vec and GloVE.

After the publication of the attention paper BERT was released in 2018. This model’s architecture introduced embeddings of 768 dimensions. Although previous RNN and LSTM models had been trained on GPUs, BERT was one of the first larger embedding models to be trained on GPUs (and TPUs), which meant that GPU optimization now became increasingly important.

The key behind training Transformer models efficiently is the ability to efficiently move data onto GPU and parallelize their matrix multiplication operations between several pipelines, aka attention heads, where each attention head can focus on understanding and defining a different part of the embedding feature space. As such, the embedding needs to be able to be partitioned evenly between the number of attention heads.

BERT has 12 attention heads, so 768 dimensions was selected from a combination of trial and error and efficiently parallelizing computation to “attend” to different parts of the feature space, meaning that, in model training, each head operates on a 64-dimensional subspace of the original 768-dimensional input embedding. This itself comes from the Transformer paper, where each sub-embedding size per head is commonly chosen as 64.

As a result, many BERT-style models, and related model families, used 768 as a baseline for embedding dimensions. Although it had a much larger training dataset, GPT-2 also implemented 768. And, although CLIP uses an embedding size of 768 as derived from the Vision Transformer architecture that CLIP uses for its image encoder, and for consistency, the text encoder also uses this dimension size[^1].

Even though training cycles for BERT were fairly small (4 days for the original BERT model) compared to the months-long pretraining processes that LLMs require these days, it was still hard computationally to infer these embedding sizes, even with GPU optimizations. For BERT, for example,

Finding the most similar pair in a collection of 10,000 sentences requires about 50 million inference computations (~65 hours).

So in 2019, UKP created and optimized a model, SBERT, focusing on sentence-level representations, which unlocked faster inference, and Minilm became the standard baseline model for document-level chunk embeddings at 384 dimensions and still performs extremely well for a number of tasks.

What else changed was the rise of the open-source HuggingFace platform where model artifacts could be shared. Previously, they were either hosted in undiscoverable repos in GitHub or locked away in scientific repositories missing metadata. The rise of HuggingFace as a platform and the transformers library as a centralized API into training and inference for PyTorch-based models unlocked a level of standardization: now, many more people could simply download and replicate the existing models instead of rebuilding arcane research code from scratch. This led to much more standard embedding and architecture sizes used both in academia and industry.

Although 768 held for a fairly long time, with upwards pressure from market competition on model sizes as the result of the release of GPT-2 (the backbone of ChatGPT), we started seeing standard embedding sizes increase.

Part of this was the fact that companies no longer had to infer their own embeddings. Previously, embeddings had been custom-learned in labs or in companies whose core competency was processing large amounts of information that could be culled in retrieval through search or recommendation.

But now, with the advent of ChatGPT and API-based model availability, embeddings became a commodity available with a GET request, and the most popular embeddings became OpenAI’s, which used 1536 dimensions, in line with GPT-3, which used much more training data than any previous model. (570GB versus GPT-2 which was 40GB, and 96 attention heads.)

It used to be the case that people only learned or fine-tune their own embeddings, but now all of the major AI providers have their own hosted sets of embeddings. Particularly common ones are OpenAI, with Cohere and Nomic close behind. Google also recently released embeddings for Gemini, with both API and local versions being available.

In addition to standardization via HuggingFace and APIs, MTEB, which benchmarks embeddings publicly, has grown as a resource where you can compare embedding models.

If we look at MTEB, the embedding benchmark today, we can see embedding sizes anywhere from our classic 768 to 4096 and beyond (note all of them are neatly divisible by 2, in line with architectural constraints)

Finally, another consideration has been the change in the landscape with vector databases, which used to be huge, are now becoming more commoditized features in software and platforms like Postgres, S3, and Elasticsearch, leading to less out of the box necessity in vector storage and performance tuning.

With the constant upward pressure on embedding sizes not limited by having to train models in-house, it’s not clear where we’ll slow down: Qwen-3, along with many others is already at 4096. It does appear that we are beginning to trim down on growth. OpenAI implemented a concept called matryoshka representation learning that trains embeddings with the “most important” concepts first, and additional embeddings adding incremental gains, meaning that an embedding learned in 1024 dimensions might be just as useful in 64, as long as the first 64 dimensions compress most of the information efficiently, and also making sure we re-normalize them. There are also research that indicates that not all embeddings are necessary in retrieval + search tasks, and that we can in fact, in some cases, truncate 50% of them anyway.

It’s been fascinating to watch the rise of dimensionality and the constant struggle in tradeoffs between creating ever-larger models and then the need for those embedding sizes to perform at inference and storage time, aka continuously coming up against the age-old machine learning tradeoff of recall versus precision, and the engineering tradeoff of hardware limitations versus software versus business considerations.

As these architectures have matured and internal models have become inference points of public-facing paid APIs, embeddings have gone from a mysterious byproduct of internal machine learning systems in companies with lots of data to a commodity used across many AI-powered applications across numerous application stacks.

All in all, it’s been so much fun watching this space evolve, even if it means it looks like I’ll be having to update my paper over and over again.

Enabling Hugo static site search with Lunr.js

Fri, 08 Aug 2025 00:00:00 +0000

There comes a time in every woman’s life when she only wants one thing: for her mininmal static site to finally have some of the same features that dynamic blogging platforms do, namely search.

So now I’ve implemented search on this blog, you should see it in the top right and the results render in-line in the drop-down.

I’m going to tweak the results because the boosting is not quite where I’d like it to be yet, but try it out!

Some implmenetation details:

My blog runs on Hugo, so I looked for solutions that work with Jekyll/Hugo. I picked very fun and lightweight lunr.js, which is a single file that implements BM-25 search based on Solr primitives (it’s called Lunr because it’s smaller and “less bright” than Solr, which I love.)

To create the index that Lunr uses for search, I had Claude write a quick Python script that traverses my post directories and outputs a single JSON file of all indexes posts.

The entries look like this:

{
 "id": "2018/07/23/good-small-datasets/",
 "title": "Good small datasets",
 "ref": "2018/07/23/good-small-datasets/",
 "content": "John Lavery, The Chess Players (1929) I've been working on a project that, like most projects, requires testing with a dataset. My personal criteria are: + Relatively small size (Less than 100 KB, or 100ish rows) + At least 5-6 features (columns) + Should have both numerical and text-based features + Ideally a range of different kinds of numbers + Has good documentation + Is open and available to the public + Relatively available for both R and as individual CSV files or Python imports (APIs and",
 "summary": "John Lavery, The Chess Players (1929) I've been working on a project that, like most projects, requires testing with a dataset. My personal criteria are: + Rela...",
 "date": "2018-07-23T00:00:00Z",
 "creator": "",
 "site": "",
 "url": "/blog/2018-07-23-small-datasets"
 }

following this logic:

return {
 'id': slug,
 'title': frontmatter.get('title', 'Untitled'),
 'ref': f"{slug}", 
 'content': text[:500], 
 'summary': text[:160] + '...' if len(text) > 160 else text,
 'date': frontmatter.get('date', ''),
 'creator': frontmatter.get('creator', ''),
 'site': frontmatter.get('site', '')
 }

I keep that code separate from my Hugo site, in a search directory that’s initialized as a uv project:

├── README.md
├── create_index.py
├── pyproject.toml
└── uv.lock

and generates the lunr.json index into my static repo.

This code is run at build time in my GitHub Actions before the hugo build step:

- name: Install uv
 uses: astral-sh/setup-uv@v4
- name: Generate search index
run: |
 echo "Generating search index..."
 uv run search/create_index.py lunr-index.json
 echo "Search index generated successfully"

Generating the script correctly took several passes because Claude missed some data cleaning nuances (I don’t want any HTML output, I wanted slightly longer summarization outputs, I didn’t want to include base pages, etc.), but now it regenerates the index in <5 sec every time.

Then, all of that is rendered in my navbar via Hugo templates, namely by including the logic to serve the search bar in the header partial. I’m abstracting away a ton of “draw the owl” features here, such as doing passes to make sure the CSS for the search bar matches my main blog theme, and rendering the dropdown results.

LLMs helped move this feature along a fair bit here because:

It’s an extremely small feature where I can clearly test the output and see the generated code
The context window for changes was fairly small
The biggest lift was in automating index creation and getting the right syntax for the Python script and the data was easy to check.
It has a tight local testing loop: hugo build and hugo serve are extremely fast locally and offer 95% parity to the served site in GH pages.

This has been a fun addition and I’m excited to do more such as potentially add typeahead suggestions and tune the query.

My favorite use-case for AI is writing logs

Wed, 16 Jul 2025 00:00:00 +0000

One of my favorite AI dev products today is Full Line Code Completion in PyCharm (bundled with the IDE since late 2023). It’s extremely well-thought out, unintrusive, and makes me a more effective developer. Most importantly, it still keeps me mostly in control of my code. I’ve now used it in GoLand as well. I’ve been a happy JetBrains customer for a long time now, and it’s because they ship features like this.

I frequently work with code that involves sequential data processing, computations, and async API calls across multiple services. I also deal with a lot of precise vector operations in PyTorch that shape suffixes don’t always illuminate. So, print statement debugging and writing good logs has been a critical part of my workflows for years.

As Kerningan and Pike say in The Practice of Programming about preferring print to debugging,

…[W]e find stepping through a program less productive than thinking harder and adding output statements and self-checking code at critical places. Clicking over statements takes longer than scanning the output of judiciously-placed displays. It takes less time to decide where to put print statements than to single-step to the critical section of code, even assuming we know where that is.

One thing that is annoying about logging is that f-strings are great but become repetitive to write if you have to write them over and over, particularly if you’re formatting values or accessing elements of data frames, lists, and nested structures, and particularly if you have to scan your codebase to find those variables. Writing good logs is important but also breaks up a debugging flow.


from loguru import logger

logger.info(f'Adding a log for {your_variable} and {len(my_list)} and {df.head(0)}')

The amount of cognitive overhead in this deceptively simple log is several levels deep: you have to first stop to type logger.info (or is it logging.info? I use both loguru and logger depending on the codebase and end up always getting the two confused.) Then, the parentheses, the f-string itself, and then the variables in brackets. Now, was it your_variable or your_variable_with_edits from five lines up? And what’s the syntax for accessing a subset of df.head again?

With full-line-code completion, JetBrains’ model auto-infers the log completion from the surrounding text, with a limit of 384 characters. Inference starts by taking the file extension as input, combined with the filepath, and then the part of the code above the input cursor, so that all of the tokens in the file extension, plus path, plus code above the caret, fit. Everything is combined and sent to the model in the prompt.

The constrained output good enough most of the time that it speeds up my workflow a lot. An added bonus is that it often writes a much clearer log than I, a lazy human, would write, logs. Because they’re so concise, I often don’t even remove when I’m done debugging because they’re now valuable in prod.

Here’s an example from a side project I’m working on. In the first case, the is autocomplete inferring that I actually want to check the Redis URL, a logical conclusion here.

redis = aioredis.from_url(settings.redis_url, decode_responses=True)

In this second case, it assumes I’d like the shape of the dataframe, also a logical conclusion because the profiling dataframes is a very popular use-case for logs.

from pandas import DataFrame

data = [['apples', '2.5'], ['oranges', '3.0'], ['bananas', '4.0']]
column_names = ['Fruit', 'Quantity']
df = pd.DataFrame(data, columns=column_names)

Implementation

The coolest part of this feature is that the inference model is entirely local to your machine.

This enforces a few very important requirements on the development team, namely compression and speed.

The model has to be small enough to bundle with the IDE for desktop memory footprints (already coming in at around ~1GB for the MacOS binary), which eliminates 99% of current LLMs
And yet, the model has to be smart enough to interpolate lines of code from its small context window
The local requirement eliminates any model inference engines like vLLM, SGLM, or Ray which implement KV cache optimization like PagedAttention
It has to be a model that’s fast enough to produce its first token (and all subsequent tokens) extremely quickly,
Finally, it has to be optimized for Python specifically since this model is only available in PyCharm

This is drastically different from the current assumptions around how we build and ship LLMs: that they need to be extremely large, general-purpose models served over proprietary APIs. we We find ourselves in a very constrained solution space because we no longer have to do all this other stuff that generalized LLMs have to do: write poetry, reason through math problems, act as OCR, offer code canvas templating, write marketing emails, and generate Studio Ghibli memes.

All we have to do is train a model to complete a single line of code with a context of 384 characters! And then compress the crap out of that model so that it can fit on-device and perform inference.

So how did they do it? Luckily, JetBrains published a paper on this, and there are a bunch of interesting notes. The work is split into two parts, model training, and then the integration of the plugin itself.

The model is trained is done in PyTorch and then quantized.

First, they train a GPT-2 style Transformer decoder-only model of 100 million parameters, including a tokenizer (aka autoregressive text completion like you’d get from Claude, OpenAI, Gemini, and friends these days). They later changed this architecture to Llama2 after the success of the growing llama.cpp and GGUF community, as well as the better performance of the newer architecture.
The original dataset they used to train the model was a subset of The Stack, a code dataset across permissive licenses with 6TB of code in 30 programming languages
- The initial training set was “just” 45 GB and in preparing the data for training, in data cleaning, for space constraints, they remove all code comments in the training data specifically to focus on code generation
- They do a neat trick for tokenizing Python (using a BPE-style tokenizer optimized for character pairs rather than bytes, since code is made up of smaller snippets and idioms than natural language text) which is indentation-sensitive, by converting spaces and tabs to start-end <SCOPE_IN><SCOPE_OUT> tokens, to remove tokens that might be different only because they have different whitespacing. They ended up going with a tokenizer vocab size of 16,384.
- They do another very cool step in training which is to remove imports because they find that developers usually add imports in after writing the actual code, a fact that the model needs to anticipate
They then split into train/test for evaluation and trained for several days on 8 NVidia A100 GPU with a cross-entropy loss objective function

Because they were able to so clearly focus on the domain and understanding of how code inference works, focus on a single programming languages with its own nuances, they were able to make the training data set smaller, make the output more exact, and spend much less time and money training the model.

The actual plugin that’s included in PyCharm “is implemented in Kotlin, however, it utilizes an additional native server that is run locally and is implemented in C++” for serving the inference tokens.

In order to prepare the model for serving, they:

Quantized it from FP32 to INT8 which compressed the model from 400 MB to 100 MB
Prepared as a served ONNX RT artifact, which allowed them to use CPU inference, which removed the CUDA overhead tax(later, they switched to using llama.cpp to serve the llama model architecture for the server.
Finally, in order to perform inference on a sequence of tokens, they use beam search. Generally, Transformer-decoders are trained on predicting the next token in any given sequence so any individual step will give you a list of tokens along with their ranked probabilities (cementing my long-running theory that everything is a search problem).

Since this is computationally impossible at large numbers of tokens, a number of solutions exist to solve the problem of decoding optimally. Beam search creates a graph of all possible returned token sequences and expands at each node with the highest potential probability, limiting to k possible beams. In FLCC, the max number of beams, k, is 20, and they chose to limit generation to collect only those hypotheses that end with a newline character.

Additionally they made use of a number of caching strategies, including initializing the model at 50% of total context - i.e. it starts by preloading ~192 characters of previous code, to give you space to either go back and edit old code, which now no longer has to be put into context, or to add new code, which is then added to the context. That way, if your cursor clicks on code you’ve already written, the model doesn’t need to re-infer.

There are a number of other very cool architecture and model decisions from the paper that are very worth reading and that show the level of care put into the input data, the modeling, and the inference architecture.

The bottom line is that, for me as a user, this experience is extremely thoughtful. It has saved me countless times both in print log debugging and in the logs I ship to prod.

In LLM land, there’s both a place for large, generalist models, and there’s a place for small models, and while much of the rest of the world writes about the former, I’m excited to also find more applications built with the latter.

20 years of YC

Mon, 17 Mar 2025 00:00:00 +0000

I saw recently that YCombinator celebrated its 20th anniversary.

Hacker News is slightly younger, but to me the two go hand in hand.

As far as I can tell, I actively started reading Hacker News around 2011. I don’t remember how I heard about it. It was probably on Reddit or Digg. Once I found it, I started reading every day, mostly because the comment sections were so full of smart people in tech.

At the time I was working as a data analyst, mostly with SQL and Excel. I understood that I would need to learn much more to move into engineering, which I was getting excited about, but didn’t quite understand how to bridge the gap.

When I first started reading HN, I didn’t understand 99% of the linked content, the jargon in the discussions, or the companies mentioned, but I was determined to learn. My general approach was to skim the top headlines and headlines that I could understand, read the post, and then read the HackerNews discussion.

As I read the discussion, I would come upon terms I had no idea about: Big O notation, collaborative filtering, caching, Hindley-Milner type systems, lambda architectures, CI/CD pipelines, cryptography, generics, build or buy, B-trees, Bloom filters, trunk-based development, red/green deploys. I would look all of them up and go down countless rabbit holes.

HN is a bit bigger these days, but, thanks to contributors and the tireless efforts of dang, still so perfectly ambiently captures what tech is thinking about in the current moment. If you don’t have a supportive community at work － like many of us data scientists － HN was the perfect ambient watercooler to be near senior technically excellent people in the industry.

In the first few years, I read maybe 2 links out of the 30 ever on the front page. But, because I read HN 4-5 times a day, after a few years, things started falling into place.

Other than constantly studying, Hacker News was one of the main things that pulled me up out of my bootstraps from a non-technical major afraid of SSHing into a server, to a person confidently deploying code to millions of users. There is only one other thing that has brought as much value to my career in tech, and that was the connections I made on old Twitter.

Over time, I gained enough confidence to start blogging on technical topics, HN gave me something new: being aware that HN might pick apart that content forced me to learn to write clearly and precisely for a highly technical, educated audience.

My favorite feeling is when, via a link, something I have written or created has made an impact on someone out there in the ether. Or, from the reader side, when I read something that makes me go “I’m not alone. This person has also thought about these problems,” and I can’t wait to see if there are any good discussions on the topic.

YC is proud of the companies it launched: Reddit, Airbnb, Instacart, Doordash, Stripe. Ironically, the biggest multiples it’s generated for the industry have been from a low-dazzle text forum written in Arc Lisp.

Thank you for everything, HN.

You can just hack on ATProto

Thu, 23 Jan 2025 00:00:00 +0000

Icon by iconixar

Since I signed up for Bluesky last year, I’ve been wanting to make something using the AT Protocol that the platform is built on top of.

I finally had a chance to do it over the holiday break and built GitFeed, a small Go app that filters the Bluesky network firehose by posts that have GitHub links and renders them into a refreshable, ephemeral feed.

You can see GitFeed here, but it might not actually be running since I didn’t build it for scale, and it’s running on a tiny DigitalOcean Droplet that’s specced out at 1 GB Memory / 1 Intel vCPU / 35 GB Disk , is entirely unloadbalanced, un-load-tested and has zero observability or alerting when my very scientific process of running nohup fails.

The cool part of an open protocol is that you can also just clone the repo and run a hosted version of it yourself.

ATProto

Bluesky is both a decentralized protocol, called AtProto and a social media company, called Bluesky plc that develops both the protocol and one of the Apps running on the protocol, Bluesky.

There is a lot more in the AT Protocol Paper, but the basics are this:

Data for each user is hosted individually in a PDS - a personal data store - which is a database storing a collection of records that are cryptographically signed and encoded in DAG-CBOR format. The record schema is defined by a “lexicon”, which is dependent on the type of data being transferred. The records themselves are stored in a merkle search tree structure which makes it easy to rebalance records efficiently both on read and write. Its default storage engine is SQLite.
Each user has a PDS, exposed as a web service to network indexers. There is an indexer, the relay, which “scrapes” but really hits all the PDSes in the network for updates. Right now there is only one true relay, run by Bluesky the company, and there is a lot of debate around what that means for a decentralized network and efforts to diversify and decentralize. To better get a sense for how the data model works, you can play around with this tool, which is what I spent a lot of time doing during this project.

The TL; DR is that you can think of the At Proto Atmosphere as a collection of databases, or, really, websites, that the relay indexes and turns into the firehose. Data is then filtered on the firehose side for CSAM and other logic, before it’s turned into an AppView. The AppView is what you see if you sign into bsky.app.

If this sounds familiar, it’s because it’s how web crawlers, including Google work, with the exception that their crawled results are not available to everyone for access.

Steve has a very nice write-up of all of this, with a beautiful ascii diagram.

Al(most) all of the data streaming through each person’s PDS is public, and enables the creation of projects like the Bluesky firehose as a screensaver, or goodfeeds, surfacing feeds across the network., or TikTok and Instagram-like apps. As you can imagine, the protocol then lends itself to a lot of nice experimentation (make sure to check the TOS/Developer guidelines before you do so).

Let’s find all the gists

Initially, I wanted to create a custom feed on Bluesky. Generally, people create these by filtering the network to include all feeds about cats, or feeds from only mutual follows, or one of my recent favorites, gift articles, which includes links from gift articles that you can click through to read.

My idea was: collect all the posts that have a link to github gists, because people put really cool stuff in gists, so I could find and expose to other users some really cool code snippets of what people are hacking on around the platform.

Initially I thought I might be able to create a lightweight recommendation feed based on aggregate likes. Or, I could create a trending links feed. But, if you want to do any machine learning, you need to start consuming the firehose at scale, collecting the data, and setting up storage, and I wanted to learn Go, not implement distributed systems - don’t use N computers when you can use one.

Moreover, in order to implement a feed, generally, you need to also implement:

pagination
a cursor in case you lose your place in consuming the feed
latency considerations
Thinking about how you render feed objects (eventually they become large and need to be hydrated)
potentially ranking that feed’s content
retries and handling for if you skip feed elements

Additionally, specific to atproto, your feed is published at your own PDS - Personal Data Server, a LOT more about this here. You can see this at the link on the Gift Articles Feed:

https://bsky.app/profile/did:plc:o4s55v3tsfph6whswxccpsia/feed/aaaixbb5liqbu

Given that the feed is published and linked to your own data store, it made me hesitant to experiment in case I messed something up and lost all of my data.

There are lots of clients to create and publish feeds, but as a nerd, I didn’t want to have a third party handle this out of principle. Moreover, for gists only, there was not enough data to be interesting. The Bluesky network is growing, having surpassed 29 million users. But, at this scale, interesting content when you filter at this level of granularity is sparse.

After messing around with the (excellent) Python client a bit in this repo, I narrowed down to the problem I actually wanted to solve:

consuming all links with "github.com" in the link name, and consuming them via jetstream rather than the firehose.

That’s how GitFeed was born.

Why GitFeed? Why Go?

McFunley says you only have so many innovation tokens. This also applies to side projects. The other way I’ve seen this explained, is that you can either pick a new language, a new stack, a new business problem to solve, or new people to work with, but not all four at the same time, otherwise you will never ship.

I always want to get to a demo quickly. But, the goal of side projects is to do stuff you wouldn’t get to explore otherwise, so I chose to spend my innovation tokens on:

learning a new backend language, Go: this thing had to serve code fast as a binary and have a very simple deployment story. I have a background in Java/Scala and initially thought about Java, but unfortunately there is no small light-weight Java server that I’m aware of (Spring doesn’t count) and Go has steadily been growing in popularity since its inception and especially after the language added features like generics and I wanted to check it out. Just as importantly, the backend of Bluesky and the protocol were specced out and written in Typescript and Go, and I thought it might be easier to navigate the Atmosphere in those languages.
getting slightly better at front-end dev: I’d dabbled a bit with front-end design for Viberary but used almost no Javascript, which I’d need to render a feed.
understanding the At Proto data model

I didn’t need to:

Understand the problem of working with a feed of streamable content - I did a lot of feeds work when I worked on recommendations at Tumblr and I find it to be one of the most fun and satisfying problems in engineering to work on.
Understand how to run/deploy on DigitalOcean using GitHub Actions - I already did this for Viberary and the path was pretty smooth
Understand SQLite
Finally, I wanted to track and catalogue my usage of LLMs for this project. I’m constantly evaluating their use-cases.

The Jetstream

I didn’t want to spend time implementing pagination, hydration, latency mitigation, and a data storage strategy. I wanted to get going quickly, so I decided to use the Jetstream. Jetstream is a relatively new content source for Bluesky content (relatively because everything is brand-new and being built on the fly.)

Working with the Bluesky firehose has a set of complications other than pagination: it also has its own data formats (CBOR and CAR for all the full merkle trees in the git repos) that take time in learning how to parse.

Moreover, the sheer volume of firehose events has grown to the point where folks consuming it need to invest heavily in scaling strategies for downstream application consumers.

Jetstream instead streams content in JSON, with reduced bandwidth and costs.

As a tradeoff, Jetstream is less stable, doesn’t contain content that needs API verification, and don’t offer pagination, activity offets, or uptime guarantees. As the docs say, it’s good for low-stakes side projects that don’t require heavy authentication or veracity, aka gitfeed.

Serving GitFeed

We’re building a simple web app with two components: that are two separate go processes:

An ingest go module that will ingest posts and writes them to a SQLite DB that we instantiate as part of the application
A serve go module that serves the app via API.

The serving module is just a few Javascript files with static HTML pages. I was very overwhelmed by all the choices in the front-end ecosystem (although the developer docs helped a ton!), so I just ended up going with plain old Vanilla JS.

When we refresh the site, we make a call to the DB (via the posts API endpoint) to surface the posts in reverse chronological order.

export async function fetchPosts() {
 const container = document.getElementById('postContainer');
 container.innerHTML = '<div class="loading">Loading posts...</div>';
 try {
 console.log('Fetching new posts...');
 const response = await fetch('/api/v1/posts');
 if (!response.ok) {
 throw new Error(`HTTP error! status: ${response.status}`);
 }
 const posts = await response.json();
 container.innerHTML = '';

And here’s the actual call site:

import {
 fetchPosts,
 updateTimestamp
} from './feed.js';

console.log('Main.js loaded');

document.addEventListener('DOMContentLoaded', async () => {
 console.log('DOM Content Loaded');
 try {
 await fetchPosts();
 await updateTimestamp();
 } catch (error) {
 console.error('Error in main initialization:', error);
 }
});

I was surprised at how much about JS was already familiar to me from Python and PHP, but where I really got stuck was in understanding how the DOM and Javascript work together, what a Javascript app structure looks like, and the Javascript ecosystem.

Using LLMs to build GitFeed:

I’ve done a lot of work with llamafile, and recently, I’ve also been enjoying the local LLM stack of: Ollama for the model backend and OpenWebUI for the front-end. Ollama serves versioned GGUF model weights wrapped in a Docker-like paradigm that hits an API wrapping llama.cpp in an (of course) Go interface. For this project, I used mistral:latest and qwen2.5-coder:latest , the best code model at the time (in the ancient space of 3 months ago, Deepseek3 wasn’t out). I did reasonably well between the two of them, with only 5% of requests that I had to bypass and send to Claude. I did find myself getting frustrated because I couldn’t clearly articulate the unknown unknowns I had about Javascript, though, and eventually I just gave up and bought Eloquent Javascript which I’m hoping to dig into this year to better understand what Qwen and I wrote together and how I can improve it.

Ingest

Consuming the Jetstream is easy. There are four public instances of Jetstream, hosted by Bluesky the company, so we need to connect to one of these, consume the content, process and filter the data, and save it to our database for serving.

You can check what’s up in the Jetstream with this nifty command line tool:

websocat wss://jetstream2.us-west.bsky.network/subscribe\?wantedCollections=app.bsky.feed.post | jq .

This is everything people are posting on Bluesky! (It sometimes gets very, very weird so be careful if you don’t want to look at NSWF texts.)

And now let’s look at GitHub posts. There aren’t a lot of them, so you might not get an even for at least a few minutes:

websocat wss://jetstream2.us-west.bsky.network/subscribe\?wantedCollections=app.bsky.feed.post | grep "github" | jq .

Jetstream is implemented as a websocket connection to the source.

wsManager := NewWebSocketManager(
 "wss://jetstream2.us-west.bsky.network/subscribe?wantedCollections=app.bsky.feed.post",
 pr,
)

A websocket is a protocol (like HTTP) that enables client-server communication over TCP but works best for streaming data without the need for continuous polling or webhooks (unlike HTTP).

A connection is instantiated with a handshake between the client and the server in HTTP first, and when the request is processed, both switch to websockets for communication.

I used the gorilla/websocket implementation of the websocket procool. for handling the core websocket logic.

We need to be able to read and write to/from the websocket:

The main thing to internalize about working with WebSockets in Go is that each client connection should get at least two goroutines: one that continuously processes messages coming from the client (i.e., a “read pump”), and one that continuously processes message going out to the client (i.e., a “write pump”).

However, this becomes easier since for GitFeed, we’re only reading from and not writing to the websocket, we need to implement logic to readPump

func (w *WebSocketManager) readPump(ctx context.Context) {

	w.Connect(ctx)
	counter := 0
	for {
		select {
		case <-ctx.Done():
			log.Printf("Exiting readPump: got kill signal\n")
			return
		default:
			var post db.ATPost
			if err := w.conn.ReadJSON(&post); err != nil {
				w.Connect(ctx)
				continue
			}
			counter++
			if counter%100 == 0 {
				log.Printf("Read %d posts\n", counter)
			}

			// Process the post
			dbPost := ProcessPost(post)

			if err := w.postRepo.WritePost(dbPost); err != nil {
				w.errorHandler(fmt.Errorf("failed to write post: %v", err))
				continue
			}
			log.Printf("Wrote Post %v", dbPost.Did)
		}
	}
}

However, unlike an HTTP call, websocksets are open persistently and don’t offer any guarantees of retries, so we have to implement this logic ourselves. Fortunately, gorilla has a lot of good examples.

You’ll notice a couple key points here: first, we log and handle the case where the web socket disconnects. Then, we do some ultra-fancy print logging to keep track of how many posts we’ve actually processed. And finally, we now get to the actual data, an var post db.ATPost that we process, parse, and write to the database, which we initialize as a post repository, a fancy word for “database with dependency injection”.

Golang tooling:

Go just works out of the box. Unlike my beloved Python, it doesn’t need uv, formatting, linting, or special build processes. At least, for a fairly small project, everything is batteries included. In fact, its boringness and rigidity allowed me to move really quickly. What surprised me is that I thought that VSCode would work really well with go, but actually didn’t as code I imported wouldn’t get loaded automatically, and there were a couple bugs that made me switch to Goland, which works extremely smoothly, without fail, and its local autocomplete at the line level is much better than PyCharm’s equivalent, likely because Go is much smaller, and statically-typed.

Jetbrains LLMs:

Jetbrains’ local LLMs are extremely well-done and I’d encourage anyone interested to check out the paper.

What is an AtProto Post

Now we get to the heart of the matter: once our websocket is open, we are ingesting a stream of JSON objects. AtProto has its own data model, defined using schemas called “Lexicons”. For posts and actions, they look like this.

{
 "did": "did:plc:eabmaihciaxprqvxpfvl6flk",
 "time_us": 1725911162329308,
 "kind": "commit",
 "commit": {
 "rev": "3l3qo2vutsw2b",
 "operation": "create",
 "collection": "app.bsky.feed.like",
 "rkey": "3l3qo2vuowo2b",
 "record": {
 "$type": "app.bsky.feed.like",
 "createdAt": "2024-09-09T19:46:02.102Z",
 "subject": {
 "cid": "bafyreidc6abdkkbchcyg62v77wbhzvb2mvytlmsychqgwf2xojjtirmzj4",
 "uri": "at://did:plc:ab7b35aakoll7hugkrjtf3xf/app.bsky.feed.post/3l3pte3p2e325"
 }
 },
 "cid": "abfyreidwaivazkwu67xztlmuobx35hs2lnfh3kolmgfmucldvhd3sgzcqi"
 }
}

The DID is the ID of the PDS (user repository) where the action happened, the record collection type of app.bsky.feed.post is what we care about, and each record has both a text entry, which truncates the text, and a facet, which has all the contained links and rich text elements in the post.

{
 "did": "did:plc:",
 "time_us": 1735494134541,
 "type": "com",
 "kind": "commit",
 "commit": {
 "rev": "",
 "type": "c",
 "operation": "create",
 "collection": "app.bsky.feed.post",
 "rkey": "",
 "record": {
 "$type": "app.bsky.feed.post",
 "createdAt": "2024-12-29T17:42:14.541Z",
 "embed": {
 "$type": "app.bsky.embed.external",
 "external": {
 "description": "",
 "thumb": {
 "$type": "blob",
 "ref": {
 "$link": ""
 },
 "mimeType": "image/jpeg",
 "size": 
 },
 "title": "",
 "uri": ""
 }
 },
 "facets": [
 {
 "features": [
 {
 "$type": "app.bsky.richtext.facet#link",
 "uri": ""
 }
 ],
 "index": {
 "byteEnd": 85,
 "byteStart": 54
 }
 }
 ],
 "langs": [
 "en"
 ],
 "text": "..."
 },
 "cid": ""
 }
}

So we parse these JSON objects and store them as Go structs. Go has a super handy tool where you can paste a JSON object and get back the Go struct.

And then we write that struct to a DB with a posts table that we’ve already instantiated for this purpose.

if err := w.postRepo.WritePost(dbPost); err != nil {
 w.errorHandler(fmt.Errorf("failed to write post: %v", err))
 continue
}

SQLite

Enough has been written about why SQLite is awesome and amazing for smaller, and even larger projects so I’ll skip that here and say that I didn’t even consider using anything else for GitFeed.

There are a ton of optimizations you can perform on SQlite to really juice performance, and I set a few of them in anticipation of many users hitting the server, and also easing read-write contention on the db.

Serving

After working extensively with FastAPI and Flask, I was really excited to learn just how batteries-included net/http module was in Go. I didn’t need to install anything extra - I was immediately writing routes and handlers.

We set up several routes to deal with our saved posts here.

// Start post service
	fmt.Println("Connect to post service...")
	pr := db.NewPostRepository(database)
	postService := &handlers.PostService{PostRepository: pr}

	// Create web routes
	routes.CreateRoutes(postService)

	log.Printf("Starting gitfeed server...")
	log.Fatal(http.ListenAndServe(":80", nil))

And then we’re up and running as soon as we build and run our Go executable.

Building and Running Go Artifacts

So easy. Just a small Makefile and we’re rebuilding and testing, and then serving the binaries.

Data Considerations

I don’t want to keep any data or manage it, this is meant to be an ephemeral snapshot, so a goroutine deletes the oldest data once there are more than 10 posts in the database.

GitHub

One last piece we need, which early GitFeed testers suggested, was a way to, after you load the posts from the database, render the associated GitHub metadata with the repo, so we also, after we load the posts from the DB, hit the GitHUB API to enrich the post.

Ops Considerations

I run all of this on a small DigitalOcean droplet, and redeploy to the droplet with new code code merged to main via GitHub actions. There’s no monitoring or alerting, something I’d like to add for the future.

Final Reflections

This app was so much fun to develop and I learned an enormous amount of stuff. Small, self-contained apps are a joy, and especially when there’s a front-end component where you have a self-reinforcing feedback loop. Since us machine learning engineers work at what a friend called “the back-end of the backend”, we don’t often get to experience UI changes, and seeing through something end-to-end was a joy.

There were points of friction: It was definitely frustrating getting up and going with a whole new language and tech stack, but once I got back into the flow, it was great.

As always, the hardest part of this project, as with any project, was understanding the data model and the business logic, and parsing out those objects correctly. The second-hardest was aligning elements in CSS.

I’d love to get to a point where the app can surface Trending GitHub repos.. And maybe add some unit tests.

Upon writing this, I realized I have like three other posts I want to write about this process, so I’ll leave this as-is for now.

How FastAPI path operations work

Tue, 14 Jan 2025 00:00:00 +0000

If you’re building a new Python web app these days, there’s a good chance you’re using FastAPI. There are a lot of features that make FastAPI easy to get started with. There are also a lot of nuances that take a while to understand. One feature I’ve been untangling is the way FastAPI manages calls to API routes via decorated path parameters. The new year is a perfect time to take a deeper dive.

What happens in a web server

When we build a web app, one of the critical components is a web server, a program that listens for incoming requests from the network. It then translates those requests into methods that are called in the backend.

To better understand what’s going on under the covers, we can first implement a simple web server using the http.server module included in Python’s standard library.

We need to write a program that listens on a port and accepts HTTP requests. It accepts the request, parses the path route, and parses any data attached to the HTTP call. Or, “All I want is to cURL and parse a JSON object”.

import json
from http.server import BaseHTTPRequestHandler, HTTPServer
from urllib.parse import urlparse, parse_qs

class RequestHandler(BaseHTTPRequestHandler):

 def parse_path(self, request_path: str)-> dict:
 """
 Parse request path
 """
 parsed = urlparse(request_path)
 params_dict = parse_qs(parsed.query)
 return params_dict

 def store_urls(self, request_path: str)-> None:
 """
 Parse URLs and store them
 """
 params = self.parse_path(request_path)
 for key, val in params.items():
 self.data_store.put_data(val[0])

 def return_k_json(self, k:dict)-> BinaryIO:
 """
 Return json response
 """
 self.send_response(200)
 self.send_header("Content-type", "application/json")
 self.end_headers()
 # Contains the output stream for writing a response back to the client. 
 # BufferedIOBase that writes to a stream
 # See https://docs.python.org/3/library/io.html#io.BufferedIOBase.write
 self.wfile.write(json.dumps(k).encode('utf-8'))

 def bad_request(self):
 """
 Handle bad request
 """
 self.send_response(400)
 self.send_header("Content-type", "application/json")
 self.end_headers()

 def do_GET(self):
 
 request_path = self.path

 if self.path == "/":
 self.return_k_json({"ciao": "mondo"})
 if request_path.startswith("/get"):
 key = self.parse_path(request_path)
 self.return_k_json({"jars": key["key"]})
 self.send_response(200)
 else:
 self.bad_request()
 self.end_headers()

 def do_POST(self):
 request_path = self.path

 if request_path.startswith("/set"):
 self.store_urls(request_path)
 self.send_response(200)
 else:
 self.bad_request()

if __name__ == "__main__":
 host = "localhost"
 port = 8000

 server = HTTPServer((host, port), RequestHandler)
 print("Server started http://%s:%s" % (host, port))
 server.serve_forever()

What’s going on here?

Let’s say that we produce Nulltella, an artisinal hazlenut spread for statisticians, and are looking to build a web app that keeps track of all of our Nulltella jars so we can later stand up a prediction service.

We would start by designing a super simple API: As users,

we want to test the server and get back a simple response
we’d like to add jars to our inventory, and
to see the jars we added.

We translate these actions to GET and PUT requests so we can write HTTP calls for them. For simplicity’s sake, we won’t actually store them server-side but we will write them so we can can very simply see how to send data to our app:

We want to test the server:

> python serve.py
> curl -X POST http://localhost:8000/
> {"ciao": "mondo"}

We want to store items:

> curl -X POST http://localhost:8000/set\?key\=8
200 OK

And get back the stored items:

> curl -X GET http://localhost:8000/get\?key\=8 
> {"jars": ["8"]}

Our server needs a way to parse the key pieces of information it receives:

They type of request. do_GET and do_POST handle this implicitly in the HTTP implementation.
The parameters we pass to the path request so that we can do something with them
A route to a method inside our application itself that processes the data

In our simple server, the heart of the routing happens at the method level. If we send a base path, we return {"ciao": "mondo"} . Otherwise, we return the amount of jars we’ve passed in via the request path by parsing the parameters in the path.

def do_GET(self) -> None:
 request_path = self.path

 if self.path == "/":
 self.return_k_json({"ciao": "mondo"})
 if request_path.startswith("/get"):
 key = self.parse_path(request_path)
 # action performed within the web app here
 self.return_k_json({"jars": key["key"]})
 self.send_response(200)
 else:
 self.bad_request()
 self.end_headers()

We can see how this can become complicated quickly. For example, what if we have multiple operations we perform during a GET : what if we get data from a database, or a cache, or we retrieve assets? We’ll have different methods that we process depending on how the path is parsed. What if we also have PUT/DELETE verbs? What if we need authentication? To write to a database? Static pages? Our code complexity relative to our starting point starts to grow, and we now need a framework.

Starlette

Early Python web dev frameworks include juggernauts Django and Flask. More recently, since Python’s async story has grown stronger, frameworks like Starlette have come onto the scene to include async functionality out of the box.

Starlette was built by the creator of Django Rest Framework and includes lightweight operations for the core functionality of HTTP calls and additional operations like web sockets, with the added bonus of being async by default.

To manage an HTTP call the same way we would with our simple server, we can do the following with Starlette:

from starlette.applications import Starlette
from starlette.responses import JSONResponse
from starlette.routing import Route

async def homepage(request):
 return JSONResponse({'ciao': 'mondo'})

app = Starlette(debug=True, routes=[Route('/', homepage),])

We start an instance of a Starlette application, which has processes routes. Each route is linked, at the path level, to the actual method it calls. If Starlette sees that specific route, it calls the method, taking into account logic for parsing and reading HTTP request headers and bodies.

What if we want to add a second method call based on a different route, getting our jar count again?

from starlette.applications import Starlette
from starlette.responses import JSONResponse
from starlette.routing import Route

async def homepage(request):
 return JSONResponse({'ciao': 'mondo'})

async def get_jars(request):
 return JSONResponse({'jars': ['8']})

app = Starlette(debug=True, routes=[
 Route('/', homepage),
 Route('/get_jars', get_jars)
])

We see that we are also passing and processing params, and there is logic that processes the path params based on the method as they come in from the request.

FastAPI’s implementation

FastAPI wraps Starlette - “as it is basically Starlette on steroids” per the docs - and includes Pydantic type validation at the logical boundaries of the application.

Under the covers, when we instantiate a FastAPI application, it’s really “just” an instance of a Starlette application with properties that we override at the application level.

from fastapi import FastAPI

app = FastAPI()

@app.get("/")
async def root():
 return {"ciao": "mondo"}

@app.get("/jars/{id}")
async def get_jars(id):
 return {"message": f"jars: {id}"}

In development, FastAPI uses uvicorn , an ASGI server to listen for incoming requests and handle them according to the routes defined in your application.

Uvicorn initializes the ASGI server, binds it to a socket connections on port 8000 , and starts listening for incoming connections. So, when we send a GET request to the main route hosted by default on port 8000, we expect to get back ciao mondo as a response.

Like our previous applications, FastAPI is still delegating path operations and methods to a router that processes them and parses parameters, but it wraps these in a Python decorator. This is easier to write, but adds a level of complexity at the layer of understanding how the path processing actually happens.

When we perform a path operation in FastAPI, we’re performing the equivalent work of routing that we do with our simple method, but with a lot more rigor and nested definitions.

Within our simple server, we:

Start the server
Listen on port 8000 for incoming requests
When we receive a request, we route it to the do_GET method
Depending on the path of the request, we route it to "/"
We return the results to the client via a 200 status

In FastAPI, we:

Start the uvicorn web server (if in development mode, if production we have to choose gunicorn using the compatible worker class)
Listen on port 8000 for incoming requests
We instantiate an instance of the FastAPI application
This in turn instantiates an instance of Starlette
When we receive a GET request, it’s routed to the application’s self.get method
This in turn calls self.router.get with the path operation
The router is an instance of routing.APIRouter
The .get method on APIRouter takes the path and retunrs return self.api_route. This is the point where the decorater is actually called - we can see the decorator in that method takes a DecoratedCallable function as input and returns a decorated add_api_route, which actually appends the route to the list of routes.

This is purely the set of steps that happens for correct routing - and we didn’t yet address how the path parameters in the path are processed.

Path Parameter Routing

Path parameter routing happens in Starlette, where path parameters are parsed out of the request into a dictionary (just like we do in our simple web application), via the magic of Jinja Templating.

TL; DR

When we write a route in FastAPI that accepts path parameters, we are creating a lengthy callstack that goes through several levels of logic in FastAPI using decorators as input into an application that routes requests and appends methods using decorators to a group of route methods; those requests are then passed onto Starlette which does the work of parsing the path variables, using Jinja templates, into dictionaries which the application can then work with and return data to you!

Everything I did in 2024

Fri, 03 Jan 2025 00:00:00 +0000

I want to get back into writing more regularly this year, so in light of that, here’s my last year in review.

Evaluating LLMs

Like many of us in tech, I spent a large portion of 2024 thinking about and working with LLMs, but I was lucky enough to do it for work. I spent the year designing, building, open-sourcing, (and naming! 🐊) an application to evaluate LLMs, Lumigator.

In support of that work, I did open-source work in the LLM ecosystemand learned a ton of stuff along the way. Just when I thought I had reached machine learning enlightenment, I learned a ton about the weird nondeterministic properties of LLMs, their evaluation methods, Ray + Ray Serve, vLLM, Llamafile, GGUF, FastAPI, OpenAPI-compatible APIs, and much, much, much more. I am hoping to translate some of these into blog posts as well.

There is always more to learn in machine learning, particularly in the fast-moving world at the bleeding edge. Lucikly, the fundamentals always remain the same.

My biggest takeaways from this year are that:

Machine learning in industry is mostly still made up of good engineering, and in order to work with LLMs, you need to have both machine learning context and engineering discipline. There’s some debate whether the “old” machine learning skills (gradient boosted trees anyone? 👵) still apply in this Brave New World of NonTabular Scraped Internet, and I am of the opinion that these skills are more valuable than ever, particularly as engineers without ML context enter the frustrating world of building with non-deterministic experimental loops.

But what’s even more valuable is being able to ship things that hold engineering rigor. Even if you are deep in academic research, taking an engineering approach will make your life significantly easier.

As part of this, I spent a significant amount of the year widening my own context window about engineering best practices: Over the summer, I learned C and this fall, I learned JavaScript and Go and built my first working project, Gitfeed. I am particularly enamored with Go: it’s so boring, so clean, and so easy to build things that are simple and move quickly out of the box compared to the Python ecosystem.

This is not to say that the Python world, which is my first love, is not changing and evolving. I moved to uv from pyenv for both work and personal projects in 2024 and have never looked back, particularly as uv continues to work to gain support for PyTorch and all of its cuda-related install issues.

There is too much going on in the model landscape: too many models, too much choice, too many platforms, too much money, too much drama. My main strategy is to follow what’s going on on a daily basis to keep my finger on the pulse and then basically ignore it. If people mention it again over the course of 2-3 months, continue to pay attention.

In light of this, some contradictary advice if you’re in the space is that you should also always be playing around with stuff, trying it out, building it and breakign it. There are some classic papers and books for understanding the context and theory, but most of the stuff I read is still in r/LocalLLaMA or in various tweets/skeets and blog posts. We are not at the level yet for this stuff where cannon exists, although I have noticed that there are technical books being published about AI engineering this year, which means we are starting to solidify.

Some LLM tools I tried out and loved this year were llamafile, for getting started with GGUF transformer and embedding models with a server extremely quickly, and ollama + openwebui for an experience that is nearly identical to using Claude. In fact, I’ve switched over from Claude to using mistral:latest locally for most of my LLM usage, which is basically code search. Mistral is a model that has consistently worked well for me both at work and for my own purposes, and I’ve enjoyed the chance to pit it against newer models like Qwen2.5 Coder, Llama2, and phi.

I am still having a hard time wrapping my mind around what LLM models are useful for, in general, and for me personally. I spend a lot of time reading blog posts about how other (smarter) people use LLMS and against academic papers that study the success of these models at machin learning tasks stemming in their roots as NLP models focusing on problems like text completion, classification, translation, and summarization.

A very cool and fun related thing I did while reviewing the “classics” was edit this wonderful series of posts by Katharine about model memorization which you should read if you’re interested in their internals.

I also spent a significant part of the year working on web development. I wrote earlier in the year that there is a bifurcation in how we consume LLMs:

The LLM ecosystem is currently bifurcated between HuggingFace and OpenAI compatibility: An interesting pattern has developed in my development work on open-source in LLMs. It’s become clear to me that, in this new space of developer tooling around transformer-style language models at an industrial scale, you are generally conforming to be downstream of one of two interfaces: models that are trained and hosted using HuggingFace libraries and particularly the HuggingFace hub as infrastructure - in practicality, this means dependence on PyTorch’s programming paradigm, which HuggingFace tools wrap (although they now also provide interop between Tensorflow and JAX)

Models that are available via API endpoints, particularly as hosted by OpenAI. Given that OpenAI was a first mover in the product LLM space, they currently have the API advantage, and many tools that have developed have developed OpenAI-compatible endpoints which don’t always mean using OpenAI, but conform to the same set of patterns that the Chat Completions API v1/chat/completions offers. For example, adding OpenAI-style interop chat completions allowed us to stand up our own vLLM OpenAI-compatible server that works against models we’ve started with on HuggingFace and fine-tuned locally.

If you want to be successful in this space today and you’d like to cater to a broad audience, you as a library or service provider have to be able to interface with both of these.

I still believe this is true and it will be interesting to see how this develops over the course of the next year as APIs and their model ecosystems become even broader.

These models will integrate with, not replace traditional machine learning systems. In the beginning, everyone thought that we could replace whole classes of engineering and machine learning problems with LLMs. It’s becoming increasingly clearer that one big-ass model is not as a good as many smaller models for specific tasks performed in industry, which makes complete sense because LLMs as a concept rose out of academic labs whose goals are to generalize on out-of-sample problems, whereas the task of industry is to specialize in healthcare or OCR document scanning for legal, or sentiment analysis for social media. Even if fine-tuning did decline as a practice this year, people are making models specialized in a lot of deifferent ways, which I think the rise of agents at the end of this year really makes clear.

An area that has particularly interested me is the integration of LLMs into recommender systems, my favorite area of applied machine learning. Earlier last year, I, along with James and Ravi, started a position paper on what will change in recommender systems in the light of LLMs based on a joke tweet I had made,

personalized recommendations based on implicit matrix factorization from data acquired through large log streaming architectures were a low interest rate phenomenon

My hypothesis was that companies used to collect a lot of streaming user data, particularly in social and other B2C settings, that were then used for personalization. Now, all the data collection happens publicly at the level of the interet, where it is re-compressed into LLMs. What does this mean for personalization, now?

Unfortunately we didn’t finish the post (maybe someday?), but our main hypothesis was that traditional recsys will not be replaced with, but instead augmented with LLMs, that LLMs could help fill the cold-start gap because they’re so good at zero-shot retrieval and general topic classification and can serve as places to augment cold-start problems and in onboarding, and that LLMs could offer more explainability in recommendations. This has so far proven to be true.

Italy + Learning Italian

I had the extreme pleasure of being invited to give a keynote at PyCon Italia in Florence in May. I took all these learnings that had been running on crazed hamster wheels in my brain and threw them into a talk based loosely on the Decameron.

In addition to subjecting my poor audience to jokes about the Medicis, I also gave the intro in Italian to an audience of +100, which was extremely nervewracking. I’ve been wanting to study Italian since I was 18 and the stars aligned two years ago for me to seriously start. and I’ve been learning ever singe. This year, I also managed to read my first (Level A1/A2) books in Italian.

My family also came to Italy. It was the kids’ first trip to Europe. Everyone had an amazing time (at least after my talk was over), ate a lot of gelato, and experienced Rome and Florence in the spring. One of my favorite moments of the trip (and of my year) was listening to a Ricchi e Poveri cover band playing at a disco outside our hotel at 11:30 at night in Florence.

Inspired by all of this, and to take a break from LLMs, I also started learning how to mix music, and hopefully will have something more concrete to write/share about DJing this year.

I also managed to read some books.

As I spent the year trying my best to keep up with the wild, unruly, and growing LLM landscape, another part of my ecosystem was wilting. It’s clear for all intents and purposes that Twitter as we knew her is dead. It doesn’t have the juice anymore, as the kids say. I’m surprised that it died from human intervention rather than engineering failure, but as anyone who’s read my newsletter knows, this is by far the most likely outcome for techno-social systems. I have been wanting to write a long eulogy on my personal experinece and what Twitter meant to me: it was a place where I made friends, learned to be a serious programmer, worked out my best ideas, got leads for jobs, started a newsletter, organized Normconf, and most importantly, had fun. But in the end, it all died with a whimper, and I, like many others, just quietly left.

I’ve always been a firm believer in owning your own internet space, and I continued to do that by blogging, and moving from Substack to Buttondown, but I also need micro social interaction because I get energy and ideas from it. So, I engaged more on Bluesky, where I’ve been since last year, and was pleasantly surprised to see the machine learning community start to migrate there. Amost immediately as soon as I came back to the platform, I had a conversation that led to an idea for a post.

A lot of people have (rightfully so) sworn off agglomerated social in favor of group chats. This makes complete sense to me, yet as a creator and someone who thrives on community, I was sorely missing this space until I came back to Bluesky. There is a lot of conversation around protocol versus platform and what it means for Bluesky the app versus Bluesky the protocol to succeed.

In particular, I encourage anyone interested in the space to read this series of exchanges on the question but I remain hopeful that that space grows and remains a place where users can choose and experiment. It allowed me, at the end of the year, to hack on gitfeed, which is very, very very cool.

2025

What next? Who knows, who can even predict the future? Not machine learning models, and certainly not me. I’m just excited to keep learning and building and making and dreaming, and, hopefully writing more about it all.

Write code with your Alphabet Radio on

Mon, 16 Dec 2024 00:00:00 +0000

There is a lot of debate in the software community around whether LLMs can replace developers. Part of the reason is the way we formulate the problem of what it means to write software. In industry, we still give outsize cultural deference to software developers as lone wizards who come into the room, put on their hoodie, crank up the techno, and write the application on their own.

When we sit down at our keyboard, we assume that we sit down alone, with our precious treasure trove of programming knowledge that we have built up over the years, hoarding the nuggets about byte size and data types like a dragon guarding its gold. It is us and our hard-earned brain patterns from years of algorithm classes and hundreds of pages spent trawling StackOverflow versus the machine, working alone, against each other.

We sit down, we crack our knuckles and deserialize the knowledge from our brain into clear streams of beautiful code, in solitude and blessed silence. These days, we are spending less time with books and search results and more time with ChatGPT or Claude, and it’s this prompting process, we assume, that will ultimately replace us. But that’s only true if we consider the developer as an individual unit of work separate from anything else, without our own context window.

Machine learning is compression and LLMs doubly so. When we sit with an LLM, we are not only asking the average of the internet but the average, as Karpathy puts it, of every human data labeler that has contributed to moving LLMs forward.

When we ask LLMs to write code, we get back compressed representations of the entire field, optimized via RLHF, sorted by descending probability, and limited by the size of the model’s context window. So we are not alone, but we are also often don’t ascend outside a local minimum of average code, sometimes not even outside the local minima of suckiness. Not to mention, these models have no access to hundreds and thousands and millions of lines of elegant code that has never seen the public internet because it sits quietly, undiscoverable in corporate git servers. When we ask the model, almost any model (with the exception of finetunes on our own data), we are peeking into a very small window of what good code could be.

When I sit down at the keyboard, I also sit down with the average of the local minima, but I also have a secret superpower. I’ve had the enormous good fortune to work with and learn from very good developers, and it’s the average habits of all of these developers that lift me out of the depths of compressed scraped GitHub.

These developers’ good habits stream from my memories into my consciousness. As I write code, they enrich my decision-making process, like a radio station of good advice that never turns off.

A insists that I need to learn fundamental algorithms and data structures inside and out because they are the basis for what makes good code fast. “Even if you never implement a LinkedList in your own code, you’ll be able to more clearly reason through the decisions others have made and use the libraries that make the most efficient use of them.”

B tells me, through hundreds of PR comments over years and years, that it’s not good enough to have a PR that just passes tests. The code needs to be clean, reasonable, and legible for others because we read code more than we write it.

C tells me to write code that is elegant. C says it will take longer to write this kind of code than something that just ships, but that it is our professional responsibility as developers to carve out that time, to demand it. “Write your code, then write a second time. Make it work, make it right, make it fast.”

D tells me not to mess with fancy tools. A simple print debug statement goes a long way. A 2-line unit test fixes 100 lines of obtuse code. Code completion editors don’t work when you don’t know what you want to autocomplete, yet.

E tells me to master my tools, to dig into problems I don’t understand, to get to a reproducible example, to play with things in the terminal, take them apart, write smaller and smaller pieces of code until they make sense.

F tells me to take time to analyze the data correctly, and to be crisp and concrete about my analysis.

G and H tell me to start without machine learning if I can, and if I have to use it, to use a simple model first of all.

I tells me to get to a local development loop as soon as possible.

J tells me to take my craft seriously, but never myself.

K tells me to be patient and have grace, both for others, and for myself. Code is hard, but humans are harder.

Above all of them is Grug, constantly whispering “complexity bad” with every line I add.

Over and over my Alphabet Radio plays my greatest hits, fixing a variable name, digging through a class hierarchy, adding a test, using dependency injection, avoiding unnecessary libraries, deleting lines of code until the number of lines I add is less than the number I subtract, refactoring, refactoring, refactoring.

I am so lucky to have my Alphabet Radio. Engineers like me who come from data land sometimes don’t get the privilege of building a playlist like this because we don’t work in teams, and for a long time at the beginning of my career, I was one of these people.

It was very lonely and I felt wrong a lot of the time without guardrails or intuition as to why what I was doing was inefficient and incorrect. LLMs don’t and can’t help with this part. Repeated exposure to the best practices of good developers does.

When I am at the keyboard, my joyous, cacophonous radio station keeps streaming the greatest hits on an infinite loop and allows me to rise above the local minimum, encouraging me to work harder to produce code that is not just functional, but easy for others to read, fast to modify, and doesn’t break for longer.

Nothing is black and white. Code is not precious, nor the be-all end-all. The end goal is a functioning product. All code is eventually thrown away. LLMs help with some tasks, if you already know what you want to do and give you shortcuts. But they can’t help with this part. They can’t turn on the radio. We have to build our own context window and make our own playlist.

When LLMs can stream advice as clearly and well as my Alphabet Radio, then, I’ll worry. Until then, I build with my radio on.