steel

💥 “They Would Never Use the Death Star on Us”: Alderaan Residents Reflect on Their Support for the Empire as a Large Imperial Installation Enters the System ↗

2026-05-05T20:11:00Z

Jack Loftus over at McSweeny's:

Lesser of two evils. The Senate was ineffective, and the liberal Jedi were out of touch. The Emperor said he’d cut through all that. And he did—sometimes literally. You have to give him that. Things moved. Maybe a little too much moving right now, with the Death Star repositioning every few minutes to maintain a firing solution on our planet, but still.

Yeah.

🏷️ Tags: Fun, Politics, Links

📄 Permalink to this post

I Built My Own Hair Electrolysis Machine

2026-04-27T00:00:00Z

Analogical reasoning

2026-04-26T04:00:00Z

Turns out, competition works

2026-04-23T07:33:20Z

I’m settling into Austin here and preparing a move into my new apartment. As I was working through internet setup, I was impressed to find an all-you-can-eat menu of four easy fiber options for internet, with symmetrical gigabit starting at $60 and no data caps.

Last year when I left San Francisco, I was paying over $100/month for a 100 mbps / 20 mbps cable connection, no better than what I’d first gotten in the late 90s, and with a strict 1.2 TB cap (overage cost: $10 every 50 GB).

I’d tried to find another ISP, any other ISP, but there were none. Previously I’d had MonkeyBrains, but this landlord wouldn’t allow the invasive rooftop antenna installation, and the unit didn’t have good line of sight anyway. Not even the other dinosaurs like AT&T provided service.

Why were there no alternatives? Because “progressive” San Francisco banned them, handing a de facto monopoly on a silver platter to Comcast, which was more than willing to take advantage of that by offering 1990s speeds, punitive data caps, and sky high pricing.

In Austin, competition is legal. Comcast (a garbage company full of garbage people) would love to charge $100 for a 100 mbps connection with no upload, but they offer $70 fiber instead. Why? Because every resident in the building has 3+ other options to choose from. They could try to pull the same racket here, and wouldn’t get a single taker. Turns out, competition works. Who could possibly have known.

On a related note, the apartment I’m renting is $200 cheaper than the San Francisco rent I paid 15 years ago back in 2011. The unit’s in a better neighborhood (right on the park), is larger, considerably better built (concrete between units so you aren’t sharing TV programming with your neighbors), and with full amenities. How? Because again, unlike “progressive” SF, building is legal. It’s so legal that despite booming population growth, rents between 2021-25 are down 4%, or a whopping 19% inflation adjusted.

My Nix Config Is Intimate

2026-04-22T00:00:00Z

Caveman

2026-04-12T16:41:03Z

An excerpt from Michael Crichton’s Congo (1980):

“I don’t understand,” Elliot said. Ross explained that the “M” meant that there was more message, and he had to press the transmit button again. He pushed the button several times before he got the message, which in its entirety read:

REVUWD ORGNL TAPE HUSTN NU FINDNG RE AURL SIGNL INFO-COMPUTR ANLYSS COMPLTE THNK ITS LNGWGE.

Elliot found he could read the compressed shortline language by speaking it aloud: “Reviewed original tape Houston, new finding regarding aural signal information, computer analysis complete think it’s language.” He frowned. “Language?”

Crichton was a gear guy. The story’s protagonists took high tech satellite uplinks into the field, allowing transmission back to HQ, but due to the extreme expense of satellite bandwidth, having to read messages in shorthand like, “REVUWD ORGNL TAPE HUSTN NU FINDNG”.

I always found it ridiculous. Although these words have had their vowels removed, they’re still uniquely intelligible in the English language. It’d be trivial to write a short algorithm that’d use a dictionary to expand the message back to uncompressed English on the receiving end. Or better yet, stop with the vowel thing and use a standard compression algorithm ¹. You’d get better results.

Yesterday, I came across Caveman. Its job is to save tokens in Claude by having the LLM speak like a caveman, removing filler words and other niceties that make up a more fluently legible human language.

Before:

“Sure! I’d be happy to help you with that. The issue you’re experiencing is most likely caused by your authentication middleware not properly validating the token expiry. Let me take a look and suggest a fix.”

After:

“Bug in auth middleware. Token expiry check use < not <=. Fix:”

Crichton would’ve loved it. 45 years later we’ve come full circle, are back to speaking like cavemen again, and as an at-least-somewhat legitimate technical workaround. I don’t know what I thought I knew anymore.

🔊 Hostile Volume ↗

2026-04-10T04:11:00Z

Just set the volume to 25%. Just fold in the cheese. UX is my passion.

🏷️ Tags: UX, Fun, Links

📄 Permalink to this post

🏎️ International Chess Federation Adds Race Car Piece ↗

2026-04-10T03:46:00Z

The Onion:

In all officially sanctioned matches played from today forward, the pawn immediately in front of a player’s king will be replaced with a sick little hot rod that can move any number of squares horizontally, vertically, or in a circle like it’s doing donuts[.]

Delightful piece. Do pay them a visit to learn all the new rules.

🏷️ Tags: Chess, Fun, Links

📄 Permalink to this post

🎲 Claude mixes up who said what, and that's not OK ↗

2026-04-10T03:34:00Z

Gareth Dwyer, in a post about LLMs not keeping track of who said what:

Yes, of course AI has risks and can behave unpredictably, but after using it for months you get a ‘feel’ for what kind of mistakes it makes, when to watch it more closely, when to give it more permissions or a longer leash.

Sure, you can get a “feel” for what kind of mistakes LLMs make, but that feeling means absolutely nothing and cannot be trusted. Randomness is literally built in. The temperature setting of an LLM determines how random the token selection process will be, and every token selection is a fork in the road for the rest of the output.

This kind of widespread misunderstanding about how LLMs work terrifies me. To be clear, I’m not picking on Gareth specifically—this is just a good example of the misplaced trust in LLMs I see running rampant these days. These systems are intentionally designed to be nondeterministic, with randomness as a key component of how they work. Thinking you can get a “feel” for when you can trust them is like getting a “feel” for how the dice are rolling or a “feel” for how the cards are being shuffled.

🏷️ Tags: Generative AI, Slop Generators, Links

📄 Permalink to this post

🔴 FBI Extracts Suspect’s Deleted Signal Messages Saved in iPhone Notification Database ↗

2026-04-10T03:00:00Z

Joseph Cox at 404 Media:

The FBI was able to forensically extract copies of incoming Signal messages from a defendant’s iPhone, even after the app was deleted, because copies of the content were saved in the device’s push notification database, multiple people present for FBI testimony in a recent trial told 404 Media.

9to5Mac has some additional details if you hit 404 Media’s paywall.

There’s a setting in Signal to prevent actual message content from showing up in notifications, but this person didn’t have it enabled.

Stay safe out there, kids.

🏷️ Tags: Apple, iPhone, iOS, Security, Links

📄 Permalink to this post

🎮 Porting Mac OS X to the Nintendo Wii ↗

2026-04-09T01:33:00Z

Bryan Keller got the first version of Mac OS X (10.0, Cheetah) running on a Nintendo Wii:

Before figuring out how to tackle this project, I needed to know whether it would even be possible. According to a 2021 Reddit comment:

There is a zero percent chance of this ever happening.

Feeling encouraged, I started with the basics […]

Hell yeah. That’s a great attitude to have. And it paid off!

Something about seeing Mac OS X running on a Wii seems correct to me in a weird way. Whenever I saw Mac OS X running on other things, like a hackintosh, it always felt a little off, but the Wii and Mac OS X seem like kindrid spirits.

If you want to learn a lot about how old versions of Mac OS X or the Wii work, read Bryan’s entire post. If you want to skip all the technical details, I understand, but the last paragraph is a good reminder for everyone:

In the end, I learned (and accomplished) far more than I ever expected - and perhaps more importantly, I was reminded that the projects that seem just out of reach are exactly the ones worth pursuing.

🏷️ Tags: Apple, Mac OS X, Nintendo, Wii, Links

📄 Permalink to this post

💻 This Is Not The Computer For You ↗

2026-04-08T02:17:00Z

Sam Henri Gold, in a wonderful review of the MacBook Neo:

Nobody starts in the right place. You don’t begin with the correct tool and work sensibly within its constraints until you organically graduate to a more capable one. That is not how obsession works. Obsession works by taking whatever is available and pressing on it until it either breaks or reveals something. The machine’s limits become a map of the territory. You learn what computing actually costs by paying too much of it on hardware that can barely afford it.

When the first version of World of Warcraft was released in 2004, the PC I had at the time didn't meet the minimum system requirements. Not by a long shot. My CPU was too slow, I didn't have enough RAM, and my graphics card was a generation or two out of date.

I bought World of Warcraft and forced it to run anyway.

Even with the resolution cranked way down, the frame rate was abysmal. Visual glitches were abundant. But it ran. I was in Azeroth. It was blurry, fuzzy, and glitchy, but I was doing quests, finding loot, and chatting with other players.

It wasn't good enough. Every update or change—even tiny ones—tended to break everything and require hours of adjustments, experimentation, and tweaks to get things working again. I installed so many different drivers. I made so many registry and INI file tweaks. So, so many. Even when the game was technically working, I had to avoid certain areas because they were too visually demanding or too crowded and would cause a crash.

It was, in many ways, terrible. But terrible got me in the door. Terrible taught me a lot about how games worked, about how graphics drivers worked, about how Windows worked, and about how computers in general worked. Terrible forced me to figure out how to wring every last bit of performance from outdated hardware that had every right to flip me the bird as it committed thermal suicide.

The MacBook Neo is far from terrible. If anything, it’s far more capable than most people think when they hear things like “iPhone chip” and “8 GB of RAM.” It does have more limits than any other current-gen Mac, but it’s also scrappy, fun, extremely well-designed, and makes the Mac accessible to a much larger audience.

When it comes to performance and specs, there’s no way anyone can argue it’s the best Mac. When it comes to people and practicality, it’s the best Mac Apple has made in years.

🏷️ Tags: Apple, Mac, Craft, Constraints, Links

📄 Permalink to this post

"Somewhere" (2010) review

2026-04-07T00:09:44Z

I’ve often cited Sofia Coppola’s Lost in Translation (2003) as one of my favorite movies. I’d never dug much into Coppola’s other work, so imagine my delight to discover that she’s made another movie, Somewhere (2010) with a similar premise.

I excitedly got to watching it, but was ultimately disappointed. There’s room for two movies to have similar premises, but Somewhere takes that to another level. It’s functionally the same film.

The macro/themes are the same – disengaged, burned-out actor stays long-term at a hotel. A young woman comes into his life with whom he feels a genuine human connection. She helps break his sad routine and rediscover joy. One is in Tokyo, one is in LA. In one the woman is a much younger stranger, in the other his daughter.

But overarching story aside, even specific scenes are strongly derivative:

There’s an absurdist foreign interview of the lead in each.
Both heavily feature scenes of characters lying in beds.
Each has meta-scenes of leads watching TV.
Both include a scene of another woman sleeping over with the lead, and the awkward morning after interaction with the young woman about it.
There are scenes of the characters swimming around in upscale hotel pools.
Each has a scene of the lead watching strippers.

I understand having a few callbacks in there to the filmmaker’s previous work, but this is something else.

Lost in Translation is clearly the distantly better movie. My takeaway is that although it had a good script, Bill Murray and the overwhelming chemistry between him and Scarlett Johansson carried that movie. Switch out those two leads, and it’s very possible that like Somewhere, almost no one would have heard of it.

at arm’s length

2026-04-05T04:00:00Z

The Second Wave of the API-first Economy

2026-03-27T15:11:08Z

Fifteen years ago, when some colleagues and I were building Heroku’s V3 API, we set an ambitious goal: the public API should be powerful enough to run our own dashboard. No private endpoints, no escape hatches.

It was a stretch, but it worked. A new version of the company’s dashboard shipped on V3, and an unaffiliated developer who we’d never met before built Heroku’s first iOS app on it, without a single feature request sent our way.

The first wave

Our dashboard-on-public-APIs-only seems needlessly idealistic nowadays, but it was an objective born of the time. The year was 2011, and the optimism around the power of APIs was palpable. A new world was opening up. One of openness, interconnectivity, unbounded possibility.

And we weren’t the only ones thinking that way:

Only a year before (2010) Facebook released its original Open Graph API, providing immensely powerful insights into its platform data.
Twitter’s API at the time was almost completely open. You didn’t even need an OAuth token — just authenticate on API endpoints with your username/password and get access to just about anything.
GitHub was doing really impressive API design work, providing an expansive, feature-complete API with access to anything developers could need, and playing with forward-thinking ideas like hypermedia APIs/HATEOAS.

You can still find traces of this bygone era, standing like some cyclopean ruins from a previous age. Hit the root GitHub API and you’ll find an artifact over a decade old — a list of links that were intended to be followed as hypermedia:

$ curl https://api.github.com | jq

{
  "current_user_url": "https://api.github.com/user",
  "current_user_authorizations_html_url": "https://github.com/settings/connections/applications{/client_id}",
  "authorizations_url": "https://api.github.com/authorizations",
  "code_search_url": "https://api.github.com/search/code?q={query}{&page,per_page,sort,order}",
  "commit_search_url": "https://api.github.com/search/commits?q={query}{&page,per_page,sort,order}",
  "emails_url": "https://api.github.com/user/emails",
  "emojis_url": "https://api.github.com/emojis",
  "events_url": "https://api.github.com/events",
  ...

This wasn’t a pre-planned, stack-ranked feature that a product team spent half a year putting together. It was one or two early engineers who got really excited about an API idea, and shipped it, probably without even asking for permission.

Part of the push for open APIs was simple good will towards the rest of the world. The engineers building them were brought up in the earliest days of the internet, steeped in its original counterculture, and had an innate bias for radical openness.

There was also a feeling from the companies involved that the APIs would be beneficial for their bottom lines. Users and third parties would use APIs to supplement the core product with add-ons and extensions that’d drive growth and increase product retention and satisfaction.

Sites like the now defunct ProgrammableWeb popped up to discuss and catalog the newly appearing APIs, and the “programmable web” wasn’t only a website, it was a principle.

In the near future, all platforms would be API-first, providing full programmatic access and opening a new wave of interoperability across the web that’d let any service talk to any other service and massively accelerate the scope and reach of the internet. APIs would help expand everything from freedom to communication to commerce. An overwhelming force for good in the world.

API winter

Of course, it didn’t last. The programmable web went through a phase of expansion, reached its maximum extent, and began to contract.

Twitter’s famous API, which used to be an API tinkerer’s dream, leveled off and began to dip as the company struggled to find ways to generate revenue. New features no longer got first-class API treatment. Access to the firehose was closed. Third-party Twitter clients were restricted and eventually locked out.
The power of Facebook’s Graph API was hugely constricted post-Cambridge Analytica where a single rogue app was able to suck up data on millions of users and put it up for sale. Strict app review procedures were implemented. The API went from open access to a walled garden.
Even more extreme, Instagram’s previously public API was deprecated totally. Realizing they had a real money maker on their hands, they saw no reason to share ad revenue with anyone else. Use Instagram through the first-party app or not at all.
Even APIs like GitHub’s that stayed quite open had to crack down to a degree. Endpoints became authenticated by necessity and aggressive rate limiting was put in to curb abuse and reduce operational toil. And even when APIs were still largely accessible, using them to build a full-scale third-party app became more difficult as limiters flattened heavy (even if legitimate) use.

The rationale for why APIs were being declawed or disappearing completely varied—abuse, monetization pressure, competitive risk, privacy, etc.—but the pattern was clear. Walls were going up across the world.

APIs didn’t disappear, but it was a cold winter for them. The expectation of an API became more limited to developer-focused platforms whose users paid them — Stripe, Twilio, Slack, etc. When new consumer products appeared on the market (e.g. TikTok), no one expected them to have much in the way of an API.

The coming second wave

For many years this was the status quo. If you were using Twitter, you’d use it from Twitter.com. Facebook, from Facebook.com. Instagram or TikTok, from their respective iOS/Android apps. Developer products like GitHub and Stripe continued strong, but elsewhere, APIs weren’t enough of a competitive advantage for anyone who didn’t have one to suffer.

But around mid-2025, the world changed. The last half year especially has been distinguished by the rise of indescribably powerful LLMs, which now dominate discourse as the most useful new tool in a generation.

They’re already useful enough as incredible trivia machines or code generators, but they really start to shine when they integrate with things. It’s pretty neat having one generate a valid Kubernetes configuration for your new app, but it’s really neat watching it provision an EKS cluster via awscli and send out its first production deploy on your behalf.

Suddenly, an API is no longer liability, but a major saleable vector to give users what they want: a way into the services they use and pay for so that an agent can carry out work on their behalf. Especially given a field of relatively undifferentiated products, in the near future the availability of an API might just be the crucial deciding factor that leads to one choice winning the field.

Picking my future bank

Let’s think about banks. I have a couple bank accounts, each offering a standard set of features largely unchanged since the 60s. If I call them, they’ll send me some checks. I can request a transfer between two internal accounts and they will transfer the money … in 1-5 business days. Nowadays, they even offer ultra-modern features (from 2010) like gasp, MFA, just as long as it’s through a provider that’s paid them off (Symantec VIP). Suffice it to say, they’re comfortable in the status quo. My banks do not have good APIs.

So far this has worked out okay for them. People aren’t known to migrate banks often, and even if they did, regulatory moats make new incumbents rare.

But in the modern age, can it last? When I want to move $100 from one bank to another, my banks put me through a humiliating ritual of logging into both accounts, and bypassing multiple security checks and captchas before I can perform any operation. All this despite me having just logged into both accounts from this exact location and biometrically-secured computer the day before.

The world I want is to instruct an LLM: “move $100 from Wells Fargo checking to Charles Schwab brokerage” and it will just happen. And to be fair, LLMs are already so absurdly good at reverse engineering things that this might already work today. But you know what’d work better? If both banks shipped with APIs, LLM-friendly usage instructions (through MCP or the like), and a strong auth layer to give me confidence that the whole process is secure.

If I were choosing a bank today, some considerations would be the same as they’ve always been—competent security, free checking, no foreign transaction fees—but I’d also futureproof the choice by picking one that’s established technical bona fides by providing an API. Even if I’m not quite ready to trust my banking credentials to an agent quite yet, I assume that this day is coming.

Ubiquitous again

Now apply the same principle to every service you use during the course of a week, or ever:

Online marketplaces: Robot, schedule my normal Amazon Fresh order for the first available slot tomorrow morning.
Office co-working: Robot, book me a desk at Embarcadero Center today.
Ski resorts: Robot, buy me a day pass for tomorrow and load it to my resort card. Confirm the price with me first.
Restaurants: Robot, put in my usual lunch order at Musubi Kai. Get me the unadon!

Where wouldn’t you want an API?

Forecasting the future is infamously hazardous, but based on the adoption patterns of myself and the people around me, I expect the demand to interact with services through LLMs is going to be overwhelming, and services aiming to provide a good product experience or which face competitive pressure (i.e. someone else could provide that experience instead) will offer APIs.

I used to wish that we’d gone down an alternative branch of web technology and adopted a protocol like Gopher so we’d have a more standardized web experience instead of every product you use producing its own unique UX, most bad. I think we will see more standardization, just not in the form I expected. The convention of the future will be human language, fed into what looks a lot like a terminal, and fulfilled via API.

On behalf of people

Notably, this is different than the first wave of APIs that I described above. Instead of APIs being to offer infinitely flexible access for inter-service communication, scrape data, or build apps on top of someone else’s platform, their primary use will be to fulfill requests on behalf of a primary user. Exactly like what they’d be doing through a first-party app, but in a programmatic way.

During the first wave, APIs were largely aimed at third parties who'd use them to extend and augment the underlying platform to provide additional features for users.

In the second wave, APIs map cleanly to normal product capabilities. They provide programmatic access for agents that act on behalf of people.

It may seem like a subtle distinction, but there are considerable differences. The second model better incentivizes APIs to exist:

APIs aren’t for building a product that aims to displace the offerings of the underlying platform, but rather for giving users an alternative way to access it.
Security models are simplified because they’re the same ones used by the product itself. Users have the same visibility that they’d have through a first-party app, and no more.
Aiming to support access patterns for a single person, platforms can rate limit much more aggressively to curb expenses and operational problems associated with offering an API.

APIs should aim to provide a little more leeway than they would for a human, but only nominally so. An agent acting on my behalf should be able to occasionally poll LinkedIn for old colleagues that I should be reconnecting with and send them connect requests, but if someone’s set up their ClawBot to scrape the entire social graph on their behalf, platforms should feel more than free to throttle the hell out of them and give them a strike towards a permanent ban.

Slack’s rate limits are a good example of this, supporting numbers like 50 channel or 100 profile reads per minute. You can’t build a multi-user app with 50 channel reads per minute, but it’s plenty for a single user to access their own account.

Limits of the model

While can expect many products and services to offer APIs for good agentic interoperability, it won’t be forthcoming everywhere.

Don’t expect much out of Instagram, TikTok, or other platforms that power themselves with ads. Neither from monopolies that won’t feel any serious pressure to change — you won’t be reliably paying your Xfinity bill via agent anytime soon.

Hints of the future, today

In this section I figured I’d call out a few services that are already pulling this future forward:

As I was in the middle of writing this essay, I got a note from Basecamp that they’d revamped themselves for LLM accessibility, including new API, new CLI, and bundled skill to instruct agents on their use.
Salesforce introduced their “Headless 360” initiative, which purports to have made every Salesforce feature accessible by API, MCP, or CLI command.

API spring

Fifteen years ago, us API maximalists thought that APIs were going to eat the world, ushering in a new paradigm of interoperability that would vastly expand our capabilities as users, and even change the world for the better.

What we got instead was an API winter. As useful as APIs were in some situations, that usefulness was outweighed by concerns around revenue, privacy, and abuse.

But as scary of a thought as it was that this might be the end, it wasn’t. We’re at the beginning of a new spring of APIs that’ll appear to support use by agents acting on behalf of people. As this mode of operation gets more popular, expect the availability of an API to be a competitive edge that differentiates a service from its competitors. The result will be a global proliferation of APIs and expanding product capability like never before seen.

Video game March

2026-03-22T04:00:00Z

What I've Worked On at Stainless

2026-03-21T00:00:00Z

At Stainless, I build compiler-like generators that transform API specifications into idiomatic client libraries that feel as though they were hand-written by a language expert who had the time to get…

In and of itself

2026-03-18T04:00:00Z

📊 Nobody Gets Promoted for Simplicity ↗

2026-03-09T15:24:00Z

Matheus Lima in a piece that had my head nodding up and down the whole way through:

If you’re an engineer, learn that simplicity needs to be made visible. The work doesn’t speak for itself; not because it’s not good, but because most systems aren’t designed to hear it.

Start with how you talk about your own work. “Implemented feature X” doesn’t mean much. But “evaluated three approaches including an event-driven architecture and a custom abstraction layer, determined that a straightforward implementation met all current and projected requirements, and shipped in two days with zero incidents over six months”, that’s the same simple work, just described in a way that captures the judgment behind it. The decision not to build something is a decision, an important one! Document it accordingly.

I've met a lot of great engineers who do absolutely amazing work. Many of them have no idea how to articulate and celebrate what they accomplish in a way that resonates with leadership. This post should be required reading for every software developer.

🏷️ Tags: Engineering, Software Development, Code, Design, Links

📄 Permalink to this post

🏛️ Archive of Archives ↗

2026-03-08T15:39:00Z

One of the best things about the web are the digital collections various people curate and share. Now we have an archive of those archives:

Archive of Archives is a curated collection of digital archives developed by SHAPE. All are welcome to contribute to the project. Archive submission guidelines can be found on GitHub.

Superb!

🏷️ Tags: Web, Creativity, Inspiration, Fun, Resources, Links

📄 Permalink to this post

Minimum force

2026-03-08T05:00:00Z

Professional vision

2026-02-27T05:00:00Z

🏙️ Why is the sky blue? ↗

2026-02-24T21:16:00Z

Erik Kennedy goes into incredible detail to explain exactly why the sky appears blue.

But...

As you saw above, violet scatters more than blue. So why isn’t the sky purple? The dumb but true answer is: our eyes are just worse at seeing violet. It’s the very highest frequency of light we can see; it’s riiight on the edge of our perception.

But! – if we could see violet as well as blue, the sky would appear violet.

Neat!

🏷️ Tags: Science, Nature, Links

📄 Permalink to this post

Notes on senses

2026-02-15T05:00:00Z

Building a 24-bit Arcade CRT Display Adapter, From Scratch

2026-02-04T00:00:00Z

On the MIT Mystery Hunt 2026

2026-02-01T05:00:00Z

the cruelty of air

2026-01-11T05:00:00Z

Some stories from developing puzlink.js

2026-01-03T05:00:00Z

Actual Lines (And GIFs) From Tech Recruiter Emails

2025-12-13T00:00:00Z

Inspired by CJ Quines's original blog post, which may or may not be a joke, hard to tell... Anyway, the following are real, I promise! "Tomer?" (this was the whole email) "Transform an unsexy…

dd is my fdisk

2025-12-01T00:00:00Z

I Program on the Subway

2025-11-16T00:00:00Z

Introducing Glide, an extensible, keyboard-focused web browser

2025-09-30T20:00:00Z

TL;DR: Glide is a Firefox fork with a TypeScript config that lets you build anything.

invisible-heading

Download - Docs - Cookbook - Source

Browsers should be hackable, just like your editor.

glide.keymaps.set("normal", "gC", async () => {
  // extract the owner and repo from a url like 'https://github.com/glide-browser/glide'
  const [owner, repo] = glide.ctx.url.pathname.split("/").slice(1, 3);
  if (!owner || !repo) throw new Error("current URL is not a github repo");

  // * clone the current github repo to ~/github.com/$owner/$repo
  // * start kitty with neovim open at the cloned repo
  const repo_path = glide.path.join(glide.path.home_dir, "github.com", owner, repo);
  await glide.process.execute("gh", ["repo", "clone", glide.ctx.url, repo_path]);
  await glide.process.execute("kitty", ["-d", repo_path, "nvim"], { cwd: repo_path });
}, { description: "open the GitHub repo in the focused tab in Neovim" });

Glide keymapping to clone the GitHub repo in the current tab and open it in kitty + neovim.

For my job, I have to clone many different repos. This keymapping saves me a couple seconds each time, multiple times a day.

Although it probably doesn't save me much time, adding this mapping only took a couple of minutes and using it makes me happy

Why I built Glide

I was using tridactyl within Firefox and generally enjoying it, but occasionally I would run into frustrating issues due to security constraints imposed by Firefox on web extensions. For example, extensions are completely disabled on addons.mozilla.org, so all of my mappings could break depending on the website I had open. Additionally, tridactyl wouldn't work with a custom homepage.

These security constraints imposed on tridactyl, and every other extension, are fundamental to how extensions operate. For example, it would be very bad if an extension could prevent itself from being uninstalled by modifying addons.mozilla.org, so browsers have to protect users from potentially malicious extension writers.

At that point, I realised there was an opportunity to make a browser that's truly customisable at its core, with no restrictions on what you can accomplish. From customising the browser UI itself, to calling out to other tools—anything should be possible.

How Glide is different

Glide holistically solves these usability issues by supporting a TypeScript config that lets you do anything¹.

Glide can support APIs and functionality that will never be supported in web extensions as the security model is fundamentally different. You—the end user—are responsible for the config, so there's no reason to restrict what you can do.

In the Glide config you can define custom key mappings, access the web extensions API, spawn arbitrary processes, define macros, and more.

Here's a small example that adds g+c as a key mapping to switch to the calendar tab:

// ~/.config/glide/glide.ts
glide.keymaps.set("normal", "gc", async () => {
  const tab = await glide.tabs.get_first({ url: "https://calendar.google.com/*" });
  assert(tab && tab.id);
  await browser.tabs.update(tab.id, { active: true });
}, { description: "[g]o to [c]alendar.google.com" });

For more examples, see the cookbook or my own dotfiles.

The cherry on top is that all of this is built on top of Firefox. If you already use Firefox, then your existing extensions and workflows will still work in Glide.

Modes

Glide borrows the concept of modes from (neo)vim; every key mapping you define will be attached to a specific mode.

Glide switches between modes automatically when you interact with the browser. For example, in the default mode, normal, if you click on an <input> element, Glide will switch to insert mode, so that key mappings don't interfere with entering text.

If a website doesn't play well with your key mappings you can switch to ignore mode by pressing Shift+Escape, in this mode the only default key mapping is Shift+Escape to exit ignore mode.

Navigating

Glide supports a hint mode that lets you operate web pages entirely using the keyboard.

Press f to enter hint mode and Glide will overlay text labels over every hintable element, e.g. links and buttons. Typing the label for a hint will then focus and click the element.

Personal favourite features

gI focuses the largest visible input element on the page and it feels like magic every time I use it.
<space><space> opens a tab fuzzy finder, which is invaluable for finding that one tab you always lose.
<c-i> and <c-o> are also invaluable for navigating previously open tabs.
:repl for quickly testing out config changes.
hints for when I don't want to reach for my mouse.
A which-key inspired UI for reminding you of the many key mappings.

I've been daily driving Glide for ~6 months now and while I'm biased, I love it. It's still in a very early alpha stage but if you'd like to try it out, you can download it for macOS / Linux today. I'd recommend checking out the tutorial with :tutor to get your bearings, although it's far from complete yet.

p.s. sorry Linux folks, Glide isn't in any package repositories yet, so you'll have to untar it and set up Glide manually for now.

As Glide is in very early alpha there are missing APIs so you can't literally do everything yet, but enabling full control is one of the main goals.↩

How to make a rare birds hotline in 2025

2025-09-06T00:00:00Z

I had an idea the other day when working on some AI voice stuff... what if you could re-create the old birding hotlines but have them automatically update upon recent sightings? What would it be like to be able to call a number and hear which birds are currently being seen in an area, knowing that the information was always up to do? How feasible is this with current tooling and would it even be fun to use? Let's see!

Beginnings

The other day at work I was helping a client launch a new feature for a "real time" AI voice API. I was helping update their SDKs to support some new features and as part of that I needed to use some quick examples to test them out. I had never really used an "audio" forms of AI before so I honestly had no idea what was even possible here in mid 2025. I fired up one of our "push to talk" examples, and was prompted to record a prompt to the AI. As usual, my thought turned to birds, so I asked "what are some good birds to see in New York right now?" I was quite surprised when a very convincing voice replied back with some genuinely good advice about fall migrants.

I quickly moved on to polishing the examples up and getting the release out the door, but the experience stuck with me. I, like so many others no doubt, am more than a bit uncertain about our strange new future we live in. At the same time, I've also seen real, tangible ways that AI has helped speed up my work or just get rid of hassles or blockers (like setting up the frontend for half of this site). So I got to thinking, how could I use this voice stuff when it comes to birding?

Backstory on birding hotlines

I'm honestly not the best person to explain these. They predated my time in birding by probably 2 decades. However, I've heard them detailed in more than a few birding stories or memoirs so I have a good idea of how they came about and how they worked. Back around the 1970's when listing and Big Years really started taking off, one of the biggest enablers of those were the ability to get more and more up to date information about where rare birds might be. As birders started tearing across the country to try and see more and more birds, it became quite helpful to answer questions like "What rare birds are in Florida today?" or "Is the Amur Stonechat continuing in Texas right now?" I believe answering this first was possible as more birders came to know each other and would just call around to their friends or network to find out information. As the field grew though, I imagine it got more than a bit annoying (and expensive) to always be "on call" for a location or a rarity. Thus, the hotlines were born. Once a day or so, an area's birding club or society would record a message about the day's rarities:

The Hudsonian Godwit found by Leo continues on Jamaica Bay's East Pond today. Additionally a Wilson's Phalarope was reported just west of there at Floyd Bennet field. The Red-footed Booby first found by Tim has not been seen since Tuesday morning

Not only was this helpful for Big Year birders, who might be trying to make the call of diverting a route to pick up another species, but it was also useful for local birders too to stay up to date with their local rarities.

These hotlines continued for quite a while, up until the 2000's when they began to be replaced by quicker and more easily distributed communication methods. Email list-serves required much less effort to update, and then even more-so once larger messaging platforms like Whatsapp, GroupMe and now Discord entered the picture. eBird itself also has become instrumental in helping track and find rare birds. You can set up email alerts for rare birds across the world and with a few clicks check and see just when the last reporting of a rare species was.

All in all, this progression has been a great one for basically everyone involved. If you're a lister or working on a Big Year it's never been quicker nor easier to track the status of a rare bird whether it's in your local patch or halfway around the world. I imagine the only person's job that might've gotten difficult is the moderators of these communication mediums. There are countless more birders today and there are bound to be a few mistakes (just like a rare Cuckoo in my patch yesterday seemed to morph into a Blue Jay once I had better viewing conditions). I'm sure the moderators of yore had to deal with this in their capacity too, though maybe it was over a panicked phone call rather than a spurious Discord post.

So why go back down the path of a hotline? There's virtually no reason in practical terms. But, there's just something quaint and nice to me about potentially being able to call in to my local patch to hear what's been going on. Additionally, I imagine a big reason these hotlines fell away is that it was just too much work to record a new update every day, much less every few minutes when a new observation comes in. Today, though, that work might be largely mitigated by what's been coming out with the latest technology. And, I'm really genuinely curious to see if an outdated form of technology might have a second chance when it's paired with the newest of technology we have today

Making this work

Like always, I'd like to start simple before going too far with this idea. I'd like to only work with one place first, my local patch McGolrick Park. I'd love to stick with this first cause I'm going to have a deep sense of what kind of updates and contents would resonate the most with our birders here. I'd like to run this at some sort of regular cadence. Maybe I'll try hourly to start before seeing if I should make that more frequent, or even experiment with having it update in real time? Fetching this data should be pretty straightforward. I can just ask the eBird API for the observations in the park from the last day. I can also ask for any rarities too (those these are very rare in this park). From that, I think then I can send this over to an LLM to have it do two things: first, turn the raw observational data into a human readable (or really, hearable) form. Then, I can use its text-to-speech capabilities to convert that into audio. I'm arguably thinking about this wrong, since both of those steps will probably just happen in one call. Last, I'll just need to get that recording into a place where Twilio or some other phone provider can pick it up and play it over a "recorded line" of sorts. This last part, the phone bits, is funnily enough the part I'm least familiar with.

Fetching the data.

Let's start with the data. Qualitatively, what I'm looking for here is something like:

first, and most importantly, are there any rare birds in the park?
overall, which bird species are being seen right now?
if something interesting is here, did it just show up or has it been here?
if we can know, where specifically is it in the park? (hello helpful comments!)
also, who saw the bird? Can we give them credit?

I think roughly we'll want to make a few calls to the eBird API:

bird species over the last few days in the hotspot
rare bird species in the last few days in the hotspot

And maybe that's it? I might have to make a few follow up calls if I want to see the history of a bird over time (to say something like "The Cerulean Warbler first found by P continues in the Magic Bushes" [what a dream if it did]). But let's walk before we run.

Given I already have the eBird API set up nicely as a Python SDK, getting this up and going is pretty easy:

async def fetch_observations_for_regions_from_phoebe(
    region_code: str,
) -> list[PhoebeObservation]:
    return await get_phoebe_client().data.observations.recent.list(
        back=7,
        cat="species",
        hotspot=True,
        region_code=region_code,
        include_provisional=True,
    )


async def fetch_notable_observations_for_area_from_phoebe(
    region_code: str,
) -> list[PhoebeObservation]:
    return await get_phoebe_client().data.observations.recent.notable.list(
        back=7,
        hotspot=True,
        region_code=region_code,
    )


async def run_bird_calls_job(region_code: str):
    species_observations = await fetch_observations_for_regions_from_phoebe(region_code)
    print(f"Fetched {len(species_observations)} observations")
    species_found = {obs.species_code for obs in species_observations}
    print(f"Found these species codes: {species_found}")

    notable_observations = await fetch_notable_observations_for_area_from_phoebe(
        region_code
    )
    print(f"Fetched {len(notable_observations)} notable observations")

    if notable_observations:
        notable_species_found = {obs.species_code for obs in notable_observations}
        print(f"Found these notable species codes: {notable_species_found}")

Running that produces some pretty useful results!

% uv run src/cloaca/api/bird_calls/main.py
Fetched 30 observations
Found these species codes: {'grcfly', 'bawwar', 'amered', 'eursta', 'dowwoo', 'chswar', 'amerob', 'amecro', 'rthhum', 'laugul', 'veery', 'scatan', 'amhgul1', 'chiswi', 'norcar', 'eawpew', 'rethaw', 'houspa', 'camwar', 'yelwar', 'norwat', 'norpar', 'magwar', 'yebcuc', 'rocpig', 'comyel', 'blujay', 'olsfly', 'comgra', 'moudov'}
Fetched 0 notable observations

The 0 notable observations is expected, so let's swap over to Jamaica Bay's East Pond to make sure that part of the query is working:

% uv run src/cloaca/api/bird_calls/main.py
Fetched 101 observations
Found these species codes: {'comyel', 'merlin', 'cangoo', 'dowwoo', 'gbbgul', 'bkbplo', 'comter', 'snoegr', 'lessca', 'mallar3', 'eursta', 'mutswa', 'blkski', 'rewbla', 'wilpha', 'bubsan', 'leasan', 'fiscro', 'lobdow', 'pecsan', 'baleag', 'carwre', 'semplo', 'stisan', 'greegr', 'whrsan', 'bcnher', 'rudduc', 'norwat', 'sposan', 'rocpig', 'grycat', 'osprey', 'swaspa', 'buwtea', 'houspa', 'forter', 'willet1', 'buggna', 'amewig', 'amerob', 'marwre', 'amgplo', 'balori', 'barswa', 'margod', 'amecro', 'amhgul1', 'sonspa', 'amwpel', 'cedwax', 'laugul', 'hudgod', 'brnthr', 'norhar2', 'amered', 'gloibi', 'norcar', 'shbdow', 'treswa', 'houwre', 'ribgul', 'sora', 'amekes', 'renpha', 'normoc', 'norsho', 'pibgre', 'moudov', 'lesyel', 'gadwal', 'btbwar', 'whevir', 'gresca', 'caster1', 'rebwoo', 'botgra', 'wessan', 'amegfi', 'purmar', 'gnwtea', 'perfal', 'doccor', 'rethaw', 'royter1', 'grbher3', 'easpho', 'norfli', 'coohaw', 'greyel', 'wooduc', 'ambduc', 'yelwar', 'houfin', 'ycnher', 'redkno', 'ameoys', 'easkin', 'killde', 'comgra', 'semsan'}
Fetched 88 notable observations
Found these notable species codes: {'lessca', 'wessan', 'purmar', 'amwpel', 'margod', 'wilpha', 'caster1', 'hudgod', 'bubsan', 'sora'}

Looks like it is (and also I really need to get back over there!).

Let's do some quick and dirty filtering on the McGolrick data though to remove our most common species (I don't think anyone's calling in to hear what the Starlings are up to). I can mostly manually figure these out, though it's funny when Copilot tries to help...

Something interesting I noticed right away... someone noticed a Yellow-billed Cuckoo this morning! Maybe that darn Blue Jay turned back... But, for McGolrick, this and the Olive-sided Flycatcher are genuine patch rarities. We get 1-2 of these per year, so I'd like to flag these as rare. Unfortunately, I think I'd have to use the EBD to do that... but fortunately, I already have that loaded up and working so it shouldn't be too difficult to add?

Before embarking on that and probably ending up writing another new blog post about just that, I'd like to proceed a bit further with trying and proving out that idea.

Human hearable reports

For this next part we need to take this data and get it into the LLM to make it something more human appealing. Let's try and turn this into a form that an LLM like ChatGPT can use. One though I'm toying with is giving the LLM a "tier" of possible birds like so:

rarities: genuinely rare birds that would show up on eBird's rarity reports
patch_rarities: birds that are rare for the park (like the Cuckoo or Flycatcher today)
patch_favorites: birds that aren't as rare, but everyone loves to see when they show up (warblers, tanagers, etc.) For now I'll just let this be anything that isn't one of the prior tiers and isn't a common bird.

So maybe I can define that in a dataclass like so:

class BirdRarityTier(enum.Enum):
    # genuinely rare birds that would show up on eBird's rarity reports
    RARITY = "rarity"
    # birds that are rare for the park (like the Cuckoo or Flycatcher today)
    PATCH_RARITY = "patch_rarity"
    # birds that aren't as rare, but everyone loves to see when they show up (warblers, tanagers, etc.)
    PATCH_FAVORITE = "patch_favorite"
    # the rest
    COMMON = "common"


class PatchObservation:
    common_name: str
    date_last_seen: str
    taxonomic_order: int
    scientific_name: str
    species_code: str
    rarity_tier: BirdRarityTier

Then we can just convert our two API responses into this form. I can't actually distinguish patch_rarities just yet, but that'll come shortly. Here's a sample of this data for Jamaica Bay:

+----------------+-----------------------------+------------------+
| patch_favorite | Osprey                      | 2025-09-06 10:11 |
+----------------+-----------------------------+------------------+
| patch_favorite | Gray Catbird                | 2025-09-06 10:11 |
+----------------+-----------------------------+------------------+
| patch_favorite | Northern Waterthrush        | 2025-09-06 10:11 |
+----------------+-----------------------------+------------------+
| rarity         | Marbled Godwit              | 2025-08-31 09:30 |
+----------------+-----------------------------+------------------+
| rarity         | Caspian Tern                | 2025-08-31 09:30 |
+----------------+-----------------------------+------------------+
| rarity         | American White Pelican      | 2025-08-31 09:30 |
+----------------+-----------------------------+------------------+
| rarity         | Purple Martin               | 2025-08-31 09:30 |
...

(of course I haven't set up common birds for Jamaica Bay, hence why an Osprey is there)

This is just about there! All that's left is a prompt to tell the LLM how to report this. I'm in no ways talented at writing these, but let's take a first stab:

You are a rare bird alert hotline that Birders will call into to hear about rare, notable or interesting birds at a Hotspot. The hotspot name and the list of birds will be provided to you in this input. Your job is to turn that list of birds into a brief summary of the birds at the hotspot today. You always want to focus on reporting `rarity` level birds first, then `patch_rarity`'s, followed by `patch_favorites`. Prioritize birds that have been seen the most recently too. Don't report anything that hasn't been seen in the last 3 days.


Here's an example input:
<!-- cSpell:disable -->
\`\`\`
Hotspot: McGolrick Park

+----------------+---------------------------+------------------+
| Rarity         | Common Name               | Last Seen        |
+================+===========================+==================+
| patch_favorite | Baltimore Oriole          | 2025-08-30 18:25 |
+----------------+---------------------------+------------------+
| patch_favorite | Ovenbird                  | 2025-08-30 18:25 |
+----------------+---------------------------+------------------+
| patch_favorite | Black-and-white Warbler   | 2025-09-01 07:00 |
+----------------+---------------------------+------------------+
| patch_favorite | Chestnut-sided Warbler    | 2025-09-01 07:00 |
+----------------+---------------------------+------------------+
| patch_favorite | Scarlet Tanager           | 2025-09-01 07:00 |
+----------------+---------------------------+------------------+
| patch_favorite | Great Crested Flycatcher  | 2025-09-02 06:30 |
+----------------+---------------------------+------------------+
| patch_favorite | Common Yellowthroat       | 2025-09-02 06:30 |
+----------------+---------------------------+------------------+
| patch_favorite | Magnolia Warbler          | 2025-09-02 06:30 |
+----------------+---------------------------+------------------+
| patch_favorite | Ruby-throated Hummingbird | 2025-09-03 06:30 |
+----------------+---------------------------+------------------+
| patch_favorite | Yellow Warbler            | 2025-09-06 07:55 |
+----------------+---------------------------+------------------+
| patch_favorite | Northern Parula           | 2025-09-06 09:52 |
+----------------+---------------------------+------------------+
| patch_favorite | American Redstart         | 2025-09-06 10:37 |
+----------------+---------------------------+------------------+
| patch_favorite | Northern Waterthrush      | 2025-09-06 11:08 |
+----------------+---------------------------+------------------+
| patch_favorite | Cape May Warbler          | 2025-09-06 11:08 |
+----------------+---------------------------+------------------+
| patch_rarity   | Olive-sided Flycatcher    | 2025-09-05 18:26 |
+----------------+---------------------------+------------------+
| patch_rarity   | Yellow-billed Cuckoo      | 2025-09-06 11:08 |
+----------------+---------------------------+------------------+
\`\`\`
<!-- cSpell:enable -->

And here's an example of how you would report this

> Thanks for calling McGolrick Park rare bird alert. A Yellow-Billed cuckoo was seen starting this morning. Yesterday, an Olive-sided flycatcher was seen. For the warbler lovers, Yellow Warblers, Redstarts, Northern Waterthrushes, Cape Mays and Northern Parulas are all being seen. Happy birding!

That's probably woefully insufficient, but let's try it! It's funny how I spend all day working with OpenAI's SDKs yet I still failed to get this working the first time...

Hey there! Looks like McGolrick Park is buzzing with bird activity right now! We’ve got some real gems spotted recently. Just today, you’ve had sightings of the Northern Parula, American Redstart, Northern Waterthrush, Cape May Warbler, and even the Yellow Warbler.

And check this out—there’s a rare visitor: the Yellow-billed Cuckoo was spotted today, too! Plus, yesterday you had an Olive-sided Flycatcher—another rare one in the area. If you head over to the park now, you might still catch those patch favorites like the Ruby-throated Hummingbird, Scarlet Tanager, and Magnolia Warbler, all seen just in the last few days.

It’s prime time for birding out there. Grab your binoculars and happy birdwatching!
Wow! McGolrick Park is a birding hotspot right now! Let me give you the highlights in a fun rundown.

First off, some of the favorite feathered friends that have shown up recently include the Yellow Warbler, Northern Parula, and American Redstart—all seen today. These warblers are in peak migration mode, so now’s a fantastic time to spot them. Plus, you’ve got the Northern Waterthrush and Cape May Warbler sighted today too—talk about a warbler bonanza!

Then, looking at yesterday, you had the Common Grackle pop up, and earlier this week, the Magnolia Warbler and Great Crested Flycatcher were around, along with that dazzling Scarlet Tanager.

But here’s where it gets really exciting: we’ve got a couple of rarities on the list! Just yesterday, an Olive-sided Flycatcher was seen, and today—drumroll please—a Yellow-billed Cuckoo made an appearance. That’s a rare bird for the area and a real treat if you can catch a glimpse.

So, if you’re heading out to McGolrick Park, bring those binoculars and keep your eyes peeled. You’ve got a great mix of patch favorites and some special visitors right in the heart of migration season. Happy birding!
Alright, let’s break it down! McGolrick Park is absolutely alive with birds right now. Here’s the quick scoop.

We’ve got some super fresh sightings today—September 6th. The Yellow Warbler showed up early in the morning at 7:55. Then, the Northern Parula swooped in around 9:52. Not long after, the American Redstart popped up at 10:37. And get this—just a bit later, at 11:08, someone spotted both a Northern Waterthrush and a Cape May Warbler. That’s a warbler party!

And for the rare birds? Today’s jackpot: the Yellow-billed Cuckoo was seen at 11:08 as well. That’s a big one. Plus, just yesterday—September 5th—an Olive-sided Flycatcher was spotted around 6:26 in the evening. Those are some rare treats for the park!

And so on... it was just running in a loop. And that's entirely my fault cause I copied the example for "push to talk" including the while True infinite loop: 🤦‍♂️.

openai = AsyncOpenAI()
async with openai.realtime.connect(
    model="gpt-realtime",
) as connection:
    await connection.session.update(
        session={
            "output_modalities": ["text"],
            "model": "gpt-realtime",
            "type": "realtime",
        }
    )
    while True:
        await connection.conversation.item.create(
            item={
                "type": "message",
                "role": "user",
                "content": [{"type": "input_text", "text": user_prompt}],
            }
        )
        await connection.response.create()
        async for event in connection:
            if event.type == "response.output_text.delta":
                print(event.delta, flush=True, end="")
            elif event.type == "response.output_text.done":
                print()
            elif event.type == "response.done":
                break

Also that's not really the way we want this hotline to sound like... oh right, I forgot to to provide a system prompt! Adding that in and...

Thanks for calling the McGolrick Park rare bird alert. The standout sighting today is a Yellow-billed Cuckoo, seen this morning. Also notable, an Olive-sided Flycatcher was spotted yesterday. Warbler enthusiasts will be thrilled: Northern Parula, American Redstart, Northern Waterthrush, Cape May Warbler, and Yellow Warbler have all been seen today. Other favorites recently spotted include a Common Grackle, Ruby-throated Hummingbird, and Veery. Happy birding!

Hey that's pretty good! I have this just outputting as text for now, so let's add audio too. Getting the audio output added was simple: I just needed to add audio as an output_modalities and also process the output_audio events. Getting this into an audio file was slightly trickier but still pretty straightforward:

def save_pcm_audio_chunks(pcm_bytes_array):
    print(f"Received {len(pcm_bytes_array)} audio chunks")
    # Open WAV file for writing
    with wave.open("bird_report_audio.wav", "wb") as wav_file:
        # Set WAV parameters
        wav_file.setnchannels(1)  # mono
        wav_file.setsampwidth(2)  # 2 bytes per sample (16-bit)
        wav_file.setframerate(24000)  # sample rate

        # Write PCM data
        wav_file.writeframes(b"".join(pcm_bytes_array))

And...

Your browser does not support the audio element.

It works! Though why is our fairly obnoxious response back. One thing I noticed when playing around with the various voice options though is that they're heavily scripting out things like tone, affect and emotion. Life goals honestly:

Affect: Deep, commanding, and slightly dramatic, with an archaic and reverent quality that reflects the grandeur of Olde English storytelling.

I added some extra input like so:

    Affect: deep, informed, wise.

    Tone: informative, terse and to the point.

    Emotion: dry, direct, concise.

    Pronunciation: articulate with a slight twang of a Southern US accent.

Unfortunately, this seemed to have no effect. I eventually realized that this seems to be a bug when providing both text & audio as the output modality. I mistakenly thought this was necessary to get a transcript... but turns out I just need audio. Once I provided that, we had something much better to work with!

Thanks for calling McGolrick Park rare bird alert. A Yellow-billed Cuckoo was seen late this morning. Yesterday, an Olive-sided Flycatcher was spotted. Today’s patch favorites include Northern Waterthrush, Cape May Warbler, American Redstart, Northern Parula, and Yellow Warbler. Happy birding.

This is possibly too formulaic, but let's go with it for now.

Getting this in the phones

Okay, we've got an audio file now we can work with. All that's needed is for us to shove this into a phone call! Initially, I thought I might explore a solution where I just upload the file to something periodically. However, doing a bit more research (like 15 seconds more), maybe I'll just have Twilio read the audio data from an API. This is maybe slightly more work, but also opens the door up to have this data updated in "real time" whenever someone calls. Poking around, I really think what I want is whatever this TwiML thing is called... maybe this will let me eventually set up some sort of dynamic phone tree even?

Seems like the main thing I need to do now is to set up a new API endpoint that returns their custom XML stuff:

async def get_bird_call(location_code: str | None) -> str:
    if location_code is None:
        location_code = MC_GOLRICK_PARK_HOTSPOT_ID

    response = VoiceResponse()
    response.say("Hello you're talking to Bird!")

    return str(response)

And... that works:

"<?xml version=\"1.0\" encoding=\"UTF-8\"?><Response><Say>Hello you're talking to Bird!</Say></Response>"

Now we just need to map that to the audio file itself. I'm not sure exactly the best way to host a file like this, but for now I can just push the file up to the server and then read it from there. In some ideal world I guess this would live on a service like S3 better suited for serving data. But, this will do for now:

@Cloaca_App.get("/v1/bird_calls/audio_file")
async def get_bird_call_audio_file(location_code: str | None):
    try:
        audio_files_path = os.environ["AUDIO_FILES_PATH"]
    except KeyError:
        raise RuntimeError(
            "AUDIO_FILES_PATH environment variable is required. "
            "Please set it to the path of your audio files."
        )
    file_path = f"{audio_files_path}/{location_code}.wav"
    print(f"Looking for audio file at {file_path}")
    if os.path.exists(file_path):
        return FileResponse(file_path, media_type="audio/wav")
    else:
        return {"message": "Audio file not found"}, 404

From here, the main work was just going through some extra steps in Twilio to set up a phone number. Then, I simply routed that phone number to an ngrok-forwarded URL to my local machine and... 🥁🥁🥁

It works! I can't really show a great demo of this given you can't record the audio of a phone call in a screen recording on iOS, but it really does work!

Polishing this up

Alright, now on to the final step of getting this working for real. There are a few things we need to finish up. First, we need the alert to run periodically. Then, we need to make sure the file is getting persisted in the right place. Last, we need to wire this all up correctly in Twilio to not be using ngrok for this.

One forgotten API key, one mis-formatted URL, one forgotten deploy and one long session debugging why Twilio didn't like my Content-Type later... and it's live! You can now call 9148637717 and get some local bird intel!

This was quite fun and quick. It could've been quicker if it were simpler to just pass audio straight back to Twilio. All in all, this was pretty straightforward and painless though.

If you want to see the full details of this, I merged in the Pull Request for it here:

https://github.com/birds-eye-app/cloaca/pull/8

I think it'd be fun in the future to possibly extend this further:

Certainly adding more supported hotspots. It'd be nice to add a feature to type in or even just say the hotspot you want to see information for.
I also need to actually add in the DuckDB information to more accurately show patch rarities.
I'd love to figure out a way to access who submitted the observation. These hotlines love giving credit to who found the bird and it'd be cool to replicate that. I get why eBird might be a bit stingy on handing user details out though.
Similarly, I wish I could access the species comments on these too. It'd be very nice to include that so people could know where to look: "A Scarlet Tanager is being seen above the playground".
It might be cool to eventually support some more true real time voice stuff, enabling someone to talk over the phone?

Typing select queries in Python

2022-02-24T08:30:00Z

The recently accepted PEP 646 introduced variadic generics! While the main motivation for this feature was to improve typing for numerical libraries it also means that is now possible to accurately type the results of an SQL select query statement in Python!

There are some limitations to this however, unfortunately we cannot return named properties and instead must use a tuple form.

This post will walk through the process of typing and implementing the equivalent of an SQL select function, if you just want to see the final results then click here.

Getting Started

This post assumes that you are already familiar with generics in python, if you are not then I suggest you read through this tutorial.

You will also need a recent version of typing-extensions and pyright.

Introduction to PEP 646

The acceptance of PEP 646 means we can now create generics with an arbitrary number of type variables instead of being limited to a pre-defined number!

The new features that this introduces is TypeVarTuple and Unpack which must be used in combination with each other to represent variadic generics.

A not particularly useful example of this is inserting an integer to the start of a tuple (yes tuples are immutable but lets assume for this example that we would return a new tuple):

from typing import Tuple
from typing_extensions import TypeVarTuple, Unpack


T = TypeVarTuple('T')


def insert_int(seq: Tuple[Unpack[T]]) -> Tuple[int, Unpack[T]]:
    ...


insert_int(('a', 1))  # return type: Tuple[int, str, int]

Minimal example

The minimum code required to represent a select satement statically is actually fairly small.

from typing import Tuple
from typing_extensions import Unpack, TypeVarTuple, reveal_type


FieldT = TypeVarTuple('FieldT')


class User:
    id: int
    name: str
    email: str
    hashed_password: str

    @classmethod
    def select(cls, *fields: Unpack[FieldT]) -> Tuple[Unpack[FieldT]]:
        ...


user = User.select(User.id, User.name)
reveal_type(user)

Now running this code through pyright will give us the following output:

$ pyright typed_select.py
typed_select.py:20:13 - information: Type of "user" is "Tuple[int, str]"

Wow, this is exactly what we were looking for!

Now lets try actually running this code and see what happens:

$ python typed_select.py
Traceback (most recent call last):
  File "/Users/robert/code/craigie.dev/sources/typing-select/001.py", line 19, in <module>
    user = User.select(User.id, User.name)
AttributeError: type object 'User' has no attribute 'id'

Hmmmm that is annoying.

This is happening because we've given the User class type hints but haven't actually given these fields any values.

Let's assign some default values for the fields we are selecting:

from typing import Tuple
from typing_extensions import Unpack, TypeVarTuple, reveal_type


FieldT = TypeVarTuple('FieldT')


class User:
    id: int = 0
    name: str = ''
    email: str
    hashed_password: str

    @classmethod
    def select(cls, *fields: Unpack[FieldT]) -> Tuple[Unpack[FieldT]]:
        ...


user = User.select(User.id, User.name)
reveal_type(user)

We can now sucessfuly run this code:

$ python typed_select.py
Runtime type is 'NoneType'

Implementation

Now let's implement the select() function so that we can actually use it at runtime.

Lets start by defining a Field class so that we don't have to assign weird defaults for each field and so we can store a reference to the field's name in the database.

class Field:
    name: str

    def __init__(self, name: str) -> None:
        self.name = name

We now have to define a wrapper function over the Field class so that type checkers don't complain that we're assigning a Field instance when we can only assign strings for example. This works as we tell the type checker that we are returning Any which disables type checking.

def field(*, name: str) -> Any:
    return Field(name=name)

Now lets update our User model to use the new Field class

class User:
    id: int = field(name='id')
    name: str = field(name='name')
    email: str = field(name='email')
    hashed_password: str = field(name='hashed_password')

To make this easier to implement we'll define a list of records as our database instead of using a real database driver.

fake_db = [
    {
        'id': 1,
        'name': 'Robert',
        'email': 'robert@craigie.dev',
        'hashed_password': 'foo',
    },
    {
        'id': 2,
        'name': 'Tegan',
        'email': 'tegan@craigie.dev',
        'hashed_password': 'bar',
    },
]

Now all that we need to do is implement the select() function

    @classmethod
    def select(cls, *fields: Unpack[FieldT]) -> Tuple[Unpack[FieldT]]:
        if any(not isinstance(field, Field) for field in fields):
            raise TypeError(
                'Expected all select arguments to be an instance of Field'
            )

        # for simplicites sake we'll just take the first record
        user = fake_db[0]

        # we have to add a type: ignore comment here as the type checker
        # cannot understand the relationship between this expression
        # and the input.
        return tuple(  # type: ignore
            user[field.name] for field in fields if isinstance(field, Field)
        )

Let's type check it and make sure we haven't made any mistakes.

$ pyright typed_select.py
typed_select.py:60:13 - information: Type of "user" is "Tuple[str, int]"

All good! Now lets run it and see what we get!

$ python typed_select.py
Runtime type is 'tuple'
('Robert', 1)

Amazing! We've just managed to statically type and implement a select query!

There are some potential improvements that could be made to this, for example, returning a custom object that provides some helper methods instead of a raw tuple.

Final Implementation

from typing import Tuple, Any
from typing_extensions import Unpack, TypeVarTuple, reveal_type


FieldT = TypeVarTuple('FieldT')


class Field:
    name: str

    def __init__(self, name: str) -> None:
        self.name = name


def field(*, name: str) -> Any:
    return Field(name=name)


fake_db = [
    {
        'id': 1,
        'name': 'Robert',
        'email': 'robert@craigie.dev',
        'hashed_password': 'foo',
    },
    {
        'id': 2,
        'name': 'Tegan',
        'email': 'tegan@craigie.dev',
        'hashed_password': 'bar',
    },
]


class User:
    id: int = field(name='id')
    name: str = field(name='name')
    email: str = field(name='email')
    hashed_password: str = field(name='hashed_password')

    @classmethod
    def select(cls, *fields: Unpack[FieldT]) -> Tuple[Unpack[FieldT]]:
        if any(not isinstance(field, Field) for field in fields):
            raise TypeError(
                'Expected all select arguments to be an instance of Field'
            )

        # for simplicites sake we'll just take the first record
        user = fake_db[0]

        # we have to add a type: ignore comment here as the type checker
        # cannot understand the relationship between this expression
        # and the input.
        return tuple(  # type: ignore
            user[field.name] for field in fields if isinstance(field, Field)
        )


user = User.select(User.name, User.id)
reveal_type(user)  # Revealed type is Tuple[str, int]
print(user)        # ('Robert', 1)

Digging into species likelihood with eBird and DuckDB

2025-08-27T00:00:00Z

In part 2 of this exploration we dug deep into how to answer the question "where do all the birders go?" We found many ways to make queries run very slowly on DuckDB but ended up hitting our initial goal of starting to use the EBD in production. Now, we're going to extend that further to see if we can show people how many birds might be likely in a given place and time.

Best practices when working with eBird data

I'll admit from the start that this is far more of a daunting exercise than the last one. The analysis part of the hotspot popularity was pretty straightforward: count the checklists and group them by the hotspot and month (though, that didn't stop me from making plenty of mistakes). The challenge there was more on the technical side of getting queries to run quickly and efficiently. This part will be much more challenging on both fronts. For one, answering the question of "which bird can be seen where" with any sort of correctness is quite hard. Second, doing this in a way that will work in production is not going to be easy.

One advantage that I do have this time around is the fact that the eBird team have spent a lot of time discussing and documenting how to go about this exact sort of analysis. I'll be borrowing heavily from their fantastic resource here on Best Practices for Using eBird Data.

A particular highlight I'd recommend going over is the various biases or imbalances that creep in when using citizen science data like eBird. I'll summarize briefly here, but it's very readable if you want to check it out directly. The underlying thing we're trying to measure is when and where a species of bird will be at any given point. We can't just know this, so we're forced to estimate this in some way. The data source we're using here is humans going out and writing down which birds they saw and how many. Many of them do this with a scientific mindset, but they're also humans with preferences and needs that influence when and where they do this. So we end up with lots of trends in the observations that are more human than avian. We humans prefer to bird on the weekends, during migration when there are plenty of birds, when the weather isn't horrendous and so on. And we tend to notice, well, more noticeable birds. A loud or beautiful bird song will get noticed better than an owl sleeping nearly invisibly at the top of a tree.

Writing a query for species likelihood

Bringing this to the task at hand, let's start with a naive analysis of species likelihood. We'll count the number of checklists in a hotspot for a certain month. Then we'll count how many times a species appeared on a checklist in that place. Dividing those we should get a rough idea of "what are the most likely birds to see then and there?"

Here's a quick way I got to this data, with the regular caveat that I've certainly made mistakes:

with
  total_checklists as (
    select
      extract (
        month
        from
          "OBSERVATION DATE"
      ) as month,
      "LOCALITY ID",
      count(distinct "SAMPLING EVENT IDENTIFIER") as total_checklists
    from
      ebird_ny."full"
    where
      "OBSERVATION DATE" > current_date - interval '5 year'
      and month = 8
      and "LOCALITY ID" = 'L165143'
    group by
      1,
      2
  ),
  species_checklists as (
    select
      extract (
        month
        from
          "OBSERVATION DATE"
      ) as month,
      "LOCALITY ID",
      "COMMON NAME",
      count(distinct "SAMPLING EVENT IDENTIFIER") as species_checklists
    from
      ebird_ny."full"
    where
      "OBSERVATION DATE" > current_date - interval '5 year'
      and month = 8
      and "LOCALITY ID" = 'L165143'
    group by
      1,
      2,
      3
  )
select
  "COMMON NAME",
  species_checklists,
  total_checklists,
  species_checklists / total_checklists
from
  total_checklists
  join species_checklists using ("LOCALITY ID", month)
order by
  species_checklists desc
limit
  10

There's probably a quicker way to write that too, but using something like window functions gets a little tricky since I don't want to just add up the checklists counts for all species. Instead I want the unique list of checklists. Anyways, here's what that returns:

COMMON NAME	species_checklists	total_checklists	likelihood
Laughing Gull	593	669	0.8863976083707026
Mute Swan	548	669	0.8191330343796711
Great Egret	539	669	0.8056801195814649
Osprey	533	669	0.796711509715994
Double-crested Cormorant	532	669	0.7952167414050823
Gray Catbird	527	669	0.7877428998505231
Canada Goose	519	669	0.7757847533632287
Mallard	516	669	0.7713004484304933
Northern Mockingbird	509	669	0.7608370702541106
Snowy Egret	506	669	0.7563527653213752

(if the numbers seem high that's cause the totals are across the past 5 years or so of data)

Sorted the other way (least popular first) we have these:

COMMON NAME	species_checklists	total_checklists	(species_checklists / total_checklists)
Whimbrel	1	669	0.0014947683109118087
Monk Parakeet	1	669	0.0014947683109118087
Calidris sp.	1	669	0.0014947683109118087
Wilson's Warbler	1	669	0.0014947683109118087
Piping Plover	1	669	0.0014947683109118087
Hooded Warbler	1	669	0.0014947683109118087
White-faced Ibis	1	669	0.0014947683109118087
Black-throated Green Warbler	1	669	0.0014947683109118087
Dickcissel	1	669	0.0014947683109118087
Tricolored Heron	1	669	0.0014947683109118087

This matches my gut level instincts of what might show up at Jamaica Bay in August. But there are some additional steps we should consider.

The main one is that we can't naively assume that most people were trying to do complete checklists. Put another way, if someone birded for 2 minutes in Jamaica Bay and saw 3 species, a Laughing Gull, a a House Sparrow and an Osprey, we should not assume that a Mallard wasn't in Jamaica Bay that day. The eBird docs go much further into what to think about here, but being able to do this conversion, in the right way, is one of the most valuable aspects of eBird and why complete checklists are so valuable to science!

Oh and also we probably shouldn't factor in Calidris sp. type reports either. So let's apply a few recommended filters:

we'll only include species
no incidental checklists
keep it under 10km

Last, I'll throw in a distinct at the top for 2 reasons. 1, this should condense "shared" checklists down to only 1 report per species. 2, I think this will also collapse sub-species into the regular species name. Both of these are definitely areas to followup on, though I hope it'll be good enough for now.

I'll just define those in a new input_data expression to make it easier to keep things consistent:

select
    distinct -- distinct is here to exclude duplicate (shared) checklists
    extract (
    month
    from
        "OBSERVATION DATE"
    ) as month,
    "LOCALITY ID" as locality_id,
    "SAMPLING EVENT IDENTIFIER" as checklist_id,
    "COMMON NAME" as common_name
    from ebird_ny."full"
    where
    "OBSERVATION DATE" > current_date - interval '5 year'
    and CATEGORY = 'species'
    and "PROTOCOL TYPE" in ('Stationary', 'Traveling')
    and "EFFORT DISTANCE KM" < 10

Once we have that, we can now run this across all the hotspots for August (I'm using a narrow New York only dataset to make this easier). This way we can see something like "which hotspot has the highest number of species likely to be found at it?" And we get something surprising:

month	LOCALITY	locality_id	common_species	uncommon_species	total_checklists
8	St. Lawrence River Dike Trail at Richard's Landing	L1810696	100	0	12
8	Freshkills Park--North Park (Phase l)	L27548440	96	0	6
8	Ulster Park	L16301078	95	16	34
8	Jamaica Bay Wildlife Refuge	L165143	94	40	619
8	Montezuma NWR--Towpath Rd.	L266832	93	54	70
8	Montezuma NWR--Knox-Marsellus and Puddler Marshes	L679571	92	14	44
8	Lower La Chute River & Ticonderoga Marsh	L4910422	91	0	11
8	Spring Farm Nature Sanctuary	L6624443	89	0	16
8	River Rd. marshes and RR tracks	L2457005	89	0	15
8	Columbia University, Nevis Laboratories Environs	L27674356	89	0	16

The top hotspot is only one with 12 checklists in it. I haven't checked those checklists myself to see if they're valid, but there's nothing to say there's anything wrong with something like this. There are plenty of very rich hotspots out there that don't get visited that often. Still, it seems less than ideal if we'd rank this spot as more rich than something like Jamaica Bay where not only does it have a similar count of species, but we're also far more certain of the numbers you can see there. We're gonna need... statistics for this.

Error rate / confidence

What I think I really want here is not necessarily a way to alter the number of species we show for lower frequency hotspots. Instead, we really just want to return additional data alongside it that shows how confident we are in the data. It's been a long while since I was in stats class and unfortunately I'm drawing a bit of blank on how best to model data like this. What's confusing to me is that we're not really dealing with "random" data per se. A birder seeing 90 species at a location over 3 hours is a good sign that there's going to be a lot of species at that location. But it's not obvious to me how to rate that finding if 100 other birders hit that level, or if no one else does. It would feel like a mistake to throw that observation out, or even to water it down with 10 other quicker trips if they only spent a few minutes and recorded 10 species.

I think what I'll do for now is do a very rough "error" calculation (rough cause I'm truly not sure if it's right), and return that to my map alongside the unmodified count of "common" species. Here's how I calculate that:

sqrt(common_species) / total_checklists

Getting this to the front end

Now, I had planned originally on making another table for this data, but I think with the learnings from last time, it might make sense to just throw this data into our existing popular hotspots table. It will definitely increase the table size, but I won't have to, hopefully, worry too much about performance with a new JOIN.

But, let's just see how this looks now. I haven't talked much about the frontend side of displaying things in this series, and that's mostly cause I haven't thought too much about it. It's mostly been me (Claude) copy and pasting the existing map code I have here. I don't want to do this forever, but it's been "good enough" for now. So let's keep doing that.

So with a decent prompt, we're up and running... except these numbers are a bit off. Why is Floyd Bennet showing 109 on the map but 53 in the tooltip?

Looking at the DB for that location and month:

┌─────────────┬─────────────────────┬───────────────────────────────────┬────────────────┐
│ locality_id │    locality_name    │ avg_weekly_number_of_observations │ common_species │
│   varchar   │       varchar       │              double               │     int64      │
├─────────────┼─────────────────────┼───────────────────────────────────┼────────────────┤
│ L152773     │ Floyd Bennett Field │                 7.869565217391305 │             53 │
│ L152773     │ Floyd Bennett Field │                 7.869565217391305 │             93 │
│ L152773     │ Floyd Bennett Field │                 7.869565217391305 │             70 │
│ L152773     │ Floyd Bennett Field │                 7.869565217391305 │             55 │
│ L152773     │ Floyd Bennett Field │                 7.869565217391305 │             63 │
│ L152773     │ Floyd Bennett Field │                 7.869565217391305 │             92 │
│ L152773     │ Floyd Bennett Field │                 7.869565217391305 │             71 │
│ L152773     │ Floyd Bennett Field │                 7.869565217391305 │             85 │
│ L152773     │ Floyd Bennett Field │                 7.869565217391305 │             62 │
│ L152773     │ Floyd Bennett Field │                 7.869565217391305 │            109 │
│ L152773     │ Floyd Bennett Field │                 7.869565217391305 │             67 │
│ L152773     │ Floyd Bennett Field │                 7.869565217391305 │             59 │
├─────────────┴─────────────────────┴───────────────────────────────────┴────────────────┤
│ 12 rows                                                                      4 columns │
└────────────────────────────────────────────────────────────────────────────────────────┘

We should not have 12 rows for this. Oh the ole 1:many accidental JOIN issue. I'm creating my final table like so:

SELECT
  locality_id,
  month,
...
FROM
  localities l
  JOIN hotspot_popularity hp USING (locality_id_int)
  LEFT JOIN hotspots_richness hr using (locality_id_int)
...

The join on the ID is fine, but I forgot I need to make sure the same month is used for popularity and richness. This ought to fix it:

LEFT JOIN hotspots_richness hr using (locality_id_int, month)

Zooming out a bit things are looking overall pretty good. It's hard to say how accurate this is... I'm learning from just looking around on the map which hotspots really shine at which time of year. I'm also struggling a bit to figure out how best to show an error bar or confidence interval at this level. Should I fade out lower confidence hotspots? Should I make the edges blurrier? Should I de-saturate them?

For now I think I'm going to focus on the more core problem of making sure this data is accurate. If I quickly compare some hotspots it seems my methodology is vastly under-counting compared to eBird. It's been fairly hard for me to figure out how eBird and Merlin calculate the numbers here. I've seen that for a species to not be a yellow dot, it has to appear on 6% or more of checklists. And that's actually what I used initially in my data. However, if I actually look into the eBird mobile species under "Likely" for a hotspot they include these yellow dots. So maybe I should include them in mine? Or really what I want to do is to include both?

Adding those numbers in brings my numbers much closer to eBird's. I think it also makes since to display both in the tooltip. Maybe I can even add an option shortly to toggle the map from displaying either just common birds or both common and uncommon?

Scaling this up

Before going too much further though I want to stop and check in on the performance of this. I'm a little wary of adding too much more data if I can't actually deploy it. I do most of my development off of a more limited NY-only dataset. This is plenty for testing and poking around, but I still need to run the full thing to make sure I'm not gonna drain the bank on storage fees.

The first run of this... wasn't promising. I very quickly drained my computer memory and the spillover disk storage. I hadn't taken a cleanup or refactoring pass through the likelihood query, so I did quick look through to remove some extraneous columns and CTE's. One quick snack break later... and no luck. Still running out of space. Off to the DuckDB docs again!

The docs were helpful as always, but there weren't too many tricks I hadn't already tried. The job is already getting as much memory and disk space as I can provide and I'm not going anything woefully incorrect with my setup from what I can tell. A few things that did stick out to me: when I initially set up the full dataset, I didn't do anything to help optimize by using the correct types, sorting by timestamps or filtering out data or columns I'm not going to use. That's probably long overdue here so I set about trying to define a more intermediate table to work off of. Unfortunately, with me stuck with the already too-large dataset on my hard drive I wasn't left with enough room to spin off a new one, pruned though it might be. Thankfully, I already needed to regenerate the underlying dataset with the latest data from July so we might as well take that on and clean up and format the data a bit more as we go.

A long side quest on better data import

Back in Part 1, I was just trying to get to the point of being able to query the EBD in a DuckDB dataset. It was a slow but fun journey, but I'd only done this once. Thus, my methods of doing so were sloppy at best and I knew the next time I was going to do this I should take a few passes and cleaning up what I was going and also making it a bit more efficient. This latter part really piqued my interest this time around, almost certainly to a negative degree. I had the external disk space to just do this in a slow, but sure way, converting the compressed form into an uncompressed TSV and then into a DuckDB database. Still, that felt quite wasteful and slow (in reality it would just mean firing off two scripts before going to bed one night and then waking up to a fresh dataset ready for use). So, like many times before, I set off to spend a bunch of mostly unnecessary time speeding this up and making it more efficient.

The core of what I didn't like about the old flow was the necessity of reading through the data 2 or even 3 times to get to a working DuckDB:

Once to un-tar the files (I'm not entirely positive if this requires reading all of the underlying bytes)
To decompress the compressed TSV file inside of that TAR
To load that TSV file into DuckDB

What would be great is if we could go from the .tar file to the full DB in one step. Theoretically I knew this was possible but I wasn't sure how feasible it would be with the existing tools I had. Something that gave me early confidence was the fact that DuckDB can read directly from stdin when invoked from the command line like so:

cat ebird.tsv | duckdb -c "SELECT * FROM read_csv('/dev/stdin')"

Neat! All that's really needed then is to selectively read and decompress the .gz file inside the .tar and pass that into DuckDB. Overall this was pretty straightforward and it wasn't long before I had a simple working solution. One aside here is that I did all of this myself in bash as a first pass, but then opted to let Claude Code then take a pass at polishing the script to have more logging, a progress bar using pv and good timing tracking. Claude Code really seems at its best in moments like this where I have a very clear overall direction for what I want to do, but can rely on it to introduce extra details like args, or logging that would be finicky or frustrating for me to get just right with my intermediate bash knowledge. Here's how it looked once I had it up and running for real:

 % ./queries/parse_ebd.sh ebd_relJul-2025 ./dbs/ ./dbs/
Starting eBird data processing at Sat Aug 30 09:28:23 EDT 2025
Processing: ebd_relJul-2025
Output database: ./dbs/ebd_relJul-2025.db
[09:28:23] Removing existing database file...
[09:28:23] Database file removed
[09:28:23] Starting data extraction and processing...
  - Extracting ebd_relJul-2025.txt.gz from tar archive
  - Archive size: 198.9G
  - Compressed .gz file size: 198.9G
23.1GiB 0:11:20 [34.5MiB/s] [==>                 ]  11% ETA 1:25:42

I'm pretty happy with how this script came together! It's much more efficient with disk space and quicker than doing this in 3 steps. It takes about 60-90 minutes to complete which is the difference between leaving it for a night vs leaving it for a run or a long walk. Here's the full version if you care to look.

It wasn't all success, though. One important goal I had for this refactor was to introduce better ordering at this stage. DuckDB really emphasizes in their docs and blogs the importance of table ordering for good performance, especially in this great article: https://duckdb.org/2025/05/14/sorting-for-fast-selective-queries.html. Naively, I thought I could just throw in an order by observation_date at the end of my new script and get this for free. However, this just resulted in all of the free disk space on my machine getting eaten up at an alarmingly fast pace. I think what was going on here is that DuckDB had to basically read in the entire contents of the DB into either memory or spillover disk space to determine the right order before it could begin to start committing anything into the more space-efficient DB.

Fresh data but no progress

So, while it was good to have a much better import script and the latest EBD data, I still didn't feel like I was getting any closer to being able to generate the full data I wanted here if I couldn't actually set up ordering. Now the challenge is entirely about how to get this data ordered in a reasonable way. I searched around for ideas about how to order this in place, but didn't find anything mentioning this as a feature of DuckDB. I couldn't just write a new query to just "create a new table with everything in the old table but order it" since I'd just run out of disk space again.

Thankfully, while on a walk a rather simple solution occurred to me. Just do this in batches. I can easily read a "chunk" of the old dataset, push it into a new DB and order it there. Since I'm only working with a portion of the data at a time then I shouldn't eat up all the available disk space, just what's necessary to sort the portion of data I chose. And, if disk space gets precious, I can actually just delete those rows from the old DB once I insert them so I'm not actually net adding more data to my machine.

I thought for a few minutes on the right elegant solution to reliably and repeatedly split my data up before opting to just do it by the month of the observation. In hindsight I think this might've been really silly though... I want to be sorting by observation date but if I'm doing this across months then I think I'm probably having to read across wildly different chunks or pages of the data? In any case, the logic of this was pretty straightforward.

First, create the new DB copying the schema over from the old DB:

CREATE TABLE ebd_sorted AS
FROM
  ebd_unsorted.ebd_full
WITH
  NO DATA;

Then, just iterate through the months of the years and push them into the new DB:

months_of_year=(1 2 3 4 5 6 7 8 9 10 11 12)
for MONTH in "${months_of_year[@]}"; do
  duckdb path_to_output_db <<EOF
    INSERT INTO
      ebd_sorted
    FROM
      ebd_unsorted.ebd_full
    where
      extract month = $MONTH
    ORDER BY
      observation_date;
EOF
done

Thankfully when used like this DuckDB will just compute the percentage for you:

[07:58:42] Processing month: 1
100% ▕████████████████████████████████████████████████████████████▏
[08:05:58] Processing month: 1 completed
Total execution time: 7m 16s
[08:05:58] Processing month: 2
10% ▕██████

Un-thankfully, it's entirely inaccurate when you're adding sorting at the end. The first time I ran this I crashed my machine since I had very little storage left. Deleting a couple dozen old or broken DuckDB's fixed this and the second time this ran to completion in about 90 mins. That's honestly a bit surprisingly slow, though a bit reason for that could be my quite silly chunking strategy. Another confusing thing is that the insertions get faster rather than slower in later batches. You can see there the first insertion took 7+ minutes while the last month took less than 5. I'm not entirely sure why this might be. I would have assumed DuckDB would have to do more work at the end to figure out the right insertion order?

In any case this worked, but leaves a lot to be improved upon in future iterations:

picking a better chunking strategy
figuring out how the raw TSV is sorted if at all. If it is it'd be great to use that for my chunking strategy
the dream: could I figure out a way to merge this step in with the prior one so that I'm iteratively sorting the data as I read it in from the TSV?

Here's the script if you care to take a look!

Back to the real purpose

The prior few sections were over a few days of intermittent work, so I was a little fearful that all of that might have been a pointless adventure into premature optimization that went actually nowhere in terms of fixing the actual problem: being able to show species likelihood across the full data set.

Thankfully, the sorting did appear to pay off and the first run with the full data set was successful! This species richness table creation is still painfully slow, but it finishes:

  Creating localities table with spatial columns... ✅ Created in 13.1s!
  Creating taxonomy table... ✅ Created in 0.1s!
  Creating hotspot popularity table... ✅ Created in 61.0s!
  Creating hotspots richness table... ✅ Created in 525.9s!
  Creating localities_hotspots optimization table... ✅ Created in 7.2s!
  Dropping hotspots richness table... ✅ Dropped in 0.2s!

Nearly 10 minutes to create that one table, but that's a small price to pay to be able to see this many hotspots on a map at once!

Conclusion

I really need to take a pass now at cleaning things up on the display side of this. I also should absolutely be limiting this query... each one of these runs returns 20MB of data. But, I think this is a nice place to wrap up this post. I'm not entirely certain where I'll go from here next. There are more than a few places to explore:

I'd love to be able to show more information to users about not just the number of species but characteristics about them.
I'd love to be able to use this data to now compute "where can I see the most lifers"? I'm already showing that on another page, but it's using the eBird API and is woefully slow to run. This is going to be hard though because I can't just aggregate the species count for efficiency's sake. I'll need to report back the actual list of species for the hotspot, which would add a whole lot of data.
My hope here isn't to have a bunch of different maps with disparate information on them. Instead, I'd like to support a single view for "Where should I go?" that takes into account richness, popularity, lifers to various data points on a hotspot.
Most of all, I really need to take a pass on polishing things end to end so I can actually share this with some birding friends and have them use it!

Continuing with DuckDB and eBird

2025-08-25T00:00:00Z

In the last part we looked at the "basics" of getting DuckDB set up with the full eBird dataset. It was slow, it used a lot of data, but we finally got to answer the real questions, like "Who's seen the most ducks?" Now, it's on to slightly more practical explorations. The goal in this post is to go from a large, still relatively unwieldy dataset, to something we can actually deploy and host on the personal site

I knew from the get-go we were going to have to get very creative in how we approached pruning down the dataset here. There are two primary reasons I knew we'd have to do this. One: there was no way I was going to be able to affordably deploy 200GB of data for my personal site. If it was a for-profit tech company, deploying something like this would be an afterthought. I could get this up and running on S3 or MotherDuck in a matter of minutes. However, doing so would incur major costs, or at least costs that sound major when it's your blog and not a venture-backed startup. I set myself a goal of getting this under 10GB from the start given that was the threshold where you could meet the MotherDuck "free tier". As with many things in my programming journey, this starting principle turned out to be woefully incorrect. While they do offer up to 10 GB of storage, this only comes with a finite limit of 10 compute hours. Hopefully I wouldn't blow through that too soon, but still I'd rather not deploy this to something with a known expiration date. Deploying it to Render, the existing place where I host my site, would still be fairly easy, but there I'd be paying a fixed cost per GB, so I have a real interest to get this data down to the smallest size possible

The other big limitation I had to think through was performance. DuckDB and tools like it are great for, and explicitly built for, offline analysis. In other words, they're perfect for ad-hoc questions, but they're not as purpose-built for repeatable, online analysis like when you're offering a tool to the internet that might need to answer the same questions over and over again in, ideally, a quick manner. Thankfully, these two ends can go hand-in-hand. Reducing the size of the data will very often result in speeding up the queries themselves too.

My initial plan for getting started was to use the DuckDB data to actually produce a different database: Postgres. Postgres is extremely popular, well-documented and suited to the exact thing I was doing here. It's also trivially easy to dump data out of DuckDB into Postgres with something like this:

-- attach to a postgres DB
ATTACH 'postgresql://localhost:5432/cloaca' AS postgres_db (TYPE postgres);

-- dump a table into it
insert into public."ebird" (
  select *
  from ebd_full."full"
  where
)

I think if I was focusing on doing this the "right" way I would have kept going this way. But something dawned on me as I was fiddling around with DuckDB. The queries I was running on it were already seemingly fast enough for "production" use. If so, why waste all the effort to get it into Postgres? Furthermore... I use Postgres all the time and while I wouldn't mind spinning it up again, getting my tables neatly defined and honing my indexes to perfection, it just sounds more fun to see if I could just get DuckDB up and running all on its own. I haven't checked if this is remotely a good idea. It might be. It might be a standard part of the tool? The main pro I could think of was a vague notion of cost savings. DuckDB (I think) compresses data a bit better than Postgres so I'd be having to spend less on the disk usage if I went this way. There's far more cons that came to mind: it's probably slower, harder to tune, harder to run myself and won't be taking advantage of the super rich ecosystem of Postgres monitoring and maintenance tools... All in all, sounds like a bad decision future-me will regret.

The two initial datasets

I knew there were two overall types of data sets I wanted to produce for the app out of the gate:

Hotspot popularity
Species likelihood

Hotspot popularity:

Something I really like to see when going to a new area is simply which hotspots are more popular. Sure, it's nice to go where the popular kids are, but this is really more a valid and super easy signal to finding a good place to bird. The most popular birding spots, at least in my area, are truly really good. They're where I like to go and where I'd recommend others to go too. It's also a nice signal that a place is safe and accessible. To state the obvious: this isn't a guarantee of any of these. Some spots are just popular cause, well, they're popular. Central Park is a great example of both. It truly is a really good spot to see birds, but also, is really, really popular too.

Here's what I wrote to calculate this:

    select
        "LOCALITY ID" as locality_id,
        extract(month from "OBSERVATION DATE") as month,
        -- I think i'm doing this right here?
        count(*) / count(distinct date_trunc('week', "OBSERVATION DATE")) as avg_weekly_number_of_observations
    from
        ebd_full."full"
    where
        "OBSERVATION DATE" > current_date - interval '5 year'
    group by
        1,2

A few quick explanation notes: this is "grouping" all observations in the last 5 years by their Locality ID (the unique ID in eBird for a hotspot) and the month of the observation. This is basically a way of saying "how many checklists were submitted in Central Park in October for the last 5 years?". I don't exactly want that though. I think a more intuitive metric is "how many checklists typically get submitted in a week there?" A hotspot with limited popularity might only have 1 or 2 per week, something with moderate popularity between 10-50, and then something that's very popular should be 100 or above. Some quick benchmarking against the "real" data on eBird supports this to be true. In May of this year, Green-wood cemetery saw 536 checklists. That's roughly 125 per week. McGolrick Park, my local patch, is far less popular but still sees steady traffic, so it'd be moderate with 114 total checklists or 29 or so per week. Then, the more peripheral hotspots would see somewhere between 2-20 checklists per month.

A few things already stick out to me as issues here and point to the fact that you can spend a LOT of time refining metrics and queries like this. This is an exercise I absolutely love. We start these analyses out with such a rich internal model of what feels like the "true" metric here. But then getting that metric written out and working requires so much refining. One thing about the McGolrick park numbers is that it shows the under-valuing of group checklists. The McGolrick park bird club can have 200+ people in attendance in a given week. This should arguably reflect in the popularity, but it won't. Simply multiplying the checklists count by the number of observers would solve for this. Another thing is the conversion to weekly numbers. In hindsight, I probably should have gone with monthly data purely as a convenient way to test my results against eBird. However, to me the "weekly number of checklists" seems just more intuitive. Other questions: should we count checklists or number of unique observers? When I see a semi-popular hotspot, would it be confusing to realize it's just 1-2 birders' local patch? I could spend days more digging into this, but for now I should just mostly move on...

Except there are at least two glaring bugs I wrote here. In my defense I was a bit doubtful in my own comment on that line.

1: count(*): this isn't counting checklists, this is counting observations! So my numbers were wayyy too high.

Oh yeah and they are. We need to count distinct checklists here instead: count(distinct "SAMPLING EVENT IDENTIFIER") should do the trick.

2: This is a more subtle bug but one I should know better than to do... given I've done this a million times before when writing this kind of time based average: / count(distinct date_trunc('week', "OBSERVATION DATE")). Naively, this seems right: the average is number of checklists / number of weeks. However, we're grouping by the hotspot and the month. Thus, if we're using the hotspots own data to determine the number of weeks for that month, that number might not be the actual months in that year. If that hotspot only had 1 checklist that month, then we could just be dividing 1/1. What we actually need to do is just use a constant number of weeks per month for every hotspot.

Here's the "final" query here that I ended up with. I'm gonna hedge that still since I know I'll probably find some more bugs:

with number_of_weeks_in_each_month as (
    select
        extract(month from "OBSERVATION DATE") as month,
        count(distinct date_trunc('week', "OBSERVATION DATE")) as num_weeks
    from
        ebd_full."full"
    where
        "OBSERVATION DATE" > current_date - interval '5 year'
    group by
        1
)
select
    "LOCALITY ID" as locality_id,
    extract(month from "OBSERVATION DATE") as month,
    -- I think i'm doing this right here?
    count(distinct "SAMPLING EVENT IDENTIFIER") / max(num_weeks) as avg_weekly_number_of_observations
from
    ebd_full."full"
    join number_of_weeks_in_each_month on extract(month from "OBSERVATION DATE") = month
where
    "OBSERVATION DATE" > current_date - interval '5 year'
group by
    1,2

number_of_weeks_in_each_month is still not logically perfect, but in actuality should be. I should just figure out how to generate a series here in DuckDB... but I can just throw that on the future list of things to do.

Having this, we could now "materialize" this into a more condensed dataset. By only storing the actual data we need, we'd be greatly reducing the size of the data and also the time it takes to analyze it. I used the first query here to create a table for "hotspot popularity" and then, at run time, I joined it against a separate table that tracks the actual details for a hotspot. Here's how that second table is defined:

select
    distinct LOCALITY,
    "LOCALITY ID" as locality_id,
    "LOCALITY TYPE" as locality_type,
    LATITUDE,
    LONGITUDE,
    ST_Point(LONGITUDE, LATITUDE) as geometry
from
    ebd_full."full"
where
    "OBSERVATION DATE" > current_date - interval '2 year'
    and "LOCALITY TYPE" = 'H'
    AND LATITUDE IS NOT NULL
    AND LONGITUDE IS NOT NULL
order by
    locality_id

There's nothing too much to comment on here. I'm only looking at the last 2 years of data to determine if a hotspot should be included (one of many steps to reduce the vast number of hotspots with little to no data in them). I'm also only considering public hotspots, not personal locations. The most interesting thing is the last column I'm selecting: ST_Point(LATITUDE, LONGITUDE). As I started playing around with this data I realized one extension of DuckDB I almost certainly want to use is its spatial tooling. I've never really worked with Geo data like this before, but I knew that a very important point of optimization would be quickly determining if a hotspot was within the area I wanted to search. You can mathematically figure this out using the raw decimal data of latitude and longitude but that would be quite error-prone for me to work out myself and also probably not nearly as performant.

Turns out doing this the way I did still resulted in both issues. Starting out, I had the distance filter entirely wrong. So even when I was looking for hotspots within 5km of a point in New York, I was returning basically every hotspot on earth. This was a pain to debug but eventually turned out to be me just not realizing that I was providing 1,000 times the correct value to ST_Distance_Sphere. A far more subtle bug was when my query resulted in a weird, cats-eye shaped ellipsis instead of a perfect circle. I honestly never figured out what caused this and I tried numerous fixes. Claude really went out of its way to suggest incorrect answers here, telling me I needed to make custom Mercator projections to fix it. What ended up working was to swap the latitude and longitude when I created the table up above in ST_Point. It feels like when faced with most Geo data bugs, the first trick should always be to try swapping the latitude and longitude.

In any case, here's what I ended up with for the run-time query to analyze this data:

SELECT
    localities.locality_id,
    LOCALITY as locality_name,
    LATITUDE as latitude,
    LONGITUDE as longitude,
    avg_weekly_number_of_observations
FROM localities
JOIN hotspot_popularity ON localities.locality_id = hotspot_popularity.locality_id
WHERE locality_type = 'H'  -- Only hotspots
    AND ST_Distance_Sphere(geometry, ST_Point(?, ?)) <= ?  -- Great circle distance in meters
    AND hotspot_popularity.month = ?
    and hotspot_popularity.avg_weekly_number_of_observations >= 1

We're just joining my localities table against the hotspot popularity one, providing parameters for latitude, longitude, the max distance in meters and lastly the month to use. I'm also filtering out any hotspots with less than 1 avg weekly number of observations since it's just kinda weird to show 0 on the map. Note: this is something to revisit (probably really soon), or else sparsely visited areas might not have much useful to display. Bigger note: it feels like sparsely visited places are one of the hardest challenges for displaying useful information for birding.

One last note on the backend implementation: I'm also defining an RTREE index on these geometric points too, using the following:

CREATE INDEX idx_localities_spatial ON localities USING RTREE (geometry)

I experimented very briefly with defining other indexes but found that their extra size (nearly doubling the DB size), paid virtually no benefit. This was not at all scientific though and is another area for potential investigation especially if these queries end up bogging down and have some EXPLAIN'ing to do.

But... that's a lot of text and not a lot of showing the results. So with a quick 🥁, here's how it looks:

That looks roughly right for typical May activity in Brooklyn. Performance wise it's quite snappy too:

Executing get_popular_hotspots with lat: 40.65808990435005, lon: -73.96840797919282, radius: 10.0km, month: 5
[DuckDB Spatial] Query execution took 0.057s, returned 74 rows
[DuckDB Spatial] Result conversion took 0.000s
Time took to process the request and return response is 0.10329604148864746 sec
INFO:     127.0.0.1:51295 - "GET /v1/popular_hotspots?latitude=40.65808990435005&longitude=-73.96840797919282&radius_km=10&month=5 HTTP/1.1" 200 OK

I can then zoom out to the entire eastern seaboard too with similar perf:

Executing get_popular_hotspots with lat: 41.92017945145008, lon: -71.16751310883416, radius: 1000.0km, month: 5
[DuckDB Spatial] Query execution took 0.067s, returned 10138 rows
[DuckDB Spatial] Result conversion took 0.003s
Time took to process the request and return response is 0.1633131504058838 sec
INFO:     127.0.0.1:51327 - "GET /v1/popular_hotspots?latitude=41.92017945145008&longitude=-71.16751310883416&radius_km=1000&month=5 HTTP/1.1" 200 OK

Rendering this in an explainable way is a bit more challenging:

I'm just using my existing map plotting tools and I think they could use some adjustments here to show a less data at once, especially when zoomed all the way out. Still, it's rather cool to very quickly get a ranking of the popularity of spots at such a broad range as this!

Oh and one last note? How much space does this take? Well right now with just these tables, for the entire world, it's only a measly 466MB. That's far lower than I expected, though I really am not throwing much into at all yet. I'm more than a bit worried what will happen when I need a row for each species, for each month for each hotspot...

Getting this live

Now, you might notice, past-writer me and present me are in a bit of conflict here. See, now I'm supposed to go ahead and move on to the species likelihood mapping. But I do have one big concern. I've sunk a few hours into this exercise thus far and while I have promising results locally, I really have no idea how this might be perform on a hosted site. So I think before we spend more time adding data, I think it might be worth pausing here to see if we can even deploy what we currently have. My rough plan is to just send the "parsed" DuckDB up to Render as an attached volume and then read it from there... but I am also starting from square 1 this arena.

Render makes this quite easy though and I can get 1GB of storage attached at < $0.25/month. Creating this took about 5 seconds and hopefully loading it with data should be just as easy since I can SCP data in from my local machine. I didn't have an SSH key added to my account yet but this also was quite easy. Honestly I have no interest to shill for Render, but kudos to them on their docs in this whole process. This can always be a pain to get right and their docs were the right level of "here's an example that's probably what you want" along with great deep links from their docs to the right place in their product.

Within 1 minute I was SCP'ing the file up to the disk:

% scp -s ~/parsed_full.db srv@ssh.oregon.render.com:/var/data
parsed_full.db    46%  207MB   4.2MB/s   00:56 ETA

It took longer (though not even that long), just to upload the parsed dataset than it did to get the drive working. Now I just need to tell Cloaca where to look for this data and we should be good! Thankfully I was already just reading this from an ENV variable DUCK_DB_PATH so as long as I provide that to the server in Render this should work... In a real deployment, we'd want to have some sort of separate environment to test something like this. But as of this writing this service has a 1:1 ratio of developers to users so it's gonna be fine if we break a few things.

Once this was deployed it was simple enough to test with a quick request to the new endpoint... and, as expected, there's a bit of trouble:

IO Error: Extension "/root/.duckdb/extensions/v1.3.2/linux_amd64/spatial.duckdb_extension" not found.
Extension "spatial" is an existing extension.

This is somewhat expected. While I've delivered the DB file to the server, and that server does have duckdb installed to it via the python wrapper, I can imagine there are some extra steps to actually getting it running there. With a quick google, and boy do I ever have to give the Docs teams of the world credit today, we found the exact solution: https://duckdb.org/docs/stable/clients/python/overview.html#loading-and-installing-extensions. We just need to add these two lines and we should be good:

con.install_extension("spatial")
con.load_extension("spatial")

And trying again... it works! But it took 5 seconds. This is certainly less than ideal, but I still think it's worth getting out the door so we can learn a bit more from more production-like usage. Speaking of which...

Using it live

As of that writing, I took an ice cream break with E. She asked for a quick demo and I started showing her the NYC area. However, we both were curious to apply this to a real-life use case. We're planning a trip in a few months to New Zealand, so I spun the globe over there to see how it looked. It was honestly so great to very easily have the entire country in the view and then to see the top 10 or so hotspots. What was even cooler was to see that at least half of them, and all the top 5 locations, were places we were planning on going! This hopefully is a good validation that the data this is showing is actually relevant for trip planning.

That said, though, this is far too slow. So let's try and make this faster.

To look into this, I'll run these queries locally and also on the Render VM. Thankfully it's pretty easy to just run DuckDB in both and compare the results from explain / analyze. Here's the initial result from the VM:

┌────────────────────────────────────────────────┐
│┌──────────────────────────────────────────────┐│
││               Total Time: 2.78s              ││
│└──────────────────────────────────────────────┘│
└────────────────────────────────────────────────┘
┌───────────────────────────┐
│           QUERY           │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│      EXPLAIN_ANALYZE      │
│    ────────────────────   │
│           0 Rows          │
│          (0.07s)          │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│         PROJECTION        │
│    ────────────────────   │
│__internal_decompress_strin│
│           g(#0)           │
│             #1            │
│             #2            │
│             #3            │
│             #4            │
│                           │
│          60 Rows          │
│          (0.00s)          │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│          ORDER_BY         │
│    ────────────────────   │
│      parsed_full.main     │
│    .hotspot_popularity    │
│.avg_weekly_number_of_obser│
│        vations DESC       │
│                           │
│          60 Rows          │
│          (0.00s)          │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│         PROJECTION        │
│    ────────────────────   │
│__internal_compress_string_│
│        uhugeint(#0)       │
│             #1            │
│             #2            │
│             #3            │
│             #4            │
│                           │
│          60 Rows          │
│          (0.00s)          │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│         PROJECTION        │
│    ────────────────────   │
│        locality_id        │
│       locality_name       │
│          latitude         │
│         longitude         │
│avg_weekly_number_of_observ│
│           ations          │
│                           │
│          60 Rows          │
│          (0.00s)          │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│         HASH_JOIN         │
│    ────────────────────   │
│      Join Type: INNER     │
│                           │
│        Conditions:        ├──────────────┐
│ locality_id = locality_id │              │
│                           │              │
│          60 Rows          │              │
│          (0.02s)          │              │
└─────────────┬─────────────┘              │
┌─────────────┴─────────────┐┌─────────────┴─────────────┐
│         TABLE_SCAN        ││         TABLE_SCAN        │
│    ────────────────────   ││    ────────────────────   │
│           Table:          ││     Table: localities     │
│     hotspot_popularity    ││   Type: Sequential Scan   │
│                           ││                           │
│   Type: Sequential Scan   ││        Projections:       │
│                           ││        locality_id        │
│        Projections:       ││          LOCALITY         │
│        locality_id        ││          LATITUDE         │
│avg_weekly_number_of_observ││         LONGITUDE         │
│           ations          ││                           │
│                           ││          Filters:         │
│          Filters:         ││     locality_type='H'     │
│          month=5          ││    (ST_Distance_Sphere    │
│avg_weekly_number_of_observ││  (geometry, '\x00\x00\x00 │
│        ations>=1.0        ││  \x00\x00\x00\x00\x00\x00 │
│                           ││  \x00\x00\x00\x01\x00\x00 │
│                           ││  \x00\xF46\xB5p\xC8\xF5D@ │
│                           ││ \x04\x06\xE7\x88\xB8\xCAQ │
│                           ││ \xC0'::GEOMETRY) <= 30000 │
│                           ││            .0)            │
│                           ││                           │
│         52280 Rows        ││          362 Rows         │
│          (2.45s)          ││          (0.19s)          │
└───────────────────────────┘└───────────────────────────┘

Something I noticed right away is my index isn't being used! I glanced over it above, but I actually initially started off using ST_DWithin but stopped using it since it wasn't really what I wanted. Someone halfway proficient in Geo data could probably spot the issue and solution right away... but I'm only seeing the first half. Basically if I want to use an index I should use ST_DWithin but if I want accuracy I need to use ST_Distance_Sphere. I wonder if I could get away with using both? Use the former for an initial efficient narrowing of the points and then ST_Distance_Sphere to narrow that down to an actually accurate picture? I played around with this and while it sometimes was used by the query planner, most of the time DuckDB would just skip the index and default to a sequential scan. Claude seems to think I just need to define an index on the month of the observation and everything will be fine. I think it just thinks it's operating on a Postgres DB and is wrong. It's probably actually right and I'm ignoring it incorrectly, but let's try and John Henry this a bit longer.

DuckDB does have excellent docs across the board, so I took a glance at their docs on performance. Two things stuck out to me: 1) they really highlight the importance of ordering here when possible. 2) They talk about the inefficiency of joining on strings here. I know we're not explicitly ordering these tables and I also know we're using strings as our IDs in most cases (Locality IDs). 1) is because I didn't think about and 2) is because the key is a string... or does it have to be? So let's try both. Let's order both tables when parsing them. And then let's try converting the hotspot ID to an integer.

Ordering Data

For ordering... well it turns out I already was ordering the localities table but not the hotspots one. Let's see what happens there. Just saying this again, these databases are fascinating, truly. Ordering the hotspot popularity table vastly reduced the initial DB creation performance (going from a few seconds to over 2 minutes), but the DB is now nearly 1/3 the original size (250MB vs 650MB before).

Let's just replicate that on the VM to test things out by creating a quick new version of that table:

create or replace table hotspot_popularity_v2 as (
      select * from hotspot_popularity
      order by locality_id, month
  );
...
Connection to ssh.oregon.render.com closed by remote host.
Connection to ssh.oregon.render.com closed.

Oh no. Ran out of memory (used over 512MB) while running your code.

Might have to just rebuild the DB and push it back up. Thankfully it's a lot smaller now!

Unfortunately, if there was a speedup here it was a modest one. On to the IDs!

Using integer IDs

Let's peak into the locality IDs:

┌─────────────┐
│ locality_id │
│   varchar   │
├─────────────┤
│ L1          │
│ L10         │
│ L10000087   │
│ L1000010    │
│ L10000105   │
│ L1000013    │
│    ·        │
│    ·        │
│ L10003217   │
│ L10003228   │
│ L10003279   │
│ L10003410   │
│ L1000354    │
└─────────────┘

So these should be pretty easy to just turn into integers with something like:

CAST(SUBSTRING(locality_id, 2) AS INTEGER) as locality_id_int

I think it might be nice to only store these as integers and then be able to convert back to the string version. However, I'm not seeing an obvious pattern there in how the string version is being created. I'm sure I could finesse some sort of left-padding... or I could store both for now. We'll store both in the localities table and then we'll only use the integer ID in hotspots. Somehow that further reduced the DB size, now to just 165 MB? I should state at this point I am in no ways confident in my method of sizing the database...

Nor am I that confident in how I'm measuring performance. It feels like I halved the performance here again, but these queries are still taking 1-2 seconds. It's at this point that I'm starting to feel truly silly for spending more and more time trying to get an analytic database to perform like this.

Trying other tactics

Stepping back and looking at that explain again, I'm thinking that maybe the join isn't really the expensive part. What looks to me to be more problematic is the sequence scan on the localities table:

┌─────────────┴─────────────┐
│         TABLE_SCAN        │
│    ────────────────────   │
│           Table:          │
│     hotspot_popularity    │
│                           │
│   Type: Sequential Scan   │
│                           │
│        Projections:       │
│      locality_id_int      │
│avg_weekly_number_of_observ│
│           ations          │
│                           │
│          Filters:         │
│          month=5          │
│avg_weekly_number_of_observ│
│        ations>=1.0        │
│                           │
│         52421 Rows        │
│          (0.26s)          │
└───────────────────────────┘

I would love to somehow force DuckDB to try and use the index I defined on the lat/long geometry for this table I defined with:

CREATE INDEX idx_localities_spatial ON localities USING RTREE (geometry)

However, I'm not seeing an easy way to force the query planner to use this index like you can in other DBs. Digging through the docs, I noticed 1 concern right away:

One of the arguments to the spatial predicate function must be a “constant” (i.e., an expression whose result is known at query planning time). This is because the query planner needs to know the bounding box of the query region before the query itself is executed in order to use the R-tree index scan.

Oof. I don't think this would work in my query case since I'm using parameters like so:

...
AND ST_DWithin(geometry, ST_Point(?, ?), ?)
...

Maybe I could just do the query templating myself before hand to make this a more static query?

Playing around a bit with some simplified queries, though, I realized that even a simple, static ST_DWithin with a ST_Point wasn't using the RTREE index on either my local DB nor on the slower VM. However, if I just threw in an ST_Envelope instead suddenly the index would be used. This is surprising and confusing. It turns out that ST_DWithin isn't supported but ST_Within is... oof... More oof's ensued after this as I tried many various query patterns to try and get an index to be used when joining against the hotspot data. I found a lot of different ways to get an index to apply to a query just against localities (the table with the lat/long geometry on it), but nothing when a join was applied to it. I very well could have missed something, but I, and subsequently Claude, tried many different combinations of query definitions with no dice.

I did have another thought when trying out various ideas: what if I could just add the geometry data to the hotspot data. Or better yet... why even have 2 tables if really only need 1. I had started off with separate tables because of separation of concerns and whatnot, but really if I continue down the path of "only return data for a hotspot" then I might realistically can get away with only querying a single table. If I did that then the join condition might be entirely obviated and I might get better index performance?

So I created a simple, unified table... and presto:

┌────────────────────────────────────────────────┐
│┌──────────────────────────────────────────────┐│
││              Total Time: 0.0530s             ││
│└──────────────────────────────────────────────┘│
└────────────────────────────────────────────────┘

That's on the VM. That plan didn't even use the index nor does it seem like it even needs it!

Conclusion

At this point, it's probably time to wrap up this rambling post. I sure did learn a lot doing this, though it's a bit difficult to sum up neatly. Here are some sundry bullet points firing around at the end:

DuckDB is still very cool. It's super easy to get up and running, the docs are great and I was able to try things out, deploy them, adjust them live all with minimal effort.
Using it as a production style DB is fairly silly. I'm sure I could avoided half the mistakes and rabbit holes here if I just started using Postgres sooner.
On using Claude / AI for things like this: I don't think Claude has a great idea of how to work with DuckDB. Most of its strategies were built for relational DBs, like its tendency to just say "you need an index for this" even though DuckDB strongly recommends against using indexes for most things. Even giving it direct access to the DuckDB docs really didn't help. What it really shines at though is being a test bench of sorts for various strategies. It can write, run and analyze queries so much faster than I can. Giving it a few ideas and test strategies and then watching it go performed really nicely. Asking it to solve my problems left a bit more to be desired.
Like with a lot of criticisms (valid or not) of AI, they also apply to humans. My instincts from the get-go were "oh this query isn't using an index, let's jump through hoops to get it to". DuckDB doesn't really tend to want to work that way, and when you're working with an analytics DB for the first time it might make sense to read the docs more.

My Review of the PATH

2025-08-25T00:00:00Z

Cursed Knowledge

2025-08-25T00:00:00Z

Cursed knowledge I have learned over time that I wish I never knew. Inspired by Immich's Cursed Knowledge. The knowledge is ordered from most to least recently learned. \> Ajv multipleOf validation is…

Analyzing the full eBird Dataset with DuckDB

2025-08-16T00:00:00Z

Ever since I started birding, I’ve had an ever growing list of questions. Questions about the species I’ve seen, the ones I haven’t, the habitats they’re found in, the activity and the participants itself. When do Eastern Phoebes typically show up in New York? Where do they go when they leave? Where do they like to nest? How frequently are they seen in New York? Why does that frequency dip in the summertime?

The Journey

I love having this mounting list of questions as it’s one of the key drivers of a “mid life renaissance” of sorts in my curiosity. I’d love to write about that spark even more soon, but for now I’d like to talk about one of the avenues I’ve turned to both quench and deepen this curiosity. Soon after getting into birding, I started getting quite curious about the underlying data that birding both depends upon and also adds to. Birding, more so than any other hobby I’ve encountered, focuses on both the reading and contribution of data. Birders study weather forecasts, migration timelines and previous year’s records to answer questions like: “when are the first migrants of the year going to show up?”, “are the winds out of the South this weekend?” (a good sign here in New York in the spring that tropical migrants will be coming in, born along by the favorable winds) or “I just saw an owl species in Brooklyn. Which species might be most likely that matches the brief glimpse I got?” Just as interesting, and particularly compelling to many of us, is the fact that by recording our checklists, taking photos and recording audio that we’re also adding to the ever growing data set of birding observations (most commonly here in the US via Cornell’s eBird).

As I got started into birding I was participating in both of these, but I was also starting to get curious about what the underlying data might look like. Soon into my birding journey I found out that eBird had an API you could use to access this data. I’ve already made more than a few fun projects off of that, but one of the limitations I ran into there is that an API like this is best suited for answering questions served by targeted, smaller data sets. It was great to see a list of the most recent species in an area or to get more information on hotspots around me. But, for questions that stretched across larger areas or across long time periods, it would take quite a long time (and more than likely hitting some rate limits) and not be that efficient to use an API for this. APIs like this are great at serving small-to-medium amounts of data very quickly based on specific filters or conditions. They’re not so great at providing deeper or richer datasets spanning thousands or millions of entries. And while my questions weren’t of the “millions” (yet), I was running into the limitations of the API when it came to many of my questions. Thankfully, eBird offers another route to accessing its data, via pre-packaged downloads of large swaths of data. This wouldn’t have the immediate, quick accessibility of the API, but would more than make up for it in its comprehensive and rich dataset.

What’s funny to me is that it took me nearly a year before I actually downloaded my first segment of the dataset. I can’t say the exact reason why, but one of the reasons why surely had to do with my unfamiliarity around digging through data like this. This isn’t to say I wasn’t comfortable or even happy around data analysis. While working at Stripe I had the distinction (happy or not) of being one of the top users of our shared analytics system (being summoned on more than one occasion to explain I insisted on running so many SQL queries every day). But it’s that last part, SQL, that maybe best explains my hesitancy with this form of data querying. See, the main recommendation for working with the eBird dataset was with R. No shade meant to the language, it’s just not something I had really ever worked with before. So, while I had a little fun poking around at the dataset before, I very quickly got frustrated by both the slowness of asking relatively simple questions via R (and by slowness I mean my slowness in figuring out just how to get anything working with it). I put this aside and promised myself I’d return to it another day when maybe my curiosity was willing to drive me a bit further.

Sometime later, however, my curiosity actually led me in a different direction. If I was having such a hard time (and honestly not a fun time) trying to look at this data in this way… why not just try and do it the way that I was used to? To be fair to me, I think I assumed that using what I used at Stripe would be just too cost prohibitive and difficult for a small personal project (I mean it’s called a data warehouse for starters). On a whim, I asked if what I wanted was even possible on my own humble computer. Not only was I not discouraged, but right away I was pointed to a good solution.

Enter DuckDB. Okay, right from the get go, it has to be prefect right? It has Duck in the name. But, before we go further, let’s take a brief aside to even talk about what’s going on here in a more detailed manner.

The problem is basically this: in its raw form the ebird data set is just lines and lines of text separated by tabs (the sister file format to the very commonly used CSV). In fact, you probably can read this here and get a sense of what the data includes!

URN:CornellLabOfOrnithology:EBIRD:OBS2919749158	2025-03-01 23:25:39.781016	21333	species	avibase-69544B59	American Crow	Corvus brachyrhynchos				2					United States	US	Alabama	US-AL	Tuscaloosa	US-AL-125		27			Lakefield Drive, Tuscaloosa	L10510092	H	33.1559565	-87.6336990	2025-03-01	06:21:00	obsr1073731		S215984764	Traveling	Traveling	P22			24	1.092		2	1	G14142220	0	1	0
URN:CornellLabOfOrnithology:EBIRD:OBS2933256785	2025-03-05 07:43:54.669907	21333	species	avibase-69544B59	American Crow	Corvus brachyrhynchos				1					United States	US	Alabama	US-AL	Tuscaloosa	US-AL-125		27			23rd Street Area, Tuscaloosa	L12228602	P	33.1882745	-87.5396724	2025-03-05	06:27:00	obsr451424		S216708215	Stationary	Stationary	P21			15			1	1		0	1	0
URN:CornellLabOfOrnithology:EBIRD:OBS3009418896	2025-03-30 17:15:29.683739	21333	species	avibase-69544B59	American Crow	Corvus brachyrhynchos				3					United States	US	Alabama	US-AL	Tuscaloosa	US-AL-125		27			River Bend Turf	L5039140	H	33.1335213	-87.6534175	2025-03-30	15:11:00	obsr451424		S221934152	Traveling	Traveling	P22			60	13.532		2	1	G14355008	0	1	0

That’s nice for general readability and it also makes working with this across a broad audience and technical background quite feasible. The downside here is that, in this form, this file is prohibitively large and also quite inefficient to use. I’m not going to go too deep into the technical reasons why, but suffice it to say that, by default, most any analysis of this file is going to require reading each and every line: all 1,809,934,873 of them for the full file. There are lots of different ways we could make this easier! Possibly the most commonly used method would be to throw it into a general purpose database like Postgres. This would certainly speed things up and reduce the size of the file. However, a database like this isn’t really suited for what I want to do here. I want to be able to ask near arbitrary types of questions without spending time before hand defining things like indexes which are basically telling the database which axes I want to analyze data along. I could define an index for the date of the observation and then looking only for observations on a certain date would be blazingly fast! But this comes with 2 downsides: first, I’d need to wait a lengthy amount of time for the database to create that index (it needs to sort through all the data to see where it lives on that axis) and, moreover, I’m interested in so many more axes than that!

The thing I really wanted was a columnar database! Not only would a columnar index not really require indexes, but it was perfectly suited for this kind of “Ask me anything” style approach. The thing that I didn’t know was that columnar databases can be used in any variety of uses cases: from the large corporate behemoths to me poking around to see where the best place to see an Eastern Phoebe in March was. The exact reason why columnar databases are so good is honestly still beyond me. But I do know one of their biggest advantages is how they treat the data they store. See, in a traditional database, all elements in one of the rows in that TSV are going to be stored together. That’s quite useful when you’re trying to do something like look at one specific observation or a few to see all the relevant details about it: the species, the date, the observer, the location etc. Columnar databases flip this idea on its head though and treat everything by column and not by row. In an analytics use case, I don’t care as much about all the details of one observation; instead, I’m far more interested in a few details about a large list of observations. By storing data by column, they make it far more efficient to analyze along these. So, instead of thinking of these entries as one row (“I saw an Eastern Phoebe, on May 1st at Prospect Park on this checklist”), it flips this to instead sort things by the species, or date, or location. (How do they do this so quickly? I still have no idea… I believe it has something to do with the fact that they’re storing all this data along an index by default?)

In any case, we’ve got our data principles set: now we can talk about ducks again. DuckDB caught my interest right away based on the simple install instructions (geared toward a local user), the ease at which I could work with CSV files, the fact they had just shipped a UI for running queries (I’m all for running my queries in a terminal, but in 2025 I don’t really wanna be interpreting tab and pipe characters in one) and, I mean, come on, they have Duck in the name! I loaded up the very small starter dataset I had and, glancing at the instructions, I had it working in less than 30 seconds. Here’s where I should start to really emphasize my unfamiliarity with what I was doing (if I haven’t laid it on thick already). I started trying to figure out if I needed to put this in S3 (a cloud-based storage system that probably backs nearly everything on the Internet in some way) or some other remote file storage system. I started playing around with just the right way to store this in Parquet (a newer file storage format far more suited for this problem than a TSV). I was game-planning the right way to stream the data from eBird’s server, through my machine as a TSV, up into the cloud as Parquet… when it dawned on me. This TSV was large, but not prohibitively so: 137GB. And while I didn’t have a spare cloud server sitting around… I did have quite a number of unused hard drives from my color editing work with my brother. Why not just throw the file on that?

I thought I’d go from that step to querying data in an hour or two… but boy was that over-confident. For one, the download would end up taking far longer than that. Not only cause I didn’t have the best internet speed, but my drive also kept going to sleep halfway through it… in any case, it was actually for the best, since it meant I spent more time testing and playing around with a more modest 25GB data set instead. This really paid off in the long run since I actually figured out a good plan with a quicker dataset than just trying and failing repeatedly on the first one. Here’s the first thing that threw me off: I had assumed that I needed to figure out my own specific method of data storage. It turned out that, while DuckDB will happily spit out Parquet (and other forms of data), that’s not really the “database” it’s working off of! Instead the Database is its own file… in any case, it was definitely fun to see how Parquet can drastically reduce the size of the download! With the 25gb NY data set it reduced it down to just 3gb… and I’m not even sure I was doing any compression with Parquet either! (See the questions below for areas I still need to explore) [and an editor’s note; he certainly was doing compression].

So I didn’t need Parquet to store the data… but, as usual when I’m doing anything technical (or honestly, just anything) I was far ahead of myself. I was already thinking of how to store the data when I first needed to not only finish downloading it… but then figure out how I’d even get that huge TSV file uncompressed and into DuckDB. The first part, decompression, on any modern computer is typically just as easy as double clicking the file, but that’s not as straightforward when you’re dealing with output that will end up exceeding 600gb (the native decompressor on my Mac never worked on this and was painfully slow). One obvious problem here was that I didn’t exactly have 600gb to spare on my local machine. However, if I moved this over to one of those aforementioned external drives then I ran into another issue: that these drives get real bogged down if you’re trying to read from them while writing to them. Put another way, if I’m trying to decompress the data into another file I need to tell the drive “Hey take these chunks of data and tell me what they are. Oh also, while you’re doing that, also remember these other chunks.” For, again, complicated reasons, a lot of drives don’t love doing this. They’d prefer one operation or another, but not both. A straightforward workaround would be then to read the file from my local drive and write to the external one. An even better find, however, was when I started looking for a tool to do this and realized that there were far quicker de-compressors available than the default one that lives on my machine (they use multiple cores to accomplish this speedup). Enter pigz (continuing with the animal-named tools on this one I guess):

@����n+Y�%���@>��Xdƺ�E�b�"���[���U�h$Bd���`ɭT���yn��6�l����…

Uhh that’s not quite right… oh yeah, you need to tell this tool to decompress, but default it will just recompress things. Okay adding the right arg to pigz, and we’ve got data! Loading this into DuckDB was a bit more involved as I had to fiddle a bit more with the loading arguments to get everything right. The main issue was around how restricted characters in the TSV are quoted (isn’t always the case when dealing with these?). But… we’ve got the data in! Again, this is with the more limited NY only data set, yet still, it only took 30s to create the DuckDB table for this. That’s way faster than any operation using R ever took! We can ask our first question… When was the first Eastern Phoebe observation in NY?

“Mounted at Wards [$1.25]; destroyed February 1907”

Always a nice reminder about how ornithology used to be just a bit more grisly. We’re close to realizing the dream! Writing that question was quick and the result was even quicker: 58ms. Now, we just need to work on the final goal, loading up the full data set.

The key challenge here is still around file size. The uncompressed full dataset is around 600 gb. While I do have drives that can handle that, it’s a bit more difficult to figure out how best to decompress that file. What I ended up doing to avoid filling up my hard drive (which I mistakenly had already done twice) and to have this finish in a relatively speedy fashion was to actually keep the compressed file on my computer’s hard drive and then decompress the file directly to the external hard drive. This ended up taking about 2 hours which is far from great, but at least we have something that works! Something I’d like to figure out in the future is a better story around downloading -> decompression -> loading into DuckDB. Ideally one that’s quicker and involves less need for intermediate storage. You should be able to stream or pipe more of these processes to each other so that we don’t have to wait for the entire 600gb TSV file to be ready before loading it into DuckDB. The main reason I didn’t spend much time on this yet is that, a) it’s easier to reason about what I did b) premature optimization is the death of so many of my projects and c) dealing with issues is a bit harder in the streaming case than it is when I have more discrete check points along the way.

Setting that aside, I’m now one step away from the final dream. I need to get this data into DuckDB. One downside I discovered here with DuckDB is that it’s pretty hard to track the timing of larger tasks like this. There’s not a way to get a progress bar, that I was able to find. A strategy I used for this, and a few other tasks here, was to use times from the smaller NY dataset to then estimate the time for the full dataset. This isn’t a perfect strategy, given the non-linear impact of compression and other things like disk caches that will have more of an impact on the smaller file sizes. Still, it was actually pretty good at predicting things like the final row length of the DB (off by about 5%), but less good at predicting the disk size. It ended up being around 214GB which is way bigger than I was planning for. It turns out DuckDB isn’t going to magically solve my disk size needs. Still, that’s a problem for future David (a fun problem no doubt) and not the focus of the day.

The focus of the day is answering some burning questions, like who has seen the most ducks in a day? It surprisingly isn’t a tie… it’s this checklist here with 11 ducks: https://ebird.org/checklist/S16191277. Wood Duck, Mottled Duck, Ruddy Duck… Great work here. Plumed Whistling-Duck, Mandarin Duck, Ruddy Shelduck, what on earth is going on here? Oh all of these are escapees… that doesn’t feel right in the spirit of “most ducks.” I think that should mean wild ducks. If we filter those pesky escapees out… we’re down to 10 ducks reported. Still, surprisingly, there’s no tie! Only 1 leader emerges and it’s from 1996 in Ethiopia: https://ebird.org/checklist/S77656343. Here’s the query:

select
    "SAMPLING EVENT IDENTIFIER",
    country,
    state,
    "OBSERVATION DATE",
    count(distinct "SCIENTIFIC NAME") as no_of_ducks_seen,
    array_agg("COMMON NAME"),
    array_agg("OBSERVATION DATE")
from
    ebd.full
where
    "COMMON NAME" ilike '%duck%'
    and category not in ('slash', 'spuh')
    and APPROVED = 1 -- no escapees here!
    and (
        "EXOTIC CODE" is null
        or "EXOTIC CODE" = 'N'
    )
group by
    1,
    2,
    3,
    4
order by
    no_of_ducks_seen desc
limit
    100

But another problem you say! That’s not the true definition of duck! You need to filter to the species in Anatidae… true that! But definitely another challenge for another day.

Speaking of which, boy is there more work to do here. I’ve loved being able to scratch data questions easily and quickly. But it’s quick impracticable to have a 200+GB database sitting around your hard drive. It’s also unwieldy to have to leave things running for hours to refresh the dataset each month. Beyond that, I’d love to be able to use this in more broadly reachable tools. DuckDB offers a hosted version of their service and they have a free tier you can use if your dataset is below 10GB. A 5% reduction sounds really difficult, but I haven’t engaged much at all in reducing the dataset. For one, there are numerous columns here I don’t really need (comments, redundant taxonomic information, checklist IDs). In fact, it’d be a quite fun activity to see just what columns or portions of the DB actually contribute to its size. I also probably don’t need all data to answer most questions I have.

Here are some other outstanding tasks I’d love to look into: - All the questions around performance here. Could I use ordering for better indexing? https://duckdb.org/docs/stable/guides/performance/indexing#the-effect-of-ordering-on-zonemaps - Why is the first eBird checklist from Thailand on 1481-03-24 - Is there a form of Parquet compression that would reduce the size even more? - I should clean up data types more (move from integer to binary). This will reduce size more! Probably shifting things to enums could be massive too? - Convert column names to underscore to make them easier to auto_complete. - Figure out how to get x/y coords into data to perform distance analysis more quickly!

info

One post-writing note that I figured out a few days later: at a few points I mention that I needed to change the file format or data types to take advantage of DuckDB’s compression. This isn’t necessarily true. I found out subsequently that DuckDB is already quite smart at inferring the right data type for a column all on its own. So there’s less tuning needed at the onset (great!), but it also means there aren’t as many quick wins as I imagined.

Step-by-step guide on how to load the EBD into DuckDB

Ingredients

pigz or another tool for decompressing the EBD into a TSV. If you're not using a huge data set, you might can just use your native OS's decompression tool.
duckdb: download instructions here
the EBD! Sign up and request data here

Steps

Step 1: Download your Data

This can take a few minutes or a few hours. Note, Cornell's download speeds seem to be capped at around 8-10MB/S so plan accordingly!

Step 2: Decompress your data

You can do this whatever tool you like. I liked using pigz for performance. Here's a handy snippet I used to use pigz with progress tracking:

pv ebd_relJan-2025.tsv.gz | pigz -d > ebd_relJan-2025.tsv

pv is used to monitor the progress of data coming through the pipe.

At the end of this step you should have the "raw" TSV, looking something like this:

URN:CornellLabOfOrnithology:EBIRD:OBS2919749158	2025-03-01 23:25:39.781016	21333	species	avibase-69544B59	American Crow	Corvus brachyrhynchos				2					United States	US	Alabama	US-AL	Tuscaloosa	US-AL-125		27			Lakefield Drive, Tuscaloosa	L10510092	H	33.1559565	-87.6336990	2025-03-01	06:21:00	obsr1073731		S215984764	Traveling	Traveling	P22			24	1.092		2	1	G14142220	0	1	0
URN:CornellLabOfOrnithology:EBIRD:OBS2933256785	2025-03-05 07:43:54.669907	21333	species	avibase-69544B59	American Crow	Corvus brachyrhynchos				1					United States	US	Alabama	US-AL	Tuscaloosa	US-AL-125		27			23rd Street Area, Tuscaloosa	L12228602	P	33.1882745	-87.5396724	2025-03-05	06:27:00	obsr451424		S216708215	Stationary	Stationary	P21			15			1	1		0	1	0
URN:CornellLabOfOrnithology:EBIRD:OBS3009418896	2025-03-30 17:15:29.683739	21333	species	avibase-69544B59	American Crow	Corvus brachyrhynchos				3					United States	US	Alabama	US-AL	Tuscaloosa	US-AL-125		27			River Bend Turf	L5039140	H	33.1335213	-87.6534175	2025-03-30	15:11:00	obsr451424		S221934152	Traveling	Traveling	P22			60	13.532		2	1	G14355008	0	1	0

Step 3: Load the data into DuckDB

There's a wide variety of ways to do this. I'm including the simplest version I found here:

create TABLE ebd.full AS
SELECT
    *
FROM
    read_csv(
        '/path/to/decompressed/ebd.tsv',
        store_rejects = true,
        quote = ''
    );

The three things I'd call out there are:

We're using read_csv to well, read the TSV. DuckDB is smart enough to detect the delimiter.
We're storing the rejected lines. It's always good to review these to make sure everything went alright!
The last, most important part is quote = ''. If you don't do this most if not all lines will get rejected for some reason...

Step 4: Write a query against your data!

The most fun part. Time to learn from the data!

Here's a query to see the first observation in the EBD:

select *
from ebd.full
order by "OBSERVATION DATE"
limit 1

The special hell of Bolt, Europe's Uber clone

2025-07-15T14:50:08Z

I was in Latvia a few weeks ago. Riga’s one of the Europeans cities without a good transit link from the airport into city. Snooping around online, I found that the recommended way to get a ride was the use of an app called Bolt, a European clone of Uber. I realize now that I didn’t actually check that Uber wasn’t available in Latvia, but I’m not against experimenting with a new app here and there.

I used it twice to get to and from the city center, and it worked perfectly. Neither of my drivers spoke English and I didn’t speak a word of Latvian, but that’s what technology’s for. The rides went off without a hitch and I got exactly where I was supposed to be both times.

I arrived in Lyon recently and figured, hey, this is Europe, why not try the European app again, and used Bolt.

Ride attempt no. 1

Car pulls into airport, drives to the waiting spot, stops up ahead of me, I walk over to it, driver pulls away, and leaves the airport. Mystified, I photograph the guy’s license plate as he drives off figuring I might need it for dispute evidence.

The driver doesn’t cancel the ride as he rides off into the distance, leaving me to do it, presumably so it falls to me to pay the app’s €7 cancellation fee.

Ride attempt no. 2

I cancel and try again. I get a ride parked not far off, but with a message: “This is an automated acceptance. This car is set to charge for another 45 minutes.” Sure enough, it’s unmoving and unresponsive, and eventually the ride times out (thankfully, avoiding another €7 charge).

Ride attempt no. 3

No message this time, but another car that appears to be charging and/or long term parked (it’s a Tesla, so I suspect charging again). I leave the app, waiting for the pick up to time out.

Ride attempt no. 4

I give up on Bolt, and switch to Uber. I match a driver right away. It’s almost suspicious how quickly I matched him. But this is good! Progress. He drives over and I walk up to meet him. I get in the car and we start moving. Finally, this fiasco is over.

But then a guy runs up to the driver’s window. Hey, he shouts, you’re our ride! We booked you on Bolt. We just talked about on the phone a few minutes ago, remember?

Knowing that his license plate and photo matches what’s on their screen, the driver doesn’t bother denying it, and instead just points to his phone’s screen and says, I pick up Brandur. See?

Even as the car’s “winner” (I’m not sure if this was because I got to the car first or the Uber fare was more favorable for the driver), I have principles, and of course don’t love this situation either, but my only alternative would be to get out and cancel the ride, for which I’d surely get hit with another fee. Unfortunately my best option is to stay quiet about it, let the Bolt user get another ride, and give the driver a low rating later. Naturally, the driver didn’t cancel the other guy’s Bolt ride (at least as far as I observed from the back seat), which would’ve left the user to eat the €7 fee.

As we drove away from the airport, I suddenly realized: wait! this must be what happened to me during my first ride.

Ride attempt no. 1, part 2

I go back into the Bolt app and open a support conversation. This option is purposely hidden deep inside submenus of submenus of submenus, so it took me five minutes to find it. I explain what happened and include the photographic evidence. From the first response it’s obvious they have me talking to an AI. I drop all formality, and type only the minimum viable number of characters to get the next response. The AI promises me a refund for my €7 cancellation fee, then proceeds to provide no refund.

Eventually I’m escalated to a human operator, who somehow manages to be worse than the AI. After explaining the situation again, I’m told that fine, in this extremely rare, never-before-seen, once-in-a-cosmic-era situation, they’ll refund the €7 fee. But don’t fuck up again!

Don’t worry Bolt, I won’t. My days of using you scam peddlers are over.

When something works well enough, it’s easy to take it for granted. As much flak as Uber and Lyft take, my experience with Bolt made me stop and think that even given 10+ years and hundreds of rides on both apps, my bad experiences have numbered like maybe, two? That sort of quality bar isn’t an easy thing to maintain.

Non-Euclidean Monitor Layout With Cursor Wormholes

2025-07-12T00:00:00Z

Occasionally injected clocks in Postgres

2025-06-29T17:48:16Z

In a standard app deployment that’s scaled horizontally across many nodes, we can expect the clocks to be a little askew across the fleet. It’s generally not a huge problem these days because our use of NTP is so good and so widespread, but minor drift is still present.

Where a single source of time authority is desired, a nice trick is to use the database. A single database is shared across all deployed nodes, so by using the database’s now() function instead of time.Now() in code, we can expect perfect consistency across all created records.

But a downside of this approach is that it makes time hard to stub because Postgres’ time is hard to stub. Stubbing time is often a necessity in tests and not being able to do so is a deal breaker.

We’ve been using a hybrid approach with some success. A call to coalesce prefers an injected timestamp if there is one, but falls back on now() most of the time (including in production) to share a clock.

Step 1: SQL + sqlc

Here’s a sample query showing the coalesce in action. sqlc.narg defines a parameter as nullable.

-- name: QueuePause :execrows
UPDATE queue
SET paused_at = CASE
                WHEN paused_at IS NULL THEN coalesce(
                    sqlc.narg('now')::timestamptz,
                    now()
                )
                ELSE paused_at
                END
WHERE name = @name;

In sqlc.yaml, tell sqlc to emit nullable timestamps as *time.Time pointers:

version: "2"
sql:
  - engine: "postgresql"
    queries: ...
    schema: ...
    gen:
      go:
        overrides:
          - db_type: "timestamptz"
            go_type:
              type: "time.Time"
              pointer: true
            nullable: true

Which generates this code:

const queuePause = `-- name: QueuePause :execrows
UPDATE queue
SET
    paused_at = CASE WHEN paused_at IS NULL THEN coalesce($1::timestamptz, now()) ELSE paused_at END
WHERE CASE WHEN $2::text = '*' THEN true ELSE name = $2 END
`

type QueuePauseParams struct {
    Now  *time.Time
    Name string
}

func (q *Queries) QueuePause(ctx context.Context, db DBTX, arg *QueuePauseParams) (int64, error) {
    result, err := db.Exec(ctx, queuePause, arg.Now, arg.Name)
    if err != nil {
        return 0, err
    }
    return result.RowsAffected(), nil
}

Step 2: Stubabble time generator

Working in Go, define a TimeGenerator interface:

When unstubbed, it returns the current time from NowUTC() or nil from NowUTCOrNil().
When stubbed, it returns the stubbed time from NowUTC() or a pointer version of the same from NowUTCOrNil().

// TimeGenerator generates a current time in UTC. In test
// environments it's implemented by TimeStub which lets the
// current time be stubbed. Otherwise, it's implemented as
// UnstubbableTimeGenerator which doesn't allow stubbing.
type TimeGenerator interface {
    // NowUTC returns the current time. This may be a stubbed
    // time if the time has been actively stubbed in a test.
    NowUTC() time.Time

    // NowUTCOrNil returns if the currently stubbed time _if_
    // the current time is stubbed, and returns nil otherwise.
    // This is generally useful in cases where a component may
    // want to use a stubbed time if the time is stubbed, but
    // to fall back to a database time default otherwise.
    NowUTCOrNil() *time.Time
}

A stubbable implementation for tests:

type TimeStub struct {
    nowUTC *time.Time
}

func (t *TimeStub) NowUTC() time.Time {
    if t.nowUTC == nil {
        return time.Now().UTC()
    }

    return *t.nowUTC
}

func (t *TimeStub) NowUTCOrNil() *time.Time {
    return t.nowUTC
}

func (t *TimeStub) StubNowUTC(nowUTC time.Time) time.Time {
    t.nowUTC = &nowUTC
    return nowUTC
}

An unstubbable time generator for production:

type UnstubbableTimeGenerator struct{}

func (g *UnstubbableTimeGenerator) NowUTC() time.Time       { return time.Now() }
func (g *UnstubbableTimeGenerator) NowUTCOrNil() *time.Time { return nil }

func (g *UnstubbableTimeGenerator) StubNowUTC(nowUTC time.Time) time.Time {
    panic("time not stubbable outside tests")
}

Step 3: Distributing a shared time generator

The next key aspect is that all code needs to share a single instance of TimeGenerator so that when it’s stubbed from a test, all services and subservices get the same stubbed value.

We put a TimeGenerator on a base service archetype that’s automatically injected from top-level services to subservices:

func (c *Client[TTx]) QueuePauseTx(ctx context.Context, tx TTx, name string, opts *QueuePauseOpts) error {
    executorTx := c.driver.UnwrapExecutor(tx)

    if err := executorTx.QueuePause(ctx, &QueuePauseParams{
        Name:   name,
        Now:    c.baseService.Time.NowUTCOrNil(), // <-- accessed here
        Schema: c.config.Schema,
    }); err != nil {
        return err
    }

By default, it’s instantiated as UnstubbableTimeGenerator. From tests, it’s a TimeStub:

func BaseServiceArchetype(tb testing.TB) *baseservice.Archetype {
    tb.Helper()

    return &baseservice.Archetype{
        Logger: Logger(tb),
        Time:   &TimeStub{},
    }
}

In a test, time is stubbed like:

stubbedNow := client.baseService.Time.StubNowUTC(time.Now().UTC())

Loose conviction

Consider this one a loose recommendation. It’s useful in some situations where timestamp consistency is critically important, but not in others where it isn’t. Server clocks tend to be pretty good nowadays, and it’s a lot of code to avoid a few tens of microseconds worth of drift.

Also, consider that there might be a downside to using the database clock. In SQL, CURRENT_TIMESTAMP and now() in Postgres represent the current time at the start of the current transaction rather than the current time. This might be a benefit as all records created during a transaction are assigned the same created time, but it’s just as often undesirable because depending on the duration of the transaction, timestamps can be wildly unrepresentative of when things actually happened.

Schoenberg: The MIDI Esoteric Programming Language

2025-06-23T00:00:00Z

Schoenberg is an esoteric programming language where programs are written as MIDI files. A MIDI file is basically digital sheet music that tells a computer which notes to play when and how loudly. The…

Testing the graceful handling of request cancellation in Go, 499s

2025-06-19T22:16:09Z

We had a situation a few days ago where a lazy loading problem in our Ruby code led to long running requests that our Dashboard, with an optimistic five second deadline on backend requests, was timing out. This raised a question in Slack: if our frontend does time out a backend request, does the request keep running? Or does the API know how to save resources by abandoning it midway through?

If the API stack’s being bombarded by expensive requests that are largely being canceled early, it’s a huge optimization to make sure that they only use the resources that they absolutely to. Requests discarded early stop executing immediately and no further effort is put toward servicing them.

In most code I’ve ever worked in, I could quite confidently answer the question above with a definitive and resounding “no”. Doing a good job of request cancellation requires it be baked quite deeply into language and low level libraries, which isn’t common. And even when those handle it well, userland code usually doesn’t. Also, cancelling a request midway in services that don’t use transactions would be unacceptably dangerous – mutated state would be left mutated, and that’d cause untold trouble later on.

Cancellation in Go

But in a Go stack, the built-in HTTP server should handle cancellations using context:

For incoming server requests, the context is canceled when the client’s connection closes, the request is canceled (with HTTP/2), or when the ServeHTTP method returns.

And with our code being widely safeguarded by transactions, the feature should even be safe to use!

Now prove it

Theory is one thing, but reality is another. If request cancellations indeed work, we should be able to prove it, so I set up a little bootstrap in pursuit of that. To make testing easy, add an artificial API endpoint waiting on sleep or context finished:

select {
case <-time.After(5 * time.Second):
case <-ctx.Done():
        return nil, ctx.Err()
}

Start the API server. Then from another terminal, run cURL and interrupt it after a few seconds:

$ curl -i http://localhost:5222/sleep
^C

I found that we were handling canceled requests reasonably well, but that the error we were logging wasn’t right. The code was checking context cancellation, but getting confused between context that was canceled from the HTTP server versus one canceled by our built-in timeout middleware, improperly sending a 408 Request timeout to logs.

Local vs. request context

After a little refactoring, I ended up with this code:

func (e *APIEndpoint[TReq, TResp]) Execute(w http.ResponseWriter, r *http.Request) {
    ctx := r.Context()

    // Add a default timeout for all API requests to ensure there's
    // always a backstop in case of degenerate behavior. Rescued
    // below and turned into a more user-friendly error.
    ctx, cancel := context.WithTimeout(ctx, RequestTimeout)
    defer cancel()
    
    ...
    
    ret, err := e.serviceHandler(ctx, req)
    if err != nil {
        if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
            // Distinct error message when the request itself was
            // canceled above the API stack versus we had a
            // cancellation/timeout occur within the API endpoint.
            if r.Context().Err() != nil {
                // This is a non-standard status code (499), but
                // fairly widespread because Nginx defined it.
                err = apierror.NewClientClosedRequestError(ctx, errMessageRequestCanceled).WithSpecifics(ctx, err)
            } else {
                err = apierror.NewRequestTimeoutError(ctx, errMessageRequestTimeout).WithSpecifics(ctx, err)
            }
        }

        WriteError(ctx, w, err)
        return
    }

Should a context error occur, we return a 408 Request timeout in case of a timeout on local ctx, but a 499 Client closed request if context was canceled upstream by the HTTP server canceling r.Context().

499 isn’t real status code, but rather one invented by Nginx which happens to be useful here. It doesn’t really matter what status code we use because the end user (who canceled the request before the status code returned) will never see it. It’s purely for our own logging and telemetry.

Looking at local logs running the sleep/cancel routine, I now see this:

canonical_api_line GET /sleep -> 499 (4.162702459s)
    api_error_cause="context canceled"
    api_error_internal_code=client_closed_request
    api_error_message="Context of incoming request canceled; API endpoint stopped executing."

Generalizing cancellation handling

Although our demo uses an artificial sleep statement, importantly this still works for any normal requests. Our code isn’t littered with <-ctx.Done() checks all over the place, but it does have a great many database operations like this one:

account, err := dbsqlc.New().AccountTouchLastSeenAt(ctx, e, apiKey.AccountID)
if err != nil {
    return nil, xerrors.Errorf("error looking up account: %w", err)
}

These call into Sqlc which call into Pgx, and Pgx detects a canceled context and sends back an error. In the event of a canceled request, the first database operation will come back with an error that’ll bubble back up the stack to our API endpoint infrastructure. There it’ll be turned it into a 499. Subsequent database operations won’t run, saving time and resources.

// API service handler error handling. Repeated from above.
ret, err := e.serviceHandler(ctx, req)
if err != nil {
    if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
        // Distinct error message when the request itself was
        // canceled above the API stack versus we had a
        // cancellation/timeout occur within the API endpoint.
        if r.Context().Err() != nil {
            // This is a non-standard status code (499), but
            // fairly widespread because Nginx defined it.
            err = apierror.NewClientClosedRequestError(ctx, errMessageRequestCanceled).WithSpecifics(ctx, err)
        } else {
            err = apierror.NewRequestTimeoutError(ctx, errMessageRequestTimeout).WithSpecifics(ctx, err)
        }
    }

    WriteError(ctx, w, err)
    return
}

Pgx is one example of a library that’ll check context cancellation, but it’ll generally occur in any low level library that’s doing I/O. As another example, SDKs like AWS or Stripe will usually go through net/http, which will catch them.

With code exercised (and adequate new testing in place), I was confident returning to Slack and declaring that “yes”, request cancellation is handled smoothly. I can’t say the same about our Ruby code, but that’s an adventure for another day.

Be careful with Dropbox

2025-05-26T06:28:04Z

I’ve been a Dropbox users going on fifteen years now. It’s one of the most frustrating products in my arsenal because fifteen years ago it was perfect, but every new release just makes it a little bit worse than it was before. It’s still fine to use, but you can see the writing on the wall as the long term trend is all in the wrong direction.

Despite that, I previously would’ve lavished it with praise in that I’ve never once had trouble with data loss or data integrity. Despite increasing feature bloat, it did what it was supposed to, syncing files to the right places, and doing so reliably, which is pretty much all I need out of it.

That ended Friday, when I was installing Dropbox on a new laptop. My Dropbox size runs ~500 GB, so when credentialing a new machine, I copy it from another computer on the network for speed, and to conserve precious bandwidth ¹ :

Rsync ~/Dropbox from an existing computer to the new one.
brew install dropbox. Open it, log in, close it.
Replace the contents of ~/Dropbox with the rsynced copy.
Open Dropbox, let it sync against the new data. It should find everything it needs already there.

Dropbox made a change in the last couple years wherein they moved the standard ~/Dropbox on Mac to a new ~/Library/CloudStorage/Dropbox location. I now know that folders in this directory are meant for use with Apple’s File Provider API.

Apparently the change had been introduced for macOS Ventura (two major versions ago), but there must’ve been an incremental roll out because I set up a computer last year and didn’t run into it then. Once you’ve been opted into the feature, you cannot opt out. Changing back to ~/Dropbox is not an option.

Normally I do a wholesale swap of ~/Dropbox with my locally copied version, but seeing this new magic folder in ~/Library, I worried there’d be some irreversible effect if I did it the normal way. Instead, I closed Dropbox, cded into the folder to rm all the files acting as cloud “stubs”, intending to replace them with materialized versions from my local copy.

What a mistake. I dumbly assumed that with Dropbox closed, any changes I made to the folder would be safe, just like they were in every previous version of Dropbox. Not so. At all.

I got suspicious after about ten seconds. Normally an rm even on gigantic directories is near instant, but this one was running long. I SIGINTed it, but the damage was done.

I’m sure you guessed what happened already. ~/Library/CloudStorage is a magic location, and folders in it use macOS extension voodoo to make arbitrary changes in a cloud storage API. Despite Dropbox not being open, it’d used a Mac API to intercept the rm and started to remove everything. My other computers had already synced the deletions. 100s of GBs gone in seconds.

Dropbox has a good “undelete” function, so I was able to log into their web UI and recover all the deleted files, but I was left with the problem of all my other computers having purged their local contents, with potentially 100s of GBs on each needing to be synced back down (and I thought I was saving bandwidth when I started doing this). Worse yet, Dropbox puts any files it deletes into a ~/Dropbox/.dropbox.cache directory, but can’t reuse any of that data when files are recovered, so it just makes a copy. Dropbox doesn’t purge its cache often, even if disk space gets critically low, so every computer potentially needed 2 * 500 GB =~ 1 TB of free space for the full recovery, which they didn’t have.

Two days later, I got everything back to where it should be, but all I could think afterwards was what a stupid, unforced error this all was. A mandatory move to ~/Library/CloudStorage/Dropbox/File Provider API has no marginal utility for the user ², even if it makes product managers at the company feel good about themselves.

Being particular incensed at this moment, I started looking into alternatives immediately. There’s dozens out there, but my approximate evaluation is that there isn’t one that’s a crystal clear, unambiguous win that’d I’d be excited about doing the work to switch over to, which is too bad.

What I’m really looking for is Dropbox circa 2011. The one without the gratuitous/dangerous product changes, without an Electron app, and without the nags to upgrade my account which I already pay $120/year for.

Anyway, I doubt most users will run into this one as it was a confluence of stupid things that led me down this path, but I’d just caution like the title says: be careful with Dropbox. Don’t rm too much. Don’t assume intuitive cause and effect. Don’t assume operations are safe even if the app is closed.

Migrating My Servers (and me!) to NYC

2025-05-19T00:00:00Z

My Second M68k SBC

2025-04-12T00:00:00Z

Toasting QR Codes onto Tortillas

2025-04-06T00:00:00Z

My Time at The Recurse Center

2025-04-01T00:00:00Z

Optimizing JPEGs with MozJPEG for local archival

2025-03-29T19:35:10Z

Call me old fashioned, but I like to keep my photo collection as local files on disk rather than symbolic pointers in the cloud, or sent off to deep storage on large archival drives, neither of which I’m likely to ever look at again. It’s nice having quick access to them that still works over a bad internet link or on an airplane.

It’s a great system, but it’s been getting more difficult as time goes by. My photo collection grows year by year, but Apple’s hard drive sizes stay frozen circa 2012. I’m running the same 1 TB drive that I was five years ago, which is only incrementally larger than five years before that (and even the mizerly 1 TB is still a $200 upcharge over the default 512 GB that’s somehow a thing that Apple sells in 2025).

Realistically, I know that I’ll never look at the majority of these photos again, so I already prune the collections aggressively to keep only the highlights, but was looking for storage opportunities beyond that. Years ago I wrote about optimizing JPEGs for this site using MozJPEG, and knowing that a lot of cameras produce suboptimally compressed JPEGs, realized there was a similar opportunity for archival.

I ended up writing a wrapper around around MozJPEG that saves about 80% of space compared to what comes out of my camera. Here’s a sample run:


    $ optimize 001-ana-nuevo/*
    created: 001-ana-nuevo/2W4A6210.jpg (9.02MB -> 2.11MB / saved 77%)
    created: 001-ana-nuevo/2W4A6212.jpg (8.21MB -> 1.79MB / saved 78%)
    created: 001-ana-nuevo/2W4A6216.jpg (11.0MB -> 2.68MB / saved 76%)
    created: 001-ana-nuevo/2W4A6218.jpg (6.36MB -> 1.29MB / saved 80%)
    created: 001-ana-nuevo/2W4A6219.jpg (12.11MB -> 3.01MB / saved 75%)
    created: 001-ana-nuevo/2W4A6224.jpg (7.3MB -> 1.69MB / saved 77%)
    created: 001-ana-nuevo/2W4A6228.jpg (7.75MB -> 1.72MB / saved 78%)
    created: 001-ana-nuevo/2W4A6230.jpg (8.62MB -> 1.99MB / saved 77%)
    created: 001-ana-nuevo/2W4A6236.jpg (8.14MB -> 1.87MB / saved 77%)
    created: 001-ana-nuevo/2W4A6237.jpg (6.65MB -> 1.48MB / saved 78%)
    created: 001-ana-nuevo/2W4A6238.jpg (7.59MB -> 1.69MB / saved 78%)
    created: 001-ana-nuevo/2W4A6240.jpg (9.38MB -> 2.21MB / saved 76%)
    created: 001-ana-nuevo/2W4A6242.jpg (9.26MB -> 2.22MB / saved 76%)
    created: 001-ana-nuevo/2W4A6243.jpg (10.17MB -> 2.44MB / saved 76%)
    created: 001-ana-nuevo/2W4A6247.jpg (10.49MB -> 2.56MB / saved 76%)
    created: 001-ana-nuevo/2W4A6251.jpg (7.92MB -> 1.84MB / saved 77%)
    created: 001-ana-nuevo/2W4A6252.jpg (8.97MB -> 2.12MB / saved 76%)
    created: 001-ana-nuevo/2W4A6253.jpg (7.74MB -> 1.75MB / saved 77%)
    created: 001-ana-nuevo/2W4A6254.jpg (9.43MB -> 2.3MB / saved 76%)
    created: 001-ana-nuevo/2W4A6255.jpg (10.78MB -> 2.65MB / saved 75%)
    created: 001-ana-nuevo/2W4A6258-pups.jpg (9.13MB -> 2.22MB / saved 76%)
    created: 001-ana-nuevo/2W4A6259.jpg (10.46MB -> 2.55MB / saved 76%)
    created: 001-ana-nuevo/2W4A6260.jpg (8.54MB -> 2.04MB / saved 76%)
    created: 001-ana-nuevo/2W4A6262.jpg (10.3MB -> 2.59MB / saved 75%)
    created: 001-ana-nuevo/2W4A6266.jpg (8.81MB -> 2.19MB / saved 75%)
    created: 001-ana-nuevo/2W4A6267.jpg (9.64MB -> 2.31MB / saved 76%)
    created: 001-ana-nuevo/2W4A6268.jpg (9.83MB -> 2.33MB / saved 76%)
    created: 001-ana-nuevo/2W4A6269.jpg (8.93MB -> 2.14MB / saved 76%)
    created: 001-ana-nuevo/2W4A6271.jpg (7.38MB -> 1.74MB / saved 76%)
    created: 001-ana-nuevo/2W4A6272.jpg (7.19MB -> 1.68MB / saved 77%)
    created: 001-ana-nuevo/2W4A6283-water-fight.jpg (7.65MB -> 1.73MB / saved 77%)
    created: 001-ana-nuevo/2W4A6284.jpg (8.02MB -> 1.77MB / saved 78%)
    created: 001-ana-nuevo/2W4A6286.jpg (5.82MB -> 1.11MB / saved 81%)
    created: 001-ana-nuevo/2W4A6287.jpg (6.03MB -> 1.14MB / saved 81%)

I’m sure there’s some subtle downside to the extra compression, but I’ve tried zooming all the way in on a couple samples before and after, and I can see differences right at the pixel level, but the optimized version isn’t clearly worse to my eye.

My script’s use-at-your-own-risk me-ware that I’m not publishing in any official sense, but here it is for reference.

Some gotchas I ran into and which might save someone else time/trouble:

The MozJPEG binary to compress JPEGs is called cjpeg. This is an old Linux style project, and naming the binary after the project would make things too easy and too obvious for users. Under the strict edicts of 1970s Unix philosophy, that’s completely unacceptable.
You might have multiple packages on your system providing cjpeg. Make sure you’re using MozJPEG’s because it offers much better compression than libjpeg or libjpeg-turbo. You can see here that my default cjpeg is not MozJPEG’s:

$ which cjpeg
/opt/homebrew/bin/cjpeg

$ ls -l /opt/homebrew/bin/cjpeg
lrwxr-xr-x@ 1 brandur  admin    36B Feb 10 11:45 /opt/homebrew/bin/cjpeg -> ../Cellar/jpeg-turbo/3.1.0/bin/cjpeg

The original libjpeg cjpeg didn’t support reading JPEGs, only writing them, and would encourage you to read JPEGs with another binary called djpeg and pipe that into cjpeg (again, the wonders of Unix philosophy). You can do that with MozJPEG too, but DO NOT DO THAT! Piping will strip EXIF data, which you shouldn’t do. Unlike libjpeg’s version, MozJPEG’s cjpeg does read JPEGs, so piping is not necessary.
If you’re writing to a new a file and then replacing the original after (which you probably should for safety), make sure to copy the original create/modify timestamps to the new file. The easiest way to do this is with touch -r <original> <new>`

TOUCH(1)						      General Commands Manual							  TOUCH(1)

NAME
     touch – change file access and modification times

SYNOPSIS
     touch [-A [-][[hh]mm]SS] [-achm] [-r file] [-t [[CC]YY]MMDDhhmm[.SS]] [-d YYYY-MM-DDThh:mm:SS[.frac][tz]] file ...

DESCRIPTION
     The touch utility sets the modification and access times of
     files. If any file does not exist, it is created with default
     permissions.

     By default, touch changes both modification and access times.
     The -a and -m flags may be used to select the access time or
     the modification time individually.  Selecting both is
     equivalent to the default.  By default, the timestamps are set
     to the current time. The -d and -t flags explicitly specify a
     different time, and the -r flag specifies to set the times
     those of the specified file.  The -A flag adjusts the values
     by a specified amount.

     The following options are available:

     ...

     -r      Use the access and modifications times from the
             specified file instead of the current time of day.

Another approach would be to do away with JPEG completely and go to HEIC or WebP, but I’m still finding support for those a little spotty, and navigating them in a file browser feels slow because the compression takes longer to render. I’ll check in on that again in a year or two.

The right way to do data fixtures in Go

2025-03-20T15:56:52Z

Every test suite should start early in building a strong convention to generate data fixtures. If it doesn’t, data fixtures will still emerge (they’re that necessary), but in a way that’s poorly designed, with no API (or a poorly designed one), and not standardized.

Other languages tend to have common libraries for fixture generation. As if often does, Go goes its own way and doesn’t have a ubiquitous fixtures package, but especially when combining sqlc and validator, it does well without one.

Here’s one of our project’s 130 fixtures:

package dbfactory

type MultiFactorOpts struct {
    ID          *uuid.UUID              `validate:"-"`
    AccountID   uuid.UUID               `validate:"required"`
    ActivatedAt *time.Time              `validate:"-"`
    ExpiresAt   *time.Time              `validate:"-"`
    Kind        *dbsqlc.MultiFactorKind `validate:"-"`
}

func MultiFactor(ctx context.Context, t *testing.T, e db.Executor, opts *MultiFactorOpts) *dbsqlc.MultiFactor {
    t.Helper()

    validateOpts(t, opts)

    var (
        num          = nextNumSeq()
        numFormatted = formatNumSeq(num)
    )

    multiFactor, err := dbsqlc.New().MultiFactorInsert(ctx, e, dbsqlc.MultiFactorInsertParams{
        ID:          ptrutil.ValOrDefaultFunc(opts.ID, func() uuid.UUID { return ptesting.ULID(ctx).New() }),
        AccountID:   opts.AccountID,
        ActivatedAt: ptrutil.TimeSQLNull(opts.ActivatedAt),
        ExpiresAt:   ptrutil.TimeSQLNull(opts.ExpiresAt),
        Kind:        string(ptrutil.ValOrDefault(opts.Kind, dbsqlc.MultiFactorKindTOTP)),
        Name:        fmt.Sprintf("%s no. %s", ptrutil.ValOrDefault(opts.Kind, dbsqlc.MultiFactorKindTOTP), numFormatted),
    })
    require.NoError(t, err)

    return multiFactor
}

The minimum viable use of the fixture needs only AccountID:

mf := dbfactory.MultiFactor(ctx, t, tx, &dbfactory.MultiFactorOpts{
    AccountID: account.ID,
})

But all salient properties are settable, so a more elaborate use just involves sending more overrides:

expiredMF := dbfactory.MultiFactor(ctx, t, bundle.tx, &dbfactory.MultiFactorOpts{
    AccountID: account.ID,
    ExpiresAt: ptrutil.Ptr(time.Now().Add(-5 * time.Minute)),
    Kind:      ptrutil.Ptr(dbsqlc.MultiFactorKindWebAuthn),
})

Observations

A few aspects worth calling out:

Under the principle of not mocking the database, fixtures are real live data records. They’re queryable using the full expressiveness of SQL, are valid according to the schema’s data types/checks/triggers, and satisfy foreign keys.
Fixtures never return an error, instead failing their input t so that generating a fixture is a one liner for the caller and doesn’t need an if err != nil { ... } check.
Inputs are annotated with the Go validate framework to demarcate required versus non-required or more complex validations as needed. This is a godsend because it keeps validations short (zero additional lines instead of a minimum of three for an if statement) and fast/easy to write.
As few properties are made validate:"required" as possible, with non nullable fields given defaults instead of marked mandatory for the caller to fill. This makes fixtures easier to use and reduces boilerplate at call sites. e.g. name is a required property on multi_factor above, but the fixture generates a sane default.
Insert statements are generated with sqlc.

-- name: MultiFactorInsert :one
INSERT INTO multi_factor (
    id,
    account_id,
    activated_at,
    expires_at,
    kind,
    name
) VALUES (
    @id,
    @account_id,
    @activated_at,
    @expires_at,
    @kind,
    @name
) RETURNING *;

We use of a lot of custom pointer helpers like ptrutil.TimeSQLNull (changes a pointer to a sql.NullTime) and ptrutil.ValOrDefault. Each one of these changes a ~4 line local variable declaration and if block to one LOC that it’s inlined into the insert. True Go dogmatists won’t like this, but it saves dozens of lines per test fixture, and given hundreds of test fixtures, this adds up to thousands of lines saved overall.
Each test case gets its own lazily marshaled monotonic ULID generated based on t. Separate generators guarantee monotonicity even if some test cases rewind their generators to generate ULIDs at particular times.

Organizing with var blocks

Typically, fixtures are generated together in a var ( ... ) block, keeping tests looking nice and tidy:

t.Run("SetNameSSOJoinSCIMError", func(t *testing.T) {
    t.Parallel()

    bundle, ctx := setup(t)

    var (
        org  = dbfactory.Organization(ctx, t, bundle.tx, &dbfactory.OrganizationOpts{SCIMEnabled: true})
        team = dbfactory.Team(ctx, t, bundle.tx, &dbfactory.TeamOpts{OrganizationID: &org.ID})
        _    = dbfactory.AccessGroupAccount_Admin(ctx, t, bundle.tx, team.ID, bundle.account.ID)
    )

    _, err := pservicetest.InvokeHandler(bundle.svc.Update, ctx, &TeamUpdateRequest{
        Name:   ptrutil.Ptr("new name"),
        TeamID: eid.EID(team.ID),
    })
    prequire.APIErrorWithMessage(t, &apierror.BadRequestError{}, fmt.Sprintf(errMessageTeamUpdateSCIM, "name"), err)
})

Standardize conventions, even the small ones

We have a few helpers that are used in almost every test fixture. These are so trivial that they almost don’t need to be extracted into their own functions, but we’ve done so to prevent implementations from drifting and keep code maximally succinct.

// Formats a number like "000007". Typically used in conjunction
// with nextNumSeq to make identifiers prettier and so they align
// better.
func formatNumSeq(num int64) string {
    return fmt.Sprintf("%06d", num)
}

var numSeq int64

// Gets a unique number that can be used in names, etc. and which
// is more friendly to look at than a UUID.
func nextNumSeq() int64 {
    return atomic.AddInt64(&numSeq, 1)
}

func validateOpts(t *testing.T, opts any) {
    t.Helper()

    err := validate.Struct(opts)
    require.NoError(t, err)
}

EXTREME SERVER SIDE RENDERING

2025-02-16T00:00:00Z

An LLM, a Formal Grammar, and 40 Sacred Words

2025-02-07T00:00:00Z

Profiling production for memory overruns + canonical log stats

2025-02-03T01:02:27Z

You’re only lucky for so long. After four years of running our Go API in production with no memory trouble whatsoever, last week we started seeing instantaneous bursts of ~1.5 GB suddenly allocated, enough to cause Heroku to kill the dyno for being “vastly over quota” (our steady state memory use sits around ~50 MB, so we run on 512 MB dynos).

This was of course, concerning. We were only experiencing a few of these a day, but with no idea what was causing them, and having appeared very suddenly, we had to assume that they might get more frequent. Not only is the API suddenly being taken offline at any moment is a bad place to be UX-wise, and even with our careful use of transactions, makes resource leaks between components possible.

Alloc delta

To localize the problem, I used Go’s runtime.MemStats in conjunction with our canonical API lines, making a new total_alloc_delta property available to see how many allocations took place during the period of an API request:

func (m *CanonicalAPILineMiddleware) Wrapper(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        var (
            memStats      runtime.MemStats
            memStatsBegin = m.TimeNow()
        )
        runtime.ReadMemStats(&memStats)
        var (
            memStatsBeginDuration = m.TimeNow().Sub(memStatsBegin)

            // TotalAlloc doesn't decrement on heap frees, so it gives
            // us useful info even if the GC runs during the request.
            totalAllocBegin = memStats.TotalAlloc
        )

        // API request served here
        next.ServeHTTP(w, r)
    
        // Middleware continues ...
        memStatsEnd := m.TimeNow()
        // Since we're only interested in one field, reuse the same
        // struct so we don't need to allocate a second one.
        runtime.ReadMemStats(&memStats)
        var (
            memStatsEndDuration = m.TimeNow().Sub(memStatsEnd)
            totalAllocDelta     = memStats.TotalAlloc - totalAllocBegin
        )
        
        logData := &CanonicalAPILineData{
            ID:                   m.ULID.New(),
            HTTPMethod:           r.Method,
            HTTPPath:             r.URL.Path,
            ...
            ReadMemStatsDuration: timeutil.PrettyDuration(memStatsBeginDuration + memStatsEndDuration),
            TotalAllocDelta:      totalAllocDelta,
        }

        plog.Logger(ctx).WithFields(structToFields(logData)).
            Infof(
                "canonical_api_line %s %s -> %v %s(%s)",
                r.Method,
                routeOrPath,
                logData.Status,
                idempotencyReplayStr,
                duration,
            )

MemStats provides a large bucket of properties to pick from, but TotalAlloc’s a useful one because it represents bytes allocated to the heap, but unlike similar stats like HeapAlloc, it’s monotonically increasing. It’s not decremented as objects are freed:

// TotalAlloc is cumulative bytes allocated for heap objects.
//
// TotalAlloc increases as heap objects are allocated, but
// unlike Alloc and HeapAlloc, it does not decrease when
// objects are freed.
TotalAlloc uint64

This is good because it means that all API requests will end up with the same memory heuristic, and made roughly comparable. Garbage collection may or may not occur during a request. Using TotalAlloc makes it irrelevant whether it did or not.

With that deployed, I can search logs for outliers (:>500000000 means greater than 5 MB):

source:platform app:app[web] canonical_api_line (-http_route:/health-checks/{name})
    total_alloc_delta:>500000000

And voila, we turn up the bad ones. Here, an API request that spiked memory a full 5 GB!

Jan 29 10:18:33 platform app[web] info canonical_api_line
    POST /queries -> 503 (2.53252138s)
total_alloc_delta=5008335944

Parallel allocations

The use of TotalAlloc is imperfect because it not only tracks allocations of the current API request, but allocations across the current API request and all parallel requests.

We can see this effect through false positives:

Feb 1 23:07:18 platform app[web] info canonical_api_line
    GET /clusters/{cluster_id}/databases -> 504 (2m57.322010348s)
total_alloc_delta=743772480

It looks like this API request allocated 744 MB, but what actually happened is that it was a bad timeout that executed for a full three minutes ¹. During that time, other API requests served in the interim allocated the majority of that memory. It didn’t crash our 512 MB dyno because multiple GCs also occurred during that time.

Pprof to S3

Getting our memory overruns localized to a particular endpoint was good, but even having done that, I’d need a little more help to figure out where the rogue memory was going. To that end, I put in one more clause in the middleware so that in case of a huge overrun, the process dumps a pprof heap profile to S3:

    ...

    // If we used a particularly huge amount of memory during the
    // request, upload a profile to S3 for analysis. Buckets have a
    // configured life cycle so objects will expire out after some
    // time.
    if err := m.maybeUploadPprof(ctx, logData.RequestID, totalAllocDelta); err != nil {
        plog.Logger(ctx).Errorf(m.Name+": Error uploading pprof profile: %s", err)
}

...

const pprofTotalAllocDeltaThreshold = 1_000_000_000

func (m *CanonicalAPILineMiddleware) maybeUploadPprof(ctx context.Context, requestID uuid.UUID, totalAllocDelta uint64) error {
    if !m.pprofEnable || totalAllocDelta < m.pprofTotalAllocDeltaThreshold {
        return nil
    }

    profKey := fmt.Sprintf("%s/pprof/%s.prof", m.EnvName, requestID)

    var buf bytes.Buffer
    if err := pprof.WriteHeapProfile(&buf); err != nil {
        return xerrors.Errorf("error writing heap profile: %w", err)
    }

    if _, err := m.aws.S3_PutObject(ctx, &s3.PutObjectInput{
        Body:   &buf,
        Bucket: ptrutil.Ptr(awsclient.S3Bucket),
        Key:    &profKey,
    }); err != nil {
        return xerrors.Errorf("error putting heap profile to S3 at path %q: %w", profKey, err)
    }

    plog.Logger(ctx).Infof(m.Name+": pprof_profile_generated_line: TotalAlloc delta %d exceeded %d; generated pprof profile to S3 key %q",
        totalAllocDelta, m.pprofTotalAllocDeltaThreshold, profKey)

    return nil
}

Our memory problem ended up being a queries endpoint that was overly willing to read giant result sets into memory, then serialize the whole thing into a big JSON buffer for response, which was also pretty indented (and in Go’s encoding/json, indenting a JSON response requires a second giant buffer 2x the size of the first one). I fixed it by reducing the maximum number of rows we were willing to read into the response.

I’m not expecting to run into new memory overruns or leaks anytime soon, but I left the pprof code in place for the time being. It only does work in case of huge memory increases so there’s no performance penalty most of the time, and it might come in handy again.

Stop the world

A token glance at the implementation of runtime.ReadMemStats looks a little concerning:

// ReadMemStats populates m with memory allocator statistics.
//
// The returned memory allocator statistics are up to date as of the
// call to ReadMemStats. This is in contrast with a heap profile,
// which is a snapshot as of the most recently completed garbage
// collection cycle.
func ReadMemStats(m *MemStats) {
    _ = m.Alloc // nil check test before we switch stacks, see issue 61158
    stw := stopTheWorld(stwReadMemStats)

    systemstack(func() {
        readmemstats_m(m)
    })

    startTheWorld(stw)
}

To produce accurate stats, the runtime needs to “stop the world”, meaning that all active goroutines are paused, a sample taken, and resumed.

Intuitively, that seems like it could be pretty slow, and some initial googling seemed to confirm that. However, I later found a patch from 2017 that’d improved the situation considerably by doing cumulative tracking of relevant stats so only a very brief stop the world was required. It indicated a reduction in timing down to 25µs, even at 100 concurrent goroutines.

I added a separate log stat to see how long my two ReadStatMems calls were taking, and found they were averaging ~100µs for both:

read_mem_stats_duration=0.000098s
read_mem_stats_duration=0.000110s
read_mem_stats_duration=0.000113s
read_mem_stats_duration=0.000126s
read_mem_stats_duration=0.000123s
read_mem_stats_duration=0.000084s
read_mem_stats_duration=0.000091s
read_mem_stats_duration=0.000092s
read_mem_stats_duration=0.000090s
read_mem_stats_duration=0.000083s

That’s 50µs per invocation instead of 25µs, but given that a single DB query takes an order or two of magnitude longer at 1-10ms, a little delay to get memory stats is acceptable. If our stack was hyper performance sensitive or saturated with huge request volume, I’d take it out.

The Taylorator - All Your Frequencies Are Belong to Us

2025-01-27T00:00:00Z

Sierpinski Triangles on a Pen Plotter

2025-01-10T00:00:00Z

Rubic - a BASIC-inspired Ruby REPL

2025-01-09T00:00:00Z

Go's bytes.Buffer vs. strings.Builder

2025-01-03T05:34:32Z

I was writing some Go code today that generated other Go code. Writing it line by line, mostly in a loop, but with pre- and post-matter.

My usual go to for this type of thing is bytes.Buffer, but after I’d finished the implementation, given that I was working entirely with strings, I started to wonder if I should’ve used strings.Builder instead.

I realized that I had no idea whether one was faster than the other, so I wrote a quick benchmark to check:

package main

import (
    "bytes"
    "strings"
    "testing"
)

var fragments = []string{
    "This",
    "is a series of",
    "string fragments",
    "that will be concatenated together",
    "into a single larger string",
    "so that we can",
    "determine which of Go's various",
    "tools for doing this",
    "is most efficient.",
    "I found a few articles",
    "online",
    "but most were poorly cited",
    "or",
    "behind a Medium login wall",
    "or otherwise",
    "not of admirable quality.",
}

func BenchmarkBytesBuffer(b *testing.B) {
    for range b.N {
        var buf bytes.Buffer

        for _, fragment := range fragments {
            buf.WriteString(fragment)
            buf.WriteString(" ")
        }

        _ = buf.String()
    }
}

func BenchmarkConcatenateStrings(b *testing.B) {
    for range b.N {
        var str string

        for _, fragment := range fragments {
            str += fragment
            str += " "
        }
    }
}

func BenchmarkStringBuilder(b *testing.B) {
    for range b.N {
        var sb strings.Builder

        for _, fragment := range fragments {
            sb.WriteString(fragment)
            sb.WriteString(" ")
        }

        _ = sb.String()
    }
}

$ go test -bench=. -benchmem
goos: darwin
goarch: arm64
pkg: github.com/brandur/go-builder-vs-buffer
cpu: Apple M4
BenchmarkBytesBuffer-10           5013081    217.3 ns/op    1280 B/op    5 allocs/op
BenchmarkConcatenateStrings-10    1603748    753.5 ns/op    5557 B/op    31 allocs/op
BenchmarkStringBuilder-10         6916813    146.9 ns/op    752 B/op     6 allocs/op
PASS
ok      github.com/brandur/go-builder-vs-buffer 4.724s

So there you have it. At least when it comes to concatenating only strings at relatively modest sizes, strings.Builder is about 33% faster, and 80% faster than ¹ than concatenating strings. Given that the DX is identical between the two, I’ll make it my new default go to.

Postgres UUIDv7 + per-backend monotonicity

2024-12-31T22:32:43Z

An implementation for UUIDv7 was committed to Postgres earlier this month. These have all the benefits of a v4 (random) UUID, but are generated with a more deterministic order using the current time, and perform considerably better on inserts using ordered structures like B-trees.

A nice surprise is that the random portion of the UUIDs will be monotonic within each Postgres backend:

In our implementation, the 12-bit sub-millisecond timestamp fraction is stored immediately after the timestamp, in the space referred to as “rand_a” in the RFC. This ensures additional monotonicity within a millisecond. The rand_a bits also function as a counter. We select a sub-millisecond timestamp so that it monotonically increases for generated UUIDs within the same backend, even when the system clock goes backward or when generating UUIDs at very high frequency. Therefore, the monotonicity of generated UUIDs is ensured within the same backend.

This is a hugely valuable feature in practice, especially in testing. Say you want to generate five objects for testing an API list endpoint. It’s possible they’re generated in-order by virtue of being across different milliseconds or by getting lucky, but probability is against you, and the likelihood is that some will be out of order. A test case has to generate the five objects, then do an initial sort before making use of them. That’s not the end of the world, but it’s more test code and adds noise.

test_accounts = 5.times.map { TestFactory.account }

# maybe IDs were in order, but maybe not, so do an initial sort
test_accounts.sort_by! { |a| a.id }

# API endpoint will return accounts ordered by ID
resp = make_api_request :get, "/accounts"
expect(resp.map { _1["id"] }).to eq(test_accounts.map(&:id))

With Postgres ensuring monotonicity for UUIDv7s, the five generated objects get five in-order IDs, making the test safer ¹ and faster to write. Montonicity isn’t guaranteed across backends, but that’s okay in well written test suites. Patterns like test transactions will guarantee that each test case speaks to exactly one backend.

12 bits more clock

My grasp on monotonicity has always been tenuous at best, so I was curious how it was implemented here. I looked at the patch, and its approach was more obvious than I expected:

/*
 * Generate UUID version 7 per RFC 9562, with the given timestamp.
 *
 * UUID version 7 consists of a Unix timestamp in milliseconds (48
 * bits) and 74 random bits, excluding the required version and
 * variant bits. To ensure monotonicity in scenarios of high-
 * frequency UUID generation, we employ the method "Replace
 * LeftmostRandom Bits with Increased Clock Precision (Method 3)",
 * described in the RFC. This method utilizes 12 bits from the
 * "rand_a" bits to store a 1/4096 (or 2^12) fraction of sub-
 * millisecond precision.
 *
 * ns is a number of nanoseconds since start of the UNIX epoch.
 * This value is used for time-dependent bits of UUID.
 */
static pg_uuid_t* generate_uuidv7(int64 ns) {

...

/*
 * sub-millisecond timestamp fraction (SUBMS_BITS bits, not
 * SUBMS_MINIMAL_STEP_BITS)
 */
increased_clock_precision = ((ns % NS_PER_MS) * (1 << SUBMS_BITS)) / NS_PER_MS;

/* Fill the increased clock precision to "rand_a" bits */
uuid->data[6] = (unsigned char) (increased_clock_precision >> 8);
uuid->data[7] = (unsigned char) (increased_clock_precision);

/* fill everything after the increased clock precision with random bytes */
if (!pg_strong_random(&uuid->data[8], UUID_LEN - 8))
    ereport(ERROR,
            (errcode(ERRCODE_INTERNAL_ERROR),
            errmsg("could not generate random values")));

UUIDv7 dictates an initial 48 bits that encodes a timestamp down to millisecond precision. A millisecond is a short amount of time for a human, but quite long for a computer, and many UUIDs could easily be generated with the space of a single ms.

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                      48 bits unix_ts_ms                       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|   48 bits unix_ts_ms (cont)   |  ver  |    12 bits rand_a     |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|var|                    62 bits rand_b                         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                     62 bits rand_b (cont)                     |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

The Postgres patch solves the problem by repurposing 12 bits of the UUID’s random component to increase the precision of the timestamp down to nanosecond granularity (filling rand_a above), which in practice is too precise to contain two UUIDv7s generated in the same process. It makes a repeated UUID between processes more likely, but there’s still 62 bits of randomness left to make use of, so collisions remain vastly unlikely.

The wait is on

UUIDv7s are going to make a great core addition to Postgres, and I can’t wait to start using them. Quite unfortunately, their commit was delayed past the freeze for Postgres 17, so they won’t make it into an official version until Postgres 18 is cut in late 2025. So now, we wait.

Stripe V2

2024-12-29T06:56:24Z

I happened to notice by way of a Slack bot today that Stripe released a V2 version of their API. I thought this must’ve been a soft launch right before the holidays, surely to be followed up by a more formal blog post, but the Way Back Machine clocked the page in early October, making it three months old. It’s been there all along, I just hadn’t seen it before.

The V1 and V2 APIs are separate namespaces and what’s available in V2 is currently very minimal (only events and event destinations), so integrations will still use V1 for almost everything, but the overview page tells us about its aspirational design intentions.

JSON, with a sprinkling of HATEOAS

A few highlights:

By far the best and biggest change is that request bodies are sent as JSON instead of application/x-www-form-urlencoded. Form encoding isn’t the worst thing in the world, but it falls flat on its face when encoding complex data types like arrays and maps (or worse, nested arrays and maps). It’s also just weird and out of place in 2024. This change should’ve happened ten years ago.
Pagination has picked up a hypermedia-esque veneer (see HATEAOS), returning a next_page_url that’s requested directly instead of a cursor and having the caller build the next URL themselves.
The new API is trying to move away from a model where sub-objects in an API resource are expanded by default, to one where they need to be requested with an include parameter. We had plenty of discussions about this before I left. The purpose of the change is to make API requests faster (Stripe’s API is quite slow) by rendering less for most requests. I counted only two places where this is actually used so far though, so time will tell whether the gambit actually succeeds or not.
Endpoints will try for “real” idempotency where callers can converge failed operations to either success or definitive failure by calling them again:
- When you provide the same idempotency key for two requests:
  - API v1 always returns the previously-saved response of the first API request, even if it was an error.
  - API v2 attempts to retry any failed requests without producing side effects (any extraneous change or observable behavior that occurs as a result of an API call) and provide an updated response.
Previously (and still for most endpoints), failures from an intermittent blip or bug were a big problem. The idempotency layer dumbly returned whatever canned response had been recorded on the initial go around (including internal server errors), so users wouldn’t get closure on what exactly happened. Their best hope would that be a Stripe engineer would eventually repair their charge manually at some later time, and send a webhook about it.

REST-ish v4-ever

Lots of positive progress there, but a new API version also presents an opportunity to clear out blemishes, and I expected to see more of that. A few points that are less good:

I was hoping they’d fix their verbs to play more nicely with modern REST conventions. Instead of using POST everywhere, use POST for endpoints that are knowingly not idempotent (without an idempotency key), PUT for mutation endpoints that are, and PATCH for mutation endpoints that aren’t. I admit it’s pedantic, but it’s so absolutely trivial to implement, and the use of a good verb signals more information than a reader would otherwise have with a cursory glance at API structure.
They’re still doing the RPC-style calls like:
```
POST /v2/core/event_destinations/:id/enable
```
Also pedantic, but enable here should theoretically be reserved for a nested resource. I think it’s cleaner to model actions as IDs under a shared “actions” subresource:
```
POST /v2/core/event_destinations/:id/actions/enable
```

Nouveau DX

Frankly, I was a bit shocked by how little attention this got. There was a time not too long ago when Stripe cutting a new API version would’ve been a major event in the tech world, but in three months I didn’t come across a single person who mentioned it.

A major part of this is that Stripe is no longer a great technical leader in the same sense that it used to be. But also, as Colin points out:

This is an undeniable sign that “a great REST API” is no longer the benchmark for great DX

That’s got to be true too. Few of us want to be making manual HTTP calls out to APIs anymore. These days a great SDK, not a great API, is a hallmark, and maybe even a necessity, of a world class development experience.

Go's maximum time.Duration

2024-12-21T17:30:01Z

While working on a River bug related to retry policy, I came across a case where it was actually plausible to overflow Go’s built-in time.Duration and wrap back around to negative number.

A duration has a much simpler representation than a timestamp. It’s an int64 counted in nanoseconds:

// A Duration represents the elapsed time between two instants
// as an int64 nanosecond count. The representation limits the
// largest representable duration to approximately 290 years.
type Duration int64

As the comment states, the maximum duration is about 290 years. More precisely, 292 (non-leap) years, 171 days, and 23 hours:

func main() {
    const (
        maxDuration time.Duration = 1<<63 - 1

        day  = 24 * time.Hour
        year = 365 * day
    )

    var (
        years        = maxDuration / year
        withoutYears = maxDuration % year

        days        = withoutYears / day
        withoutDays = withoutYears % day
    )

    fmt.Printf("max duration: %dy%dd%s\n", years, days, withoutDays)
}

$ go run main.go
max duration: 292y171d23h47m16.854775807s

292 years is a long time, and it’s not likely most programs will need more than that, but our retry algorithm is exponential, and crosses that threshold after 310 retries.

Compile v. runtime overflow

When performing a direct calculation on a constant, the compiler will detect the overflow:

func main() {
    const maxDuration time.Duration = 1<<63 - 1
    var maxDurationSeconds = float64(maxDuration / time.Second)

    notOverflowed := time.Duration(maxDurationSeconds) * time.Second
    fmt.Printf("not overflowed: %+v\n", notOverflowed)

    overflowed := time.Duration(int64(maxDuration)+1) * time.Second
    fmt.Printf("overflowed: %+v\n", overflowed)
}

$ go run main.go
./main.go:15:30: int64(maxDuration) + 1 (constant 9223372036854775808 of type int64) overflows int64

But performing the same operation on a variable will happily wrap around:

overflowed := time.Duration(maxDurationSeconds+1) * time.Second
fmt.Printf("overflowed: %+v\n", overflowed)

$ go run main.go
not overflowed: 2562047h47m16s
overflowed: -2562047h47m16.709551616s

Little practical use, but well defined

I fixed River’s back offs at large attempt counts by using Go 1.21’s min function combined with the maximum known number of seconds that’ll fit in a time.Duration:

// The maximum value of a duration before it overflows. About 292 years.
const maxDuration time.Duration = 1<<63 - 1

// Same as the above, but changed to a float represented in seconds.
var maxDurationSeconds = maxDuration.Seconds()

func (p *DefaultClientRetryPolicy) NextRetry(job *rivertype.JobRow) time.Time {
    return time.Now().Add(timeutil.SecondsAsDuration(
        p.retrySeconds(len(job.Errors) + 1),
    ))
}

func (p *DefaultClientRetryPolicy) retrySeconds(attempt int) float64 {
    retrySeconds := math.Pow(float64(attempt), 4)
    return min(retrySeconds, maxDurationSeconds)
}

After hitting retry attempt 310, the algorithm backs off 292 years at a time. This behavior will never be of any real use to anybody, but I changed it to be well defined behavior of no real use to anybody, with no risk of odd bugs that might otherwise result from an overflow.

ERROR: invalid byte sequence for encoding UTF8: 0x00 (and what to do about it)

2024-12-19T21:58:05Z

One of the oldest errors I ever remember seeing in an error tracker:

ERROR: invalid byte sequence for encoding “UTF8”: 0x00

Through my time at Heroku it was like a distant friend. Not one that you’d see every day, but one who’d appear to be surprise you a few dozen times a year. Since it didn’t seem to be causing any major fallout and I never heard a user complain about it, I’m somewhat embarrassed to say that in four years neither myself nor anyone else ever bothered to look into it.

These days, on a Go stack and with much better control and insight into any changes we make, we’re pretty aggressive about trying to prune Sentry errors down to zero. Over a few months I’d see the 0x00 error come and go, and finally decided to look into it.

The problem comes from Postgres raising an error when a caller tries to insert a text/varchar value containing a value of 0x00, or zero byte. The same value that’s used to terminate a string in plain old C. Postgres explicitly disallows it:

The character with the code zero cannot be in a string constant.

The tricky part is that although Postgres won’t take a zero byte, almost every programming language ever created will, thereby creating a natural asymmetry between database and language stack.

As far as I know, there aren’t any legitimate uses for sending a zero byte to an API or web app. Looking back through our logs, the main places I’ve seen it are from bots out on the internet, presumably using common attack patterns to probe for weaknesses, or from pentest teams that we paid to do the same.

Validating at the edges

We’re using the validate framework for Go to check that API inputs are sound, like that they’re present, below a max length, or within bounds. In a language known for its verbosity, validate annotations are succinct and quick to write.

The custom validations apistring200, apistrong2000, apistring20000, etc. are assigned to API string parameters in order of magnitude tiers. Their implementation denies \x00s that come in with request payloads:

// API strings are meant to provide a reasonable default validation
// for strings that come in via the API that aren't already
// validated more strictly. The main idea is to make sure that
// we're not getting long, unbounded input that'll either store a
// very invalid value to the database or be rejected by a DB-level
// constraint (which would bubble up as a 500 with little context).
//
// They also validate that strings contain no invalid unicode
// sequences, and that no `\x00` zero bytes are present, both of
// which Postgres will reject.
must(registerAPIString("apistring200", 200))
must(registerAPIString("apistring2000", 2_000))
must(registerAPIString("apistring20000", 20_000))
must(registerAPIString("apistring200000", 200_000))

const (
    apiStringErrorMessage = "`{0}` should be a non-empty string with a maximum length of %d characters, and contain no invalid unicode sequences or zero bytes"
)

func registerAPIString(tag string, maxLength int) error {
    if err := validate.RegisterValidation(tag, func(fl validator.FieldLevel) bool {
        val := fl.Field().String()

        if len(val) == 0 || len(val) > maxLength {
            return false
        }

        if !utf8.ValidString(val) {
            return false
        }

        // A zero (0x00) rune is valid UTF-8 and won't be caught
        // by the unicode check above, but Postgres will refuse
        // to insert it.
        if strings.Contains(val, "\x00") {
            return false
        }

        return true
    }); err != nil {
        return err
    }

    return registerTranslation(tag, fmt.Sprintf(apiStringErrorMessage, maxLength))
}

Notably, it also denies invalid UTF-8 byte sequences (\x00 is not desirable, but it is valid UTF-8), another common malformed input that internet bots like to send, and which will cause its own Postgres error.

Struct fields are tagged with validations, making use easy and concise:

// Request for creating a new account.
type AccountCreateRequest struct {
    // Full name for the new account.
    Name *string `json:"name" validate:"apistring200"`
    
    ...

Storing raw request properties

That takes care of input forms, but another place we’d see the problem is when trying to insert canonical API lines to the database for operational visibility. Even where we denied a request with invalid input with a 400, we record a canonical line for it, invalid input and all.

For this case, we take anything invalid in the input and replace it with a placeholder token that’s safely storable to Postgres:

// TrimInvalidUTF8 replaces any invalid UTF-8 or \x00 bytes with
// symbolic stand-in tokens. This lets strings that contain invalid
// UTF-8 be stored to Postgres, which normally won't tolerate
// invalid UTF-8 in string-like fields.
func TrimInvalidUTF8(s string) string {
    if !utf8.ValidString(s) {
        s = strings.ToValidUTF8(s, "[invalid UTF-8]")
    }

    // A zero (0x00) rune is valid UTF-8 and won't be caught by the
    // check above, but Postgres will refuse to insert it. Replace
    // all instances with a marker that Postgres can tolerate and
    // which is indicative of what happened. This should only ever
    // happen because of random probing from malicious internet
    // actors sending garbage into HTTP paths and what not.
    if strings.Contains(s, "\x00") {
        s = strings.ReplaceAll(s, "\x00", "[0x00 UTF-8 rune]")
    }

    return s
}

This is combined with another helper to that samples inputs longer than we’re willing to store:

// Returns a string that's been truncated the given max length and
// stripped of any invalid UTF-8 that Postgres might balk at.
// Returns an empty string on `nil` for purposes of the batch
// insert will treat empty strings as NULL.
validTruncatedStringOrEmpty := func(sPtr *string, maxLength int) string {
    if sPtr == nil {
        return ""
    }

    return stringutil.SampleLongN(stringutil.TrimInvalidUTF8(*sPtr), maxLength)
}

When inserting a canonical line for a request, inputs are sanitized and truncated. This happens for obvious fields where an invalid input can be sent like a query string or form body, but for less obvious ones as well. Invalid input can come in almost anywhere, including headers like Content-Type or User-Agent:

insertParams.ContentType[i] =
    validTruncatedStringOrEmpty(logData.ContentType, 200)
insertParams.HTTPPath[i] =
    validTruncatedStringOrEmpty(&logData.HTTPPath, 200)
insertParams.QueryString[i] =
    validTruncatedStringOrEmpty(logData.QueryString, 2000)
insertParams.UserAgent[i] =
    validTruncatedStringOrEmpty(logData.UserAgent, 200)

0x01 down

This is one of those little housekeeping tasks that may not be that important, but is quite gratifying. With the steps above we’ve eradicated “invalid byte sequence” errors, taking us a step closer to our target steady state of zero Sentry issues.

Building an Over-engineered Basement Monitor

2024-11-24T00:00:00Z

The parallel test bundle, a convention for Go testing

2024-10-27T23:06:21Z

A year ago we went through of process of getting every test case in our project tagged with t.Parallel and ratcheted with paralleltest. I was initially skeptical about this being worth the effort because testing across Go packages was already happening in parallel, but it turned out to be a major boon for running large packages individually where we reduced test time by 30%+. We did one more step from there to tag every subtest with t.Parallel too. The gains from that weren’t as big, but it helps when running tests with many subtests one off, and isn’t much effort to sustain now that it’s in place.

We’re running close to 5,000 tests at this point. Large scale code refactoring tools aren’t widespread in Go, so I did most of the refactoring with some very gnarly multi-line regexes, and even with those, the only reason that it was possible was that we’re obsessive with keeping strong code convention. Most test cases were structured with an identical layout, which might’ve seemed like unnecessary pedantry when it was first going in, but later paid off in reams as I refactored thousands of tests in hours instead of weeks.

Let me showcase a test convention that we’ve found to be useful for making subtests parallel-safe, keeping them DRY (unlike many languages, Go doesn’t have built-in facilities for setup/teardown blocks in tests), and keeping code readable. I try to be honest in the assessment of programming conventions and am not always certain about new ones, but we’ve been using the parallel test bundle for months and I’d rate it a ¹⁰⁄₁₀ strong recommendation. Better yet, it’s all just plain Go code and doesn’t require the adoption of anything weird/novel.

The test bundle struct

The test bundle itself is simple struct containing the object under test and useful fixtures to have available across subtests:

type testBundle struct {
    account *dbsqlc.Account
    svc     *playgroundTutorialService
    team    *dbsqlc.Team
    tx      db.Tx
}

The setup function

It’s paired with a setup helper function that returns a bundle:

setup := func(t *testing.T) (*testBundle, context.Context) {
    t.Helper()

    // These two vars are standard across almost every test case.
    var (
        ctx = ptesting.Context(t)
        tx  = ptesting.TestTx(ctx, t)
    )

    // Group of data fixtures.
    var (
        team    = dbfactory.Team(ctx, t, tx, &dbfactory.TeamOpts{})
        account = dbfactory.Account(ctx, t, tx, &dbfactory.AccountOpts{})
        _       = dbfactory.AccessGroupAccount_Admin(ctx, t, tx, team.ID, account.ID)
    )
    ctx = authntest.Account(account).Context(ctx)

    return &testBundle{
        account: account,
        svc:     pservicetest.InitAndStart(ctx, t, NewPlaygroundTutorialService(), tx.Begin, nil),
        team:    team,
        tx:      tx,
    }, ctx
}

Along with a test bundle, the function also returns a context ¹, which is useful for seeding context with a context logger that makes sure all logging output is collated with the test being run instead of stdout where its output would be interleaved with that of other tests running parallel. Tests that don’t need a context omit the second return value.

Subtest invocations

Each subtest marks itself as parallel, and calls setup to procure a test bundle:

t.Run("AllProperties", func(t *testing.T) {
    t.Parallel()

    bundle, ctx := setup(t)
    
    ...

Each instance of a test bundle is fully insulated from every other instance, ensuring that no side effects from a test can leak into any other. Every test case uses a test transaction so that it’s got its own private snapshot into the database for purposes of raising fixtures or querying.

We tend to put test bundles in every test case, even where the bundle contains only a single field. This is a courtesy to a future developer who might need to augment the test and where a preexisting test bundle makes that faster to do. It also keeps convention strong in case we need to do another broad refactor down the line.

Complete example

Here’s a full code sample with all the steps together:

func TestPlaygroundTutorialServiceCreate(t *testing.T) {
   t.Parallel()

   type testBundle struct {
      account *dbsqlc.Account
      svc     *playgroundTutorialService
      team    *dbsqlc.Team
      tx      db.Txer
   }

   setup := func(t *testing.T) (*testBundle, context.Context) {
      t.Helper()

      var (
         ctx = ptesting.Context(t)
         tx  = ptesting.TestTx(ctx, t)
      )

      var (
         team    = dbfactory.Team(ctx, t, tx, &dbfactory.TeamOpts{})
         account = dbfactory.Account(ctx, t, tx, &dbfactory.AccountOpts{})
         _       = dbfactory.AccessGroupAccount_Admin(ctx, t, tx, team.ID, account.ID)
      )
      ctx = authntest.Account(account).Context(ctx)

      return &testBundle{
         account: account,
         svc:     pservicetest.InitAndStart(ctx, t, NewPlaygroundTutorialService(), tx.Begin, nil),
         team:    team,
         tx:      tx,
      }, ctx
   }

   t.Run("AllProperties", func(t *testing.T) {
      t.Parallel()

      bundle, ctx := setup(t)

      resp, err := pservicetest.InvokeHandler(bundle.svc.Create, ctx, &PlaygroundTutorialCreateRequest{
         BootstrapSQL: ptrutil.Ptr(`SELECT unnest(array[1,2,3]);`),
         Name:         "My playground tutorial",
         Content:      "# My tutorial\n\nThis is my SQL tutorial, created by **me**.",
         IsPinned:     true,
         IsPublic:     true,
         TeamID:       eid.EID(bundle.team.ID),
         Weight:       ptrutil.Ptr(int32(100)),
      })
      require.NoError(t, err)
      prequire.PartialEqual(t, &apiresourcekind.PlaygroundTutorial{
         BootstrapSQL: ptrutil.Ptr(`SELECT unnest(array[1,2,3]);`),
         Content:      "# My tutorial\n\nThis is my SQL tutorial, created by **me**.",
         IsPinned:     true,
         IsPublic:     true,
         Name:         "My playground tutorial",
         TeamID:       eid.EID(bundle.team.ID),
         Weight:       ptrutil.Ptr(int32(100)),
      }, resp)

      _, err = dbsqlc.New().PlaygroundTutorialGetByID(ctx, bundle.tx, uuid.UUID(resp.ID))
      require.NoError(t, err)

      prequire.EventForActor(ctx, t, bundle.tx, "playground_tutorial.created", bundle.account.ID)
   })
}

See also the PartialEqual helper which I wasn’t completely sure about when I first put it in, but am now fully bought into now because it’s shown itself to be so effective at keeping many consecutive assertions very tidy.

Mutability Isn’t Variability

2024-10-24T08:00:00Z

How programmers confuse ideas about things changing over time.

Rails World 2024

2024-10-06T20:17:03Z

I attended Rails World again this year, this time in Toronto. A quick recap while it’s still fresh.

What a great event. Both this year and last the organizers went out of their way to pick some of the most incredible venues I’ve ever seen. Many places are adequate to the task of containing a conference for a few days, but few make your mouth go wide with a “wow” as you walk into the place.

This year’s was held at Evergeen Brick Works, an old factory that lapsed into a state of disrepair for many years, and later converted to event venue. Its renovators decided to keep some aspects of the previous abandoned wreck. Its roof that’d fallen in wasn’t replaced, leaving the evergreens that’d grown in the interim stretching through up into the sky (unclear what would’ve happened if it’d rained). Derelict machinery and the more tasteful graffiti had been left in place to add to the character. Meanwhile, ultra-modern acoustics and AV equipment made for excellent talks, and clashed nicely with the exposed brick.

Attention was paid to every detail. Quality drinks and delicious snacks were always on offer between sessions, and three food trucks operated all day outside (and good choices too: pizza served out of a decommissioned fire truck, beaver tails, and poutine, only Canada’s best! ¹). One of my favorite details that was a holdover from the conference’s first year is that all breakfast and lunch food is edible standing up, and served out of the same area that made up the convention floor. With few tables available, people mingle organically while eating, preventing a common conference lunch problem of groups self-siloing at tables where they stay immobile for 30+ minutes and meet few new people, if any. Organizers responded dynamically to fix problems as they arose. For example, lunch lines were too long on the first day, so by day two there were double the number of food stations. Pair programming sessions were available all day through Test Double.

This was all a nice change after attending RailsConf a few years back. There you couldn’t even get coffee outside a tight 30 minute availability window in the morning. This was understandable because money was tight. Ruby Central was spending it on more important things, like paying out $500k cancellation penalties to send a political “fuck you” to the entire state of Texas, which happily took their money and proceeded to not notice at all. (It may not be a big surprise to hear that 2025 will be the last year of RailsConf.)

DHH is pretty transparent on numbers, and was up front that Rails World operates at a loss that’s backstopped by the large companies that form Rails Foundation:

Rails Foundation, the founding core members listed above, as well as the contributing members […], were willing to happily underwrite a loss of over $100,000 on the conference itself.

I love it. This is one of the best ways for companies getting good leverage out of Ruby/Rails to give back to the community. We’re not contributing anywhere near what a colossus like Shopify is, but it felt great to have Crunchy sponsoring the event.

Tech highlights

I spent most of the conference at our booth, so I mostly only got a chance to catch the keynotes, but that was enough to catch the broad themes. A few notable highlights.

Solid Cache

Like last year, David touched upon Solid Cache. This is such a great concept: caches traditionally always needed to be memory bound using a component like memcached or Redis because memory was fast and disks were slow. Now, memory is still fast, but with modern SSDs, disk is also fast, and available in much larger denominations. 37 Signal’s products like Hey put their cache in MySQL, where they run it on a 30 TB disk with 60 days retention, and which has a 96% cache hit rate. This especially improves cache hits for the long tail of older keys that would’ve been long since the evicted given a less spacious in-memory data set.

Solid Cache also dovetails well with the single dependency stack. Three years later we still run one and exactly one persistence component: Postgres. It’s amazing just how plausible this is even for a mature stack, and it makes you realize that even the most fundamental belief systems of the programming world should be reevaluated every once in a while.

37 Signals stubbornly cargo cults Oracle products, but as Andrew covers, Solid Cache can be made workable on Postgres too. Although let me caveat that to say I’ve never done it, and suspect that there might be issues with long-lived deletion expiration queries at the scale of 30 TB of data since Postgres isn’t particularly good at efficiently deleting rows (a big reason that recent partitioning improvements are so important).

Server-phobia

For the last few months David’s been on an anti-cloud mission. One of keynote slides highlights the size, capacity and cost of a Performance M dyno (1 core/2 threads w/ 2.5GB for $250/mo.), with the next showing a rough equivalent on Hetzner (48 cores/96 threads w/ 256GB for $220/mo.), the clear message being that the Hetzner box is 50-100x more capable, and also cheaper. A big new piece of Rails is Kamal, a system that’s meant to make deployment to raw metal as simple as it is on Heroku. Kamal bundles the new Kamal Proxy, a reverse proxy that coordinates deploys, terminates TLS, and handles graceful restarts.

He’s got a point with this one. For a long time servers represented a huge capital investment and distraction from building an actual product, and in that context AWS and its ancillaries are an attractive idea. But as anyone who’s used a lot of AWS could tell you, it may be cheap in the beginning, but it’s only a matter of time until that inverts, and AWS bills become a recurring nightmare.

That said, if I were trying to send this message I’d be careful to make it clear that this is a trade off. You’re unquestionably going to save money on hardware, but you’ll spend more time on management. Someone’s also going to be the one carrying the pager for all these boxes, and presumably that’s not the 37 Signals CEO or any of its executive team.

Rails 8.1: Et tu search?

Rails 8 was released that day, and he closed the keynote by touching on some expected features for its next major release, 8.1. Next in its sights is the beast that no sane person wants to run: ElasticSearch, with the promise of bringing a sophisticated search engine into Rails itself. Also up for inclusion is “House (MD)”, which would make Markdown a more native piece of the Rails stack.

# search on any field
Post.search "announcement"

# by specific fields
Post.search title: "announcement", content: "solid search"

Twenty min

Rails World was bigger this year than last, but it’s far from a huge conference, as shown by the competitive ticketing process, where tickets were gone 20 minutes after going on sale.

Hard-to-get tickets are bad, but a positive side effect is that everyone at Rails World really wanted to be at Rails World. You don’t get there by accident. The result is that every single person you spoke to had something interesting to say. In one case I’d randomly started talking to a couple Dutch guys staying at the same hotel I was, and 15 minutes later we were talking about the trade offs of Aurora versus vanilla Postgres. This will sound self-serving, but I met quite a few people that were already familiar with this website, and they’d ask me about topics I’d written about recently like Postgres 17 bulk B-tree lookups or generating a couple secure bytes with gen_random_uuid().

I love it. The passion and expertise is the closest I’ve experienced at any event to what we used to get in the halcyon days of the early 2010s, before tech was so obviously the most important industry in the world, and became ludicrously financialized as every venture firm and Stanford graduate jumped to get a piece of it.

Unpopular opinion: Toronto

Going on its second year now, there’s a traditional announcement in the closing keynote of where the next Rails World will be held. In 2025, it’ll be back in Amsterdam, and I admit to breathing a sigh of relief (assuming I can even get in).

The Evergreen Brickworks venue is gorgeous, Shopify’s Toronto office is fabulous, and I had a good time visiting the city. But. Toronto’s downtown is enormous, and it’s the kind of place where every street, at every hour day or night, is characterized by the constant roar of total, all-encompassing, gridlock traffic. And like anywhere, when traffic is bad and tempers are heated, roads are never enough space for the pinnacle of human innovation, the automobile, and cars spill over onto every crosswalk and bike lane. With the bike lanes full, bike traffic moves onto the sidewalks, 90%+ of which is also motorized, with few riders even bothering to give lip service to those little foot rest doodads on the bottom of the bike that before the advent of the lithium battery and lightweight motor, used to be for peddling. Stop signs, red lights, and traffic priority all become the loosest of possible suggestions.

I’d be exploring an inner city suburb, with leafy canopy and the most gorgeous, stately houses that positively ooze history in all directions. Amazing! Beautiful! Except, these otherwise quiet streets are filled to the brim with hundreds of bumper-to-bumper SUVs (no self-respecting Canadian drives anything smaller than an SUV, and a family of two or more should ideally upgrade to something a little more size appropriate, like an F-350) inching their way onward at a pace only marginally faster than a brisk walk. I’d cross a bridge over a deep, forested ravine. Look over the edge, expecting to see a peaceful, bubbling brook far below. What do I see instead? A highway of course, which Torontonians have seen fit to plough through each of the city’s precious few parks.

After one of the evening parties I found myself talking to a guy who was professing his undying love for the city of Toronto. Me: what exactly do you like about it? Him: the diversityyyyyy man. Me: … okay, … anything else?

Sorry, I can’t help myself. But also, Amsterdam is the correct answer.

To recap, great event, great people. I hope to see many of you there next year.

TIL: Variables in custom VSCode snippets

2024-10-04T18:18:21Z

This blog is entirely driven by Markdown, TOML, and Git. Publishing an atom or sequence involves popping open a TOML file, adding a new item to the top, committing to Git, and pushing to origin to trigger a CI action that deploys the site:

[[atoms]]
  published_at = 2024-10-04T10:24:22-07:00
  description = """\
Hello, world!
"""

This generally works quite well, and in this developer’s humble opinion, far preferable to something involving a web UI with a little text box, but when I’m being honest with myself, I have to admit that the friction to editing is a little too high, and prevents me from publishing posts that I would’ve done if I was on a platform with a web UI and a little text box, like Twitter.

I’d been using VSCode snippets to speed up inserting a new TOML item, but the published_at date wasn’t automated, so I’d have to jump to a terminal, get a timestamp with date, then jump back and paste it. Not a big deal, but a little slow and mildly annoying.

I went back and RTFMed. It turns out that custom snippets support a number of built-in variables like $TM_FILENAME, $CURRENT_SECONDS_UNIX, or even $UUID for a random V4 UUID.

With a few more variables I got it to insert RFC3339 dates exactly like the ones I’d been grabbing from my terminal:

{
	"New atom": {
		"prefix": "at",
		"body": [
			"",
			"[[atoms]]",
			"  published_at = $CURRENT_YEAR-$CURRENT_MONTH-${CURRENT_DATE}T$CURRENT_HOUR:$CURRENT_MINUTE:$CURRENT_SECOND$CURRENT_TIMEZONE_OFFSET",
			"  description = \"\"\"\\",
			"$1",
			"\"\"\"",
			""
		],
		"description": "New atom"
	}
}

There’s quite a few other useful built-ins (e.g. currently selected text, contents of clipboard, start comment), and transformations with regex are supported.

I also took the time to get the whitespace around the inserted block exactly right, so no extra time is needed to correct it after insertion. All in all I probably saved myself about ten seconds for each snippet use, but it’s enough of a gain to make myself marginally more likely to do it.

Next up (hopefully): a mobile publishing workflow, something that’s been sorely missing for years.

Automatic Transcription and Accident Detection from Radio Chatter

2024-10-02T00:00:00Z

A few secure, random bytes without `pgcrypto`

2024-09-24T18:38:37Z

In Postgres it’s common to see the SQL random() function used to generate a random number, but it’s a pseudo-random number generator, and not suitable for cases where real randomness is required critical. Postgres also provides a way of getting secure random numbers as well, but only through the use of the pgcrypto extension, which makes gen_random_bytes available.

Pulling pgcrypto into your database is probably fine—at least it’s a core extension that’s distributed with Postgres itself—but while testing the RC version of Postgres 17 last week, I found that it was surprisingly difficult to build Postgres against OpenSSL, which is required to build pgcrypto, thereby making pgcrypto itself hard to build.

I’m broadly against the use of Postgres extensions because they make upgrades harder and projects less portable ¹, so we have a minimal posture when it comes to them, depending only on btree_gist and pgcrypto. Like pgcrypto, btree_gist is also distributed with Postgres, but unlike pgcrypto, doesn’t have an OpenSSL dependency, making it trivial to build.

Rather than wasting more time trying to get OpenSSL configured, I did a quick code audit to find out where we were using pgcrypto, and found that we were using it in exactly one place to generate random bytes for use in a ULID:

-- 10 entropy bytes
ulid = timestamp || gen_random_bytes(10);

Needing a whole extension for generating a few random bytes seems like a waste, but unfortunately Postgres doesn’t offer a built-in way to get cryptographically secure random bytes in any other way … or does it?

Secure bytes, just not for you

Internally, Postgres has a module called pg_strong_random.c that exports a pg_strong_random() function that will use OpenSSL if available, but can fall back to /dev/urandom in case it’s not, which is perfectly fine for our purposes:

/*
 * pg_strong_random & pg_strong_random_init
 *
 * Generate requested number of random bytes. The returned bytes are
 * cryptographically secure, suitable for use e.g. in authentication.
 *
 * Before pg_strong_random is called in any process, the generator must first
 * be initialized by calling pg_strong_random_init().
 *
 * We rely on system facilities for actually generating the numbers.
 * We support a number of sources:
 *
 * 1. OpenSSL's RAND_bytes()
 * 2. Windows' CryptGenRandom() function
 * 3. /dev/urandom
 *
 * Returns true on success, and false if none of the sources
 * were available. NB: It is important to check the return value!
 * Proceeding with key generation when no random data was available
 * would lead to predictable keys and security issues.
 */

So secure randomness is available without needing to dip into OpenSSL or pgcrypto. Postgres just doesn’t make it available to you.

Roundabout randomness

Luckily, there’s a workaround. pg_strong_random() is called through another function that’s exported to userspace, Postgres 13’s gen_random_uuid() which generates a V4 UUID that’s secure, random data with the exception of six variant/version bits in the middle:

Datum
gen_random_uuid(PG_FUNCTION_ARGS)
{
    pg_uuid_t  *uuid = palloc(UUID_LEN);

    if (!pg_strong_random(uuid, UUID_LEN))
        ereport(ERROR,
                (errcode(ERRCODE_INTERNAL_ERROR),
                 errmsg("could not generate random values")));

    /*
     * Set magic numbers for a "version 4" (pseudorandom) UUID, see
     * http://tools.ietf.org/html/rfc4122#section-4.4
     */
    uuid->data[6] = (uuid->data[6] & 0x0f) | 0x40;    /* time_hi_and_version */
    uuid->data[8] = (uuid->data[8] & 0x3f) | 0x80;    /* clock_seq_hi_and_reserved */

    PG_RETURN_UUID_P(uuid);
}

Given our use of pgcrypto is so limited, and we only need ten random bytes at a time for a ULID, I changed our gen_ulid() implementation to find ten bytes of randomness by pulling five bytes off the front and back of a V6 UUID:

-- 10 entropy bytes
--
-- We extract these by generating a random UUID and extracting
-- the first five bytes and last bytes out of it (thus avoiding
-- versioning bits in the middle). This is a roundabout way of
-- doing this, but is done to avoid a dependency on the pgcrypto
-- extension just to get `gen_random_bytes()`.
--
-- `uuid_send()` changes `uuid` to `bytea`.
random_uuid = uuid_send(gen_random_uuid());
ulid = timestamp ||
    substring(random_uuid FROM 1 FOR 5) ||
    substring(random_uuid FROM 12 FOR 5);

Which then lets us rid ourselves of pgcrypto, along with OpenSSL:

DROP EXTENSION pgcrypto;

Making tests against a locally built version of Postgres considerably easier.

I’m hoping we can ditch this hack as soon as V7 UUIDs land in core (they didn’t make Postgres 17, which is very sad), but in the mean time, this trick might be useful to someone else.

Direnv's `source_env`, and how to manage project configuration

2024-09-20T11:53:58Z

For years I’ve been using Direnv to manage configuration in projects. It’s a small program that loads env vars out of an .envrc file on a directory by directory basis, using a shell hook to load vars as you enter a folder, and unload them as you leave.

A typical .envrc:

export API_URL="http://localhost:5222"
export DATABASE_URL="postgres://localhost:5432/project-db"
export ENV_NAME=dev

The beauty of Direnv is not only that it’s 12-factor friendly, but that it’s language agnostic, and unlike its language-specific alternatives that hook into program code in various creative ways, Direnv makes configuration available to your main program and anything else you need to run with it.

So configuration is available for your project’s core programs:

# gets DATABASE_URL from env
make build/api && build/api

And for all adjacent utilities, including ones that you didn’t write, and would otherwise have no way of hooking into a bespoke configuration system:

# still works fine!
goose -dir ./migrations/main postgres $DATABASE_URL

Uneven distribution

For years I’ve recommended in project READMEs to get started by copying an .envrc template and running the program:

cp .envrc.sample .envrc
direnv allow
go test ./...

.envrc.sample is committed to Git while .envrc is not due to the presumption that it may eventually be edited to include user-specific secrets.

That works fine, but has always had the downside in that if configuration changes and .envrc.sample is updated, other developers don’t get those changes unless they copy a fresh .envrc.sample, and they almost certainly won’t think to do that. This is an advantage that I’d thought language-specific configuration systems like Dotenv have had over Direnv, where they can often read multiple env files, some of which may contain shared configuration that’s versioned with the repo.

The missing piece of the puzzle: `source_env`

Well, after being a Direnv user for ten years, yesterday I learnt of the existence of source_env, a special directive that can go in an .envrc and which will read out out of another envrc file.

This simplifies the configuration of my projects dramatically. They have an .envrc.sample, but it’s stripped down to almost nothing, containing only a source_env statement and room to add customization.

# Common configuration for al developers, committed to Git.
source_env .envrc.local

# Custom env values go here.

Meanwhile, all default configuration migrates to a .envrc.local (the .local suffix not having any special meaning, but rather just a convention to use):

#
# .envrc.local
#
# Shared env vars commmitted to Git and made available to all
# developers. As # much configuration should go here as possible
# so that new env vars don't break # anyone and everyone gets to
# benefit from improvements, but don't add anything too secret or
# too custom.
#

export API_URL="http://localhost:5222"
export DATABASE_URL="postgres://localhost:5432/project-db"
export ENV_NAME=dev

.envrc.local is committed to Git, and when anyone changes configuration, all other developers get the updates the next time they pull from master.

This doesn’t account for truly sensitive configuration that shouldn’t be stored in a Git repository, but my advice on that: projects should always be able to gracefully degrade so they can run (at least in development mode) with no sensitive secrets at all. And certainly the test suite should be able to. If your project can’t do that, something is wrong.

For my money, Direnv + source_env is a perfect dev configuration system, and one that works cleanly in any language ecosystem.

Engineering Mindset

2024-09-20T00:00:00Z

I recently left Google and during my last two weeks several people asked me the same question: How did you get to where you are? What they were asking is how I developed my software engineering skills…

The 1 Hour per Year Bug (But Only in Pacific Time!)

2024-08-15T00:00:00Z

:::note This article was updated in response to some helpful comments on Hacker News. I originally incorrectly formatted the Pacific Time Zone daylight saving times relative to the Eastern Time Zone!…

lonely cosmopolite

2024-08-05T00:00:00Z

I composed, produced, and released an instrumental lofi track! https://www.youtube.com/watch?v=g54wSssQCTA You can also find the track on Spotify, Apple Music, YouTube Music, and other music streaming…

HP 1651B Boot Disk Creation

2024-07-10T00:00:00Z

To Dedupe Then Sort or Sort Then Dedupe?

2024-06-30T00:00:00Z

I recently came across a deceptively simple problem. I wanted to dedupe and sort a list of integers in-place. My initial instinct was to first remove duplicates and then sort using the programming…

Eradicating N+1s: The Two-phase Data Load and Render Pattern in Go

2024-05-28T18:50:47Z

Author’s note: This is a longer piece that starts off with exposition into the nature of the N+1 query problem. If you’re already well familiar with it, you may want to skip my description of N+1 to a story involving a creative use of Ruby fibers at Stripe to try and plug this hole, or the two-phase load and render that I’ve put in my current company’s Go codebase, a pattern we’ve been using for two years now that’s rid of us N+1s, and for which I’d have trouble citing any deficiency (aside from Go’s normal trouble with verbosity). It works.

N+1 in a nutshell

Let’s say we have a model Product that can render a public-facing API resource for itself by implementing #render. I’ll be talking about API resources a lot because that’s what I’m used, but keep in mind that this could also be an object that’s used to render an HTML view and all the same concepts apply.

class Product < ApplicationRecord
  belongs_to :owner # needs to lazy load an owner

  def render
    {
      id:          self.id,
      name:        self.name,
      owner_id:    self.owner_id,
      owner_email: self.owner.email,
    }
  end
end

Some of the properties in #render like id or name come directly from the model itself, and nothing beyond the initial model needs to be loaded from the database. But some, like owner_email must be accessed through an associated record (product.owner), which the data framework (ActiveRecord in this case) will happily lazy load.

Now, say ten products are rendered in a loop:

Product.limit(10).map do |product|
  product.render
end

In this naive loop, the number of database queries issued to render all products is one (Product.limit(10)) plus ten as owner is lazily loaded on each product. That’s where we get “N+1” – one initial fetch, and N as its objects are iterated and do their own loading.

This practically invisible problem is probably number two to only forgotten indexes as the most common reason for poor performance of web apps around. It’s an easy mistake to make, and there’s a broad lack of guard rails to protect against it.

N*M+1 and more

11 queries doesn’t sound like much, but in the real world it never stops there. Let’s look at a more complicated example where Product now has multiple associated resources along with a Widget subresource that has its own associations.

class Product < ApplicationRecord
  belongs_to :owner
  belongs_to :team
  has_many :widgets # has many widgets

  def render
    {
      id:          self.id,
      name:        self.name,
      owner_id:    self.owner_id,
      owner_email: self.owner.email,
      team_id:     self.team_id,
      team_name:   self.team.name,
      widget:      self.widgets.map { |w| w.render },
    }
  end
end

class Widget < ApplicationRecord
  belongs_to :factory # needs to lazy load a factory

  def render
    {
      id:           self.id,
      factory_id:   self.factory_id,
      factory_name: self.factory.name,
      name:         self.name,
    }
  end
end

We’re now at more like N*M+1. This is the more realistic example, and in real life it just keeps snowballing from there. Models have dozens of associations, and their subresources have subresources which have subresources. Rendering a single API resource/web page might take hundreds, or even thousands, of database queries.

Luckily for all of us, databases are pretty fast, and even when abused in this fashion can still tend get the job done in a timely manner. ORMs like ActiveRecord also have features like eager loading, that can be used to prefetch what otherwise would’ve been loaded lazily.

Product.includes(owner: [], team: [], widget: [:factory]).limit(10)

But even these sophisticated strategies have their own problems. In a large application with lots of layers, it’s not obvious from any particular query if the right prefetching is happening, and it’s easy to forget eager loads or put them in the wrong place.

A digression: Fibers and intents

Sometimes you have to get creative to solve N+1s.

A story from Stripe: due to an architecture built around Mongo, records were almost always point loaded by nothing more complex than a point index lookup (i.e. no fancy joins, eager loading, or anything else, just the equivalent of WHERE id = @id). N+1s were the rule, not the exception, but with fast hardware and modest performance expectations, it’s amazing how far you can get with this brute force approach. An API request could easily run thousands of database ops.

It’s a good example of how pernicious N+1s can be. Databases are fast, and especially in the beginning, you can have the least sophisticated internal practices imaginable and they’ll still be viable. A request might be making 50 database calls, 45 of which would be unnecessary in a better-designed system, but with each taking only 1-2 ms, everything’s still done in 50-100 ms.

But over the years 50 calls becomes 1,000, and users start to notice that things are slow. And once things are this far gone, there’s no obvious fix. The latency isn’t due to only one factor, it’s a confluence of years worth of haphazardly written code, and now there’s millions of lines of it.

With no easy solutions in sight, one of my colleagues came up with what to this day is still the most novel and effective hack I’ve ever seen work in production.

API endpoints mapped to an API resource that they render. API resources were backed by a database model. Sometimes properties on the API resource mapped directly 1:1 to properties on the model, but especially over time, these representations tended to diverge, and custom overrides were required to map internal schema to public representation.

class Charge < APIResource
  prop :amount_total                               # maps to model directly
  prop :refund_total, render: :render_refund_total # renders with custom function
  prop :user_email, render :render_user_email      # renders with custom function
  
  def render_refund_total
    @model.refunds.sum { |r| r.amount_total }
  end
  
  def render_user_email
    @model.user.email
  end
end

It was these custom overrides where N+1s were most pervasive. Models used an ORM similar to ActiveRecord or Sequel that lazily loaded related records, and rendering would more often than not require loading relations. Custom overrides often rendered subresources of their own, each of which might have its own N+1s, amplifying expense to unbounded proportions.

Dynamic aggregates

This is where the innovation came in. Ruby has a construct called fibers which are coroutines with a smaller memory footprint than a thread (using only small 4 kB stacks), and which can be paused and started again. The devised scheme:

Every custom #render_* override would be wrapped in a fiber during invocation.
If the fiber called into the database layer, it’d be paused. Its “intent” to query was recorded, and the next fiber started.
After every fiber was either paused or completed, paused fibers were examined and their database intents aggregated into batch operations.
Batch operations were invoked. Their results were disaggregated, and the appropriate data distributed back to each parked fiber.
Paused fibers were continued. If new database calls were made, the sequence would start over again.

So from the example above, if 10 charges were rendered that mapped to 10 separate users, the users were bulked loaded with user_id IN (?, ?, ?, ...) instead of a single user_id = ?, but each fiber would get back a single account as if it’d performed a point load.

class Charge < APIResource
  ...
  
  def render_user_email
    #
    # fiber paused, N charge renders become `user_id IN (?, ?, ?)`, results
    # disaggregated and handed to fibers, which are then continued
    #
    @model.user.email   
  end
end

The system had broad limitations (e.g. only point loads could be aggregated; no complex queries were supported), but despite some gnarly code, it worked, and helped knock considerable latency off API calls.

Importantly, options were limited and this was one of the few ways to have a large effect across millions of lines of code. The time where the situation could’ve been rescued with a prettier/more optimal abstraction was long since past.

Rails strict loading

N+1s are a constant threat in frameworks like ActiveRecord where lazy loading is common. Lazy loading is preventable with eager loading like #includes / #eager_load / #preload, but is difficult to guarantee because even if all relations were eager loaded initially, it’s easy to accidentally regress as a new lazy load is introduced.

To help ratchet down on the problem, Rails 6.1 introduced strict loading, wherein lazy loading becomes an error. The idea is that tests will exercise code which will fail if it performs a lazy load, allowing all instances of it to be banished before deployment.

config.active_record.strict_loading_by_default = true

class Article < ApplicationRecord
  self.strict_loading_by_default = true

  has_many :comments
end

Strict loading is an important feature and major innovation in this area, but not a panacea. Test coverage needs to be substantial to make sure problems are caught before hitting production.

Loading data in Go, exceptional verbosity

This brings us to Go, where loading data is hard even without considering N+1s.

Go can aptly be described as a newer, safer C, but with even less flexibility. You couldn’t write a good ORM for the language if you wanted to (they do exist, but rely on a lot of untyped any shenanigans, which defeats the type advantages of Go in the first place since problems are only caught at runtime), and in the absence of one, the Go philosophy is to avoid abstraction. If you need something like an API resource, piece it together query-by-query, with requisite if err != nil { ... } blocks after every statement.

For larger applications with dozens or hundreds of associations, the default result is a breathtaking amount of boilerplate to accomplish what would be a modest amount of code in a language with more succinct syntax and a dynamic ORM.

The increased verbosity does nothing to make N+1s less likely, which are still easy to introduce in a loop, especially with layers of indirection. It also makes them harder to fix because there might be a lot of refactoring involved. One of the first bugs I ever fixed coming onto the job was an N+1:

commit de58e3552eaef78c9b3d7779ddf9c646d5009985
Author: Brandur <brandur@brandur.org>
Date:   Thu Jun 3 13:06:56 2021 -0700

    Fix N+1 query getting replicas on cluster list

    We currently have an N+1 situation when listing clusters wherein we query
    replicas for every cluster picked up in the original list. This leads to
    poor performance where a user has many clusters.

    Here we fix the problem by introducing a new query that's able to select
    replicas based on a set of input IDs, and after fetching them, we assign
    them to cluster objects appropriately.

It was about as classic of a mistake as is possible. A query in a loop:

for _, cluster := range clusters {
    replicas, err := svc.getReplicasByClusterID(ctx, svc.executor(), cluster.ID)

    if err != nil {
        plog.Logger(ctx).Errorf("could not retrieve replicas for cluster id=[%s]: %s",
            cluster.ID, err.Error())
        continue
    }

    cluster.Replicas = replicas
}

This one’s is easy to spot, but once queries are folded into functions and other abstractions, they get less visible and harder to address.

The fix was to query many clusters at once before the loop, and piece them together inside of it, requiring an impressive amount of code for quite a commonplace operation. (This was before generics arrived in 1.18, so even basic tasks like mapping a slice to a keyed map wasn’t possible with less than four lines of code.)

// Code in this block retrieves any replicas for these clusters and assigns
// them appropriately. All replicas are selected in one query to avoid an N+1
// problem. It would be nice to generalize this pattern because it's not pretty.
{
    clusterIDs := make([]pgtype.UUID, len(clusters))
    for i, cluster := range clusters {
        clusterIDs[i] = db.MakeUUID(cluster.ID).UUID
    }

    replicas, err := svc.getReplicasByClusterIDs(ctx, svc.executor(), clusterIDs)
    if err != nil {
        return nil, err
    }

    clusterMap := make(map[string]*dbops.Cluster)
    for _, cluster := range clusters {
        clusterMap[cluster.ID] = cluster
    }

    for _, replica := range replicas {
        cluster := clusterMap[replica.ClusterID]
        cluster.Replicas = append(cluster.Replicas, replica)
    }
}

Beyond the eyesore, this case-by-case approach doesn’t scale well code wise either. Even this example for a single API resource with one sub-list is already messy. What would happen for one with dozens of subresources, each of which might have dozen of subresources of their own? Then add a half dozen different developers into the equation, none of whom will have perfect insight into or understanding of code that anyone else wrote.

Despite Go’s ad nauseum verbosity, it’s no less susceptible to N+1s than a language heavy in metaprogramming like Ruby.

Two-phase load and render

This is where our generalized data loading pattern comes in. It doesn’t make N+1s impossible, but it forces developers to break convention to introduce them, making adding a new one harder than not doing so.

As the name suggests, it’s broken down into two distinct render phases:

Load phase: Generates a load bundle from the database containing everything needed to render an arbitrary number of resources. Load phases always load data for N resources, even if only a single one is being rendered.
Render phase: Using a load bundle, renders a single resource. No database access is allowed.

The key insight is that the load phase knows how to load data to a bundle that’s sufficient to render N resources. For a list endpoint, render may then be called using that bundle for N resources in the list. For a point retrieval endpoint, it’ll render only one resource. Either way, the process is the same.

Let’s look at a basic example. A product API resource, each of which has one admin and belongs to a team:

package apiresourcekind

type Product struct {
    apiresource.APIResourceBase

    ID         uuid.UUID `json:"id"`
    Name       string    `json:"name"`
    OwnerID    uuid.UUID `json:"owner_id"`
    OwnerEmail string    `json:"owner_email"`
    TeamID     uuid.UUID `json:"team_id"`
    TeamName   string    `json:"team_email"`
}

//
// Phase 1: Load data into a bundle
//

type ProductLoadBundle struct {
    accounts map[uuid.UUID]*dbsqlc.Account // account ID -> account
    teams    map[uuid.UUID]*dbsqlc.Team    // team ID -> team
}

func (_ *Product) LoadBundle(
    ctx context.Context, e db.Executor, baseParams *pbaseparam.BaseParams, products []*dbsqlc.Product
) (*ProductLoadBundle, error) {
    var (
        bundle  = &ProductLoadBundle{}
        queries = dbsqlc.New(e)
    )

    // Load owners for all products, map them in bundle by ID.
    {
        accounts, err := queries.AccountGetByIDMany(ctx,
            sliceutil.Map(products, func(p *dbsqlc.Product) uuid.UUID { return p.OwnerID }))
        if err != nil {
            return nil, xerrors.Errorf("error getting accounts: %w", err)
        }
        bundle.accounts = sliceutil.KeyBy(accounts, func(a *dbsqlc.Account) uuid.UUID { return a.ID })
    }

    // Load teams for all products, map them in bundle by ID.
    {
        teams, err := queries.TeamGetByIDMany(ctx,
            sliceutil.Map(products, func(p *dbsqlc.Product) uuid.UUID { return p.TeamID }))
        if err != nil {
            return nil, xerrors.Errorf("error getting teams: %w", err)
        }
        bundle.teams = sliceutil.KeyBy(teams, func(t *dbsqlc.Team) uuid.UUID { return t.ID })
    }

    return bundle, nil
}

(Once again, please forgive the verbosity – there is literally no way to make this code more succinct in Go. It’s already boiled down as far as possible.)

//
// Phase 2: Use a bundle to render a single resource
//

func (_ *Product) Render(
    ctx context.Context, baseParams *pbaseparam.BaseParams, bundle *ProductLoadBundle, product *dbsqlc.Product
) (*Product, error) {
    return &Product{
        ID:         product.ID,
        Name:       product.Name,
        OwnerID:    product.OwnerID,
        OwnerEmail: bundle.accounts[product.OwnerID].Email,
        TeamID:     product.TeamID,
        TeamName:   bundle.teams[product.TeamID].Name,
    }, nil
}

A Product is rendered from a ProductLoadBundle bundle and dbsqlc.Product database model. Some properties like ID and Name are inherent to the product itself and are reflected directly into the API resource, but others like OwnerEmail and TeamName are only accessible by loading other database records and accessing their properties.

So, the full render process is:

LoadBundle is invoked once (regardless of the number of products being rendered).
- Owner and team records are loaded in bulk for every product (e.g. queries.AccountGetByIDMany is generated by sqlc, and maps to roughly SELECT * FROM account WHERE id = any(@id::uuid[])).
- Owners and teams are placed into maps on ProductLoadBundle key to their IDs.
Render is invoked for each product individually, but reusing the same load bundle from (1).
- Properties like ID and Name map directly from model to API resource.
- Indirect properties like OwnerEmail and TeamName are pulled off the records added to the load bundle in (1).

Renderable

Implementing a full two-phase render involves a fair bit of code (again, it’s Go), but once it’s done, that type of API resource can easily be rendered from anywhere else:

resource, err := apiresource.Render[*apiresourcekind.Product](
    ctx, tx, svc.BaseParams, product
)
if err != nil {
    return nil, err
}

And rendering many API resources at once (like on a list endpoint) looks like:

resources, err := apiresource.RenderMany[*apiresourcekind.Product](
    ctx, tx, svc.BaseParams, products
)
if err != nil {
    return nil, err
}

Returned API resources implement Renderable, which holds types for bundle, model, and API resource:

package apiresource

// Renderable is an API resource that can be rendered by Render or RenderMany.
type Renderable[TLoadBundle any, TModel any, TResource any] interface {
    // LoadBundle loads a load bundle for the given models, usually from a
    // database, which can then be used along with a model to render a full API
    // resource.
    //
    // It may seem odd that this takes a slice of models instead of a model, but
    // this is for a good reason: it lets us batch load all data dependencies
    // all at once instead of loading them one-by-one, causing an N+1 problem.
    LoadBundle(ctx context.Context, e db.Executor, baseParams *pbaseparam.BaseParams, models []TModel) (TLoadBundle, error)

    // Render renders an API resource using a load bundle and model as input.
    Render(ctx context.Context, baseParams *pbaseparam.BaseParams, bundle TLoadBundle, model TModel) (TResource, error)
}

From there, implementations for Render and RenderMany are trivial, each loading a bundle once, and then rendering either a single or slice of API resources:

package apiresource

// Render renders an API resource.
//
// The type parameters may appear to be in a weird order as you might expect
// TModel before TRenderable, but it's like this for a good reason. Type
// parameters that can be inferred can be omitted, and in general use of Render
// only TRenderable needs to be included. Both TModel and TRenderable are
// inferred and should be omitted.
func Render[TRenderable Renderable[TLoadBundle, TModel, TRenderable], TLoadBundle any, TModel any](
    ctx context.Context, e db.Executor, baseParams *pbaseparam.BaseParams, model TModel,
) (TRenderable, error) {
    var renderable TRenderable

    bundle, err := renderable.LoadBundle(ctx, e, baseParams, []TModel{model})
    if err != nil {
        return renderable, xerrors.Errorf("error loading bundle: %w", err)
    }

    resource, err := renderable.Render(ctx, baseParams, bundle, model)
    if err != nil {
        return renderable, xerrors.Errorf("error rendering resource: %w", err)
    }

    return resource, nil
}

// RenderMany is similar to Render, but renders many API resources at once.
func RenderMany[TRenderable Renderable[TLoadBundle, TModel, TRenderable], TLoadBundle any, TModel any](
    ctx context.Context, e db.Executor, baseParams *pbaseparam.BaseParams, models [TModel,
) ([]TRenderable, error) {
    var renderable TRenderable

    bundle, err := renderable.LoadBundle(ctx, e, baseParams, models)
    if err != nil {
        return nil, xerrors.Errorf("error loading bundle: %w", err)
    }

    resources := make([]TRenderable, len(models))

    for i := range resources {
        resources[i], err = renderable.Render(ctx, baseParams, bundle, models[i])
        if err != nil {
            return nil, xerrors.Errorf("error rendering resource: %w", err)
        }
    }

    return resources, nil
}

Edit (2024/06/14): This section was updated after Roman pointed out that by swapping the positions of two generic parameters, most of them can be inferred by the compiler, and Render can be called with only a single generic parameter.

Nested resources

But what about subresources? If we need to call apiresource.Render inside the Render implementation of another resource, N+1s boomerang right back.

This is where the pattern shines. N+1s are avoided by composing load bundles onto other load bundles so the Load implementation of a resource invokes Load for its subresources as well, always ensuring that there is never more than one Load per resource type.

This is best demonstrated by example. Let’s augment Product above so that it renders a list of Widget subresources. Widgets need to do some data loading of their own, to get the location of the factory they’re produced at. Widget’s Renderable implementation (widget is a leaf resource so there’s nothing exotic here):

package apiresourcekind

type Widget struct {
	apiresource.APIResourceBase

	ID              uuid.UUID `json:"id"`
	FactoryID       uuid.UUID `json:"factory_id"`
	FactoryLocation string    `json:"factory_location"`
	Name            string    `json:"name"`
}

//
// Renderable implementation
//

type WidgetLoadBundle struct {
	factories map[uuid.UUID]*dbsqlc.Factory // factory ID -> factory
}

func (_ *Widget) LoadBundle(ctx context.Context, e db.Executor, baseParams *pbaseparam.BaseParams, widgets []*dbsqlc.Widget) (*WidgetLoadBundle, error) {
	var (
		bundle  = &WidgetLoadBundle{}
		queries = dbsqlc.New(e)
	)

	// Load factories for all widgets, map them in bundle by ID.
	{
		factories, err := queries.FactoryGetByIDMany(ctx,
			sliceutil.Map(widgets, func(w *dbsqlc.Widget) uuid.UUID { return w.FactoryID }))
		if err != nil {
			return nil, xerrors.Errorf("error getting factories: %w", err)
		}
		bundle.factories = sliceutil.KeyBy(factories, func(f *dbsqlc.Factory) uuid.UUID { return f.ID })
	}

	return bundle, nil
}

func (_ *Widget) Render(ctx context.Context, baseParams *pbaseparam.BaseParams, bundle *WidgetLoadBundle, widget *dbsqlc.Widget) (*Widget, error) {
	return &Widget{
		ID:              widget.ID,
		FactoryID:       widget.FactoryID,
		FactoryLocation: bundle.factories[widget.FactoryID].Location,
		Name:            widget.Name,
	}, nil
}

Now, back to product’s (the parent resource) Renderable implementation, now modified to include widgets. WidgetLoadBundle is embedded on ProductLoadBundle and populated on Load. Product’s Render invokes Render for each of its embedded widgets, passing through the common load bundle:

package apiresourcekind

type Product struct {
	apiresource.APIResourceBase

	ID         uuid.UUID `json:"id"`
	Name       string    `json:"name"`
	OwnerID    uuid.UUID `json:"owner_id"`
	OwnerEmail string    `json:"owner_email"`
	TeamID     uuid.UUID `json:"team_id"`
	TeamName   string    `json:"team_email"`
	Widgets    []*Widget `json:"widget"`     // NEW!!
}

//
// Renderable implementation
//

type ProductLoadBundle struct {
	accounts     map[uuid.UUID]*dbsqlc.Account  // account ID -> account
	teams        map[uuid.UUID]*dbsqlc.Team     // team ID -> team
	widgetBundle *WidgetLoadBundle              // <-- the product load bundle has a widget load bundle!
	widgets      map[uuid.UUID][]*dbsqlc.Widget // product ID -> widgets
}

func (_ *Product) LoadBundle(ctx context.Context, e db.Executor, baseParams *pbaseparam.BaseParams, products []*dbsqlc.Product) (*ProductLoadBundle, error) {
	var (
		bundle  = &ProductLoadBundle{}
		queries = dbsqlc.New(e)
	)

    ...

	// Load widgets for all products, group them in bundle by product ID, and load widget bundle.
	{
		widgets, err := queries.WidgetGetByProductIDMany(ctx,
			sliceutil.Map(products, func(p *dbsqlc.Product) uuid.UUID { return p.ID }))
		if err != nil {
			return nil, xerrors.Errorf("error getting widgets: %w", err)
		}
		bundle.widgets = sliceutil.GroupBy(widgets, func(w *dbsqlc.Widget) uuid.UUID { return w.ProductID })

		bundle.widgetBundle, err = (&Widget{}).LoadBundle(ctx, e, baseParams, widgets)
		if err != nil {
			return nil, err
		}
	}

	return bundle, nil
}

func (_ *Product) Render(ctx context.Context, baseParams *pbaseparam.BaseParams, bundle *ProductLoadBundle, product *dbsqlc.Product) (*Product, error) {
	// Render widget subresources.
	var widgetResources []*Widget
	if widgets, ok := bundle.widgets[product.ID]; ok {
		widgetResources := make([]*Widget, len(widgets))
		for i, widget := range widgets {
			var err error
			widgetResources[i], err = (&Widget{}).Render(ctx, baseParams, bundle.widgetBundle, widget)
			if err != nil {
				return nil, err
			}
		}
	}

	return &Product{
		ID:         product.ID,
		Name:       product.Name,
		OwnerID:    product.OwnerID,
		OwnerEmail: bundle.accounts[product.OwnerID].Email,
		TeamID:     product.TeamID,
		TeamName:   bundle.teams[product.TeamID].Name,
		Widgets:    widgetResources,
	}, nil
}

The beauty of this approach is that even if your resources which have subresources which have subresources, it’s still okay. All load bundles map 1:1:1, and regardless of number of resources or hierarchy, we still perform a constant number of database operations. Predictable performance is always maintained.

Beyond Go

Go is special because of its overwhelming verbosity and total lack of dynamic features. Even if we hadn’t designed a framework to avoid N+1s, we would’ve had to build one to help with basic data loading, so with the two-phase load and render approach we kill two birds with one stone.

With that said, Rails’ strict loading feature is a bit of an abberation. Many ORMs offer similar dynamic APIs that perform lazy loading, but without safety rails, which practically makes N+1s the default. Common practice is to live with them, and if a particular hot spot becomes a performance problem, to go in and whack-a-mole N+1s one at a time.

The two-phase approach could be extended to other languages to help make N+1s less common and more easily addressable. The syntax above looks intimidating, but once again that’s mostly a Go verbosity problem. In most languages, you could do something similar with half the lines of code.

The specific code above is meant more for inspiration than anything else, and I’m not providing any particular package prescriptions. But it involves only a few plain Go structs, one interface, and two functions, so it’s easy to reproduce.

The Notifier Pattern for Applications That Use Postgres

2024-05-06T05:54:07Z

Listen/notify in Postgres is an incredible feature that makes itself useful in all kinds of situations. I’ve been using it a long time, started taking it for granted long ago, and was somewhat shocked recently looking into MySQL and SQLite to learn that even in 2024, no equivalent exists.

In a basic sense, listen/notify is such a simple concept that it needs little explanation. Clients subscribe on topics and other clients can send on topics, passing a message to each subscribed client. The idea takes only three seconds to demonstrate using nothing more than a psql shell:

=# LISTEN test_topic;
LISTEN
Time: 2.828 ms

=# SELECT pg_notify('test_topic', 'test_message');
 pg_notify
-----------

(1 row)

Time: 17.892 ms
Asynchronous notification "test_topic" with payload "test_message" received from server process with PID 98481.

But despite listen/notify’s relative simplicity, when it comes to applications built on top of Postgres, it’s common to use it less than optimally, eating through scarce Postgres connections and with little regard to failure cases.

Here’s where the notifier pattern for Postgres comes in. It’s an extremely simple idea, but in my experience, one that’s rarely seen in practice. Let’s start with these axioms:

LISTENs are affixed to specific connections. After listening, the original connection must still be available somewhere to successfully receive messages.
There may be many components within an application that’d like to listen on topics for completely orthogonal uses.
Despite optimizations over the years, connections in Postgres are still somewhat of a precious, limited resource, and should be conserved. We’d like to minimize the number of them required for listen/notify use.
A single connection can listen on any number of topics.

With those stated, we can explain the role of the notifier. Its job is to hold a single Postgres connection per process, allow other components in the same program to use it to subscribe to any number of topics, wait for notifications, and distribute them to listening components as they’re received.

The “single Postgres connection per process” piece is key. Use of a notifier keeps the number of Postgres connections dedicated to use with listen/notify down to one per program, a major advantage compared to the naive version, which is one connection per topic per program. Especially for languages like Go that make a in-process concurrency easy and cheap, the notifier reduces listen/notify connection overhead to practically nil.

A few implementation details

From a conceptual standpoint, the notifier’s not difficult to understand, and with only this high level description, most readers would be able to implement it themselves. I’m not going to go through an implementation in full detail, but let’s look at a few important aspects of one. (For a complete reference, you can take a look at River’s notifier, which is quite well vetted.)

Here’s a listen function to establish a new subscription:

// Listen returns a subscription that lets a caller receive values from a
// notification channel.
func (l *Notifier) Listen(channel string) *Subscription {
    l.mu.Lock()
    defer l.mu.Unlock()

    existingSubs := l.subscriptions[channel]

    sub := &Subscription{
        channel:        channel,
        listenChan:     make(chan string, 100),
        notifyListener: l,
    }
    l.subscriptions[channel] = append(existingSubs, sub)

    if len(existingSubs) > 0 {
        // If there's already another subscription for this channel, reuse its
        // established channel. It may already be closed (to indicate that the
        // connection is established), but that's okay.
        sub.establishedChan = existingSubs[0].establishedChan
        sub.establishedChanClose = func() {} // no op since not channel owner

        return sub
    }

    // The notifier will close this channel after it's successfully established
    // `LISTEN` for the given channel. Gives subscribers a way to confirm a
    // listen before moving on, which is especially useful in tests.
    sub.establishedChan = make(chan struct{})
    sub.establishedChanClose = sync.OnceFunc(func() { close(sub.establishedChan) })

    l.channelChanges = append(l.channelChanges,
        channelChange{channel, sub.establishedChanClose, channelChangeOperationListen})

    // Cancel out of blocking on WaitForNotification so changes can be processed
    // immediately.
    l.waitForNotificationCancel() 

    return sub
}

A few key details to notice:

Subscriptions use a buffered channel like make(chan string, 100) and non-blocking sends (using select with default). A notifier may receive a high volume of notifications, and if it were to block on every component successfully receiving and processing each one, it could easily fall behind. Instead, a received notification is sent into the channel using a non-blocking send. The non-blocking send means that the send operation will never block: instead the notification is discarded if the channel is full. The buffer provides a tunable amount of slack to make sure this won’t happen too easily. It’s each component’s job to make sure its processing its inbox in a timely manner. This is important because even in the event of one component falling behind, the system as a whole stays healthy.
Multiple components may want to subscribe to the same topic. Since only one connection is in use, the notifier only needs to issue one LISTEN per topic. Internally, it organizes subscriptions by topic, and if it notices that a topic already exists, a new subscription is added without issuing LISTEN.
Subscriptions provide an established channel that’s closed when a LISTEN has been successfully issued and the notifier is up and listening. This isn’t strictly necessary for most production uses, but it’s invaluable for use in testing. If a test case issues pg_notify before the notifier has started listening, that notification is lost – a problem that can lead to tortuous test intermittency ¹. Instead, a test case tells the notifier to listen, waits for the listen to succeed, then moves on to send pg_notify.

// EstablishedC is a channel that's closed after the notifier's successfully
// established a connection. This is especially useful in test cases, where it
// can be used to wait for confirmation that not only that the listener is
// started, but that it's successfully established started listening on a
// channel before continuing. For a new subscription on an already established
// channel, EstablishedC is already closed, so it's always safe to wait on it.
//
// There's no full guarantee that the notifier can ever successfully establish a
// listen, so callers will usually want to `select` on it combined with a
// context done, a stop channel, and/or a timeout.
//
// The channel is always closed as a notifier is stopping.
func (s *Subscription) EstablishedC() <-chan struct{} { return s.establishedChan }

Interruptible receives

There’s no standard SQL for waiting for a notification. Typically, it’s accomplished using a special driver-level function like Pgx’s WaitForNotification.

These commonly block until receiving a notification, which can be problem since we’re only using a single connection. What if the notifier is in a blocking receive loop, but another component wants to add a new subscription that requires LISTEN be issued?

You’ll want to handle this case by making sure that the wait loop is interruptible. Here’s one way to accomplish that in Go:

func (l *Notifier) runOnce(ctx context.Context) error {
    if err := l.processChannelChanges(ctx); err != nil {
        return err
    }

    // WaitForNotification is a blocking function, but since we want to wake
    // occasionally to process new `LISTEN`/`UNLISTEN` operations, we put a
    // context deadline on the listen, and as it expires don't treat it as an
    // error unless it's unrelated to context expiration.
    notification, err := func() (*pgconn.Notification, error) {
        const listenTimeout = 30 * time.Second

        ctx, cancel := context.WithTimeout(ctx, listenTimeout)
        defer cancel()

        // Provides a way for the blocking wait to be cancelled in case a new
        // subscription change comes in.
        l.mu.Lock()
        l.waitForNotificationCancel = cancel
        l.mu.Unlock()

        notification, err := l.conn.WaitForNotification(ctx)
        if err != nil {
            return nil, xerrors.Errorf("error waiting for notification: %w", err)
        }

        return notification, nil
    }()
    if err != nil {
        // If the error was a cancellation or the deadline being exceeded but
        // there's no error in the parent context, return no error.
        if (errors.Is(err, context.Canceled) ||
            errors.Is(err, context.DeadlineExceeded)) && ctx.Err() == nil {
            return nil
        }

        return err
    }

    l.mu.RLock()
    defer l.mu.RUnlock()

    // Notify subscribers (this is a no-op if no subs/empty slice).
    for _, sub := range l.subscriptions[notification.Channel] {
        sub.listenChan <- notification.Payload
    }

    return nil
}

The inner closure calls into WaitForNotification, but has a default context timeout of 30 seconds that automatically cycles the function periodically. It also stores the special context cancellation function l.waitForNotificationCancel.

When Listen is invoked and a new subscription needs to be added, l.waitForNotificationCancel is called. The wait is cancelled immediately, new subscriptions are processed, and the closure is reentered to wait anew.

Let it crash

Given there’s now a single master connection that’s handling all notifications for a program, it’s fairly critical that its health be monitored, and the notifier reacts appropriately. If not, all uses of listen/notify would degrade simultaneously.

The obvious way to react would be to close the connection, use a connection pool to procure a new connection, reissue LISTENs for each active subscription, then reenter the wait loop.

It can be a little tricky sometimes to guarantee that state is reset cleanly, so another possibility is to adhere to the “let it crash” school of thought. If the connection becomes irreconcilably unhealthy, stop the program, and have it come back to a healthy state by virtue of its normal start up.

// If the notifier gets unhealthy, restart the worker. This will generally
// never happen as the notifier has a built-in retry loop that try its best
// to keep established before giving up.
notifier.AddUnhealthyCallback(closeShutdown)

We’ve found this sort of edge to be so rare (I’ve only seen it happen once in a year+ of use) that letting the program crash when it does happen hasn’t produced any undue disruption.

PgBouncer

Using PgBouncer, LISTEN is only supported using session pooling (as opposed to transaction pooling) because notifications are only sent to the original session that issued a LISTEN for them.

Use of a notifier requires an app to dedicate a single connection per program for listen/notify, but every other part of the application is free to use PgBouncer in transaction pooling or statement pooling mode, thereby maximizing the efficiency of connection use.

Web APIs: Enriched DX By Disallowing Unknown Fields

2024-05-05T06:21:27Z

Go’s JSON library provides the decoder option DisallowUnknownFields which even if not intuitively obvious, is a handy option fo adding a layer of improved DX to web APIs. As the name would suggest, it causes a decoder to error when encountering a property in a JSON object being decoded that’s not present in the struct being decoded to.

type Request struct {
    Message string `json:"message"`
}

data := `{"message":"Hello.","unknown":"Not a field on the struct."}`

decoder := json.NewDecoder(bytes.NewReader([]byte(data)))
decoder.DisallowUnknownFields()

var req Request
if err := decoder.Decode(&req); err != nil {
    log.Fatal(err) // json: unknown field "unknown"
}

When a user is integrating a web API, especially in the beginning, it’s common for the initial prototype to be written by a human, and humans are prone to making mistakes. Say you’re trying to programmatically procure an access token against POST /access-tokens. The endpoint takes an optional parameter called expires_in which is a number of seconds after which the new access token will expire automatically. By virtue of reading the documentation slightly wrong, you’re accidentally sending expires: 3600 instead of expires_in: 3600. The result is that your requested expiry time is silently ignored, not only producing the wrong result, but possibly even a security leak as your account accidentally amasses access tokens that never expire.

DisallowUnknownFields widely fixes this class of mistake for all an API’s users. Some code extracted from our API:

decoder := json.NewDecoder(bytes.NewReader(rawPayload))

// Balks if finding fields in the request payload that don't map to anything
// in the target request structure. Acts as a small DX aid for users who may
// have accidentally misnamed a field.
//
// Specific API endpoints can invert this behavior through and option while
// defining the endpoint.
if !allowUnknownJSONFields {
        decoder.DisallowUnknownFields()
}

if err := decoder.Decode(v); err != nil {
    apierror.NewBadRequestError(
        r.Context(),
        fmt.Sprintf("Invalid JSON in request body: %s.", err),
    ).Write(r.Context(), w)
    return nil, false
}

Now, sending expires instead of expires_in is an error that tells the user exactly what’s wrong:

$ curl -i -H "Authorization: Bearer $CRUNCHY_API_KEY" \
    -H "Content-Type: application/json"
    -X POST $CRUNCHY_API_URL/access-tokens -d '{"expires":3600}'

HTTP/2 400
{
    "message":"Invalid JSON in request body: json: unknown field \"expires\".",
    "request_id":"5d2078fe-6ea5-4f41-816e-4717cf6c22b7"
}

It’s a feature that’s not needed every day, but it’s easy to implement, and the day it is, it’ll save hours worth of time and frustration.

Caveats and edges

There are a few sharp edges to be aware of. They’re easy to avoid once you know about them, but aren’t totally apparent for those integrating the pattern for the first time.

Turning it on safely

If you have an existing API with existing users, DisallowUnknownFields isn’t universally safe to turn on because there may be integrations out there that have been sending invalid JSON fields for years, but which was never a problem before. Those previously happy users become unhappy when disallowing unknown fields suddenly breaks all their requests.

You can still turn it on, but doing so takes a few more steps:

Start by organizing the API by pre and post DisallowUnknownFields. New API endpoints get the check automatically while existing ones default to it off.
Add logging probes to existing endpoints that fire when they encounter an unknown parameter. Search your logs for these later to see what unknown parameters are present, if any, and how many.

if err := decoder.Decode(v); err != nil {
    if strings.Contains(err.Error(), "unknown field") {
        plog.Logger(ctx).WithFields(logrus.Fields{
            "api_endpoint_method": r.Method,
            "api_endpoint_path":   r.URL.Path,
        }).Warnf("Unknown field error: %s.", err)

        decoderAllowingUnknown := json.NewDecoder(bytes.NewReader(rawPayload))
        err = decoderAllowingUnknown.Decode(v)
    }

    if err != nil {
        apierror.NewBadRequestError(
            r.Context(),
            fmt.Sprintf("Invalid JSON in request body: %s.", err),
        ).Write(r.Context(), w)
        return nil, false
    }
}

Reaching out to individual users and asking them to correct bad parameters is possible, but probably more trouble than it’s worth. A cheaper solution is to grandfather in existing errors by adding hidden fields to JSON structs that’ll let DisallowUnknownFields be enabled for the endpoint, but keep existing integrations compatible.

// Request parameters for creating a new access token.
type AccessTokenCreateRequest struct {
    ...

    // When activating strict JSON parameter validation we found that Customer X
    // was accidentally sending `expires` instead of `expires_in`. We've asked
    // them to stop, but in the meantime we allow this parameter so we don't
    // break them.
    Expires int `json:"expires" openapi:"hide" validate:"-"`
}

There’s a point where doing this for too many unknown fields becomes impractical, but for all but the largest APIs, unknown fields will be an edge that with a little luck, isn’t that common.

Deprecating fields carefully

When removing an old field from the API it might be tempting to strip it out request structs completely. It just makes sense right? If it’s ignored anyway and not used anywhere then why should it be in there.

DisallowUnknownFields will require more care in deprecating fields. Even if the parameter hasn’t been doing anything useful in years, it may still be sent by users, and if it’s removed, those existing integrations break.

The workaround is to keep deprecated parameters passed their expiration date, but mark them as such in a way that bubbles up to public documentation and generated bindings that makes it clear that they’re not useful and should no longer be used.

// Request parameters for creating a new access token.
type AccessTokenCreateRequest struct {
    ...

    // Client ID is the unique identifier of the API key that the new access
    // token should be associated with.
    //
    // Deprecated: This field used to be required, but an associated access
    // token is now inferred automatically using the secret included as part of
    // the `Authorization` header. This parameter is now ignored.
    ClientID *eid.EID `json:"client_id" validate:"-"`
}

Once again, logging probes come in handy here. Add a unique string like access_token_client_id_received that’s easily searchable in logs, and some time later once it hasn’t been seen in a long time, do a clean up pass and strip the old parameter out.

Prepare an escape hatch

Use of DisallowUnknownFields is suitable for most API endpoints, but an escape hatch will be required, so prepare for it.

A common place where DisallowUnknownFields should not be applied are webhook receive endpoints. Although in a fashion they’re technically part of your API’s surface area, they’re really more like the push API of another vendor, and because adding a new field to an API is widely considered to not be a breaking change, that vendor may add new parameters to their webhook pushes anytime.

The problem can be especially insidious because the webhook APIs of many large vendors are quite stable, so your receiver will be working fine with DisallowUnknownFields for many months or years, before suddenly every request starts failing overnight as a new parameter is added.

Our in house API endpoint framework takes the option AllowUnknownJSONFields to indicate that JSON requests should not ban unknown fields:

// Webhook endpoint where Stripe broadcasts asynchronous message about customer
// payment information.
type StripeWebhookEndpoint struct{}

func (e *StripeWebhookEndpoint) Materialize() apiendpoint.APIEndpointer {
    return &apiendpoint.APIEndpoint[StripeWebhookRequest, StripeWebhookResponse]{
        Extras: apiendpoint.APIEndpointExtras{
            AllowUnknownJSONFields: true, // <-- unknown fields allowed
        },
        Method: http.MethodPost,
        Route:  "/webhook",
        ServiceHandler: func(svc any) func(ctx context.Context, req *StripeWebhookRequest) (*StripeWebhookResponse, error) {
            return svc.(StripeService).Webhook
        },
        SuccessStatusCode: http.StatusOK,
        Title:             "Stripe webhook receiver",
    }
}

Use outside Go

DisallowUnknownFields is obviously an option specific to Go, but this pattern is widely reusable in other languages, and easy to implement yourself if it’s not built into the ecosystem’s dominant JSON package.

Augmentation with Levenshtein distance

An obvious next augmentation is not only to indicate that a parameter name doesn’t exist, but to use the Levenshtein distance to known parameter names to suggest one. So a user who sends expires is told that they probably meant expires_in, giving them a path to resolution that takes seconds instead of minutes.

Invalid JSON in request body: unknown field "expires". Did you mean "expires_in"?"

MIT Flea Market Survival Guide

2024-04-25T00:00:00Z

The 29 Days per Year Bug (30 Days for Leap Years!)

2024-04-23T00:00:00Z

A few years ago my coworker was working on a Google Docs feature when they came across a bewildering bug. The feature 📅 For one component of this feature, users were able to pick any date, save it,…

Validating Markdoc attribute values with Zod

2024-04-19T00:00:00Z

In a React application written in TypeScript, the language’s type system is the first line of defense against type errors in component props. In Markdoc, where content is decoupled from rendering logic and can’t take advantage of TypeScript types, the schema-based validation system plays an important role in bridging the gap between declarative tag definitions and typed React components. The Markdoc validator ensures that tag attribute values in a document match the types declared in the corresponding schema.

Accessing parent nodes in a Markdoc validation function

2024-04-18T00:00:00Z

A recent Markdoc release introduced a new feature that makes it possible for validation functions to access the parents of the current node. This feature is useful for writing custom validation logic that analyzes document structure or enforces rules based on a node’s position in the hierarchy. For example, you might want to impose restrictions on where an image can be nested in a document, prohibiting the use of images in specific tags like callouts and asides.

My Total Solar Eclipse Balloon Launch

2024-04-09T00:00:00Z

ASMR and vulnerability

2024-03-15T19:12:47Z

I’ve been really into ASMR videos recently, which if you’ve been living under a rock are videos that contain sounds that promote the tingly sensation in the back of your head. Not everyone experiences this and it’s still a little unknown (from my brief review of literature). This is my pseudo-scientific attempt to explain it. First, if you’ve never experienced it, I’d say that the feeling is oddly similar to the tingly feeling you get if you point a sharp object (or even someone else’s finger) between your eyes.

The end of Airplane.dev

2024-03-03T20:42:00Z

I worked at Airplane, an internal tooling startup, for nearly two years. Earlier this year, it was announced that the company was being acquired by Airtable and that the product would be shut down. In this blog post, I want to explain what happened from my perspective as a former employee.

Background

I joined Airplane in March of 2022 because I was excited by the internal tooling space and, after some time at Twilio, wanted to work at a smaller company.

For the first 16 months or so, it felt like the company was doing well. We were onboarding lots of new customers, we expanded the team from 12 people to more than 20, and there was a general sense of excitement about our prospects both internally and in conversations with users.

Personally, I was also pretty happy. Although I had had some negative interactions with the CTO, I really liked all of my other coworkers, the mission of the company, and the technologies that I got to use day-to-day. I felt like I was learning a lot as an engineer, and I was hopeful that the company would continue to expand and provide growth opportunities for me in the future.

Initial headwinds

In the summer of 2023, the company hit some headwinds. First, although we were still adding customers, our revenue growth rate had noticeably slowed down. From what I understand, this was a trend across the entire SaaS tooling space that wasn’t unique to Airplane, but it still hurt morale and made us less optimistic about the company’s near-term future.

In addition, people on the team started quitting. Up until that summer, we had had zero employee attrition. Then, within a three month period, we lost 4 engineers and the company’s head of growth. These colleagues had all been at the company for a while and all had legitimate reasons for resigning unrelated to the revenue slowdown (several, for instance, wanted to start their own companies), but the latter didn’t soften the blow. They were all well-respected and great at their jobs, so we were sad to see them go.

To top it all off, the CEO (who was also a co-founder) announced that he was leaving so that he could work on a personal, AI-related passion project. The CTO (the other co-founder) would be taking his place. His departure wasn’t a huge surprise- he had been very clearly burnt out and disengaged for months. But, it’s still never a good sign to lose the leader of your organization.

My colleagues and I were obviously upset by these developments, but we figured that the new CEO knew what he was doing and would successfully navigate us through.

Stabilization

By the fall of 2023, things had stabilized a bit. The employee attrition wave died down, and we added several new engineers and a new growth lead to the team. We also had some substantial product launches and signed two big, new enterprise customer deals. Our Q3 revenue numbers were strong, and we had new customers excited to sign with us in Q4.

In early November, we did a company retreat in Napa Valley to hang out together in-person and hack on various new product features. The CEO was a little less engaged than usual, but otherwise morale was high and there were no signs that anything was wrong.

Brewing trouble

After we got back from the retreat, we got an abrupt message from the CEO in our team’s Slack workspace. Effective immediately, we were winding down our hiring pipeline. We were also rescinding two engineering offers that we had just extended.

My colleagues and I knew that this was a bad sign. Rescinding offers is something that causes a lot of reputational harm and isn’t done casually. Companies generally only do this if they’re about to do layoffs or go through some sort of big transformation like an acquisition. The former didn’t make any sense, though, because we had lots of money in the bank (enough to last for many years at our current burn rate), had already slimmed the team down via voluntary attrition, and had an overflowing backlog of new features to work on. That left an acquisition as the most likely explanation.

Several people on the team confronted the CEO, but he denied that there was an acquisition brewing. Instead, he made vague statements about “figuring out the direction of the company” and potentially doing some sort of product pivot to reignite growth. He claimed that any update would be good news and that he’d make sure we were taken care of.

The announcement

On the morning of December 4, around 3 weeks after the new hire offers were rescinded, the CEO scheduled an all-hands for 1PM that day. He also sent out a Slack message that this was a very important meeting and that we should cancel other things if needed to be there. We knew that we would finally be getting answers about what was going on.

The CEO, who was presenting from his home in SF, started with his usual statement of Airplane’s mission, “We’re here because we believe that software is a force multiplier…”, but ended with the clause that we’d be continuing that mission as part of Airtable.

He then explained that the Airplane product would be shut down, but that most of us would be getting “extremely strong” offers from Airtable. However, we’d have to do interviews for leveling purposes, and the financial details wouldn’t be made available to us until later. This was a great outcome for us, he said, much better than the alternatives he had considered, so we should be happy. The goal was to wrap everything up before the holidays.

After the Zoom was turned off, our room in the NYC office was silent. While we were optimistic at the beginning when we heard about Airtable, the mood had clearly soured as we got more information. Basically, our product was being shut down, we’d have to interview for our new jobs, it was unclear what we’d be doing at the new company, and our stock was potentially worthless. It was also unclear why, exactly, we were doing this given that Airplane had many happy customers and tens of millions of dollars in the bank.

For the remainder of the day, we met 1:1 with the CEO to talk more about our personal situations. For me and the other engineers, the CEO told us that he enjoyed working with us and hoped that we’d continue at Airtable together. Our common stock was worth $0, but we’d be getting extra cash bonuses from Airtable to compensate. The exact details of our levels and roles were not known and would depend on the interviews to be done later in the week.

Based on these discussions, we put the pieces together and realized that this was an acqui-hire. Airtable wasn’t interested in our product, our technology, or our customers. They wanted our CEO to lead their new AI effort, and the rest of the company was baggage that would be accommodated to the minimum degree required. It was depressing for all of us.

Figuring out next steps

From this point forward, all regular work stopped. We continued supporting our existing customers (who didn’t yet know anything was happening), but there would be no more feature development and all conversations with prospective customers were cut off.

For the next week, we sat in the office doing interview prep and chatting with each other about our options.

I was personally not very excited about Airtable. I had interviewed there and gotten an offer back in 2019 but had decided at the time that it wasn’t a fit for me. Since then, the company had expanded greatly but then contracted abruptly via two large layoffs. Airtable’s headcount was about half of what it had been a year before, which seemed kind of depressing to me.

The interviews took place a week later and, thankfully, were fairly low stress. We were mostly asked about our previous experiences and projects. In my case, it was clear they were probing how high they would move me in the IC chain. My interviewers were interested in hearing about how I managed cross-company projects at the large companies on my resume as opposed to any of my technical work at Airplane.

Later that week, we got our Airtable offers. They were strong but definitely not as out-of-this-world as the CEO had promised. In addition to a base salary and equity, we were each offered a $50-75k signing bonus that would have to be paid back in full if we quit in the first year. We were given three business days to make a decision, and there would be no severance if we declined.

I said no, without much hesitation. As mentioned previously, I wasn’t excited about Airtable to begin with, and the whole acquisition process had left a sour taste in my mouth. In addition to shutting down the product and abandoning our customers, Airtable had never given us a product demo, detailed financial information about the company, or precise details on what we’d be working on aside from “AI-related features”. The whole process seemed like a rushed, disorganized mess, and I knew that there were better things out there.

Several of my colleagues came to a similar conclusion and also declined.

Winding down

On January 3, the acquisition and product shutdown was publicly announced. As expected, customers were shocked and upset. Many had been using Airplane for critical workflows within their organizations, and they now had to replace huge chunks of internal tooling before the final shutoff on March 1. The CEO tried to steer people to alternatives like windmill.dev and Retool, but neither of these is a drop-in replacement for Airplane. Even if users found replacement tools, it would take a lot of time and effort for them to do the migration.

Two days later was our last day of payroll. Our email and Slack access was shut off, and we were no longer Airplane employees. We were never given any termination paperwork to sign, and there were no formal goodbyes. The company had disintegrated with a sad whisper.

Why?

We were never told directly why the company was acquired under such unfavorable terms. Contrary to what one would logically think in these circumstances, we didn’t run out of money or even come close to that. In fact, we had tens of millions of dollars in the bank (allowing for years of runway) in addition to a great team, relatively happy customers, low churn, and solid revenue growth.

When pressed in 1:1 conversations, the CEO said that he didn’t see a path to significantly higher revenue. He claimed we would have to do a significant product pivot and/or change our sales strategy to grow beyond a particular point, and that this transformation would be risky and hard. It was easier to quit while we were ahead, shut everything down, and work on something new.

To me and others, it seemed like the CEO was tired of the startup grind and simply giving up. Ever since his fellow co-founder quit, he’d been running both the technical and go-to-market sides of the company, and it was clearly very draining. We felt that he wanted a less stressful position with a lower risk payday, and this kind of deal was the best way to achieve that.

Final thoughts

Overall, I had a good experience at Airplane and up until the last month I was pretty happy. In looking back, the thing that upsets me the most is not the loss of money (I never put much value on my stock options anyways) but rather the multitude of customer relationships that were flushed down the drain by abandoning the product. Many of these customers had personally stuck their necks out within their organizations to drive adoption of Airplane, and had spent hundreds of hours migrating critical, internal workflows to our product. Now, they were being thrown under the bus.

Were there any alternatives to what the CEO did? I obviously don’t know all the details of the offers that the CEO considered, but I feel like he could have either: (1) held out for an acquisition offer that involved keeping the product running, (2) separately sold the Airplane technology to another entity that would continue developing it, or (3) open-sourced the product. In the end, he unilaterally chose an outcome that I believe was not the best for our product or our customers, and it was sad that we all had to go along with it.

Building Airplane was a great experience, and I hope to work at a small startup again at some point in my career. For now, though, I’m going to decompress at a bigger company.

Radiosonde Hunting - Success!

2024-03-03T00:00:00Z

Leverage

2024-02-26T00:00:00Z

I composed, produced, and released an instrumental jazz track! https://www.youtube.com/watch?v=mOxsT5-jnc8 You can also find the track on Spotify, Apple Music, YouTube Music, and other music streaming…

Radiosonde Hunting - A Post-Mortem

2024-02-24T00:00:00Z

River: a Fast, Robust Job Queue for Go + Postgres

2023-11-20T14:18:48Z

Years ago I wrote about my trouble with a job queue in Postgres, in which table bloat caused by long-running queries slowed down the workers’ capacity to lock jobs as they hunted across millions of dead tuples trying to find a live one.

A job queue in a database can have sharp edges, but I’d understated in that writeup the benefits that came with it. When used well, transactions and background jobs are a match made in heaven and completely sidestep a whole host of distributed systems problems that otherwise don’t have easy remediations.

Consider:

In a transaction, a job is emitted to a Redis-based queue and picked up for work, but the transaction that emitted it isn’t yet committed, so none of the data it needs is available. The job fails and will need to be retried later.

A job is emitted from a transaction which then rolls back. The job fails and will also fail every subsequent retry, pointlessly eating resources despite never being able to succeed, eventually landing the dead letter queue.

In an attempt to work around the data visibility problem, a job is emitted to Redis after the transaction commits. But there’s a brief moment between the commit and job emit where if the process crashes or there’s a bug, the job is gone, requiring manual intervention to resolve (if it’s even noticed).

If both queue and store are non-transactional, all of the above and more. Instead of data not being visible, it may be that it’s in a partially ready state. If a job runs in the interim, all bets are off.

Work in a transaction has other benefits too. Postgres’ NOTIFY respects transactions, so the moment a job is ready to work a job queue can wake a worker to work it, bringing the mean delay before work happens down to the sub-millisecond level.

Despite our operational trouble, we never did replace our database job queue at Heroku. The price of switching would’ve been high, and despite blemishes, the benefits still outweighed the costs. I then spent the next six years staring into a maelstrom of pure chaos as I worked on a non-transactional data store. No standard for data consistency was too low. Code was a morass of conditional statements to protect against a million possible (and probable) edges where actual state didn’t line up with expected state. Job queues “worked” by brute force, bludgeoning jobs through until they could reach a point that could be tacitly called “successful”.

I also picked up a Go habit to the point where it’s now been my language of choice for years now. Working with it professionally during that time, there’s been more than a few moments where I wished I had a good framework for transactional background jobs, but didn’t find any that I particularly loved to use.

River is born

So a few months ago, Blake and I did what one should generally never do, and started writing a new job queue project built specifically around Postgres, Go, and our favorite Go driver, pgx. And finally, after long discussions and much consternation around API shapes and implementation approaches, it’s ready for beta use.

I’d like to introduce River (GitHub link), a job queue for building fast, airtight applications.

Designed for generics

One of the relatively new features in Go (since 1.18) that we really wanted to take full advantage of was the use of generics. A river worker takes a river.Job[JobArgs] parameter that provides strongly typed access to the arguments within:

type SortWorker struct {
    river.WorkerDefaults[SortArgs]
}

func (w *SortWorker) Work(ctx context.Context, job *river.Job[SortArgs]) error {
    sort.Strings(job.Args.Strings)
    fmt.Printf("Sorted strings: %+v\n", job.Args.Strings)
    return nil
}

No raw JSON blobs. No json.Unmarshal boilerplate in every job. No type conversions. 100% reflect-free.

Jobs are raw Go structs with no embeds, magic, or shenanigans. Only a Kind implementation that provides a unique, stable string to identify the job as it round trips to and from the database:

type SortArgs struct {
    // Strings is a slice of strings to sort.
    Strings []string `json:"strings"`
}

func (SortArgs) Kind() string { return "sort" }

Beyond the basics, River supports batch insertion, error and panic handlers, periodic jobs, subscription hooks for telemetry, unique jobs, and a host of other features.

Job queues are never really done, but we’re pretty proud of the API design and initial feature set. Check out the project’s README and getting started guide.

With performance in mind

One of the reasons we like to write things in Go is that it’s fast. We wanted River to be a good citizen of the ecosystem and designed it to use fast techniques where we could:

It takes advantage of pgx’s implementation of Postgres’ binary protocol, avoiding a lot marshaling to and parsing from strings.
It minimizes round trips to the database, performing batch selects and updates to amalgamate work.
Operations like bulk job insertions make use of COPY FROM for efficiency.

We haven’t even begun to optimize it so I won’t be showing any benchmarks (which tend to be misleading anyway), but on my commodity MacBook Air it works ~10k trivial jobs a second. It’s not slow.

What's different now?

You might be thinking: Brandur, you’ve had trouble with job queues in databases before. Now you’re promoting one. Why?

A few reasons. The first is, as described above, transactions are really just a really good idea. Maybe the best idea in robust service design. For the last few years I’ve been putting my money where my mouth is and building a service modeled entirely around transactions and strong data constraints. Data inconsistencies are still possible, but especially in a relative sense, they functionally don’t exist. The amount of time this saves operators from having to manually mess around in consoles fixing things cannot be overstated. It’s the difference between night and day.

Single dependency stacks

Another reason is that dependency minimization is great. I’ve written previously about how at work we run a single dependency stack. No ElastiCache, no Redis, no bespoke queueing components, just Postgres. If there’s a problem with Postgres, we can fix it. No need to develop expertise in how to operate rarely used, black box systems.

This idea isn’t unique. An interesting development in Ruby on Rails 7.1 is the addition of Solid Cache, which 37 Signals uses to cache in the same database that they use for the rest of their data (same database, but different instances of it of course). Ten years ago this would’ve made little sense because you’d want a hot cache that’d serve content from memory only, but advancements in disks (SSDs) has been so great that they measured a real world difference in the double digits (25-50%) moving their cache from Redis to MySQL, but with a huge increase in cache hits because a disk-based system allows cache space to widen expansively.

Ruby non-parallelism

A big part of our queue problem at Heroku was the design of the specific job system we were using, and Ruby deployment. Because Ruby doesn’t support real parallelism, it’s commonly deployed with a process forking model to maximize performance, and this was the case for us. Every worker was its own Ruby process operating independently.

This produced a lot of contention and unnecessary work. Running independently, every worker was separately competing to lock every new job. So for every new job to work, every worker contended with every other worker and iterated millions of dead job rows every time. That’s a lot of inefficiency.

A River cluster may run with many processes, but there’s orders of magnitude more parallel capacity within each as individual jobs are run on goroutines. A producer inside each process consolidates work and locks jobs for all its internal executors, saving a lot of grief. Separate Go processes may still contend with each other, but many fewer of them are needed thanks to superior intra-process concurrency.

Improvements in Postgres

During my last queue problems we would’ve been using Postgres 9.4. We have the benefits of nine new major versions since then, which have brought a lot of optimizations around performance and indexes.

The most important for a queue was the addition of SKIP LOCKED in 9.5, which lets transactions find rows to lock with less effort by skipping rows that are already locked. This feature is old (although no less useful) now, but we didn’t have it at the time.
Postgres 12 brought in REINDEX CONCURRENTLY, allowing queue indexes to be rebuilt periodically to remove detritus and bloat.
Postgres 13 added B-tree deduplication, letting indexes with low cardinality (of which a job queue has multiple of) be stored much more efficiently.
Postgres 14 brought in an optimization to skip B-tree splits by removing expired entries as new ones are added. Very helpful for indexes with a lot of churn like a job queue’s.

And I’m sure there’s many I’ve forgotten. Every new Postgres release brings dozens of small improvements and optimizations, and they add up.

Also exciting is the potential addition of a transaction timeout setting. Postgres has timeouts for individual statements and being idle in a transaction, but not for the total duration of a transaction. Like with many OLTP operations, long-lived transactions are hazardous for job queues, and it’ll be a big improvement to be able to put an upper bound them.

Try it

Anyway, check out River (see also the GitHub repo and docs) and we’d appreciate it if you helped kick the tires a bit. We prioritized getting the API as polished as we could (we’re really trying to avoid a /v2), but are still doing a lot of active development as we refactor internals, optimize, and generally nicen things up.

When Two Bugs Cancel Out

2023-09-18T00:00:00Z

I encountered a perplexing bug while implementing table cell splitting in Google Docs. The bug report 🐛 The QA team found a bug where some of a table's borders would be missing after a table cell…

Real Time Video From a High Altitude Balloon

2023-09-10T00:00:00Z

On Using Go's `t.Parallel()`

2023-08-26T20:48:45Z

One of Go’s best features is not only that it does parallelism well, but that it’s deeply baked in. It’s best exemplified by primitives like goroutines and their dead simple ease of use, but extends all the way up the chain to the built-in tooling. When running tests for many packages with go test ./..., packages automatically run in parallel up to a maximum equal to the number of CPUs on the machine. Between that and the language’s famously fast compilation, test suites are fast by default instead of something that needs to be painstakingly optimized later on.

Within any specific package, tests run sequentially, and as long as packages aren’t too mismatched in test suite size, that’s generally good enough.

But having uniformly sized package test suites isn’t always a given, and some packages can grow to be quite large. We have a ./server/api package that contains the majority of our product’s API and ~200 tests to exercise it, and it’s measurably slower than most packages in the project.

For cases like this, Go has another useful parallel facility: t.Parallel(), which lets specific tests within a package be flagged to run in parallel with each other. When applied to our large package, it reduced the time needed for a single run by 30-40% or by 2-3x for ten consecutive runs.

Before t.Parallel():

$ go test ./server/api -count=1
ok      github.com/crunchydata/priv-all-platform/server/api     1.486s
$ go test ./server/api -count=10
ok      github.com/crunchydata/priv-all-platform/server/api     11.786s

After t.Parallel():

$ go test ./server/api -count=1
ok      github.com/crunchydata/priv-all-platform/server/api     0.966s
$ go test ./server/api -count=10
ok      github.com/crunchydata/priv-all-platform/server/api     3.959s

These tests were already pretty fast (to beat a dead horse again: running every API test for this project is 3-5x+ faster than it took to run a single test case during my time at Stripe; language choice and infrastructure design makes a big difference), but this is one of the packages that we run tests on most frequently, so a 30-40% speed up makes a noticeable difference in DX when iterating.

After adding t.Parallel() to this one package, we then went through and added it to every test in every package, and then put in a ratchet with the paralleltest linter to mandate it for future additions.

Should you bother adding t.Parallel() like we did? Maybe. It’s a pretty easy standard to adhere to when starting from scratch, and for existing ones it’ll be easier to add it today than at any point later on, so it’s worth considering.

Is `t.Parallel()` broadly recommended practice?

As far as I can tell, no.

I like to use the Go language’s own source code to glean convention, and by my rough measurement only about 1/10th of its test suite uses t.Parallel():

# total number of tests
$ ag --no-filename --nobreak 'func Test' | wc -l
    7786
    
# total number of uses of `t.Parallel()`
$ ag --no-filename --nobreak 't\.Parallel\(\)' | wc -l
     620

This isn’t too surprising. As discussed above, parallelism across packages is usually good enough, and when iterating tests in one specific package, Go’s already pretty fast. For smaller packages adding parallelism is probably a wash, and for very small ones the extra overhead probably makes them slower (although trivially so).

Still, it might not be a bad idea. As some packages grow to be large, parallel testing will keep them fast, and annotating tests with t.Parallel() from the beginning is a lot easier than going back to add it to every test case and fix parallelism problems later on.

Sharp edges

Sharing a database with test transactions

The biggest difficulty for many projects will be to have a strategy for the test database that can support parallelism. It’s easy to build a system where multiple tests target the same test database and insert data that conflicts with each other.

We use test transactions to avoid this. Each test opens a transaction, runs everything inside it, and rolls the transaction back as it finishes up. A simplified test helper looks like:

func TestTx(ctx context.Context, t *testing.T) pgx.Tx {
    tx, err := getPool().Begin(ctx)
    require.NoError(t, err)

    t.Cleanup(func() {
        err := tx.Rollback(ctx)
        if !errors.Is(err, pgx.ErrTxClosed) {
            require.NoError(t, err)
        }
    })

    return tx
}

Invocations of the helper share a package-level pgx pool that’s automatically parallel-safe (but still has a mutex to make sure that only one test case initializes it):

var (
    dbPool   *pgxpool.Pool
    dbPoolMu sync.RWMutex
)

Usage is succinct and idiot-proof thanks to Go’s test Cleanup hook:

tx := TestTx(ctx, t)

Deadlocks across transactions

The trickiest problem I had to fix while enabling t.Parallel() involved Postgres upsert. We have a number of places where we seed data with an upsert to guarantee that it’s always in the database regardless of whether the program has run before or is starting for the first time. In the test suite, individual test cases would upsert a “known” resource:

plan := dbfactory.Plan_AWS_Hobby2(ctx, t, tx)

Implemented as:

func Plan(ctx context.Context, t *testing.T, e db.Executor, opts *PlanOpts) *dbsqlc.Plan {
    validateOpts(t, opts)

    configPlan := providers.Default.MustGet(opts.ProviderID).MustGetPlan(opts.PlanID, true)

    plan, err := dbsqlc.New(e).PlanUpsert(ctx, dbsqlc.PlanUpsertParams{
        CPU:         int32(configPlan.CPU),
        Disabled:    configPlan.Disabled,
        DisplayName: configPlan.DisplayName,
        Instance:    configPlan.Instance,
        Memory:      configPlan.Memory,
        ProviderID:  opts.ProviderID,
        PlanID:      configPlan.ID,
        Rate:        int32(configPlan.Rate),
    })
    require.NoError(t, err)
    return &plan
}

To my surprise, adding t.Parallel() would fail many tests at these invocations. Despite every test case running in its own transaction, it’s still possible for them to deadlock against other as they tried to upsert exactly the same data.

We resolved the problem by moving to a fixture seeding model, so when the test database is being created, in addition to loading a schema and running migrations, we also load a common set of test data in it that all tests will share (test transactions ensure that any changes to it are rolled back):

.PHONY: db/test
db/test:
    psql --echo-errors --quiet -c '\timing off' -c "DROP DATABASE IF EXISTS platform_main_test WITH (FORCE);"
    psql --echo-errors --quiet -c '\timing off' -c "CREATE DATABASE platform_main_test;"
    psql --echo-errors --quiet -c '\timing off' -f sql/main_schema.sql
    go run ./apps/pmigrate
    go run ./tools/src/seed-test-database/main.go

So the implementation becomes a lookup instead:

func Plan(ctx context.Context, t *testing.T, e db.Executor, opts *PlanOpts) *dbsqlc.Plan {
    validateOpts(t, opts)

    _ = providers.Default.MustGet(opts.ProviderID).MustGetPlan(opts.PlanID, true)

    // Requires test data is seeded.
    provider, err := dbsqlc.New(e).PlanGetByID(ctx, dbsqlc.PlanGetByIDParams{
        PlanID:     opts.PlanID,
        ProviderID: opts.ProviderID,
    })
    require.NoError(t, err)

    return &provider
}

Logging and `t.Log`

We make fairly extensive use of logging, and previously we’d just log to everything in tests to stdout. This is fine because Go automatically suppresses output to stdout without an additional -test.v verbose flag, and because tests ran sequentially, even when testing verbosely the output looked fine, with logs for each test case correctly appearing within their begin/end banners.

But with t.Parallel(), everything became mixed together into a big log soup:

=== RUN   TestClusterCreateRequest/StorageTooSmall
--- PASS: TestClusterCreateRequest (0.00s)
    --- PASS: TestClusterCreateRequest/StorageTooSmall (0.00s)
=== CONT  TestMultiFactorServiceList
=== RUN   TestMultiFactorServiceList/Success
=== RUN   TestMultiFactorServiceUpdate/SuccessWebAuthn
time="2023-08-20T22:26:28Z" level=info msg="password_hash_line: Match result: success [account: eee5c815-b7c6-4f19-8e1d-92428eed32ab] [hash time: 0.000496s]" account_id=eee5c815-b7c6-4f19-8e1d-92428eed32ab hash_duration=0.000496s hash_match=true
=== RUN   TestClusterServiceDelete/Owl410Gone
=== RUN   TestMultiFactorServiceList/Pagination
time="2023-08-20T22:26:28Z" level=info msg="sessionService: password_hash_upgrade_line: Upgraded password from \"argon2id\" to \"argon2id\" [account: eee5c815-b7c6-4f19-8e1d-92428eed32ab] [hash time: 0.000435s]" account_id=eee5c815-b7c6-4f19-8e1d-92428eed32ab new_algorithm=argon2id new_argon2id_memory=1024 new_argon2id_parallelism=4 new_argon2id_time=1 new_hash_duration=0.000435s old_algorithm=argon2id old_hash_iterations=0
=== RUN   TestClusterUpgradeServiceCreate/HobbyMaximum100GB
=== RUN   TestClusterServiceCreate/WithPostgresVersionID
=== RUN   TestMultiFactorServiceUpdate/WrongAccountNotFoundError
=== RUN   TestClusterServiceForkCreate/WithTargetTime
--- PASS: TestMultiFactorServiceList (0.01s)
    --- PASS: TestMultiFactorServiceList/Success (0.00s)
    --- PASS: TestMultiFactorServiceList/Pagination (0.00s)
=== CONT  TestClusterServiceActionTailscaleDisconnect
=== RUN   TestClusterServiceActionTailscaleDisconnect/Success
time="2023-08-20T22:26:28Z" level=info msg="password_hash_line: Match result: success [account: eee5c815-b7c6-4f19-8e1d-92428eed32ab] [hash time: 0.000828s]" account_id=eee5c815-b7c6-4f19-8e1d-92428eed32ab hash_duration=0.000828s hash_match=true

This isn’t usually a problem because you’re not reading the logs anyway, but quickly becomes one if you get a test failure, and only have senseless noise around it to help you debug.

The fix for this is t.Logf, which makes sure to collate log output for to the particular test case that emitted it. This will generally require a shim to use with a logging library like:

// tlogWriter is an adapter between Logrus and Go's testing package,
// which lets us send all output to `t.Log` so that it's correctly
// collated with the test that emitted it. This helps especially when
// using parallel testing where output would otherwise be interleaved
// and make debugging extremely difficult.
type tlogWriter struct {
    tb testing.TB
}

func (lw *tlogWriter) Write(p []byte) (n int, err error) {
    // Unfortunately, even with this call to `t.Helper()` there's no
    // way to correctly attribute the log location to where it's
    // actually emitted in our code (everything shows up under
    // `entry.go`). A good explanation of this problem and possible
    // future solutions here:
    //
    // https://github.com/neilotoole/slogt#deficiency
    lw.tb.Helper()

    lw.tb.Logf((string)(p))
    return len(p), nil
}

Then with Logrus for example:

func Logger(tb testing.TB) *logrus.Entry {
    logger := logrus.New()
    logger.SetOutput(&tlogWriter{tb})
    return logrus.NewEntry(logger)
}

Now when a test fails, any logs it produced are grouped correctly:

--- FAIL: TestSessionServiceCreate (0.05s)
    --- FAIL: TestSessionServiceCreate/PasswordHashAlgorithmUpgrade (0.05s)
        entry.go:294: time="2023-08-20T22:34:15Z" level=info msg="password_hash_line: Match result: success [account: 81b967f7-4f5c-4ab4-b1d7-3c455db35767] [hash time: 0.000694s]" account_id=81b967f7-4f5c-4ab4-b1d7-3c455db35767 hash_duration=0.000694s hash_match=true
        entry.go:294: time="2023-08-20T22:34:15Z" level=info msg="sessionService: password_hash_upgrade_line: Upgraded password from \"argon2id\" to \"argon2id\" [account: 81b967f7-4f5c-4ab4-b1d7-3c455db35767] [hash time: 0.011716s]" account_id=81b967f7-4f5c-4ab4-b1d7-3c455db35767 new_algorithm=argon2id new_argon2id_memory=19456 new_argon2id_parallelism=4 new_argon2id_time=2 new_hash_duration=0.011716s old_algorithm=argon2id old_hash_iterations=0
        session_service_test.go:197:
                Error Trace:    /Users/brandur/Documents/crunchy/platform/server/api/session_service_test.go:197
                                                        /Users/brandur/Documents/crunchy/platform/server/api/session_service_test.go:158
                Error:          artificial failure
                Test:           TestSessionServiceCreate/PasswordHashAlgorithmUpgrade

Bridges for common loggers like slog are usually available as public packages. Slogt, for example.

goleak

Our tests use goleak to detect any accidentally leaked goroutines, a practice that I’d recommend since leaking goroutines without realizing it is easily one of Go’s top footguns.

Previously, we had a pattern in which every test case would check itself for goroutine leaks, but adding t.Parallel() broke the pattern because test cases running in parallel would detect each other’s goroutines as leaks.

The fix was to use goleak’s built-in TestMain wrapper:

func TestMain(m *testing.M) {
    goleak.VerifyTestMain(m)
}

Leaked goroutines are only detected at package-level granularity, but as long as you’re starting off from a baseline of no leaks, that’s good enough to detect regressions.

Other notes

Requiring `t.Parallel()` in tests, but not subtests

By default the paralleltest lint will not only require that every test case define t.Parallel(), but that every subtest (i.e. t.Run("Subtest", func(t *testing.T) { ... })) define it as well. This is generally the right thing to do because it means that parallelism has better granularity and therefore more likely to produce more optimal throughput and lower the total runtime.

Due to a historical tech decision made long ago, we were ubiquitously using a testing convention within test cases where we had plenty of subtests, but subtests were not parallel safe because they were all sharing a single var block.

Refactoring to total parallel-safety would’ve taken dozens of hours and wasn’t a good use of time, so we declared t.Parallel() at the granularity of test cases but not subtests to be “good enough”. I added an ignoremissingsubtests option to paralleltest to support that, and if your set up is anything like ours, maybe that’ll help you:

linters-settings:
  paralleltest:
    # Ignore missing calls to `t.Parallel()` in subtests. Top-level
    # tests are still required to have `t.Parallel`, but subtests are
    # allowed to skip it.
    #
    # Default: false
    ignore-missing-subtests: true

Takeaways

As noted above, it’s not exactly Go convention to make ubiquitous use of t.Parallel(). That said, it’s reduced our test iteration time for large packages by 30-40%, and that’s enough of a development win that I personally intend to use it for future Go projects.

And although increased test speed is its main benefit, when combined with go test . -race it’s actually managed to help suss out some tricky parallel safety bugs that weren’t being caught with sequential-only test runs. That’s a big advantage because that whole class of bug is very difficult to debug in production.

Activating t.Parallel() everywhere for an existing project could be a big deal, but integrating it from the beginning has very little ongoing cost, and might yield substantials benefits later on.

Driving Poles Into the Ground

2023-07-23T00:00:00Z

The Making of Keyalesce

2023-07-02T00:00:00Z

Have you ever wanted to use tuples or objects for the keys of a Map or the values of a Set? It's a very common question because the following code doesn't do what you might expect: The code behaves…

Radio Mischief in the Montreal Metro

2023-06-21T00:00:00Z

Goodbye, San Francisco

2023-06-18T20:30:00Z

After nearly 10 years of living in San Francisco, I decided to leave at the end of last year. In this post, I explain what motivated my departure and how the city can improve in the future.

View of my former neighborhood from my apartment.

Problems

For the majority of my time in SF, I was pretty happy with life there. I had a great apartment in a great location. Although I had a car, I rarely used it because I could easily walk to the train, the city’s main shopping district (Union Square), and the offices of my various employers over the years.

Starting 4-5 years ago, however, things started changing for the worse, and then they accelerated downwards during the pandemic. It’s hard to pin down exactly what happened and when, but I think the core problem for me was that my sense of safety significantly degraded.

Physical safety

My biggest problem with SF was simply not feeling physically safe while walking down the street or sitting in a train. SF has a pretty low violent crime rate, at least by US standards, so I wasn’t worried about getting shot or robbed or anything like that.

Instead, the problem was that SF had (and still has today), thousands of people wandering around who are suffering from untreated substance abuse and/or severe mental illness. The vast majority of these folks were completely harmless, but a small percentage were hostile, threatening, and, in some cases, violent.

In multiple cases over the last few years, I was followed, screamed at, and threatened in broad daylight. Thankfully, nothing physically happened to me, but it’s a really jarring experience to have this happen. Each time I reported these incidents to the police, they never responded. I got the sense that as long as no one was physically harmed, they didn’t really care. Thus, there were no mechanisms in place to control, contain, or treat this behavior.

Over time, I realized that you needed to be hyper-vigilant every time you left the house, constantly evaluating everyone around you and being ready to cross the street or reverse direction at a moment’s notice. Is that person screaming and throwing trash at the side of a bus shelter a danger, or are they going to keep to themselves if I walk quickly past? What about that group that’s selling drugs in the middle of the sidewalk?

It just got really stressful and draining dealing with this after a while. And, the threats seemed to get worse over time.

Drugs

Closely related to the issue of physical safety is drugs. When I moved to my neighborhood (Mission Street between 7th and 8th), it wasn’t fancy but at least it was pretty clean and quiet. Then, a few years ago, gangs of drug dealers moved in. They first took over the 7th street corner and then, during the pandemic, also expanded to the 8th street side of my block. By the end, I couldn’t leave the house without walking through them.

Most of the time, this activity was peaceful. However, it brought more people to the neighborhood who were loud, destructive, and potentially threatening. The sidewalks became blocked in places, and I often had to walk in the street to get around the dealers, their customers, and the piles of trash they left behind.

Occasionally, violence did flare up. There were two, drug-related murders on my block in the last year, one of which I heard the gunshots for. And, people were constantly overdosing, and often dying. In my final month, I walked by a dead body on the sidewalk being attended to by the coroner’s office.

Traffic safety

The final safety dimension that affected me was my interactions with drivers in the street. As I mentioned above, I had a car but tried to walk or take transit whenever possible. I also biked a lot, particularly if I was going to places that were more than a few miles away.

San Francisco was never the most pedestrian or bike friendly city in the world, but at least you could be reasonably confident that you wouldn’t be hit by a car when going outside. Then, a few years ago, the city made a conscious decision to stop nearly all traffic enforcement ¹. As a result, the behavior of some drivers got terrible, to the point of being dangerous. I would routinely see people running red lights, going the wrong way down one way streets, driving on roads closed to private vehicles, and speeding at 60+ in 25mph zones.

In my last year, after nearly two decades of incident-free biking, a driver making an illegal right turn hit me while I was in a bike lane on Market Street. Thankfully, I escaped with only scrapes and bruises, but that was really upsetting for me, and I felt that it was only a matter of time before something worse happened.

Why it’s frustrating

Many cities in the US and in other places around the world have problems with crime, drugs, safety, homelessness, and other issues. But, there are a couple aspects of San Francisco’s situation that make its problems especially frustrating.

First, the city is incredibly wealthy and has a massive budget at its disposal- nearly $14 billion dollars ² for a population of only slightly more than 800,000 people, which, on a per-capita basis, is one of the highest in the country ³. Despite all this money and despite a budget that has grown much faster than the city’s population over the last few years, SF can’t keep its central areas safe and clean, and doesn’t seem to be providing much help to the thousands of people suffering on its streets.

Second, and even worse, the people in charge including the mayor, the police leadership, and a majority of the Board of Supervisors (the legislative body for SF), really just don’t seem to care one bit about what’s happening. Sure, they will occasionally rant in public and promise to fix things ⁴, but after a quick surge of activity, the streets just go back to their previous state or get worse. I got the sense that they just enjoyed the prestige and money of their jobs (SF officials are among the highest paid in the country ⁵), and were coasting until they could find something longer-term.

Why it matters

In the pre-pandemic days, most companies required employees to be in the office at least 4 days a week. Many Bay Area companies, including most of the big tech employers (Google, Facebook, etc.), had large offices in downtown San Francisco, and as a result, many of their employees lived in the city to avoid long commutes.

With the pandemic, much of this disappeared. Companies that previously had office-centric cultures started allowing fully remote work or at least allowed people to come in less frequently than before.

The result is that many tech employees, myself included, are no longer tied to locations within easy commuting distance of an SF office. They can live further away (e.g., in the suburbs) or just totally leave the area, as I did. There is less foot traffic in the central areas of the city, which has hurt retail businesses, who are now shutting down or moving away as well.

People now have the power to vote with their feet, and many have taken advantage of and will continue to take advantage of this power as conditions deteriorate. The city can no longer rest on its laurels and spend from an infinite pile of money as it did in the good old days, it now has to control spending and actively fight to retain businesses and residents. Unfortunately, it hasn’t figured out how to do that yet.

Moving away

After 10 years, I decided that I had had enough of feeling unsafe, of seeing people suffering and dying in the streets, of watching businesses in my neighborhood reduce their hours or shut down completely, and of seeing no significant response from the city leadership. I packed up my stuff, cancelled my lease, and moved back to the East Coast.

I have a lot of great memories of my time in SF, and I feel really sad about what’s happened to the city. Will it ever recover? Maybe with completely new leadership and drastic policy changes it can. But, until that happens, I’m not holding my breath.

FAQ

Why didn’t you just move to a different part of SF?

I wanted to live in a dense, central area and not in a single family house that’s 4 miles from downtown. Also, my neighborhood was perfectly nice when I moved in, and then got bad. What’s to prevent that from happening to other places in the city as well?

What about homelessness? You didn’t discuss it at all above despite the attention that issue has gotten.

Yes, homelessness is a very serious problem in San Francisco. However, it wasn’t super prevalent in my immediate neighborhood. I also felt that the safety and drug issues were more critical, and that homelessness was more of a symptom than a cause of the city’s problems.

What about property crime?

Property crime in San Francisco is terrible, but it didn’t affect me personally because I lived in a building with multiple security people on-site 24/7. I also kept my car in a secure garage and not on the street. If you’re moving to SF or visiting, I would suggest you do the same.

The sad thing about the city is that many people, particularly those with lower incomes, don’t have this luxury, and it’s a big financial burden when their possessions are stolen from their cars or homes. The city leadership, however, doesn’t really care because burglaries and car break-ins are considered “victimless” crimes.

I drove near (insert some place mentioned above) and it was perfectly fine. Are you exaggerating about how bad it is?

No, I’m not. If anything, I’m understating how terrible things were in my neighborhood and others nearby.

One quirky thing to note about SF is that there can be significant variation in street conditions by both location and time. One block could be perfectly fine, and a block one street over could be a disaster zone. Or, a corner could be quiet now but then get taken over by drug dealing next week.

The city (when it does anything) typically just shuffles people around, does a quick cleanup, and then forgets about the area for a while. The conditions have little to do with the buildings, and more to do with whatever the city is allowing or not allowing on the sidewalks at any particular time and place.

The stats show that SF is doing well. Assaults, burglaries (substitute some other metric here) are down!

First of all, I care more about what I see with my eyes than any statistics. Secondly, when the police take hours to respond to even serious crimes like commercial burglaries ⁶ or shootings ⁷, the incentive to report things goes way down.

Other cities in the US are just as bad. Why are you picking on SF?

It’s simply false that SF is “just as bad as everywhere else”. I’ve spent a good amount of time over the last year visiting many other places in the US, including a bunch that the media love to hate on like Seattle, Baltimore, and New York, and I felt significantly safer in all of those places. Also, they don’t have giant, open-air drug markets in their city centers.

Yes, lots of cities in the US have problems. But SF’s are genuine outliers in multiple dimensions.

SF was much worse in the 90’s (substitute some other, long-past decade here). Shouldn’t we be grateful about the progress made over the last few decades?

I don’t think that’s relevant. The SF of today is a very different place from the one of 30+ years ago, with many more resources, technologies, and policy learnings at its disposal. By this same logic, we shouldn’t care about people dying from infectious diseases today because many more died from them in some past epoch (e.g., the early 20th century), which most would argue is a ridiculous assertion.

Wow, this is so depressing. Is San Francisco going to become the next Detroit?

It’s hard to predict the future, but I think the chance of this happening is low. SF’s geography and weather are pretty unique, and the broader metropolitan area is extremely prosperous and continues to grow. However, I have no idea how low it will go before it stabilizes and then, maybe, gets better again.

Why are you complaining without providing solutions?

With all due respect, that’s not my job. I’m not an expert on criminology or substance abuse or urban planning or whatever, nor am I in a position where I control policy. When I lived in SF, I voted and paid taxes with the expectation that the people in charge would try to figure these things out or hire smart people who could. And, they failed miserably.

You’re a terrible, negative person- good riddance!

This kind of sentiment just drives more people and businesses away. Each departure reduces the city’s revenue (personally, I paid tens of thousands of dollars a year in state and local taxes) and activity, which makes the situation there even worse.

Notes

500 Kbps From a High Altitude Balloon

2023-06-13T00:00:00Z

A basic 70cm antenna

2023-06-01T00:00:00Z

A tiny QRP low-pass filter

2023-05-28T00:00:00Z

Avoid Layout Shifts Caused by Web Fonts With PostCSS Fontpie

2023-05-24T00:00:00Z

When your CSS references a web font before it finishes downloading, the browser renders text using a fallback system font instead, causing a layout shift if the text container's height changes once…

steel

💥 “They Would Never Use the Death Star on Us”: Alderaan Residents Reflect on Their Support for the Empire as a Large Imperial Installation Enters the System ↗

I Built My Own Hair Electrolysis Machine

Analogical reasoning

Turns out, competition works

My Nix Config Is Intimate

Caveman

🔊 Hostile Volume ↗

🏎️ International Chess Federation Adds Race Car Piece ↗

🎲 Claude mixes up who said what, and that's not OK ↗

🔴 FBI Extracts Suspect’s Deleted Signal Messages Saved in iPhone Notification Database ↗

🎮 Porting Mac OS X to the Nintendo Wii ↗

💻 This Is Not The Computer For You ↗

"Somewhere" (2010) review

at arm’s length

The Second Wave of the API-first Economy

Video game March

What I've Worked On at Stainless

In and of itself

📊 Nobody Gets Promoted for Simplicity ↗

🏛️ Archive of Archives ↗

Minimum force

Professional vision

🏙️ Why is the sky blue? ↗

Notes on senses

Building a 24-bit Arcade CRT Display Adapter, From Scratch

On the MIT Mystery Hunt 2026

the cruelty of air

Some stories from developing puzlink.js

Actual Lines (And GIFs) From Tech Recruiter Emails

dd is my fdisk

I Program on the Subway

Introducing Glide, an extensible, keyboard-focused web browser

invisible-heading

Why I built Glide

How Glide is different

Modes

Navigating

Personal favourite features

How to make a rare birds hotline in 2025

Beginnings​

Backstory on birding hotlines​

Making this work​

Fetching the data.​

Human hearable reports​

Getting this in the phones​

Polishing this up​

Typing select queries in Python

Getting Started

Introduction to PEP 646

Minimal example

Implementation

Final Implementation

Digging into species likelihood with eBird and DuckDB

Best practices when working with eBird data​

Writing a query for species likelihood​

Error rate / confidence​

Getting this to the front end​

Scaling this up​

A long side quest on better data import​

Fresh data but no progress​

Back to the real purpose​

Conclusion​

Continuing with DuckDB and eBird

The two initial datasets​

Hotspot popularity:​

Getting this live​

Using it live​

Ordering Data​

Using integer IDs​

Trying other tactics​

Conclusion

My Review of the PATH

Cursed Knowledge

Analyzing the full eBird Dataset with DuckDB

The Journey​

Step-by-step guide on how to load the EBD into DuckDB​

Ingredients​

Steps​

Step 1: Download your Data​

Beginnings

Backstory on birding hotlines

Making this work

Fetching the data.

Human hearable reports

Getting this in the phones

Polishing this up

Best practices when working with eBird data

Writing a query for species likelihood

Error rate / confidence

Getting this to the front end

Scaling this up

A long side quest on better data import

Fresh data but no progress

Back to the real purpose

Conclusion

The two initial datasets

Hotspot popularity:

Getting this live

Using it live

Ordering Data

Using integer IDs

Trying other tactics

The Journey

Step-by-step guide on how to load the EBD into DuckDB

Ingredients

Steps

Step 1: Download your Data

Step 2: Decompress your data

Step 3: Load the data into DuckDB

Step 4: Write a query against your data!