PgBouncer is a connection pooler for Postgres. Postgres uses one process per connection, which can become problematic when creating large numbers of client connections. It can also be expensive to repeatedly drop and recreate these connections each time a client connects.

A common solution is to use a pooler that sits in front of your database. PgBouncer is one of the most popular options for Postgres. It maintains a pool of connections to Postgres that are reused across clients. Clients connect to PgBouncer, and from their perspective, it behaves the same as connecting directly to Postgres.

One of the things you might want to understand when deploying PgBouncer is the status of the various connections being made through it. This is similar to the information provided by the pg_stat_activity view in Postgres.

PgBouncer does not provide a pg_stat_activity view. Instead, it offers the SHOW CLIENTS command, which you can run to inspect client connections. Here is an example of what that output looks like when using psql:

pgbouncer=# SHOW CLIENTS;
 type |   user   | database  | replication | state  |     addr      | port  |  local_addr   | local_port |      connect_time       |      request_time       | wait | wait_us | close_needed |      ptr       | link | remote_pid | tls | application_name | prepared_statements | id
------+----------+-----------+-------------+--------+---------------+-------+---------------+------------+-------------------------+-------------------------+------+---------+--------------+----------------+------+------------+-----+------------------+---------------------+----
 C    | postgres | pgbouncer | none        | active | 192.168.107.1 | 59281 | 192.168.107.3 |       6432 | 2026-03-21 21:35:54 UTC | 2026-03-21 21:35:57 UTC |    0 |       0 |            0 | 0xaaaadf56c9b0 |      |          0 |     | psql             |                   0 |  1

What if you want to view this information over time? That is where the Prometheus exporter for PgBouncer comes in. I recently contributed a feature that collects data from the SHOW CLIENTS command, which was included in the 0.12.0 release. You can now use the exporter to visualize SHOW CLIENTS data over time.

The metric name is pgbouncer_client_connections, and it includes the labels database, user, application_name, and state. Here is an example PromQL query:

sum by (database, user, application_name, state) (pgbouncer_client_connections)

Here is an example visualization where each series here shows # of connections for each application name: psql, app1, and app2.

You can use my repo, pgbouncer-show-clients-demo to try this out yourself.

I’ve been ramping up on Node.js for my new team at Supabase, and I’ve learned the first rule of Node.js club: Don’t block the event loop.

I’m not going to get into the details of how the Node.js event loop works. Instead, I’ll walk through my experience debugging an issue where a library was blocking it.

We made a change to enable OpenTelemetry using the open source npm libraries @opentelemetry/*. After deploying this change, our max CPU spiked to 100%. By the time I came online, a discussion was already underway with various hypotheses about the cause.

Coincidentally, I recently read a blog post, Introducing Next-Generating Flamegraph Visualization in Node.js, and this seemed like the perfect use case for CPU profiling.

This isn’t my first time using flamegraphs. If you’ve read my blog post on Tyler Tries Web Development, I wrote about my experience with memray, a Python memory profiler. I linked some good resources there about what they are and how to read them.

Here is how I set up our app for profiling:

➜ git clone https://github.com/supabase/storage.git
➜ cd storage
➜ cp .env.sample .env
➜ npm add -D @platformatic/flame
➜ npm install
➜ npm run infra:start
➜ npx flame run --manual dist/start/server.js

I then ran a k6 script to generate some load on our app in a separate terminal:

➜ node src/test/k6/run-benchmark.cjs

Once the benchmark was over, I hit CTRL+C to stop the profiling and exit. This produced a CPU profile: cpu-profile-2026-03-20T15-59-05-224Z.pb. To create the flamegraph I ran:

➜ npx flame generate cpu-profile-2026-03-04T16-04-45-113Z

This produces an HTML and markdown file. Here is what the flamegraph looked like:

My main mental model when looking at flamegraphs is to look for the widest bottom of the flame. With this in mind, the stack frame labeled spawn caught my eye:

Going down the call stack, spawn is being called from getDefaultNetworkInterface() in node_modules/systeminformation/lib/network.js and it gets there via execSync(). Remember, the first rule of Node.js club is don’t block the event loop. Yet, here we have a synchronous call.

Another nice benefit of the flame tool is the markdown report it generates which confirms our interpretation of the flamegraph:

It’s clear that spawn is the main culprit, accounting for 10.3% of total CPU samples.

Unfortunately, the flamegraph doesn’t show the full call path and stops at node_modules/systeminformation/lib/network.js so we have to do some grepping:

➜ rg "systeminformation/lib/network" -u -l 
node_modules/@opentelemetry/host-metrics/build/src/stats/si.js.map
node_modules/@opentelemetry/host-metrics/build/src/stats/si.js

Object.defineProperty(exports, "__esModule", { value: true });
exports.getNetworkData = void 0;
// Import from the network file directly as bundlers trigger the 'osx-temperature-sensor' import in the systeminformation/lib/cpu.js,
// resulting in the following warning: "Can't resolve 'osx-temperature-sensor'"
// See https://github.com/open-telemetry/opentelemetry-js-contrib/pull/2071
const network_1 = require("systeminformation/lib/network");
function getNetworkData() {
    return new Promise(resolve => {
        (0, network_1.networkStats)()
            .then(resolve)
            .catch(() => {
            resolve([]);
        });
    });
}
exports.getNetworkData = getNetworkData;

This confirms @opentelemetry/host-metrics is what’s calling into the systeminformation/lib/network module. This makes sense given our recent change to add OpenTelemetry. Now we just need to find where in our app we use it:

➜ rg "@opentelemetry/host-metrics" -l   
package.json
package-lock.json
src/internal/monitoring/otel-metrics.ts

Here is a condensed view of src/internal/monitoring/otel-metrics.ts:

import { HostMetrics } from '@opentelemetry/host-metrics'

// Initialize host metrics for Node.js runtime metrics
const hostMetrics = new HostMetrics({
  meterProvider,
  name: 'storage-api-host-metrics',
})
hostMetrics.start()

We found it! This is what the full call path ends up looking like:

  src/internal/monitoring/otel-metrics.ts:235
    └─ hostMetrics.start()
       └─ @opentelemetry/host-metrics → _createMetrics()
          └─ registers a batchObservableCallback
             └─ getNetworkData()               [si.js:25]
                └─ networkStats()              [systeminformation/lib/network.js]
                   └─ getDefaultNetworkInterface()
                      └─ execSync → spawnSync → spawn

The fix we did for now was a simple one liner to emit the network metrics group:

const hostMetrics = new HostMetrics({
  meterProvider,
  name: 'storage-api-host-metrics',
+ metricGroups: ['system.cpu', 'system.memory', 'process.cpu', 'process.memory'],
})

Here is a new flamegraph with the change:

Now for my favorite part, the event loop p99 latency across each of our app instances after we rolled out the fix:

Event loop latency measures how long a callback has to wait before the event loop picks it up. When the event loop is blocked by synchronous work, that wait time grows, meaning every request queued behind it is delayed.

We saw consistent spikes to ~1s and you can infer when the deployment took place based on the nosedive in the charts.

So a reminder, don’t block the event loop.

select 
    package_name, 
    package_summary
from 
    pypi_packages
where 
    package_name like ?
order by 
    package_name
limit 
    10

package_name is the primary key and has an index on it. The table has 864,900 rows and is 166MB in size.

90ms for a query with a filter on the index, limited to 10 rows, on a table that is only 166 MB in size!? I would have expected something in the single digit millisecond range.

This query powers the active search results in pypacktrends, so as you type, package names start to appear. It takes the current value of the search input and appends % to match package names that start with the current prefix.

I ran a quick explain query plan to see what’s going on:

.timer on
explain query plan 
select 
    package_name, 
    package_summary
from 
    pypi_packages
where 
    package_name like 'duck%'
order by 
    package_name
limit 
    10
-- QUERY PLAN
-- SCAN pypi_packages USING INDEX sqlite_autoindex_pypi_packages_1
-- Run Time: real 0.057 user 0.012548 sys 0.005846

Huh, 57ms and it’s using a full index scan. After some research, it turns out like is not case sensitive by default. Indexes, however, use binary collation by default. Collation is just the set of rules SQLite uses to compare and sort text (for example, whether A equals a, and how strings are ordered).

Because the search is case insensitive but the index uses binary collation, SQLite isn’t able to use the index efficiently. The simple fix is to make like case sensitive:

.timer on
PRAGMA case_sensitive_like = ON;
explain query plan 
select 
    package_name, 
    package_summary
from 
    pypi_packages
where 
    package_name like 'duck%'
order by 
    package_name
limit 
    10
-- QUERY PLAN
-- SEARCH pypi_packages USING INDEX sqlite_autoindex_pypi_packages_1 (package_name>? AND package_name<?)
-- Run Time: real 0.004 user 0.000486 sys 0.001236

4ms and (package_name>? AND package_name<?) shows we are now doing a range scan on the index. Much better.

One common way to do that is PXE booting. Instead of booting from a USB stick, a client asks the network where to boot from.

PXE booting usually involves two protocols:

• Dynamic Host Configuration Protocol (DHCP) for network settings and boot metadata
• Trivial File Transfer Protocol (TFTP) for downloading early boot files

At home, my TP-Link Deco router (192.168.68.1) is the main DHCP server. It hands out IP addresses, gateway, DNS, and lease info. The issue is my Deco does not expose DHCP options for PXE boot settings.

So I used dnsmasq on another host (192.168.68.57) as a Proxy DHCP server. In this setup dnsmasq only supplies PXE boot details (boot file + next server) while my router supplies the rest.

sudo ipconfig set en0 dhcp

sequenceDiagram
  participant Client
  participant DHCP_Server as DHCP Server (192.168.68.1)
  Client->>DHCP_Server: Discover (0.0.0.0 -> 255.255.255.255)
  DHCP_Server-->>Client: Offer (IP lease proposal)
  Client->>DHCP_Server: Request (accept offer)
  DHCP_Server-->>Client: ACK (lease confirmed)

sudo tcpdump -i eno1 -w /tmp/proxydhcp-demo.pcap '(port 67 or port 68 or port 69 or port 4011)'

dhcp
chain tftp://192.168.68.57/boot.ipxe

I’m still coming to terms with this experience. I have never programed in Rust and I would be lying if I said I fully understood the code that it generated. I just wrote a post where I called out how many notable projects such as Ghostty, Node.js, curl, tldraw have limited or shut down external contributions after being overwhelmed with AI slop PRs.

I did not want to be one of those people. At the end of the day it’s my name being attached to the PR and I take full responsibility for the code.

On the flip side, I have seen how AI can help. The same project, Ghostty, shows the benefits of AI and how it enables users who don’t know anything about the underlying technology contribute high quality issue reports.

A very similar Ghostty example (though a different person) was actually discussed by David Crespo on the Oxide and Friends podcast: Engineering Rigor in the LLM Age.

I resonated with many of the things David said:

And I so that was what really unsettled me was that this was an area where I really knew nothing and just using my sort of, like, sense of what sounds like it makes sense to validate that I wasn’t gonna be posting AI slop on the ghosty GitHub, I was able to come to, you know, three real, bug reports without really putting myself putting very much into the process.

I didn’t know enough about how Ghostty worked or how Zig worked to Yeah. To really evaluate. So I was nervous, but I was, you know, upfront with a lot of humility of, like, I’m really not sure about this, but I but it sounds so good that I cannot hold it back.

How can I be more like these contributors and use AI to produce high quality work?

I think it really boils down transparency. Be transparent with your use of AI, what you don’t know and show that you put some honest effort in. You can’t just generate AI slop and toss it over.

Here’s what my experience looked like going from question to merged PR.

In mise, you can run the mise list to show all tools mise “knows about”, buy many of the tools don’t display their source files unless you’re in the directory where that source config is located. For example:

Tool        Version           Source                      Requested 
aws-vault   7.2.0            
bun         1.3.0             ~/.config/mise/config.toml  latest
kubectl     1.34.0           

but I want to do something like mise list --all-sources

Tool        Version           Source                                Requested 
aws-vault   7.2.0             ~/code/work/infrastructure/mise.toml  latest
bun         1.3.0             ~/.config/mise/config.toml            latest
                              ~/code/work/infrastructure/mise.toml 
kubectl     1.34.0            ~/code/work/infrastructure/mise.toml  1.34.0

This would allow me to see all sources for each tool across all my config files. I wanted this so I can see if there are some projects I can get rid of to clean up the amount of tools I have installed.

Mise has great docs on their contributing guidelines. Before submitting a PR, they ask contributors to first open a discussion or mention the idea in their Discord.

I read all the docs on Mise and scanned the existing discussions to see if mise already allowed this. I couldn’t find anything so I opened a discussion asking if this was something currently possible, if not, if they would be open to adding this feature.

I followed up with a screenshare of my fork of mise that added the feature to demo what I was looking for. The code was entirely AI generated and I wasn’t too concerned with the correctness at this point as I simply wanted to demo the feature.

I used opencode with GPT-5.2 Codex, you can see all my prompts here. I was very transparent with my use of AI when sharing the screenshare and even linked my session in the discussion.

I got a reply back from the maintainer “seems fine to me” which felt like a green light to open a PR.

This part was tough for me though because as I said I don’t know Rust, I have zero confidence if the code is “right” and was torn about opening a PR. Having code out there with my name on it that I don’t fully understand is unsettling.

I scoured the contributing guidelines over and over to make sure my change was properly tested, documentation was updated, the PR title was correct and so on. I ran the tests several times to make sure they were passing. I had several different AI agents review the code locally to make sure everything look good.

I opened the PR and noticed a few tests were failing and got some AI code review that I quickly addressed with a follow up commit. All CI was now passing. Now, all I had to do was wait.

To my surprise I woke up the next morning to this:

The PR was merged and has been added to the next release of mise.

My first reaction was, is this fair? Why should I be added as a contributor when AI did all the work? I was simply the “idea guy”. Is this how the future of software development is going to be?

It’s still unsettling to know there is code out there with my name attached to it that I don’t fully understand. The maintainer’s approval gives me some assurance that this code isn’t entirely AI slop as I assumed it wouldn’t have been merged otherwise.

I wanted to share this experience in hopes to promote more high quality AI generated PRs to open source repos. Remember it’s your name being attached to those PRs, your reputation on the line. Do you want to be known as the person who slings AI slop all over? “But Claude wrote it…” “ChatGPT told me…” is not an excuse. Take some responsibility.

The specs are dated, but they’re still perfect for hands-on learning with Linux, Docker, and Kubernetes. They’re also very serviceable. It often takes just one screw to open the enclosure and get to the hardware.

I first came across these mini PCs through Project MINI Rack by Jeff Geerling. Instead of the traditional 19” server racks you see in data centers, there’s been a push toward 10” racks that are great for homelabs and portable network setups. These mini PCs are a great fit for that form factor, especially if you have access to a 3D printer and can print your own shelves.

Here’s what mine looks like:

It’s not just me getting interested in these. There’s been a growing movement in the tech community around self-hosting and moving away from cloud providers because of cost. Mini PCs are a great way to learn those skills.

One recent example I saw was how DHH was able to run Rails World campfire chat on a Beelink EQR5 out of his closet. I’m not here to say “Cloud is bad, owning your own hardware is better”. Like most things, it depends on what you’re building.

For me this whole homelab, Linux, self-hosting journey is primarily educational. It’s hard to really understand the value cloud providers add through all their services and abstractions on top of a basic VPS if you’ve never done it the hard way yourself.

While digging into mini PC content, I came across this video: TinyMiniMicro Home Lab Revolution Introduction. It mentioned how Chick-fil-A uses Intel NUC devices in their restaurants to host Kubernetes. That led me to a KubeCon talk, Renovating Edge Infrastructure at The Home Depot, which describes a very similar architecture.

I know the term “edge” can come with some baggage, which is why I intentionally didn’t use it in the blog title. To me, “edge” just means compute that doesn’t live in a traditional data center.

A couple similarities that stood out to me between Chick-fil-A and Home Depot:

• Eventual consistency: In store devices can’t constantly call back to the cloud, where the durable source of truth lives, because of latency and unreliable networks. Instead, they work off a local dataset that syncs periodically.
• The iPhone storage model: Applications on these devices treat local storage as temporary. Backups happen periodically, so data loss between syncs is possible. What matters is that the device can restart, reprovision, and get back to a working state quickly because the apps aren’t relying on local state.

What really intrigues me about this style of infrastructure is that both the hardware and software are accessible enough that I can build something very similar in my own homelab.

Now that I have my mini rack put together, I’ve been looking for projects that lean toward Platform, Infrastructure, and Site Reliability work. That’s the direction I want to take my career, and building my own edge infrastructure platform at home feels like a good way to learn. More blog posts to come on this.

One thing I’ve been thinking about after going down this rabbit hole was this blog post title I came across:

The Edge is not the cloud: Stop pretending they are the same thing.

And a point called out in the Home Depot talk:

“There are tons of storage solutions for Kubernetes. I almost need a sandbox up here on top of the stage. There are tons of storage solutions for Kube, but they’re all designed for not edge.”

To prove the difficulty of the problem, TenBrink recounted conversations with vendors from KubeCon: “Oh, what cloud are you running in?” TenBrink’s response: “I’m not. I’m running on my own boxes. We’re not for you.”

It got me wondering: why can’t the edge be the cloud?

What if someone took the same philosophy Oxide did with their rack scale computer that provides API driven elastic infrastructure that you own and packaged it into a device that could be dropped into a local Chick-fil-A?

Something to think about.

If you are looking to learn more I recommend checking out the following resources:

• Edge Monsters Blog
• brianchambers.substack.com
• chick-fil-atech blog

This post made the rounds recently, landing on the front page of Hacker News and all over my Twitter “For You” feed. It articulated some of the same feelings I’ve been having lately but haven’t been able to put into words.

When code gets cheap operational excellence wins.

This line stuck with me.

The genie is out of the bottle. People are going to be writing code at an unprecedented scale with AI agents and we’re already seeing the downsides. Several notable open source projects such as Ghostty, Node.js, curl, tldraw have limited or shut down external contributions after being inundated with AI generated PRs.

This begs the question, how do we gain confidence this AI generated code “works”?

To me, this is fundamentally ops work. The real problem is “confidence” is subjective. It’s not about 100% test coverage, it’s the feeling when I ship this, everything is going to be okay.

If a company has a reliable CI/CD pipeline, strong observability, and safe rollback procedures, it almost doesn’t matter where the code came from. Human. AI. Intern. Vendor. What matters is whether the product can safely absorb change.

So lets bring back ops pride and get ready for software engineering in this new era.

There is nothing more embarrassing sharing a link with someone with a bunch of utm query parameters https://www.tylerhillery.com/?utm_source=share&utm_content=share_button. Which is why I built URL Clipr, a chrome extension that removes unwanted query parameters.

Users can define a list of patterns (currently limited to “starts with” patterns) that will be excluded when copying the URL. There are three ways to copy the URL:

1. Shortcut CTRL+Shift+U / Command+Shift+U which can be modified at chrome://extensions/shortcuts
2. Copy Button in the top right corner of the popup
3. Context Menu when right clicking a link on a web page.

I could add a pattern such as utm and when I copy the URL it would go from https://www.tylerhillery.com/?utm_source=share&utm_content=share_button to https://www.tylerhillery.com

I never built a chrome extension before so the first thing I did was created a playground repo to see how chrome extensions work. Starting a playground repo is one of the first things I do when learning anything new. It provides a no stress way to experiment. I can often get caught up in the ancillary tasks of a project like CI/CD, code formatting, repo structure, testing etc. Playgrounds allow me to ignore all of that and focus solely on the thing I am learning.

The get started docs for chrome extension development is where I began. Once I finished this guide I started some searching to see if anyone had a preexisting template I could use with built in CI/CD to help with publishing the extension. Eventually I came across this framework called WXT which has the tagline:

an open source tool that makes web extension development faster than ever before.

At first it seemed a little overkill for the extension I was trying to build but I liked the idea that I could eventually publish this extension to multiple browsers. WXT does allow a pretty bare bones setup so you can ignore using a frontend framework like React, Vue, Svelte. I was actually surprised to learn that chrome extension popups are just plain HTML/CSS/JS. Not sure why I was expecting anything different.

WXT has a good guide, and my biggest takeaway was learning how a Chrome extension can have multiple entry points. These entry points define how the extension starts and where its functionality lives. For URL Clipr, the two I needed were the background and popup entry points.

The background entrypoint is responsible for adding the copy button in the context menu on right clicks. This is done by using browser.contextMenus.create. Then you can add an onClicked listener to apply the cleaning logic and add to clipboard. The other listener the background entrypoint sets up is for the copy-url event which is created as a shortcut in the wxt.config.ts file:

export default defineConfig({
  srcDir: "src",
  imports: false,
  manifest: {
    name: "URL Clipr",
    permissions: ["storage", "activeTab", "contextMenus", "scripting"],
    commands: {
      "copy-url": {
        suggested_key: {
          default: "Ctrl+Shift+U",
          mac: "Command+Shift+U",
        },
        description: "Copy URLs without unwanted query params",
      },
    },
  },
});

The only difference from the context menu on click event is the copy-url event has to use browser.tabs.query to get the active URL.

The popup entry point shows the UI when you click on the extension. It’s where the user can specify the list of patterns to exclude. In full transparency the main UI code for the popup was AI generated. I wouldn’t say “vide coded” as I did thoroughly review the code, even though HTML/CSS is not my strong suite.

The patterns are stored in local storage and retrieved whenever a URL needs to be cleaned. What surprised me was how much trickier the cleaning logic turned out to be than I expected. I initially planned to use a flexible regex-based approach that could handle cases like “starts with” “ends with,” “contains,” and “equals.” My idea was to build something like [?&])(?:${sources.join('|')})=[^&]*(?=&|$), where sources would come from the saved patterns, along with a dropdown that lets the user specify the pattern type. That way, the patterns could be stored like this:

const patterns = [
  { startsWith: 'utm' },
  { contains: '__' },
  { endsWith: 'with' },
  { exact: 'foo*bar' }
];

I have a confession to make though, I don’t understand regex that well and every time I use it I have to use regexr.com or https://regex101.com/ to breakdown the pattern for me. The pattern I came up with felt overly complex so I kept it simple for the first version and only allowed “starts with” patterns which allows me to use very basic JS string methods

.filter((param) => !patterns.some((pattern) => param.startsWith(pattern)));

Another aspect that wasn’t as straight forward as I anticipated was decoding and encoding URLs. The core function that cleans the URL takes in a string version of the URL with the list of patterns to filter out.

function urlClipr(urlStr: string, patterns: string[]): string

I convert the urlStr to a URL object. My original plan was to use the URLSearchParams object but I want to see if you can notice something in the below example:

personal/url-clipr git:(main) 
➜ node
Welcome to Node.js v24.11.1.
Type ".help" for more information.
> const url = new URL("https://example.com?utm_source=email&message=hello%20world#heading")
undefined
> url.searchParams.toString()
'utm_source=email&message=hello+world'

Did you spot it? The encoding is different for searchParams, you get hello+world vs hello%20world. This is actually mentioned in the docs:

However, URL.search encodes a subset of characters that URLSearchParams does, and encodes spaces as %20 instead of +. This may cause some surprising interactions—if you update searchParams, even with the same values, the URL may be serialized differently.

This made it tricky to use any built in objects so instead I operated on the raw string itself to “keep it simple stupid”.

export function urlClipr(urlStr: string, patterns: string[]): string {
  const url = new URL(urlStr);

  const raw = url.search.slice(1);

  const params = raw
    .split("&")
    .filter((param) => param !== "")
    .filter((param) => !patterns.some((pattern) => param.startsWith(pattern)));

  return `${url.origin}${url.pathname}${
    params.length ? "?" + params.join("&") : ""
  }${url.hash}`;
}

If it wasn’t for setting up a comprehensive test suite I wouldn’t have caught this subtle difference! Speaking of testing, WXT has great built in tools for testing. WXT spins up a browser with the extension preinstalled in dev mode. I was very impressed by WXT as you can tell the creators thought deeply about entire DX from development to publishing.

The toughest part for me was figuring out how to publish the extension to the chrome web store. WXT makes this easy and has a guide on how to do so with GitHub Actions but the part I got tripped up on was where to get the following environment variables:

- name: Submit to stores
        run: |
          pnpm wxt submit \
            --chrome-zip .output/*-chrome.zip \
        env:
          CHROME_EXTENSION_ID: ${{ secrets.CHROME_EXTENSION_ID }}
          CHROME_CLIENT_ID: ${{ secrets.CHROME_CLIENT_ID }}
          CHROME_CLIENT_SECRET: ${{ secrets.CHROME_CLIENT_SECRET }}
          CHROME_REFRESH_TOKEN: ${{ secrets.CHROME_REFRESH_TOKEN }}

To get the extension ID I had to upload the extension manually for the first time. The CLIENT_ID, CLIENT_SECRET and REFRESH_TOKEN were the tougher parts. That was until I found this guide on publishing chrome extensions programmatically using the Chrome Web Store API. It walks you through everything step by step.

URL Clipr is live now, so give it try. Right now it only supports “starts with” patterns which honestly covers everything for me (mainly utm and __) but if there is enough demand for other patterns I will look into adding them.

Now feel free to save and share those URLs without those unwanted query parameters!

As I have been making my way through the learn section on the Node.js website, I came across a pleasant surprise in the Backpressuring in Streams section:

That’s right, dtrace mentioned! Over the past ~5 months, I have gone through the entire Oxide and Friends podcast backlog, and it’s left a lasting impact on how I think about technology and software engineering. I shared more about my thoughts in this blog post.

DTrace comes up frequently in the podcast since Bryan Cantrill, one of the co-hosts, was its creator and Adam Leventhal, the other co-host, was also one of its original developers. I’m familiar with the tool from those discussions, even though I haven’t used it myself.

When learning a new programming language, I personally like to type out the examples myself and run them. However, the backpressure example didn’t go through how to instrument it with DTrace to get the results they mentioned, so I used this as an opportunity to give DTrace a spin!

First, what is DTrace? DTrace is a dynamic tracing framework, and we’ll be using it here to measure GC (garbage collection) times in Node.js to see the effects of backpressure disabled.

As described in the docs, backpressure is the buildup of data behind a buffer during data transfer. Streams in Node.js allow you to process data incrementally versus loading the entire dataset into memory. Backpressure in Node.js occurs when the Readable stream produces too much data for the Writable stream to handle.

When the Writable stream can’t keep up the data gets put into a queue waiting to be processed. This queue will continue to grow as long as the Writable stream stays behind. Managing this queue from building up is what Node’s backpressure mechanism does.

The example provided in the docs show compressing a video file because writing to disk is a lot slower than reading from disk. This provides a good way to demonstrate the effects of backpressure. If we turn off the backpressure mechanism, memory usage spikes and garbage collection runs more often, which is exactly what we’ll measure with DTrace.

Of course, before I could actually run the example, I ran into my first problem. The example uses a 9GB video file, The.Matrix.1080p.mkv. Unfortunately, I don’t just have a 9 GB video lying around. I asked ChatGPT where I could download a free large video file, which led me to something called “Big Buck Bunny.”

Ummm… what? I risked it and clicked the suspicious looking link which greeted me with this:

After some further research, it appears to be a legit free open movie from the Blender Foundation licensed under the Creative Commons Attribution 3.0 license. The more you know!

Anywho, I downloaded what I thought would be the largest file available:

➜ curl -O http://distribution.bbb3d.renderfarming.net/video/mp4/bbb_sunflower_native_60fps_stereo_abl.mp4
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1020M  100 1020M    0     0  1190k      0  0:14:38  0:14:38 --:--:--  639k

It’s only 1020MB, so I used ffmpeg to repeat it 8 more times to get 9GB as I wanted to have a similar file size as the example.

node-js-notes/intro-to-node-js git:(main) ✗ 
➜ ffmpeg -stream_loop 8 -i bbb_sunflower_native_60fps_stereo_abl.mp4 -c copy bbb_repeat_9_times.mp4
node-js-notes/intro-to-node-js git:(main) ✗ 
➜ ls -lh bbb_repeat_9_times.mp4
-rw-r--r--@ 1 tyler  staff   8.9G Oct 15 18:11 bbb_repeat_9_times.mp4

8.9GB should do.

After looking into the example more, it looks like we need to compile a custom build of Node.js taking out the return value of the .write() function, which effectively disables backpressure support.

I decided to use the latest LTS version, which at the time of writing this is v22.20.0.

git clone --branch v22.20.0 https://github.com/nodejs/node.git

Doesn’t look like node/lib/_stream_writable.js contains that function anymore.

// https://github.com/nodejs/node/blob/v22.20.0/lib/_stream_writable.js
'use strict';

// Keep this file as an alias for the full stream module.

module.exports = require('stream').Writable;

The link they provided was a permalink to a commit that took place 9 years ago.

Using go to definition, it takes me to node/lib/stream.js which contains another import from /node/lib/internal/streams/writable.js. Ah finally, the Writable.prototype.write function, but wait no return ret line:

After doing a couple more go to definitions, I found the little sucker defined in the writeOrBuffer() function in the /node/lib/internal/streams/writable.js file.

// Return false if errored or destroyed in order to break
// any synchronous while(stream.write(data)) loops.
return ret && (state[kState] & (kDestroyed | kErrored)) === 0;

I replaced this line with return true instead.

The next step is to build Node.js from source. Luckily, Node.js has some great documentation on how to do so. I am pretty sure I had all the required prerequisites to run the build process, so I went forward with the commands outlined in the guide:

export CXX=g++-12
./configure
make -j4

Buttt it blew up on me when I ran ./configure

intro-to-node-js/node git:(v22.20.0) ✗ 
➜ ./configure
Node.js configure: Found Python 3.13.8...
Traceback (most recent call last):
  File "/Users/tyler/code/learning/node-js-notes/intro-to-node-js/node/./configure", line 28, in <module>
    import configure
  File "/Users/tyler/code/learning/node-js-notes/intro-to-node-js/node/configure.py", line 2274, in <module>
    check_compiler(output)
    ~~~~~~~~~~~~~~^^^^^^^^
  File "/Users/tyler/code/learning/node-js-notes/intro-to-node-js/node/configure.py", line 1237, in check_compiler
    ok, is_clang, clang_version, gcc_version = try_check_compiler(CXX, 'c++')
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: too many values to unpack (expected 4)

Hmmm, I set up a breakpoint on that line to step into what’s happening with try_check_compiler(CXX, 'c++'). There are only two return statements in there, and only one of them returns more than four values, which implies I was hitting an OSError:

  except OSError:
    return (False, False, '', '', False)

Oddly enough I didn’t hit that error running it directly through the VS Code debugger. This is puzzling. I never used the built in pdb before but I thought I would give that a try and indeed it raised the OSError.

➜ ./configure
Node.js configure: Found Python 3.13.9...
> /Users/tyler/code/learning/node-js-notes/intro-to-node-js/node/configure.py(1238)check_compiler()
-> import pdb; pdb.set_trace()
(Pdb) n
> /Users/tyler/code/learning/node-js-notes/intro-to-node-js/node/configure.py(1239)check_compiler()
-> ok, is_clang, clang_version, gcc_version = try_check_compiler(CXX, 'c++')
(Pdb) s
--Call--
> /Users/tyler/code/learning/node-js-notes/intro-to-node-js/node/configure.py(1115)try_check_compiler()
-> def try_check_compiler(cc, lang):
(Pdb) n
> /Users/tyler/code/learning/node-js-notes/intro-to-node-js/node/configure.py(1116)try_check_compiler()
-> try:
(Pdb) n
> /Users/tyler/code/learning/node-js-notes/intro-to-node-js/node/configure.py(1117)try_check_compiler()
-> proc = subprocess.Popen(shlex.split(cc) + ['-E', '-P', '-x', lang, '-'],
(Pdb) n
> /Users/tyler/code/learning/node-js-notes/intro-to-node-js/node/configure.py(1118)try_check_compiler()
-> stdin=subprocess.PIPE, stdout=subprocess.PIPE)
(Pdb) n
> /Users/tyler/code/learning/node-js-notes/intro-to-node-js/node/configure.py(1117)try_check_compiler()
-> proc = subprocess.Popen(shlex.split(cc) + ['-E', '-P', '-x', lang, '-'],
(Pdb) n
FileNotFoundError: [Errno 2] No such file or directory: 'g++-12'
> /Users/tyler/code/learning/node-js-notes/intro-to-node-js/node/configure.py(1117)try_check_compiler()
-> proc = subprocess.Popen(shlex.split(cc) + ['-E', '-P', '-x', lang, '-'],
(Pdb) n
> /Users/tyler/code/learning/node-js-notes/intro-to-node-js/node/configure.py(1119)try_check_compiler()
-> except OSError:
(Pdb) n
> /Users/tyler/code/learning/node-js-notes/intro-to-node-js/node/configure.py(1120)try_check_compiler()
-> return (False, False, '', '', False)

Ahhh I’m an idiot. FileNotFoundError: [Errno 2] No such file or directory: 'g++-12' I blindly just copied the command export CXX=g++-12, but I don’t have that installed. The VS Code debugger didn’t have CXX set because I just set that environment variable within my terminal session.

After unsetting CXX it completed successfully.

ntro-to-node-js/node git:(v22.20.0) ✗ 
➜ ./configure
Node.js configure: Found Python 3.13.9...
INFO: configure completed successfully

The next step was running make:

make -j4

A couple of minutes after running the command, I left hoping it would be finished when I got back. 10 minutes later I returned, and I thought my MacBook was getting ready for takeoff. Seriously, listen:

This was the first time I have ever had my fans turn on for this machine. Eventually, the build did complete in about ~30min.

I copied the stream/promises example they had and ran the script for both versions, modified (backpressure disabled) and an unmodified version of Node.js:

   trial (#)  | `node` binary (ms) | modified `node` binary (ms)
=================================================================
      1       |      137040        |          139900   
      2       |      136362        |          139999   
      3       |      141825        |          142009   
      4       |      136131        |          143897   
      5       |      139041        |          142208   
=================================================================
average time: |      138080        |          141603 

Slower than the example, but I am going to chalk that up to “my machine vs their machine” and user error. The numbers between the unmodified and modified versions are comparable, which is what I wanted to see.

Now it’s time to setup DTrace to reproduce the GC measured time. I recall in a talk by Adam Leventhal, DTrace was “castrated” when macOS released System Integrity Protection (SIP). Disabling SIP made me feel uneasy. After some googling though, I came across this blog post, which used UTM to create a macOS virtual machine with SIP disabled.

This was something I was much more comfortable with, so I spun up a VM in recovery mode and ran csrutil disable to disable SIP.

macOS comes with dtrace preinstalled, so with a little help from GitHub Copilot, I asked if it could show me how to measure Node.js GC time.

tylerhillery@Tylers-Virtual-Machine intro-to-node-js % sudo dtrace -n 'node*:::gc-start { self->ts = timestamp; }
                node*:::gc-done /self->ts/ { printf("%d ms\n", (timestamp - self->ts)/1000000); self->ts = 0; }' -p 15088
dtrace: invalid probe specifier node*:::gc-start { self->ts = timestamp; }
                node*:::gc-done /self->ts/ { printf("%d ms\n", (timestamp - self->ts)/1000000); self->ts = 0; }: probe description node*:::gc-start does not match any probes

Hmmm, doesn’t look like my Node.js process is generating any DTrace probes. After some more googling, it appears Node.js doesn’t enable DTrace probes by default, but apparently I can enable them with a flag. I couldn’t find any mentions of DTrace in the Node.js codebase, but after grepping through the commit history, I was able to find this commit:

intro-to-node-js/node git:(v22.20.0) ✗ 
➜ git show --no-patch aa3a572e6be
commit aa3a572e6bee116cde69508dc29478b40f40551a
Date:   Sun Jul 17 18:33:52 2022 +0200

    build: remove dtrace & etw support
    
    There are no clear indicators anyone is using the dtrace USDT probes.
    
    ETW support is very intertwined with the dtrace infrastructure. It's not
    clear if anyone uses ETW so to keep things simple it too is removed.
    
    Fixes: https://github.com/nodejs/node/issues/43649
    
    PR-URL: https://github.com/nodejs/node/pull/43652

You got to be kidding me. Well, it was at this moment that I found out Node.js recommends using the --trace-gc flag for measuring GC nowadays.

But I have come so far, and I really wanted to see if I could get DTrace working on the latest LTS release. I could use an older version of Node.js that had DTrace enabled, but that’s no fun. Do it for the gram blog!

The problem is my C++ skills are lacking… to say the least, and the codebase has changed so much since DTrace was removed that a simple git revert wouldn’t help here. This seemed like a good use case for AI though, as I can provide it the context of the diff where it was removed and see if it can add it back in.

I gave Copilot the task, and a few minutes later I had a successful build of Node.js v22.20.0 with DTrace enabled. No clue if it’s actually right, but I don’t really care at this point. I just wanted to see those Node.js probes!

You might be asking, what is a probe anyway? The Dynamic Tracing Guide has a nice section on them:

DTrace helps you understand a software system by enabling you to dynamically modify the operating system kernel and user processes to record additional data that you specify at locations of interest, called probes. A probe is a location or activity to which DTrace can bind a request to perform a set of actions, like recording a stack trace, a timestamp, or the argument to a function. Probes are like programmable sensors scattered all over your illumos system in interesting places.

With this helpful context, let’s see if we can see those probes now:

tylerhillery@Tylers-Virtual-Machine node-js-notes % sudo dtrace -l -n 'node*:::gc*'     
   ID   PROVIDER            MODULE                          FUNCTION NAME
 2904   node2008              node _ZN4node14dtrace_gc_doneEPN2v87IsolateENS0_6GCTypeENS0_15GCCallbackFlagsEPv [node::dtrace_gc_done(v8::Isolate*, v8::GCType, v8::GCCallbackFlags, void*)] gc-done
 2905   node2008              node _ZN4node15dtrace_gc_startEPN2v87IsolateENS0_6GCTypeENS0_15GCCallbackFlagsEPv [node::dtrace_gc_start(v8::Isolate*, v8::GCType, v8::GCCallbackFlags, void*)] gc-start

We are back in business baby.

Let’s try tracking GC times again and break down each step of this dtrace command:

tylerhillery@Tylers-Virtual-Machine node-js-notes % sudo dtrace -n 'dtrace:::BEGIN { start = timestamp; } node2008:::gc-start { self->ts = timestamp; } node2008:::gc-done /self->ts/ { printf("%d,%d\n", (timestamp - start)/1000000, (timestamp - self->ts)/1000000); self->ts = 0; }'
dtrace: description 'dtrace:::BEGIN ' matched 3 probes
CPU     ID                    FUNCTION:NAME
  2   2904 _ZN4node14dtrace_gc_doneEPN2v87IsolateENS0_6GCTypeENS0_15GCCallbackFlagsEPv:gc-done 173,0
  0   2904 _ZN4node14dtrace_gc_doneEPN2v87IsolateENS0_6GCTypeENS0_15GCCallbackFlagsEPv:gc-done 429,0

• -n enabled or list probes matching the specified probe name
• dtrace:::BEGIN { start = timestamp; } The BEGIN probe fires once each time you start a new tracing request. timestamp is a DTrace variable that returns the value of an always incrementing nanosecond counter. We are saving it here to start so we can keep track of the time elapsed since the start of the DTrace tracing request
• node2008:::gc-start { self->ts = timestamp; } very similar to what we are doing above but this time saving the timestamp of the node2008:::gc-start probe
• node2008:::gc-done /self->ts/ { ... } This fires when GC completes, with a predicate /self->ts/ to ensure we only process gc-done events that have a corresponding gc-start. Let’s break down what happens inside:
  • printf("%d,%d\n", (timestamp - start)/1000000, (timestamp - self->ts)/1000000) Prints two comma-separated values:
    • Time elapsed since tracing started (in milliseconds)
    • Duration of this GC cycle (in milliseconds)
  • self->ts = 0 Resets the GC start timestamp for the next cycle

I measured the GC times with DTrace on the modified (backpressure disabled) and unmodified versions of Node.js. For completeness sake, I also measured GC times with the --trace-gc flag as well. Here are the results:

Now, I wouldn’t take the numbers too seriously as I have broken many benchmarking sins here, but the general takeaway is true regardless. Disabling backpressure clearly makes the GC work much harder.

This was interesting to see, but it has me left wondering, how does the Node.js backpressure mechanism work? Well, this gets explained later on in the docs.

Essentially, when a false is returned from the writeOrBuffer(), it will pause the Readable stream from sending any data and wait until the consumer is ready again, which gets notified by the drain event. The writeOrBuffer() function will return false when the data buffer has exceeded the highWaterMark or the write queue is internally busy.

Hopefully, this post demystifies a bit of how backpressure works in Node.js and how you can use DTrace to measure GC. Now it’s time to go watch Big Buck Bunny!

I want to walk through how I went about debugging this issue and how git bisect was crucial for pinpointing the exact commit that caused it.

First, I tried to reproduce the issue locally on my machine. The user linked the GitHub action where they first noticed this issue occurring. I was able to look at the logs to find out the version of supabase storage they were running Downloaded newer image for ghcr.io/supabase/storage-api:v1.27.4. Now that I know the version I checked out that specific commit and ran the storage api locally.

work/storage git:(master) 
➜ git checkout v1.27.4
work/storage git:(v1.27.4) 
➜ npm run infra:start
work/storage git:(v1.27.4) 
➜ npm run dev

With the server running, I copied the repro script from the GitHub issue to my local environment. The script required an additional package, so I temporarily added it to the package.json just to get the script working locally, not as a permanent dependency.

work/storage git:(v1.27.4) 
➜ npx jsr add @bradenmacdonald/s3-lite-client

I changed some of the hardcoded defaults because it was setup to run against the supabase cli and I also added some code to handle creating the bucket and uploading the example file for convenience.

After running the script this was the result.

work/storage git:(v1.27.4) ✗
➜ node src/scripts/repro.ts
Creating bucket: your-bucket
Bucket 'your-bucket' already exists
Uploading file to: orgs/demo-app/example.txt
File uploaded successfully to orgs/demo-app/example.txt
Presigned PUT URL: http://127.0.0.1:5000/s3/your-bucket/orgs/demo-app/example.txt?X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&x-id=PutObject&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=b585f311d839730f8a980a3457be2787%2F20251004%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251004T193230Z&X-Amz-Expires=60&X-Amz-SignedHeaders=host&X-Amz-Signature=75b7bd2d33b66855952bad41bc56f17aaa98cd96a50f49ad96540c030e27ee19
Local storage response: 200
Response body:

Huh, completed successfully. This had me puzzled but then I recalled the user was running into issues when running with the supabase cli command. So, instead of running directly against the storage api I thought I try against the supabase cli.

work/storage git:(v1.27.4)
➜ npx supabase init
work/storage git:(v1.27.4)
➜ npx supabase start

I reran the script.

work/storage git:(v1.27.4) ✗
➜ node src/scripts/repro.ts
Creating bucket: your-bucket
Bucket 'your-bucket' already exists
Uploading file to: orgs/demo-app/example.txt
File uploaded successfully to orgs/demo-app/example.txt
Presigned PUT URL: http://127.0.0.1:5000/s3/your-bucket/orgs/demo-app/example.txt?X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&x-id=PutObject&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=b585f311d839730f8a980a3457be2787%2F20251004%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251004T193230Z&X-Amz-Expires=60&X-Amz-SignedHeaders=host&X-Amz-Signature=75b7bd2d33b66855952bad41bc56f17aaa98cd96a50f49ad96540c030e27ee19
Local storage response: 200
Response body:

What!? Succeeded again. Then I realized the user was running supabase cli version 2.45.5 so I tried it again.

work/storage git:(v1.27.4)
➜ npx supabase@2.45.5 start
work/storage git:(v1.27.4) ✗
➜ node src/scripts/repro.ts
Creating bucket: your-bucket
Bucket 'your-bucket' created successfully
Uploading file to: orgs/demo-app/example.txt
File uploaded successfully to orgs/demo-app/example.txt
Presigned PUT URL: http://127.0.0.1:54321/storage/v1/s3/your-bucket/orgs/demo-app/example.txt?X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&x-id=PutObject&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=625729a08b95bf1b7ff351a663f3a23c%2F20251004%2Flocal%2Fs3%2Faws4_request&X-Amz-Date=20251004T201956Z&X-Amz-Expires=60&X-Amz-SignedHeaders=host&X-Amz-Signature=1f2ff5cca5aad73d7e1e53224e6b6a3f96f89bbf144a3cdebc2ef9825d15c7de
Local storage response: 400
Response body: <?xml version="1.0" encoding="UTF-8" standalone="yes"?><Error><Resource>your-bucket/orgs/demo-app/example.txt</Resource><Code>InvalidSignature</Code><Message>Unsupported authorization type</Message></Error>

Bingo! So this tells me it has to be an issue with the supabase cli and it was already fixed in the most recent version. But how do I figure out what change caused the issue and what changed fixed it?

git bisect baby. Git bisect is a tool that runs a binary search to find the commit that introduced a bug. You give it a good commit, bad commit, a script to run and it will take care of the rest.

I checked the capgo “Run tests” GitHub Action to see that last time it succeeded to figure out the last known working version. Looks like last time the GHA passed was supabase cli version 2.40.7

I cloned the supabase cli repo into a dedicated repro directory.

code/work/debugging/repro-presigned-url-local-771
➜ git clone https://github.com/supabase/cli.git

I had Copilot generate a quick bash script I could use with git bisect

#!/bin/bash

set -e

# Get the current directory (should be the CLI repo)
CLI_DIR=$(pwd)
TEST_DIR="/Users/tyler/code/work/debugging/repro-presigned-url-local-771"

echo "Building CLI at commit $(git rev-parse --short HEAD)..."

# Build the CLI
go build -o supabase_cli main.go

# Stop any existing supabase instance
./supabase_cli stop --no-backup || true

# Wait a moment for cleanup
sleep 2

# Start supabase with your specific exclusions
echo "Starting Supabase services..."
./supabase_cli start -x imgproxy,studio,mailpit,realtime,postgres-meta,supavisor,studio,logflare,vector,realtime,gotrue,edge-runtime

# Wait for services to be ready
echo "Waiting for services to start..."
sleep 15

# Run the test
echo "Running presigned URL test..."
cd "$TEST_DIR"
pnpm run dev

# Capture the exit code
result=$?

# Stop supabase
echo "Stopping Supabase services..."
cd "$CLI_DIR"
./supabase_cli stop --no-backup

# Return result (0 = good/success, non-zero = bad/failure)
if [ $result -eq 0 ]; then
    echo "✅ Test PASSED - presigned URL works"
    exit 0
else
    echo "❌ Test FAILED - presigned URL broken"
    exit 1
fi

I then started the bisect.

debugging/repro-presigned-url-local-771 git:(main) ✗
➜ git bisect start
status: waiting for both good and bad commits
debugging/repro-presigned-url-local-771 git:(main) ✗
➜ git bisect good v2.40.7
status: waiting for bad commit, 1 good commit known
debugging/repro-presigned-url-local-771 git:(main) ✗
➜ git bisect bad v2.45.5
git bisect run ./test_presigned_url.sh

ccec464034cc90832dcb8022f78419929d5e3f7d is the first bad commit
commit ccec464034cc90832dcb8022f78419929d5e3f7d
Date:   Wed Sep 17 10:03:52 2025 +0800

    feat: support publishable and secret key locally

 internal/start/start.go           |  9 +++++
 internal/start/templates/kong.yml | 71 ++++++++++++++++++++++++++++++++++++++-
 internal/status/status.go         | 28 ++++++++++++---
 pkg/config/apikeys.go             |  7 ++++
 pkg/config/auth.go                |  2 ++
 5 files changed, 112 insertions(+), 5 deletions(-)
bisect found first bad commit

Once I found the problematic commit, I wanted to pinpoint exactly when the issue was resolved. To do this, I flipped the logic in my repro script, now it exits with 0 when the test fails and 1 when it succeeds. This way, git bisect can help me track down the first commit where things start working again.

debugging/repro-presigned-url-local-771 git:(main) ✗
➜ git bisect start
status: waiting for both good and bad commits
debugging/repro-presigned-url-local-771 git:(main) ✗
➜ git bisect good v2.45.5
status: waiting for bad commit, 1 good commit known
debugging/repro-presigned-url-local-771 git:(main) ✗
➜ git bisect bad develop
debugging/repro-presigned-url-local-771 git:(main) ✗
➜ git bisect run ./test_presigned_url.sh

f5e1774848af5f97f286cae5d41b2aa3ef050830 is the first bad commit
commit f5e1774848af5f97f286cae5d41b2aa3ef050830
Date:   Thu Sep 25 17:17:29 2025 +0800

    fix: replace auth header for backwards compatibility

 internal/start/start.go           | 11 ++++++++---
 internal/start/templates/kong.yml | 28 +++++++++++++++-------------
 2 files changed, 23 insertions(+), 16 deletions(-)
bisect found first bad commit

Ah, the good ole fix: commit. We were able to easily close the issue by letting the user know to upgrade to the most recent version.

Next time you want to find the commit where it all went wrong (or right) try out git bisect.

Change Computing Forever:

Computing is our shared passion; our calling is to advance the state of the art, bringing those advances to the broadest possible audience.

This is part of Oxide’s mission statement. I believe their podcast, Oxide and Friends, will have a bigger impact on computing than their rack scale computer.

Kramer: Do you ever yearn?
George: Yearn? Do I yearn?
Kramer: I yearn.
George: You yearn.
Kramer: Oh, yes. Yes, I yearn. Often, I… I sit… and yearn. Have you yearned?

Like Kramer, I yearn.

I consider myself a young technologist. I’ve been programming professionally for 1.5 years, recreationally for 4 years. I made the switch from finance into tech July 2021, well after everything had gone remote. Ever since, I felt something was missing.

I yearned for the lunchtime chats, spontaneous whiteboard sessions, after work drinks, and office games. The Oxide and Friends podcast has helped fill this gap, recreating these conversations I felt I was missing. A unique format where the podcast is live and listeners can participate in chat.

The topics span production war stories, hiring processes, culture, security incidents, baseball, silicon valley lore, heterogeneous computing, operating systems, rust and much much more.

The podcast has given me a new found interest in computing history, a greater appreciation for hardware and deepened my love for technology. Inspiring other technologists, that is how you change computing forever.

Not everyone can buy an Oxide computer but they can listen to Oxide and Friends.

Thank you to all those involved in making the Oxide and Friends podcast.

I understand many people just want access to the underlying data and not my beautiful chart. That’s okay, I understand. I’ve added a new subdomain data.pypacktrends.com for just this reason.

Now, the official PyPI download data is already publicly available in BigQuery thanks to the Linehaul project. The catch is you still need a BigQuery account and, if you’re not careful, you can end up with a surprise bill.

My export is aggregated to weekly downloads. It’s less granular than Linehaul, but it’s free and no sign up required.

Here is an example with DuckDB:

set variable list_of_files = (
  select 
    array_agg(url) as urls 
  from 
    read_parquet('https://data.pypacktrends.com/pypi-weekly-downloads/manifest.parquet')
);

select 
  * 
from 
  read_parquet(getvariable('list_of_files'), filename := True);

I would have loved to allow for file globbing patterns with DuckDB but because I am using R2 Public Buckets with a custom domain, file globbing doesn’t work like it does with the S3 API.

I’ve opted for this approach vs. adding a data API, since with static files I don’t have to worry about bots overwhelming my precious $5 VPS.

Instead I provide a manifest.parquet file. This file has two columns week and url. You can use this file to construct an array of parquet files to fetch. If you only want certain weeks you can add a filter:

set variable list_of_files = (
  select 
    array_agg(url) as urls 
  from 
    read_parquet('https://data.pypacktrends.com/pypi-weekly-downloads/manifest.parquet')
  where
    week = '2025-08-11'
);

The dataset updates automatically every Monday at 10:00am UTC via a GitHub Action.

Here is what the schema of the parquet files look like:

If you want more documentation on how this dataset is generated checkout dbtdocs.pypacktrends.com.

I hope you find the data useful!

Inbox zero is a methodology of email management with the goal of having zero emails in your inbox. For me it’s simple, I treat my inbox like a todo list. If an email comes in and requires an action either I do it or leave it in my inbox until I do. If no action is required I either delete it or move it to a folder if it’s something I might want to reference later.

I do that exact same thing for consuming the internet, which for the sake of this blog post I am deeming “Internet Zero.”

But the internet doesn’t have an inbox… or does it? Anything you’re subscribed to is your inbox. RSS Feeds, Podcasts, YouTube Channels, Newsletters etc. When something new is posted that piece of content is now in your inbox.

When something is posted in my internet inbox it’s time for me to decide, is this something I want to consume or not? If so I file it away to a “consume it later” folder. Most of the time I don’t consume new content as I discover it but rather when I have dedicated time to do so. If it’s not something I want to consume I delete it. If I find I have been deleting a lot of content from a particular source I unfollow.

What about discovery? The problem with only consuming content I subscribe to is I am not exposing myself to content outside of that circle.

Personally I use X, Discord groups, Bluesky and Hackernews as discovery mechanisms. Hackernews is nice because I mainly limit myself to the front page. Discord groups are nice as well because they don’t expose you to an endless feed.

On the other hand, X & Bluesky I use the “For You” feeds which are unbounded. Refresh on and more content appears. It’s endless content like this that can lead to doom scrolling where you find yourself wondering why you have been consuming 3 hours of GCP & Cloudflare outage memes. This is why I have time limits on these sources because there is no real “end” to them.

I stay diligent with the posts I interact with and people I follow so I don’t upset the algo. When I find content I want to consume through these discovery sources they get saved straight to my later library, bypassing my inbox.

That’s the gist. When I have spare time now, instead of heading straight to discovery sources, I go to my later library first or check my internet inbox for new content. I created a lists page where you can check out some of my personal favorites and what I’m currently following.

Much of the hype about AI tools for programming often comes from two ends of a spectrum, those who don’t know programming and those who already have deep expertise.

People who don’t know how to program are blown away because they can accomplish something that was unfeasible to them before.

The experts, on the other hand, can quickly discern whether the output is good or bad. AI removes much of the programming toil so they can focus on the bigger picture.

Couple of quotes that really stuck with me.

Simon Willison
“LLMs amplify existing expertise”
Source

Charlie Marsh
“These tools are a magnifier. But if I were using them without my existing programming experience, it’s so obvious to me that I’d be generating mountains of garbage. And I probably wouldn’t even see it.”
Source

Ankur Goyal
“AI seems to multiply the quality of code that someone would write without AI.

Bad programmer AI = Lots of bad code*

Medium programmer AI = Lots of medium code*

Great programmer AI = Lots of great code”*
Source

What if you’re not already great but have aspirations to be great?

Or the better question is, can AI help you become great?

I think so, as long as you ruthlessly question it. If you use AI to truly understand.

I am not talking about the hand waving, you get the gist, surface level understanding. You’ll know if you truly understand something if you don’t need AI.

You’re not going to become an expert from vibe coding. Vibe coding is the exact opposite of this. Vibe coding is all about forgetting the code even exists.

Some may not have the goal of becoming a great programmer, then by all means, give into the vibes.

Some may argue AI will become so advanced knowing how to program will become meaningless.

Even so, if you had two people, both can use AI, but one knows nothing about programming and the other is an expert in programming, who do you think will do better?

If everyone is going to access to these AI tools in the future, how will you stand out?

I know how I am. By being the one who truly understands.

There’s nothing more exhilarating than knowing one SQL query could end up costing you $2,750. But if you don’t like living on the edge, here are some techniques to help reduce your query costs.

BigQuery’s On-demand pricing model costs $6.25 per TiB, with the first TiB per month being free. I have been building PyPack Trends, a web app to compare Python package downloads inspired by npm trends. This app served as a learning exercise to explore web development.

A core part of this app is being able to cost-effectively serve the PyPI package downloads dataset. The dataset is made available on BigQuery for free, thanks to the Linehaul project. The downloads dataset alone is 440TB, which comes out to $2,750 if you run a simple select * from bigquery-public-data.pypi.file_downloads.

We can confirm this by either running a dry run query or if you’re using BigQuery studio, in the upper right-hand corner of the editor, it will display the estimated amount of bytes to be scanned if you run the query.

This is where understanding how the internals of your database can help you write cost-effective and performant queries.

One way to reduce the amount of data scanned is by selecting only the columns you need. BigQuery stores table data in a columnar format. This means when we select only one column, BigQuery will only scan that column’s worth of data. If I changed the query from doing a select * from bigquery-public-data.pypi.file_downloads to select project from bigquery-public-data.pypi.file_downloads this drops the data scanned from 440TB to 13.39TB which is $2,666 less.

Another way to reduce the amount of data scanned is by taking advantage of partition pruning. A partitioned table divides the table into “chunks” called partitions. A standard column to partition on is typically a date column such as order_date, in our case, the PyPI file downloads table is partitioned on the timestamp column representing download time. This query will only scan 463.50 GB of data, select * from bigquery-public-data.pypi.file_downloads where timestamp_trunc(timestamp, day) = timestamp("2024-12-14").

While partitioning divides the data into smaller segments, clustering defines the table’s sort order. The formal definition from the BigQuery docs

A clustered column is a user-defined table property that sorts storage blocks based on the values in the clustered columns.

Like partition pruning, BigQuery will also prune these blocks when filtering on this column. Running the query select * from bigquery-public-data.pypi.file_downloads where project = 'duckdb' results in 762.27 GB scanned.

Columns commonly used for clustering are those frequently used in join operations. This image does a great job visualizing table partitioning and clustering:

Even with all these techniques implemented, BigQuery is not designed for frequent small-point queries targeting an individual row but rather large aggregate queries running across many rows. The terms to often describe these workloads are OLTP and OLAP.

Further optimization can be done by incrementally pre-aggregating the data and then exporting it to another more optimized database for OLTP workloads.

Daily download counts are too “nosiy” making it look like there are dramatic increases and decreases in daily downloads for a given package. To help smooth out the curve, I opted to pre-aggregate weekly per package.

Incremental processing is essential here because once we calculate one week’s download counts there should be no need to recompute that data again. To calculate the weekly downloads per project takes ~230 GB scanned. Ran 4 times a month would result in 80 GB shy of the free 1 TB per month. Perfect.

select
    project,
    date_trunc(date(timestamp), week (monday)) as package_downloaded_week,
    count(*)                                   as downloads
from
    bigquery-public-data.pypi.file_downloads
where true
    and timestamp >= '2024-12-02'
    and timestamp < '2024-12-09'
group by
    1, 2

I prefer to use a tool like dbt to manage these queries, as it helps apply software engineering best practices to SQL code. Dbt even has a built-in incremental materialization method, which materializes as a table in your database and only transforms the data you tell dbt to filter for.

A best practice to help minimize the chance of running an expensive query is setting the maximum_bytes_billed in your dbt profile. Here is how I have configured my dev profile.

  target: dev
  outputs:
    dev:
      type: bigquery
      method: oauth
      project: pypacktrends-prod
      dataset: "dbt_{{ env_var('BIGQUERY_USER') }}"
      priority: batch
      threads: 4
      timeout_seconds: 300
      maximum_bytes_billed: 300000000000

I have also developed a dbt macro called pypi_package_filter which checks the dbt target to see if it should filter for only a subset of packages. For the dev and ci targets I don’t need to be querying the full range of packages.

{% macro pypi_package_filter(column_name) -%}
    {% set package_list = [
        'dask',
        'datafusion',
        'duckdb',
        'getdaft',
        'ibis-framework',
        'pandas',
        'polars',
        'pyspark'
    ] %}

    {%- if target.name != 'prod' -%}
        {{ column_name }} in ('{{ package_list | join("', '") }}')
    {%- else -%}
        true
    {%- endif -%}

{% endmacro %}

with
source as (
    select * from {{ source('pypi', 'file_downloads') }}
),

renamed as (
    select
        timestamp    as package_downloaded_at,
        country_code as package_download_country_code,
        url          as package_download_url_path,
        project      as package_name,
        file         as package_download_file_details,
        details      as package_download_details,
        tls_protocol as package_download_tls_protocol,
        tls_cipher   as package_download_tls_cipher
    from
        source
    where 
        {{ pypi_package_filter('project') }}
)

select * from renamed

Lastly, I use dbt clone which takes advantage of BigQuery’s table clone feature.

A table clone is a lightweight, writable copy of another table (called the base table). You are only charged for storage of data in the table clone that differs from the base table, so initially there is no storage cost for a table clone. Other than the billing model for storage, and some additional metadata for the base table, a table clone is similar to a standard table—you can query it, make a copy of it, delete it, and so on.

This allows me to create a full copy of the expensive weekly downloads model into my own dev dataset at zero cost without having to recreate it from scratch. This feature requires access to the production dbt manifest JSON file, which fully represents your dbt project’s resources.

I run my dbt jobs through GitHub Actions and save the generated manifest file to my VPS ( I even have the dbt docs publicly available at dbtdocs.pypacktrends.com ). To retrieve the manifest file I rsync the files to my local machine.

After the data is pre-aggregated, I sync the data to an SQLite file on the VPS, essentially using SQLite as a read-only cache. I’m still experimenting with ways to relax SQLite settings to improve query performance, given the heavy read workload with infrequent writes ( once a week). Data reliability is also not a concern here as the source of truth is BigQuery, and I can always resync the data if I have to.

The downside of all this is that I no longer get the thrill when I run my queries. In tech, there are always tradeoffs.

To learn about web development, I set out to build a production grade web app.

What does “production grade” mean? For me, it means the app includes:

• Tests
• Linting, formatting, and type-checking
• Application monitoring
• Observability
• Logging
• Infrastructure as Code
• Automated deployment pipeline
• Zero downtime deploys
• Secrets management
• Documentation
• Containerization

Before building the web app, I started by creating the deployment pipeline. When working on a project, it bothers me if I don’t know how I’ll deploy it. I want to know when the final commit is merged into main, the app will be automatically deployed within minutes, live for the world to see.

One of the first things I set up for a new project is pre-commit and GitHub Actions. GitHub Actions can run workflows triggered by repo activity, like a pull request. Pre-commit is a tool for managing git commit hooks to perform actions before committing.

Pre-commit hooks can be a polarizing topic, I like the tool because it conveniently packages other tools to enforce linting, type checking, formatting etc. on your code. Whether or not these are ran locally, I use the pre-commit/action to enforce them on PRs.

name: Test

on:
  push:
    branches:
      - main
  pull_request:
    types:
      - opened
      - synchronize

jobs:
  pre-commit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version-file: ".python-version"
      - uses: pre-commit/action@v3.0.1

I prefer to enable them locally so I know ahead of time if this action will fail. Most pre-commit hooks will auto fix the errors they detect anyway.

Containerizing the app gives it flexibility on where it can be deployed. With the recent rise in self-hosting, I was drawn to using a plain old VPS. While many PaaS and cloud providers make deploying apps trivial, it’s tough to appreciate the value they add without experiencing solutions without all the bells and whistles. I don’t want to be a pink elephant¹.

Since I planned to have at least two containers running on this VPS, I needed a way to orchestrate them. Tools like Kubernetes or Docker Swarm felt too heavy for my needs, so Docker Compose felt like the right solution, especially as availability isn’t critical for this app. One nice benefit of Docker Compose is how easy it’s to run locally. Using the same technology for both production and development provides numerous benefits.

I knew I wouldn’t set up my VPS through ClickOps. Instead, I used Pulumi for infrastructure as code and secrets management. Pulumi does a lot for my app, it handles building and publishing the image to ghcr.io, generating SSH keys, provisioning the VPS, setting up DNS records, configuring email routing, and running remote commands. The remote command triggers on any image changes and runs the update_service.sh script. ( more on that later)

On startup, the VPS runs a cloud-init script which configures the VPS to disallow password login, runs apt update apt upgrade, installs docker, installs tailscale, installs uv, clones repo, creates logging directory, starts containers, and configures ufw.

Pulumi is also great because I can use the same programming language for my IaC as I use for the backend. This lets me use additional packages, like jinja2, to turn the cloud-init file into a template, allowing Pulumi to inject configuration values.

On each PR, pulumi preview runs via GitHub Actions and outputs the summary as a PR comment, so I know what changes will occur when the PR is merged. After merging the PR, this triggers another GitHub action to run pulumi up to deploy any changes.

When I first got the update_service.sh script to work, it felt like magic. The script itself is inspired by a great video from Tom Delalande, The cloud is over-engineered and overpriced. This script is what enables zero-downtime deploys. It first pulls the new image from ghcr.io which makes the old image “dangling”. We can find the image ID of dangling image by filtering for it.

dangling_image=$(docker images --filter "dangling=true" --filter "reference=${CONTAINER_REGISTRY_PREFIX}/${SERVICE_NAME}" --format "{{.ID}}")

Once we get the old image ID we can get the old container name by running.

old_container=$(docker ps --filter "ancestor=$dangling_image" --format "{{.Names}}")

Now, we can scale up the service to two containers, but we need to specify to not restart the currently running container. The new container will use the latest image we just pulled.

docker compose -f docker-compose.yml up -d --no-deps --scale "$SERVICE_NAME"=2 --no-recreate "$SERVICE_NAME"

Once that’s done, we can get the new container name.

new_container=$(docker ps --filter "ancestor=${FULL_IMAGE_NAME}" --format "{{.Names}}" | grep "$SERVICE_NAME" | tail -n 1)

We need to update our reverse proxy, caddy, to direct traffic to our new container by changing the container name env var.

CONTAINER_NAME=$(echo "${SERVICE_NAME}" | tr '[:lower:]' '[:upper:]')_CONTAINER_NAME
docker exec "$caddy_container" /bin/sh -c "export ${CONTAINER_NAME}=$new_container && caddy reload --config /etc/caddy/Caddyfile --adapter caddyfile"

Lastly, we clean up the old container and image.

docker container rm -f "$old_container"
docker rmi "$dangling_image"

Tailscale was added because it provides a more secure and convenient way to handle SSH. It enables the GitHub Action to run remote commands and allows me to SSH into the VPS if needed. I generated the SSH key in Pulumi only as a backup in case I needed to debug the cloud-init script before Tailscale gets installed. Additionally, DigitalOcean requires you to create a password if you don’t assign an SSH key when provisioning the droplet.

So there you have it, CI/CD for your $5 VPS.

Shifting left in data refers to addressing issues in a data pipeline as close as possible to where the data is produced. Data pipelines are responsible for the movement of data from source to destination, with sources drawn on the left, hence “shifting left.”

Sources could be anything from your application database, CRM, billing system, etc. Destinations are typically a data warehouse, but the data doesn’t stop there. Transformations are then done in the data warehouse to make the data into a more usable format for downstream reporting or enriching data in other systems.

I graduated college with a degree in finance and intended to pursue a career path in investment management. I landed my first job out of college in a back-office role in the investment department of a mid-size insurance company.

This role gave me my first exposure to programming and databases. I automated mundane tasks that took our team several hours every week down to a few minutes. This experience was a revelation, showing me how technology could solve real-world problems. But more importantly, it demystified the black boxes I once thought computers were.

It has given me a new appreciation for all the technology surrounding us. I sit here typing this blog post on my laptop, seeing characters appear on my screen, while those changes are reflected realtime in a local version of my website. This shit is mind-blowing to me.

The most mind-blowing part isn’t the technology itself. It’s the fact that behind all of this technology are people who built it, people who are no different from you or me. This realization prompted me to switch careers.

Going to college for 4 years and then deciding 1 year after to change career paths is no easy decision. I didn’t want to start over, return to school, or take time off to do a bootcamp. Being a young, recent college grad with few responsibilities, I had one major advantage, time. I did not need to become a developer in as little as “3 months”. I was in it for the long haul.

Given my current background, I evaluated my skill set to see which technical roles I could get. The data space appealed to me because it’s the middle ground between the technical and business domains. It’s an area where I could leverage my finance background and newfound technical interests.

Most people view career paths as a ladder with a single axis, you can go up or down. I didn’t view my career that way. Instead, I saw it as a series of diagonal moves, each role being more technical than the last, shifting left and moving up the pipeline.

It’s been over 3 years since I decided to switch career paths. I went from that back-office investment role to a Business Intelligence Developer, Analytics Engineer, and my current role as a Data Engineer. Writing this now, 3 years doesn’t seem like a long time, but boy, it has felt like an eternity.

While recently rewatching my favorite movie, The Shawshank Redemption, Andy Dufresne’s journey resonated with me.

Andy, armed with nothing more than a rock hammer, spent nearly 20 years patiently chipping away at the walls of his prison cell, eventually tunneling a hole that would lead to his escape through a sewer pipeline. What resonates most about Andy’s journey is his patience, persistence, and meticulous nature. It also showcases the slow but steady journey of self-discovery.

Instead of a rock hammer, I have been armed with something much more powerful, a computer and the internet. While Andy was tunneling through walls, I was tunneling through the layers of abstraction, chipping away the black boxes of technology that once felt incomprehensible.

Unlike Andy, who emerged with a clear plan, my journey isn’t complete. I’m still figuring out what I want to do in tech.

I’ve enjoyed my time in various data roles, but something inside of me stills yearns for more. My experience in data was merely a gateway to something else I have yet to discover. I am in no rush though, I’m only year 3 into my 20 year escape.

Confidence is the most valuable thing I have gained over the past three years. Given enough time and pressure, I can learn anything I want. That’s all it really takes is time and pressure. Well, that and a big goddamn poster.

When I started out my career in finance, I learned a valuable lesson while studying for the CFA exam. I learned this from Mark Meldrum, who was the CFA prep provider I used to help study for the exam. One of things he stressed was immersing yourself in the domain.

This can be done through numerous outlets such as watching the news or listening to a podcast. I distinctly remember Mark including a “Jelly Beans” section on his website which has links to movies, documentaries and university lectures. My two favorite resources at the time were Mark Meldrum’s Market Outlook on YouTube and the Bloomberg Surveillance news show.

One major benefit I gained from this was picking up the jargon of the domain. Many jobs have this unspoken language you are expected to know and there is no Duolingo course you can learn from. This language is only learned through experience.

Another key benefit was seeing the concepts being applied in real world scenarios. We’ve all sat in class wondering, “When will I use this?” but this experience eliminated that feeling.

It also transformed me into an active learner. Encountering familiar terminology in practice helped jog my memory and deepen my understanding, as I connected the dots between theory and application.

I carried this same practice when I transitioned into a career in tech. However, I quickly realized immersing yourself in the tech community is much different than the finance world. The news travels in unique ways and there are many pockets of specialized domains.

By far the biggest resource for me has been Twitter. To this day I am shocked by the people I get to interact with on the platform, from creators of well known open source libraries to CEOs. It has been arguable the most important part of my career because of the connections I have made.

Twitter is what you make of it though. The algorithm feeds you the content you interact with. I strictly use the “For You” feed so I get exposed to content outside of my circle. To have a good feed though it’s important to curate it. One of the first things I did when I decided to go all in on immersing myself in tech was to unfollow any account posting unrelated content.

If you find a hard time figuring out who to follow I recommend finding a few people you find interesting and looking at who they follow. There are too many people I want to call out so you can check out my following list here.

An interesting aspect of tech is the many communities within it. I initially became involved in the data community, primarily engaging through the dbt and Locally Optimistic Slack. The dbt slack was how I discovered the Senior Analytics Engineer role at Nasdaq. I reached out to the hiring manager who posted the position in the jobs channel to express my interest which helped me get pass the initial resume screening phase.

This is another key benefit you will find while immersing yourself in your domain. You’ll make invaluable connections that can open doors to opportunities you might not have encountered otherwise.

Every community seems to have there go to hang out spot. I encourage you to find the hang out spot in your community and if there isn’t one create one!

They are many other ways to immerse yourself as well. Watch a documentary, read a book, listen to podcast. If you are looking for some resources to help immerse yourself checkout my newly created Lists, where I shared the podcasts & blogs I currently follow.

After years of immersing yourself, you’ll be shocked at how in tune you feel with the domain.