Dan Snow

OpenTimes: Free travel times between U.S. Census geographies

Mon, 17 Mar 2025 00:00:00 +0000

Today I’m launching OpenTimes, a free database of pre-computed, point-to-point travel times between major U.S. Census geographies. In addition to letting you visualize travel isochrones, OpenTimes also lets you download massive amounts of travel time data for free and with no limits. Visit the dedicated about page to learn more about the project.

The primary goal here is to enable research and fill a gap I noticed in the open-source spatial ecosystem. Researchers (social scientists, economists) use large travel time matrices to quantify things like access to healthcare, but they often end up paying Google or Esri for the necessary data. By pre-calculating times between commonly-used research geographies (i.e. Census) and then making those times easily accessible via SQL, I hope to make large-scale accessibility research cheaper and simpler.

OpenTimes covers all 50 states (and D.C.), 3 travel modes (driving, biking, and walking), and 6 Census geographies. Here’s what the actual data looks like as a table:

origin_id	destination_id	duration_sec
060750328021	060750328021	0
060750328021	060750328023	284.1
060750328021	060750328022	322.5
060750328021	060750326023	479.7
…	…	…

And here’s that same data on the homepage map:

OpenTimes also has some interesting technical stuff going on, most of which I haven’t seen replicated elsewhere:

The entire OpenTimes backend is just static Parquet files on Cloudflare’s R2. There’s no RDBMS or running service, just files and a CDN. The whole thing costs about $10/month to host and costs nothing to serve. In my opinion, this is a great way to serve infrequently updated, large public datasets at low cost (as long as you partition the files correctly).
All travel times were calculated by pre-building the inputs (OSM, OSRM networks) and then distributing the compute over hundreds of GitHub Actions jobs. This worked shockingly well for this specific workload (and was also completely free).
The query layer uses a single DuckDB database file with views that point to static Parquet files via HTTP. This lets you query a table with hundreds of billions of records after downloading just the ~5MB pointer file.
The map frontend uses a Javascript Parquet library called hyparquet to query the same static Parquet files as DuckDB. Once the Parquet files are cached the map is actually super responsive.

I built most of OpenTimes during a 6-week stint at the Recurse Center, where it was my main project. Many thanks to the wonderful folks there!

Grafana creates strange DNS traffic for releases >= 10.2.3

Sun, 16 Jun 2024 00:00:00 +0000

Edit: This is resolved and nothing malicious is happening. See here for the GitHub issue update. The issue here appears to be the result of Safari pre-fetching DNS for Grafana’s forums, which are linked on the Grafana landing page and do use advertising scripts.

I use AdGuard Home for DNS-based ad-blocking and recently noticed some suspicious looking traffic getting consistently blocked in my DNS query logs. Lots of outbound queries to fishy Russian and Greek ad servers (see the full list below):

After some investigating, I found that these requests were generated by Grafana, which is visualization software I use to monitor server health. Specifically, the requests are generated only by desktop Safari clients that load any local Grafana page from Docker image 10.2.3 or above (including latest at the time of this posting). Some details:

The requests are triggered by a refresh (clicking in the UI does not trigger them).
All Safari extensions are disabled.
Safari version 17.4.1, macOS version 14.4.1.
Safari iOS clients do not produce the requests.
Chromium and Firefox do not produce the requests.
Grafana has no plugins installed.
Provisioning from YAML on a fresh image does not solve the issue.
tcpdump confirms these are client-side DNS requests coming from my MacBook.
Docker images 10.2.2 and below do not produce the unusual requests, they only perform DNS queries for grafana.com.

I confirmed these results using an entirely separate MacBook client, and they’ve also been confirmed independently on reddit. I reached out to the Grafana security team on April 3rd and again on April 8th, but received no response. I also created a GitHub issue.

I’m not qualified to speculate about the implications here, security or otherwise, but I do hope someone looks into this more. Even if the behavior isn’t malicious, it’s still incredibly strange. Crossing my fingers that this is a misconfiguration and not something worse, like a supply chain attack or the start of ads in Grafana.

🤞

Queried domains

This is the full list of domains queried when refreshing a page. The list seems to be static as it hasn’t changed between 10.2.3 and latest, as far as I can tell.

ad.letmeads.com
ad.planbplus.co.kr
ad2.trafficgate.net
adserv.ontek.com.tr
adserver.html.it
adv.imadrep.co.kr
advmanager.techfun.pl
affiliazioniads.snai.it
afftrk.altex.ro
app.readpeak.com
axiabanners.exodus.gr
bd742.com
blackfridaysales.ro
click.hotlog.ru
event.2performant.com
febrare.ru
g1.v.fwmrm.net
hitcounter.ru
interactive.forthnet.gr
izlenzi.com
l.profitshare.ro
pay4results24.eu
promo.vador.com
telegram.me
top.mail.ru
utimg.ru
www.installads.net
www.salidzini.lv
www.stumbleupon.com
www.trizer.pl
xltube.nl

Exporting Jellyfin playback stats to Prometheus

Sat, 25 Nov 2023 00:00:00 +0000

GitHub gist for source code

Edit: It turns out there’s a much simpler way to do this. See the update below.

I run a small Jellyfin server at home to watch movies and other media I’ve collected over the years. Recently, in order to debug some playback/networking issues, I wanted a way to answer the following questions via Prometheus and Grafana:

What media is Jellyfin currently playing?
Who is playing that media?
How much bandwidth is the playback using?

Basically, I wanted to make this graph in Grafana:

Your browser does not support the video tag.

This turned out to be way harder than I thought, but I eventually got it working using the Jellyfin REST API and Prometheus JSON exporter. This post walks through what I did.

The Jellyfin REST API

Jellyfin has a built-in Prometheus endpoint at $BASE_URL/metrics, but it’s extremely bare-bones and didn’t return the playback stats I wanted. There are also full-featured statistics applications like Jellystat, but they seemed like overkill for my needs.

Eventually, I stumbled across the Jellyfin REST API. It’s an extensive API that lets you basically do whatever you want in Jellyfin. However, it’s very poorly documented and I couldn’t find many examples of actually using it to do stuff.

This blog post by James Harvey (creator of Finamp) turned out to be a lifesaver. It walks through the authorization setup and gives an overview of some of the API endpoints. I also found this Gist that walks through the various Jellyfin API authorization methods. I used these resources to construct a simple curl request:

TOKEN=example-token-here
BASE_URL=https://jellyfin.example.com

curl -X "GET" \
  "$BASE_URL/Sessions" \
  -H "Authorization: MediaBrowser Token=$TOKEN" |
  jq '.[]'

The TOKEN value here is a Jellyfin API key from Admin Dashboard > Advanced > API Keys. The request outputs a JSON object with some playback stats, capabilities, last seen times, etc. Here’s the result nicely parsed with jq:

{
  "PlayState": {
    ...
  },
  "AdditionalUsers": [],
  "Capabilities": {
    ...
  },
  "RemoteEndPoint": "1.1.1.1",
  "Id": ...,
  "UserId": ...,
  "UserName": "Dan",
  "Client": "Jellyfin Roku",
  "LastActivityDate": "2023-10-29T01:26:17.2969131Z",
  "LastPlaybackCheckIn": "0001-01-01T00:00:00.0000000Z",
  "DeviceName": "50 TCL Roku TV",
  ...
}

This is useful stuff, but not exactly the nicely summarized playback information I was hoping for. It’s possible to use the REST API to gather all the relevant stats, but it takes multiple API calls to different endpoints. There turned out to be an easier way.

Fetching aggregate stats

The Jellyfin Playback Reporting Plugin (PRP) aggregates and exposes playback stats via a dashboard on the Jellyfin admin panel. These stats are stored in a SQLite database in the Jellyfin data directory. Querying this database lets you get basically any playback stats you want. All I had to do is figure out a way to query the database remotely.

Fortunately, installing the PRP also adds dedicated API endpoints at /user_usage_stats/. The most useful/relevant of these is the POST endpoint /user_usage_stats/submit_custom_query, which lets you pass any arbitrary SQL query to the underlying SQLite database.

After lots of trial and error, I was able to update the curl request to hit the new endpoint:

curl -X "POST" \
  "$BASE_URL/user_usage_stats/submit_custom_query" \
  -H "accept: application/json" \
  -H "Content-Type: application/json" \
  -H "Authorization: MediaBrowser Token=$TOKEN" \
  -d '{
    "CustomQueryString": "SELECT * FROM PlaybackActivity LIMIT 2",
    "ReplaceUserId": true
  }' |

  # The extra jq stuff here is just to parse
  # the output into something more readable
  jq '.results | map ({ "date":.[0],
    "user":.[1],
    "item_id":.[2],
    "type":.[3],
    "item_name":.[4],
    "method":.[5],
    "client":.[6],
    "device":.[7],
    "duration":.[8]
  })'

Here’s what the parsed output looked like:

[
  {
    "DateCreated": "2023-10-29 20:08:19.1115194",
    "UserName": "Dan",
    "ItemId": ...,
    "ItemType": "Movie",
    "ItemName": "She's the Man",
    "PlaybackMethod": "Transcode (v:direct a:aac)",
    "ClientName": "Jellyfin Mobile (iOS)",
    "DeviceName": "iPhone",
    "PlayDuration": "1088"
  },
  {
    "DateCreated": "2023-10-30 22:26:29.7812735",
    "UserName": "Dan",
    "ItemId": ...,
    "ItemType": "Movie",
    "ItemName": "My Neighbor Totoro",
    "PlaybackMethod": "Transcode (v:direct a:aac)",
    "ClientName": "Jellyfin Mobile (iOS)",
    "DeviceName": "iPhone",
    "PlayDuration": "33"
  }
]

Note that the test query above just returned a couple arbitrary rows from the PlaybackActivity table. I needed a more complicated SQLite query to fetch “live” stats. I came up with the query below, which only returns rows for things that are currently playing and also includes a timestamp for use in Prometheus.

SELECT
    -- Return the timestamp of the record/stream to
    -- replace the default Prometheus timestamp
    unixepoch(DATETIME(
        DateCreated, "+" || PlayDuration || " seconds"
    )) AS Timestamp,
    UserId,
    ItemType,
    ItemName,
    PlaybackMethod,
    ClientName,
    DeviceName,
    PlayDuration,
    TRUE AS IsActive
FROM PlaybackActivity
-- Only return "active" streams, i.e. ones playing
-- in the last 60 seconds
WHERE DATETIME(DateCreated, "+" || PlayDuration || " seconds") >=
    DATETIME("now", "localtime", "-60 seconds")

Hitting the /user_usage_stats/submit_custom_query endpoint with this query effectively returns all currently playing Jellyfin media, which is exactly what I needed. The final piece of the puzzle was figuring out how to export the results to Prometheus.

If your Grafana instance is on the same machine as Jellyfin, it might be easier to skip Prometheus entirely. Simply use the Grafana SQLite plugin to add the Playback Reporting Plugin database file as a Grafana datasource directly.

Prometheus JSON exporter

This part of the project was actually the easiest. I used the community json_exporter to load JSON output from the custom query into Prometheus. I basically just translated the curl request above into a Prometheus module. There are lots of ways to do this, but here’s one possible example. Note the comments, they’re important:

modules:
  jellyfin:
    headers:
      # The Token value here needs to be the API key from the
      # curl request. I hard-coded the value but I'm sure there's
      # a better way
      Authorization: MediaBrowser Token=ADD_TOKEN_HERE
      Content-Type: application/json
      accept: application/json

    # This is the query from above, but condensed to a single line
    # NOTE: The string escaping/lack of newlines is
    # required for the exporter to work
    body:
      content: '{"CustomQueryString": "SELECT unixepoch(DATETIME(DateCreated, \"+\" || PlayDuration || \" seconds\")) AS Timestamp, UserId, ItemType, ItemName, PlaybackMethod, ClientName, DeviceName, PlayDuration, TRUE AS IsActive FROM PlaybackActivity WHERE DATETIME(DateCreated, \"+\" || PlayDuration || \" seconds\") >= DATETIME(\"now\", \"localtime\", \"-60 seconds\")", "ReplaceUserId": true}'

    metrics:
    - name: jellyfin
      type: object
      help: User playback metrics from Jellyfin
      path: '{ .results[*] }'
      # Optionally use the SQL-generated timestamp instead of
      # the Prometheus "query collected at" timestamp
      epochTimestamp: '{ [0] }'
      # The JSON parsing here is basically identical to the jq
      # call above, just using JSONPath syntax instead
      labels:
        user_name: '{ [1] }'
        item_type: '{ [2] }'
        item_name: '{ [3] }'
        playback_method: '{ [4] }'
        client_name: '{ [5] }'
        device_name: '{ [6] }'
      values:
        play_duration: '{ [7] }'

Finally, I added the exporter to my Prometheus config file using the multi-target exporter pattern with relabeling and modules:

scrape_configs:
  - job_name: json
    metrics_path: /probe
    params:
      module: [jellyfin] # The module from above
    static_configs:
      - targets:
        # The Jellyfin PRP custom query API endpoint
        - https://jellyfin.example.com/user_usage_stats/submit_custom_query
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: HOSTNAME:9115 # The exporter's hostname:port

And that’s it! A curl request to http://$HOSTNAME:9115/probe now returns Prometheus metrics for any currently playing Jellyfin media, along with the user information and transcoding settings of the player. Note that metrics only exist if media is currently playing, otherwise the return body will be blank.

Translating the resulting metrics into useable Grafana graphs is a project by itself, but it is possible with the extensive use of transformations. I’ll save that topic for a future post.

Update (2024-02-23)

After digging around in the API docs for awhile, I realized that using the Playback Reporting Plugin is totally unnecessary. The Sessions/ endpoint returns everything you need and works more consistently.

The trick is to grab stuff from the NowPlayingItem key of each session, which only exists when something is playing. Here’s the updated YAML to reflect the new target:

modules:
  jellyfin:
    headers:
      Authorization: MediaBrowser Token=ADD_TOKEN_HERE
      Content-Type: application/json
      accept: application/json

    # The body is no longer needed since this is now GET

    # This will return all active sessions regardless of
    # whether something is playing. You can use a combination
    # of label and value filters in Grafana to only get actively
    # playing sessions
    metrics:
    - name: jellyfin
      type: object
      help: User playback metrics from Jellyfin
      # Only look at sessions with the NowPlayingItem key
      path: '{ [?(@.NowPlayingItem)] }'
      labels:
        user_name: '{ .UserName }'
        # Use PromQL label_join and label_replace to concatenate
        # these values into a nice item description
        item_type: '{ .NowPlayingItem.Type }'
        item_name: '{ .NowPlayingItem.Name }'
        item_path: '{ .NowPlayingItem.Path }'
        series_name: '{ .NowPlayingItem.SeriesName }'
        episode_index: 'e{ .NowPlayingItem.IndexNumber }'
        season_index: 's{ .NowPlayingItem.ParentIndexNumber }'
        client_name: '{ .Client }'
        device_name: '{ .DeviceName }'
        # Include the unique session ID in case the above
        # labels aren't a unique combination
        session_id: '{ .Id }'
      values:
        is_paused: '{ .PlayState.IsPaused }'

And the Prometheus config:

scrape_configs:
  - job_name: json
    metrics_path: /probe
    params:
      module: [jellyfin]
    static_configs:
      - targets:
        # Change the target to point at the sessions endpoint
        - https://jellyfin.example.com/Sessions
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: HOSTNAME:9115

Comparing 500 billion rows with data.table

Sat, 17 Jun 2023 00:00:00 +0000

Recently, I’ve been working on a project that requires the element-wise comparison of many rows. The goal is to calculate a sort of similarity score between two tables. A simplified version of the data/problem looks like this:

Table X

ID	V1	V2	V3	V4	V5
X1	9	3	2	4	5
X2	9	1	2	1	1

Table Y

ID	V1	V2	V3	V4	V5
Y1	9	3	3	1	1
Y2	1	1	1	1	1
Y3	9	9	7	9	8

Each row of Table X is compared to each row of Table Y and the elements are replaced with a boolean. This happens element-wise, so X1, V1 is compared to Y1, V1, X1, V2 is compared to Y1, V2, etc. The comparison of row X1 to all rows in Table Y looks like this:

ID_X	ID_Y	V1	V2	V3	V4	V5
X1	Y1	TRUE	TRUE	FALSE	FALSE	FALSE
X1	Y2	FALSE	FALSE	FALSE	FALSE	FALSE
X1	Y3	TRUE	FALSE	FALSE	FALSE	FALSE

Each row of this boolean table is multiplied by a vector of pre-determined weights (the same length as the number of columns) and then summed to determine the final similarity between row X1 and the rows of Table Y. If the weights are 5, 4, 3, 2, 1, then the resulting table becomes:

ID_X	ID_Y	V1	V2
X1	Y1	5	4
X1	Y2	0	0
X1	Y3	5	0

Summed by row, the total similarity score between row X1 and all of Table Y is:

ID_X	ID_Y	score
X1	Y1	9
X1	Y2	0
X1	Y3	5

And for all rows of Table X and all rows of Table Y:

ID_X	ID_Y	score
X1	Y1	9
X1	Y2	0
X1	Y3	5
X2	Y1	8
X2	Y2	7
X2	Y3	5

Great, seems simple enough, right? We can loop through the rows of Table X and sum the resulting rows for each comparison to Table Y. What’s the problem?

Problem

The real table X and Y are big, and the product of their comparison is absolute huge.

Table X is 1.1M rows by 1,500 columns.
Table Y is 450K rows by 1,500 columns.

Comparing just one row of Table X to Table Y requires 450K comparisons, and each comparison is between a 1,500 element vector. Comparing all of Table X to Table Y requires 1.1M * 450K comparisons and thus results in around 500 billion similarity scores.

At that scale, many of the obvious solutions to this problem just sort of… break. They quickly run out of memory or have runtimes measured in weeks. In order to make this data useful, we must find a solution that not only runs but that also runs extremely fast.

Caveats

To make the problem slightly more tractable for this post, we will:

Only use R, though there are probably much faster solutions in compiled languages.
Generate much smaller test data than the real data so we can benchmark solutions without waiting for days. Here’s the data generation in R:

library(data.table)
library(microbenchmark)

n_col <- 500
n_rows_x <- 100
n_rows_y <- 10000
weights <- runif(n_col)

# Table X becomes 100 x 500
x_df <- data.table(
  ID = paste0("X", seq_len(n_rows_x)),
  matrix(
    sample.int(n_col, n_col * n_rows_x, replace = TRUE),
    ncol = n_col,
    nrow = n_rows_x
  )
)

# Table Y becomes 10000 x 500
y_df <- data.table(
  ID = paste0("Y", seq_len(n_rows_y)),
  matrix(
    sample.int(n_col, n_col * n_rows_y, replace = TRUE),
    ncol = n_col,
    nrow = n_rows_y
  )
)

Solutions

The goal is to perform a weighted comparison of each row of Table X to each row of Table Y, and to do so quickly and with the maximum possible memory efficiency. Here are some of the solutions I tried as well as the one I eventually landed on.

For loop

The first thing I (or more accurately, my intern) tried was a simple for loop. This is very memory-efficient since it’s basically just looping through columns of existing data and sticking the results into a pre-allocated output matrix. However, it takes around 8 seconds to finish on the test data, meaning it would likely take weeks to finish on the real data. No good.

calc_sim_for <- function(x, y, weights) {
  x_nrow <- nrow(x)
  score <- matrix(0, nrow = x_nrow, ncol = nrow(y))

  for (row in seq_len(x_nrow)) {
    i <- 1
    for (col in colnames(y)) {
      matches <- y[[col]] == x[row, ][[col]]
      score[row, ] <- score[row, ] + weights[i] * matches
      i <- i + 1
    }
  }

  return(t(score))
}

microbenchmark(
  "for" = calc_sim_for(x_df[, -1], y_df[, -1], weights),
  unit = "millisecond",
  times = 5
)

Unit: milliseconds
  expr  min   lq mean median   uq  max neval
   for 7478 7493 7553   7515 7612 7664     5

Apply and matrix ops

Next, I tried using apply() to compare each table column-wise and then using matrix multiplication for the weights. This turned out to be even slower than the for loop.

calc_sim_apply <- function(x, y, weights) {
  x_nrow <- nrow(x)
  score <- matrix(0, nrow = x_nrow, ncol = nrow(y))
  t_y <- as.matrix(t(y))
  m_x <- as.matrix(x)

  for (row in seq_len(x_nrow)) {
    match_matrix <- apply(t_y, MARGIN = 2, "==", m_x[row, ])
    score[row, ] <- weights %*% as.matrix(match_matrix)
  }

  return(t(score))
}

microbenchmark(
  "apply" = calc_sim_apply(x_df[, -1], y_df[, -1], weights),
  unit = "millisecond",
  times = 5
)

Unit: milliseconds
  expr   min    lq  mean median    uq   max neval
 apply 16197 16213 16407  16422 16543 16659     5

Vector recycling

Next, I tried to take advantage of R’s fast matrix operations by comparing each row of Table X to a transposed version of Table Y. This works because vector-to-matrix comparisons are column-wise, and the single row of X (converted to a vector) is recycled automatically. This is much faster than apply() or the for loop, but still isn’t nearly fast enough.

calc_sim_mat <- function(x, y, weights) {
  x_nrow <- nrow(x)
  score <- matrix(0, nrow = x_nrow, ncol = nrow(y))
  t_y <- t(y)

  for (row in seq_len(x_nrow)) {
    vec_x <- as.numeric(x[row, ])
    score[row, ] <- unname(colSums((t_y == vec_x) * weights))
  }

  return(t(score))
}

microbenchmark(
  "mat" = calc_sim_mat(x_df[, -1], y_df[, -1], weights),
  unit = "millisecond",
  times = 5
)

Unit: milliseconds
  expr  min   lq mean median   uq  max neval
   mat 2118 2131 2142   2133 2153 2176     5

Pure data.table

I really like R’s data.table package. These days I find myself reaching for it whenever speed and memory efficiency are a concern. However, I wasn’t sure it was suitable for this problem. After messing around with lapply() and some other options, I found an approach that is incredibly simple and fast. Here’s how it works:

Pivot the input matrices from wide to long using data.table::melt(). The original Table X becomes the table shown below.
Use data.table’s keys to index the resulting long tables.
Join the long Table X and long Table Y on V and IDX.
Replace the value of V with the corresponding weight, then sum for each group of X and Y IDs.

    ID  V IDX
 1: X1 V1   9
 2: X2 V1   9
 3: X1 V2   3
 4: X2 V2   1
 5: X1 V3   2
 6: X2 V3   2
 7: X1 V4   4
 8: X2 V4   1
 9: X1 V5   5
10: X2 V5   1

This approach gains most of its speed by taking advantage of data.table’s indexed joins and aggregations. But just how fast is it?

calc_sim_dt <- function(x, y, id_col, weights) {
  x_nrow <- nrow(x)
  y_nrow <- nrow(y)
  id_col_i <- paste0("i.", id_col)
  names(weights) <- colnames(x)[-1]

  # Pivot X and Y from wide to long, add keys to the results
  x_m <- data.table::melt(
    data = x,
    id.vars = id_col,
    variable.name = "V",
    value.name = "IDX"
  )
  data.table::setkeyv(x_m, cols = c("V", "IDX"), physical = TRUE)
  y_m <- data.table::melt(
    data = y,
    id.vars = id_col,
    variable.name = "V",
    value.name = "IDX"
  )
  data.table::setkeyv(y_m, cols = c("V", "IDX"), physical = TRUE)

  # Join on the pivoted columns, then aggregate to get scores
  out <- y_m[
    x_m,
    on = .(V, IDX),
    nomatch = NULL,
    allow.cartesian = TRUE
  ][
    , c("V", "IDX") := .(weights[V], NULL)
  ][
    , .(score = sum(V)),
    keyby = c(id_col_i, id_col)
  ]

  # Clean up the results
  data.table::setnames(
    x = out,
    old = c(id_col_i, id_col),
    new = c(paste0(id_col, "_X"), paste0(id_col, "_Y"))
  )

  return(out)
}

# Use all threads for data.table
setDTthreads(0)

# Comparing all solutions
microbenchmark(
  "for" = calc_sim_for(x_df[, -1], y_df[, -1], weights),
  "apply" = calc_sim_apply(x_df[, -1], y_df[, -1], weights),
  "mat" = calc_sim_mat(x_df[, -1], y_df[, -1], weights),
  "dt" = calc_sim_dt(x_df, y_df, "ID", weights),
  unit = "millisecond",
  times = 5
)

Unit: milliseconds
  expr   min    lq  mean median    uq   max neval
   for  7547  7651  7802   7856  7887  8067     5
 apply 17223 20289 20026  20447 20697 21476     5
   mat  2169  2204  2214   2214  2217  2266     5
    dt   141   144   168    150   201   206     5

Oh. It’s a full order of magnitude faster than everything else. Looks like data.table is our winning solution!

Conclusion

So, that’s the test data, but what about the original problem with 500 billion rows? Can the data.table solution actually finish running it?

Yes it can! Though with some caveats:

data.table has a limit of 2 ^ 31 rows resulting from a join. To stay below that limit, the input Table X needs to be processed in chunks, which slows things down a bit.
Saving every score between rows isn’t actually necessary. What we’re really after is the top N most similar rows of Table Y given a row in X. This significantly shrinks the required memory and ultimately the size of the output.

I wrote some additional code to handle these caveats and ran it on the full data. Given 1.1M rows in X and 450K rows in Y, each with 1,500 columns, the code took 31H 27M 18S to run on a beefy server (128G RAM, 16 cores of a Xeon Silver 4208 using data.table’s built-in parallelism). Still slow, but totally manageable given the nature of the task.

That said, I’m sure this could be done faster using a lower-level language. I did some quick experiments using Rcpp, but I don’t think I’m skilled enough to beat the times I got with data.table. If anyone does manage to find a faster solution, feel free to email me; I’ll gladly buy you a beer.

Update (2023-06-21)

I owe someone a beer. My friend pointed out that a simple nested for loop in Python using numpy is nearly as fast as the data.table solution.

Click to view setup code

library(reticulate)
x <- as.matrix(x_df[, -1])
y <- as.matrix(y_df[, -1])

import time
import numpy as np

# Import objects from R to numpy using reticulate
x = r.x
y = r.y
w = np.array(r.weights)

# Define janky microbenchmark analogue
def benchmark(func, x, y, w, expr, times):
    exp_j = max(len(expr), 4) + 1
    timings = np.empty(times, np.float32)
    for i in range(times):
        start = time.perf_counter()
        func(x, y, w).shape
        end = time.perf_counter()
        timings[i] = round((end - start) * 100, 3)

    print("Unit: milliseconds")
    print("expr".rjust(exp_j, " ") +
      " min  lq mean median  uq max neval\n",
      expr.rjust(exp_j - 1, " ") +
      str(np.int16(np.min(timings))).rjust(4, " ") +
      str(np.int16(np.quantile(timings, 0.25))).rjust(4, " ") +
      str(np.int16(np.mean(timings))).rjust(5, " ") +
      str(np.int16(np.median(timings))).rjust(7, " ") +
      str(np.int16(np.quantile(timings, 0.75))).rjust(4, " ") +
      str(np.int16(np.max(timings))).rjust(4, " ") +
      str(times).rjust(6, " ")
    )

def calc_sim_py(x, y, w):
    output = np.zeros((len(x), len(y)), np.float64)
    for idx_x, x_row in enumerate(x):
        for idx_y, y_row in enumerate(y):
            output[idx_x][idx_y] = np.sum((x_row == y_row) * w)
    return output

benchmark(calc_sim_py, x, y, w, "py", times = 5)

Unit: milliseconds
  expr min  lq mean median  uq max neval
    py 299 301  302    302 302 305     5

And that compiling the same loop with numba’s @njit decorator reduces the time even further, down to around a half the time of data.table. Pretty wild!

from numba import njit, prange

@njit
def calc_sim_py_njit(x, y, w):
    output = np.zeros((len(x), len(y)), dtype=np.float64)
    for x_idx, x_row in enumerate(x):
        for y_idx, y_row in enumerate(y):
            output[x_idx][y_idx] = np.sum((x_row == y_row) * w)
    return output

benchmark(calc_sim_py_njit, x, y, w, "py_njit", times = 5)

Unit: milliseconds
    expr min  lq mean median  uq max neval
 py_njit  80  80   87     80  80 114     5

Update (2023-06-23)

I owe two beers. My coworker has further sped up the numpy loops using numba’s built-in parallelization. So now we’ve dropped three orders of magnitude from the original R for loop. Can we go even faster?

from numba import njit, prange

@njit(parallel=True, fastmath=True)
def calc_sim_py_par(x, y, w):
    output = np.zeros((len(x), len(y)), dtype=np.float64)
    for x_i in prange(len(x)):
        for y_i in range(len(y)):
            output[x_i][y_i] = np.sum((x[x_i] == y[y_i]) * w)
    return output

benchmark(calc_sim_py_par, x, y, w, "py_par", times = 50)

Unit: milliseconds
    expr min  lq mean median  uq max neval
  py_par   6   7    8      7   8  43    50

Pivoting 100GB text files from long to wide

Mon, 28 Nov 2022 00:00:00 +0000

Recently, I’ve been working on a project involving travel time matrices. These matrices are the output of specialized routing software and are used in social science research, urban planning, and policy analysis. They look like this:

origin	destination	minutes
A	A	0
A	B	9
A	C	15
B	A	10
B	B	0
B	C	21
C	A	13
C	C	0

Where origin and destination are the identifiers of geographic points (usually Census geography centroids) and minutes is the travel time between the points.

This is the “long” travel time matrix format. It’s the typical output of routing software, but instead of 6 rows, each matrix has millions or even billions of rows. The software outputs each matrix as a plain CSV file, which is then compressed to a more manageable size.

The long matrix format is simple and easy to work with, but has some major drawbacks. Mainly, it:

Repeats identifiers (A is repeated 3 times in the matrix above), which costs space.
Makes lookups difficult, since finding a specific origin-destination pair requires either scanning the entire file or using an index (which costs more space).
Doesn’t include pairs missing from the Cartesian product of origins and destinations. Such missing pairs are often useful for debugging routing issues.

We can fix all these issues by converting our long format to “wide” format, with origins as rows and destinations as columns. The wide version of the matrix above looks like this:

origin	A	B	C
A	0	9	15
B	10	0	21
C	13	-	0

This format saves space, compresses better, and shows missing origin-destination pairs. It can also be converted into a modern columnar storage format like Parquet for easy column lookups, built-in compression, and handy metadata.

Problem

Pivoting these matrices is hard.

In most cases, data can be easily pivoted from long to wide using functions from popular data manipulation libraries. However, such libraries typically work with data in-memory, and travel time matrices can be over 100GB uncompressed, making them impossible to fit in-memory on most machines.

As such, we need to find a method to pivot the travel time data without storing it in-memory. Our method should also:

Handle missing pairs i.e. each origin doesn’t go to all destinations.
Handle origin and destination lists of any size. Some matrices have 100K+ destinations for each origin.
Read from a compressed file, do its work, then write to a compressed file. This is critical, as it prevents mangling disks by writing hundreds of gigabytes of uncompressed text data for each matrix.
Be extremely fast, since each matrix is huge and my project involves a lot of matrices.

Fortunately, we have some conditions and additional information that make this task easier. Each matrix:

Is numerically ordered by origin, then destination (as seen in the example long table).
Includes a separate file containing the unique set of destinations.

I couldn’t find any off-the-shelf or obvious solutions to this problem. I imagine it’s too niche to warrant much attention. We’ll have to roll our own.

Solutions

Given our requirements, the simplest (IMO) solution is text stream processing using bash pipes. This approach saves almost no data in-memory, is fast (enough), and uses pre-existing CLI tools to handle compression. All we need to do is write code to perform the long-to-wide pivot on the text stream.

Awk

awk is first tool I reach for when doing any sort of text stream processing. It’s stupidly powerful, if a bit esoteric and finicky.

Here we can use it to iterate through rows and match each value’s position to an array of possible destinations (columns). This approach stores only the destinations and a temporary array of values in-memory.

# Create an array of columns (destinations) from a
# separate input file
BEGIN {
    FS=OFS=","
    while ((getline line <dests) > 0) {
        contents = contents line
    }
    numCols = split(contents,cols)
}

# Create the CSV header by printing each destination
FNR == 1 {
    printf "%s", $1
    for (c=1; c<=numCols; c++) {
        dest = cols[c]
        printf "%s%s", OFS, dest
    }
    print ""
    next
}

# Loop through rows, appending each value to an array
# based on its corresponding destination. Once the
# origin changes, print the array, then start a new row
# in the output stream
$1 != prev[1] {
    if ( FNR > 2 ) {
        prt()
    }
    split($0,prev)
}
{ values[$2] = $3 }
END { prt() }

function prt(destination, value, c) {
    printf "%s", prev[1]
    for (c=1; c<=numCols; c++) {
        destination = cols[c]
        value = values[destination]
        printf "%s%d", OFS, value
    }
    print ""
    delete values
}

The script (saved as pivot.awk) is called using the following bash command:

# Read the file and add progress bar with pv, then decompress
# the input stream (bz2), pivot with awk, and recompress with zstd
# The list of unique destinations is passed to awk as a variable
pv in.csv.bz2 \
  | pbzip2 -dc \
  | awk -v dests=<(cat dests.csv | paste -s -d, -) -f pivot.awk \
  | zstd \
  > out.csv.zst

This approach works, but is extremely slow for large matrices (see results). Let’s find something better.

Rust approach A (naive)

Given the need for performance, a lower-level language is the obvious next step. Let’s try using Rust.

I don't have much experience with compiled languages (most of my work uses Python/R), but I took this as an opportunity to learn something new. I chose Rust because it has great documentation, easy-to-use tooling, and a (seemingly) large community. Also, I was intimidated by the complexity of older languages like C/C++.

I have to say, I'm incredibly impressed by Rust. I managed to get a working prototype binary in about 2 hours despite almost no experience with compiled languages, mostly thanks to the excellent book, linting, and compiler hints. That said, please forgive any code smell; I'm learning (in public) as I go.

My naive approach does nearly the same thing as the awk script. First, it stores the destinations as keys in a dictionary (BTreeMap in Rust). Then, it takes lines from stdin, populates the dictionary values, and then prints the values to stdout for each origin. The code is a bit more extensive, so I’ll link to GitHub rather than showing it here.

GitHub gist for naive rust code

The resulting binary (named pivot) is called similarly to the awk script:

pv in.csv.bz2 \
  | pbzip2 -dc \
  | ./pivot dests.csv \
  | zstd \
  > out.csv.zst

This approach is nearly 10x faster than the awk script. However, there are a couple things to improve:

Cloning a fresh, unpopulated dictionary (only keys, no values) for each origin seems incredibly wasteful, but I’m not sure how else to clear the dictionary values.
All origins and destinations are strings. Making them unsigned integers or similar might be faster.
Lots of .to_string() calls which may be unnecessary.

Rust approach B (improved)

Given my relative lack of experience with compiled languages, I figured it might be worthwhile to seek input from various Rust-specific forums. I was a little intimidated at first, but the Rust community turned out to be unreasonably helpful.

I got dozens of responses, ranging from correcting small syntax issues to literally rewriting my entire project. My favorite response (credit to theiz) had a relatively straightforward approach:

Create a vector of all destinations from the separate argument file.
Allocate a vector of commas with the same length as (1).
Pre-allocate a big string that can hold all the minute values for a given origin.
For each line from stdin, push a value into the string from (3) when the destination matches the expected value from (1), else skip and fill with commas from (2).

GitHub gist for improved rust code

This approach uses minimal allocations, is 8x faster than the naive approach, and is almost 60x faster than the awk script. I can’t think of many ways to improve it besides multithreading or similar complexity.

It does have one downside: the input file must be sorted by origin and destination. If the destinations are not sorted, then the loop will break. The naive approach doesn’t have this restriction.

Results

We can measure the speed of each approach using two test files. Each one is sorted (by origin, then destination) and has its unique destinations extracted to a separate file.

File A - 17031-output-TRACT-CAR.csv.bz2
- 13MB compressed, 108M uncompressed
- 3.8 million rows (origin-destination pairs)
- 1318 unique origins, 2869 unique destinations
File B - 06037-output-BLOCK-WALK.csv.bz2
- 1.1GB compressed, 11GB uncompressed
- ~300 million rows (origin-destination pairs)
- 105,957 unique origins, 115,936 unique destinations

The results shown below are from a 2022 MacBook Air M2 with a 1TB SSD. Speed is measured with pv and total time with time. Parallel bzip2 (pbzip2) is used for decompression, and Z-standard (zstd) is used for recompression. You can find more long matrix files on the PySAL spatial access resources page.

	File A	File B
Method	Time (Real)	Time (Real)	Avg. Speed
awk	6.45s	3h22m8.07s	124KiB/s
Rust A (naive)	1.86s	15m23.42s	1.11MiB/s
Rust B (improved)	1.24s	2m22.02s	7.13MiB/s
Baseline (no pivoting)	1.15s	0m25.51s	40.6MiB/s

Some thoughts:

awk is blown out of the water by the two Rust solutions, so it’s firmly out of the running for this particular task.
Rust solution A is nearly as fast as B for smaller files. Considering that it’s slightly more robust (doesn’t expect sorted destinations), I may use it for matrices smaller than 1GB.
Rust solution B is the only real option for super large matrices. As far as I can tell it has O(n) time complexity. I tested it on a 121GB Census block matrix and found only a slight decrease in average speed. Super impressive.

Notably, the compressed wide matrix files are typically 30-50% smaller than their long equivalents, depending on sparsity.

Conclusion

So that’s it. Our matrices are pivoted and resaved as .zst files. They can be loaded into any software that accepts wide matrices or converted to Parquet/ORC (more on that later).

This was a fun, challenging little project and turned into a great opportunity to learn a new language. I’m really happy I chose Rust. It’s such a delight to work in, even if I don’t yet fully grok it. Feel free to email or comment on the gist if I’ve missed any obvious improvements.

Favorite digital things

Mon, 30 Aug 2021 00:00:00 +0000

This is a list of my favorite digital things, excluding media. It’s basically a blogroll, /uses page, and bookmark dump combined into one. It’s purpose is three-fold:

Make cool things discoverable for others
Serve as a personal bookmark page for things I like
Force me to be intentional about the things I use and read

This list is constantly updated. When I find a cool new thing, I’ll add it. When I lose interest in something, I’ll remove it. The history of this page is tracked via git.

As usual, indicates an absolute favorite.

Blogs

These are the blogs I read regularly. They’ve influenced my thinking and helped form who I am today. If they support RSS, I subscribe to them using FreshRSS and NetNewsWire.

Art

Joel Simon - Beautiful generative art mixed with code
Larsen Husby - Love me some maps
Matt Keeter - Weird shaders, raytracers, and games
Paul Stamatiou - Photography + hands down the best travelogues I’ve ever seen

General

Ana Rodrigues - Bookmarks, bookmarks everywhere
Evening Sends - Rock climbing writing, essays, and opinions
Francesco Di Lorenzo - Bite-sized posts on life and programming
Jeremy Maluf - Lists for onebag packing and other stuff. Inspired this post
Kyle E. Mitchell - Interesting posts and takes on law ∪ technology
Paul Graham - Classic essays on tech, life, and startups
Steve Vance - Urban planning, travelogues, and Chicago thoughts
Terence Eden - Life advice, book reviews, and technology. Running for decades
Terry Tao - Time and career advice from the world’s best mathematician
Tom MacWright - Spatial data, development, miscellany. Inspired this blog

Coding and tech

Bradley Taunt - Small websites and minimal UX
Cassidy Williams - Huge inspiration, amazing dev/streamer. Excellent newsletter
Dan Luu - Classic engineering blog. Mostly on measuring interesting things
Dan McKinley - Tech talker and creator of many influential slide decks
Fabrice Bellard - Open-source hero and creator of ffmpeg. Enough said
Intuitive Explanations - Wonderful math explainers for calculus and linear algebra
Joel On Software - Classic wide-ranging blog on software, teams, and work
Josh Thompson - Everything from urban parking to finding a remote job
Julia Evans - Zines, life advice, and technical posts. Incredible human
Julia Silge - Tidymodels champion. Neat posts on R and predicting home prices
Kate Rose Morley - Simple games, data viz, and CSS tricks
Marianne Bellotti - Computer science ∪ government ∪ management
Nick Babcock - Lessons on performance + some viz stuff
Nikita Prokopov - Creator of Fira Code. Posts on software performance and design
The Pragmatic Engineer - Huge collection of useful career and programming advice
Tom Ryder - Sys admin stuff and all things Unix, shell, and terminal
Vicki Boykis - Data science, ML, R stuff, books, and essays

Spatial

Katie Jolly - Maps and geography tips for R
Kyle Walker - Creator of tidycensus, mapper extraordinaire
Paul Ramsey - Weird, weedsy posts on PostGIS performance
Topi Tjukanov - Utterly useless (but beautiful) map visualizations

Posts and articles

These posts have inspired or influenced me in some way. They’re things I reread and share with others. I tend to like concise posts (lists) or posts that graphically explain something complicated.

Chicago

Portrait of Chicago: Uptown - History of Uptown. A little dated, but still well-done
Scenes from Chicago’s Ewing Annex Hotel - Stories from one the last SROs
UChicago’s Secret Places - Cool photos and urbex on the UChicago campus

Life

100 Tips for a Better Life - Things I wish I’d known at 20
Don’t End the Week With Nothing - Make things you can show
Doing a Job - The best thing ever written on how to work and lead
Embrace the Grind - Sometimes you just have to do the boring stuff
Focus - Precisely describes my situation re: maintaining and cultivating focus
How Complex Systems Fail - Short, prescient read on why things break
Leverage Points: Places to Intervene in a System - How to make things break
The Illustrated Guide to a Ph.D. - Visual perspective on what a Ph.D. really means
Stop Learning - You know enough, start doing things
Work Hard - After a certain level, there are no shortcuts for effort

Coding and tech

All the Best Engineering Advice I Stole From Non-Technical People - Big wisdom
Choose Boring Technology - Stop reinventing ssh and grasping for shiny things
Drunk Post: Things I’ve learned as a Sr Engineer - Strong opinions, mostly true
FAANG Interview Advice - Good general interview advice, not just for FAANG
Fast Software, the Best Software - Speed > lots of features
I Wish My Web Server Were in the Corner of My Room - Mix virtual and physical
The Infrastructure of a One-Man SaaS - Load balancing and AWS and K8s
Kaminsky DNS Vulnerability - The most engaging thing ever written on DNS
The Local Minima of Suckiness - How I think about teaching and learning coding
The Rise of User-Hostile Software - Software now is better for devs, worse for users
Teaching Programming - Extremely helpful tips for teaching a class how to code
The Tech Stack of a One-Woman Hardware Company - Microcontrollers and C
Thoughts After 6 Years - Short post on lessons learned. Inspired my post
The Website Obesity Crisis - Why does reddit load 15 MB on every page
The World’s Smallest Hash Table - Optimizing code beyond all reason
What I’ve Learned in 45 Years in the Software Industry - People > technology

Data

Building a Data Team at a Mid-Stage Startup - The DS experience, a short story
CLI Tools Can Be 235x Faster Than Your Hadoop Cluster - awk > hadoop
Data Science Is Different Now - DS has a supply bubble and schools aren’t helping
Engineers Shouldn’t Write ETL - They should write tools and platforms
Good Data Scientist, Bad Data Scientist - What makes a great DS anyways?
The Modern Data Stack - Where things are and where they’re going

Visualization

Building Your Color Palette - Systemic approach for picking colors for UIs
Cell Tower Distribution - Beautiful post on cell tower data viz
Compressing and Enhancing Hand-Written Notes - Shrink those scans
Ditherpunk - Monochrome image dithering and how it works
Mechanical Watch - One of the most technically and visually impressive treatments of a complex topic I’ve ever seen

Math

The Bitter Lesson - Computation power and general methods > complex methods
The Most Important Statistical Ideas of the Past 50 Years - Stats summary
The Unreasonable Effectiveness of Mathematics - Plus Hamming’s response

Other

The American Abyss - Long political essay on Jan. 6th and how we got there
How to Gain More From Your Reading - Tips for attentive, intentional reading
MMAcevedo - Horrifying short fiction on the perils of saving a brain’s state
The Plague Year - Summary of the institutional failures that made COVID so bad
Some Blogging Myths - The most motivating post on blogging I’ve ever found
The Tyranny of Structurelessness - All groups develop informal structure
What I Think About When I Edit - Sound and simple writing advice
Your Climbing Shoes Are Too Tight - Treat your feet right, don’t size down
Cocktail Party Ideas - Basically a long-form of this xkcd. I think about this post all the time and try to remember to stay humble about the limits of my own knowledge

Software

These are the things I use every day. It’s a list whittled down over about 5 years. It’s not exhaustive, but it’s pretty close; I still use other stuff like Word and Teams.

I have a strong preference for software that is fast, outwardly simple, and that does one thing extremely well.

Mobile

ALL - Bitwarden - Open-source password manager. Great CLI integration
iOS - Brew Timer - Coffee brewing timer for V60 and Aeropress
ALL - Down Dog - Procedurally-generated at-home yoga and HIIT
AND - Loop Habit Tracker - Minimal, yet feature-rich and flexible habit tracker
ALL - Mountain Project - Find crags and track climbs, works offline
iOS - NetNewsWire - Tiny, simple, open-source RSS reader. Also on Mac
AND - Niagara Launcher - Minimalist, fast, well-designed launcher
ALL - Todoist - To-do list I’ve used for over a decade. Simple design, great UX
iOS - WorkOutDoors - Offline maps for Apple Watch. Great for hiking and biking
ALL - YNAB - Straightforward budgeting with bank and brokerage integration

Desktop

MAC - IINA - Minimal Mac-only alternative to VLC
MAC - Little Snitch - Decide how apps talk to the internet. Must-have for any Mac
ALL - Obsidian - Fast notes. Free, works everywhere, plain markdown
MAC - Swish - Resize and move windows with super intuitive gestures
MAC - Transmit - Fast, versatile file transfer utility
ALL - Positron - VS Code fork specifically for data science
WIN - Terminal + WSL 2 - Linux virtualization on Windows that just works

Other

FreshRSS - Free, self-hosted RSS feed aggregator. Lightweight, fast, and simple
JetBrains Mono - Monospaced font with ligatures. Works in nearly every editor
Parquet - Fast, space-efficient columnar storage format
Postgres - Preferred RDBMS for any big project with structured data
PostGIS - Spatial data extension for Postgres
SQLite - The go-to DB engine for any small project that requires SQL
WireGuard - Miracle VPN. Crazy fast and easy-to-use
ZFS - Filesystem for storage pooling and management

CLI tools

ffmpeg - Multimedia toolkit for just about everything
fzf - Blazingly fast fuzzy finding. Why cd when you can Alt-C?
hugo - Simple, self-contained static site builder. Used to make this site
neovim - Text editor of choice. Config and plugins also work with vim
ripgrep - grep but better. Search every file in a directory in milliseconds
stow - Symlink, config, and dotfiles management
tmux - Terminal multiplexer + keep remote terminal sessions alive
uv - Python project management. pip but 100x better/faster
zstd - Compression algorithm of choice. Replaced bzip2 as a favorite

R packages

data.table - Impossible data manipulation magic. Faster than everything else
dplyr - Clear, intuitive data munging that plays well with other Tidyverse libraries
lubridate - Intuitive, consistent date-time handling. Save yourself from footguns
ggplot2 - The plotting lib for R. Blows matplotlib (and many others) out of the water
r5r - Robust, realistic transportation routing using Conveyal’s R5 engine
renv - Wonderful environment/package manager for R (finally)
sf - Geospatial manipulation, maps, and geometry features
tidycensus - Useful interface for the Census API
tidytransit - GTFS feed reader. Make trains go brrr

Last updated 2026-02-01

Lessons learned in local government

Mon, 24 May 2021 00:00:00 +0000

I’ve been working as a data scientist in local government for around a year and a half now. Not long, but long enough to learn some things and update a few of my opinions.

None of these lessons are specific to my organization or role. Friends in other agencies confirm they generalize well to most positions at the intersection of data and local government.

I’m sharing these lessons with two goals:

To help myself remember them and to keep a running log of what I’ve learned.
To (hopefully) help other people avoid the same painful mistakes.

Things I’ve learned

Local government is incredibly risk averse and therefore difficult to change.
- On the elected side, there’s an information problem. Voters are more likely to notice (and thus punish) bad outcomes, while good outcomes are often overlooked by both media and voters. This reduces the incentive to change the status quo (high risk, low potential reward) and turns policy into a communications problem.
- On the career side, there’s an incentives problem. Most career local government employees value stability. They may not prefer the status quo, but they know that shaking things up is riskier than doing nothing. They’re also unlikely to be rewarded for status quo improvements or additional effort (due to pay schedules and a slow promotion process).
Big technology changes in local government usually happen through contracting. This is because:
1. Elected and appointed employees use contracting as a way to mitigate personal risk.
2. Contracting helps ensure the start/survival/continuity of status-quo-changing projects.
3. Most local governments lack the technical capacity/talent to do major tech overhauls in-house.
Small things take a long time if they haven’t been done before. New tasks require new processes. New processes require delegation of responsibility, updated budgets, and lots of communication. All of this is very slow. It once took me around 7 months to get a VM approved and spun up.
Attention causes change. Most local government bodies (besides cities and police departments) receive little to no public attention. As a result, they tend to be fairly responsive when they do receive attention, positive (helping an organization improve, providing feedback) or negative (vocal criticism, the threat of media scrutiny).
The boring stuff often matters the most. Most people aren’t affected by the sexy, hot-button social or cultural issues covered by national media. Conversely, local things like taxes, zoning, housing, and garbage pickup have a profound impact on people’s everyday lives.
Local government data is worse than you think. The systems for storing most local government data are antiquated, messy, and often have decades worth of various kludges. Government’s main advantage when collecting data is the ability to mandate compliance by law (usually from businesses).
Job satisfaction comes from impact. Impact comes from leverage. Leverage is built through rapport with others, ownership of projects/processes relevant to the organization’s mission, and trust that you can implement/get things done.

Things I’ve changed my mind about

People skills are just as important as technical skills. I wish I could go back and slap myself in the face with this knowledge. Unlike at a cool startup or some FAANG company, you will not be recognized in local government for your PhD-level knowledge of the latest ML techniques.

Day-to-day stuff in local government gets done through collaboration and clear communication with lots of different stakeholders. That means the understandability of your work is more important than the purity of your code. The success of a project is dependent on your leverage and how well you explain the project’s utility to people above you.

If you really want to do technical work in government without dealing with people, find a manager who understands the value of your technical skills and will boost/cover/hype for you.
Legacy systems aren’t necessarily bad. They can be crazy fast and also very good at the specific thing they were designed for. The issues start when new requirements pop up and you need to shoehorn some horrible new process into your 1980s mainframe.
Domain knowledge is vital, especially for data people. Career employees may not be able to build a cutting-edge ML model, but they probably have more knowledge about the data and its constraints than you ever will. Go talk to them, in-person if possible. If you don’t, you might spend a year thinking that APTS = 5 means five apartments, rather than six apartments (5 is the categorical encoding for the numeric value 6).
Management skills are incredibly valuable. I entered government thinking of managers as less valuable than technical employees. I was wrong. So much local government work is about people, processes, and dealing with stakeholders; a manager can really make (or break) a small organization or department. Unfortunately, one of local government’s biggest problems is a lack of good managers (on all levels).
Old technology is good actually. By “old” here I mean tech that’s 5-10 years old i.e. not brand new. Such technology is usually super stable, well-performing, and well-documented. Chasing the newest stack is for startup folks.
The simplest possible solution is the best solution. Someone after you will have to maintain your code. Moreover, if you’re producing public-facing work, it should be simple for the sake of transparency. Remove complexity (and technologies) whenever possible.
Decision makers (rightly) do not care about the technical details. This was a tough lesson. Decision-makers care about options, their implications, and their potential risks. They do not care about how a boosted tree model works.

Opinions that haven’t changed

Avoid vendor lock-in, prefer open-source. Local governments tend to prefer SaaS tools. They offload support costs to a third-party and (arguably) require less technical, less independent users. However, with the right team, open-source can be more transparent, cheaper, and can help build connections to the wider developer community. On a related note, open-source GIS tools > ESRI.
The potential impact of a motivated, smart person is extremely high. Local government is bereft of competent data people despite being awash in data and data-related problems. A sufficiently motivated person can accomplish a lot in local government even with an average data science skill set. The first (and hardest) step is to find a team with the leverage and mandate to create change.

Creating moving transit maps with R and GTFS feeds

Wed, 03 Mar 2021 00:00:00 +0000

GitHub gist for source code

I’ve been on a bit of a transit kick this month. I read Human Transit (the blog and the book) and have been playing a lot of Mini Metro.

I’ve also been playing around with GTFS feeds, which are a standardized data format for public transit systems. They provide schedules, stop locations, route shapes, and more.

I wanted to see if I could take a GTFS feed and turn it into an animated map in the style of Mini Metro (I love its aesthetic). Turns out it’s actually pretty easy.

Your browser does not support the video tag.

These maps are made using R and ggplot2. Each frame of animation is a separate plot combined using ffmpeg. There are probably better/faster ways to animate besides rendering each frame, but this way is easy and flexible.

Walkthrough

You can find the code to make these maps at the GitHub gist at the top of the page. It only works for single feeds (so no New York, which has many separate feeds) and only for trains, though it could be easily adapted to other route types.

It’s relatively simple code, so rather than explain it step-by-step, I’ll walkthrough the general process of creating a map. Anything in snake_case refers to an object in the code.

Create the data

First, we need to download a GTFS feed and convert it into a format useable in R. The map needs three data sets to work:

A spatial data frame of stop locations (stop_shapes)
A spatial data frame of track shapes (route_shapes)
A data frame of the point location of all trains at any time within the specified window (final_df)

The first two data sets are contained in the feed, but the third data set is not. In order to create it, we need locations and arrival times for not only stops, but also the points between stops. GTFS feeds don’t contain that information, but they do contain data we can use to figure it out, mainly:

The geographic location of each stop (stops_df)
Each train’s scheduled arrival time for a given stop (stops_df)
The shape of the track between stops, which we can convert to waypoints for the train to follow. These waypoint don’t have arrival times (waypoints_df)

Combining these two data sets gives us the position and arrival time of stops and the position (but not time) of waypoints between stops. We can then use Stineman interpolation to fill in the missing times.

The final data set (final_df) is what creates the actual animation. It contains a list of geographic points and corresponding arrival times for each train.

Generate the animation frames

Finally, we create a ggplot2 object and add animation with gganimate. There are two important components here:

transition_components(time), which tells gganimate to cycle through the window of time defined in final_df
geom_point() with a group = trip_id aesthetic, which tells gganimate to treat each train as a single point that moves along through time

The resulting object is passed to the animate() function, which then generates the inidividual frames. gganimate will use the tweenr package to further interpolate point locations for smoother animations.

Convert to video

The output of animate() is a folder of individual .png plots. These can be combined into a single video using ffmpeg. I used the following settings:

ffmpeg -framerate 60 \
  -pattern_type glob -i './gganim_plot*.png' \
  -c:v libx264 -profile:v main -preset fast \
  -strict experimental -movflags +faststart -pix_fmt yuv420p \
  output.mp4

And then one more pass to create a WebM file as a fallback for HTML <video> tags:

ffmpeg -i output.mp4 -b:v 0 -crf 45 -an -f webm output.webm

Overall, this method works fairly well. It takes awhile to generate the individual frames, but the resulting animation is crisp, smooth, and beautiful. Pretty good for less than 200 lines of code.

Hello (again) world

Sat, 27 Feb 2021 00:00:00 +0000

I think I’ve made a personal website 3-4 times now. Each one sat on the internet dusty and unused. I made them out of a sense of obligation to have a site, not out of genuine interest in creating content.

Well, now I finally feel like making some stuff and need a place to put it. So this site is my attempt at creating that space. It has five main purposes. It is:

An outlet for rare creative impulses (photos, writing, or otherwise)
A place to put coding projects and other technical work/thoughts
A way to keep track of books I read and which ones I like the most
An occasional diary and place to keep track of goals, life events, or whatever I’m consuming at the time
A way for me to learn modern web design and best practices

Plus, this site is just a way for me to document my own life. I have awful long-term memory, so I want a way to look back at what I was up to during a given time.

I plan to post something roughly every other week. Hopefully I can contribute at least a small bit to the vast pool of knowledge and content that is the modern internet.

Visualizing America's rural physician shortage

Wed, 10 Apr 2019 00:00:00 +0000

Note: This is adapted from an essay I wrote as a research assistant during graduate school. You can see the original essay here.

Rural America is facing a massive and growing shortage of physicians. For the past few decades, doctors have flocked to cities seeking better career opportunities, higher pay, and urban amenities. As a result, only 10% of physicians now live in rural areas, compared to 20% of the overall U.S. population. This maldistribution has potentially disastrous consequences: studies have shown that having physical access to a doctor is an important determinant of health. For many rural Americans, getting this access requires a long drive and a potentially longer wait, resulting in potential health consequences such as a higher likelihood of infant mortality, obesity, and disability. The shortage and its effects are likely to get worse in the coming decades, as an increasing percentage of physicians choose to practice in urban areas.

In response to this crisis, policymakers have implemented two primary measures designed to stem the flow of physicians moving to urban areas. First, the Department of Health and Human Services and others have created wage incentives for doctors practicing in shortage areas. These incentives are designed to attract new doctors to rural areas by offering them a slight wage premium, and to keep rural doctors rural by offsetting their opportunity costs. Second, many medical schools have opened rural branch campuses or programs, ostensibly designed to accustom students to rural life and erode their preference for urban areas.

The assumption of these policies is that doctors live in urban areas simply because they prefer higher density and urban amenities. Unfortunately, this assumption is incomplete and misinterprets the nature and causes of the rural doctor shortage. The rural doctor shortage is not a result of rurality per se, but rather of the urban/rural education and class divide. In short, doctors have an overriding preference for urban areas because urban areas, like doctors, tend to be richer and more highly educated than rural areas. Current policies aimed at changing this preference have been unsuccessful because class is largely an immutable characteristic. Paying doctors more in rural areas will not override their desire to be around other doctors and highly educated people. Instead, policymakers concerned about the rural doctor shortage should focus on recruiting doctors with pre-developed preferences for rural areas and on finding appropriate doctor substitutes, such as nurse practitioners.

Maintaining current policies will waste millions of dollars and do little to ameliorate the rural doctor shortage or improve rural health. Policymakers should pivot toward a new set of policy solutions based on a more nuanced understanding of the shortage and its causes. To that end, the rest of this document is split into three parts: first, describing where the shortage is and what its causes are; second, discussing why current policies have failed; and third, offering new potential solutions.

Mapping the U.S. doctor shortage

Effective policy responses begin by correctly identifying the problem they’re trying to solve. Policymakers, health professionals, and academics have spent decades studying the spatial distribution of physicians. As early as the 1960s, academics such as Rimlinger and Steele (1961) lamented the shortage of rural U.S. physicians and its potential impact on rural health. Since then, countless academics have updated the methods used for quantifying the shortage, but their shared description of the shortage as being rural has remained the same (where rural typically means lower population density).

Unfortunately, this definition is incomplete. While it’s generally true that rural areas have fewer doctors, there are also rural areas with a high number of doctors relative to their population. These oversupplied areas aren’t just small anomalies, they often encompass entire regions of the United States. The map below shows the physician-to-population ratio (physicians per 100K people) of each U.S. state. The ratio is grouped into quantiles for easier viewing. Most rural southern states (Mississippi and Arkansas) and plain states (Iowa, Nebraska) do have a low number of doctors relative to their population. However, notably rural regions such as the Pacific Northwest (Washington, Oregon) and the far Northeast (Maine, Vermont) are oversupplied.

This distribution indicates that the rural doctor shortage is not driven entirely by rurality, but rather by a combination of rurality and other factors such as educational attainment level. Research shows that educational attainment explains over 40% of the variation in physician supply, while population density (rurality) explains less than 10%. In other words, the rural doctor shortage may be better characterized as a doctor shortage in less-educated areas. Rural areas do have doctor shortages, but they’re largely caused by rural areas being less-educated, rather than by rurality itself. Thus, policymakers have misidentified the cause of the shortage, attributing it to rurality when the truth is likely more nuanced.

The true nature of the doctor shortage can be seen more clearly at the county level. The map below shows an index of the relative level of primary care accessibility for every county in the United States. This index is calculated using a new technique which minimizes the driving time and cost of congestion for each patient. In other words, it allows patients to make a trade-off between distance and crowdedness, with patients moving farther to avoid extremely crowded providers. This method adds to previous literature on spatial accessibility, which is often measured simply as the patient-to-provider ratio of a given space.

Since the algorithm is applied to the entire U.S., the index number it yields is nationally relative and can be used to compare regional or county-level variation in accessibility. In highly accessible (purple) areas, patients drive only a short distance to their doctor and the doctor’s office is unlikely to be crowded when they arrive. In inaccessible (pink) areas, patients may drive for hours to their doctor, only to find themselves competing for attention with the many other patients that doctor serves.

The results from this new method support the hypothesis that physicians prefer to live in highly educated areas. On the map, this can be seen most clearly in the New England region. Maine, Vermont, and New Hampshire have excellent primary care accessibility despite being relatively rural states. Similarly, states in the upper Midwest such as Minnesota, Wisconsin, and Iowa are highly accessible despite being mostly rural. All of these states contain a large number of highly educated people relative to their population. Conversely, Texas and much of the South have poor accessibility despite being relatively more urban. These areas have lower average educational attainment than the rest of the U.S.

Nonetheless, framing the problem as a rural shortage can still be useful. Certain regions notwithstanding, the U.S. generally has a large overlap between rurality and low educational attainment, and areas with the most severe shortages tend to be both very rural and very undereducated. As such, rurality can serve as a useful heuristic, capturing the educated/undereducated area divide in a way that is more intuitive to most people. However, this characterization becomes a problem when used as a basis for policy, where misidentification of the underlying problem can yield ineffective and costly results.

(For the remainder of the article, “rural” is used as a shorthand for “rural and/or undereducated”.)

Age is also an important aspect of the rural doctor shortage story. In the past three decades, more and more young doctors have settled in cities, while very few have moved to rural areas. This sorting is likely driven by homophily (the desire to interact with and live among people like yourself) and existing demographic shifts in American society. Put bluntly, young, educated people like to live around other young, educated people, most of whom live in cities. Doctors follow this same trend.

Unfortunately, these young, urban doctors tend to stay put rather than move to rural areas as they get older. As a result, rural doctors tend to be an average of 18 years older than their urban peers. In the plot below, the bulk of the distribution of rural doctors is around 60 years old. Many of these doctors will retire in the next few decades and will not be replaced, severely exacerbating the existing crisis.

The rise in specialist physicians has also contributed to the rural doctor shortage. As more medical students choose high-paying specialties, the proportion of primary care physicians trained each year has decreased, particularly since 2001. The plot below shows this trend.

Specialists are much more likely to practice in urban areas than primary care physicians. Rural areas rarely produce the demand needed to support a specialist. Even if rural demand was sufficient, most rural hospitals do not have the infrastructure, equipment, or funds necessary to support full-time specialists. As a result, specialists are much more likely to live and practice in urban areas.

Doctor preferences and the current policy response

Policymakers have spent millions of dollars trying to solve the rural doctor shortage. Unfortunately, current policies are misguided and focus on the wrong problem, relying on the assumption that geographic variation in physician supply is driven by rurality itself. These policies have each attempted to change physician preferences for rurality but have ignored physicians’ homophily and preference for educated areas. As a result, these policies have largely failed and the rural doctor shortage persists.

One of the most well-known policies designed to address the rural shortage is increasing rural doctor pay. Health and Human Services (HHS) and other healthcare payers reimburse doctors practicing in shortage areas at a higher rate, typically around 10%. The goal of these payments is to incentivize physicians to move to rural areas by offsetting their input and opportunity costs. However, these payments have not worked as intended.

The plot below shows the median wage of urban and rural doctors over time. Since 2000, the wage gap between urban and rural doctors has expanded dramatically, with rural doctors now making roughly $40K more than their urban counterparts. Yet the rural doctor shortage has only worsened, indicating that such payment policies have mostly subsidized already-rural doctors while inducing few urban doctors to move.

By ignoring wage incentives, urban doctors display an overriding hidden preference for urban areas. Previous research (Lombardo et al.) suggests this preference is a combination of doctors’ preferences for urban amenities and their preference for being around other highly educated people. This partially explains the maldistribution visible in the previous maps. Places such as New England have a high level of educational attainment and nice urban amenities, they therefore have a large number of doctors. Places like Mississippi have low average educational attainment and few urban amenities, they therefore have fewer doctors.

Doctors have this preference for urban areas regardless of their birthplace. The plot below shows the rurality of where different cohorts of physicians end up attending medical school, doing their residency, and practicing. Each line represents a cohort of physicians born in 1 of 10 core-based statistical area (CBSA) deciles. For example, the pink line represents the most rural-born cohort of doctors, while the purple line represents the most urban-born. Each shaded region represents a different level of rurality.

All of the doctor cohorts, regardless of their birthplace rurality, end up practicing in urban/suburban areas. After being mechanically forced to urban areas for school (almost all medical schools are in cities), doctors never return to the same rurality level as their birthplace. This indicates that doctors’ preferences for urban areas may develop as a result of training in those areas. Based on this assumption, some medical schools have developed rural programs intended to give doctors a taste of rural life. However, there is little evidence to indicate that such programs increase rural practice retention.

Medical schools also play a role in determining where a doctor chooses to practice. The plot below shows the average MCAT score of each medical school compared to the rurality of the average graduate’s practice location. Graduates of medical schools with higher average MCAT scores tend to end up practicing in more urban areas.

The reason for this is two-fold. First, better medical schools tend to be located in large cities. Doctors at these schools likely develop preferences for big city amenities as well as business and social networks that make leaving undesirable. Second, graduates of better medical schools are more likely to be matched with their preferred residency programs, i.e. they are unlikely to be forced to attend less competitive rural programs.

How do we fix the rural doctor shortage?

The current state of rural American medicine is bleak. Doctors are overwhelmingly clustered in urban and highly educated areas. The existing cohort of rural doctors is rapidly aging and is unlikely to be replaced. More and more doctors are becoming specialists who can only viably practice in cities. Solving the rural doctor shortage seems like an impossible task.

Previous solutions have focused on increasing the compensation of rural doctors, yet this strategy seems to be ineffective. Doctors have such a strong preference for urban areas that it overrides any incentives created by relatively small rural wage premiums. Given this preference, the high earnings of most physicians, and the marginal utility of money, it would likely take a massive wage premium to begin attracting marginal doctors to rural areas.

Instead, two solutions should be implemented. First, medical schools should recruit more students from rural areas. The sixth plot shows that such students are the most likely to practice in rural areas. These students have pre-existing preferences for rurality, i.e. they are not as prone to developing strong preferences for urban areas because they already prefer rural ones. Additionally, students from rural areas are already invested in and familiar with rural communities. Recruiting more of these students would undoubtedly bolster the number of rural physicians.

Second, states should focus on training and licensing more nurse practitioners (NPs). The plot below shows that nurse practitioners already serve as substitutes in states with relatively few doctors. NPs can perform the same basic tasks as a primary care physicians but are cheaper and easier to train and license. Substituting NPs in rural areas could reduce the impact of the rural physician shortage and improve basic health outcomes for rural Americans.

The worsening maldistribution of U.S. physicians has potentially devastating health consequences for millions of Americans. States and the medical community should act quickly to implement practical solutions that increase the recruitment of rural medical students and NPs.

Dan Snow

OpenTimes: Free travel times between U.S. Census geographies

Grafana creates strange DNS traffic for releases >= 10.2.3

Queried domainsLink to queried-domains section

Exporting Jellyfin playback stats to Prometheus

GitHub gist for source code

The Jellyfin REST APILink to the-jellyfin-rest-api section

Fetching aggregate statsLink to fetching-aggregate-stats section

Prometheus JSON exporterLink to prometheus-json-exporter section

Update (2024-02-23)Link to update-2024-02-23 section

Comparing 500 billion rows with data.table

ProblemLink to problem section

CaveatsLink to caveats section

SolutionsLink to solutions section

For loopLink to for-loop section

Apply and matrix opsLink to apply-and-matrix-ops section

Vector recyclingLink to vector-recycling section

Pure data.tableLink to pure-datatable section

ConclusionLink to conclusion section

Update (2023-06-21)Link to update-2023-06-21 section

Update (2023-06-23)Link to update-2023-06-23 section

Pivoting 100GB text files from long to wide

ProblemLink to problem section

SolutionsLink to solutions section

AwkLink to awk section

Rust approach A (naive)Link to rust-approach-a-naive section

Rust approach B (improved)Link to rust-approach-b-improved section

ResultsLink to results section

ConclusionLink to conclusion section

Favorite digital things

BlogsLink to blogs section

ArtLink to art section

GeneralLink to general section

Coding and techLink to coding-and-tech section

SpatialLink to spatial section

Posts and articlesLink to posts-and-articles section

ChicagoLink to chicago section

LifeLink to life section

Coding and techLink to coding-and-tech-2 section

DataLink to data section

VisualizationLink to visualization section

MathLink to math section

OtherLink to other section

SoftwareLink to software section

MobileLink to mobile section

DesktopLink to desktop section

OtherLink to other-2 section

CLI toolsLink to cli-tools section

R packagesLink to r-packages section

Lessons learned in local government

Things I’ve learnedLink to things-ive-learned section

Things I’ve changed my mind aboutLink to things-ive-changed-my-mind-about section

Opinions that haven’t changedLink to opinions-that-havent-changed section

Creating moving transit maps with R and GTFS feeds

GitHub gist for source code

WalkthroughLink to walkthrough section

Create the dataLink to create-the-data section

Generate the animation framesLink to generate-the-animation-frames section

Convert to videoLink to convert-to-video section

Hello (again) world

Visualizing America's rural physician shortage

Mapping the U.S. doctor shortageLink to mapping-the-us-doctor-shortage section

Doctor preferences and the current policy responseLink to doctor-preferences-and-the-current-policy-response section

How do we fix the rural doctor shortage?Link to how-do-we-fix-the-rural-doctor-shortage section

Queried domains

The Jellyfin REST API

Fetching aggregate stats

Prometheus JSON exporter

Update (2024-02-23)

Problem

Caveats

Solutions

For loop

Apply and matrix ops

Vector recycling

Pure data.table

Conclusion

Update (2023-06-21)

Update (2023-06-23)

Problem

Solutions

Awk

Rust approach A (naive)

Rust approach B (improved)

Results

Conclusion

Blogs

Art

General

Coding and tech

Spatial

Posts and articles

Chicago

Life

Coding and tech

Data

Visualization

Math

Other

Software

Mobile

Desktop

Other

CLI tools

R packages

Things I’ve learned

Things I’ve changed my mind about

Opinions that haven’t changed

Walkthrough

Create the data

Generate the animation frames

Convert to video

Mapping the U.S. doctor shortage

Doctor preferences and the current policy response

How do we fix the rural doctor shortage?