Blast from the past: the Heroku buildpacks I forked

Recently, going through my GitHub account, I found three repositories I had completely forgotten; forks of Heroku's Python, Node.js and Go buildpacks. The Go one (2020) has no commits of mine at all, so I must have forked it just to read it. The other two have 38 commits between them, from November 2013 to February 2015. When I opened them, I remembered about half of what they did. So this post is a small exercise in archaeology. The git log remembers more than I do, and I want to write it down before I forget the other half.

One app, one buildpack

For the younger readers (and for my own memory), Heroku was the PaaS (Platform as a Service) that turned deployment into a git push. You pushed your repository to Heroku, and the slug compiler turned it into a slug; a tarball with your application and everything it needed, unpacked into /app on a dyno (Heroku's word for a container).

The part that did the work was the buildpack, and a buildpack was three scripts. bin/detect says whether it recognises the application. bin/compile receives the build directory, a cache directory and the configuration variables, and installs everything into the first. bin/release tells the platform how to start the result. Anything a buildpack leaves in .profile.d/ is sourced when the dyno boots, which is how it puts things on the PATH at runtime.

Underneath was the stack, Cedar, an Ubuntu 10.04 image. And here is the catch; an application got exactly one buildpack. If your Python application needed a C library the stack did not have, or a second language runtime next to the first, there was no setting for that (there was heroku-buildpack-multi, a third-party buildpack that ran others in sequence, but if I remember correctly Heroku did not support it). So you forked the official buildpack and added a step. Everybody did, and GitHub is still full of these forks.

added luajit

The Python fork started on 25 November 2013, at 13:00. By 19:31 I had made 25 commits, and nine of them are called "added luajit". The rest are called "minor", "modified the script", "still messing with heroku stuff" and, at 15:40, "back to basics" :)

Those messages are the development loop. There was no easy way to run a buildpack locally, so every attempt was the same cycle; edit the step, commit, push the buildpack to GitHub, push a test application to Heroku, read the build log. Between 14:31 and 14:53 there are nine commits, two to four minutes apart, which is how long one round took.

The goal was simple. Lupa, Stefan Behnel's Python binding for Lua, needs LuaJIT, and the stack did not have it. So the step downloads the LuaJIT 2.0.2 source and runs make install PREFIX=/app/.heroku/vendor. The first version fetched the tarball from my own server, bkarak.wizhut.com, and a later one from luajit.org. A mdkir vendor appears twice, the second time in the commit called "back to basics".

Why /app, when the build runs in a temporary directory? Because the Python buildpack of 2013 moved the build directory to /app for the duration of the compile and moved it back at the end. The paths baked into compiled libraries were therefore the runtime paths, and that trick is the reason the step works at all.

The commit that actually fixed it is the one at 19:18, "fixed pkgconfig". Lupa's setup.py finds LuaJIT by asking pkg-config luajit, and LuaJIT installs its luajit.pc into lib/pkgconfig. The upstream buildpack, however, exported a PKG_CONFIG_PATH pointing at lib/pkg-config, with a hyphen; a directory nothing ever creates. I appended the right one and the build passed. Six hours, for a hyphen.

There is also a leftover. At 15:30 I committed vendor/lupa-0.20, the whole lupa source tree with LuaJIT 2.0.2 inside it; 207 files and 138,275 lines. As far as I can read the plan from that commit, it was to let lupa build against its own copy (its setup.py looks for a LuaJIT directory next to itself first). Ten minutes later came "back to basics", and no script has referenced that directory since. It is still in the repository, thirteen years later.

What did I need Lua inside a Python application for? Scripting. It was a Python service, and the Lua was the scripting for the server; I think it was called Luminous at the time. None of that is in the git log, which is probably why I had forgotten it.

Noticed cffi

In January 2015 the same fork got a second step, and its README a second line; libffi, "required by pyopenssl". This script was not mine. Upstream had added its own cryptography support in August 2014, but my fork was based on November 2013 and did not have it, so I took a step that was going around in other people's forks. It downloads a pre-compiled libffi 3.0 from somebody's Amazon S3 (Simple Storage Service) bucket into /app/vendor and adds it to PKG_CONFIG_PATH. The same file, with a different bucket, still sits in other forks on GitHub. Mine has no check for cffi in requirements.txt (the copies in other forks have one), so every build announced "Noticed cffi. Bootstrapping libffi.", cffi or not.

Then, on 25 February 2015 at 02:19, comes "replaced libffi". I swapped the pre-compiled binary for the libffi 3.1 source tarball from Fedora's package archive. The problem is that the step only extracts; it never runs configure or make. I downloaded that tarball today to be sure. It is 937 KB of source in a libffi-3.1/ directory, with a libffi.pc.in template and no library in it. So from that commit on, the step unpacked source code into the slug and built nothing, and its message that "libffi-based builds should work!" could never print. Whatever my builds linked against after that, it was not my step. Did the build break and I gave up, or had I already moved on? I do not remember (a commit at two in the morning suggests the first).

And why pyOpenSSL? Apple. The same Python service had to validate in-app purchases, and an App Store receipt was a PKCS #7 (Public-Key Cryptography Standards #7) container signed by Apple. Validating one means checking that signature and its certificate chain up to Apple's root certificate, and that is OpenSSL's job. In Python that meant pyOpenSSL; pyOpenSSL needs cryptography, cryptography needs cffi, and cffi needs libffi. Four packages down, there was a C library the stack did not have.

Not for use as a primary buildpack

The Node.js fork is shorter, and better behaved. Eight commits, from 23:18 on 27 October 2014 to 00:08 the next day, and a final diff of 14 lines. The application was a Node.js one that had to run Java programs next to it, so its slug needed a JDK (Java Development Kit).

Heroku's buildpacks for the JVM languages shared a library, jvm-buildpack-common, published as a tarball on S3. In October 2014 it was not a buildpack of its own. Its bin/compile was echo "noop", and its bin/detect printed "JVM Common - Not for use as a primary buildpack." It was meant to be sourced, which is what Heroku's own JVM buildpacks did, and what I did in the Node.js one:

# bkarak - prepare for Java
curl --silent --location http://heroku-jvm-common.s3.amazonaws.com/jvm-buildpack-common.tar.gz | tar xz
. bin/util
. bin/java

export_env_dir $env_dir

BUILD_DIR=$build_dir

status "-----> Installing OpenJDK ${javaVersion}..."
install_java ${BUILD_DIR} 1.8
jdk_overlay ${BUILD_DIR}

The commits record the three things I had to learn in that hour. First, the library's install_java writes its runtime profile into $BUILD_DIR, in capitals, while the Node.js buildpack calls the same directory $build_dir; hence "fixing stuff with build dir" and the BUILD_DIR=$build_dir line. Second, the JDK lands in .jdk, which is not on the dyno's PATH, so the Node.js profile script got /app/.jdk/bin in front of everything else ("adding java to path"; it took two commits, because the first one let $HOME expand at build time). Third, Java 8 was seven months old and had to be asked for by name; the library's default was still 1.6.

Reading it today, I can also see the bugs I did not notice then. The tarball is extracted into the current directory, which at that point is the application's build directory, so every slug shipped the library's bin/util and bin/java next to the application's own files (and would have overwritten an application script with the same name). ${javaVersion} is set inside install_java, so the status line printed before it always said "Installing OpenJDK ...". And status prints its own arrow, so the log said -----> -----> Installing OpenJDK.

Eight days later, on 4 November 2014, Cedar-14 (Ubuntu 14.04) became the default stack. On 3 December 2014, jvm-common got a compile, a detect and a release of its own, and became a buildpack you could actually use as one. And on 7 May 2015, Heroku gave multiple buildpacks per application first-class support. Within seven months of my fork, the reason for it was gone.

Running them in 2026

So, can they be updated? Before answering, I ran them. Heroku publishes its stack images, so I took heroku/heroku:24-build (Ubuntu 24.04, the x86-64 image, under emulation on my Apple M4 Max) in Docker. I ran each buildpack's detect and compile against a small test application the way the platform does, then sourced .profile.d and started the application the way a dyno does.

Both forks fail within seconds, and neither gets as far as my code. The Python fork dies on its first helper:

===== compile
/usr/bin/env: 'python2': No such file or directory
compile exit 127

That is bpwatch, the build-timing tool the buildpack shipped with; it is a Python 2 script, and Ubuntu 24.04 has no python2. The Node.js fork exits with status 1 and no message at all. Tracing it shows why. The first thing it does is ask semver.io which Node.js version matches the range in package.json, and semver.io no longer resolves. (The 2014 jq binary vendored in that buildpack, on the other hand, still runs fine on a 2024 Ubuntu.)

Then I checked every URL the two forks download during a build. There are eight, and seven are dead.

The one that survived is the one with a checksum in its path. Fedora's archive stores source tarballs under their MD5 (Message Digest 5) hash, and the file I downloaded today still hashes to f5898b29bbfd70502831a212d9249d10, the value in the URL. Every other URL named a place. That one named the content.

Ok, so the forks are dead. What does the same application need today?

For the Python one, nothing. A requirements.txt with lupa and pyOpenSSL, a .python-version that says 3.13, and the stock buildpack. The tail of the build, then a small script that runs a Lua loop and prints the versions:

       Successfully installed cffi-2.1.1 cryptography-50.0.2 lupa-2.8 pyOpenSSL-26.4.0 pycparser-3.12
===== runtime
lupa 2.8 | LuaJIT 2.1.1774896198 | Lua 5.1
lua sum: 500000500000
pyOpenSSL 26.4.0 | cryptography 50.0.2
python ssl: OpenSSL 3.0.13 30 Jan 2024 | HAS_SNI = True
pyOpenSSL links: OpenSSL 4.0.3 29 Sep 2026

The whole build took 24 seconds, and nothing was compiled. Every package arrived as a wheel, and the three with native code as manylinux wheels, binaries built to run on any reasonably modern Linux. Lupa's wheel carries LuaJIT 2.1 inside it. cffi's wheel links libffi statically (its extension module does not even ask for libffi.so). And cryptography brings its own OpenSSL, 4.0.3, while Python's ssl module uses the stack's 3.0.13; two OpenSSLs in one process, and nobody had to configure either. The two native libraries I fought with in 2013 and 2015 now ship inside the packages that need them. And the receipts? Apple moved on as well. Its verifyReceipt endpoint was deprecated in June 2023, StoreKit 2 delivers transactions signed as JWS (JSON Web Signature), and Apple publishes an App Store Server Library, with a Python version among others, that verifies them.

For the Node.js one, configuration replaces the fork. A system.properties with java.runtime.version=21, and the JVM buildpack in front of the Node.js one:

heroku buildpacks:set heroku/nodejs
heroku buildpacks:add --index 1 heroku/jvm

===== 0: detect
JVM Common
===== compile
-----> Installing Azul Zulu OpenJDK 21.0.13
===== 1: detect
Node.js
===== compile
       Downloading and installing node 22.23.4...
===== runtime
node v22.23.4
java -version (from node): openjdk version "21.0.13" 2026-10-20 LTS
java on PATH: /app/.jdk/bin/java

Same /app/.jdk as my 2014 version, and the same .profile.d trick to put it on the PATH. Even the detect string survived, minus the warning; "Not for use as a primary buildpack" became "JVM Common". The mechanism did not change. It moved out of my fork and into a buildpack that Heroku maintains.

So, technically, they can be updated. I could rebase both forks onto today's upstream and carry the steps forward. But there is nothing left to carry. The LuaJIT step is a wheel, the libffi step is a wheel, and the JDK step is a buildpack of its own. And the ground is moving again. Fir, Heroku's new generation (generally available since April 2025), builds with Cloud Native Buildpacks into OCI (Open Container Initiative) images and does not run classic buildpacks at all, and since September 2026 Cedar applications can opt into the same thing. The bin/compile interface I forked twice is now the legacy one.

So I did not rebase them. I fixed the bugs above in both forks instead, and added a note to each README saying what replaced them. The libffi step now actually builds libffi, both Python steps download from Fedora's checksum-addressed archive, the JDK comes from Heroku's JVM buildpack run as a sub-buildpack in a directory of its own, Node.js versions are resolved against nodejs.org's own release index, every download is checked against a SHA-256, and the calls to dead hosts are gone. I have not run the fixed versions; the Python one still needs a stack that no longer exists. The git history keeps the 2013 and 2014 versions exactly as they were.

Epilogue

A fork looks free the day you make it. It is one commit, and the build passes. The bill comes later, and it is paid in everything upstream fixes and you do not get. By February 2015, upstream had Cedar-14, newer Pythons and its own libffi handling, while my fork was still a 2013 copy of everything, with a pkg-config path fixed by hand. In my case nobody paid that bill, because the applications are long gone; that is the one way a fork ends well.

The other lesson is about the URLs. Everything a build downloads is a dependency, even when it is not in requirements.txt, and of the eight my forks needed, the only one still alive is the one that named its content instead of its location.

The forks are still on GitHub, heroku-buildpack-python and heroku-buildpack-nodejs, if you want to see the nine "added luajit" for yourself. The git log remembers the times, the typos and the commit at two in the morning. It does not remember why; the Lua scripting and the Apple receipts I had to remember myself. I suppose that is the division of labour between a git log and a memory.