Git Granary

Captain’s log: it’s been 10 days since I entered the Git LFS rabbit hole. Following an emergency in-prod GitHub replacement I got nerd sniped by the Git Large File Storage API.

Git Granary

🌾 Git Granary is the end result. I coded my own LFS server!

I plan to self-host Granary locally alongside my Gitea instance. Why not just use Gitea’s LFS? Because I honestly forgot Gitea supported LFS …and that I’d configured it. It’s still sitting there empty. I may move to Forgejo I need to catch up on the drama.

Anyway, I’ll have separate Git and LFS servers on my local network. They’re not exposed to the internet 24/7. I will need to temporarily tunnel the LFS server when I deploy my website from the public GitHub repo. I haven’t figured that part out yet. Can I install Tailscale in an action runner?

Granary is designed for self-hosted personal use. Technically it can support multiple users but they must use the same basic HTTP auth. The only multi-user issue I see is if two users try to upload the exact same object simultaneously. In that case I’m guessing the file would be locked by the JavaScript runtime and one request would fail.

Runtimes

Granary can run under Bun, Deno, and Node. I’ve coded it with an adapter pattern to provide runtime specific filesystem API and HTTP server functionality. Bun and Node adapters are minimum viable implementations.

Deno is the primary runtime. serveFile from the standard library does the heavy lifting for download operations. For uploads I’m using the extended Web Crypto that support streaming through digest. This allows me to tee the request.body and stream it to disk whilst calculating the SHA-256.

I found Bun’s file API to be lacking so it uses the same node:fs code from the Node adapter. Bun and Node don’t check the hash. That’s the main difference from Deno. They could but I’m not all that interested maintaining multiple adapters. I just wanted to prove the concept.

Each runtime uses it’s own HTTP server; Bun.serve, Deno.serve, and Node’s createServer. I had to fudge the Node one because it doesn’t use standard Fetch API objects. I borrowed from @hono/node-server to polyfill Request. I’m not actually using Hono for routing. Simple URL pattern matching is sufficient in the root handler.

For example:

async function handle(request: Request): Promise<Response> {
  const pathname = '/:repo([a-zA-Z0-9._-]+)/objects/batch';
  const batchMatch = new URLPattern({pathname}).exec(url);
  if (match) {
    const {repo} = match.pathname.groups;
    /* Do something... */
  }
}

Earlier this year I experimented with a request/response router based on URL patterns. They’re slower than string or regular expression matching. “Slower” in the synthetic benchmark sense. Real world you wouldn’t be using JavaScript if you needed that many requests per second.

Open Source

Granary is MIT licensed feel free to use it if you’re daring! I will be testing it myself. I don’t foresee many changes or bug fixes. The Git LFS API is remarkably simple.

If I don’t report back in a few months you’ll know I’ve binned it.