Everything except the bytes

When you first build a large upload, you think about bandwidth. The file is big, the network is slow, and the obvious work is moving bytes from a laptop to object storage as fast as the connection allows.

That part turned out to be the easy half. Once the bytes go straight from the browser to storage and never touch an application server, throughput stops being interesting. What remains is coordination: agreeing on how the file is split, proving the client is allowed to write it, deciding when it is finished, and cleaning up when any of that fails halfway. Almost every bug I have fixed in this area was in the coordination, not the transfer.

The round trips nobody counts

Multipart uploads work by splitting a file into parts, each of which gets its own presigned URL. The textbook shape is that the client asks the server to sign each part as it needs it.

For one 200 MB file at 5 MiB parts, that is forty extra requests to sign forty URLs. For a wedding gallery of twelve hundred photos it is hundreds of round trips that exist purely to hand out permission slips.

The fix was embarrassingly simple once I noticed it. Presigning is not a call to storage. It is a local HMAC over a string, computed in microseconds, with no network involved. There is no reason to do it one at a time. So the create call now signs every part up front and returns the whole list.

The per-part signing endpoint still exists, which I like as a detail. It is now the route the happy path never calls. It only runs when a browser refreshes mid-upload and needs URLs re-minted for the parts that have not landed yet.

Authorization on the hot path

The other cost was authorization. Checking that someone may write to a gallery means a call to the auth service and a database read. Doing that on every part sign is roughly forty of each per file, and it is the same answer every time.

So the create call, which does run the full check, also mints a capability token: an HMAC over the gallery, the object key, the upload id, and an expiry. Every later call verifies that token in process instead of re-deriving the same permission from scratch. The token cannot be pointed anywhere else because the key it covers was generated server side under the original check.

It expires after twelve hours, and that number then shows up somewhere I did not expect, which I will come back to.

The finalize tail

Here is the thing I got most wrong. I assumed a large batch was slow because it was large. When I actually measured a twelve-hundred-photo upload, the bytes finished at a reasonable rate and then the batch sat there, finishing, for a long time afterwards.

Every completed file was independently taking a row lock on the gallery to check its storage quota. Uploads that run in parallel all pass the initial quota check before any of them has stored anything, so the real check has to happen at the end, under a lock. Twelve hundred files meant twelve hundred serialized lock acquisitions, each with a storage call and a queue publish attached.

The batch now completes in groups. The client buffers finished files and flushes at eight of them, or after 250 ms, whichever comes first. The server takes one lock for the whole group and walks it, and both outcomes land as single set-based updates.

The flush threshold is not arbitrary. Completion calls share a queue with part uploads, so waiting for "enough" files with no deadline could deadlock once every slot in that queue holds a pending completion. Eight matches the upload concurrency exactly, so a full wave flushes immediately.

Failure is not symmetric

Two things can go wrong at the end of an upload, and they want opposite responses.

If the storage service fails to assemble the parts, the upload stays pending and the client can retry. Nothing durable exists yet.

If it assembles them but the resulting object is the wrong size, the file is marked failed and actively deleted. That asymmetry matters because a finished object is a real object. The bucket rule that reclaims abandoned multipart uploads will never touch it, so if I do not delete it myself it becomes storage nobody is aware of and nobody is billing anyone for on purpose.

The size check itself gets three attempts with a linear 200 ms backoff, which is not a network retry. Storage in this setup ingests near the client and then copies to a home region, so a brief consistency lag should cost a short wait rather than a wrongly failed upload.

Aborts that die with the network

The failure I am fondest of is the one where the cleanup shares a fate with the thing it is cleaning up.

A network drops mid-upload. The upload library notices, gives up, and fires an abort so the server can release the partial upload. That abort travels over the same dead network, and fails. Now there is a row that says "uploading" forever and a partial upload in storage that nothing will reclaim for days.

So failed aborts go into a queue and get retried on the next signal that connectivity is genuinely back: either the browser's online event, or the next successful create call. Both are proof rather than optimism. The abort request is also sent with keepalive so it survives the page being closed, and only retries on server errors, since a client error means the server has already made up its mind.

There is a matching problem on the client. The upload library, when cancelled while the create call is still in flight, discards the result and its own abort never learns which upload to cancel. The fix was to stop trusting the library's bookkeeping and keep my own ledger of what the server opened, so cleanup is always deterministic.

That 13 hour number is the twelve hour token expiry plus an hour. It is not a guess about user behaviour. A pending row older than the token that authorizes it cannot possibly be completed by anyone, so it is provably garbage.

Part size is an invariant

One constraint caught me off guard: object storage rejects a multipart upload whose non-final parts are not all the same size. That turns part size from a tuning knob into something closer to a contract.

The client picks 5 MiB on a slow connection and 25 MiB otherwise. But if someone refreshes the page mid-upload and their connection has changed class in the meantime, recalculating would slice the remaining file differently and the upload would be rejected on completion. So the chosen size is written to local storage and reused for as long as there is recovery state to resume, even if the "correct" answer is now different.

The right value is the one you already committed to.

Concurrency is a fairness problem

The last thing that surprised me is that upload concurrency is not really about upload speed.

Running eight parts in parallel saturates the uplink. A saturated uplink also strangles downloads, and the rest of the application is still trying to fetch thumbnails and pages. The moment that hurts most is exactly when someone has left the upload running in the background and gone off to browse their gallery, because that is when they are least aware of the upload and least tolerant of everything else being slow.

So concurrency drops from eight to four when the upload panel is not on screen. It is the same total work, arranged so it is invisible.

Reflections

The lesson I would keep is that the interesting failures in a system like this cluster around the edges of an operation rather than in the middle of it. The middle is a data transfer, and data transfers are a solved problem. Starting, finishing, resuming, and cancelling are where the state lives, and state is where things go wrong.

I would also say that measuring earlier would have saved me weeks. "Big uploads are slow" is the kind of statement that sounds like it explains itself, and it stopped me from asking which part was slow for far too long. The answer, that it was the finalize tail and not the bytes, was not something I would have guessed.

If you want to see it running, it is the upload path behind Lensdrop.