Cancelled Rust Uploads Need Durable Cleanup

Article · Interrupted execution

The file reached storage. Before the service made it visible as an attachment, the request was cancelled.

There are two bad repairs. Deleting every old-looking file can remove an upload which is still valid. Leaving cleanup to the request task can forget the file when that task disappears.

The useful question is who can find the unfinished work later. An upload record saved in the database gives a recovery worker somewhere to start.

This lab uses PostgreSQL and files in a temporary directory. It investigates publication and cleanup after storage has acknowledged a complete file. It does not implement an S3 client, an HTTP multipart parser, or the lifecycle of partially uploaded cloud objects.

Give each stored attempt its own identity

The upload row has an ID, a storage key which never changes, and a state. The public attachment table is separate. An object existing in storage does not by itself make that object a published attachment.

The demonstration starts with a stored file and a row in stored state. In a full upload handler, create the durable attempt record before starting storage work. Record enough information to check later whether storage finished. A process can stop after the store succeeds but before it records that acknowledgement.

Use a new server-generated key for each storage attempt. Do not reuse a customer's filename as the storage key. Otherwise delayed cleanup for yesterday's attempt can delete today's replacement at the same key.

The crash and write article explains why the point where an operation stops matters. Here the database and the file store have separate operations. A PostgreSQL transaction does not make a file operation commit with it.

Publication and cancellation must compete for the same state

Publishing the attachment first claims the stored upload and inserts the public record in one transaction:

pub async fn publish(pool: &PgPool, id: i64) -> Result<bool, sqlx::Error> {
    let mut tx = pool.begin().await?;
    let key: Option<String> = sqlx::query_scalar(
        "UPDATE uploads SET state = 'ready'
         WHERE id = $1 AND state = 'stored' RETURNING object_key",
    )
    .bind(id)
    .fetch_optional(&mut *tx)
    .await?;
    if let Some(key) = key {
        sqlx::query("INSERT INTO attachments (id, object_key) VALUES ($1, $2)")
            .bind(id)
            .bind(key)
            .execute(&mut *tx)
            .await?;
        tx.commit().await?;
        Ok(true)
    } else {
        tx.rollback().await?;
        Ok(false)
    }
}

Cancellation uses the same starting state:

pub async fn cancel(pool: &PgPool, id: i64) -> Result<bool, sqlx::Error> {
    Ok(
        sqlx::query("UPDATE uploads SET state = 'cleanup' WHERE id = $1 AND state = 'stored'")
            .bind(id)
            .execute(pool)
            .await?
            .rows_affected()
            == 1,
    )
}

It cannot change a ready upload. The database decides which transition wins; the request does not make that decision from a stale status it read earlier.

Once cleanup has been chosen, this example never changes the upload back to stored. A new attempt gets a new identity and key. That rule lets a recovery worker remove the old object without racing a later publication of the same attempt.

The unique key constraint prevents two upload records from sharing a key. The service also needs to preserve the no-reuse rule over retention and retries. Removing old upload records must not make the key generator start reusing old names.

The allowed paths are small enough to write down:

There is no transition back from cleanup to stored. These rules protect the file only while every writer follows them.

A missing file can be a successful retry

Here is the local cleanup operation:

pub async fn cleanup(
    pool: &PgPool,
    root: &Path,
    id: i64,
) -> Result<bool, Box<dyn std::error::Error>> {
    let key: Option<String> =
        sqlx::query_scalar("SELECT object_key FROM uploads WHERE id = $1 AND state = 'cleanup'")
            .bind(id)
            .fetch_optional(pool)
            .await?;
    let Some(key) = key else {
        return Ok(false);
    };
    // Keys are generated by the service. The lab accepts one simple filename.
    if key.is_empty() || !key.bytes().all(|c| c.is_ascii_alphanumeric() || c == b'-') {
        return Err("invalid generated object key".into());
    }
    match std::fs::remove_file(root.join(key)) {
        Ok(()) => {}
        Err(error) if error.kind() == std::io::ErrorKind::NotFound => {}
        Err(error) => return Err(error.into()),
    }
    sqlx::query("UPDATE uploads SET state = 'cleaned' WHERE id = $1 AND state = 'cleanup'")
        .bind(id)
        .execute(pool)
        .await?;
    Ok(true)
}

The worker deletes the file only when its upload row is marked for cleanup. A file-not-found result is accepted because a previous deletion may have succeeded before the process recorded cleaned in PostgreSQL. Other filesystem errors are returned, leaving the durable cleanup request available for investigation or retry.

The filename validation is intentionally narrow, and the directory belongs to the fixture. This is not a general filesystem sandbox. A production file service should use a trusted storage root and a design which does not turn request paths into arbitrary filesystem access.

Also, this function uses synchronous filesystem calls for tiny fixture files. A latency-sensitive async service needs an appropriate storage adapter or blocking-work boundary. The test checks changes between upload states and the retry decision, not the performance of filesystem cleanup on a runtime thread.

Three endings, one visible attachment

The executable test creates three stored files.

For the first, cancellation chooses cleanup. The recovery operation removes the file. Running it again does not create new work or publish an attachment.

For the second, publication wins. A later cancellation cannot change its state, and cleanup leaves the file in place.

For the third, the test deletes the file after marking cleanup, then runs the worker as if deletion had succeeded before a crash. The repeated deletion is accepted, and the row reaches cleaned. Publication is still refused.

At the end, the public attachment table contains only the second ID. The test checks both stored rows and actual files. A successful cleanup response alone would be much weaker evidence.

The complete fixture is in the lab. It constructs the interruption boundary through stored state; it does not kill a process or simulate a power failure.

Cloud multipart uploads have another unfinished state

An S3 multipart upload is not yet a completed object. Its upload ID and uploaded parts need their own handling. AWS documents the separate initiate, upload, complete, and abort lifecycle.

For that adapter, persist the provider upload identity and track in-flight parts. AWS notes that parts already in flight can still finish after an abort request. An unknown storage result therefore needs reconciliation with the provider; a cleanup row alone cannot prove that no later bytes will arrive. Aborting a multipart upload and deleting a completed object are different operations. Provider lifecycle rules can provide a backstop for incomplete uploads; they do not replace the application's publication decision.

Before connecting this code to a remote store, add adapter tests for an uncertain completion, a failed abort, retries, and the provider's version or conditional-operation rules. The local example gives those operations an owner and a durable record. It does not establish how a cloud provider behaves at each boundary.