Test generated code from an empty output directory

Article · Derived state

The generated client builds on the developer's machine. A fresh CI checkout cannot find one of its modules.

The missing file was generated weeks ago and never removed. The current generator no longer creates it, but the old build directory kept the mistake hidden.

I would check both directions: can the generator produce everything from nothing, and can it handle a directory containing output from the previous input?

Treat generated files as outputs

Cargo gives a build script an output directory. Put generated source there and include it deliberately. This small Rust build-script example shows the basic path:

use std::{env, fs, path::PathBuf};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    println!("cargo::rerun-if-changed=schema/api.txt");
    println!("cargo::rerun-if-changed=template.cfg");
    let schema = fs::read_to_string("schema/api.txt")
        .map_err(|error| format!("required schema/api.txt: {error}"))?;
    let value: i32 = fs::read_to_string("template.cfg")
        .map_err(|error| format!("required template.cfg: {error}"))?
        .trim()
        .parse()?;
    let mut generated = String::new();
    for name in schema.lines() {
        if name.is_empty()
            || name.len() > 16
            || !name
                .bytes()
                .all(|byte| byte.is_ascii_alphanumeric() || byte == b'_')
        {
            return Err("invalid fixture API name".into());
        }
        generated.push_str(&format!("pub fn api_{name}() -> i32 {{ {value} }}\n"));
    }
    let output = PathBuf::from(env::var("OUT_DIR")?);
    let temporary = output.join("client.rs.tmp");
    fs::write(&temporary, generated)?;
    fs::rename(temporary, output.join("client.rs"))?;
    Ok(())
}

Both the schema and template are declared here. If another generator reads configuration or environment variables, declare those inputs too. A visible schema path is not enough when a hidden template controls the result.

Cargo's build-script documentation notes that OUT_DIR can persist across builds. Do not rely on it starting empty. At the same time, the generator must not require files left there by an earlier run.

The build-script rerun article explains change detection. Here the additional question is whether the generator owns a complete, reproducible set of output files.

Test creation, replacement, and removal

In the existing CI workflow, begin with a fresh build directory and generate the client. Compile a small consumer that uses the expected interface.

Then change the schema and run again using the same directory. Confirm that the changed interface is present. Remove a schema item and check that obsolete generated modules do not remain available accidentally.

If the generator produces several files, keep an output manifest or generate into a temporary subdirectory that it owns before replacing its previous output. Do not recursively delete a broad directory supplied by an untrusted input.

Handle failed generation too. A partial write must not look like a valid client from the new schema. Writing to a temporary file and renaming it can help for a single file, subject to the filesystem and platform contract. A multi-file set needs a coherent publication strategy.

Fresh builds are evidence, not the daily fix

Deleting every cache before every build can hide incorrect change detection and make CI unnecessarily expensive. It is useful as one controlled check, not as the only supported workflow.

Likewise, a successful reused build does not prove a new checkout works. Keep both cases because they detect different bugs.

Record the generator version and schema identity in diagnostics or the generated header when useful. Avoid embedding a changing timestamp if it makes identical inputs produce different source for no product reason.

The useful contract is straightforward: declared inputs produce the required files, old outputs cannot supply missing work, and a failed run cannot masquerade as a successful new generation. Once that is true, caching becomes a performance choice instead of part of the generator's correctness.

What the empty-output trial actually checks

The complete build script reads two declared inputs. The schema names the generated functions. The template supplies their returned value. It builds the whole client as one string, writes a temporary file in OUT_DIR, then replaces client.rs by renaming that file. There is no undeclared helper producing half of the example.

The CI driver creates a new temporary package with no previous generated directory. A schema containing alpha and beta, with template value seven, produces a program that prints generation=7:7. Changing only the template to eight produces 8:8. This checks the second declared input rather than relying on a schema edit to trigger generation accidentally.

Next, the trial removes beta from the schema. The unchanged consumer fails to compile at its call to api_beta. That failure is expected. It demonstrates that an obsolete function did not survive from an earlier output. Updating the consumer to use alpha alone then succeeds.

The trial also deletes the required template while previous output still exists. The build fails with a missing-template diagnostic. Restoring the template with value nine succeeds. A generator should not quietly reuse an old client when its required inputs are unavailable.

These observations cover creation, replacement, removal, and a missing input for this small generator. They do not prove that an arbitrary OpenAPI generator is deterministic or that rename alone gives crash durability on every filesystem. A real generator may have includes, environment variables, or external tools that also belong in its declared input set.

Keep this trial in CI beside ordinary incremental builds. Fresh output checks the completeness of generation. Incremental changes check whether Cargo learns when to rerun it. Both are useful, and neither requires deleting the entire workspace on every edit.