Your agent test should not be able to send a real email

Article · Through the layers

The test prompt says, “This is a simulation. Do not send real messages.” The application still loads its normal mail credential.

That test environment can send real messages. The prompt is asking it not to.

For end-to-end agent tests, remove the real effect capability where it is unnecessary. Let the workflow exercise the send decision against a capture adapter instead.

Make the environment choose the adapter

Select the adapter using trusted deployment configuration. Do not let generated arguments name an environment, credential, or arbitrary mail endpoint.

A Rust interface can keep capture separate from delivery:

fn mail_valid(recipient: &str, subject: &str, body: &str) -> bool {
    recipient.len() <= 64
        && recipient.ends_with("@example.test")
        && recipient.bytes().filter(|byte| *byte == b'@').count() == 1
        && !recipient.bytes().any(|byte| byte.is_ascii_whitespace())
        && subject.len() <= 128
        && body.len() <= 1024
}
pub struct Mail {
    pub recipient: String,
    pub subject: String,
    pub body: String,
}
#[derive(Default)]
pub struct CapturedMail {
    pub messages: Vec<Mail>,
}
impl CapturedMail {
    pub fn send(&mut self, mail: Mail) -> Result<(), &'static str> {
        if !mail_valid(&mail.recipient, &mail.subject, &mail.body) {
            return Err("invalid captured message");
        }
        self.messages.push(mail);
        Ok(())
    }
}

This example captures the intended effect in memory. A real test adapter should also enforce size limits and expose assertions about the selected recipient and content. Use synthetic addresses and documents.

Do not initialize a production mail client before deciding which adapter to use. Test configuration should fail closed if a production credential or forbidden endpoint is supplied.

The internal-tool security article explains the surrounding gateway. This environment boundary belongs below the model's wording and above any external effect.

Check the paths around the adapter

A fake send method is not enough if another tool can execute a shell command, call arbitrary HTTP endpoints, or retrieve the production secret. Review the capabilities the test process actually has.

Keep production credentials out of its environment and secret store access. Restrict network egress according to the infrastructure's supported controls. Verify those controls in the test environment rather than assuming an adapter covers every route.

OWASP's excessive-agency guidance is relevant here: a tool set can expose more functionality and authority than the workflow requires.

Exercise the send decision

Run a workflow that should prepare and send a message. Assert one captured message with the expected authorized recipient. Then use a forbidden recipient, missing approval, and a document that asks the agent to send elsewhere.

The denied cases should capture nothing and cause no external request. The approval-boundary article explains binding approval to the resolved action.

Also inspect the actual adapter selected by startup configuration. Testing CapturedMail directly does not prove the end-to-end workflow uses it.

Keep provider delivery as a separate check

A test of CapturedMail alone establishes only that the adapter stores a message. It does not check authorization or approval. To test those rules, run the real gateway with the capture adapter and check the recipient, rejected calls, and absence of external requests.

Neither test proves SMTP behaviour, provider authentication, delivery, or inbox placement.

Use a separate controlled provider integration environment when those are the questions. Give it dedicated credentials and approved destinations, with explicit limits.

The useful separation is between deciding to send and possessing the ability to deliver to real people. Most agent regression tests need the first. Giving them the second by default adds a consequence the prompt cannot reliably remove.

Run the capture process with no external route

The capture adapter is also exercised through the shared gateway. Missing approval captures zero messages. An approved synthetic notification captures one message for the expected example-test recipient. A real-domain recipient is refused before another captured effect. The fixture adapter validates field lengths as well as the destination convention.

The CI driver builds a small static Rust probe with the same gateway module. It starts that probe in a Docker container with network mode none. Startup accepts only capture mode; requesting SMTP mode exits with an error. The probe checks that its named production-secret environment variables are absent.

Inside that isolated process, a loopback connection succeeds as a positive control. Connecting to an external documentation address fails with Linux ENETUNREACH, and the IPv4 routing table has no routes. This distinguishes a process with working sockets but no external path from a test that merely never tried to send anything.

The probe and CI driver retain these checks. Isolation applies to this capture process, not the whole CI job. The surrounding job still needs network access for dependencies and artifact upload. The probe has no shell tool, production secret-store connection, or production mail client.

This does not test SMTP authentication, provider delivery, or inbox placement. Nor does checking a few environment names prove an arbitrary application's entire secret inventory is empty. A real test deployment must remove the actual credentials and alternate connectors its process could reach.

The useful boundary is below the prompt. Generated content can request a send, but it cannot switch this process into delivery mode. The gateway still checks permission and approval, so removing network access has not replaced application-policy testing. Both controls run around the same captured action, with synthetic data and a bounded result.