emailmarketing.net

Shared IP Pool Recovery — Disaster Runbook for a Poisoned Pool

The disaster-recovery runbook for a shared IP pool torched by a bad tenant — distinguishing a poisoned pool from one struggling sender, attributing and isolating the offender, containing them, delisting the pool's IPs across blocklist zones, evacuating clean tenants to a healthy pool, and deciding whether to rehabilitate or retire the burned IPs.

Operationalesp-operator

When a single bad tenant torches a shared IP pool, the damage lands on every clean tenant riding those IPs. This is the disaster-recovery runbook for that specific scenario: a shared pool where one (or a few) customers' behavior has driven the pool's IPs onto blocklists, into spam-foldering, or under provider throttling, and the innocent co-tenants are now paying for it.

This article is composition, not new mechanics — it sequences procedures that live in full elsewhere and adds only the coordination logic a shared-pool disaster needs. It deliberately does not restate:

What makes this different from single-account recovery: Reputation Incident Recovery assumes one account owns the burned asset and can pause everything while it rebuilds. A shared pool cannot: the offender must be excised without pausing the clean tenants, whose only fault was sharing IPs. The whole procedure is dominated by that constraint — isolate the guilty, protect the innocent, decide the IPs' fate last.

The recovery sequence at a glance

Order matters; each step assumes the previous is done. Delisting before containment just re-lists; evacuating clean tenants before delisting moves them onto still-warm IPs but does not fix the pool.

# Phase Goal Primary cross-link
1 Detect & classify Confirm the pool is poisoned, not one sender struggling Outbound Monitoring
2 Attribute Identify which tenant(s) caused it Spam-Trap Incident Response
3 Contain Stop the bleeding — throttle/pause/evict the offender, purge their queue Account Enforcement, Abuse Desk
4 Protect clean tenants Evacuate innocents to a healthy pool (concurrent with 5) Multi-Tenant Architecture
5 Delist Clear the pool's IPs across every zone, root cause first Spamhaus Listings Deep Dive
6 Rehabilitate or retire Re-warm the cleaned IPs or replace them Reputation Incident Recovery

Steps 3 and 4–5 run in parallel once attribution is firm: containment stops new damage while evacuation and delisting proceed. Do not delist while the offender is still injecting.

Phase 1 — Detect and classify: poisoned pool vs. one struggling sender

The first decision is diagnostic, and getting it wrong is expensive: treating a whole-pool poisoning as one sender's problem lets the fire keep burning the co-tenants; treating one sender's ordinary bad week as a pool disaster triggers a needless evacuation. The monitoring stack that produces these signals is in Outbound Monitoring; this is how to read them for the pool-vs-sender question. The discriminator is scope: does the damage track the pool's IPs regardless of which tenant sent, or does it track one tenant regardless of which IP they used?

Signal Points to a poisoned pool Points to one struggling sender
Blocklisting A shared-pool IP is listed; the listing's evidence spans multiple tenants' mail Only a dedicated IP is listed, or the listing traces to one tenant's domains/content
Placement / engagement collapse Every tenant on the pool sees the same provider degrade simultaneously One tenant degrades while co-tenants on the same IPs hold steady
Deferral / rejection spike Per-destination rejections rise for all senders on the pool at one provider Rejections concentrate in one tenant's traffic
Complaint / trap rate Pool-aggregate rate breaches threshold but no single tenant dominates it One tenant's rate is an outlier; pool-aggregate is fine once they're removed
DMARC / auth failures The pool's shared sending domain is being flagged A customer's own delegated domain is failing

The load-bearing tool is per-pool scoping with mandatory customer attribution — an alert on a shared asset is only half an alert until it is mapped to the tenant(s) behind it (see Outbound Monitoring § Scoping). Practical test: strip the top-suspect tenant's traffic from the pool aggregate and recompute. If the pool metrics fall back inside threshold, you have one offender (a containment problem); if they stay red, the pool is broadly contaminated (a full-evacuation problem, possibly multiple offenders, and a harder root cause). Watch the shared estate hardest — a shared-pool IP listing punishes every tenant on it, which is why blocklist and trap watchers page at platform severity for shared assets.

Severity gate — is this actually a disaster? Escalate into this runbook (rather than the routine enforcement loop) when any of: a shared-pool IP hits an SBL/CSS-class blocking listing (not merely informational); a major provider (Gmail/Microsoft/Yahoo) starts spam-foldering or hard-rejecting the whole pool; or two or more clean tenants report inbox loss they did not cause. Below that bar, this is ordinary per-tenant enforcement, not pool recovery.

Phase 2 — Attribution: find the offender(s)

Containment and delisting both require knowing who did it — you cannot cite an AUP clause against an unknown tenant, and Spamhaus will demand to know how the problem was solved before delisting (see below). Attribution mechanics for the specific case of a trap-driven poisoning are covered in full in Spam-Trap Incident Response § Detection and attribution — the same per-send tagging (tenant IDs, campaign IDs, per-customer DKIM d=, Feedback-ID-style headers) that makes any shared-asset signal actionable is what maps the poisoning to a tenant.

Shared-pool specifics to resolve before acting:

  • One offender or several? The strip-and-recompute test from Phase 1 answers this. A pool contaminated by many marginal tenants is a stratification failure, not a single bad actor — the fix is re-tiering the pool (see Avoiding Blocklistings § Risk-tiered IP segmentation), not evicting one customer.
  • Compromise vs. malice. A tenant whose prior behavior was clean and who suddenly spikes is likely a compromised account, not a spammer — same pool symptoms, completely different remediation and a different tone of customer contact. Prior good history is the tell; resolve this before you send the eviction notice.
  • Which IPs carried the bad mail. In a pool, the offender's traffic may have touched only some of the pool's IPs. Map bad sends to specific IPs — it scopes the delisting work in Phase 5 and informs the retire-vs-rehab call in Phase 6.

Preserve trap confidentiality throughout: blocklist/reputation vendors report trap-hit metrics without burning trap identities, and that discipline must survive into your customer notification (see Spam-Trap Incident Response).

Phase 3 — Containment: stop the bleeding

Once attribution is firm, sever the offender from the pool immediately. This is the Outbound Monitoring response ladder and the Account Enforcement state machine applied under disaster pressure — the difference from routine enforcement is that here every hour the offender keeps sending re-lists the IPs you are about to try to delist, so containment is not negotiable and precedes any delisting request.

  1. Throttle-to-zero or pause the offending tenant — customer- or tenant-scoped, human-gated resumption. For a confirmed poisoning this jumps straight to pause, skipping the gentle throttle used for ambiguous cases. Fraudulent or clearly malicious accounts skip the courtesies entirely (Abuse Desk § Remediation workflow); a suspected compromise is paused and secured (credentials rotated, API keys revoked) rather than terminated.
  2. Purge the offender's queued and deferred mail. Deferred bad mail retrying for up to 72 hours after you "stopped" it keeps generating complaints, trap hits, and fresh listings under the pool's IPs — draining the queue is part of stopping the bleeding, not a later cleanup (the queue-purge discipline from Reputation Incident Recovery § Stop the bleeding).
  3. Notify with the specific AUP/ToS clause cited — this keeps the customer agreement intact and protects the platform against dispute from either side (Account Enforcement, Abuse Desk). The eviction decision itself (suspend to remediate vs. terminate) follows the enforcement ladder; termination survives dispute only with documented communication, multiple intervention attempts, and stakeholder alignment (Avoiding Blocklistings § Enforcement of last resort).
  4. Rebuild suppressions from the incident window. Every bounce, complaint, and unsubscribe the offender generated must be honored on the pool going forward — merge them in before any tenant resumes at volume (Reputation Incident Recovery § Rebuild the data layer, Suppression-List Architecture).

Containment is complete when no bad mail is entering or leaving the pool. Only then does delisting stick.

Phase 4 — Protect the clean tenants: evacuate to a healthy pool

The innocents cannot wait out the delisting timeline on burned IPs — every day on a listed shared IP costs them inbox placement they did nothing to lose. Migrate them to a healthy pool concurrently with the delisting work. The routing and isolation primitives are in Multi-Tenant Architecture; the migration is a re-binding of the routing object, not a rebuild of the tenants.

Migration mechanics:

  • Re-point the routing object, not the tenant. A tenant sends through whatever pool its routing binding names (configuration set / ip_pool_name). Move clean tenants by re-binding them to a healthy pool — their identities, suppressions, and metrics are logical and stay put; only the physical IP routing changes (see the logical-tenant-vs-physical-pool split in Multi-Tenant Architecture § The three isolation layers).
  • Destination pool must have headroom and matching warmth. Dropping evacuees onto a pool that is itself near capacity, or cold for the providers they send to, just moves the problem. Warm-up state is per IP × per receiving ISP and it decays — a healthy pool that has not carried Gmail volume is not warm for Gmail (see IP Warm-Up, Multi-Tenant Architecture § Managed pools). If no warm pool has headroom, the shared-pool overflow mechanism is the shock absorber: spill evacuee volume across warmed capacity and ramp the rest rather than blasting cold IPs.
  • Prioritize by exposure. Evacuate the highest-volume and most placement-sensitive clean tenants first — they lose the most per hour on listed IPs. Low-volume tenants can tolerate a short hold.
  • Do not evacuate onto another risky tenant's pool. Preserve the risk-tiering: clean, established senders go to a clean, established pool, not to whatever has space. Contamination stays contained within a tier (Avoiding Blocklistings § Risk-tiered IP segmentation).
  • Update authentication if the sending domain changes. If evacuees moved to IPs under a different shared sending domain, re-verify SPF/DKIM/DMARC alignment before they resume (customers break their own DNS routinely — the customer-auth watcher catches this; see Customer Domain Authentication).

A well-tiered platform makes this cheap: the healthy pools already exist, and evacuation is a routing change. A platform with one undifferentiated shared pool has nowhere to evacuate to — which is the argument for stratification before an incident, not during one.

Phase 5 — Delist the pool's IPs across the blocklist zones

With the offender contained and the queue purged, work delisting. The per-zone criteria, return codes, and self-service-vs-investigated mechanics are in Spamhaus Listings Deep Dive and Blocklists & Spamhaus — do not restate them; the shared-pool wrinkles are:

  • Root cause fixed first, and provably. Delisting a shared-pool IP while the cause persists burns your limited self-removals and, for the SBL, invites a wider re-listing. The abuse desk must be able to describe how the problem was solved — for a spammer tenant, Spamhaus expects genuine deprovisioning, not just a sending pause (Spamhaus Listings Deep Dive § SBL delisting). This is why Phase 3 comes first.
  • Every listed IP, every relevant zone. Enumerate each shared-pool IP the bad mail touched (from Phase 2) and check each against the applicable zones. Match the workflow to the list:
Listing class Removal path Shared-pool note
CSS (127.0.0.3, automated, low-reputation email) Fix cause → self-remove via checker, or wait ~72 h auto-expiry The list a sloppy-tenant poisoning most often hits; self-removals are limited — don't burn them before containment is real
SBL (127.0.0.2, manual) Responsible network's abuse desk writes to the SBL Removals Team explaining the fix The escalation risk: tolerating the offender can widen the listing from one IP to your whole allocation — the existential case for eviction
DBL (127.0.1.x, domains) Auto-expiry once criteria stop matching, or checker form Relevant when the poisoning implicated the shared sending/click-tracking domain, not just IPs
Provider-internal (Gmail/Microsoft/Yahoo reputation) Postmaster/support escalation with incident + remediation description Not a DNSBL; clears on its own timeline as clean volume rebuilds — see Escalation & Mitigation Channels
  • Compromise-driven listings clear fast; spammer-driven ones do not. If the poisoning was a documented, remediated account compromise, operators and providers routinely reverse listings quickly once you can show the pool is secured. A deliberate-spammer poisoning is harder and slower — the delisting request must show the customer was actually removed.
  • Delisting ≠ instant unblock. Removal from a zone propagates on the receiver's refresh schedule (minutes for DQS/ZEN subscribers, up to 24 h for laggards), and receiver-side reputation damage decays separately on top of that. Do not tell evacuated or returning tenants the pool is "clean" the moment the zone clears.
  • Escalate delisting requests to a human, always. Mishandled or repeated automated delisting requests can make a listing permanent (Outbound Monitoring § Escalate-to-human criteria).

Phase 6 — Rehabilitate or retire the poisoned IPs

After the offender is gone, the clean tenants are evacuated, and the zones are clearing, decide what to do with the burned IPs. The general replace-vs-rehabilitate analysis — new assets start negative not neutral, reputation follows the mail not just the asset, compromise-driven listings clear quickly — is in Reputation Incident Recovery § Replace or rehabilitate?. The default is still rehabilitate; the shared-pool-specific factors that push the decision:

Factor Favors rehabilitating the pool's IPs Favors retiring/replacing them
Cause Single contained offender, cleanly evicted; compromise, documented and remediated Broad multi-tenant contamination (stratification failure); repeat poisoning of the same pool
Listing status Delistable; operators responsive to the remediation evidence Permanent/repeat SBL listing after the offender was tolerated too long; IPs on lists with no practical removal path
History depth Short incident window on otherwise clean IPs Long abuse history, or IPs the offender had been quietly degrading for months
Cost Re-warm beats acquiring and warming fresh IPs from zero Sustained hard blocks at major providers despite a clean multi-week re-warm

Rehabilitation path (chosen pool): treat the cleaned IPs as a re-warm, not a cold start — receivers have concrete bad history tied to them, so ramp only recently-engaged traffic, hold or step back at any provider still deferring, and re-check blocklists before each volume increase (the re-warm-vs-cold-warm-up differences are tabulated in Reputation Incident Recovery § Re-warming vs cold warm-up). Note the dormancy rule: an IP idle over 30 days during the incident-and-recovery window needs a fresh warm-up regardless of its prior status — a burned pool held out of rotation for weeks self-inflicts a re-warm requirement.

Retirement path: if the pool is beyond rehabilitation, relinquish the burned IPs cleanly and stand up a fresh pool warmed from zero, then migrate the (already-evacuated) tenants onto it once warm. Retire the burned IPs rather than quietly re-issuing them to new tenants — new senders inheriting an IP with recent abuse history burn faster than the pool recovered. When acquiring replacement IP space, vet its reputation history first (IP Acquisition Diligence); a fresh pool built on inherited-dirty space is no fresh start.

Either way, the offender does not come back onto shared infrastructure. A tenant capable of poisoning a shared pool belongs on a dedicated IP (reputation attaches to them alone), on a stricter risk tier, or off the platform — never back into the pool they burned.

Related articles

#esp-operations#ip-pools#shared-pool#incident-response#disaster-recovery#blocklists#delisting#tenant-isolation#migration#re-warming#containment