How to Test an SMTP Provider Before You Commit

A 14-day structured trial tells you more about an SMTP provider than any sales call, and it costs you one campaign's worth of effort. Run authentication on day 1, seed placement on days 2 to 3, a real send to your most engaged 5,000 on days 4 to 7, a volume ramp on days 8 to 11, and failure plus support testing on days 12 to 14. Seven numbers come out the other end, and they decide it.

I have run this on SendGrid, Mailgun, Amazon SES and self-hosted Postfix. Every time, at least one provider failed on something the pricing page never mentioned.

Why a trial beats a sales call

The sales page gives you a monthly allowance, a per-email price and an uptime number. None of those are what breaks in production.

Questions to ask an SMTP provider before you buy covers what you get them to say. This article covers what you make them prove.

The 14-day test plan

Run it in this order. Each phase depends on the one before it, so do not skip ahead.

Fourteen days is the floor. Under a week you only test the happy path, and that is not where providers fail.

Day 1: authentication and one message

Do not send anything else until this passes. Publish SPF, DKIM and DMARC on a sending subdomain, never the root domain, then send exactly one message to a Gmail address you control and read the raw headers.

Check the PTR first. A dedicated IP whose reverse DNS does not resolve back to your HELO hostname gets rejected at Gmail and Microsoft no matter what else you do right.

dig -x 203.0.113.45 +short
dig TXT selector1._domainkey.mail.example.com +short
dig TXT _dmarc.example.com +short

Then send one message with swaks, which shows you the full SMTP conversation.

swaks --to [email protected] \
      --from [email protected] \
      --server smtp.provider.example \
      --port 587 --auth LOGIN \
      --auth-user "$SMTP_USER" --auth-password "$SMTP_PASS" \
      --h-Subject "day 1 auth test" \
      --tls

Open the original in Gmail and look for spf=pass, dkim=pass and dmarc=pass. Anything less than all three on day 1 is a config problem, and a provider whose docs cannot get you there in an hour has told you something about days 2 to 14.

Days 2 to 3: seed placement per receiver

Send the same content to 25 to 40 seed mailboxes spread across Gmail, Outlook and Yahoo, weighted roughly to match your real list's provider mix. Log inbox, spam or missing per provider.

Microsoft is what separates providers. Outlook weighs IP reputation far more heavily than Gmail, so a pool with noisy neighbours shows up as Outlook-only spam while Gmail looks fine. That split tells you more about the pool than anything the provider will say.

Seeds have no engagement history, so treat the number as a floor rather than the truth. The full method is in inbox placement testing with seed lists, and the gap between seed numbers and what your real subscribers see is covered in seed list vs real inbox placement.

Days 4 to 7: a real send to 5,000 engaged contacts

Seeds test the plumbing. Real recipients test the reputation. Take your 5,000 most engaged contacts, anyone who opened or clicked in the last 90 days, and send one campaign a day for four days.

Engaged contacts are the right pick because they are the least likely to complain. A complaint rate above 0.1% on this segment is a provider signal, not a list signal.

Record per send:

Google's own sender guidelines put the hard complaint limit at 0.30% and the target at 0.10%. On your most engaged segment you should be nowhere near either. If you are, the mail is landing badly enough that people are hitting report spam instead of ignoring it.

Days 8 to 11: ramp and measure throughput

Push volume up on consecutive days and watch the 4xx rate rather than the delivered count. Delivered is a lagging number. Deferrals move first.

Time the whole send, not the API calls. A provider that accepts 60,000 messages in four minutes then takes nine hours to deliver them has just moved the queue to their side, which is fine until you need a campaign out before lunch.

Two numbers to be strict about. Sustained throughput of 40 to 100 messages per second is normal on decent dedicated infrastructure, and API time to first byte should sit under 400ms. Monthly allowances are marketing; the hourly throttle decides whether a 400,000-address campaign goes out in a morning or over three days.

Days 12 to 14: break it on purpose

Working systems all look the same. The differences show up in failure.

Bounce classification. Send to a mix of deliberately bad addresses: a nonexistent mailbox on a domain that exists, a nonexistent domain, and a full mailbox if you can arrange one. A provider that reports a 550 user unknown as a soft bounce will keep retrying dead addresses and quietly poison your reputation. Check the codes it hands back, not the labels in the dashboard.

Complaint reporting delay. Mark one of your own seed messages as spam at Gmail and time how long it takes to appear in the provider's feed. Under 24 hours is fine. Over 72 hours means you learn about a spike three days late.

API failure behaviour. Send a malformed payload, an oversized message and a burst past your rate limit. You want a clear 4xx with a machine-readable reason, not a 500 or a silent accept.

Suppression handling. Unsubscribe one address, then send to it again. It should be blocked at the provider. If it goes out, the suppression list is decorative.

How to test support properly

Most people test support with a billing question during office hours, the one question every provider answers fast. That tells you nothing.

Open a real deliverability ticket at 2am on a Saturday. Ask something technical, such as why you are seeing 421 4.7.0 Try again later from Gmail on the IP they assigned you, with the timestamp and the IP included.

The tell is whether the first reply engages with your IP and timestamp or points you at a generic warm-up article. A provider whose weekend tier can only forward tickets will be doing exactly that on the morning your campaign stalls. What actually matters in an SMTP provider SLA covers what the contract does and does not promise.

Check the IP and the pool before you commit

Do this on day 1, before your first real send, and again on day 14. Five minutes.

1. Look the assigned IP up on the Spamhaus reputation checker and on Barracuda's lookup. A listed IP handed to you on day 1 is grounds to ask for a different one.
2. Confirm the PTR resolves to a hostname you control or that at least matches the sending domain. Generic reverse DNS like 45.113.0.203.provider-pool.net is a shared-pool signal.
3. Check the neighbours. Run reverse lookups across the surrounding /24 and see who else is there. A /24 full of unrelated hostnames on cheap PTRs is a crowded pool.

for i in $(seq 40 50); do
  echo -n "203.0.113.$i "
  dig -x 203.0.113.$i +short
done

On a genuinely dedicated IP the neighbours are quiet, the PTR is yours, and nothing in the range is listed. On a shared pool sold as "dedicated-ish", you find out here.

Score two providers side by side

Run the same 14 days on two providers with the same list split 50/50. Score each line 0, 1 or 2, where 2 is a clear pass.

Maximum is 46. My rule: 39 or above is a go, 31 to 38 is a go only if the misses are fixable from your side, and under 31 is a no regardless of price. A zero on any 3x line is an automatic no on its own, because those five are the things you cannot work around.

Price is deliberately absent. A provider that is 30% cheaper and scores 12 points lower is more expensive within a quarter, once you count the campaigns that land in spam.

What the trial answers that the sales page never does

None of these have an answer on a pricing page, and all six have cost someone a quarter of email revenue.

One discipline holds it together: change one variable at a time. Swap the template, trim the list and switch IPs between two sends, and the numbers mean nothing.

How BulkEmailSetup helps

We build the infrastructure you would be testing for: your own server, your own IPs that stay yours, SPF, DKIM, DMARC and PTR verified before the first send, MTA tuning, bounce classification that gets the codes right, and a warm-up plan sized to your list. Day 1 of the plan above is our handover checklist, not something you chase support about.

Basic is $549 one-time, covering 1 SMTP server, 3 dedicated IPs, 25,000 emails/day and unlimited contacts. Higher tiers scale to 15 IPs and 200,000 emails/day. See pricing, or 12 questions to ask an SMTP provider before you pay for what to get in writing before the trial starts.

Frequently asked questions