SMTP Relay Testing & Diagnostics – Technical Knowledge Base Guide

This article provides a technical reference for testing, validating, and troubleshooting SMTP relay configurations. It is intended for IT professionals, system administrators, and support engineers responsible for mail servers, application mail delivery, monitoring systems, and security enforcement.

The guide covers:

• SMTP relay fundamentals

• Diagnostic methodology

• Command-line testing techniques

• Common failure scenarios

• Security risks & hardening

• Best practices

Product / System / Feature Overview

What is SMTP Relay?

SMTP relay is the process by which an SMTP server accepts email from a client/system and forwards it to another mail server for final delivery.

Relaying may occur:

• Between internal systems and external domains

• Between mail gateways

• From applications to mail servers

• Via authenticated or IP-based trust

Technical Explanation

SMTP Relay Architecture

Typical mail flow:

Application / Client → SMTP Relay Server → Destination Mail Server → Recipient

Key components:

• SMTP Client (MUA / App / Device)

• SMTP Relay / Smart Host

• DNS / MX Resolution

• Destination MTA

Relay Control Mechanisms

SMTP servers regulate relay using:

Relay Limitations & Behaviors

SMTP relay may enforce:

• Rate limits

• Recipient restrictions

• Sender restrictions

• TLS requirements

• Spam filtering

• Greylisting

Use Cases & Environments

SMTP relay is commonly used in:

• Enterprise mail gateways

• Application servers

• Printers / scanners

• Cloud mail services

• Monitoring / alert systems

• Backup mail routing

Diagnostic & Testing Methodology

When validating SMTP relay, test:

✔ TCP connectivity
✔ SMTP handshake
✔ Authentication
✔ TLS negotiation
✔ MAIL FROM / RCPT TO acceptance
✔ Relay permissions
✔ Message submission

Step-by-Step SMTP Relay Testing

✅ 1. Network Connectivity Test

Windows (PowerShell)

Test-NetConnection smtp.server.com -Port 587

Expected:

• TcpTestSucceeded : True

Failure indicates:

• Firewall block

• ISP port filtering

• Incorrect hostname/port

Linux / macOS

nc -zv smtp.server.com 587

✅ 2. Basic SMTP Handshake (Telnet)

telnet smtp.server.com 25

Expected response:

220 smtp.server.com ESMTP Postfix

Test commands:

EHLO test.local

✅ 3. TLS / SSL Testing (OpenSSL)

openssl s_client -connect smtp.server.com:587 -starttls smtp

Validate:

• Certificate chain

• TLS negotiation

• Cipher suites

✅ 4. Full SMTP Relay Test (Swaks)

Swaks = Preferred diagnostic tool

Example:

swaks --to user@example.com \
      --from admin@yourdomain.com \
      --server smtp.server.com \
      --port 587 \
      --auth LOGIN \
      --auth-user username \
      --auth-password password \
      --tls

Validates:

✔ Authentication
✔ TLS
✔ Relay permissions
✔ SMTP conversation

✅ 5. Application-Level Testing

Check:

• SMTP host

• Port

• Encryption method

• Authentication method

• Credentials

• Sender address format

SMTP Relay Verification Checklist

• Server reachable

• Correct port

• TLS policy satisfied

• Auth accepted

• MAIL FROM accepted

• RCPT TO accepted

• DATA accepted

• No relay denial

Common SMTP Relay Errors

❌ Relay Access Denied

Example:

554 Relay access denied

Root Causes:

• IP not trusted

• Missing authentication

• Incorrect relay policy

Fix:

✔ Enable SMTP auth
✔ Add IP to whitelist
✔ Review relay restrictions

❌ Authentication Failed

535 Authentication failed

Root Causes:

• Incorrect credentials

• Disabled SMTP auth

• Wrong auth mechanism

Fix:

✔ Verify username/password
✔ Check auth method (LOGIN / PLAIN / CRAM-MD5)
✔ Enable SMTP AUTH

❌ Connection Timeout

Root Causes:

• Firewall block

• ISP port filtering

• Wrong port

Fix:

✔ Test via PowerShell / nc
✔ Verify outbound rules
✔ Try port 587 instead of 25

❌ TLS Required but Not Provided

530 Must issue STARTTLS first

Fix:

✔ Enable TLS in client/app
✔ Use STARTTLS port (587)

❌ Certificate Errors

Root Causes:

• Expired certificate

• Hostname mismatch

• Missing intermediate CA

Fix:

✔ Renew certificate
✔ Correct FQDN
✔ Install CA chain

Logs & Diagnostics

Check mail server logs:

Linux (Postfix)

tail -f /var/log/mail.log

Exchange Server

Event Viewer → Transport Logs

SMTP Gateway

Review:

• Connection attempts

• Auth failures

• TLS negotiation errors

• Policy rejections

Security Considerations & Risks

? Open Relay Risk

An improperly configured relay server may allow:

• Spam abuse

• Blacklisting

• Reputation damage

Test for open relay:

swaks --to external@gmail.com --server smtp.server.com

? Credential Exposure

Without TLS:

• Credentials transmitted in plaintext

• Vulnerable to interception

Mitigation:

✔ Enforce TLS
✔ Disable insecure auth methods

? Brute Force Attacks

Mitigation:

✔ Rate limiting
✔ Fail2ban / IDS
✔ Strong password policy

Best Practices & Recommendations

✔ Always require SMTP Authentication OR IP Trust
✔ Enforce TLS encryption
✔ Disable legacy auth where possible
✔ Implement rate limits
✔ Monitor SMTP logs
✔ Validate reverse DNS (PTR records)
✔ Avoid port 25 dependency for outbound mail
✔ Use port 587 (submission)
✔ Regularly test relay functionality
✔ Maintain certificate validity

Conclusion

SMTP relay failures typically arise from:

• Network restrictions

• Authentication issues

• TLS misconfiguration

• Relay policy enforcement

A structured diagnostic approach using:

• Connectivity testing

• SMTP handshake validation

• TLS verification

• Swaks testing

provides reliable root cause identification.