To set up an email server on a VPS you need to install Postfix (SMTP sending) and Dovecot (IMAP/POP3 receiving), configure your domain's DNS records (SPF, DKIM, and DMARC), and obtain a TLS certificate. With those four pillars in place, your domain sends and receives email independently.
Running your own mail server gives you complete control over your mailboxes, removes per-seat fees from third-party services, and reinforces your brand identity. That said, it's a setup that demands attention to detail — one wrong step can send your messages straight to spam or get them rejected outright. This guide walks you through the right path.
Is Running Your Own Email Server Worth It?
Before diving in, it's worth being honest about what's involved:
- Advantages: complete privacy, no per-mailbox cost, full control over spam filters and retention policies.
- Disadvantages: a new VPS IP has low reputation; you need to keep the server updated and monitor blacklists regularly.
If you only need a handful of business mailboxes and are willing to do an IP warm-up, this approach makes excellent sense. For high-volume marketing (thousands of emails per day), consider pairing your inbound server with a relay like Amazon SES or Mailgun for outbound.
Check the available infrastructure options in our VPS servers section before choosing a plan.
Prerequisites
- VPS running Ubuntu 22.04 or Debian 12 (this guide assumes Ubuntu).
- Root SSH access.
- A registered domain with access to its DNS zone.
- A fixed public IP address (included with most VPS plans).
- Port 25 unblocked — check with your provider first; many block it by default and require a support request to open it.
Step 1 — Prepare the Server
Update the system and set the hostname
The server hostname must match the PTR (rDNS) record for your IP; otherwise, many remote servers will reject your mail outright.
apt update && apt upgrade -y
hostnamectl set-hostname mail.yourdomain.comNext, ask your VPS provider to set the PTR record for your IP to point to mail.yourdomain.com. Most providers allow this from the control panel.
Install Postfix
apt install postfix -yDuring installation, choose Internet Site and type your domain (yourdomain.com) when asked for the mail system name.
Install Dovecot
apt install dovecot-core dovecot-imapd dovecot-pop3d -yStep 2 — Configure Postfix for Sending and Receiving
Edit /etc/postfix/main.cf. The key parameters to set or verify:
myhostname = mail.yourdomain.com
mydomain = yourdomain.com
myorigin = $mydomain
inet_interfaces = all
mydestination = $myhostname, localhost.$mydomain, localhost, $mydomain
home_mailbox = Maildir/
smtpd_tls_cert_file = /etc/letsencrypt/live/mail.yourdomain.com/fullchain.pem
smtpd_tls_key_file = /etc/letsencrypt/live/mail.yourdomain.com/privkey.pem
smtpd_use_tls = yes
smtpd_tls_security_level = maySave and restart: systemctl restart postfix
Step 3 — TLS Certificate with Let's Encrypt
Without TLS, modern mail servers will flag your messages as insecure or reject them entirely.
apt install certbot -y
certbot certonly --standalone -d mail.yourdomain.comCertbot will renew the certificate automatically every 60 days. Add a renewal hook to reload Postfix and Dovecot after each renewal.
Step 4 — Essential DNS Records
Without the right DNS records, your mail will land in spam or be rejected. Set up these four entries in your DNS zone:
| Record | Type | Value / Purpose |
|---|---|---|
| MX | MX | mail.yourdomain.com (priority 10) — tells the internet where to deliver incoming mail |
| A | A | mail.yourdomain.com → YOUR_VPS_IP — resolves the mail server hostname |
| SPF | TXT | v=spf1 mx ~all — authorizes your VPS to send on behalf of your domain |
| DMARC | TXT | v=DMARC1; p=quarantine; rua=mailto:[email protected] |
The DKIM record requires one extra step: install opendkim, generate the key pair, and publish the public key as a TXT record with your chosen selector. DKIM cryptographically signs every outgoing message and is the single most important factor for deliverability.
apt install opendkim opendkim-tools -y
opendkim-genkey -t -s mail -d yourdomain.comThe generated mail.txt file contains the TXT record value you need to add to your DNS zone.
Step 5 — Test and Monitor
Once everything is configured, verify it works:
- Send a test email to mail-tester.com for a deliverability score (aim for 9+/10).
- Use MXToolbox to confirm that MX, SPF, and DKIM are correctly published.
- Tail logs in real time:
tail -f /var/log/mail.log - Set up blacklist monitoring alerts (recommended tool: MultiRBL).
The team at elenlace.com handles this entire setup for you — from installation to ongoing IP reputation monitoring.
Key Takeaways
- A VPS mail server needs four components: Postfix (SMTP), Dovecot (IMAP), TLS (Let's Encrypt), and correct DNS records (MX, SPF, DKIM, DMARC).
- The PTR (rDNS) record is just as important as the MX record — without it, many servers will reject your messages.
- DKIM is the single most impactful factor for deliverability; do not skip it.
- Port 25 may be blocked on your VPS by default; check and request it be opened before starting.
- Monitor blacklists regularly — a new IP has neutral reputation that you need to build gradually.
Would you rather have an experienced team handle the full setup? Contact us at elenlace.com — we configure your mail server, secure it, and monitor its reputation so your emails always land in the inbox.
FAQ
How long does DNS propagation take?
DNS records can take anywhere from 15 minutes to 48 hours to propagate globally, depending on the configured TTL and your provider. For quick testing, set low TTLs (300 seconds) while configuring; raise them to 3600 or higher once everything is stable.
Can I use the mail server with Outlook or Thunderbird?
Yes. Configure your client with: incoming IMAP server mail.yourdomain.com port 993 (SSL/TLS), outgoing SMTP server mail.yourdomain.com port 587 (STARTTLS). You'll also need to enable the submission port (587) in Postfix and configure SASL authentication.
Will my emails still end up in spam even after correct configuration?
They might at first, because your IP is new and has no sending history. The "warm-up" process means starting with low sending volumes and increasing gradually. With SPF, DKIM, and DMARC properly configured, your reputation improves within a few weeks.
What's the difference between running Postfix and using a service like Google Workspace?
Postfix on your VPS gives you full control and zero per-mailbox cost, but requires ongoing technical administration. Google Workspace or Zoho Mail are managed services with high deliverability from day one, ideal if you don't want to manage infrastructure. Many businesses use both: a VPS for transactional email, and a managed provider for team mailboxes.
Useful resources
Other providers and guides worth comparing:
