This is is going to be short and sweet. If you are running opensmtpd or dovecot inside FreeBSD jails you may need specific changes for this configuration, depending on your jail setup.
For dovecot I had to add listen = * to the config.
For opensmtpd I needed to have listen on localhost port <portnumber> instead of listen on lo0 port <portnumber> for smtpd.conf.