Mail relay to customer server

[ispcomm]

Hi, all

I have implemented a feature for VHCS that I would like to push into omega.

The feature permits vhcs to become an internet MX host that cleans mail and delivers if to another (customer) server instead of dropping it in the mail boxes.

The change is not localized in a single file, but affects a couple of engine files and the php frontend. A column is added in the domain definition to hold the destination server.

Integrating this code in Omega is trivial, but requires change to several files and a single patch might not be possible.

What would be the best way to do it?

ispcomm

[kilburn]

I would LOVE this feature! Single patch should be possible by downloading a fresh svn tree, applying the changes and using 'svn diff' command.

If you attach the patch to this post I will test and commit it to the trunk...

[ispcomm]

Well.. it's not that easy because the mods involve:

1. A change in the database
2. A change in the engine (to accomodate new logic + change of dbase)
3. A change in the postfix template (new table is generated)
4. A change in the php frontend and corresponding templates
5. A new update script (i.e. RC5->RC6).

At the current rate of development, I don't have the time to actually implement it as a single patch against svn (head) as that changes often and I don't have developer access to implement the change gradually (step by step) in parallel with others.

Apart from getting some "sponsors" for becoming a dev (hmm... anyone wants to help?) I only see one possible solution: To open a new ticket were I'll add several patches as soon as they're available and then have them pushed in svn quickly by you or someone else (rats?).

You'll have to trust me in the mean time that the end result will be good and workable (I use it in production).

Other than that I can only branch ispcp (well... I have done so already, but only for my small personal mods) and keep it in sync with the official development until the full patch is ready. At that point it could be released as a single monolithic patch or as a series of patches. This approach is also good, but will take more time, and will need to be done against a known good version (like RC6).

Personally I think that the best would be to try to implement it as soon as RC5 becomes official (unless consensus is that this feature will have to wait until 1.0.0).

What do you think could be the best way?

ispcomm

[Cube]

Sounds complicated. I put the relevant domains into the existing transport-table and things work.

[kilburn]

I'm sorry ispcomm but I'm not an admin so I can't give svn accounts to anyone. By the way, there's people using the trunk version on their servers and we're at RC phase so changes should be as "atomic" as possible. Commiting partial features is not a good idea, but svn merge should help here.

Cube Wrote:Sounds complicated. I put the relevant domains into the existing transport-table and things work.

You should also set the "relay_domains" and "relay_recipients_maps" to tell postfix wich accounts are valid an wich aren't (otherwise you'll be relaying a lot of mails to unexistant accounts, generating bounces, etc.).

I don't know wich is your approach ispcomm, but I see this needed changes:

1. CONFIGS: Modify the postfix config adding two hash tables: "relay_domains" (relay_domains key) and "relay_recipients" (relay_recipients_map) key.
2. CONFIGS: Modify the dns templates to allow relaying. I would point the MX record to mx.domain.tld., set the mx.domain.tld record to BASE_SERVER_IP and the mail.domain.tld to point to a new RELAY_IP (the customer one).
3. GUI: Allow the admins & resellers (I wouldn't allow this at client level) to set a domain as "relayed to XXXXXX". Sections modified: domain creation, domain edition (this one on the admin level too, but the "edit domain" link is only visible if you set the HOSTING_PLANS_LEVEL to "admin" in "gui/includes/ispcp-lib.php"). "Domain" table should be modified at this point, adding a "mail_relay" field to the "domain" table. '_no_' would mean "domain not relayed" and 'xxx.xxx.xxx.xxx' would mean "relayed to this ip"
4. ENGINE: Modify the dmn_mgr to add the new "RELAY_IP" tag in bind parts (it should point to BASE_SERVER_IP if the domain isn't relayed, customers IP if it is).
5. ENGINE: Modify the dmn_mgr to check the "mail_relay" config and if it's on create the domain entry in the "relay_domains" hash db (warning: you must ensure that it's not added as a virtual domain too!) and create the appropiate entry in the transport table.
6. GUI: Allow the client to provide a list of valid recipients (you could re-use the mail_users db table, just add a new 'mail_type')
7. ENGINE: Modify the mbox-mgr to manage the "relay_recipients" hash db using mail_users ispcp table.

Maybe I forgot something. Wich is your aproach ispcomm?

As I'm only offering this service to customers who have their own Exchange server, right now I'm using a quite different approach. I install them a script that exports the valid mail addresses from AD, uploads them to the server (scp) and runs a sudoed script that fills the relay_recipients table and reloads postfix. If anyone is interested in the actual scripts, etc. I could write a how-to....

[ispcomm]

Kilburn,

my approach si similar to yours. The only shortcuts I have taken is:

1. I do not modify the domain template. I just use an fixed ip or a dyndns service for the customer relay.

2. Do not export recipient maps from the "Exchange" but rely on the verify daemon to verify valid recipients before accepting the inbound mail.

The rest is very similar to what you describe and I don't do it by hand as the virtual domain table (in vhcs) is overwritten each time a new domain is created and a domain in the virtual table cannot also be delivered outside.

I'll go forward and implement the needed changed and try to keep them smerged for the time being. The change is simple enough to make in one of the RC versions (I hope)

ispcomm.

[kilburn]

ispcomm Wrote:1. I do not modify the domain template. I just use an fixed ip or a dyndns service for the customer relay.

I did this because otherwise the {mail,smtp,pop3,imap}.domain.tld entries would point to the primary MX (our mail cleaner) and not the user's server which has the real mailboxes. Without this change the user can't set up their mail client using those records...

ispcomm Wrote:2. Do not export recipient maps from the "Exchange" but rely on the verify daemon to verify valid recipients before accepting the inbound mail.

I must admit that I didn't take a look at it, because I had gone to the postfix documentation solution directly. As your solution seemed cleaner I've taken a look at it, but on the verify manpage you can read:

Quote:BUGS
The address verification service is suitable only for
sites that handle a low mail volume. Verification probes
add additional traffic to the mail queue and perform
poorly under high load. Servers may blacklist sites that
probe excessively, or that probe excessively for non-exis-
tent recipient addresses.

If the persistent database ever gets corrupted then the
world comes to an end and human intervention is needed.
This violates a basic Postfix principle.

As both reasons are show-stoppers for me I'll keep my current setup Do you want me to write the how-to? It's basically an ispcp adapted version of the howto linked on the official postfix docs.

[rbtux]

Well it's always better to have all valid recipient addresses on the relay. If thats not possible postfix brings the parameter reject_unverified_recipient (or similar)... This should work too but it also increases the network traffic...

[ispcomm]

Kilburn,

I think yours is the proper way of doing things. I made more shortcuts because I have few customers with this requirement.

Off course verify_recipient is done only on domains that we're the relay for, and it's good practice to put our ispcp server in the whitelist of the customers server.

OTOH, the postfix server will cache the result of a verify and the database does not corrupt so often (I would say never). At the end you can always delete it and restart from scratch.

The traffic volume is OK until you have tens of thousands emails per day on a that particular domain. Postfix checks the recipient in a parallel session before accepting it to the sender so you still get the bonus of not accepting undeliverable and forged recipient mail.

As for writing a howto, it will be usefull to many and might serve as guideline to implement the feature in ispcp later.

rxbux: It's a compromise when you don't want or cannot deal with the customer's internal mailserver and you don't trust their administrator for timely updates (or automatic updates). It's a way to be sure things work with as little intervention as possible. If the volume of incoming mails is high (~10000 mail/day) they you can customize it better but for the majorance of the sites this is not necessary.

ispcomm

[kilburn]

ispcomm Wrote:The traffic volume is OK until you have tens of thousands emails per day on a that particular domain. Postfix checks the recipient in a parallel session before accepting it to the sender so you still get the bonus of not accepting undeliverable and forged recipient mail.

My customers using this service receive an avg of ~10k delivery attempts to unexistant accounts each day. As currently greylisting is done AFTER validating the recipient address (good thing if the machine has this info because it avoids filling the greylisting database with known-to-be-rejected delivery attemps), most of this ~10k attempts would generate a verify request, and the Exchange server of our customers will probably die (You know exchange is a really big piece of shit, don't you?)...

If you reconfigure postfix to greylist before validating recipients the customer server's won't die, but your postgrey database would grow several times bigger and the waste of resources will be noticeable.

IMHO this is a "setup/admin cost" vs "bare resources cost" issue.