Interal workings of mailing lists

[Jodrik]

Lately we have been extending our usage of mailinglists (or rather, our marketing of them to custommers). But I have been running into some issues regarding the delivery of e-mails to several servers. Does Ability deliver one mail at a time or does it use multiple connections to deliver several mails at once?

Code: Select all

Sat, 31 Oct 2009 12:43:06 -> Success: Action=[SMTP Transfer], Details=[Domain=12move.nl, Host=mail-mx-00.tiscali.nl:25, IP=195.241.79.130: Connection accepted.]
Sat, 31 Oct 2009 12:43:06 -> Success: Action=[SMTP Transfer], Details=[Domain=12move.nl, Host=mail-mx-00.tiscali.nl:25, IP=195.241.79.130: Starting SSL.]
Sat, 31 Oct 2009 12:43:08 -> Failed: Action=[SMTP Transfer], Details=[Domain=12move.nl, Host=mail-mx-00.tiscali.nl:25, IP=195.241.79.130: Connection closed, failed to start SSL.]
Sat, 31 Oct 2009 12:43:08 -> Success: Action=[SMTP Transfer], Details=[Domain=12move.nl, Host=mail-mx-00.tiscali.nl:25, IP=195.241.79.130: Reconnection accepted.]
Sat, 31 Oct 2009 12:43:15 -> Failed: Action=[SMTP Transfer], Details=[Domain=12move.nl, Host=mail-mx-00.tiscali.nl:25, IP=195.241.79.130: Recipient Rejected: recipient1@12move.nl (550 Mailbox unavailable.)]
Sat, 31 Oct 2009 12:43:15 -> Failed: Action=[SMTP Transfer], Details=[Domain=12move.nl, Host=mail-mx-00.tiscali.nl:25, IP=195.241.79.130: Recipient Rejected: recipient2@12move.nl (550 Only one rcpt allowed for <> sender.)]

Sat, 31 Oct 2009 12:43:15 -> Failed: Action=[SMTP Transfer], Details=[Domain=12move.nl, Host=mail-mx-00.tiscali.nl:25, IP=195.241.79.130: All remaining recipients were rejected.]


(Changed the mail addresses due to privacy reasons)

From what I see / understand here these mails are being blocked due to the fact that the mailserver attempts to deliver all mails in one go. Now I've been looking into this and I would like to know how Ability is delivering these mails because the TO address that is being used is not actually the TO address being issues by the mailing list. Could this be causing the problems?

According to RFC 821

When the same message is sent to multiple recipients the SMTP encourages the transmission of only one copy of the data for all the recipients at the same destination host.

This command is used to identify an individual recipient of the mail data; multiple recipients are specified by multiple use of this command.

We are talking about a major Dutch ISP so I would think they work conform to the RFCs (Although I've seen alot of cases they generally go around quite a few sometimes). I would also asume and read from the logs that Ability delivers according to these RFC regulations, but several issues came to light when I started checking the actual mail-flow of a mailinglist generated e-mail message:

SMTP Connection logs from the recipient Ability server

Code: Select all

Fri, 06 Nov 2009 11:06:03 -> 195.242.98.70 -> Success: Action=[Accept Connection], Details=[Port 25]
Fri, 06 Nov 2009 11:06:04 -> 195.242.98.70 -> Success: Action=[Accept Connection], Details=[Port 25]
Fri, 06 Nov 2009 11:06:04 -> 195.242.98.70 -> Success: Action=[Close Connection]
Fri, 06 Nov 2009 11:06:04 -> 195.242.98.70 -> Success: Action=[Received Hello], Details=[Host=KO3.net]
Fri, 06 Nov 2009 11:06:04 -> 195.242.98.70 -> Success: Action=[Received Sender]
Fri, 06 Nov 2009 11:06:13 -> 195.242.98.70 -> Success: Action=[SPF Check], Details=[Domain=KO3.net, Result=PASS]
Fri, 06 Nov 2009 11:06:13 -> 195.242.98.70 -> Success: Action=[Received Recipient], Details=[xxxxx@AMN.nl]
Fri, 06 Nov 2009 11:06:13 -> 195.242.98.70 -> Success: Action=[Start Mail Transaction]
Fri, 06 Nov 2009 11:06:13 -> 195.242.98.70 -> Success: Action=[Complete Mail Transaction], Details=[From Host=KO3.net, Size=1 KB, From=, To=xxxxx@AMN.nl]
Fri, 06 Nov 2009 11:06:13 -> 195.242.98.70 -> Success: Action=[Close Connection]


Mail Header

Code: Select all

Received: from AMN.nl (213.189.22.5) by poseidon.arnhem.ad.amn.nl (10.10.10.6)
with Microsoft SMTP Server id 8.1.393.1; Fri, 6 Nov 2009 11:05:13 +0100
Received: from KO3.net ([195.242.98.70]) by AMN.nl    with SMTP (Code-Crafters
Ability Mail Server 2.62);    Fri, 06 Nov 2009 11:06:13 +0100
Received: from KO3.net ([62.58.182.82]) by KO3.net    with HTTP (KO3.net
MX1.KO3.net 2.70.0);    Fri, 06 Nov 2009 11:06:01 +0100
From: <xxxxx@KO3.net>
To: <mailinglistadress@KO3.net>
Subject: Testing TO behavior
Date: Fri, 6 Nov 2009 11:06:01 +0100
Message-ID: <0118251202.20091106110601@KO3.net>
MIME-Version: 1.0
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
X-Originating-IP: [62.58.182.82]
X-Mailer: MX1.KO3.net 2.70.0 WebMail (by KO3.net)
Received-SPF: NONE () client-ip=62.58.182.82; envelope-from=sender@Krelekamp.net; helo=;
KO3-From-IP: 62.58.182.82
KO3-SMTP-Sender: sender@Krelekamp.net
KO3-SMTP-Host-Address:
KO3-Hop-Count: 1
KO3-Bayesian-Score: Unknown
KO3-SPF: NONE
Reply-To: <xxxxx@KO3.net>
Return-Path: <>


This was mailed from an address in the @Krelekamp.net domain which is on the same Ability server that hosts the mailing-list of the @KO3.net domain. I altered the addresses due to privacy issues but the idea should remain the same.

One thing grabs my attention:
The TO address delivered in the e-mail is not being changed to represent the current recipient, this ofcourse cannot be done when multiple recipients are recieving the same e-mail on a single SMTP connection so this would mean a re-write of the mailing list code to either send mails seperately with different TO addresses or to skip putting in the TO address in the DATA section of the mail which would result in an empty TO address which is most likely unwanted.

My suggestion there for is that mailing lists should send their e-mails per recipient and that the TO address is then re-written in the e-mails based on a per user basis so to make the e-mail appear as personal as possible, or some option is made available to allow for this kind of behavior.

I also made several suggestions earlier this week by mail about changing the feed-back that is recieved from triggering mailing lists such as a report on the mail that was send being directed at the person who triggered it, automatically removing addresses that are not available anymore due to delivery errors and also reporting this. Also a capability to make a user or several users within a domein responsible for the mailing list and it's behavior via a GUI that is now only included within the web-admin console, this would then also require setting for the maximum amount of mailing lists that one can use on a per-domain level that can be set by the server admin account. Also the option for several server-wide admin accounts would be nice.

Well, looks like in the source of writing this mail I solved my own question and I'm left with the suggestion on the re-writing of the recipient. This is a feature that several of my clients have now been looking for, I would very much like to see such an option quickly. Also the report feature should also be made a rather important suggestion / hot-fix. I myself and my clients love the ease of managing and maintaining the ability mailing lists but these aspects that are missing are very big issues for using this mailing list feature on a large scale. We currently are moving to a situation where several of our big clients are in the process of putting up mailing lists with several thousands of recipients on our platform, I hope Ability mailserver steps up to the name and keeps being the best mailing-list platform we can provide to our clients as I highly prefer it over our Hosted Exchange due to it's ease of usage and management for our clients.

[rob]

Your deductions are correct and the suggestion that you produced is something we are already looking at adding the future (sorry I can't give an estimate of when due to the large number of updates we already have lined up to). Basically mailing lists do collect similar email addresses together and hence why the to address is actually that of the mailing list address itself. What we intend in the future is an option to switch the mailing list to individual mails per recipient, and so open up the ability to modify the to address on a per delivery basis (this will help overcome some issues where mailing lists are more suspectible to SPAM blocking). As to administrating mailing lists by users, this is something we are keen to add but does require a few changes to the system to implement. This means extending domian administrators' control to mailing lists and may not appear in the short term but it is something we are certianly looking at in the long term.

[Jodrik]

To add insult to injury I figured out that indeed the ISP is not accepting the mails in the proper fashion required by the RFCs. Although I have not yet tested wether this would also be the case if all addresses used are actually accepted since in this example use was made of an address that did not exists which may have triggered the server to think it was SPAM. (Ofcourse this would be highly advanced and we all know ISPs are not highly advanced in any way, shape or form).

[Jodrik]

Okay another small note on mailing lists:

From what I can tell in the logs the following scenario happens:

One of our servers attempts to connect to ISP host, starts with a delivery listing, during this listing an unaccepted name is used and the ISP forces the connection to close.

This results in the mailing to this entire ISP to fail due to the batching in this manner. Again I geuss this calls for a more active response in the mailing list software as it should then drop the rejected recipient from the listing.

[rob]

A server dropping the connection from one address shouldn't occur according to the SMTP protocol, but some SPAM protection software of course will have less regard. The mailing list does indeed have much room for improvement and a future change to the outoing mail system will help this alot (no ETA yet sorry). Basically presently permanent errors do not stop receipients being tried again, but with we do intend to improve the system so that only temporary errors will be retried. This means a mailing list with a bad user, would generally get dropped automatically.

[Jodrik]

Hi Rob,

Is there an ETA yet on any of the mailing list issues addressed above. I know alot of the stuff mentioned here is cuased by ISPs and not ability but it would be very nice if this could be "worked around"

[rob]

Sorry still no ETA on this just yet, of course we will post an update to the mailing list should this be added.

[m1byo]

you could drop the maximum number of mails per transaction to 1, that would then only send one email at a time, but would ensure that you only lost the "spam" mailing list messages.