Troubleshooting SSL/TLS version on SMTP / Outmail service

[EKjellquist]

I'm looking for any help i can get in terms of how I can effectively troubleshoot or look further..

I want to at least require TLS 1.0 at a minimum on the SMTP and outgoing mail services, but we get a small, but consistent number of contacts either whose mail doesn't get received by us (their server(s) appear to close connection after STARTTLS) and some who we can't send mail to (same thing happens, but when Outgoing Mail tries to send, we get a '503: Bad Sequence of commands).

I should add that every Outgoing Mail send that attempts to use SSL/TLS, the logs always report 'Starting SSL' then 'Connection Closed, failed to start SSL' then 'Reconnection accepted', 'Starting TLS' etc. Mostly these DO complete OK, it appears to just be a few recipients that Outgoing Mail fails on any use of SSL/TLS other than 'none' - There could be reasons on the recipient server's end that are at issue, but we receive no message failure (as one would expect with a refusal from antispam controls, mailboxes not present, etc). I do have the box unchecked that allows non-secure connections to be used as a backup. Generalized sample of what I see in the Outmail log for failures:

Tue, 31 Jan 2017 15:38:04 -> Success: Action=[Add Mail To Queue], Details=[5 KB]
Tue, 31 Jan 2017 15:38:04 -> Success: Action=[Process Mail], Details=[5 KB: Start transfer.]
Tue, 31 Jan 2017 15:38:04 -> Success: Action=[Detect DNS's], Details=[Found 2 entries.]
Tue, 31 Jan 2017 15:38:04 -> Success: Action=[MX Lookup], Details=[DNS=Using automatically detected DNS's, Domain=mail.domain01.com: Found 1 records]
Tue, 31 Jan 2017 15:38:05 -> Success: Action=[SMTP Transfer], Details=[Domain=domain01.com, Host=mail.domain01.com:25, IP=1.2.3.4: Connection accepted.]
Tue, 31 Jan 2017 15:38:05 -> Success: Action=[SMTP Transfer], Details=[Domain=domain01.com, Host=mail.domain01:25, IP=1.2.3.4: Starting SSL.]
Tue, 31 Jan 2017 15:38:05 -> Failed: Action=[SMTP Transfer], Details=[Domain=domain01.com, Host=mail.domain01:25, IP=1.2.3.4: Connection closed, failed to start SSL.]
Tue, 31 Jan 2017 15:38:06 -> Success: Action=[SMTP Transfer], Details=[Domain=domain01.com, Host=mail.domain01:25, IP=1.2.3.4: Reconnection accepted.]
Tue, 31 Jan 2017 15:38:11 -> Failed: Action=[SMTP Transfer], Details=[Domain=domain01.com, Host=mail.domain01:25, IP=1.2.3.4: Transaction rejected with: 503 Bad sequence of commands]

A successful send consistently looks like this:

Tue, 31 Jan 2017 15:38:52 -> Success: Action=[Add Mail To Queue], Details=[301 KB]
Tue, 31 Jan 2017 15:38:52 -> Success: Action=[Process Mail], Details=[301 KB: Start transfer.]
Tue, 31 Jan 2017 15:38:52 -> Success: Action=[Detect DNS's], Details=[Found 2 entries.]
Tue, 31 Jan 2017 15:38:52 -> Success: Action=[MX Lookup], Details=[DNS=Using automatically detected DNS's, Domain=domain01.com: Found 2 records]
Tue, 31 Jan 2017 15:38:53 -> Success: Action=[SMTP Transfer], Details=[Domain=domain01.com, Host=mail.domain01.com:25, IP=1.2.3.4: Connection accepted.]
Tue, 31 Jan 2017 15:38:53 -> Success: Action=[SMTP Transfer], Details=[Domain=domain01.com, Host=mail.domain01.com:25, IP=1.2.3.4: Starting SSL.]
Tue, 31 Jan 2017 15:38:53 -> Failed: Action=[SMTP Transfer], Details=[Domain=domain01.com, Host=mail.domain01.com:25, IP=1.2.3.4: Connection closed, failed to start SSL.]
Tue, 31 Jan 2017 15:38:53 -> Success: Action=[SMTP Transfer], Details=[Domain=domain01.com, Host=mail.domain01.com:25, IP=1.2.3.4: Reconnection accepted.]
Tue, 31 Jan 2017 15:38:54 -> Success: Action=[SMTP Transfer], Details=[Domain=domain01.com, Host=mail.domain01.com:25, IP=1.2.3.4: Starting TLS.]
Tue, 31 Jan 2017 15:38:55 -> Success: Action=[SMTP Transfer], Details=[Domain=domain01.com, Host=mail.domain01.com:25, IP=1.2.3.4: TLS started.]

using https://www.checktls.com/perl/TestReceiver.pl does confirm that the domain in question is able to accept TLS 1.2, so I'm not sure exactly what the issue could be...

[EKjellquist]

I should add that AFAIK setting the SSL/TLS version on any of the services works as the MINIMUM standard the service will attempt, but will use the highest version that's able to be negotiated? Do I have that wrong, and the server ONLY uses / accepts the specified version before falling back to 'none'?

[Code Crafters]

As you are probably aware, Outgoing Mail tries to deliver in the following sequence until successful:

1. Implicit SSL on port 465 with SSL from the start of the connection.
2. Explicit SSL on port 25 with SSL negotiated after the initial connection using protocol commands.
3. None SSL if the checkbox to "Only Allow Secure Connections" is not checked (recommended).

If you choose TLS 1.0 though, this is the only version of SSL that will be used. We could implement starting with TLS 1.2 and working backwards but this could result in a lot of connection attempts. You can however, add a static route to use a different SSL mode for the domain you're having trouble with to force TLS 1.2 for that domain only.

[EKjellquist]

I guess that's what I'm getting at the heart of - Since servers that use TLS are typically backwards compatible (depending on their settings), so that if our server is attempting to communicate on TLS 1.2, and their server only goes up to TLS 1.1, is it standard practice for AMS (or other mail servers) to then attempt to connect with 1.1?

The way I read the SSL/TLS version setting for each service was setting the MINIMUM security level AMS would use; is it actually the reverse, and should we be using TLS 1.2 as the setting if we want to start negotiations with 1.2 and move backwards?

My interpretation of the errors was that I was attempting to use too new of a security standard (TLS 1.0) for some servers that could only use SSL 2/3, but could it simply be that the majority of servers are really using TLS 1.2 and that's it nowadays? As of like July 2016 TLS 1.2 is now required for PCI compliance as far as HTTPS goes, but not sure about email communications (I would assume so)...

[Code Crafters]

The Ability Mail Server setting for SSL mode is the minimum and only setting used currently. Some server SSL modes may be compatible with more than one client mode but generally the 2 have to match exactly. The server can only listen on one SSL mode so it's the responsibility of the client to try TLS 1.2 (for example) and work backwards until an SSL mode is accepted at an approved minimum version.

In the scenario of Outgoing Mail, Ability Mail Server is effectively the client here. We can of course change the behaviour so that outgoing mail could always start at TLS 1.2 and work backwards to the selected SSL mode as a minimum. We have to be careful that too many connection attempts don't trigger SPAM filters blocking the connections though too. We are always open to improvement suggestions.

In general, you should set your services to listen on the maximum SSL/TLS mode that your mail clients support. In the case of SMTP of course, the clients are anybody sending to your mail server. Sending without SSL is always an fall-back option for clients too of course. We've currently recommended TLSv1.0 as the default based on clients like outlook not always working with the latest TLS 1.2.

As the settings suggest SSLv2/3 is the most compatible with clients but deemed insecure and no longer recommended for use. Ideally TLS 1.2 will be used but if older clients don't support this you have to work backwards to find the most secure that still works with your mail clients.

[EKjellquist]

OK, that makes it more clear for me. I guess one feature that would be nice to have, then, would be an exception list as far as that TLS protocol to use based on some combination of settings. I'm seeing that the majority of other servers we interact with are using TLS 1.2 because of compliance requirements and tightening up security, so really all we'd need is a way for each service to utilize a lesser version of SSL/TLS for this user / sender or that domain.

I would think a good way to handle it would be similar to the exception list for SPAM Filtering, where you can identify a sender, IP, domain, etc explicitly or using wildcards, and choose an alternative SSL/TLS version to use if there's a match. For us, the login-based services such as POP3, IMAP, Webmail etc are no issue because all the AMS accounts are internal and our client software can handle TLS 1.2, but SMTP and Outgoing mail would be most critical since they have to interact with myriad configurations of servers outside our control. For those running AMS configurations where people can create accounts / sign up, then all services would really need it.

I'm not sure that there's a standard refusal code for a client-server or server-server connection that gets dropped because SSL/TLS capability doesn't match (I can't say I've ever seen a failure message or log entry to that effect), but would it also be possible to build in (at least from AMS' end) a log entry for a failure due to SSL/TLS incompatibility? That would help at least in getting a heads-up about those types of failures so I could shoot myself a notification and take steps to put in the exception as defined above...

[Code Crafters]

Listening services like SMTP, POP3, IMAP4 and WebMail can only use a single SSL mode for implicit connections where the SSL negotiation is done on connect. However, it would be possible to add an IP based filter for the SSL/TLS mode to be used for explicit SSL done after connect on the standard port via STARTTLS type commands. So, we'd probably have a default SSL mode as now per service but then an extra list with IP (with * wildcards) / SSL mode pairs to override this.

Outgoing Mail is a client service so it can retry connections as many times as needed with different SSL modes. Currently, you pick one to try then allow non-SSL after that if you want. We could add a multi-select option for this mode so that several (starting with the most secure working backwards) could be tried but obviously this can lead to more connection attempts which could trigger SPAM filters on the destination mail server so we have to be careful. Note that you already can do static routes per destination domain to use a different mode though. It's probably still better to do domain based rather than IP based SSL modes for Outgoing Mail too.

Let us know what you think about these suggested features and we'll add them to our feature request list.

[EKjellquist]

Having an IP filter would certainly help, but there's still the possibility that those IPs could change, more than likely w/o notice, whereas an email address or domain change for a contact would more than likely be given to us as a heads-up by those contacts. Moreover, some domains have myriad IPs or subnets we'd have to put in exceptions for, whereas specifying the exception to trigger for a specific address or wildcard / address would be more manageable, imo.

One of the reasons we couldn't use outgoing mail encryption in years past was indeed the 'multiple connections seen as spam' issue, where a minority of recipient servers of the day were rejecting mail from us; not a large number, but enough to not make using SSL/TLS on outgoing worth the hassle. These days I'm seeing the vast majority of recipient servers not having an issue with attempting SSL/ TLS and defaulting to an unencrypted connection, it's just those handful of exceptions where either on incoming or outgoing SMTP the connection is closed abruptly (w/o any apparent fault on our end).

[Code Crafters]

We are still considering options for change in the next update around outgoing mails SSL modes including allowing multiple modes to be tried in descending order.

There already are static routes to allow different modes for particular domains although you currently have to add a static route per domain. We may change this to allow multiple domains for a static route and allow wildcards for domains (e.g. *.domain.com) if not already possible.

While looking into this we also discovered that the following order is tried.

1. Implicit SSL/TLS on port 25 (unless relay mode specifies a port)
2. Explicit TLS via STARTTLS on port 25 (unless relay mode specifies a port)
3. Non-SSL on port 25 (unless relay mode specifies a port)

Implicit SSL in option 1 above is usually done on 465 not 25 so we are investigating if this is a bug and 465 should be used (unless relay mode specifies a port).

The SSL modes based on IP filters (including IP of domain etc as with other IP block lists) would only be useful for listening services but we are still considering these as an option too.

[EKjellquist]

Here's a recent SMTP log that could help illustrate the issue further (with debug on); a very small handful of senders try to send us email, and mails seem to consistently fail on the TLS handshake..

(*.xxx.yyy.zzz is a public IP for the sending SMTP, MyMailServer is the AMS receiving server)
Thu, 23 Feb 2017 10:15:44 -> *.xxx.yyy.zzz -> Success: Action=[Accept Connection], Details=[Port 25]
Thu, 23 Feb 2017 10:15:44 -> *.xxx.yyy.zzz -> ***DEBUG*** -> Success: Action=[Sent Command], Details=[220 MyMailServer ESMTP (Code Crafters Ability Mail Server)]
Thu, 23 Feb 2017 10:15:44 -> *.xxx.yyy.zzz -> ***DEBUG*** -> Success: Action=[Recv Command], Details=[EHLO server907.appriver.com]
Thu, 23 Feb 2017 10:15:44 -> *.xxx.yyy.zzz -> Success: Action=[Received Hello], Details=[Host=server907.appriver.com]
Thu, 23 Feb 2017 10:15:44 -> *.xxx.yyy.zzz -> ***DEBUG*** -> Success: Action=[Sent Command], Details=[250-MyMailServer]
Thu, 23 Feb 2017 10:15:44 -> *.xxx.yyy.zzz -> ***DEBUG*** -> Success: Action=[Sent Command], Details=[250-PIPELINING]
Thu, 23 Feb 2017 10:15:44 -> *.xxx.yyy.zzz -> ***DEBUG*** -> Success: Action=[Sent Command], Details=[250-8BITMIME]
Thu, 23 Feb 2017 10:15:44 -> *.xxx.yyy.zzz -> ***DEBUG*** -> Success: Action=[Sent Command], Details=[250-AUTH PLAIN LOGIN CRAM-MD5 CRAM-SHA1]
Thu, 23 Feb 2017 10:15:44 -> *.xxx.yyy.zzz -> ***DEBUG*** -> Success: Action=[Sent Command], Details=[250-STARTTLS]
Thu, 23 Feb 2017 10:15:44 -> *.xxx.yyy.zzz -> ***DEBUG*** -> Success: Action=[Sent Command], Details=[250 OK]
Thu, 23 Feb 2017 10:15:44 -> *.xxx.yyy.zzz -> ***DEBUG*** -> Success: Action=[Recv Command], Details=[STARTTLS]
Thu, 23 Feb 2017 10:15:44 -> *.xxx.yyy.zzz -> ***DEBUG*** -> Success: Action=[Sent Command], Details=[220 Starting TLS...]
Thu, 23 Feb 2017 10:15:44 -> *.xxx.yyy.zzz -> Failed: Action=[Start TLS]
Thu, 23 Feb 2017 10:15:44 -> *.xxx.yyy.zzz -> Success: Action=[Close Connection]

In this case, it's from email contacts that have hosted email that ultimately relays through appriver servers. If I reboot the server AMS runs on, mails from the above example seem to come through for 30-60 minutes OK but go back to failing as above.

Observations:
- Running current version of AMS, 4.2.1 as of this writing
- server907.appriver.com comes through as the same domain, but different IPs (same behavior regardless of IP)
- email sender addresses are whitelisted, this includes senders from multiple hosted email domains that come through the above server
- We're using all antispam controls except for greylisting; after a reboot when the mails come through, whitelisting allows them through fine
- This pattern occurs regardless of what TLS version I set SMTP to (1.0, 1.1, 1.2)
- I'm using Outgoing Mail at TLS 1.2, and outgoing mails to the addressees affected by the above are delivered w/o issue
- We use a self-signed certificate for this mail server, which was generated directly in AMS
- Plenty of other sending servers are able to send to us via TLS/SSL or unencrypted w/o any kind of issue

My best guess would normally be that the sending server gets to TLS and doesn't like the handshake because of the weaker SHA1 / RSA certificate cipher that AMS uses (diagnostic sites like checktls etc have given us warnings about that for some time), and is closing the connection on the sending side (but I don't have access to the sending server's logs). BUT that wouldn't explain why the connection would work again for a little while after a reboot.

So I guess I'm looking for ideas on what else I can look for here - CheckTLS and other sites don't come up with any exceptions save for the certificate issue, the server that AMS runs on doesn't appear to have any issues (e.g. firewall, ports, service crashes, etc). I'd really rather not turn off encryption for SMTP altogether if at all possible, and I can't know for sure if purchasing a legit certificate would definitely clear this up....

[Code Crafters]

Often the TLS version being used by the sending mail server / client is incompatible with the version being used on Ability Mail Server. The sending mail server / client may even be using a version older than TLSv1.0. The new beta release of OpenSSL has now completely removed versions older than TLSv1.0 but we won't be using this until it's official release and then will have to consider if our customers still want that older client compatibility vs the security risks.

The other thing as you said is that some mail servers / clients will refuse the SSL connection on the basis that the certificate is not-trusted and signed by a CA. However, as you said, there is no way to tell that this is the issue or the only issue. I don't know if any Certificate Authorities give 7 day trial certificates or similar to allow you to try out this theory. With self-signed certificates you usually have to either install the certificate on the client machine or authorise use of it in your outlook etc.

We have made a couple of changes that are going into the next update for Outgoing Mail. Firstly, Implicit SSL is now optional and specified on a separate port instead of 25 or whatever relay port you use. Also, failure messages were not always sent but now this will be fixed.

[EKjellquist]

Any chance of a self-signed certificate generator using more up-to-date ciphers?

[Code Crafters]

You can generate these outside of Ability Mail Server in other ways but we will look into if the certificates can be made stronger.

[EKjellquist]

I did end up trying out a free Comodo cert (they use SHA256, which is ok), but I'm not certain how to install the intermediate cert bundle with AMS? Checktls.com comes up with '[000.624] Cert NOT VALIDATED: unable to get local issuer certificate'. The hostname validation comes up fine, and I have the intermediate cert installed on the server's certificate store. I'm using a paid wildcard cert with AMS on another installation / different domain, comes up with the same error...

[EKjellquist]

I also tried checktls' sender test options both from an AMS account and from a Hotmail account to try and find out more.

This is a response containing the header of the test email from AMS:

<-- 220 ts4.checktls.com ESMTP TestSender Fri, 03 Mar 2017 10:47:39 -0500
--> EHLO [mymailserver]
<-- 250-ts4.checktls.com Hello [50.75.95.106], pleased to meet you
<-- 250-ENHANCEDSTATUSCODES
<-- 250-8BITMIME
<-- 250-STARTTLS
<-- 250 HELP
--> MAIL FROM:<[MyEmailAddress]>
<-- 250 Ok - mail from [MyEmailAddress]
--> RCPT TO:<test@TestSender.CheckTLS.com>
<-- 250 Ok - recipient test@TestSender.CheckTLS.com
--> DATA
<-- 354 Send data. End with CRLF.CRLF

And the corresponding Outgoing mail log:

Fri, 03 Mar 2017 10:47:38 -> Success: Action=[Add Mail To Queue], Details=[4 KB]
Fri, 03 Mar 2017 10:47:38 -> Success: Action=[Process Mail], Details=[4 KB: Start transfer.]
Fri, 03 Mar 2017 10:47:38 -> Success: Action=[Detect DNS's], Details=[Found 2 entries.]
Fri, 03 Mar 2017 10:47:38 -> Success: Action=[MX Lookup], Details=[DNS=Using automatically detected DNS's, Domain=TestSender.CheckTLS.com: Found 1 records]
Fri, 03 Mar 2017 10:47:38 -> Success: Action=[SMTP Transfer], Details=[Domain=TestSender.CheckTLS.com, Host=ts4.CheckTLS.com:25, IP=216.68.85.113: Connection accepted.]
Fri, 03 Mar 2017 10:47:38 -> Success: Action=[SMTP Transfer], Details=[Domain=TestSender.CheckTLS.com, Host=ts4.CheckTLS.com:25, IP=216.68.85.113: Starting SSL.]
Fri, 03 Mar 2017 10:47:38 -> Failed: Action=[SMTP Transfer], Details=[Domain=TestSender.CheckTLS.com, Host=ts4.CheckTLS.com:25, IP=216.68.85.113: Connection closed, failed to start SSL.]
Fri, 03 Mar 2017 10:47:39 -> Success: Action=[SMTP Transfer], Details=[Domain=TestSender.CheckTLS.com, Host=ts4.CheckTLS.com:25, IP=216.68.85.113: Reconnection accepted.]
Fri, 03 Mar 2017 10:47:39 -> Success: Action=[SMTP Transfer], Details=[Domain=TestSender.CheckTLS.com, Host=ts4.CheckTLS.com:25, IP=216.68.85.113: Starting TLS.]
Fri, 03 Mar 2017 10:47:39 -> Failed: Action=[SMTP Transfer], Details=[Domain=TestSender.CheckTLS.com, Host=ts4.CheckTLS.com:25, IP=216.68.85.113: Connection closed, failed to start TLS.]
Fri, 03 Mar 2017 10:47:39 -> Success: Action=[SMTP Transfer], Details=[Domain=TestSender.CheckTLS.com, Host=ts4.CheckTLS.com:25, IP=216.68.85.113: Reconnection accepted.]
Fri, 03 Mar 2017 10:47:40 -> Success: Action=[SMTP Transfer], Details=[Domain=TestSender.CheckTLS.com, Host=ts4.CheckTLS.com:25, IP=216.68.85.113: Recipient Accepted: test@TestSender.CheckTLS.com]
Fri, 03 Mar 2017 10:47:40 -> Success: Action=[SMTP Transfer], Details=[Domain=TestSender.CheckTLS.com, Host=ts4.CheckTLS.com:25, IP=216.68.85.113: Transaction completed, sent to 1 of 1 recipients successfully.]

The above is common with all of my outgoing mail; using any type of SSL or TLS mode specified for Outgoing Mail produces the above in almost all cases (SSL/TLS consistently fails and sends unsecured). Even with the new legit certificate, firewall disabled etc it still occurs.

This is the same test using a hotmail account:

<-- 220 ts4.checktls.com ESMTP TestSender Fri, 03 Mar 2017 10:43:36 -0500
--> EHLO BAY004-OMC3S20.hotmail.com
<-- 250-ts4.checktls.com Hello [65.54.190.158], pleased to meet you
<-- 250-ENHANCEDSTATUSCODES
<-- 250-8BITMIME
<-- 250-STARTTLS
<-- 250 HELP
--> STARTTLS
<-- 220 Ready to start TLS
====tls negotiation successful (cypher: AES256-SHA256, client cert: Subject Name: undefined;Issuer Name: undefined;)
~~> EHLO BAY004-OMC3S20.hotmail.com
<~~ 250-ts4.checktls.com Hello [65.54.190.158], pleased to meet you
<~~ 250-ENHANCEDSTATUSCODES
<~~ 250-8BITMIME
<~~ 250 HELP
~~> MAIL FROM:<MyEmail@hotmail.com>
<~~ 250 Ok - mail from MyEmail@hotmail.com
~~> RCPT TO:<test@TestSender.CheckTLS.com>
<~~ 250 Ok - recipient test@TestSender.CheckTLS.com
~~> DATA
<~~ 354 Send data. End with CRLF.CRLF

[EKjellquist]

This is the same AMS test again, Outgoing mail log with Debug on, TLS 1.2

Fri, 03 Mar 2017 11:05:30 -> Success: Action=[Add Mail To Queue], Details=[4 KB]
Fri, 03 Mar 2017 11:05:30 -> Success: Action=[Process Mail], Details=[4 KB: Start transfer.]
Fri, 03 Mar 2017 11:05:30 -> Success: Action=[Detect DNS's], Details=[Found 2 entries.]
Fri, 03 Mar 2017 11:05:30 -> Success: Action=[MX Lookup], Details=[DNS=Using automatically detected DNS's, Domain=TestSender.CheckTLS.com: Found 1 records]
Fri, 03 Mar 2017 11:05:31 -> Success: Action=[SMTP Transfer], Details=[Domain=TestSender.CheckTLS.com, Host=ts4.CheckTLS.com:25, IP=216.68.85.113: Connection accepted.]
Fri, 03 Mar 2017 11:05:31 -> Success: Action=[SMTP Transfer], Details=[Domain=TestSender.CheckTLS.com, Host=ts4.CheckTLS.com:25, IP=216.68.85.113: Starting SSL.]
Fri, 03 Mar 2017 11:05:31 -> Failed: Action=[SMTP Transfer], Details=[Domain=TestSender.CheckTLS.com, Host=ts4.CheckTLS.com:25, IP=216.68.85.113: Connection closed, failed to start SSL.]
Fri, 03 Mar 2017 11:05:31 -> Success: Action=[SMTP Transfer], Details=[Domain=TestSender.CheckTLS.com, Host=ts4.CheckTLS.com:25, IP=216.68.85.113: Reconnection accepted.]
Fri, 03 Mar 2017 11:05:31 -> ***DEBUG*** -> Success: Action=[Recv Response], Details=[IP=216.68.85.113: 220 ts4.checktls.com ESMTP TestSender Fri, 03 Mar 2017 11:05:31 -0500]
Fri, 03 Mar 2017 11:05:31 -> ***DEBUG*** -> Success: Action=[Send Command], Details=[IP=216.68.85.113: EHLO [MyMailServer]]
Fri, 03 Mar 2017 11:05:32 -> ***DEBUG*** -> Success: Action=[Recv Response], Details=[IP=216.68.85.113: 250-ts4.checktls.com Hello [50.75.95.106], pleased to meet you]
Fri, 03 Mar 2017 11:05:32 -> ***DEBUG*** -> Success: Action=[Recv Response], Details=[IP=216.68.85.113: 250-ENHANCEDSTATUSCODES]
Fri, 03 Mar 2017 11:05:32 -> ***DEBUG*** -> Success: Action=[Recv Response], Details=[IP=216.68.85.113: 250-8BITMIME]
Fri, 03 Mar 2017 11:05:32 -> ***DEBUG*** -> Success: Action=[Recv Response], Details=[IP=216.68.85.113: 250-STARTTLS]
Fri, 03 Mar 2017 11:05:32 -> ***DEBUG*** -> Success: Action=[Recv Response], Details=[IP=216.68.85.113: 250 HELP]
Fri, 03 Mar 2017 11:05:32 -> ***DEBUG*** -> Success: Action=[Send Command], Details=[IP=216.68.85.113: STARTTLS]
Fri, 03 Mar 2017 11:05:32 -> ***DEBUG*** -> Success: Action=[Recv Response], Details=[IP=216.68.85.113: 220 Ready to start TLS]
Fri, 03 Mar 2017 11:05:32 -> Success: Action=[SMTP Transfer], Details=[Domain=TestSender.CheckTLS.com, Host=ts4.CheckTLS.com:25, IP=216.68.85.113: Starting TLS.]
Fri, 03 Mar 2017 11:05:32 -> Failed: Action=[SMTP Transfer], Details=[Domain=TestSender.CheckTLS.com, Host=ts4.CheckTLS.com:25, IP=216.68.85.113: Connection closed, failed to start TLS.]
Fri, 03 Mar 2017 11:05:32 -> Success: Action=[SMTP Transfer], Details=[Domain=TestSender.CheckTLS.com, Host=ts4.CheckTLS.com:25, IP=216.68.85.113: Reconnection accepted.]
Fri, 03 Mar 2017 11:05:32 -> ***DEBUG*** -> Success: Action=[Recv Response], Details=[IP=216.68.85.113: 220 ts4.checktls.com ESMTP TestSender Fri, 03 Mar 2017 11:05:32 -0500]
Fri, 03 Mar 2017 11:05:32 -> ***DEBUG*** -> Success: Action=[Send Command], Details=[IP=216.68.85.113: EHLO [MyMailServer]]
Fri, 03 Mar 2017 11:05:32 -> ***DEBUG*** -> Success: Action=[Recv Response], Details=[IP=216.68.85.113: 250-ts4.checktls.com Hello [50.75.95.106], pleased to meet you]
Fri, 03 Mar 2017 11:05:32 -> ***DEBUG*** -> Success: Action=[Recv Response], Details=[IP=216.68.85.113: 250-ENHANCEDSTATUSCODES]
Fri, 03 Mar 2017 11:05:32 -> ***DEBUG*** -> Success: Action=[Recv Response], Details=[IP=216.68.85.113: 250-8BITMIME]
Fri, 03 Mar 2017 11:05:32 -> ***DEBUG*** -> Success: Action=[Recv Response], Details=[IP=216.68.85.113: 250-STARTTLS]
Fri, 03 Mar 2017 11:05:32 -> ***DEBUG*** -> Success: Action=[Recv Response], Details=[IP=216.68.85.113: 250 HELP]
Fri, 03 Mar 2017 11:05:32 -> ***DEBUG*** -> Success: Action=[Send Command], Details=[IP=216.68.85.113: MAIL FROM:<[myemailaddress]>]
Fri, 03 Mar 2017 11:05:33 -> ***DEBUG*** -> Success: Action=[Recv Response], Details=[IP=216.68.85.113: 250 Ok - mail from [myemailaddress]]
Fri, 03 Mar 2017 11:05:33 -> ***DEBUG*** -> Success: Action=[Send Command], Details=[IP=216.68.85.113: RCPT TO:<test@TestSender.CheckTLS.com>]
Fri, 03 Mar 2017 11:05:33 -> ***DEBUG*** -> Success: Action=[Recv Response], Details=[IP=216.68.85.113: 250 Ok - recipient test@TestSender.CheckTLS.com]
Fri, 03 Mar 2017 11:05:33 -> Success: Action=[SMTP Transfer], Details=[Domain=TestSender.CheckTLS.com, Host=ts4.CheckTLS.com:25, IP=216.68.85.113: Recipient Accepted: test@TestSender.CheckTLS.com]
Fri, 03 Mar 2017 11:05:33 -> ***DEBUG*** -> Success: Action=[Send Command], Details=[IP=216.68.85.113: DATA]
Fri, 03 Mar 2017 11:05:33 -> ***DEBUG*** -> Success: Action=[Recv Response], Details=[IP=216.68.85.113: 354 Send data. End with CRLF.CRLF]
[email body text omitted]
Fri, 03 Mar 2017 11:05:34 -> ***DEBUG*** -> Success: Action=[Recv Response], Details=[IP=216.68.85.113: 250 Ok]
Fri, 03 Mar 2017 11:05:34 -> Success: Action=[SMTP Transfer], Details=[Domain=TestSender.CheckTLS.com, Host=ts4.CheckTLS.com:25, IP=216.68.85.113: Transaction completed, sent to 1 of 1 recipients successfully.]
Fri, 03 Mar 2017 11:05:34 -> ***DEBUG*** -> Success: Action=[Send Command], Details=[IP=216.68.85.113: QUIT]
Fri, 03 Mar 2017 11:05:34 -> ***DEBUG*** -> Success: Action=[Recv Response], Details=[IP=216.68.85.113: 221 ts4.checktls.com closing connection]

[EKjellquist]

OK, it does look like it ended up working with the new certificate when I switched back to TLS 1.0 with the new certificate:

<-- 220 ts4.checktls.com ESMTP TestSender Fri, 03 Mar 2017 11:28:58 -0500
--> EHLO [mymailserverdomain]
<-- 250-ts4.checktls.com Hello [50.75.95.106], pleased to meet you
<-- 250-ENHANCEDSTATUSCODES
<-- 250-8BITMIME
<-- 250-STARTTLS
<-- 250 HELP
--> STARTTLS
<-- 220 Ready to start TLS
====tls negotiation successful (cypher: DHE-RSA-AES256-SHA, client cert: Subject Name: undefined;Issuer Name: undefined;)
~~> EHLO [mymailserverdomain]
<~~ 250-ts4.checktls.com Hello [50.75.95.106], pleased to meet you
<~~ 250-ENHANCEDSTATUSCODES
<~~ 250-8BITMIME
<~~ 250 HELP
[etc]

So I'm guessing my old self-signed cert (which we generated w/i AMS) had older ciphers which were being rejected by recipient servers when my server was trying to connect on outgoing mail (on any version of SSL/TLS), because the receiving servers likely didn't like any of the available ciphers in that cert. with the updated (but still free) cert, it still didn't work with TLS 1.2 (which could either be because the recipient servers largely weren't using 1.2 or because the free cert didn't have ciphers good enough for 1.2), BUT it was able to negotiate with the DHE-RSA-AES256-SHA cipher on TLS 1.0.

Does it stand to reason that paid certificates generally have stronger ciphers available, and should I be looking for a specific set of generally-accepted ciphers in a cert to be able to use TLS 1.2 effectively? So far, this hasn't been a level of detail I've found in many places offering certs, other than being at least SHA256 capable...

[EKjellquist]

OK well this certainly has been an interesting journey this past week. I believe I solved most of my question in the previous post through a bit of trial and error, and rather than deleting all this history, hopefully there might be someone else that this helps.

We're using a single IP setup currently, so what I've traditionally been doing was forwarding all the relevant email ports (including https port 443) to my local AMS server IP, and AMS has worked fine. Recently (see above posts) Chrome was reporting that the connection was using an old RSA key exchange with the new SHA256 certificate I got. Tried a number of different things that didn't end up working.

Suffice to say, I wanted to bring AMS connections from the WAN through my Apache / PHP server anyway (rather than port-forward 443 directly to the AMS server via the edge router as I had been doing), so I rerouted the 443 port forward to my Apache server instead and added the following virtualhost (some specific items obviously renamed):

<VirtualHost *:443>
ServerName mail.mydomain.com
ServerAdmin domains@mydomain.com
SSLEngine On
SSLProxyEngine On
ProxyPreserveHost On
SSLCertificateFile C:/Apache24/certs/mail.mydomain.com/mail.mydomain.com.crt
SSLCertificateKeyFile C:/Apache24/certs/mail.mydomain.com/mail.mydomain.com.key
SSLCertificateChainFile C:/Apache24/certs/mail.mydomain.com/mail.mydomain.com.ca-bundle
ProxyPass / https://mail.mydomain.com/
ProxyPassReverse / https://mail.mydomain.com/
</VirtualHost>

My local DNS server has an entry for mail.mydomain.com towards its local IP. The above require mod_ssl, mod_proxy (possibly a few other libraries that were already on). After a little tinkering here and there, I was able to log in fine as before, but now going through my Apache server as a proxy.

However, I reran the SSL Labs test and was floored, as the following things are fixed:

(1) Chrome reports it is now using ECDHE_RSA with P-256 as the key exchange, no more security errors
(2) Downgrade attack prevention is supported (TLS_FALLBACK_SCSV)
(3) Some forward Secrecy is being done (by browser)
(4) TLS 1.0-1.2 is now possible, browsers that were erroring out before are now fine
(5) an unfathomably huge number of ciphers are supported
(6) no certificate chain issues

SO since I had been keeping up with my Apache updates all along, I'm guessing I should have been serving up webmail via this route from day 1 (tried years ago, couldn't get it to work then). Any further thoughts / suggestions are appreciated...

[Code Crafters]

Let us just clear up a few things that may be confusing in your previous replies.

1. SSL certificates on AMS are only used for incoming "listening" services (SMTP, POP3, IMAP4, WebMail, Remote Admin) and never used in Outgoing Mails or POP3 retrievals. In this case it's the destination (receiving) mail server whose certificate is to be trusted.
2. If you have separate intermediate certificates that you've purchased you need to combine these into a single certificate file before importing into Ability Mail Server. Email us for help with this if you can't google how to do it easily.
3. Ability Mail Server 3 and earlier WebMail used to have to run entirely on Ability Mail Server as it was a custom scripting language not supported by IIS etc. Ability Mail Server 4 has separate plain content (HTML, Javascript, CSS) and API calls. The plain content can be served by any web server allowing you to now do host binding on IIS etc. to use port 80 on that domain and port 80 on another for another web site. The API calls are always served by AMS of course.

Now to the main issue of TLS handshakes failing. Our research has shown that there are only certain TLS modes that are compatible with others. In general, no TLS modes are backwards compatible for security reasons. Therefore, only matching versions on server / client will succeed. We have only found it possible to listen on a single mode and haven't yet found a way to support more than one TLS mode concurrently. For listening services, we had a suggestion to allow an IP / TLS mode matching rule which we may implement in a future update but this may be difficult to maintain for admins. For Outgoing Mail, we have discussed the option of trying multiple TLS handshakes and this may be possible within a single connection (multiple connections will likely lead to temporary blocking of AMS IP). You suggest that apache can somehow work with TLS 1.0, 1.1 and 1.2 on a single listening connection? We will look into this. As for modes and ciphers, we are just using what comes with OpenSSL. The next major version (currently in beta) will remove SSL v2, v2/3 and v3 so we will have to consider if any customers still need that functionality even though it is definitely not recommended anymore.

We have done a couple of changes for the next update:

1. Implicit SSL will now be tried on a specified port (optional), then explicit TLS via and then plain are the next to be tried in that order depending on the various SSL settings. Any of the 3 can be optional but at least one of either TLS or plain connections will always be allowed.
2. Failure messages weren't always sent but this has been fixed for the next update.

[EKjellquist]

1. SSL certificates on AMS are only used for incoming "listening" services (SMTP, POP3, IMAP4, WebMail, Remote Admin) and never used in Outgoing Mails or POP3 retrievals. In this case it's the destination (receiving) mail server whose certificate is to be trusted.

Understood. Is it common for receivers to verify sender certificates or at least report on trust level? Even if not something that could be a full-fledged filter, it would be nice to be able to log them...

3. Ability Mail Server 3 and earlier WebMail used to have to run entirely on Ability Mail Server as it was a custom scripting language not supported by IIS etc. Ability Mail Server 4 has separate plain content (HTML, Javascript, CSS) and API calls. The plain content can be served by any web server allowing you to now do host binding on IIS etc. to use port 80 on that domain and port 80 on another for another web site. The API calls are always served by AMS of course.

This makes sense - I had attempted the same proxy scenario back in the 1.xx days with different ports and couldn't quite get it right, and had just port-forwarded everything from the edge router; when attempting the same with AMS 4 I couldn't figure out at first why it worked on port 80 w/o any jiggering

Now to the main issue of TLS handshakes failing. Our research has shown that there are only certain TLS modes that are compatible with others. In general, no TLS modes are backwards compatible for security reasons. Therefore, only matching versions on server / client will succeed. We have only found it possible to listen on a single mode and haven't yet found a way to support more than one TLS mode concurrently. For listening services, we had a suggestion to allow an IP / TLS mode matching rule which we may implement in a future update but this may be difficult to maintain for admins. For Outgoing Mail, we have discussed the option of trying multiple TLS handshakes and this may be possible within a single connection (multiple connections will likely lead to temporary blocking of AMS IP). You suggest that apache can somehow work with TLS 1.0, 1.1 and 1.2 on a single listening connection? We will look into this. As for modes and ciphers, we are just using what comes with OpenSSL. The next major version (currently in beta) will remove SSL v2, v2/3 and v3 so we will have to consider if any customers still need that functionality even though it is definitely not recommended anymore.

Yes, this was the major thing I noticed when running the SSLlabs test on my webmail site, first with just forwarding ports directly to the mail server (as I had been doing previously) and then after using my WAMP server as a proxy (x64 Apache Lounge builds, Apache 2.4.25 / OpenSSL 1.02j / PHP 7.1 as of this writing). There was a pretty huge difference in supported browsers, TLS modes, encryption capability, etc., but I figured this was because of the differences between the more vanilla AMS web server components vs something a bit more comprehensive. It's not that it's ever really been an issue historically, but more recently, browsers / email clients have been more proactive in dropping support for older standards, and as a result, I've been reading a LOT over a short period of time on both SSL/TLS and related Apache directives.

I don't know that Apache allows for changing the TLS version in the middle of negotiation (rather than offering TLS / Cipher versions server-side that the client negotiates); On the apache side I can at least log and identify where those handshakes fail; on TLS handshakes for SMTP, the base AMS logs don't seem to show that negotiation (just shows a failure); I'd like to be able to do that w/o enabling the full debug log, but at present there's not way to specify...

Is there a plan to update the base AMS web server core, or should we really be using web server proxies such as the above as a general rule for tightened security? So far the only major gripe Chrome seems to have with directly communicating with the AMS web server is that it uses RSA for key exchange. It might be enough to provide an option for the webmail service to be able to specify the internal AMS web server equivalent of cipher list and whether or not the server forces the client to respect its list order. Unsure if that would be appropriate for just webmail or for all AMS services that can potentially use SSL/TLS?

Going along with that, ultimately when I'm proxying through Apache 2.4.x, here's what I'm using to achieve an A+ through SSLLabs' analysis. All email-related ports are forwarded to the AMS server, only web ports (80, 443) forward to the Apache server (with relevant mods enabled), mail.mydomain.com resolves to its LAN address locally:

[Virtualhost definition]
***************************
<VirtualHost *:443>
ServerName mail.mydomain.com
ServerAdmin domains@mydomain.com

Header always set Strict-Transport-Security "max-age=63072000; includeSubdomains;"
SSLEngine On
SSLProxyEngine On
ProxyPreserveHost On
SSLCertificateFile C:/Apache24/certs/MyCertificate
SSLCertificateKeyFile C:/Apache24/certs/MyKey
SSLCertificateChainFile C:/Apache24/certs/MyIntermediateCert
ProxyPass / https://mail.mydomain.com/ retry=1 acquire=3000 timeout=600 Keepalive=On
ProxyPassReverse / https://mail.mydomain.com/
</VirtualHost>
*************************
[SSL Configuration]
*************************
Listen 443
SSLCipherSuite HIGH:MEDIUM:!MD5:!RC4
SSLProxyCipherSuite HIGH:MEDIUM:!MD5:!RC4
SSLCompression off
SSLSessionTickets off
SSLHonorCipherOrder on
SSLProtocol all -SSLv3 -TLSv1
SSLProxyProtocol all -SSLv3 -TLSv1
SSLPassPhraseDialog builtin
SSLSessionCache "shmcb:c:/Apache24/logs/ssl_scache(512000)"
SSLSessionCacheTimeout 300
SSLUseStapling On
SSLStaplingCache "shmcb:c:/Apache24/logs/ssl_stapling(128000)"
SSLStaplingStandardCacheTimeout 3600
SSLStaplingErrorCacheTimeout 600
*******************************

Everything above works OK, the only thing I can't do yet is specify only high ciphers that require ECDHE. if I limit SSLCipherSuite to only ECDHE-based keys (which would be ok based on which browsers appear to be supported), I get a 502 proxy error (which I believe is because the AMS web server supports up to TLS_RSA_WITH_AES_128_GCM_SHA256). Not that I'm really worried too much at this point, but supporting any of the following would allow for further tightening-up of the webmail service...

ECDHE-ECDSA-AES256-GCM-SHA384
ECDHE-RSA-AES256-GCM-SHA384
ECDHE-ECDSA-CHACHA20-POLY1305
ECDHE-RSA-CHACHA20-POLY1305
ECDHE-ECDSA-AES128-GCM-SHA256
ECDHE-RSA-AES128-GCM-SHA256
ECDHE-ECDSA-AES256-SHA384
ECDHE-RSA-AES256-SHA384
ECDHE-ECDSA-AES128-SHA256
ECDHE-RSA-AES128-SHA256