You might see the following error in the Windows Application Event Log:
The request failed. Mailbox:
System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send.
---> System.IO.IOException: Unable to read data from the transport connection:
An existing connection was forcibly closed by the remote host.
---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
--- End of inner exception stack trace ---
at System.Net.TlsStream.EndWrite(IAsyncResult asyncResult)
at System.Net.ConnectStream.WriteHeadersCallback(IAsyncResult ar)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
The request is successful when you try to connect to the URL provided in the error details using a browser on the Exchange server.
You can verify that the issue by trying to access the URL using the PowerShell Invoke-WebRequest cmdlet. Open a new PowerShell session and try connecting to the URL.
Invoke-WebRequest -Uri $uri
You will receive the same error message as stated in the event log by MSExchangeApplicationLogic. A successful connection returns XML as content.
The reason for this error is related to the .NET Framework TLS configuration, not Exchange Server. The .NET Framework lacks configuration for the use of TLS 1.2.
The solution for this issue is to configure the .NET Framework to correctly use TLS 1.2. You can follow the description for TLS 1.2 enforcement for Azure AD Connect, or you can simply use this Gist.
Due to the changes made to the SCHANNEL configuration you just restart the computer to bring the changes into effect.
Changing the TLS settings does not only affect outgoing connections but incoming connections as well.
Test the TLS changes in a test environment before adjusting your servers in the production environment. If you have not already enabled TLS 1.2 for your Exchange Servers, I recommend reading the 3-part series by the Exchange product group.
Enjoy Exchange Server!
Exchange Server uses Receive Connectors for providing SMTP endpoints for incoming connections. A modern Exchange Server provides a default connector on TCP port 25.
Sometimes you might have a requirement to create a new receive connector for selected incoming SMTP connections. A standard requirement is a receive connector for relaying messages to external recipients. This cannot (should not) be achieved using the default connector.
Each connector uses the RemoteIPRanges attribute to store the list of IP addresses of remote systems that can connect to that connector. The default connector utilizes the full IPv4 and IPv6 addresses ranges.
Your new receive connector requires at least a single IP address for a selected remote system that is supposed to connect to that receive connector. You can add a single IP address, address ranges, or IP addresses using CIDR notation.
The attribute RemoteIPRanges is a multi-value attribute and has a limit of IP address entries that can be added.
The maximum number of address entries that you can add to that attribute varies. You can store approximately 1,300 entries.
When you exceed the number of values you receive the following error message:
The administrative limit for this request was exceeded.
+ CategoryInfo : NotSpecified: (:) [Set-ReceiveConnector], AdminLimitExceededException
+ FullyQualifiedErrorId : [Server=EX01,RequestId=ee9d45ad-418b-4172-9235-963eca1a7830,TimeStamp=18.08.2020
20:07:54] [FailureCategory=Cmdlet-AdminLimitExceededException] AC1E336E,Microsoft.Exchange.Management.SystemConfi
+ PSComputerName : ex01.varunagroup.de
I have tested the number of values that can be stored in that multi-value attribute. Depending on the IP address format I was able to add 1,238 (172.80.x.y) or 1,244 (10.1.x.y) single IP addresses to the RemoteIPRanges attribute.
Plan your IP address configuration requirements carefully and avoid using single IP addresses. Preferably, you should use IP address ranges or IP address CIDR notation for networks.
Enjoy Exchange Server!
Are you located in Germany, Austria, or Switzerland? Join the Exchange User Group DACH to collaborate with other Exchange enthusiasts.
Follow us on Twitter @exusg, join on Meetup, or visit our website.
I was involved in a troubleshooting request for a hybrid mail flow issue. Before I take a closer look at the issue, let's talk about the hybrid setup.
A managed service provider runs separated on-premises Exchange Organizations for various clients. Also, the service provider runs it's own Exchange Organization in a hybrid setup with Exchange Online (EXO) utilizing centralized mail flow. Let's name the managed service provider Varunagroup, using the primary domain varunagroup.de.
The on-premises IT-Infrastructure consists of the following email components:
The following diagram illustrates the setup and the expected mail flow.
Let's name one of the clients Setebos AG, using setebos-ag.com as their primary domain.
Varunagroup's IT department activated journaling in Exchange Online, using an on-premises Journaling mailbox. After a few days, an IT administrator checked the inbox folder for journaling messages and journaling reports. The journaling inbox did not contain messages of Varounagroup senders or recipients only, but messages from client sender domains, e.g., setebos-ag.com.
In reality, the mail flow from on-premises to external recipients from any of the local Exchange organizations looked like shown in this diagram.
Why does the Variangoup journaling mailbox contain messages from Setebos senders sent to external recipients?
We choose a single message for troubleshooting purposes, originating from the Setebos.com domain, sent to a non-Varunagroup recipient.
The interesting piece of information is row 6.
You see that EXO resolves the target mail exchanger via DNS. The target is another Microsoft 365 tenant as we see an xxx.mail.protection.outlook.com host.
When checking the on-premises mail gateway connection log, we found the distracting information that the gateway resolved the target mail exchanger as xxx.mail.protection.outlook.com.
As a next step, we checked the extended message tracking log using the new Exchange Admin Center. We created a new custom query with the following search criteria:
When you troubleshoot connection issues with Exchange Online, always select the extended report. You'll receive the report as a CSV file attachment. Use the Data tab in Excel to import the CSV file. Do not access the content by simply clicking the received file attachment.
The interesting information is stored in the custom_data column for row source=SMTP and event_id=RECEIVE.
S:InboundConnectorData=Name=Inbound from [EXCHANGE ORG GUID];
The information in line 3 shows the actual name of the configured Varunagroup inbound connector, as shown in the Exchange Online connector configuration. The message did not enter the Varunagroup EXO tenant due to a mysterious connection, it was received by the dedicated inbound connector, configured by HCW.
The key to this question is the TLS certificate used by the centralized email gateway and the TLS common name filtering in Exchange Online.
The wildcard name *.varunagroup.de resulted in a matching string comparison for the incoming TLS common names of mx01.varunagroup.de and mx02.varunagroup.de. At the same time, the inbound connector matched the Edge Transport TLS certificate smtpo365.varunagroup.de.
Nobody knew, how the inbound connector configuration got "changed" to the wildcard name or for how long that configuration resulted in outbound messages from customer domains routed via the service provider tenant.
The solution contains two configurations.
The TLS common name behavior is by design and described in this blog post as FAQ #6(b). As a customer, you identify this as a misbehaving SMTP receive connector. But as described in the blog post, this is by design.
It is required that you understand the inbound routing behavior of Exchange Online if you have complicated outbound routing requirements. The blog post provides detailed information on how Office 365 inbound routing works and what you should be aware of.
Enjoy Exchange Online.
The use of Exchange Edge Transport Servers requires the synchronization of user and configuration data from internal Exchange Servers to the Edge Transport Servers. The synchronization utilizes secure LDAP (EdgeSync) to transmit the data securely and is based on an Edge Subscription.
When you create a new Edge Subscription on your internal Exchange Servers by importing the Edge Subscription XML-file, establishing the EdgeSync-connection might fail.
You will find the following error in the application event log of the internal Exchange Server:
Log Name: Application
Source: MSExchange EdgeSync
Event ID: 1035
Task Category: Synchronization
EdgeSync failed to synchronize because it only supports Cryptographic API certificates.
The local Hub Transport server's default certificate with thumbprint XYZ isn't a
CAPI certificate. To set a CAPI certificate as the default certificate, use the
Enable-ExchangeCertificate cmdlet with the Services parameter using the value of SMTP.
The private key of the current Exchange Transport default certificate of the internal Exchange servers uses a CNG private key. EdgeSync requires a CAPI1 based private key.
This problem occurs primarily when using an Enterprise Certificate Authority using certificate templates with individual template settings.
Get-TransportService | ft Name,InternalTransportCertificateThumbprint
certutil -v -store my > cert.txt
If both attribute are of the value 0, the certificate if a CNG certificate.
The section might look like this:
Unique container name: XYZ
Provider = Microsoft Software Key Storage Provider
ProviderType = 0
Flags = 20 (32)
CRYPT_MACHINE_KEYSET -- 20 (32)
KeySpec = 0 -- XCN_AT_NONE
Use OpenSSL to convert the CNG certificate to a CAPI1 certificate.
Using OpenSSL requires the download of the Windows release of OpenSSL. I recommend to not install the software on the Exchange Server but a separate Windows server or your administrative desktop system. Additionally, you need the certificate with its private key as a PFX-file.
Use the following steps to convert the CNG certificate to a CAPI1 certificate.
openssl pkcs12 -in CERT.pfx -out cert.pem -nodes
openssl pkcs12 -export -in cert.pem -out NEWCERT.pfx
The new PFX-file is now a CAPI1 certificate. The new certificate has the same thumbprint. Now you must replace the current certificate used by Exchange Server with the new certificate.
Replacing the certificate requires a downtime of each Exchange Server requiring the certificate replacement. This is due to the requirement to remove the CNG certificate first, following the import of the CAPI1 certificate. Afterward, you need to enable the required Exchange services.
Get-ExchangeCertificate -Server SERVERNAME
# It is mandatory to answer the query for replacing the default certificate with YES
Enable-ExchangeCertificate -Thumbprint THUMBPRINT -Services SMTP
# Restart the transport service
# It is mandatory to answer the query for replacing the default certificate with YES
Enable-ExchangeCertificate -Thumbprint NEWCERTTHUMBPRINT -Services SMTP
# Restart the transport service
Now, that you updated the local Exchange Servers there is one more step that needs to be checked on the Edge Transport Servers.
Edge Transport Servers are not domain-joined and therefore do not receive any GPO-based configuration. Each required configuration must be performed locally. To ensure that the default transport certificate of the internal Exchange servers can be used for cryptographic operations we must ensure that the certificate chain of that certificate is present in the certificate store of Edge Transport servers.
Take a look at the certificate chain of the converted CAPI1 certificate and import the Root-CA and Subordinate-CA certificates into the Edge Transport servers local certificate store. You must ensure that the certificates are placed into appropriate stores:
Next, you create a new Edge Subscription on your Edge Transport server and create a new subscription for the Active Directory site on the internal Exchange Server. The internal Exchange Servers are now able to establish an EdgeSync connection and encrypting the data transferred to the Edge Transport servers.
When you receive the TLS certificate as PFX/PKCS12 file, you import the certificate and the private key. The import process itself defines the priavte key Crypto Provider. Using the following command line you ensure that the import process suses the legacy crypto provider.
certutil -csp "Microsoft RSA SChannel Cryptographic Provider" -importpfx my MYCERTpfx
Enjoy Exchange Server and Edge Transport!
You might have the requirement to authenticate mobile devices and authorize user access to on-premises Exchange Server mailboxes using a multi-vendor strategy. This blog post focuses on the configuration of a Kemp LoadMaster located in an internal network segment. The Kemp LoadMaster ESP functionality is used to allow ActiveSync connections for members of a dedicated security group only.
This results in the following authentication and authorization endpoints:
The following diagram shows a simplified overview for mobile devices connecting to an on-premises Exchange Server. The perimeter and internal network segments are omitted for simplification reasons.
The following screenshots illustrate a working setup for a virtual service load balancing mobile device connections from MobileIron Sentry to Exchange Server. It's assumed that you've already configured the following:
The SSL Traffic is offloaded and re-encrypted as we need to authenticate the user with ESP. Ensure to select a Cipher Set that does not provide any weak or unsecure cipher suites. In this example I've selected the predefined set BestPractices.
Enable ESP to activate the ESP configuration section. The settings are as follows:
In the Real Servers section you add all member servers of your Exchange Server DAG. Ensure to use the HTTPS protocol the health checks and ensure to query the /Microsoft-Server-ActiveSync/healthcheck.htm document.
Using this configuration you've added your Kemp LoadMaster as an additional authentication endpoint to secure mobile device access to Exchange Server mailboxes.
The Skype for Business client merges contact data from different sources when displaying the contact card. The merged data is used to perform name resolution when performing a user search. In a more complex deployment scenario where an email domain name is in shared use in two in Active Directory forests you might run into a situation where the Skype for Business client tries to use the wrong target address.
Such a scenario would look like as follows.
Forest A is used as a dedicated infrastructure for developers. When logged on to the development network the users should be able to skype with colleagues currently connected to the default office client network.
Forest B is the default office client network.
Clients logged on to forest A connect to the Skype for Business infrastructure in forest B as external clients. Forest A is used for user account authentication while forest B is used for Skype for Business connection and authentication.
Forest A runs an Exchange Server 2013 organization.
Forest B runs an Exchange Server 2013 organization and a full Skype for Business 2015 server deployment.
Both Active Directory forests to not have trust of any sort established.
Such a configuration would result in forest A users trying to contact a Skype for Business user using the wrong address. Instead of using John.Doe@Sedna-Ltd.com the Skype for Business client of Jane Doe@Sedna-Ltd.com (logged on to forest A) would try to contact John.Doe@dev.Sedna-Ltd.com. As there is no endpoint defined for dev.Sedna-Ltd.com a connection could not be established. Therefore, no availability information is available and no call or chat connection could be established.
The Skype for Business client uses merged data pulled from the Skype for Business address and from the local (Forest A) Active Directory (GAL). This results in a connection attempt to John.Doe@dev.Sedna-Ltd.com.
Enjoy Skype for Business
The latest downloadable build of Exchange Server 2016 Cumulative Update 9 disclosed an information that was previously shown accidently to the public by Greg T. during his breakout session BRK3249 - Modern Authentication for Exchange Server On-Premises at Microsoft Ignite 2017.
As part of the global harmonization of the product name space of the well established Outlook brand the next release of Exchange Server will be named Outlook Server 2019.
This name change was mentioned originally on this slide:
Give it a thought and you'll realize that this change makes absolute sense as different product names for the same software function distract customers and users.
Another reason for renaming Exchange Server is a new functionality for integrating personal mailbox files (PST). It was and still is a tedious task for administrators to get hold of all those PST files in use by end users. Instead of implementing a complex and data protection safe process to import PST files to the primary users mailbox the new Outlook Server 2019 offers synchronized PST folders. A functionality we've waited for for years.
Two new functions are introduced as part the new modern Outlook Server 2019
How does it work?
The following diagram illustrates the new functionality in a simple Outlook Server 2019 setup:
The following screenshot illustrates the new PSTSync folder and some sample PST file for a user with SAMAccountName JohnDoe
It's good the see that there is a future for a email server product like Exchange Server and that after so many years of cloud only an on-premises only feature got added.
Enjoy the day and Happy Easter!