You are hopefully familiar with the new Exchange Emergency Mitigation Service (EEMS) for Exchange Server 2016 and 2019. That is a new service providing automated emergency configuration of your Exchange servers by Microsoft in the case a security risk has been identified. Such emergency mitigation is a technical workaround until a proper security patch is available.
The service responsible for fetching the current list of published mitigations is MSExchangeMitigation.
Exchange Organisation following the official guidance for deploying Exchange Server won't see any specific issues with EEMS. It simply works.
But Exchange Server runs in many different infrastructures where you might end up in a situation with a non-working EEMS.
You see the following event log error:
Exception encountered while fetching mitigations : System.AggregateException: One or more errors occurred. ---> System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel. ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.
In addition, you see the following in the diagnostic logs of the Exchange Server:
S:LogLevel=Information;S:Message=Started MSExchangeMitigation S:LogLevel=Information;S:Message=Fetching mitigations from https://officeclient.microsoft.com/getexchangemitigations S:LogLevel=Information;S:Message=Using Proxy http://[IPADDRESS]/ To Fetch Configurations S:LogLevel=Information;S:Message=No diagnostic data sent. DataCollectionEnabled is false S:LogLevel=Warning;S:Message=TLS certificate or its chain validation failed S:LogLevel=Error;S:Message=Exception encountered while fetching mitigations : One or more errors occurred.;S:Source=Microsoft.Exchange.Mitigation.Service.Mitigations.MitigationEngine
File location: V15\Logging\MitigationService
But what is the validation procedure failing? The solution is simple. The certificate revocation check for the certificate chain failed. The EEMS was not able to connect to the CRL-endpoints of each certificate in the certificate chain. CRL-endpoints are accessible by HTTP and not HTTPS for performance reasons. And outbound HTTP is often blocked for Exchange servers.
The Exchange Server must be able to validate the certificate chain successfully establish a TLS-connection to officeclient.microsoft.com. Certainly, you can disable the CRL check for the server. But this is something I do not recommend. The XML file containing the mitigation configuration is signed by an X509 certificate and your servers should be able to validate and check the CRL.
Ensure that your Exchange servers can communicate with the Internet to validate the certificate chain.
Enjoy Exchange Server.
When you move mailboxes using migration batches you might encounter a situation that your batch contains migration users that fail during batch execution. One of the possible reasons is an existing move request for the affected users. You must remove those requests to successfully move mailboxes.
The following PowerShell example gets all failed migration users from a migration batch and removes existing move requests.
$r = Get-MigrationUser -BatchId MyMigrationBatch | ?{$_.status -eq 'Failed'} $r | %{Remove-MoveRequest -Identity $_.MailboxIdentifier -Confirm:$false}
Enjoy Exchange Server!
You might see the following error in the Windows Application Event Log:
The request failed. Mailbox: Url: https://officeclient.microsoft.com/config16?CV=15.1.2308.14&Client=WAC_Outlook&corr=910a665e-fd34-4c4b-8ed9-8a38fb515483 Exception: System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult) at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult) --- End of inner exception stack trace --- at System.Net.TlsStream.EndWrite(IAsyncResult asyncResult) at System.Net.ConnectStream.WriteHeadersCallback(IAsyncResult ar) --- End of inner exception stack trace --- at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult) at Microsoft.Exchange.Data.ApplicationLogic.Extension.BaseAsyncOmexCommand.<>c__DisplayClass18_0.<EndGetResponseCallback>b__0()
The request is successful when you try to connect to the URL provided in the error details using a browser on the Exchange server.
You can verify that the issue by trying to access the URL using the PowerShell Invoke-WebRequest cmdlet. Open a new PowerShell session and try connecting to the URL.
$uri='https://officeclient.microsoft.com/config16?CV=15.1.2308.14&Client=WAC_Outlook&corr=910a665e-fd34-4c4b-8ed 9-8a38fb515483' Invoke-WebRequest -Uri $uri
You will receive the same error message as stated in the event log by MSExchangeApplicationLogic. A successful connection returns XML as content.
The reason for this error is related to the .NET Framework TLS configuration, not Exchange Server. The .NET Framework lacks configuration for the use of TLS 1.2.
The solution for this issue is to configure the .NET Framework to correctly use TLS 1.2. You can follow the description for TLS 1.2 enforcement for Azure AD Connect, or you can simply use this Gist.
Due to the changes made to the SCHANNEL configuration you just restart the computer to bring the changes into effect.
Changing the TLS settings does not only affect outgoing connections but incoming connections as well.
Test the TLS changes in a test environment before adjusting your servers in the production environment. If you have not already enabled TLS 1.2 for your Exchange Servers, I recommend reading the 3-part series by the Exchange product group.
You might face a situation during an Exchange Server migration where your Exchange Server 2019 mailbox users are not able to open their public folder favorites when using Outlook on the Web (OWA).
When your users try to access a public folder, they receive an error message.
This error occurs when the public folder mailboxes are still hosted on a previous version of Exchange Server. This includes Exchange Server 2016 and 2013.
The online documentation explains, why this is happening:
The solution to this problem is easy. Move the public folder mailboxes to Exchange Server 2019 before you migrate any user mailboxes.
This approach ensures that mailboxes hosted on Exchange Server 2019 and previous versions of Exchange Server are able to access public folders using Outlook on the Web.
When you create or update an Exchange hybrid configuration using the Hybrid Configuration Wizard magic things happen. That's why it is called a Wizard.
One essential step of the Hybrid Configuration Wizard (HCW) is the configuration of the hybrid mail-flow. The hybrid mail-flow is required for both, classic and modern Exchange hybrid.
The wizard asks you to select one or more Exchange servers that you will utilize for handling inbound mail traffic from Exchange Online to your on-premises organization. You either configure direct mail flow to your Exchange Mailbox Servers in your internal company network, or to your Edge Transport Servers located in the perimeter network.
The following screenshot example shows the selection dialogue.
You can only select a server object, but not a receive connector on that selected server. The HCW chooses the "right" receive connector on the selected servers for you. If you are using the default set of receive connectors, you will not encounter any issues. HCW will use the default frontend connector on a mailbox server. When you use an Edge Transport Server you will run into any trouble as well. There is only one receive connector which you must extend by setting some additional parameters.
But what about an Exchange Organization where each mailbox server hosts multiple receive connectors bound to TCP port 25?
When you use multiple receive connectors bound to TCP 25 you will see that HCW will choose a receive connector that you won't expect. You might think that HCW will select always the default frontend connector. That is not the case.
When you select multiple servers for hybrid mail-flow, and each server has a different receive connector configuration, you might get the impression that HCW selects the receive connector randomly. That is not the case either.
While doing some testing in a large enterprise infrastructure with five different Exchange forests (development, testing, staging, pre-production, production) we saw an interesting behavior.
From all available receive connectors having a TCP 25 binding, HCW selects the receive connector with matching RemoteIPRanges values of:
Adjusting the default receive connector does have a direct impact on how HCW selects a receive connector in your Exchange environment. When you use multiple receive connectors for internal relay purposes, your receive connectors might end up in a messing situation. As mentioned, HCW selects receive connectors with a TCP 25 binding, regardless of the transport location of the connectors, frontend, or hub transport. The enterprise environment mentioned had some deviations between the different environments and we saw TCP 25 receive connectors in frontend transport and hub transport.
Run the HCW and select only one server for hybrid mail-flow and identify the receive connector configured by HCW. Configure an appropriate receive connector on all other mailbox servers used for hybrid mail flow. Update the hybrid configuration object of your on-premises Exchange Organization accordingly.
Verify the following two Tls* parameters of the receive connector:
Get-ReceiveConnector 'EXSRV01\Default Frontend EXSRV01' | fl tls* TlsCertificateName : <I>CN=Sectigo RSA Domain Validation Secure Server CA, O=Sectigo Limited, L=Salford, S=Greater Manchester, C=GB<S>CN=mail.varunagroup.de, OU=PositiveSSL, OU=Domain Control Validated TlsDomainCapabilities : {mail.protection.outlook.com:AcceptCloudServicesMail}
You must ensure that the hybrid receive connector uses the correct TLS certificate, enabled for SMTP. Additionally, you must set the TlsDomainCapabilitiers to allow cloud mail for connections incoming connections with a TLS certificate for mail.protection.outlook.com.
Keep your receive connectors at frontend transport.