I was involved in a troubleshooting request for a hybrid mail flow issue. Before I take a closer look at the issue, let's talk about the hybrid setup.

Hybrid Setup

A managed service provider runs separated on-premises Exchange Organizations for various clients. Also, the service provider runs it's own Exchange Organization in a hybrid setup with Exchange Online (EXO) utilizing centralized mail flow. Let's name the managed service provider Varunagroup, using the primary domain varunagroup.de.

The on-premises IT-Infrastructure consists of the following email components:

• Centralized Third-Party Email Gateway Solution with two nodes
  TLS certificates in use
  • mx01.varunagroup.de
  • mx02.varunagroup.de
     
• Varunagroup on-premises Exchange Organisation
  • Hybrid setup with Exchange Online
  • Hybrid mail flow using Edge Transport Servers
    TLS certificate in use
    • smtpo365.varunagroup.de
  • Centralized mail flow with EXO inbound connector configured by HCW 
  • Tenant name: varunagroup.onmicrosoft.com
  • Internet Send Connector with address space '*' uses the centralized Third-Party gateways as smart hosts
     
•  Multiple separated on-premises Exchange Organization hosted for SPLA-clients
  • Internet Send Connector with address space '*' uses the centralized Third-Party gateways as smart hosts

The following diagram illustrates the setup and the expected mail flow.

Let's name one of the clients Setebos AG, using setebos-ag.com as their primary domain. 

The Issue

Varunagroup's IT department activated journaling in Exchange Online, using an on-premises Journaling mailbox. After a few days, an IT administrator checked the inbox folder for journaling messages and journaling reports. The journaling inbox did not contain messages of Varounagroup senders or recipients only, but messages from client sender domains, e.g., setebos-ag.com.

In reality, the mail flow from on-premises to external recipients from any of the local Exchange organizations looked like shown in this diagram.

Question

Why does the Variangoup journaling mailbox contain messages from Setebos senders sent to external recipients?

We choose a single message for troubleshooting purposes, originating from the Setebos.com domain, sent to a non-Varunagroup recipient.

Analysis

1. The first thing to check is the Exchange Online Message Trace.
   In this case, the administrator already checked the Message Trace using the legacy Exchange Online Admin Center.
   
   The Exchange Online message trace showed the Varunagroup Exchange Online tenant received the Setebos message.

• Row 1: Exchange Online received the message for Varunagroup 
• Rows 2-5: The DLP Journaling rule processed the message, and the journaling report got routed to the journaling mailbox
• Row 6: The message was sent to an external mail server using the Exchange Online DNS resolver
• Row 7: Spam diagnosis for outgoing messages

The interesting piece of information is row 6. 

You see that EXO resolves the target mail exchanger via DNS. The target is another Microsoft 365 tenant as we see an xxx.mail.protection.outlook.com host.
 

2. Why did this message end up in the Varunagroup tenant?

When checking the on-premises mail gateway connection log, we found the distracting information that the gateway resolved the target mail exchanger as xxx.mail.protection.outlook.com.

As a next step, we checked the extended message tracking log using the new Exchange Admin Center. We created a new custom query with the following search criteria:

• Time range: Last 7 days
• Message-Id: The message Id fetched from the outbound connection log 
• Report type: Extended report

When you troubleshoot connection issues with Exchange Online, always select the extended report. You'll receive the report as a CSV file attachment. Use the Data tab in Excel to import the CSV file. Do not access the content by simply clicking the received file attachment. 

The interesting information is stored in the custom_data column for row source=SMTP and event_id=RECEIVE. 

S:ProxyHop1=HE1EUR01FT049.mail.protection.outlook.com(10.152.0.221);
S:ProxyHop2=AM0PRxxCAxxxx.outlook.office365.com(2603:10a6:208:fa::40);
S:InboundConnectorData=Name=Inbound from [EXCHANGE ORG GUID];
ConnectorType=OnPremises;
TenantId=[VARUNAGROUP GUID];
S:InboundTlsDetails=TLS=SP_PROT_TLS1_2_SERVER [...];
S:CorrelationId=d9ac6a10-8de9-4308-4205-07d865e8909b;
S:MimeParts=Att/Emb/MPt:0/0/1;
S:MessageValue=MediumHigh;
S:Replication=AM6PRxxxxMBxxxx;
S:FirstForestHop=AM0PRxxxxMBxxxx.eurprd03.prod.outlook.com;
S:FromEntity=HybridOnPrem;
S:Oorg=varunagroup.de;
S:ProxiedClientIPAddress=81.173.212.44;
S:ProxiedClientHostname=mx01.varunagroup.de;
S:DeliveryPriority=Normal;
S:AccountForest=EURPRxxAxxx.PROD.OUTLOOK.COM

The information in line 3 shows the actual name of the configured Varunagroup inbound connector, as shown in the Exchange Online connector configuration. The message did not enter the Varunagroup EXO tenant due to a mysterious connection, it was received by the dedicated inbound connector, configured by HCW.

3. Why was the Hybrid Inbound Connector chosen?

The key to this question is the TLS certificate used by the centralized email gateway and the TLS common name filtering in Exchange Online.

• The email gateways use the following TLS certificate with the two following common names
  • mx01.varunagroup,de
  • mx02.varunagroup.de
    ​
• The hybrid inbound connector used the TLS common name filtering, controlled by the TlsSenderCertificateName attribute, with the following name
  • *.varunagroup.de

The wildcard name *.varunagroup.de resulted in a matching string comparison for the incoming TLS common names of mx01.varunagroup.de and mx02.varunagroup.de. At the same time, the inbound connector matched the Edge Transport TLS certificate smtpo365.varunagroup.de.

Nobody knew, how the inbound connector configuration got "changed" to the wildcard name or for how long that configuration resulted in outbound messages from customer domains routed via the service provider tenant.

Solution

The solution contains two configurations.

1. Ensuring that the FQDN attribute of the Edge Send Connector is set to smtpo365.varunagroup.de
   
   This ensures that Exchange Server Transport selects the installed and SMTP-enabled TLS certificate for that name.  
    
2. ​Changing the TlsSenderCertificateName to smtpo365.veruangroup.de 
   
   This ensures that Exchange Online selects the hybrid inbound connector for Edge Transport established connections only.
    

The TLS common name behavior is by design and described in this blog post as FAQ #6(b). As a customer, you identify this as a misbehaving SMTP receive connector. But as described in the blog post, this is by design.

It is required that you understand the inbound routing behavior of Exchange Online if you have complicated outbound routing requirements. The blog post provides detailed information on how Office 365 inbound routing works and what you should be aware of.
 

Links

• Office 365 Attribution
• Hybrid Deployment Requirements
• Hybrid Configuration Wizard

Enjoy Exchange Online.

When you run software solutions that make use of TLS secured communication channels the applications need to have access to the certificate's private key. The private key is part of the certificate stored in the local certificate store of the computer. In most cases the software solution creates a new self-signed certificate and configures access rights appropriately.

When establishing TLS communication channels to external partners, the use of a public SSL/TLS certificate is a must have requirement.

The following step-by-step instructions describe how to assign Read permisson for the Email Security Solution Gateway NoSpamProxy. In this case the solution does not utilize a classic service account, but a so-called virtual service account. Virtual service accounts provide a much better access security when executing Windows services.

Step-by-Step Instructions

Step 1

Open the local computers certificate store using the MMC Snap-Ins.

Step 2

Select the certificate to use and open the context menu (right click).

Select Manage Private Keys to manage the private key permissions.

Step 3

Click Add and add the required service accounts.

In this case the virtual service accounts are part of the local computer entity. Select the local computer and not the Active Directory domain as source when searching accounts. Virtual accounts us the prefix NT Service.

Add the follow accounts to configure read access for NoSpamProxy on a server having the Gateway and Intranet role installed.

NT Service\NetatworkMailGatewayIntranetRole
NT Service\NetatworkMailGatewayManagementService
NT Service\NetatworkMailGatewayGatewayRole
NT Service\NetatworkMailGatewayPrivilegedService

Add the follow accounts to configure read access for NoSpamProxy on a server having the Gateway role installed only.

NT Service\NetatworkMailGatewayManagementService
NT Service\NetatworkMailGatewayGatewayRole
NT Service\NetatworkMailGatewayPrivilegedService

Click Check Names to verifiy the existence of the entered service accounts.

Step 4

When correctly resolved the accounts names are replaced by theis respective display names. Click OK to add the accounts. 

Step 5

Configure read access for all added service accounts and click OK.

The software solution is now capable of accessing the private key of the certificate.

Link

• Service Accounts Step-by-Step Guide

This blog post focusses on an issue where your Exchange Online users are not able to send emails to other Exchange Online recipients outside of your organization when using a 3rd Party Centralized Email Flow Setup. The term 3rd Party Centralized Email Flow Setup describes a solution where you not follow the preferred hybrid architecture proposed by the Exchange product group, but use a 3rd party software as a centralized email gateway.

Problem

You have followed the recommendation to secure the Exchange Online inbound connector for your on-premises email servers by using a certificate name or the remote IP address of your on-premises email gateway.

Assumption

The on-premises email security gateway utilizes a self-signed certificate to secure TLS connections. The gateway is configured to use two different send connector setups:

• Internet Connector
  Use receipients domain MX records
  Use self-signed certificate
  Target address space: *
   
• Office 365 Connector
  Use tenant.mail.protection.outlook.com to route internal email messages
  Use self-signed certificate
  Target address space: tenant.mail.onmicrosoft.com

In this case Exchange Online Protection (EOP) will not be able to differentiate between tenant internal inbound mail flow and mail flow targeted to other tenants. Therefore, email messages sent from your Exchange Online users to recipients located in other Exchange Online tenants will be discarded.

Interestingly enough, this will happen silently. Your gateway solution will log a successful delivery to Exchange Online Protection. The tenant administrator of the recipient domain will not find an any information in the Exchange Online message tracking logs.

The following diagram illustrates the setup.

Solution

The solution for this problem is pretty simple. Just use dedicated certificates for each connector targetting Exchange Online.

Change the Internet Connector to fully trusted 3rd party certificate. In this case you are not required to modify the existing Exchange Online inbound connector setup.

The new connector configurations are:

• Internet Connector
  Use receipients domain MX records
  Use 3rd party certificate
  Target address space: *
   
• Office 365 Connector
  Use tenant.mail.protection.outlook.com to route internal email messages
  Use self-signed certificate
  Target address space: tenant.mail.onmicrosoft.com

The following diagram illustrates the new setup:

Links

• Important notice for Office 365 email customers who have configured connectors, https://blogs.technet.microsoft.com/exchange/2016/03/29/important-notice-for-office-365-email-customers-who-have-configured-connectors/
• Mail flow best practices for Exchange Online and Office 365 (overview), https://technet.microsoft.com/en-us/library/jj937232(v=exchg.150).aspx

Enjoy!

Problem

When you integrate Skype for Business Server instant messaging with Exchange Server 2013 or Exchange Server 2016 you might encounter the following error in the OWA InstantMessaging log files.

ERROR:UCWEB Failure: Code=TlsFailure, SubCode=TlsRemoteDisconnected, Reason=\r\n
Microsoft.Rtc.Internal.UCWeb.Utilities.UCWException: Unknown error (0x80131500) 
---> Microsoft.Rtc.Signaling.TlsFailureException: Unknown error (0x80131500) 
---> Microsoft.Rtc.Internal.Sip.RemoteDisconnectedException: Peer disconnected while outbound capabilities negotiation was in progress 
---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host\r\n   
at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)\r\n   
at Microsoft.Rtc.Internal.Sip.TcpTransport.OnReceived(Object arg)\r\n   
--- End of inner exception stack trace ---\r\n   
--- End of inner exception stack trace ---\r\n   
at Microsoft.Rtc.Signaling.SipAsyncResult`1.ThrowIfFailed()\r\n   
at Microsoft.Rtc.Signaling.Helper.EndAsyncOperation[T](Object owner, IAsyncResult result)\r\n   
at Microsoft.Rtc.Internal.UCWeb.UCWAuthenticatedEndpoint.OotyUserEndpointEstablish_callback(IAsyncResult asyncResult)\r\n   
--- End of inner exception stack trace ---\r\n   
at Microsoft.Rtc.Internal.UCWeb.Utilities.AsyncHelper.EndAsyncCall[T](IAsyncResult asyncResult, String methodName, T ucwScopeInstance)\r\n   
at Microsoft.Rtc.Internal.UCWeb.UCWAuthenticatedEndpoint.EndSignIn(IAsyncResult asyncResult)\r\n   
at Microsoft.Exchange.Clients.Owa2.Server.Core.InstantMessageOCSProvider.<>c__DisplayClass33.<SignInCallback>b__32(RequestDetailsLogger logger)

The log files are located at

\Program Files\Microsoft\Exchange Server\V15\Logging\OWA\InstantMessaging

Solution

The Exchange Server OWA host name must be the common name (CN) of the SSL certificate used securing OWA communication.

Example for a non working IM configuration

• OWA host name: owa.varunagroup.de
• SSL certificate CN: mobile.varunagroup.de

Example for a working IM configuration

• OWA host name: owa.varunagroup.de
• SSL certificate CN: owa.varunagroup.de

Links

• Integrate Skype for Business Server 2015 and Microsoft Outlook Web App
  https://technet.microsoft.com/en-us/library/jj688055.aspx
• Configure instant messaging integration with Outlook on the web in Exchange 2016
  https://technet.microsoft.com/en-us/library/mt607043(v=exchg.160).aspx