I was involved in a troubleshooting request for a hybrid mail flow issue. Before I take a closer look at the issue, let's talk about the hybrid setup.
A managed service provider runs separated on-premises Exchange Organizations for various clients. Also, the service provider runs it's own Exchange Organization in a hybrid setup with Exchange Online (EXO) utilizing centralized mail flow. Let's name the managed service provider Varunagroup, using the primary domain varunagroup.de.
The on-premises IT-Infrastructure consists of the following email components:
The following diagram illustrates the setup and the expected mail flow.
Let's name one of the clients Setebos AG, using setebos-ag.com as their primary domain.
Varunagroup's IT department activated journaling in Exchange Online, using an on-premises Journaling mailbox. After a few days, an IT administrator checked the inbox folder for journaling messages and journaling reports. The journaling inbox did not contain messages of Varounagroup senders or recipients only, but messages from client sender domains, e.g., setebos-ag.com.
In reality, the mail flow from on-premises to external recipients from any of the local Exchange organizations looked like shown in this diagram.
Why does the Variangoup journaling mailbox contain messages from Setebos senders sent to external recipients?
We choose a single message for troubleshooting purposes, originating from the Setebos.com domain, sent to a non-Varunagroup recipient.
The interesting piece of information is row 6.
You see that EXO resolves the target mail exchanger via DNS. The target is another Microsoft 365 tenant as we see an xxx.mail.protection.outlook.com host.
When checking the on-premises mail gateway connection log, we found the distracting information that the gateway resolved the target mail exchanger as xxx.mail.protection.outlook.com.
As a next step, we checked the extended message tracking log using the new Exchange Admin Center. We created a new custom query with the following search criteria:
When you troubleshoot connection issues with Exchange Online, always select the extended report. You'll receive the report as a CSV file attachment. Use the Data tab in Excel to import the CSV file. Do not access the content by simply clicking the received file attachment.
The interesting information is stored in the custom_data column for row source=SMTP and event_id=RECEIVE.
S:InboundConnectorData=Name=Inbound from [EXCHANGE ORG GUID];
The information in line 3 shows the actual name of the configured Varunagroup inbound connector, as shown in the Exchange Online connector configuration. The message did not enter the Varunagroup EXO tenant due to a mysterious connection, it was received by the dedicated inbound connector, configured by HCW.
The key to this question is the TLS certificate used by the centralized email gateway and the TLS common name filtering in Exchange Online.
The wildcard name *.varunagroup.de resulted in a matching string comparison for the incoming TLS common names of mx01.varunagroup.de and mx02.varunagroup.de. At the same time, the inbound connector matched the Edge Transport TLS certificate smtpo365.varunagroup.de.
Nobody knew, how the inbound connector configuration got "changed" to the wildcard name or for how long that configuration resulted in outbound messages from customer domains routed via the service provider tenant.
The solution contains two configurations.
The TLS common name behavior is by design and described in this blog post as FAQ #6(b). As a customer, you identify this as a misbehaving SMTP receive connector. But as described in the blog post, this is by design.
It is required that you understand the inbound routing behavior of Exchange Online if you have complicated outbound routing requirements. The blog post provides detailed information on how Office 365 inbound routing works and what you should be aware of.
Enjoy Exchange Online.
Are you located in Germany, Austria, or Switzerland? Join the Exchange User Group DACH to collaborate with other Exchange enthusiasts.
Follow us on Twitter @exusg, join on Meetup, or visit our website.