MVP - Most Valuable Professional
rss

Just can't get enough of IT

This blog is about mostly anything in IT. But the primary focuses are Microsoft technologies like Exchange Server, Microsoft 365, Microsoft Teams, and Cloud Security.

I was involved in a troubleshooting request for a hybrid mail flow issue. Before I take a closer look at the issue, let's talk about the hybrid setup.

Hybrid Setup

A managed service provider runs separated on-premises Exchange Organizations for various clients. Also, the service provider runs it's own Exchange Organization in a hybrid setup with Exchange Online (EXO) utilizing centralized mail flow. Let's name the managed service provider Varunagroup, using the primary domain varunagroup.de.

The on-premises IT-Infrastructure consists of the following email components:

  • Centralized Third-Party Email Gateway Solution with two nodes
    TLS certificates in use
    • mx01.varunagroup.de
    • mx02.varunagroup.de
       
  • Varunagroup on-premises Exchange Organisation
    • Hybrid setup with Exchange Online
    • Hybrid mail flow using Edge Transport Servers
      TLS certificate in use
      • smtpo365.varunagroup.de
    • Centralized mail flow with EXO inbound connector configured by HCW 
    • Tenant name: varunagroup.onmicrosoft.com
    • Internet Send Connector with address space '*' uses the centralized Third-Party gateways as smart hosts
       
  •  Multiple separated on-premises Exchange Organization hosted for SPLA-clients
    • Internet Send Connector with address space '*' uses the centralized Third-Party gateways as smart hosts

The following diagram illustrates the setup and the expected mail flow.

Diagram showing the expected Exchange Online mail flow

 

Let's name one of the clients Setebos AG, using setebos-ag.com as their primary domain. 

 

The Issue

Varunagroup's IT department activated journaling in Exchange Online, using an on-premises Journaling mailbox. After a few days, an IT administrator checked the inbox folder for journaling messages and journaling reports. The journaling inbox did not contain messages of Varounagroup senders or recipients only, but messages from client sender domains, e.g., setebos-ag.com.

In reality, the mail flow from on-premises to external recipients from any of the local Exchange organizations looked like shown in this diagram.

Diagram showing the mail flow relayed through the Varunagroup tenant

 

Question

Why does the Variangoup journaling mailbox contain messages from Setebos senders sent to external recipients?

We choose a single message for troubleshooting purposes, originating from the Setebos.com domain, sent to a non-Varunagroup recipient.

 

Analysis

  1. The first thing to check is the Exchange Online Message Trace.
    In this case, the administrator already checked the Message Trace using the legacy Exchange Online Admin Center.

    The Exchange Online message trace showed the Varunagroup Exchange Online tenant received the Setebos message.

Screenshot - Exchange Online Message Trace

  • Row 1: Exchange Online received the message for Varunagroup 
  • Rows 2-5: The DLP Journaling rule processed the message, and the journaling report got routed to the journaling mailbox
  • Row 6: The message was sent to an external mail server using the Exchange Online DNS resolver
  • Row 7: Spam diagnosis for outgoing messages

The interesting piece of information is row 6. 

You see that EXO resolves the target mail exchanger via DNS. The target is another Microsoft 365 tenant as we see an xxx.mail.protection.outlook.com host.
 

  1. Why did this message end up in the Varunagroup tenant?

When checking the on-premises mail gateway connection log, we found the distracting information that the gateway resolved the target mail exchanger as xxx.mail.protection.outlook.com.

As a next step, we checked the extended message tracking log using the new Exchange Admin Center. We created a new custom query with the following search criteria:

  • Time range: Last 7 days
  • Message-Id: The message Id fetched from the outbound connection log 
  • Report type: Extended report

When you troubleshoot connection issues with Exchange Online, always select the extended report. You'll receive the report as a CSV file attachment. Use the Data tab in Excel to import the CSV file. Do not access the content by simply clicking the received file attachment. 

The interesting information is stored in the custom_data column for row source=SMTP and event_id=RECEIVE

S:ProxyHop1=HE1EUR01FT049.mail.protection.outlook.com(10.152.0.221);
S:ProxyHop2=AM0PRxxCAxxxx.outlook.office365.com(2603:10a6:208:fa::40);
S:InboundConnectorData=Name=Inbound from [EXCHANGE ORG GUID];
ConnectorType=OnPremises;
TenantId=[VARUNAGROUP GUID];
S:InboundTlsDetails=TLS=SP_PROT_TLS1_2_SERVER [...];
S:CorrelationId=d9ac6a10-8de9-4308-4205-07d865e8909b;
S:MimeParts=Att/Emb/MPt:0/0/1;
S:MessageValue=MediumHigh;
S:Replication=AM6PRxxxxMBxxxx;
S:FirstForestHop=AM0PRxxxxMBxxxx.eurprd03.prod.outlook.com;
S:FromEntity=HybridOnPrem;
S:Oorg=varunagroup.de;
S:ProxiedClientIPAddress=81.173.212.44;
S:ProxiedClientHostname=mx01.varunagroup.de;
S:DeliveryPriority=Normal;
S:AccountForest=EURPRxxAxxx.PROD.OUTLOOK.COM

The information in line 3 shows the actual name of the configured Varunagroup inbound connector, as shown in the Exchange Online connector configuration. The message did not enter the Varunagroup EXO tenant due to a mysterious connection, it was received by the dedicated inbound connector, configured by HCW.

 

  1. Why was the Hybrid Inbound Connector chosen?

The key to this question is the TLS certificate used by the centralized email gateway and the TLS common name filtering in Exchange Online.

  • The email gateways use the following TLS certificate with the two following common names
    • mx01.varunagroup,de
    • mx02.varunagroup.de
  • The hybrid inbound connector used the TLS common name filtering, controlled by the TlsSenderCertificateName attribute, with the following name
    • *.varunagroup.de

The wildcard name *.varunagroup.de resulted in a matching string comparison for the incoming TLS common names of mx01.varunagroup.de and mx02.varunagroup.de. At the same time, the inbound connector matched the Edge Transport TLS certificate smtpo365.varunagroup.de.

Nobody knew, how the inbound connector configuration got "changed" to the wildcard name or for how long that configuration resulted in outbound messages from customer domains routed via the service provider tenant.

 

Solution

The solution contains two configurations.

  1. Ensuring that the FQDN attribute of the Edge Send Connector is set to smtpo365.varunagroup.de

    This ensures that Exchange Server Transport selects the installed and SMTP-enabled TLS certificate for that name.  
     
  2. Changing the TlsSenderCertificateName to smtpo365.veruangroup.de 

    This ensures that Exchange Online selects the hybrid inbound connector for Edge Transport established connections only.
     

The TLS common name behavior is by design and described in this blog post as FAQ #6(b). As a customer, you identify this as a misbehaving SMTP receive connector. But as described in the blog post, this is by design.

It is required that you understand the inbound routing behavior of Exchange Online if you have complicated outbound routing requirements. The blog post provides detailed information on how Office 365 inbound routing works and what you should be aware of.
 

The simple rule is: 
Always use dedicated TLS certificates for separating mail flow to Exchange Online. Especially when using centralized mail flow for your Microsoft 365 tenant.

 

Links

 

Enjoy Exchange Online.

 

 

 

Read More »

The Microsoft 365 Virtual Marathon took place on May 27-28 2020.

The recording of my session "Exchange Hybrid - What, Why, and How" is available on YouTube. 

 

Browse all recordings of the Microsoft 365 Virtual Marathon here: https://www.youtube.com/channel/UCrtmT6Ir1MIs0ZES7sKMmqA

Enjoy!

 

 

Read More »

Microsoft 365 Groups are the backbone of various Microsoft 365 workloads. As you might know, each group utilizes a SharePoint site collection, and an Exchange shared mailbox.

When you create a new Microsoft 365 group, SharePoint Online must store the associated site collection somewhere. SharePoint Online uses predefined paths to determine the storage location. These paths are called: Managed Paths.

SharePoint Online uses two different pre-configured managed paths:

  • /sites
  • /teams

With /sites as the default setting for the Microsoft 365 tenant.

Whenever you create, e.g., a new team in Microsoft Teams, the associated site collection is stored in https://TENANTNAME.sharepoint.com/sites/TEAMNAME. As a SharePoint administrator, you see the site collection paths in the list of active sites in the SharePoint Admin Center.

Screenshot SharePoint Online Active Sites

 

But what can you do, if you want to store the associated site collections in the /teams managed path?

 

Changing the Managed Path

The SharePoint Admin Center provides you with an option to change the managed path for sites, created by users. 

Open the SharePoint Admin Center, navigate to Settings -> Site Creation.

Screenshot SharePoint Admin Center Settings -> Site Creation

 

Change the setting for Create team sites under to /teams/.

Screenshot SharePoint Admin Center - Site creation

 

The description of this setting is misleading. This setting affects not only SharePoint team site creation initiated by users on the SharePoint start page or OneDrive, but site collections created by Microsoft 365 Groups as well.

You do not need to enable the checkbox to let users create sites from the SharePoint start page and OneDrive. This setting is only required, when you want to enable self-service site creation of modern SharePoint sites for users. The modern SharePoint sites are based on Microsoft 365 Groups.

After changing the path, SharePoint Online creates new associated site collections for Microsoft 365 Groups in /teams/.

Screenshot - SharePoint Admin Center - Active Sites

 

Note:
Changing this setting affects Microsoft 365 Groups in general. It does not, which Microsoft 365 app you use to create a new group.
The associated site collection for a new plan in Planner, a new team in Microsoft Teams, a new Group in Outlook on the Web, or even a new website in OneDrive, is created using this configured path.

 

Enjoy SharePoint Online.

 

 

Read More »

Logo - Microsoft 365 Virtual ConferenceThe Microsoft 365 Virtual Marathon is happening on May 27-28 2020.

This is a free online event, providing you with 36 hours of non-stop sessions from speakers around the globe. You can join every time. 

  • 300+ Speakers
  • 400+ Sessions
  • Keynotes by Jeff Teper, Naomi Moneypenny, Bill Baer, Jon Levesque, Laurie Pottmeyer, and Michael Holste
  • Sessions are primarily in Englisch, but there are session in six additional languages
    • French, German, Japanese, Korean, Portuguese, and Spanish 
  • Hashtag: #M365VM

The virtual conference is a joint effort with SPC.

 

I speak at Microsoft 365 Virtual Marathon about:

  • Exchange Hybrid - What, Why, and How
  • Wednesday, May 27th
  • 23:00h (CEST)

 

The marathon session plan is available here.

Register now.

 

Link

 

Enjoy!

 

 

Read More »

Icon Exchange Server 2019When you plan to implement an Exchange Hybrid Configuration between your on-premises Exchange Organization and Exchange online you have to choose between two variants and five operating modes. It is not as complicated as it sounds.

I have written a blog post about the different options available. 

The post is published in ENow's ESE blog.

Enjoy.

 

 

Read More »