MVP - Most Valuable Professional
rss

Just can't get enough of IT

This blog is about mostly anything in IT. But the primary focuses are Microsoft technologies like Exchange Server, Microsoft 365, Microsoft Teams, and Cloud Security.

These are the results of the  Exchange Server Questionnaire from August 2021.

First of all, I want to thank all of you who participated in the questionnaire. The results are pretty interesting. Even though, that the results are not 100% representative they provide a high-level view of the Exchange Organizations, the mail flow configurations, and the future plans regarding hybrid and Exchange Online.

With 55 replies the questionnaire is far from being a comprehensive representation of the Exchange organizations. But the answers provide an idea of the Exchange landscape used by organizations globally.

  

1. Exchange Server Versions in use (Production)

Exchange Server 2016 is the dominant version currently in use, followed by Exchange Server 2019. The vast majority of 93% runs modern Exchange Server versions. But there are still older and unsupported Exchange Server versions in use. 7% use Exchange Server 2010 and older. 

 

Diagram Exchange Server Versions in use (Production)

 

2. How many Exchange Server systems do you operate?

76% of the organizations maintain up to ten Exchange servers. 20% prefer to rely on just one Exchange server. It is interesting that only 2 (not percent) plan to go hybrid or to move to Exchange Online.  

 

Diagram How many Exchange Servers do you operate?

 

3. How many mailboxes do your Exchange Servers host?

The majority of on-premises Exchange organizations are in the 1,000 - 10,000 mailboxes range. Nevertheless, the SMBs with 1 to 1,000 mailboxes adds up to 50% of the Exchange organizations that took part in this questionnaire. There are just a few organizations that host more than 50,000 mailboxes.    

 

Diagram How many mailboxes do your Exchange Servers host?

 

4. Do you use an on-premises or cloud-based SMTP gateway solution?

There are Exchange organizations that do not use an SMTP-Gateway solution as part of the mail-flow implementation. Thor organizations that do not use a gateway solution run 1 to 10 Exchange servers on-premises. The majority of those have less than 1,000 mailboxes but there are a few that are responsible for more than 1,000 mailboxes. That leaves the question of why an organization prefers to not secure mal-flow with a gateway.

 

Diagram Do you use an on-premises or cloud-based SMTP gateway solution?

 

5. Which product do you use as a gateway solution?

The use of SMTP gateways is a must, as you do not want to expose your domain member servers to the Internet, not even for the SMTP protocol. A majority of 28 answers for other gateways shows, that there are so many products available and that I did not choose valid answer options upfront. 

 

Diagram Which product do you use as a gateway solution?

The Other answers include:

  • Cisco ESA
  • Clearswift
  • Eleven
  • Fortigate
  • IronPort
  • Postfix
  • Reddoxx
  • Trustwave

 

6. Is your current Exchange organization using a hybrid configuration with Exchange Online?

65% of the Exchange organizations of this questionnaire already run in a hybrid configuration with Exchange Online. Only 35% are (still) not using a hybrid setup.  

 

Diagram Is your current Exchange organisation using a hybrid configuration with Exchange Online?

 

7. Do you plan to implement a hybrid Exchange configuration or to move to Exchange Online?

Of those who currently do not run a hybrid configuration only 37% plan on implementing Exchange Hybrid or migrate fully to Exchange Online. Staying on-premises is the only option.

 

Diagram Do you plan to implement a hybrid Exchange configuration or to move to Exchange Online?

 

8. Until when do you plan to implement a hybrid configuration or go cloud-only?

The majority of the organizations still running only an on-premises Exchange organization plan on implementing Exchange Hybrid or migrating to Exchange Online by the end of 2021. None of the participating organizations has plans scheduled after 2022.

Diagram Until when do you plan to implement a hybrid configuration or go cloud-only?

 

9. Which hybrid model did you choose?

It is no surprise that Classic Full Hybrid is the most adopted hybrid configuration. And, no surprise either, none of the other classic hybrid options is implemented. The modern hybrid approach is implemented but with lesser.

Diagram Which hybrid model did you choose?

 

10. What are the reasons for staying 100% on-premises?

The reasons for staying with an on-premises Exchange organization vary. the reasons mentioned are:

  • Enclosed environment, external access with BlackBerry UEM, due to public sector data security requirements
  • Mailbox data is classified as too sensitive
  • Too expensive and low internet bandwidth
  • Legal and clients audits 

There are still organizations that choose an on-premises Exchange organization in favor of Exchange Online. I wonder if company policies for reducing the carbon footprint might drive the migration of on-premises data center resources to hosted cloud services.  

 

11. Will you implement Exchange Server vNEXT?

Exchange Server vNEXT is in scope for 47% of the organizations. When comparing it with the used Exchange Server version currently in use (~50% Exchange Server 2016) it is an indicator that some companies just skip Exchange Server 2019. Some organizations prefer not to follow the full life-cycle of Exchange Server. s7% of those who do not want to implement Exchange Server vNEXT and want to stay on-premises are single server implementations of Exchange. 

Diagram Will you implement Exchange Server vNEXT?

 

 

Summary

The product Exchange Server is still widely used in on-premises deployments. The reasons vary from legal and compliance requirements, network bandwidth constraints, and the overall costs for Exchange Online. Exchange Server vNEXT is a must-have for nearly 50% of the organizations participating in this questionnaire. There are still older and unsupported versions in productive use. Why this is the case is unanswered in this questionnaire.

Organizations running a hybrid Exchange configuration primarily use a Classic Full Hybrid configuration. This might be due to an early implementation in those days when nothing else was available, or due to requirements using Microsoft Teams with on-premises mailboxes. The adoption of Modern Hybrid shows that the Hybrid Agent approach helps organizations that cannot implement a Classic Full Hybrid. 

I leave the results of this questionnaire to your interpretation and look forward to your replies, either to this blog post or by social media on Twitter and LinkedIn. Please use the hashtag #ExchangeQuest2021.

There will be a new Exchange Server questionnaire in early 2022, covering various implementation scenarios in more detail. If you want to see a specific Exchange topic covered in the 2022 questionnaire, just let me know.

Again, thank you all for participating in this questionnaire.

 

 

Read More »

IllustrationThe Problem

Mail flow from on-premises devices and applications to Exchange Online is a tricky topic. The documentation allows for different solutions.

Recently a client ran into a situation where an on-premises application was not able to deliver messages to a configured inbound connector in the Exchange Online tenant. The connector was configured for remote IP address selection.

Exchange Online responded to each connection attempt with the following error message:

  • 451 4.4.3 Temporary server error. Please try again later ATTR3.1

There weren't any changes on the on-premises configuration and the setup was in use for multiple months without any issues.

 

The Solution

It took some time to identify the solution, but in the end, the solution was easy.

Disabling and re-enabling solved the issue.  

 

Enjoy Exchange Online.

 

Read More »

PowerShellYou sometimes need some (or even many) test user objects in Active Directory.

This script helps you create any number of test users in your Active Directory domain, which you can easily enable for on-premises or remote mailboxes afterward.

 

The Script

# Number of user accounts to create
$UserCount = 5
$RandomPassword = $true
$DefaultPassword = 'Pa55w.rd'

# User name prefix
# New user object will be named TestUser1, TestUser2, ...
$TestUserPrefix = 'TestUser'

# User object properties
$GivenName = 'Test'
$Surname = 'User'
$Company = 'Varunagroup'
$JobTitle = @('Junior Consultant','Senior Consultant','Technical Consultant','Business Consultant')
$PreferredLanguage = 'de-DE'

# Name of the new organizational unit for test user object
$TestOU = 'Test User'

# Target OU path where the script creates the new OU 
$TargetOU = 'OU=IT,dc=varunagroup,dc=de'

# Import Active Directory PowerShell Module
Import-Module -Name ActiveDirectory 

# Build OU Path
$UserOUPath = ("OU={0},{1}" -f $TestOU, $TargetOU)

# Check if OU exists
$OUExists = $false

try {
   $OUExists = [adsi]::Exists("LDAP://$UserOUPath")
}
catch {
   $OUExists =$true   
}

if(-not $OUExists) { 
   # Create new organizational unit for test users
   New-ADOrganizationalUnit -Name $TestOU -Path $TargetOU -ProtectedFromAccidentalDeletion:$false -Confirm:$false
}
else {
   Write-Warning ('OU {0} exists please delete the OU and user objects manually, before running this script.' -f $UserOUPath)
   Exit
}

Write-Output ("Creating {0} user object in {1}" -f $UserCount, $UserOUPath)

# Create new user objects
1..$UserCount | ForEach-Object {

   # Get a random number for selecting a job title
   $random = Get-Random -Minimum 0 -Maximum (($JobTitle | Measure-Object). Count - 1)

   # Set user password
   if($RandomPassword) {
      # Create a random password
      $UserPassword = ConvertTo-SecureString -String (-join ((33..93) + (97..125) | Get-Random -Count 25 | % {[char]$_})) -AsPlainText -Force
   }
   else {
      # Use a fixed password
      $UserPassword = ConvertTo-SecureString -String $DefaultPassword -AsPlainText -Force
   }

   # Create a new user object
   # Adjust user name template and other attributes as needed
   New-ADUser -Name ("{0}{1}" -f $TestUserPrefix, $_) `
   -DisplayName ("{0} {1}" -f $TestUserPrefix, $_) `
   -GivenName $GivenName `
   -Surname ("$Surname{0}" -f $_) `
   -OtherAttributes @{title=$JobTitle[$random];company=$Company;preferredLanguage=$PreferredLanguage} `
   -Path $UserOUPath `
   -AccountPassword $UserPassword `
   -Enabled:$True `
   -Confirm:$false
}

 

Enable mailboxes

Use your on-premises Exchange Management Shell to enable all test users with an on-premises mailbox.

$UserOU = 'OU=Test User,OU=IT,dc=varunagroup,dc=de'
Get-User -OrganizationalUnit $UserOU | Enable-Mailbox -Confirm:$false

 

Use your on-premises Exchange Management Shell to enable all test users with a new remote mailbox in Exchange Online. Do not forget to change the tenant name of the remote routing address.

Get-User -OrganizationalUnit 'OU=Test User,OU=IT,dc=varunagroup,dc=de' | %{Enable-RemoteMailbox
 -Identity $_ -Confirm:$false -RemoteRoutingAddress "$($_.SamAccountName)@TENANT.mail.onmicrosoft.com"}

 

You find the most recent version of the script at GitHub.

 

Links

 

Enjoy.

 

Read More »
Use this script with modern public folders only. See this post for legacy public folders.

 

Exchange Server 2013Exchange Server 2016Exchange Server 2019Description

When you want to migrate your modern public folders from Exchange 2013 or newer to modern public folders in Exchange Online, you must prepare the public folder names for migration.

Public folder names are not allowed to contain the following:

  • A backslash "\"
  • A forward slash "/"
  • A semicolon ";"
  • A comma ","
  • A colon ":"
  • Leading or trailing spaces

The script Fix-ModernPublicFolderNames.ps1 fixes the public folder names to prepare migration to modern public folders in Exchange Online.

 

Examples

# EXAMPLE 1
# Rename and trim public folders

.\Fix-ModernPublicFolderNames.ps1

# EXAMPLE 2
# Rename and trim public folders, export list of renamed 
# folders and folders with renaming errors as text files

.\Fix-ModernPublicFolderNames.ps1 -ExportFolderNames

 

Version History

  • 1.0, Initial community release

 

Links

The script for updating modern public folder names and legacy public folder names share the same repository.

 

 

Follow

 

Community

Are you located in Germany, Austria, or Switzerland? Join the Exchange User Group DACH to collaborate with other Exchange enthusiasts.
Follow us on Twitter @exusg, join on Meetup, or visit our website

 

 

 

 

Read More »

I was involved in a troubleshooting request for a hybrid mail flow issue. Before I take a closer look at the issue, let's talk about the hybrid setup.

 

Hybrid Setup

A managed service provider runs separated on-premises Exchange Organizations for various clients. Also, the service provider runs it's own Exchange Organization in a hybrid setup with Exchange Online (EXO) utilizing centralized mail flow. Let's name the managed service provider Varunagroup, using the primary domain varunagroup.de.

The on-premises IT-Infrastructure consists of the following email components:

  • Centralized Third-Party Email Gateway Solution with two nodes
    TLS certificates in use
    • mx01.varunagroup.de
    • mx02.varunagroup.de
       
  • Varunagroup on-premises Exchange Organisation
    • Hybrid setup with Exchange Online
    • Hybrid mail flow using Edge Transport Servers
      TLS certificate in use
      • smtpo365.varunagroup.de
    • Centralized mail flow with EXO inbound connector configured by HCW 
    • Tenant name: varunagroup.onmicrosoft.com
    • Internet Send Connector with address space '*' uses the centralized Third-Party gateways as smart hosts
       
  •  Multiple separated on-premises Exchange Organization hosted for SPLA-clients
    • Internet Send Connector with address space '*' uses the centralized Third-Party gateways as smart hosts

The following diagram illustrates the setup and the expected mail flow.

Diagram showing the expected Exchange Online mail flow

 

Let's name one of the clients Setebos AG, using setebos-ag.com as their primary domain. 

 

The Issue

Varunagroup's IT department activated journaling in Exchange Online, using an on-premises Journaling mailbox. After a few days, an IT administrator checked the inbox folder for journaling messages and journaling reports. The journaling inbox did not contain messages of Varounagroup senders or recipients only, but messages from client sender domains, e.g., setebos-ag.com.

In reality, the mail flow from on-premises to external recipients from any of the local Exchange organizations looked like shown in this diagram.

Diagram showing the mail flow relayed through the Varunagroup tenant

 

Question

Why does the Variangoup journaling mailbox contain messages from Setebos senders sent to external recipients?

We choose a single message for troubleshooting purposes, originating from the Setebos.com domain, sent to a non-Varunagroup recipient.

 

Analysis

  1. The first thing to check is the Exchange Online Message Trace.
    In this case, the administrator already checked the Message Trace using the legacy Exchange Online Admin Center.

    The Exchange Online message trace showed the Varunagroup Exchange Online tenant received the Setebos message.

Screenshot - Exchange Online Message Trace

  • Row 1: Exchange Online received the message for Varunagroup 
  • Rows 2-5: The DLP Journaling rule processed the message, and the journaling report got routed to the journaling mailbox
  • Row 6: The message was sent to an external mail server using the Exchange Online DNS resolver
  • Row 7: Spam diagnosis for outgoing messages

The interesting piece of information is row 6. 

You see that EXO resolves the target mail exchanger via DNS. The target is another Microsoft 365 tenant as we see an xxx.mail.protection.outlook.com host.
 

  1. Why did this message end up in the Varunagroup tenant?

When checking the on-premises mail gateway connection log, we found the distracting information that the gateway resolved the target mail exchanger as xxx.mail.protection.outlook.com.

As a next step, we checked the extended message tracking log using the new Exchange Admin Center. We created a new custom query with the following search criteria:

  • Time range: Last 7 days
  • Message-Id: The message Id fetched from the outbound connection log 
  • Report type: Extended report

When you troubleshoot connection issues with Exchange Online, always select the extended report. You'll receive the report as a CSV file attachment. Use the Data tab in Excel to import the CSV file. Do not access the content by simply clicking the received file attachment. 

The interesting information is stored in the custom_data column for row source=SMTP and event_id=RECEIVE

S:ProxyHop1=HE1EUR01FT049.mail.protection.outlook.com(10.152.0.221);
S:ProxyHop2=AM0PRxxCAxxxx.outlook.office365.com(2603:10a6:208:fa::40);
S:InboundConnectorData=Name=Inbound from [EXCHANGE ORG GUID];
ConnectorType=OnPremises;
TenantId=[VARUNAGROUP GUID];
S:InboundTlsDetails=TLS=SP_PROT_TLS1_2_SERVER [...];
S:CorrelationId=d9ac6a10-8de9-4308-4205-07d865e8909b;
S:MimeParts=Att/Emb/MPt:0/0/1;
S:MessageValue=MediumHigh;
S:Replication=AM6PRxxxxMBxxxx;
S:FirstForestHop=AM0PRxxxxMBxxxx.eurprd03.prod.outlook.com;
S:FromEntity=HybridOnPrem;
S:Oorg=varunagroup.de;
S:ProxiedClientIPAddress=81.173.212.44;
S:ProxiedClientHostname=mx01.varunagroup.de;
S:DeliveryPriority=Normal;
S:AccountForest=EURPRxxAxxx.PROD.OUTLOOK.COM

The information in line 3 shows the actual name of the configured Varunagroup inbound connector, as shown in the Exchange Online connector configuration. The message did not enter the Varunagroup EXO tenant due to a mysterious connection, it was received by the dedicated inbound connector, configured by HCW.

 

  1. Why was the Hybrid Inbound Connector chosen?

The key to this question is the TLS certificate used by the centralized email gateway and the TLS common name filtering in Exchange Online.

  • The email gateways use the following TLS certificate with the two following common names
    • mx01.varunagroup,de
    • mx02.varunagroup.de
  • The hybrid inbound connector used the TLS common name filtering, controlled by the TlsSenderCertificateName attribute, with the following name
    • *.varunagroup.de

The wildcard name *.varunagroup.de resulted in a matching string comparison for the incoming TLS common names of mx01.varunagroup.de and mx02.varunagroup.de. At the same time, the inbound connector matched the Edge Transport TLS certificate smtpo365.varunagroup.de.

Nobody knew, how the inbound connector configuration got "changed" to the wildcard name or for how long that configuration resulted in outbound messages from customer domains routed via the service provider tenant.

 

Solution

The solution contains two configurations.

  1. Ensuring that the FQDN attribute of the Edge Send Connector is set to smtpo365.varunagroup.de

    This ensures that Exchange Server Transport selects the installed and SMTP-enabled TLS certificate for that name.  
     
  2. Changing the TlsSenderCertificateName to smtpo365.veruangroup.de 

    This ensures that Exchange Online selects the hybrid inbound connector for Edge Transport established connections only.
     

The TLS common name behavior is by design and described in this blog post as FAQ #6(b). As a customer, you identify this as a misbehaving SMTP receive connector. But as described in the blog post, this is by design.

It is required that you understand the inbound routing behavior of Exchange Online if you have complicated outbound routing requirements. The blog post provides detailed information on how Office 365 inbound routing works and what you should be aware of.
 

The simple rule is: 
Always use dedicated TLS certificates for separating mail flow to Exchange Online. Especially when using centralized mail flow for your Microsoft 365 tenant.

 

Links

 

Enjoy Exchange Online.

 

Are you located in Germany, Austria, or Switzerland? Join the Exchange User Group DACH to collaborate with other Exchange enthusiasts.
Follow us on Twitter @exusg, join on Meetup, or visit our website

 

Read More »