I am honored to speak at TeamsFest 2020.
TeamsFest 2020 is a TeamsFest is a 100% free, 100% community-driven conference dedicated to Microsoft Teams. It aims to bring together exceptional technical talent and thought leaders to democratize knowledge about Microsoft Teams, encourage participation in the Microsoft Teams community, and give those who are struggling financially an opportunity to attend a first-class Teams conference.
My session will cover the requirements for implementing an Exchange Hybrid configuration with Exchange Online when utilizing on-premises mailboxes with Microsoft Teams.
See you at TeamsFest 2020!
Exchange Server 2016 introduced the PowerShell cmdlet Get-MailboxServerRedundancy. This cmdlet helps you plan and prepare for Exchange Server maintenance by querying the current maintenance readiness of the database availability group (DAG).
Interestingly, there is no PowerShell help available for this vital cmdlet. Microsoft Docs or Exchange Management Shell's Get-Help provide any valuable information.
When querying a DAG about the server redundancy status, the cmdlet's default output provides you with the essential information.
The default output contains information about:
This example shows the Get-MailboxServerRedundancy output of a six server DAG, before activating maintenance mode for server LOCEXS06.
Get-MailboxServerRedundancy -DatabaseAvailabilityGroup EXDAG01 Identity IsServerFound IsInMainten RepairUrgency SafeForMaintenance HealthInfoLastUpdateTime InAD ance -------- ------------- ----------- ------------- ------------------ ------------------------ LOCEXS01 True False Prohibited False 17.02.2020 09:10:11 LOCEXS02 True False Normal True 17.02.2020 09:10:11 LOCEXS03 True False Normal True 17.02.2020 09:10:11 LOCEXS06 True False Normal True 17.02.2020 09:10:11 LOCEXS05 True False Normal True 17.02.2020 09:10:11 LOCEXS04 True False Prohibited False 17.02.2020 09:10:11
As Exchange Administrator, you are most interested in the information displayed in columns RepairUrgency and SafeForMaintenance.
As you can see in this screenshot, no server is in maintenance mode. Servers S01 and S04 have a RepairUrgency state of Prohibited, and a SafeForMaintenance state of False. The latter tells us that we cannot activate maintenance mode for servers safely without risking mailbox database redundancy.
What is the reason for this? Let's have a look.
You can use the same cmdlet to query detailed information for each member server of the DAG. The default output for a single server does not provide any additional information on the server status.
Get-MailboxServerRedundancy -DatabaseAvailabilityGroup EXDAG01 -Identity LOCEXS01 Identity IsServerFound IsInMainten RepairUrgency SafeForMaintenance HealthInfoLastUpdateTime InAD ance -------- ------------- ----------- ------------- ------------------ ------------------------ LOCEXS01 True False Prohibited False 17.02.2020 09:11:11
Because we cannot activate maintenance mode for server LOCEXS01 safely, we are interested in identifying which redundancy state is responsible.
You can find this information by displaying the detailed server information.
Use the Format-List, or short FL, cmdlet to display the Get-MailboxServerRedundancy cmdlet output as a formatted list.
Get-MailboxServerRedundancy -DatabaseAvailabilityGroup EXDAG01 -Identity LOCEXS01 | FL RunspaceId : 70d82f8d-e6ca-4bfc-863f-11300a9784ff Identity : LOCEXS01 IsServerFoundInAD : True IsInMaintenance : False RepairUrgency : Prohibited SafeForMaintenance : False ServerContactedFqdn : LOCEXS04.VARUNAGROUP.DE HealthInfoCreateTime : 15.06.2018 15:16:19 HealthInfoLastUpdateTime : 17.02.2020 09:11:11 ServerFoundInAD : CurrentState: Active; LastActiveTransition: 15.06.2018 15:22:16; LastInactiveTransition: InMaintenance : CurrentState: Inactive; LastActiveTransition: 17.01.2020 09:07:02; LastInactiveTransition: 17.01.2020 10:42:02 AutoActivationPolicyBlocked : CurrentState: Inactive; LastActiveTransition: 09.01.2020 10:14:50; LastInactiveTransition: 09.01.2020 11:00:51 ActivationDisabledAndMoveNow : CurrentState: Inactive; LastActiveTransition: ; LastInactiveTransition: 15.06.2018 15:22:16 HighAvailabilityComponentStateOffline : CurrentState: Inactive; LastActiveTransition: 17.01.2020 09:07:02; LastInactiveTransition: 17.01.2020 10:42:02 CriticalForMaintainingAvailability : CurrentState: Inactive; LastActiveTransition: 31.01.2020 16:52:49; LastInactiveTransition: 31.01.2020 16:56:49 CriticalForMaintainingRedundancy : CurrentState: Active; LastActiveTransition: 29.01.2020 11:43:06; LastInactiveTransition: 29.01.2020 11:42:06 PotentiallyCriticalForMaintainingRedundancy : CurrentState: Active; LastActiveTransition: 01.02.2020 05:49:37; LastInactiveTransition: CriticalForRestoringAvailability : CurrentState: Inactive; LastActiveTransition: 06.05.2019 09:16:36; LastInactiveTransition: 06.05.2019 09:20:36 CriticalForRestoringRedundancy : CurrentState: Inactive; LastActiveTransition: 29.01.2020 11:42:06; LastInactiveTransition: 29.01.2020 11:43:06 HighForRestoringAvailability : CurrentState: Inactive; LastActiveTransition: 29.01.2020 11:42:06; LastInactiveTransition: 29.01.2020 11:43:06 HighForRestoringRedundancy : CurrentState: Inactive; LastActiveTransition: 10.02.2020 09:05:02; LastInactiveTransition: 10.02.2020 09:06:02 IsSafeForMaintenance : CurrentState: Inactive; LastActiveTransition: 03.11.2019 09:42:35; LastInactiveTransition: 12.11.2019 06:29:58 IsValid : True ObjectState : Unchanged
The lines 24-27 show the information we want to know. Both, the CriticalForMaintainingRedundancy and PotentiallyCriticalForMaintainingRedundancy parameters have a CurrentState value of Active. The Primary Activation Manager (PAM) considers the server availability critical to provide redundant availability of the database copies hosted by this server.
Each of state-parameter shows three pieces of information:
But there is still the bothering question of why are two of the six servers not safe for activating maintenance?
The reason is simple. The mailbox databases mounted by the member servers of the DAG have a different number of database copies. This configuration is due to data storage capacity constraints.
The mailbox databases storing primary user mailboxes use four database copies per database. Those copies are evenly distributed across all six mailbox servers. Mailbox database storing online archive mailboxes use three copies per database. This database copy layout allows for safely activating server maintenance for one server at a time without risk to database redundancy.
The servers LOCEXS01 and LOCEXS04 hold mailbox databases with just two copies per configured database. Placing one of those two servers into maintenance mode reduces the database availability for these mailbox databases to one. Therefore, PAM informs us that database redundancy is at risk when activating maintenance for those two servers.
This example shows the member server redundancy state while LOCEXS06 is in maintenance. The reason for monthly maintenance for installing Windows updates.
Maintenance was activated using the StartDagServerMaintenance.ps1 PowerShell script.
Get-MailboxServerRedundancy -DatabaseAvailabilityGroup indag01 Identity IsServerFound IsInMainten RepairUrgency SafeForMaintenance HealthInfoLastUpdateTime InAD ance -------- ------------- ----------- ------------- ------------------ ------------------------ LOCEXS01 True False Prohibited False 17.02.2020 11:04:12 LOCEXS02 True False Normal True 17.02.2020 11:04:12 LOCEXS03 True False Prohibited False 17.02.2020 11:04:12 LOCEXS06 True True High True 17.02.2020 11:04:12 LOCEXS05 True False Prohibited False 17.02.2020 11:04:12 LOCEXS04 True False Prohibited False 17.02.2020 11:04:12
Having a single server in maintenance has a significant impact on all other servers in the DAG. The servers LOCEXS03 and LOCEXS05 are not safe for maintenance as well. Activating maintenance for those two servers would affect the database redundancy for the databases hosted by those two servers.
After completing all maintenance tasks, e.g., installing Windows Updates or a new Exchange Server Cumulative Update, you end server maintenance using the PowerShell script StopDagServerMaintenance.ps1.
We query the server redundancy state again.
Get-MailboxServerRedundancy -DatabaseAvailabilityGroup indag01 Identity IsServerFound IsInMainten RepairUrgency SafeForMaintenance HealthInfoLastUpdateTime InAD ance -------- ------------- ----------- ------------- ------------------ ------------------------ LOCEXS01 True False Prohibited False 17.02.2020 11:23:12 LOCEXS02 True False Normal True 17.02.2020 11:23:12 LOCEXS03 True False Normal True 17.02.2020 11:23:12 LOCEXS06 True False High True 17.02.2020 11:23:12 LOCEXS05 True False Normal True 17.02.2020 11:23:12 LOCEXS04 True False Prohibited False 17.02.2020 11:23:12
Server LOCEXS06 is not in maintenance, but the RepairUrgency state is High. The local Exchange Server replication engine is still busy replicating and processing log files, and updating the search indices. When CopyQueueLength and ReplayQueueLength are back to 0, and ContentIndexStates are back to Healthy, the RepairUrgency switches to Normal.
You receive an error message when activating maintenance for an Exchange Server not safe for maintenance using StartDagServerMaintenance.ps1 -serverName [SERVER] In this case, you must use:
.\StartDagServerMaintenance.ps1 -serverName SERVERNAME -overrideMinimumTwoCopies:$true
Enjoy Exchange Server!
Once upon a time at an Exchange Conference near you, a member of the Exchange Product Group (PG) announced that the very last Exchange Server will go away when having an active Exchange hybrid setup.
This was a hot topic for discussions at the Microsoft Exchange Conferences (MEC, @IamMEC) in 2012 and 2014, already. Since then the Exchange PG came up with a number of reasons why this is not possible. The question on when we will finally be able to remove the very last Exchange Server from the on-premises Exchange organization was asked every year at the Ignite Conference.
Currently, the supported scenario for hybrid configurations between your on-premises Exchange organization and Exchange Online requires that you keep the last Exchange Server for creating, and managing Exchange related objects, even if those objects are located in Exchange Online.
The following diagram illustrates the current requirements:
In the past, there was communication on certain interim solutions that were supposed to support you in removing the last Exchange Server from your Exchange organization. Such interim solutions were:
At Ignite those solutions even made it into the official session catalog:
All those interim solutions leave your on-premises Exchange organization and the Active Directory configuration in an uncomfortable twilight-zone. It was still something that worked somehow, but you knew it was officially not supported, and the secure and stable operation of the hybrid configuration was at risk.
But wait...
Removing the last Exchange Server is supported! (at least when all components are released)
The new approach for managing your Exchange Online tenancy after migrating your on-premises Exchange organization to Exchange Online does not require an on-premises Exchange Server.
The new mode of operation reduces your on-premises requirements to:
The following diagram illustrates the new modern Exchange Online Management experience:
Simply you remove the requirement to use on-premises Exchange Server to write to your on-premises Active Directory. Instead, Azure AD Connect uses a new synchronization capability to handle the new Exchange Management experience in the AAD Connect MetaVerse. The on-premises AD-connector writes the changes to Active Directory which keeps the Active Directory up-to-date for all other on-premises solutions that require identities to have a proper state.
You execute all Exchange-related actions using the new Exchange Online Management PowerShell module, or, if needed, the new Modern Exchange Admin Center (EAC, which was announced at Ignite 2019.
Before you uninstall the last Exchange Server from your on-premises Exchange organization, ensure that you
PS C:\> Get-WindowsFeature Display Name Name Install State ------------ ---- ------------- [ ] Active Directory Certificate Services AD-Certificate Available [ ] Certification Authority ADCS-Cert-Authority Available [ ] Certificate Enrollment Policy Web Service ADCS-Enroll-Web-Pol Available [ ] Certificate Enrollment Web Service ADCS-Enroll-Web-Svc Available [ ] Certification Authority Web Enrollment ADCS-Web-Enrollment Available [ ] Network Device Enrollment Service ADCS-Device-Enrollment Available [ ] Online Responder ADCS-Online-Cert Available [ ] Active Directory Domain Services AD-Domain-Services Available [ ] Active Directory Federation Services ADFS-Federation Available [ ] Active Directory Lightweight Directory Services ADLDS Available [ ] Active Directory Rights Management Services ADRMS Available [ ] Active Directory Rights Management Server ADRMS-Server Available [ ] Identity Federation Support ADRMS-Identity Available [ ] Device Health Attestation DeviceHealthAttestat... Available [ ] DHCP Server DHCP Available [ ] DNS Server DNS Available [ ] Exchange Online Remote Features EXORemote Available [ ] Fax Server Fax Available [X] File and Storage Services FileAndStorage-Services Installed [X] File and iSCSI Services File-Services Installed [X] File Server FS-FileServer Installed [ ] BranchCache for Network Files FS-BranchCache Available [...]
PS C:\> Install-WindowsFeature -Name EXORemote Display Name Name Install State ------------ ---- ------------- [ ] Active Directory Certificate Services AD-Certificate Available [ ] Certification Authority ADCS-Cert-Authority Available [ ] Certificate Enrollment Policy Web Service ADCS-Enroll-Web-Pol Available [ ] Certificate Enrollment Web Service ADCS-Enroll-Web-Svc Available [ ] Certification Authority Web Enrollment ADCS-Web-Enrollment Available [ ] Network Device Enrollment Service ADCS-Device-Enrollment Available [ ] Online Responder ADCS-Online-Cert Available [ ] Active Directory Domain Services AD-Domain-Services Available [ ] Active Directory Federation Services ADFS-Federation Available [ ] Active Directory Lightweight Directory Services ADLDS Available [ ] Active Directory Rights Management Services ADRMS Available [ ] Active Directory Rights Management Server ADRMS-Server Available [ ] Identity Federation Support ADRMS-Identity Available [ ] Device Health Attestation DeviceHealthAttestat... Available [ ] DHCP Server DHCP Available [ ] DNS Server DNS Available [X] Exchange Online Remote Features EXORemote Installed [ ] Fax Server Fax Available [X] File and Storage Services FileAndStorage-Services Installed [X] File and iSCSI Services File-Services Installed [X] File Server FS-FileServer Installed [ ] BranchCache for Network Files FS-BranchCache Available [...]
Even though not explicitly stated, you should restart the server after installing the Windows feature.
As part of the next AAD Connect synchronization cycle, the magic happens.
Verify that you can edit the Exchange related attributes of synchronized Active Directory objects in Exchange Online or Azure AD before you remove your last Exchange Server.
Whey ready to uninstall the last Exchange Server you must use the following command line parameters to remove the server as intended. Otherwise, you'll leave the Exchange organization in an inchoate state. Ensure that you use an administrative PowerShell session.
./Setup.exe /mode:uninstall /SwitchToMEMA /IAcceptExchangeOnlineLicenseTerms
Normally, you do not have to accept license terms when uninstalling Exchange Server, but in this case, you have to accept the Exchange Online license terms.
Enjoy the modern experience and management options of Exchange Online!
Exchange Conferences
The Microsoft 365 Virtual Marathon took place on May 27-28 2020.
The recording of my session "Exchange Hybrid - What, Why, and How" is available on YouTube.
Browse all recordings of the Microsoft 365 Virtual Marathon here: https://www.youtube.com/channel/UCrtmT6Ir1MIs0ZES7sKMmqA
Enjoy!
The Microsoft 365 Virtual Marathon is happening on May 27-28 2020.
This is a free online event, providing you with 36 hours of non-stop sessions from speakers around the globe. You can join every time.
The virtual conference is a joint effort with SPC.
I speak at Microsoft 365 Virtual Marathon about:
The marathon session plan is available here.
Register now.
The use of Exchange Edge Transport Servers requires the synchronization of user and configuration data from internal Exchange Servers to the Edge Transport Servers. The synchronization utilizes secure LDAP (EdgeSync) to transmit the data securely and is based on an Edge Subscription.
When you create a new Edge Subscription on your internal Exchange Servers by importing the Edge Subscription XML-file, establishing the EdgeSync-connection might fail.
You will find the following error in the application event log of the internal Exchange Server:
Log Name: Application Source: MSExchange EdgeSync Event ID: 1035 Task Category: Synchronization Level: Error Keywords: Classic Description: EdgeSync failed to synchronize because it only supports Cryptographic API certificates. The local Hub Transport server's default certificate with thumbprint XYZ isn't a CAPI certificate. To set a CAPI certificate as the default certificate, use the Enable-ExchangeCertificate cmdlet with the Services parameter using the value of SMTP.
The private key of the current Exchange Transport default certificate of the internal Exchange servers uses a CNG private key. EdgeSync requires a CAPI1 based private key.
This problem occurs primarily when using an Enterprise Certificate Authority using certificate templates with individual template settings.
Get-TransportService | ft Name,InternalTransportCertificateThumbprint
certutil -v -store my > cert.txt
If both attribute are of the value 0, the certificate if a CNG certificate.
The section might look like this:
Unique container name: XYZ Provider = Microsoft Software Key Storage Provider ProviderType = 0 Flags = 20 (32) CRYPT_MACHINE_KEYSET -- 20 (32) KeySpec = 0 -- XCN_AT_NONE
Use OpenSSL to convert the CNG certificate to a CAPI1 certificate.
Using OpenSSL requires the download of the Windows release of OpenSSL. I recommend to not install the software on the Exchange Server but a separate Windows server or your administrative desktop system. Additionally, you need the certificate with its private key as a PFX-file.
Use the following steps to convert the CNG certificate to a CAPI1 certificate.
openssl pkcs12 -in CERT.pfx -out cert.pem -nodes
openssl pkcs12 -export -in cert.pem -out NEWCERT.pfx
The new PFX-file is now a CAPI1 certificate. The new certificate has the same thumbprint. Now you must replace the current certificate used by Exchange Server with the new certificate.
Replacing the certificate requires a downtime of each Exchange Server requiring the certificate replacement. This is due to the requirement to remove the CNG certificate first, following the import of the CAPI1 certificate. Afterward, you need to enable the required Exchange services.
Get-ExchangeCertificate -Server SERVERNAME
# It is mandatory to answer the query for replacing the default certificate with YES Enable-ExchangeCertificate -Thumbprint THUMBPRINT -Services SMTP # Restart the transport service Restart-Service MSExchangeTransport
# It is mandatory to answer the query for replacing the default certificate with YES Enable-ExchangeCertificate -Thumbprint NEWCERTTHUMBPRINT -Services SMTP # Restart the transport service Restart-Service MSExchangeTransport
Now, that you updated the local Exchange Servers there is one more step that needs to be checked on the Edge Transport Servers.
Edge Transport Servers are not domain-joined and therefore do not receive any GPO-based configuration. Each required configuration must be performed locally. To ensure that the default transport certificate of the internal Exchange servers can be used for cryptographic operations we must ensure that the certificate chain of that certificate is present in the certificate store of Edge Transport servers.
Take a look at the certificate chain of the converted CAPI1 certificate and import the Root-CA and Subordinate-CA certificates into the Edge Transport servers local certificate store. You must ensure that the certificates are placed into appropriate stores:
Next, you create a new Edge Subscription on your Edge Transport server and create a new subscription for the Active Directory site on the internal Exchange Server. The internal Exchange Servers are now able to establish an EdgeSync connection and encrypting the data transferred to the Edge Transport servers.
When you receive the TLS certificate as PFX/PKCS12 file, you import the certificate and the private key. The import process itself defines the priavte key Crypto Provider. Using the following command line you ensure that the import process suses the legacy crypto provider.
certutil -csp "Microsoft RSA SChannel Cryptographic Provider" -importpfx my MYCERTpfx
Enjoy Exchange Server and Edge Transport!
Exchange Server uses Receive Connectors for providing SMTP endpoints for incoming connections. A modern Exchange Server provides a default connector on TCP port 25.
Sometimes you might have a requirement to create a new receive connector for selected incoming SMTP connections. A standard requirement is a receive connector for relaying messages to external recipients. This cannot (should not) be achieved using the default connector.
Each connector uses the RemoteIPRanges attribute to store the list of IP addresses of remote systems that can connect to that connector. The default connector utilizes the full IPv4 and IPv6 addresses ranges.
Your new receive connector requires at least a single IP address for a selected remote system that is supposed to connect to that receive connector. You can add a single IP address, address ranges, or IP addresses using CIDR notation.
The attribute RemoteIPRanges is a multi-value attribute and has a limit of IP address entries that can be added.
The maximum number of address entries that you can add to that attribute varies. You can store approximately 1,300 entries.
When you exceed the number of values you receive the following error message:
The administrative limit for this request was exceeded. + CategoryInfo : NotSpecified: (:) [Set-ReceiveConnector], AdminLimitExceededException + FullyQualifiedErrorId : [Server=EX01,RequestId=ee9d45ad-418b-4172-9235-963eca1a7830,TimeStamp=18.08.2020 20:07:54] [FailureCategory=Cmdlet-AdminLimitExceededException] AC1E336E,Microsoft.Exchange.Management.SystemConfi gurationTasks.SetReceiveConnector + PSComputerName : ex01.varunagroup.de
I have tested the number of values that can be stored in that multi-value attribute. Depending on the IP address format I was able to add 1,238 (172.80.x.y) or 1,244 (10.1.x.y) single IP addresses to the RemoteIPRanges attribute.
Plan your IP address configuration requirements carefully and avoid using single IP addresses. Preferably, you should use IP address ranges or IP address CIDR notation for networks.