I am honored to speak at TeamsFest 2020.
TeamsFest 2020 is a TeamsFest is a 100% free, 100% community-driven conference dedicated to Microsoft Teams. It aims to bring together exceptional technical talent and thought leaders to democratize knowledge about Microsoft Teams, encourage participation in the Microsoft Teams community, and give those who are struggling financially an opportunity to attend a first-class Teams conference.
My session will cover the requirements for implementing an Exchange Hybrid configuration with Exchange Online when utilizing on-premises mailboxes with Microsoft Teams.
See you at TeamsFest 2020!
Exchange Server 2016 introduced the PowerShell cmdlet Get-MailboxServerRedundancy. This cmdlet helps you plan and prepare for Exchange Server maintenance by querying the current maintenance readiness of the database availability group (DAG).
Interestingly, there is no PowerShell help available for this vital cmdlet. Microsoft Docs or Exchange Management Shell's Get-Help provide any valuable information.
When querying a DAG about the server redundancy status, the cmdlet's default output provides you with the essential information.
The default output contains information about:
This example shows the Get-MailboxServerRedundancy output of a six server DAG, before activating maintenance mode for server LOCEXS06.
Get-MailboxServerRedundancy -DatabaseAvailabilityGroup EXDAG01 Identity IsServerFound IsInMainten RepairUrgency SafeForMaintenance HealthInfoLastUpdateTime InAD ance -------- ------------- ----------- ------------- ------------------ ------------------------ LOCEXS01 True False Prohibited False 17.02.2020 09:10:11 LOCEXS02 True False Normal True 17.02.2020 09:10:11 LOCEXS03 True False Normal True 17.02.2020 09:10:11 LOCEXS06 True False Normal True 17.02.2020 09:10:11 LOCEXS05 True False Normal True 17.02.2020 09:10:11 LOCEXS04 True False Prohibited False 17.02.2020 09:10:11
As Exchange Administrator, you are most interested in the information displayed in columns RepairUrgency and SafeForMaintenance.
As you can see in this screenshot, no server is in maintenance mode. Servers S01 and S04 have a RepairUrgency state of Prohibited, and a SafeForMaintenance state of False. The latter tells us that we cannot activate maintenance mode for servers safely without risking mailbox database redundancy.
What is the reason for this? Let's have a look.
You can use the same cmdlet to query detailed information for each member server of the DAG. The default output for a single server does not provide any additional information on the server status.
Get-MailboxServerRedundancy -DatabaseAvailabilityGroup EXDAG01 -Identity LOCEXS01 Identity IsServerFound IsInMainten RepairUrgency SafeForMaintenance HealthInfoLastUpdateTime InAD ance -------- ------------- ----------- ------------- ------------------ ------------------------ LOCEXS01 True False Prohibited False 17.02.2020 09:11:11
Because we cannot activate maintenance mode for server LOCEXS01 safely, we are interested in identifying which redundancy state is responsible.
You can find this information by displaying the detailed server information.
Use the Format-List, or short FL, cmdlet to display the Get-MailboxServerRedundancy cmdlet output as a formatted list.
Get-MailboxServerRedundancy -DatabaseAvailabilityGroup EXDAG01 -Identity LOCEXS01 | FL RunspaceId : 70d82f8d-e6ca-4bfc-863f-11300a9784ff Identity : LOCEXS01 IsServerFoundInAD : True IsInMaintenance : False RepairUrgency : Prohibited SafeForMaintenance : False ServerContactedFqdn : LOCEXS04.VARUNAGROUP.DE HealthInfoCreateTime : 15.06.2018 15:16:19 HealthInfoLastUpdateTime : 17.02.2020 09:11:11 ServerFoundInAD : CurrentState: Active; LastActiveTransition: 15.06.2018 15:22:16; LastInactiveTransition: InMaintenance : CurrentState: Inactive; LastActiveTransition: 17.01.2020 09:07:02; LastInactiveTransition: 17.01.2020 10:42:02 AutoActivationPolicyBlocked : CurrentState: Inactive; LastActiveTransition: 09.01.2020 10:14:50; LastInactiveTransition: 09.01.2020 11:00:51 ActivationDisabledAndMoveNow : CurrentState: Inactive; LastActiveTransition: ; LastInactiveTransition: 15.06.2018 15:22:16 HighAvailabilityComponentStateOffline : CurrentState: Inactive; LastActiveTransition: 17.01.2020 09:07:02; LastInactiveTransition: 17.01.2020 10:42:02 CriticalForMaintainingAvailability : CurrentState: Inactive; LastActiveTransition: 31.01.2020 16:52:49; LastInactiveTransition: 31.01.2020 16:56:49 CriticalForMaintainingRedundancy : CurrentState: Active; LastActiveTransition: 29.01.2020 11:43:06; LastInactiveTransition: 29.01.2020 11:42:06 PotentiallyCriticalForMaintainingRedundancy : CurrentState: Active; LastActiveTransition: 01.02.2020 05:49:37; LastInactiveTransition: CriticalForRestoringAvailability : CurrentState: Inactive; LastActiveTransition: 06.05.2019 09:16:36; LastInactiveTransition: 06.05.2019 09:20:36 CriticalForRestoringRedundancy : CurrentState: Inactive; LastActiveTransition: 29.01.2020 11:42:06; LastInactiveTransition: 29.01.2020 11:43:06 HighForRestoringAvailability : CurrentState: Inactive; LastActiveTransition: 29.01.2020 11:42:06; LastInactiveTransition: 29.01.2020 11:43:06 HighForRestoringRedundancy : CurrentState: Inactive; LastActiveTransition: 10.02.2020 09:05:02; LastInactiveTransition: 10.02.2020 09:06:02 IsSafeForMaintenance : CurrentState: Inactive; LastActiveTransition: 03.11.2019 09:42:35; LastInactiveTransition: 12.11.2019 06:29:58 IsValid : True ObjectState : Unchanged
The lines 24-27 show the information we want to know. Both, the CriticalForMaintainingRedundancy and PotentiallyCriticalForMaintainingRedundancy parameters have a CurrentState value of Active. The Primary Activation Manager (PAM) considers the server availability critical to provide redundant availability of the database copies hosted by this server.
Each of state-parameter shows three pieces of information:
But there is still the bothering question of why are two of the six servers not safe for activating maintenance?
The reason is simple. The mailbox databases mounted by the member servers of the DAG have a different number of database copies. This configuration is due to data storage capacity constraints.
The mailbox databases storing primary user mailboxes use four database copies per database. Those copies are evenly distributed across all six mailbox servers. Mailbox database storing online archive mailboxes use three copies per database. This database copy layout allows for safely activating server maintenance for one server at a time without risk to database redundancy.
The servers LOCEXS01 and LOCEXS04 hold mailbox databases with just two copies per configured database. Placing one of those two servers into maintenance mode reduces the database availability for these mailbox databases to one. Therefore, PAM informs us that database redundancy is at risk when activating maintenance for those two servers.
This example shows the member server redundancy state while LOCEXS06 is in maintenance. The reason for monthly maintenance for installing Windows updates.
Maintenance was activated using the StartDagServerMaintenance.ps1 PowerShell script.
Get-MailboxServerRedundancy -DatabaseAvailabilityGroup indag01 Identity IsServerFound IsInMainten RepairUrgency SafeForMaintenance HealthInfoLastUpdateTime InAD ance -------- ------------- ----------- ------------- ------------------ ------------------------ LOCEXS01 True False Prohibited False 17.02.2020 11:04:12 LOCEXS02 True False Normal True 17.02.2020 11:04:12 LOCEXS03 True False Prohibited False 17.02.2020 11:04:12 LOCEXS06 True True High True 17.02.2020 11:04:12 LOCEXS05 True False Prohibited False 17.02.2020 11:04:12 LOCEXS04 True False Prohibited False 17.02.2020 11:04:12
Having a single server in maintenance has a significant impact on all other servers in the DAG. The servers LOCEXS03 and LOCEXS05 are not safe for maintenance as well. Activating maintenance for those two servers would affect the database redundancy for the databases hosted by those two servers.
After completing all maintenance tasks, e.g., installing Windows Updates or a new Exchange Server Cumulative Update, you end server maintenance using the PowerShell script StopDagServerMaintenance.ps1.
We query the server redundancy state again.
Get-MailboxServerRedundancy -DatabaseAvailabilityGroup indag01 Identity IsServerFound IsInMainten RepairUrgency SafeForMaintenance HealthInfoLastUpdateTime InAD ance -------- ------------- ----------- ------------- ------------------ ------------------------ LOCEXS01 True False Prohibited False 17.02.2020 11:23:12 LOCEXS02 True False Normal True 17.02.2020 11:23:12 LOCEXS03 True False Normal True 17.02.2020 11:23:12 LOCEXS06 True False High True 17.02.2020 11:23:12 LOCEXS05 True False Normal True 17.02.2020 11:23:12 LOCEXS04 True False Prohibited False 17.02.2020 11:23:12
Server LOCEXS06 is not in maintenance, but the RepairUrgency state is High. The local Exchange Server replication engine is still busy replicating and processing log files, and updating the search indices. When CopyQueueLength and ReplayQueueLength are back to 0, and ContentIndexStates are back to Healthy, the RepairUrgency switches to Normal.
You receive an error message when activating maintenance for an Exchange Server not safe for maintenance using StartDagServerMaintenance.ps1 -serverName [SERVER] In this case, you must use:
.\StartDagServerMaintenance.ps1 -serverName SERVERNAME -overrideMinimumTwoCopies:$true
Enjoy Exchange Server!
The Microsoft 365 Virtual Marathon took place on May 27-28 2020.
The recording of my session "Exchange Hybrid - What, Why, and How" is available on YouTube.
Browse all recordings of the Microsoft 365 Virtual Marathon here: https://www.youtube.com/channel/UCrtmT6Ir1MIs0ZES7sKMmqA
Enjoy!
The Microsoft 365 Virtual Marathon is happening on May 27-28 2020.
This is a free online event, providing you with 36 hours of non-stop sessions from speakers around the globe. You can join every time.
The virtual conference is a joint effort with SPC.
I speak at Microsoft 365 Virtual Marathon about:
The marathon session plan is available here.
Register now.
Recently I had to support the uninstall of Exchange Server 2016 CU10 on a Windows Server 2019 system. That this setup is not supported is a different topic. In this case, a new Exchange Server 2016 system was placed in service, and the old system needed to be removed from the on-premises Exchange organization.
We mounted the Exchange 2016 CU10 ISO, and ran the following command from an administrative command line:
Setup.exe /mode:uninstall
The prerequisites check failed with an odd error:
http://terenceluk.blogspot.com/2017/01/attempting-to-delete-exchange-server.html
Querying for any incompleted public folder migration requests returned no results. But the prerequisites check insisted that there was an existing public folder migration request. In such a case you already know that you have to use ADSIEdit to find the object in question.
It turned out that the prerequisites check was right, as we found a single public folder migration request in the Active Directory configuration partition. The request was an artifact of an unsuccessful migration attempt in 2019. After we have checked that the current modern public folder hierarchy worked as expected, we deleted the artifact from Active Directory.
Now the uninstall procedure passed the prerequisites check successfully and the uninstaller moved on removed Exchange Server 2016 step by step.
Until...
The uninstall step Language Files an Access Denied exception while executing MSIEXEC uninstall actions for each Language Pack.
Language Files FAILED
The following error was generated when "$error.Clear(); $regPath='HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall'; $PackageGUIDRegEx = "{DEDFFB[0-9a-fA-F]{2}-42EC-4E26-[0-9a-fA-F]{4}-430E86DF378C}"; $InstallPath = (Get-ItemProperty 'HKLM:\SOFTWARE\Microsoft\ExchangeServer\v15\setup').MsiInstallPath;
if(test-path ($regPath)) { Write-ExchangeSetupLog -info ("Removing " + $RoleLanguagePackType + " Language Packs."); Get-ChildItem ($regPath) | foreach{ if($_ -match "(?<ProductCode>$PackageGUIDRegEx)") { $langPackPackageCode = $matches['ProductCode']; if($langPackPackageCode -ne $null -and $langPackPackageCode.Length -ne 0) { Write-ExchangeSetupLog -info ("Removing package $langPackPackageCode"); $language = $langPackPackageCode.Substring(20,4); $logFilePath = [IO.Path]::Combine($RoleLogFilePath,"Uninstall") + '.' + $language + '.' + "Client" + "." + $RoleLogDateTime + ".msilog"; uninstall-MsiPackage -ProductCode ($langPackPackageCode) -LogFile ($logFilePath); }; }; }; Get-Childitem -Path $InstallPath -include ".Localized.js",".Localized.min.js" -recurse | foreach ($) {remove-item $.fullname}; Write-ExchangeSetupLog -info "Remove Language Packs completed."; }; " was run: "**System.UnauthorizedAccessException: Access is denied** ---> System.ComponentModel.Win32Exception: Access is denied --- End of inner exception stack trace --- at System.Management.Automation.Utils.NativeDirectoryExists(String path) at System.Management.Automation.SessionStateInternal.IsItemContainer(CmdletProvider providerInstance, String path, CmdletProviderContext context)".
Interestingly, the ExchangeSetup log file showed that the uninstaller wrote the informational text Remove Language Packs completed successfully.
After following an idea to remove the language pack-related registry keys and other fancy approaches, we did something trivial. We restarted the server, mounted the ISO file, and ran Setup.exe /mode:uninstall again.
The uninstaller process now passed the step Language Files without any issues.
I sometimes like simple solutions.
Enjoy Exchange Server.
I am honored to be a speaker at the Virtual Scottish Summit 2021 conference, taking place on Saturday, 27. February.
The Scottish Summit is a truly European event. You can choose from 365 sessions covering mostly any about Microsoft 365 workloads in seven languages:
Attend my session when you are interested in the challenges of implementing Exchange Server Hybrid, and the requirements to make it work with Microsoft Teams and on-premises mailboxes.
My session at Scottish Summit 2021:
See you online for the Virtual Scottish Summit.
The use of Exchange Edge Transport Servers requires the synchronization of user and configuration data from internal Exchange Servers to the Edge Transport Servers. The synchronization utilizes secure LDAP (EdgeSync) to transmit the data securely and is based on an Edge Subscription.
When you create a new Edge Subscription on your internal Exchange Servers by importing the Edge Subscription XML-file, establishing the EdgeSync-connection might fail.
You will find the following error in the application event log of the internal Exchange Server:
Log Name: Application Source: MSExchange EdgeSync Event ID: 1035 Task Category: Synchronization Level: Error Keywords: Classic Description: EdgeSync failed to synchronize because it only supports Cryptographic API certificates. The local Hub Transport server's default certificate with thumbprint XYZ isn't a CAPI certificate. To set a CAPI certificate as the default certificate, use the Enable-ExchangeCertificate cmdlet with the Services parameter using the value of SMTP.
The private key of the current Exchange Transport default certificate of the internal Exchange servers uses a CNG private key. EdgeSync requires a CAPI1 based private key.
This problem occurs primarily when using an Enterprise Certificate Authority using certificate templates with individual template settings.
Get-TransportService | ft Name,InternalTransportCertificateThumbprint
certutil -v -store my > cert.txt
If both attribute are of the value 0, the certificate if a CNG certificate.
The section might look like this:
Unique container name: XYZ Provider = Microsoft Software Key Storage Provider ProviderType = 0 Flags = 20 (32) CRYPT_MACHINE_KEYSET -- 20 (32) KeySpec = 0 -- XCN_AT_NONE
Use OpenSSL to convert the CNG certificate to a CAPI1 certificate.
Using OpenSSL requires the download of the Windows release of OpenSSL. I recommend to not install the software on the Exchange Server but a separate Windows server or your administrative desktop system. Additionally, you need the certificate with its private key as a PFX-file.
Use the following steps to convert the CNG certificate to a CAPI1 certificate.
openssl pkcs12 -in CERT.pfx -out cert.pem -nodes
openssl pkcs12 -export -in cert.pem -out NEWCERT.pfx
The new PFX-file is now a CAPI1 certificate. The new certificate has the same thumbprint. Now you must replace the current certificate used by Exchange Server with the new certificate.
Replacing the certificate requires a downtime of each Exchange Server requiring the certificate replacement. This is due to the requirement to remove the CNG certificate first, following the import of the CAPI1 certificate. Afterward, you need to enable the required Exchange services.
Get-ExchangeCertificate -Server SERVERNAME
# It is mandatory to answer the query for replacing the default certificate with YES Enable-ExchangeCertificate -Thumbprint THUMBPRINT -Services SMTP # Restart the transport service Restart-Service MSExchangeTransport
# It is mandatory to answer the query for replacing the default certificate with YES Enable-ExchangeCertificate -Thumbprint NEWCERTTHUMBPRINT -Services SMTP # Restart the transport service Restart-Service MSExchangeTransport
Now, that you updated the local Exchange Servers there is one more step that needs to be checked on the Edge Transport Servers.
Edge Transport Servers are not domain-joined and therefore do not receive any GPO-based configuration. Each required configuration must be performed locally. To ensure that the default transport certificate of the internal Exchange servers can be used for cryptographic operations we must ensure that the certificate chain of that certificate is present in the certificate store of Edge Transport servers.
Take a look at the certificate chain of the converted CAPI1 certificate and import the Root-CA and Subordinate-CA certificates into the Edge Transport servers local certificate store. You must ensure that the certificates are placed into appropriate stores:
Next, you create a new Edge Subscription on your Edge Transport server and create a new subscription for the Active Directory site on the internal Exchange Server. The internal Exchange Servers are now able to establish an EdgeSync connection and encrypting the data transferred to the Edge Transport servers.
When you receive the TLS certificate as PFX/PKCS12 file, you import the certificate and the private key. The import process itself defines the priavte key Crypto Provider. Using the following command line you ensure that the import process suses the legacy crypto provider.
certutil -csp "Microsoft RSA SChannel Cryptographic Provider" -importpfx my MYCERTpfx
Enjoy Exchange Server and Edge Transport!
Are you located in Germany, Austria, or Switzerland? Join the Exchange User Group DACH to collaborate with other Exchange enthusiasts. Follow us on Twitter @exusg, join on Meetup, or visit our website.