Just can't get enough of IT

Exchange Server 2016 introduced the PowerShell cmdlet Get-MailboxServerRedundancy. This cmdlet helps you plan and prepare for Exchange Server maintenance by querying the current maintenance readiness of the database availability group (DAG). 

Interestingly, there is no PowerShell help available for this vital cmdlet. Microsoft Docs or Exchange Management Shell's Get-Help provide any valuable information.

When querying a DAG about the server redundancy status, the cmdlet's default output provides you with the essential information.

The default output contains information about:

• Identity
  Name of the DAG member server
   
• IsServerFoundInAD
  Indicates if the corresponding server computer object exists Active Directory
   
• IsInMaintenance
  Indicates if the server is currently in maintenance mode
   
• RepairUrgency
  Indicates an aggregated state of the mailbox database and search index repair modes 
   
• SafeForMaintenance
  Indicates if you can safely activate the maintenance mode for this server
   
• HealthInfoLastUpdateTime
  Timestamp when the server's health state was last updated
   

Example - Prior Maintenance

This example shows the Get-MailboxServerRedundancy output of a six server DAG, before activating maintenance mode for server LOCEXS06.

Get-MailboxServerRedundancy -DatabaseAvailabilityGroup EXDAG01

Identity        IsServerFound IsInMainten RepairUrgency SafeForMaintenance HealthInfoLastUpdateTime
                InAD          ance
--------        ------------- ----------- ------------- ------------------ ------------------------
LOCEXS01        True          False       Prohibited    False              17.02.2020 09:10:11
LOCEXS02        True          False       Normal        True               17.02.2020 09:10:11
LOCEXS03        True          False       Normal        True               17.02.2020 09:10:11
LOCEXS06        True          False       Normal        True               17.02.2020 09:10:11
LOCEXS05        True          False       Normal        True               17.02.2020 09:10:11
LOCEXS04        True          False       Prohibited    False              17.02.2020 09:10:11

As Exchange Administrator, you are most interested in the information displayed in columns RepairUrgency and SafeForMaintenance.

As you can see in this screenshot, no server is in maintenance mode. Servers S01 and S04 have a RepairUrgency state of Prohibited, and a SafeForMaintenance state of False. The latter tells us that we cannot activate maintenance mode for servers safely without risking mailbox database redundancy. 

What is the reason for this? Let's have a look.

Server Information

You can use the same cmdlet to query detailed information for each member server of the DAG. The default output for a single server does not provide any additional information on the server status. 

Get-MailboxServerRedundancy -DatabaseAvailabilityGroup EXDAG01 -Identity LOCEXS01

Identity        IsServerFound IsInMainten RepairUrgency SafeForMaintenance HealthInfoLastUpdateTime
                InAD          ance
--------        ------------- ----------- ------------- ------------------ ------------------------
LOCEXS01        True          False       Prohibited    False              17.02.2020 09:11:11

Because we cannot activate maintenance mode for server LOCEXS01 safely, we are interested in identifying which redundancy state is responsible.

You can find this information by displaying the detailed server information.

Detailed Server Information

Use the Format-List, or short FL, cmdlet to display the Get-MailboxServerRedundancy cmdlet output as a formatted list.

Get-MailboxServerRedundancy -DatabaseAvailabilityGroup EXDAG01 -Identity LOCEXS01 | FL

RunspaceId                                  : 70d82f8d-e6ca-4bfc-863f-11300a9784ff
Identity                                    : LOCEXS01
IsServerFoundInAD                           : True
IsInMaintenance                             : False
RepairUrgency                               : Prohibited
SafeForMaintenance                          : False
ServerContactedFqdn                         : LOCEXS04.VARUNAGROUP.DE
HealthInfoCreateTime                        : 15.06.2018 15:16:19
HealthInfoLastUpdateTime                    : 17.02.2020 09:11:11
ServerFoundInAD                             : CurrentState: Active; LastActiveTransition: 15.06.2018 15:22:16;
                                              LastInactiveTransition:
InMaintenance                               : CurrentState: Inactive; LastActiveTransition: 17.01.2020 09:07:02;
                                              LastInactiveTransition: 17.01.2020 10:42:02
AutoActivationPolicyBlocked                 : CurrentState: Inactive; LastActiveTransition: 09.01.2020 10:14:50;
                                              LastInactiveTransition: 09.01.2020 11:00:51
ActivationDisabledAndMoveNow                : CurrentState: Inactive; LastActiveTransition: ; LastInactiveTransition:
                                              15.06.2018 15:22:16
HighAvailabilityComponentStateOffline       : CurrentState: Inactive; LastActiveTransition: 17.01.2020 09:07:02;
                                              LastInactiveTransition: 17.01.2020 10:42:02
CriticalForMaintainingAvailability          : CurrentState: Inactive; LastActiveTransition: 31.01.2020 16:52:49;
                                              LastInactiveTransition: 31.01.2020 16:56:49
CriticalForMaintainingRedundancy            : CurrentState: Active; LastActiveTransition: 29.01.2020 11:43:06;
                                              LastInactiveTransition: 29.01.2020 11:42:06
PotentiallyCriticalForMaintainingRedundancy : CurrentState: Active; LastActiveTransition: 01.02.2020 05:49:37;
                                              LastInactiveTransition:
CriticalForRestoringAvailability            : CurrentState: Inactive; LastActiveTransition: 06.05.2019 09:16:36;
                                              LastInactiveTransition: 06.05.2019 09:20:36
CriticalForRestoringRedundancy              : CurrentState: Inactive; LastActiveTransition: 29.01.2020 11:42:06;
                                              LastInactiveTransition: 29.01.2020 11:43:06
HighForRestoringAvailability                : CurrentState: Inactive; LastActiveTransition: 29.01.2020 11:42:06;
                                              LastInactiveTransition: 29.01.2020 11:43:06
HighForRestoringRedundancy                  : CurrentState: Inactive; LastActiveTransition: 10.02.2020 09:05:02;
                                              LastInactiveTransition: 10.02.2020 09:06:02
IsSafeForMaintenance                        : CurrentState: Inactive; LastActiveTransition: 03.11.2019 09:42:35;
                                              LastInactiveTransition: 12.11.2019 06:29:58
IsValid                                     : True
ObjectState                                 : Unchanged

The lines 24-27 show the information we want to know. Both, the CriticalForMaintainingRedundancy and PotentiallyCriticalForMaintainingRedundancy parameters have a CurrentState value of Active. The Primary Activation Manager (PAM) considers the server availability critical to provide redundant availability of the database copies hosted by this server. 

Each of state-parameter shows three pieces of information:

• CurrentState
  The current state, either Active  or Inactive
   
• LastActiveTransition
  The timestamp of the last state change to Active
   
• LastInactiveTransition
  The timestamp of the last state change to Inactive

But there is still the bothering question of why are two of the six servers not safe for activating maintenance?

The reason is simple. The mailbox databases mounted by the member servers of the DAG have a different number of database copies. This configuration is due to data storage capacity constraints.

The mailbox databases storing primary user mailboxes use four database copies per database. Those copies are evenly distributed across all six mailbox servers. Mailbox database storing online archive mailboxes use three copies per database. This database copy layout allows for safely activating server maintenance for one server at a time without risk to database redundancy.

The servers LOCEXS01 and LOCEXS04 hold mailbox databases with just two copies per configured database. Placing one of those two servers into maintenance mode reduces the database availability for these mailbox databases to one. Therefore, PAM informs us that database redundancy is at risk when activating maintenance for those two servers. 

Example - During Maintenance

This example shows the member server redundancy state while LOCEXS06 is in maintenance. The reason for monthly maintenance for installing Windows updates.

Maintenance was activated using the StartDagServerMaintenance.ps1 PowerShell script.

Get-MailboxServerRedundancy -DatabaseAvailabilityGroup indag01

Identity        IsServerFound IsInMainten RepairUrgency SafeForMaintenance HealthInfoLastUpdateTime
                InAD          ance
--------        ------------- ----------- ------------- ------------------ ------------------------
LOCEXS01        True          False       Prohibited    False              17.02.2020 11:04:12
LOCEXS02        True          False       Normal        True               17.02.2020 11:04:12
LOCEXS03        True          False       Prohibited    False              17.02.2020 11:04:12
LOCEXS06        True          True        High          True               17.02.2020 11:04:12
LOCEXS05        True          False       Prohibited    False              17.02.2020 11:04:12
LOCEXS04        True          False       Prohibited    False              17.02.2020 11:04:12

Having a single server in maintenance has a significant impact on all other servers in the DAG. The servers LOCEXS03 and LOCEXS05 are not safe for maintenance as well. Activating maintenance for those two servers would affect the database redundancy for the databases hosted by those two servers.

Example - After Maintenance

After completing all maintenance tasks, e.g., installing Windows Updates or a new Exchange Server Cumulative Update, you end server maintenance using the PowerShell script StopDagServerMaintenance.ps1.

We query the server redundancy state again. 

Get-MailboxServerRedundancy -DatabaseAvailabilityGroup indag01

Identity        IsServerFound IsInMainten RepairUrgency SafeForMaintenance HealthInfoLastUpdateTime
                InAD          ance
--------        ------------- ----------- ------------- ------------------ ------------------------
LOCEXS01        True          False       Prohibited    False              17.02.2020 11:23:12
LOCEXS02        True          False       Normal        True               17.02.2020 11:23:12
LOCEXS03        True          False       Normal        True               17.02.2020 11:23:12
LOCEXS06        True          False       High          True               17.02.2020 11:23:12
LOCEXS05        True          False       Normal        True               17.02.2020 11:23:12
LOCEXS04        True          False       Prohibited    False              17.02.2020 11:23:12

Server LOCEXS06 is not in maintenance, but the RepairUrgency state is High. The local Exchange Server replication engine is still busy replicating and processing log files, and updating the search indices. When CopyQueueLength and ReplayQueueLength are back to 0, and ContentIndexStates are back to Healthy, the RepairUrgency switches to Normal.

Tip

• You receive an error message when activating maintenance for an Exchange Server not safe for maintenance using
  ​StartDagServerMaintenance.ps1 -serverName [SERVER]
  
  In this case, you must use:

.\StartDagServerMaintenance.ps1 -serverName SERVERNAME -overrideMinimumTwoCopies:$true

Enjoy Exchange Server!

Once upon a time at an Exchange Conference near you, a member of the Exchange Product Group (PG) announced that the very last Exchange Server will go away when having an active Exchange hybrid setup.

This was a hot topic for discussions at the Microsoft Exchange Conferences (MEC, @IamMEC) in 2012 and 2014, already. Since then the Exchange PG came up with a number of reasons why this is not possible. The question on when we will finally be able to remove the very last Exchange Server from the on-premises Exchange organization was asked every year at the Ignite Conference. 

Current Situation

Currently, the supported scenario for hybrid configurations between your on-premises Exchange organization and Exchange Online requires that you keep the last Exchange Server for creating, and managing Exchange related objects, even if those objects are located in Exchange Online. 

The following diagram illustrates the current requirements:

• Domain Controller Servers, with the most recent Active Directory Schema Update for a supported Exchange Server version, and stored Exchange-extended AD objects.
• DirSync Server, with AAD Connect, synchronizing Exchange-extended AD objects stored in on-premises Active Directory with Azure AD.
• Last Exchange Server, running a supported version of Exchange Server, managing the objects stored in on-premises Active Directory.

Interim solutions

In the past, there was communication on certain interim solutions that were supposed to support you in removing the last Exchange Server from your Exchange organization. Such interim solutions were:

• ADSI Edit
• Identity Management Solutions (IDM)
• PowerShell

At Ignite those solutions even made it into the official session catalog:

• Ignite 2018 - THR2145 - Why do we need to keep an Exchange Server on-premises when we move to the cloud?

All those interim solutions leave your on-premises Exchange organization and the Active Directory configuration in an uncomfortable twilight-zone. It was still something that worked somehow, but you knew it was officially not supported, and the secure and stable operation of the hybrid configuration was at risk.

But wait...

Removing the last Exchange Server is supported!

(at least when all components are released)

Solution

The new approach for managing your Exchange Online tenancy after migrating your on-premises Exchange organization to Exchange Online does not require an on-premises Exchange Server. 

The new mode of operation reduces your on-premises requirements to:

• Domain Controller Servers, running the most recent release Windows Server 2019 with the most recent Active Directory Schema Update for Exchange Server 2019 CU 5. This updates the Exchange-extended AD objects and the hybrid configuration object to support modern EXO management. Also, you must enable the Windows Feature "Exchange Online Remote Features" as described below.
   
• DirSync Server, with AAD Connect, synchronizing Exchange-extended AD objects stored in on-premises Active Directory with Azure AD. You must run the most recent release of Azure AD Connect (1.4.38.0) and have Exchange Hybrid enabled in AAD Connect.
  See release notes for Azure AD Connect. 
   
• Administrative PC, running the most recent version of the Exchange Online PowerShell v2 (aka ExchangeOnlineManagement)

The following diagram illustrates the new modern Exchange Online Management experience:

Simply you remove the requirement to use on-premises Exchange Server to write to your on-premises Active Directory. Instead, Azure AD Connect uses a new synchronization capability to handle the new Exchange Management experience in the AAD Connect MetaVerse. The on-premises AD-connector writes the changes to Active Directory which keeps the Active Directory up-to-date for all other on-premises solutions that require identities to have a proper state.

You execute all Exchange-related actions using the new Exchange Online Management PowerShell module, or, if needed, the new Modern Exchange Admin Center (EAC, which was announced at Ignite 2019.

Enable Modern Exchange Admin Experience (MEAE)

Before you uninstall the last Exchange Server from your on-premises Exchange organization, ensure that you

• Updated the Active Directory Schema Version to Exchange Server 2019 CU5
• Prepare the Active Directory and all domains in the Active Directory Forest that contain Exchange objects
• Ensure that there are not pre-Windows 2019 domain controller in use, and patch all domain controller to the March 2020 release
• Install the new EXORemote Windows Feature 

PS C:\> Get-WindowsFeature

Display Name                                            Name                       Install State
------------                                            ----                       -------------
[ ] Active Directory Certificate Services               AD-Certificate                 Available
    [ ] Certification Authority                         ADCS-Cert-Authority            Available
    [ ] Certificate Enrollment Policy Web Service       ADCS-Enroll-Web-Pol            Available
    [ ] Certificate Enrollment Web Service              ADCS-Enroll-Web-Svc            Available
    [ ] Certification Authority Web Enrollment          ADCS-Web-Enrollment            Available
    [ ] Network Device Enrollment Service               ADCS-Device-Enrollment         Available
    [ ] Online Responder                                ADCS-Online-Cert               Available
[ ] Active Directory Domain Services                    AD-Domain-Services             Available
[ ] Active Directory Federation Services                ADFS-Federation                Available
[ ] Active Directory Lightweight Directory Services     ADLDS                          Available
[ ] Active Directory Rights Management Services         ADRMS                          Available
    [ ] Active Directory Rights Management Server       ADRMS-Server                   Available
    [ ] Identity Federation Support                     ADRMS-Identity                 Available
[ ] Device Health Attestation                           DeviceHealthAttestat...        Available
[ ] DHCP Server                                         DHCP                           Available
[ ] DNS Server                                          DNS                            Available
[ ] Exchange Online Remote Features                     EXORemote                      Available
[ ] Fax Server                                          Fax                            Available
[X] File and Storage Services                           FileAndStorage-Services        Installed
    [X] File and iSCSI Services                         File-Services                  Installed
        [X] File Server                                 FS-FileServer                  Installed
        [ ] BranchCache for Network Files               FS-BranchCache                 Available
[...]

PS C:\> Install-WindowsFeature -Name EXORemote

Display Name                                            Name                       Install State
------------                                            ----                       -------------
[ ] Active Directory Certificate Services               AD-Certificate                 Available
    [ ] Certification Authority                         ADCS-Cert-Authority            Available
    [ ] Certificate Enrollment Policy Web Service       ADCS-Enroll-Web-Pol            Available
    [ ] Certificate Enrollment Web Service              ADCS-Enroll-Web-Svc            Available
    [ ] Certification Authority Web Enrollment          ADCS-Web-Enrollment            Available
    [ ] Network Device Enrollment Service               ADCS-Device-Enrollment         Available
    [ ] Online Responder                                ADCS-Online-Cert               Available
[ ] Active Directory Domain Services                    AD-Domain-Services             Available
[ ] Active Directory Federation Services                ADFS-Federation                Available
[ ] Active Directory Lightweight Directory Services     ADLDS                          Available
[ ] Active Directory Rights Management Services         ADRMS                          Available
    [ ] Active Directory Rights Management Server       ADRMS-Server                   Available
    [ ] Identity Federation Support                     ADRMS-Identity                 Available
[ ] Device Health Attestation                           DeviceHealthAttestat...        Available
[ ] DHCP Server                                         DHCP                           Available
[ ] DNS Server                                          DNS                            Available
[X] Exchange Online Remote Features                     EXORemote                      Installed
[ ] Fax Server                                          Fax                            Available
[X] File and Storage Services                           FileAndStorage-Services        Installed
    [X] File and iSCSI Services                         File-Services                  Installed
        [X] File Server                                 FS-FileServer                  Installed
        [ ] BranchCache for Network Files               FS-BranchCache                 Available
[...]

Even though not explicitly stated, you should restart the server after installing the Windows feature.

• Connect to the modern Exchange Online Admin Center as a Global Administrator or Exchange Services Administrator and add at least one of your on-premises domain controllers to the list of modern management servers.
  https://admin.exchange.microsoft.com/#/modernmanagement

As part of the next AAD Connect synchronization cycle, the magic happens.

• The added EXO modern management server is validated
  • You'll see an error sign in Modern Exchange Admin Center if the validation has failed
•  Azure AD removes the "on-premises"-lock from the Exchange related attributes, so you can start managing these attributes in EXO
• The Exchange hybrid configuration object is set to "Modern Coexistence", ensuring that a newly added on-premises Exchange Server will not interfere with the setup

Verify that you can edit the Exchange related attributes of synchronized Active Directory objects in Exchange Online or Azure AD before you remove your last Exchange Server. 

Whey ready to uninstall the last Exchange Server you must use the following command line parameters to remove the server as intended. Otherwise, you'll leave the Exchange organization in an inchoate state. Ensure that you use an administrative PowerShell session. 

./Setup.exe /mode:uninstall /SwitchToMEMA /IAcceptExchangeOnlineLicenseTerms

Normally, you do not have to accept license terms when uninstalling Exchange Server, but in this case, you have to accept the Exchange Online license terms.

Prerequisites

• Windows Server 2019 Desktop Experience Editon or Core Edition, March 2020 Update
  Core Edition recommended
• Azure AD Connect, release 1.4.38.0
• Exchange Server 2019 CU5, as the last Exchange Server

Links

• Exchange Online Management PowerShell Module v2
• Hybrid deployment prerequisites
• Exchange Server hybrid deployments
• Install-WindowsFeature
• You must follow the instructions given in the follow article:
  Final instructions for enabling this awesome new MEMA-experience

Enjoy the modern experience and management options of Exchange Online!

Exchange Conferences

• MEC 2014 - Austin
• MEC 2012 - Orlando

The Microsoft 365 Virtual Marathon took place on May 27-28 2020.

The recording of my session "Exchange Hybrid - What, Why, and How" is available on YouTube. 

Browse all recordings of the Microsoft 365 Virtual Marathon here: https://www.youtube.com/channel/UCrtmT6Ir1MIs0ZES7sKMmqA

Enjoy!

The Microsoft 365 Virtual Marathon is happening on May 27-28 2020.

This is a free online event, providing you with 36 hours of non-stop sessions from speakers around the globe. You can join every time. 

• 300+ Speakers
• 400+ Sessions
• Keynotes by Jeff Teper, Naomi Moneypenny, Bill Baer, Jon Levesque, Laurie Pottmeyer, and Michael Holste
• Sessions are primarily in Englisch, but there are session in six additional languages
  • French, German, Japanese, Korean, Portuguese, and Spanish 
• Hashtag: #M365VM

The virtual conference is a joint effort with SPC.

I speak at Microsoft 365 Virtual Marathon about:

• Exchange Hybrid - What, Why, and How
• Wednesday, May 27th
• 23:00h (CEST)

The marathon session plan is available here.

Register now.

Link

• Register for Microsoft 365 Virtual Marathon

Enjoy!

Recently I had to support the uninstall of Exchange Server 2016 CU10 on a Windows Server 2019 system. That this setup is not supported is a different topic. In this case, a new Exchange Server 2016 system was placed in service, and the old system needed to be removed from the on-premises Exchange organization.

We mounted the Exchange 2016 CU10 ISO, and ran the following command from an administrative command line:

Setup.exe /mode:uninstall

Prerequisites Checks

The prerequisites check failed with an odd error:

http://terenceluk.blogspot.com/2017/01/attempting-to-delete-exchange-server.html

Querying for any incompleted public folder migration requests returned no results. But the prerequisites check insisted that there was an existing public folder migration request. In such a case you already know that you have to use ADSIEdit to find the object in question. 

It turned out that the prerequisites check was right, as we found a single public folder migration request in the Active Directory configuration partition. The request was an artifact of an unsuccessful migration attempt in 2019. After we have checked that the current modern public folder hierarchy worked as expected, we deleted the artifact from Active Directory.

Now the uninstall procedure passed the prerequisites check successfully and the uninstaller moved on removed Exchange Server 2016 step by step.

Until...

Uninstall Error

The uninstall step Language Files an Access Denied exception while executing MSIEXEC uninstall actions for each Language Pack.

Language Files                                                                                    FAILED

The following error was generated when "$error.Clear();
$regPath='HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall';
$PackageGUIDRegEx = "{DEDFFB[0-9a-fA-F]{2}-42EC-4E26-[0-9a-fA-F]{4}-430E86DF378C}";
$InstallPath = (Get-ItemProperty 'HKLM:\SOFTWARE\Microsoft\ExchangeServer\v15\setup').MsiInstallPath;

if(test-path ($regPath))
{
Write-ExchangeSetupLog -info ("Removing " +  $RoleLanguagePackType + " Language Packs.");
Get-ChildItem ($regPath) | foreach{
if($_ -match "(?<ProductCode>$PackageGUIDRegEx)") {
$langPackPackageCode = $matches['ProductCode'];
if($langPackPackageCode -ne $null -and $langPackPackageCode.Length -ne 0) {
Write-ExchangeSetupLog -info ("Removing package $langPackPackageCode");
$language = $langPackPackageCode.Substring(20,4);
$logFilePath = [IO.Path]::Combine($RoleLogFilePath,"Uninstall") + '.' + $language +
'.' + "Client" + "." + $RoleLogDateTime + ".msilog";
uninstall-MsiPackage -ProductCode ($langPackPackageCode) -LogFile ($logFilePath);
};
};
};
Get-Childitem -Path $InstallPath -include ".Localized.js",".Localized.min.js" -recurse | foreach ($) {remove-item $.fullname};
Write-ExchangeSetupLog -info "Remove Language Packs completed.";
};
" was run: "**System.UnauthorizedAccessException: Access is denied** ---> System.ComponentModel.Win32Exception: Access is denied
--- End of inner exception stack trace ---
at System.Management.Automation.Utils.NativeDirectoryExists(String path)
at System.Management.Automation.SessionStateInternal.IsItemContainer(CmdletProvider providerInstance, String path, CmdletProviderContext context)".

Interestingly, the ExchangeSetup log file showed that the uninstaller wrote the informational text Remove Language Packs completed successfully. 

Solution

After following an idea to remove the language pack-related registry keys and other fancy approaches, we did something trivial. We restarted the server, mounted the ISO file, and ran Setup.exe /mode:uninstall again. 

The uninstaller process now passed the step Language Files without any issues.

I sometimes like simple solutions.

Enjoy Exchange Server. 

I am honored to be a speaker at the Virtual Scottish Summit 2021 conference, taking place on Saturday, 27. February.

The Scottish Summit is a truly European event. You can choose from 365 sessions covering mostly any about Microsoft 365 workloads in seven languages:

• Englisch
• Spanish
• German
• French
• Italian
• Portuguese
• Polish

Attend my session when you are interested in the challenges of implementing Exchange Server Hybrid, and the requirements to make it work with Microsoft Teams and on-premises mailboxes.

My session at Scottish Summit 2021:

• Exchange Hybrid - What, Why, and How
• Session schedule details
  • 13:00h (UK) 

Links

• Scottish Summit 2021 Website
• Register for Scottish Summit
• Scottish Summit Sessions
• Scottish Summit 2020 YouTube Channel

See you online for the Virtual Scottish Summit.