Services of third-party software solutions often interfere with installing a new Exchange Server cumulative update, because these services have a file lock active.
To avoid any issues when installing a CU, or having the prerequisites check fail due to open files, you simply stop the Windows services and ensure that those services do not restart automatically. Especially monitoring solutions that use some kind of watchdog service are a candidate that you must disable for installing an Exchange Server CU.
The following two PowerShell examples help you to prepare the Windows services for installing an Exchange Server CU.
In preparation for the installation of an Exchange Server cumulative update, you can use the following PowerShell commands.
# Disable and stop services or just stop services # Add other services as needed # Set SMEX service to manual and stop services Get-Service -Name 'ScanMail*' | Set-Service -StartupType Manual Get-Service -Name 'ScanMail*' | ?{$_.Status -eq 'Running'} | Stop-Service -Force # Stop SMEX SQL Express instance Get-Service -Name 'MSSQL*' | ?{$_.Status -eq 'Running'} | Stop-Service -Force # Disable and stop ENow monitoring services Get-Service 'ENow*' | Set-Service -StartupType Disabled Get-Service 'ENow*' | ?{$_.Status -eq 'Running'} | Stop-Service -Force # Stop NetBackup service Get-Service -Name 'NetBackup*' | ?{$_.Status -eq 'Running'} | Stop-Service -Force
After installing the Exchange Server cumulative update you should restart your computer. I recommend initiating a check for additional Windows Updates for the CU. This helps to ensure that you do not only have the latest CU installed, but required security updates as well.
# Enabling and starting services # Adjust the list of services as needed # Enable and start SMEX services Get-Service -Name 'ScanMail*' | Set-Service -StartupType Automatic Get-Service -Name 'ScanMail*' | Start-Service # Enable and start ENow Monitoring services Get-Service -Name 'ENow*' | Set-Service -StartupType Automatic Get-Service -Name 'ENow*' | Start-Service
Enjoy Exchange Server.
Once upon a time at an Exchange Conference near you, a member of the Exchange Product Group (PG) announced that the very last Exchange Server will go away when having an active Exchange hybrid setup.
This was a hot topic for discussions at the Microsoft Exchange Conferences (MEC, @IamMEC) in 2012 and 2014, already. Since then the Exchange PG came up with a number of reasons why this is not possible. The question on when we will finally be able to remove the very last Exchange Server from the on-premises Exchange organization was asked every year at the Ignite Conference.
Currently, the supported scenario for hybrid configurations between your on-premises Exchange organization and Exchange Online requires that you keep the last Exchange Server for creating, and managing Exchange related objects, even if those objects are located in Exchange Online.
The following diagram illustrates the current requirements:
In the past, there was communication on certain interim solutions that were supposed to support you in removing the last Exchange Server from your Exchange organization. Such interim solutions were:
At Ignite those solutions even made it into the official session catalog:
All those interim solutions leave your on-premises Exchange organization and the Active Directory configuration in an uncomfortable twilight-zone. It was still something that worked somehow, but you knew it was officially not supported, and the secure and stable operation of the hybrid configuration was at risk.
But wait...
Removing the last Exchange Server is supported! (at least when all components are released)
The new approach for managing your Exchange Online tenancy after migrating your on-premises Exchange organization to Exchange Online does not require an on-premises Exchange Server.
The new mode of operation reduces your on-premises requirements to:
The following diagram illustrates the new modern Exchange Online Management experience:
Simply you remove the requirement to use on-premises Exchange Server to write to your on-premises Active Directory. Instead, Azure AD Connect uses a new synchronization capability to handle the new Exchange Management experience in the AAD Connect MetaVerse. The on-premises AD-connector writes the changes to Active Directory which keeps the Active Directory up-to-date for all other on-premises solutions that require identities to have a proper state.
You execute all Exchange-related actions using the new Exchange Online Management PowerShell module, or, if needed, the new Modern Exchange Admin Center (EAC, which was announced at Ignite 2019.
Before you uninstall the last Exchange Server from your on-premises Exchange organization, ensure that you
PS C:\> Get-WindowsFeature Display Name Name Install State ------------ ---- ------------- [ ] Active Directory Certificate Services AD-Certificate Available [ ] Certification Authority ADCS-Cert-Authority Available [ ] Certificate Enrollment Policy Web Service ADCS-Enroll-Web-Pol Available [ ] Certificate Enrollment Web Service ADCS-Enroll-Web-Svc Available [ ] Certification Authority Web Enrollment ADCS-Web-Enrollment Available [ ] Network Device Enrollment Service ADCS-Device-Enrollment Available [ ] Online Responder ADCS-Online-Cert Available [ ] Active Directory Domain Services AD-Domain-Services Available [ ] Active Directory Federation Services ADFS-Federation Available [ ] Active Directory Lightweight Directory Services ADLDS Available [ ] Active Directory Rights Management Services ADRMS Available [ ] Active Directory Rights Management Server ADRMS-Server Available [ ] Identity Federation Support ADRMS-Identity Available [ ] Device Health Attestation DeviceHealthAttestat... Available [ ] DHCP Server DHCP Available [ ] DNS Server DNS Available [ ] Exchange Online Remote Features EXORemote Available [ ] Fax Server Fax Available [X] File and Storage Services FileAndStorage-Services Installed [X] File and iSCSI Services File-Services Installed [X] File Server FS-FileServer Installed [ ] BranchCache for Network Files FS-BranchCache Available [...]
PS C:\> Install-WindowsFeature -Name EXORemote Display Name Name Install State ------------ ---- ------------- [ ] Active Directory Certificate Services AD-Certificate Available [ ] Certification Authority ADCS-Cert-Authority Available [ ] Certificate Enrollment Policy Web Service ADCS-Enroll-Web-Pol Available [ ] Certificate Enrollment Web Service ADCS-Enroll-Web-Svc Available [ ] Certification Authority Web Enrollment ADCS-Web-Enrollment Available [ ] Network Device Enrollment Service ADCS-Device-Enrollment Available [ ] Online Responder ADCS-Online-Cert Available [ ] Active Directory Domain Services AD-Domain-Services Available [ ] Active Directory Federation Services ADFS-Federation Available [ ] Active Directory Lightweight Directory Services ADLDS Available [ ] Active Directory Rights Management Services ADRMS Available [ ] Active Directory Rights Management Server ADRMS-Server Available [ ] Identity Federation Support ADRMS-Identity Available [ ] Device Health Attestation DeviceHealthAttestat... Available [ ] DHCP Server DHCP Available [ ] DNS Server DNS Available [X] Exchange Online Remote Features EXORemote Installed [ ] Fax Server Fax Available [X] File and Storage Services FileAndStorage-Services Installed [X] File and iSCSI Services File-Services Installed [X] File Server FS-FileServer Installed [ ] BranchCache for Network Files FS-BranchCache Available [...]
Even though not explicitly stated, you should restart the server after installing the Windows feature.
As part of the next AAD Connect synchronization cycle, the magic happens.
Verify that you can edit the Exchange related attributes of synchronized Active Directory objects in Exchange Online or Azure AD before you remove your last Exchange Server.
Whey ready to uninstall the last Exchange Server you must use the following command line parameters to remove the server as intended. Otherwise, you'll leave the Exchange organization in an inchoate state. Ensure that you use an administrative PowerShell session.
./Setup.exe /mode:uninstall /SwitchToMEMA /IAcceptExchangeOnlineLicenseTerms
Normally, you do not have to accept license terms when uninstalling Exchange Server, but in this case, you have to accept the Exchange Online license terms.
Enjoy the modern experience and management options of Exchange Online!
Exchange Conferences
You are not able to list public folders in a co-existence scenario with Exchange Server 2007 and Exchange Server 2010/2013 using the Exchange 2007 EMS or EMC.
When you try to execute Get-PublicFolder you receive the following error:
Get-PublicFolder " There is no existing PublicFolder that matches the following Identity: '\'. Please make sure that you specified the correct PublicFolder Identity and that you have the necessary permissions to view PublicFolder.
This might happen after you have removed the first Exchange 2007 mailbox server, but not the last Exchange 2007 mailbox server.
Exchange Server 2007 uses the Exchange System Attendant to access the public folder store and fails if the System Attendant discovery in Active Directory does not provide a proper configuration.
KB 2621350 describes the discovery process:
There two annoying things about these steps
The magic System Attendant mailbox has been removed from Exchange 2010. But the System Attendent configuration node does still exist in the Active Directory Configuration Partition for compatibility reasons. The configured attributes of the System Attendant entry vary depending on the version of the installed Exchange Server.
In regards to the public folder issue, we need to focus on the following:
To fix the public folder access issue for Exchange Server 2007, set the homeMDB and homeMTA attributes. Set the Exchange System Attendant attributes to appropriate values for your Exchange servers.
Repeat steps 4 to 8 for each Exchange 2013 server in your environment.
Repeat steps 4 to 13 for each Exchange 2010 server in your environment.
Wait for Active Directory replication and retry to access the public folders using Get-PublicFolder in an Exchange Server 2007 Management Shell.
It might be required to restart the Exchange 2007 Information Store and System Attendant service of the Exchange 2007 server in question
Use an administrative PowerShell
Restart-Service MSExchangeIS Restart-Service MSExchangeSA
I haven’t noticed any issues in production environments so far. If you encounter any issues in your environment, feel free to leave a comment.
Do you need assistance with your Exchange Server setup? You have questions about your Exchange Server infrastructure and going hybrid? You are interested in what Exchange Server 2016 has to offer for your environment?
Contact me at thomas@mcsmemail.de Follow at https://twitter.com/stensitzki
You might encounter EventId 106 errors on Exchange 2013/2016 servers.
There are some article available on how to fix this issue on servers having a full installationof Exchange Server 2013/2016.
But you might encounter this error on servers having the Exchange 2013/2016 Management Tools installed only as well. In this case the solutions described will not work as expected.
Running the New-PerfCounters cmdlet requires the Microsoft.Exchange.Management.PowerShell.Setup PowerShell snapin. This PowerShell snapin cannot be loaded on servers having the Exchange Management Tools installed.
Add-PsSnapin : The Windows PowerShell snap-in 'Microsoft.Exchange.Management.PowerShell.Setup' is not installed on this computer. At line:1 char:1 + Add-PsSnapin Microsoft.Exchange.Management.PowerShell.Setup + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : InvalidArgument: (Microsoft.Excha...owerShell.Setup:String) [Add-PSSnapin], PSArgumentEx ception + FullyQualifiedErrorId : AddPSSnapInRead,Microsoft.PowerShell.Commands.AddPSSnapinCommand
The required code library is available on servers having the Exchange Management Tools installed. But the library is not added to the registry by default.
Registry of a server running Exchange Management Tools only:
Registry of a fully installed Exchange Server:
Just export the registry key Microsoft.Exchange.Management.PowerShell.Setup from an Exchange Server, fix Exchange file paths (if required) and import the regiytry file on the server having the Exchange Management Tools installed.
Example registry file:
Windows Registry Editor Version 5.00 [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\PowerShell\1\PowerShellSnapIns\Microsoft.Exchange.Management.PowerShell.Setup] "CustomPSSnapInType"="Microsoft.Exchange.Management.PowerShell.SetupPSSnapIn" "ApplicationBase"="D:\\Program Files\\Microsoft\\Exchange Server\\V15\\bin" "AssemblyName"="Microsoft.Exchange.PowerShell.Configuration, Version=15.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" "Description"="Setup Tasks for the Exchange Server" "ModuleName"="D:\\Program Files\\Microsoft\\Exchange Server\\V15\\bin\\Microsoft.Exchange.PowerShell.configuration.dll" "PowerShellVersion"="1.0" "Vendor"="Microsoft" "Version"="15.0.0.0"
After adding the registry key, you can successfully fix the performance counter issue.
Add-PsSnapin Microsoft.Exchange.Management.PowerShell.Setup New-PerfCounters -DefinitionFileName "$exinstall\setup\perf\WorkerTaskFrameworkPerfCounters.xml”
You have an Exchange Server 2016 organization and plan to upgrade to Cumulative Update 10. You log on to an Exchange Server, activate DAG maintenance and prepare the Server Component States for installing the new Cumulative Update.
You open an elevated PowerShell Session and start the Setup using
./Setup.exe /Mode:Upgrade /IAcceptExchangeServerLicenseTerms
Out of a sudden the Exchange Server CU Setup fails while executing setup step Stopping Services with an error:
Microsoft Exchange Server 2016 Cumulative Update 10 Unattended Setup Copying Files... File copy complete. Setup will now collect additional information needed for installation. Languages Management tools Mailbox role: Transport service Mailbox role: Client Access service Mailbox role: Unified Messaging service Mailbox role: Mailbox service Mailbox role: Front End Transport service Mailbox role: Client Access Front End service Performing Microsoft Exchange Server Prerequisite Check Configuring Prerequisites COMPLETED Prerequisite Analysis COMPLETED Configuring Microsoft Exchange Server Preparing Setup COMPLETED Stopping Services FAILED The following error was generated when "$error.Clear(); & $RoleBinPath\ServiceControl.ps1 -Operation:DisableServices -Roles:($RoleRoles.Replace('Role','').Split(',')) -SetupScriptsDirectory:$RoleBinPath; & $RoleBinPath\ServiceControl.ps1 -Operation:Stop -Roles:($RoleRoles.Replace('Role','').Split(',')) -IsDatacenter:([bool]$RoleIsDatacenter) " was run: "Microsoft.PowerShell.Commands.ProcessCommandException: Cannot stop process "fms (2496)" because of the following error: Access is denied ---> System.ComponentModel.Win32Exception: Access is denied at System.Diagnostics.ProcessManager.OpenProcess(Int32 processId, Int32 access, Boolean throwIfExited) at System.Diagnostics.Process.GetProcessHandle(Int32 access, Boolean throwIfExited) at System.Diagnostics.Process.get_HasExited() at Microsoft.PowerShell.Commands.StopProcessCommand.ProcessRecord() --- End of inner exception stack trace ---". The Exchange Server setup operation didn't complete. More details can be found in ExchangeSetup.log located in the <SystemDrive>:\ExchangeSetupLogs folder.
Why would an error like Access Denied happen while executing the Setup.exe as a member of the local Administrators security group from within an elevated PowerShell session?
The PowerShell code executed as part of the CU Setup sets the startup type of Exchange and some Windows services to Disabled. This ensures that in the case of a server reboot an automatic start of the service will not interfere partially executed the setup. After setting the startup type to Disabled the services are stopped.
The services are controlled by the ServiceControl.ps1 script which is located on the Exchange Server installation media in \Setup\ServerRoles\Common\.
The function StopServices stops services using the Stop-Service cmdlet. Due to some timing issues some services are stopped by killing the running processes using Stop-Process -Force.
The services stopped by stopping the running process are:
Executing the Stop-Process cmdlet results in the Access Denied error.
The issue is related to fact that the user account logged on the server and executing the Exchange Server Cumulatice Update does not have the local User Rights Assignment to Debug Programs.
By default the right to debug programs is assigned to the local Adminstrators security group. In secured Active Directory infrastructures the user rights assignments and local security groups are often managed using Group Policy Objects (GPO). The GPOs manage the names of local security groups, group memberships and even user rights assignments.
The client encountering the issue described above hasn't had any issues installing Cumulative Updates for Exchange Server 2013 in the past. So this is solution is related to the setup of Exchange Server 2016 Cumulative Updates on Windows Server 2016 only. If you have any information regarding Exchange Server 2013, let me know using the comments section below.
Enjoy Exchange Server!
The use of Exchange Edge Transport Servers requires the synchronization of user and configuration data from internal Exchange Servers to the Edge Transport Servers. The synchronization utilizes secure LDAP (EdgeSync) to transmit the data securely and is based on an Edge Subscription.
When you create a new Edge Subscription on your internal Exchange Servers by importing the Edge Subscription XML-file, establishing the EdgeSync-connection might fail.
You will find the following error in the application event log of the internal Exchange Server:
Log Name: Application Source: MSExchange EdgeSync Event ID: 1035 Task Category: Synchronization Level: Error Keywords: Classic Description: EdgeSync failed to synchronize because it only supports Cryptographic API certificates. The local Hub Transport server's default certificate with thumbprint XYZ isn't a CAPI certificate. To set a CAPI certificate as the default certificate, use the Enable-ExchangeCertificate cmdlet with the Services parameter using the value of SMTP.
The private key of the current Exchange Transport default certificate of the internal Exchange servers uses a CNG private key. EdgeSync requires a CAPI1 based private key.
This problem occurs primarily when using an Enterprise Certificate Authority using certificate templates with individual template settings.
Get-TransportService | ft Name,InternalTransportCertificateThumbprint
certutil -v -store my > cert.txt
If both attribute are of the value 0, the certificate if a CNG certificate.
The section might look like this:
Unique container name: XYZ Provider = Microsoft Software Key Storage Provider ProviderType = 0 Flags = 20 (32) CRYPT_MACHINE_KEYSET -- 20 (32) KeySpec = 0 -- XCN_AT_NONE
Use OpenSSL to convert the CNG certificate to a CAPI1 certificate.
Using OpenSSL requires the download of the Windows release of OpenSSL. I recommend to not install the software on the Exchange Server but a separate Windows server or your administrative desktop system. Additionally, you need the certificate with its private key as a PFX-file.
Use the following steps to convert the CNG certificate to a CAPI1 certificate.
openssl pkcs12 -in CERT.pfx -out cert.pem -nodes
openssl pkcs12 -export -in cert.pem -out NEWCERT.pfx
The new PFX-file is now a CAPI1 certificate. The new certificate has the same thumbprint. Now you must replace the current certificate used by Exchange Server with the new certificate.
Replacing the certificate requires a downtime of each Exchange Server requiring the certificate replacement. This is due to the requirement to remove the CNG certificate first, following the import of the CAPI1 certificate. Afterward, you need to enable the required Exchange services.
Get-ExchangeCertificate -Server SERVERNAME
# It is mandatory to answer the query for replacing the default certificate with YES Enable-ExchangeCertificate -Thumbprint THUMBPRINT -Services SMTP # Restart the transport service Restart-Service MSExchangeTransport
# It is mandatory to answer the query for replacing the default certificate with YES Enable-ExchangeCertificate -Thumbprint NEWCERTTHUMBPRINT -Services SMTP # Restart the transport service Restart-Service MSExchangeTransport
Now, that you updated the local Exchange Servers there is one more step that needs to be checked on the Edge Transport Servers.
Edge Transport Servers are not domain-joined and therefore do not receive any GPO-based configuration. Each required configuration must be performed locally. To ensure that the default transport certificate of the internal Exchange servers can be used for cryptographic operations we must ensure that the certificate chain of that certificate is present in the certificate store of Edge Transport servers.
Take a look at the certificate chain of the converted CAPI1 certificate and import the Root-CA and Subordinate-CA certificates into the Edge Transport servers local certificate store. You must ensure that the certificates are placed into appropriate stores:
Next, you create a new Edge Subscription on your Edge Transport server and create a new subscription for the Active Directory site on the internal Exchange Server. The internal Exchange Servers are now able to establish an EdgeSync connection and encrypting the data transferred to the Edge Transport servers.
When you receive the TLS certificate as PFX/PKCS12 file, you import the certificate and the private key. The import process itself defines the priavte key Crypto Provider. Using the following command line you ensure that the import process suses the legacy crypto provider.
certutil -csp "Microsoft RSA SChannel Cryptographic Provider" -importpfx my MYCERTpfx
Enjoy Exchange Server and Edge Transport!
I came across an interesting issue when setting up a new Exchange 2013 server in an Exchange organization having the cmdlet extension agent enabled.
As mentioned in my last post Exchange setup checks for the existence of the ScriptingAgentConfig.xml file when agent extenstion is enabled in the Exchange organization. It turned out that this ist not only true when you install an Exchange update using /mode:update, but as well when installing a new Exchange server using /mode:install.
The following error occurs when Exchange Management Tools are provisioned.
Configuring Microsoft Exchange Server Preparing Setup COMPLETED Stopping Services COMPLETED Copying Exchange Files COMPLETED Language Files COMPLETED Restoring Services COMPLETED Language Configuration COMPLETED Exchange Management Tools FAILED The following error was generated when "$error.Clear(); Set-WERRegistryMarkers; " was run: "Microsoft.Exchange.Provisioning.ProvisioningBrokerException: Provisioning layer initialization failed: '"Scripting Agent initialization failed: "File is not found: 'C:\Program File s\Microsoft\Exchange Server\V15\Bin\CmdletExtensionAgents\ScriptingAgentConfig.xml'.""' ---> Microso ft.Exchange.Provisioning.ProvisioningException: "Scripting Agent initialization failed: "File is not found: 'C:\Program Files\Microsoft\Exchange Server\V15\Bin\CmdletExtensionAgents\ScriptingAgentConf ig.xml'."" ---> System.IO.FileNotFoundException: "File is not found: 'C:\Program Files\Microsoft\Exc hange Server\V15\Bin\CmdletExtensionAgents\ScriptingAgentConfig.xml'." at Microsoft.Exchange.ProvisioningAgent.ScriptingAgentConfiguration.Initialize(String xmlConfigPa th) at Microsoft.Exchange.ProvisioningAgent.ScriptingAgentConfiguration..ctor(String xmlConfigPath) --- End of inner exception stack trace --- at Microsoft.Exchange.ProvisioningAgent.ScriptingAgentConfiguration..ctor(String xmlConfigPath) at Microsoft.Exchange.ProvisioningAgent.ScriptingAgentClassFactory.get_Configuration() at Microsoft.Exchange.ProvisioningAgent.ScriptingAgentClassFactory.GetSupportedCmdlets() at Microsoft.Exchange.Provisioning.ProvisioningBroker.BuildHandlerLookupTable(CmdletExtensionAgen t[] enabledAgents, Exception& ex) --- End of inner exception stack trace --- at Microsoft.Exchange.Provisioning.ProvisioningLayer.GetProvisioningHandlersImpl(Task task) at Microsoft.Exchange.Provisioning.ProvisioningLayer.GetProvisioningHandlers(Task task) at Microsoft.Exchange.Configuration.Tasks.Task.<BeginProcessing>b__4() at Microsoft.Exchange.Configuration.Tasks.Task.InvokeNonRetryableFunc(Action func, Boolean termin atePipelineIfFailed)".
As expected a fresh Exchange install contains the sample file only. The following screenshot shows the Exchange Management Shell and the releated folder in the background.
The only solution currently known to me is to disable the cmdlet extension agent until the setup of the new Exchange server has finished.
Disable-CmdletExtensionAgent "Scripting Agent"
Having the cmdlet extension agent disabled the setup finishes without any issues. Don't forget to copy the cmdlet extension Xml file to the newly built server and to enable the cmdlet extension agent again.
Enable-CmdletExtensionAgent "Scripting Agent"
You need assistance with your Exchange Server setup? You have questions about your Exchange Server infrastructure and going hybrid? You are interested in what Exchange Server 2016 has to offer for your environment?