Services of third-party software solutions often interfere with installing a new Exchange Server cumulative update, because these services have a file lock active.
To avoid any issues when installing a CU, or having the prerequisites check fail due to open files, you simply stop the Windows services and ensure that those services do not restart automatically. Especially monitoring solutions that use some kind of watchdog service are a candidate that you must disable for installing an Exchange Server CU.
The following two PowerShell examples help you to prepare the Windows services for installing an Exchange Server CU.
In preparation for the installation of an Exchange Server cumulative update, you can use the following PowerShell commands.
# Disable and stop services or just stop services # Add other services as needed # Set SMEX service to manual and stop services Get-Service -Name 'ScanMail*' | Set-Service -StartupType Manual Get-Service -Name 'ScanMail*' | ?{$_.Status -eq 'Running'} | Stop-Service -Force # Stop SMEX SQL Express instance Get-Service -Name 'MSSQL*' | ?{$_.Status -eq 'Running'} | Stop-Service -Force # Disable and stop ENow monitoring services Get-Service 'ENow*' | Set-Service -StartupType Disabled Get-Service 'ENow*' | ?{$_.Status -eq 'Running'} | Stop-Service -Force # Stop NetBackup service Get-Service -Name 'NetBackup*' | ?{$_.Status -eq 'Running'} | Stop-Service -Force
After installing the Exchange Server cumulative update you should restart your computer. I recommend initiating a check for additional Windows Updates for the CU. This helps to ensure that you do not only have the latest CU installed, but required security updates as well.
# Enabling and starting services # Adjust the list of services as needed # Enable and start SMEX services Get-Service -Name 'ScanMail*' | Set-Service -StartupType Automatic Get-Service -Name 'ScanMail*' | Start-Service # Enable and start ENow Monitoring services Get-Service -Name 'ENow*' | Set-Service -StartupType Automatic Get-Service -Name 'ENow*' | Start-Service
Enjoy Exchange Server.
You are not able to list public folders in a co-existence scenario with Exchange Server 2007 and Exchange Server 2010/2013 using the Exchange 2007 EMS or EMC.
When you try to execute Get-PublicFolder you receive the following error:
Get-PublicFolder " There is no existing PublicFolder that matches the following Identity: '\'. Please make sure that you specified the correct PublicFolder Identity and that you have the necessary permissions to view PublicFolder.
This might happen after you have removed the first Exchange 2007 mailbox server, but not the last Exchange 2007 mailbox server.
Exchange Server 2007 uses the Exchange System Attendant to access the public folder store and fails if the System Attendant discovery in Active Directory does not provide a proper configuration.
KB 2621350 describes the discovery process:
There two annoying things about these steps
The magic System Attendant mailbox has been removed from Exchange 2010. But the System Attendent configuration node does still exist in the Active Directory Configuration Partition for compatibility reasons. The configured attributes of the System Attendant entry vary depending on the version of the installed Exchange Server.
In regards to the public folder issue, we need to focus on the following:
To fix the public folder access issue for Exchange Server 2007, set the homeMDB and homeMTA attributes. Set the Exchange System Attendant attributes to appropriate values for your Exchange servers.
Repeat steps 4 to 8 for each Exchange 2013 server in your environment.
Repeat steps 4 to 13 for each Exchange 2010 server in your environment.
Wait for Active Directory replication and retry to access the public folders using Get-PublicFolder in an Exchange Server 2007 Management Shell.
It might be required to restart the Exchange 2007 Information Store and System Attendant service of the Exchange 2007 server in question
Use an administrative PowerShell
Restart-Service MSExchangeIS Restart-Service MSExchangeSA
I haven’t noticed any issues in production environments so far. If you encounter any issues in your environment, feel free to leave a comment.
Do you need assistance with your Exchange Server setup? You have questions about your Exchange Server infrastructure and going hybrid? You are interested in what Exchange Server 2016 has to offer for your environment?
Contact me at thomas@mcsmemail.de Follow at https://twitter.com/stensitzki
These are the results of the Exchange Server Questionnaire from August 2021.
First of all, I want to thank all of you who participated in the questionnaire. The results are pretty interesting. Even though, that the results are not 100% representative they provide a high-level view of the Exchange Organizations, the mail flow configurations, and the future plans regarding hybrid and Exchange Online.
With 55 replies the questionnaire is far from being a comprehensive representation of the Exchange organizations. But the answers provide an idea of the Exchange landscape used by organizations globally.
Exchange Server 2016 is the dominant version currently in use, followed by Exchange Server 2019. The vast majority of 93% runs modern Exchange Server versions. But there are still older and unsupported Exchange Server versions in use. 7% use Exchange Server 2010 and older.
76% of the organizations maintain up to ten Exchange servers. 20% prefer to rely on just one Exchange server. It is interesting that only 2 (not percent) plan to go hybrid or to move to Exchange Online.
The majority of on-premises Exchange organizations are in the 1,000 - 10,000 mailboxes range. Nevertheless, the SMBs with 1 to 1,000 mailboxes adds up to 50% of the Exchange organizations that took part in this questionnaire. There are just a few organizations that host more than 50,000 mailboxes.
There are Exchange organizations that do not use an SMTP-Gateway solution as part of the mail-flow implementation. Thor organizations that do not use a gateway solution run 1 to 10 Exchange servers on-premises. The majority of those have less than 1,000 mailboxes but there are a few that are responsible for more than 1,000 mailboxes. That leaves the question of why an organization prefers to not secure mal-flow with a gateway.
The use of SMTP gateways is a must, as you do not want to expose your domain member servers to the Internet, not even for the SMTP protocol. A majority of 28 answers for other gateways shows, that there are so many products available and that I did not choose valid answer options upfront.
The Other answers include:
65% of the Exchange organizations of this questionnaire already run in a hybrid configuration with Exchange Online. Only 35% are (still) not using a hybrid setup.
Of those who currently do not run a hybrid configuration only 37% plan on implementing Exchange Hybrid or migrate fully to Exchange Online. Staying on-premises is the only option.
The majority of the organizations still running only an on-premises Exchange organization plan on implementing Exchange Hybrid or migrating to Exchange Online by the end of 2021. None of the participating organizations has plans scheduled after 2022.
It is no surprise that Classic Full Hybrid is the most adopted hybrid configuration. And, no surprise either, none of the other classic hybrid options is implemented. The modern hybrid approach is implemented but with lesser.
The reasons for staying with an on-premises Exchange organization vary. the reasons mentioned are:
There are still organizations that choose an on-premises Exchange organization in favor of Exchange Online. I wonder if company policies for reducing the carbon footprint might drive the migration of on-premises data center resources to hosted cloud services.
Exchange Server vNEXT is in scope for 47% of the organizations. When comparing it with the used Exchange Server version currently in use (~50% Exchange Server 2016) it is an indicator that some companies just skip Exchange Server 2019. Some organizations prefer not to follow the full life-cycle of Exchange Server. s7% of those who do not want to implement Exchange Server vNEXT and want to stay on-premises are single server implementations of Exchange.
The product Exchange Server is still widely used in on-premises deployments. The reasons vary from legal and compliance requirements, network bandwidth constraints, and the overall costs for Exchange Online. Exchange Server vNEXT is a must-have for nearly 50% of the organizations participating in this questionnaire. There are still older and unsupported versions in productive use. Why this is the case is unanswered in this questionnaire.
Organizations running a hybrid Exchange configuration primarily use a Classic Full Hybrid configuration. This might be due to an early implementation in those days when nothing else was available, or due to requirements using Microsoft Teams with on-premises mailboxes. The adoption of Modern Hybrid shows that the Hybrid Agent approach helps organizations that cannot implement a Classic Full Hybrid.
I leave the results of this questionnaire to your interpretation and look forward to your replies, either to this blog post or by social media on Twitter and LinkedIn. Please use the hashtag #ExchangeQuest2021.
There will be a new Exchange Server questionnaire in early 2022, covering various implementation scenarios in more detail. If you want to see a specific Exchange topic covered in the 2022 questionnaire, just let me know.
Again, thank you all for participating in this questionnaire.
You might encounter EventId 106 errors on Exchange 2013/2016 servers.
There are some article available on how to fix this issue on servers having a full installationof Exchange Server 2013/2016.
But you might encounter this error on servers having the Exchange 2013/2016 Management Tools installed only as well. In this case the solutions described will not work as expected.
Running the New-PerfCounters cmdlet requires the Microsoft.Exchange.Management.PowerShell.Setup PowerShell snapin. This PowerShell snapin cannot be loaded on servers having the Exchange Management Tools installed.
Add-PsSnapin : The Windows PowerShell snap-in 'Microsoft.Exchange.Management.PowerShell.Setup' is not installed on this computer. At line:1 char:1 + Add-PsSnapin Microsoft.Exchange.Management.PowerShell.Setup + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : InvalidArgument: (Microsoft.Excha...owerShell.Setup:String) [Add-PSSnapin], PSArgumentEx ception + FullyQualifiedErrorId : AddPSSnapInRead,Microsoft.PowerShell.Commands.AddPSSnapinCommand
The required code library is available on servers having the Exchange Management Tools installed. But the library is not added to the registry by default.
Registry of a server running Exchange Management Tools only:
Registry of a fully installed Exchange Server:
Just export the registry key Microsoft.Exchange.Management.PowerShell.Setup from an Exchange Server, fix Exchange file paths (if required) and import the regiytry file on the server having the Exchange Management Tools installed.
Example registry file:
Windows Registry Editor Version 5.00 [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\PowerShell\1\PowerShellSnapIns\Microsoft.Exchange.Management.PowerShell.Setup] "CustomPSSnapInType"="Microsoft.Exchange.Management.PowerShell.SetupPSSnapIn" "ApplicationBase"="D:\\Program Files\\Microsoft\\Exchange Server\\V15\\bin" "AssemblyName"="Microsoft.Exchange.PowerShell.Configuration, Version=15.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" "Description"="Setup Tasks for the Exchange Server" "ModuleName"="D:\\Program Files\\Microsoft\\Exchange Server\\V15\\bin\\Microsoft.Exchange.PowerShell.configuration.dll" "PowerShellVersion"="1.0" "Vendor"="Microsoft" "Version"="15.0.0.0"
After adding the registry key, you can successfully fix the performance counter issue.
Add-PsSnapin Microsoft.Exchange.Management.PowerShell.Setup New-PerfCounters -DefinitionFileName "$exinstall\setup\perf\WorkerTaskFrameworkPerfCounters.xml”
You have an Exchange Server 2016 organization and plan to upgrade to Cumulative Update 10. You log on to an Exchange Server, activate DAG maintenance and prepare the Server Component States for installing the new Cumulative Update.
You open an elevated PowerShell Session and start the Setup using
./Setup.exe /Mode:Upgrade /IAcceptExchangeServerLicenseTerms
Out of a sudden the Exchange Server CU Setup fails while executing setup step Stopping Services with an error:
Microsoft Exchange Server 2016 Cumulative Update 10 Unattended Setup Copying Files... File copy complete. Setup will now collect additional information needed for installation. Languages Management tools Mailbox role: Transport service Mailbox role: Client Access service Mailbox role: Unified Messaging service Mailbox role: Mailbox service Mailbox role: Front End Transport service Mailbox role: Client Access Front End service Performing Microsoft Exchange Server Prerequisite Check Configuring Prerequisites COMPLETED Prerequisite Analysis COMPLETED Configuring Microsoft Exchange Server Preparing Setup COMPLETED Stopping Services FAILED The following error was generated when "$error.Clear(); & $RoleBinPath\ServiceControl.ps1 -Operation:DisableServices -Roles:($RoleRoles.Replace('Role','').Split(',')) -SetupScriptsDirectory:$RoleBinPath; & $RoleBinPath\ServiceControl.ps1 -Operation:Stop -Roles:($RoleRoles.Replace('Role','').Split(',')) -IsDatacenter:([bool]$RoleIsDatacenter) " was run: "Microsoft.PowerShell.Commands.ProcessCommandException: Cannot stop process "fms (2496)" because of the following error: Access is denied ---> System.ComponentModel.Win32Exception: Access is denied at System.Diagnostics.ProcessManager.OpenProcess(Int32 processId, Int32 access, Boolean throwIfExited) at System.Diagnostics.Process.GetProcessHandle(Int32 access, Boolean throwIfExited) at System.Diagnostics.Process.get_HasExited() at Microsoft.PowerShell.Commands.StopProcessCommand.ProcessRecord() --- End of inner exception stack trace ---". The Exchange Server setup operation didn't complete. More details can be found in ExchangeSetup.log located in the <SystemDrive>:\ExchangeSetupLogs folder.
Why would an error like Access Denied happen while executing the Setup.exe as a member of the local Administrators security group from within an elevated PowerShell session?
The PowerShell code executed as part of the CU Setup sets the startup type of Exchange and some Windows services to Disabled. This ensures that in the case of a server reboot an automatic start of the service will not interfere partially executed the setup. After setting the startup type to Disabled the services are stopped.
The services are controlled by the ServiceControl.ps1 script which is located on the Exchange Server installation media in \Setup\ServerRoles\Common\.
The function StopServices stops services using the Stop-Service cmdlet. Due to some timing issues some services are stopped by killing the running processes using Stop-Process -Force.
The services stopped by stopping the running process are:
Executing the Stop-Process cmdlet results in the Access Denied error.
The issue is related to fact that the user account logged on the server and executing the Exchange Server Cumulatice Update does not have the local User Rights Assignment to Debug Programs.
By default the right to debug programs is assigned to the local Adminstrators security group. In secured Active Directory infrastructures the user rights assignments and local security groups are often managed using Group Policy Objects (GPO). The GPOs manage the names of local security groups, group memberships and even user rights assignments.
The client encountering the issue described above hasn't had any issues installing Cumulative Updates for Exchange Server 2013 in the past. So this is solution is related to the setup of Exchange Server 2016 Cumulative Updates on Windows Server 2016 only. If you have any information regarding Exchange Server 2013, let me know using the comments section below.
Enjoy Exchange Server!
The use of Exchange Edge Transport Servers requires the synchronization of user and configuration data from internal Exchange Servers to the Edge Transport Servers. The synchronization utilizes secure LDAP (EdgeSync) to transmit the data securely and is based on an Edge Subscription.
When you create a new Edge Subscription on your internal Exchange Servers by importing the Edge Subscription XML-file, establishing the EdgeSync-connection might fail.
You will find the following error in the application event log of the internal Exchange Server:
Log Name: Application Source: MSExchange EdgeSync Event ID: 1035 Task Category: Synchronization Level: Error Keywords: Classic Description: EdgeSync failed to synchronize because it only supports Cryptographic API certificates. The local Hub Transport server's default certificate with thumbprint XYZ isn't a CAPI certificate. To set a CAPI certificate as the default certificate, use the Enable-ExchangeCertificate cmdlet with the Services parameter using the value of SMTP.
The private key of the current Exchange Transport default certificate of the internal Exchange servers uses a CNG private key. EdgeSync requires a CAPI1 based private key.
This problem occurs primarily when using an Enterprise Certificate Authority using certificate templates with individual template settings.
Get-TransportService | ft Name,InternalTransportCertificateThumbprint
certutil -v -store my > cert.txt
If both attribute are of the value 0, the certificate if a CNG certificate.
The section might look like this:
Unique container name: XYZ Provider = Microsoft Software Key Storage Provider ProviderType = 0 Flags = 20 (32) CRYPT_MACHINE_KEYSET -- 20 (32) KeySpec = 0 -- XCN_AT_NONE
Use OpenSSL to convert the CNG certificate to a CAPI1 certificate.
Using OpenSSL requires the download of the Windows release of OpenSSL. I recommend to not install the software on the Exchange Server but a separate Windows server or your administrative desktop system. Additionally, you need the certificate with its private key as a PFX-file.
Use the following steps to convert the CNG certificate to a CAPI1 certificate.
openssl pkcs12 -in CERT.pfx -out cert.pem -nodes
openssl pkcs12 -export -in cert.pem -out NEWCERT.pfx
The new PFX-file is now a CAPI1 certificate. The new certificate has the same thumbprint. Now you must replace the current certificate used by Exchange Server with the new certificate.
Replacing the certificate requires a downtime of each Exchange Server requiring the certificate replacement. This is due to the requirement to remove the CNG certificate first, following the import of the CAPI1 certificate. Afterward, you need to enable the required Exchange services.
Get-ExchangeCertificate -Server SERVERNAME
# It is mandatory to answer the query for replacing the default certificate with YES Enable-ExchangeCertificate -Thumbprint THUMBPRINT -Services SMTP # Restart the transport service Restart-Service MSExchangeTransport
# It is mandatory to answer the query for replacing the default certificate with YES Enable-ExchangeCertificate -Thumbprint NEWCERTTHUMBPRINT -Services SMTP # Restart the transport service Restart-Service MSExchangeTransport
Now, that you updated the local Exchange Servers there is one more step that needs to be checked on the Edge Transport Servers.
Edge Transport Servers are not domain-joined and therefore do not receive any GPO-based configuration. Each required configuration must be performed locally. To ensure that the default transport certificate of the internal Exchange servers can be used for cryptographic operations we must ensure that the certificate chain of that certificate is present in the certificate store of Edge Transport servers.
Take a look at the certificate chain of the converted CAPI1 certificate and import the Root-CA and Subordinate-CA certificates into the Edge Transport servers local certificate store. You must ensure that the certificates are placed into appropriate stores:
Next, you create a new Edge Subscription on your Edge Transport server and create a new subscription for the Active Directory site on the internal Exchange Server. The internal Exchange Servers are now able to establish an EdgeSync connection and encrypting the data transferred to the Edge Transport servers.
When you receive the TLS certificate as PFX/PKCS12 file, you import the certificate and the private key. The import process itself defines the priavte key Crypto Provider. Using the following command line you ensure that the import process suses the legacy crypto provider.
certutil -csp "Microsoft RSA SChannel Cryptographic Provider" -importpfx my MYCERTpfx
Enjoy Exchange Server and Edge Transport!
Are you located in Germany, Austria, or Switzerland? Join the Exchange User Group DACH to collaborate with other Exchange enthusiasts. Follow us on Twitter @exusg, join on Meetup, or visit our website.
I came across an interesting issue when setting up a new Exchange 2013 server in an Exchange organization having the cmdlet extension agent enabled.
As mentioned in my last post Exchange setup checks for the existence of the ScriptingAgentConfig.xml file when agent extenstion is enabled in the Exchange organization. It turned out that this ist not only true when you install an Exchange update using /mode:update, but as well when installing a new Exchange server using /mode:install.
The following error occurs when Exchange Management Tools are provisioned.
Configuring Microsoft Exchange Server Preparing Setup COMPLETED Stopping Services COMPLETED Copying Exchange Files COMPLETED Language Files COMPLETED Restoring Services COMPLETED Language Configuration COMPLETED Exchange Management Tools FAILED The following error was generated when "$error.Clear(); Set-WERRegistryMarkers; " was run: "Microsoft.Exchange.Provisioning.ProvisioningBrokerException: Provisioning layer initialization failed: '"Scripting Agent initialization failed: "File is not found: 'C:\Program File s\Microsoft\Exchange Server\V15\Bin\CmdletExtensionAgents\ScriptingAgentConfig.xml'.""' ---> Microso ft.Exchange.Provisioning.ProvisioningException: "Scripting Agent initialization failed: "File is not found: 'C:\Program Files\Microsoft\Exchange Server\V15\Bin\CmdletExtensionAgents\ScriptingAgentConf ig.xml'."" ---> System.IO.FileNotFoundException: "File is not found: 'C:\Program Files\Microsoft\Exc hange Server\V15\Bin\CmdletExtensionAgents\ScriptingAgentConfig.xml'." at Microsoft.Exchange.ProvisioningAgent.ScriptingAgentConfiguration.Initialize(String xmlConfigPa th) at Microsoft.Exchange.ProvisioningAgent.ScriptingAgentConfiguration..ctor(String xmlConfigPath) --- End of inner exception stack trace --- at Microsoft.Exchange.ProvisioningAgent.ScriptingAgentConfiguration..ctor(String xmlConfigPath) at Microsoft.Exchange.ProvisioningAgent.ScriptingAgentClassFactory.get_Configuration() at Microsoft.Exchange.ProvisioningAgent.ScriptingAgentClassFactory.GetSupportedCmdlets() at Microsoft.Exchange.Provisioning.ProvisioningBroker.BuildHandlerLookupTable(CmdletExtensionAgen t[] enabledAgents, Exception& ex) --- End of inner exception stack trace --- at Microsoft.Exchange.Provisioning.ProvisioningLayer.GetProvisioningHandlersImpl(Task task) at Microsoft.Exchange.Provisioning.ProvisioningLayer.GetProvisioningHandlers(Task task) at Microsoft.Exchange.Configuration.Tasks.Task.<BeginProcessing>b__4() at Microsoft.Exchange.Configuration.Tasks.Task.InvokeNonRetryableFunc(Action func, Boolean termin atePipelineIfFailed)".
As expected a fresh Exchange install contains the sample file only. The following screenshot shows the Exchange Management Shell and the releated folder in the background.
The only solution currently known to me is to disable the cmdlet extension agent until the setup of the new Exchange server has finished.
Disable-CmdletExtensionAgent "Scripting Agent"
Having the cmdlet extension agent disabled the setup finishes without any issues. Don't forget to copy the cmdlet extension Xml file to the newly built server and to enable the cmdlet extension agent again.
Enable-CmdletExtensionAgent "Scripting Agent"
You need assistance with your Exchange Server setup? You have questions about your Exchange Server infrastructure and going hybrid? You are interested in what Exchange Server 2016 has to offer for your environment?