I was involved in a troubleshooting request for a hybrid mail flow issue. Before I take a closer look at the issue, let's talk about the hybrid setup.
A managed service provider runs separated on-premises Exchange Organizations for various clients. Also, the service provider runs it's own Exchange Organization in a hybrid setup with Exchange Online (EXO) utilizing centralized mail flow. Let's name the managed service provider Varunagroup, using the primary domain varunagroup.de.
The on-premises IT-Infrastructure consists of the following email components:
The following diagram illustrates the setup and the expected mail flow.
Let's name one of the clients Setebos AG, using setebos-ag.com as their primary domain.
Varunagroup's IT department activated journaling in Exchange Online, using an on-premises Journaling mailbox. After a few days, an IT administrator checked the inbox folder for journaling messages and journaling reports. The journaling inbox did not contain messages of Varounagroup senders or recipients only, but messages from client sender domains, e.g., setebos-ag.com.
In reality, the mail flow from on-premises to external recipients from any of the local Exchange organizations looked like shown in this diagram.
Why does the Variangoup journaling mailbox contain messages from Setebos senders sent to external recipients?
We choose a single message for troubleshooting purposes, originating from the Setebos.com domain, sent to a non-Varunagroup recipient.
The interesting piece of information is row 6.
You see that EXO resolves the target mail exchanger via DNS. The target is another Microsoft 365 tenant as we see an xxx.mail.protection.outlook.com host.
When checking the on-premises mail gateway connection log, we found the distracting information that the gateway resolved the target mail exchanger as xxx.mail.protection.outlook.com.
As a next step, we checked the extended message tracking log using the new Exchange Admin Center. We created a new custom query with the following search criteria:
When you troubleshoot connection issues with Exchange Online, always select the extended report. You'll receive the report as a CSV file attachment. Use the Data tab in Excel to import the CSV file. Do not access the content by simply clicking the received file attachment.
The interesting information is stored in the custom_data column for row source=SMTP and event_id=RECEIVE.
S:ProxyHop1=HE1EUR01FT049.mail.protection.outlook.com(10.152.0.221); S:ProxyHop2=AM0PRxxCAxxxx.outlook.office365.com(2603:10a6:208:fa::40); S:InboundConnectorData=Name=Inbound from [EXCHANGE ORG GUID]; ConnectorType=OnPremises; TenantId=[VARUNAGROUP GUID]; S:InboundTlsDetails=TLS=SP_PROT_TLS1_2_SERVER [...]; S:CorrelationId=d9ac6a10-8de9-4308-4205-07d865e8909b; S:MimeParts=Att/Emb/MPt:0/0/1; S:MessageValue=MediumHigh; S:Replication=AM6PRxxxxMBxxxx; S:FirstForestHop=AM0PRxxxxMBxxxx.eurprd03.prod.outlook.com; S:FromEntity=HybridOnPrem; S:Oorg=varunagroup.de; S:ProxiedClientIPAddress=81.173.212.44; S:ProxiedClientHostname=mx01.varunagroup.de; S:DeliveryPriority=Normal; S:AccountForest=EURPRxxAxxx.PROD.OUTLOOK.COM
The information in line 3 shows the actual name of the configured Varunagroup inbound connector, as shown in the Exchange Online connector configuration. The message did not enter the Varunagroup EXO tenant due to a mysterious connection, it was received by the dedicated inbound connector, configured by HCW.
The key to this question is the TLS certificate used by the centralized email gateway and the TLS common name filtering in Exchange Online.
The wildcard name *.varunagroup.de resulted in a matching string comparison for the incoming TLS common names of mx01.varunagroup.de and mx02.varunagroup.de. At the same time, the inbound connector matched the Edge Transport TLS certificate smtpo365.varunagroup.de.
Nobody knew, how the inbound connector configuration got "changed" to the wildcard name or for how long that configuration resulted in outbound messages from customer domains routed via the service provider tenant.
The solution contains two configurations.
The TLS common name behavior is by design and described in this blog post as FAQ #6(b). As a customer, you identify this as a misbehaving SMTP receive connector. But as described in the blog post, this is by design.
It is required that you understand the inbound routing behavior of Exchange Online if you have complicated outbound routing requirements. The blog post provides detailed information on how Office 365 inbound routing works and what you should be aware of.
Links
Enjoy Exchange Online.
Are you located in Germany, Austria, or Switzerland? Join the Exchange User Group DACH to collaborate with other Exchange enthusiasts. Follow us on Twitter @exusg, join on Meetup, or visit our website.
Troubleshooting Outlook connectivity issues with Office 365 is tricky. Administrators can use two valuable tools provided by Microsoft to identify and even fix client related connectivity issues.
Start with the Outlook account problems test page in the Office 365 portal. You need to log on as the Office 365 user having issues.
The site tests for the following:
If no issues are identified after you've logged on to Office 365, move to the next step.
The Microsoft Support and Recovery Assistant (SARA) for Office 365 is click to run tool that is installed and executed locally.
These two tools fix most of the Outlook connectivity issues you are facing as an Office 365 administrator.
Enjoy Office 365
The SMTP Simulator project has been started due to a specific demand during a customer project. We needed a solution to test native transport of Exchange Server 2013 and third party addons to Exchange in an isolated lab envrionment having no internet access at all.
While it is pretty easy to send test emails using PowerShell, we wanted to create an automated service which is capable of:
The SMTP Simulator can be used with any Message Transfer Agents (MTA), not only with Exchange. Besides testing the MTA itself, we needed to test some of the following third-party solutions:
The Visual Studio solution creates a MSI installer file. The MSI package created installs the SMTP Service itself, but not the required web application (see issue #49).
Documentation is provided by the SMTP Simulator Wiki.
The code has been published as open source at Github. Feel free to fork the solution and contribute to the code.
Report any issues or feature requests at Github.
The project still has some open ends and needs some love and attention. Open issues are part of the issue tracker at Github.
Main topics are:
You have an Exchange Server 2016 organization and plan to upgrade to Cumulative Update 10. You log on to an Exchange Server, activate DAG maintenance and prepare the Server Component States for installing the new Cumulative Update.
You open an elevated PowerShell Session and start the Setup using
./Setup.exe /Mode:Upgrade /IAcceptExchangeServerLicenseTerms
Out of a sudden the Exchange Server CU Setup fails while executing setup step Stopping Services with an error:
Microsoft Exchange Server 2016 Cumulative Update 10 Unattended Setup Copying Files... File copy complete. Setup will now collect additional information needed for installation. Languages Management tools Mailbox role: Transport service Mailbox role: Client Access service Mailbox role: Unified Messaging service Mailbox role: Mailbox service Mailbox role: Front End Transport service Mailbox role: Client Access Front End service Performing Microsoft Exchange Server Prerequisite Check Configuring Prerequisites COMPLETED Prerequisite Analysis COMPLETED Configuring Microsoft Exchange Server Preparing Setup COMPLETED Stopping Services FAILED The following error was generated when "$error.Clear(); & $RoleBinPath\ServiceControl.ps1 -Operation:DisableServices -Roles:($RoleRoles.Replace('Role','').Split(',')) -SetupScriptsDirectory:$RoleBinPath; & $RoleBinPath\ServiceControl.ps1 -Operation:Stop -Roles:($RoleRoles.Replace('Role','').Split(',')) -IsDatacenter:([bool]$RoleIsDatacenter) " was run: "Microsoft.PowerShell.Commands.ProcessCommandException: Cannot stop process "fms (2496)" because of the following error: Access is denied ---> System.ComponentModel.Win32Exception: Access is denied at System.Diagnostics.ProcessManager.OpenProcess(Int32 processId, Int32 access, Boolean throwIfExited) at System.Diagnostics.Process.GetProcessHandle(Int32 access, Boolean throwIfExited) at System.Diagnostics.Process.get_HasExited() at Microsoft.PowerShell.Commands.StopProcessCommand.ProcessRecord() --- End of inner exception stack trace ---". The Exchange Server setup operation didn't complete. More details can be found in ExchangeSetup.log located in the <SystemDrive>:\ExchangeSetupLogs folder.
Why would an error like Access Denied happen while executing the Setup.exe as a member of the local Administrators security group from within an elevated PowerShell session?
The PowerShell code executed as part of the CU Setup sets the startup type of Exchange and some Windows services to Disabled. This ensures that in the case of a server reboot an automatic start of the service will not interfere partially executed the setup. After setting the startup type to Disabled the services are stopped.
The services are controlled by the ServiceControl.ps1 script which is located on the Exchange Server installation media in \Setup\ServerRoles\Common\.
The function StopServices stops services using the Stop-Service cmdlet. Due to some timing issues some services are stopped by killing the running processes using Stop-Process -Force.
The services stopped by stopping the running process are:
Executing the Stop-Process cmdlet results in the Access Denied error.
The issue is related to fact that the user account logged on the server and executing the Exchange Server Cumulatice Update does not have the local User Rights Assignment to Debug Programs.
By default the right to debug programs is assigned to the local Adminstrators security group. In secured Active Directory infrastructures the user rights assignments and local security groups are often managed using Group Policy Objects (GPO). The GPOs manage the names of local security groups, group memberships and even user rights assignments.
The client encountering the issue described above hasn't had any issues installing Cumulative Updates for Exchange Server 2013 in the past. So this is solution is related to the setup of Exchange Server 2016 Cumulative Updates on Windows Server 2016 only. If you have any information regarding Exchange Server 2013, let me know using the comments section below.
Enjoy Exchange Server!
Recently I was facing an issue where Windows Server 2012 R2 reported remaining 22% of free disk space of one of the Exchange Server data volumes. The Exchange Server data volumes are connected using mount points.
Before trying to identify any issues in regards to hidden system files or streams, I checked the volume shadow copy configuration using the Disk Management MMC.
Windows Disk Management showed that volume C: was using a mounted volume as shadow copy target.
Examining the available disk space using the Get-Diskspace.ps1 PowerShell script, it showed that WMI was reporting the FreeSpace the same way as the Disk Management.
[PS] D:\SCRIPTS\Get-Diskspace>.\Get-Diskspace.ps1 -ComputerName EX01 Fetching Volume Data from EX01 Name Capacity (GB) FreeSpace (GB) BootVolume SystemVolume FileSystem ---- ------------- -------------- ---------- ------------ ---------- E:\ExchangeDatabases\DatabaseSet1\ 2048 1083 False False NTFS E:\ExchangeDatabases\DatabaseSet2\ 2048 445 False False NTFS E:\ExchangeDatabases\DatabaseSet3\ 2048 1219 False False NTFS E:\ExchangeDatabases\DatabaseSet4\ 2048 1091 False False NTFS
The only viable solution to reclaim the wasted disk space was to remove the shadow copy of volume C.
First I checked the current list of shadow copies using the vssadmin command line tool.
D:\>vssadmin list shadows vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool (C) Copyright 2001-2013 Microsoft Corp. Contents of shadow copy set ID: {eb09ce08-f8a6-47ea-b48d-2d6da7591d4e} Contained 1 shadow copies at creation time: 23.06.2017 16:05:56 Shadow Copy ID: {3684b224-bca2-42c4-a0b3-43b7d0db2d96} Original Volume: (C:)\\?\Volume{df40ac48-f610-11e3-80ce-806e6f6e6963}\ Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy1 Originating Machine: ex01.granikossolutions.eu Service Machine: ex01.granikossolutions.eu Provider: 'Microsoft Software Shadow Copy provider 1.0' Type: ApplicationRollback Attributes: Persistent, No auto release, Differential
The vssadmin tool does not clearly state the path to the shadow copy volume. Therefore, it is much more convenient to identify the shadow copy target using Disk Management. But the output shows that the shadow copy is nearly six months old. So it's safe to delete this orphaned shadow copy.
You can easily delete all shadow copies of a selected volume using the following command
vssadmin delete shadows /for=c: /all
But it turned out that the shadow copy could not be deleted, even with administrative credentials in use.
D:\>vssadmin delete shadows /for=c: /all vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool (C) Copyright 2001-2013 Microsoft Corp. Error: Snapshots were found, but they were outside of your allowed context. Try removing them with the backup application which created them.
Needless to say, that a well-known third party solution is in use to backup the servers. The shadow copy remainers are copies created by the backup solution and were not properly removed after backup due to a system failure during backup.
But how can you remove the current shadow copy without tempering the exisiting permissions for your account?
Simply use the Disk Management MMC to modifx the current shadow copy configuration and the shadow copy is removed.
Switch to the command line and check for existing shadow copies.
D:\>vssadmin list shadows vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool (C) Copyright 2001-2013 Microsoft Corp. No items found that satisfy the query.
The oprhaned shadow copy is gone.
Now open the Settings windows for the source volume again and change the Located on this volume to be the same as the source volume anfd change the Use Limit to the same value for the volume that is configured on other servers.
Enjoy Volume Shadow Copies
When you run your Exchange Organization in hybrid mode with Office 365 and you migrate your on-premise Public Folders to Office 365, you are required to configure a remote Public Folder Mailbox in the Exchange Organization settings.
With Public Folders on-premise your Exchange Online Org looks like this:
Get-OrganizationConfig | fl *public* DefaultPublicFolderAgeLimit : DefaultPublicFolderIssueWarningQuota : 1.7 GB (1,825,361,920 bytes) DefaultPublicFolderProhibitPostQuota : 2 GB (2,147,483,648 bytes) DefaultPublicFolderMaxItemSize : Unlimited DefaultPublicFolderDeletedItemRetention : 30.00:00:00 DefaultPublicFolderMovedItemRetention : 7.00:00:00 PublicFoldersLockedForMigration : False PublicFolderMigrationComplete : False PublicFoldersEnabled : Remote PublicComputersDetectionEnabled : False RootPublicFolderMailbox : 00000000-0000-0000-0000-000000000000 RemotePublicFolderMailboxes : {PublicFolder-Mailbox001}
With Public Folders on-premise your On-Premise Exchange Org looks like this:
Get-OrganizationConfig | fl *public* DefaultPublicFolderAgeLimit : DefaultPublicFolderIssueWarningQuota : Unlimited DefaultPublicFolderProhibitPostQuota : Unlimited DefaultPublicFolderMaxItemSize : Unlimited DefaultPublicFolderDeletedItemRetention : 30.00:00:00 DefaultPublicFolderMovedItemRetention : 7.00:00:00 PublicFoldersLockedForMigration : True PublicFolderMigrationComplete : True PublicFoldersEnabled : Local PublicComputersDetectionEnabled : False RootPublicFolderMailbox : ae0ef819-90d2-45d0-92b6-8b2062cf71a3 RemotePublicFolderMailboxes : {}
With Public Folders in Exchange Online your Exchange Online Org looks like this:
Get-OrganizationConfig | fl *public* DefaultPublicFolderAgeLimit : DefaultPublicFolderIssueWarningQuota : 1.7 GB (1,825,361,920 bytes) DefaultPublicFolderProhibitPostQuota : 2 GB (2,147,483,648 bytes) DefaultPublicFolderMaxItemSize : Unlimited DefaultPublicFolderDeletedItemRetention : 30.00:00:00 DefaultPublicFolderMovedItemRetention : 7.00:00:00 PublicFoldersLockedForMigration : False PublicFolderMigrationComplete : False PublicFoldersEnabled : Local PublicComputersDetectionEnabled : False RootPublicFolderMailbox : 5810bb30-cdda-4287-85a4-8a2547bb9b01 RemotePublicFolderMailboxes : {}
With Public Folders in Exchange Online your Exchange On-Premise Org looks like this:
Get-OrganizationConfig | fl *public* DefaultPublicFolderAgeLimit : DefaultPublicFolderIssueWarningQuota : Unlimited DefaultPublicFolderProhibitPostQuota : Unlimited DefaultPublicFolderMaxItemSize : Unlimited DefaultPublicFolderDeletedItemRetention : 30.00:00:00 DefaultPublicFolderMovedItemRetention : 7.00:00:00 PublicFoldersLockedForMigration : True PublicFolderMigrationComplete : True PublicFoldersEnabled : Remote PublicComputersDetectionEnabled : False RootPublicFolderMailbox : 00000000-0000-0000-0000-000000000000 RemotePublicFolderMailboxes : {mcsmemail.de/Users/PF365-Mailbox-01-55e3d544-ed5a-4557-9008-d8c1b6f06d86}
The remote public folder mailbox has been added to the on-premise Exchange confguration by using:
Set-OrganizationConfig -RemotePublicFolderMailboxes PF365-Mailbox-001 -PublicFoldersEnabled Remote
To be able to add the remote public folder mailbox in a hybrid configuration you are required to add the public folder mailbox (or mailboxes, if you have more than one serving the hierarchy) as a mail user.
Microsoft provides a PowerShell script as part of a script collection here.
When you run the Import-PublicFolderMailboxes.ps1 script you might run into the following error:
Cannot bind parameter 'Name' to the target. Exception setting "Name": "The length of the property is too long. The maximum length is 64 and the length of the value provided is 65." + CategoryInfo : WriteError: (:) [New-MailUser], ParameterBindingException + FullyQualifiedErrorId : ParameterBindingFailed,Microsoft.Exchange.Management.RecipientTasks.NewMailUser + PSComputerName : ex2013.mcsmemail.de
The name attribute for a mail user is limited to 64 characters. But why are you exceeding the length when the mailbox name is only 17 characters long?
It turns out that the PowerSheel script adds a prefix name "" and ther mailbox GUID as a suffix. And voilá, the name exceeds the allowed length for the mail user name attribute.
Don't use more than 16 characters when naming the Public Folder mailboxes in Office 365.
Or modify the Import-PublicFolderMailboxes.ps1 script to fit your needs.
$hasPublicFolderServingHierarchy = $true; $displayName = $publicFolderMailbox.Name.ToString().Trim(); # ORIG: $name = "RemotePfMbx-" + $displayName + "-" + [guid]::NewGuid(); $name = $displayName + "-" + [guid]::NewGuid(); $externalEmailAddress = $publicFolderMailbox.PrimarySmtpAddress.ToString();
You need assistance with your Exchange Server setup? You have questions about your Exchange Server infrastructure and going hybrid? You are interested in what Exchange Server 2016 has to offer for your environment? Contact me at thomas@mcsmemail.de Follow at https://twitter.com/stensitzki
This script deletes all Exchange and IIS logs older than X days from all Exchange 2013+ servers that are fetched using the Get-ExchangeServer cmdlet.
The Exchange log file location is read from the environment variable and used to build an administrative UNC path for file deletions.
Optionally, you can use the Active Directory configuration partition to determine the Exchange install path dynamically, if supported in your Active Directory environment.
# EXAMPLE 1 # Delete Exchange and IIS log files older than 14 days .\Purge-LogFiles -DaysToKeep 14 # EXAMPLE 3 # Delete Exchange and IIS log files older than 7 days with automatic discovery .\Purge-LogFiles -DaysToKeep 7 -Auto # EXAMPLE 3 # Delete Exchange and IIS log files older than 7 days with automatic discovery and send email report .\Purge-LogFiles -DaysToKeep 7 -Auto -SendMail -MailFrom postmaster@sedna-inc.com -MailTo exchangeadmin@sedna-inc.com -MailServer mail.sedna-inc.com # EXAMPLE 4 # Delete Exchange and IIS log files older than 14 days, but copy files to a central repository and compress the log files before final deletion .\Purge-LogFiles -DaysToKeep 14 -RepositoryRootPath \\OTHERSERVER\OtherShare\LOGS -ArchiveMode CopyZipAndDelete # EXAMPLE 5 # Delete Exchange Server, IIS, and HTTPERR log files older than 7 days, and send an HTML email. Identify Exchange file paths using AD configuration objects. .\Purge-LogFiles.ps1 -DaysToKeep 7 -SendMail -MailFrom postmaster@sedna-inc.com -MailTo exchangeadmin@sedna-inc.com -MailServer mail.sedna-inc.com -UseDynamicExchangePaths -IncludeHttpErr
Brian Reid, C7 Solutions, http://www.c7solutions.com/2013/04/removing-old-exchange-2013-log-files-html