Fixing Broken BackEnd Site

Hi folks,

I would love to share with you my recent experience that I had with Remote PowerShell on an Exchange 2016 box. As you well know since Exchange 2013 Exchange server is using 2 web sites in IIS: Default WebSite and BackEnd. The prior is listening on ports 80 and 443 while the latter is listening on ports 81 and 444.

In my case when I launched Exchange Management Shell I was greeted with the below error:

My investigation led me to the following error events in System log

And Application log:

The errors were pointing to the BackEnd site where actual data rendering and processing for all protocols including PowerShell. As I was looking for solution of this problem this post and this TechNet article. According to it we need to retrieve information of all certificates that are used by IIS. We will need to use the below command for it.

netsh http show sslcert

You will get output as below:

When retrieved we will need to record certificate hash and application ID (appid) for 127.0.0.1:443. Usually I prefer to dump the above command to a test file to make it easier. Also I was not able to find a certificate which was listed for 0.0.0.0:444 in the certificate store of the affected server. So the certificate needs to be removed and replace with the existing one which is used for the Default Web Site.

After this  you will need to delete cert for the back-end site represented by 0.0.0.0:444 by running the below command:

netsh http delete sslcert ipport=0.0.0.0:444

After cert has been removed we will need to configure certificate for our back-end web site to use the same certificate that is used  by the Default Web Site. In the output above it is presented as 127.0.0.1:443.
Let's imagine that certificate hash for 127.0.0.1:443  is 1234567890abcdef3456787asabaec4e8ba and application id is  "{1abc2e345-a14b-4c22-b022-59fc885b0974}"

netsh http add sslcert ipport=0.0.0.0:444 certhash=1234567890abcdef3456787asabaec4e8ba appid="1234567890abcdef3456787asabaec4e8ba"{1abc2e345-a14b-4c22-b022-59fc885b0974}"

Make sure that all the brackets and quotes are in place.

I hope you will find this article helpful.

Enjoy!

Exporting Mailboxes in Multi-Domain Environment

Hi folks,

Just wanted to share with you about an issue I have recently encountered. In the AD forest which has multiple domains when remotely connected to Exchange server via PowerShell and running New-MailboxExportRequest command you may experience the issue as below:

Basically, PowerShell will complain that it can't find mailbox on a DC from the same domain as Exchange server is. And this is because that by default EMS is by default talking to the same domain where Exchange servers are located and where it was launched. As I previously posted to see mailboxes cross forest you will need to execute this command:

Set-ADServerSettings -ViewEntireForest:$true

However, in this case it doesn't become a magic bullet to solve the problem.

Google-ing took me to this thread on MS TechNet forum. According to it in addition to seeing the whole forest you also need to talk to a domain controller in a child domain where your mailbox is located. So imagine and in the contoso.com forest you have a domain child.contoso.com which has Exchange mailboxes the code for exporting mailbox to PST file will be as follows;

Set-ADServerSettings -ViewEntireForest:$true
New-MailboxExportRequest user@child.contoso.com –FileName \\servername\pst\test.pst -DomainController DC01.child.contoso.com

I hope you will find it useful.

Enjoy!

And Back to the Subject of ActiveSync Devices Reporting

Hi folks,

I have already shared with you about creating reports for Exchange ActiveSync devices using PowerShell. You can read this post about it.

I have recently discovered this reporting script for reporting ActiveSync devices published here by Paul Cunningham. All the information about it can be found in the article as well as in this Paul's post. The script is amazing and perfectly fits the purpose.

However, there're couple of the things that are missing in the script. First, if you have a multi-domain forest it is possible that not all of your mailboxes are retrieved and added to the report. To fix it you will need to edit the script and add the following line:

Set-AdServerSettings -ViewEntireForest:$true

I have added this line after the section which loads Exchange 2013 management shell as below:

This script is also good for creating ActiveSync reports from Office 365. However when trying to run this from my Windows 10 laptop I would be greeted with error like below:

This is quite easy to fix. Simply remove the whole section which loads on-premises EMS and run script from the remote PowerShell session connected to Office 365 endpoint. To be clear this is what you need to remove from the script to run it successfully against the cloud:

Enjoy!

Creating Reports from Admin Audit Logs

Hi folks,

In this post I would love to share with you some of experience that I had with Audit logs in Exchange. You well know about admin audit and mailbox audit logs available in Exchange and their usage so I won't spend too much time discussing them, but rather jump into some examples of retrieving info from admin audit logs.

So I once was requested about preparing report on who removed user mailboxes and also who added extra permissions to them. The 2 below lines provide code which retrieve when the commands which remove and disable mailboxes and also that assign permissions to the mailbox and also who ran them and when.

Search-AdminAuditLog -Cmdlets Disable-Mailbox, Remove-Mailbox -StartDate 09/20/2016 -EndDate 09/22/2016 |select ObjectModified,CmdletName,@{N="CmdletParameters";E={$_.CmdletParameters}},Caller,ExternalAccess,Succeeded,RunDate,OriginatingServer,ObjectState |export-csv C:\Reports\Lost-MBX-Report.csv


Search-AdminAuditLog -Cmdlets Add-MailboxPermission, Add-ADPermission -StartDate 11/18/2016 -EndDate 11/21/2016 |select ObjectModified,CmdletName,@{N="CmdletParameters";E={$_.CmdletParameters}},Caller,ExternalAccess,Succeeded,RunDate,OriginatingServer,ObjectState |export-csv C:\Reports\AD-Perm-Report.csv

There is also a nicer way to create reports from Admin Audit Logs and this article was very helpful as it points to this script which allows creating quite a nice and readable reports. Imagine you were requested to generate a report on which databases mailboxes have been moved. You will need to execute the code below which will generate reports similar to the ones that are generated by one-liners above:

$AdmSearch = Search-AdminAuditLog -Cmdlets new-moverequest,New-MigrationBatch
$AdmSearch | .\Get-SimpleAuditLogReport.ps1 –agree |Export-csv MigrationCmdlets.csv

The most valuable information is that the full command is being reported which gives you a good picture what was exactly happening in your environment.

Of course you can modify your request for admin log search to anything you like or need. Just use this article for more information.

Enjoy

Reinstalling Failed DAG Node from Scratch

Hi folks,

Just wanted to share with you about another scenario which you may face when you work with DAGs. Sometimes you can have your Exchange servers dead which you need to reinstall . There're 2 options of doing it. The first one is using /RecoverServer parameter while in the other case you will need to scrap all the Exchange server related information from AD. In this post I will cover the second scenario.

First, you will need to remove all the database copies from the failed server. You will need to run command as below:

Get-MailboxDatabaseCopyStatus -Server Server01 |foreach {Remove-MailboxDatabaseCopy -Identity $_.Name}

After this server should be removed from DAG. Since server is offline and can't be accessed by the cluster service you will need to use the -ConfigurationOnly parameter and command will be as below:

Remove-DatabaseAvailabilityGroupServer -Identity DAG01 -MailboxServer Server01 -Confirm:$false -ConfigurationOnly

After this you will need to verify that the server has been indeed removed from DAG by running the below command:

Get-DatabaseAvailabilityGroup DAG01 |select Name,Servers |fl

After DAG server has been removed it should be evicted from cluster:

Get-ClusterNode Server01 |Remove-ClusterNode -Force

Finally, we will need to clean up AD object for Exchange server. For this purpose ADSIEDIT tool will be needed. You will need to connect ADSIEDIT to Configuration partition and then navigate to the server object, something like below:

CN=Server01,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=CONTOSO,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=contoso,DC=com

Delete server object and click Yes for deleting container object. Please don't forget to be extremely careful when using ADSIEDIT, because one wrong mouse click can destroy your Exchange org and you will need to restore AD from backup authoritatively which is not a lot of fun.

Finally, in this scenario we will need to delete computer object of a dead Exchange server. Merely resetting password for it is not enough. And this is because when Exchange is installed it registers a bunch of Exchange-related SPN records. You can see it as below:

setspn -l SERVER01
Registered ServicePrincipalNames for CN=SERVER01,OU=Servers,DC=contoso,DC=com
:
        MSServerClusterMgmtAPI/SERVER01.contoso.com
        MSServerClusterMgmtAPI/SERVER01
        IMAP/SERVER01
        IMAP/SERVER01.contoso.com
        IMAP4/SERVER01
        IMAP4/SERVER01.contoso.com
        POP/SERVER01
        POP/SERVER01.contoso.com
        POP3/SERVER01
        POP3/SERVER01.contoso.com
        exchangeRFR/SERVER01
        exchangeRFR/SERVER01.contoso.com
        exchangeAB/SERVER01
        exchangeAB/SERVER01.contoso.com
        exchangeMDB/SERVER01
        exchangeMDB/SERVER01.contoso.com
        SMTP/SERVER01
        SMTP/SERVER01.contoso.com
        SmtpSvc/SERVER01
        SmtpSvc/SERVER01.contoso.com

If brand new computer object is not created Exchange installation may fail or in case of its success, services may fail starting, therefore I highly insist that it is deleted before the OS is re-imaged.

I hope you will find this post helpful for your dealings with Exchange servers.

Enjoy!

RBAC Effective Permissions

Hi folks,

You are well aware with the effective permissions in the NTFS and also on other objects that are controlled by ACLs (like AD objects or registry).

Now the same kind of check and report is available for Exchange RBAC. What it does, it checks all the role group membership of an admin account and then provides as an output roles that they were assigned to.

The below code should do this magic:

Get-ManagementRoleAssignment -GetEffectiveUsers | Where-Object {$_.EffectiveUserName -eq "Admin"} | Select-Object Role

And it provides code as below:

Enjoy!

A Great and Easy Way to Retrieve Average Message Size

Hi folks,

I have recently been working on one sizing exercise for Exchange 2016. In particular I needed to size servers that will be used for email routing only, no storage of mailboxes whatsoever. As you well know in Exchange 2013 and what's more on 2016 transport is no longer a separate role, however there are no tools available for sizing transport only servers.

The key parameter for any Exchange calculations is an average message size. Previously Profile Analyzer was a very helpful tool to achieve this purpose. As you know Profile Analyzer no longer works for Exchange 2010 and later. Microsoft has released this nice tool as a replacement of the Profile Analyzer, however it doesn't work against sites without mailboxes.

Therefore the only way to get average mail size on the transpport only servers is to use message tracking log. After some searching I have found this great article that came to my help.

So if you have only a single site with the Exchange calculating average size is pretty easy. The following code needs to be executed to achieve this:

For Exchange 2007 and 2010:

Get-TransportServer | Get-MessageTrackingLog -resultsize unlimited | measure-object -Property TotalBytes -Maximum –Average

For Exchange 2013 and 2016:

Get-TransportService | Get-MessageTrackingLog -resultsize unlimited | measure-object -Property TotalBytes -Maximum –Average

Nice and easy, isn't it.

Now, let's imagine we will need to calculate average size of all the messages that went trough the server over the course of a business week and that we have more than one AD site. We will need to change our query to all servers within a site, let's call it NEWYORK. the script will look something like this:

$TrpSrv = Get-ExchangeServer  |Where-Object {$_.Site -like "*NEWYORK*"} | |Get-TransportService

$TrpSrv |foreach {Get-MessageTrackingLog -Server $_.Name -Start "01/09/2017 09:00:00" -End "01/13/2017 17:00:00" -resultsize unlimited | measure-object -Property TotalBytes -Maximum –Average |out-file D:\Scripts\Average.txt -append}

Alternatively you will need to run the same command for each individual server in a site and then use tools like calculator or Excel to calculate average size of the message. In this case the code will be like below:

Get-TransportService SERVER01 | Get-MessageTrackingLog -Start "01/09/2017 09:00:00" -End "01/13/2017 17:00:00" -resultsize unlimited | measure-object -Property TotalBytes -Maximum –Average |out-file D:\Scripts\Average.txt

I hope you will find this post useful for your sizing adventures.

Enjoy!