Figting High CPU Usage on Mailbox Servers Caused By Uneven Spread of Mailbox Database Copies

Hi folks,

Sounds like a time to share my recent adventure.

I have recently run into an interesting issue. On 2 of servers that are the part of Database Availability Group (DAG) one was running at the top of CPU. It would never go below 70% which is not healthy. At the same time its pair hosting copies was consuming only 20-30% of CPU.

I had to resolve such injustice and make sure that CPU usage is equal and within norm on both of the pairs. First of all I needed to determine how many of copies each of the pairs has mounted and why are they mounted this way.

This is where my 2 good friends came into play e.g. PowerShell and Microsoft Excel.

First of all I ran against each server hosting the copy to make sure that this number is equal:

(Get-MailboxDatabase -Server MbxSrvHostname).count

An it actually was equal.

Now I wanted to understand how copies are mounted on both servers and why are they mounted there. I ran the following command (please don't do it yourself):

Get-MailboxDatabase -Server ServerName |select Name,Server,Servers,Activation* |export-csv D:\Software\DBCopyN4P4.csv. 

And the result was quite ugly and unreadable.

I had to tweak output for Servers and ActivationPreference parameters as follows to ensure the proper output: @{N="Servers";E={$_.Servers}},@{N="ActivationPreference";E={$_.ActivationPreference}}.

My final command for outputting databases and their copies and preferences turned to be as follows:

Get-MailboxDatabase -Server ServerName |select Name,Server,@{N="Servers";E={$_.Servers}},@{N="ActivationPreference";E={$_.ActivationPreference}} |export-csv D:\Software\DBCopyN4P4.csv

I won't be able to show the output here due to ethics, but I can guarantee that this is working.

Now the time came to redistribute mailbox database copies across both members of the pair. To make my life easier I generated CSV file that contains the list of the database copies and in particular their names which I needed for configuration command:

Get-MailboxDatabase -Server ServerName |Get-MailboxDatabaseCopyStatus |Select Name |Sort Name |Export-Csv D:\Software\DbCopy.csv

I have got output as below

I have done some processing with the file. First, I removed ugly 1st row. Secondly I have added column and added preference numbers for database copies in the following format: 1-2 2-1. This is because copies were sorted in CSV file by name and the copy on the 1st server was always listed before the one on the 2nd.

Something like

Name	Preference
DB01\SERVER01	1
DB01\SERVER02	2
DB02\SERVER01	2
DB02\SERVER02	1

Or

After that the time for magic came. I have reconfigured activation of every database copy using CSV file I had created in the previous step. I imported it into pipeline and then executed Set-MailboxDatabaseCopy command against every copy in the file. Something like this:

Import-Csv D:\Software\DbCopyPref.csv |foreach {Set-MailboxDatabaseCopy -Identity $_.Name -ActivationPreference $_.Preference}

After this you can use either Move-ActiveMailboxDatabase command or RedistributeActiveDatabases.ps1 from Exchange scripts to redistribute copies based on their preference. Because I was dealing only with 2 mailbox servers from DAG and didn't want other copies/servers to be affected  I was using Move-ActiveMailboxDatabase against every mailbox database copy. Boring, but safe.

And all of this was worthy of effort. CPU usage on both servers in pair went down to 30-50%.

Hope it will be helpful.