Monday, December 7, 2015

First Glance on Exchange 2016 (Part 5). AutoReseed in Play

Hi folks,

I have covered configuring DAG and AutoReseed in my previous article. In this one I will share with you on how AutoReseed Actually works. You can read this and this articles on the work flow behind it. I will simply share with you on how I tested it experimentally.

In order to emulate scenario that would activate AutoReseed, I first have removed virtual disk from one of my lab servers.

As the result all the mailbox database copies on this disk get into the state of Failed and Suspended.

The following event log is used to monitor events related to AutoRessed; Applications and Services Logs->Microsoft->Exchange->HighAvailability->Seeding

AutoReseed will start attempts to automatically reseed database copies that were stored on the list volume. It will log series of events with the source of HighAvailability.

First event 1109 will be logged as process starts.

It will follow by the event 1124 where attempt for resuming DBs will take place

Afterwards event 1110 will be logged showing that process has completed successfully.

But don't be fooled by the "success" word. In 15 minutes the whole process will be repeated again.

Overall, this process will be repeated 3 times every 15 minutes as AutoReseed is trying to resurrect the lost disk. After 3 attempts have failed the similar work process will start. You will again see event 1109 you saw previous 3 times.

Event 1119 will be logged with the information about attempt to assign spare disk

After this event 1124 will be logged, however it will attempt to assign spare drive and log the AssignSpare event.

This will follow with the event 1149 which will list all the available volumes:

If Autoreseed successfully finds spare volume and assigns it. Event 1125 will be logged saying that the spare volume has been successfully assigned

The process will finish with the event 1110 about the successful finishing of the process, but is not yet the end.

In 15 minutes database copies reseed will be started. The new cycle will start with our beloved event 1109. This will be followed by event 1124 with the difference that InPlaceReseed process has started.

The events 825, 1127 1246 and 1238 will be logged for every database copy as witness of the reseeding process.

Should DB copies replicate successfully event 826 will be booked for each of them

After this you will see event 827 which give evidence that context index is being replicated.

Finally after all replication is over events 1109, 1183 and 1110 are logged as evidence that HealthyCopyCompletedSeed event has completed. It means that all the mailbox database copies that were affected by the loss of the disk have been successfully replicated.


To verify health of your mailbox databases you can execute Get-MailboxDatabaseCopy against databases that were affected by this process to confirm that their state is healthy. You should get something as the below:

And when you check volume mount points settings on your spare drive you will notice that the database volume mount points previously assigned to the lost volume are now assigned to the spare volume.

And now all you need to do is to replace a faulty disk and assign it to the volume mount point of the lost volume.

To be continued...

Tuesday, November 17, 2015

Resolving Some of Exchange 2013 Uninstall Problems

Hi folks,

In this post I would love to share with you on the issues that I had when uninstalling on Exchange 2013 server. The server version was Exchange 2013 CU9 and it was multi-role server. As you may know Exchange 2013 mailbox server is a combination of all Exchange 2010 server roles.

To uninstall Exchange 2013 binary files I have navigated to the installation files and executed the following command:

setup.exe /m:Uninstall /IAcceptExchangeServerLicenseTerms

So when I launched uninstall all prereqs proved to be successful and installation started. Mailbox services were uninstalled straight away. When the UM services uninstall turn came it prompted me with the following window:

Exchange 2013 uninstall program in order to uninstall UM services component was looking to the following files:
- MSSpeech_SR_en-US_TELE.msi
- MSSpeech_TTS_en-US_Helen.msi

These MSI files belong to UCMA 4.0 which is a prereq for Exchange 2013 installation. When UCMA is being installed it creates the following folder in the Temp folder within end user's profile under which the Exchange 2013 installation is happening. For example: C:\Users\ExchAdmin\AppData\Local\Temp\2\UcmaRuntimeSetup.  After UCMA installation UcmaRuntimeSetup folder is being removed by installation program. To resolve this I had to install UCMA 4.0 on the other computer. I didn't rush to click Next button before file installation to take place. I have copied the whole UcmaRuntimeSetup folder under my account's user profile Temp folder and re-ran the uninstall again.

As you can see UM uninstall completed successfully. 

However I faced another issue when CAS component was being uninstalled I faced the following error displayed both in screen and logged to the Exchange setup log.

This is where this article came to help. The problem was with IIS not being able to load exppw.dll module. You can read more details about it in the article itself. As long as I was concerned I had to edit configuration file for IIS located at C:\Windows\System32\inetsrv\config\applicationHost.config  and add the following into the <globalModules> section:

<add name="exppw" image="C:\Program Files\Microsoft\Exchange Server\V15\ClientAccess\Owa\auth\exppw.dll" />

I have executed IISRESET and installation continued for CAS component was successfully uninstall.

First I encountered the following error when uninstalling Front-End Transport component

This is because Microsoft Exchange Active Directory Topology wouldn't stop because of FIP-FS 
Microsoft Filtering Management Service not being properly removed from the server. It's state was running while startup type was set to Disable. After stopping Microsoft Filtering Management Service I was able to do the same with Microsoft Exchange Active Directory Topology.

Uninstall process moved forward and stumbled over the missing cafe_exppw module in IIS.

I had to go back to C:\Windows\System32\inetsrv\config\applicationHost.config and added the following into the </globalModules>  section:

<add name="cafe_exppw" image="D:\Program Files\Microsoft\Exchange Server\FrontEnd\HttpProxy\bin\exppw.dll" />

After this I ran IISRESET and restarted uninstall program. Front-End CAS component has been successfully uninstalled.

Finally uninstall of the language packs went successfully and server has been sucessfully removed from the AD so there was no need to scrapping it out manually


Restoring Exchange 2013 After Restoring Broken Trust

Hi folks,

I have another adventure to be shared with you. I had an experience when my Exchange server has lost trust with the AD domain. It has been improperly rejoined back to AD by removing AD account to the computer and re-adding it back to the AD by creating a new AD object which resulted in the brand new SID and default group membership.

As you probably know Exchange 2013 (as other versions since 2000) keeps all its configuration settings in the Configuration partition in the AD. After such rejoin a server couldn't retrieve information from the AD and as the result it wasn't able to start any of services.

The resolution of this problem was simple add server back to the following Exchange-related security groups and restarting it (to update access ticket with groups' SIDs):

After server restart I noticed that all the services have restarted successfully and Test-ServiceHealth along the Event Viewer showed no error.

I hope you'll find it helpful.


Monday, November 2, 2015

Cleaning Up Ghost Move Requests

Hi folks,

I recently had another adventure which I would love to share with you.

I was working on getting rid of the default mailbox databases I ended up with some ghost move requests. When executing Get-MoveRequest cmdlet I saw the following warning:

Removing these move requests was not successful. When attempting to remove move requests I was presented with the error as below:

My investigation of this issue lead me to this thread on TechNet forum. It turned to be very close to what I was having, except for the value of msExchMailboxMoveStatus attribute which was different, however my scenario was different from the one that was discussed there. I was getting value of 1 which probably corresponds to the status value of Queued.

I have created query like below in Active Directory users and computers to retrieve these user objects in ADUC (alternatively ADSIEdit can be used to edit AD attributes on the low level):

When checking move request related attributes in AD I have discovered the following:

msExchMailboxMoveFlags 10
msExchMailboxMoveStatus 1

I have edited each of them and cleared the value as follows:

After clearing each object I would run Get-MoveRequest cmdlet which would clearly show me that the number of ghost mailbox move objects reduces till it finally disappeared when I cleaned all of the user objects.

So now, we are left with the orphaned user accounts. We simply delete them using Active Directory Users and Computers.

I hope this helps you as it helped me.


First Glance on Exchange 2016 (Part 4). Storage and DAG

Hi folks,

I continue my series on my first experience with Exchange 2016. I have finally found time to write this article. Now as I have configured transport and client access protocol in my lab, the time came to configure mailbox databases and DAGs.

Autoreseed is a great feature since Exchange 2013 which allows to automatically replace failed volume and This is a great article from TechNet Blog that gives a good guidance on how to configure AutoReseed. Autoreseed is managed by 3 important attributes that can be configured on DAG:

 - AutoDagVolumesRootFolderPath is the root folder for the volumes and the default value is E:\ExchVols (in my scenario)

- AutoDagDatabasesRootFolderPath is the root path for the databases and default value is E:\ExchDB (in my scenario)

- AutoDagDatabaseCopiesPerVolume is the amount of databases per volume. I used 2 which allows me to host 2 DBs on volume which is yet another great Exchange 2013/16 feature.

As you know we need to have file share witness and file share directory to be created File Share Witness can't be member of DAG and since we have only one Exchange 2016 role now we will need to use a member server for this purpose.

When a member server is used as a file share witness the Exchange Trusted Subsystem group should be added as a member of the local admins group

I have also created directory C:\FSW\DAG001 which I will use as a witness directory.

As file share witness has been prepared, I moved to configuring of the storage on my Exchange servers.

Create  root folders for mount points collection

Within each root folders I have created folders as follows:

For volumes' mount points:
- E:\ExchVols\Vol001
- E:\ExchVols\Vol002

For Databases mount points:

After folders have been formatted both volumes with REFS and 64K as unit size (this is now supported according to Exchange 2016 PA but still not a best practice for Exchange 2013) and have mounted both DB and spare volumes to  E:\ExchVols\Vol001 and E:\ExchVols\Vol002:

After this I mounted volume which will store my mailbox databases to E:\ExchDB\DB001 and  E:\ExchDB\DB002 while I left the second volume alone:

Volume that hosts DBs has 3 mount points (one for volume mount point and another 2 for mailbox databases mount points)

Volume that corresponds to spare volume has only one:

After configuring mount points I have created folders for mailbox databases and transaction logs as follows:


These 4 folders correspond to the mailbox DBs and transaction logs that will be hosted on my 2016 box. Please note that file structure under E:\ExchDB\DB001, E:\ExchDB\DB002 and E:\ExchVols\Vol001 will be the same as both folders are used to map the same physical volume.

Each mailbox databases will be having its own point point, yet being located on the same physical drive.

As soon as the storage was ready I moved to creating and configuring DAG. I created DAG without CNO and IP address, which is yet another new Exchange 2013 SP1 and 2016 feature which is now recommended by MS. About benefits about having DAG without IP address and CNO you can read here.

Below is the command for creating DAG w/o IP address and CNO:

New-DatabaseAvailabilityGroup -Name DAG001 -DatabaseAvailabilityGroupIPAddresses ([System.Net.IPAddress]::None) -WitnessServer EX2016-FSW01 -WitnessDirectory C:\FSW\DAG001

After DAG was ready I ran the following commands to configure parameters for AutoReseed:

Set-DatabaseAvailabilityGroup DAG001 -AutoDagVolumesRootFolderPath E:\ExchVols

Set-DatabaseAvailabilityGroup DAG001 -AutoDagDatabasesRootFolderPath E:\ExchDB

Set-DatabaseAvailabilityGroup DAG001 -AutoDagDatabaseCopiesPerVolume 2

Configure Datacenter Activation Coordination mode (aka DAC anout which you can read here) for DAG :

Set-DatabaseAvailabilityGroup DAG001 -DatacenterActivationMode DagOnly

As DAG had been created I have added both of my servers into DAG by running the following command for each  of them:

Add-DatabaseAvailabilityGroupServer DAG001 -MailboxServer EX2016-EX01
Add-DatabaseAvailabilityGroupServer DAG001 -MailboxServer EX2016-EX02

After Exchange has installed all the necessary features (Failover Clustering) and added both servers to DAG we check its status by:

Get-DatabaseAvailabilityGroup DAG001 -Status

We need to ensure that both servers are listed under Member Servers and Operational Servers:

After DAG has been successfully created and configured we can create mailbox databases and add copies.

The following code helped me with easy and quick mailbox database creation:

$Server = $(Get-WmiObject Win32_Computersystem).name

New-MailboxDatabase -Name DB001 -Server $Server -EdbFilePath E:\ExchDB\DB001\DB001.db\DB001.edb -LogfolderPath E:\ExchDB\DB001\DB001.log

New-MailboxDatabase -Name DB002 -Server $Server -EdbFilePath E:\ExchDB\DB002\DB002.db\DB002.edb -LogfolderPath E:\ExchDB\DB002\DB002.log

Restart-Service MsExchangeIs

Get-MailboxDatabase Db00* |mount-Database

After we have got our mailbox databases created we can create their copies on other mailbox servers:

Add-MailboxDatabaseCopy DB001 -MailboxServer EX2016-EX001
Add-MailboxDatabaseCopy DB002 -MailboxServer EX2016-EX001

After seeding is completed we need to make sure that all mailbox databases are in good shape and have healthy and working copies though all the environment. It can be achieved by this command:

Get-MailboxDatabase |Get-MailboxDatabaseCopyStatus

Finally we need to get rid of the default mailbox databases. First we need to move arbitration and monitoring mailboxes to the highly available mailbox databases (otherwise we will be presented with the error when we attempt to delete default mailbox database. The following code helps to move them: 

Get-Mailbox -Arbitration |foreach {new-moverequest -Identity $_.Name -TargetDatabase DB001}
Get-Mailbox -Monitoring |foreach {new-moverequest -Identity $_.Name -TargetDatabase DB002}

We can monitor process of the above mailboxes move by executing the following cmdlet:


As soon as it's completed for all mailboxes we can remove move requests and then remove all mailbox databases by using the Remove-MailboxDatabase cmdlet.

To be continued...

Friday, October 30, 2015

Automatic Installation of Exchange 2013/2016

Hi folks,

I would love to share with you a quick and easy way for installing Exchange 2013/2016 servers automatically. It can be achieved by using this script that can be downloaded from the TechNet Gallery. Here is the author's site where he explains about using of the script and all its parameters in more details.

To successfully perform automatic installation you will need to create a folder, let's say D:\ExInstall. You will need to download Office 2010 Filter PackOffice 2010 Filter Pack SP1 and UCMA 4.0 and store them into that folder. Additionally you will need to place this script in to the same folder. Alternatively you may need Internet access from the server on which you are installing Exchange. If one of these components is not found in the installation folder script will try to connect to internet and download installation files from there and then execute it.

After this you will need to create subfolder, let's call it Exch2013. You will need to extract Exchange 2013 or 2016 binaries there.

As long as Windows 2012 R2 components are concerned, they will also be automatically installed as the part of this installation, so no need to worry for them.

As soon as everything is in place, the final thing that is needed is administrative account under which you will need to perform Exchange installation. If you are installing a new server into existing Exchange 2013/2016 organization you will need to be a member of Organization Management group. Please note that this script also extends schema and AD, so if you are introducing a first Exchange 2013/2016 server into existing 2010 organization or performing a greenfield installation you will also need to have permission to update AD and schema. This will require membership in Domain Admins and Schema Admins groups. If schema is prepared before you execute the install script will notice it and skip this step.

During the installation the script also sets local PowerShell policy for script execution to Not Configured which is one of the prereqs for successful Exchange 2013/2016 installation.

Finally it's worthy to mention that the script creates 2 files in the installation folder:

- Install-Exchange15.ps1_20150918122654.log
- Install-Exchange15.ps1_state.xml

The first one of the is used as log for the installation process and the second one is the installation state. They're very useful when you re-run your installation due to some kind of interruption as well as they give you a good picture of what is happening during the script execution which can be very necessary for troubleshooting.

Below is the sample code which installs multi-role (CAS and MBX) Exchange 2013 server along with all prerequisites to the folder where you want it to be installed. At the end of the installation it also registers URL for Autodiscover SCP which saves yet another step for Exchange deployment:

.\Install-Exchange15.ps1 -InstallMultiRole -InstallPath 'D:\ExInstall' -InstallFilterPack -SourcePath 'D:\ExInstall\Exch2013' -TargetPath 'C:\Program Files\Microsoft\Exchange Server\V15' -SCP -AutoPilot -Verbose

I won't explain every single parameter as all of them are clearly described in the author's article shared above.

During the execution script will bounce a server as many times as needed to complete installation of components and continue at the point of stop after the restart.


Uninstalling Exchange 2010 and IIS Metabase Error

Hi folks,

I would love to share with you about my recent adventure. I was working on the project which required uninstall of Exchange 2010. When uninstalling a server with the CAS role I faced the following error:

Looking it up in Google pointed me to the cases when people were trying to install Exchange 2010 (which is running the same installation program as in uninstall). Most of them like this one were pointing that some of Exchange prerequisites, especially IIS related were missing.

From the first glance all was fine IIS was available and all the components were around. I could clearly see in the Server Manager or when running Get-WindowsFeature cmdlet (please don't forget to import Server Manager PowerShell module if you're running this cmdlet in Windows 2008 R2). However when opening Server Manager I paid attention to the Roles Summary section which showed red cross next to the Web Server (IIS).

Digging further revealed that the IISADMIN service was stopped and disabled.

The solution was very simple. Set startup type for the service to Automatic and start this service. This can be done either by using services.msc console or by running the following 2 commands:

Get-Service IISADMIN |Set-Service -StartupType Automatic

Start-Service IISADMIN

After IIS admin has started I clicked on the Retry button to check prerequisites again and I have got so long desired state:

So my final advice would be when uninstalling Exchange 2010 not to stop and/or any of the services on which Exchange is depending. All you need is simply have it drained of connections and removed from all the load balancers so that no client access traffic comes to it. Un-installation program will do all the magic for you.