In this case it is Exchange 2010 and Active Sync phones.

A little background:  Exchange 2010 adds a passel of schema and security extensions.  I am not going to discuss the schema extensions here but focus on the security additions.  The security that is of interest here is all at the domain level – on the domain object.  Two new security groups are added for Exchange 2010: ‘Exchange Trusted Subsystem’ and ‘Exchange Windows Permissions’ each with 17 aces (if I count correctly).    Additional aces are also added for the ‘Exchange Servers’ group.  The important Aces here are:

• Allow create/delete rights for msExchangeActiveSyncDevices child objects to a user object,
• Allow create/delete rights for msExchangeActiveSyncDevice child objects to the above msExchangeActiveSyncDevices object, and
• Read/ write all properties of these new objects.

I am not privy to why Microsoft does things but prior to Exchange 2010 there was no structure to contain ActiveSync device information other than the user account or the mailbox itself.  With the msExchangeActiveSyncDevices object acting as a container a user can now have multiple msExchangeActiveSyncDevice objects for multiple Active Sync devices.  I.e. a Phone and an iPad.  Forward thinking.  Very good Microsoft.  Oops, I said I wasn’t going to discuss schema extensions.

So why does any of this matter?  Here is why.  The security that allows Exchange to create, delete, and use these new objects is all inherited from the Domain object.  If it does not inherit to a user object, Exchange cannot create the new the objects.  The result is simple.  Move the user’s mailbox to an Exchange 2010 server and their phone stops syncing.

There can be various reasons the security is not inherited.  On one extreme, you could be in a hosted environment running some variant of Microsoft Provisioning Services (MPS).  On the other, a network admin that is no longer there was too clever by half and blocked security on OU’s for reasons no one remembers.  Whatever the case, you can allow the security to inherit and then block it again if there is good reason.  In any case, this represents a relatively permanent solution.

Finally to the issue I have seen multiple times already.  There is this process that lives in relative obscurity and has been running on DC’s since the NT5 Betas—the Security Descriptor Propagator (SDPROP).  It actually does the inheritance of security from parent to child objects in Active Directory.  On the PDC role holder it has an additional function — it protects the security on security principles that have elevated rights.  ‘Elevated Rights’ is defined as the user being a member of one of a list of built-in groups.  These groups are defined as ‘Protected Groups’.  Any ‘User’ that is a member of a ‘Protected Group’ has its security overwritten by the security descriptor on the AdminSDHolder object in the ‘System’ container by SDPROP and its ‘adminCount’ attribute set to ‘1’.  Since inheritance is blocked on the AdminSDHolder’s security descriptor, inheritance is also blocked on the user object.   This happens every time SDPROP runs on the PDC role holder.  By default, that is once an hour.

The list of Protected Groups has varied from Windows 2000 RTM till now.  The groups in 2008 R2 are:

• Account Operators,
• Administrators,
• Backup Operators,
• Domain Admins,
• Domain Controllers,
• Enterprise Admins,
• Print Operators,
• Read-Only Domain Controllers,
• Replicator,
• Schema Admins, and
• Server Operators. 

If a user with a phone is (or has ever been) in any of these groups, directly or transitively through group nesting, even via distribution group, their phone will not work with their mailbox on an Exchange 2010 MBX role server.

The solution is obvious.  Give people that need administrative rights, administrative accounts and tie their phone to a normal user account.  This is best practice anyway.

If you are determined to not do this, you have several options:

• Remove the users from the groups that are causing the issue.  This also requires you clear the ‘adminCount’ attribute in their account since SDPROP will not clear it for you.  Then enable inheritance.
• Alter the Security on the AdminSDHolder object security descriptor by adding the new Exchange security to it. Since the aces added in Exchange 2010 are numerous, this can be a bit tedious but it will work if you get it right.  Don’t forget to maintain it in the future.
• Change the behavior of SDPROP.  Only four groups were protected in Windows 2000 RTM.  As the list of Groups has expanded, Microsoft has provided a vehicle to modify the list.  This is done by setting the dsHeuristics attribute of the Directory Service object in the configuration container.  Using this attribute, you can selectively exclude Account Operators, Print Operators, Server Operators, or Backup Operators from the list of Protected Groups.  Realize when you do this you are weakening the security of your forest.
• Alter the Security on the AdminSDHolder object security descriptor by allowing inheritance.  Again, realize when you do this you are weakening the security of your forest.

Details of these options are discussed here: 

http://support.microsoft.com/kb/817433

If you want to be proactive before you start moving mailboxes, you can search for all users in you forest that have an ‘adminCount’ of ‘1’.  Lots of ways to do this.  Here is an Ldifde example:

ldifde -f AdminUsers.txt -d dc=<domain DN>  -r “(&(objectcategory=person)(objectclass=user)(admincount=1))” -l  samAccountName

Nick

The ‘msExchQueryBaseDN’ attribute is used to restrict Outlook Web Access’ (OWA) search for mail enabled objects in Active Directory (when simulating the Global Address List) — or at least that is what it was originally used for.  Rather than searching for all mail enabled objects, it will search only a portion of Active Directory.  The attribute is usually not set because most Exchange organizations have only one Global Address List – The ‘Default Global Address List’ which contains all mail enabled objects.

If you don’t understand why there would be more than one GAL, stop reading here. 

Hosting companies supply services to many groups or enterprises and therefore have many GAL’s.  They set this attribute in each user account to control that users ‘view’ of other users via OWA. It is also sometime set in very large organizations when the GAL becomes very large.

Prior to Exchange 2007, when set, the attribute contained a Distinguished Name (DN) of an OU that was the base of the LDAP query.  In Exchange 2007, this attribute could also be set to the DN of an address list in the ‘All Address Lists Container’.  In this situation, OWA would use the Address List’s filter for the query.  This provided more flexibility in controlling the ‘experience’ of users, especially when they were in different OUs.

Notice that everything I have discussed here is about OWA:  Not Outlook.  Outlook accesses GAL’s based on security.  It had, to my knowledge, never been affected by this attribute.  It appears that the new ‘Availability Service’ in Exchange 2007 and 2010 does use it.

When an Outlook 2007 or above user wants the free/busy information for a user it asks the Availability Service on a Clint Access Server (CAS) in its site for the information.  When that user’s mailbox is on a 2007 or above Exchange server, the CAS server gets the information directly from the user’s calendar in the user’s mailbox.  If the Users mailbox is in another site, the request is proxied to a CAS in that site which gets the information from the users mailbox and passes it back to the requesting CAS and then the requesting user.  Note: the requesting user can be an OWA, Outlook Anywhere, or Outlook user. 

Nothing controls whether the ‘msExchQueryBaseDN’ attribute for a particular user, the Global Address List for that user, or the Offline Address List (if the user is cached) for that user produce the same result.

So here is what happens:  User A tries to set up a meeting with User B in Outlook 2007 or above.  User B is in their GAL and is added to the meeting request’s scheduling assistant.  Outlook asks a local CAS for the User B’s free/busy information.   The CAS does not return it.  Outlook shows it as unavailable.  Why?

In these cases it was discovered that the ‘msExchQueryBaseDN’ attribute for User A was more restrictive than the GAL.  It did not include User B so User B’s free/busy was not returned.  Change the user’s ‘msExchQueryBaseDN’ attribute to include User B and User A retrieves it. 

One reason for this behavior could be that, as mentioned, The Availability Service is servicing requests from multiple sources so it applies the most restrictive rules for all of them.  Another possibility is that Auto Discover is somehow involved.  If you run ‘Test E-mail Configuration’ from User A’s Outlook and put in User B as the E-mail Address being tested, it is not successful if the ‘msExchQueryBaseDN’ attribute excludes User B.  It is successful when the msExchQueryBaseDN’ attribute includes User B. 

No matter what the reason, it is necessary make sure that the ‘msExchQueryBaseDN’ attribute is not more restrictive than a user’s GAL if you want to avoid this problem. 

Nick

[ERROR] An unexpected error occurred while the forms-based authentication settings for path /LM/W3SVC/1 were being modified. The error returned was 5506

After a few failed attempts, I solved the problem by removing the certificate binding to 443 in IIS and restarting the install.  When the install was done, I found the original ‘Exchange’ generated certificate bound to 443 rather than the certificate I was using.   I just bound the correct one and everything seemed to work.

It seems the install is trying to bind a different certificate than the one in the meta base.  The HASH doesn’t match causing the error.

Looking deeper, I found I had BOTH the self signed ‘Exchange’ certificate and my current certificate enabled for IMAP and POP.  Since I do not use either IMAP or POP, I had never removed the certificate generated by the original Exchange install.  I believe that caused the confusion.

The lesson, check to see if multiple certificates are enabled for any CAS services and eliminate the unused ones.

Here is the scenario, a loyal administrator somewhere deletes Robert Smith when they meant to delete Roberta Smith.  It could also be that someone was confused by the ‘newspeak’ in the Exchange Management Console (EMC) and did not realize that ‘Remove’ equates to ‘Delete User’ not to ‘Remove Exchange Attributes’ under ‘Exchange Tasks…’ in the 2000/2003 Exchange extended ADUC and deleted the user object by mistake.  The conflicting terminology is another subject I will pass over here.

Your phone rings.  Easy right?  Just put it all back the way it was.  Well, it is a little more complicated than that.  Let’s get started…

If you have Exchange 2007 installed with the defaults, Robert’s mailbox is safe for the moment and should remain in the exchange database for 30 days.

The only short term consequence it that that new mail to Robert is getting NDR’s because he no longer exists either in Exchange or Active Directory.

So, the first task is to get Robert back in Active Directory.  If you are running AD 2003, this is most easily accomplished using the ADRestore utility written by Mark Russinovich.  Originally a Winternals tool, it is available from Microsoft as a free since Winternals was purchased by Microsoft.

From a command prompt, on a DC run ADRestore to find the deleted account:

C:\>adrestore “smith, robert”

AdRestore v1.1
by Mark Russinovich
Sysinternals – www.sysinternals.com

Enumerating domain deleted objects:cn: Smith, Robert
DEL:5f9f2273-1302-4bbc-99b9-9aa6dd2ade6d
distinguishedName: CN=Smith\, Robert\0ADEL:5f9f2273-1302-4bbc-99b9-9aa6dd2ade6d,
CN=Deleted Objects,DC=pennic,DC=com
lastKnownParent: CN=Users,DC=pennic,DC=com
Found 1 item matching search criteria.

Now that you have found the deleted account, rerun the command with the -r option and restore it:

C:\>adrestore “smith, robert” -r

AdRestore v1.1
by Mark Russinovich
Sysinternals – www.sysinternals.com

Enumerating domain deleted objects:cn: Smith, Robert
DEL:5f9f2273-1302-4bbc-99b9-9aa6dd2ade6d
distinguishedName: CN=Smith\, Robert\0ADEL:5f9f2273-1302-4bbc-99b9-9aa6dd2ade6d,
CN=Deleted Objects,DC=pennic,DC=com
lastKnownParent: CN=Users,DC=pennic,DC=com

Do you want to restore this object (y/n)? y

Restore succeeded.

Found 1 item matching search criteria.

The AD account is restored but it is not 100%.  The important thing is that the account has the same SID and GUID and all Active Directory related function that utilizes these will still work.  Three dimensions of the pre-deletion account, however, are missing and one is different:

1. The user is no longer mail enabled or mailbox enabled. There are some artifacts in that the attributes ms-ExchMailboxGuid, legacyExchangeDN, and ms-ExchHomeServer are still set.  Nonetheless, Robert is just a normal user.
2. The user is no longer a member of any groups accept Doman Users.  AD group membership actually exists in the ‘group’ object(s).  The attribute memberOf is back populated based on those memberships.  When the account was deleted, Active directory deleted the users membership from each group. the restored user’s membersOf attribute is empty. 
3.  If the user had a primary group of anything other than Domain Users, that has also been lost and must be recreated after the user is added to the appropriate group.
4.  The account is disabled

Items 2, and 3 must be repaired manually and can be done easily with ADUC although, depending on the complexity of your environment, you may need to do some sleuthing to determine the proper group memberships.  Microsoft KB840001 discusses the complexity involved in group membership recovery.

Item 4 is simple.  Enable the user before proceeding.

Item 1 should be easy too.  That mailbox is still there.  You just need to open the EMC, browse to the Recipient Configuration container, Disconnected Mailbox item, select the correct server, refresh the screen, select Robert’s mailbox, and reconnect Optimal Results.  Oops, this is where the problem starts.  It is not there?  Why?

My testing indicates that as of E2K7 SP1, only users deleted ( ‘removed’ — remember that terminology stuff ) using the EMC will appear immediately in the EMC as Disconnected Mailboxes.  A user deleted in ADUC whether extended with the Exchange 2003 ESM, or native, does not appear.  Also, even a user deleted using the EMC may ‘disappear’ soon after the account is restored using ADRestore.  I can only speculate on what is going on, but a disconnected mailbox that appears in EMC has the DisconnectDate attribute set: One that does not appear does not.  The EMC must set the attribute: ADUC does not.  Also, restoring the user object previously associated with the mailbox appears to clear the attribute.

In any case, all the mailboxes will appear as disconnected ‘later’ — as in the next day.  The once-a-day database maintance activity will identify mailboxes that do not have a cooresponding AD account and set the DisconnectDate.

This process can be started manually using the ‘Clean-MailboxDatabase’ cmdlet (analagous to ‘Run Cleanup Agent’ in the 2003 ESM). If your orginization has a lot of Servers and Mailbox Databases this can take a while unless you know what database the mailbox is in.

So much for conjecture — if it is not exposed in the EMC and Mail to Robert is still getting NDRs.  How do you get it back?

There are two options:

First, try the easy way — if you know what database the mailbox is in, fire up the Exchange Management Shell (EMS) and enter the following:

[PS]c:\>clean-mailboxdatabase <Mailbox Database Name>
Check the disconnected mailbox display in the EMC.  If the mailbox is esposed, use the EMC to reconnect it.  You are done.

If not, do the harder way — in the EMS enter the following:

[PS] C:\>get-mailboxdatabase | get-mailboxstatistics  |  select  Displayname,DisconnectDate,mailboxguid,database

This will give you a list of all user mailboxes with their GUIDs, and databases.  If you have 20 users this will be a useable list if you have 20,000, it will not.

You can pipe the output to a file:

[PS] C:\>get-mailboxdatabase | get-mailboxstatistics |  select Displayname,DisconnectDate,mailboxguid ,database>> c:\MyOutput.txt

Or filter on an attribute like display name to make it more manageable:

[PS]c:\get-mailboxdatabase | get-mailboxstatistics  | where {$_.Displayname -eq “Smith, Robert”} | select Displayname,mailboxguid,database
DisplayName       MailboxGuid                                                 Database
———–                ———-                                                       ——–
Smith, Robert      b61027c3-78ac-4b12-963d-c97f3fb30ed4  gemini-mbx\SG01\MBX01

Now you have the mailboxGUID of the disconnected mailbox and you know the database it is in (note: gemini-mbx is the virtual server on my SCC cluster).  You can now do the easy option since you know the database that contains the mailbox, or you can just connect it in the EMS.

To reconnect it:

[PS] C:\>connect-mailbox -database gemini-mbx.pennic.com\mbx01 -identity b61027c3-78ac-4b12-963d-c97f3fb30ed4 -user “Smith, Robert”
WARNING: The operation completed successfully but the change will not become effective until Active Directory replication occurs.

(Note: you must be specific enough to uniquely qualify the mailbox database. If all the databases were mbx01 then I would need to include the storage group.  Just one case, among many, for naming all mailbox databases in your organization uniquely.)

And you are done!

The reconnect mailbox-enabled the user account.  RUS will re-stamp the email addresses from email policy.

Wait for replication in AD and the mailbox should be useable.

Don’t forget, you still need to fix those group memberships.  

Then you can sit back and wait for that phone to ring again.  It won’t be long, you are an Exchange Administrator.

Nick

But, if you configure your OWA site to require SSL, when they type http://owa.myorg.com or just owa.myorg.com (which will default to http) they will not connect successfully.

So, how to allow them to use the lowest common denominator of owa.myorg.com but still connect them SSL?

Microsoft has a couple of KB articles describing method(s) that accomplish this by using a custom Active Server Page and in one case modified security: KB555053 and KB839357. They will both work but seem like overkill to solve a simple problem.

I have been accomplishing this for a number of clients over a number of years using the simple technique described here.  It does not change the OWA site.  It works for all versions of Exchange.  It works for IIS 5, and 6, and should work for IIS 4 and 7.  It works in an NLBWLB environment.  It is compatible with Default Web Site applications other than Exchange installed.

Here how it would work for Exchange 2003 and IIS 6…

Log on as an administrator and open the IIS management snapin on the OWA server.

Create a new site:

Right click on the Web Sites node and select New>Web Site…  When the Wizard opens select Next and give the new site a name such as ‘OWA Redirect’.  Select Next and enter a host header that reflects your OWA URL such as owa.myorg.com.  Select Next again and enter a path.  Any path will do, we are going to change this in a moment.  I just enter ‘c:’, leave the ‘Allow anonymous access to this site’ box checked and select Next twice and select Finish.

Modify the site:

Right Click on the newly created site and select ‘Properties’.  Select the ‘Home Directory’ tab.  Change ‘The content for this resource should come from:’ to ‘A redirection to a URL’.  In the ‘Redirect to:’ box enter the same the same host header you used above preceded by ‘https://’ and followed by ‘/exchange’.  For example:  https://owa.myorg.com/exchange.  Check the boxes ‘The exact URL entered above’ and ‘A permanent redirection for this resource’.  Select Apply and OK to exit.  Restart IIS.

If you now open a browser and connect with just owa.myorg.com, the host header will take you to the redirect site you just created which will in turn send you to https://owa.myorg.com/exchange.  Since it is SSL (port 443) the host header is not applicable, you will connect to the Default Web Site and be prompted to log in.

Simple.

If you have more than one OWA server you will obviously need to repeat this process on each of them.

Don’t forget to lock down the Default Web Site containing OWA per Microsoft’s recommendations and always use Forms Authentication where it is an option.  This technique only makes it easier on the user (and hopefully you), it does not secure OWA.

I have yet to use this technique with an IIS 7 CAS server (installed on Server 2008) but it works for Exchange 2007 installed on 2003 and IIS 6.  I have also never used it for IIS 4.

Whenever I think something simple will work and Microsoft recommends something more complex, I think I must be missing something.

Maybe I am.  

If anyone can find an issue with this technique, please leave a comment.

Nick