Delay on Enterprise Vault Web Pages on first access each day.

After deploying a geographically dispersed cluster toward the end of last year, the next thing on my to do list, was to provide archiving services.

We use Symantec Enterprise Vault across the organisation, so I set about installing and configuring that. The installation went smoothly and all was working without any major issues.

The only issue reported by users was in regard to the search and browse vault web pages, usually viewed through Outlook were taking a long time to load for the first attempt each day.

I called Symantec Support and was pointed in the direction of the article below.

http://seer.entsupport.symantec.com/docs/324255.htm

In our case this didn’t apply. This article (http://seer.entsupport.symantec.com/docs/351109.htm) was closer to our solution, but still not our complete solution.

The runtime config below will be familiar to admins who run Exchange 2007 Servers where no internet access is available. For those who are not familiar with certificate signed code, at regular intervals a certificate revocation list will be checked to see if the certificate has been revoked. Where no internet access is available the delay occurs as the server attempts the connection anyway.

Symantec advised us to place the following inside the runtime tags inside the machine.config file. The reason this is different from the latter support article above is because were running the server on Windows 2008.

<runtime>
     <generatePublisherEvidence enabled="false"/>
</runtime>

This solved our delays nicely.

Convert Linked Mailboxes to User Mailboxes in Bulk

My organisation has gone through a massive migration project to unify Active Directories and Exchange organisations. As a result of these migrations a lot of mailbox migrations have resulted in a lot of mailboxes ending up as linked mailboxes even though their not.

The official TechNet article on this explains how to disconnect the mailbox and re-attach it to the user account correctly as a user mailbox. http://technet.microsoft.com/en-us/library/bb201749(EXCHG.80).aspx

Another way to make this appear to be corrected is to manually change the “Recipient Type” AD property on the affected mailboxes. This though, is unsupported.

Using the official method from Microsoft results in the loss of any specific mailbox information such as SMTP, x400 & x500 addresses, mailbox sizes and any other individual mailbox settings.

Only e-mail addresses and mailbox sizes were important to me (I must admit, I forgot about mailbox sizes at first).

I came up with the script below that would properly convert all Linked Mailboxes on a particular server to user mailboxes in a supported way. The script is very effective, but you will want to check out the list of considerations below before running it, they might lead you to amend the script slightly. You will have to modify the $exchangeserver and $userdomain variables though.

$exchangeserver = "exchccr1"
$userdomain = "domain\"

$linkedmailboxes = get-mailbox -server $exchangeserver -resultsize unlimited|where {$_.recipienttypedetails -eq "LinkedMailbox"}

foreach ($mailbox in $linkedmailboxes){
Disable-Mailbox -Identity $mailbox.displayname -confirm:$false
}

Get-MailboxDatabase -server $exchangeserver|Clean-MailboxDatabase

start-sleep -s 90

foreach ($mailbox in $linkedmailboxes){
$usernamestring = $userdomain + $mailbox.samaccountname
Connect-Mailbox -Identity $mailbox.exchangeguid -Database $mailbox.database -User $usernamestring
set-mailbox -identity $mailbox.displayname -emailaddresses $mailbox.emailaddresses
}

start-sleep -s 90

foreach ($mailbox in $linkedmailboxes){
set-mailbox -identity $mailbox.displayname -EmailAddressPolicyEnabled $mailbox.EmailAddressPolicyEnabled -emailaddresses $mailbox.emailaddresses -UseDatabaseQuotaDefaults $mailbox.UseDatabaseQuotaDefaults -ProhibitSendQuota $mailbox.ProhibitSendQuota -ProhibitSendReceiveQuota $mailbox.ProhibitSendReceiveQuota -IssueWarningQuota $mailbox.IssueWarningQuota
}

Some things to consider…

• The filter on this script doesn’t consider legitimate linked mailboxes.
• Only E-Mail addresses and mailbox sizes are re-applied to the freshly attached mailbox. More can be added to the script though.
• You can’t attach a mailbox to a disabled account. The script won’t stop, but will error on disabled user accounts.
• Don’t stop the script as it’s running, even if it’s choosing the wrong selection of accounts. It will mean more manual work after if you do.
• Ensure you have "Exchange Server Administrator" permissions on the server you wish to run this script.
• Ensure you have permissions to run the Clean-MailboxDatabase permissions on the server you wish to run this script (You don't get this by default with "Exchange Sever Administrator" permissions). If you're an admin in my 0rg, we can give you this permission if you if you don't have it already.

There you go, a script that will do all your linked mailboxes in one go. I’ve not been able to find another online, so I hope this helps you.

Upgrading Delegated Exchange 2007 Clusters to SP2 - FIX

Concerning a previous post “Upgrading Exchange 2007 Clusters to SP2 – Workaround”, Microsoft provided a fix for the issue of upgrading an Exchange 2007 Cluster using delegated privileges.

Microsoft provided us with a fix that allows this to happen without using the workaround described in the article above.

Download the file from here http://downloads.daiowen.co.uk/ExBPA.PreReqs.xml

How to use the XML…

1. Copy all the E12SP2 setup files to local disk.
2. Replace the original ExBPA.PreReqs.xml with the one available above.
3. Run the setup from local disk.

Exchange 2007 Public Folder Mail Routing

We had a report recently that mail from outside the Exchange organisation destined for Public Folders was being returned in the form of an NDR, but all other mail was flowing fine.

To explain the problem, here’s a little background about the Exchange 2007 topology. We have two HUB servers that handle mail heading inbound and outbound of the organisation. Beneath that we have a lots of exchange deployments at physical sites with varying local configurations. To complicate things we have firewalls sat in front of these other deployments with some more strict than others. As we add more exchange deployments it can be a considerable task getting these firewalls adjusted to allow the new hub transport servers to communicate with the old, usually leading local administrators to notice queues forming on their sites.

I had all the information I needed to track the messages, so started by tracking the message at our two hub transports handling mail into and out of the system. The Public Folder that the message was being delivered to, only had one replica. I discovered that the message was being sent to what seemed to be a completely random hub server, not to the site where the replica existed. The messages were queuing there as the complaining administrators hadn’t opened their firewalls as requested. Fine I thought, get them to open the firewalls properly, but I wanted to figure out why the message was being sent to this strange server in the first place.

The answer lay in the following Microsoft TechNet Article - http://technet.microsoft.com/en-us/library/bb232041(EXCHG.80).aspx

The article explains how messages are routed for public folders. The start of our problems were because that our two Hub Servers that were receiving mail from the internet didn’t have a copy of the Public Folder Hierarchy to know where to route the message, in this instance it will look at the values of msExchOwningPFTreeBL a property of CN=Public Folders,CN=Folder Hierarchies,CN=First Administrative Group,CN=Administrative Groups,CN=Cymru,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=cymru,DC=nhs,DC=uk . All of the public folder stores should be listed in that property and the Exchange 2007 SP1 or SP2 categoriser filters them out in the following way…

1. Ranking by the age of the public folder database   By default, public folder databases that have an age threshold of less than two days are not considered unless the age of all public folder databases is less than the threshold or the age is unknown.

2. Proximity   The local server is preferred. If the local server does not contain a replica of the public folder database, a server in the same Active Directory site is preferred. If the local Active Directory site does not contain a replica of the public folder database, a server in a remote Active Directory site or routing group is selected as the preferred destination.

3. Cost   If more than one remote Active Directory site or routing group contains a replica of the public folder database, the server in the Active Directory site or routing group that has the least cost routing path from the local Active Directory site is selected as the preferred destination.

In the long term, I’d want the messages routed directly from our two entry point Hub Servers, but in the short term point 1 stopped us from just creating a Public Folder Database to store only the Hierarchy for routing purposes, two days might have been a problem. I created the databases anyway.

Our AD site layout is fairly simple , its a snowflake design where all of the AD sites with connections to our central site had all the same costs. The quick way to resolve this was to drop the cost of a site where you wanted these messages to be routed via, this solved the problem short term until the mandatory two days expired until the newly created PF Databases could route the messages itself.

OR the local admin could have opened the firewalls properly, but that would have been too easy. :-)

Upgrading Exchange 2007 Clusters to SP2 – Workaround

I posted last month about a problem delegating installs of Exchange 2007 SP2. Delegated Admins will receive an error message stating the following…

You must be a member of the 'Exchange Organization Administrators' or 'Enterprise Administrators' group to continue.

Have been looking into the issue and have had a case open with Microsoft. Turns out that you only get this issue on a fully patched server. If you try upgrading or installing as a delegated admin on a fresh install of either server 2008 or 2003 you don’t see the problem either with Exchange SP1 or SP2. I haven’t had time to identify exactly what patch causes this yet, if I’ll bother at all.

If you have patched your server though, MS came up with this workaround.

1. Disable update checking for the BPA by heading into the registry and HKCU\Software\Microsoft\Exchange\ExBPA and either creating or modifying a DWORD named “VersionCheckAlways” and set it to ‘0’
2. Copy the installation files to a local drive and replace Setup\ServerRoles\Common\en\ExBPA.PreReqs.xml with this Modified XML File [I have since updated this link to a properly working file provided to us MS, please read the accompanying post]

Once you’ve done this you can ignore all Pre-Requisite Checking for the install. I was strongly advised my Microsoft that you should ensure that there are no other Pre-Requisite Failures by running an unmodified setup before making the changes above.

Microsoft have said that they’ll pass this to the product group for a fix.

Upgrading Exchange 2007 Clusters to SP2 - Continued

Regarding my previous post delegated installs and upgrades to SP2, see here - http://daiowen.blogspot.com/2009/11/upgrading-exchange-2007-clusters-to-sp2.html

Microsoft has informed us that this will be classed as a bug and is working on discovering the cause before saying if they will fix the problem or not.

Geographically Dispersed CCR Cluster

I recently had the opportunity to install a geographically dispersed CCR Exchange 2007 cluster.

Server 2008’s cluster features can now handle clusters on separate subnet’s making the fact that the only data centres available were operating on Layer 3 wasn’t a problem. I didn’t need to stretch a VLAN across physical sites.

Configuring the networking for the cluster went slightly against the grain for me. Essentially the Private networking element has gone for these types of clusters, because all traffic, heartbeat and all has to go over the public network. That said, it was a simple process. I configured the networking using four NIC’s, three were teamed and another was on its own but it was set not to register in DNS. I didn’t want client traffic coming over the single NIC.

When you set up the cluster you simply enter two IP addresses that the cluster can use, and on failover, one, the one that’s not on the subnet the active node is in, will stay offline, sounds nice doesn’t it, but wait.

Even though you don’t have to stretch a VLAN anymore for this type of cluster. Exchange 2007 still requires cluster nodes to be in the same Active Directory site. This means that if you are planning for the disaster of losing a site, then you’ll need two DC’s in each site in the same AD site so that each node will always have a DC in the event that you loose one of the physical sites. You can’t use DC siteCoverage for this, as I discovered.

With the cluster set up I set up a combined HUB CAS in each physical site. Exchange will load balance mail flow to each HUB Transport Server by itself, but what about CAS connectivity. Autodiscovery service will handle Outlook Web Services, such as OAB & Out of Office etc, but what about Outlook Web Access. On the same subnet you’d use NLB to provide users with a single resilient point of entry to OWA. That’s no good on separate subnets unless you have a hardware load balancer, which I didn’t. So the OWA failover process became a manual process using CName’s in DNS. Not the nicest of solutions.

Another issue… You can’t put a Public Folder Database on a CCR unless it’s the on CCR in the Exchange Organisation. So Public Folders were to be sat on the HUB/CAS servers with content replication between each server. But in the event of a loss of one of those PF servers, it’s a manual failover process to get PF access back. You need to change the Default Public Folder Database for each Mailbox Database in the CCR. But that’s the same for any Public Folder failure.

So now we have two parts of the failover that requires manual failover, not nice, was starting to not like separating my Cluster over different subnets.

Issue number 3… When cluster failover occurs, the cluster IP changes. Meaning that unless all your clients are sat on the same AD site this change of DNS record will take time to replicate to them. By default the TTL of cluster DNS names is 20 Minutes. Meaning that in the worst scenario, your clients could be waiting 15 minutes for AD replication plus 20 Minutes for the DNS record to expire on their machines. 35 Minutes is a long time. Not really acceptable either. You can alleviate this issue by reducing the TTL of the record. I reduced mine to 3 Minutes. Another change you can make is by enabling change notification on the AD site links between the Cluster’s AD site and the AD site/sites where the clients sit. This brings the failover time down to 3 Minutes. Another change we made was in group policy… We created a GPO that configured Outlook not to complain about connectivity issues for 4 Minutes after disconnection from the Exchange Server.

This configuration meant that during a failover the majority of clients would not notice a problem unless they were sending emails and noticing that they were sitting in their outbox.

So with the exception of OWA and Public Folders, the system was quite acceptable. Just after covering off all of the above problems, space became available in our main data centre. We could now stretch a VLAN between these sites. So I reconfigured the networking and put each node in the same subnet. And guess what, most of the problems above went away. With the exception of Public Folder failover, but I can’t get these people to use the SharePoint servers available in the organisation, so I’m afraid that they’ll just have to live with that  :-).