Exchange performance degrades with large numbers of Entourage clients

Robert Liebsch, an IT Manager with a large set of Entourage users, reported that his 75 Entourage users were taxing the Exchange Server, causing performance problems. He sent us an article he wrote that looks at the technical issues involved. Based on some analysis, he offered this observation:

I think MS implemented WebDAV in Entourage in a way that isn't jibing with Exchange 2003. I don't think the volume of data, number of Mac users on Exchange, etc, were thought through. I would guess that not only are Mac users suffering, but Windows users are seeing degraded performance.

He also offers some strategies for dealing with the problem in his article below. Reader responses with some suggestions are below that.

Problems with WebDAV in Exchange

By Robert Liebsch

April 11, 2007

I am the Exchange admin for 75 Entourage users. I find that Exchange has problems with large numbers of Entourage clients and very large mailboxes. Our situation is exacerbated by everyone having a Blackberry.

I have all my users on the most recent version of Entourage connecting to Exchange 2003 (fully updated, thank you dst).

I'd recommend having the users talk with their Exchange Admins. Things to have the admins check using Perfmon:

1. MSExchangeIS Connection Count

2. MSExchangeIS Mailbox HTTP/DAV statistics

3. Physical Disk Avg. Disk Quwue Length, Avg. Disk sec/Read, Avg. Disk sec/Write, Disk Read Bytes/sec, Disk Write Bytes/sec

Related links at MacWindows

- MS dropped direct Palm syncing in Entourage 2008; issues with BlackBerry, Win Mobile

- iPhone to become direct Exchange client

- Office 2008 12.0.1 fixes bugs with Exchange, Word Win compatibility; Office 2004 also updated

- Microsoft releases Office 2008 with new Windows compatibility features, but macros are gone

- New Office Mac to offer limitted compatiblity with Win Office 2007

- MacTech to help with lack of VBA in Office Mac 2008

- Beta converter turns Word .docx files into RTF, but can't save to .docx

- No MAPI for Entourage 2008, but Kerberos added

- Entourage 2008 supports kerberos for AD, GAL authentication

- Entourage and Exchange Tips and Reports

I think what the admins will find is that Exchange is getting POUNDED by Entourage. I see sustained reads and write correlating with decreases in DAV performance, subsequently users will have to log in again.

According to Microsoft, my server should handle roughly 3000 users with mailboxes of recommended size of about 250 to 300MB. I have one RAID card for the OS and Log disk arrays. I have another dedicated RAID card for 14 drive array housing the Database files. RPC Averaged Latency hovers in the 200s. My average Disk Queue is in the 200s. In theory, it should be less than 20, technically it should be never higher than 15 (one more than the disks I have in array). My average disk sec read and write are both well over what Microsoft calls acceptable. My server only has about 75 active users, but about 130GB total storage.

I have been working/struggling with these problems for 6 years now.

I think MS implemented WebDAV in Entourage in a way that isn't jibing with Exchange 2003. I don't think the volume of data, number of Mac users on Exchange, etc, were thought through. I would guess that not only are Mac users suffering, but Windows users are seeing degraded performance. It is, though, a magnitude of improvement over Exchange2000 ability to handle the DAV traffic.

I know 2003 is designed for WebDAV, but I think there is something subtly different/wrong in the Entourage is making requests. Each Entourage client spawns a DAV connection (apparently) for each folder (Inbox, Sent, Calendar, Contacts) and subfolder (June, Clients, etc) every few minutes to find and reconcile differences. I think this is generating fairly significant DAV traffic and Disk I/O because of the reads/writes. Multiply this by your number of users and factor in depth and number of subfolders, then add for delegates, then add for FreeBusy checks, then add again for blackberry users on a BES. Let's not forget, that those people with the most complex file structures and largest databases have delegates who have access and cached copies of their databases.

Rapidly, with a small office of 75, you could be looking at 1000 connections. And this affects the Outlook users too. I'll see RPC latencies in the 300s at times. My 4 outlook users will complain of waiting up to 1 minute for calendars to appear.

Another oddity. I watch inbound and outbound mail from our gateways. Additionally, because I have the SMTP queue on its own disk and the transaction logs on a separate array, I can see that the writes to disk do not correspond with the volume of inbound mail. We get a good deal of mail, but not enough to correspond with the amount of Disk Writes/Sec. What is Entourage writing to the DB?

My server is almost an example of how to implement Exchange 2003, and it is buried by the Entourage clients. One disk for OS and SMTP queue, four disks for transaction logs in RAID10. Those 2 logical disks are on a dedicated RAID controller. I have 14 Ultra320 at 15Krpm drives in RAID10 on a dedicated controller. I have teamed Gigabit adapters for network connectivity, 4GB of RAM. The Exchange server has two stores (new Micrsoft recommendations say fewer stores is better).

The only real solutions are: 1. Multiple servers. I had moved 1/3 of the users base to another server and the problems nearly vanished. Thanks to DST, I had to move them back until I get the resources for a second Exchange 2003/2007 server. But, this is indicative that once the volume of data/connections resumes performance will slowly degrade until the login requests or timeouts resume.

Or 2. STRICT Storage Policies. Since this appears to be related to volume of data or number of connections, we need to reduce the volume significantly. But this is counter to the trend in email services offered by corporations. We are always pointed at Yahoo! And Google offering well over 2GB for free. Users don't realize the complexity of Exchange versus the simple webmail offering of Yahoo and Google.

--Robert Liebsch

NOTE: Liebsch sent us several updates responding suggestions by other readers. You can read his follow-up notes below.

More on flawed WebDAV in Entourage degrading Exchange performance

April 13, 2007

Several readers responded to Wednesday's article about large numbers of Entourage users degrading Exchange Server performance. After some testing, the author concluded that there is a problem with the way Microsoft implemented WevDAV. Several readers agreed, pointing to Microsoft knowledgebase articles and their own experience. Several readers also provide some techniques that help. One reader, however is not seeing the problem with 120 Entourage clients.

Ben Lachoff agrees, and said that Microsoft has verified this and has called it a "design flaw" in Exchange:

We have recently grappled with this issue at several clients, and the issue does not happen just by having Entourage clients; we have had servers with 200 Entourage users without any problems. The server slowdown ONLY happens when you move mailboxes or defrag the Exchange database on an exchange box with lots of Entourage users. These actions will cause the exchange database .stm file to be converted back into an .edb file. When the Entourage users access their exchange data after either a mailbox move or information store defrag the Entourage client software will start slowly converting all emails out of the edb file back into the stm file. This causes the server to CRAWL. You can confirm this is happening in a variety of ways, including the performance tests that Robert Liebsch mentions in his write up. You can also test with no Entourage clients open on the network and the server magically returns to normal speeds. Finally, you can check on the relative sizes of the edb/stm files and see the stm file is growing slowly over time while the edb file size gets smaller.

We called Microsoft to discuss the issue and they are aware of it. They are calling it a DESIGN FLAW in Exchange, and they currently have no time frame on a fix. The issue affects Exchange 2000, Exchange 2003, and Exchange 2007. There are similar implications for servers that are heavily used by OWA, POP3, and IMAP users, not just Entourage users.

For more info on this issue, please refer to the following articles from Microsoft: KB329067 and KB317722

There are a few ways to work around the performance issues, but they do not entirely solve the issue. One way to partially resolve the issue is to move Entourage users mailboxes to an information store on a different server or a different drive on the same server. This will help to isolate the heavy disk I/O away from the rest of the system. Another workaround is to add the DAV key suggested in KB329067 and set the value to 1GB (the maximum) This will stop WebDAV clients like Entourage from converting every email in their mailbox to the .stm file, but it will not totally resolve the issues.

Another possible solution would be to implement something like Kerio Mail Server for the Entourage users and try to integrate that into your existing Exchange environment.

Arne has some suggestions:

Just saw your article and I'd like to commiserate with Robert Liebsch.

The issue that we have found at the bottom of it is the conversion from MIME (Entourage, stm) to MAPI (Outlook, edb) and vice-versa. For us the issue is amplified as we run Quest Archive Manager which talks to the IS using MAPI in just the same way that he has Blackberries that also talk to the IS (via a BES) via MAPI, so we have a constant conversion cycle going going.

Hi is quite correct in that it isn't related to the volume of email.

We had some small gains by manually aligning storage boundaries however we are planning an Exchnage 2007 migration to alleviate this issue.

However when Disabled Archive Manager for three weeks as an extended test, the difference was phenomenal, after it had finished coverting everything back to the stm for the Entourage users, the Exchange Server was behaving the way a Dual Xeon 3.2 with 4gb memory and Exchnage Databases on a 14x36gb 10k RAID5 array should do (it was flying).

Peter Gilber also believes there is a problem with WebDAV:

I have a client, MS Small Business Server 2003, 7 PCs in the office. I convinced one user to move to a Mac at home (iMac 20" mactel). He loves it. I received an email from his ISP asking what is causing all of this increased traffic? Turned out he leaves Entourage 2004 open all of the time at home and while he does get a lot of email, he doesn't receive many large attachments. Well, after digging through the ISP's logs/reports they sent me -- his Entourage downloaded 5GB during one day. So I can attest to a flawed WebDav implementation here. Let's hope Entourage 2008 utilizes RPC over HTTP like Outlook2003+ does. (Best excuse to use Exchange2003/ Outlook2003/2007). Maybe we Mac users will even get to sync Tasks and Notes too!

Dedrick Allen has 120 Entourage clients but doesn't see a problem:

What Mr. Liebsch describes in his article about Entourage taxing his Exchange server is pretty bizarre. We have Exchange 2003 running on a much less server than he has and we don't have any performance problems.

Our server is a Dual Xeon 3Ghz with 4 GB of RAM. 5 HD's with 4 in a RAID5 (total 250GB) for the OS and database files and a single drive for the transaction logs. 100 Mbit network We run spam filters and antivirus filters on the same server with very minimal taxing of the server.

We have about 120 Entourage users about 230 Outlook and OWA users. Our users have a 512MB mailbox limit. Some have up to 1GB depending on their job. Several use Palm's and several use Blackberry's in addition to their desktops. We also allow them to use POP services to access their mail from other clients in or outside the network. Even with all this traffic, our server is still VERY underutilized. We are also able to perform every 4 hour incremental backups on the Exchange server without slowing down user interaction.

We have NEVER seen a server performance problem. I would suggest he take a look at his IIS configuration and several of the OS config options. WebDAV is served via IIS and if IIS is not configured properly it can cause problems. Also if he has some sort of IIS filtering/preprocessing going on it can interfere. Also, clear all the event logs and reboot the server. Then go back and look for an entry that Exchange says your server is not properly tuned. Exchange 2003 can sometimes detect bad configuration and will report so in the event viewer.

I agree with him however that Entourage is not an ideal Exchange client. I wish MS would have just ported Outlook 2001 for Mac to a native OS X application. It worked much better and had more Exchange specific functionality than Entourage does. Other than the lack of some of the Exchange specific features the only problems we have with Entourage is database corruption when storing the MUD folders in a users network home directory. We moved the MUD folder to the local machine and the problem went away.

If you'd like to comment on this article

Another twist to Entourage and Exchange 2003 performance issues

April 23, 2007

Tim Wilton responded to our reports about Entourage causing performance degradation on Exchange Server. While other readers reported the problem with large numbers of Mac clients, Wilton only has one:

I found your article interesting in that we have experienced exactly the same performance hit on our SBS 2003 server running Exchange 2003 SP2. With only one Mac user, we have found that performance almost grinds to halt when this client connects to the mail store. Disk Queues averaging about 150 to 200 on an IBM X205 with 5x80GB drives in a raid 5 set with a 5i raid controller!

This had us stumped for a while until I noticed that when Entourage started in the morning and logged in, BANG, up went the disk queues. So now I am looking for a resolution. I might try to move the mailbox to a separate store on another disk to free up the other users.

Another reader previously pointed to a Microsoft Knowledge Base article with some information about the issue.

Two more readers told us how they got around the problem of Entourage clients degrading performance of Exchange Server. They also provide more information as to why it happens.

Nick Pitman decided to dump Entourage altogether and have the Mac clients use Outlook for Windows via an application server:

We discovered the effect Entourage has on Exchange servers about 18 months ago. On our site under 10% of users have a Mac. If memory serves me correctly one Entourage user was estimated to be creating the same Exchange server load as several hundred PC clients. We found same issues as your correspondent Robert Liebsch. After investigation our only solution was to remove Entourage support and give our Mac community Outlook email access via Citrix. This was not as popular as native Entourage for our Mac community but a blessing for our many PC users who were having serious exchange performance issues prior to this decision.

Rod Austria worked with Microsoft engineers and made several discoveries about the problems. He provide a suggested fix:

I too had this exact issue, after recently moving about 60 mailboxes from an existing Exchange 2003 server to a new one. I opened a ticket with Microsoft support, and discusses with engineers from both the Exchange support group and the Entourage group. After about a week of dealing with the, issue we not much better off from when the problem first started. I was finally escalated to a higher level of support with the Exchange group, and worked with an engineer who was very familiar with the problem.

The server I was working on did not have an ideal hard drive configuration: 2 SATA drives mirrored. According to the engineer, KB329067 HAS INACCURATE INFORMATION, in the section titled: Add the MaximumResponseSizeInBytes registry value in Exchange Server 2003 for WebDAV clients such as Entourage. This section suggests that this registry setting sets a max email size value, that determines when MAPI-to-MIME conversion runs, when a non-MAPI client views the Inbox or folder, containing mail. THIS IS NOT THE CASE. The registry setting is actually setting the maximum amount of memory used on the Exchange server, for IIS. IIS is used during the conversion process and adds significant hard drive load if improperly set. I had set this to 1MB thinking that only mail smaller than 1MB would be converted. What should be done is to set this registry key to a value between 500MB and 1GB. I set it to 800MB, using the formula stated in the article. Note, this is the max amount of memory that IIS will use, so it will not immediately allocate this memory. After setting this value and rebooting the server, the non-MAPI clients, including Entourage, started connecting to the Exchange server, with very little degradation of performance on the server. I too was using perfmon to monitor disk activity, and very little activity.

One other thing the engineer mentioned is that the email that was moved will never permanently convert to MIME. The non-MAPI clients will always convert this mail from MAPI to MIME every time they access their Exchange mailbox. Hopefully, this will be a non-issue if the registry change fixes the performance issue. He did provide article KB903707, which suggests to move mail from the Inbox into custom folders. Entourage will constantly compare the Inbox of the local cache copy of a mailbox every minute, which triggers the conversion. Mail that is in custom folders will only run a conversion when the folder is selected, displaying the mail header contents. So in large environments, or for users that have large amounts of mail in their Inbox, this may be an additional step. Our environment of 60 mailboxes did not seem to need this additional step.

Read the rest of current news at MacWindows.

Robert Liebsch (author of the original article at the top of the page) reported success with suggestions we reported on April 26 for fixing the issue of Entourage clients degrading performance of Exchange Server. He said:

I'd like to update that Rod Austria's work and recommendations are making a difference. I did the registry addition and the early numbers are looking quite promising. RPC averaged latency has dropped from nearly 200 to consistently under 50 (hovering under 10).

This is the first work day after the addition of the key and updating the OS to SP2 (W2K3).

Disk I/O and Queue length have decreased substantially, and when they do jump, the arrays recover much more quickly.

Again, it is early, but I am going to remain hopeful.

Meanwhile, Rod Austria informed us of a typo in one of the two Knowledge Base articles that he referenced in his April 26 article about Exchange performance problems caused by Entourage. KB320967should be KB329067. We've made the change to the text of the article and the link.

Update on the fix for Entourage clients slowing Exchange

May 31, 2007

Robert Liebsch has an update to his report about the problem of Entourage clients degrading performance of Exchange Server. Liebsch was the first to report the problem, then responded positively to a suggestion by another reader. He sent us another update:

Now that it has been a few weeks, many thanks to Rod Austria for the solution. So many problems have been resolved by adding the DAV key to the registry with the 800MB value.

Entourage is faster sending and receiving mail updates (faster than Outlook at times). Blackberry delivery latency has dropped to insignificant levels. My road warriors no longer time out or have unresponsive Entourage clients. Attachment downloads and uploads have improved significantly. My Administrative staff can move through the calendaring system with appropriate responsiveness.

John Zsakany saw the issue, tried the fix, and had success:

I knew the Macs users were the issue. When they are logged in my disk queue goes up to 800! Turn off the store, and it’s below 1.

So in my environment I moved the Mac users over, 20 of them. Brand new quad core 2 RAID 1 drives and a raid 10 drive, iSCSI drives. 20 users killed the resources.

I attempted the reg entry fix, which seemed to fix it. The Disk queue peaked 4-5 times in the last few days and I moved almost everyone over Sunday (about 50 mailboxes).

Adam Blevins confirmed the previously reported fix for the problem of Entourage clients degrading performance of Exchange Server:

I was also seeing the exact same performance issues on our Exchange Server. We have about 500 Entourage 2004 clients and 200 OWA clients.

It has been over 5 weeks now working with Microsoft on this issue. Microsoft stated in KB329067 to set the value to 50MB. This did not resolve our problem.

Only modifying the registry value set to 800MB = 838860800 Decimal fixed the performance issue.

Issue:

Disk I/O over .700
RPC over 300
Current Disk Queue Length over 150.

Needless to say server was crawling by. After registry fix:

Disk I/O under .020
RPC under .010
Current Disk Queue Length under 8

Other MacWindows Departments