Migrating from courier imap to exchange 2007 using imapsync

The following blogpost describes my experience when migrating from courier imap to an exchange 2007 server with the help of imapsync. The intend is not to provide a complete howto but rather reflect my thought and experiences during the migration phase.

Exchange actively rewrites headers, e.g. the Date heade is rewritten like this:

C: Date: Tue, 01 Sep 2009 18:25:48 +0200
E: Date: Tue, 1 Sep 2009 18:25:48 +0200

According to RFC2822 this day of month can contain 1 or 2 digits, so I think its OK.

This counts also for the subject header or maybe other headers. Interesting detail: User-Agent is changed to the lowercase version user-agent. I think at least this breaks with the RFCs.

skipsize option must be used, because Exchange adds additional received headers and recodes messages to quoted printable. It also adds X-MS-Has-Attach and X-MS-TNEF-Correlator headers. Therefore the message size differs.

Because of the rewrite useheader to include only specific headers can also not be used. But by default – if you don’t specify useheader – ALL headers will be considered which is safe if only one time copy is done and no syncronization is needed. As an result of this massive header rewriting you cannot restart imapsync and hope it will detect already copied messages and sync only those that has been changed since last run. Before restarting you should delete the destination server folders.

If you are using Thunderbird to check the Exchange server make sure that you activate the option “Mark messages as deleted”. You need to compact the folder from time to time on the exchange server when you want to start again from a clean state. Otherwise your deleted messages would interfere. See also my blogpost.

When you are already logged in into the exchange server and run run imapsync afterwards it may happen that you don’t see all messages created by imapsync in the other session. There seems to be no way in thunderbird to explicitely refresh a folder. So you have to go in offline working mode and online working mode again.

Exchange servers do not to know custom IMAP flags. Trying to APPEND a message with thunderbirds NonJunk flags set results in an error (use debug and debugimap options to see the full error message):

19 APPEND test (\Answered \Seen NonJunk) "04-Sep-2009 16:02:06 +0200" {5876}
Read:   19 BAD Command Argument Error. 11
ERROR: 19 BAD Command Argument Error. 11 at /usr/lib/perl5/vendor_perl/5.10.0/Mail/IMAPClient.pm line 1328

Custom imap flags are those flags not starting with a leading backslash character. You can circumvent this problem by just ommiting any custom flags with --regexflag. The regular expression is matches against the message flags. The flags are given as one single string seperated by whitespace. So to clear all custom flags you have to give the following three regular expressions:

1. "s/\s[^\\]\w+//g" deletes “SomeCustomFlag” in “\SystemFlag SomeCustomFlag”
2. "s/^[^\\]\w+\s//g" deletes “SomeCustomFlag” in “SomeCustomFlag \SystemFlag”
3. "s/^[^\\]\w+//g" deletes “SomeCustomFlag” in “SomeCustomFlag”

With Imapsync version 1.255 I experienced TCP connection drops on exchange server side during the folder structure creation. I suppose exchange server silenty drops the connection when it detects to much errors in an IMAP session. Probably the cause for the errors was the mechanism how imapsync detected if a folder already existed. The problem went away when switching to imapsync 1.286. Maybe 1.267 (“Better test to check non existing folders on destination server.”) already fixed this issue.

Imapsync version 1.286 did not work on my computer as is. The error was Undefined subroutine &Term::ReadKey::ReadMode called at imapsync when the script was asking me for a passphrase. The fix was quite easy: Uncomment the use Term::ReadKey on line 436. This is definitely a bug in imapsync!

Note that imapsync still tries to detect duplicate messages even though Exchange changes headers. Imapsync will happily produce duplicates between source and destination server by re-copying the same message again and again if called multiple times, but it will detect duplicate messages already existing on source server side. So it will not mirror two messages having the same headers in one folder. If you have such date you will get a lot of “Skipping” messages.

Exchange and Outlook expect the well known folders like “Sent” or “Drafts” to be on the same hierarchy level as INBOX and not like courier underneath INBOX. (OK, technically this is not true. Exchange simply does not have a private namespace – it does provide an empty string “” – while courier has “INBOX.”. Its solely up to the client how it visualizes the INBOX and private namespace. But most clients simply show the namespace as being part of the folder structure.) For this reason we provide the option --regextrans2 's/INBOX\///' to imapsync.

Unfourtunately accessing public folders over IMAP is not longer possible with Exchange 2007, see this blogpost and (stated explicitely) here. This concerns also user shared folders. It is also not possible for Administrator to login via POP3/IMAP.

Fourtunately courier imap fully supports an administrator / superuser account, see one of my previous blog posts.

Because of this two limitations I could not implement my original plan to migrate from courier imap shared/#shared folders to whatever exchange has as concept. So the preliminary solution is to migrate all shared folders and user accounts (#shared namespace) into one exchange account and let the Exchange/Windows administrator do the rest, i.e. split them again to different exchange mailboxes.

I observed warnings like Warning: ignoring folder #shared.user.INBOX because it is not selectable and Warning: ignoring folder #shared.user.Sent.2005 because it is not selectable. The cause for the first one was a mixed up Maildir with a directory .INBOX.Sent inside! Don’t know how this could happened at all. However this confused courier a bit, because the valid directory .Sent was already present. As .INBOX.Sent was a valid maildir folder but contained no mail I simply deleted it and the first warnings went away. The second warning is not really an issue. It means that #shared.user.Sent is an folder containing only subfolder like #shared.user.Sent.2005.March, not messages itself and cannot be selected for copying (\Noselect \HasChildren in IMAP speak).

Trailing whitespace is another source of error messages like Couldn't create [#shared/user/foldername ]: 29 BAD Command Argument Error. 11. Exchange just cannot handle those foldernames. We could rewrite them using regextrans2 to e.g. underscores, but there is no way to detect if the rewritten folder is not existing yet. So better inform the user to rename those folders before themself before migration.

It is advisable to do the migration in small steps, perhaps only a couple of accounts with --debug --debugimap activated. Look carefully at the logs. First migrate the administrators users account (the courier account belonging to imap user group “administrators”) and the public shared folders and then migrate the private shared folders from #shared namespace. To tell imapsync what to copy for the first migration step we provide the option --include '^INBOX.*' --include '^shared.*'. Courier does not respond with the shared private folder #shared to a LIST or LSUB call – even if you are subscribed. So you need to explicitely provide the other users folder with --folderrec '#shared.' (note the trailing dot).

Finally, these were the command lines I used to migrate it: 🙂

$ imapsync --host1 127.0.0.1 --noauthmd5 --user1 username --host2 serverto  --user2 username -ssl2 \
   --include '^INBOX.*' --include '^shared.*' --regextrans2 's/INBOX\///' --skipsize --nofoldersizes  --syncinternaldates \
   --regexflag 's/\s[^\\]\w+//g' --regexflag 's/^[^\\]\w+\s//g' --regexflag 's/^[^\\]\w+//g'
$ imapsync --host1 127.0.0.1 --noauthmd5 --user1 username --host2 serverto  --user2 username --ssl2 \
   --folderrec '#shared.' --regextrans2 's/INBOX\///' --skipsize --nofoldersizes  --syncinternaldates\
   --regexflag 's/\s[^\\]\w+//g' --regexflag 's/^[^\\]\w+\s//g' --regexflag 's/^[^\\]\w+//g'