Update (10/24/2007): This post has become unexpectedly popular. After 12K page visits on day one, half a year later it still receives 5-600 visitors every day. However, now that Gmail supports IMAP, it has mostly become obsolete, so I suggest you read my Simplified Guide to Importing All Your Archive Email Into Gmail instead.

This post still has value, mostly in the comments section, where 120 or so readers help out each other on numerous related issues.

The original post:

I finally got sick of all the problems with Outlook, bit the bullet and transferred all my historical email online. Having spent a few days using “native” Gmail (vs. POP to Outlook) I already feel a lot more productive. Ironically I’m writing this on the very day when Yahoo announced unlimited storage – but I’m with Mike on this: message threading, labels and powerful search still make Gmail (the Google Apps flavor) the best choice for me. At least for now – but I keep an eye for the next incarnation of another product – will name it in due course (if you guessed which one, you’re probably right ).

Migrating to a new email service wouldn’t be complete if you couldn’t move all your old “baggage” with you. Apparently this is a burning problem for many, as a year-old post I wrote on the subject is one of my most popular hits ever. Back then I was still happily (?) POP-ing it down to Outlook, but wanted a fast all-in-one searchable archive, and Gmail was the perfect solution. But none of the solutions were perfect – until now. There are several “gmail-loader” tools on the Net, but some simply don’t work, others change the original sender information to the email account they use for the transfer – pretty bad, IMHO. My simple solution a year ago was using Thunderbird with a redirect extension. You can read the steps to achieve this here. Even this solution wasn’t flawless: gmail listed all historical mail with the date of the transfer – the original date was sill preserved and searchable, you just got the list display messed up. This still appears to be the biggest hurdle users face according to this new discussion on Lifehacker.

The final solution comes from Google themselves: now that they quietly expanded Mail Fetcher to Google Apps accounts, and removed the “non-gmail source” restriction, there is a simple yet perfect two-steps process to get it all done. Gmail Mail Fetcher fixes the date problem, so now in two steps and using two email accounts you can get it all right.

Step 1: Load all your client-based email to a temporary Gmail account either using my Thunderbird procedure, or, for an easier and elegant solution, get hold of an IMAP account. Gmail does not support IMAP, but my old provider, 1and1.com is not a bad choice: 5 email accounts, 2G each with IMAP support $0.99 / month.

In Outlook (or whatever email client) set up an IMAP account according to the instructions from your online provider. Then folder by folder copy all email into the Inbox on the newly created IMAP account. Don’t forget your Sent Mail folder: yes, that goes into the IMAP Inbox, too. Open all your archives and repeat the same process. Don’t worry if it takes a wile: Outlook doesn’t simply copy between local folders, it shoots up all your email to your temporary IMAP server on the web, and you’ll be constrained by your upstream speed (typically lower than downstream). If you have a spare PC, it’s a good idea to use that one.

Step 2: Now that your email is online, make sure POP access is enabled from your temporary account. If this is a gmail account (not IMAP), this is the setting you need:

“Enable POP for all mail (even mail that’s already been downloaded)”

Then in your Gmail target account – the final destination where you want to have all your archive mail – set up Gmail Mail Fetcher to pick up all mail from your temporary account. The dates will magically be fixed!

Here are Google’s instructions on setting up Mail Fetcher. Do NOT check the button for “Leave a copy of retrieved messages on the server” – you do want Mail Fetcher to “eat” them all from the temporary account, in fact that will be one of your indicators that the transfer is finished. Be prepared for a slow process – Gmail will poll your temporary account at 60-90 minute intervals, fetching 200 emails at a time. At Settings > Accounts you can follow the progress, but ignore the “nnn mails remaining” indicator, as it’s totally wrong. When all done, don’t be alarmed that the number of fetched emails is less than what you started with: your email client (and the IMAP server) counted individual emails, while Gmail will group them into thread, and reports the thread count, which could be significantly lower.

Last, but not least a word on labels / categories: if you nicely organized your Outlook archive in folders, Gmail has no way to preserve that structure. The trick here is to do Steps 1 and 2 in iterations, completely transferring one folder at a time. Then you can set a label for all your fetched email to match the original Outlook folder, and keep on changing it folder by folder.

Finally there is the issue of backup: after all we heard of disappearing Gmail… If you trust Gmail, just worried about what may happen to your individual account, there is always the option of setting up a shadow-gmail account which will fetch everything from your primary one. If you want a local archive, “just in case”, either run Outlook to periodically POP your mail down, or I believe Thunderbird has a plugin that allows it to be minimized to the system tray permanently and check the POP server in the background.

Update (3/28): One potential problem I forgot to mention is that all the fetched email becomes “unread”. Hard to believe, but Gmail does not have a “set all read” feature, and while there are some scripts, I’ve read stories of user accounts being suspended for 24 hours for scripting activity. If anyone has an idea how to changed all mail to “read” please comment below, I’ll bring it up here. Thanks.

Update to the update: The solution comes from Jason Brown, and it’s a surprise: Gmail has added a trick, I have no idea when. In Inbox (or wherever the messages are) choose “Select: All” from the list just above the message list. That will select all of the messages that are visible in the list – but here’s the surprise: At the top and bottom of the list where so it used to only say “All 100 conversations on this page are selected”, there is an additional clickable message: “Select all xxxx conversations in Inbox”, which will in fact select *all* the messages in the Inbox. Then choose “Mark as read” from the “More actions…” drop-down list. Job done! You can do this on Inbox, labels, or if you select All Mail, then on the entire account in a single step. Thanks, Jason!

Somewhat related: The Yahoo Mail announcement (unlimited storage) is picking up steam on Techmeme: Google Blogoscoped, PC World: Techlog,Techdirt, Google Operating System, Monkey Bites, CyberNet Technology News, michael parekh on IT, PaulStamatiou.com, Web Strategy, Download Squad, WebProNews, franticindustries, The Webpreneur, Search Engine Land, Liquidmatrix Security Digest, Conversion Rater, larry borsato, Gizmodo, CrunchGear, CenterNetworks and parislemon

Update (4/7): It’s somewhat obvious, but here’s a tip for backing up your archive online: create another (a third, fourth ..etc) Gmail account, configure Mail Fetcher there with your main account as the source, and voila! – you have a second, third..etc backup copy of all your email. I felt the need to spell this out upon reading Using Google Groups To Backup Gmail by The Google Tutor. It’s an interesting concept and nicely written up, but I think it’s built on fundamentally flawed logic:

• If you’re worried about losing content in your particular gmail account, why not get a second /third backup as I described above? You have the full gmail functionality, which you don’t get with Groups.. What’s the chance of losing all the accounts at the same time? Besides, this method will backup your “Sent” mail, too, which forwarding to Groups can’t help with.
• On the other hand, if you’re worried about Google in general, then why trust yet-another Google service? Groops is no safer than Gmail in that case.

Tags: email, email archive, email tools, gafyd, gayd, gmail, gmail loader, gml, Google, google apps, mail fetcher, Outlook, Personal Productivity, thunderbird