A lively discussion and exchange of information occurred recently on Hawk Wings, the blog site mostly devoted to news and resources for users of Apple’s terrific Mail program. A colleague at work sent me a message on Tuesday, excited when word on Hawk Wings started circulating about a “vacuum” process available for SQLite databases that appeared to dramatically speed up Apple Mail. He had tried the recommended vacuuming and definitely noticed peppier Mail performance. One thing led to another, and before I knew it, I’d become engrossed in developing and polishing up an AppleScript utility to automate a periodic vacuuming of my Mail, which I’m of course dubbing VacuumMail.

As the Hawk Wings discussion unfolded, we learned that Mail maintains an SQLite database called “Envelope Index” in your ~/Library/Mail folder, which gradually grows as the number of emails in your mailbox does. Natively, Mail performs no optimizations on this critical database, which contains pointers to all of your mail that become fragmented and somewhat disorganized over time. At the office, my Envelope Index file was over 100mb, and at home it’s about 30mb. SQLite offers a “vacuum” command that rewrites the Envelope Index, optimizing and reorganizing it for faster access. It sounds a bit like what happens when Mac OS X defragments your hard drive periodically.

At first, news of this function took the form of a shell command you can run in Terminal. It was quite interesting and exciting to see how the Mac users reading of this learned more about it as information was shared, and the command itself became more concise and precise as the day went on. Other users discovered that SQLite offers an “autovacuum” process that can do vacuuming without prompting, and I’m sure that’s a great thing as well. However, we also learned that vacuuming is a more robust and thorough optimizing of the file, since it actually analyzes and rewrites the whole thing, whereas autovacuuming acts only on a certain recent portion of mail pointers. The basic Terminal command turns out to be:

sqlite3 ~/Library/Mail/Envelope Index vacuum;

One very important lesson learned is that vacuuming is only available in the version of SQLite that comes with Mac OS X 10.4 (”Tiger”), and that “autovacuuming” is only available starting with 10.4.8. Of course, this should be fine with Panther users, since the problems I’ve had with Mail only really started with Tiger. It may be that use of Envelope Index itself started then as well.

So it wasn’t long before several people offered up AppleScripts to help automate the process, because who wants to fire up Terminal if you don’t have to? And by the end of the day a helpful reader had prepared and offered for download an Automator action you can use, with instructions for scheduling it through iCal.

Me, I saw an opportunity for combining some of the best ideas offered and for refining the way the AppleScript interacts with the Mail user. For one thing, I wanted a script that would run with no interaction at all, since sometimes that’s what I’d want. For example, if Mail isn’t running, the script doesn’t have to close Mail or start Mail up again afterwards. Further, I wanted a script I could schedule through Apple’s terrific, but still underused Launchd service, yet could also be run manually if necessary.

So, I took the opportunity to learn a little AppleScript and a little more about Launchd, as well as even a bit more about SQLite and Unix as well, to modify the script to my needs. Not being satisfied with that, I decided to also learn how to make an application bundle out of the script, so I could add a custom icon and maybe further refinements later on.

As my colleague noted, this whole episode is the latest that demonstrates what a great resource blog sites like Hawk Wings have become for Mac users. An outgrowth of one man’s personal interest and desire to develop a useful resource around one application (Apple Mail), Hawk Wings is an information resource that never could have existed prior to the advent of blogging. Mail is simply too narrow a topic to deserve a whole magazine or newsletter, and none of the traditional news outlets for Mac users would have been able or willing to devote the resources to the care and feeding of a website like Hawk Wings. So, my little foray is an attempt to give something back and to say “thanks” to Hawk Wings and its many readers.

The end result of my small effort is VacuumMail, which can be used as a regular application or automated through the included LaunchAgent for use with Launchd. I also took a few minutes to document the script and provide some information about Launchd and how to manage LaunchAgents and Services. The download package has three files: VacuumMail, the launch agent (which is just a .plist file), and a page of documentation.

Enjoy!

Download VacuumMail

Note: The following is the VacuumMail documentation included with the application package:

VacuumMail

Version 0.9.6
© 2007 by Leland Scott
Based on OptimizeMail by Sebastian Morsch, OptimiseEnvelope by pmbuko, and various other ideas and scraps of code on the HawkWings website.

Basic Operation

VacuumMail is an AppleScript application that performs the “vacuum” command on Apple Mail’s underlying SQLite Envelope Index database. This must not be done while Mail is running, so this scriptlet first checks to see if Mail is running and if so, quits Mail. VacuumMail first notifies the user that it will be shutting Mail down, giving the user an opportunity to cancel the vacuum operation. This dialog window will close automatically, quit Mail, and commence vacuuming in 8 seconds if the user doesn’t click on the “Cancel” button.

VacuumMail is designed to be incorporated into an automated maintenance routine and run at regular intervals. According to the SQLite documentation, the vacuum command does a more thorough optimization of Mail’s database than the “autovacuum” setting that’s supported in Mac OS X beginning with version 10.4.8.

Alternatively, you can launch VacuumMail interactively if you need to do a manual run. I keep VacuumMail in my Finder toolbar, so it’s handy to launch.

If Mail was running when the process begins, VacuumMail will restart Mail when it’s finished. The vacuum process can take several minutes if your Envelope Index is very large, and in this first version VacuumMail doesn’t display any kind of progress indicator. However, when it’s finished, VacuumMail will display a dialog showing the size of your Envelope Index file before the latest vacuuming as well as afterwards. This dialog will close after a few seconds if you don’t close it manually.

VacuumMail can be installed anywhere on your system (unless you use the Launch Agent) but expects to find your ~/Library/Mail folder at that location. I did modify the shell commands so they’ll work with symbolic links, in case you, like me, store your Mail folder on a different partition but have it symbolically linked to your Library folder.

Using the VacuumMail LaunchAgent

The VacuumMail installer also includes a Launch Agent for launchd that you can use to automate the running of VacuumMail. This file, org.musingsfrommars.vacuumMail.plist, will be installed in your ~/Library/LaunchAgents folder. The installer will create this folder for you if you don’t already have one.

This agent assumes that you will put VacuumMail in your /Applications/Utilities/ folder, so that’s where this installer puts the application. If you want to put it somewhere else, you’ll need to edit the .plist file… either using the terrific, open source launchd editor Lingon, or in your favorite .plist editor. (I recommend one called PrefSetter, a free .plist editor that’s a big step up from the tool Apple distributes with its Developer Tools package.)

This agent will run VacuumMail every Tuesday afternoon at 1:00pm. I wanted to vacuum my mail weekly, and that time was chosen somewhat arbitrarily. If you want a different day and time, or a different time interval, it’s easy enough to change that behavior using Lingon. The other launchd option this agent utilizes is its ability to run in low priority mode. The launch agent runs with a “nice” priority setting of 15, so it will readily yield processor privileges to other applications on your system when it runs.

About Lingon and Launchd

Lingon is a graphical interface for creating launchd configuration files and controlling them through launchctl for Mac OS X Tiger. It was developed by Peter Borg and is available as open source from SourceForge. The Lingon website has good documentation for its use, as does the application itself.

The VacuumMail installer includes an option for you to install Lingon.

Launchd is a new system startup program Apple introduced in Tiger. The launchd daemon takes over many tasks from cron, xinetd, mach_init, and init, which are UNIX programs that traditionally have handled system initialization, called systems scripts, run startup items, and generally prepared the system for the user. And they still exist on Mac OS X Tiger, but launchd has superseded them in many instances. Apple’s use of launchd instead of the traditional UNIX programs provides a big performance boost to Tiger users. At any given time, only those daemons that are actually used are launched; combined with the fact that daemons can shut themselves down and be relaunched as needed means that launchd can reduce the average memory footprint of the system. For more information, see Apple’s excellent documentation on Launchd, from which this description is summarized. In addition, I noticed tonight that Lingon is one of the projects managed through the relatively new MacOS forge website, which is a SourceForge-like gathering place for Apple-initiated open source projects: Definitely checkout the Lingon site on MacOS forge for the latest and greatest developments with Launchd, or if you’d like to contribute some of your brains to the project.

License

VacuumMail is © 2007 by Leland Scott. It is released as freeware under creative commons license 2.5, which means you can modify it and use it freely except

• In commercial projects
• Without giving proper credit (Leland Scott)

Lingon is © 2006 by Peter Borg. It is released through SourceForge under the BSD license and can be used subject to those terms. It is included in the VacuumMail package with the generous permission of its author.

Contact Information

If you have feedback about VacuumMail, contact Leland Scott at llscotts [at] fastmail.fm. However, I won’t be able to provide support for the fundamental components of the application: SQLite, Apple Mail, Unix commands, AppleScript, Launchd, or Lingon. I suggest you read through the very interesting dialog about all of this on the HawkWings website, linked previously.

Peter Borg can be contacted through the Lingon website.

Change Log