Optimizing buildouts with shared eggs and download caches

Most users should have this user-default configuration containing option settings that make Buildout work better:

[buildout]
eggs-directory = ~/.buildout/eggs
download-cache = ~/.buildout/download-cache
abi-tag-eggs = true

You might be wondering why these settings aren’t the default, if they’re recommended for everyone. They probably should be the default, and perhaps will be in version 3 of buildout. Making them the default now might break existing buildouts.

Shared eggs directory

You can save a lot of time and disk space by sharing eggs between buildouts. You can do this by setting the eggs-directory option, as shown above. This will override the default value for this option which puts eggs in the eggs buildout subdirectory. By sharing eggs, you can avoid reinstalling the same popular packages in each and every buildout that uses them.

ABI tag eggs

If you use a shared eggs directory, it’s a good idea to set the abi-tag-eggs option to true. This causes eggs to be segregated by ABI tag. This has two advantages:

  1. If you alternate between Python implementations (PyPy versus C Python) or between build configurations (normal versus debug), ABI tagging eggs will avoid mixing incompatible eggs.
  2. ABI tagging eggs makes Buildout run faster. Because ABI tags include Python version information, eggs for different Python versions are kept separate, causing the shared eggs directory for a given Python version to be smaller, making it faster to search for installed eggs.

Download cache

When buildout installs distributions, it has to download them first. Specifying a download-cache option in your user-default configuration causes the download to be cached. This can be helpful when multiple installations might be performed for a source distribution.

Some recipes download information. For example, a number of recipes download non-Python source archives and user configure, and make to install them. Most of these recipes can leverage a download cache to avoid downloading the same information over and over.