Using an index to speed up searching for easyconfigs¶

EasyBuild often needs to search for Easyconfig files (or accompanying files like patches), either based on command line arguments (like the name of an easyconfig file) or options to the eb command (like --search, see Searching for easyconfigs), or to resolve dependencies for a particular easyconfig file that was parsed already.

Searching for easyconfig files or patches may take a while, since it can potentially involve weeding through thousands of files, which may be located on a shared filesystem (where metadata-intensive operations like file searching can be rather slow).

If EasyBuild turns out to be sluggish when searching for easyconfig files in your setup, using an index file could make a big difference.

Note

Support for index files was implemented in EasyBuild version 4.2.0 .

Creating an index¶

(--create-index)

eb --create-index can be used to create an index file for a particular directory that holds a (large) set of easyconfig files.

The index file (a hidden file named .eb-path-index) will be created in the specified directory. It will contain a list of (relative) paths for all files in that directory, along with some metadata: the time at which the index file was created and a timestamp that indicates how long the index is considered to be valid (see Controlling how long the index is valid).

Note

Make sure to create an index for the correct path.

The search for easyconfig files performed by EasyBuild will not recurse into subdirectories of the locations it considers (see Searching for easyconfigs: the robot search path), other than those with a name matching the software for which it is trying to find an easyconfig file (like t/TensorFlow/ when searching for an easyconfig file for TensorFlow).

Hence, if the directory housing your easyconfig files has an easybuild/easyconfigs subdirectory (for example, a working copy of the easybuild-easyconfigs repository), create the index in that subdirectory, rather than in the higher-level directory.

Example of creating an index¶

$ eb --create-index $HOME/easybuild-easyconfigs/easybuild/easyconfigs
== temporary log file in case of crash /tmp/eb-tUu6_w/easybuild-SKBnVO.log
Creating index for /home/example/easybuild-easyconfigs/easybuild/easyconfigs...
== found valid index for /home/example/easybuild-easyconfigs/easybuild/easyconfigs, so using it...
Index created at /home/example/easybuild-easyconfigs/easybuild/easyconfigs/.eb-path-index (738 files)

$ head -n 5 $HOME/easybuild-easyconfigs/easybuild/easyconfigs/.eb-path-index
# created at: 2020-04-13 14:19:57.008981
# valid until: 2020-04-20 14:19:57.008981
a/Arrow/Arrow-0.16.0-intel-2019b-Python-3.7.4.eb
b/Boost/Boost-1.71.0-gompi-2019b.eb
b/bokeh/bokeh-1.4.0-foss-2019b-Python-3.7.4.eb

Note

The "found valid index ..." message being printed occurs because EasyBuild tries to load the index file right after creating it, as a sanity check.

Updating an existing index¶

(--create-index --force)

To update an existing index, you can use --create-index --force.

Without using --force, EasyBuild will refuse to overwrite the existing index file:

$ eb --create-index $HOME/easybuild-easyconfigs/easybuild/easyconfigs
== temporary log file in case of crash /tmp/eb-tUu6_w/easybuild-SKBnVO.log
Creating index for /home/example/easybuild-easyconfigs/easybuild/easyconfigs...
ERROR: File exists, not overwriting it without --force: /home/example/easybuild-easyconfigs/easybuild/easyconfigs/.eb-path-index

Using index files¶

EasyBuild will automatically pick up and use any index file that it runs into while searching for easyconfig files or patches. If an index file is found, it will be preferred over walking through the directory tree to check for the target file, which is likely to significantly speed up the search operation.

When a (valid) index file is found for a particular path, a message will be printed mentioning "found valid index for...":

$ eb --search TensorFlow-2.1.0-foss-2019b
== found valid index for /home/example/easybuild-easyconfigs/easybuild/easyconfigs, so using it...
* /home/example/easybuild-easyconfigs/easybuild/easyconfigs/t/TensorFlow/TensorFlow-2.1.0-foss-2019b-Python-3.7.4.eb

Ignoring indices¶

(--ignore-index)

One potential issue with having an index in place is that it may get outdated: new files may have been added to the directory since the index was created or last updated.

If updating the indexes is not an option (see Updating an existing index), you can instruct EasyBuild to ignore any existing indices using the --ignore-index configuration option.

The only downside of this option is that searching for easyconfig files may be significantly slower. Any existing index files are left untouched (they will not be updated or removed).

Controlling how long the index is valid¶

(--index-max-age)

When creating an index file, you can specify how long the index should be considered valid.

Using the --index-max-age configuration option, you can specify how long after the creation time the index remains valid (in seconds).

By default, EasyBuild will consider index files to remain valid for 1 week (7 * 24 * 60 * 60 = 604,800 seconds).

To create an index that always remains valid (never expires), use zero (0) as value for --index-max-age:

$ eb --index-max-age=0 --create-index $HOME/easybuild-easyconfigs/easybuild/easyconfigs

$ head -n 2 $HOME/easybuild-easyconfigs/easybuild/easyconfigs/.eb-path-index
# created at: 2020-04-13 15:10:07.173191
# valid until: 9999-12-31 23:59:59.999999

Note

Trust us here, December 31st 9999 is the end of times. Better get prepared.

Index included with EasyBuild releases¶

Each EasyBuild release (since EasyBuild v4.2.0) comes with an index file for the easyconfig (and patch) files that are included with that release.

Hence, you only need to use --create-index to create/update the index file for any additional directories with easyconfig files you may have on the side (and only if searching those easyconfig files is rather slow).

Should I create an index?¶

Whether or not you should create an index file for your directories housing additional easyconfig files depends on a number of factors, including:

how often files are added and/or removed in those directories, since files listed in the index are assumed to be there and any files not included in the index will be overlooked by EasyBuild when it's searching for files;
the filesystem on which those directories are located, since an index file will only make a significant difference on filesystems where metadata-intensive operations are relatively slow;
how many files there are in those directories, since performance benefits will only be apparent if the number if files is sufficiently large;

Note

Keep in mind that creating an index implies also updating it frequently, to ensure that EasyBuild will take all available files in account.