easybuild.framework.easyblock module

Generic EasyBuild support for building and installing software. The EasyBlock class should serve as a base class for all easyblocks.

author:Stijn De Weirdt (Ghent University)
author:Dries Verdegem (Ghent University)
author:Kenneth Hoste (Ghent University)
author:Pieter De Baets (Ghent University)
author:Jens Timmerman (Ghent University)
author:Toon Willems (Ghent University)
author:Ward Poelmans (Ghent University)
author:Fotis Georgatos (Uni.Lu, NTUA)
author:Damian Alvarez (Forschungszentrum Juelich GmbH)
author:Maxime Boissonneault (Compute Canada)
author:Davide Vanzo (Vanderbilt University)
class easybuild.framework.easyblock.EasyBlock(ec)

Bases: object

Generic support for building and installing software, base class for actual easyblocks.


Apply post-install patch files that are specified via the ‘postinstallpatches’ easyconfig parameter.


List of shared libraries which are not allowed to be linked in any installed binary/library. Supported values are pure library names without ‘lib’ prefix or extension (‘example’), file names (‘libexample.so’), and full paths (‘/usr/lib64/libexample.so’).


List of subdirectories for binaries and libraries for this software installation. This is used during the sanity check to check RPATH linking and banned/required linked shared libraries.


Build software (abstract method).

check_accepted_eula(name=None, more_info=None)

Check whether EULA for this software is accepted in current EasyBuild configuration.


Check whether a SHA256 checksum is available for all sources & patches (incl. extensions).

Returns:list of strings describing checksum issues (missing checksums, wrong checksum type, etc.)
check_checksums_for(ent, sub='', source_cnt=None)

Utility method: check whether SHA256 checksums for all sources/patches are available, for given entity


Verify if all is ok to start build.


Verify checksum of sources and patches, if a checksum is available.


Clean up fake module.


Cleanup leftover mess: remove/clean build directory

except when we’re building in the installation directory or cleanup_builddir is False, otherwise we remove the installation


Shutdown the logger.

collect_exts_file_info(fetch_files=True, verify_checksums=True)

Collect information on source and patch files for extensions.

  • fetch_files – whether or not to fetch files (if False, path to files will be missing from info)
  • verify_checksums – whether or not to verify checksums

list of dict values, one per extension, with information on source/patch files.


Configure build (abstract method).


Determine iteration count based on configure/build/install options that may be lists.

dry_run_msg(msg, *args)

Print dry run message.

extensions_step(fetch=False, install=True)

After make install, run this. - only if variable len(exts_list) > 0 - optionally: load module that was just created using temp module file - find source for extensions, in ‘extensions’ (and ‘packages’ for legacy reasons) - run extra_extensions

static extra_options(extra=None)

Extra options method which will be passed to the EasyConfig constructor.


Unpack the source files.


Fetch source and patch files for extensions (DEPRECATED, use collect_exts_file_info instead).

fetch_patches(patch_specs=None, extension=False, checksums=None)

Add a list of patches. All patches will be checked if a file exists (or can be located)

fetch_source(source, checksum=None, extension=False, download_instructions=None)

Get a specific source (tarball, iso, url) Will be tested for existence or can be located

  • source – source to be found (single dictionary in ‘sources’ list, or filename)
  • checksum – checksum corresponding to source
  • extension – flag if being called from collect_exts_file_info()
fetch_sources(sources=None, checksums=None)

Add a list of source files (can be tarballs, isos, urls). All source files will be checked if a file exists (or can be located)

  • sources – list of sources to fetch (if None, use ‘sources’ easyconfig parameter)
  • checksums – list of checksums for sources

Fetch source files and patches (incl. extensions).


Fix shebang lines for specified files.


Full module name (including subdirectory in module install path)


Generate the (unique) name for the builddir


Generate the name of the installation directory.

get_checksum_for(checksums, filename=None, index=None)

Obtain checksum for given filename.

  • checksums – a list or tuple of checksums (or None)
  • filename – name of the file to obtain checksum for (Deprecated)
  • index – index of file in list
static get_steps(run_test_cases=True, iteration_count=1)

Return a list of all steps to be performed.


Return the directory where to start the whole configure/make/make install cycle from - typically self.src[0][‘finalpath’] - start_dir option – if abspath: use that – else, treat it as subdir for regular procedure


Handle options relevant during iterated part of build/install procedure.


Initialise easyblock instance for performing a dry run.


Create class instances for all extensions.


Install extensions.

Parameters:install – actually install extensions, don’t just prepare environment for installing

Install extensions in parallel.

Parameters:install – actually install extensions, don’t just prepare environment for installing

Install extensions sequentially.

Parameters:install – actually install extensions, don’t just prepare environment for installing

Install built software (abstract method).


Helper method to invalidate module caches for specified module path.


Load dependency modules.

load_fake_module(purge=False, extra_modules=None, verbose=False)

Create and load fake module.

  • purge – boolean indicating whether or not to purge currently loaded modules first
  • extra_modules – list of extra modules to load (these are loaded before loading the ‘self’ module)
load_module(mod_paths=None, purge=True, extra_modules=None, verbose=True)

Load module for this software package/version, after purging all currently loaded modules.

  • mod_paths – list of (additional) module paths to take into account
  • purge – boolean indicating whether or not to purge currently loaded modules first
  • extra_modules – list of extra modules to load (these are loaded before loading the ‘self’ module)
  • verbose – print modules being loaded when trace mode is enabled

Create the build directory.


Create a develop module file which sets environment based on the build Usage: module load name, which loads the module you want to use. $EBDEVELNAME should then be the full path to the devel module file. So now you can module load $EBDEVELNAME.

WARNING: you cannot unload using $EBDEVELNAME (for now: use module unload basename $EBDEVELNAME)

make_dir(dir_name, clean, dontcreateinstalldir=False)

Create the directory.


Create the installation directory.


Make the dependencies for the module file.

Parameters:unload_info – dictionary with full module names as keys and module name to unload first as corr. value

Add specific ‘module use’ actions to module file, in order to find dependencies outside the end user’s MODULEPATH.


Create the module description.


Include prepend-path statements for extending $MODULEPATH.

make_module_extra(altroot=None, altversion=None)

Set extra stuff in module file, e.g. $EBROOT*, $EBVERSION*, etc.

  • altroot – path to use to define $EBROOT*
  • altversion – version to use to define $EBVERSION*

Sets optional variables for extensions.

Insert a footer section in the module file, primarily meant for contextual information


Create the necessary group check.


Generate the environment-variables to run the module.


A dictionary of possible directories to look for.


Generate module file

Parameters:fake – generate ‘fake’ module in temporary location, rather than actual module file

Subdirectory in module install path


Module generator (DEPRECATED, use self.module_generator instead).


Shortcut the get the module name.

obtain_file(filename, extension=False, urls=None, download_filename=None, force_download=False, git_config=None, download_instructions=None)

Locate the file with the given name - searches in different subdirectories of source path - supports fetching file from the web if path is specified as an url (i.e. starts with “http://:”) :param filename: filename of source :param extension: indicates whether locations for extension sources should also be considered :param urls: list of source URLs where this file may be available :param download_filename: filename with which the file should be downloaded, and then renamed to <filename> :param force_download: always try to download file, even if it’s already available in source path :param git_config: dictionary to define how to download a git repository


Package installed software (e.g., into an RPM), if requested, using selected package tool.

patch_step(beginpath=None, patches=None)

Apply the patches


Finalize installation procedure: adjust permissions as configured, change group ownership (if requested). Installing user must be member of the group that it is changed to.


Run post-initialization tasks.


Do some postprocessing - run post install commands if any were specified


Restore options that were iterated over


Also do this before (eg to set the template)

prepare_step(start_dir=True, load_tc_deps_modules=True)

Pre-configure step. Set’s up the builddir just before starting configure

  • start_dir – guess start directory based on unpacked sources
  • load_tc_deps_modules – load modules for toolchain and dependencies in build environment

Remove module file (if it exists), and check for ghost installation directory (and deal with it).


Report a failing test either via an exception or warning depending on ignore-test-failure

Parameters:msg_or_error – failure description (string value or an EasyBuildError instance)

List of shared libraries which must be linked in all installed binaries/libraries. Supported values are pure library names without ‘lib’ prefix or extension (‘example’), file names (‘libexample.so’), and full paths (‘/usr/lib64/libexample.so’).


Reset environment. When iterating over builddependencies, every time we start a new iteration we need to restore the environment to where it was before the relevant modules were loaded.


Build and install this software. run_test_cases (bool): run tests after building (e.g.: make test)


Run post install commands that are specified via ‘postinstallcmds’ easyconfig parameter.

run_step(step, step_methods)

Run step, returns false when execution should be stopped


Check whether specific shared libraries are (not) linked into installed binaries/libraries.


Sanity check binaries/libraries w.r.t. RPATH linking.

sanity_check_step(*args, **kwargs)

Do a sanity check on the installation - if any of the files/subdirectories in the installation directory listed

in sanity_check_paths are non-existent (or empty), the sanity check fails

Set ‘parallel’ easyconfig parameter to determine how many cores can/should be used for parallel builds.


Set up CUDA PTX cache.


Short module name (not including subdirectory in module install path)


Skip already installed extensions, by removing them from list of Extension instances to install (self.ext_instances).

This is done in parallel when EasyBuild is configured to install extensions in parallel.


Skip already installed extensions (checking in parallel), by removing them from list of Extension instances to install (self.ext_instances).


Skip already installed extensions (checking sequentially), by removing them from list of Extension instances to install (self.ext_instances).

skip_step(step, skippable)

Dedice whether or not to skip the specified step.


Install in a stage directory before actual installation.


Start directory in build directory


Run provided test cases.


Run unit tests provided by software (if any).


Toolchain used to build this easyblock


Update the the easyconfig template dictionary with easyconfig.TEMPLATE_NAMES_EASYBLOCK_RUN_STEP names

update_exts_progress_bar(info, progress_size=0, total=None)

Update extensions progress bar with specified info and amount of progress made


Shortcut the get the module version.

exception easybuild.framework.easyblock.StopException

Bases: Exception

StopException class definition.

easybuild.framework.easyblock.build_and_install_one(ecdict, init_env)

Build the software :param ecdict: dictionary contaning parsed easyconfig + metadata :param init_env: original environment (used to reset environment)

easybuild.framework.easyblock.build_easyconfigs(easyconfigs, output_dir, test_results)

Build the list of easyconfigs.

easybuild.framework.easyblock.copy_easyblocks_for_reprod(easyblock_instances, reprod_dir)

Get an instance for this easyconfig :param easyconfig: parsed easyconfig (EasyConfig instance)

returns an instance of EasyBlock (or subclass thereof)

easybuild.framework.easyblock.inject_checksums(ecs, checksum_type)

Inject checksums of given type in specified easyconfig files

  • ecs – list of EasyConfig instances to inject checksums into corresponding files
  • checksum_type – type of checksum to use
easybuild.framework.easyblock.print_dry_run_note(loc, silent=True)

Print note on interpreting dry run output.

easybuild.framework.easyblock.reproduce_build(app, reprod_dir_root)

Create reproducibility files (processed easyconfig and easyblocks used) from class instance

  • app – easyblock class instance
  • reprod_dir_root – root directory in which to create the ‘reprod’ directory
Return reprod_dir:

directory containing reproducibility files