easyblock

Generic EasyBuild support for building and installing software. The EasyBlock class should serve as a base class for all easyblocks.

Authors:

• Stijn De Weirdt (Ghent University)
• Dries Verdegem (Ghent University)
• Kenneth Hoste (Ghent University)
• Pieter De Baets (Ghent University)
• Jens Timmerman (Ghent University)
• Toon Willems (Ghent University)
• Ward Poelmans (Ghent University)
• Fotis Georgatos (Uni.Lu, NTUA)
• Damian Alvarez (Forschungszentrum Juelich GmbH)
• Maxime Boissonneault (Compute Canada)
• Davide Vanzo (Vanderbilt University)
• Caspar van Leeuwen (SURF)

EasyBlock

Bases: object

Generic support for building and installing software, base class for actual easyblocks.

full_mod_name property

Full module name (including subdirectory in module install path)

mod_subdir property

Subdirectory in module install path

moduleGenerator property

Module generator (DEPRECATED, use self.module_generator instead).

name property

Shortcut the get the module name.

short_mod_name property

Short module name (not including subdirectory in module install path)

start_dir property

Start directory in build directory

toolchain property

Toolchain used to build this easyblock

version property

Shortcut the get the module version.

__init__(ec)

Initialize the EasyBlock instance.

PARAMETER	DESCRIPTION
ec	a parsed easyconfig file (EasyConfig instance)

apply_post_install_patches(patches=None)

Apply post-install patch files that are specified via the 'postinstallpatches' easyconfig parameter.

banned_linked_shared_libs()

List of shared libraries which are not allowed to be linked in any installed binary/library. Supported values are pure library names without 'lib' prefix or extension ('example'), file names ('libexample.so'), and full paths ('/usr/lib64/libexample.so').

bin_lib_subdirs()

List of subdirectories for binaries and libraries for this software installation. This is used during the sanity check to check RPATH linking and banned/required linked shared libraries.

build_step()

Build software (abstract method).

check_accepted_eula(name=None, more_info=None)

Check whether EULA for this software is accepted in current EasyBuild configuration.

check_checksums()

Check whether a SHA256 checksum is available for all sources & patches (incl. extensions).

RETURNS	DESCRIPTION
	list of strings describing checksum issues (missing checksums, wrong checksum type, etc.)

check_checksums_for(ent, sub='', source_cnt=None)

Utility method: check whether SHA256 checksums for all sources/patches are available, for given entity

check_readiness_step()

Verify if all is ok to start build.

checksum_step()

Verify checksum of sources and patches, if a checksum is available.

clean_up_fake_module(fake_mod_data)

Clean up fake module.

cleanup_step()

Cleanup leftover mess: remove/clean build directory

except when we're building in the installation directory or cleanup_builddir is False, otherwise we remove the installation

close_log()

Shutdown the logger.

collect_exts_file_info(fetch_files=True, verify_checksums=True)

Collect information on source and patch files for extensions.

PARAMETER	DESCRIPTION
fetch_files	whether or not to fetch files (if False, path to files will be missing from info) DEFAULT: True
verify_checksums	whether or not to verify checksums DEFAULT: True

RETURNS	DESCRIPTION
	list of dict values, one per extension, with information on source/patch files.

configure_step()

Configure build (abstract method).

det_iter_cnt()

Determine iteration count based on configure/build/install options that may be lists.

dry_run_msg(msg, *args)

Print dry run message.

extensions_step(fetch=False, install=True)

After make install, run this. - only if variable len(exts_list) > 0 - optionally: load module that was just created using temp module file - find source for extensions, in 'extensions' (and 'packages' for legacy reasons) - run extra_extensions

extra_options(extra=None) staticmethod

Extra options method which will be passed to the EasyConfig constructor.

extract_step()

Unpack the source files.

fetch_extension_sources(skip_checksums=False)

Fetch source and patch files for extensions (DEPRECATED, use collect_exts_file_info instead).

fetch_patches(patch_specs=None, extension=False, checksums=None)

Add a list of patches. All patches will be checked if a file exists (or can be located)

fetch_source(source, checksum=None, extension=False, download_instructions=None)

Get a specific source (tarball, iso, url) Will be tested for existence or can be located

PARAMETER	DESCRIPTION
source	source to be found (single dictionary in 'sources' list, or filename)
checksum	checksum corresponding to source DEFAULT: None
extension	flag if being called from collect_exts_file_info() DEFAULT: False

fetch_sources(sources=None, checksums=None)

Add a list of source files (can be tarballs, isos, urls). All source files will be checked if a file exists (or can be located)

PARAMETER	DESCRIPTION
sources	list of sources to fetch (if None, use 'sources' easyconfig parameter) DEFAULT: None
checksums	list of checksums for sources DEFAULT: None

fetch_step(skip_checksums=False)

Fetch source files and patches (incl. extensions).

fix_shebang()

Fix shebang lines for specified files.

gen_builddir()

Generate the (unique) name for the builddir

gen_installdir()

Generate the name of the installation directory.

get_checksum_for(checksums, filename=None, index=None)

Obtain checksum for given filename.

PARAMETER	DESCRIPTION
checksums	a list or tuple of checksums (or None)
filename	name of the file to obtain checksum for DEFAULT: None
index	index of file in list DEFAULT: None

get_checksums_from_json(always_read=False)

Get checksums for this software that are provided in a checksums.json file

PARAMETER	DESCRIPTION
always_read	always read the checksums.json file, even if it has been read before DEFAULT: False

get_steps(run_test_cases=True, iteration_count=1) staticmethod

Return a list of all steps to be performed.

guess_start_dir()

Return the directory where to start the whole configure/make/make install cycle from - typically self.src[0]['finalpath'] - start_dir option -- if abspath: use that -- else, treat it as subdir for regular procedure

handle_iterate_opts()

Handle options relevant during iterated part of build/install procedure.

init_dry_run()

Initialise easyblock instance for performing a dry run.

init_ext_instances()

Create class instances for all extensions.

install_extensions(install=True)

Install extensions.

PARAMETER	DESCRIPTION
install	actually install extensions, don't just prepare environment for installing DEFAULT: True

install_extensions_parallel(install=True)

Install extensions in parallel.

PARAMETER	DESCRIPTION
install	actually install extensions, don't just prepare environment for installing DEFAULT: True

install_extensions_sequential(install=True)

Install extensions sequentially.

PARAMETER	DESCRIPTION
install	actually install extensions, don't just prepare environment for installing DEFAULT: True

install_step()

Install built software (abstract method).

invalidate_module_caches(modpath)

Helper method to invalidate module caches for specified module path.

load_dependency_modules()

Load dependency modules.

load_fake_module(purge=False, extra_modules=None, verbose=False)

Create and load fake module.

PARAMETER	DESCRIPTION
purge	boolean indicating whether or not to purge currently loaded modules first DEFAULT: False
extra_modules	list of extra modules to load (these are loaded before loading the 'self' module) DEFAULT: None

load_module(mod_paths=None, purge=True, extra_modules=None, verbose=True)

Load module for this software package/version, after purging all currently loaded modules.

PARAMETER	DESCRIPTION
mod_paths	list of (additional) module paths to take into account DEFAULT: None
purge	boolean indicating whether or not to purge currently loaded modules first DEFAULT: True
extra_modules	list of extra modules to load (these are loaded before loading the 'self' module) DEFAULT: None
verbose	print modules being loaded when trace mode is enabled DEFAULT: True

make_builddir()

Create the build directory.

make_devel_module(create_in_builddir=False)

Create a develop module file which sets environment based on the build Usage: module load name, which loads the module you want to use. $EBDEVELNAME should then be the full path to the devel module file. So now you can module load $EBDEVELNAME.

WARNING: you cannot unload using $EBDEVELNAME (for now: use module unload basename $EBDEVELNAME)

make_dir(dir_name, clean, dontcreateinstalldir=False)

Create the directory.

make_extension_string(name_version_sep='-', ext_sep=', ', sort=True)

Generate a string with a list of extensions.

The name and version are separated by name_version_sep and each extension is separated by ext_sep

make_installdir(dontcreate=None)

Create the installation directory.

make_module_dep(unload_info=None)

Make the dependencies for the module file.

PARAMETER	DESCRIPTION
unload_info	dictionary with full module names as keys and module name to unload first as corr. value DEFAULT: None

make_module_deppaths()

Add specific 'module use' actions to module file, in order to find dependencies outside the end user's MODULEPATH.

make_module_description()

Create the module description.

make_module_extend_modpath()

Include prepend-path statements for extending $MODULEPATH.

make_module_extra(altroot=None, altversion=None)

Set extra stuff in module file, e.g. $EBROOT, $EBVERSION, etc.

PARAMETER	DESCRIPTION
altroot	path to use to define $EBROOT* DEFAULT: None
altversion	version to use to define $EBVERSION* DEFAULT: None

make_module_extra_extensions()

Sets optional variables for extensions.

make_module_footer()

Insert a footer section in the module file, primarily meant for contextual information

make_module_group_check()

Create the necessary group check.

make_module_req()

Generate the environment-variables to run the module.

make_module_req_guess()

A dictionary of possible directories to look for.

make_module_step(fake=False)

Generate module file

PARAMETER	DESCRIPTION
fake	generate 'fake' module in temporary location, rather than actual module file DEFAULT: False

obtain_file(filename, extension=False, urls=None, download_filename=None, force_download=False, git_config=None, no_download=False, download_instructions=None, alt_location=None, warning_only=False)

Locate the file with the given name - searches in different subdirectories of source path - supports fetching file from the web if path is specified as an url (i.e. starts with "http://:")

PARAMETER	DESCRIPTION
filename	filename of source
extension	indicates whether locations for extension sources should also be considered DEFAULT: False
urls	list of source URLs where this file may be available DEFAULT: None
download_filename	filename with which the file should be downloaded, and then renamed to DEFAULT: None
force_download	always try to download file, even if it's already available in source path DEFAULT: False
git_config	dictionary to define how to download a git repository DEFAULT: None
no_download	do not try to download the file DEFAULT: False
download_instructions	instructions to manually add source (used for complex cases) DEFAULT: None
alt_location	alternative location to use instead of self.name DEFAULT: None

package_step()

Package installed software (e.g., into an RPM), if requested, using selected package tool.

patch_step(beginpath=None, patches=None)

Apply the patches

permissions_step()

Finalize installation procedure: adjust permissions as configured, change group ownership (if requested). Installing user must be member of the group that it is changed to.

post_init()

Run post-initialization tasks.

post_install_step()

Do some postprocessing - run post install commands if any were specified

post_iter_step()

Restore options that were iterated over

prepare_for_extensions()

Ran before installing extensions (eg to set templates)

prepare_step(start_dir=True, load_tc_deps_modules=True)

Pre-configure step. Set's up the builddir just before starting configure

PARAMETER	DESCRIPTION
start_dir	guess start directory based on unpacked sources DEFAULT: True
load_tc_deps_modules	load modules for toolchain and dependencies in build environment DEFAULT: True

print_post_install_messages()

Print post-install messages that are specified via the 'postinstallmsgs' easyconfig parameter.

remove_module_file()

Remove module file (if it exists), and check for ghost installation directory (and deal with it).

report_test_failure(msg_or_error)

Report a failing test either via an exception or warning depending on ignore-test-failure

PARAMETER	DESCRIPTION
msg_or_error	failure description (string value or an EasyBuildError instance)

required_linked_shared_libs()

List of shared libraries which must be linked in all installed binaries/libraries. Supported values are pure library names without 'lib' prefix or extension ('example'), file names ('libexample.so'), and full paths ('/usr/lib64/libexample.so').

reset_env()

Reset environment. When iterating over builddependencies, every time we start a new iteration we need to restore the environment to where it was before the relevant modules were loaded.

run_all_steps(run_test_cases)

Build and install this software. run_test_cases (bool): run tests after building (e.g.: make test)

run_post_install_commands(commands=None)

Run post install commands that are specified via 'postinstallcmds' easyconfig parameter.

run_step(step, step_methods)

Run step, returns false when execution should be stopped

sanity_check_linked_shared_libs(subdirs=None)

Check whether specific shared libraries are (not) linked into installed binaries/libraries.

sanity_check_load_module(extension=False, extra_modules=None)

Load module to prepare environment for sanity check

sanity_check_rpath(rpath_dirs=None, check_readelf_rpath=True)

Sanity check binaries/libraries w.r.t. RPATH linking.

sanity_check_step(*args, **kwargs)

Do a sanity check on the installation - if any of the files/subdirectories in the installation directory listed in sanity_check_paths are non-existent (or empty), the sanity check fails

set_parallel()

Set 'parallel' easyconfig parameter to determine how many cores can/should be used for parallel builds.

set_up_cuda_cache()

Set up CUDA PTX cache.

skip_extensions()

Skip already installed extensions, by removing them from list of Extension instances to install (self.ext_instances).

This is done in parallel when EasyBuild is configured to install extensions in parallel.

skip_extensions_parallel(exts_filter)

Skip already installed extensions (checking in parallel), by removing them from list of Extension instances to install (self.ext_instances).

skip_extensions_sequential(exts_filter)

Skip already installed extensions (checking sequentially), by removing them from list of Extension instances to install (self.ext_instances).

skip_step(step, skippable)

Decide whether or not to skip the specified step.

stage_install_step()

Install in a stage directory before actual installation.

test_cases_step()

Run provided test cases.

test_step()

Run unit tests provided by software (if any).

update_config_template_run_step()

Update the the easyconfig template dictionary with easyconfig.TEMPLATE_NAMES_EASYBLOCK_RUN_STEP names

update_exts_progress_bar(info, progress_size=0, total=None)

Update extensions progress bar with specified info and amount of progress made

StopException

Bases: Exception

Exception thrown to stop running steps

build_and_install_one(ecdict, init_env)

Build the software

PARAMETER	DESCRIPTION
ecdict	dictionary contaning parsed easyconfig + metadata
init_env	original environment (used to reset environment)

build_easyconfigs(easyconfigs, output_dir, test_results)

Build the list of easyconfigs.

get_easyblock_instance(ecdict)

Get an instance for this easyconfig

PARAMETER	DESCRIPTION
ecdict	parsed easyconfig (EasyConfig instance) returns an instance of EasyBlock (or subclass thereof)

inject_checksums(ecs, checksum_type)

Inject checksums of given type in specified easyconfig files

PARAMETER	DESCRIPTION
ecs	list of EasyConfig instances to inject checksums into corresponding files
checksum_type	type of checksum to use

inject_checksums_to_json(ecs, checksum_type)

Inject checksums of given type in corresponding json files

PARAMETER	DESCRIPTION
ecs	list of EasyConfig instances to calculate checksums and inject them into checksums.json
checksum_type	type of checksum to use

print_dry_run_note(loc, silent=True)

Print note on interpreting dry run output.

reproduce_build(app, reprod_dir_root)

Create reproducibility files (processed easyconfig and easyblocks used) from class instance

PARAMETER	DESCRIPTION
app	easyblock class instance
reprod_dir_root	root directory in which to create the 'reprod' directory

RETURNS	DESCRIPTION
	reprod_dir directory containing reproducibility files