easybuild.framework.easyblock module¶
Generic EasyBuild support for building and installing software. The EasyBlock class should serve as a base class for all easyblocks.
author: | Stijn De Weirdt (Ghent University) |
---|---|
author: | Dries Verdegem (Ghent University) |
author: | Kenneth Hoste (Ghent University) |
author: | Pieter De Baets (Ghent University) |
author: | Jens Timmerman (Ghent University) |
author: | Toon Willems (Ghent University) |
author: | Ward Poelmans (Ghent University) |
author: | Fotis Georgatos (Uni.Lu, NTUA) |
author: | Damian Alvarez (Forschungszentrum Juelich GmbH) |
author: | Maxime Boissonneault (Compute Canada) |
author: | Davide Vanzo (Vanderbilt University) |
-
class
easybuild.framework.easyblock.
EasyBlock
(ec)¶ Bases:
object
Generic support for building and installing software, base class for actual easyblocks.
-
apply_post_install_patches
(patches=None)¶ Apply post-install patch files that are specified via the ‘postinstallpatches’ easyconfig parameter.
List of shared libraries which are not allowed to be linked in any installed binary/library. Supported values are pure library names without ‘lib’ prefix or extension (‘example’), file names (‘libexample.so’), and full paths (‘/usr/lib64/libexample.so’).
-
bin_lib_subdirs
()¶ List of subdirectories for binaries and libraries for this software installation. This is used during the sanity check to check RPATH linking and banned/required linked shared libraries.
-
build_step
()¶ Build software (abstract method).
-
check_accepted_eula
(name=None, more_info=None)¶ Check whether EULA for this software is accepted in current EasyBuild configuration.
-
check_checksums
()¶ Check whether a SHA256 checksum is available for all sources & patches (incl. extensions).
Returns: list of strings describing checksum issues (missing checksums, wrong checksum type, etc.)
-
check_checksums_for
(ent, sub='', source_cnt=None)¶ Utility method: check whether SHA256 checksums for all sources/patches are available, for given entity
-
check_readiness_step
()¶ Verify if all is ok to start build.
-
checksum_step
()¶ Verify checksum of sources and patches, if a checksum is available.
-
clean_up_fake_module
(fake_mod_data)¶ Clean up fake module.
-
cleanup_step
()¶ Cleanup leftover mess: remove/clean build directory
except when we’re building in the installation directory or cleanup_builddir is False, otherwise we remove the installation
-
close_log
()¶ Shutdown the logger.
-
collect_exts_file_info
(fetch_files=True, verify_checksums=True)¶ Collect information on source and patch files for extensions.
Parameters: - fetch_files – whether or not to fetch files (if False, path to files will be missing from info)
- verify_checksums – whether or not to verify checksums
Returns: list of dict values, one per extension, with information on source/patch files.
-
configure_step
()¶ Configure build (abstract method).
-
det_iter_cnt
()¶ Determine iteration count based on configure/build/install options that may be lists.
-
dry_run_msg
(msg, *args)¶ Print dry run message.
-
extensions_step
(fetch=False, install=True)¶ After make install, run this. - only if variable len(exts_list) > 0 - optionally: load module that was just created using temp module file - find source for extensions, in ‘extensions’ (and ‘packages’ for legacy reasons) - run extra_extensions
-
static
extra_options
(extra=None)¶ Extra options method which will be passed to the EasyConfig constructor.
-
extract_step
()¶ Unpack the source files.
-
fetch_extension_sources
(skip_checksums=False)¶ Fetch source and patch files for extensions (DEPRECATED, use collect_exts_file_info instead).
-
fetch_patches
(patch_specs=None, extension=False, checksums=None)¶ Add a list of patches. All patches will be checked if a file exists (or can be located)
-
fetch_source
(source, checksum=None, extension=False, download_instructions=None)¶ Get a specific source (tarball, iso, url) Will be tested for existence or can be located
Parameters: - source – source to be found (single dictionary in ‘sources’ list, or filename)
- checksum – checksum corresponding to source
- extension – flag if being called from collect_exts_file_info()
-
fetch_sources
(sources=None, checksums=None)¶ Add a list of source files (can be tarballs, isos, urls). All source files will be checked if a file exists (or can be located)
Parameters: - sources – list of sources to fetch (if None, use ‘sources’ easyconfig parameter)
- checksums – list of checksums for sources
-
fetch_step
(skip_checksums=False)¶ Fetch source files and patches (incl. extensions).
-
fix_shebang
()¶ Fix shebang lines for specified files.
-
full_mod_name
¶ Full module name (including subdirectory in module install path)
-
gen_builddir
()¶ Generate the (unique) name for the builddir
-
gen_installdir
()¶ Generate the name of the installation directory.
-
get_checksum_for
(checksums, filename=None, index=None)¶ Obtain checksum for given filename.
Parameters: - checksums – a list or tuple of checksums (or None)
- filename – name of the file to obtain checksum for (Deprecated)
- index – index of file in list
-
static
get_steps
(run_test_cases=True, iteration_count=1)¶ Return a list of all steps to be performed.
-
guess_start_dir
()¶ Return the directory where to start the whole configure/make/make install cycle from - typically self.src[0][‘finalpath’] - start_dir option – if abspath: use that – else, treat it as subdir for regular procedure
-
handle_iterate_opts
()¶ Handle options relevant during iterated part of build/install procedure.
-
init_dry_run
()¶ Initialise easyblock instance for performing a dry run.
-
init_ext_instances
()¶ Create class instances for all extensions.
-
install_extensions
(install=True)¶ Install extensions.
Parameters: install – actually install extensions, don’t just prepare environment for installing
-
install_extensions_parallel
(install=True)¶ Install extensions in parallel.
Parameters: install – actually install extensions, don’t just prepare environment for installing
-
install_extensions_sequential
(install=True)¶ Install extensions sequentially.
Parameters: install – actually install extensions, don’t just prepare environment for installing
-
install_step
()¶ Install built software (abstract method).
-
invalidate_module_caches
(modpath)¶ Helper method to invalidate module caches for specified module path.
-
load_dependency_modules
()¶ Load dependency modules.
-
load_fake_module
(purge=False, extra_modules=None, verbose=False)¶ Create and load fake module.
Parameters: - purge – boolean indicating whether or not to purge currently loaded modules first
- extra_modules – list of extra modules to load (these are loaded before loading the ‘self’ module)
-
load_module
(mod_paths=None, purge=True, extra_modules=None, verbose=True)¶ Load module for this software package/version, after purging all currently loaded modules.
Parameters: - mod_paths – list of (additional) module paths to take into account
- purge – boolean indicating whether or not to purge currently loaded modules first
- extra_modules – list of extra modules to load (these are loaded before loading the ‘self’ module)
- verbose – print modules being loaded when trace mode is enabled
-
make_builddir
()¶ Create the build directory.
-
make_devel_module
(create_in_builddir=False)¶ Create a develop module file which sets environment based on the build Usage: module load name, which loads the module you want to use. $EBDEVELNAME should then be the full path to the devel module file. So now you can module load $EBDEVELNAME.
WARNING: you cannot unload using $EBDEVELNAME (for now: use module unload basename $EBDEVELNAME)
-
make_dir
(dir_name, clean, dontcreateinstalldir=False)¶ Create the directory.
-
make_installdir
(dontcreate=None)¶ Create the installation directory.
-
make_module_dep
(unload_info=None)¶ Make the dependencies for the module file.
Parameters: unload_info – dictionary with full module names as keys and module name to unload first as corr. value
-
make_module_deppaths
()¶ Add specific ‘module use’ actions to module file, in order to find dependencies outside the end user’s MODULEPATH.
-
make_module_description
()¶ Create the module description.
-
make_module_extend_modpath
()¶ Include prepend-path statements for extending $MODULEPATH.
-
make_module_extra
(altroot=None, altversion=None)¶ Set extra stuff in module file, e.g. $EBROOT*, $EBVERSION*, etc.
Parameters: - altroot – path to use to define $EBROOT*
- altversion – version to use to define $EBVERSION*
-
make_module_extra_extensions
()¶ Sets optional variables for extensions.
Insert a footer section in the module file, primarily meant for contextual information
-
make_module_group_check
()¶ Create the necessary group check.
-
make_module_req
()¶ Generate the environment-variables to run the module.
-
make_module_req_guess
()¶ A dictionary of possible directories to look for.
-
make_module_step
(fake=False)¶ Generate module file
Parameters: fake – generate ‘fake’ module in temporary location, rather than actual module file
-
mod_subdir
¶ Subdirectory in module install path
-
moduleGenerator
¶ Module generator (DEPRECATED, use self.module_generator instead).
-
name
¶ Shortcut the get the module name.
-
obtain_file
(filename, extension=False, urls=None, download_filename=None, force_download=False, git_config=None, download_instructions=None)¶ Locate the file with the given name - searches in different subdirectories of source path - supports fetching file from the web if path is specified as an url (i.e. starts with “http://:”) :param filename: filename of source :param extension: indicates whether locations for extension sources should also be considered :param urls: list of source URLs where this file may be available :param download_filename: filename with which the file should be downloaded, and then renamed to <filename> :param force_download: always try to download file, even if it’s already available in source path :param git_config: dictionary to define how to download a git repository
-
package_step
()¶ Package installed software (e.g., into an RPM), if requested, using selected package tool.
-
patch_step
(beginpath=None, patches=None)¶ Apply the patches
-
permissions_step
()¶ Finalize installation procedure: adjust permissions as configured, change group ownership (if requested). Installing user must be member of the group that it is changed to.
-
post_init
()¶ Run post-initialization tasks.
-
post_install_step
()¶ Do some postprocessing - run post install commands if any were specified
-
post_iter_step
()¶ Restore options that were iterated over
-
prepare_for_extensions
()¶ Also do this before (eg to set the template)
-
prepare_step
(start_dir=True, load_tc_deps_modules=True)¶ Pre-configure step. Set’s up the builddir just before starting configure
Parameters: - start_dir – guess start directory based on unpacked sources
- load_tc_deps_modules – load modules for toolchain and dependencies in build environment
-
remove_module_file
()¶ Remove module file (if it exists), and check for ghost installation directory (and deal with it).
-
report_test_failure
(msg_or_error)¶ Report a failing test either via an exception or warning depending on ignore-test-failure
Parameters: msg_or_error – failure description (string value or an EasyBuildError instance)
List of shared libraries which must be linked in all installed binaries/libraries. Supported values are pure library names without ‘lib’ prefix or extension (‘example’), file names (‘libexample.so’), and full paths (‘/usr/lib64/libexample.so’).
-
reset_env
()¶ Reset environment. When iterating over builddependencies, every time we start a new iteration we need to restore the environment to where it was before the relevant modules were loaded.
-
run_all_steps
(run_test_cases)¶ Build and install this software. run_test_cases (bool): run tests after building (e.g.: make test)
-
run_post_install_commands
(commands=None)¶ Run post install commands that are specified via ‘postinstallcmds’ easyconfig parameter.
-
run_step
(step, step_methods)¶ Run step, returns false when execution should be stopped
Check whether specific shared libraries are (not) linked into installed binaries/libraries.
-
sanity_check_rpath
(rpath_dirs=None)¶ Sanity check binaries/libraries w.r.t. RPATH linking.
-
sanity_check_step
(*args, **kwargs)¶ Do a sanity check on the installation - if any of the files/subdirectories in the installation directory listed
in sanity_check_paths are non-existent (or empty), the sanity check fails
-
set_parallel
()¶ Set ‘parallel’ easyconfig parameter to determine how many cores can/should be used for parallel builds.
-
set_up_cuda_cache
()¶ Set up CUDA PTX cache.
-
short_mod_name
¶ Short module name (not including subdirectory in module install path)
-
skip_extensions
()¶ Skip already installed extensions, by removing them from list of Extension instances to install (self.ext_instances).
This is done in parallel when EasyBuild is configured to install extensions in parallel.
-
skip_extensions_parallel
(exts_filter)¶ Skip already installed extensions (checking in parallel), by removing them from list of Extension instances to install (self.ext_instances).
-
skip_extensions_sequential
(exts_filter)¶ Skip already installed extensions (checking sequentially), by removing them from list of Extension instances to install (self.ext_instances).
-
skip_step
(step, skippable)¶ Dedice whether or not to skip the specified step.
-
stage_install_step
()¶ Install in a stage directory before actual installation.
-
start_dir
¶ Start directory in build directory
-
test_cases_step
()¶ Run provided test cases.
-
test_step
()¶ Run unit tests provided by software (if any).
-
toolchain
¶ Toolchain used to build this easyblock
-
update_config_template_run_step
()¶ Update the the easyconfig template dictionary with easyconfig.TEMPLATE_NAMES_EASYBLOCK_RUN_STEP names
-
update_exts_progress_bar
(info, progress_size=0, total=None)¶ Update extensions progress bar with specified info and amount of progress made
-
version
¶ Shortcut the get the module version.
-
-
exception
easybuild.framework.easyblock.
StopException
¶ Bases:
Exception
StopException class definition.
-
easybuild.framework.easyblock.
build_and_install_one
(ecdict, init_env)¶ Build the software :param ecdict: dictionary contaning parsed easyconfig + metadata :param init_env: original environment (used to reset environment)
-
easybuild.framework.easyblock.
build_easyconfigs
(easyconfigs, output_dir, test_results)¶ Build the list of easyconfigs.
-
easybuild.framework.easyblock.
copy_easyblocks_for_reprod
(easyblock_instances, reprod_dir)¶
-
easybuild.framework.easyblock.
get_easyblock_instance
(ecdict)¶ Get an instance for this easyconfig :param easyconfig: parsed easyconfig (EasyConfig instance)
returns an instance of EasyBlock (or subclass thereof)
-
easybuild.framework.easyblock.
inject_checksums
(ecs, checksum_type)¶ Inject checksums of given type in specified easyconfig files
Parameters: - ecs – list of EasyConfig instances to inject checksums into corresponding files
- checksum_type – type of checksum to use
-
easybuild.framework.easyblock.
print_dry_run_note
(loc, silent=True)¶ Print note on interpreting dry run output.
-
easybuild.framework.easyblock.
reproduce_build
(app, reprod_dir_root)¶ Create reproducibility files (processed easyconfig and easyblocks used) from class instance
Parameters: - app – easyblock class instance
- reprod_dir_root – root directory in which to create the ‘reprod’ directory
Return reprod_dir: directory containing reproducibility files