easyconfig

Easyconfig module that contains the EasyConfig class.

Authors:

• Stijn De Weirdt (Ghent University)
• Dries Verdegem (Ghent University)
• Kenneth Hoste (Ghent University)
• Pieter De Baets (Ghent University)
• Jens Timmerman (Ghent University)
• Toon Willems (Ghent University)
• Ward Poelmans (Ghent University)
• Alan O'Cais (Juelich Supercomputing Centre)
• Bart Oldeman (McGill University, Calcul Quebec, Compute Canada)
• Maxime Boissonneault (Universite Laval, Calcul Quebec, Compute Canada)
• Victor Holanda (CSCS, ETH Zurich)

ActiveMNS

Bases: BaseActiveMNS

Wrapper class for active module naming scheme.

__init__(*args, **kwargs)

Initialize logger.

check_ec_type(ec, raise_error=True)

Obtain a full parsed easyconfig file to pass to naming scheme methods if provided keys are insufficient.

PARAMETER	DESCRIPTION
ec	available easyconfig parameter specifications (EasyConfig instance or dict value)
raise_error	boolean indicating whether or not an error should be raised if a full easyconfig is required but not found DEFAULT: True

det_devel_module_filename(ec, force_visible=False)

Determine devel module filename.

det_full_module_name(ec, force_visible=False, require_result=True)

Determine full module name by selected module naming scheme, based on supplied easyconfig.

det_init_modulepaths(ec)

Determine initial modulepaths according to module naming scheme.

det_install_subdir(ec)

Determine name of software installation subdirectory.

det_modpath_extensions(ec)

Determine modulepath extensions according to module naming scheme.

det_module_subdir(ec)

Determine module subdirectory according to module naming scheme.

det_module_symlink_paths(ec)

Determine list of paths in which symlinks to module files must be created.

det_short_module_name(ec, force_visible=False)

Determine short module name according to module naming scheme.

det_user_modpath_extensions(ec)

Determine user-specific modulepath extensions according to module naming scheme.

expand_toolchain_load(ec=None)

Determine whether load statements for a toolchain should be expanded to load statements for its dependencies. This is useful when toolchains are not exposed to users.

is_short_modname_for(short_modname, name)

Determine whether the specified (short) module name is a module for software with the specified name.

requires_full_easyconfig(keys)

Check whether specified list of easyconfig parameters is sufficient for active module naming scheme.

EasyConfig

Bases: object

Class which handles loading, reading, validation of easyconfigs

all_dependencies property

Return list of all dependencies, incl. hidden/build deps & toolchain, but excluding filtered deps.

name property

returns name

toolchain property

returns the Toolchain used

version property

returns version

__contains__(key)

Check whether easyconfig parameter is defined

__eq__(ec)

Is this EasyConfig instance equivalent to the provided one?

__getitem__(key)

Return value of specified easyconfig parameter (without help text, etc.)

__hash__()

Return hash value for a hashable representation of this EasyConfig instance.

__init__(path, extra_options=None, build_specs=None, validate=True, hidden=None, rawtxt=None, auto_convert_value_types=True, local_var_naming_check=None)

initialize an easyconfig.

PARAMETER	DESCRIPTION
path	path to easyconfig file to be parsed (ignored if rawtxt is specified)
extra_options	dictionary with extra variables that can be set for this specific instance DEFAULT: None
build_specs	dictionary of build specifications (see EasyConfig class, default: {}) DEFAULT: None
validate	indicates whether validation should be performed (note: combined with 'validate' build option) DEFAULT: True
hidden	indicate whether corresponding module file should be installed hidden ('.'-prefixed) DEFAULT: None
rawtxt	raw contents of easyconfig file DEFAULT: None
auto_convert_value_types	indicates wether types of easyconfig values should be automatically converted in case they are wrong DEFAULT: True
local_var_naming_check	mode to use when checking if local variables use the recommended naming scheme DEFAULT: None

__ne__(ec)

Is this EasyConfig instance equivalent to the provided one?

__setitem__(key, value)

Set value of specified easyconfig parameter (help text & co is left untouched)

__str__()

Return a string representation of this EasyConfig instance

asdict()

Return dict representation of this EasyConfig instance.

builddependencies()

Return a flat list of the parsed build dependencies When builddependencies are iterable they are flattened lists with duplicates removed outside of the iterating process, because the callers want simple lists.

check_deprecated(path)

Check whether this easyconfig file is deprecated.

copy(validate=None)

Return a copy of this EasyConfig instance.

count_files()

Determine number of files (sources + patches) required for this easyconfig.

dep_is_filtered(dep, filter_deps_specs)

Returns True if a dependency is filtered according to the filter_deps_specs

dependencies(build_only=False)

Returns an array of parsed dependencies (after filtering, if requested) dependency = {'name': '', 'version': '', 'system': (False|True), 'versionsuffix': '', 'toolchain': ''} Iterable builddependencies are flattened when not iterating.

PARAMETER	DESCRIPTION
build_only	only return build dependencies, discard others DEFAULT: False

dependency_names(build_only=False)

Return a set of names of all (direct) dependencies after filtering. Iterable builddependencies are flattened when not iterating.

PARAMETER	DESCRIPTION
build_only	only return build dependencies, discard others DEFAULT: False

disable_templating()

Temporarily disable templating on the given EasyConfig

Usage: with ec.disable_templating(): # Do what you want without templating # Templating set to previous value

dump(fp, always_overwrite=True, backup=False, explicit_toolchains=False)

Dump this easyconfig to file, with the given filename.

PARAMETER	DESCRIPTION
always_overwrite	overwrite existing file at specified location without use of --force DEFAULT: True
backup	create backup of existing file before overwriting it DEFAULT: False

extend_params(extra, overwrite=True)

Extend list of known parameters via provided list of extra easyconfig parameters.

filename()

Determine correct filename for this easyconfig file.

filter_deps(deps)

Filter dependencies according to 'filter-deps' configuration setting.

filter_hidden_deps()

Replace dependencies by hidden dependencies in list of (build) dependencies, where appropriate.

generate_template_values()

Try to generate all template values.

get(key, default=None, resolve=True)

Gets the value of a key in the config, with 'default' as fallback.

PARAMETER	DESCRIPTION
resolve	if False, disables templating via calling get_ref, else resolves template values DEFAULT: True

get_cuda_cc_template_value(key)

Get template value based on --cuda-compute-capabilities EasyBuild configuration option and cuda_compute_capabilities easyconfig parameter. Returns user-friendly error message in case neither are defined, or if an unknown key is used.

get_parsed_multi_deps()

Get list of lists of parsed dependencies that correspond with entries in multi_deps easyconfig parameter.

get_ref(key)

Obtain reference to original/untemplated value of specified easyconfig parameter rather than a copied value with templated values.

handle_external_module_metadata(mod_name)

Helper function for _parse_dependency; collects metadata for external module dependencies.

PARAMETER	DESCRIPTION
mod_name	name of external module to collect metadata for

handle_multi_deps()

Handle lists of dependency versions of which we should iterate specified in 'multi_deps' easyconfig parameter.

This is basically just syntactic sugar to prevent having to specify a list of lists in 'builddependencies'.

is_mandatory_param(key)

Check whether specified easyconfig parameter is mandatory.

local_var_naming(local_var_naming_check)

Deal with local variables that do not follow the recommended naming scheme (if any).

parse()

Parse the file and set options mandatory requirements are checked here

parse_filter_deps()

Parse specifications for which dependencies should be filtered.

parse_version_range(version_spec)

Parse provided version specification as a version range.

probe_external_module_metadata(mod_name, existing_metadata=None)

Helper function for handle_external_module_metadata.

Tries to determine metadata for external module when there is not entry in the metadata file, by looking at the variables defined by the module file.

This is mainly intended for modules provided in the Cray Programming Environment, but it could also be useful in other contexts.

The following pairs of variables are considered (in order, first hit wins), where 'XXX' is the software name in capitals: 1. $CRAY_XXX_PREFIX and $CRAY_XXX_VERSION 1. $CRAY_XXX_PREFIX_DIR and $CRAY_XXX_VERSION 2. $CRAY_XXX_DIR and $CRAY_XXX_VERSION 2. $CRAY_XXX_ROOT and $CRAY_XXX_VERSION 5. $XXX_PREFIX and $XXX_VERSION 4. $XXX_DIR and $XXX_VERSION 5. $XXX_ROOT and $XXX_VERSION 3. $XXX_HOME and $XXX_VERSION

If none of the pairs is found, then an empty dictionary is returned.

PARAMETER	DESCRIPTION
mod_name	name of the external module
existing_metadata	already available metadata for this external module (if any) DEFAULT: None

set_keys(params)

Set keys in this EasyConfig instance based on supplied easyconfig parameter values.

If any unknown easyconfig parameters are encountered here, an error is raised.

PARAMETER	DESCRIPTION
params	a dict value with names/values of easyconfig parameters to set

start_iterating()

Start iterative mode.

stop_iterating()

Stop iterative mode.

update(key, value, allow_duplicate=True)

Update an easyconfig parameter with the specified value (i.e. append to it). Note: For dictionary easyconfig parameters, 'allow_duplicate' is ignored (since it's meaningless).

validate(check_osdeps=True)

Validate this easyonfig - ensure certain easyconfig parameters are set to a known value (see self.validations) - check OS dependencies - check license

validate_iterate_opts_lists()

Configure/build/install options specified as lists should have same length.

validate_license()

Validate the license

validate_os_deps()

validate presence of OS dependencies osdependencies should be a single list

clean_up_easyconfigs(paths)

Clean up easyconfigs (in place) by filtering out comments/buildstats included by EasyBuild in archived easyconfigs (cfr. FileRepository.add_easyconfig in easybuild.tools.repository.filerepo)

PARAMETER	DESCRIPTION
paths	list of paths to easyconfigs to clean up

copy_easyconfigs(paths, target_dir)

Copy easyconfig files to specified directory, in the 'right' location and using the filename expected by robot.

PARAMETER	DESCRIPTION
paths	list of paths to copy to git working dir
target_dir	target directory

RETURNS	DESCRIPTION
	dict with useful information on copied easyconfig files (corresponding EasyConfig instances, paths, status)

copy_patch_files(patch_specs, target_dir)

Copy patch files to specified directory, in the 'right' location according to the software name they relate to.

PARAMETER	DESCRIPTION
patch_specs	list of tuples with patch file location and name of software they are for
target_dir	target directory

create_paths(path, name, version)

Returns all the paths where easyconfig could be located is the basepath should be a string can be a '*' if you use glob patterns, or an install version otherwise

det_file_info(paths, target_dir)

Determine useful information on easyconfig files relative to a target directory, before any actual operation (e.g. copying) is performed

PARAMETER	DESCRIPTION
paths	list of paths to easyconfig files
target_dir	target directory

RETURNS	DESCRIPTION
	dict with useful information on easyconfig files (corresponding EasyConfig instances, paths, status) relative to a target directory

det_installversion(version, toolchain_name, toolchain_version, prefix, suffix)

Deprecated 'det_installversion' function, to determine exact install version, based on supplied parameters.

det_location_for(path, target_dir, soft_name, target_file)

Determine path to easyconfigs directory for specified software name, using specified target file name.

PARAMETER	DESCRIPTION
path	path of file to copy
target_dir	(parent) target directory, should contain easybuild/easyconfigs subdirectory
soft_name	software name (to determine location to copy to)
target_file	target file name

RETURNS	DESCRIPTION
	full path to the right location

det_subtoolchain_version(current_tc, subtoolchain_names, optional_toolchains, cands, incl_capabilities=False)

Returns unique version for subtoolchain, in tc dict. If there is no unique version: * use '' for system, if system is not skipped. * return None for skipped subtoolchains, that is, optional toolchains or system toolchain without add_system_to_minimal_toolchains. * in all other cases, raises an exception.

disable_templating(ec)

Temporarily disable templating on the given EasyConfig

Usage: with disable_templating(ec): # Do what you want without templating # Templating set to previous value

fix_deprecated_easyconfigs(paths)

Fix use of deprecated functionality in easyconfigs at specified locations.

get_easyblock_class(easyblock, name=None, error_on_failed_import=True, error_on_missing_easyblock=None, **kwargs)

Get class for a particular easyblock (or use default)

get_module_path(name, generic=None, decode=True)

Determine the module path for a given easyblock or software name, based on the encoded class name.

PARAMETER	DESCRIPTION
generic	whether or not the easyblock is generic (if None: auto-derive from specified class name) DEFAULT: None
decode	whether or not to decode the provided class name DEFAULT: True

get_toolchain_hierarchy(parent_toolchain, incl_capabilities=False)

Determine list of subtoolchains for specified parent toolchain. Result starts with the most minimal subtoolchains first, ends with specified toolchain.

The system toolchain is considered the most minimal subtoolchain only if the add_system_to_minimal_toolchains build option is enabled.

The most complex hierarchy we have now is goolfc which works as follows:

goolfc
/     \

gompic golfc() \ / \ () optional toolchains, not compulsory for backwards compatibility gcccuda golf() \ / GCC / | GCCcore() | \ | (system: only considered if --add-system-to-minimal-toolchains configuration option is enabled)

PARAMETER	DESCRIPTION
parent_toolchain	dictionary with name/version of parent toolchain
incl_capabilities	also register toolchain capabilities in result DEFAULT: False

handle_deprecated_or_replaced_easyconfig_parameters(ec_method)

Decorator to handle deprecated/replaced easyconfig parameters.

is_generic_easyblock(easyblock)

Return whether specified easyblock name is a generic easyblock or not.

is_local_var_name(name)

Determine whether provided variable name can be considered as the name of a local variable:

One of the following suffices to be considered a name of a local variable: * name starts with 'local_' or '_' * name consists of a single letter * name is builtins (which is always defined)

letter_dir_for(name)

Determine 'letter' directory for specified software name. This usually just the 1st letter of the software name (in lowercase), except for funky software names, e.g. ones starting with a digit.

process_easyconfig(path, build_specs=None, validate=True, parse_only=False, hidden=None)

Process easyconfig, returning some information for each block

PARAMETER	DESCRIPTION
path	path to easyconfig file
build_specs	dictionary specifying build specifications (e.g. version, toolchain, ...) DEFAULT: None
validate	whether or not to perform validation DEFAULT: True
parse_only	only parse easyconfig superficially (faster, but results in partial info) DEFAULT: False
hidden	indicate whether corresponding module file should be installed hidden ('.'-prefixed) DEFAULT: None

resolve_template(value, tmpl_dict)

Given a value, try to susbstitute the templated strings with actual values. - value: some python object (supported are string, tuple/list, dict or some mix thereof) - tmpl_dict: template dictionary

robot_find_easyconfig(name, version)

Find an easyconfig for module in path, returns (absolute) path to easyconfig file (or None, if none is found).

robot_find_subtoolchain_for_dep(dep, modtool, parent_tc=None, parent_first=False)

Find the subtoolchain to use for a dependency

PARAMETER	DESCRIPTION
dep	dependency target dict (long and short module names may not exist yet)
parent_tc	toolchain from which to derive the toolchain hierarchy to search (default: use dep's toolchain) DEFAULT: None
parent_first	reverse order in which subtoolchains are considered: parent toolchain, then subtoolchains DEFAULT: False

RETURNS	DESCRIPTION
	minimal toolchain for which an easyconfig exists for this dependency (and matches build_options)

toolchain_hierarchy_cache(func)

Function decorator to cache (and retrieve cached) toolchain hierarchy queries.

triage_easyconfig_params(variables, ec)

Triage supplied variables into known easyconfig parameters and other variables.

Unknown easyconfig parameters that have a single-letter name, or of which the name starts with 'local_' are considered to be local variables.

PARAMETER	DESCRIPTION
variables	dictionary with names/values of variables that should be triaged
ec	dictionary with set of known easyconfig parameters

RETURNS	DESCRIPTION
	2-tuple with dict of names/values for known easyconfig parameters + unknown (non-local) variables

verify_easyconfig_filename(path, specs, parsed_ec=None)

Check whether parsed easyconfig at specified path matches expected specs; this basically verifies whether the easyconfig filename corresponds to its contents

PARAMETER	DESCRIPTION
path	path to easyconfig file
specs	expected specs (dict with easyconfig parameter values)
parsed_ec	(list of) EasyConfig instance(s) corresponding to easyconfig file DEFAULT: None