easybuild.tools.modules module¶
- This python module implements the environment modules functionality:
- loading modules
- checking for available modules
- …
author: | Stijn De Weirdt (Ghent University) |
---|---|
author: | Dries Verdegem (Ghent University) |
author: | Kenneth Hoste (Ghent University) |
author: | Pieter De Baets (Ghent University) |
author: | Jens Timmerman (Ghent University) |
author: | David Brown (Pacific Northwest National Laboratory) |
-
class
easybuild.tools.modules.
EnvironmentModules
(mod_paths=None, testing=False)¶ Bases:
easybuild.tools.modules.EnvironmentModulesTcl
Interface to environment modules 4.0+
-
COMMAND
= 'MODULESHOME_NOT_DEFINED/libexec/modulecmd.tcl'¶
-
COMMAND_ENVIRONMENT
= 'MODULES_CMD'¶
-
MAX_VERSION
= None¶
-
NAME
= 'Environment Modules'¶
-
REQ_VERSION
= '4.0.0'¶
-
VERSION_REGEXP
= '^Modules\\s+Release\\s+(?P<version>\\d\\S*)\\s'¶
-
check_module_output
(cmd, stdout, stderr)¶ Check output of ‘module’ command, see if if is potentially invalid.
-
-
class
easybuild.tools.modules.
EnvironmentModulesC
(mod_paths=None, testing=False)¶ Bases:
easybuild.tools.modules.ModulesTool
Interface to (C) environment modules (modulecmd).
-
COMMAND
= 'modulecmd'¶
-
MAX_VERSION
= '3.99'¶
-
NAME
= 'Environment Modules'¶
-
REQ_VERSION
= '3.2.10'¶
-
VERSION_REGEXP
= '^\\s*(VERSION\\s*=\\s*)?(?P<version>\\d\\S*)\\s*'¶
-
get_setenv_value_from_modulefile
(mod_name, var_name)¶ Get value for specific ‘setenv’ statement from module file for the specified module.
Parameters: - mod_name – module name
- var_name – name of the variable being set for which value should be returned
-
run_module
(*args, **kwargs)¶ Run module command, tweak output that is exec’ed if necessary.
-
update
()¶ Update after new modules were added.
-
-
class
easybuild.tools.modules.
EnvironmentModulesTcl
(mod_paths=None, testing=False)¶ Bases:
easybuild.tools.modules.EnvironmentModulesC
Interface to (Tcl) environment modules (modulecmd.tcl).
-
COMMAND
= 'modulecmd.tcl'¶
-
COMMAND_SHELL
= ['tclsh']¶
-
NAME
= 'ancient Tcl-only Environment Modules'¶
-
REQ_VERSION
= None¶
-
TERSE_OPTION
= (1, '-t')¶
-
VERSION_OPTION
= ''¶
-
VERSION_REGEXP
= '^Modules\\s+Release\\s+Tcl\\s+(?P<version>\\d\\S*)\\s'¶
-
available
(mod_name=None)¶ Return a list of available modules for the given (partial) module name; use None to obtain a list of all available modules.
Parameters: name – a (partial) module name for filtering (default: None)
-
remove_module_path
(path, set_mod_paths=True)¶ Remove specified module path (using ‘module unuse’).
Parameters: - path – path to remove from $MODULEPATH via ‘unuse’
- set_mod_paths – (re)set self.mod_paths
-
run_module
(*args, **kwargs)¶ Run module command, tweak output that is exec’ed if necessary.
-
set_path_env_var
(key, paths)¶ Set environment variable with given name to the given list of paths.
-
-
class
easybuild.tools.modules.
Lmod
(*args, **kwargs)¶ Bases:
easybuild.tools.modules.ModulesTool
Interface to Lmod.
-
COMMAND
= 'lmod'¶
-
COMMAND_ENVIRONMENT
= 'LMOD_CMD'¶
-
DEPR_VERSION
= '7.0.0'¶
-
NAME
= 'Lmod'¶
-
REQ_VERSION
= '6.5.1'¶
-
REQ_VERSION_DEPENDS_ON
= '7.6.1'¶
-
SHOW_HIDDEN_OPTION
= '--show-hidden'¶
-
USER_CACHE_DIR
= '/home/docs/.lmod.d/.cache'¶
-
VERSION_REGEXP
= '^Modules\\s+based\\s+on\\s+Lua:\\s+Version\\s+(?P<version>\\d\\S*)\\s'¶
-
available
(mod_name=None)¶ Return a list of available modules for the given (partial) module name; use None to obtain a list of all available modules.
Parameters: name – a (partial) module name for filtering (default: None)
-
check_module_function
(*args, **kwargs)¶ Check whether selected module tool matches ‘module’ function definition.
-
check_module_output
(cmd, stdout, stderr)¶ Check output of ‘module’ command, see if if is potentially invalid.
-
compose_cmd_list
(args, opts=None)¶ Compose full module command to run, based on provided arguments
Parameters: args – list of arguments for module command Returns: list of strings representing the full module command to run
-
get_setenv_value_from_modulefile
(mod_name, var_name)¶ Get value for specific ‘setenv’ statement from module file for the specified module.
Parameters: - mod_name – module name
- var_name – name of the variable being set for which value should be returned
-
module_wrapper_exists
(mod_name)¶ Determine whether a module wrapper with specified name exists. First check for wrapper defined in .modulerc.lua, fall back to also checking .modulerc (Tcl syntax).
-
prepend_module_path
(path, set_mod_paths=True, priority=None)¶ Prepend given module path to list of module paths, or bump it to 1st place.
Parameters: - path – path to prepend to $MODULEPATH
- set_mod_paths – (re)set self.mod_paths
- priority – priority for this path in $MODULEPATH (Lmod-specific)
-
unuse
(path)¶ Remove a module path
-
update
()¶ Update after new modules were added.
-
use
(path, priority=None)¶ Add path to $MODULEPATH via ‘module use’.
Parameters: - path – path to add to $MODULEPATH
- priority – priority for this path in $MODULEPATH (Lmod-specific)
-
-
class
easybuild.tools.modules.
Modules
(*args, **kwargs)¶ Bases:
easybuild.tools.modules.EnvironmentModulesC
NO LONGER SUPPORTED: interface to modules tool, use modules_tool from easybuild.tools.modules instead
-
class
easybuild.tools.modules.
ModulesTool
(mod_paths=None, testing=False)¶ Bases:
object
An abstract interface to a tool that deals with modules.
-
COMMAND
= None¶
-
COMMAND_ENVIRONMENT
= None¶
-
COMMAND_SHELL
= None¶
-
DEPR_VERSION
= None¶
-
MAX_VERSION
= None¶
-
NAME
= None¶
-
REQ_VERSION
= None¶
-
TERSE_OPTION
= (0, '--terse')¶
-
USER_CACHE_DIR
= None¶
-
VERSION_OPTION
= '--version'¶
-
VERSION_REGEXP
= None¶
-
add_module_path
(path, set_mod_paths=True)¶ Add specified module path (using ‘module use’) if it’s not there yet.
Parameters: - path – path to add to $MODULEPATH via ‘use’
- set_mod_paths – (re)set self.mod_paths
-
available
(mod_name=None, extra_args=None)¶ Return a list of available modules for the given (partial) module name; use None to obtain a list of all available modules.
Parameters: mod_name – a (partial) module name for filtering (default: None)
-
buildstats
()¶ Return tuple with data to be included in buildstats
-
check_cmd_avail
()¶ Check whether modules tool command is available.
-
check_loaded_modules
()¶ Check whether any (EasyBuild-generated) modules are loaded already in the current session
-
check_module_function
(allow_mismatch=False, regex=None)¶ Check whether selected module tool matches ‘module’ function definition.
-
check_module_output
(cmd, stdout, stderr)¶ Check output of ‘module’ command, see if if is potentially invalid.
-
check_module_path
()¶ Check if MODULEPATH is set and change it if necessary.
-
compose_cmd_list
(args, opts=None)¶ Compose full module command to run, based on provided arguments
Parameters: args – list of arguments for module command Returns: list of strings representing the full module command to run
-
exist
(mod_names, mod_exists_regex_template=None, skip_avail=False, maybe_partial=True)¶ Check if modules with specified names exists.
Parameters: - mod_names – list of module names
- mod_exists_regex_template – DEPRECATED and unused
- skip_avail – skip checking through ‘module avail’, only check via ‘module show’
- maybe_partial – indicates if the module name may be a partial module name
-
get_setenv_value_from_modulefile
(mod_name, var_name)¶ Get value for specific ‘setenv’ statement from module file for the specified module.
Parameters: - mod_name – module name
- var_name – name of the variable being set for which value should be returned
-
get_value_from_modulefile
(mod_name, regex, strict=True)¶ Get info from the module file for the specified module.
Parameters: - mod_name – module name
- regex – (compiled) regular expression, with one group
-
interpret_raw_path_lua
(txt)¶ Interpret raw path (Lua syntax): resolve environment variables, join paths where pathJoin is specified
-
interpret_raw_path_tcl
(txt)¶ Interpret raw path (TCL syntax): resolve environment variables
-
list
()¶ Return result of ‘module list’.
-
load
(modules, mod_paths=None, purge=False, init_env=None, allow_reload=True)¶ Load all requested modules.
Parameters: - modules – list of modules to load
- mod_paths – list of module paths to activate before loading
- purge – whether or not a ‘module purge’ should be run before loading
- init_env – original environment to restore after running ‘module purge’
- allow_reload – allow reloading an already loaded module
-
loaded_modules
()¶ Return a list of loaded modules.
-
mk_module_cache_key
(partial_key)¶ Create a module cache key, using the specified partial key, by combining it with the current $MODULEPATH.
-
modpath_extensions_for
(mod_names)¶ Determine dictionary with $MODULEPATH extensions for specified modules. All potential $MODULEPATH extensions are included, even the ones guarded by a condition (which is not checked). Only direct $MODULEPATH extensions are found, no recursion if performed for modules that load other modules. Modules with an empty list of $MODULEPATH extensions are included in the result.
Parameters: mod_names – list of module names for which to determine the list of $MODULEPATH extensions Returns: dictionary with module names as keys and lists of $MODULEPATH extensions as values
-
module_wrapper_exists
(mod_name, modulerc_fn='.modulerc', mod_wrapper_regex_template=None)¶ Determine whether a module wrapper with specified name exists. Only .modulerc file in Tcl syntax is considered here.
-
modulefile_path
(mod_name, strip_ext=False)¶ Get the path of the module file for the specified module
Parameters: - mod_name – module name
- strip_ext – strip (.lua) extension from module fileame (if present)
-
path_to_top_of_module_tree
(top_paths, mod_name, full_mod_subdir, deps, modpath_exts=None)¶ Recursively determine path to the top of the module tree, for given module, module subdir and list of $MODULEPATH extensions per dependency module.
For example, when to determine the path to the top of the module tree for the HPL/2.1 module being installed with a goolf/1.5.14 toolchain in a Core/Compiler/MPI hierarchy (HierarchicalMNS):
- starting point:
- top_paths = [‘<prefix>’, ‘<prefix>/Core’] mod_name = ‘HPL/2.1’ full_mod_subdir = ‘<prefix>/MPI/Compiler/GCC/4.8.2/OpenMPI/1.6.5’ deps = [‘GCC/4.8.2’, ‘OpenMPI/1.6.5’, ‘OpenBLAS/0.2.8-LAPACK-3.5.0’, ‘FFTW/3.3.4’, ‘ScaLAPACK/…’]
- 1st iteration: find module that extends $MODULEPATH with ‘<prefix>/MPI/Compiler/GCC/4.8.2/OpenMPI/1.6.5’,
- => OpenMPI/1.6.5 (in ‘<prefix>/Compiler/GCC/4.8.2’ subdir); recurse with mod_name = ‘OpenMPI/1.6.5’ and full_mod_subdir = ‘<prefix>/Compiler/GCC/4.8.2’
- 2nd iteration: find module that extends $MODULEPATH with ‘<prefix>/Compiler/GCC/4.8.2’
- => GCC/4.8.2 (in ‘<prefix>/Core’ subdir); recurse with mod_name = ‘GCC/4.8.2’ and full_mod_subdir = ‘<prefix>/Core’
- 3rd iteration: try to find module that extends $MODULEPATH with ‘<prefix>/Core’
- => ‘<prefix>/Core’ is in top_paths, so stop recursion
Parameters: - top_paths – list of potentation ‘top of module tree’ (absolute) paths
- mod_name – (short) module name for starting point (only used in log messages)
- full_mod_subdir – absolute path to module subdirectory for starting point
- deps – list of dependency modules for module at starting point
- modpath_exts – list of module path extensions for each of the dependency modules
-
prepend_module_path
(path, set_mod_paths=True, priority=None)¶ Prepend given module path to list of module paths, or bump it to 1st place.
Parameters: - path – path to prepend to $MODULEPATH
- set_mod_paths – (re)set self.mod_paths
- priority – priority for this path in $MODULEPATH (Lmod-specific)
-
purge
()¶ Purge loaded modules.
-
read_module_file
(mod_name)¶ Read module file with specified name.
-
remove_module_path
(path, set_mod_paths=True)¶ Remove specified module path (using ‘module unuse’).
Parameters: - path – path to remove from $MODULEPATH via ‘unuse’
- set_mod_paths – (re)set self.mod_paths
-
run_module
(*args, **kwargs)¶ Run module command.
Parameters: - args – list of arguments for module command; first argument should be the subcommand to run
- kwargs – dictionary with options that control certain aspects of how to run the module command
-
set_and_check_version
()¶ Get the module version, and check any requirements
-
set_mod_paths
(mod_paths=None)¶ Set mod_paths, based on $MODULEPATH unless a list of module paths is specified.
Parameters: mod_paths – list of entries for $MODULEPATH to use
-
set_path_env_var
(key, paths)¶ Set path environment variable to the given list of paths.
-
show
(mod_name)¶ Run ‘module show’ for the specified module.
-
unload
(modules=None)¶ Unload all requested modules.
-
unuse
(path)¶ Remove module path via ‘module unuse’.
-
update
()¶ Update after new modules were added.
-
use
(path, priority=None)¶ Add path to $MODULEPATH via ‘module use’.
Parameters: - path – path to add to $MODULEPATH
- priority – priority for this path in $MODULEPATH (Lmod-specific)
-
-
class
easybuild.tools.modules.
NoModulesTool
(*args, **kwargs)¶ Bases:
easybuild.tools.modules.ModulesTool
Class that mock the module behaviour, used for operation not requiring modules. Eg. tests, fetch only
-
available
(*args, **kwargs)¶ No modules, so nothing available
-
check_loaded_modules
()¶ Nothing to do since no modules
-
exist
(mod_names, *args, **kwargs)¶ No modules, so nothing exists
-
list
()¶ No modules loaded
-
-
easybuild.tools.modules.
avail_modules_tools
()¶ Return all known modules tools.
-
easybuild.tools.modules.
curr_module_paths
(normalize=False)¶ Return a list of current module paths.
Parameters: normalize – Normalize the paths
-
easybuild.tools.modules.
get_software_libdir
(name, only_one=True, fs=None)¶ Find library subdirectories for the specified software package.
Returns the library subdirectory, relative to software root. It fails if multiple library subdirs are found, unless only_one is False which yields a list of all library subdirs.
Parameters: - name – name of the software package
- only_one – indicates whether only one lib path is expected to be found
- fs – only retain library subdirs that contain one of the files in this list
-
easybuild.tools.modules.
get_software_root
(name, with_env_var=False)¶ Return the software root set for a particular software name.
-
easybuild.tools.modules.
get_software_root_env_var_name
(name)¶ Return name of environment variable for software root.
-
easybuild.tools.modules.
get_software_version
(name)¶ Return the software version set for a particular software name.
-
easybuild.tools.modules.
get_software_version_env_var_name
(name)¶ Return name of environment variable for software root.
-
easybuild.tools.modules.
invalidate_module_caches_for
(path)¶ Invalidate cache entries related to specified path.
-
easybuild.tools.modules.
mk_module_path
(paths)¶ Create a string representing the list of module paths.
-
easybuild.tools.modules.
modules_tool
(mod_paths=None, testing=False)¶ Return interface to modules tool (environment modules (C, Tcl), or Lmod)
-
easybuild.tools.modules.
reset_module_caches
()¶ Reset module caches.