easybuild.tools.modules module

This python module implements the environment modules functionality:

• loading modules
• checking for available modules
• …

author:Stijn De Weirdt (Ghent University) author:Dries Verdegem (Ghent University) author:Kenneth Hoste (Ghent University) author:Pieter De Baets (Ghent University) author:Jens Timmerman (Ghent University) author:David Brown (Pacific Northwest National Laboratory)

class easybuild.tools.modules.EnvironmentModules(mod_paths=None, testing=False)

Bases: easybuild.tools.modules.EnvironmentModulesTcl

Interface to environment modules 4.0+

COMMAND = 'MODULESHOME_NOT_DEFINED/libexec/modulecmd.tcl'

COMMAND_ENVIRONMENT = 'MODULES_CMD'

MAX_VERSION = None

NAME = 'Environment Modules v4'

REQ_VERSION = '4.0.0'

VERSION_REGEXP = '^Modules\\s+Release\\s+(?P<version>\\d\\S*)\\s'

check_module_output(cmd, stdout, stderr)

Check output of ‘module’ command, see if if is potentially invalid.

class easybuild.tools.modules.EnvironmentModulesC(mod_paths=None, testing=False)

Bases: easybuild.tools.modules.ModulesTool

Interface to (C) environment modules (modulecmd).

COMMAND = 'modulecmd'

MAX_VERSION = '3.99'

NAME = 'Environment Modules v3'

REQ_VERSION = '3.2.10'

VERSION_REGEXP = '^\\s*(VERSION\\s*=\\s*)?(?P<version>\\d\\S*)\\s*'

get_setenv_value_from_modulefile(mod_name, var_name)

Get value for specific ‘setenv’ statement from module file for the specified module.

Parameters:

• mod_name – module name
• var_name – name of the variable being set for which value should be returned

run_module(*args, **kwargs)

Run module command, tweak output that is exec’ed if necessary.

update()

Update after new modules were added.

class easybuild.tools.modules.EnvironmentModulesTcl(mod_paths=None, testing=False)

Bases: easybuild.tools.modules.EnvironmentModulesC

Interface to (Tcl) environment modules (modulecmd.tcl).

COMMAND = 'modulecmd.tcl'

COMMAND_SHELL = ['tclsh']

NAME = 'ancient Tcl-only Environment Modules'

REQ_VERSION = None

TERSE_OPTION = (1, '-t')

VERSION_OPTION = ''

VERSION_REGEXP = '^Modules\\s+Release\\s+Tcl\\s+(?P<version>\\d\\S*)\\s'

available(mod_name=None)

Return a list of available modules for the given (partial) module name; use None to obtain a list of all available modules.

Parameters:name – a (partial) module name for filtering (default: None)

remove_module_path(path, set_mod_paths=True)

Remove specified module path (using ‘module unuse’).

Parameters:

• path – path to remove from $MODULEPATH via ‘unuse’
• set_mod_paths – (re)set self.mod_paths

run_module(*args, **kwargs)

Run module command, tweak output that is exec’ed if necessary.

set_path_env_var(key, paths)

Set environment variable with given name to the given list of paths.

class easybuild.tools.modules.Lmod(*args, **kwargs)

Bases: easybuild.tools.modules.ModulesTool

Interface to Lmod.

COMMAND = 'lmod'

COMMAND_ENVIRONMENT = 'LMOD_CMD'

DEPR_VERSION = '7.0.0'

NAME = 'Lmod'

REQ_VERSION = '6.5.1'

REQ_VERSION_DEPENDS_ON = '7.6.1'

SHOW_HIDDEN_OPTION = '--show-hidden'

USER_CACHE_DIR = '/home/docs/.lmod.d/.cache'

VERSION_REGEXP = '^Modules\\s+based\\s+on\\s+Lua:\\s+Version\\s+(?P<version>\\d\\S*)\\s'

available(mod_name=None)

Return a list of available modules for the given (partial) module name; use None to obtain a list of all available modules.

Parameters:name – a (partial) module name for filtering (default: None)

check_module_function(*args, **kwargs)

Check whether selected module tool matches ‘module’ function definition.

check_module_output(cmd, stdout, stderr)

Check output of ‘module’ command, see if if is potentially invalid.

compose_cmd_list(args, opts=None)

Compose full module command to run, based on provided arguments

Parameters:args – list of arguments for module command Returns:list of strings representing the full module command to run

get_setenv_value_from_modulefile(mod_name, var_name)

Get value for specific ‘setenv’ statement from module file for the specified module.

Parameters:

• mod_name – module name
• var_name – name of the variable being set for which value should be returned

module_wrapper_exists(mod_name)

Determine whether a module wrapper with specified name exists. First check for wrapper defined in .modulerc.lua, fall back to also checking .modulerc (Tcl syntax).

prepend_module_path(path, set_mod_paths=True, priority=None)

Prepend given module path to list of module paths, or bump it to 1st place.

Parameters:

• path – path to prepend to $MODULEPATH
• set_mod_paths – (re)set self.mod_paths
• priority – priority for this path in $MODULEPATH (Lmod-specific)

unuse(path)

Remove a module path

update()

Update after new modules were added.

use(path, priority=None)

Add path to $MODULEPATH via ‘module use’.

Parameters:

• path – path to add to $MODULEPATH
• priority – priority for this path in $MODULEPATH (Lmod-specific)

class easybuild.tools.modules.Modules(*args, **kwargs)

Bases: easybuild.tools.modules.EnvironmentModulesC

NO LONGER SUPPORTED: interface to modules tool, use modules_tool from easybuild.tools.modules instead

class easybuild.tools.modules.ModulesTool(mod_paths=None, testing=False)

Bases: object

An abstract interface to a tool that deals with modules.

COMMAND = None

COMMAND_ENVIRONMENT = None

COMMAND_SHELL = None

DEPR_VERSION = None

MAX_VERSION = None

NAME = None

REQ_VERSION = None

TERSE_OPTION = (0, '--terse')

USER_CACHE_DIR = None

VERSION_OPTION = '--version'

VERSION_REGEXP = None

add_module_path(path, set_mod_paths=True)

Add specified module path (using ‘module use’) if it’s not there yet.

Parameters:

• path – path to add to $MODULEPATH via ‘use’
• set_mod_paths – (re)set self.mod_paths

available(mod_name=None, extra_args=None)

Return a list of available modules for the given (partial) module name; use None to obtain a list of all available modules.

Parameters:mod_name – a (partial) module name for filtering (default: None)

buildstats()

Return tuple with data to be included in buildstats

check_cmd_avail()

Check whether modules tool command is available.

check_loaded_modules()

Check whether any (EasyBuild-generated) modules are loaded already in the current session

check_module_function(allow_mismatch=False, regex=None)

Check whether selected module tool matches ‘module’ function definition.

check_module_output(cmd, stdout, stderr)

Check output of ‘module’ command, see if if is potentially invalid.

check_module_path()

Check if MODULEPATH is set and change it if necessary.

compose_cmd_list(args, opts=None)

Compose full module command to run, based on provided arguments

Parameters:args – list of arguments for module command Returns:list of strings representing the full module command to run

exist(mod_names, mod_exists_regex_template=None, skip_avail=False, maybe_partial=True)

Check if modules with specified names exists.

Parameters:

• mod_names – list of module names
• mod_exists_regex_template – DEPRECATED and unused
• skip_avail – skip checking through ‘module avail’, only check via ‘module show’
• maybe_partial – indicates if the module name may be a partial module name

get_setenv_value_from_modulefile(mod_name, var_name)

Get value for specific ‘setenv’ statement from module file for the specified module.

Parameters:

• mod_name – module name
• var_name – name of the variable being set for which value should be returned

get_value_from_modulefile(mod_name, regex, strict=True)

Get info from the module file for the specified module.

Parameters:

• mod_name – module name
• regex – (compiled) regular expression, with one group

interpret_raw_path_lua(txt)

Interpret raw path (Lua syntax): resolve environment variables, join paths where pathJoin is specified

interpret_raw_path_tcl(txt)

Interpret raw path (TCL syntax): resolve environment variables

list()

Return result of ‘module list’.

load(modules, mod_paths=None, purge=False, init_env=None, allow_reload=True)

Load all requested modules.

Parameters:

• modules – list of modules to load
• mod_paths – list of module paths to activate before loading
• purge – whether or not a ‘module purge’ should be run before loading
• init_env – original environment to restore after running ‘module purge’
• allow_reload – allow reloading an already loaded module

loaded_modules()

Return a list of loaded modules.

mk_module_cache_key(partial_key)

Create a module cache key, using the specified partial key, by combining it with the current $MODULEPATH.

modpath_extensions_for(mod_names)

Determine dictionary with $MODULEPATH extensions for specified modules. All potential $MODULEPATH extensions are included, even the ones guarded by a condition (which is not checked). Only direct $MODULEPATH extensions are found, no recursion if performed for modules that load other modules. Modules with an empty list of $MODULEPATH extensions are included in the result.

Parameters:mod_names – list of module names for which to determine the list of $MODULEPATH extensions Returns:dictionary with module names as keys and lists of $MODULEPATH extensions as values

module_wrapper_exists(mod_name, modulerc_fn='.modulerc', mod_wrapper_regex_template=None)

Determine whether a module wrapper with specified name exists. Only .modulerc file in Tcl syntax is considered here.

modulefile_path(mod_name, strip_ext=False)

Get the path of the module file for the specified module

Parameters:

• mod_name – module name
• strip_ext – strip (.lua) extension from module fileame (if present)

path_to_top_of_module_tree(top_paths, mod_name, full_mod_subdir, deps, modpath_exts=None)

Recursively determine path to the top of the module tree, for given module, module subdir and list of $MODULEPATH extensions per dependency module.

For example, when to determine the path to the top of the module tree for the HPL/2.1 module being installed with a goolf/1.5.14 toolchain in a Core/Compiler/MPI hierarchy (HierarchicalMNS):

• starting point:

  top_paths = [‘<prefix>’, ‘<prefix>/Core’] mod_name = ‘HPL/2.1’ full_mod_subdir = ‘<prefix>/MPI/Compiler/GCC/4.8.2/OpenMPI/1.6.5’ deps = [‘GCC/4.8.2’, ‘OpenMPI/1.6.5’, ‘OpenBLAS/0.2.8-LAPACK-3.5.0’, ‘FFTW/3.3.4’, ‘ScaLAPACK/…’]

• 1st iteration: find module that extends $MODULEPATH with ‘<prefix>/MPI/Compiler/GCC/4.8.2/OpenMPI/1.6.5’,

  => OpenMPI/1.6.5 (in ‘<prefix>/Compiler/GCC/4.8.2’ subdir); recurse with mod_name = ‘OpenMPI/1.6.5’ and full_mod_subdir = ‘<prefix>/Compiler/GCC/4.8.2’

• 2nd iteration: find module that extends $MODULEPATH with ‘<prefix>/Compiler/GCC/4.8.2’

  => GCC/4.8.2 (in ‘<prefix>/Core’ subdir); recurse with mod_name = ‘GCC/4.8.2’ and full_mod_subdir = ‘<prefix>/Core’

• 3rd iteration: try to find module that extends $MODULEPATH with ‘<prefix>/Core’

  => ‘<prefix>/Core’ is in top_paths, so stop recursion

Parameters:

• top_paths – list of potentation ‘top of module tree’ (absolute) paths
• mod_name – (short) module name for starting point (only used in log messages)
• full_mod_subdir – absolute path to module subdirectory for starting point
• deps – list of dependency modules for module at starting point
• modpath_exts – list of module path extensions for each of the dependency modules

prepend_module_path(path, set_mod_paths=True, priority=None)

Prepend given module path to list of module paths, or bump it to 1st place.

Parameters:

• path – path to prepend to $MODULEPATH
• set_mod_paths – (re)set self.mod_paths
• priority – priority for this path in $MODULEPATH (Lmod-specific)

purge()

Purge loaded modules.

read_module_file(mod_name)

Read module file with specified name.

remove_module_path(path, set_mod_paths=True)

Remove specified module path (using ‘module unuse’).

Parameters:

• path – path to remove from $MODULEPATH via ‘unuse’
• set_mod_paths – (re)set self.mod_paths

run_module(*args, **kwargs)

Run module command.

Parameters:

• args – list of arguments for module command; first argument should be the subcommand to run
• kwargs – dictionary with options that control certain aspects of how to run the module command

set_and_check_version()

Get the module version, and check any requirements

set_mod_paths(mod_paths=None)

Set mod_paths, based on $MODULEPATH unless a list of module paths is specified.

Parameters:mod_paths – list of entries for $MODULEPATH to use

set_path_env_var(key, paths)

Set path environment variable to the given list of paths.

show(mod_name)

Run ‘module show’ for the specified module.

unload(modules=None)

Unload all requested modules.

unuse(path)

Remove module path via ‘module unuse’.

update()

Update after new modules were added.

use(path, priority=None)

Add path to $MODULEPATH via ‘module use’.

Parameters:

• path – path to add to $MODULEPATH
• priority – priority for this path in $MODULEPATH (Lmod-specific)

class easybuild.tools.modules.NoModulesTool(*args, **kwargs)

Bases: easybuild.tools.modules.ModulesTool

Class that mock the module behaviour, used for operation not requiring modules. Eg. tests, fetch only

available(*args, **kwargs)

No modules, so nothing available

check_loaded_modules()

Nothing to do since no modules

exist(mod_names, *args, **kwargs)

No modules, so nothing exists

list()

No modules loaded

easybuild.tools.modules.avail_modules_tools()

Return all known modules tools.

easybuild.tools.modules.curr_module_paths(normalize=False)

Return a list of current module paths.

Parameters:normalize – Normalize the paths

easybuild.tools.modules.get_software_libdir(name, only_one=True, fs=None)

Find library subdirectories for the specified software package.

Returns the library subdirectory, relative to software root. It fails if multiple library subdirs are found, unless only_one is False which yields a list of all library subdirs.

Parameters:

• name – name of the software package
• only_one – indicates whether only one lib path is expected to be found
• fs – only retain library subdirs that contain one of the files in this list

easybuild.tools.modules.get_software_root(name, with_env_var=False)

Return the software root set for a particular software name.

easybuild.tools.modules.get_software_root_env_var_name(name)

Return name of environment variable for software root.

easybuild.tools.modules.get_software_version(name)

Return the software version set for a particular software name.

easybuild.tools.modules.get_software_version_env_var_name(name)

Return name of environment variable for software root.

easybuild.tools.modules.invalidate_module_caches_for(path)

Invalidate cache entries related to specified path.

easybuild.tools.modules.mk_module_path(paths)

Create a string representing the list of module paths.

easybuild.tools.modules.modules_tool(mod_paths=None, testing=False)

Return interface to modules tool (environment modules (C, Tcl), or Lmod)

easybuild.tools.modules.reset_module_caches()

Reset module caches.