Skip to content

modules

This python module implements the environment modules functionality: - loading modules - checking for available modules - ...

Authors:

  • Stijn De Weirdt (Ghent University)
  • Dries Verdegem (Ghent University)
  • Kenneth Hoste (Ghent University)
  • Pieter De Baets (Ghent University)
  • Jens Timmerman (Ghent University)
  • David Brown (Pacific Northwest National Laboratory)

EnvironmentModules

Bases: EnvironmentModulesTcl

Interface to environment modules 4.0+

__init__(*args, **kwargs)

Constructor, set Environment Modules-specific class variable values.

available(mod_name=None, extra_args=None)

Return a list of available modules for the given (partial) module name; use None to obtain a list of all available modules.

PARAMETER DESCRIPTION
mod_name

a (partial) module name for filtering (default: None)

DEFAULT: None

check_module_function(allow_mismatch=False, regex=None)

Check whether selected module tool matches 'module' function definition.

check_module_output(cmd, stdout, stderr)

Check output of 'module' command, see if if is potentially invalid.

get_setenv_value_from_modulefile(mod_name, var_name)

Get value for specific 'setenv' statement from module file for the specified module.

PARAMETER DESCRIPTION
mod_name

module name

var_name

name of the variable being set for which value should be returned

EnvironmentModulesC

Bases: ModulesTool

Interface to (C) environment modules (modulecmd).

get_setenv_value_from_modulefile(mod_name, var_name)

Get value for specific 'setenv' statement from module file for the specified module.

PARAMETER DESCRIPTION
mod_name

module name

var_name

name of the variable being set for which value should be returned

run_module(*args, **kwargs)

Run module command, tweak output that is exec'ed if necessary.

update()

Update after new modules were added.

EnvironmentModulesTcl

Bases: EnvironmentModulesC

Interface to (Tcl) environment modules (modulecmd.tcl).

available(mod_name=None, extra_args=None)

Return a list of available modules for the given (partial) module name; use None to obtain a list of all available modules.

PARAMETER DESCRIPTION
mod_name

a (partial) module name for filtering (default: None)

DEFAULT: None

remove_module_path(path, set_mod_paths=True)

Remove specified module path (using 'module unuse').

PARAMETER DESCRIPTION
path

path to remove from $MODULEPATH via 'unuse'

set_mod_paths

(re)set self.mod_paths

DEFAULT: True

run_module(*args, **kwargs)

Run module command, tweak output that is exec'ed if necessary.

set_path_env_var(key, paths)

Set environment variable with given name to the given list of paths.

Lmod

Bases: ModulesTool

Interface to Lmod.

__init__(*args, **kwargs)

Constructor, set lmod-specific class variable values.

available(mod_name=None)

Return a list of available modules for the given (partial) module name; use None to obtain a list of all available modules.

PARAMETER DESCRIPTION
mod_name

a (partial) module name for filtering (default: None)

DEFAULT: None

check_module_function(*args, **kwargs)

Check whether selected module tool matches 'module' function definition.

check_module_output(cmd, stdout, stderr)

Check output of 'module' command, see if if is potentially invalid.

compose_cmd_list(args, opts=None)

Compose full module command to run, based on provided arguments

PARAMETER DESCRIPTION
args

list of arguments for module command

RETURNS DESCRIPTION

list of strings representing the full module command to run

get_setenv_value_from_modulefile(mod_name, var_name)

Get value for specific 'setenv' statement from module file for the specified module.

PARAMETER DESCRIPTION
mod_name

module name

var_name

name of the variable being set for which value should be returned

module_wrapper_exists(mod_name)

Determine whether a module wrapper with specified name exists. First check for wrapper defined in .modulerc.lua, fall back to also checking .modulerc (Tcl syntax).

prepend_module_path(path, set_mod_paths=True, priority=None)

Prepend given module path to list of module paths, or bump it to 1st place.

PARAMETER DESCRIPTION
path

path to prepend to $MODULEPATH

set_mod_paths

(re)set self.mod_paths

DEFAULT: True

priority

priority for this path in $MODULEPATH (Lmod-specific)

DEFAULT: None

unuse(path)

Remove a module path

update()

Update after new modules were added.

use(path, priority=None)

Add path to $MODULEPATH via 'module use'.

PARAMETER DESCRIPTION
path

path to add to $MODULEPATH

priority

priority for this path in $MODULEPATH (Lmod-specific)

DEFAULT: None

Modules

Bases: EnvironmentModulesC

NO LONGER SUPPORTED: interface to modules tool, use modules_tool from easybuild.tools.modules instead

ModulesTool

Bases: object

An abstract interface to a tool that deals with modules.

__init__(mod_paths=None, testing=False)

Create a ModulesTool object

PARAMETER DESCRIPTION
mod_paths

A list of paths where the modules can be located @type mod_paths: list

DEFAULT: None

__str__()

String representation of this ModulesTool instance.

add_module_path(path, set_mod_paths=True)

Add specified module path (using 'module use') if it's not there yet.

PARAMETER DESCRIPTION
path

path to add to $MODULEPATH via 'use'

set_mod_paths

(re)set self.mod_paths

DEFAULT: True

available(mod_name=None, extra_args=None)

Return a list of available modules for the given (partial) module name; use None to obtain a list of all available modules.

PARAMETER DESCRIPTION
mod_name

a (partial) module name for filtering (default: None)

DEFAULT: None

buildstats()

Return tuple with data to be included in buildstats

check_cmd_avail()

Check whether modules tool command is available.

check_loaded_modules()

Check whether any (EasyBuild-generated) modules are loaded already in the current session

check_module_function(allow_mismatch=False, regex=None)

Check whether selected module tool matches 'module' function definition.

check_module_output(cmd, stdout, stderr)

Check output of 'module' command, see if if is potentially invalid.

check_module_path()

Check if MODULEPATH is set and change it if necessary.

compose_cmd_list(args, opts=None)

Compose full module command to run, based on provided arguments

PARAMETER DESCRIPTION
args

list of arguments for module command

RETURNS DESCRIPTION

list of strings representing the full module command to run

exist(mod_names, mod_exists_regex_template=None, skip_avail=False, maybe_partial=True)

Check if modules with specified names exists.

PARAMETER DESCRIPTION
mod_names

list of module names

mod_exists_regex_template

DEPRECATED and unused

DEFAULT: None

skip_avail

skip checking through 'module avail', only check via 'module show'

DEFAULT: False

maybe_partial

indicates if the module name may be a partial module name

DEFAULT: True

get_setenv_value_from_modulefile(mod_name, var_name)

Get value for specific 'setenv' statement from module file for the specified module.

PARAMETER DESCRIPTION
mod_name

module name

var_name

name of the variable being set for which value should be returned

get_value_from_modulefile(mod_name, regex, strict=True)

Get info from the module file for the specified module.

PARAMETER DESCRIPTION
mod_name

module name

regex

(compiled) regular expression, with one group

interpret_raw_path_lua(txt)

Interpret raw path (Lua syntax): resolve environment variables, join paths where pathJoin is specified

interpret_raw_path_tcl(txt)

Interpret raw path (TCL syntax): resolve environment variables

list()

Return result of 'module list'.

load(modules, mod_paths=None, purge=False, init_env=None, allow_reload=True)

Load all requested modules.

PARAMETER DESCRIPTION
modules

list of modules to load

mod_paths

list of module paths to activate before loading

DEFAULT: None

purge

whether or not a 'module purge' should be run before loading

DEFAULT: False

init_env

original environment to restore after running 'module purge'

DEFAULT: None

allow_reload

allow reloading an already loaded module

DEFAULT: True

loaded_modules()

Return a list of loaded modules.

mk_module_cache_key(partial_key)

Create a module cache key, using the specified partial key, by combining it with the current $MODULEPATH.

modpath_extensions_for(mod_names)

Determine dictionary with $MODULEPATH extensions for specified modules. All potential $MODULEPATH extensions are included, even the ones guarded by a condition (which is not checked). Only direct $MODULEPATH extensions are found, no recursion if performed for modules that load other modules. Modules with an empty list of $MODULEPATH extensions are included in the result.

PARAMETER DESCRIPTION
mod_names

list of module names for which to determine the list of $MODULEPATH extensions

RETURNS DESCRIPTION

dictionary with module names as keys and lists of $MODULEPATH extensions as values

module_wrapper_exists(mod_name, modulerc_fn='.modulerc', mod_wrapper_regex_template=None)

Determine whether a module wrapper with specified name exists. Only .modulerc file in Tcl syntax is considered here.

modulefile_path(mod_name, strip_ext=False)

Get the path of the module file for the specified module

PARAMETER DESCRIPTION
mod_name

module name

strip_ext

strip (.lua) extension from module fileame (if present)

DEFAULT: False

path_to_top_of_module_tree(top_paths, mod_name, full_mod_subdir, deps, modpath_exts=None)

Recursively determine path to the top of the module tree, for given module, module subdir and list of $MODULEPATH extensions per dependency module.

For example, when to determine the path to the top of the module tree for the HPL/2.1 module being installed with a goolf/1.5.14 toolchain in a Core/Compiler/MPI hierarchy (HierarchicalMNS):

  • starting point: top_paths = ['', '/Core'] mod_name = 'HPL/2.1' full_mod_subdir = '/MPI/Compiler/GCC/4.8.2/OpenMPI/1.6.5' deps = ['GCC/4.8.2', 'OpenMPI/1.6.5', 'OpenBLAS/0.2.8-LAPACK-3.5.0', 'FFTW/3.3.4', 'ScaLAPACK/...']

  • 1st iteration: find module that extends $MODULEPATH with '/MPI/Compiler/GCC/4.8.2/OpenMPI/1.6.5', => OpenMPI/1.6.5 (in '/Compiler/GCC/4.8.2' subdir); recurse with mod_name = 'OpenMPI/1.6.5' and full_mod_subdir = '/Compiler/GCC/4.8.2'

  • 2nd iteration: find module that extends $MODULEPATH with '/Compiler/GCC/4.8.2' => GCC/4.8.2 (in '/Core' subdir); recurse with mod_name = 'GCC/4.8.2' and full_mod_subdir = '/Core'

  • 3rd iteration: try to find module that extends $MODULEPATH with '/Core' => '/Core' is in top_paths, so stop recursion

PARAMETER DESCRIPTION
top_paths

list of potentation 'top of module tree' (absolute) paths

mod_name

(short) module name for starting point (only used in log messages)

full_mod_subdir

absolute path to module subdirectory for starting point

deps

list of dependency modules for module at starting point

modpath_exts

list of module path extensions for each of the dependency modules

DEFAULT: None

prepend_module_path(path, set_mod_paths=True, priority=None)

Prepend given module path to list of module paths, or bump it to 1st place.

PARAMETER DESCRIPTION
path

path to prepend to $MODULEPATH

set_mod_paths

(re)set self.mod_paths

DEFAULT: True

priority

priority for this path in $MODULEPATH (Lmod-specific)

DEFAULT: None

purge()

Purge loaded modules.

read_module_file(mod_name)

Read module file with specified name.

remove_module_path(path, set_mod_paths=True)

Remove specified module path (using 'module unuse').

PARAMETER DESCRIPTION
path

path to remove from $MODULEPATH via 'unuse'

set_mod_paths

(re)set self.mod_paths

DEFAULT: True

run_module(*args, **kwargs)

Run module command.

PARAMETER DESCRIPTION
args

list of arguments for module command; first argument should be the subcommand to run

DEFAULT: ()

kwargs

dictionary with options that control certain aspects of how to run the module command

DEFAULT: {}

set_and_check_version()

Get the module version, and check any requirements

set_mod_paths(mod_paths=None)

Set mod_paths, based on $MODULEPATH unless a list of module paths is specified.

PARAMETER DESCRIPTION
mod_paths

list of entries for $MODULEPATH to use

DEFAULT: None

set_path_env_var(key, paths)

Set path environment variable to the given list of paths.

show(mod_name)

Run 'module show' for the specified module.

unload(modules=None)

Unload all requested modules.

unuse(path)

Remove module path via 'module unuse'.

update()

Update after new modules were added.

use(path, priority=None)

Add path to $MODULEPATH via 'module use'.

PARAMETER DESCRIPTION
path

path to add to $MODULEPATH

priority

priority for this path in $MODULEPATH (Lmod-specific)

DEFAULT: None

NoModulesTool

Bases: ModulesTool

Class that mock the module behaviour, used for operation not requiring modules. Eg. tests, fetch only

available(*args, **kwargs)

No modules, so nothing available

check_loaded_modules()

Nothing to do since no modules

exist(mod_names, *args, **kwargs)

No modules, so nothing exists

list()

No modules loaded

avail_modules_tools()

Return all known modules tools.

curr_module_paths(normalize=False)

Return a list of current module paths.

PARAMETER DESCRIPTION
normalize

Normalize the paths

DEFAULT: False

get_software_libdir(name, only_one=True, fs=None)

Find library subdirectories for the specified software package.

Returns the library subdirectory, relative to software root. It fails if multiple library subdirs are found, unless only_one is False which yields a list of all library subdirs. If only_one is True and fs is None, select the one subdirectory with shared or static libraries, if possible.

PARAMETER DESCRIPTION
name

name of the software package

only_one

indicates whether only one lib path is expected to be found

DEFAULT: True

fs

only retain library subdirs that contain one of the files in this list

DEFAULT: None

get_software_root(name, with_env_var=False)

Return the software root set for a particular software name.

get_software_root_env_var_name(name)

Return name of environment variable for software root.

get_software_version(name)

Return the software version set for a particular software name.

get_software_version_env_var_name(name)

Return name of environment variable for software root.

invalidate_module_caches_for(path)

Invalidate cache entries related to specified path.

mk_module_path(paths)

Create a string representing the list of module paths.

modules_tool(mod_paths=None, testing=False)

Return interface to modules tool (environment modules (C, Tcl), or Lmod)

reset_module_caches()

Reset module caches.