NAME

    HPCI

VERSION

Version 0.31

SYNOPSIS

    use HPCI;

    my $group = HPCI->group(
        cluster => ($ENV{HPCI_CLUSTER} // 'uni'),
        ...
    );
    $group->stage(
        name => 'analysis_A',
        command => '...'
    );
    $group->stage(
        name => 'analysis_B',
        command => '...'
    );
    $group->stage(
        name => 'analysis_C',
        command => '...'
    );
    $group->stage(
        name => 'report',
        command => '...'
    );
    $group->add_deps(
        pre_reqs => [ qw(analysis_A analysis_B analysis_C) ],
        dep => 'report'
    );
    my $status_info = $group->execute;

    my $exit_status = 0;
    for my $stage ( qw(analysis_A analysis_B analysis_C report) ) {
        if (my $stat = $status_info->{$stage}[-1]{exit_status}) {
            $exit_status ||= $stat;
            print stderr "Stage $stage failed, status $stat!\n";
        }
    }

OVERVIEW

HPCI (High Performance Computing Interface) provides an interface to a range of types of computer aggregations (clusters, clouds, ...). (The rest of this document will use cluster henceforth to refer to any type of aggregation that is supported by HPCI.)

A cluster is defined as a software interface that allows running multiple programs on separate compute elements (nodes).

HPCI uses an HPCD (High Performance Computing Driver) module to translate its standard interface into the appropriate access mechanisms for the type of cluster that is selected. (If you have used the DBI/DBD modules for accessing databases, this will seem very familiar.)

The goal of this HPCI/HPCD split is to allow users to write programs that make use of cluster facilities in a portable manner. If there is a reason to run the same program using a different type of cluster, it should only require change the cluster definition attributes provided to one parent object creation; the rest of code need not know or care about the changed cluster type. Programs which are likely to be run on different cluster types will usually be written to get the cluster attribute information from a configuration file, or command line arguments - so the program itself need not change at all.

Running a program on different types of clusters can happen for a number of reasons. An organization might have access to multiple types of cluster, such as an in-house cluster plus an external cloud. Scholarly research often shares programs both to allow similar research, or to validate existing research results.

HPCD modules can provide cluster-specific extensions. That can either be a different kind of functionality, or it can be as simple as allowing the teminology familiar to users of that cluster type to be used in place of the generic terminology provided by HPCI. However, using such extensions makes it harder to move to a different cluster type. So, actually making use of such extensions must be considered carefully.

INTERFACE

The life cycle of running a group of commands on a cluster is as follows:

create group: A group object is created using the HPCI "class method" group. HPCI isn't really a class, it just appears to be one. Its group "class method" actually delegates creation of a group object to the HPCD module that is indicated by the cluster attribute and it returns an cluster-specific group object that supports the HPCI interface.
create stages: A stage is created for each command that is to be executed on a separate node of the cluster. This is created using the group object's method stage.
define dependency ordering: An important reason for running a group of jobs on a cluster is the ability to use multiple computers to run portions of the computation at the same time, rather than having them compete for the rsources of a single computer. However, often some stages will depend upon the output of other stages. Such a dependent stage cannot start executing until all pre-requisite stages have completed. Specifying such dependency requirements is done with the group method add_deps.
execution: Finally, the group method execute will run the entire set of stages. It does not return until all stages have completed (or have been skipped). Each stage will normally be run once. However, it is possible for there to be retries under some recoverable failure conditions causing some stages to be run .multiple times. Alternatively, a hard failure of one stage can be a trigger for completely skipping the execution of other stages. Each separate execution of a stage is managed with an internal object (called a job - but a user program won't see job objects directly).

OUTPUT DIRECTORY LAYOUT

There are a number of output files created during a group execution.

The default layout of these files is:

  <base_dir>                     "."
    <group_dir>                  <base_dir>/<name>-<YYYYMMDD-hhmmss>
      <log>                      <group_dir>/<name>.log
      <stage_dir>                <group_dir>/<stage_name>
        <script_file>            <stage_dir>/script.sh
        <job_dir>                <stage_dir>/<retry_number>
          stdout
          stderr
        final_retry              symlink to final <job_dir>

Many of these files/directories can be re-assigned to different location using group or stage attributes - shown avove is the default layout. Commonly, you will specifically use the base_dir attribute to choose a location other than the current directory for placing the tree; or else use the group_dir attribute if you want to choose a location that does not create a sub-directory for you. (If this is an already existing directory that is being re-used you may end up with a mixture of old and new contents that are hard to figure out.)

base_dir

The top level of all the generated output. It defaults to ".", but can be specified explicitly when the group is created.

group_dir

By default, a new directory is created under base_dir. Its name is name-YYYYMMDD-hhmmss - the name of the group along with a timestamp of when the execution started.

log

The automatically provided log is written to the file name.log directly under group_dir. This logs information about the execution of the entire group of stages.

stage_dir

Each stage creates a sub-directory beneath group_dir with the same name as the stage.

script_file

The script created to be executed on the cluster node. This wraps the specified command with additional logic to pass on environment and config info, and to set output redirection. It is called "script.sh" and placed in stage_dir.

job_dir

A sub-directory is created under stage_dir for each attempt to run the command. Usually, there will only be a single attempt. However, if the cluster driver provides mechanisms for detecting recoverable issues and then retries a command there can be more than one attempt; or alternately, if a pre-requisite stage fails there might be no attempt made (in that case, though, the entire stage_dir directory would not even get created). These directories are simply named with the retry number ("0", "1", ...).

stdout/stderr

Within each job_dir, the files "stdout" and "stderr" collect the standard output and standard error output from that (re)try attempt to run the command.

final_retry

A symlink named "final_retry" is created within stage_dir that points to the job_dir of the final (re)try. Since you often don't care as much about the initial run tries as you do about the last one, this symlink provides a consistant access path to that final retry.

HPCI "Class"

You can pretend that HPCI is a class with one class method named group.

METHOD group

The group method creates and returns a group object, which you can treat like a HPCI::Group object. (In fact, it really returns an object of class HPCD::cluster::Group, but if you ignore that fact then you can trivially have your program run on some other cluster type.)

The group object is described further below.

Group Object

A group object is created with the HPCI class method group (and not using a new method directly.

Cluster Attributes

The one necessary attribute is cluster. For some specific cluster types there may be additional attributes required for connecting to the cluster software (authentification, usage class info, etc.).

cluster

The cluster attribute specifies which type of cluster is to be used. This is the only required attribute. (Some cluster types may have additional attributes that are required for specifying connection and authentification info.)

Basic Attributes

name

The name you give to a group is used for creating the directory where output is stored, and also in log messages. A default name "default_group_name" is provided if you do not specific an explicit name. Using the default name is adequate in simple programs which only create one group, but for more complicated programs giving separate names to each group is necessary to easily identify the output of each group.

Directory Layout Attributes

base_dir

If none of the other directory layout attributes are used to over-ride this, this attribute specifies the directory in which all output directories and files will be created. This is usually an existing directory; it defaults to the current directort ".".

group_dir

This directory is usually created to contain the outputs of the group execution. By default, it is directly under base_dir with a name that consists of the group name attribute and a timestamp (e.g. "T_Definition-20150521-153256").

If you provide an explicit value for this parameter, then it should not be an existing directory containing previous results. (If it is, the log file will be appended to the previous one, but the stage directories will over-write equivalently named directories and files that are created in this run, while leaving unchanged any that did not recur, so you'll have a mix of old and new contents.)

Logging Attributes

An HPCI group logs its activities using a Log::Log4perl logger. The logger can either be provided by the caller, or else HPCI will create its own.

log

This a Log::Log4perl::Logger object. If it is provided as an attribute to the group creation call, it will be used as it is, and the other logging attributes will be ignored.

If it is not provided by the user, a new Log::Log4perl::Logger object will be created using the attributes below to define where it is logged to. This created logger will send all log entries to a file, as well as sending all info and higher log entries to stderr.

log_path

If this attribute is provided (and the log attribute is not provided) it will be used as the full pathname of a file where the log will be written. If it is not provided, it will use the path log_dir/log_file by default.

log_dir

If neither log or log_path is provided, this attribute can be used to specify the directory where the log file is to be written. By default, it uses group_dir.

log_file

If neither log or log_path is provided, this attribute can be used to specify the file name to be written in the log directory. By default, it uses name.log.

log_level

You can provide this attribute to change the default log level setting from "info" to any of debug info warn error fatal.

Operational Attributes

max_concurrent

This attribute specifies the maximum number of stages that will be executing at one time. The default setting of 0 allows as many stages as possible (all those that are not waiting for a pre-requisite stage to complete) to run at the same time.

status

This attribute is set internally while stages are executed. It contains the final result status from each stage run that has completed. The execute method returns this value when execution completes, so you will usually not need to access it explicitly yourself.

This value is a hashref (indexed by stage name). The values are arrayrefs (indexed by run number 0..n). For each run, there is a hash. The key exit_status contains the exit status of the run. If the stage was never run, exit_status instead contains a text message listing the reason that it was skipped.

Environment Passing Attributes

You can set up a set of enviroment variables that will be provided to all stages. (You can also set variables that are only for individual stages - if so, they will modfy any set you provide in the group.)

See HPCI::Env for a description of these.

Methods

stage

The method stage is used to create a new stage object. Its attributes are described below.

add_deps

The method add_deps is used to specify pre-requisite/dependent relationships. It takes either a hashref or a list containing two pairs. One key must be either pre_req or pre_reqs, the other must be either dep or deps.

If the singular key is used (either pre_req or dep), then the corresponding value must be a scalar - either the string name of a stage object, or a reference to a stage object.

If the plural key is used (either pre_reqs or deps), then the corresponding value must be an arrayref containing a list of scalars, each one being either the string name of a stage or a reference to a stage object.

The plural forms are provided for convenience - often the output file from one preparation stage is required by many others, or the output from many processing stages is needed by a stage that merges results into a summary report.

It is an error to provide a sequence of dependencies that form a cycle.

execute

The execute method is the final goal of building the group. It schedules the execution of individual stages. It allows for dependencies before running a stage. It provides for re-running a stage if a soft failure has occurred that allows a retry. If a failure that cannot be retried occurrs, it skips scheduling dependent stages, or stops scheduling all new stages.

Stage Object

Attributes

name

The name attribute must be provided for stages. It is a string. Each stage within a group must have a unique name. There is no default value provided.

command

The command attribute must be provided before the group is executed. It can either be provided as a string attribute in the stage method call that creates the stage, or using the one of the command-setting methods provided by the stage class.

See HPCI::Stage for more details about the command setting methods.

dir

The dir attribute is optional. It specifies the direcory in which files related to the stage are placed. By default, it is group_dir/stage_name. You will usually not need to change this.

resources_required

retry_resources_required

The resources_required and retry_resources_required are used to define resources that will be required by the stage when it executes. These attributes are somewhat cluster specific - each cluster has its own set of requirements for how a job submission must specify the sort of resources that it will require.

The resources_required attribute is a hash, specifying the value for each resource that is to be considered.

The retry_resources_required attribute is also a hash. For each resource, you can specify an array of values. If the cluster driver is able to detect that a run failed because the resource was inadequate, it will retry the run with the next larger value from this list.

See HPCI::Stage for more details about resources.

failure_action ('abort_group', 'abort_deps'*, or 'ignore')

Specifies the action to take if this stage fails (terminates with a non-zero status).

There are three string values that it can have:

- abort_deps (default): If the stage fails, then any stages which depend upon it (recursively) are not run. The group continues executing until all stages which are not dependent upon this stage (including those that have not yet been initiated) complete execution.
- abort_group: If the stage fails, then no other stages are started. The group simply waits until stages that have already been started complete and then returns.
- ignore: Execution continues unchanged, any dependent stages will be run when they are no longer blocked.

As an alternative to providing a value to the failute_action attribute when you create a stage, you can instead provide one of the pseudo-attributes 'abort_group_on_failure', 'abort_deps_on_failure', or 'ignore_failure' with a true value to specify 'abort_group', 'abort_deps', or 'ignore' respectively.

Methods

command creation

See HPCI::Stage for details.

Env

See HPCI::Env for details.

Additional

This is an early public release of HPCI, and at present, there are only two drivers available.

Only one cluster type is directly included within the HPCI package. The cluster type HPCD::uni runs on a "cluster" of only one machine. It simply uses fork to submit individual stages and has facility for retries and timeouts. This is the default cluster type used for testing, as it will work natively on all types of Unix systems. It is also possible to use this driver as a fallback, in cases where the only available "real" cluster is not accessable for some reason.

Additionally, there is the HPCD::SGE driver available on CPAN. It has seen heavy use within Boutros Lab.

Now that these packages have been released, it is likely new cluster drivers will be written. People interested in developing drivers for additional cluster types should contact the authors of this package to co-ordinate releases, features needed, etc. at mailto:BoutrosLabSoftware@oicr.on.ca.

Additionally, you may wish to subscribe to the email list mentioned at https:://lists.oicr.on.ca/mailman/listinfo/hpci-discuss. This is expected to be a low volume discussion group, although the future will tell what the actual volume will be.

As additional capabilities of new cluster types are addressed, and as different control needs used at other organizations are identified; this interface will surely change. As far as possible, such changes will be done in an upwardly compatible manner, but until a few more drivers have been integrated there is the possibility of changes that are not fully backward compatible. Watch the release notes for warnings of such issues. At some point there will be a 1.0.0 release, at which point this expectation of (limited) incompatible future change will be dropped. After that point, incompatible changes will only be made for critical reasons.

The reason for separate distribution of cluster-specific HPCD packages are fairly obvious:

-: The maintainers of the HPCI package do not have access to every possible cluster type, and it unlikely that anyone will have access to all supported cluster types from one location, so the driver modules will need to be tested separately anyhow.
-: A user of HPCI is equally not going to have need to access every type of cluster that exists, so they will probably prefer to only download the driver modules that they actually need.

AUTHOR

Christopher Lalansingh - Boutros Lab

John Macdonald - Boutros Lab

ACKNOWLEDGEMENTS

Paul Boutros, Phd, PI - Boutros Lab

The Ontario Institute for Cancer Research

To install HPCI, copy and paste the appropriate command in to your terminal.

cpanm

cpanm HPCI

CPAN shell

perl -MCPAN -e shell
install HPCI

For more information on module installation, please visit the detailed CPAN module installation guide.

	Global
`s`	Focus search bar
`?`	Bring up this help dialog

	GitHub
`g` `p`	Go to pull requests
`g` `i`	go to github issues (only if github is preferred repository)

	POD
`g` `a`	Go to author
`g` `c`	Go to changes
`g` `i`	Go to issues
`g` `d`	Go to dist
`g` `r`	Go to repository/SCM
`g` `s`	Go to source
`g` `b`	Go to file browse

	Search terms
module: (e.g. module:Plugin)
distribution: (e.g. distribution:Dancer auth)
author: (e.g. author:SONGMU Redis)
version: (e.g. version:1.00)

NAME

VERSION

SYNOPSIS

OVERVIEW

INTERFACE

OUTPUT DIRECTORY LAYOUT

base_dir

group_dir

log

stage_dir

script_file

job_dir

stdout/stderr

final_retry

HPCI "Class"

METHOD group

Group Object

Cluster Attributes

cluster

Basic Attributes

name

Directory Layout Attributes

base_dir

group_dir

Logging Attributes

log

log_path

log_dir

log_file

log_level

Operational Attributes

max_concurrent

status

Environment Passing Attributes

Methods

stage

add_deps

execute

Stage Object

Attributes

name

command

dir

resources_required

retry_resources_required

failure_action ('abort_group', 'abort_deps'*, or 'ignore')

Methods

command creation

Env

Additional

SEE ALSO

AUTHOR

ACKNOWLEDGEMENTS

Module Install Instructions