/ 
EMDIS-ECS-0.41
River stage zero No dependents
/ EMDIS::ECS

NAME

EMDIS::ECS - ECS utility module

SYNOPSIS

use vars qw($ECS $ECS_CFG $ECS_NODE_TBL);
use EMDIS::ECS;
$err = EMDIS::ECS::load_config("ecs.cfg");
die "Unable to initialize ECS: $err\n" if $err;

ECS::log_error("This is an error.");

$err = EMDIS::ECS::send_admin_email("Something happened.\n",
    "Here are details.\n");
ECS::log_error("error sending admin email: $err") if $err;

$err = EMDIS::ECS::send_ecs_message('UX', '', "msg_type=READY\n",
    "# comment\n");
ECS::log_error("error sending ECS message: $err") if $err;

DESCRIPTION

This module contains a bunch of miscellaneous ECS related subroutines. However, most of the documentation found here pertains to the Perl ECS implementation in general, not those specific subroutines.

Introduction

This Perl implementation of the EMDIS Communication System (ECS), herein referred to as "Perl-ECS", is generally compatible with the ECS specification published by the ZKRD, though it differs from the specification in some of its implementation details. A PDF document containing the original ECS specification is available from the ZKRD web site (see http://www.zkrd.de/).

Getting Started

Before Perl-ECS can be used, a number of pre-requisites must be satisfied.

Install Perl-ECS

Install Perl, preferably version 5.6.1 or higher. Then install the EMDIS::ECS package. (Presumably already done if you're reading this documentation online.)

Email Account

Acquire an email account to be used by the ECS system. Perl-ECS uses SMTP to send outbound mail, so a SMTP server will need to be available for this purpose.

To read incoming email, Perl-ECS can use IMAP protocol, POP3 protocol, or a DIRECTORY method. If IMAP or POP3 protocol is used, that service will also need to be available.

Encryption Software

Install and configure PGP and/or GnuPG encryption software. Refer to http://www.pgp.com/, http://www.pgpi.org/, http://www.gnupg.org/, and http://www.philzimmermann.com/ for more information on the topic of PGP and related software.

GnuPG Version 2.2 - Additional Notes

The default OpenPGP configuration used by Perl-ECS is intended for use with GnuPG (gpg) versions 1.4 and 2.0. However, gpg version 2.2 is a standard component of newer Linux systems such as Ubuntu 18.

For systems using gpg version 2.2, configuration adjustments are needed in order to enable Perl-ECS to transmit the passphrase to gpg via stdin (pinentry-mode loopback).

1. Create or edit $GNUPGHOME/gpg-agent.conf, adding the line:

allow-loopback-pinentry

2. Execute the command:

gpg-connect-agent /bye

3. In the ecs.cfg configuration file, revise the OPENPGP_CMD_ENCRYPT and OPENPGP_CMD_DECRYPT settings to add the following. (If needed, first uncomment those settings.):

--pinentry-mode loopback

4. If upgrading from an earlier gpg version, use ecstool --tweak to modify all (addr_r) key IDs in the node table, because the IDs change when the keyring is converted to gpg 2.2.

AMQP Messaging

As an experimental new feature, version 0.41 added support for use of AMQP messaging as an alternative to email.

To use AMQP messaging, the ENABLE_AMQP setting must be set to YES or TRUE. AMQP communications utilize a mboxes/amqp_staging directory, which will need to be created manually, e.g.:

mkdir mboxes/amqp_staging

Additionally, the node table now accepts new AMQP-related settings. These can be added via the "ecstool --tweak" command, e.g.:

ecstool --tweak BB amqp_addr_meta emdis.bb.meta
ecstool --tweak BB amqp_addr_msg emdis.bb.msg

AMQP settings configured at the individual node level override equivalent global settings when communicating with that node. The presence of amqp_addr_meta and amqp_addr_msg in the node configuration, respectively, enable use of AMQP for transmission of META and regular EMDIS messages to that node (assuming ENABLE_AMQP is also set in ecs.cfg).

The node table also recognizes an amqp_only yes/no option. If enabled, the amqp_only option disables use of email when transmitting meta-messages, documents, or regular messages messages to that node.

Document Exchange

As an experimental new feature, version 0.41 added support for document exchange.

The ecs_scan_mail program, when processing files in the mboxes/to_dir/to_XX subdirectories, now looks for filenames with the suffix .doc or .doc.xml and sends those files as documents.

Similarly, the "ecstool --send" command now sends files with a .doc or .doc.xml suffix as documents, e.g. "ecstool --send EE test01.doc"

The ecs_scan_mail program copies documents received to the mboxes/from_dir/from_XX subdirectories, to a filename with a .doc suffix.

Document exchange uses a Subject header of the form EMDIS:AA:123:DOC to indicate the presence of a document and its sequence number. DOC_MSG_ACK and DOC_RE_SEND are meta messages used for document exchange.

Configuration

Once the above prerequisites are in place, it's time to configure your ECS system. Create a directory to hold the ECS data files and then run the ecs_setup program to help create a basic configuration file. The ECS configuration file can also be created and edited using a regular text editor.

NODE_TBL

The NODE_TBL used by Perl-ECS contains several additional fields not described in the ECS specification. See below for descriptions of NODE_TBL fields; the names of added fields are shown emphasized. The ecstool program provides commands to manipulate the NODE_TBL.

ack_seq

The highest sequence number acknowledged by this node.

addr

The email address of this node.

addr_r

The PGP or GnuPG userid of this ECS node.

amqp_addr_doc

Name of AMQP queue to use when sending ECS document messages to this node. (Note: document exchange is not yet implemented.)

amqp_addr_meta

Name of AMQP queue to use when sending ECS meta messages to this node.

amqp_addr_msg

Name of AMQP queue to use when sending regular ECS messages to this node.

amqp_broker_url

Node-specific AMQP_BROKER_URL configuration. See also EMDIS:ECS::Config documentation.

amqp_cmd_recv

Node-specific AMQP_CMD_RECV configuration. See also EMDIS:ECS::Config documentation.

amqp_cmd_send

Node-specific AMQP_CMD_SEND configuration. See also EMDIS:ECS::Config documentation.

amqp_debug_level

Node-specific AMQP_DEBUG_LEVEL configuration. See also EMDIS:ECS::Config documentation.

amqp_password

Node-specific AMQP_PASSWORD configuration. See also EMDIS:ECS::Config documentation.

amqp_recv_timeout

Node-specific AMQP_RECV_TIMEOUT configuration. See also EMDIS:ECS::Config documentation.

amqp_send_timelimit

Node-specific AMQP_SEND_TIMELIMIT configuration. See also EMDIS:ECS::Config documentation.

amqp_sslcert

Node-specific AMQP_SSLCERT configuration. See also EMDIS:ECS::Config documentation.

amqp_sslkey

Node-specific AMQP_SSLKEY configuration. See also EMDIS:ECS::Config documentation.

amqp_sslpass

Node-specific AMQP_SSLPASS configuration. See also EMDIS:ECS::Config documentation.

amqp_truststore

Node-specific AMQP_TRUSTSTORE configuration. See also EMDIS:ECS::Config documentation.

amqp_username

Node-specific AMQP_USERNAME configuration. See also EMDIS:ECS::Config documentation.

amqp_vhost

Node-specific AMQP_VHOST configuration. See also EMDIS:ECS::Config documentation.

contact

Email address of administrator for this node.

doc_in_seq

Sequence number assigned to the most recent document received from this node (intended for internal use only).

doc_out_seq

Sequence number assigned to the most recent document sent to this node (intended for internal use only).

encr_meta

Indicates whether meta-messages to/from this node should be encrypted ("true" or "false").

encr_out_keyid

ID of secret key to be used for encrypted messages to/from this node. For this node only, overrides the ECS configuration file's GPG_KEYID or PGP_KEYID.

encr_out_passphrase

Passphrase for encr_out_keyid. For this node only, overrides the ECS configuration file's GPG_PASSPHRASE or PGP_PASSPHRASE.

encr_sig

Identifies the PGP or GnuPG signature for this node.

encr_typ

The type of encryption used by this node. Currently supported encryption types are "PGP2", "PGP2-verify", "OpenPGP", and "OpenPGP-verify" ("verify" indicates messages from the node are cryptographically signed and the signature should be verified).

in_seq

The sequence number of the last message accepted from this node.

in_seq_ack

The sequence number of the last message from this node for which a MSG_ACK has been sent.

last_in

Date and time when the last message from this node was accepted.

last_in_adm

Date and time when local ECS administrator was notified of communication loss with this node.

last_out

Date and time when the last message from this node was received.

msg_part_size

Maximum message part size, in bytes, for this node. Defaults to MSG_PART_SIZE_DFLT from ECS config file if not specified or zero. Refer to the EMDISCORD email parts RFC (RFC-20091021-EmailParts.pdf) for additional information about message parts.

node

The ECS node name. (Primary key)

node_disabled

Indicates whether processing is currently disabled for this node (YES or NO).

out_seq

The sequence number of the last message sent to this node.

q_first_file

Name of first file in processing queue. The filename shows the date and time when the file was retrieved from the email inbox. This field is automatically updated by the ecs_scan_mail program, once per T_SCN interval.

q_gap_seq

Minimum message seq_num of "early" message in processing queue. This value is updated only when a gap in message seq_num values is encountered. During message processing, if the "early" message situation persists longer than the number of seconds specified by the T_RESEND_DELAY configuration parameter and q_gap_seq does not change during this time period, the ecs_scan_mail program will automatically generate a batch of RE_SEND requests.

q_gap_time

Date and time when q_gap_seq value was observed. This value is used to compute whether the T_RESEND_DELAY time interval has elapsed.

q_max_seq

Maximum message seq_num in processing queue. This field is automatically updated by the ecs_scan_mail program, once per T_SCN interval.

q_min_seq

Minimum message seq_num in processing queue. This field is automatically updated by the ecs_scan_mail program, once per T_SCN interval.

q_size

Number of messages in processing queue. This field is automatically updated by the ecs_scan_mail program, once per T_SCN interval.

proc_file

File containing message currently being processed. For THIS_NODE only, this field is automatically updated by the ecs_scan_mail program during message processing.

proc_node

Node id for message currently being processed. For THIS_NODE only, this field is automatically updated by the ecs_scan_mail program during message processing.

proc_seq

Sequence number of message currently being processed. For THIS_NODE only, this field is automatically updated by the ecs_scan_mail program during message processing.

ready_num_disabled

Indicates whether READY meta-messages sent to this node will include last_recv_num and last_sent_num values (YES or NO).

Special Features

A few notes regarding differences between Perl-ECS and the ECS specification.

Serialized Message Processing

Incoming messages are processed one at a time, with a processing time limit. Because of this, Perl-ECS is not susceptible to "fork bomb" problems that could occur when many messages are received in a short period of time.

No MSG_TBL

There is no MSG_TBL to track "early" messages. Instead, the ecs_scan_mail program performs a brute force analysis of the mboxes/store directory once during each T_SCN interval.

RE_SEND Protocol

If a gap in message seq_num values for a given node is encountered, and the lowest incoming message seq_num for that node has not changed for T_RESEND_DELAY seconds, the ecs_scan_mail program automatically issues a batch of up to 100 RE_SEND requests. Because missing email messages may indicate the existence of unusual problems, the ecs_scan_mail program also sends email to notify the ECS administrator when this happens.

Email Protocols

To send email, Perl-ECS requires a SMTP server. Likewise, to read email, it requires a POP3 or IMAP server. It does not use mailx or other system specific software.

Expanded NODE_TBL

The NODE_TBL contains several additional fields: encr_meta, encr_sig, encr_typ, last_in_adm. See the above NODE_TBL section for details.

mboxes/in_fml

Incoming FML messages are archived in the mboxes/in_fml subdirectory.

The ecstool Program

The ecstool program has been given additional capabilities. For details, refer to the ecstool documentation (e.g. "perldoc ecstool" or "man ecstool").

ECS Configuration File

A significant number of new settings have been added to the ECS configuration file. For details, refer to the EMDIS::ECS::Config documentation (e.g. "perldoc EMDIS::ECS::Config" or "man EMDIS::ECS::Config"). The ecs_setup program is designed to help in the creation of a basic ECS configuration file.

PID Files

The files in ECS_DAT_DIR that contain the PIDs for the ecs_chk_com and ecs_scan_mail daemons are ecs_chk_com.pid and ecs_scan_mail.pid, respectively.

Log Files

The ecs_chk_com and ecs_scan_mail daemons do not share common log and err files.

The ecs_off.lck File

The ECS daemons do not check for the presence of an ecs_off.lck file.

MSG_PROC

The ecs_scan_mail daemon sets ADAPTER_CMD and ECS_DRP_DIR environment variables during execution of the MSG_PROC command. The default MSG_PROC script provided with Perl-ECS ("ecs_proc_msg") executes the command specified by ADAPTER_CMD when processing a message. ECS_DRP_DIR specifies the location of the ECS_DAT_DIR/maildrop directory.

ECS_DAT_DIR/maildrop

The "maildrop" directory holds outbound messages generated by the MSG_PROC, ADAPTER_CMD, or "ecstool --maildrop" commands. During each T_SCN interval, after messages in the mboxes/store directory have been processed, FML files found in the maildrop directory are automatically sent to the appropriate destination, based on the values of the HUB_SND and HUB_RCV fields. (However, the maildrop FML parser is currently unable to process HUB_SND and HUB_RCV values formatted using /FIELDS or /CONST.)

ADM_ADDR

Any incoming non-ECS messages are passed to ADM_ADDR.

mboxes/active

There is no mboxes/active subdirectory. This directory is not needed when reading mail from a POP3 or IMAP inbox.

ecs, ecs_send_msg, ecs_resend_req

Scripts or programs named ecs, ecs_send_msg, and ecs_resend_req are not provided.

BUGS

Possibly.

SEE ALSO

EMDIS::ECS::Config, EMDIS::ECS::FileBackedMessage, EMDIS::ECS::LockedHash, EMDIS::ECS::Message, ecs_chk_com, ecs_proc_meta, ecs_proc_msg, ecs_scan_mail, ecs_setup, ecstool

AUTHOR

Neil Smeby <nsmeby@nmdp.org>

Joel Schneider <jschneid@nmdp.org> - modifications, refactoring, documentation, etc.

COPYRIGHT AND LICENSE

THIS PACKAGE IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF MERCHANTIBILITY AND FITNESS FOR A PARTICULAR PURPOSE.

Copyright (C) 2002-2020, National Marrow Donor Program. All rights reserved.

See LICENSE file for license details.

HISTORY

ECS, the EMDIS Communication System, was originally designed and implemented by the ZKRD (http://www.zkrd.de/). This Perl implementation of ECS was developed by the National Marrow Donor Program (http://www.marrow.org/).

2004-03-12 Canadian Blood Services - Tony Wai Added MS Windows support for Windows 2000 and Windows XP Added "DIRECTORY" inBox Protocol. This can interface with any mail system that can output the new messages to text files.

2007-08-01 ZKRD - emdisadm@zkrd.de Added new error report management using the new variable MAIL_LEVEL. All email to admin statements are removed. In relation to the error code ECS.pm will send an email to admin or not. Bugfix for the regular expression in sub read_ecs_message_id(): The regular expression now ignores spam tags in the subject line. Hold lock in send_ecs_message until msg. is successfully send.