NAME

MARC::Errorchecks

DESCRIPTION

Module for storing MARC error-checking subroutines, based on MARC21, AACR2R, and LCRIs. These are used to find errors not easily checked by the MARC::Lint and MARC::Lintadditions modules, such as those that cross field boundaries.

Each subroutine should generally be passed a MARC::Record object.

Returned warnings/errors are generated as follows: push @warningstoreturn, join '', ($field->tag(), ": [ERROR TEXT]\t"); return \@warningstoreturn;

SYNOPSIS

 use MARC::Batch;
 use MARC::Errorchecks;

 #See also MARC::Lintadditions for more checks
 #use MARC::Lintadditions;

 #change file names as desired
 my $inputfile = 'marcfile.mrc';
 my $errorfilename = 'errors.txt';
 my $errorcount = 0;
 open (OUT, ">$errorfilename");
 #initialize $infile as new MARC::Batch object
 my $batch = MARC::Batch->new('USMARC', "$inputfile");
 my $errorcount = 0;
 #loop through batch file of records
 while (my $record = $batch->next()) {
  #if $record->field('001') #add this if some records in file do not contain an '001' field
  my $controlno = $record->field('001')->as_string();   #call MARC::Errorchecks subroutines

  my @errorstoreturn = ();

  # check everything

  push @errorstoreturn, (@{MARC::Errorchecks::check_all_subs($record)});

  # or only a few
  push @errorstoreturn, (@{MARC::Errorchecks::check_010($record)});
  push @errorstoreturn, (@{MARC::Errorchecks::check_bk008_vs_bibrefandindex($record)});

  # report results
  if (@errorstoreturn){
   #########################################
   print OUT join( "\t", "$controlno", @errorstoreturn, "\t\n");

   $errorcount++;
  }

 } #while

TO DO

Maintain check-all subroutine, a wrapper that calls all the subroutines in Errorchecks, to simplify calling code in .pl.

Verify each of the codes in the data against current lists and lists of changes. Maintain code list data when future changes occur. Possibly move the code list data into a separate file (e.g., MARC::Errorchecks::CodeData)

Determine whether extra tabs are being added to warnings. Examine how warnings are returned and see if a better way is available.

Add functionality.

 -Ending punctuation (in Lintadditions.pm, and 300 dealt with here, and now 5xx (some)).
 -Matching brackets and parentheses in fields?
 -Geographical headings miscoded as subjects.
 
 Possibly rewrite as object-oriented?
 If not, optimize this and the Lintadditions.pm checks.
 Example: reduce number of repeated breaking-out of fields into subfield parts.
 So, subroutines that look for double spaces and double punctuation might be combined.

Deal with other TO DO items found below. This includes fixing problem of "bibliographical references" being required if 008 contents has 'b'.

check_all_subs

Calls each error-checking subroutine in Errorchecks. Gathers all errors and returns those errors in an array (reference).

TO DO (check_all_subs)

Make sure to update this subroutine as additional subroutines are added.

check_double_periods($record)

Looks for more than one period within subfields after 010. Exception: Exactly 3 periods together are treated as ellipses.

Looks for multiple commas.

TO DO (check_double_periods)

Find exceptions where double periods may be allowed. Find exceptions where more than 3 periods can be next to each other. Deal with the exceptions.

check_internal_spaces($record)

Looks for more than one space within subfields after 010. Ignores 035 field, since multiple spaces could be allowed.

TO DO (check_internal_spaces)

check_trailing_spaces($record)

Looks for extra spaces at the end of fields greater than 010. Ignores 016 extra space at end.

TO DO (check_trailing_spaces)

Rewrite to incorporate 010 and 016 space checking.

Consider allowing trailing spaces in 035 field.

check_008($record)

Code for validating 008s in MARC records. Validates each byte of the 008, based on MARC::Errorchecks::validate008($field008, $mattype, $biblvl)

TO DO (check_008)

Improve validate008 subroutine (see that sub for more information): -Revise error message reporting. -Break byte 18-34 checking into separate sub so it can be used for 006 validation as well. -Optimize efficiency.

check_010($record)

Verifies 010 subfield 'a' has proper spacing.

TO DO (check_010)

Think about whether subfield 'z' needs proper spacing.

Deal with non-digit characters in original 010a field. Currently these are simply reported and the space checking is skipped.

Maintain date ranges in checking validity of numbers.

Modify date ranges according to local catalog needs.

Determine whether this subroutine can be implemented in MARC::Lintadditions/Lint--I don't remember why it is here rather than there?

NAME

check_end_punct_300($record)

DESCRIPTION

Reports an error if an ending period in 300 is missing if 4xx exists, or if 300 ends with closing parens-period if 4xx does not exist.

NAME

check_bk008_vs_300($record)

DESCRIPTION

300 subfield 'b' vs. presence of coding for illustrations in 008/18-21.

Ignores CIP records completely. Ignores non-book records completely (for the purposes of this subroutine).

If 300 'b' has wording, reports errors if matching 008/18-21 coding is not present. If 008/18-21 coding is present, but similar wording is not present in 300, reports errors.

Note: plates are an exception, since they are noted in $a rather than $b of the 300. So, they need to be checked twice--once if 'f' is the only code in the 008/18-21, and again amongst other codes.

Also checks for 'p.' or 'v.' in subfield 'a'

LIMITATIONS

Only accounts for a single 300 field (300 was recently made repeatable).

Older/more specific code checking is limited due to lack of use (by our catalogers). For example, coats of arms, facsim., etc. are usually now given as just 'ill.' So the error check allows either the specific or just ill. for all except maps.

Depends upon 008 being coded for book monographs.

Subfield 'a' and 'c' wording checks ('p.' or 'v.'; 'cm.', 'in.', 'mm.') only look at first of each kind of subfield.

TO DO (check_bk008_vs_300($record))

Take care of case of 008 coded for serials/continuing resources.

Find exceptions to $a having 'p.' or 'v.' for books.

Find exceptions to $c having 'cm.', 'mm.', or 'in.' preceded by digits.

Deal with other LIMITATIONS.

Account for upcoming rule change in which metric units have no punctuation. When that rule goes into effect, move 300$c checking to check_end_punct_300($record).

Reverse checks to report missing 008 code if specific wording is present in 300.

Reverse check for plates vs. 'f'

NAME

 parse008vs300b($illcodes, $field300subb)
 

DESCRIPTION

008 illustration parse subroutine

checks 008/18-21 code against 300 $b

WHY?

To simplify the check_bk008_vs_300($record) subroutine, which had many if-then statements. This moves the additional checking conditionals out of the way. It may be integrated back into the main subroutine once it works. This was written while constructing check_bk008_vs_300($record) as a separate script.

Synopsis/Usage description

        parse008vs300b($illcodes, $field300subb)

 #$illcodes is bytes 18-21 of 008
 #$subfieldb is subfield 'b' of record's 300 field

TO DO (parse008vs300b($$))

Integrate code into check_bk008_vs_300($record)?

Verify possibilities for 300 text

Move 'm' next to 'f' since it is likely to be indicated in subfield 'e' not 'b' of the 300. Our catalogers do not generally code for sound recordings in this way in book records.

check_490vs8xx($record)

If 490 with 1st indicator '1' exists, then 8xx should exist.

check_240ind1vs1xx($record)

If 1xx exists then 240 1st indicator should be '1'. If 1xx does not exist then 240 should not be present.

However, exceptions to this rule are possible, so this should be considered an optional error.

check_245ind1vs1xx($record)

If 1xx exists then 245 1st indicator should be '1'. If 1xx does not exist then 245 1st indicator should be '0'.

However, exceptions to this rule are possible, so this should be considered an optional error.

matchpubdates($record)

Date matching 008, 050, 260

Attempts to match date of publication in 008 date1, 050 subfield 'b', and 260 subfield 'c'.

Reports errors when one of the fields does not match. Reports errors if one of the dates cannot be found

Handles cases where 050 or 260 (or 260c) does not exist. -Currently if the subroutine is unable to get either the date1, any 050 with $b, or a 260 with $c, it returns (exits). -Future, or better, behavior, might be to continue processing for the other fields.

Handles cases where 050 is different due to conference dates. Conference exception handling is currently limited to presence of 111 field or 110$d.

KNOWN PROBLEMS

May not deal well with serial records (problem not even approached).

Only examines 1st 260, does not account for more than one 260 (recent addition).

Relies upon 260$c date being the first date in the last 260$c subfield.

Has problem finding 050 date if it is not last set of digits in 050$b.

Process of getting 008date1 duplicates similar check in validate_008 subroutine.

TO DO

Improve Conference publication checking (limited to 111 field or 110$d being present for this version) This may include comparing 110$d or 111$d vs. 050, and then comparing 008date1 vs. 260$c.

Fix parsing for 050$bdate.

For CIP, if 260 does not exist, compare only 050 and 008date1. Currently, CIP records without 260 are skipped.

Account for undetermined dates, e.g. [19--?] in 260 and 008.

Account for older 050s with no date present.

check_bk008_vs_bibrefandindex($record)

 Ignores non-book records (other than cartographic materials).
 For cartographic materials, checks only for index coding (not bib. refs.).

 Examines 008 book-contents (bytes 24-27) and book-index (byte 31).
 Compares with 500 and 504 fields.
 Reports error if 008contents has 'b' but 504 does not have "bibliographical references."
 Reports error if 504 has "bibliographical references" but no 'b' in 008contents.
 Reports error if 008index has 1 but no 500 or 504 with "Includes .* index."
 Reports error if a 500 or 504 has "Includes .* index" but 008index is 0. 
 Reports error if "bibliographical references" appears in 500.
 Allows "bibliographical reference."

TO DO/KNOWN PROBLEMS

 As with other subroutines, this one treats all 008 as being coded for monographs.
 Serials are ignored for the moment.

 Account for records with "Bibliography" or other wording in place of "bibliographical references."
 Currently 'b' in 008 must match with "bibliographical reference" or "bibliographical references" in 504 (or 500--though that reports an error).

 Reverse check for other wording (or subject headings) vs. 008 'b' in contents.

 Check for other 008contents codes.

 Check for misspelled "bibliographical references."

 Check spacing if pagination is given in 504.

check_041vs008lang($record)

Compares first code in subfield 'a' of 041 vs. 008 bytes 35-37.

check_5xxendingpunctuation($record)

Validates punctuation in various 5xx fields.

Currently checks 500, 501, 504, 508, 511, 538, 546.

For 586, see check_nonpunctendingfields($record)

TO DO (check_5xxendingpunctuation)

Add checks for the other 5xx fields.

Verify rules for these checks.

findfloatinghypens($record)

Looks at various fields and reports fields with space-hypen-space as errors.

TO DO (findfloatinghypens($record))

Find exceptions.

video007vs300vs538($record)

Comparison of 007 coding vs. 300abc subfield data and vs. 538 data for video records (VHS and DVD).

DESCRIPTION

Focuses on videocassettes (VHS) and videodiscs (DVD and Video CD). Does not consider coding for motion pictures.

If LDR/06 is 'g' for projected medium, (skipping those that aren't) and 007 is present, at least 1 007 should start with 'v'

If 007/01 is 'd', 300a should have 'videodisc(s)'. 300c should have 4 3/4 in. Also, 538 should have 'DVD' If 007/01 is 'f', 300a should have 'videocassette(s)' 300c should have 1/2 in. Also, 538 should have 'VHS format' or 'VHS hi-fi format' (case insensitive on hi-fi), plus a playback mode.

LIMITATIONS

Checks only videocassettes (1/2) and videodiscs (4 3/4). Current version reports problems with other forms of videorecordings.

Accounts for existence of only 1 300 field.

Looks at only 1st subfield 'a' and 'c' of 1st 300 field.

TO DO

Account for motion pictures and videorecordings not on DVD (4 3/4 in.) or VHS cassettes.

Check proper plurality of 300a (1 videodiscs -> error; 5 videocassette -> error)

Monitor need for changes to sizes, particularly 4 3/4 in. DVDs.

Expand allowed terms for 538 as needed and revise current VHS allowed terms.

Update to allow SMDs of conventional terminology ('DVD') if such a rule passes.

Deal with multiple 300 fields.

Check GMD in 245$h

Clean up redundant code.

ldrvalidate($record)

Validates bytes 5, 6, 7, 17, and 18 of the leader against MARC code list valid characters.

DESCRIPTION

Checks bytes 5, 6, 7, 17, and 18.

$ldrbytes{$key} has keys "\d\d", "\d\dvalid" for each of the bytes checked (05, 06, 07, 17, 18)

"\d\dvalid" is a hash ref containing valid code linked to the meaning of that code.

print $ldrbytes{'05valid'}->{'a'}, "\n"; yields: 'Increase in encoding level'

TO DO (ldrvalidate)

Customize (comment or uncomment) bytes according to local needs.

Examine other Lintadditions/Errorchecks subroutines using the leader to see if duplicate checks are being done.

Move or remove such duplicate checks.

Consider whether %ldrbytes needs full text of meaning of each byte.

geogsubjvs043($record)

Reports absence of 043 if 651 or 6xx subfield z is present.

TO DO (geogsubjvs043)

Update/maintain list of exceptions (in the hash, %geog043exceptions).

findemptysubfields($record)

 Looks for empty subfields.
 Skips 037 in CIP-level records and tags < 010.

check_040present($record)

Reports error if 040 is not present. Can not use Lintadditions check_040 for this since that relies upon field existing before the check is executed.

check_nonpunctendingfields($record)

Checks for presence of punctuation in the fields listed below. These fields are not supposed to end in punctuation unless the data ends in abbreviation, ___, or punctuation.

Fields checked: 240, 246, 440, 490, 586.

TO DO (check_nonpunctendingfields)

Add exceptions--abbreviations--or deal with them. Currently all fields ending in period are reported.

check_fieldlength($record)

Reports error if field is longer than 1870 bytes. (1879 is actual limit, but I wanted to leave some extra room in case of miscalculation.)

This check relates to certain system limitations.

TO DO (check_fieldlength($record))

Use directory information in raw MARC to get the field lengths.

Add new subs with code below.

sub {

        #get passed MARC::Record object

        my $record = shift;

        #declaration of return array

        my @warningstoreturn = ();

        push @warningstoreturn, ("");

        return \@warningstoreturn;

} #

NAME

readcodedata() -- Read Country, Geographic Area Code, Language Data

DESCRIPTION

Subroutine for reading data to build an array of country codes, geographic area codes, and language codes, valid and obsolete, for use in validate008 (in MARC::Errorchecks) and 043 validation (in MARC::Lintadditions).

SYNOPSIS

 my @dataarray = MARC::Errorchecks::readcodedata();
## or 
 #MARC::Errorchecks::readcodedata();
 #my @countrycodes = split "\t", $MARC::Errorchecks::dataarray[1];
 
 my @countrycodes = split "\t", $dataarray[1];
 my @oldcountrycodes = split "\t", $dataarray[3];
 my @geogareacodes = split "\t", $dataarray[5];
 my @oldgeogareacodes = split "\t", $dataarray[7];
 my @languagecodes = split "\t", $dataarray[9];
 my @oldlanguagecodes = split "\t", $dataarray[11];

DATA Outline

 Data lines:
 0: __CountryCodes__
 1: countrycodes (tab-delimited)
 2: __ObsoleteCountry__
 3: oldcountrycodes (tab-delimited)
 4: __GeogAreaCodes__
 5: gacodes (tab-delimited)
 6: __ObsoleteGeogAreaCodes__
 7: oldgacodes (tab-delimited)
 8: __LanguageCodes__
 9: languagecodes (tab-delimited)
 10: __LanguageCodes__
 11: oldlanguagecodes (tab-delimited)

TO DO (readcodedata())

 Evaluate need for GeogAreaCodes in this module.
 These may not be needed, since 043 is validated in MARC::Lintadditions.

 Move this and the codes to separate data file?
 

NAME

parse008date($field008string)

DESCRIPTION

Subroutine parse008date returns four-digit year, two-digit month, and two-digit day. It requres an 008 string at least 6 bytes long.

SYNOPSIS

 my ($earlyyear, $earlymonth, $earlyday);
 print ("What is the earliest create date desired (008 date, in yymmdd)? ");
 while (my $earlydate = <>) {
 chomp $earlydate;
 my $field008 = $earlydate;
 my $yyyymmdderr = MARC::Errorchecks::parse008date($field008);
 my @parsed008date = split "\t", $yyyymmdderr;
 $earlyyear = shift @parsed008date;
 $earlymonth = shift @parsed008date;
 $earlyday = shift @parsed008date;
 my $errors = join "\t", @parsed008date;
 if ($errors) {
 if ($errors =~ /is too short/) {
 print "Please enter a longer date, $errors\nEnter date (yymmdd): ";
 }
 else {print "$errors\nEnter valid date (yymmdd): ";}
 } #if errors
 else {last;}
 }

validate008 ($field008, $mattype, $biblvl)

Checks the validity of 008 bytes.

DESCRIPTION

Checks the validity of 008 bytes. Depends upon 008 being based upon LDR/06, so continuing resources/serials records may not work. Check LDR/07 for 's' for serials

Returns hash with named 008 positions, cleaned 008 array, and $hasbadchars string, with tab-separated errors.

OTHER INFO

Character positions 00-17 and 35-39 are defined the same across all types of material, with special consideration for position 06.

Steps in validation code for format specific positions:

 1. add hash key and value pair for byte position(s)
 2. verify each byte against list of valid codes
 3. add valid characters as individual positions of cleaned array
 4. add any error to scalar containing tabbed errors

Synopsis

 use MARC::Record;
 use MARC::Errorchecks;

 #$mattype and $biblvl are from LDR/06 and LDR/07
 #my $mattype = substr($leader, 6, 1); 
 #my $biblvl = substr($leader, 7, 1);
 #my $field008 = $record->field('008')->as_string();
 my $field008 = '000101s20002000nyu                 eng d';
 my ($validatedhashref, $cleaned008ref, $badcharsref) =  MARC::Errorchecks::validate008($field008, $mattype, $biblvl);
 my %validatedhash = %$validatedhashref;
 my @cleaned008arr = @$cleaned008ref;
 my $badchars = $$badcharsref;
 foreach my $key (sort keys %validatedhash) {
 print "$key => $validatedhash{$key}\n";
 }
 print join ('', @cleaned008arr, "\n");
 print "$badchars\n";

 print $field008hash{pubctry};

TO DO (validate008)

 Add requirement that 40 char string needs to be passed in.
 Add error checking for less than 40 char string.
 --Partially done--Less than 40 characters leads to error.
 Verify datetypes that allow multiple dates.
 Deal with problem of Serials/Continuing resources--
 currently seriality is checked late in process, 
 so any records with serial 008 might report unnecessary errors.
 Determine whether it might be better to add invalid warnings to array rather than scalar string.
 Reconsider what the subroutine returns.

 Separate byte 18-34 checking so the same code can be used for 006 byte checking.

TEST CODE

 #test code
 sub validate008;
 my $leader = '00050nam';
 my $field008 = '000101s20002000nyu                 eng d';
 my $mattype = substr($leader, 6, 1); 
 my $biblvl = substr($leader, 7, 1);

 print "$field008\n";
 my ($validatedhashref, $cleaned008ref, $badcharsref) = validate008($field008, $mattype, $biblvl);
 my %validatedhash = %$validatedhashref;
 my @cleaned008arr = @$cleaned008ref;
 my $badchars = $$badcharsref;
 foreach my $key (sort keys %validatedhash) {
 print "$key => $validatedhash{$key}\n";
 }
 print join ('', @cleaned008arr, "\n");
 print "$badchars\n";

CHANGES/VERSION HISTORY

Version 1.03: Updated Aug. 30-Oct. 16, 2004. Released Oct. 17. First CPAN version.

 -Moved subs to MARC::QBIerrorchecks
 --check_003($record)
 --check_CIP_for_stockno($record)
 --check_082count($record)
 -Fixed bug in check_5xxendingpunctuation for first 10 characters.
 -Moved validate008() and parse008date() from MARC::BBMARC (to make MARC::Errorchecks more self-contained).
 -Moved readcodedata() from BBMARC (used by validate008)
 -Moved DATA from MARC::BBMARC for use in readcodedata() 
 -Remove dependency on MARC::BBMARC
 -Added duplicate comma check in check_double_periods($record)
 -Misc. bug fixes
 Planned (future versions):
 -Account for undetermined dates in matchpubdates($record).
 -Cleanup of validate008
 --Standardization of error reporting
 --Material specific byte checking (bytes 18-34) abstracted to allow 006 validation.
  

Version 1.02: Updated Aug. 11-22, 2004. Released Aug. 22, 2004.

 -Implemented VERSION (uncommented)
 -Added check for presence of 040 (check_040present($record)).
 -Added check for presence of 2 082s in full-level, 1 082 in CIP-level records (check_082count($record)).
 -Added temporary (test) check for trailing punctuation in 240, 586, 440, 490, 246 (check_nonpunctendingfields($record))
 --which should not end in punctuation except when the data ends in such.
 -Added check_fieldlength($record) to report fields longer than 1870 bytes.
 --This should be rewritten to use the length in the directory of the raw MARC.
 -Fixed workaround in check_bk008_vs_bibrefandindex($record) (Thanks again to Rich Ackerman).
 

Version 1.01: Updated July 20-Aug. 7, 2004. Released Aug. 8, 2004.

 -Temporary (or not) workaround for check_bk008_vs_bibrefandindex($record) and bibliographies.
 -Removed variables from some error messages and cleanup of messages.
 -Code readability cleanup.
 -Added subroutines:
 --check_240ind1vs1xx($record)
 --check_041vs008lang($record)
 --check_5xxendingpunctuation($record)
 --findfloatinghypens($record)
 --video007vs300vs538($record)
 --ldrvalidate($record)
 --geogsubjvs043($record)
 ---has list of exceptions (e.g. English-speaking countries)
 --findemptysubfields($record)
 -Changed subroutines:
 --check_bk008_vs_300($record): 
 ---added cross-checking for codes a, b, c, g (ill., map(s), port(s)., music)
 ---added checking for 'p. ' or 'v. ' or 'leaves ' in subfield 'a'
 ---added checking for 'cm.', 'mm.', 'in.' in subfield 'c'
 --parse008vs300b
 ---revised check for 'm', phono. (which our catalogers don't currently use)
 --Added check in check_bk008_vs_bibrefandindex($record) for 'Includes index.' (or indexes) in 504
 ---This has a workaround I would like to figure out how to fix
 

Version 1.00 (update to 0.95): First release July 18, 2004.

 -Fixed bugs causing check_003 and check_010 subroutines to fail (Thanks to Rich Ackerman)
 -Added to documentation
 -Misc. cleanup
 -Added skip of 787 fields to check_internal_spaces
 -Added subroutines:
 --check_end_punct_300($record)
 --check_bk008_vs_300($record)
 ---parse008vs300b
 --check_490vs8xx($record)
 --check_245ind1vs1xx($record)
 --matchpubdates($record)
 --check_bk008_vs_bibrefandindex($record)

Version 1 (original version (actually version 0.95)): First release, June 22, 2004

SEE ALSO

MARC::Record -- Required for this module to work.

MARC::Lint -- In the MARC::Record distribution and basis for this module.

MARC::Lintadditons -- Extension of MARC::Lint for checks involving individual tags. (vs. cross-field checking covered in this module). Available at http://home.inwave.com/eija (and may be merged into MARC::Lint).

MARC pages at the Library of Congress (http://www.loc.gov/marc)

Anglo-American Cataloging Rules, 2nd ed., 2002 revision, plus updates.

Library of Congress Rule Interpretations to AACR2R.

MARC Report (http://www.marcofquality.com) -- More full-featured commercial program for validating MARC records.

LICENSE

This code may be distributed under the same terms as Perl itself.

Please note that this module is not a product of or supported by the employers of the various contributors to the code.

AUTHOR

Bryan Baldus eijabb@cpan.org

Copyright (c) 2003-2004

cpanm MARC::Errorchecks

perl -MCPAN -e shell
install MARC::Errorchecks