Revision history for MARC::Charset
1.35 Tue Aug 13 19:50:55 PDT 2013
- improve conversion of certain composed characters to MARC8
Some characters should not be fully decomposed
before converting them to MARC8. This patch adds
a table of such characters, based on Annex A of
http://www.loc.gov/marc/marbi/2006/2006-04.html
and on some sample records provided by Jason
Stephenson of MVLC.
- recognize G0 and G1 characters properly
When converting from MARC8 to UTF8, MARC::Charset now
properly recognizes if a (single-byte) MARC8 character falls
in G0 or G1.
This is part of the fix for RT#63271 (converting characters
in the Extended Cyrillic character set), but should also
fix similar issues with converting characters in the extended
Arabic set.
This commit also means that all MARC8 character sets that support
both G0 and G1 wll be properly converted, regardless of whether
they're currently set as the G0 or G1 character set. For example,
it is now possible to convert Extended Latin as G0 or Basic Latin
as G1.
This fixes RT#63271
- have MARC::Charset::Code->marc_value() handle G0/G1 conversion
Since there's at present no need to do things like have
ANSEL be the G0 character set when converting from UTF8 to
MARC8, this commit centralizes the logic for deciding
whether to return the G0 or G1 MARC8 representation of a
character.
Also add MARC::Charset::Code->g0_marc_value(), which returns
the G0 representation of the character for use by the
character DB.
- New test cases for converting Vietnamese and Extended Cyrillic
text.
1.34 Mon Feb 11 09:10:35 PST 2013
- RT#83257: use AnyDBM_File rather than hardcode GDBM_File
To improve portability, use AnyDBM_File to select a DBM
rather than rely on GDBM_File. GDBM_File apparently used
to be a core module, but not all distributions included it,
particularly OS X. In any event, GDBM_File is no longer
core.
This patch also includes a tweak to allow MARC::Charset to
work with NDBM_File and ODBM_File, neither of which
support 'exists'.
I've tested MARC::Charset successfully on the following
DBMs:
- GDBM_File
- DB_File
- NDBM_File
- ODBM_File
- SDBM_File
This is also my preferred order; SDBM_File is selected last
because it produces the biggest data file on disk.
- RT#38912: fix mapping of double diacritics (ligature and double
tilde).
Thanks to Thomas P. Ventimiglia for the bug report and test case.
1.33 Thu Aug 4 23:25:14 EDT 2011
- move build_db() to separate .PL script so that module can be
built even if Class::Accessor and other dependencies aren't
available before Makefile.PL is run.
- list GDBM_File as an explicit dependency, as some distributions
like ActivePerl don't include it even though it is a
core module.
1.32 Thu Jun 30 16:38:32 EDT 2011
- make sure utf8 flag set in output of marc8_to_utf8
1.31 Thu Sep 30 10:53:00 EDT 2010
- minor revision to get v1.3 Changes into the CPAN distro :-)
1.3 Wed Sep 29 10:26:49 EDT 2010
- added latest codetables.xml from
http://www.loc.gov/marc/specifications/codetables.xml (thanks to
Mark Muehlhaeusler for noticing that there were some Arabic updates.
I reapplied the changes that François Charette suggested in v0.98
which are still not present in LC's codetable :-(
1.2 Tue May 11 22:21:13 EDT 2010
- use Storable::nfreeze instead of Storable::freeze to get a more
portable character set database. Reported and fixed by Niko Tyni
of debian http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=579517
1.1 Tue Jun 30 15:51:16 EDT 2009
- Patch from Damyan Ivanov of the Debian Perl Group to trim the size
of the character mapping db by 70 times! Uses GDBM_File instead of
SDBM_File.
1.00 Tue May 27 08:30:56 EDT 2008
- Forgot to add etc/additional-iii-characters.xml to the MANIFEST
- seems like as good a time as any for a v1.0 release :-)
0.99 Sun May 25 08:08:05 EDT 2008
- Addition of characters used by III ILSs which are not covered
by the official LoC codetables. Thanks go to Galen Charlton.
- Removed PREREQ_FATAL from Makefile.PL to make CPAN testers happy
0.98 Tue Aug 7 08:28:24 EDT 2007
- addition of two code elements to etc/codetables.xml that enable
the conversion of some Arabic records that contain 0x8D and 0x8E
which ought to map to 0x200D and 0x200C in Unicode. These mappings
are present for Basic and Extended Latin, but are not present
in Arabic codetables. There are actually some records that seem
to prove the need for these rules (LCCN 2006552991). Thanks to
François Charette <fcharette@ankabut.net> for finding and proposing
the fix. Rules were forwarded on to LC for inclusion in canonical
character set mapping.
- added t/farsi.t and t/farsi.marc to enable testing of new
code rules. Hopefully this will fail if the codetables.xml is
inadvertently removed without LC having added the new rules.
0.97 Sun May 20 13:48:31 EDT 2007
- added t/null.t
- fixed Charset::Compiler to use the <alt> element when <ucs> is not
defined. Previous versions of MARC::Charset would convert valid MARC8
to null when it encountered a mapping that lacked a UCS value
many thanks to Michael O'Connor.
- allow carriage return and line feeds to pass unmolesteed, much the
same as spaces today. Apparently, UNIMARC records embed these
formatting characters on a regular basis.
0.96 Wed Mar 14 01:24:48 EDT 2007
- added ignore_errors() to skip MARC8 -> UTF8 snafus
- added assume_encoding() to treat transcoding failures as if they
are from a known, specific encoding. Useful if you have a set of
records that, for instance, report being MARC8 but are actually
encoded in Latin1 (which, btw, is completely invalid and also very
common). Only in effect when ignore_errors() is true.
- added assume_unicode() to treat invalid MARC8 as UTF8. This is a
convenience function based on assume_encoding().
0.92 Sat Feb 4 19:34:19 CST 2006
- marc8_to_utf8 and utf8_to_marc8 needed to pass along spaces
without translation
- added tests to t/escape2.t and t/utf8.t to test space behavior
0.91 Fri Feb 3 23:10:59 EST 2006
- fix in marc8_to_utf8 for error reporting when no mapping is found
0.9 Fri Feb 3 22:25:39 EST 2006
- the utf8->marc8 will prefer the first mapping it runs across in the
LoC XML mapping table. v0.8 preferred the last mapping found which
meant that utf8_to_marc8 would escape to non-ascii character sets
for some punctuation. Thanks Mike Rylander for helping isolate
this problem.
- added a test that makes sure punctuation is working properly
to no_escape.t
- modified test of multiple combining characters in utf8.t to
actually test for correct result
0.8 Tue Dec 6 07:10:19 CST 2005
- complete overhaul to make MARC::Charset use LoC XML mapping table.
0.7 Wed Sep 7 21:34:18 2005
- pod fixes
0.6 Thu Feb 26 10:26:22 2004
- fixed MARC::Charset::EastAsian to not hexify results of character lookup
since we are now storing hex values in the BerkeleyDB.
- also fixed the method for looking up the location of the BerkeleyDB
so that the testing version takes precedence over one that is
installed. This is why the above error was not detected during
testing.
0.5 Fri Apr 11 06:47:00 2003
- all Charset classes inherit from MARC::Charset::Generic
- added MARC::Charset::UTF8
- added MARC::Charset::to_marc8() for conversion of UTF8 back to MARC8
- t/115.utf8.t basic tests of to_marc8()
- modified Makefile.PL to create a reverse mapping database for mapping
UTF8 characters back to their MARC8 equivalent.
0.3 Tue Dec 3 17:09:23 2002
- revamped to_utf8() to handle multibyte character sets. It is no longer
recursive, and didn't really need to be in the first place.
- created MARC::Charset::EastAsian!
0.2 Sat Oct 19 03:24:22 2002
- Added the 'Final character' to identify Extended Latin.
0.1 Fri Jul 26 09:10:36 2002
- Original version