NAME
Release::Checklist - A QA checklist for CPAN releases
What follows is a list of categories or subjects that touch the quality - or what an end-user percieves as such - of a distribution end might need some attention.
This document is aiming at up-river authors that want/need improving the release status of their distribution.
Work is underway in merging this list with The Berlin Consensus into a toolset and/or service.
Some of those can be (semi-)automated (in tests), others need action from the maintainer. Not all subjects may apply to your project.
Each subject will mention or list modules that might help in improving or preserving high standards.
Various guidelines are also available in the perl core documentation and the basics for preparing a (new) module for distribution can be found in perlnewmod.
$ perldoc perlmodstyleTest
Test, test and test. The more you test, the lower the chance you will break your code with small changes.
use strict;
use warnings;
use Test::More;
:
done_testing ();Separate your module tests and your author tests. This will lower the number of dependencies. Check your Pod syntax, documentation coverage, spelling, and minimum perl version requirements in xt/. Prevent declaring dependencies that are only used inxt/. There is no need to check prerequisites when the distribution runs its tests in user-space.
t/
xt/If possible, do not use Test::* modules that you do not actually require, however fancy they may be. See the point about dependencies.
If you are still using any additional Test:: module, do not mix your own code with the functionality from a/the module. Be consistent: use all or use nothing. That is: if the module you (now) use comes with features you had scripted yourself before using that module, replace them too.
If adding tests after a bug-fix, add at least two tests: one that tests that the required (fixed) behavior now passes and that the invalid behavior fails.
Check to see if your tests support running in parallel
$ prove -vwb -j8If you have Test2::Harness installed, also test with yath
$ yath
$ yath -j8Add tests cases from issues or tickets to your own tests. Adding references to the tickets or issues creates a self-documenting structure, reasoning and history.
Dependencies
Every module you use is a module your release will depend on. If a new release of such module fails, the likelihood of your release being unable to install increases. So only use modules that you need. Each dependency should be well-considered.
Depending on modules that are included in the perl5 CORE distribution do not have this problem, as you are sure that this module is available already. However, be careful to ensure that that module is a core module in the lowest version of perl that you claim to support. Check it with Module::CoreList
$ corelist File::Temp
Data for 2018-04-20
File::Temp was first released with perl v5.6.1Then there are two types of modules you can depend on: functional modules, like DBI and XML::LibXML, and developer convenience modules, like Modern::Perl
You cannot get around needing the first type of modules, but the convenience modules should only be used in (local) perl scripts and not in CPAN modules, so please do not add additional dependencies on modules/pragma's like sanity, Modern::Perl, common::sense, exact, or nonsense.
However useful they might be in your own working environment and force you into behaving well, adding them as a requirement to a CPAN module will increase the complexity of the requirements to probably no good use, as they are unlikely to be found on all your targeted systems and add a chance to break.
There is no problem with you using those in your own (non-CPAN) scripts and modules, but please do not add needless dependencies.
Test::Prereq is a fine module to test your dependencies, but please do not ship it in a test-file on t/.
Finally, make sure all your dependencies are declared in your META structure, so all CPAN clients know what to do before your module can be tested. META files are preferably not created manually, but generated by authoring tools like ExtUtils::MakeMaker, Dist::Zilla, or App::ModuleBuildTiny etc.
"prereqs" : {
"build" : {
"requires" : {
"ExtUtils::MakeMaker" : "0"
}
},
"configure" : {
"requires" : {
"ExtUtils::MakeMaker" : "0"
}
},
"runtime" : {
"requires" : {
"perl" : "5.006",
"Test::More" : "0.88"
}
}
}Legal issues
Be very careful in choosing dependencies that have a different license than your own distribution. However useful a module might be, if it has a more restrictive license than your distribution, it may be off limits to some users. That would mean that your distribution cannot be used either.
The reverse is also true: if your release has a more restrictive license than most releases on CPAN, it may lead to others avoiding depending on your code.
The perl/perl_5 licence is a reasonable default if you do not have a preferred license.
Documentation
Make sure that you have a clear, concise, and short SYNOPSIS section. This section should show the most important code as simple and clear as possible. If you have 3500 methods in your class, do not list all of the there. Just show how to create the object and show the 4 methods that a beginner would use. Note that the user that reads the manual will appreciate complete, correct, and up-to-date documentation per method more than a complete list of available methods in the SYNOPSIS.
Make sure your documentation covers all your methods and/or functions and all edge cases are documented. If you have private functions, mentions that in the documentation, so users can read that they might disappear without warning. A special section about errors, error messages, and/or exceptions is also a very nice thing to have.
Make sure your pod is correct and can also be parsed by the pod-modules in the lowest version of perl you support (or mention that you need at least version whatever to read the pod as intended).
The web is dynamic and some (external) references might cease to exist not telling you, so checking links used in your documentation once in a while might hint you to updated sites.
Neutrality
Do not use offensive or political statements anywhere in your code, your documentation, or your changelog(s). No one gains from statements about certain operating system(s), editor(s), president(s) or whatever are not to your liking. Make neutral claims about not being able to test your module on system X instead of saying system X should die a horrible death.
Do not use profane language. Do not promote any deity. A religion to one might be an insult to others. Keep neutral.
Spelling
Not every developer is of native English tongue. And even if, they also make (spelling) mistakes. There are enough tools available to prevent public display of misspellings and typos. Use them.
It is a good plan to have someone else proofread your documentation. If you can, ask three readers: one who knows about what the module is about, one who can be seen as an end-user of this modules without any knowledge about the internals, and last someone who has no clue about programming. You might be surprised of what they will find in the documentation as weird, unclear, or even plain wrong.
- - pod-spell-check
- - Pod::Spelling::Aspell
- - Pod::Escapes
- - Pod::Parser
- - Pod::Spell
- - Pod::Spell::CommonMistakes
- - Pod::Wordlist
- - Test::Synopsis
- - Text::Aspell
- - Text::Ispell
- - Text::Wrap
Examples
Have examples of your code. Preferably both in the EXAMPLES section of the pod, as in a folder named examples.
It is good practice to use your example code/scripts in your documentation too, as that gives you a two-way check (additional to your tests). Even better if the test scripts can be used as examples.
Test coverage
Do not just test what you think would be used. There will be users that try to bend the rules and invent ways for your module to be useful that you would never think of.
If every line of your code is tested, not only do you prevent unexpected breakage, but you also make sure that most corner cases are tested. Besides that, it will probably confront you with questions like "What can I possibly do to get into this part of my code?", Which may lead to optimizations and other fun.
Remove dead code. It is in your (git) repository anyway.
Version coverage
This is a hard one. If your release/dist requires specific versions of other modules, try to create an environment where you test your distribution against the required version and a version that does not meet the minimum version.
If your module requires Foo::Bar-0.123 because it supports correct UTF-8 encoding/decoding, and you wrote a test for that, your release is apt to fail in an environment where Foo::Bar-0.023 is installed.
This gets really hard to set up if your release has different code for versions of perl and for versions of required modules, but it pays off eventually. Note that monitoring CPANTESTERS can be a huge help.
If your code resides on GitHub, you can setup hooks to Travis CI. Just compose a .travis.yml and enable the hook. This supports a variety of perl versions and an environment where you can install modules for just the tests you need.
Minimal perl support
Your Makefile.PL (or whatever initial file the build system you use requires) will have to state a minimal supported perl version that ends up in META.json and META.yml
Do not guess. It is easy to check with
- - Test::MinimumVersion
- - Test::MinimumVersion::Fast.
- - Perl::MinimumVersion comes with the perlver tool:
$ perlver --blame test.pl
--------------------------------------------------
File : test.pl
Line : 3
Char : 14
Rule : _perl_5010_operators
Version : 5.010
--------------------------------------------------
//
--------------------------------------------------Multiple perl versions
If you host your perl code on github, you can integrate the services of Travis. Read their instructions on how to set up for multiple perl versions and or (environment) options.
If you have multiple perls installed on your system, test your module or release with all of the versions that you claim to support before doing the release. Best would be to test with a threaded perl and a non-threaded perl. If you can test with a mixture of -Duselongdouble and 32bit/64bit perls, that would be even better.
$ perl -wc lib/Foo/Bar.pmRepeat this on as many architectures as you can (i586, x64, IA64, PA-RISC, Sparc, PowerPC, …)
Repeat this on as many Operating Systems as you can (Linux, NetBSD, OSX, HP-UX, Solaris, Windows, OpenVMS, AIX, …)
Testing against a -Duselongdouble compiled perl will surface bad tests, e.g. tests that match against NVs like 2.1:
use Test::More;
my $i = 21000000000000001;
$i /= 10e15;
is ($i, 2.1);
done_testing;with -Uuselongdouble:
ok 1
1..1with -Duselongdouble:
not ok 1
# Failed test at -e line 1.
# got: '2.1000000000000001'
# expected: '2.1'
1..1
# Looks like you failed 1 test of 1.will show that 2.25 is probably a better choice that 2.1.
XS
If you use XS, make sure you (try to) support the widest range of perl versions. As using XS is quite often using more delicate areas of perl and perl internals, it is especially important to test for both threaded and unthreaded perl and - if supported - on operating systems that have different linking procedures than Linux. AIX and Windows are known to show deficiencies early.
- - Devel::PPPort (most recent version)
Leak tests
Your code allocates memory. Not only for the code itself, but also for all data-structures, a stack and other resources (modules you use or require).
Creating circular references or (in XS) variables that do not get freed will cause leaks. You might not notice in your tests, but if a long running process hits leaks and crashes with out-of-memory after 4 days, that is a problem. Tracing memory leaks might be hard, but some help is available
If you have a perl available with ASAN (Address Sanitizer) enabled, your may find corruptions during compilation.
Release tests
Some see CPANTS as a game, but many of the tests it puts on your release have a reason. Before you upload, you can check most of that to prevent unhappy users.
- - Test::Package - planned
- - Test::Kwalitee
- - Module::CPANTS::Analyse
- - cpants_lint.pl
$ perl Makefile.PL
$ make
$ make test
$ make dist
$ cpants_lint.pl Foo-Bar-0.01.tar.gz
Checked dist: Foo-Bar-0.01.tar.gz
Score: 144.44% (26/18)
Congratulations for building a 'perfect' distribution!
$Other that doing all these tests before you upload to CPAN, there are some awesome metrics available on cpants that trigger after you have uploaded to CPAN. This also shows more metrics and has lots of good documentation for the kwalitee indicators.
Clean dist
Some problems only surface when you do a make clean or make distclean. The develop cycle normally only adds and changes files, and if you forget to add those to the MANIFEST, your distribution will be incomplete and is likely to fail on other systems, whereas your tests locally still keep passing.
Check MANIFEST and MANIFEST.skip are complete.
$ make dist
$ make distclean$ perl -MTest::DistManifest -we'manifest_ok'
1..4
ok 1 - Parse MANIFEST or equivalent
ok 2 - All files are listed in MANIFEST or skipped
ok 3 - All files listed in MANIFEST exist on disk
ok 4 - No files are in both MANIFEST and MANIFEST.SKIPFile naming
Do not make it hard for people and modules to use the files in your distribution by using spaces and special characters in file names. Also try to not use overly long names and deep directory structures.
If spaces or weird characters in file or folder names are required by your module and/or your test cases, consider creating these files from the test itself and cleaning them up when the test is done.
$ perl -MTest::More -MTest::Portability::Files -w \
-e 'options (all_tests => 1);' \
-e 'options (test_dos_length => 0);' \
-e 'run_tests ();'Code style consistency
Add a CONTRIBUTING.md or similar file to guide others to consistency that will match your style (or, in case of joint effort, the style as agreed upon by the developers).
There are helper modules to enforce a style (given a configuration) or try to help contributors to come up with a path/change than matches the project's style and layout. Again: consistency helps. A lot.
- - Perl::Tidy
- - Perl::Critic + plugins, lot of choices
- - Test::Perl::Critic
- - Test::Perl::Critic::Policy
- - Test::TrailingSpace
- - Perl::Lint
- - .perltidy
- - .perlcritic
META
Make sure your meta-data matches the expected requirements. That can be achieved by using a generator that produces conform the most recent specifications or by using tools to check handcrafted META-files against the META spec 1.4 (2008) or META spec 2.0 (2011):
- - CPAN::Meta::Converter
- - CPAN::Meta::Validator
- - JSON::PP
- - Parse::CPAN::Meta
- - Test::CPAN::Meta::JSON
- - Test::CPAN::Meta::YAML
- - Test::CPAN::Meta::YAML::Version
- - YAML::Syck
It is highly appreciated if you declare resources, like your public repository URL and the preferred way to communicate in your META.json:
"resources" : {
"x_IRC" : "irc://irc.perl.org/#toolchain",
"repository" : {
"type" : "git",
"url" : "https://github.com/Tux/Release-Checklist",
"web" : "https://github.com/Tux/Release-Checklist"
},
"bugtracker" : {
"web" : "https://github.com/Tux/Release-Checklist/issues"
}
}Those are recognized and shown in the top-left section on meta::cpan.
Versions
Use a sane versioning system that the rest of the world might understand. Do not use the MD5 of the current date and time related to the phase of the moon or versions that include quotes or spaces. Keep it simple and clear.
Make sure it is a versioning system that increments
Changes
Make sure your Changes or ChangeLog file is up-to-date. Your release procedure might check the most recent mentioned date in that. At least every release should have an entry. Put the most recent change(s) on top (sort in reverse date order), so people can easily spot the most recent change(s).
Performance
Check if your release matches previous performance (if appropriate)
- - between different versions of perl
- - between different versions of the module
- - between different versions of dependencies
License
Make a clear statement about your license. (or choose a default, but at least state it).
Some target areas require a license in order to allow a CPAN module to be installed.
README / README.md
Add a file that states the purpose of your distribution.
The README should state the purpose, the minimal envirenment to test, build, and run and possible license issue. If there is a need to amend or create configuration files or set up databases, that should be mentioned too.
Tickets
If your module is on github or similar, regularly check on submitted issues and deal with them: either explain why it is not an issue or comment with alternatives or even better: this has been fixed now. Additionally, you should check on possible forks of your code and check the commits in that fork. There are people that make very interesting changes that you could use to improve your code. If the changes they made are already taken into your own code, or your code offers a (better) alternative to their change you could comment on their commits with additional information.
Likewise if you use RT as bug tracker.
If any ticket is entered as a vulnerability issue, be sure to make clear in the feedback to that issue that it is indeed a security issue and how you dealt with it or that you do/did not acknowledge this as a vulnerability, but as a (plain/simple) bug and deal with at with the appropriate actions.
Whatever you choose, make it clear in your META information, so the link to issues on metacpan points to the correct location.
META.yml================================================================================ resources: bugtracker: https://github.com/Tux/Release-Checklist/issues
META.json================================================================================ "resources" : { "bugtracker" : { "web" : "https://github.com/Tux/Release-Checklist/issues" } }
Downriver
You have had reasons to make the changes leading up to a new distribution. If you really care about the users of your module, you should check if your new release would break any of the CPAN modules that (indirectly) depend on your module by testing with your previous release and your upcoming release and see if the new release would cause the other module(s) to break.
used_by.pl will check the depending modules with the upcoming version.
Of course it is impossible to cover every possible situation here. The DarkPAN (uses of your module beyond what is registered on CPAN) is huge.
LICENSE
Copyright (C) 2015-2021 H.Merijn Brand. All rights reserved.
This library is free software; you can redistribute and/or modify it under the same terms as Perl itself.