NAME

App::Fetchware - App::Fetchware is Fetchware's API used to make extensions.

VERSION

version 1.016

SYNOPSIS

### App::Fetchware's use inside a Fetchwarefile.
### See fetchware's new command for an easy way to create Fetchwarefiles.
use App::Fetchware;

# Only program, lookup_url, one or more mirrors, and some method of
# verification are required.
program 'Your program';
lookup_url 'http://whatevermirror.your/program/is/on';
gpg_keys_url 'http://whatevermirror.your/program/gpg/key/url.asc';
mirror 'http://whatevermirror1.your/program/is/on';
mirror 'http://whatevermirror2.your/program/is/on';
mirror 'http://whatevermirror3.your/program/is/on';
mirror 'http://whatevermirror4.your/program/is/on';
mirror 'http://whatevermirror5.your/program/is/on';

# filter is not required, but is often needed to tell fetchware which
# program in the lookup_url directory or what specific version you would
# want to install. For example, Apache maintains 3 versions 2.0, 2.2, and
# 2.4. filter is what allows you to select which version you want fetchware
# to use.
filter 'version-2.0';

# Below are some popular options that may interest you.
make_options '-j 4';

### This is how Fetchwarefile's can replace lookup()'s or any other
### App::Fetchware API subroutine's default behavior.
### Remember your coderef must take the same parameters and return the same
### values.
hook lookup => sub {
    # Callback that replaces lookup()'s behavior.
    # Callback receives the same arguments as lookup(), and it must return
    # the same number and type of arguments that lookup() returns.
    return $download_path;
};


### See EXTENDING App::Fetchware WITH A MODULE for details on how to extend
### fetchware with a module to install software that cannot be expressed
### using App::Fetchware's configuration file syntax.

DESCRIPTION

App::Fetchware represents fetchware's API. For ducumentation on how to use App::Fetchware's fetchware command line interface see fetchware.

It is the heart and soul of fetchware where all of fetchware's main behaviors are kept. It is fetchware's API, which consists of the subroutines new(), new_install(), check_syntax(), start(), lookup(), download(), verify(), unarchive(), build(), install(), uninstall(), upgrade() and end().

App::Fetchware stores both details about fetchware's configuration file syntax, documents how to create a fetchware extension, and documents the internal workings of how App::Fetchware implements fetchware's package management behavior:

For details on App::Fetchware's configuration file syntax see the section "CREATING A App::Fetchware FETCHWAREFILE" and the section "MANUALLY CREATING A App::Fetchware FETCHWAREFILE" for more details, and how to create one in a text editor without fetchware new's help.
If the needs of your program overcome the capabilities of App::Fetchware's configuration options, then see the section "FURTHER CUSTOMIZTING YOUR FETCHWAREFILE" for details on how to overcome those limitations.
Straightforward and more complicated Fetchware file examples are available in the "EXAMPLE FETCHWAREFILES" section.
For instructions on how to create a fetchware extension see the section "CREATING A FETCHWARE EXTENSION".
For details on App::Fetchware's API that is useful for those customizing their Fetchwarefile's and to those who are implementing a fetchware extension please see the section "FETCHWAREFILE API SUBROUTINES".

CREATING A App::Fetchware FETCHWAREFILE

In order to create a new fetchware package, you need to create a new Fetchwarefile. You can easily do this with the fetchware new command, which works as follows.

1. This command will ask you a few questions, and use the answers you provide to create a Fetchwarefile for you.
2. After it does so, it gives you a chance to edit its autogenerated Fetchwarefile manually in an editor of your choice.
3. Afterwards, it will ask you if you would like to go ahead and use your newly created Fetchwarefile to install your new program as a fetchware package. If you answer yes, the default, it will install it, but if you anwer no; instead, it will simply print out the location to the Fetchwarefile that it created for you. You can then copy that file to a location of your choice, or use that path as an option to additional fetchware commands.

You can also create your Fetchwarefile manually in a text editor if you want to. See the section "MANUALLY CREATING A App::Fetchware FETCHWAREFILE" for the details. Some programs require greater customization of Fetchware's behavior than is available in its configuration options in these cases see the section "FURTHER CUSTOMIZTING YOUR FETCHWAREFILE" for the specific details on how to make fetchware do what it needs to do to manage your source code distributions.

MANUALLY CREATING A App::Fetchware FETCHWAREFILE

fetchware provides a new command that allows you to easily create a Fetchwarefile by simply answering a bunch of simple questions. This new command even will let you manually edit the Fetchwarefile it generates for you, so you can later customize it however you want to. Then it will ask if you want to install it. If you answer yes, then it will install it for you, if you answer no, it will print out the path to the newly created Fetchwarefile it created for you. Then you can call fetchware install path/to/Fetchwarefile to install it later on. Or you can follow the instructions below and manually create a Fetchwarefile in your text editor of choice.

Fetchware's Fetchwarefile's configuration option syntax

The syntax for setting configuration options is easy. It's just the name of the configuration option you want to specify like so:

program

And then you add a space, and then whatever value you want it to have in quotes.

program 'Apache';

And don't forget the semicolon ; on then end. The semicolon is required

You can use comments as needed to help document you Fetchwarefile like so:

# Fetchwarefile for Apache.
program 'Apache';

1. Name it

Use your text editor to create a file with a .Fetchwarefile file extension. Use of this convention is not required, but it makes it obvious what type of file it is. Then, just copy and paste the example text below, and replace [program] with what you choose the name of your proram to be. program is simply a configuration option that simply names your Fetchwarefile. It is not actually used for anything other than to name your Fetchwarefile to document what program or behavior this Fetchwarefile manages.

use App::Fetchware;

# [program] - explain what [program] does.
program '[program]';

Fetchwarefiles are actually small, well structured, Perl programs that can contain arbitrary perl code to customize fetchware's behavior, or, in most cases, simply specify a number of fetchware or a fetchware extension's configuration options. Below is my filled in example App::Fetchware fetchwarefile.

use App::Fetchware;

# apache 2.2 - Web Server.
program 'apache-2.2';

Notice the use App::Fetchware; line at the top. That line is absolutely critical for this Fetchwarefile to work properly, because it is what allows fetchware to use Perl's own syntax as a nice easy to use syntax for Fetchwarefiles. If you do not use the matching use App::Fetchware; line, then fetchware will spit out crazy errors from Perl's own compiler listing all of the syntax errors you have. If you ever receive that error, just ensure you have the correct use App::Fetchware; line at the top of your Fetchwarefile.

2. Determine your lookup_url

At the heart of App::Fetchware is its lookup_url, which is the URL to the FTP or HTTP mirror you want App::Fetchware to use to obtain a directory listing to see if a new version of your program is available for download. To figure this out just use your browser to find the program you want fetchware to manage for you's Web site. Skip over the download link, and instead look for the gpg, sha1, or md5 verify links, and copy and paste one of those between the single quotes above in the lookup_url. Then delete the file portion--from right to left until you reach a /. This is necessary, because fetchware uses the lookup_url as a basis to download your the gpg, sha1, or md5 digital signatures or checksums to ensure that the packages fetchware downloads and installs are exactly the same as the ones the author uploads.

lookup_url '';

And then after you copy the url.

lookup_url 'http://www.apache.org/dist/httpd/';

3. Determine your filter configuration option

The filter option specifies a perl regex that is matched against the list of the files in the directory you specify in your lookup_url. This sorts through the directory to pick out which program or even which version of the same program you want this Fetchwarefile to manage.

This is needed, because some programs such as Apache have multiple versions available at the same time, so you would need to specify which version of apache you want to download. So, you'd specify filter 'httpd-2.2'; to make fetchware download and manage Apache 2.2, or you could specify filter 'httpd-2.4'; to specify the newer 2.4 series version.

This option also exists to allow fetchware to pick out our program if you specify a mirror directory that has more than one program in it. If you do this, then fetchware can use filter to pick out the program you want to download and install from the crowd.

Just write your perl regex in between the single quotes ' below. You don't need to master regular expressions to specify this option. Just specify the name of the program and/or the main part of the version number. Do not specify the entire version number or fetchware will never update your program properly.

filter '';

And then after you type in the text pattern.

filter 'httpd-2.2';

4. Add mandatory verification settings

Verification of software downloads is mandatory, because fetchware, in order to install the software that is downloaded, must execute the build and installation scripts on your computer sometimes even as the root administrator! Therefore, fetchware will refuse to build and install any software package that cannot be verified. This limitation can be bypassed by setting the verify_failure_ok configuration option to true, but this is not recommended.

Instead, if standard verification fails, please set up one or more of the configuration options below that may allow verification to succeed if the author has his download site set up differently then fetchware expects.

gpg_keys_url - Should list a URL to a file most likely named KEYS that contains versions of the author's gpg verification keys that is suitable to be imported into gpg using gpg --import [name of file]. An example would be:

gpg_keys_url 'http://www.apache.org/dist/httpd/KEYS';

users_keyring - Tells fetchware to use the user who calls fetchware's gpg keyring instead of fetchware's own keyring. This is handy for when you want to install a program, but the author has no easily accessible KEYS file, but the author has listed his gpg key on his Website. With this option, you can import this key into your own keyring using gpg --import [name of file], and then specify this option in your Fetchwarefile as shown below.

users_keyring 'On';

gpg_sig_url - Should list a URL to a directory (not a file) that has files with the same names as the software archives that contain your program, but with a .asc, .sig, or .sign file extension. An example would be:

gpg_sig_url 'http://www.apache.org/dist/httpd/';

sha1_url - Should list a URL to a directory (not a file) that has files with the same names as the software archives that contain your program, but with a .sha or .sha1 file extension. An example would be:

sha1_url 'http://www.apache.org/dist/httpd/';

md5_url - Should list a URL to a directory (not a file) that has files with the same names as the software archives that contain your program, but with a .md5 file extension. An example would be:

md5_url 'http://www.apache.org/dist/httpd/';

NOTICE: There is no configuration option to change what filename fetchware uses. You're stuck with its default of what fetchware determines your $download_path to be with the appropriate .asc, sha1, or .md5 added to it.

Just copy and paste the example above replacing the example between the single quotes ' with the actual value you need.

5. Specify at least one mirror

Because fetchware's lookup_url must be the author's main mirror instead of a 3rd party mirror for verification purposes, you must also add a mirror option that specifies one 3rd party mirror. I recommend picking one near your physical geographical location or at least in your own country or one close by.

mirror can be specified more than once, you you can have more than one mirror. An example is below.

mirror 'http://apache.mesi.com.ar//httpd/';
mirror 'http://apache.osuosl.org//httpd/';
mirror 'ftp://apache.mirrors.pair.com//httpd/';
mirror 'http://mirrors.sonic.net/apache//httpd/';
mirror 'http://apache.mirrors.lucidnetworks.net//';

You can specify as many mirrors as you want to. You could perhaps include all the mirrors your source code distribution has. And the mirrors are tried in the order they are specified in your Fetchwarefile.

6. Specifiy other options

That's all there is to it unless you need to further customize App::Fetchware's behavior to modify how your program is installed.

If your Fetchwarefile is now finished, you can install your new Fetchwarefile as a fetchware package with:

fetchware install [path to your new fetchwarefile]

Or you can futher customize it further as shown next if needed.

7. Optionally add build and install settings

If you want to specify additional settings the first to choose from are the build and install settings. These settings control how fetchware builds and installs your software. They are briefly listed below. For further details see the section "App::Fetchware FETCHWAREFILE CONFIGURATION OPTIONS".

temp_dir - Specifies the temporary directory fetchware will use to create its own working temporary directory where it downloads, unarchives, builds, and then installs your program from a directory inside this directory.
user - (UNIX only) - Specifies a non-root user to drop privileges to when downloading, verifying, unarchive, and building your program. Root priveedges are kept in the parent process for install if needed.
prefix - Specifies the --prefix option for AutoTools (./configure) based programs.
configure_options - Specifies any additional options that fetchware should give to AutoTools when it runs ./configure to configure your program before it is built and installed.
make_options - Specifies any command line options you would like to provide to make when make is run to build and install your software. -j 4 is quite popular to do a paralled make to build and install the program faster.
build_commands - Specifies a list of commands that fetchware will use to build your program. You only need this option if your program uses a build system other than AutoTools such as cmake or perhaps a custom one like Perl's Configure
install_commands - Specifies a list of commands that fetchware will use to install your program. You only need this option if your program uses a build system other than AutoTools such as cmake or perhaps a custom one like Perl's Configure
uninstall_commands - Specifies a list of commands that fetchware will use to uninstall your program. You only need this option if your source code distribution does not provide a make uninstall target, which not every source code distribution does.
no_install - Specifies a boolean (true or false) value to turn off fetchware installing the software it has downloaded, verified, unarchvied, and built. If you specify a true argument (1 or 'True' or 'On'), then fetchware will not install your program; instead, it will leave its temporary directory intact, and print out the path of this directory for you to inspect and install yourself. If you don't specify this argument, comment it out, or provide a false argument (0 or 'False' or 'Off'), then fetchware will install your program.

Just copy and paste the example below replacing [new_directive] with the name of the new directive you would like to add, and fill in the space between the single quotes '.

[new_directive] '';

After pasting it should look like.

[new_directive] '~/wallpapers';

USING YOUR App::Fetchware FETCHWAREFILE WITH FETCHWARE

After you have created your Fetchwarefile as shown above you need to actually use the fetchware command line program to install, upgrade, or uninstall your App::Fetchware Fetchwarefile.

install

A fetchware install [path/to/Fetchwarefile] while using a App::Fetchware Fetchwarefile causes fetchware to install the program specified in your fetchwarefile to your computer as you have specified any build or install options.

upgrade

A fetchware upgrade [installed program name] while using a App::Fetchware Fetchwarefile will simply run the same thing as install all over again, which ill upgrade your program if a new version is available.

uninstall

A fetchware uninstall [installed program name] will cause fetchware to run the command make uninstall, or run the commands specified by the uninstall_commands configuration option. make uninstall is only available from some programs that use AutoTools such as ctags, but apache, for example, also uses AutoTools, but does not provide a uninstall make target. Apache for example, therefore, cannot be uninstalled by fetchware automatically.

upgrade-all

A fetchware upgrade-all will cause fetchware to run fetchware upgrade for all installed packages that fetchware is tracking in its internal fetchware database. This command can be used to have fetchware upgrade all currently installed programs that fetchware installed.

If you would like fetchware upgrade-all to be run every night automatically by cron, then just create a file say fetchware with the contents below in it, and add it to /etc/cron.daily.

#!/bin/sh
# Update all already installed fetchware packages.
fetchware upgrade-all

And if you don't want to run it system wide as root, you can add it to your user crontab by pasting the snippet below in to your crontab by executing crontab -e.

# Check for updates using fetchware every night at 2:30AM.
# Minute   Hour   Day of Month     Month          Day of Week     Command    
# (0-59)  (0-23)     (1-31)  (1-12 or Jan-Dec) (0-6 or Sun-Sat)
    30      2          *              *               *           fetchware upgrade-all

App::Fetchware'S FETCHWAREFILE CONFIGURATION OPTIONS

App::Fetchware has many configuration options. Most were briefly described in the section "MANUALLY CREATING A App::Fetchware FETCHWAREFILE". All of them are detailed below.

program 'Program Name';

program simply gives this Fetchwarefile a name. It is availabe to fetchware after parsing your Fetchwarefile, and is used to name your Fetchwarefile when using fetchware new. It is required just like lookup_url, mirror, perhaps filter, and some method to verify downloads are.

filter 'perl regex here';

Specifies a Perl regular expression that fetchware uses when it determines what the latest version of a program is. It simply compares each file in the directory listing specified in your lookup_url to this regular expression, and only matching files are allowed to pass through to the next part of fetchware that looks for source code archives to download.

See perlretut for details on how to use and create Perl regular expressions; however, actual regex know how is not really needed just paste verbatim text between the single quotes '. For example, filter 'httpd-2.2'; will cause fetchware to only download Apache 2.2 instead of the version for Windows or whatever is in the weird httpd-deps-* package.

temp_dir '/tmp';

temp_dir tells fetchware where to store fetchware's temporary working directory that it uses to download, verify, unarchive, build, and install your software. By default it uses your system temp directory, which is whatever directory File::Temp's tempdir() decides to use, which is whatever File::Spec's tmpdir() decides to use.

fetchware_db_path '~/.fetchwaredb';

fetchware_db_path tells fetchware to use a different directory other than its default directory to store the installed fetchware package for the particular fetchware package that this option is specified in your Fetchwarefile. Fetchware's default is /var/log/fetchware on Unix when run as root, and something like /home/[username]/.local/share/Perl/dist/fetchware/ when run nonroot.

This option is not recommended unless you only want to change it for just one fetchware package, because fetchware also consults the FETCHWARE_DATABASE_PATH environment variable that you should set in your shell startup files if you want to change this globally for all of your fetchware packages. For sh/bash like shells use:

export FETCHWARE_DATABASE_PATH='/your/path/here'

user 'nobody';

Tells fetchware what user it should drop privileges to. The default is nobody, but you can specify a different username with this configuration option if you would like to.

Dropping privileges allows fetchware to avoid downloading files and executing anything inside the downloaded archive as root. Except of course the commands needed to install the software, which will still need root to able to write to system directories. This improves security, because the downloaded software won't have sytem privileges until after it is verified, providing that what you downloaded is exactly what the author uploaded.

Note this only works for unix like systems, and is not used on Windows and other non-unix systems.

Also note, that if you are running fetchware on Unix even if you do not specify the user configuration option to configure what user you will drop privileges to, fetchware will still drop privileges using the ubiquitous nobody user. If you do not want to drop privileges, then you must use the stay_root configuration option as described below.

stay_root 'On';

Tells fetchware to not drop privileges. Dropping privileges when run as root is fetchware's default behavior. It improves security, and allows fetchware to avoid exposing the root account by downloading files as root.

Do not use this feature unless you are absolutely sure you need it.

SECURITY NOTICE

stay_root, when turned on, causes fetchware to not drop privileges when fetchware looks up, downloads, verifies, and builds your program. Instead, fetchware will stay root through the entire build cycle, which needlessly exposes the root account when downloading files from the internet. These files may come from trusted mirrors, but mirrors can, and do get cracked:

http://www.itworld.com/security/322169/piwik-software-installer-rigged-back-door-following-website-compromise?page=0,0

http://www.networkworld.com/news/2012/092612-compromised-sourceforge-mirror-distributes-backdoored-262815.html

http://www.csoonline.com/article/685037/wordpress-warns-server-admins-of-trojans

http://www.computerworld.com/s/article/9233822/Hackers_break_into_two_FreeBSD_Project_servers_using_stolen_SSH_keys

lookup_url 'ftp://somedomain.com/some/path

This configuration option specifies a url of a FTP or HTTP or local (file://) directory listing that fetchware can download, and use to determine what actual file to download and perhaps also what version of that program to download if more than one version is available as some mirrors delete old versions and only keep the latest one.

This url is used for:

1. To determine what the actual download url is for the latest version of this program
2. As the base url to also download a cryptographic signature (ends in .asc) or a SHA-1 or MD5 signature to verify the contents match what the SHA-1 or MD5 checksum is.

You can use the mirror configuration option to specify additional mirrors. However, those mirrors will only be used to download the large software archives. Only the lookup_url will be used to download directory listings to check for new versions, and to download digital signatures or checksums to verify downloads.

lookup_method 'timestamp';

Fetchware has two algorithms it uses to determine what version of your program to download:

timestamp: The timestamp algorithm simply uses the mtime (last modification time) that is availabe in FTP and HTTP directory listings to determine what file in the directory is the newest. timestamp is also the default option, and is the one used if lookup_method is not specified.
versionstring: Versionstring parses out the version numbers that each downloadable program has, and uses them to determine the downloadable archive with the highest version number, which should also be the newest and best version of the archive to use.

gpg_keys_url 'lookup_url.com/some/path';

Specifies a file not a directory URL for a KEYS file that lists all of the authors' gpg keys for fetchware to download and import before using them to verify the downloaded software package.

If you come accross a software package whoose author uses gpg to sign his software packages, but he does not include it in the form of a file on his main mirror, then you can specify the user_keyring option. This option forces fetchware to use the user who runs fetchware's keyring instead of fetchware's own keyring. This way you can then import the author's key into your own keyring, and have fetchware use that keyring that already has the author's key in it to verify your downloads.

user_keyring 'On';

When enabled fetchware will use the user who runs fetchware's keyring instead of fetchware's own keyring. Fetchware uses its own keyring to avoid adding cruft to your own keyring.

This is needed when the author of a software package does not maintain a KEYS file that can easily be downloaded and imported into gpg. This option allows you to import the author's key manually into your own gpg keyring, and then fetchware will use your own keyring instead of its own to verify your downloads.

LIMITAITON

user_keyring when set to true requires that the user that fetchware is running under have a real gpg keyring with keys that have been imported into it. This is not the case unless the user option has been specified with a user account with a proper home directory and gpg keyring for gpg to use. Because of this limitation if you need to specify user_keyring be sure to also specify the user option to specify a real user account instead of the default fake one nobody.

Typically you would import the keys into your own user accounts gpg keyring, and then you would specify your own username with the user option to tell fetchware to drop privs to your own user account to have access to your own gpg keys.

gpg_sig_url 'mirror.com/some/path';

Specifies an alternate url to use to download the cryptographic signature that goes with your program. This is usually a file with the same name as the download url with a .asc file extension added on. Fetchware will also append the extensions sig and sign if .asc is not found, because some pgp programs and authors use these extensions too.

sha1_url 'mastermirror.com/some/path';

Specifies an alternate url to download the SHA-1 checksum. This checksum is used to verify the integrity of the archive that fetchware downloads.

You must specify the master mirror site, which is your programs main mirror site, because if you download it from a mirror, its possible that both the archive and the checksum could have been tampered with.

md5_url 'mastermirror.com/some/path';

Specifies an alternate url to download the MD5 checksum. This checksum is used to verify the integrity of the archive that fetchware downloads.

user_agent 'Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0'

Specifies what user_agent you would like fetchware to pretend to be when downloading files using the HTTP protocol. Some sites annoying prevent some user agents from working while allowing others. This allows you to pretend to be a real browser such as Firefox if you need to.

verify_method 'gpg';

Chooses a method to verify your program. The default is to try gpg, then sha1, and finally md5, and if all three fail, then the default is to exit fetchware with an error message, because it is insecure to install archives that cannot be verified. The availabel options are:

gpg - Uses the gpg program to cryptographically verify that the program you downloaded is exactly the same as its author uploaded it.
sha1 - Uses the SHA-1 hash function to verify the integrity of the download. This is much less secure than gpg.
md5 - Uses the MD5 hash function to verify the integrity of the download. This is much less secure than gpg.

verify_failure_ok 'True';

Fetchware's default regarding failing to verify your downloaded Archive with gpg, sha1, or md5 is to exit with an error message, because installing software that cannot be cryptographically verified should never be done.

SECURITY NOTICE

However, if the author of a program you want to use fetchware to manage for you does not offer a gpg, sha1, or md5 file to verify its integrity, then you can use this option to force Fetchware to install this program anyway. However, do not enable this option lightly. Please scour the program's mirrors and homepage to see which gpg_keys_url, gpg_sig_url, sha1_url, md5_url, or user_keyring you can use to ensure that your archive is verified before it is compiled and installed. Even mirrors from sites large and small get hacked regularly:

http://www.itworld.com/security/322169/piwik-software-installer-rigged-back-door-following-website-compromise?page=0,0

http://www.networkworld.com/news/2012/092612-compromised-sourceforge-mirror-distributes-backdoored-262815.html

http://www.csoonline.com/article/685037/wordpress-warns-server-admins-of-trojans

http://www.computerworld.com/s/article/9233822/Hackers_break_into_two_FreeBSD_Project_servers_using_stolen_SSH_keys

So, Please give searching for a gpg_keys_url, gpg_sig_url, sha1_url, md5_url, or user_keyring for your program another try before simply enabling this option.

NOTICE: verify_failure_ok is a boolean configuration option, which just means its values are limited to either true or false. True values are 'True', 'On', 1, and false values are 'False', 'Off', and 0. All other values are syntax errors.

prefix '/opt/';

Controls the AutoTools ./configuration --prefix=... option, which allows you to change the base directory that most software (software that uses AutoTools) uses as the base directory for when they install themselves.

For example, most programs copy binaries to prefix/bin, documentation to prefix/docs, manpages to prefix/man, and so on.

WARNING: prefix only supports source code distributions that use GNU AutoTools. These can easily be determined by the presence of certain files in the the distributions main directory such as configure, configure.in, and acinclude.m4, and others. So, if your program uses a different build system just include that system's version of AutoTools' --prefix option in your build_commands configuration option.

configure_options '--datadir=/var/mysql';

Provides options to AutoTools ./configure program that configures the source code for building. Most programs don't need this, but some like Apache and MySQL need lots of options to configure them properly. In order to provide multiple options do not separate them with spaces; instead, separate them with commas and keep single quotes ' around them like in the example below.

configure_options '--datadir=/var/mysql', '--mandir=/opt/man',
    '--enable-module=example';

This option is not compatible with build_commands. If you use build_commands, than this option will not be used.

WARNING: configure_options only supports source code distributions that use GNU AutoTools. These can easily be determined by the presence of certain files in the the distributions main directory such as configure, configure.in, and acinclude.m4, and others. So, if your program uses a different build system just include that system's version of AutoTools' ./configure program in your build_commands configuration option.

make_options '-j4';

This option exists mostly just to enable parallel make using the -j jobs option. But any list of options make accepts will work here too. Separate them using commas and surround each one with single quotes ' like in the example above.

build_commands './configure', 'make';

Specifies what commands fetchware will use to build your program. Building your program includes configuring the source code to be compiled, and then the actual compiling too. It does not include installing the compiled source code. Use install_commands, which is described below, for that. Specify multiple options just like configure_options does.

The default build_commands is simply to run ./configure followed by make.

install_commands 'make install';

This single command or perhaps list of commands will be run in the order your specifyto install your program. You can specify multiple options just like configure_options does.

The default install_commands is simply to run make install.

uninstall_commands 'make uninstall';

This command or list of commands will be run instead of fetchware's default of make uninstall. Many source code distributions do not provide a uninstall make target, so they can not easily be uninstalled by fetchware without such support. In these cases, you could look into paco, src2pkg, or fpm. These all aid turning a source code distribution into your operating system's package format, or somehow magically monitoring make install to track what files are installed where, and then using this information to be able to uninstall them.

no_install 'On';

This boolean, see below, configuration option determines if fetchware should install your software or not install your software, but instead prints out the path of its build directory, so that you can QA test or review the software before you install it.

NOTICE: no_install is a boolean configuration option, which just means its values are limited to either true or false. True values are 'True', 'On', 1, and false values are 'False', 'Off', and 0. All other values are syntax errors.

mirror 'somemirror0.com/some/optional/path';

Your Fetchwarefile needs to have at least one mirror specified. Although you can specify as many as you want to.

This configuration option, unlike all the others, can be specified more than once. So, for example you could put:

mirror 'somemirror1.com';
mirror 'somemirror2.com';
mirror 'somemirror3.com';
mirror 'somemirror4.com';
mirror 'somemirror5.com';

When fetchware downloads files or directories it will try each one of these mirrors in order, and only fail if all attempts at all mirrors fail.

If you specify a path in addition to just the hostname, then fetchware will try to get whatever it wants to download at that alternate path as well.

mirror 'somemirror6./com/alternate/path';

FURTHER CUSTOMIZING YOUR FETCHWAREFILE

Because fetchware's configuration files, its Fetchwarefiles, are little Perl programs, you have the full power of Perl at your disposal to customize fetchware's behavior to match what you need fetchware to do to install your source code distributions.

Not only can you use arbitrary Perl code in your Fetchwarefile to customize fetchware for programs that don't follow most FOSS mirroring's unwritten standards or use a totally different build system, you can also create a fetchware extension. Creating a fetchware extension even allows you to turn your extension into a proper CPAN distribution, and upload it to CPAN to share it with everybody else. See the section below, "CREATING A FETCHWARE EXTENSION", for full details.

How Fetchware's configuration options are made

Each configuration option is created with App::Fetchware::CreateConfigOptions This package's import() is a simple code generator that generates configuration subroutines. These subroutines have the same names as fetchware's configuration options, because that is exactly what they are. Perl's Prototypes are used in the code that is generated, so that you can remove the parentheses typically required around each configuration subroutine. This turns what looks like a function call into what could believably be considered a configuration file syntax.

These prototypes turn:

lookup_url('http://somemirror.com/some/path');

Into:

lookup_url 'http://somemirror.com/some/path';

Perl's prototypes are not perfect. The single quotes and semicolon are still required, but the lack of parens instantly makes it look much more like a configuration file syntax, then an actual programming language.

The magic of `use App::Fetchware;`

The real magic power behind turning a Perl program into a configuration file sytax comes from the use App::Fetchware; line. This line is single handedly responsible for making this work. This one line imports all of the configuration subroutines that make up fetchware's configuration file syntax. And this mechanism is also behind fetchware's extension mechanism. (To use a App::Fetchware extension, you just use it. Like use App::FetchwareX::HTMLPageSync;. That's all there is to it. This other App::Fetchware is responsible for exporting subroutines of the same names as those that make up App::Fetchware's API. These subroutines are listed in the section "FETCHWAREFILE API SUBROUTINES" as well as their helper subroutines. See the section below "CREATING A FETCHWARE EXTENSION" for more information on how to create App::Fetchware extensions.

So how do I add some custom Perl code to customize my Fetchwarefile?

You use hook() to override one of fetchware's API subroutines. Then when fetchware goes to call that subroutine, your own subroutine is called in its place. You can hook() as many of fetchware's API subroutines as you need to.

Remember your replackement subroutine must take the exact same arguments, and return the same outputs that the standard fetchware API subroutines do!

All of the things these subroutines return are later used as parameters to later API subroutines, so failing to return a correct value may cause fetchware to fail.

hook()

# Inside a Fetchwarefile...
hook lookup => sub {
    # Your own custom lookup handler goes here!
};

hook() allows you to replace fetchware's API subroutines with whatever Perl code reference you want to. But it must take the same arguments that each API subroutine takes, and provide the same return value. See the section "FETCHWAREFILE API SUBROUTINES" for the details of what the API subroutine's parameters are, and what their return values should be.

hook() should be used sparingly, and only if you really know what you're doing, because it directly changes fetchware's behavior. It exists for cases where you have a software package that exceeds the abilities of fetchware's configuration options, but creating a fetchware extension for it would be crazy overkill.

hook lookup => sub {
    # Your replacement for lookup() goes here.
};

However, that is not quite right, because some of App::Fetchware's API subroutines take important arguments and return important arguments that are then passed to other API subroutines later on. So, your replacement lookup() must take the same arguments and must return the same values that the other App::Fetchware subroutines may expect to be passed to them. So, let's fix lookup(). Just check lookup()'s documentation to see what its arguments are and what it returns by checking out the section "FETCHWAREFILE API SUBROUTINES":

hook lookup => sub {
    # lookup does not take any arguments.
    
    # Your replacement for lookup() goes here.
    
    # Must return the same thing that the original lookup() does, so
    # download() and everything else works the same way.
    return $download_path;
};

Some App::Fetchware API subroutines take arguments, so be sure to account for them:

hook download => sub {
    # Take same args as App::Fetchware's download() does.
    my $download_path = shift;
    
    # Your replacement for download() goes here.
    
    # Must return the same things as App::Fetchware's download()
    return $package_path;
};

If changing lookup()'s behavior or one of the other App::Fetchware subroutines, and you only want to change part of its behavior, then consider importing one of the :OVERRIDE_* export tags. These tags exist for most of the App::Fetchware API subroutines, and are listed below along with what helper subroutines they import with them. To check their documentation see the section "FETCHWAREFILE API SUBROUTINES".

WARNING: If you specify a OVERRIDE_* export tag to App::Fetchware be sure to add the :DEFAULT export tag to also export App::Fetchware's default exports, which must be properly exported for fetchware to work properly.

OVERRIDE_LOOKUP - "get_directory_listing()", "parse_directory_listing()", "determine_download_path()", "ftp_parse_filelist()", "http_parse_filelist()", "file_parse_filelist()", "lookup_by_timestamp()", "lookup_by_versionstring()", "lookup_determine_downloadpath()"
OVERRIDE_DOWNLOAD - "determine_package_path()"
OVERRIDE_VERIFY - "gpg_verify()", "sha1_verify()", "md5_verify()", "digest_verify()"
OVERRIDE_UNARCHIVE - "check_archive_files()", "list_files()", "list_files_tar()", "list_files_zip()", "unarchive_package()", "unarchive_tar()", "unarchive_zip()"
OVERRIDE_BUILD - "run_star_commands()" and "run_configure()".
OVERRIDE_INSTALL - "chdir_unless_already_at_path()".
OVERRIDE_UNINSTALL - uninstall() uses build()'s and install()'s API's, but does not add any subroutines of its own..

An example:

use App::Fetchware qw(:DEFAULT :OVERRIDE_LOOKUP);

...

hook lookup => sub {

    ...

    # ...Download a directory listing....

    # Use same lookup alorithms that lookup() uses.
    return lookup_by_versionstring($filename_listing);
    
    # Return what lookup() needs to return.
    return $download_path;
};

Feel free to specify a list of the specifc subroutines that you need to avoid namespace polution, or install and use Sub::Import if you demand more control over imports.

A real example

See the section "EXAMPLE FETCHWAREFILES" for real examples of using hook() to change fetchware's behavior enough to make it work properly with different source code distributions that are popular.

EXAMPLE FETCHWAREFILES

Below are example Fetchwarefiles. They use a range of features to show you what Fetchware can do, and how it's Fetchwarefile can be manipulated to work with any source-code distribution.

Apache Web Server

This Apache Fetchwarefile includes a few extra mirrors just in case one is down. The fairly common -j 4 make_option to make the build go faster, and a gigantic configure_options telling ./configure excatly how I want my Apache built and configured. It also uses a heredoc to make its configure_options configuration option much more legible.

use App::Fetchware;

program 'Apache';
lookup_url 'http://www.apache.org/dist/httpd/';
filter 'httpd-2.2';
mirror 'http://apache.mirrors.pair.com/httpd/';
mirror 'http://mirrors.ibiblio.org/apache/httpd/';
mirror 'ftp://apache.cs.utah.edu/apache.org/httpd/';

verify_method 'gpg';
gpg_keys_url 'http://www.apache.org/dist/httpd/KEYS';

make_options '-j 4';
prefix '/home/dly/software/apache2.2';
# You can use heredocs to make gigantic options like this one more legible.
configure_options <<EOO;
--with-mpm=prefork
--enable-modules="access alias auth autoindex cgi logio log_config status vhost_alias userdir rewrite ssl"
--enable-so
EOO

NGINX Web Server

nginx has its distribution set up differently than Apache, so some changes are needed. First, nginx does not seem to use any mirrors at all, which means nginx's Fetchwarefile is going to look kind of stupid with the same exact URL being used for both the lookup_url and the mirror, but such a configuration is supported. Next, nginx does not have a KEYS file, but it does list it's developer's keys on its Website. So, they have to be imported manually into your keyring, and then specify the user_keyring option to switch fetchware from usings its own keyring to using your own keyring. Also, note the comment regarding having to use the user option to specify a real user account. This is needed, because the verify step is done by fetchware's child after that child drops its root privileges. Th default user is nobody, and nobody has no real home, and therefore no keyring, so gpg won't be able to read the keys you ask it to by using the user_keyring option; therefore, user must be specified to change it to a real user, whoose keyring has had these keys imported into it. Also, worth noting that this nginx configuration does not use a filter option. This is not actually needed, because the only source-code packages availabe at the lookup_url are the nginx software packages themselves, but it might be a good idea to include one, because the nginx developers could always change how their download server is structured. So, including it is always a good idea.

use App::Fetchware;

program 'nginx';

# lookup_url and mirror are the same thing, because nginx does not seem to have
# mirrors. Fetchware, however, requires one, so the same URL is simply
# duplicated.
lookup_url 'http://nginx.org/download/';
mirror 'http://nginx.org/download/';


# Must add the developers public keys to my own keyring. These keys are
# availabe from http://nginx.org/en/pgp_keys.html Do this with:
# gpg \
# --fetch-keys http://nginx.org/keys/aalexeev.key\
# --fetch-keys http://nginx.org/keys/is.key\
# --fetch-keys http://nginx.org/keys/mdounin.key\
# --fetch-keys http://nginx.org/keys/maxim.key\
# --fetch-keys http://nginx.org/keys/sb.key\
# --fetch-keys http://nginx.org/keys/glebius.key\
# --fetch-keys http://nginx.org/keys/nginx_signing.key
# You might think you could just set gpg_keys_url to the nginx-signing.key key,
# but that won't work, because like apache different releases are signed by
# different people. Perhaps I could change gpg_keys_url to be like mirror where
# you can specify more than one option?
user_keyring 'On';
# user_keyring specifies to use the user's own keyring instead of fetchware's.
# But fetchware drops privileges by default using he user 'nobody.' nobody is
# nobody, so that user account does not have a home directory for gpg to read a
# keyring from. Therefore, I'm using my own account instead.
user 'dly';
# The other option, which is commented out below, is to use root's own keyring,
# and the no_install option to ensure that root uses its own keyring instead of
# nobody's.
# noinstall 'On';
verify_method 'gpg';

PHP Programming Language

PHP annoyingly uses a custom Web application on each of its mirror sites to serve HTTP downloads. No simple directory listing is available. Therefore, to use php with fetchware, custom lookup, download, and verify hooks are needed that override fetchware's internal behavior to customize fetchware as needed so that it can work with how PHP's site is up.

The lookup hook downloads and parses the http://www.php.net/downloads.php page, which lists files availabe for download. This file is parsed using HTML::TreeBuilder to determine the latest version. The MD5 sum is also parsed out to verify the downloaded file as well.

The download hook is only needed, because http_download_file() presumes that the last part of the path is the filename you're downloading. And this is annoyingly not the case with the way PHP has its downloading system set up.

The verify hook just uses Digest::MD5 to calculate the md5sum of the downloaded file, and compares it with the one lookup parses out.

use App::Fetchware qw(
    :OVERRIDE_LOOKUP
    :OVERRIDE_DOWNLOAD
    :OVERRIDE_VERIFY
    :DEFAULT
);
use App::Fetchware::Util ':UTIL';
use HTML::TreeBuilder;
use URI::Split qw(uri_split uri_join);
use Data::Dumper;
use HTTP::Tiny;

program 'php';

lookup_url 'http://us1.php.net/downloads.php';
mirror 'http://us1.php.net';
mirror 'http://us2.php.net';
mirror 'http://www.php.net';

# php does *not* use a standard http or ftp mirrors for downloads. Instead, it
# uses its Web site, and some sort of application to download files using URLs
# such as: http://us1.php.net/get/php-5.5.3.tar.bz2/from/this/mirror
#
# Bizarrely a URL like
# http://us1.php.net/get/php-5.5.3.tar.bz2/from/us2.php.net/mirror
# gets you the same page, but on a different mirror. Weirdly, these are direct
# downloads without any HTTP redirects using 300 codes, but direct downloads.
# 
# This is why using fetchware with php you needs a custom lookup handler.
# The files you download are resolved to a [http://us1.php.net/distributions/...]
# directory, but trying to access a apache styple auto index at that url fails
# with a rediret back to downloads.php.
my $md5sum;
hook lookup => sub {
    die <<EOD unless config('lookup_url') =~ m!^http://!;
php.Fetchwarefile: Only http:// lookup_url's and mirrors are supported. Please
only specify a http lookup_url or mirror.
EOD

    msg "Downloading lookup_url [@{[config('lookup_url')]}].";
    my $dir_list = download_dirlist(config('lookup_url'));

    vmsg "Parsing HTML page listing php releases.";
    my $tree = HTML::TreeBuilder->new_from_content($dir_list);

    # This parsing code assumes that the latest version of php is the first one
    # we find, which seems like a dependency that's unlikely to change.
    my $download_path;
    $tree->look_down(
        _tag => 'a',
        sub {
            my $h = shift;
            
            my $link = $h->as_text();

            # Is the link a php download link or something to ignore.
            if ($link =~ /tar\.(gz|bz2|xz)|(tgz|tbz2|txz)/) {

                # Set $download_path to this tags href, which should be
                # something like: /get/php-5.5.3.tar.bz2/from/a/mirror
                if (exists $h->{href} and defined $h->{href}) {
                    $download_path = $h->{href};
                } else {
                    die <<EOD;
php.Fetchwarefile: A path should be found in this link [$link], but there is no
path it in. No href [$h->{href}].
EOD
                }

                # Find and save the $md5sum for the verify hook below.
                # It should be 6 elements over, so it should be the sixth index
                # in the @right array below (remember to start counting from 0.).
                my @right = $h->right();
                my $md5_span_tag = $right[5];
                $md5sum = $md5_span_tag->as_text();
                $md5sum =~ s/md5:\s+//; # Ditch md5 header.
            }
        }
    );

    # Delete the $tree, so perl can garbage collect it.
    $tree = $tree->delete;

    # Determine and return a proper $download_path.
    # Switch it from [/from/a/mirror] to [/from/this/mirror], so the mirror will
    # actually return the file to download.
    $download_path =~ s!/a/!/this/!;

    vmsg "Determined download path to be [$download_path]";
    return $download_path;
};


# I also must hook download(), because fetchware presumes that the filename of
# the downloaded file is the last part of the $path, but that is not the case
# with the path php uses for file downloads, because it ends in mirror, which is
# *not* the name of the file; therefore, I must  hook download() to fix this
# problem.
hook download => sub {
    my ($temp_dir, $download_path) = @_;

    my $http = HTTP::Tiny->new();
    my $response;
    for my $mirror (config('mirror')) {
        my ($scheme, $auth, $path, $query, $fragment) = uri_split($mirror);
        my $url = uri_join($scheme, $auth, $download_path, undef, undef);
        msg <<EOM;
Downloading path [$download_path] using mirror [$mirror].
EOM
        $response = $http->get($url);
        
        # Only download it once.
        last if $response->{success};
    }

    die <<EOD unless $response->{success};
php.Fetchwarefile: Failed to download the download path [$download_path] using
the mirrors [@{[config('mirror')]}]. The response was:
[@{[Dumper($response->{headers})]}].
EOD
    die <<EOD unless length $response->{content};
php.Fetchwarefile: Didn't actually download anything. The length of what was
downloaded is zero. status [$response->{status}] reason [$response->{reason}]
HTTP headers [@{[Dumper($response->{headers})]}].
EOD

    msg 'File downloaded successfully.';

    # Determine $filename from $download_path
    my @paths = split('/', $download_path);
    my ($filename) = grep /php/, @paths;

    vmsg "Filename determined to be [$filename]";

    open(my $fh, '>', $filename) or die <<EOD;
php.Fetchwarefile: Failed to open [$filename] for writing. OS error [$!].
EOD

    print $fh $response->{content};
    close $fh or die <<EOD;
php.Fetchwarefile: Huh close($filename) failed! OS error [$!].
EOD

    my $package_path = determine_package_path($temp_dir, $filename);

    vmsg "Package path determined to be [$package_path].";

    return $package_path
};


# The above lookup hook parses out the md5sum on the php downloads.php web
# site, and stores it in $md5sum, which is used in the the verify hook below.
hook verify => sub {
    # Don't need the $download_path, because lookup above did that work for us.
    # $package_path is the actual php file that we need to ensure its md5
    # matches the one lookup determined.
    my ($download_path, $package_path) = @_;

    msg "Verifying [$package_path] using md5.";

    dir <<EOD if not defined $md5sum;
php.Fetchwarefile: lookup failed to figure out the md5sum for verify to use to
verify that the php version [$package_path] matches the proper md5sum.
The md5sum was [$md5sum].
EOD

    my $package_fh = safe_open($package_path, <<EOD);
php.Fetchwarefile: Can not open the php package [$package_path]. The OS error
was [$!].
EOD

    # Calculate the downloaded php file's md5sum.
    my $digest = Digest::MD5->new();
    $digest->addfile($package_fh);
    my $calculated_digest = $digest->hexdigest();

    die <<EOD unless $md5sum eq $calculated_digest;
php.Fetchwarefile: MD5sum comparison failed. The calculated md5sum
[$calculated_digest] does not match the one parsed of php.net's Web site
[$md5sum]! Do not trust this downloaded file! Perhaps there's a bug somewhere,
or perhaps the php mirror you downloaded this php package from has been hacked.
Mirrors do get hacked occasionally, so it is very much possible.
EOD

    msg "ms5sums [$md5sum] [$calculated_digest] match.";

    return 'Package Verified';
};

PHP Programming Language using its git VCS instead of download mirrors.

PHP like most open source software you can easily download off the internet uses a version control system to track changes to its source code. This source code repository is basically the same thing as a normal source code distribution would be except VCS commands like git pull are used to update it instead of checking a mirror for a new version. The Fetchwarefile below for php customizes Fetchware to work with php's VCS instead of the traditional downloading of actual source code archives.

It overrides lookup() to use a local git repo stored in the $git_repo_dir variable. To create a repo just clone php's git repo (see http://us1.php.net/git.php for details.). It runs git pull to update the repo, and then it runs git tags, and ditches some older junk tags, and finds only the tags used for new versions of php. These are sorted using the versonstring lookup() algorithm, and the latest one is returned.

download() uses git checkout [latesttag] to "download" php by simply changing the working directory to the latest tag. verify() uses git's cool verify-tag command to verify the gpg signature. unarchive() is updated to do nothing since there is no archive to unarchive. However, because we reuse build(), archive() must return a $build_path that build() will change its directory to. start() and end() are also overridden, because managing a temporary directory is not needed, so, instead, they just do a git checkout master to switch from whatever the latest tag is back to master, because git pull bases what it does on what branch you're in, so we must actually be a real branch to update git.

# php-using-git.Fetchwarefile: example fetchwarefile using php's git repo
# for lookup(), download(), and verify() functionality.
use App::Fetchware qw(:DEFAULT :OVERRIDE_LOOKUP);
use App::Fetchware::Util ':UTIL';
use Cwd 'cwd';

# The directory where the php source code's local git repo is.
my $git_repo_dir = '/home/dly/Desktop/Code/php-src';

# By default Fetchware drops privs, and since the source code repo is stored in
# the user dly's home directory, I should drop privs to dly, so that I have
# permission to access it.
user 'dly';

# Determine latest version by using the tags developers create to determine the
# latest version.
hook lookup => sub {
    # chdir to git repo.
    chdir $git_repo_dir or die <<EOD;
php.Fetchwarefile: Failed to chdir to git repo at
[$git_repo_dir].
OS error [$!].
EOD

    # Pull latest changes from php git repo.
    run_prog('git pull');

    # First determine latest version that is *not* a development version.
    # And chomp off their newlines.
    chomp(my @tags = `git tag`);

    # Now sort @tags for only ones that begin with 'php-'.
    @tags = grep /^php-/, @tags;

    # Ditch release canidates (RC, alphas and betas.
    @tags = grep { $_ !~ /(RC\d+|beta\d+|alpha\d+)$/ } @tags;

    # Sort the tags to find the latest one.
    # This is quite brittle, but it works nicely.
    @tags = sort { $b cmp $a } @tags;

    # Return $download_path, which is only just the latest tag, because that's
    # all I need to know to download it using git by checking out the tag.
    my $download_path = $tags[0];

    return $download_path;
};


# Just checkout the latest tag to "download" it.
hook download => sub {
    my ($temp_dir, $download_path) = @_;

    # The latest tag is the download path see lookup.
    my $latest_tag = $download_path;

    # checkout the $latest_tag to download it.
    run_prog('git checkout', "$latest_tag");

    my $package_path = cwd();
    return $package_path;
};


# You must manually add php's developer's gpg keys to your gpg keyring. Do
# this by  going to the page: http://us1.php.net/downloads.php . At the
# bottom the gpg key "names are listed such as "7267B52D" or "5DA04B5D."
# These are their key "names." Use gpg to download them and import them into
# your keyring using: gpg --keyserver pgp.mit.edu --recv-keys [key id]
hook verify => sub {
    my ($download_path, $package_path) = @_;

    # the latest tag is the download path see lookup.
    my $latest_tag = $download_path;

    # Run git verify-tag to verify the latest tag
    my $success = eval { run_prog('git verify-tag', "$latest_tag"); 1;};

    # If the git verify-tag fails, *and* verify_failure_ok has been turned on,
    # then ignore the thrown exception, but print an annoying message.
    unless (defined $success and $success) {
        unless (config('verify_failure_ok')) {
            msg <<EOM;
Verification failure ok, becuase you've configured fetchware to continue even
if it cannot verify its downloads. Please reconsider, because mirror and source
code repos do get hacked. The exception that was caught was:
[$@]
EOM
        }
    }
};


hook unarchive => sub {
    # there is nothing to archive due to use of git.
    do_nothing(); # But return the $build_path, which is the cwd().
    my $build_path = $git_repo_dir;
    return $build_path;
};

# It's a git tag, so it lacks an already generated ./configure, so I must use
# ./buildconf to generate one. But it won't work on php releases, so I have to
# force it with --force to convince ./buildconf to run autoconf to generate the
# ./configure program to configure php for building.
build_commands './buildconf --force', './configure', 'make';

# Add any custom configure options that you may want to add to customize
# your build of php, or control what php extensions get built.
#configure_options '--whatever you --need ok';

# start() creates a tempdir in most cases this is exactly what you want, but
# because this Fetchwarefile is using git instead. I don't need to bother with
# creating a temporary directory.
hook start => sub {
    # But checkout master anyway that way the repo can be in a known good state
    # so lookup()'s git pull can succeed.
    run_prog('git checkout master');
};


# Switch the local php repo back to the master branch to make using it less
# crazy. Furthermore, when using git pull to update the repo git uses what
# branch your on, and if I've checked out a tag, I'm not actually on a branch
# anymore; therefore, I must switch back to master, so that the git pull when
# this fetchwarefile is run again will still work.
hook end => sub {
    run_prog('git checkout master');
};

MariaDB Database

This example MariaDB Fetchwarefile parses the MariaDB download page to determine what the latest version is based on what filter option you set up. Once this is determined, the download path is created based on the weird path that MariaDB uses on its mirrors.

Like PHP MariaDB uses some annoying software on their Web site to presumably track downloads. This software makes use of AJAX, which is vastly beyone the capabilities of HTML::TreeBuilder to parse, because it needs a working JavaScript environment. Therefore, the example Fetchwarefile below has no way of verifying the MySQL downloads. This could be fixed by using a Perl Web scraping module that can deal with JavaScript.

use App::Fetchware;

program 'MariaDB';

# MariaDB uses ccache, which wants to create a ~/.ccache cache, which it can't
# do when it's running as nobody, so use a real user account to ensure ccache
# has a cache directory it can write to.
user 'dly';

lookup_url 'https://downloads.mariadb.org/';

# Below are the two USA mirrors where I live. Customize them as you need based
# on the mirrors listed on the download page (https://downloads.mariadb.org/ and
# then click on which version you want, and then click on the various mirrors
# by country. All you need is the scheme (ftp:// or http:// part) and the
# hostname without a slash (ftp.osuosl.org or mirror.jmu.edu). Not the full path
# for each mirror.
mirror 'http://ftp.osuosl.org';
mirror 'http://mirror.jmu.edu';

# The filter option is key to the custom lookup hook working correctly. It must
# represent the text that corresponds to the latest GA release of MariaDB
# available. It should be 'Download 5.5' for 5.5 or 'Download 10.0' for the
# newver but not GA 10.0 version of MariaDB.
filter 'Download 5.5';

hook lookup => sub {
    vmsg "Downloading HTML download page listing MariaDB releases.";
    my $dir_list = http_download_dirlist(config('lookup_url'));

    vmsg "Parsing HTML page listing MariaDB releases.";
    my $tree = HTML::TreeBuilder->new_from_content($dir_list);

    # This parsing code assumes that the latest version of php is the first one
    # we find, which seems like a dependency that's unlikely to change.
    my @version_number;
    $tree->look_down(
        _tag => 'a',
        sub {
            my $h = shift;
            
            my $link = $h->as_text();

            # Find the filter which should be "Download\s[LATESTVERSION]"
            my $filter = config('filter');
            if ($link =~ /$filter/) {
                # Parse out the version number.
                # It's just the second space separated field.
                push @version_number, (split ' ', $link)[1];
            }
        }
    );

    # Delete the $tree, so perl can garbage collect it.
    $tree = $tree->delete;

    # Only one version should be found.
    die <<EOD if @version_number > 1;
mariaDB.Fetchwarefile: multiple version numbers detected. You should probably
refine your filter option and try again. Filter [@{[config('filter')]}].
Versions found [@version_number].
EOD

    # Construct a download path using $version_number[0].
    my $filename = 'mariadb-' . $version_number[0] . '.tar.gz';

    # Return a proper $download_path, so That I do not have to hook download(),
    # but can reuse Fetchware's download() subroutine.
    my $weird_prefix = '/mariadb-' . $version_number[0] . '/kvm-tarbake-jaunty-x86/';
    my $download_path = '/pub/mariadb' . $weird_prefix .$filename;
    return $download_path;
};

# Make verify() failing to verify MariaDB ok, because parsing out the MD5 sum
# would require a Web scraper that supports javascript, which HTML::TreeBuilder
# obviously does not.
verify_failure_ok 'On';

# Use build_commands to configure fetchware to use MariaDB's BUILD script to
# build it. See https://mariadb.com/kb/en/generic-build-instructions/ for
# instructions on the different BUILD  cmake scripts that are available.
build_commands 'BUILD/compile-pentium64-max';

# Use install_commands to tell fetchware how to install it. I could leave this
# out, but it nicely documents what command is needed to install MariaDB
# properly.
install_commands 'make install';

PostgreSQL Database

Below is a example Fetchwarefile that overrides lookup() to determine the latest version, but manages to avoid overriding anything else. It uses the same style as the rest downloading an HTML page that lists the version numbers on it somewhere. Then it parses the HTML with HTML::TreeBuilder. It populates an array, and then uses App::Fetchware's lookup_by_versionstring() to determine which version is the latest one. This is then concatenated with a bunch of other stuff to determine the $download_path.

MD5 verification is supported by simply specifying a md5_url option, because by default fetchware uses the lookup_url to determine where to download the md5sum from, but that won't work with PostgreSQL, because it's download system has the md5sum on the download mirror instead of the lookup_url.

use App::Fetchware qw(:DEFAULT :OVERRIDE_LOOKUP);
use App::Fetchware::Util ':UTIL';

use HTML::TreeBuilder;

program 'postgres';

# The Postgres file browser URL lists the available versions of Postgres.
lookup_url 'http://www.postgresql.org/ftp/source/';

# Mirror URL where the file browser links to download them from.
my $mirror = 'http://ftp.postgresql.org';
mirror $mirror;

# The Postgres file browser URL that is used for the lookup_url lists version
# numbers of Postgres like v9.3.0. this lookup hook parses out the list of
# theses numbers, determines the latest one, and constructs a $download_path to
# return for download to use to download based on what I set my mirror to.
hook lookup => sub {
    my $dir_list = no_mirror_download_dirlist(config('lookup_url'));

    my $tree = HTML::TreeBuilder->new_from_content($dir_list);

    # Parse out version number directories.
    my @ver_nums;
    my @list_context = $tree->look_down(
        _tag => 'a',
        sub {
            my $h = shift;

            my $link = $h->as_text();

            # Is this link a version number or something to ignore?
            if ($link =~ /^v\d+\.\d+(.\d+)?$/) {
                # skip version numbers that are beta's, alpha's or release
                # candidates (rc).
                return if $link =~ /beta|alpha|rc/i;
                # Strip useless "v" that just gets in the way later when I
                # create the $download_path.
                $link =~ s/^v//;
                push @ver_nums, $link;
            }
        }
    );

    # Turn @ver_num into the array of arrays that lookup_by_versionstring()
    # needs its arguments to be in.
    my $directory_listing = do {
        my $arrayref_of_arrays_directory_listing = [];
        for my $ver_num (@ver_nums) {
            push @$arrayref_of_arrays_directory_listing,
                [$ver_num];
        }
        $arrayref_of_arrays_directory_listing;
    };
    # Find latest version.
    my $latest_ver = lookup_by_versionstring($directory_listing);

    # Return $download_path.
    my $download_path = '/pub/source/'. "v$latest_ver->[0][0]" .
        "/postgresql-$latest_ver->[0][0].tar.bz2";
    return $download_path;
};

# MD5sums are stored on the download site, so use them to verify the package.
verify_method 'md5';
# But they are *not* stored on the original "lookup_url" site, so I must provide
# a md5_url pointing to the download site.
md5_url $mirror;

CREATING A FETCHWARE EXTENSION

WARNING: Currently, fetchware's extension system is BETA, and unlikely to change, but changes may happen most likely just in the form of bug fixes. This, however, is unlikely, but it could happen. Most likely, just some minor bug fixes will occur to the API not any major changes or refactors or rewrites. I'm considering "genericizing" make_test_dist() so Fetchware extension authors can reuse it to test their extensions.

Fetchware's main program fetchware uses App::Fetchware's short and simple API to implement fetchware's default behavior; however, other styles of source code distributions exist on the internet that may not fit inside App::Fetchware's capabilities. That is why, in addition to its flexible configuration file syntax, fetchware allows modules other than App::Fetchware to provide it with its behavior.

How the API works

When fetchware installs or upgrades something it executes the API subroutines check_syntax(), start(), lookup(), download(), verify(), unarchive(), build(), install(), and end() in that order. And when fetchware uninstalls an installed package it executes the API subroutines check_syntax(), start(), unarchive(), uninstall(), and end(). Upgrade is basically the exact same thing as install(), but it compares version numbers using the upgrade() API subroutine. And fetchware new obviously just calls new() and new_install().

Extending App::Fetchware

This API can be overridden inside a user created Fetchwarefile by using hook() as explained above. hook() simply takes a Perl code reference that takes the same parameters, and returns the same results that the subroutine that you've hook()ed takes and returns.

For more extensive changes you can create a App::Fetchware module that "subclasses" App::Fetchware. Now App::Fetchware is not an object-oriented module, so you cannot use parent or base to add it to your program's inheritance tree using @INC. You can, however, use App::Fetchware::ExportAPI to import whatever subroutines from App::Fetchware that you want to reuse such as start() and end(), and then simply implement the remaining subroutines that make up App::Fetchware's API. Just like the CODEREF extensions mentioned above, you must take the same arguments and return the same values that fetchware expects or using your App::Fetchware extension will blow up in your face.

This is described in much more detail below in "CHANGING FETCHWARE'S BEHAVIOR".

Essential Terminology

App::Fetchware manages to behave like an object oriented module would with regards to its extension system without actually using perl's object-oriented features especially using @INC for subclassing, which App::Fetchware does not use.

The same terminology as used in OOP is also used here in App::Fetchware, because the concepts are nearly the same--they're simply implemented differently.

API subroutines

These are the subroutines that App::Fetchware implements, and that fetchware uses to implement its desired behavior. They are new(), new_install(), check_syntax(), start(), lookup(), download(), verify(), unarchive(), build(), install(), uninstall(), upgrade(), and end(). All must be implemented or "inherited" from App::Fetchware using App::Fetchware::ExportAPI as discussed below in "Implement your fetchware extension." in a App::Fetchware subclass.

override

Means the same thing it does in object-oriented programming. Changing the definition of a method/subroutine without changing its name. In OOP this is simply done by subclassing something, and then defining one of the methods that are in the superclass in the subclass. In App::Fetchware extensions this is done by simply defining a subroutine with the same name as one or more of the API subroutines that are defined above. There is no method resolution order and @INC is not consulted.

subclass

Means the same thing it does in object-oriented programming. Taking one class and replacing it with another class. Only since App::Fetchware is not object-oriented, it is implemented differently. You simply use App::Fetchware::ExportAPI to specify the "API subroutines" that you are not going to override, and then actually implement the remaining subroutines, so that your App::Fetchware subclass has the same interface that App::Fetchware does.

To create a fetchware extension you must understand how they work:

1. First a Fetchwarefile is created, and what module implements App:Fetchware's API is declared with a use App::Fetchware...; line. This line is use App::Fetchware for default Fetchwarefiles that use App::Fetchware to provide fetchware with the API it needs to work properly.
2. To use a fetchware extension, you simply specify the fetchware extension you want to use with a use App::FetchwareX::...; instead of specifying use App::Fetchware line in your Fetchwarefile. You must replace the App::Fetchware import with the extension's. Both cannot be present. Fetchware will exit with an error if you use more than one App::Fetchware line without specifying specific subroutines in all but one of them.
3. Then when fetchware parses this Fetchwarefile when you use it to install, upgrade, or uninstall something, This use App::FetchwareX::...; line is what imports App::Fetchware's API subroutines into fetchware's namespace.

That's all there is to it. That simple use App::Fetchware...; imports from App::Fetchware or a App::Fetchware extension such as App::FetchwareX::HTMLPageSync the API subroutines (such as start(), lookup(), ..., install(), and uninstall()) fetchware needs to use to install, upgrade, or uninstall whatever program your Fetchwarefile specifies.

After understanding how they work, simply follow the instructons and consider the recommendations below. Obviously, knowing Perl is required. A great place to start is chromatic's Modern Perl.

Develop your idea keeping in mind fetchware's package manager metaphor

Fetchware is a package manager like apt-get, yum, or slackpkg. It installs, upgrades, or uninstalls programs. Fetchware is not a Plack for command line programs. It has a rather specific API meant to fit its package manager metaphor. So, keep this in mind when developing your idea for a fetchware extension.

Map your extension's behavior to App::Fetchware's API

App::Fetchware has a specific behavior consisting of just a few subroutines with specific names that take specific arguments, and return specific values. This API is how you connect your extension to fetchware.

Just consider the description's of App::Fetchware's API below, and perhaps consult their full documentation in "FETCHWAREFILE API SUBROUTINES".

my ($program_name, $fetchwarefile) = new($term, $program_name) - Along with new_install() implements Fetchware's new command for helping users create new Fetchwarefiles using a Q&Z wizard interface.
my $fetchware_package_path = new_install($program_name, $fetchwarefile) - new()'s sister subroutine for implementing the Q&A wizard interface for Fetchware's new command. Just used to keep root to make ask_to_install_now_to_test_fetchwarefile() work for system level installs while keeping drop privs enabled by default.
'Syntax Ok' = check_syntax() - Checks the user's Fetchwarefile, but only at Fetchware's level. Perl level syntax errors are not checked. PPI is not used to parse the file. Instead only high-level Fetchware specific syntax errors are checked.
my $temp_dir = start(KeepTempDir = 0 | 1)> - Gives your extension a chance to do anything needed before the rest of the API subroutines get called. App::Fetchware's start() manages App::Fetchware's temporary directory creation. If you would like to also use a temporary directory, you can just use App::Fetchware::ExportAPI to "inherit" App::Fetchware's start() instead of implementing it yourself.
my $download_url = lookup() - Determines and returns a download url that download() receives and uses to download the archive for the program.o
my $package_path = download($tempd_dir, $download_url) - Downloads its provided $download_url argument.
verify($download_url, $package_path) - Verifies the integrity of your downloaded archive using gpg, sha1, or md5.
my $build_path = unarchive($package_path) - Unpacks the downloaded archive.
build($build_path) - Configures and compiles the downloaded archive.
install() - Installs the compiled archive.
end() - Cleans up the temporary directory that start() created. Can be overridden to do any other clean up tasks that your archive needs.
uninstall($build_path) - Uninstalls an already installed program installed with the same App::Fetchware extension.
$upgrade = upgrade($download_path, $fetchware_package_path) - uses its two arguments to determine if a new version is available to upgrade to, or if the currently installed version is the latest version, and no additional upgrades are needed.

Also, keep in mind the order in which these subroutines are called, what arguments they receive, and what their expected return value is.

new = Just new() and new_install().
install - check_syntax(), start(), lookup(), download(), verify(), unarchive(), build(), install(), and end().
upgrade - check_syntax(), start(), lookup(), upgrade(), download(), verify(), unarchive(), build(), install(), and end().
upgrade-all - Is the same as upgrade, because it just call upgrade for each fetchware package that is installed in the fetchware package database.
uninstall - You might think its just uninstall(), but it does not. check_syntax(), start(), unarchive(), uninstall(), and end().

Use the above overview of App::Fetchware's API to design what each API subroutine keeping in mind its arguments and what its supposed to return.

Determine your fetchware extension's Fetchwarefile configuration options.

App::Fetchware has various configuration options such as temp_dir, prefix, and so on. Chances are your fetchware extension will also need such configuration options. These are easily created with App::Fetchware::CreateConfigOptions, which manufactures these to order for your convenience. There are four different types of configuration options:

ONE - Takes only one argument, and can only be used once.
ONEARRREF - Can only be called once, but can take multiple agruments at once.
MANY - Takes only one argument, but can be called more than once. The only example is mirror.
BOOLEAN - Takes only one arguement, and can only be called once just like ONE. The difference is that BOOLEANs are limited to only boolean true or false values such as 'On' or 'Off', 'True' or 'False', or 1 or 0. App::Fetchware examples include no_install and vefify_failure_ok.

Using the documentation above and perhaps also the documentation for App::Fetchware::CreateConfigOptions, determine the names of your configuration options, and what type of configuraton options they will be.

Implement your fetchware extension.

Since you've designed your new fetchware extension, now it's time to code it up. The easiest way to do so, is to just take an existing extension, and just copy and paste it, and then delete its specifics to create a simple extension skeleton. Then just follow the steps below to fill in this skeleton with the specifics needed for your fetchware extension.

1. Set up proper exports and imports.

Because fetchware needs your fetchware extension to export all of the subroutines that make up the fetchware's API, and any configuration options (as Perl subroutines) your extension will use, fetchware uses the helper packages App::Fetchware::ExportAPI and App::Fetchware::CreateConfigOptions to easily manage setting all of this up for you.

First, use App::Fetchware::ExportAPI to be sure to export all of fetchware's API subroutines. This package is also capable of "inheriting" any of App::Fetchware's API subroutines that you would like to keep. An example.

# Use App::Fetchware::ExportAPI to set up proper exports this fetchware
# extension.
use App::Fetchware::ExportAPI KEEP => [qw(new_install start end)],
    OVERRIDE => [qw(new check_syntax lookup download verify
    unarchive build install uninstall upgrade)]
;

There two types of subroutines ExportAPI helps you with:

KEEP - Specifies which API subroutines you will "keep" or perhaps "inherit" from App::Fetchware. Usually this is just start() and end() to manage the temp_dir for you.
OVERRIDE = Specifies which API subroutines you will "override," or implement yourself.

Second, use App::Fetchware::CreateConfigOptions to create all of the configuration options (such as temp_dir, no_install, and so on.) you want your fetchware extension to have.

There are four types of configuration options.

ONE - Take one an only ever one argument, and can only be called once per Fetchwarefile.: Examples: temp_file, prefix, and lookup_url.
ONEARRREF - Takes one or more arguments like MANY, but unlike MANY can only be called once.: Examples: configure_options, make_options, and build_commands.
MANY - Takes one or more arguments, and can be called more than once. If called more than once, then second call's arguments are added to the existing list of arguments.: Examples: mirror.
BOOLEAN - Just like ONE except it will convert /off/i and /false/i to 0 to support more than just Perl's 0 or undef being false.: Examples: verify_failure_ok, no_install, and stay_root.

Additionally, there is another type of configuration option that only App::Fetchware::CreateConfigOptions uses:

IMPORT - Allows you to import already defined configuration options from App::Fetchware into your Fetchware extension. For exmaple, you might also want a temp_dir or no_install configuration option, and the IMPORT type allows you to easily import one and document the fact that it is imported.

An example.

use App::Fetchware::CreateConfigOptions
    IMPORT => [qw(temp_dir no_install)],
    ONE => [qw(repository directory)],
    ONEARRREF => [qw(build_options install_options)],
    BOOLEAN => [qw(delete_after_download)],
;

These 2 simple use()'s are all it takes to set up proper exports for your fetchware extension.

2. Code any App::Fetchware API subroutines that you won't be reusing from App::Fetchware.

Use their API documentation from the section "FETCHWAREFILE API SUBROUTINES" to ensure that you use the correct subroutine names, arguments and return the correct value as well.

An example for overriding lookup() is below.

=head2 lookup()

    my $download_url = lookup();

# New lookup docs go here....

=cut

sub lookup {

    # New code for new lookup() goes here....

    # Return the required $download_url.
    return $download_url;
}

Use Fetchware's Own Libraries to Save Developement Time.

Fetchware includes many libraries to save development time. These libraries are well tested by Fetchware's own test suite, so you too can use them to save development time in your own App::Fetchware extensions.

These libraries are:

App::Fetchware::Util

Houses various logging subroutines, downloading subroutines, security subroutines, and temporary directory managment subroutines that fetchware itself uses, and you can also make use of them in your own fetchware extensions.

Logging subroutines - msg(), vmsg(), and run_prog().
- These subroutines support fetchware's command line options such as -v (--verbose) and -q (--quiet).
- msg() should be used to print a message to the screen that should always be printed, while vmsg() should only be used to print messages to the screen when the -v (--verbose) command line option is turned on.
- run_prog() is a system() wrapper that also supports -v and -q options, and should be used to run any external commands that your App::Fetchware extension needs to run.
Downloading subroutines - download_file() and download_dirlist().
- download_file() should be used to download files. It supports the following schemes ftp://, http://, or file://. It simply downloads the file using Net::Ftp or HTTP::Tiny to the current working directory.
- download_dirlist() should be used to download FTP, HTTP, or local directory listings. It is mostly just used by lookup() to determine if a new version is available based on the information it parses from the directory listing.
Security subroutines - safe_open() and drop_privs().
- safe_open() opens the file and then runs a bunch of file and directory tests to ensure that only the user running fetchware or root can modify the file or any of that file's containing directories to help prevent Fetchwarefiles from perhaps being tampered by other users or programs.
- drop_privs() forks and drops privileges. It's used by fetchware to drop privs to avoid downloading and compiling software as root. It most likely should not be called by App::Fetchware extensions, because fetchware already calls it for you, but it is there if you need it.
Temporary Directory subroutines - create_tempdir(), original_cwd(), and cleanup_tempdir(). Only needed if you don't reuse App::Fetchware's own start() and end() subroutines that make use of thse temporary directory subroutines for you.
- create_tempdir() creates and chdir()'s into a temporary directory using File::Temp's tempdir() function. It also deals with creating a Fetchware semaphore file to keep fetchware clean from deleting any still needed temporary directories.
- original_cwd() simply returns what fetchware's current working directory was before create_tempdir() created and chdir()'d into the temporary directory.
- cleanup_tempdir() deals with closing the fetchware semaphore file.

Test::Fetchware

Test::Fetchware includes utility subroutines that fetchware itself uses to test itself, and they are shared with extension writers through this module.

eval_ok() - A poor man's Test::Exception in one simple subroutine. Why require every user to install a dependency only used for testing when one simple subroutine does the trick.
print_ok() - A poor man's Test::Output in one simple subroutine.
skip_all_unless_release_testing() - Does just what it's name says. If fetchware's internal release/author only Environment variables are set, only then will any Test::More subtests that call this subroutine skip the entire subtest. This is used to skip running tests that install real programs on the testing computer's system. Many of Fetchware's tests actually install a real program such as Apache, and I doubt any Fetchware user would like to have Apache installed and uninstalled a bagillion times when they install Fetchware. Use this subroutine in your own App::Fetchware extension's to keep that from happening.
make_clean() - Just run_prog('make', 'clean') in the current working directory just as its name suggests.
make_test_dist() - Use this subroutine or craft your own similar subroutine, if you need more flexibility, to test actually installing a program on user's systems that just happens to execute all of the proper installation commands, but supplies installation commands that don't actually install anything. It's used to test actually installing software without actually installing any software.
md5sum_file() - Used to provide a md5sum file for make_test_dist() create software packages in order to pass fetchware's verify checks.
verbose_on() - Make's all vmsg()'s actually print to the screen even if -v or --verbose was not actually provided on the command line. Used to aid debugging.

App::Fetchware::Config

App::Fetchware::Config stores and manages fetchware's parsed configuration file. parse_fetchwarefile() from fetchware does the actual parsing, but it stores the configuration file inside App::Fetchware::Config. Use the subroutines below to access any configuration file options that you create with App::Fetchware::CreateConfigOptions to customize your fetchware extension. Also feel free to reuse any names of App::Fetchware configuration subroutines such as temp_dir or lookup_url

config() - Sets and gets values from the currently parsed fetchware configuration file. If there is one argument, then it returns that configuration options value or undef if there is none.If there are more than one argument, then the first argument is what configuration option to use, and the rest of the arguments are what values to set that configuration option to.
config_iter() - returns a configuration iterator. that when kicked (called, like $config_iter->()) will return one value from the specifed configuration option. Can be kicked any number of times, but once the number of configuration values is exhausted the iterator will return undef.
config_replace() - config() is used to set configuration options, and once set they cannot be changed by config(). This is meant to catch and reduce errors. But sometimes, mostly in test suites, you need to change the value of a configuration option. That's what config_replace() is for.
config_delete() - deletes the specified configuration option. Mostly just used for testing.
__clear_CONFIG() - An internal only subroutine that should be only used when it is really really needed. It clears (deletes) the entire internal hash that the configuration options are stored in. It really should only be used during testing to clear App::Fetchware::Config's intenal state between tests.
debug_CONFIG() - prints App::Fetchware::Config's internal state directly to STDOUT. Meant for debugging only in your test suite.

App::Fetchware::Fetchwarefile

Helper OO class for new() API subroutine. Allows you to programatically build a Fetchwarefile for the user using a small API instead of manually concatenating tons of strings constantly worrying about wordwrap and whitespace.

new() constructor - App::Fetchware::Fetchwarefile is actually Object-oriented unlike the rest of Fetchware's internals, so new() is its constructor.
config_options() - Used to add the actual configuration options and their values to the Fetchwarefile object.
generate() - Returns the "generated" Fetchwarefile. This method does all of the string concatenation for you in order to create the Fetchwarefile specfied in your new() and config_options() calls.

App::Fetchware's OVERRIDE_* export tags.

App::Fetchware's main API subroutines, especially the crazy complicated ones such as lookup(), are created by calling and passing data among many component subroutines. This is done to make testing much much easier, and to allow App::Fetchware extensions to also use some or most of these component subroutines when they override a App::Fetchware API subroutine.

new()'s OVERRIDE_NEW export tag.

The export tag exports all of the helper subroutines new() uses to implement its functionality. Some like get_lookup_url(), get_verification(), and get_filter_option() are quite specific to App::Fetchware, but extension_name, fetchwarefile_name(), opening_message(), prompt_for_other_options(), and edit_manually() are nice and generic, and should be appropriate for any Fetchware extension.

new_install()'s OVERRIDE_NEW_INSTALL export tag.

new_install() only exports ask_to_install_now_to_test_fetchwarefile(), and most Fetchware extensions should probably just "inherit" new_install(), because new_install()'s only real purpose was making ask_to_install_now_to_test_fetchwarefile() work when fetchware drops privs.

check_syntax()'s OVERRIDE_CHECK_SYNTAX export tag

check_syntax() only has the check_config_options() helper subroutine that is meant for reuse by Fetchware extensions, so they can check their syntax just as Fetchware itself does.

lookup()'s OVERRIDE_LOOKUP export tag.

This export tag is the largest, and perhaps the most important, because it implements fetchware's ability to determine if a new version of your software package is available. Its default is just a clever use of HTTP and FTP directory listings.

See the section "lookup() API REFERENCE" for more details on how to use these subroutines to determine if new versions of your software is available automatically.

download()'s OVERRIDE_DOWNLOAD export tag.

Only exports the subroutine determine_package_path(), which simply comcatenates a $tempdir with a $filename to return a properl $package_path, which unarchive later uses. This is mostly its own subroutine to better document how this is done, and to allow easier code reuse.

verify()'s OVERRIDE_VERIFY export tag.

Exports a family of subroutines to verify via MD5, SHA1, or GPG the integrity of your downloaded package. MD5 and SHA1 are supported for legacy reasons. All software packages should be GPG signed for much much much better security. GPG signatures when verified actually prove that the software package you downloaded is exactly what the author of that software package created, whereas MD5 and SHA1 sums just verify that you downloaded the bunch of bits in the same order that they are stored on the server.

digest_verify() an be used to add support for any other Digest::* modules that CPAN has a Digest based module for that correctly follow Digest's API.

unarchive()'s OVERRIDE_UNARCHIVE export tag.

Exports subroutines that will help you unarchive software packages in tar and zip format. The most important part to remember is to use list_files() to list the files in your archive, and pass that list to check_archive_files() to ensure that the archive will not overwrite any system files, and contains no absolute paths that could cause havok on your system. unarchive_package() does the actual unarchiving of software packages.

build()'s OVERRIDE_BUILD export tag.

Provides run_star_commands(), which is meant to execute common override commands that fetchware provides with the build_commands, install_commands, and uninstall_commands configuration file directives. These directives are of type ONEARRREF where they can only be called once, but you can supply a comma separated list of commands that fetchware will install instead of the standard commands default AutoTools commands (build() => ./configure, make; install() => make install; uninstall() => ./configure, make uninstall). See its documentation for more details.

install()'s OVERRIDE_INSTALL export tag.

install() only exports chdir_unless_already_at_path(), which is of limited use. install() also uses build()'s run_star_commands().

uninstall()'s OVERRDIE_UNINSTALL export tag.

uninstall() actually has no exports of its own, but it does make use of build() and install()'s exports.

upgrade()'s OVERRIDE_UNINSTALL export tag.

upgrade() also has no exports of its own, and does not use anyone elses.

Write your fetchware extension's documentation

Fill in all of the skeleton's missing POD to ensure that fetchware extension has enough documentation to make it easy for user's to use your fetchware extension. Be sure to document:

All of Perl's standard POD sections such as SYNOPSIS, DESCRIPTION, AUTHOR, and all of the others. See perlpodstyle for more details.
Give each subroutine its own chunk of POD before it explaining its arguments, any App::Fetchware configuration options it uses, and what its return value is.
Be sure to document both its external interface, its Fetchwarefile, and its internal interface, what subroutines it has and uses.

Write tests for your fetchware extension

Use perls venerable Test::More, and whatever other Perl TAP testing modules you need to be sure your fetchware extension works as expected.

"" in Test::Fetchware has a few testing subroutines that fetchware itself uses in its test suite that you may find helpful. These include:

"eval_ok()" in Test::Fetchware - A poor man's Test::Exception. Captures any exceptions that are thrown, and compares them to the provided exception text or regex.
"print_ok()" in Test::Fetchware - A poor man's Test::Output. Captures STDOUT, and compares it to the provided text.
"skip_all_unless_release_testing()" in Test::Fetchware - Fetchware is a package manager, but who wants software installed on their computer just to test it? This subroutine marks test files or subtests that should be skipped unless fetchware's extensive FETCHWARE_RELEASE_TESTING environement variables are set. This funtionality is described next.
"make_clean()" in Test::Fetchware - Just runs make clean in the current directory.
"make_test_dist()" in Test::Fetchware - Creates a temporary distribution that is used for testing. This temporary distribution contains a ./configure and a Makefile that create no files, but can still be executed in the standard AutoTools way.
"md5sum_file()" in Test::Fetchware - Just md5sum's a file so verify() can be tested.
"expected_filename_listing()" in Test::Fetchware - Returns a string of crazy Test::Deep subroutines to test filename listings. Not quite as useful as the rest, but may come in handy if you're only changing the front part of lookup().

Your tests should make use of fetchware's own FETHWARE_RELEASE_TESTING environment variable that controls with the help of skip_all_unless_release_testing() if and where software is actually installed. This is done, because everyone who installs fetchware or your fetchware extension is really gonna freak out if its test suite installs apache or ctags just to test its package manager functionality. To use it:

1. Set up an automated way of enabling FETCHWARE_RELEASE_TESTING.

Just paste the frt() bash shell function below. Translating this to your favorite shell should be pretty straight forward. Do not just copy and paste it. You'll need to customize the specific FETCHWARE_* environment variables to whatever mirrors you want to use or whatever actual programs you want to test with. And you'll have to point the local (file://) urls to directories that actually exist on your computer.

# Sets FETCHWARE_RELEASE_TESTING env vars for fully testing fetchware.
frt() {
    if [ -z "$FETCHWARE_RELEASE_TESTING" ]
    then
        echo -n 'Setting fetchware_release_testing environment variables...';
        export FETCHWARE_RELEASE_TESTING='***setting this will install software on your computer!!!!!!!***'
        export FETCHWARE_FTP_LOOKUP_URL='ftp://carroll.cac.psu.edu/pub/apache/httpd'
        export FETCHWARE_HTTP_LOOKUP_URL='http://www.apache.org/dist/httpd/'
        export FETCHWARE_FTP_MIRROR_URL='ftp://carroll.cac.psu.edu/pub/apache/httpd'
        export FETCHWARE_HTTP_MIRROR_URL='http://mirror.cc.columbia.edu/pub/software/apache//httpd/'
        export FETCHWARE_FTP_DOWNLOAD_URL='ftp://carroll.cac.psu.edu/pub/apache/httpd/httpd-2.2.26.tar.bz2'
        export FETCHWARE_HTTP_DOWNLOAD_URL='http://mirrors.ibiblio.org/apache//httpd/httpd-2.2.26.tar.bz2'
        export FETCHWARE_LOCAL_URL='file:///home/dly/software/httpd-2.2.22.tar.bz2'
        export FETCHWARE_LOCAL_ZIP_URL='file:///home/dly/software/ctags-zip/ctags58.zip'
        export FETCHWARE_LOCAL_BUILD_URL='/home/dly/software/ctags-5.8.tar.gz'
        export FETCHWARE_LOCAL_UPGRADE_URL='file:///home/dly/software/fetchware-upgrade'
        export FETCHWARE_NONROOT_USER='YOURUSERNAME'
        echo 'done.'
    else
        echo -n 'Deleting fetchware_release_testing environment variables...';
        unset FETCHWARE_RELEASE_TESTING
        unset FETCHWARE_FTP_LOOKUP_URL
        unset FETCHWARE_HTTP_LOOKUP_URL
        unset FETCHWARE_FTP_MIRROR_URL
        unset FETCHWARE_HTTP_MIRROR_URL
        unset FETCHWARE_FTP_DOWNLOAD_URL
        unset FETCHWARE_HTTP_DOWNLOAD_URL
        unset FETCHWARE_LOCAL_URL
        unset FETCHWARE_LOCAL_ZIP_URL
        unset FETCHWARE_LOCAL_BUILD_URL
        unset FETCHWARE_LOCAL_UPGRADE_URL
        unset FETCHWARE_NONROOT_USER
        echo 'done.'
    fi
}

Just run frt with no args to turn FETCHWARE_RELEASE_TESTING on, and run it once more to turn it off. Don't forget to reload your shell's configuration with:

$ . ~/.bashrc # Or whatever file you added it to is named.

Then inside your test suite just import skip_all_unless_release_testing() from Test::Fetchware:

use Test::Fetchware ':TESTING';

2. Call skip_all_unless_release_testing() as needed

In each subtest where you actually install anything other than a test distribution made with make_test_dist(), begin that subtest with a call to skip_all_unless_release_testing(), which will skip the whole subtest if FETCHWARE_RELEASE_TESTING is not setup properly.

subtest 'test install() for real' => sub {
    # Only test if during release testing.
    skip_all_unless_release_testing();
    
    # Tests that actually install non-trivial distributions such as ones
    # made by make_test_dist() go here.

    ....
};

If you dislike subtests, or otherwise don't want to use them, then put all of the tests that actually install something into a SKIP block after the other tests that all users will run at the end of your test file.

Fetchware has no Web site or any other place to share fetchware extensions. But fetchware is written in Perl, so fetchware can just use Perl's CPAN. To learn how to create modules and upload them to CPAN please see Perl's own documentation. perlnewmod shows how to create new Perl modules, and how to upload them to CPAN. See Module::Starter for a simple way to create a skeleton for a new Perl module, and dzil is beyond amazing, but has insane dependencies and a significant learning curve.

Fetchware Extension Best Practices

Below are a few points to keep in mind when creating a Fetchware extension that the documentation itself fails to mention prominently or that otherwise slip through the cracks, but are still important to keep in mind while designing and implementing a Fetchware extension.

Actually implement the new() API subroutine to help your Fetchware extension's users create new Fetchwarefiles for your Fetchware extension. And use App::Fetchware::Fetchwarefile to help you generate the Fetchwarefile.
Only use run_prog() to execute external programs, because it supports Fetchware's verbose (-v) and quiet (-q) options. If you really do need something else, then checkout run_prog()'s documentation for how to see if verbose or quiet options are enabled.
Use App::Fetchware::Util's logging subroutines msg() and vmsg() to print messages out to your user to tell them what you are doing. I recommened a msg() or two in each or of your main API subroutines, and then at least one vmsg() in each and any helper subroutines you may write and use.

FETCHWAREFILE API SUBROUTINES

The subroutines below are Fetchwarefile's API subroutines or helper subroutines for App::Fetchware's API subroutines. If you want information on fetchware's configuration file syntax, then see the section "FETCHWAREFILE CONFIGURATION SYNTAX" for more information. For additional information on how to keep or override these API subroutines in a fetchware extension see the section "CREATING A FETCHWARE EXTENSION"

new()

my ($program_name, $fetchwarefile) = new($term, $program_name);

# Or in an extension, you can return whatever list of variables you want,
# and then cmd_new() will provide them as arguments to new_install() except
# a $term Term::ReadLine object will precede the others.
my ($term, $program_name, $fetchwarefile, $custom_argument1, $custom_argument2)
    = new($term, $program_name);

new() is App::Fetchware's API subroutine that implements fetchware's new command. new() calls a bunch of helper subroutines that implement the algorithm fetchware uses to build new Fetchwarefiles automagically for the user. The algorithm is dead stupid:

1. Ask for lookup_url & download it.
2. Analyze the contents of the output from the lookup_url.
3. Build the Fetchwarefile according to the output.
4. Ask other questions as needed.

new() uses Term::UI, which in turn uses Term::ReadLine to implement the character based question and anwser wizard interface. A Term::ReadLine/Term::UI object is passed to new() as its first argument. new() also uses App::Fetchware::Fetchwarefile to help create and help generate the Fetchwarefile for the user.

new()'s argument is the program name that the user has specified on the command line. It will be undef if the user did not specify one on the command line.

Whatever scalars (not references just regular strings) that new() returns will be shared with new()'s sister API subroutine new_install() that is called after new() is called by cmd_install(), which implements fetchware's new command. new_install() is called in the parent process, so it does have root permissions, so be sure to test it as root as well.

drop_privs() NOTES

This section notes whatever problems you might come accross implementing and debugging your Fetchware extension due to fetchware's drop_privs mechanism.

See Util's drop_privs() subroutine for more info.

This subroutine is not run as root; instead, it is run as a regular user unless the stay_root configuration option has been set to true.

new() API REFERENCE

Below are the API routines that new() uses to create the question and answer interface for helping to build new Fetchwarefiles and fetchware packages.

extension_name();

extension_name(__PACKAGE__);

This subroutine sets the name of the extension that this implementation of new() wants to be called by. It should be App::FetchwareX::ExtensionName the full name of your extension to make looking it up in documentaiton easier.

All of new()'s API subroutines (everything in App::Fetchware's OVERRIDE_NEW export tag) use extension_name() to deterime what the this extension should be called. This is really only used in error messages and occasionally in some of the questions that new's API subroutines will ask the user. But this subroutine is important, because it allows extension authors to change all of the App::Fetchware references in the error messages to their own fetchware extensions name.

extension_name() is a singleton, and can only be set once. After being set only once any attempts to set it again will result in an exception being thrown. Furthermore, any calls to it without arguments will result in it returning the one scalar argument that was set the first time it was called.

opening_message();

opending_message($opening_message);

opening_message() takes the specified $opening_message and just prints it to STDOUT. This subroutine may seem useless, you could just use print, but using it instead of print helps document that what you're doing is printing the opening message to the user.

fetchwarefile_name();

# $fetchwarefile_name comes from the first command line option to fetchware
# new: fetchware new $fetchwarefile_name, which is optional, hence the need
# for fetchwarefile_name().
sub new {
    my $fetchwarefile_name = shift;

    my $program_name = fetchwarefile_name($term, program => $fetchwarefile_name)

If when the user called fetchware new $fetchwarefile_name, if they provided a $fetchwarefile_name name on the command line, then fetchwarefile_name() simply returns what the user provided, but if they did not, then $fetchwarefile_name in the call to fetchwarefile_name() is going to be undef. In that case, fetchwarefile_name() uses the provided Term::UI in the provided $term variable, to ask the user what the name of their fetchwarefile should be, which it returns back to the caller.

get_lookup_url()

my $lookup_url = get_lookup_url($term);

Uses $term argument as a Term::ReadLine/Term::UI object to interactively explain what a lookup_url is, and to ask the user to provide one and press enter.

download_lookup_url()

my $filename_listing = download_lookup_url($term, $lookup_url);

Attempts to download the lookup_url the user provides. Returns it after parsing it using parse_directory_listing() from App::Fetchware that lookup() itself uses.

get_mirrors()

my $mirrors_hashref = get_mirrors($term, $filename_listing);

# $mirrors_hashref = (
#   mirrors => [
#       'ftp://some.mirror/mirror',
#       'http://some.mirror/mirror',
#       'file://some.mirror/mirror',
#   ],
# );

Asks the user to specify at least one mirror to use to download their archives. It also reiterates to the user that the lookup_url should point to the author's original download site, and not a 3rd party mirror, because md5sums, sha1sums, and gpg signatures should only be downloaded from the author's download site to avoid them being modified by a hacked 3rd party mirror. While mirror should be configured to point to a 3rd party mirror to lessen the load on the author's offical download site.

After the user enters at least one mirror, get_mirrors() asks the user if they would like to add any additional mirrors, and it adds them if the user specifies them.

The list of the mirrors the user specified is returned as a hash with only one key mirror, and a value that is an arrayref of mirrors that the user has specified. The caller, then should call append_options_to_fetchwarefile() to add this list of mirrors to the user's Fetchwarefile.

get_verification()

my $verification_hashref = get_verification($term, $filename_listing, $lookup_url);

# $verification_hashref = (
#   gpg_keys_url => 'http://main.mirror/distdir',
#   verification_method => 'gpg',
# );

Parses $filename_listing to determine what type of verification is available. Prefering gpg, but falling back on sha1, and then md5 if gpg is not available.

If the type is gpg, then get_verification() will ask the user to specify a gpg_keys_url, which is required for gpg, because fetchware needs to be able to import the needed keys to be able to use those keys to verify package downloads. If this URL is not provided by the author, then get_verification() will ask the user if they would like to import the author's key into their own gpg public keyring. If they would, then get_verification() will use the user_keyring 'On' option to use the user's public keyring instead of fetchware's own keyring. And if the user does not want to use their own gpg public keyring, then get_verification will fall back to sha1 or md5 setting verify_method to sha1 or md5 as needed.

Also, adds a gpg_keys_url option if a KEYS file is found in $filename_listing.

If no verification methods are available, fetchware will print a big nasty warning message, and offer to use verify_failure_ok to make such a failure cause fetchware to continue installing your software.

Returns a hashref of options for the user's Fetchwarefile. You're responsible for calling append_options_to_fetchwarefile() to add them to the user's Fetchwarefile, or perhaps the caller could analyze them in some way, before adding them if needed. The keys are the names of the configuration options, and the values are their values.

get_filter_option()

$filter_hashref = get_filter_option($term, $filename_listing);

# $filter_hashref = (
#   filter => 'user specfied filter option',
# );

Analyzes $filename_listing and asks the user whatever questions are needed by fetchware to determine if a filter configuration option is needed, and if it is what it should be. filter is simply a perl regex that the list of files that fetchware downloads is checked against, and only files that match this regex will fetchware consider to be the latest version of the software package that you want to install. The filter option is needed, because some mirrors will have multiple software packages in the same directory or multitple different versions of one piece of software in the same directory. An example would be Apache, which has Apache versions 2.0, 2.2, and 2.4 all in the same directory. The filter option is how you differentiate between them.

If a filter was provided by the user than it is returned as a hashref with filter as the key for use with append_options_to_fetchwarefile(), or for further analysis by extension authors.

prompt_for_other_options()

prompt_for_other_options($term,
    temp_dir => {
        prompt => <<EOP,
What temp_dir configuration option would you like? 
EOP
        print_me => <<EOP
temp_dir is the directory where fetchware creates a temporary directory that
stores all of the temporary files it creates while it is building your software.
The default directory is /tmp on Unix systems and C:\\temp on Windows systems.
EOP
    },
        ...
);

Accepts a Term::Readline/Term::UI object as an argument to use to ask the user questions, and a gigantic hash of hashes in list form. The hash of hashes, %option_description, argument incluedes the prompt and print_me options that are then passed through to Term::UI to ask the user what argument they want for each specified option in the %option_description hash.

The user's answers are tallied up an returned as a hash reference.

edit_manually()

$fetchwarefile = edit_manually($term, $fetchwarefile);

edit_manually() asks the user if they would like to edit the specified $fetchwarefile manually. If the user answers no, then nothing is done. But if the user answers yes, then fetchware will open their favorit editor either using the $ENV{EDITOR} environment variable, or fetchware will ask the user what editor they would like to use. Then this editor, and a temporary fetchwarefile are opened, and the user can edit their Fetchwarefile as they please. If they are not satisfied with their edits, and wan to undo them, they can delete the entire file, and write a size 0 file, which will cause fetchware to ignore the file they edited. If the write a file with a size greater than 0, then the file the user wrote, will be used as their Fetchwarefile.

new_install()

my $fetchware_package_path = new_install($program_name, $fetchwarefile);

Configuration subroutines used:

All of them, because it calls bin/fetchware's cmd_install(), which in turn calls all of the API subroutines that fetchware's install command does.

Exists separate from new(), because new() drops privileges like most other fetchware commands do. But the new command includes the ability to ask the user if they want to install the associated program from their newly created Fetchwarefile, which requires root privileges. Therefore, we must also have a API subroutine that runs in the root privileged parent to that the install commands will run with proper permissions when fetchware is run as root.

drop_privs() NOTES

This section notes whatever problems you might come accross implementing and debugging your Fetchware extension due to fetchware's drop_privs mechanism.

See Util's drop_privs() subroutine for more info.

When fetchware is run as root, new_install() is called in the parent process with root permissions so that you can call the ask_to_install_now_to_test_fetchwarefile() helper subroutine. I suppose you could do something else in your extension if it makes sense, but that's what this API sub is intended for. The ask_to_install_now_to_test_fetchwarefile() helper subroutine needs root permissions (Unless the Fetchwarefile has been setup so that the user running it has access.), because it will call fetchware's cmd_install() to directly cause fetchware to go ahead and install the previoulsy generated Fetchwarefile.

new_install() API REFERENCE

The subroutines below are used by new_install() to provide the new_install functionality for fetchware. If you have overridden the new_install() handler, you may want to use some of these subroutines so that you don't have to copy and paste anything from new_install.

App::Fetchware is not object-oriented; therefore, you can not subclass App::Fetchware to extend it!

ask_to_install_now_to_test_fetchwarefile()

my $fetchware_package_path = ask_to_install_now_to_test_fetchwarefile($term, \$fetchwarefile, $program_name);
my $fetchwarefile_filename = ask_to_install_now_to_test_fetchwarefile($term, \$fetchwarefile, $program_name);

This subroutine asks the user if they want to install the Fetchwarefile that this subroutine has been called with. If they say yes, then the Fetchwarefile is passed on to cmd_install() to do all of the installation stuff. If they say no, then fetchware saves the file to "$program_name.Fetchwarefile" or ask_to_install_now_to_test_fetchwarefile() will ask the user where to save the file until the user picks a filename that does not exist.

If you answer yes to install your Fetchwarefile, then ask_to_install_now_to_test_fetchwarefile() will return the full path to the fetchware package that has been installed.

start()

my $temp_dir = start();

Configuration subroutines used:

temp_dir

Creates a temp directory using File::Temp, and sets that directory up so that it will be deleted by File::Temp when fetchware closes.

Returns the $temp_file that start() creates, so everything else has access to the directory they should use for storing file operations.

EXTENSION OVERRIDE NOTES

start() calls App::Fetchware::Util's create_tempdir() subroutine that cleans up the temporary directory. If your fetchware extension overrides start() or end(), you must call create_tempdir() or name your temproary directories in a manner that fetchware clean won't find them, so something that does not start with fetchware-*.

If you fail to do this, and you use some other method to create temporary directories that begin with fetchware-*, then fetchware clean may delete your temporary directories out from under your feet. To fix this problem:

Use App::Fetchware::Util's create_tempdir() in your start() and cleanup_tempdir() in your end().
Or, be sure not to name your temprorary directory that you create and manage yourself to begin with fetchware-*, which is the glob pattern fetchware clean uses. I recommend using something like App-FetchwareX-NameOfExtension-$$-XXXXXXXXXXXXXX as the name you would use in your File::Temp::temdir $pattern, with $$ being the special perlvar for the curent processes id.

drop_privs() NOTES

This section notes whatever problems you might come accross implementing and debugging your Fetchware extension due to fetchware's drop_privs mechanism.

See Util's drop_privs() subroutine for more info.

start() is called in the parent with root privileges. This is done, so that when the parent calls cleanup_tempdir() in its end() call, cleanup_tempdir() still has a valid filehandle to the fetchware semaphore file, which is used to keep fetchware clean from deleting fetchware's temporary directories out from under if you run a fetchware clean while another process is running another fetchware comand at the same time.

lookup()

my $download_path = lookup();

Configuration subroutines used:

lookup_url
lookup_method
filter
user_agent

Accesses lookup_url as a http/ftp directory listing, and uses lookup_method to determine what the newest version of the software is available. This is combined with the path given in lookup_url, and return as $download_path for use by download().

LIMITATIONS lookup_url is a web browser like URL such as http://host.com/a/b/c/path, and it must be a directory listing not a actual file. This directory listing must be a listing of all of the available versions of the program this Fetchwarefile belongs to.

Only ftp://, http://, and file:// URL scheme's are supported.

And the HTML directory listings Apache and other Web Server's return are exactly what lookup() uses to determine what the latest version available for download is.

drop_privs() NOTES

This section notes whatever problems you might come accross implementing and debugging your Fetchware extension due to fetchware's drop_privs mechanism.

See Util's drop_privs() subroutine for more info.

Under drop_privs() lookup() is executed in the child with reduced privileges.

lookup_method can be either 'timestamp' or 'versionstring', any other values will result in fetchware throwing an exception.

lookup() API REFERENCE

The subroutines below are used by lookup() to provide the lookup functionality for fetchware. If you have overridden the lookup() handler, you may want to use some of these subroutines so that you don't have to copy and paste anything from lookup.

App::Fetchware is not object-oriented; therefore, you can not subclass App::Fetchware to extend it!

get_directory_listing()

my $directory_listing = get_directory_listing();

Downloads a directory listing that lookup() uses to determine what the latest version of your program is. It is returned.

SIDE EFFECTS

get_directory_listing() returns a SCALAR REF of the output of HTTP::Tiny or a ARRAY REF for Net::Ftp downloading that listing. Note: the output is different for each download type. Type 'http' will have HTML crap in it, and type 'ftp' will be the output of Net::Ftp's dir() method.

Type 'file' will just be a listing of files in the provided lookup_url directory.

parse_directory_listing()

my $file_listing = parse_directory_listing($directory_listing);

Based on URL scheme of 'file', 'http', or 'ftp', parse_directory_listing() will call file_parse_filelist(), ftp_parse_filelist(), or http_parse_filelist(). Those subroutines do the heavy lifting, and the results are returned.

determine_download_path()

my $download_path = determine_download_path($filename_listing);

Runs the lookup_method to determine what the lastest filename is, and that one is then concatenated with lookup_url to determine the $download_path, which is then returned to the caller.

Also calls lookup_determine_downloadpath() to determine the actual download path from the $sorted_filename_listing returned by lookup_by_*().

ftp_parse_filelist()

$filename_listing = ftp_parse_filelist($ftp_listing);

Takes an array ref as its first parameter, and parses out the filenames and timstamps of what is assumed to be Net::FTP-dir()> long directory listing output.

Returns an array of arrays of filenames and timestamps.

http_parse_filelist()

$filename_listing = http_parse_filelist($http_listing);

Takes an scalar of downloaded HTML output, and parses it using HTML::TreeBuilder to build and return an array of arrays of filenames and timestamps.

file_parse_filelist()

my $filename_listing = file_parse_filelist($file_listing);

Parses the provided filelist by stating each file, and creating a properly formatted timestamp to return.

Returns an array of arrays of filenames and timestamps.

lookup_by_timestamp()

my $download_url = lookup_by_timestamp($filename_listing);

Implements the 'timestamp' lookup algorithm. It takes the timestamps placed in its first argument, normalizes them into a standard descending format (YYYYMMDDHHMM), and then cleverly uses sort to determine the latest filename, which should be the latest version of your program.

lookup_by_versionstring()

my $sorted_filename_listing = lookup_by_versionstring($filename_listing);

Determines the $sorted_filename_listing used by determine_download_path() by cleverly spliting the filenames (the first value of the array of arrays input, $filename_listing) on /\D+/, which will return a list of version numbers. Then, the cleverly splitted filename is pushed on to the input array, and then the array is sorted based on this new value using a custom sort block.

lookup_by_versionstring() also discards any entries from $filename_listing that do not have a version number in them, because without version numbers that entry can not be sorted properly. And if it is left in, it could confuse fetchware into figuring out the correct download path.

Also note, that both $filename_listing and lookup_by_versionstring()'s return value, $sorted_filename_listing, must both be arrayrefs of arrays. That is a scallar pointing to an array of arrays.

lookup_determine_downloadpath()

my $download_path = lookup_determine_downloadpath($file_listing);

Given a $file_listing of files with the same timestamp or versionstring, determine which one is a downloadable archive, a tarball or zip file. And support some hacks to make fetchware more robust. These are the filter configuration subroutine, ignoring "win32" on non-Windows systems, and supporting Apache's CURRENT_IS_ver_num and Linux's LATEST_IS_ver_num helper files.

download()

my $package_path = download($temp_dir, $download_path);

Configuration subroutines used:

mirror
user_agent

Downloads $download_path to tempdir 'whatever/you/specify'; or to whatever File::Spec's tempdir() method tries. Supports ftp and http URLs as well as local files specified like in browsers using file://

Also, returns $package_path, which is used by unarchive() as the path to the archive for unarchive() to untar or unzip.

LIMITATIONS Uses Net::FTP and HTTP::Tiny to download ftp and http files. No other types of downloading are supported, and fetchware is stuck with whatever limitations or bugs Net::FTP or HTTP::Tiny impose.

drop_privs() NOTES

This section notes whatever problems you might come accross implementing and debugging your Fetchware extension due to fetchware's drop_privs mechanism.

See Util's drop_privs() subroutine for more info.

Under drop_privs() download() is executed in the child with reduced privileges.

download() API REFERENCE

The subroutines below are used by download() to provide the download functionality for fetchware. If you have overridden the download() handler, you may want to use some of these subroutines so that you don't have to copy and paste anything from download.

App::Fetchware is not object-oriented; therefore, you can not subclass App::Fetchware to extend it!

determine_package_path()

my $package_path = determine_package_path($tempdir, $filename)

Determines what $package_path is based on the provided $tempdir and $filename. $package_path is the path used by unarchive() to unarchive the software distribution download() downloads.

$package_path is returned to caller.

verify()

verify($download_path, $package_path)

Configuration subroutines used:

gpg_keys_url
user_keyring
gpg_sig_url
sha1_url
md5_url
verify_method
verify_failure_ok
user_agent

Verifies the downloaded package stored in $package_path by downloading $download_path.{asc,sha1,md5}> and comparing the two together. Uses the helper subroutines {gpg,sha1,md5,digest}_verify().

LIMITATIONS Uses gpg command line, and the interface to gpg is a little brittle. Crypt::OpenPGP is buggy and not currently maintainted again, so fetchware cannot make use of it, so were stuck with using the command line gpg program.

drop_privs() NOTES

This section notes whatever problems you might come accross implementing and debugging your Fetchware extension due to fetchware's drop_privs mechanism.

See Util's drop_privs() subroutine for more info.

Under drop_privs() verify() is executed in the child with reduced privileges.

verify() API REFERENCE

The subroutines below are used by verify() to provide the verify functionality for fetchware. If you have overridden the verify() handler, you may want to use some of these subroutines so that you don't have to copy and paste anything from verify().

App::Fetchware is not object-oriented; therefore, you can not subclass App::Fetchware to extend it!

gpg_verify()

'Package Verified' = gpg_verify($download_path);

Uses the command-line program gpg to cryptographically verify that the file you download is the same as the file the author uploaded. It uses public-key priviate-key cryptography. The author signs his software package using gpg or some other OpenPGP compliant program creating a digital signature file with the same filename as the software package, but usually with a .asc file name extension. gpg_verify() downloads the author's keys, imports them into fetchware's own keyring unless the user sets user_keyring to true in his Fetchwarefile. Then Fetchware downloads a digital signature that usually ends in .asc. Afterwards, fetchware uses the gpg command line program to verify the digital signature. gpg_verify returns true if successful, and throws an exception otherwise.

You can use gpg_keys_url to specify the URL of a file where the author has uploaded his keys. And the gpg_sig_url can be used to setup an alternative location of where the .asc digital signature is stored.

sha1_verify()

'Package verified' = sha1_verify($download_path, $package_path);
undef = sha1_verify($download_path, $package_path);

Verifies the downloaded software archive's integrity using the SHA Digest specified by the sha_url 'ftp://sha.url/package.sha' config option. Returns true for sucess dies on error.

SECURITY NOTE If an attacker cracks a mirror and modifies a software package, they can also modify the MD5 sum of that software package on that same mirror. Because of this limitation MD5 sums can only tell you if the software package was corrupted while downloading. This can actually happen as I've had it happen to me once.: If your stuck with using MD5 sum, because your software package does not provide gpg signing, I recommend that you download your SHA1 sums (and MD5 sums) from your software package's master mirror. For example, Apache provides MD5 and SHA1 sums, but it does not mirror them--you must download them directly from Apache's servers. To do this specify a sha1_url 'master.mirror/package.sha1'; in your Fetchwarefile.

md5_verify()

'Package verified' = md5_verify($download_path, $package_path);
undef = md5_verify($download_path, $package_path);

Verifies the downloaded software archive's integrity using the MD5 Digest specified by the md5_url 'ftp://sha.url/package.sha' config option. Returns true for sucess and dies on error.

SECURITY NOTE If an attacker cracks a mirror and modifies a software package, they can also modify the MD5 sum of that software package on that same mirror. Because of this limitation MD5 sums can only tell you if the software package was corrupted while downloading. This can actually happen as I've had it happen to me once.: If your stuck with using MD5 sum, because your software package does not provide gpg signing, I recommend that you download your MD5 sums (and SHA1 sums) from your software package's master mirror. For example, Apache provides MD5 and SHA1 sums, but it does not mirror them--you must download them directly from Apache's servers. To do this specify a md5_url 'master.mirror/package.md5'; in your Fetchwarefile.

digest_verify()

'Package verified' = digest_verify($digest_type, $download_path, $package_path);
undef = digest_verify($digest_type, $download_path, $package_path);

Verifies the downloaded software archive's integrity using the specified $digest_type, which also determines the "$digest_type_url" 'ftp://sha.url/package.sha' config option. Returns true for sucess and returns false for failure.

OVERRIDE NOTE If you need to override verify() in your Fetchwarefile to change the type of digest used, you can do this easily, because digest_verify() uses Digest, which supports a number of Digest::* modules of different Digest algorithms. Simply do this by override verify() to call digest_verify('Digest's name for your Digest::* algorithm');

SECURITY NOTE If an attacker cracks a mirror and modifies a software package, they can also modify the $digest_type sum of that software package on that same mirror. Because of this limitation $digest_type sums can only tell you if the software package was corrupted while downloading. This can actually happen as I've had it happen to me once.: If your stuck with using $digest_type sum, because your software package does not provide gpg signing, I recommend that you download your $digest_type sums (and SHA1 sums) from your software package's master mirror. For example, Apache provides MD5 and SHA1 sums, but it does not mirror them--you must download them directly from Apache's servers. To do this specify a $digest_type_url 'master.mirror/package.$digest_type';' in your Fetchwarefile.

unarchive()

my $build_path = unarchive($package_path)

Configuration subroutines used:

none

Uses Archive::Tar or Archive::Zip to turn .tar.{gz,bz2,xz} or .zip into a directory. Is intelligent enough to warn if the archive being unarchived does not contain all of its files in a single directory like nearly all software packages do. Uses $package_path as the archive to unarchive, and returns $build_path.

drop_privs() NOTES

This section notes whatever problems you might come accross implementing and debugging your Fetchware extension due to fetchware's drop_privs mechanism.

See Util's drop_privs() subroutine for more info.

Under drop_privs() unarchive() is executed in the child with reduced privileges.

unarchive() API REFERENCE

The subroutine below are used by unarchive() to provide the unarchive functionality for fetchware. If you have overridden the unarchive() handler, you may want to use some of these subroutines so that you don't have to copy and paste anything from unarchive().

App::Fetchware is not object-oriented; therefore, you can not subclass App::Fetchware to extend it!

list_files()

# Remember $files is a array ref not a regular old scalar.
my $files = list_files($package_path;

list_files() takes $package_path as an argument to a tar'ed and compressed package or to a zip package, and calls list_files_{tar,zip}() accordingly. list_files_{tar,zip}() in turn uses either Archive::Tar or Archive::Zip to list all of the files in the archive and return a arrayref to that list.

list_files_tar()

my $tar_file_listing = list_files_tar($path_to_tar_archive);

Returns a arrayref of file names that are found in the given, $path_to_tar_archive, tar file. Throws an exception if there is an error.

It uses Archive::Tar->iter() to avoid reading the entire tar archive into memory.

list_files_zip()

my $zip_file_listing = list_files_zip($path_to_zip_archive);

Returns a arrayref of file names that are found in the given, $path_to_zip_archive, zip file. Throws an exception if there is an error.

unarchive_package()

unarchive_package($format, $package_path)

unarchive_package() unarchive's the $package_path based on the type of $format the package is. $format is determined and returned by list_archive(). The currently supported types are tar and zip. Nothing is returned, but unarchive_{tar,zip}() is used to to use Archive::Tar or Archive::Zip to unarchive the specified $package_path.

unarchive_tar()

my @extracted_files = unarchive_tar($path_to_tar_archive);

Extracts the given $path_to_tar_archive. It must be a tar archive. Use unarchive_zip() for zip archives. It returns a list of files that it extracted.

unarchive_zip()

'Extraced files successfully.' = unarchive_zip($path_to_zip_archive);

Extraces the give $path_to_zip_archive. It must be a zip archive. Use unarchive_tar() for tar archives. It only returns true for success. It does not return a list of extracted files like unarchive_tar() does, because Archive::Zip's extractTree() method does not.

check_archive_files()

my $build_path = check_archive_files($files);

Checks if all of the files in the archive are contained in one main directory, and spits out a warning if they are not. Also checks if one or more of the files is an absolute path, and if so it throws an exception, because absolute paths could potentially overwrite important system files messing up your computer.

build()

'build succeeded' = build($build_path)

Configuration subroutines used:

prefix
configure_options
make_options
build_commands

Changes directory to $build_path, and then executes the default build commands of './configure', 'make', 'make install' unless your Fetchwarefile specifies some build options like build_commands 'exact, build, commands';, make_options '-j4';, configure_options '--prefix=/usr/local';, or prefix '/usr/local'.

You can only specify build_commands or any of the other three build options. You cannot combine build_commands with any of the other build options.

LIMITATIONS build() like install() inteligently parses build_commands, prefix, make_options, and configure_options by just using Test::Parsewords to parse out the string considering quotes, and then execute them using fetchware's built in run_prog() to properly support -q.: Also, simply executes the commands you specify or the default ones if you specify none. Fetchware will check if the commands you specify exist and are executable, but the kernel will do it for you and any errors will be in fetchware's output.

drop_privs() NOTES

This section notes whatever problems you might come accross implementing and debugging your Fetchware extension due to fetchware's drop_privs mechanism.

See Util's drop_privs() subroutine for more info.

Under drop_privs() build() is executed in the child with reduced privileges.

The $build_path it returns is very important, because the parent process will need to know this path, which is in turn provided to install() as an argument, and install will then chdir to $build_path.

build() API REFERENCE

Below are the subroutines that build() exports with its BUILD_OVERRIDE export tag. run_configure() runs ./configure, and is needed, because AutoTools uses full paths in its configuration files, so if you move it to a different location or a different machine, the paths won't match up, which will cause build and install errors. Rerunning ./configure with the same options as before will recreate the paths to get rid of the errors.

run_star_commands()

run_star_commands(config('*_commands'));

run_star_commands() exists to remove some crazy copy and pasting from build(), install(), and uninstall(). They all loop over the list accessing a ONEARRREF configuration option such as config('build_commands'), and then determine what the individual star_commands are, and then run them with run_prog().

The "star" simply refers to shell globbing with the "star" character * meaning "any."

run_configure()

run_configure();

Runs ./configure as part of build() or uninstall(), which also annoying needs to run it.

NOTE run_configure() is a piece of build() that was chopped out, because uninstall() needs to run ./configure too. The only reason uninstall() must do this is because Autotools uses full paths in the Makefile it creates. Examples from Apache are pasted below.

top_srcdir   = /tmp/fetchware-5506-8ROnNQObhd/httpd-2.2.22
top_builddir = /tmp/fetchware-5506-8ROnNQObhd/httpd-2.2.22
srcdir       = /tmp/fetchware-5506-8ROnNQObhd/httpd-2.2.22
builddir     = /tmp/fetchware-5506-8ROnNQObhd/httpd-2.2.22
VPATH        = /tmp/fetchware-5506-8ROnNQObhd/httpd-2.2.22

Why arn't relative paths good enough for Autotools?

install()

'install succeeded' = install($build_path);

Configuration subroutines used:

install_commands
make_options
prefix

Executes make install, which installs the specified software, or executes whatever install_commands 'install, commands'; if its defined.

install() takes $build_path as its argument, because it must chdir() to this path if fetchware drops privileges.

LIMITATIONS

install() like build() inteligently parses install_commands by split()ing on /,\s+/, and then split()ing again on ' ', and then execute them fetchware's built-in run_prog(), which supports -q.

Also, simply executes the commands you specify or the default ones if you specify none. Fetchware will check if the commands you specify exist and are executable, but the kernel will do it for you and any errors will be in fetchware's output.

drop_privs() NOTES

This section notes whatever problems you might come accross implementing and debugging your Fetchware extension due to fetchware's drop_privs mechanism.

See Util's drop_privs() subroutine for more info.

install() is run in the parent process as root, because most server programs must be installed as root. install() must be called with $build_path as its argument, because it may need to chdir to $build_path, the same path that build() built the package in, in order to be able to install the program.

install() API REFERENCE

Below are the subroutines that install() exports with its INSTALL_OVERRIDE export tag. fillmein!!!!!!!!

chdir_unless_already_at_path()

chdir_unless_already_at_path() takes a $path as its argument, and determines if that path is currently part of the current processes current working directory. If it is, then it does nothing. Buf if the given $path is not in the current working directory, then it is chdir()'d to.

If the chdir() fails, an exception is thrown.

uninstall()

'uninstall succeeded' = uninstall($build_path)

Configuration subroutines used:

uninstall_commands

Cd's to $build_path, and then executes make uninstall, which installs the specified software, or executes whatever uninstall_commands 'uninstall, commands'; if its defined.

LIMITATIONS

uninstall() like install() inteligently parses install_commands by split()ing on /,\s+/, and then split()ing again on ' ', and then execute them fetchware's built-in run_prog(), which supports -q.

drop_privs() NOTES

This section notes whatever problems you might come accross implementing and debugging your Fetchware extension due to fetchware's drop_privs mechanism.

See Util's drop_privs() subroutine for more info.

uninstall() is run in the parent process as root, because most server programs must be uninstalled as root. uninstall() must be called with $build_path as its argument, because it may need to chdir to $build_path, the same path that build() built the package in, in order to be able to uninstall the program.

upgrade()

my $upgrade = upgrade($download_path, $fetchware_package_path)

if ($upgrade) {
    ...
}

Configuration subroutines used:

none

Uses its two arguments ($download_path and $fetchware_package_path) to determine if the new version of your program that has been downloaded (from $download_path) is newer than the currently installed version (from $fetchware_package_path).

Returns true if $download_path is newer than $fetchware_package_path.

Returns false if $download_path is not newer than $fetchware_package_path.

FETCHWARE EXTENSION WARNING: upgrade() must return 0 or '' not undef, because fetchware forks and drop priveleges, and the simple communication scheme between child and parent does not support the Perl concept of undef as undef is special "perlness", and can't just be stored in a string as "undef", and then read back in and magically become a variable that define() says is undef. The string "undef" is not undef.

drop_privs() NOTES

This section notes whatever problems you might come accross implementing and debugging your Fetchware extension due to fetchware's drop_privs mechanism.

See Util's drop_privs() subroutine for more info.

upgrade() is run in the child process as nobody or user, because the child needs to know if it should actually bother running the rest of fetchware's API subroutines.

check_syntax()

'Syntax Ok' = check_syntax()

Configuration subroutines used:

none

Calls check_config_options() to check for the following syntax errors in Fetchwarefiles. Note by the time check_syntax() has been called parse_fetchwarefile() has already parsed the Fetchwarefile, and any syntax errors in the user's Fetchwarefile will have already been reported by Perl.

This may seem like a bug, but it's not. Do you really want to try to use regexes or something to try to parse the Fetchwarefile reliably, and then report errors to users? Or add PPI of all insane Perl modules as a dependency just to write syntax checking code that most of the time says the syntax is Ok anyway, and therefore a complete waste of time and effort? I don't want to deal with any of that insanity.

Instead, check_syntax() uses config() to examine the already parsed Fetchwarefile for "higher level" syntax errors. Syntax errors that are Fetchware syntax errors instead of just Perl syntax errors.

For yours and my own convienience I created check_config_options() helper subroutine. Its data driven, and will check Fetchwarefile's for three different types of common syntax errors that occur in App::Fetchware's Fetchwarefile syntax. These errors are more at the level of logic errors than actual syntax errors. See its POD below for additional details.

Below briefly lists what App::Fetchware's implementation of check_syntax() checks.

Mandatory configuration options
- program, lookup_url, and mirror are required for all Fetchwarefiles.
Conflicting configuration options
- build_commands conflicts with any of prefix, configure_options, and make_options.
Ensures some options have only allowable options specified.
- lookup_method can only have 'timestamp' or 'versionstring'. as options.
- And verify_method can only have 'gpg', 'sha1', or 'md5' specified.

drop_privs() NOTES

This section notes whatever problems you might come accross implementing and debugging your Fetchware extension due to fetchware's drop_privs mechanism.

See Util's drop_privs() subroutine for more info.

check_syntax() is run in the parent process before even start() has run, so no temporary directory is available for use.

check_syntax() API REFERENCE

check_syntax() uses check_config_options() to do the heavy lifting of Set Theory to check that you do not use some configuration options that are not compatible with other ones that you have used.

check_config_options()

check_config_options(
    BothAreDefined => [ [qw(build_commands)],
        [qw(prefix configure_options make_options)] ],
    Mandatory => [ 'program', <<EOM ],
App-Fetchware: Your Fetchwarefile must specify a program configuration
option. Please add one, and try again.
EOM
    Mandatory => [ 'mirror', <<EOM ],
App-Fetchware: Your Fetchwarefile must specify a mirror configuration
option. Please add one, and try again.
EOM
    Mandatory => [ 'lookup_url', <<EOM ],
App-Fetchware: Your Fetchwarefile must specify a lookup_url configuration
option. Please add one, and try again.
EOM
    ConfigOptionEnum => ['lookup_method', [qw(timestamp versionstring)] ],
    ConfigOptionEnum => ['verify_method', [qw(gpg sha1 md5)] ],
);

Uses config() to test that no configuration options that clash with each other are used.

It's parameters are specified in a list with an even number of parameters. Each group of 2 parameters specifies a type of test that check_config_options() will test for. There are three types of tests. Also, note that the parameters are specified as a list not as a hash, because multiple "keys" are allowed, and hash keys must be unique; therefore, the parameters are a list instead of a hash.

BothAreDefined

Is used to test if one or more elements from both provided arrayrefs are defined at the same time. This is needed, because if you specify build_commands any value you specify for prefix, configure_options, make_options will be ignored, which may not be what you expect or want, so BothAreDefined is used to check for this.

Also, unlike Mandatory and ConfigOptionEnum this syntax checker does not take a string argument that specifies an error message, because it takes the two other values that you specifiy, and uses them to fill in its own error message.

Mandatory

Is used to check for mandatory options, which just means that these options absolutely must be specified in user's Fetchwarefiles, and if they are not, then the provided error message is thrown as an exception.

ConfigOptionEnum

Tests that enumerations are valid. For example, lookup_method can only take two values timestamp or versionstring, and ConfigOptionEnum is used to test for this.

end()

end();

Configuration subroutines used:

none

end() is called after all of the other main fetchware subroutines such as lookup() are called. It's job is to cleanup after everything else. It just calls cleanup_tempdir(), which mostly just closes the fetchware.sem fetchware semaphore file used to lock each fetchware temporary directory so fetchware clean does not delete it out from under an concurrently running Fetchware process.

It also calls the very internal only __clear_CONFIG() subroutine that clears App::Fetchware's internal %CONFIG variable used to hold your parsed Fetchwarefile.

EXTENSION OVERRIDE NOTES

end() calls App::Fetchware::Util's cleanup_tempdir() subroutine that cleans up the temporary directory. If your fetchware extension overrides end() or start(), you must call cleanup_tempdir() or name your temproary directories in a manner that fetchware clean won't find them, so something that does not start with fetchware-*.

Use App::Fetchware::Util's create_tempdir() in your start() and cleanup_tempdir() in your end().
Or, be sure not to name your temprorary directory that you create and manage yourself to begin with fetchware-*, which is the glob pattern fetchware clean uses.

drop_privs() NOTES

This section notes whatever problems you might come accross implementing and debugging your Fetchware extension due to fetchware's drop_privs mechanism.

See Util's drop_privs() subroutine for more info.

end() runs in the parent process similar to start(). It needs to be able to close the lockfile that start() created so that fetchware clean cannot delete fetchware's temporary directory out from under it.

FAQ

Why doesn't fetchware and App::Fetchware use OO or Moose?

One of my goals for fetchware was that its guts be pragmatic. I wanted it to consist of a bunch of subroutines that simply get executed in a specific order. And these subroutines should be small and do one and only one thing to make understanding them easier and to make testing them a breeze.

OO is awesome for massive systems that need to take advandtage of inheritance or perhaps composition. But fetchware is small and simple, and I wanted its implementation to also be small and simple. It is mostly just two four thousand line programs with some smaller utility files. If it were OO there would be half or even a whole dozen number of smaller files, and they would have complicated relationships with each other. I did not want to bother with needless abstration that would just make fetchware more complicated. It is a simple command line program, so it should be written as one.

Moose was out, because I don't need any of its amazing features. I could use those amazing features, but fetchware's simple nature does not demand them. Also, unless Moose and its large chunk of dependencies are in your OS's file system cache, Moose based command line apps take a few seconds to start, because perl has to do a bunch of IO on slow disks to read in Moose and its dependencies. I don't want to waste those few seconds. Plus fetchware is a program not intended for use by experienced Perl developers like dzil is, which does use Moose, and has a few second start up overhead, which is acceptable to its author and maintainer. I also use it, and I'm ok with it, but those few seconds might be off putting to users who have no Perl or Moose knowledge.

What's up with the insane fetchware extension mechanism?

Extension systems are always insane. dzil, for example, uses a configuration file where you list names of dzil plugins, and for each name dzil loads that plugin, and figures out what dzil roles it does, then when dzil executes any commands you give it, dzil executes all of those roles that the plugins registered inside you configuration file.

I wanted a configuration file free of declaring what plugins you're using. I wanted it to be easy to use. dzil is for Perl programmers, so it requiring some knowledge of Perl and Moose is ok. But fetchware is for end users or perhaps system administrators not Perl programmers, so something easier is needed.

The extension mechanism was designed for ease of use by people who use your fetchware extension. And it is. Just "use" whatever fetchware extension you want in your Fetchwarefile, and then supply whatever configuration options you need.

This extension mechanism is also very easy for Perl programmers, because you're basically subclassing App::Fetchware, only you do it using App::Fetchware::ExportAPI and App::Fetchware::CreateConfigOptions. See section "Implement your fetchware extension." for full details.

How do I fix the verification failed error.

Fetchware is designed to always attempt to verify the software archives it downloads even if you failed to configure fetchware's verification settings. It will try to guess what those setting should be using simple heuristics. First it will try gpg verificaton, then sha1 verification, and finally md5 verification. If all fail, then fetchware exit failure with an appropriate error message.

When you get this error message read fetchware's documentation on how to set this up.

How do I make fetchware log to a file instead of STDOUT?

You can't fetchware does not have any log file support. However, you can simply redirect STDOUT to a file to make your shell redirect STDOUT to a file for you.

fetchware install <some-program.Fetchwarefile> > fetchware.log

This would not prevent any error messages from STDERR being printed to your screen for that:

fetchware install <some-program.Fetchwarefile> 2>&1 fetchware.log

And to throw away all messages use:

fetchware -q install <some-progra.Fetchwarefile>

or use the shell

fetchware install <some-program.Fetchwarefile 2>&1 /dev/null

Why don't you use Crypt::OpenPGP instead of the gpg command line program?

I tried to use Crypt::OpenPGP, but I couldn't get it to work. And getting gpg to work was a breeze after digging through its manpage to find the right command line options that did what I need it to.

Also, unfortunately Crypt::OpenPGP is buggy, out-of-date, and seems to have lost another maintainer. If it ever gets another maintainer, who fixes the newer bugs, perhaps I'll add support for Crypt::OpenPGP again. Because of how fetchware works it needs to use supported but not popular options of Crypt::OpenPGP, which may be where the bugs preventing it from working reside.

Supporting Crypt::OpenPGP is still on my TODO list. It's just not very high on that list. Patches are welcome to add support for it, and the old code is still there commented out, but it needs updating if anyone is interested.

In the meantime if you're on Windows without simple access to a gpg command line program, try installing gpg from the gpg4win project, which packages up gpg and a bunch of other tools for easier use on Windows.

Does fetchware support Windows?

Yes and no. I intend to support Windows, but right now I'm running Linux, and my Windows virtual machine is broken, so I can't easily test it on Windows. The codebase makes heavy use of File::Spec and Path::Class, so all of its file operations should work on Windows.

I currently have not tested fetchware on Windows. There are probably some test failures on Windows, but Windows support should be just a few patches away.

So the real answer is no, but I intend to support Windows in a future release.

How do I configure a HTTP proxy for use with Fetchware?

Fetchware uses Perl's HTTP::Tiny module to download files over HTTP. So, all you need to do is configure HTTP::Tiny for use with proxies. This is done with environment variables that can easily be set from your shell.

Set http_proxy in the format http://host:port. You can do this permanantly for this session with:

export http_proxy='http://example.com:8080'

Or once just for this invocation of fetchware:

http_proxy='http://example.com:8080 fetchware new

See your OS's shell for more details regarding using and exporting environment variables.

How do I get Fetchware to not delete its temporary directory, so I can troubleshoot a fetchware failure.

Use fetchware's -v (verbose) and --keep-temp command-line options. -v will turn on extra verbose messages that fetchware prints out detailing what its doing and why. These messages will also list where fetchware's temporary directory is. --keep-temp tell fetchware to keep its temporary files and directories and not delete them. This way you can examine them to help troubleshoot the problem, and see what's going on.

--keep-temp should not be needed when the installation or upgrade of a fetchware package or Fetchwarefile fails, because the temp file deletion code will not run after an error occurs before fetchware exits failure.

Use fetchware clean to delete this and any other left over fetchware temporary directories or files.

SECURITY

App::Fetchware is inherently insecure, because its Fetchwarefile's are executable Perl code that actually is executed by Perl. These Fetchwarefiles are capable of doing everything someone can program Perl to do. This is why App::Fetchware will refuse to execute any Fetchwarefile's that are writable by anyone other than the owner who is executing them. It will also exit with a fatal error if you try to use a Fetchwarefile that is not the same user as the one who is running fetchware. These saftey measures help prevent fetchware being abused to get unauthorized code executed on your computer.

App::Fetchware also features the user configuration option that tells fetchware what user you want fetchware to drop privileges to when it does everything but install (install()) and clean up (end()). The configuration option does not tell fetchware to turn on the drop privelege code; that code is always on, but just uses the fairly ubuiquitous nobody user by default. This feature requires the OS to be some version of Unix, because Windows and other OSes do not support the same fork()ing method of limiting what processes can do. On non-Unix OSes, fetchware won't fork() or try to use some other way of dropping privileges. It only does it on Unix. If you use some version of Unix, and do not want fetchware to drop privileges, then specify the stay_root configuration option.

ERRORS

App::Fetchware does not return any error codes; instead, all errors are die()'d if it's App::Fetchware::Config's error, or croak()'d if its the caller's fault. These exceptions are short paragraphs that give full details about the error instead of the vague one liner that perl's own errors give.

BUGS

The official bug tracker for fetchware is its github issues page.

AUTHOR

David Yingling <deeelwy@gmail.com>

COPYRIGHT AND LICENSE

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.

To install App::Fetchware, copy and paste the appropriate command in to your terminal.

cpanm

cpanm App::Fetchware

CPAN shell

perl -MCPAN -e shell
install App::Fetchware

For more information on module installation, please visit the detailed CPAN module installation guide.

	Global
`s`	Focus search bar
`?`	Bring up this help dialog

	GitHub
`g` `p`	Go to pull requests
`g` `i`	go to github issues (only if github is preferred repository)

	POD
`g` `a`	Go to author
`g` `c`	Go to changes
`g` `i`	Go to issues
`g` `d`	Go to dist
`g` `r`	Go to repository/SCM
`g` `s`	Go to source
`g` `b`	Go to file browse

	Search terms
module: (e.g. module:Plugin)
distribution: (e.g. distribution:Dancer auth)
author: (e.g. author:SONGMU Redis)
version: (e.g. version:1.00)

NAME

VERSION

SYNOPSIS

DESCRIPTION

CREATING A App::Fetchware FETCHWAREFILE

MANUALLY CREATING A App::Fetchware FETCHWAREFILE

Fetchware's Fetchwarefile's configuration option syntax

USING YOUR App::Fetchware FETCHWAREFILE WITH FETCHWARE

App::Fetchware'S FETCHWAREFILE CONFIGURATION OPTIONS

program 'Program Name';

filter 'perl regex here';

temp_dir '/tmp';

fetchware_db_path '~/.fetchwaredb';

user 'nobody';

stay_root 'On';

lookup_url 'ftp://somedomain.com/some/path

lookup_method 'timestamp';

gpg_keys_url 'lookup_url.com/some/path';

user_keyring 'On';

gpg_sig_url 'mirror.com/some/path';

sha1_url 'mastermirror.com/some/path';

md5_url 'mastermirror.com/some/path';

user_agent 'Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0'

verify_method 'gpg';

verify_failure_ok 'True';

prefix '/opt/';

configure_options '--datadir=/var/mysql';

make_options '-j4';

build_commands './configure', 'make';

install_commands 'make install';

uninstall_commands 'make uninstall';

no_install 'On';

mirror 'somemirror0.com/some/optional/path';

FURTHER CUSTOMIZING YOUR FETCHWAREFILE

How Fetchware's configuration options are made

The magic of use App::Fetchware;

So how do I add some custom Perl code to customize my Fetchwarefile?

hook()

A real example

EXAMPLE FETCHWAREFILES

Apache Web Server

NGINX Web Server

PHP Programming Language

PHP Programming Language using its git VCS instead of download mirrors.

MariaDB Database

PostgreSQL Database

CREATING A FETCHWARE EXTENSION

How the API works

Extending App::Fetchware

Essential Terminology

API subroutines

override

subclass

Develop your idea keeping in mind fetchware's package manager metaphor

Map your extension's behavior to App::Fetchware's API

Determine your fetchware extension's Fetchwarefile configuration options.

Implement your fetchware extension.

Use Fetchware's Own Libraries to Save Developement Time.

Write your fetchware extension's documentation

Write tests for your fetchware extension

Share it on CPAN

Fetchware Extension Best Practices

FETCHWAREFILE API SUBROUTINES

new()

new() API REFERENCE

extension_name();

opening_message();

fetchwarefile_name();

get_lookup_url()

download_lookup_url()

get_mirrors()

get_verification()

get_filter_option()

prompt_for_other_options()

edit_manually()

new_install()

new_install() API REFERENCE

ask_to_install_now_to_test_fetchwarefile()

start()

lookup()

The magic of `use App::Fetchware;`