NAME
Smolder::Manual - How to use Smolder more effectively
UPLOAD FILE FORMAT
This describes the file format for test reports expected by Smolder. This report file can either be uploaded using the web interface or the smolder_smoke_signal
utility for automating the upload.
TAP
TAP is the "Test Anything Prototcol" and is the basic format behind all test reports. Whatever language you use for your testing or whatever infrastructure or harness you have running your tests, if it can output TAP then it can work with Smolder. There are TAP emiters for almost all currently popular languages (C, Java, PHP, Perl, Python, Ruby, etc). And even if your language of choice doesn't have an existing TAP emiter, TAP is such a simple protocol that it's a fairly trivial task to write one.
The full documentation for TAP can be found at http://search.cpan.org/perldoc?TAP with more information available at http://testanything.org.
The basic idea is that a single file of TAP output is equivalent to a single test script or module. These files are collected together in a tar (optionally compressed with gzip) or a zip archive. This archive file can have an arbitrary number of these TAP files (the file extension doesn't matter, but you can use .tap if you need to use something) in an arbitrary number of directories (nice for grouping and organizing your tests).
Simple Example
Here's a quick example of using a few bash
commands to create a Smolder test report file from a typical Perl test suite (typical Perl test suites use Test::Builder which output TAP) :
$] foreach i in t/*.t; do perl $i > `basename .t`.tap; done
$] tar -zc *.tap > test_run.tar.gz
YAML
Pretty straight forward. Now, there are some additional details which you may or may not want Smolder to keep track of. What order did my test files run in? How long did the test suite take to run? Since these are questions about the whole test run in general it's not possible to really associate them with any one test's output.
This is why we also allow this test suite meta information to be presented to Smolder in a small and simple YAML file. This file is usually named meta.yml, but you can use anything really as long as there's only one file with the .yml extension. It looks something like this:
---
file_order:
- foo.tap
- bar.tap
- extra/baz.tap
- extra/froob.tap
start_time: 1178159475
stop_time: 1178159983
Since this entire file is optional, if you do include it you can include which ever parts are of interest to you. Smolder can care about the following keys of the YAML map:
file_order
This is a list of the TAP files from your test run in the order you desire to display them. This will usually be the order in which they are run. The path to the TAP file is relative to the directory of the TAP archive.
start_time
The time the test suite started to run. This time is given in epoch seconds (seconds since 01/01/1970).
stop_time
The time the test suite finished its run. This time is given in epoch seconds (seconds since 01/01/1970).
Full Example
Here's a slightly more complicated example (written in Perl) that demonstates not only saving the TAP output to files, but also generating a meta.yml file to go along with it. We do this with the TAP::Harness::Archive module which does most of the work for us:
#!/usr/bin/perl
use warnings;
use strict;
use TAP::Harness::Archive;
my @files = glob('t/*.t');
my $harness = TAP::Harness::Archive->new({
lib => [ 'lib', 'blib/lib', 'blib/arch' ],
archive => 'my_test_run.tar.gz',
});
$harness->runtests(@files);
Or you can use the runtests utility that comes with TAP::Harness to this with the --archive
or -a
arguments.
]$ runtests t/*.t --archive my_test_run.tar.gz
RUNNING YOUR TESTS
Smolder tries as hard as possible to not dictacte how you write or structure your tests. It even tries hard not to care how you run your tests too much. But it does need some help to get the right information in the right format (a TAP Archive)
Running
prove
The easiest way to run your tests for Smolder is to use the prove
utility that comes with the Test::Harness Perl module. It has a --archive
option that tells it to generate an archive of the TAP files.
]$ prove --archive my_test_run.tar.gz
Module::Build::TAPArchive
If you are using Module::Build in Perl you could instead use the Module::Build::TAPArchive subclass which provides an extra build action test_archive
.
]$ perl Build.PL && ./Build test_archive --archive_file my_test_run.tar.gz
Full Diagnostic Messages
Many times a failing test will output diagnostics messages giving more information (ie, expected vs. received return values, etc). TAP output is expected to be on STDOUT, but diagnostic information is usually on STDERR. To capture all of this, simply use the --merge
flag for prove
.
SmokeRunner::Multi
Sometimes you have many different projects or different branches of the same project that you would like to test from an SVN source checkout. The Perl module SmokeRunner::Multi was designed for just this purpose.
It can be used simply to run tests and output them to your screen, or create TAP archives and even automatically upload it to a running Smolder server.
Buildbot
Buildbot (http://buildbot.net) is a neat Python tool that makes creating a build farm nice and easy. With it you can setup a master build server that has multiple slave build servers (running different platforms and architectures). This is overkill for lots of scenarios, but is really nice if your project needs to run on lots of systems.
Having Buildbot submit the test results is essentially the same as anything else. You need to create a TAP archive and have it uploaded to Smolder.
Uploading to Smolder
The easiest way to automate the uploading of test results to a running Smolder is to use the smolder_smoke_signal utiltiy that comes with Smolder. It takes a TAP archive file and uploads it to the Smolder server of your choice.
]$ smolder_smoke_signal --server smolder.foo.com \
--username myself --password s3cr3t --file test_report.tar.gz \
--project MyProject
Or you can use the LWP
Perl library directly (and there's probably a way to do it just using curl
from the command line.
LWP::UserAgent->new()->post(
'http://smolder.project.org/app/projects/add_report/$project_id',
Content_Type => 'form-data',
Content => [
architecture => '386',
platform => 'Linux',
comments => $comments,
username => 'my-user',
password => 's3cr3t',
report_file => ['tap_archive.tar.gz'],
]
);
Altogether Now
Typically, your automated test/upload code can be a bash script as simple as this that you can run from cron:
!#/bin/bash
prove --archive test_run.tar.gz
smolder_smoke_signal --server smolder.foo.com \
--username myself --password s3cr3t --file test_run.tar.gz \
--project MyProject
Pretty easy.
CONFIGURATION
TODO