/ 
Template-Benchmark-0.99_05
River stage one • 1 direct dependent • 1 total dependent
/ Template::Benchmark::Engine

NAME

Template::Benchmark::Engine - Base class for Template::Benchmark template engine plugins.

SYNOPSIS

package Template::Benchmark::Engines::TemplateSandbox;

use warnings;
use strict;

use base qw/Template::Benchmark::Engine/;

our $VERSION = '0.99_02';

our %feature_syntaxes = (
    literal_text              => <<END_OF_TEMPLATE,
foo foo foo foo foo foo foo foo foo foo foo foo
foo foo foo foo foo foo foo foo foo foo foo foo
foo foo foo foo foo foo foo foo foo foo foo foo
foo foo foo foo foo foo foo foo foo foo foo foo
foo foo foo foo foo foo foo foo foo foo foo foo
END_OF_TEMPLATE
    scalar_variable           =>
        '<: expr scalar_variable :>',
    );

#  rest of module...

DESCRIPTION

Provides a base class for Template::Benchmark template engine plugins, and provides a handy place to document how to write your own plugin.

SUBCLASSING

To write your own Template::Benchmark plugin you'll need to subclass this class (Template::Benchmark::Engine) and put your package in the Template::Benchmark::Engines:: namespace.

The naming convention within that namespace is to strip the :: from the name of the template engine and retain capitalization, thus Template::Sandbox becomes plugin Template::Benchmark::Engines::TemplateSandbox, HTML::Template becomes Template::Benchmark::Engines::HTMLTemplate.

The notable exception is that Template becomes Template::Benchmark::Engines::TemplateToolkit, because everyone calls it Template::Toolkit rather than Template.

Supported or Unsupported?

Throughout the sections below are references to whether a template feature or benchmark type is supported or unsupported in the template engine.

Indicating that something is unsupported is fairly simple, you just return an undef value in the appropriate place, but what constitutes "unsupported"?

It doesn't neccessarily mean that it's impossible to perform that task with the given template engine, but generally if it requires some significant chunk of DIY code or boilerplate or subclassing by the developer using the template engine, it should be considered to be unsupported by the template engine itself.

This of course is a subjective judgement, but a general rule of thumb is that if you can tell the template engine to do it, it's supported; and if the template engine allows you to do it, it's unsupported, even though it's possible.

Methods To Subclass

$template_snippet = Plugin->feature_syntax( $template_feature )

Your plugin doesn't need to provide this method directly, it can be inherited from Template::Benchmark::Engine where it will access the %feature_syntaxes variable in your plugin's namespace, using $template_feature as a key.

Obviously %feature_syntaxes can't be a private variable for this to work, so declare it as a global or with our.

For example:

our %feature_syntaxes = (
    literal_text              => <<END_OF_TEMPLATE,
foo foo foo foo foo foo foo foo foo foo foo foo
foo foo foo foo foo foo foo foo foo foo foo foo
foo foo foo foo foo foo foo foo foo foo foo foo
foo foo foo foo foo foo foo foo foo foo foo foo
foo foo foo foo foo foo foo foo foo foo foo foo
END_OF_TEMPLATE
    scalar_variable           =>
        '<: expr scalar_variable :>',
    hash_variable_value       =>
        '<: expr hash_variable.hash_value_key :>',
# ...
    );

Any feature that isn't supported by the template engine should have an undef value.

Please see the section "Feature Syntaxes" for a list of the different template features and what their requirements are.

$descriptions = Plugin->benchmark_descriptions()

This method must be defined, to return a hashref of benchmark function names to a useful description, for example:

sub benchmark_descriptions
{
    return( {
        TS    =>
            "Template::Sandbox ($Template::Sandbox::VERSION) without caching",
        TS_CF =>
            "Template::Sandbox ($Template::Sandbox::VERSION) with " .
            "Cache::CacheFactory ($Cache::CacheFactory::VERSION) caching",
        TS_CHI =>
            "Template::Sandbox ($Template::Sandbox::VERSION) with " .
            "CHI ($CHI::VERSION) caching",
        TS_FMM =>
            "Template::Sandbox ($Template::Sandbox::VERSION) with " .
            "Cache::FastMmap ($Cache::FastMmap::VERSION) caching",
        } );
}

It's generally very useful to include version numbers like the example, don't waste half an hour figuring out why your benchmarks are slow before you realise it's benchmarking the last version from CPAN rather than your fancy new development copy, all because you forgot to set your library path.

Uh, not that I'd make such a basic mistake.

The name of the different benchmarks shouldn't clash with anyone else's otherwise things will get confusing, and an underscore (_) should only be used to distinguish between different functions returned by the same plugin.

Best bet is to look at the plugins already written and choose something written along similar lines, but distinctive to your template engine, it needs to be short though because it will be being used as a column heading and row title for the benchmark results, and they're pretty wide already.

The convention I've used is to use initials of the package's namespace, T for Template::, H for HTML::, Te for Text::, then sufficient initials to disambiguate the rest of the package name.

So Template::Toolkit is TT, Template::Sandbox is TS, HTML::Template is HT, Text::Template is TeTe as opposed to Text::Tmpl being TeTmpl.

Additional initials might be added if the template engine can accept diffent template syntaxes, which is handled by several plugins:

Template::Alloy gets TATT (running in Template::Toolkit mode) and TAHT (running in HTML::Template mode), and if you want long-winded there's TeMMTeTe - Text::MicroMason running in Text::Template mode.

It should be obvious by now why there needs to be a nice clear description to accompany these names.

Within a plugin, if there are several different configuration options, caching choices, or other tweaks to be benchmarked, this is indicated by a suffix after an underscore (_).

So Template::Sandbox using Cache::CacheFactory for caching becomes TS_CF, and using CHI for caching is TS_CHI, this lets you easily see within the results that both TS_CF and TS_CHI are produced using the same template engine plugin.

$template_functions = Plugin->benchmark_functions_for_uncached_string()

This method needs to return a hashref of names to benchmark function references, if the benchmark type is unsupported it should return undef.

Each name needs to be listed in the hashref returned from Plugin->benchmark_descriptions().

Each benchmark function needs to accept the contents of the template as the first argument and then two hashrefs of template variables to set.

The benchmark function should return the content of the processed template.

For example:

sub benchmark_functions_for_uncached_string
{
    my ( $self ) = @_;

    return( {
        TS =>
            sub
            {
                my $t = Template::Sandbox->new();
                $t->set_template_string( $_[ 0 ] );
                $t->add_vars( $_[ 1 ] );
                $t->add_vars( $_[ 2 ] );
                ${$t->run()};
            },
        } );
}

Please see the section "Benchmark Types" for a list of the different benchmark types and what restrictions apply to the benchmark functions in each.

$template_functions = Plugin->benchmark_functions_for_disk_cache( $template_dir, $cache_dir )
$template_functions = Plugin->benchmark_functions_for_shared_memory_cache( $template_dir, $cache_dir )
$template_functions = Plugin->benchmark_functions_for_memory_cache( $template_dir, $cache_dir )
$template_functions = Plugin->benchmark_functions_for_instance_reuse( $template_dir, $cache_dir )

Each of these methods need to return a hashref of names to benchmark function references like Plugin->benchmark_functions_for_uncached_string(), however they have slightly different arguments. If the benchmark type is unsupported it should return undef.

$template_dir provides you with the location of the temporary directory in which the template will be written.

$cache_dir provides you with the location of the temporary directory created for any cache you wish to create.

Note that the cache directory is unique to your plugin, however it is shared for every benchmark function returned by your plugin, if you have multiple benchmark functions you will need to set things up yourself within that cache directory to prevent them stomping on each-other's toes.

Each benchmark function needs to accept the leaf filename of the template as the first argument and then two hashrefs of template variables to set.

The leaf filename is the final bit after the directory if you're wondering, so if the template was /tmp/KJJFKav/TemplateSandbox/TemplateSandbox.txt then you'd get /tmp/KJJFKav/TemplateSandbox as $template_dir, and TemplateSandbox.txt passed as the first argument to your benchmark function.

The benchmark function should return the content of the processed template.

For example:

sub benchmark_functions_for_disk_cache
{
    my ( $self, $template_dir, $cache_dir ) = @_;
    my ( $cf, $chi );

    $cf = Cache::CacheFactory->new(
        storage    => { 'file' => { cache_root => $cache_dir, }, },
        );
    $chi = CHI->new(
        driver   => 'File',
        root_dir => $cache_dir,
        );

    return( {
        TS_CF =>
            sub
            {
                my $t = Template::Sandbox->new(
                    cache         => $cf,
                    template_root => $template_dir,
                    template      => $_[ 0 ],
                    ignore_module_dependencies => 1,
                    );
                $t->add_vars( $_[ 1 ] );
                $t->add_vars( $_[ 2 ] );
                ${$t->run()};
            },
        TS_CHI =>
            sub
            {
                my $t = Template::Sandbox->new(
                    cache         => $chi,
                    template_root => $template_dir,
                    template      => $_[ 0 ],
                    ignore_module_dependencies => 1,
                    );
                $t->add_vars( $_[ 1 ] );
                $t->add_vars( $_[ 2 ] );
                ${$t->run()};
            },
        } );
}

Please see the section "Benchmark Types" for a list of the different benchmark types and what restrictions apply to the benchmark functions in each.

Benchmark Types

Comparing a template engine that's running with a memory cache to a completely uncached engine is like comparing apples with oranges, so each benchmark type is designed to simulate a different environment in which the template engine is running, this lets Template::Benchmark group results so that a fair comparison can be made between engines at performing a similar task.

With this in mind, each benchmark type has its own restrictions that should be adhered to when writing a plugin.

Common to each benchmark type is the requirement to accept two seperate hashrefs of template variables, and behave as if they might be different between invocations of the benchmark function. (Currently the contents of the variable hashrefs do not change, however that may change in future versions, and regardless, they should be treated as if they have changed each time.)

uncached_string

This benchmark type explicitly disallows caching of any kind, and must take the template as the supplied scalar value and process it "from scratch" each time.

This broadly simulates running in an uncached CGI environment if you're thinking of web applications, or the performance of a cache-miss in a cached environment.

disk_cache

This benchmark type requires that the template be read from disk, from the filename given, and may cache intermediate stages on the disk too.

No template data may be kept in-memory between invocations.

This broadly simulates running in a CGI environment with a disk cache to store compiled templates between requests.

shared_memory_cache

This benchmark type requires that the template be read from disk, from the filename given, and may cache intermediate stages in shared memory.

No template data may be kept in non-shared memory between invocations.

It's quite normal for template engines not to provide shared memory support, so not many plugins provide this benchmark type.

This broadly simulates running in a mod_perl environment with the templates loaded into a shared memory cache before the webserver forks.

memory_cache

This benchmark type requires that the template be read from disk, from the filename given, and may cache intermediate stages in memory.

While template data may be stored in-memory, it must be accessed by instantiating a new copy of the template engine with the provided filename, and not simply by reusing an instance or compiled subroutine reference from a previous run.

ie: The benchmark function itself should store no stateful information.

This broadly simulates running in a mod_perl environment without using shared memory.

An example:

sub benchmark_functions_for_memory_cache
{
    my ( $self, $template_dir, $cache_dir ) = @_;
    my ( @template_dirs );

    @template_dirs = ( $template_dir );

    return( {
        HT =>
            sub
            {
                my $t = HTML::Template->new(
                    type              => 'filename',
                    path              => \@template_dirs,
                    source            => $_[ 0 ],
                    case_sensitive    => 1,
                    cache             => 1,
                    die_on_bad_params => 0,
                    );
                $t->param( $_[ 1 ] );
                $t->param( $_[ 2 ] );
                $t->output();
            },
        } );
}

As can be seen, on each invocation the same code is run, with no stateful information carried between invocations.

instance_reuse

This benchmark type requires that the template be read from disk, from the filename given, and may cache intermediate stages in memory.

This benchmark type should be provided only if there is some degree of reuse of data-structures from a previous invocation, such as reusing a previously-created template instance, or a compiled subroutine reference, by the benchmark function itself.

For example, if the benchmark function instantiates a copy of the template engine on the first run, loading the filename given, and then stores that instance in a local variable, then on subsequent invocations reuses that instance rather than starting from the beginning each time.

Or, an instance is created, the template loaded and compiled to a subroutine reference, that reference is stored, and subsequent invocations resume from that point.

Some examples:

sub benchmark_functions_for_instance_reuse
{
    my ( $self, $template_dir, $cache_dir ) = @_;
    my ( $tt, $tt_x, $tt_xcet, @template_dirs );

    @template_dirs = ( $template_dir );

    $tt     = Template->new(
        STASH        => Template::Stash->new(),
        INCLUDE_PATH => \@template_dirs,
        );
    return( {
        TT =>
            sub
            {
                my $out;
                $tt->process( $_[ 0 ], { %{$_[ 1 ]}, %{$_[ 2 ]} }, \$out );
                $out || $tt->error();
            },
        } );
}

This example shows the Template::Toolkit insance being reused between invocations.

sub benchmark_functions_for_instance_reuse
{
    my ( $self, $template_dir, $cache_dir ) = @_;
    my ( $t );

    return( {
        TeMMHM =>
            sub
            {
                $t = Text::MicroMason->new()->compile(
                    file => File::Spec->catfile( $template_dir, $_[ 0 ] )
                    )
                    unless $t;
                $t->( ( %{$_[ 1 ]}, %{$_[ 2 ]} ) );
            },
        } );
}

And this example shows that an intermediate stage is preserved as a function reference on the first invocation, then subsequent invocations just invoke that reference.

This benchmark type simulates running in a mod_perl environment with some form of memory caching, but the end-user of the template system would need to write some DIY caching themselves, and the benchmark doesn't include the overhead of just what that caching might be.

Template Features

Different template engines support different template features, so Template::Benchmark allows the person performing the benchmarks to mix-and-match the features they wish to benchmark.

Those template engines that support the feature will be benchmarked and the end-user will be informed of which template engines didn't support which features.

To this end, Template::Benchmark queries the plugin for the template syntax required to implement each feature. That is, to generate the correct output from the given template variables, in the correct manner.

To ensure that like-for-like comparisons are being made, there are several variants of some basic template features, aimed to reflect nuances of common use.

literal

A chunk of literal text, dumped through to the output largely unchanged from its form in the template. ("Largely unchanged" means unescaping backslashes or the equivilent is fine.)

The block of literal text to be used is:

foo foo foo foo foo foo foo foo foo foo foo foo
foo foo foo foo foo foo foo foo foo foo foo foo
foo foo foo foo foo foo foo foo foo foo foo foo
foo foo foo foo foo foo foo foo foo foo foo foo
foo foo foo foo foo foo foo foo foo foo foo foo
scalar_variable

Interpolation of a template variable named scalar_variable.

hash_variable_value

Interpolation of a template variable stored in the hashref named hash_variable with key 'hash_value_key'.

array_variable_value

Interpolation of a template variable stored in the arrayref named array_variable with index 2.

deep_data_structure_value

Interpolation of a template variable stored in the hashref named this with nesteed keys 'is', 'a', 'very', 'deep', 'hash', 'structure'.

This feature is designed to stress the speed that the template engine traverses deep data-structures.

array_loop_value
array_loop_template

Loop through the arrayref template variable named array_loop, inserting each element into the template output in turn.

No delimiter is expected so if array_loop had value

[ 'one', 'two', 'three' ]

the output would look like

onetwothree

The reason there's no delimiter between records is to keep the template simple and to avoid any situations where differing behaviour creeps in from different template engines: for example if a newline was output after each element, some template engines would insert (or trim) additional white space as part of the flow control block, and while there may be ways to configure that behaviour within the template engine, that constitutes doing additional work over that done by other engines and would skew the benchmark's result away from being just the cost of doing the loop.

The _value version of this template feature permits any method of generating the content containing the value from the array, whereas the _template version requires that the output be produced by a block of template.

An example of this distinction would be from Template::Benchmark::Engines::TextTemplate:

array_loop_value          =>
    '{ $OUT .= $_ foreach @array_loop; }',

While a loop can be executed in the embedded perl, it can only build a literal string to be inserted back into the template output, there is no way to say that the loop means 'repeat this section of template' like that allowed, for example, in Template::Toolkit:

array_loop_template       =>
    '[% FOREACH i IN array_loop %][% i %][% END %]',

While it may be possible to coerce some embedded perl examples to do something similar by creating a new template engine instance and running a template fragment on it, that falls into the realms of DIY solutions discussed in "Supported or Unsupported?"

This distinction is important because it determines how easy it is to have large repeated sections of template without having to fall back to generating them within perl (which is presumably what you were trying to avoid by using a template system in the first place.)

hash_loop_value
hash_loop_template

Loop through the hashref template variable named hash_loop, in alphabetic order of the keys, inserting each key and value into the template output.

The key and value should be seperated by ': ' but between key/value pairs there's no delimiter.

{ 'one' => 1, 'two' => 2, 'three' => 3 }

would produce

one: 1three: 3two: 2

The _value and _template versions of this template feature follow the same rules as documented for array_loop_value and array_loop_template.

records_loop_value
records_loop_template

Loop across an arrayref of hashrefs, much like that returned from a DBI fetchall_arrayref( {} ), for each 'record' output the value of the 'name' and 'age' keys.

As with hash_loop_value, a ': ' seperates name from age, and no delimiter between records.

[
  { name => 'Andy MacAndy',  age => 12, },
  { name => 'Joe Jones',     age => 10, },
  { name => 'Jenny Jenkins', age => 11, },
]

would give

Andy MacAndy: 12Joe Jones: 10Jenny Jenkins: 11

The _value and _template versions of this template feature follow the same rules as documented for array_loop_value and array_loop_template.

constant_if_literal
constant_if_template

Conditionally choose to insert some content if a constant literal 1 is true.

In the case of constant_if_literal the content is the literal text 'true' and for constant_if_template the content is the result of a template block inserting the content of template variable template_if_true.

The distinction between _literal and _template versions of this test are similar to those between array_loop_value and array_loop_template: the _template version must result from executing a block of the template markup rather than perl string manipulation.

variable_if_literal
variable_if_template

Conditionally choose to insert some content if the template variable variable_if is true.

In the case of variable_if_literal the content is the literal text 'true' and for variable_if_template the content is the result of a template block inserting the content of template variable template_if_true.

The distinction between _literal and _template versions of this test are similar to those between array_loop_value and array_loop_template: the _template version must result from executing a block of the template markup rather than perl string manipulation.

constant_if_else_literal
constant_if_else_template

Conditionally choose to insert some content if a constant literal 1 is true, or some other content if it's false.

In the case of constant_if_else_literal the content is the literal text 'true' for true, and 'false' for false, and for constant_if_else_template the content is the result of a template block inserting the content of template variable template_if_true if true, or template_if_false if false.

The distinction between _literal and _template versions of this test are similar to those between array_loop_value and array_loop_template: the _template version must result from executing a block of the template markup rather than perl string manipulation.

variable_if_else_literal
variable_if_else_template

Conditionally choose to insert some content if the template variable variable_if_else is true, or some other content if it's false.

In the case of variable_if_else_literal the content is the literal text 'true' for true, and 'false' for false, and for variable_if_else_template the content is the result of a template block inserting the content of template variable template_if_true if true, or template_if_false if false.

The distinction between _literal and _template versions of this test are similar to those between array_loop_value and array_loop_template: the _template version must result from executing a block of the template markup rather than perl string manipulation.

constant_expression

Insert the result of the constant expression 10 + 12.

Note that the template should actually calculate this, don't just put a literal 22 in the template, as the purpose of this feature is to determine if constants are subjected to constant-folding optimizations by the template engine and to give some indication of what gains are made by the engine in that situation.

variable_expression

Insert the result of multiplying the template variables variable_expression_a and variable_expression_b, ie doing:

variable_expression_a * variable_expression_b
complex_variable_expression

Insert the result of the following expression:

( ( variable_expression_a * variable_expression_b ) +
  variable_expression_a - variable_expression_b ) /
variable_expression_b

Note that the brackets should be included, even if the template engine would sort out precedence correctly, because processing of brackets and precedence is part of what is being benchmarked by this feature.

Also note that the values of variable_expression_a and variable_expression_b are chosen so that the entire operation acts on integers and results in an integer value, so there is no need to worry about different floating-point precisions or output formats.

This feature is intended to be a slightly more stressful version of variable_expression, to allow comparision between the two results to isolate the expression engine performance of a template engine.

constant_function

Perform a function call (or equivilent, such as vmethod) within a template expression, on a constant literal.

The expression should do the equivilent of the perl:

substr( 'this has a substring', 11, 9 )

Like the difference between constant_expression and variable_expression this is to detect/benchmark any constant-folding optimizations.

variable_function

Perform a function call (or equivilent, such as vmethod) within a template expression, on the template variable variable_function_arg.

The expression should do the equivilent of the perl:

substr( $variable_function_arg, 4, 2 )

AUTHOR

Sam Graham, <libtemplate-benchmark-perl at illusori.co.uk>

BUGS

Please report any bugs or feature requests to bug-template-benchmark at rt.cpan.org, or through the web interface at http://rt.cpan.org/NoAuth/ReportBug.html?Queue=Template-Benchmark. I will be notified, and then you'll automatically be notified of progress on your bug as I make changes.

SUPPORT

You can find documentation for this module with the perldoc command.

perldoc Template::Benchmark::Engine

You can also look for information at:

ACKNOWLEDGEMENTS

Thanks to Paul Seamons for creating the the bench_various_templaters.pl script distributed with Template::Alloy, which was the ultimate inspiration for this module.

COPYRIGHT & LICENSE

Copyright 2010 Sam Graham.

This program is free software; you can redistribute it and/or modify it under the terms of either: the GNU General Public License as published by the Free Software Foundation; or the Artistic License.

See http://dev.perl.org/licenses/ for more information.