NAME

PDL::Stats::Basic -- basic statistics and related utilities

DESCRIPTION

The terms FUNCTIONS and METHODS are arbitrarily used to refer to methods that are threadable and methods that are NOT threadable, respectively.

Does not have mean or median function here. see SEE ALSO.

SYNOPSIS

use PDL::LiteF;
use PDL::NiceSlice;
use PDL::Stats::Basic;

my $stdv = $data->stdv;

my $stdv = stdv( $data );

FUNCTIONS

stdv

Signature: (a(n); float+ [o]b())

Sample standard deviation.

stdv does handle bad values. It will set the bad-value flag of all output piddles if the flag is set for any of the input piddles.

stdv_unbiased

Signature: (a(n); float+ [o]b())

Unbiased estimate of population standard deviation.

stdv_unbiased does handle bad values. It will set the bad-value flag of all output piddles if the flag is set for any of the input piddles.

var

Signature: (a(n); float+ [o]b())

Sample variance.

var does handle bad values. It will set the bad-value flag of all output piddles if the flag is set for any of the input piddles.

var_unbiased

Signature: (a(n); float+ [o]b())

Unbiased estimate of population variance.

var_unbiased does handle bad values. It will set the bad-value flag of all output piddles if the flag is set for any of the input piddles.

se

Signature: (a(n); float+ [o]b())

Standard error of the mean. Useful for calculating confidence intervals.

  # 95% confidence interval for samples with large N

  $ci_95_upper = $data->average + 1.96 * $data->se;
  $ci_95_lower = $data->average - 1.96 * $data->se;

se does handle bad values. It will set the bad-value flag of all output piddles if the flag is set for any of the input piddles.

ss

Signature: (a(n); float+ [o]b())

Sum of squared deviations from the mean.

ss does handle bad values. It will set the bad-value flag of all output piddles if the flag is set for any of the input piddles.

skew

Signature: (a(n); float+ [o]b())

Sample skewness, measure of asymmetry in data. skewness == 0 for normal distribution.

skew does handle bad values. It will set the bad-value flag of all output piddles if the flag is set for any of the input piddles.

skew_unbiased

Signature: (a(n); float+ [o]b())

Unbiased estimate of population skewness. This is the number in GNumeric Descriptive Statistics.

skew_unbiased does handle bad values. It will set the bad-value flag of all output piddles if the flag is set for any of the input piddles.

kurt

Signature: (a(n); float+ [o]b())

Sample kurtosis, measure of "peakedness" of data. kurtosis == 0 for normal distribution.

kurt does handle bad values. It will set the bad-value flag of all output piddles if the flag is set for any of the input piddles.

kurt_unbiased

Signature: (a(n); float+ [o]b())

Unbiased estimate of population kurtosis. This is the number in GNumeric Descriptive Statistics.

kurt_unbiased does handle bad values. It will set the bad-value flag of all output piddles if the flag is set for any of the input piddles.

cov

Signature: (a(n); b(n); float+ [o]c())

Sample covariance. see corr for ways to call

cov does handle bad values. It will set the bad-value flag of all output piddles if the flag is set for any of the input piddles.

corr

Signature: (a(n); b(n); float+ [o]c())

Pearson correlation coefficient. r = cov(X,Y) / (stdv(X) * stdv(Y)).

Usage:

perldl> $a = random 5, 3
perldl> $b = sequence 5,3
perldl> p $a->corr($b)

[0.20934208 0.30949881 0.26713007]

for square corr table

perldl> p $a->corr($a->dummy(1))

[
 [           1  -0.41995259 -0.029301192]
 [ -0.41995259            1  -0.61927619]
 [-0.029301192  -0.61927619            1]
]

but it is easier and faster to use corr_table.

corr does handle bad values. It will set the bad-value flag of all output piddles if the flag is set for any of the input piddles.

corr_table

Signature: (a(n,m); float+ [o]c(m,m))

Square Pearson correlation table. Gives the same result as threading using corr but it calculates only half the square, hence much faster. And it is easier to use with higher dimension pdls.

Usage:

   # 5 obs x 3 var, 2 such data tables

   perldl> $a = random 5, 3, 2
   
   perldl> p $a->corr_table
   [
    [
    [          1 -0.69835951 -0.18549048]
    [-0.69835951           1  0.72481605]
    [-0.18549048  0.72481605           1]
   ]
   [
    [          1  0.82722569 -0.71779883]
    [ 0.82722569           1 -0.63938828]
    [-0.71779883 -0.63938828           1]
    ]
   ]

for the same result using corr,

perldl> p $a->dummy(2)->corr($a->dummy(1))

This is also how to use t_corr and n_pair with such a table.

corr_table does handle bad values. It will set the bad-value flag of all output piddles if the flag is set for any of the input piddles.

t_corr

Signature: (r(); n(); [o]t())

$corr   = $data->corr( $data->dummy(1) );
$n      = $data->n_pair( $data->dummy(1) );
$t_corr = $corr->t_corr( $n );

use PDL::GSL::CDF;

$p_2tail = 2 * (1 - gsl_cdf_tdist_P( $t_corr->abs, $n-2 ));

t significance test for Pearson correlations.

t_corr does handle bad values. It will set the bad-value flag of all output piddles if the flag is set for any of the input piddles.

n_pair

Signature: (a(n); b(n); int [o]c())

Returns the number of good pairs between 2 lists. Useful with corr (esp. when bad values are involved)

n_pair does handle bad values. It will set the bad-value flag of all output piddles if the flag is set for any of the input piddles.

corr_dev

Signature: (a(n); b(n); float+ [o]c())

$corr = $a->dev_m->corr_dev($b->dev_m);

Calculates correlations from dev_m vals. Seems faster than doing corr from original vals when data pdl is big

corr_dev does handle bad values. It will set the bad-value flag of all output piddles if the flag is set for any of the input piddles.

t_test

Signature: (a(n); b(m); float+ [o]t(); [o]d())

my ($t, $df) = t_test( $pdl1, $pdl2 );

use PDL::GSL::CDF;

my $p_2tail = 2 * (1 - gsl_cdf_tdist_P( $t->abs, $df ));

Independent sample t-test, assuming equal var.

t_test does handle bad values. It will set the bad-value flag of all output piddles if the flag is set for any of the input piddles.

t_test_nev

Signature: (a(n); b(m); float+ [o]t(); [o]d())

Independent sample t-test, NOT assuming equal var. ie Welch two sample t test. Df follows Welch-Satterthwaite equation instead of Satterthwaite (1946, as cited by Hays, 1994, 5th ed.). It matches GNumeric, which matches R.

  my ($t, $df) = $pdl1->t_test( $pdl2 );

t_test_nev does handle bad values. It will set the bad-value flag of all output piddles if the flag is set for any of the input piddles.

t_test_paired

Signature: (a(n); b(n); float+ [o]t(); [o]d())

Paired sample t-test.

t_test_paired does handle bad values. It will set the bad-value flag of all output piddles if the flag is set for any of the input piddles.

METHODS

get_data

Reads either file or file handle*. Returns observation x variable pdl and var and obs ids if specified. Ids in perl @ ref to allow for non-numeric ids. Other non-numeric entries are treated as missing, which are filled with $opt{MISSN} then set to BAD*. Can specify num of data rows to read from top but not arbitrary range.

*If passed handle, it will not be closed here.

*PDL::Bad::setvaltobad only works consistently with the default TYPE double before PDL-2.4.4_04.

Default options (case insensitive):

V       => 1,        # prints simple status
TYPE    => double,
C_ID    => 1,
R_ID    => 1,
R_VAR   => 0,        # set to 1 if var in rows
SEP     => "\t",     # can take regex qr//
MISSN   => -999,
NROW    => '',

Usage:

($data, $idv, $ido) = get_data( \*STDIN, { TYPE=>long } );

$data = get_data( 'zcat big_data.txt.gz |' );

which_id

Lookup specified var (obs) id in $idv ($ido) (see get_data) and return index in $idv ($ido) as pdl if found. Useful for selecting data by var (obs) id.

my $ind = which_id $ido, ['smith', 'summers', 'tesla'];

my $data_subset = $data( $ind, );

# take advantage of perl pattern matching
# e.g. use data from people whose last name starts with s

my $i = which_id $ido, [ grep { /^s/ } @$ido ];

my $data_s = $data($i, );

REFERENCES

Hays, W.L. (1994). Statistics (5th ed.). Fort Worth, TX: Harcourt Brace College Publishers.

AUTHOR

To install PDL::Stats, copy and paste the appropriate command in to your terminal.

cpanm

cpanm PDL::Stats

CPAN shell

perl -MCPAN -e shell
install PDL::Stats

For more information on module installation, please visit the detailed CPAN module installation guide.

	Global
`s`	Focus search bar
`?`	Bring up this help dialog

	GitHub
`g` `p`	Go to pull requests
`g` `i`	go to github issues (only if github is preferred repository)

	POD
`g` `a`	Go to author
`g` `c`	Go to changes
`g` `i`	Go to issues
`g` `d`	Go to dist
`g` `r`	Go to repository/SCM
`g` `s`	Go to source
`g` `b`	Go to file browse

	Search terms
module: (e.g. module:Plugin)
distribution: (e.g. distribution:Dancer auth)
author: (e.g. author:SONGMU Redis)
version: (e.g. version:1.00)

NAME

DESCRIPTION

SYNOPSIS

FUNCTIONS

stdv

stdv_unbiased

var

var_unbiased

se

ss

skew

skew_unbiased

kurt

kurt_unbiased

cov

corr

corr_table

t_corr

n_pair

corr_dev

t_test

t_test_nev

t_test_paired

METHODS

get_data

which_id

SEE ALSO

REFERENCES

AUTHOR

Module Install Instructions