NAME

recs-decollate

recs-decollate --help-all

Help from: --help-basic:
Usage: recs-decollate <args> [<files>]
   Decollate records of input (or records from <files>) into output records.

Arguments:
   --dldeaggregator ...                Specify a domain language aggregate. See "Domain Language Integration" below.
   --deaggregator|-d <deaggregators>   Colon separated list of aggregate field specifiers. See "Deaggregates" section below.
   --list-deaggregators                Bail and output a list of deaggregators.
   --show-deaggregator <deaggregator>  Bail and output this deaggregator's detailed usage.
   --filename-key|fk <keyspec>         Add a key with the source filename (if no filename is applicable will put NONE)

  Help Options:
      --help-all             Output all help for this script
      --help                 This help screen
      --help-deaggregators   List the deaggregators
      --help-domainlanguage  Help on the recs domain language, a [very complicated] way of specifying valuations (which act like
                             keys) or aggregators

Deaggregates:
   Deaggregates are specified as <deaggregator>[,<arguments>]. See --list-deaggregators for a list of available deaggregators.

   In general, key name arguments to deaggregators may be key specs, but not key groups

Domain Lanuage Integration:
   The normal mechanism for specifying keys and aggregators allows one to concisely instantiate the objects that back them in the
   platform and is certainly the easiest way to use recs. The record stream domain language allows the creation of these objects in
   a programmatic way, with neither the syntactic issues of the normal way nor its guiding hand.

   The domain language is itself just PERL with a collection of library functions for creating platform objects included. Your
   favorite aggregators are all here with constructors matching their normal token. For convenience of e.g. last, aggregators are
   also included with a prefixed underscore.

   Below you can find documentation on all the "built in" functions. Most aggregators and deaggregators should be present with
   arguments comparable to their normal instantiation arugments, but with keyspec parameters replaced with valuations parameters.

   Deaggregates may be specified using the recs domain language. --dldeaggregator requires the code evaluate as a deaggregator.

   See --help-domainlanguage for a more complete description of its workings and a list of available functions.

   See the examples below for a more gentle introduction.

Examples:
   Split the "hosts" field into individual "host" fields
      recs-decollate --dldeaggregator '_split(hosts,qr/, */,host)'

Help from: --help-deaggregators:
split: split the provided field
unarray: split the provided array
unhash: split the provided hash

Help from: --help-domainlanguage:
DOMAIN LANGUAGE
   The normal mechanism for specifying keys and aggregators allows one to concisely instantiate the objects that back them in the
   platform and is certainly the easiest way to use recs. The record stream domain language allows the creation of these objects in
   a programmatic way, with neither the syntactic issues of the normal way nor its guiding hand.

   The domain language is itself just PERL with a collection of library functions for creating platform objects included. Your
   favorite aggregators are all here with constructors matching their normal token. For convenience of e.g. last, aggregators are
   also included with a prefixed underscore.

   Below you can find documentation on all the "built in" functions. Most aggregators and deaggregators should be present with
   arguments comparable to their normal instantiation arugments, but with keyspec parameters replaced with valuations parameters.

Special Syntax
   Where one sees a <snippet> argument below, a string scalar is expected, however quoting these can get fairly difficult and they
   can be confused with non-<snippet> scalars.

   Example:
     --dla "ucancat(',', snip('{{x}} * 2'))"

   To remedy this, one may use <<code>> to inline a snippet which will be immediately understood by the typing mechanism as being
   code. Escaping inside this is as single quotes in PERL.

   Example With <<CODE>>
     --dla 'uconcat(",", <<{{x}} * 2>>)'

Function Library
   ii_agg(<snippet>, <snippet>[, <snippet>])
   ii_aggregator(<snippet>, <snippet>[, <snippet>])
   inject_into_agg(<snippet>, <snippet>[, <snippet>])
   inject_into_aggregator(<snippet>, <snippet>[, <snippet>])
      Take an initial snippet, a combine snippet, and an optional squish snippet to produce an ad-hoc aggregator based on inject
      into. The initial snippet produces the aggregate value for an empty collection, then combine takes $a representing the
      aggregate value so far and $r representing the next record to add and returns the new aggregate value. Finally, the squish
      snippet takes $a representing the final aggregate value so far and produces the final answer for the aggregator.

      Example(s):
         Track count and sum to produce average:
            ii_agg(<<[0, 0]>>, <<[$a->[0] + 1, $a->[1] + {{ct}}]>>, <<$a->[1] / $a->[0]>>)

   for_field(qr/.../, <snippet>)
      Takes a regex and a snippet of code. Creates an aggregator that creates a map. Keys in the map correspond to fields chosen by
      matching the regex against the fields from input records. Values in the map are produced by aggregators which the snippet
      must act as a factory for ($f is the field).

      Example(s):
         To aggregate the sums of all the fields beginning with "t"
            for_field(qr/^t/, <<sum($f)>>)

   for_field(qr/.../, qr/.../, <snippet>)
      Takes two regexes and a snippet of code. Creates an aggregator that creates a map. Keys in the map correspond to pairs of
      fields chosen by matching the regexes against the fields from input records. Values in the map are produced by aggregators
      which the snippet must act as a factory for ($f1 is the first field, $f2 is the second field).

      Example(s):
         To find the covariance of all x-named fields with all y-named fields:
            for_field(qr/^x/, qr/^y/, <<covar($f1, $f2)>>)

   map_reduce_agg(<snippet>, <snippet>[, <snippet>])
   map_reduce_aggregator(<snippet>, <snippet>[, <snippet>])
   mr_agg(<snippet>, <snippet>[, <snippet>])
   mr_aggregator(<snippet>, <snippet>[, <snippet>])
      Take a map snippet, a reduce snippet, and an optional squish snippet to produce an ad-hoc aggregator based on map reduce. The
      map snippet takes $r representing a record and returns its mapped value. The reduce snippet takes $a and $b representing two
      mapped values and combines them. Finally, the squish snippet takes a mapped value $a representing all the records and
      produces the final answer for the aggregator.

      Example(s):
         Track count and sum to produce average:
            mr_agg(<<[1, {{ct}}]>>, <<[$a->[0] + $b->[0], $a->[1] + $b->[1]]>>, <<$a->[1] / $a->[0]>>)

   rec()
   record()
      A valuation that just returns the entire record.

   snip(snip)
      Takes a snippet and returns both the snippet and the snippet as a valuation. Used to distinguished snippets from scalars in
      cases where it matters, e.g. min('{{x}}') interprets it is a keyspec when it was meant to be a snippet (and then a
      valuation), min(snip('{{x}}')) does what is intended. This is used internally by <<...>> and in fact <<...>> just translates
      to snip('...').

   subset_agg(<snippet>, <aggregator>)
   subset_aggregator(<snippet>, <aggregator>)
      Takes a snippate to act as a record predicate and an aggregator and produces an aggregator that acts as the provided
      aggregator as run on the filtered view.

      Example(s):
          An aggregator that counts the number of records with a time not above 6 seconds:
             subset_agg(<<{{time_ms}} <= 6000>>, ct())

   type_agg(obj)
   type_scalar(obj)
   type_val(obj)
      Force the object into a specific type. Can be used to force certain upconversions (or avoid them).

   valuation(sub { ... })
   val(sub { ... })
      Takes a subref, creates a valuation that represents it. The subref will get the record as its first and only argument.

      Example(s):
         To get the square of the "x" field:
            val(sub{ $[0]->{x} ** 2 })

   xform(<aggregator>, <snippet>)
      Takes an aggregator and a snippet and produces an aggregator the represents invoking the snippet on the aggregator's result.

      Example(s):
         To take the difference between the first and second time fields of the record collection:
            xform(recs(), <<{{1/time}} - {{0/time}}>>)

See Also

RecordStream(3) - Overview of the scripts and the system

recs-examples(3) - A set of simple recs examples

recs-story(3) - A humorous introduction to RecordStream

SCRIPT --help - every script has a --help option, like the output above