NAME

Net::NfDump - Perl API for manipulating with nfdump files

SYNOPSIS

  use Net::NfDump;

  #
  # Example 1: reading nfdump file(s)
  # 
  
  $flow = new Net::NfDump(
        InputFiles => [ 'nfdump_file1', 'nfdump_file2' ], 
        Filter => 'icmp and src net 10.0.0.0/8',
        Fields => 'proto, bytes' ); 

  $flow->query();

  while (my ($proto, $bytes) = $flow->fetchrow_array() )  {
    $h{$proto} += $bytes;
  }
  $flow->finish();

  foreach ( keys %h ) {
    printf "%s %d\n", $_, $h{$_};
  }


  #
  # Example 2: creating and writing records into nfdump file
  #
  
  my $flow = new Net::NfDump(
        OutputFile => 'output.nfcap',
        Fields => 'srcip,dstip' );

  $flow->storerow_arrayref( [ txt2ip('147.229.3.10'), txt2ip('1.2.3.4') ] );

  $flow->finish();

  #
  # Example 3: reading/writing (merging two input files) and swap
  #            source and destination address if the destination port 
  #            is 80/http (I know it doesn't make much sense).
  #

  $flow1 = new Net::NfDump( 
		InputFiles => [ 'nfdump_file1', 'nfdump_file2' ], 
        Fields => 'srcip, dstip, dstport' ); 

  $flow2 = new Net::NfDump( 
		OutputFile => 'nfdump_file_out', 
        Fields => 'srcip, dstip, dstport' ); 

  $flow1->query();
  $flow2->create();

  while (my $ref = $flow->fetchrow_arrayref() )  {

    if ( $ref->[1] == 80 ) { 
      ($ref->[0], $ref->[1]) = ($ref->[1], $ref->[0]);
    }

	$flow2->clonerow($flow1);
	$floe2->storerow_arrayref($ref);

  }

  $flow1->finish();
  $flow2->finish();

DESCRIPTION

The module implements basic operations on binary files produced with nfdump tool. It allows read, create and write flow records on those files. The modules tries to keep naming conventions for methods same as iare used n DBI nodules/API, so developers that are used to use this interface should be famillar with the interface.

The module uses original nfdump sources to implement nescessary functions. The compatibility with original nfdump should be asisly keeps and maintance of code to cope with future version shoul be minnimal.

The architecture is following:

       APPLICATION 
+------------------------+
|                        |  
| Net::NfDump API (perl) |
|                        |
+------------------------+
|                        |  
| libnf - glue code (C)  |
|                        |
+------------------------+
|                        |  
|   nfdump sources (C)   |
|                        |
+------------------------+
      NFDUMP FILES

Net::NfDump API

Implements all methods and functions described in rest of thi document.
libnf - glue code

Creates code that converts internal nfdump structures into perl and back.
nfdump sources

Are original nfdump package source files downloaded from http://sourceforge.net/projects/nfdump/. There are no changes in thos code and all relevant changes are places into libnf code only.

This version of Net::NfDump is based on nfdump-1.6.8p1.

Options

Options can be nahdled in varios methods. The basic options ses can be handled in the constructor and than modified in methods like $obj->query() or $obj->create().

The values after => indicates the default value for the item.

InputFiles => []

List of files to read (arrayref).
Filter => 'any'

Filter taht will be applied on input records. Uses nfdump/tcpdump syntax.
Fields => '*'

List of fields to read or update any field from supported fields can be used here. See the chapter "Supported Fields" for the full list of supported fields. Special field * can be used for defining all fields.
TimeWindowStart, TimeWindowEnd => 0

Filter flows that starts or ends in the specified fied time window. The options uses unix timestamp values or 0 if the filter should not be apllied.
OutputFile => undef

Output file for storerow_* methods. Default: undef
Compressed => 1

Flag whether the otput files should be compressed or not.
Anonymized => 0

Flag indicating that output file contains anonymized data.
Ident => ''

String identificator of files stored into the header of the file.

Constructor, status informations methods

$obj = new Net::NfDump( %opts )
```
my $obj = new Net::NfDump( InputFiles => [ 'file1']  );
```
The constructor. As the parameter options can be specified.

$ref = $obj->info()

my $i = $obj->info();
print Dumper($i);

Returns the information the current state of processing input files. It returns information about already processed files, blocks, records. Those information can be usefull for guessing time of processing whole dataset. Hashref returs following items:

total_files           - total number of files to process
elapsed_time          - elapsed time 
remaining_time        - guessed remaining time to process all records
percent               - guessed percent of processed records

processed_files       - total number of processed files
processed_records     - total number of processed records
processed_blocks      - total number of processed blocks
processed_bytes       - total number of processed bytes 
                        number of bytes read from file 
                        system after uncompressing 

current_filename      - the name of the file currently processed
current_total_blocks  - the number of blocks in the currently 
                        processed file 
current_processed_blocks -  the number of processd blocks in the 
                        currently processed file

$obj->finish()
```
$obj->finish();
```
Closes all openes file handles. It is nescessary to call that method specilly when a new file is created. The method flushes to file records that remains in the memory buffer and updates file statistics in the header. Withat calling this method the output file might be corupted.

Methods for reading data

$obj->query( %opts )
```
$obj->query( Filter => 'src host 10.10.10.1' );
```
Method that have to be executed before any of the fetchrow_* method is used. Options can be handled to the method.
$ref = $obj->fetchrow_arrayref()
```
while (my $ref = $obj->fetchrow_arrayref() ) {
    print Dumper($ref);
}
```
Have to be used after query method. The method $obj->query() s called automatically if it wasn't called before.

Method returns array reference with the record and skips to the next record. Returns true if there are more records to read or undef if end of the record set have been reached.
@array = $obj->fetchrow_array()
```
while ( @array = $obj->fetchrow_arrayref() ) { 
  print Dumper(\@array);
}
```
Same functionality as fetchrow_arrayref however returns items in array instead.
$ref = $obj->fetchrow_hashref()
```
while ( $ref = $obj->fetchrow_hashref() ) {
   print Dumper($ref);
}
```
Same as fetchrow_arrayref, however the items are returned in the hash reference as the key => vallue tuples.

NOTE: This method can be very uneffective in some cases, please see PERFORMANCE section.

Methods for writing data

$obj->create( %opts )
```
$obj->create( OutputFile => 'output.nfcapd' );
```
Creates a new nfdump file. This method have to be called before any of $obj->storerow_* method is called.
$obj->storerow_arrayref( $arrayref )
```
$obj->storerow_arrayref( [ $srcip, $dstip ] );
```
Insert data defined in arrayref to the file opened by create. The number of fields and their order have to respect order defined in the Fileds option handled during $obj->new() or $obj->create() method.
$obj->storerow_array( @array )
```
$obj->storerow_array(  $srcip, $dstip  );
```
Same as storerow_arrayref, however items are handled as the single array
$obj->storerow_hashref ( \%hash )
```
$obj->storerow_hashref( { 'srcip' =>  $srcip, 'dstip' => $dstip } );
```
Inserts structure defined as hash reference into output file.

NOTE: This method can be very uneffective in some cases, please see PERFORMANCE section.
$obj->clonerow( $obj2 )
```
$obj->clonerow( $obj2 );
```
Copy the full content of the row from the source object (instance). This method is usefull for writing effective scripts (it's much faster that any of the prevous row).

Extra conversion and support functions

The module also provides extra convertion functions that allow convert binnary format of IP address, MAC address and MPLS labels tag into text format and back.

Those functions are not exported by default, so it have to be either called with full module name or imported when the module is loades. For importing all support function :all synonym can be used.

use Net::NfDump qw ':all';

$txt = ip2txt( $bin )
$bin = txt2ip( $txt )
```
$ip = txt2ip('10.10.10.1');
print ip2txt($ip);
```
Converts both IPv4 and IPv6 address into text form and back. The standart inet_ntop/inet_pton functions can be used instead to provide same results.

Function txt2ip returns binnary format of IP addres or undef if the conversion is impossible.
$txt = mac2txt( $bin )
$bin = txt2mac( $txt )
```
$mac = txt2mac('aa:02:c2:2d:e0:12');
print mac2txt($mac);
```
Converts MAC addres to xx:yy:xx:yy:xx:yy format and back. The fuction to mac2txt accepts an address in any of following format:
```
aabbccddeeff
aa:bb:cc:dd:ee:ff
aa-bb-cc-dd-ee-ff
aabb-ccdd-eeff
```
Returns the binnary format of the address or undef if confersion is impossible.
$txt = mpls2txt( $mpls )

$mpls = txt2mpls( $txt )

$mpls = txt2mpls('1002-6-0 1003-6-0 1004-0-1');
print mpls2txt($mpls);

Converts label information to format Lbl-Exp-S and back.

Where:

Lbl - Value given to the MPLS label by the router. 
Exp - Value of experimental bit. 
S   - Value of the end-of-stack bit: Set to 1 for the oldest 
      entry in the stack and to zero for all other entries.

$ref = flow2txt( \%row )
$ref = txt2flow( \%row )

The function flow2txt gets hash reference to items returned by fetchrow_hashref and converts all items into humman readable text format. Applies finctions ip2txt, mac2txt, mpl2txt to the items where it make sense. The function txt2flow does opossite functionality.

$ref = file_info( $file_name )

$ref = file_info('file.nfcap');
print Dumper($ref);

Reads information from nfdump file header. It provides various atributes like number of blocks, version, flags, statistics, etc. As the result the follwing items are returned:

version
ident
blocks
catalog
anonymized
compressed
sequence_failures

first
last

flows, bytes, packets

flows_tcp, flows_udp, flows_icmp, flows_other
bytes_tcp, bytes_udp, bytes_icmp, bytes_other
packets_tcp, packets_udp, packets_icmp, packets_other

SUPPORTED ITEMS

Time items
=====================
first - Timestamp of first seen packet 
last - Timestamp of last seen packet 
received - Timestamp when the packet was received by collector 

Statistical items
=====================
bytes - The number of bytes 
pkts - The number of packets 
outbytes - The number of output bytes 
outpkts - The number of output packets 
flows - The number of flows (aggregated) 

Layer 4 information
=====================
srcport - Source port 
dstport - Destination port 
tcpflags - TCP flags  

Layer 3 information
=====================
srcip - Source IP address 
dstip - Destination IP address 
nexthop - IP next hop 
srcmask - Source mask 
dstmask - Destination mask 
tos - Source type of service 
dsttos - Destination type of Service 
srcas - Source AS number 
dstas - Destination AS number 
nextas - BGP Next AS 
prevas - BGP Previous AS 
bgpnexthop - BGP next hop 
proto - IP protocol  

Layer 2 information
=====================
srcvlan - Source vlan label 
dstvlan - Destination vlan label 
insrcmac - In source MAC address 
outsrcmac - Out destination MAC address 
indstmac - In destintation MAC address 
outdstmac - Out source MAC address 

MPLS information
=====================
mpls - MPLS labels 

Layer 1 information
=====================
inif - SNMP input interface number 
outif - SNMP output interface number 
dir - Flow directions ingress/egress 
fwd - Forwarding status 

Exporter information
=====================
router - Exporting router IP 
systype - Type of exporter 
sysid - Internal SysID of exporter 

Extra/special fields
=====================
clientdelay - nprobe latency client_nw_delay_usec 
serverdelay - nprobe latency server_nw_delay_usec
appllatency - nprobe latency appl_latency_usec

PERFORMANCE

It is obvious tahat prformance of the perl interface is lower comparing to highly optimized nfdump utility. As nfdump is able to process up to 2 milions of records per second, the Net::NfDump is not bale to process more than 1 milion of records per second. However there are several rules to keep the code optimised:

Use $obj->fetchrow_arrayref() and $obj->storerow_arrayref() instead of *_array and *_hashref equivalents. Arrayref handles only the reference to the structure with data. Avoid of using *_hashref functions, it can by 5 times slower.
Handle to the perl API only items that are nescessary for using in the code. It is always more effective to define in Fields => 'srcip,dstip,...' intead of Fileds => '*'.

Prefer of using $obj->clonerow($obj2) method. This method copies data between two instances directly in the C code in the libnf layer.

Following code:

$obj1->exec( Fields => '*' );
$obj2->create( Fields => '*' );

while ( my $ref = $obj1->fetchrow_arrayref() ) {
  # do something with srcip 
  $obj2->storerow_arrayref($ref);
}

can be written in more effective way (several times faster):

$obj1->exec( Fields => 'srcip' );
$obj2->create( Fields => 'srcip' );

while ( my $ref = $obj1->fetchrow_arrayref() ) {
  # do something with srcip 
  $obj2->clonerow($obj1);
  $obj2->storerow_arrayref($ref);
}

NOTE ABOUT 32BIT PLATFORMS

Nfdump primary uses 64 bit counters and other items to store single integer value. However the native 64 bit support is not compiled in every perl. For thoose cases where only 32 integer values are supported the Net::NfDump uses Math::Int64 module.

The build scripts automatically detect the platform and Math::Int64 module is required only on platforms where perl do not supports 64bit integer values. =head1 SEE ALSO

http://nfdump.sourceforge.net/

AUTHOR

Tomas Podermanski, <tpoder@cis.vutbr.cz>, Brno University of Technology

COPYRIGHT AND LICENSE

This library is free software; you can redistribute it and modify it under the same terms as Perl itself.

If you are uses Net::NfDump please send us a postcard, preferably with a picture from your location / city to:

Brno University of Technology 
CVIS
Tomas Podermanski 
Antoninska 1
601 90 
Czech Republic

To install Net::NfDump, copy and paste the appropriate command in to your terminal.

cpanm

cpanm Net::NfDump

CPAN shell

perl -MCPAN -e shell
install Net::NfDump

For more information on module installation, please visit the detailed CPAN module installation guide.

	Global
`s`	Focus search bar
`?`	Bring up this help dialog

	GitHub
`g` `p`	Go to pull requests
`g` `i`	go to github issues (only if github is preferred repository)

	POD
`g` `a`	Go to author
`g` `c`	Go to changes
`g` `i`	Go to issues
`g` `d`	Go to dist
`g` `r`	Go to repository/SCM
`g` `s`	Go to source
`g` `b`	Go to file browse

	Search terms
module: (e.g. module:Plugin)
distribution: (e.g. distribution:Dancer auth)
author: (e.g. author:SONGMU Redis)
version: (e.g. version:1.00)