NAME
Net::NfDump - Perl API for manipulating with nfdump files
SYNOPSIS
use Net::NfDump;
my $src = new Net::NfDump(
InputFiles => [ 'nfdump_file' ],
Filter => 'icmp and src net 10.0.0.0/8',
Fields => [ '*' ] );
my $dst = new Net::NfDump(
OutputFile => 'nfdump_icmp_private',
Fields => [ '*' ] );
$src->query();
while (my $row = $src->fetchrow_arrayref() ) {
$dst->storerow_arrayref($row);
}
$src->finish();
$dst->finish();DESCRIPTION
METHODS
new
The constructor. As the parameter options can be specified. This options will be used as a default option set in the particular methods.
Module supports followinng options
Options for reading data
=================================================================
InputFiles => [] - List of files to read (arrayref)
Filter => 'any' - Filter taht will be applied on input
records. Uses nfdump/tcpdump syntax
Fields => [ '*' ] - List of fields to read or update
any field from supported fields can
be used here. See the chapter
"Supported Fields" for the full list
of supported fields. Special field * can
be used for defining all fields.
TimeWindowStart => 0
TimeWindowEnd => 0 - Filter flows that starts or ends in the
specified fied time window. The options uses
unix timestamp values or 0 if the filter shoul
not be apllied.
Options for writing data
=================================================================
OutputFile => undef - Output file for storerow_* methods.
Compressed => 1 - Flag whether the otput files should be
compressed or not.
Anonymized => 0 - Flag indicating that output file contains
anonymized data.
Ident => "" - String identificator of files info
Returns the information the current state of processing input files. It returns information about already processed files, blocks, records. Those information can be usefull for guessing time of processing whole dataset.
query()
Method that have to be executed before the fetchrow_* method are used.
fetchrow_arrayref
Have to be used after query method. If the query wasn't called before the method is called as $obj->query() before the first record is returned.
Method returns array reference with the record and skips to the next record. Returns true if there are more records to read or false if all record from all files have been read.
fetchrow_array
Same functionality as fetchrow_arrayref however returns items in array.
fetchrow_hashref
Same as fetchrow_arrayref, however the items are returned in the hash reference.
NOTE: This method can be very uneffective in some cases, please see PERFORMANCE section.
create
Creates a new nfdump file.
storerow_arrayref
Insert data defined in arrayref to the file opened by create. The number of fields and their order have to respect order defined in the Fileds option handled during $obj->new() or $obj->create() method.
storerow_array
Same as storerow_arrayref, however items are handled as the single array
storerow_hashref
Inserts structure defined as hash reference into output file.
NOTE: This method can be very uneffective in some cases, please see PERFORMANCE section.
clonerow
Copy the full content of the row from the source object (instance). This method is usefull for writing effective scripts (it's much faster that any of the prevous row).
finish
Closes all openes file handles. It is nescessary to call that method specilly when a new file is created. The method flushes to file records that remains in the memory buffer and updates file statistics in the header. Withat calling this method the output file might be corupted.
FLOW QUERY - NOT IMPLEMENTED YET
The flow query is language vyry simmilar to SQL to query data on nfdump files. However flow query have nothing to do with SQL. It uses only simmilar command syntax. Example of flow query
SELECT * FROM data/nfdump1.nfcap, data2/nfdump2.nfcap
WHERE src host 147.229.3.10
TIME WINDOW BETWEEN '2012-06-03' AND '202-06-04'
ORDER BY bytes
LIMIT 100
INSERT INTO data/nout_nfdump.nfcap (srcip, dstip, srcport, dstport) NOTE ABOUT 32BIT PLATFORMS
Nfdump primary uses 64 bit counters and other items to store single integer value. However the native 64 bit support is not compiled in every perl. For thoose cases where only 32 integer values are supported the Net::NfDump uses Math::Int64 module.
The build scripts automatically detect the platform and Math::Int64 module is required only on platforms where perl do not supports 64bit integer values.
EXTRA CONVERTION FUNCTIONS
The module also provides extra convertion functions that allow convert binnary format of IP address, MAC address and MPLS labels tag into text format and back.
Those functions are not exported by default
ip2txt
Converts both IPv4 and IPv6 address into text form. The standart inet_ntop function can be used instead to provide same results.
txt2ip
Inversion fuction to ip2txt. Returns binnary format of IP addres or undef if the conversion is impossible.
mac2txt
Converts MAC addres to xx:yy:xx:yy:xx:yy format.
txt2mac
Inversion fuction to mac2txt. Accept address in any of following format aabbccddeeff aa:bb:cc:dd:ee:ff aa-bb-cc-dd-ee-ff aabb-ccdd-eeff
Return the binnary format of the address or undef if confersion is impossible.
mpls2txt
Converts label information to format Lbl-Exp-S
Whwre Lbl - Value given to the MPLS label by the router. Exp - Value of experimental bit. S - Value of the end-of-stack bit: Set to 1 for the oldest entry in the stack and to zero for all other entries.
txt2mpls
Inversion function to mpls2txt. As the argiment expects the text representaion of the MPLS labels as was described in the previous function (Lbl-Exp-S)
flow2txt
Gets hash reference to items returned by fetchrow_hashref and converts all items into human readable text format. Applies finction ip2txt, mac2txt, mpl2txt to the items where it make sense.
txt2flow
Inversion function to flow2txt. It is usefull before calling storerow_hashref
file_info
Reads information from nfdump file header. It provides various atributes like number of blocks, version, flags, statistics, etc. As the result the follwing items are returned:
version
ident
blocks
catalog
anonymized
compressed
sequence_failures
first
msec_first
last
msec_last
flows, bytes, packets
flows_tcp, flows_udp, flows_icmp, flows_other
bytes_tcp, bytes_udp, bytes_icmp, bytes_other
packets_tcp, packets_udp, packets_icmp, packets_otherSUPPORTED ITEMS
Time items
=====================
first - Timestamp of first seen packet
msecfirst - Number of miliseconds of first seen packet since B<first>
last - Timestamp of last seen packet
mseclast - Number of miliseconds of last seen packet since B<last>
received - Timestamp when the packet was received by collector
Statistical items
=====================
bytes - The number of bytes
pkts - The number of packets
outbytes - The number of output bytes
outpkts - The number of output packets
flows - The number of flows (aggregated)
Layer 4 information
=====================
srcport - Source port
dstport - Destination port
tcpflags - TCP flags
Layer 3 information
=====================
srcip - Source IP address
dstip - Destination IP address
nexthop - IP next hop
srcmask - Source mask
dstmask - Destination mask
tos - Source type of service
dsttos - Destination type of Service
srcas - Source AS number
dstas - Destination AS number
nextas - BGP Next AS
prevas - BGP Previous AS
bgpnexthop - BGP next hop
proto - IP protocol
Layer 2 information
=====================
srcvlan - Source vlan label
dstvlan - Destination vlan label
insrcmac - In source MAC address
outsrcmac - Out destination MAC address
indstmac - In destintation MAC address
outdstmac - Out source MAC address
MPLS information
=====================
mpls - MPLS labels
Layer 1 information
=====================
inif - SNMP input interface number
outif - SNMP output interface number
dir - Flow directions ingress/egress
fwd - Forwarding status
Exporter information
=====================
router - Exporting router IP
systype - Type of exporter
sysid - Internal SysID of exporter
Extra/special fields
=====================
clientdelay - nprobe latency client_nw_delay_usec
serverdelay - nprobe latency server_nw_delay_usec
appllatency - nprobe latency appl_latency_usecSEE ALSO
http://nfdump.sourceforge.net/
AUTHOR
Tomas Podermanski, <tpoder@cis.vutbr.cz>, Brno University of Technology
COPYRIGHT AND LICENSE
Copyright (C) 2012 by Brno University of Technology
This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.