NAME
Search::ESsearcher::Templates::httpAccess - Provicdes support for HTTP access logs sucked down via beats.
VERSION
Version 0.0.0
LOGSTASH / FILEBEAT
This uses a logstath beasts input akin to below.
The important bit below is setting the "type" to "beats" and "fields.log" to "apache-access".
If you are using something different than "type" and "beats" you can specify that via "--field" and "--fieldv" respectively.
If you are using something different than "fields.log" and "apache-access" you can specify that via "--field2" and "--field2v" respectively.
input {
beats {
host => "192.168.14.3"
port => 5044
type => "beats"
}
}
filter {
if [fields][log] == "apache-access" {
grok {
match => {
"message" => "%{HTTPD_COMBINEDLOG}+%{GREEDYDATA:extra_fields}"
}
overwrite => [ "message" ]
}
mutate {
convert => ["response", "integer"]
convert => ["bytes", "integer"]
convert => ["responsetime", "float"]
}
geoip {
source => "clientip"
target => "geoip"
add_tag => [ "apache-geoip" ]
}
date {
match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
remove_field => [ "timestamp" ]
}
useragent {
source => "agent"
}
}
}
output {
if [type] == "beats" {
elasticsearch {
hosts => [ "127.0.0.1:9200" ]
}
}
}Then for file beats, something akin to below. The really important bits here the various values for "fields".
For "fields.vhost" and "fields.vhost_port" if you are using somethind different, you can specify that via "--field3" and "--field4" respectively.
- type: log
enabled: true
paths:
- /var/log/apache/foo.bar:80-access.log
fields:
log: apache-access
vhost: foo.bar
vhost_port: 80Options
--host <host>
The machine beasts is running on feeding info to logstash/ES.
--response <code>
The response code from the HTTP server.
--verb <verb>
The verb used with the request.
--vhost <vhost>
The domain served up.
--port <port>
The port for the vhost.
--ip <ip>
The client IP that made the request.
--os <os>
The supplied OS value that made the request.
--showos
Shows the OS value.
--req <req>
The HTTP request.
--ref <ref>
The supplied referrer for the request.
--agent <agent>
The supplied agent value that made the request.
--noagent
Do not show the agent field.
--auth <auth>
The authed user for the request.
--bgt <bytes>
Response bytes greater than.
--bgte <bytes>
Response bytes greater than or equal to.
--blt <bytes>
Response bytes less than.
--blte <bytes>
Response bytes less than or equal to.
--geoip
Require GEO IP to have worked.
--country <country>
The 2 letter country code.
--showcountry
Show country code.
--region <state>
The state/province/etc to search for.
--showregion
Show region code.
--postal <zipcode>
The postal code to search for.
--showpostal
Show postal code.
--city <cide>
The city to search for.
--showcity
Show city name.
--size <count>
The number of items to return.
--dgt <date>
Date greater than.
--dgte <date>
Date greater than or equal to.
--dlt <date>
Date less than.
--dlte <date>
Date less than or equal to.
--msg <message>
Messages to match.
AND, OR, or NOT shortcut
, OR
+ AND
! NOTA list seperated by any of those will be transformed
These may be used with program, facility, pid, or host.
example: --program postfix,spamd
results: postfix OR spamddate
date
/^-/ appends "now" to it. So "-5m" becomes "now-5m".
/^u\:/ takes what is after ":" and uses Time::ParseDate to convert it to a unix time value.
Any thing not matching maching any of the above will just be passed on.