Tag Archives: configs

logstash+kibana+rabbitmq+elasticsearch

Install logstash rabbitmq-server elasticsearch from repository.
Download kibana from its site and unpack to your webroot folder.
I'll add nginx logs to logstash.

Nginx

set ngixn logs format in nginx.conf:

1
2
log_format logstash '$http_host ' '$remote_addr [$time_local] ' '"$request" $status $body_bytes_sent ' '"$http_referer" "$http_user_agent" ' '$request_time ' '$upstream_response_time ' '$upstream_addr';
access_log /var/log/nginx/access.log logstash;

Logstash

add in /etc/logstash/conf.d 2 files,
logstash-nginx:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
input {
tcp {
type => "nginx_agupcom"
data_timeout => 10
mode => "server"
host => "127.0.0.1"
port => 14001
}
}

filter {
if [type] == "nginx_agupcom" {
grok {
match => [
"message", "%{IPORHOST:http_host} %{IPORHOST:clientip} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} %{NUMBER:request_time:float} %{NUMBER:upstream_time:float} %{HOSTPORT:upstream_addr}",
"message", "%{IPORHOST:http_host} %{IPORHOST:clientip} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} %{NUMBER:request_time:float}"
]
}
date {
match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
}
geoip {
source => "clientip"
}
}
}

output {
rabbitmq {
exchange => "logstash_agupcom"
exchange_type => direct
host => "127.0.0.1" # string (required)
key => "logstash_agupcom"
workers => 2
durable => true
persistent => true
}
}

nginx_agupcom — name of my log type
logstash_agupcom name of queue in rabbitmq.
Continue reading logstash+kibana+rabbitmq+elasticsearch