Tag Archives: example

logstash+kibana+rabbitmq+elasticsearch

Install logstash rabbitmq-server elasticsearch from repository.
Download kibana from its site and unpack to your webroot folder.
I'll add nginx logs to logstash.

Nginx

set ngixn logs format in nginx.conf:

1
2
log_format logstash '$http_host ' '$remote_addr [$time_local] ' '"$request" $status $body_bytes_sent ' '"$http_referer" "$http_user_agent" ' '$request_time ' '$upstream_response_time ' '$upstream_addr';
access_log /var/log/nginx/access.log logstash;

Logstash

add in /etc/logstash/conf.d 2 files,
logstash-nginx:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
input {
tcp {
type => "nginx_agupcom"
data_timeout => 10
mode => "server"
host => "127.0.0.1"
port => 14001
}
}

filter {
if [type] == "nginx_agupcom" {
grok {
match => [
"message", "%{IPORHOST:http_host} %{IPORHOST:clientip} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} %{NUMBER:request_time:float} %{NUMBER:upstream_time:float} %{HOSTPORT:upstream_addr}",
"message", "%{IPORHOST:http_host} %{IPORHOST:clientip} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} %{NUMBER:request_time:float}"
]
}
date {
match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
}
geoip {
source => "clientip"
}
}
}

output {
rabbitmq {
exchange => "logstash_agupcom"
exchange_type => direct
host => "127.0.0.1" # string (required)
key => "logstash_agupcom"
workers => 2
durable => true
persistent => true
}
}

nginx_agupcom — name of my log type
logstash_agupcom name of queue in rabbitmq.
Continue reading logstash+kibana+rabbitmq+elasticsearch

ansible: playbooks howto

You could create 2 types of playbooks:
1 — just a simple yaml file, example (poweroff.yml):

1
2
3
4
5
6
- hosts: vbox
  sudo
: yes
  gather_facts
: no
  tasks
:
    - name
: poweroff
      command
: /sbin/poweroff

2 — project, example:
structure:

1
2
3
4
5
6
7
8
9
10
11
12
deploy/
deploy/roles
deploy/roles/preconf
deploy/roles/preconf/vars
deploy/roles/preconf/vars/main.yml
deploy/roles/preconf/files
deploy/roles/preconf/files/.vimrc
deploy/roles/preconf/handlers
deploy/roles/preconf/tasks
deploy/roles/preconf/tasks/main.yml
deploy/roles/preconf/templates
deploy/site.yml

site.yml — something like main file where will be included roles
content of site.yml:

1
2
3
4
5
- name: First deploy, install necessary packages and users
  hosts
: vbox

  roles
:
   - preconf

roles folder — subcategories for your project, different roles for different logical steps,
in this example only 1 role with name «preconf»,
role folder contains folders:
tasks — it's a main role folder which contains tasks description.
vars — variables which could be used in tasks
files — folder with files
templates — folder with templates (I don't use it here)

vars/main.yml:

1
2
3
4
5
6
7
8
admin_team:
  - { user
: 'alter', comment: 'Roman', key: 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYIX38J/i16iedtP0wUnTK13eh1eGkCL0sa2GMi/TPA6ANFdF/D1AbUXgp88SuQNXrMa8hs9E5+D+/LtCFy+jN+RHEZmVXtnW7WckYAynw5i66Le8MUeRpjBo1m5C865WY0qCoKwkQfZSX6yecw8Gt9sw26SrL3oeLEM4zoiI+NHhahQYanwA7i6LLE6A2UEz2ni/ZNZH3eVaOVknD8D9b6MjDRtMoBkdHu2g3Xx2fOGpIQXOp99H0oofDCTB7BBwelWOYmRHUAvbyNIdJlWFFdamSWAevoHarrypZiCKc0TUYzi6etzSyJkOJoOoyy1xt2UG6jaAzKs2iqFgcBJ0n alter@M1USER0061', shell: '/bin/zsh' }
  - { user
: 'n506', comment: 'Dmitriy', key: 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCjjXrr6cZIDRRGih90P1WPjA2o5YXNG9Y2+Jhq18hBYJs6NKWaMyXsjvwBuHIJrbb9T9Rn9vVOr4fvMyGM0I+N9jr1bpGeq7BGZgqRtLoNrW/fvODKkKImDrwDKJG/B+EPRNLlOTvs2EsdVEJbJDi4ix0Ilj3D+SKe+AJ5gURY26DIKYEmMmyq2Ea8WmLam+rs8EfDkQw6NCkQvZSDzgs5pK+iwI4MVOKYisa0hakwfeS+qCiZr3nORy+TdwfJLJXovWFpQ24GacK3iK4FWdB3iafcFNc5BuHxpZeWcPUFoDf1ePlWD+sts44mzfaWyk95xV2NTUZmIsZ2L/Brucb1 n506@n506', shell: '/bin/bash' }
  - { user
: 'sticeberg', comment: 'Alexander', key: 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpFL+DrUxRsGOjC041AMx7SNwaxF7uUHGLtwq1/YkexpctkpJStYRv9mbvqRTDit5Mugy6Khk3Uux/S0r7g2ZxjgK+QU6iex9SKzm6/pRZkWEFVA0NQCBjrkRlRVCoRBpYhnO0OSwdE/QLIe9k2FwSGnMf6M5RbmlSViVDCrZ68CsArs7N6rrmAcMK0yy+BwrWSvVtvc92ao4K9li8rTg4VVy3E/F+NSNAJj90a+RAf6sb8M6qfbf0pTtxXU51kdRP2iS5QRj8KNkgdMFAD9+wYZpm9bs7vlWOaycVFlGsN9zevXaYrjkHLPYtCGGvCISfgEWAbUgdmuimUGnAMm7n sticeberg@gosts-MacBook-Air.local', shell: '/bin/bash' }

game_users
:
  - { user
: 'f1', shell: '/bin/bash', home: '/f1', uid: '6001' }
  - { user
: 'a1', shell: '/bin/bash', home: '/a1', uid: '6002' }

and tasks/main.yml:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
- name: update system packages
  sudo
: yes
  apt
: upgrade=safe update_cache=yes

- name
: installing packages
  sudo
: yes
  apt
: pkg={{ item }}
  with_items
:
   - zsh
    - htop
    - vim
    - mc
    - wget
    - curl
    - tmux
    - facter
    - tcpdump
    - iptraf
    - pbzip2
    - pigz
    - tcptraceroute


- name
: add users for project
  sudo
: yes
  user
: name="{{ item.user }}" shell="{{ item.shell }}" home="{{ item.home }}" uid="{{ item.uid }}"
  with_items
:
   - "{{ game_users }}"

- name
: add admins
  sudo
: yes
  user
: name="{{ item.user }}" shell="{{ item.shell }}" groups='sudo' comment="{{ item.comment }}"
  with_items
:
   - "{{ admin_team }}"

- name
: add ssh keys for admins
  sudo
: yes
  authorized_key
: user="{{ item.user }}" key="{{ item.key }}"
  with_items
:
   - "{{ admin_team }}"

- name
: copy vimrc file
  sudo
: yes
  copy
: src=".vimrc" dest="/home/{{ item.user }}/.vimrc" owner="{{ item.user }}"
  with_items
:
   - "{{ admin_team }}"

playbooks run by comman ansible-playbook:
1 — ansible-playbook poweoff.yml
2 — ansible-playbook site.yml

Fedora: runscript (example: openvpn client)

/etc/init.d/opevpn-home-client

#!/bin/bash
#
# Run-level Startup script for the OpenVPN client
#
# chkconfig: 345 91 19
# description: Startup/Shutdown OpenVPN client 
OVPN_HOME="/home/alter/ovpn/"
case "$1" in
    start)
        echo -n "Starting OpenVPN home client: "
        cd $OVPN_HOME && /usr/sbin/openvpn ovpn.conf
        echo "OK"
        ;;
    stop)
        echo -n "Shutdown OpenVPN home client: "
        killall openvpn
        echo "OK"
        ;;
    reload|restart)
        $0 stop
        $0 start
        ;;
    *)
        echo "Usage: $0 start|stop|restart"
        exit 1
esac
exit 0
chmod +x /etc/init.d/opevpn-home-client
chkconfig --add openvpn-home-client