Netflow collection

Install fprobe and flow-tools
start fprobe
edit /etc/flow-tools/flow-capture.conf
start flow-tools

I use postgresql db.
table structure:


traffic=# \d+ raw

Table «public.raw»
Column   |  Type   |     Modifiers      | Storage | Description
------------±--------±-------------------±--------±------------
unix_secs  | integer | not null default 0 | plain   |
unix_nsecs | integer | not null default 0 | plain   |
sysuptime  | integer | not null default 0 | plain   |
exaddr     | inet    | not null           | main    |
dpkts      | integer | not null default 0 | plain   |
doctets    | integer | not null default 0 | plain   |
srcaddr    | inet    | not null           | main    |
dstaddr    | inet    | not null           | main    |
srcport    | integer | not null default 0 | plain   |
dstport    | integer | not null default 0 | plain   |
prot       | integer | not null default 0 | plain   |
Indexes:
«raw_doctets» btree (doctets)
«raw_dpkts» btree (dpkts)
«raw_dstaddr» btree (dstaddr)
«raw_dstport» btree (dstport)
«raw_exaddr» btree (exaddr)
«raw_prot» btree (prot)
«raw_srcaddr» btree (srcaddr)
«raw_srcport» btree (srcport)
«raw_sysuptime» btree (sysuptime)
«raw_unix_nsecs» btree (unix_nsecs)
«raw_unix_secs» btree (unix_secs)
Has OIDs: no

 

# Экспортируем данные из flow-tools в базу данных.
netflow-psql-exporter.py

#!/usr/bin/python
# -*- coding: utf8 -*-

import psycopg2, time, os, logging

tablename = 'raw'
year = time.strftime("%Y", time.localtime(time.time() - 24*3600))
month = time.strftime("%m", time.localtime(time.time() - 24*3600))
day = time.strftime("%d", time.localtime(time.time() - 24*3600))

flow_dir = ('/var/log/flow/%s/%s-%s/%s-%s-%s/*')%(year, year, month, year, month, day)

log_file = ('/var/log/flow/export-to-postgres-log/%s-%s-%s.log')%(year, month, day)
logging.basicConfig(level=logging.DEBUG,format="%(asctime)s: %(message)s",filename=log_file,filemode="w")

try:
        conn = psycopg2.connect("dbname='traffic' host='pg-master' user='traffic' password='traffic_password'")
except:
        print('can not connect to database')
        logging.debug('can not connect to database')
else:
        print('connect established')
        logging.debug('connect established')

# Экспортируем данные из flow-tools в базу данных.
try:
        os.popen(""" flow-cat """+ flow_dir +""" | flow-export -f5 -m unix_secs,unix_nsecs,sysuptime,exaddr,dpkts,doctets,srcaddr,dstaddr,srcport,dstport,prot -u traffic:traffic_password:pg-master:5432:traffic:"""+tablename)
except:
        logging.debug('can not execute flow-export command')
else:
        logging.debug('execute flow-export is done')

conn.close()