Data Collection

From DreamFactory
Jump to: navigation, search
DFEData Collection
m
m
Line 1: Line 1:
 
{{DISPLAYTITLE:Data Collection System}}
 
{{DISPLAYTITLE:Data Collection System}}
  
The Data Collection system is an autonomous system that listens for, and logs, information from any instances deployed in your enterprise. This data is collated and then made available under the [[DFE/Dashboard/Reports|Reports]] tab of the [[DFE/Console|Console]].
+
DFE's Data Collection System (DCS) autonomously listens for, and logs, information from any instances deployed in your enterprise. This data is collated and then made available under the [[DFE/Dashboard/Reports|Reports]] tab of the [[DFE/Console|Console]].
  
 
[[File:data-collection-components.png]]
 
[[File:data-collection-components.png]]
 +
 +
== The Mighty Mighty ELK ==
 +
The data storage and visualization engine underneath this all is the combination of three separate pieces of software: Elasticsearch, Logstash, and Kibana. These three together are commonly referred to as the "ELK" stack (first letter of each component). The system listens for data of various types from various sources. This all depends on configuration.
 +
 +
=== Installing the ELK Stack ===
 +
The procedure is simple and well documented elsewhere:
 +
 +
* [Redhat/CentOS](https://www.digitalocean.com/community/tutorials/how-to-use-logstash-and-kibana-to-centralize-logs-on-centos-6)
 +
* [Debian/Ubuntu](https://www.digitalocean.com/community/tutorials/how-to-use-logstash-and-kibana-to-centralize-and-visualize-logs-on-ubuntu-14-04)
 +
 +
=== Configuration ===
 +
You must configure the [Logstash]() component to listen for incoming audit packets from the DFE system. These are transmitted as UDP packets from each deployed instance. To tell logstash to listen for these packets,

Revision as of 22:14, 7 October 2015


DFE's Data Collection System (DCS) autonomously listens for, and logs, information from any instances deployed in your enterprise. This data is collated and then made available under the Reports tab of the Console.

Data-collection-components.png

The Mighty Mighty ELK

The data storage and visualization engine underneath this all is the combination of three separate pieces of software: Elasticsearch, Logstash, and Kibana. These three together are commonly referred to as the "ELK" stack (first letter of each component). The system listens for data of various types from various sources. This all depends on configuration.

Installing the ELK Stack

The procedure is simple and well documented elsewhere:

Configuration

You must configure the [Logstash]() component to listen for incoming audit packets from the DFE system. These are transmitted as UDP packets from each deployed instance. To tell logstash to listen for these packets,