Reports

From DreamFactory
Jump to: navigation, search
DFEReports
 
(30 intermediate revisions by 3 users not shown)
Line 1: Line 1:
−
=Configuring Reports=
+
= Introduction =
−
DreamFactory Enterprise uses the ELK stack (Elasticsearch, Logstash, Kibana) for reporting. API calls from instances are automatically logged by the ELK stack, and Kibana can be used as an user-friendly and flexible tool to visualize the metrics.
+
DreamFactory Enterprise utilizes a set of visualization tools from Elasticsearch https://www.elastic.co/. These three tools are Elasticsearch https://www.elastic.co/products/elasticsearch, Logstash https://www.elastic.co/products/logstash, and Kibana https://www.elastic.co/products/kibana. These three applications are commonly referred to as an "ELK" stack. Once a cluster is created and instances provisioned, all API traffic is echoed to the reporting system for visualization.
 +
 
 +
== Initial Configuration ==
 +
T.B.D.
 +
 
 +
== Configuration ==
 +
The ELK stack is pre-installed in DreamFactory Enterprise. Check your Kibana configuration file (kibana.yml), it should look like this with default configurations:
 +
 
 +
<source lang="bash">
 +
server.port: 5601
 +
server.host: "0.0.0.0"
 +
elasticsearch.url: "http://localhost:9200"
 +
kibana.index: ".kibana"
 +
</source>
 +
<br />
  
 
==Introduction to Kibana==
 
==Introduction to Kibana==
−
Go to the Kibana URL in your DreamFactory Enterprise installation. Usually the URL is http://kibana.your-enterprise-domain.com.  
+
Go to the Kibana URL in your DreamFactory Enterprise installation. Usually the URL is http://kibana.subdomain.your-domain.com. Make sure to add the URL in the .env file, located in the root folder:
 +
 
 +
<source lang="bash">
 +
DFE_AUDIT_CLIENT_HOST=kibana.subdomain.your-domain.com
 +
DFE_AUDIT_CLIENT_PORT=5601
 +
</source>
 +
<br />
 
[[File:reports_kibana_1.png|thumbnail|caption|Kibana Discover]]
 
[[File:reports_kibana_1.png|thumbnail|caption|Kibana Discover]]
 
The Discover tab shows the latest API calls, the parameters indexed, and a bar chart with total API calls for the selected time period. You can filter the data based on the indexed parameters on the left side. This method is useful for getting a detailed view of the data.
 
The Discover tab shows the latest API calls, the parameters indexed, and a bar chart with total API calls for the selected time period. You can filter the data based on the indexed parameters on the left side. This method is useful for getting a detailed view of the data.
Line 9: Line 29:
 
Use the Visualize tab to view API calls as line charts, bar charts, pie charts, etc. Select types and filters for X and Y axes to present the data in the preferred way. You can save each visualization, including the selected filters, and open each visualization again later.
 
Use the Visualize tab to view API calls as line charts, bar charts, pie charts, etc. Select types and filters for X and Y axes to present the data in the preferred way. You can save each visualization, including the selected filters, and open each visualization again later.
  
−
A dashboard is collection of visualizations. You can easily insert your saved visualizations into a dashboard. Yiuy can have as many dashboards as you want and each dashboard provides an easy way to present multiple charts on the same screen.
+
A dashboard is collection of visualizations. You can easily insert your saved visualizations into a dashboard. You can have as many dashboards as you want and each dashboard provides an easy way to present multiple charts on the same screen.
  
−
==Create New Dashboard==
+
==Setup Default Dashboard==
−
The following section shows how to create a simple dashboard with two charts, a bar chart and a pie chart. Both charts show the Top 5 most active instances. The bar chart is a stacked bar chart showing daily API calls grouped by the Top 5 instances, and the pie chart shows how the API calls are distributed between the Top 5 instances.  
+
[[File:dashboard_default.png|thumbnail|caption|Default Dashboard]]
 +
There are no dashboards or visualizations pre-configured by the DreamFactory Enterprise installer, but by downloading and running this script, a default dashboard can easily be configured. The default dashboard shows total API calls, top 10 instances, top 10 applications, top 10 users etc. The default dashboard is a good starting point to get familiar with using the ELK stack with DreamFactory Enterprise.  
  
 +
'''''Note: The default dashboard setup script can not create visualizations unless there's data in the Elasticsearch database. There must be at least one instance in use before the script can setup the dashboard'''''
 +
 +
Assuming there's data in the Elasticsearch database (data is stored automatically when an instance is used), download the script setup.sh here [[File:Setup.odt]] (converted to text for security reasons), save it on your server and follow these steps to setup :
 +
<br />
 +
<br />
 +
<source lang="bash">
 +
ubuntu@server:~ $ sudo chmod -x setup.sh
 +
ubuntu@server:~ $ sudo ./setup.sh
 +
</source>
 +
<br />
 +
Open the created dashboard:
 +
<br />
 +
* Go to Kibana
 +
* Click Dashboard in the top menu
 +
* Click the Load Saved Dashboard button
 +
* Click on Dashboard in the list
 +
<br />
 +
Now the newly created dashboard is shown. This page can be bookmarked for quick access.
 +
 +
==Create Custom Dashboard==
 +
The following section shows how to create a simple dashboard with two charts, a bar chart and a pie chart. Both charts show the Top 10 most active instances. The bar chart is a stacked bar chart showing daily API calls grouped by the Top 10 instances, and the pie chart shows how the API calls are distributed between the Top 10 instances.
  
 
===Create Visualization===
 
===Create Visualization===
−
[[File:reports_kibana_2.png|thumbnail|caption|Kibana Visualize Bar Chart]]
+
[[File:Kibana_43_bar.png|thumbnail|caption|Kibana Visualize Bar Chart]]
 
Follow these steps to create a new visualization (bar chart):
 
Follow these steps to create a new visualization (bar chart):
 
<br />
 
<br />
Line 24: Line 66:
 
* The Y-axis is Count by default, no change needed.
 
* The Y-axis is Count by default, no change needed.
 
* Start setting up the X-axis filter by choosing X-axis as bucket type in the left side of the screen.
 
* Start setting up the X-axis filter by choosing X-axis as bucket type in the left side of the screen.
−
* Choose “Date Histogram” as aggregation, @timestamp as field, and Daily as interval. Click Apply in the lower left to update the chart, which now will show total daily API calls across all instances.
+
* Choose “Date Histogram” as aggregation, @timestamp as field, and Daily as interval. Click the green "Apply changes" button (top) to update the chart, which now will show total daily API calls across all instances.
−
* Click Add Sub Aggregation to add another aggregation.  
+
* Click "Add sub-buckets" to add another aggregation.  
 
* Choose “Split Bars” as bucket type, and “Terms” as sub aggregation.
 
* Choose “Split Bars” as bucket type, and “Terms” as sub aggregation.
−
* Choose “dfe.instance_id” under String, choose “Top” as Order, and set Size to 5. Click Apply to update the chart.  
+
* Choose “dfe.instance_id” under String, choose “metric: Count” as Order By, and set Order to Descending and Size to 10. Click the green "Apply changes" button (top) to update the chart   
 
* Click the Save Visualization icon in the top right menu, give your chart a title, and click Save. Your visualization is now saved and you can add it to any dashboard.
 
* Click the Save Visualization icon in the top right menu, give your chart a title, and click Save. Your visualization is now saved and you can add it to any dashboard.
 
<br />
 
<br />
−
[[File:reports_kibana_3.png|thumbnail|caption|Kibana Visualize Pie Chart]]
+
 
 
Creating a pie chart is very similar:
 
Creating a pie chart is very similar:
−
<br />
+
[[File:Kibana_43_pie.png|thumbnail|caption|Kibana Visualize Pie Chart]]
 
* Go to Visualize and choose “Pie chart”.
 
* Go to Visualize and choose “Pie chart”.
 
* Choose “From a new search” as a new search source.
 
* Choose “From a new search” as a new search source.
Line 39: Line 81:
 
* Start setting up the slices by choosing Split Slices as bucket type on the left side of the screen.
 
* Start setting up the slices by choosing Split Slices as bucket type on the left side of the screen.
 
* Choose “Terms” as aggregation.
 
* Choose “Terms” as aggregation.
−
* Choose “dfe.instance_id” under String, choose “Top” as Order, and set Size to 5. Click Apply to update the chart, which now shows how API calls are distributed between instances.   
+
* Choose “dfe.instance_id” under String, choose “metric: Count” as Order By, and set Order to Descending and Size to 10. Click the green "Apply changes" button (top) to update the chart, which now shows how API calls are distributed between instances.   
 
* Click the Save Visualization icon in the top right menu, give your chart a title, and click Save. Your visualization is now saved and you can add it to any dashboard.
 
* Click the Save Visualization icon in the top right menu, give your chart a title, and click Save. Your visualization is now saved and you can add it to any dashboard.
  
 
===Create Dashboard===
 
===Create Dashboard===
−
 
+
[[File:Kibana_43_dashboard.png|thumbnail|caption|Kibana Dashboard]]
 
You can now add the two visualizations (charts) to a dashboard. Follow these steps to create a new dashboard:
 
You can now add the two visualizations (charts) to a dashboard. Follow these steps to create a new dashboard:
−
[[File:reports_kibana_4.png|thumbnail|caption|Kibana Dashboard]]
+
 
 
* Go to Dashboard.
 
* Go to Dashboard.
 
* Click the + icon (Add Visualization) in the top menu.
 
* Click the + icon (Add Visualization) in the top menu.
Line 59: Line 101:
  
 
==Security==
 
==Security==
−
Kibana is not protected by default and can be used by anyone with the URL. It is highly recommended to password protect Kibana. A good way is to add web server authentication for the URL. It can be setup in the configuration files, see the following examples.
+
Kibana is not protected by default and can be used by anyone with the URL. It is highly recommended to password protect Kibana. A good way is to add web server authentication for the URL in the web server configuration files. See the following examples.
  
 
===NGINX===
 
===NGINX===
 
<source lang="bash">
 
<source lang="bash">
−
auth_basic "Restricted";
+
auth_basic ”Restricted”;
 
auth_basic_user_file /etc/nginx/conf.d/kibana.fabric.htpasswd;
 
auth_basic_user_file /etc/nginx/conf.d/kibana.fabric.htpasswd;
 
</source>
 
</source>
Line 72: Line 114:
 
AuthType Basic
 
AuthType Basic
 
AuthName “Restricted”
 
AuthName “Restricted”
−
AuthUserFile /var/www/…./.htpasswd-private
+
AuthUserFile /var/www/..../.htpasswd-private
 
Require valid-user
 
Require valid-user
 
</source>
 
</source>
 
<br>
 
<br>
−
This will force user authentication and the user will be prompted for username and password.
+
This will force user authentication and the user will be prompted for a username and password.
 
<br>
 
<br>
 
Authentication may be different for other web servers.
 
Authentication may be different for other web servers.

Latest revision as of 14:50, 11 May 2016

Introduction

DreamFactory Enterprise utilizes a set of visualization tools from Elasticsearch https://www.elastic.co/. These three tools are Elasticsearch https://www.elastic.co/products/elasticsearch, Logstash https://www.elastic.co/products/logstash, and Kibana https://www.elastic.co/products/kibana. These three applications are commonly referred to as an "ELK" stack. Once a cluster is created and instances provisioned, all API traffic is echoed to the reporting system for visualization.

Initial Configuration

T.B.D.

Configuration

The ELK stack is pre-installed in DreamFactory Enterprise. Check your Kibana configuration file (kibana.yml), it should look like this with default configurations: 

server.port: 5601
server.host: "0.0.0.0"
elasticsearch.url: "http://localhost:9200"
kibana.index: ".kibana"


Introduction to Kibana

Go to the Kibana URL in your DreamFactory Enterprise installation. Usually the URL is http://kibana.subdomain.your-domain.com. Make sure to add the URL in the .env file, located in the root folder: 

DFE_AUDIT_CLIENT_HOST=kibana.subdomain.your-domain.com
DFE_AUDIT_CLIENT_PORT=5601


Kibana Discover

The Discover tab shows the latest API calls, the parameters indexed, and a bar chart with total API calls for the selected time period. You can filter the data based on the indexed parameters on the left side. This method is useful for getting a detailed view of the data.

Use the Visualize tab to view API calls as line charts, bar charts, pie charts, etc. Select types and filters for X and Y axes to present the data in the preferred way. You can save each visualization, including the selected filters, and open each visualization again later.

A dashboard is collection of visualizations. You can easily insert your saved visualizations into a dashboard. You can have as many dashboards as you want and each dashboard provides an easy way to present multiple charts on the same screen.

Setup Default Dashboard

Default Dashboard

There are no dashboards or visualizations pre-configured by the DreamFactory Enterprise installer, but by downloading and running this script, a default dashboard can easily be configured. The default dashboard shows total API calls, top 10 instances, top 10 applications, top 10 users etc. The default dashboard is a good starting point to get familiar with using the ELK stack with DreamFactory Enterprise.

Note: The default dashboard setup script can not create visualizations unless there's data in the Elasticsearch database. There must be at least one instance in use before the script can setup the dashboard

Assuming there's data in the Elasticsearch database (data is stored automatically when an instance is used), download the script setup.sh here File:Setup.odt (converted to text for security reasons), save it on your server and follow these steps to setup : 

ubuntu@server:~ $ sudo chmod -x setup.sh 
ubuntu@server:~ $ sudo ./setup.sh


Open the created dashboard:

  • Go to Kibana
  • Click Dashboard in the top menu
  • Click the Load Saved Dashboard button
  • Click on Dashboard in the list


Now the newly created dashboard is shown. This page can be bookmarked for quick access.

Create Custom Dashboard

The following section shows how to create a simple dashboard with two charts, a bar chart and a pie chart. Both charts show the Top 10 most active instances. The bar chart is a stacked bar chart showing daily API calls grouped by the Top 10 instances, and the pie chart shows how the API calls are distributed between the Top 10 instances.

Create Visualization

Kibana Visualize Bar Chart

Follow these steps to create a new visualization (bar chart):

  • Go to Visualize and choose "Vertical bar chart".
  • Choose “From a new search” as a new search source.
  • A bar chart without any filters will be shown. By default, the last 15 minutes is shown. Expand the timespan selection by clicking the link in the very top right corner of the screen (where the default timespan "Last 15 minutes" is shown), and choose “Last 7 days”. Collapse the timespan selection again.
  • The Y-axis is Count by default, no change needed.
  • Start setting up the X-axis filter by choosing X-axis as bucket type in the left side of the screen.
  • Choose “Date Histogram” as aggregation, @timestamp as field, and Daily as interval. Click the green "Apply changes" button (top) to update the chart, which now will show total daily API calls across all instances.
  • Click "Add sub-buckets" to add another aggregation.
  • Choose “Split Bars” as bucket type, and “Terms” as sub aggregation.
  • Choose “dfe.instance_id” under String, choose “metric: Count” as Order By, and set Order to Descending and Size to 10. Click the green "Apply changes" button (top) to update the chart
  • Click the Save Visualization icon in the top right menu, give your chart a title, and click Save. Your visualization is now saved and you can add it to any dashboard.


Creating a pie chart is very similar:

Kibana Visualize Pie Chart
  • Go to Visualize and choose “Pie chart”.
  • Choose “From a new search” as a new search source.
  • A pie chart without any filters will be shown. By default, the last 15 minutes is shown. Expand the timespan selection by clicking the link in the very top right corner of the screen (where the default timespan "Last 15 minutes" is shown), and choose “Last 7 days”. Collapse the timespan selection again.
  • The Slice Size is Count by default, no change needed.
  • Start setting up the slices by choosing Split Slices as bucket type on the left side of the screen.
  • Choose “Terms” as aggregation.
  • Choose “dfe.instance_id” under String, choose “metric: Count” as Order By, and set Order to Descending and Size to 10. Click the green "Apply changes" button (top) to update the chart, which now shows how API calls are distributed between instances.
  • Click the Save Visualization icon in the top right menu, give your chart a title, and click Save. Your visualization is now saved and you can add it to any dashboard.

Create Dashboard

Kibana Dashboard

You can now add the two visualizations (charts) to a dashboard. Follow these steps to create a new dashboard:

  • Go to Dashboard.
  • Click the + icon (Add Visualization) in the top menu.
  • Select one or multiple visualizations (charts), in this case the bar chart and pie chart you just created.
  • Drag and resize the visualizations to position them.
  • Click the Save Dashboard icon in the upper right of the top menu, give your dashboard a title, and click Save. You can now open your dashboard again at any time.

Embed

All the settings are stored as URL parameters, so it's easy to embed and share your visualizations and dashboards. Click the Share icon in the top right menu to get a link and HTML iframe script for embedding.

If the charts are embedded, the top menu etc. will not be visible. The menu and other controls are hidden by adding “embed” as a URL parameter.

Security

Kibana is not protected by default and can be used by anyone with the URL. It is highly recommended to password protect Kibana. A good way is to add web server authentication for the URL in the web server configuration files. See the following examples.

NGINX

auth_basic ”Restricted”;
auth_basic_user_file /etc/nginx/conf.d/kibana.fabric.htpasswd;


Apache

AuthType Basic
AuthName “Restricted”
AuthUserFile /var/www/..../.htpasswd-private
Require valid-user


This will force user authentication and the user will be prompted for a username and password.
Authentication may be different for other web servers.

Retrieved from "https://wiki.dreamfactory.com/index.php?title=DFE/Reports&oldid=32207"