User:BDavis (WMF)/Projects/Logstash

Notes on Logstash in Labs

Setup server

 * Added firewall hole for ports 80 and 9200.
 * Turned on shared home dirs and project storage
 * Created x-large instance named logstash
 * ssh logstash.pmtpa.wmflabs
 * sudo puppetd --test --verbose
 * sudo reboot
 * Added puppet classes:
 * role::labsnfs::client
 * webserver::php5
 * ssh logstash.pmtpa.wmflabs
 * sudo puppetd --test --verbose
 * sudo reboot
 * ssh logstash.pmtpa.wmflabs
 * verified that nfs client mounted /home and /data/project
 * created /data/project/logstash to hold project files
 * wget https://download.elasticsearch.org/logstash/logstash/logstash-1.2.2-flatjar.jar
 * sudo apt-get install openjdk-7-jdk

Simple test of logstash
input { stdin { } } output { stdout { codec => rubydebug }   elasticsearch { embedded => true } }  ServerAdmin webmaster@localhost DocumentRoot /data/project/logstash/kibana  Options FollowSymLinks AllowOverride None   Order allow,deny allow from all 
 * created simple testing config for logstash:
 * java -jar logstash-1.2.2-flatjar.jar agent -f logstash-simple.conf
 * download kibana3 and install in /data/project/logstash/kibana
 * edit /data/project/logstash/kibana/config.js
 * make /data/project/logstash/kibana/app/dashboards/default.json a symlink to logstash.json
 * created apache reverse proxy conf in /data/project/logstash/apache:
 * created apache reverse proxy conf in /data/project/logstash/apache:

ErrorLog ${APACHE_LOG_DIR}/error.log LogLevel warn CustomLog ${APACHE_LOG_DIR}/access.log combined

   ProxyRequests Off

 ProxySet connectiontimeout=5 timeout=90 

# Proxy for _aliases and .*/_search  ProxyPassMatch http://127.0.0.1:9200/$1 ProxyPassReverse http://127.0.0.1:9200/$1 

# Proxy for kibana-int/{dashboard,temp} stuff (if you don't want auth on   # /, then you will want these to be protected)  ProxyPassMatch http://127.0.0.1:9200/$1$2 ProxyPassReverse http://127.0.0.1:9200/$1$2 




 * sudo a2enmod proxy
 * sudo a2enmod proxy_http
 * sudo service apache restart
 * cd /etc/apache2/sites-available
 * sudo ln -s /data/project/logstash/apache/proxy
 * sudo service apache2 reload
 * browse to http://logstash.instance-proxy.wmflabs.org/ and see kibana load!