User:BDavis (WMF)/Projects/Logstash

Notes on Logstash in Labs

Setup server[edit]

Added firewall hole for ports 80 and 9200.
Turned on shared home dirs and project storage
Created x-large instance named logstash
ssh logstash.pmtpa.wmflabs
sudo puppetd --test --verbose
sudo reboot
Added puppet classes:
- role::labsnfs::client
- webserver::php5
ssh logstash.pmtpa.wmflabs
sudo puppetd --test --verbose
sudo reboot
ssh logstash.pmtpa.wmflabs
verified that nfs client mounted /home and /data/project
created /data/project/logstash to hold project files
wget https://download.elasticsearch.org/logstash/logstash/logstash-1.2.2-flatjar.jar
sudo apt-get install openjdk-7-jdk

Simple test of logstash[edit]

created simple testing config for logstash:

input {
    stdin { }
}
output {
    stdout {
        codec => rubydebug
    }
    elasticsearch {
        embedded => true
    }
}

java -jar logstash-1.2.2-flatjar.jar agent -f logstash-simple.conf
download kibana3 and install in /data/project/logstash/kibana
edit /data/project/logstash/kibana/config.js
- elasticsearch: "http://"+window.location.hostname,
make /data/project/logstash/kibana/app/dashboards/default.json a symlink to logstash.json
created apache reverse proxy conf in /data/project/logstash/apache:

<VirtualHost *:80>
  ServerAdmin webmaster@localhost
  DocumentRoot /data/project/logstash/kibana
  <Directory />
    Options FollowSymLinks
    AllowOverride None
  </Directory>
  <Directory /data/project/logstash/kibana>
    Order allow,deny
    allow from all
  </Directory>

  ErrorLog ${APACHE_LOG_DIR}/error.log
  LogLevel warn
  CustomLog ${APACHE_LOG_DIR}/access.log combined

  <IfModule mod_proxy.c>
    ProxyRequests Off

    <Proxy http://127.0.0.1:9200>
      ProxySet connectiontimeout=5 timeout=90
    </Proxy>

    # Proxy for _aliases and .*/_search
    <LocationMatch "^/(_nodes|_aliases|_search|.*/_search|_mapping|.*/_mapping)$">
      ProxyPassMatch http://127.0.0.1:9200/$1
      ProxyPassReverse http://127.0.0.1:9200/$1
    </LocationMatch>

    # Proxy for kibana-int/{dashboard,temp} stuff (if you don't want auth on
    # /, then you will want these to be protected)
    <LocationMatch "^/(kibana-int/dashboard/|kibana-int/temp)(.*)$">
      ProxyPassMatch http://127.0.0.1:9200/$1$2
      ProxyPassReverse http://127.0.0.1:9200/$1$2
    </LocationMatch>

  </IfModule>

</VirtualHost>

sudo a2enmod proxy
sudo a2enmod proxy_http
sudo service apache restart
cd /etc/apache2/sites-available
sudo ln -s /data/project/logstash/apache/proxy
sudo service apache2 reload
browse to http://logstash.instance-proxy.wmflabs.org/ and see kibana load!