User:BDavis (WMF)/Projects/Logstash

From MediaWiki.org
Jump to navigation Jump to search

Notes on Logstash in Labs

Setup server[edit]

  • Added firewall hole for ports 80 and 9200.
  • Turned on shared home dirs and project storage
  • Created x-large instance named logstash
  • ssh logstash.pmtpa.wmflabs
  • sudo puppetd --test --verbose
  • sudo reboot
  • Added puppet classes:
    • role::labsnfs::client
    • webserver::php5
  • ssh logstash.pmtpa.wmflabs
  • sudo puppetd --test --verbose
  • sudo reboot
  • ssh logstash.pmtpa.wmflabs
  • verified that nfs client mounted /home and /data/project
  • created /data/project/logstash to hold project files
  • wget https://download.elasticsearch.org/logstash/logstash/logstash-1.2.2-flatjar.jar
  • sudo apt-get install openjdk-7-jdk

Simple test of logstash[edit]

  • created simple testing config for logstash:
input {
    stdin { }
}
output {
    stdout {
        codec => rubydebug
    }
    elasticsearch {
        embedded => true
    }
}
  • java -jar logstash-1.2.2-flatjar.jar agent -f logstash-simple.conf
  • download kibana3 and install in /data/project/logstash/kibana
  • edit /data/project/logstash/kibana/config.js
    • elasticsearch: "http://"+window.location.hostname,
  • make /data/project/logstash/kibana/app/dashboards/default.json a symlink to logstash.json
  • created apache reverse proxy conf in /data/project/logstash/apache:
<VirtualHost *:80>
  ServerAdmin webmaster@localhost
  DocumentRoot /data/project/logstash/kibana
  <Directory />
    Options FollowSymLinks
    AllowOverride None
  </Directory>
  <Directory /data/project/logstash/kibana>
    Order allow,deny
    allow from all
  </Directory>

  ErrorLog ${APACHE_LOG_DIR}/error.log
  LogLevel warn
  CustomLog ${APACHE_LOG_DIR}/access.log combined

  <IfModule mod_proxy.c>
    ProxyRequests Off

    <Proxy http://127.0.0.1:9200>
      ProxySet connectiontimeout=5 timeout=90
    </Proxy>

    # Proxy for _aliases and .*/_search
    <LocationMatch "^/(_nodes|_aliases|_search|.*/_search|_mapping|.*/_mapping)$">
      ProxyPassMatch http://127.0.0.1:9200/$1
      ProxyPassReverse http://127.0.0.1:9200/$1
    </LocationMatch>

    # Proxy for kibana-int/{dashboard,temp} stuff (if you don't want auth on
    # /, then you will want these to be protected)
    <LocationMatch "^/(kibana-int/dashboard/|kibana-int/temp)(.*)$">
      ProxyPassMatch http://127.0.0.1:9200/$1$2
      ProxyPassReverse http://127.0.0.1:9200/$1$2
    </LocationMatch>

  </IfModule>

</VirtualHost>
  • sudo a2enmod proxy
  • sudo a2enmod proxy_http
  • sudo service apache restart
  • cd /etc/apache2/sites-available
  • sudo ln -s /data/project/logstash/apache/proxy
  • sudo service apache2 reload
  • browse to http://logstash.instance-proxy.wmflabs.org/ and see kibana load!