ざっくりまとめ、その後自分でやってみたことも含む

今日から始める PHPエンジニアのためのアクセスログ解析基盤構築入門祖山さん

http://who-you-me.github.io/slides/phpcon2014/#/

生ログをどう解析するか？について
主な目的は
・ログ調査を効率化する
・ユーザの行動履歴を分析する
効率化は、fluentd ＆ Elasticsearch ＆ Kibana がオススメ
行動履歴分析は、fluentd ＆ BigQuery がオススメ、SQL感覚で操作可能

まずはfluentd ＆ Elasticsearch ＆ Kibana を構築してみた

GOAL

apache

LTSV形式でログを保存するようにする、これをfluentdに送る

root@hostname:/home/shimizu# cat /etc/apache2/apache2.conf
...
LogFormat "unixtime:%{%s}t\tdatetime:%{%d/%b/%Y:%H:%M:%S %z}t\tx-forwaded-for:%{X-Forwarded-For}i\thost:%h\tsize:%B\tresponse_time:%D\tstatus:%>s\tserver:%A\tHost:%{Host}i\tmethod:%m\tpath:%U%q\tprotocol:%H\tUA:%{User-Agent}i\treferer:%{Referer}i" ltsv
...
root@hostname:/home/shimizu# vi /etc/apache2/sites-enabled/000-default 
...
CustomLog /var/www/log/access.log ltsv
...
root@hostname:/home/shimizu# apache2ctl configtest
Syntax OK
root@hostname:/home/shimizu# /etc/init.d/apache2 restart
[ ok ] Restarting web server: apache2 ... waiting .

fluentdをインストール後、必要なプラグインをインストール

※fluentdのインストールは割愛

root@hostname:/home/shimizu# /usr/lib/fluent/ruby/bin/fluent-gem install fluent-plugin-filter
Fetching: fluent-plugin-filter-0.0.1.gem (100%)
Successfully installed fluent-plugin-filter-0.0.1
1 gem installed
Installing ri documentation for fluent-plugin-filter-0.0.1...
Installing RDoc documentation for fluent-plugin-filter-0.0.1...
root@hostname:/home/shimizu# /usr/lib/fluent/ruby/bin/fluent-gem install fluent-plugin-forest
Fetching: fluent-plugin-forest-0.3.0.gem (100%)
Successfully installed fluent-plugin-forest-0.3.0
1 gem installed
Installing ri documentation for fluent-plugin-forest-0.3.0...
Installing RDoc documentation for fluent-plugin-forest-0.3.0...
root@hostname:/home/shimizu# /usr/lib/fluent/ruby/bin/fluent-gem install fluent-plugin-config-expander
Fetching: fluent-plugin-config-expander-0.1.5.gem (100%)
Successfully installed fluent-plugin-config-expander-0.1.5
1 gem installed
Installing ri documentation for fluent-plugin-config-expander-0.1.5...
Installing RDoc documentation for fluent-plugin-config-expander-0.1.5...
root@hostname:/home/shimizu# /usr/lib/fluent/ruby/bin/fluent-gem install fluent-plugin-typecast
Fetching: fluent-plugin-typecast-0.1.2.gem (100%)
Successfully installed fluent-plugin-typecast-0.1.2
1 gem installed
Installing ri documentation for fluent-plugin-typecast-0.1.2...
Installing RDoc documentation for fluent-plugin-typecast-0.1.2...
root@hostname:/home/shimizu# /usr/lib/fluent/ruby/bin/fluent-gem install fluent-plugin-parser
Fetching: fluent-plugin-parser-0.3.4.gem (100%)
Successfully installed fluent-plugin-parser-0.3.4
1 gem installed
Installing ri documentation for fluent-plugin-parser-0.3.4...
Installing RDoc documentation for fluent-plugin-parser-0.3.4...

root@hostname:/home/shimizu# /usr/lib/fluent/ruby/bin/fluent-gem install fluent-plugin-elasticsearch
Fetching: patron-0.4.18.gem (100%)
Building native extensions.  This could take a while...
ERROR:  Error installing fluent-plugin-elasticsearch:
        ERROR: Failed to build gem native extension.

        /usr/lib/fluent/ruby/bin/ruby extconf.rb
checking for curl-config... no
checking for main() in -lcurl... no
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of
necessary libraries and/or headers.  Check the mkmf.log file for more
details.  You may need configuration options.

Provided configuration options:
        --with-opt-dir
        --without-opt-dir
        --with-opt-include
        --without-opt-include=${opt-dir}/include
        --with-opt-lib
        --without-opt-lib=${opt-dir}/lib
        --with-make-prog
        --without-make-prog
        --srcdir=.
        --curdir
        --ruby=/usr/lib/fluent/ruby/bin/ruby
        --with-curl-dir
        --without-curl-dir
        --with-curl-include
        --without-curl-include=${curl-dir}/include
        --with-curl-lib
        --without-curl-lib=${curl-dir}/lib
        --with-curllib
        --without-curllib
extconf.rb:39:in `<main>':   Can't find libcurl or curl/curl.h (RuntimeError)

  Try passing --with-curl-dir or --with-curl-lib and --with-curl-include
  options to extconf.


Gem files will remain installed in /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/patron-0.4.18 for inspection.
Results logged to /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/patron-0.4.18/ext/patron/gem_make.out

### いろいろインストールしたら、うまくいくようになった ###
root@hostname:/home/shimizu# aptitude install ruby1.9.1-dev
以下の新規パッケージがインストールされます:
  libruby1.9.1{a} ruby1.9.1{a} ruby1.9.1-dev 
...

root@hostname:/home/shimizu# aptitude install liberuby-dev
以下の新規パッケージがインストールされます:
  liberuby{a} liberuby-dev libreadline5{a} libruby1.8{a} 
...

root@hostname:/home/shimizu# aptitude install libcurl4-gnutls-dev
以下の新規パッケージがインストールされます:
  comerr-dev{a} krb5-multidev{a} libcurl4-gnutls-dev libgcrypt11-dev{a} libgnutls-dev{a} libgnutls-openssl27{a} libgnutlsxx27{a} 
  libgpg-error-dev{a} libgssrpc4{a} libidn11-dev{a} libkadm5clnt-mit8{a} libkadm5srv-mit8{a} libkdb5-6{a} libkrb5-dev{a} libldap2-dev{a} 
  libp11-kit-dev{a} librtmp-dev{a} libssh2-1-dev{a} libtasn1-3-dev{a} pkg-config{a} 
...

root@hostname:/home/shimizu# /usr/lib/fluent/ruby/bin/fluent-gem install fluent-plugin-elasticsearch
Building native extensions.  This could take a while...
Fetching: multi_json-1.10.1.gem (100%)
Fetching: multipart-post-2.0.0.gem (100%)
Fetching: faraday-0.9.0.gem (100%)
Fetching: elasticsearch-transport-1.0.5.gem (100%)
Fetching: elasticsearch-api-1.0.5.gem (100%)
Fetching: elasticsearch-1.0.5.gem (100%)
Fetching: fluent-plugin-elasticsearch-0.5.1.gem (100%)
Successfully installed patron-0.4.18
Successfully installed multi_json-1.10.1
Successfully installed multipart-post-2.0.0
Successfully installed faraday-0.9.0
Successfully installed elasticsearch-transport-1.0.5
Successfully installed elasticsearch-api-1.0.5
Successfully installed elasticsearch-1.0.5
Successfully installed fluent-plugin-elasticsearch-0.5.1
8 gems installed
Installing ri documentation for patron-0.4.18...
Installing ri documentation for multi_json-1.10.1...
Installing ri documentation for multipart-post-2.0.0...
Installing ri documentation for faraday-0.9.0...
Installing ri documentation for elasticsearch-transport-1.0.5...
Installing ri documentation for elasticsearch-api-1.0.5...
Installing ri documentation for elasticsearch-1.0.5...
Installing ri documentation for fluent-plugin-elasticsearch-0.5.1...
Installing RDoc documentation for patron-0.4.18...
Installing RDoc documentation for multi_json-1.10.1...
Installing RDoc documentation for multipart-post-2.0.0...
Installing RDoc documentation for faraday-0.9.0...
Installing RDoc documentation for elasticsearch-transport-1.0.5...
Installing RDoc documentation for elasticsearch-api-1.0.5...
Installing RDoc documentation for elasticsearch-1.0.5...
Installing RDoc documentation for fluent-plugin-elasticsearch-0.5.1...

Elasticsearch

分散型全文検索システム、AWSのサービスではない

root@hostname:/usr/local/src# wget -d https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.3.4.tar.gz
...
root@hostname:/usr/local/src# tar xzvf elasticsearch-1.3.4.tar.gz 
...
root@hostname:/usr/local/src# cat /usr/local/src/elasticsearch-1.3.4/config/elasticsearch.yml
...設定ファイル
root@hostname:/usr/local/src# /usr/local/src/elasticsearch-1.3.4/bin/elasticsearch
[2014-10-13 17:47:42,085][INFO ][node                     ] [Bucky III] version[1.3.4], pid[870], build[a70f3cc/2014-09-30T09:07:17Z]
[2014-10-13 17:47:42,086][INFO ][node                     ] [Bucky III] initializing ...
[2014-10-13 17:47:42,094][INFO ][plugins                  ] [Bucky III] loaded [], sites []
...

root@hostname:/home/shimizu# curl -X GET http://localhost:9200/
{
  "status" : 200,
  "name" : "Bucky III",
  "version" : {
    "number" : "1.3.4",
    "build_hash" : "a70f3ccb52200f8f2c87e9c370c6597448eb3e45",
    "build_timestamp" : "2014-09-30T09:07:17Z",
    "build_snapshot" : false,
    "lucene_version" : "4.9"
  },
  "tagline" : "You Know, for Search"
}

Kibana

Elasticsearchのデータをグラフなどに可視化するためのフロントエンドのツール
クライアントで動くjavascriptのビジュアライズツール、AjaxでサーバにあるElasticsearchとKibanaが通信する
HTML/javascript/CSSだけで動作するため導入しやすい

root@hostname:/usr/local/src# wget -d https://download.elasticsearch.org/kibana/kibana/kibana-3.1.1.tar.gz
...
root@hostname:/usr/local/src# tar xzvf kibana-3.1.1.tar.gz
...
root@hostname:/usr/local/src# cat kibana-3.1.1/config.js
...
### elasticsearchサーバを指定する。Kibanaはブラウザからjsで動作するため、localhostなどで指定してはならない
### セキュリティホールになりやすいため要注意
// elasticsearch: "http://"+window.location.hostname+":9200",
elasticsearch: "http://49.212.204.46:9200",
...
root@hostname:/usr/local/src# cat /etc/apache2/sites-enabled/000-default 
...
        # kibana用フォルダを指定
        alias /kibana /usr/local/src/kibana-3.1.1
...
root@hostname:/home/shimizu# apache2ctl configtest
Syntax OK
root@hostname:/home/shimizu# /etc/init.d/apache2 restart
[ ok ] Restarting web server: apache2 ... waiting .

アクセスすると

参考URL

今日から始めるfluentd × Elasticsearch × kibana – カジュアルな解析・高速化
http://eure.jp/blog/fluentd_elasticsearch_kibana/
Elasticsearchをリポジトリからインストールする方法
http://www.elasticsearch.org/blog/apt-and-yum-repositories/
Elasticsearch Kibana でデータの可視化
https://medium.com/hello-elasticsearch/elasticsearch-kibana-6ad80550939f
fluentd + Elasticsearch + Kibanaで始めるログ解析 (セットアップ編)
http://blog.excale.net/index.php/archives/1929

今日から始める PHPエンジニアのためのアクセスログ解析基盤構築入門 祖山さん

GOAL

apache

fluentdをインストール後、必要なプラグインをインストール

Elasticsearch

Kibana

参考URL

おすすめ

驚愕! Azure リソース マネージャを利用した Infrastructure as Code 実践 を閲覧した

Interact 2018 に参加した(1)

Security Online Day 2014 いま、企業が守るべきものとは何か？ に参加した

今日から始める PHPエンジニアのためのアクセスログ解析基盤構築入門祖山さん

驚愕! Azure リソースマネージャを利用した Infrastructure as Code 実践を閲覧した

Security Online Day 2014 いま、企業が守るべきものとは何か？に参加した