ざっくりまとめ、その後自分でやってみたことも含む

今日から始める PHPエンジニアのためのアクセスログ解析基盤構築入門祖山さん

http://who-you-me.github.io/slides/phpcon2014/#/

生ログをどう解析するか？について
主な目的は
・ログ調査を効率化する
・ユーザの行動履歴を分析する
効率化は、fluentd ＆ Elasticsearch ＆ Kibana がオススメ
行動履歴分析は、fluentd ＆ BigQuery がオススメ、SQL感覚で操作可能

まずはfluentd ＆ Elasticsearch ＆ Kibana を構築してみた

GOAL

apache

LTSV形式でログを保存するようにする、これをfluentdに送る

root@hostname:/home/shimizu# cat /etc/apache2/apache2.conf
...
LogFormat "unixtime:%{%s}t\tdatetime:%{%d/%b/%Y:%H:%M:%S %z}t\tx-forwaded-for:%{X-Forwarded-For}i\thost:%h\tsize:%B\tresponse_time:%D\tstatus:%>s\tserver:%A\tHost:%{Host}i\tmethod:%m\tpath:%U%q\tprotocol:%H\tUA:%{User-Agent}i\treferer:%{Referer}i" ltsv
...
root@hostname:/home/shimizu# vi /etc/apache2/sites-enabled/000-default 
...
CustomLog /var/www/log/access.log ltsv
...
root@hostname:/home/shimizu# apache2ctl configtest
Syntax OK
root@hostname:/home/shimizu# /etc/init.d/apache2 restart
[ ok ] Restarting web server: apache2 ... waiting .

fluentdをインストール後、必要なプラグインをインストール

※fluentdのインストールは割愛

root@hostname:/home/shimizu# /usr/lib/fluent/ruby/bin/fluent-gem install fluent-plugin-filter
Fetching: fluent-plugin-filter-0.0.1.gem (100%)
Successfully installed fluent-plugin-filter-0.0.1
1 gem installed
Installing ri documentation for fluent-plugin-filter-0.0.1...
Installing RDoc documentation for fluent-plugin-filter-0.0.1...
root@hostname:/home/shimizu# /usr/lib/fluent/ruby/bin/fluent-gem install fluent-plugin-forest
Fetching: fluent-plugin-forest-0.3.0.gem (100%)
Successfully installed fluent-plugin-forest-0.3.0
1 gem installed
Installing ri documentation for fluent-plugin-forest-0.3.0...
Installing RDoc documentation for fluent-plugin-forest-0.3.0...
root@hostname:/home/shimizu# /usr/lib/fluent/ruby/bin/fluent-gem install fluent-plugin-config-expander
Fetching: fluent-plugin-config-expander-0.1.5.gem (100%)
Successfully installed fluent-plugin-config-expander-0.1.5
1 gem installed
Installing ri documentation for fluent-plugin-config-expander-0.1.5...
Installing RDoc documentation for fluent-plugin-config-expander-0.1.5...
root@hostname:/home/shimizu# /usr/lib/fluent/ruby/bin/fluent-gem install fluent-plugin-typecast
Fetching: fluent-plugin-typecast-0.1.2.gem (100%)
Successfully installed fluent-plugin-typecast-0.1.2
1 gem installed
Installing ri documentation for fluent-plugin-typecast-0.1.2...
Installing RDoc documentation for fluent-plugin-typecast-0.1.2...
root@hostname:/home/shimizu# /usr/lib/fluent/ruby/bin/fluent-gem install fluent-plugin-parser
Fetching: fluent-plugin-parser-0.3.4.gem (100%)
Successfully installed fluent-plugin-parser-0.3.4
1 gem installed
Installing ri documentation for fluent-plugin-parser-0.3.4...
Installing RDoc documentation for fluent-plugin-parser-0.3.4...

root@hostname:/home/shimizu# /usr/lib/fluent/ruby/bin/fluent-gem install fluent-plugin-elasticsearch
Fetching: patron-0.4.18.gem (100%)
Building native extensions.  This could take a while...
ERROR:  Error installing fluent-plugin-elasticsearch:
        ERROR: Failed to build gem native extension.

        /usr/lib/fluent/ruby/bin/ruby extconf.rb
checking for curl-config... no
checking for main() in -lcurl... no
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of
necessary libraries and/or headers.  Check the mkmf.log file for more
details.  You may need configuration options.

Provided configuration options:
        --with-opt-dir
        --without-opt-dir
        --with-opt-include
        --without-opt-include=${opt-dir}/include
        --with-opt-lib
        --without-opt-lib=${opt-dir}/lib
        --with-make-prog
        --without-make-prog
        --srcdir=.
        --curdir
        --ruby=/usr/lib/fluent/ruby/bin/ruby
        --with-curl-dir
        --without-curl-dir
        --with-curl-include
        --without-curl-include=${curl-dir}/include
        --with-curl-lib
        --without-curl-lib=${curl-dir}/lib
        --with-curllib
        --without-curllib
extconf.rb:39:in `<main>':   Can't find libcurl or curl/curl.h (RuntimeError)

  Try passing --with-curl-dir or --with-curl-lib and --with-curl-include
  options to extconf.


Gem files will remain installed in /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/patron-0.4.18 for inspection.
Results logged to /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/patron-0.4.18/ext/patron/gem_make.out

### いろいろインストールしたら、うまくいくようになった ###
root@hostname:/home/shimizu# aptitude install ruby1.9.1-dev
以下の新規パッケージがインストールされます:
  libruby1.9.1{a} ruby1.9.1{a} ruby1.9.1-dev 
...

root@hostname:/home/shimizu# aptitude install liberuby-dev
以下の新規パッケージがインストールされます:
  liberuby{a} liberuby-dev libreadline5{a} libruby1.8{a} 
...

root@hostname:/home/shimizu# aptitude install libcurl4-gnutls-dev
以下の新規パッケージがインストールされます:
  comerr-dev{a} krb5-multidev{a} libcurl4-gnutls-dev libgcrypt11-dev{a} libgnutls-dev{a} libgnutls-openssl27{a} libgnutlsxx27{a} 
  libgpg-error-dev{a} libgssrpc4{a} libidn11-dev{a} libkadm5clnt-mit8{a} libkadm5srv-mit8{a} libkdb5-6{a} libkrb5-dev{a} libldap2-dev{a} 
  libp11-kit-dev{a} librtmp-dev{a} libssh2-1-dev{a} libtasn1-3-dev{a} pkg-config{a} 
...

root@hostname:/home/shimizu# /usr/lib/fluent/ruby/bin/fluent-gem install fluent-plugin-elasticsearch
Building native extensions.  This could take a while...
Fetching: multi_json-1.10.1.gem (100%)
Fetching: multipart-post-2.0.0.gem (100%)
Fetching: faraday-0.9.0.gem (100%)
Fetching: elasticsearch-transport-1.0.5.gem (100%)
Fetching: elasticsearch-api-1.0.5.gem (100%)
Fetching: elasticsearch-1.0.5.gem (100%)
Fetching: fluent-plugin-elasticsearch-0.5.1.gem (100%)
Successfully installed patron-0.4.18
Successfully installed multi_json-1.10.1
Successfully installed multipart-post-2.0.0
Successfully installed faraday-0.9.0
Successfully installed elasticsearch-transport-1.0.5
Successfully installed elasticsearch-api-1.0.5
Successfully installed elasticsearch-1.0.5
Successfully installed fluent-plugin-elasticsearch-0.5.1
8 gems installed
Installing ri documentation for patron-0.4.18...
Installing ri documentation for multi_json-1.10.1...
Installing ri documentation for multipart-post-2.0.0...
Installing ri documentation for faraday-0.9.0...
Installing ri documentation for elasticsearch-transport-1.0.5...
Installing ri documentation for elasticsearch-api-1.0.5...
Installing ri documentation for elasticsearch-1.0.5...
Installing ri documentation for fluent-plugin-elasticsearch-0.5.1...
Installing RDoc documentation for patron-0.4.18...
Installing RDoc documentation for multi_json-1.10.1...
Installing RDoc documentation for multipart-post-2.0.0...
Installing RDoc documentation for faraday-0.9.0...
Installing RDoc documentation for elasticsearch-transport-1.0.5...
Installing RDoc documentation for elasticsearch-api-1.0.5...
Installing RDoc documentation for elasticsearch-1.0.5...
Installing RDoc documentation for fluent-plugin-elasticsearch-0.5.1...

Elasticsearch

分散型全文検索システム、AWSのサービスではない

root@hostname:/usr/local/src# wget -d https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.3.4.tar.gz
...
root@hostname:/usr/local/src# tar xzvf elasticsearch-1.3.4.tar.gz 
...
root@hostname:/usr/local/src# cat /usr/local/src/elasticsearch-1.3.4/config/elasticsearch.yml
...設定ファイル
root@hostname:/usr/local/src# /usr/local/src/elasticsearch-1.3.4/bin/elasticsearch
[2014-10-13 17:47:42,085][INFO ][node                     ] [Bucky III] version[1.3.4], pid[870], build[a70f3cc/2014-09-30T09:07:17Z]
[2014-10-13 17:47:42,086][INFO ][node                     ] [Bucky III] initializing ...
[2014-10-13 17:47:42,094][INFO ][plugins                  ] [Bucky III] loaded [], sites []
...

root@hostname:/home/shimizu# curl -X GET http://localhost:9200/
{
  "status" : 200,
  "name" : "Bucky III",
  "version" : {
    "number" : "1.3.4",
    "build_hash" : "a70f3ccb52200f8f2c87e9c370c6597448eb3e45",
    "build_timestamp" : "2014-09-30T09:07:17Z",
    "build_snapshot" : false,
    "lucene_version" : "4.9"
  },
  "tagline" : "You Know, for Search"
}

Kibana

Elasticsearchのデータをグラフなどに可視化するためのフロントエンドのツール
クライアントで動くjavascriptのビジュアライズツール、AjaxでサーバにあるElasticsearchとKibanaが通信する
HTML/javascript/CSSだけで動作するため導入しやすい

root@hostname:/usr/local/src# wget -d https://download.elasticsearch.org/kibana/kibana/kibana-3.1.1.tar.gz
...
root@hostname:/usr/local/src# tar xzvf kibana-3.1.1.tar.gz
...
root@hostname:/usr/local/src# cat kibana-3.1.1/config.js
...
### elasticsearchサーバを指定する。Kibanaはブラウザからjsで動作するため、localhostなどで指定してはならない
### セキュリティホールになりやすいため要注意
// elasticsearch: "http://"+window.location.hostname+":9200",
elasticsearch: "http://49.212.204.46:9200",
...
root@hostname:/usr/local/src# cat /etc/apache2/sites-enabled/000-default 
...
        # kibana用フォルダを指定
        alias /kibana /usr/local/src/kibana-3.1.1
...
root@hostname:/home/shimizu# apache2ctl configtest
Syntax OK
root@hostname:/home/shimizu# /etc/init.d/apache2 restart
[ ok ] Restarting web server: apache2 ... waiting .

アクセスすると

参考URL

今日から始めるfluentd × Elasticsearch × kibana – カジュアルな解析・高速化
http://eure.jp/blog/fluentd_elasticsearch_kibana/
Elasticsearchをリポジトリからインストールする方法
http://www.elasticsearch.org/blog/apt-and-yum-repositories/
Elasticsearch Kibana でデータの可視化
https://medium.com/hello-elasticsearch/elasticsearch-kibana-6ad80550939f
fluentd + Elasticsearch + Kibanaで始めるログ解析 (セットアップ編)
http://blog.excale.net/index.php/archives/1929

今日から始める PHPエンジニアのためのアクセスログ解析基盤構築入門 祖山さん

GOAL

apache

fluentdをインストール後、必要なプラグインをインストール

Elasticsearch

Kibana

参考URL

おすすめ

【初心者超歓迎】Pythonで人工生命を作ろう！！ に参加した

第2回社内セキュリティ共有勉強会に参加した

[AWS Black Belt Online Seminar] AWS Summit Tokyo 2019 振り返り 2019 主要アップデートまとめ を閲覧した

今日から始める PHPエンジニアのためのアクセスログ解析基盤構築入門祖山さん

【初心者超歓迎】Pythonで人工生命を作ろう！！に参加した

[AWS Black Belt Online Seminar] AWS Summit Tokyo 2019 振り返り 2019 主要アップデートまとめを閲覧した