WSO2 log monitoring ( EFK )

When you running a distributed system with multiple applications & multiple replicas, you may face a challenge to monitor application/server logs on each server/replica. because there may be a huge number of log files for each application across all your platforms.

So there are two approaches to monitor your application using log files.

1. Log in to each instance and tail your logs
2. Setup client to collect log files, persist on a single location and visualize in a single dashboard.

My second approach is the smart one, the First approach will waste your time, effort. So I chose EFK stack to implement the second approach.

Elasticsearch Fluentd Kibana ( EFK ) contains three components.

Elasticsearch is a real-time, distributed, and scalable search engine which allows for full-text and structured search, as well as for analytics. It is commonly used to index and search through large volumes of log data, but can also be used to search many different kinds of documents.

Fluentd used to collect, transform, and ship log data to the Elasticsearch backend. Fluentd is a popular open-source data collector that we’ll set up on our Kubernetes nodes to tail container log files, filter and transform the log data, and deliver it to the Elasticsearch cluster, where it will be indexed and stored.

Kibana is a powerful data visualization frontend and dashboard for Elasticsearch. Kibana allows you to explore your Elasticsearch log data through a web interface, and build dashboards and queries to quickly answer questions and gain insight into your Kubernetes applications.

In this article, I’m sharing fluentd configurations for WSO2 APIM 2.6.0 / WSO2 IS 5.7.0 wso2carbon.log file.

First I need to install fluentd agent to my AWS ec2 ( CentOS ) instance.

Steps

1. increase the maximum number of file descriptors

$ ulimit -n

2. If you get 1024 , Increase memory configs on /etc/security/limits.conf and restart machine

root soft nofile 65536
root hard nofile 65536
* soft nofile 65536
* hard nofile 65536

3. Optimize kernel parameters in /etc/sysctl.conf file

net.core.somaxconn = 1024
net.core.netdev_max_backlog = 5000
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_wmem = 4096 12582912 16777216
net.ipv4.tcp_rmem = 4096 12582912 16777216
net.ipv4.tcp_max_syn_backlog = 8096
net.ipv4.tcp_slow_start_after_idle = 0
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 10240 65535

4. install tg-agent3

$ curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent3.sh | sh

5. Launch deamon

$ sudo systemctl start td-agent.service
$ sudo systemctl status td-agent.service
● td-agent.service - td-agent: Fluentd based data collector for Treasure Data
Loaded: loaded (/lib/systemd/system/td-agent.service; disabled; vendor preset: enabled)
Active: active (running) since Thu 2017-12-07 15:12:27 PST; 6min ago
Docs: https://docs.treasuredata.com/articles/td-agent
Process: 53192 ExecStart = /opt/td-agent/embedded/bin/fluentd --log /var/log/td-agent/td-agent.log --daemon /var/run/td-agent/td-agent.pid (code = exited, statu
Main PID: 53198 (fluentd)
CGroup: /system.slice/td-agent.service
├─53198 /opt/td-agent/embedded/bin/ruby /opt/td-agent/embedded/bin/fluentd --log /var/log/td-agent/td-agent.log --daemon /var/run/td-agent/td-agent
└─53203 /opt/td-agent/embedded/bin/ruby -Eascii-8bit:ascii-8bit /opt/td-agent/embedded/bin/fluentd --log /var/log/td-agent/td-agent.log --daemon /v
Dec 07 15:12:27 ubuntu systemd[1]: Starting td-agent: Fluentd based data collector for Treasure Data...
Dec 07 15:12:27 ubuntu systemd[1]: Started td-agent: Fluentd based data collector for Treasure Data.

Now Fluentd agent installation is a success. Now we have to create our configuration file for WSO2 logs.

1. Create a source configuration

<source>
@type tail
path wso2is-km/5.7.0/repository/logs/wso2carbon.log
pos_file /var/log/td-agent/wso2carbon.log.pos
tag iskm
<parse>
@type multiline
format_firstline /TID:\s*\[/
format1 /TID:\s*\[(?<tenent>[^ ]*)\]\s*\[\]\s*\[(?<time>[^ ]* [^ ]*)\s*(?<loglevel>[^ ]*)\s*{(?<class>[^ ]*)}\s*-\s*(?<message>.*)/
time_format %Y-%m-%d %H:%M:%S,%L
</parse>
</source>

In the source tag,

• type is tail, which means I want to tail my log file.
• path is for log file what I need to monitor.
• pos_file is for fluend agent to keep a record of the original file read.
• tag is the name event we are creating here.
• parse, is a directive that includes log formatting parameters.
• type is regex , formatting rule(expression) type.
• expression , Log formatting regex expression.
• time_format, Log time format.
• time_key, Time value for login.
• types id:integer , generate integer id value.

You can test your regex pattern using this tool http://fluentular.herokuapp.com/

2 Create publish configuration

<match iskm>
@type copy
<store>
@type elasticsearch
host <host name >
port 9200
include_tag_key true
include_timestamp true
tag_key @log_name4
logstash_format true
logstash_prefix wso2.uat.${tag}
flush_interval 5s
</store>
</match>

In the match tag,

• match , define tag name of the source.
• store , the type should be elasticsearch.
• host and port of Elasticsearch.

After including the above two tags to /etc/td-agent/td-agent.conf file,

Install Elastcsearch plugin

/usr/sbin/td-agent-gem install fluent-plugin-elasticsearch

After copy configuration file & install elasticsearch plugin, restart tg-agent plugin.

systemctl restart td-agent.service

Now, you need to create an index pattern in kibana dashboard.

Management -> Index pattern -> Create index pattern -> Search {tag name} -> create index

Then you can list log on kiban dashboard .

Thank you, Cheers !!!

References

• Fluentd : https://docs.fluentd.org/
• WSO2 IS/ APIM : https://wso2.com/