EFK+Kafka

工作流程就不写了,网上一大堆,说的比我好多了,大家先去看工作原理,我这里只介绍安装步骤,让你快速在企业内部应用起来

安装

下载官方的二进制包

cd /usr/local/src/

安装ELK

yum install -y ./kibana-6.8.1-x86_64.rpm ./logstash-6.8.1.rpm ./elasticsearch-6.8.1.rpm

ES

调整ES内存vim /etc/elasticsearch/jvm.options

-Xms8g
-Xmx8g

调整ES监听地址(外网需要防火墙增加安全)

network.host: 0.0.0.0

启动

systemctl daemon-reload
systemctl enable elasticsearch.service
vim /etc/elasticsearch/jvm.options 
vim /etc/elasticsearch/elasticsearch.y
systemctl start elasticsearch.service 
systemctl status elasticsearch.service

Kibana

设置kibana监听和ES地址

server.host: "10.24.4.54"
elasticsearch.hosts: ["http://127.0.0.1:9200"]

启动

systemctl start kibana.service 
systemctl status kibana.service

Logstash

调整内存vim /etc/logstash/jvm.options

-Xms16g
-Xmx16g

配置 vim logstash.conf

input {
  beats {
    host => '0.0.0.0'
    port => 5044 
  }
}
#input {
#  kafka {
#    bootstrap_servers => "192.168.7.232:9092"
#    topics => "test"
#    group_id => "test"
#    codec => "json"
#  }
#}

# 将收集时间改成日志的时间,并设置时区为UTC
filter {
    grok {
        match => ["message","%{TIMESTAMP_ISO8601:logdate}"]
    }
    date {
      match => ["logdate","yyyy-MM-dd HH:mm:ss,SSS"]
      target => "@timestamp"
      timezone => "UTC"
   }
}


output{
  elasticsearch {
    hosts => ["http://127.0.0.1:9200"]
    index => "%{env}-%{tag}-%{+YYYY.MM.dd}"
  }

#  stdout {
#    codec=>rubydebug
#  }
}

启动

systemctl start logstash.service 
systemctl status logstash.service

Filebeat

Filebeat 我这里是Ubantu,没用rpm,直接用官方的二进制,解压直接修改配置文件即可vim filebeat.yml

filebeat.inputs:
#################################################### 服务xxx
- type: log
  enabled: true
  paths:
      - /xxx/xxxx/xxxx/xxxx/xxxx.log
  fields:
    env: qa
    tag: admin
  fields_under_root: true
  multiline.pattern: '^[0-9]'
  multiline.negate: true
  multiline.match: after
############################ logstash ##################
output:
  logstash:
    hosts: ["10.24.4.54:5044"]
#output:
#  kafka:
#    hosts: ["10.24.4.54:9092"]
#    topic: test
#    required_acks: 1


env:那个环境,比如qa环境,dev环境,test环境,给索引标识环境而已

tag:那个服务的日志,比如测试环境的admin服务日志,给索引增加标识

fields_under_root:字段冲突覆盖

multiline:匹配Java日志

发表评论