Filebeat实现ELK日志收集

发表于2022-05-08

下载

解压

1
2
3
tar -zxvf jdk-8u171-linux-x64.tar.gz 
tar -zxvf elasticsearch-7.3.1-linux-x86_64.tar.gz 
tar -zxvf kibana-7.3.1-linux-x86_64.tar.gz

配置环境变量

1
2
3
4
5
6
7
8
9
10
11
vim /etc/profile

JAVA_HOME=/usr/java
JRE_HOME=/usr/java/jre
CLASS_PATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib
PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin
export JAVA_HOME JRE_HOME CLASS_PATH PATH

source /etc/profile

java -version

配置ElasticSearch

1
2
3
4
5
6
7
8
9
10
11
12
13
14
vim /usr/elasticsearch/config/elasticsearch.yml 

cluster.name: exam-application
node.name: node01
network.host: 0.0.0.0
http.port: 8020
http.cors.enabled: true
http.cors.allow-origin: "*"
http.cors.allow-headers: Authorization
xpack.security.enabled: true

xpack.security.transport.ssl.enabled: true
cluster.initial_master_nodes: ["node01"]

设置ES参数

1
2
vim /usr/elasticsearch/config/jvm.options
默认初始、最大堆内存为1G 根据需求更改

添加es启动用户

1
2
3
4
5
6
7
useradd  esuser

修改密码
passwd esuser

chown -R estest /usr/elasticsearch/

修改系统参数

1
2
3
4
5
6
7
8
9
10
11
vim /etc/sysctl.conf
vm.max_map_count=655360
sysctl -p

vim /etc/security/limits.conf

*               soft    nofile          65536
*               hard    nofile          65536
*               soft    nproc           4096
*               hard    nproc           4096

启动es

1
2
3
su  esuser
./bin/elasticsearch -d

配置密码

启动es后 执行elasticsearch-setup-passwords interactive进行密码设置

访问测试

1
ip:8020

安装ik分词器

1
./bin/elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v7.7.1/elasticsearch-analysis-ik-7.7.1.zip

测试ik

1
2
3
4
5
6
curl --location --request POST 'http://127.0.0.1:8020/_analyze/?pretty' \
--header 'Content-Type: application/json' \
--data-raw '{
  "analyzer": "ik_max_word",
  "text": "这是一个搜索引擎"
}'

安装Kibana

1
2
3
4
5
6
7
8
9
10
11
chown -R estest /usr/kibana/
chmod -R  777  /usr/kibana/
vim /usr/kibana/config/kibana.yml

elasticsearch.hosts: ["http://localhost:8020"]
elasticsearch.username: "elastic"
elasticsearch.password: "elastic"
server.host: "0.0.0.0"
server.port: 8019
i18n.locale: "zh-CN"

启动Kibana

1
2
su esuser
./bin/kibana

访问测试

1
ip:8019

安装FileBate

1
2
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.7.1-linux-x86_64.tar.gz
tar xzvf filebeat-7.7.1-linux-x86_64.tar.gz

配置

直接发送json日志到ES

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
vim filebeat.yml

json.keys_under_root: true
json.add_error_key: true
json.overwrite_keys: true


filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /data/soft/app/log/*.log



output.elasticsearch:
  hosts: ["localhost:8020"]
  index: "app_log_%{+YYYY-MM-dd}"
  username: "elastic"
  password: "elastic"
  protocol: "http"
setup.template.enabled: false
setup.template.name: "app"
setup.template.pattern: "app-*"
setup.ilm.enabled: false #一定要配，否则不起作用


运行

1
nohup ./filebeat -c filebeat.yml -e  &

访问测试

1
http://hw03.p2m.org.cn:8019/