[TOC]1、概述通过通过filebeat收集nginx日志，写入到redis，然后通过logstash读取数据，输入到es集群存储，最后通过grafana读取es数据，展示出来。2、服务器信息序号ip配置系统服务1192.168.1.2014核8g512gcentos7.xnginx/filebeat2192.168.1.2024核8g512gcentos7.xredis/es/logstash/grafana3、安装3.1 nginx配置nginx安装略nginx日志配置# 新增以下日志配置，并选择新的日志配置格式 $ vim /etc/nginx/nginx.conf log_format aka_logs '{"@timestamp":"$time_iso8601",' '"host":"$hostname",' '"server_ip":"$server_addr",' '"client_ip":"$remote_addr",' '"xff":"$http_x_forwarded_for",' '"domain":"$host",' '"url":"$uri",' '"referer":"$http_referer",' '"args":"$args",' '"upstreamtime":"$upstream_response_time",' '"responsetime":"$request_time",' '"request_method":"$request_method",' '"status":"$status",' '"size":"$body_bytes_sent",' '"request_body":"$request_body",' '"request_length":"$request_length",' '"protocol":"$server_protocol",' '"upstreamhost":"$upstream_addr",' '"file_dir":"$request_filename",' '"http_user_agent":"$http_user_agent"' '}'; access_log /var/log/nginx/access.log aka_logs;重新加载nginx$ nginx -t && nginx -s reload4、安装es集群因服务器有限，采用docker方式安装es集群管理为docker-compose4.1 安装docker略4.2 安装docker-compose$ yum install epel-release $ yum install docker-compose4.3 编写es的docker-compose yaml文件参照https://www.elastic.co/guide/en/elasticsearch/reference/7.5/docker.html yaml文件如下：version: '2.2' services: elasticsearch: image: elasticsearch:7.8.0 container_name: grafana_es7_01 environment: - node.name=grafana_es7_01 - cluster.name=grafana_es-cluster - discovery.seed_hosts=grafana_es7_02,grafana_es7_01 - cluster.initial_master_nodes=grafana_es7_01,grafana_es7_02 - bootstrap.memory_lock=true - "ES_JAVA_OPTS=-Xms512m -Xmx512m" ulimits: memlock: soft: -1 hard: -1 volumes: - /data/es/data/es01:/usr/share/elasticsearch/data ports: - 9200:9200 - 9300:9300 networks: - elastic elasticsearch2: image: elasticsearch:7.8.0 container_name: grafana_es7_02 environment: - node.name=grafana_es7_02 - cluster.name=grafana_es-cluster - discovery.seed_hosts=grafana_es7_01,grafana_es7_02 - cluster.initial_master_nodes=grafana_es7_01,grafana_es7_02 - bootstrap.memory_lock=true - "ES_JAVA_OPTS=-Xms512m -Xmx512m" ulimits: memlock: soft: -1 hard: -1 volumes: - /data/es/data/es02:/usr/share/elasticsearch/data networks: - elastic networks: elastic: driver: bridge external: name: elastic4.4 启动# 创建数据目录 $ mkdir /data/es/data/{es01,es02} -pv # 授权 $ chmod -R 775 /data/es/data # 创建es网卡 $ docker network create elastic # 启动es $ cd /data/es/docker-compose $ docker-compose up -d4.5 检查状态# 浏览器访问 http://192.168.1.202:92004.6 检查es集群的插件1、google浏览器插件2、elasticsearch-head插件$ docker run -d -p 9100:9100 mobz/elasticsearch-head:5至此es集群安装完成5、安装redis通过docker安装启动命令$ docker run -itd --name grafana_redis -p 6379:6379 --hostname grafana_redis --net elastic redis:latest --requirepass "qwer@12345"6、安装filebeat安装jdk略安装# 安装 $ yum install https://mirrors.tuna.tsinghua.edu.cn/elasticstack/7.x/yum/7.6.1/filebeat-7.6.1-x86_64.rpm配置filebeat# 注释掉es信息 $ vim /etc/filebeat/filebeat.yaml启动filebeat$ systemctl enable --now filebeat检查redis是否有数据输入登陆redis，查看0号库，会看到有nginx_logs的数据，就说明成功了7、安装logstash安装jdk略安装logstash$ wget https://mirrors.tuna.tsinghua.edu.cn/elasticstack/7.x/yum/7.6.1/logstash-7.6.1.rpm --no-check-certificate $ yum install logstash-7.6.1.rpmlogstash增加nginx日志配置文件$ cd /etc/logstash/conf.d $ vim nginx.conf配置logstash的jdk环境变量# 首行增jdk的环境变量 $ vim /usr/share/logstash/bin/logstash.lib.sh export JAVA_HOME=/usr/java/jdk1.8.0_331 export PATH=$JAVA_HOME/bin:$PATH export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar下载必要库GeoLite2-City.mmdb放置在/usr/share/logstash/下下载路径链接：https://pan.baidu.com/s/1Pv0dBKk_OeZyT9bwSE_UWw提取码：1234 启动logstash$ systemctl enable --now logstash8、配置grafana安装略安装必要插件$ grafana-cli plugins install grafana-piechart-panel $ grafana-cli plugins install grafana-worldmap-panel $ service grafana-server restart配置grafana数据源es9、导入grafana模板模板编号 1119010、参考文档https://www.cnblogs.com/xiao987334176/p/13588386.html