ELK 单节点 Docker 部署

May 24, 2024About 2 min

ELK 单节点 Docker 部署

用 sebp/elk 一体化镜像在单机上把 Elasticsearch + Logstash + Kibana 跑起来,再补上安全认证、license 和踩过的两个坑。

镜像与端口

镜像版本 sebp/elk:8.6.2。涉及的端口:

端口	服务
9200	Elasticsearch API
9300	ES 节点间传输
5601	Kibana 网页
5044	Logstash beats
8080	Logstash https
8082	Logstash http
9600	Logstash API

启动命令

先把 SSL 证书放到宿主机:/etc/pki/tls/certs/logstash-beats.crt 和 /etc/pki/tls/private/logstash-beats.key。然后:

sudo docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 -p 8080:8080 -p 9600:9600 -p 8082:8082 \
  -v /usr/local/elk/elasticsearch_data:/var/lib/elasticsearch \
  -v /usr/local/elk/elasticsearch_conf:/etc/elasticsearch \
  -v /usr/local/elk/logstash_opt:/opt/logstash \
  -v /usr/local/elk/logstash:/etc/logstash \
  -v /usr/local/elk/kibana:/opt/kibana \
  -v /etc/pki/tls/certs:/etc/pki/tls/certs \
  -v /etc/pki/tls/private:/etc/pki/tls/private \
  -it --name elk -d --restart=always sebp/elk:8.6.2

Logstash 配置

改完配置后重启容器。

输入插件,8080 走 https,8082 走 http:

# /usr/local/elk/logstash/conf.d/02-beats-input.conf
input {
  http {
    host => "0.0.0.0"
    port => 8080
    ssl => true
    ssl_certificate => "/etc/pki/tls/certs/logstash-beats.crt"
    ssl_key => "/etc/pki/tls/private/logstash-beats.key"
  }
  http {
    host => "0.0.0.0"
    port => 8082
  }
}

过滤,丢掉没有 indexTag 的日志:

# /usr/local/elk/logstash/conf.d/12-some-filter.conf
filter {
  if [indexTag]==""
  {
    drop {}
  }
}

输出到 ES,索引名按 indexTag + 日期切分。password 填 ES 的 elastic 用户密码:

# /usr/local/elk/logstash/conf.d/30-output.conf
output {
  elasticsearch {
    hosts => ["http://<es-host>:9200"]
    index => "%{[indexTag]}-%{+YYYY.MM.dd}"
    user => "elastic"
    password => "<YOUR_PASSWORD>"
  }
  stdout {
    codec => rubydebug
  }
}

开启单节点安全认证

单节点集群要在 elasticsearch.yml 里把 discovery.type 设成 single-node,并关掉 transport SSL。多节点的话则要启用 minimal security 并在节点间配 TLS。

编辑 elasticsearch.yml:

xpack.security.enabled: true
discovery.type: single-node

xpack.security.http.ssl:
  enabled: false

xpack.security.transport.ssl:
  enabled: false

给内置用户设密码:

sudo docker exec -it elk /bin/bash
cd /opt/elasticsearch
./bin/elasticsearch-reset-password -u elastic        # 随机密码
./bin/elasticsearch-reset-password -i -u elastic     # -i 交互式指定密码
./bin/elasticsearch-reset-password -u kibana_system

如果报 File realm configuration file [/etc/elasticsearch/users] is missing,补两个空文件:

sudo touch /etc/elasticsearch/users
sudo touch /etc/elasticsearch/users_roles

让 Kibana 用密码连 ES。编辑 kibana.yml:

elasticsearch.username: "kibana_system"
elasticsearch.password: "your_password_saved_before"

或者把密码存进 keystore:

sudo docker exec -it elk /bin/bash
cd /opt/kibana
./bin/kibana-keystore create
./bin/kibana-keystore add elasticsearch.password   # 提示时输入 kibana_system 的密码

重启 ELK,用 elastic 超级用户登录 Kibana 管理空间、用户和角色。

license 过期

Kibana 登录报错、容器日志里出现 elasticsearch license expired 时,是 X-Pack 的 30 天试用 license 到期了。ELK 6.3 之后免费 basic license 已包含在发行版里,不用下载,但默认是 30 天试用,需要手动启动无限期 basic license:

# 启动无限期 basic license
curl -X POST "localhost:9200/_license/start_basic?acknowledge=true&pretty"

# 查看 license 状态,acknowledge=true 且 status 为 active 即可
curl -X GET "localhost:9200/_license?pretty"

踩过的坑:shard 数量上限

报错 this action would add [2] shards, but this cluster currently has [1000]/[1000] maximum normal shards open。

ES 7.0 起每个节点默认上限 1000 个 shard,单节点就是 1000。官方推荐是扩节点,但开发环境只能临时调高单节点上限。在 Kibana → 开发者工具 → Console 执行:

PUT /_cluster/settings
{
   "transient": {
      "cluster": {
        "max_shards_per_node": 10000
      }
   }
}