ELK 在 k8s 上的两种部署方案
ELK 在 k8s 上的两种部署方案
在 k8s 上跑 ELK 试过两条路:一是用 sebp/elk 一体化镜像跑 DaemonSet,简单但三个组件耦合在一个 Pod;二是 Elasticsearch / Logstash / Kibana 各自独立部署,ES 用 StatefulSet + PV 持久化,更接近生产。下面分别记。
方案一:DaemonSet(一体化镜像)
镜像 sebp/elk:8.6.2。三个组件在同一 Pod 里,适合快速起一套。
自定义 ES 配置要重新打镜像
ES 启动用户是 elasticsearch,直接挂载 elasticsearch.yml 会有权限问题,所以改配置得重新 build 镜像再推到私有仓库:
FROM sebp/elk:8.6.2
ADD ./elasticsearch.yml /etc/elasticsearch/elasticsearch.yml
RUN chown elasticsearch.elasticsearch /etc/elasticsearch/elasticsearch.yml
containerd 对接私有镜像仓库
改 /etc/containerd/config.toml,把私有仓库加进 mirrors:
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."<registry-host>"]
endpoint = ["http://<registry-host>:80"]
重启生效:systemctl restart containerd,之后 crictl pull <registry-host>/library/elk:dev 即可。
ES 集群首次启动的坑
ES 集群第一次启动必须先设 cluster.initial_master_nodes,否则选不出 master。做法:第一次只起一个 Pod,打开 cluster.initial_master_nodes,注释掉 discovery.seed_hosts;之后起多 Pod 时反过来,注释 cluster.initial_master_nodes、打开 discovery.seed_hosts,这样才能正常组集群。
elasticsearch.yml 关键项:
cluster.name: elasticsearch
node.name: elk
path.repo: /var/backups # 快照备份目录
network.host: 0.0.0.0
discovery.seed_hosts: ["elk.elk"]
#cluster.initial_master_nodes: ["elk"]
xpack.security.enabled: false
k8s 配置文件
DaemonSet 本体,镜像从私有仓库拉,initContainers 处理权限和内核参数(vm.max_map_count、文件句柄数):
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: elk
namespace: elk
labels:
app: elk
spec:
selector:
matchLabels:
app: elk
template:
metadata:
labels:
app: elk
spec:
containers:
- name: elk
image: "<registry-host>/library/elk:dev"
imagePullPolicy: Always
tty: true
env:
- name: TZ
value: Asia/Shanghai
ports:
- containerPort: 5601
- containerPort: 5044
- containerPort: 9200
- containerPort: 9300
- containerPort: 8080
volumeMounts:
- name: data
mountPath: /var/lib/elasticsearch
- name: logstash-volume
mountPath: /etc/logstash/conf.d
- name: kibana-volume
mountPath: /opt/kibana/config
- name: es-backup
mountPath: /var/backups
initContainers:
- name: fix-permissions
image: busybox
command: ["sh", "-c", "chmod 777 -R /var/lib/elasticsearch"]
securityContext:
privileged: true
volumeMounts:
- name: data
mountPath: /var/lib/elasticsearch
- name: increase-vm-max-map
image: busybox
command: ["sysctl", "-w", "vm.max_map_count=262144"]
securityContext:
privileged: true
- name: increase-fd-ulimit
image: busybox
command: ["sh", "-c", "ulimit -n 65536"]
securityContext:
privileged: true
volumes:
- name: kibana-volume
configMap:
name: kibana-config
- name: data
hostPath:
path: /var/lib/elasticsearch
- name: logstash-volume
configMap:
name: logstash-config
- name: es-backup
hostPath:
path: /mnt/VM_ShareDir/ES_Backup # 各节点公共目录
Service 把五个端口暴露成 ClusterIP:
apiVersion: v1
kind: Service
metadata:
name: elk
namespace: elk
spec:
type: ClusterIP
ports:
- { name: elastic, port: 9200, targetPort: 9200, protocol: TCP }
- { name: kibana, port: 5601, targetPort: 5601, protocol: TCP }
- { name: logstash-beats, port: 5044, targetPort: 5044, protocol: TCP }
- { name: elastic-transport,port: 9300, targetPort: 9300, protocol: TCP }
- { name: logstash-http, port: 8080, targetPort: 8080, protocol: TCP }
selector:
app: elk
Ingress 按域名转发到各端口(域名换成自己的):
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-elk
namespace: elk
spec:
ingressClassName: ingress-ctr
rules:
- host: kibana.example.com
http:
paths:
- { path: /, pathType: Prefix, backend: { service: { name: elk, port: { number: 5601 } } } }
- host: elastic.example.com
http:
paths:
- { path: /, pathType: Prefix, backend: { service: { name: elk, port: { number: 9200 } } } }
- host: logstash.example.com
http:
paths:
- { path: /, pathType: Prefix, backend: { service: { name: elk, port: { number: 5044 } } } }
Kibana 的 configmap 里 elasticsearch.hosts 指向 ES 的 Ingress 地址,i18n.locale: "zh-CN" 切中文;Logstash 的 output 同样指向 ES 地址。
ES 快照与恢复
要注册共享文件系统快照库,先把同一个文件系统挂到所有 master 和 data 节点的相同位置(上面部署脚本已把公共目录挂到容器 /var/backups),再在 Kibana 网页上创建共享文件系统快照存储库,然后定快照策略。
恢复:恢复某些索引前先关闭对应索引,再选索引/数据流恢复。若数据全没了,新起一个 ES 集群,建快照库,恢复全部索引和数据流,之后重建快照策略。
常用 API 与测试命令
# 集群健康
curl http://<es-host>:30692/_cluster/health?pretty
# 看索引
curl http://<es-host>:30692/_cat/indices
# 开启某索引
curl -X POST http://<es-host>:30692/[index_name]/_open?pretty
# 往 logstash http 发测试日志
curl -v -X POST http://<logstash-host>:30692 -d "test1"
curl -v -X POST -H "Content-Type:application/json" http://<logstash-host>:30692 -d @log.json
Filebeat(可选)
Filebeat 是日志发送器,发到 Logstash 的 5044 端口。它走 TCP 连接,如果 Filebeat 在 k8s 集群外,需要改 ingress-nginx controller 加 TCP 转发(见 Ingress-Nginx 加 TCP 转发);集群内则不用改。
配 filebeat.yml 里的 log 路径和 logstash 地址即可。注意 output 里 elasticsearch 和 logstash 只能选一个,另一个要注释,否则报错。启动:管理员权限下 filebeat.exe -e -d "publish",或装成 service 用 .\install-service-filebeat.ps1。
方案二:StatefulSet + PV(组件分离)
ELK 全部 8.13.0,ES 用 StatefulSet + NFS PV 持久化,Logstash 走 SSL。前提是先准备一台 NFS server,并创建 storageclass nfs。
PV 用 NFS,三个 ES 副本各一个:
apiVersion: v1
kind: PersistentVolume
metadata:
name: es-pv1
spec:
storageClassName: nfs
capacity:
storage: 100Gi
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
nfs:
path: /PV/es/es-pv1
server: <nfs-server-ip> # 改成 NFS server 地址
# es-pv2 / es-pv3 同理
ES StatefulSet,podAntiAffinity 让三个副本分散到不同节点,通过 discovery.seed_hosts 和 cluster.initial_master_nodes 用 headless service 的 DNS 名组集群:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: elasticsearch
namespace: elk
spec:
podManagementPolicy: Parallel
serviceName: elasticsearch
replicas: 3
selector:
matchLabels:
app: elasticsearch
template:
metadata:
labels:
app: elasticsearch
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: elasticsearch
topologyKey: kubernetes.io/hostname
initContainers:
- name: fix-permissions
image: busybox
command: ["sh", "-c", "chmod 777 -R /usr/share/elasticsearch/data"]
securityContext:
privileged: true
volumeMounts:
- name: elasticsearch-data
mountPath: /usr/share/elasticsearch/data
containers:
- image: elasticsearch:8.13.0
name: elasticsearch
env:
- name: discovery.seed_hosts
value: "elasticsearch-0.elasticsearch.elk.svc.cluster.local,elasticsearch-1.elasticsearch.elk.svc.cluster.local,elasticsearch-2.elasticsearch.elk.svc.cluster.local"
- name: cluster.initial_master_nodes
value: "elasticsearch-0,elasticsearch-1,elasticsearch-2"
- name: cluster.name
value: "es-cluster"
- name: ES_JAVA_OPTS
value: "-Xms512m -Xmx512m"
- name: xpack.security.enabled
value: "false"
ports:
- { containerPort: 9200, name: db, protocol: TCP }
- { containerPort: 9300, name: inter }
volumeMounts:
- name: elasticsearch-data
mountPath: /usr/share/elasticsearch/data
volumeClaimTemplates:
- metadata:
name: elasticsearch-data
spec:
storageClassName: "nfs"
accessModes: [ "ReadWriteMany" ]
resources:
requests:
storage: 100Gi
ES 配两个 service:headless(clusterIP: None,给 StatefulSet 组集群用)和 NodePort(nodePort: 30920,对外访问)。
Kibana 是单副本 Deployment,ELASTICSEARCH_HOSTS 指向 http://elasticsearch:9200,NodePort 30601 对外。
Logstash 也是 Deployment,pipeline 配置走 configmap,SSL 证书走 secret:
data:
input.conf: |-
input {
http {
host => "0.0.0.0"
port => 8080
ssl => true
ssl_certificate => "/etc/pki/tls/certs/ssl.crt"
ssl_key => "/etc/pki/tls/certs/ssl.key"
}
http {
host => "0.0.0.0"
port => 8082
}
}
filter.conf: |-
filter {
if [indexTag]==""
{
drop {}
}
}
output.conf: |-
output {
elasticsearch {
hosts => ["elasticsearch:9200"]
index => "%{[indexTag]}-%{+YYYY.MM.dd}"
}
stdout {
codec => rubydebug
}
}
SSL 证书放进 kubernetes.io/tls 类型的 Secret(logstash-ssl-key,字段 tls.crt/tls.key),Logstash 的 8080/8082 通过 NodePort 30080/30082 暴露。