Prometheus本身不支持告警功能,主要通过插件alertmanage来实现告警。AlertManager用于接收Prometheus发送的告警并对于告警进行一系列的处理后发送给指定的用户。
Prometheus触发一条告警的过程:文章源自小柒网-https://www.yangxingzhen.cn/6720.html
prometheus--->触发阈值--->超出持续时间--->alertmanager--->分组|抑制|静默--->媒体类型--->邮件|钉钉|微信等。文章源自小柒网-https://www.yangxingzhen.cn/6720.html
文章源自小柒网-https://www.yangxingzhen.cn/6720.html
安装Alertmanager
1、下载Alertmanager文章源自小柒网-https://www.yangxingzhen.cn/6720.html
[root@localhost ~]# wget https://github.com/prometheus/alertmanager/releases/download/v0.20.0/alertmanager-0.20.0.linux-amd64.tar.gz文章源自小柒网-https://www.yangxingzhen.cn/6720.html
[root@localhost ~]# tar xf alertmanager-0.20.0.linux-amd64.tar.gz文章源自小柒网-https://www.yangxingzhen.cn/6720.html
[root@localhost ~]# mv alertmanager-0.20.0.linux-amd64 /usr/local/alertmanager文章源自小柒网-https://www.yangxingzhen.cn/6720.html
2、创建启动文件文章源自小柒网-https://www.yangxingzhen.cn/6720.html
[root@localhost ~]# vim /usr/lib/systemd/system/alertmanager.service文章源自小柒网-https://www.yangxingzhen.cn/6720.html
[Unit]
Description=alertmanager
Documentation=https://github.com/prometheus/alertmanager
After=network.target
[Service]
Type=simple
User=root
ExecStart=/usr/local/alertmanager/alertmanager --config.file=/usr/local/alertmanager/alertmanager.yml --cluster.advertise-address=0.0.0.0:9093
Restart=on-failure
[Install]
WantedBy=multi-user.target
3、配置alertmanager.yml文件文章源自小柒网-https://www.yangxingzhen.cn/6720.html
Alertmanager 安装目录下默认有 alertmanager.yml 配置文件,可以创建新的配置文件,在启动时指定即可。
[root@localhost ~]# cd /usr/local/alertmanager
[root@localhost alertmanager]# vim alertmanager.yml
global:
resolve_timeout: 5m
# 邮件SMTP配置
smtp_smarthost: 'smtp.exmail.qq.com:25'
smtp_from: 'service@yangxingzhen.com'
smtp_auth_username: 'service@yangxingzhen.com'
smtp_auth_password: '123456'
smtp_require_tls: false
# 自定义通知模板
templates:
- '/usr/local/prometheus/alertmanager/template/email.tmpl'
# route用来设置报警的分发策略
route:
# 采用哪个标签来作为分组依据
group_by: ['alertname']
# 组告警等待时间。也就是告警产生后等待10s,如果有同组告警一起发出
group_wait: 10s
# 两组告警的间隔时间
group_interval: 10s
# 重复告警的间隔时间,减少相同邮件的发送频率
repeat_interval: 1h
# 设置默认接收人
receiver: 'email'
routes: # 可以指定哪些组接手哪些消息
- receiver: 'email'
continue: true
group_wait: 10s
receivers:
- name: 'email'
email_configs:
- to: 'xingzhen.yang@yangxingzhen.com'
html: '{{ template "email.to.html" . }}'
headers: { Subject: "Prometheus [Warning] 报警邮件" }
send_resolved: true
- smtp_smarthost:是用于发送邮件的邮箱的 SMTP 服务器地址+端口;
- smtp_auth_password:是发送邮箱的授权码而不是登录密码;
- smtp_require_tls:不设置的话默认为 true,当为 true 时会有 starttls 错误,为了简单这里设置为 false;
- templates:指出邮件的模板路径;
- receivers 下html指出邮件内容模板名,这里模板名为“to.html”,在模板路径中的文件中定义。
- headers:为邮件标题;
4、配置告警模板
[root@localhost alertmanager]# mkdir -p /usr/local/prometheus/alertmanager/template
[root@localhost alertmanager]# vim /usr/local/prometheus/alertmanager/template/email.tmpl
{{ define "email.to.html" }}
{{- if gt (len .Alerts.Firing) 0 -}}
{{ range .Alerts }}
=========start==========<br>
告警程序: prometheus_alert <br>
告警级别: {{ .Labels.severity }} <br>
告警类型: {{ .Labels.alertname }} <br>
告警主机: {{ .Labels.instance }} <br>
告警主题: {{ .Annotations.summary }} <br>
告警详情: {{ .Annotations.description }} <br>
触发时间: {{ .StartsAt.Format "2006-01-02 15:04:05" }} <br>
=========end==========<br>
{{ end }}{{ end -}}
{{- if gt (len .Alerts.Resolved) 0 -}}
{{ range .Alerts }}
=========start==========<br>
告警程序: prometheus_alert <br>
告警级别: {{ .Labels.severity }} <br>
告警类型: {{ .Labels.alertname }} <br>
告警主机: {{ .Labels.instance }} <br>
告警主题: {{ .Annotations.summary }} <br>
告警详情: {{ .Annotations.description }} <br>
触发时间: {{ .StartsAt.Format "2006-01-02 15:04:05" }} <br>
恢复时间: {{ .EndsAt.Format "2006-01-02 15:04:05" }} <br>
=========end==========<br>
{{ end }}{{ end -}}
{{- end }}
5、配置告警规则
[root@localhost alertmanager]# mkdir -p /usr/local/prometheus/rules
[root@localhost alertmanager]# cd /usr/local/prometheus/rules
[root@localhost rules]# vim node.yml
groups:
- name: Node_exporter Down
rules:
- alert: Node实例已宕机
expr: up == 0
for: 10s
labels:
user: root
severity: Warning
annotations:
summary: "{{ $labels.job }}"
address: "{{ $labels.instance }}"
description: "Node_exporter 客户端在1分钟内连接失败."
在Prometheus.yml 中指定 node.yml 的路径
[root@localhost rules]# vim /usr/local/prometheus/prometheus.yml
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
# scrape_timeout is set to the global default (10s).
# Alertmanager configuration
alerting:
alertmanagers:
- static_configs:
- targets: ['localhost:9093']
# - localhost:9093
# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
- 'rules/*.yml'
# - "first_rules.yml"
# - "second_rules.yml"
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: 'prometheus'
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
static_configs:
- targets: ['localhost:9100']
6、重启 Prometheus 服务
[root@localhost rules]# systemctl restart prometheus
7、启动 Alertmanager
[root@localhost rules]# systemctl daemon-reload
[root@localhost rules]# systemctl start alertmanager
8、验证效果
此时访问prometheus管理界面可以看到如下信息:
9、停止 node_exporter 服务,然后再看效果。
[root@localhost rules]# systemctl stop node_exporter
prometheus界面的alert可以看到告警状态。
- 绿色表示正常。
- 红色状态为PENDING表示alerts还没有发送至Alertmanager,因为rules里面配置了for: 10s。
- 10秒后状态由PENDING变为FIRING,此时Prometheus才将告警发给alertmanager,在Alertmanager中可以看到有一个alert。
10、邮箱会收到告警邮件
11、再次启动node_export
[root@localhost ~]# systemctl start node_exporter
邮箱会收到恢复信息
附:
CPU使用率告警规则:
groups:
- name: CPU
rules:
- alert: CPU使用率过高
expr: (100 - (avg by (instance) (irate(node_cpu{mode="idle"}[5m])) * 100)) > 80
for: 1m
labels:
severity: Warning
annotations:
summary: "{{ $labels.instance }} CPU使用率过高"
description: "{{ $labels.instance }}: CPU使用率超过80%,当前使用率({{ $value }})."
内存使用率告警规则:
groups:
- name: Memory
rules:
- alert: 内存使用率过高
expr: (node_memory_MemTotal - (node_memory_MemFree+node_memory_Buffers+node_memory_Cached )) / node_memory_MemTotal * 100 > 80
for: 1m #告警持续时间,超过这个时间才会发送给alertmanager
labels:
severity: Warning
annotations:
summary: "{{ $labels.instance }} 内存使用率过高"
description: "{{ $labels.instance }}:内存使用率超过80%,当前使用率({{ $value }})."
若文章图片、下载链接等信息出错,请在评论区留言反馈,博主将第一时间更新!如本文“对您有用”,欢迎随意打赏,谢谢!
![weinxin](https://www.yangxingzhen.com/images/weixin.jpg)
![weinxin](https://www.yangxingzhen.com/images/weixingongzhonghao.png)
请问楼主 我这边alerts里面已经显示红色了 但是邮件一直没收到,方便看下吗?
配置文件是:
global:
# resolve_timeout:解析超时时间
resolve_timeout: 5m
# smtp_smarthost: 使用email打开服务配置
smtp_smarthost: ‘smtp.exmail.qq.com:465’
# smtp_from:指定通知报警的邮箱
smtp_from: ‘xxx@hfhj178.com’
# smtp_auth_username:邮箱用户名
smtp_auth_username: ‘xxx@hfhj178.com’
# smtp_auth_password:授权密码
smtp_auth_password: ‘uxxxxxxxx’
# smtp_require_tls:是否启用tls
smtp_require_tls: false
tes:
– ‘/usr/local/alertmanager/template/*.tmpl’
# route标记:告警如何发送分配
route:
# group_by:采用哪个标签作为分组的依据
group_by: [‘alertname’]
# group_wait:分组等待的时间
group_wait: 10s
# group_interval:上下两组发送告警的间隔时间
group_interval: 10s
# repeat_interval:重复发送告警时间。默认1h
repeat_interval: 3m
# receiver 定义谁来通知报警
receiver: ’email’
# receiver标记:告警接受者
receivers:
# name:报警来源自定义名称
– name: ’email’
# email_configs:通过邮箱发送报警
email_configs:
# to:指定接收端email
– to: ‘xxx@hfhj178.com’
headers: { Subject: “[WARN] 报警邮件”} # 接收邮件的标题
send_resolved: true
# inhibit_rules标记:降低告警收敛,减少报警,发送关键报警
#inhibit_rules:
# – source_match:
# severity: ‘critical’
# target_match:
# severity: ‘warning’
# equal: [‘alertname’, ‘dev’, ‘instance’]
@ 南柯. 请检查下邮箱授权码对不对呢?这里的授权码不是邮箱密码哦
请问楼主,我设置了这个时间触发时间: {{ .StartsAt.Format “2006-01-02 15:04:05” }},可是邮件正文里触发时间还是UTC时间,怎么转成CST时间呢?
@ 溜溜 你可以修改服务时间为CST,这个获取的就是系统时间哦。
@ 小柒博客 哪个服务时间?
不是系统时间?
Local time: Thu 2020-11-19 13:24:32 CST
Universal time: Thu 2020-11-19 05:24:32 UTC
RTC time: Thu 2020-11-19 05:24:32
Time zone: Asia/Shanghai (CST, +0800)
NTP enabled: n/a
NTP synchronized: no
RTC in local TZ: no
DST active: n/a
@ 溜溜 服务器时间哦,说错了
@ 小柒博客 你看下你的时区是不是上海?
@ 溜溜 请问你的邮件能正常通知吗?
写的不错。