https://llinux.cn/%E5%8F%AF%E8%A7%82%E6%B5%8B%E6%80%A7/%E7%9B%91%E6%8E%A7%E4%B8%8E%E5%91%8A%E8%AD%A6/ 阿征的运维笔记 zh https://llinux.cn/%E5%8F%AF%E8%A7%82%E6%B5%8B%E6%80%A7/%E7%9B%91%E6%8E%A7%E4%B8%8E%E5%91%8A%E8%AD%A6/alertmanager/ https://llinux.cn/%E5%8F%AF%E8%A7%82%E6%B5%8B%E6%80%A7/%E7%9B%91%E6%8E%A7%E4%B8%8E%E5%91%8A%E8%AD%A6/alertmanager/ <h2 id="告警数据流">告警数据流<a class="td-heading-self-link" href="#%e5%91%8a%e8%ad%a6%e6%95%b0%e6%8d%ae%e6%b5%81" aria-label="Heading self-link"></a></h2> <ul> <li><strong>Prometheus</strong> ：采集数据 -&gt; <strong>计算规则 (Rules)</strong> -&gt; 发现异常 -&gt; <strong>产生告警</strong> -&gt; 发送给 Alertmanager。</li> <li><strong>Alertmanager</strong> ：接收告警 -&gt; 分组/去重 -&gt; <strong>发送通知</strong> (发邮件、钉钉等)。</li> </ul> <h1 id="alertmanager-概述">Alertmanager 概述<a class="td-heading-self-link" href="#alertmanager-%e6%a6%82%e8%bf%b0" aria-label="Heading self-link"></a></h1> <ul> <li>Alertmanager 除了可以接受 Prometheus 的报警信息外，也支持接收来自其它工具的告警；</li> <li>Alertmanager 也是应用程序，它自身同样应该纳入prometheus的监控目标；</li> <li>Alertmanager 同样提供了可视化的 web UI，以及使用 /metrics 路径暴露了其内建的指标</li> <li>TCP/9093，Alertmanager-server</li> <li>TCP/9094，Alertmanager-cluster</li> <li><a href="https://github.com/prometheus/alertmanager">https://github.com/prometheus/alertmanager</a></li> </ul> <h1 id="alertmanager-单机部署">Alertmanager 单机部署<a class="td-heading-self-link" href="#alertmanager-%e5%8d%95%e6%9c%ba%e9%83%a8%e7%bd%b2" aria-label="Heading self-link"></a></h1> <h2 id="helm">helm<a class="td-heading-self-link" href="#helm" aria-label="Heading self-link"></a></h2> <div class="highlight code-sample"> <div class="includecode" > <pre tabindex="0"><code>kubectl create ns monitoring helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm pull prometheus-community/alertmanager --version 1.7.0 tar xf alertmanager-1.7.0.tgz helm install alertmanager ./alertmanager-1.7.0.tgz -n monitoring -f alertmanager/values.yaml</code></pre> </div> </div><h2 id="二进制">二进制<a class="td-heading-self-link" href="#%e4%ba%8c%e8%bf%9b%e5%88%b6" aria-label="Heading self-link"></a></h2> <ul> <li>直接将二进制文件解压到自定义的目录即可</li> <li>可以选择使用 service 文件来对其进行启动</li> <li>启动后验证 9093端口是否开启，同样可以访问 metrics 来查看是否有指标收集上来</li> </ul> <h2 id="service">service<a class="td-heading-self-link" href="#service" aria-label="Heading self-link"></a></h2> <ul> <li>/lib/systemd/system/prometheus-alertmanager.service</li> </ul> <div class="highlight code-sample"> <div class="includecode" > <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl"><span class="o">[</span>Unit<span class="o">]</span> </span></span><span class="line"><span class="cl"><span class="nv">Description</span><span class="o">=</span>Prometheus alertmanager </span></span><span class="line"><span class="cl"><span class="nv">Documentation</span><span class="o">=</span>https://github.com/prometheus/alertmanager </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="o">[</span>Service<span class="o">]</span> </span></span><span class="line"><span class="cl"><span class="nv">Restart</span><span class="o">=</span>always </span></span><span class="line"><span class="cl"><span class="nv">User</span><span class="o">=</span>root </span></span><span class="line"><span class="cl"><span class="nv">ExecStart</span><span class="o">=</span>/monitor/alertmanager/alertmanager --config.file<span class="o">=</span>/monitor/alertmanager/alertmanager.yml </span></span><span class="line"><span class="cl"><span class="nv">ExecReload</span><span class="o">=</span>/bin/kill -HUP <span class="nv">$MAINPID</span> </span></span><span class="line"><span class="cl"><span class="nv">TimeoutStopSec</span><span class="o">=</span>20s </span></span><span class="line"><span class="cl"><span class="nv">SendSIGKILL</span><span class="o">=</span>no </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="o">[</span>Install<span class="o">]</span> </span></span><span class="line"><span class="cl"><span class="nv">WantedBy</span><span class="o">=</span>multi-user.target</span></span></code></pre></div> </div> </div><h1 id="alertmanager-集群部署">Alertmanager 集群部署<a class="td-heading-self-link" href="#alertmanager-%e9%9b%86%e7%be%a4%e9%83%a8%e7%bd%b2" aria-label="Heading self-link"></a></h1> <ul> <li><a href="https://github.com/prometheus/alertmanager#high-availability">https://github.com/prometheus/alertmanager#high-availability</a></li> </ul> <h2 id="gossip协议">Gossip协议<a class="td-heading-self-link" href="#gossip%e5%8d%8f%e8%ae%ae" aria-label="Heading self-link"></a></h2> <ul> <li>多Alertmanager实例的场景中，告警信息可能会被重复发送；</li> <li>Gossip机制可为多个Alertmanager之间提供了信息传递的机制，以确保在多个Alertmanager分别接收到相同告警信息的情况下，只会有一个告警通知被发送给Receiver；</li> <li>Gossip是分布式系统中被广泛使用的协议，用于实现分布式节点之间的信息交换和状态同步；</li> </ul> <h2 id="部署-alertmanager-集群">部署 Alertmanager 集群<a class="td-heading-self-link" href="#%e9%83%a8%e7%bd%b2-alertmanager-%e9%9b%86%e7%be%a4" aria-label="Heading self-link"></a></h2> <ul> <li>假设将三个alertmanager节点组织成为集群： <ul> <li>node01：10.0.0.8</li> <li>node02：10.0.0.18</li> <li>node03：10.0.0.28</li> </ul> </li> <li>各节点需要的启动命令如下：（注意：下面示例中只是说明配置集群所需的必选参数，每个alertmanager节点还要使用相同的<code>alertmanager.yml</code> 配置文件，或者以共享存储的方式共享配置文件） <ul> <li>node01: alertmanager</li> <li>node02: alertmanager &ndash;cluster.peer=10.0.0.8:9094</li> <li>node03: alertmanager &ndash;cluster.peer=10.0.0.8:9094</li> </ul> </li> <li>验证： <ul> <li>http://10.0.0.8:9093/#/status</li> <li>观察 Cluster Status</li> </ul> </li> </ul> <h2 id="prometheus-指向-alertmanager-集群">Prometheus 指向 Alertmanager 集群<a class="td-heading-self-link" href="#prometheus-%e6%8c%87%e5%90%91-alertmanager-%e9%9b%86%e7%be%a4" aria-label="Heading self-link"></a></h2> <ul> <li>配置完 Alertmanagers 集群后，Prometheus 不要以负载均衡的方式访问 Alertmanagers，而是将 Prometheus 配置文件中指向所有 Alertmanagers 的列表</li> </ul> <div class="highlight code-sample"> <div class="includecode" > <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">alerting</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">alertmanagers</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">static_configs</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">targets</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="l">alertmanager1:9093</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="l">alertmanager2:9093</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="l">alertmanager3:9093</span></span></span></code></pre></div> </div> </div><h1 id="alertmanager-配置文件">Alertmanager 配置文件<a class="td-heading-self-link" href="#alertmanager-%e9%85%8d%e7%bd%ae%e6%96%87%e4%bb%b6" aria-label="Heading self-link"></a></h1> <p><a href="https://prometheus.io/docs/alerting/latest/configuration/">https://prometheus.io/docs/alerting/latest/configuration/</a></p>