Keepalived

LVS+keepalived

前言

LVS 配合 keepalived 实现高可用
keepalived中内置了LVS的规则生成工具所以可以不使用传统的方式定义LVS的规则而是通过修改keepalived配置文件的方式自动生成LVS规则

virtual_server 的文件结构

#/etc/keepalived/keepalived.conf

virtual_server IP port {
...
    real_server IP port {
        ...
    }
    real_server IP port {
        ...
    }
}

virtual_server 的配置说明

参考文档：https://www.keepalived.org/manpage.html
注意：括号必须分行写

virtual_server <IP port>|<fwmark int>|<group string> {
#IP port      定义虚拟服务器的IP地址和端口号
#fwmark int   ipvs的防火墙打标，实现基于防火墙的负载均衡集群
#group string 使用虚拟服务器组
---------------------------------------------------------------------------
    delay_loop <TIMER> #检查后端服务器的时间间隔 单位:秒 ，不指定则默认为60秒
    lb_algo <rr|wrr|lc|wlc|sh|dh|lblc> #调度算法定义
    lb_kind <NAT|DR|TUN> #工作模式
    persistence_timeout <TIMER> #持久连接超时时长(设定时间内始终往同一台RS调度) 单位:秒 不指定则默认为6分钟
    protocol TCP|UDP #指定服务协议，一般为TCP
    sorry_server <IP port> #当所有RS不可用时 备用的sorry_server服务器的地址
---------------------------------------------------------------------------
    real_server <IP port> { #RS的IP和port
        weight <num> #RS的权重 不指定默认为1，只在WRR、WLC等根据权重进行调度的算法时有效
        notify_up <STRING>|<QUOTED-STRING> #RS的上线通知脚本
        notify_down <STRING>|<QUOTED-STRING> #RS的下线通知脚本
        TCP_CHECK|MISC_CHECK|HTTP_GET|SSL_GET {  #定义当前主机监控状态监测方法
        ... #详参下面的 应用层监测 和 TCP监测
        }    
    }
}

virtual_server 应用层监测

应用层监测：HTTP_GET|SSL_GET
监测web服务的工作状态

virtual_server <IP port> {
...
    real_server <IP port> {
        ...
        HTTP_GET {  #应用层监测
            url { 
            path <URL_PATH> #定义要监控的url
            status_code <CODE> #指定上述监测机制中HTTP头中返回的状态代码，一般设为200
            }
            connect_timeout <NUM> #客户端请求的超时时间
            retry <NUM> #超时后最大的重新尝试次数
            delay_before_retry <NUM> #两次连续重试之间的延迟时长
            connect_ip <IPADDR> #指定向当前RS的哪个IP地址发起健康状态监测请求，可选项
            connect_port <PORT> #指定向当前RS的哪个port发起健康状态监测请求，可选项
            bindto <IPADDR> #指定以当前主机的哪个源IP向RS发起健康状态监测请求，可选项
            bind_port <PORT> #指定以当前主机的哪个源port向RS发起健康状态监测请求，可选项
        }    
    }
}

范例：

# /usr/share/doc/keepalived/keepalived.conf.virtualhost
...
virtual_server 10.0.0.10 80 {
    delay_loop 3
    lb_algo rr
    lb_kind NAT
    persistence_timeout 50
    protocol TCP
    sorry_server 127.0.0.1 80

    real_server 10.0.0.7 80 {
        weight 1
        HTTP_GET {
            url {
              path /monitor.html
              status_code 200
            }
            connect_timeout 3
            retry 3
            delay_before_retry 3
        }
        
    real_server 10.0.0.17 80 {
        weight 1
        HTTP_GET {
            url {
              path /monitor.html
              status_code 200
            }
            connect_timeout 3
            retry 3
            delay_before_retry 3
        }
        
    }
...

virtual_server TCP监测

监测一些不属于http协议的应用，如：mysql、redis等…
根据是否能与RS进行TCP三次握手来判断服务是否正常运行

virtual_server <IP port> {
...
    real_server <IP port> {
        ...
        TCP_CHECK {  #TCP连接监测
            connect_timeout <NUM> #客户端请求的超时时间
            retry <NUM> #超时后最大的重新尝试次数
            delay_before_retry <NUM> #两次连续重试之间的延迟时长
            connect_ip <IPADDR> #向当前RS的哪个IP地址发起健康状态监测请求
            connect_port <PORT> #向当前RS的哪个port发起健康状态监测请求
            bindto <IPADDR> #以当前主机的哪个源IP向RS发起健康状态监测请求
            bind_port <PORT> #以当前主机的哪个源port向RS发起健康状态监测请求
        }    
    }
}

范例：

virtual_server 10.0.0.10 80 {
    delay_loop 3
    lb_algo rr
    lb_kind NAT
    persistence_timeout 50
    protocol TCP
    sorry_server 127.0.0.1 80

    real_server 10.0.0.7 80 {
        weight 1
        TCP_CHECK {
            connect_timeout 3
            retry 3
            delay_before_retry 3
            connect_port 3306
        }
...

实现单主单VIP的 LVS-DR 模式

部署前言

先决条件：
- 时间同步
- 关闭selinux
部署概述：
- 状态检测报文通过组播发送

环境

IP	VIP	service	hostname
10.0.0.18	10.0.0.100	Keepalived+LVS(DR)+nginx(sorry server)	Keepalived1
10.0.0.28	10.0.0.100	Keepalived+LVS(DR)+nginx(sorry server)	Keepalived2
10.0.0.38	lo:10.0.0.100(RS)	nginx	web1.xiangzheng.org
10.0.0.38	lo:10.0.0.100(RS)	nginx	web2.xiangzheng.org

安装ipvsadm工具

虽然规则不用手动输入但是keepalived配置文件本身还是调用了ipvsadm这个工具

[root@keepalived1 ~]# yum install -y ipvsadm
[root@keepalived2 ~]# yum install -y ipvsadm

准备web服务和页面

[root@web1 ~]# echo '10.0.0.38 web page' > /usr/share/nginx/html/index.html 
[root@web1 ~]# systemctl enable --now nginx
[root@web2 ~]# echo '10.0.0.48 web page' > /usr/share/nginx/html/index.html
[root@web2 ~]# systemctl enable --now nginx

#测试访问
[root@client ~]# curl 10.0.0.38
10.0.0.38 web page
[root@client ~]# curl 10.0.0.48
10.0.0.48 web page

将RS绑定VIP

在RS执行以下脚本

#!/bin/bash
VIP="10.0.0.100"
MASK="32"
DEV="lo"

case $1 in
start)
    ip address add ${VIP}/${MASK} dev ${DEV}
    #ifconfig lo:1 ${VIP}/${MASK}
    echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
    echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
    echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
    echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
    echo "VIP:${VIP}/${MASK} is Start"
    ;;
stop)
    ip address del ${VIP}/${MASK} dev lo
    #ifconfig ${DEV} down
    echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
    echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
    echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
    echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce
    echo "VIP:${VIP}/${MASK} is Stop"
    ;;
*)
    echo "Usage: $(basename $0) {start or stop}"
    ;;
esac


[root@web1 ~]# ./rs_vip.sh start
[root@web2 ~]# ./rs_vip.sh start

keepalived1配置

sorry server准备过程省略，其实就是安装nginx服务并部署sorry server页面至本机，当然部署在其他主机也可以

root@lvs-ka1:~# cat /etc/keepalived/keepalived.conf
global_defs {
   router_id lvs-ka1
   vrrp_skip_check_adv_addr
   vrrp_garp_interval 0
   vrrp_gna_interval 0
   vrrp_mcast_group4 226.6.6.6
}

vrrp_instance azheng_nginx {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 66666
    }
    virtual_ipaddress {
        10.0.0.123/32 dev eth0 label eth0:0
    }
}

virtual_server 10.0.0.123 80 {
    delay_loop 6
    lb_algo rr
    lb_kind DR
    #persistence_timeout 30
    protocol TCP
    sorry_server 127.0.0.1 80
    real_server 10.0.0.102 80 {
        weight 1
        HTTP_GET {
            url {
                path /
                status_code 200
            }
            connect_timeout 3
            retry 3
            delay_before_retry 3
        }
    }
    real_server 10.0.0.103 80 {
        weight 1
        HTTP_GET {
            url {
                path /
                status_code 200
            }
            connect_timeout 3
            retry 3
            delay_before_retry 3
        }
    }
}

keepalived2配置

sorry server准备过程省略，其实就是安装nginx服务并部署sorry server页面至本机，当然部署在其他主机也可以

root@lvs-ka2:~# cat /etc/keepalived/keepalived.conf
global_defs {
   router_id lvs-ka2
   vrrp_skip_check_adv_addr
   vrrp_garp_interval 0
   vrrp_gna_interval 0
   vrrp_mcast_group4 226.6.6.6
}

vrrp_instance azheng_nginx {
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 80
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 66666
    }
    virtual_ipaddress {
        10.0.0.123/32 dev eth0 label eth0:0
    }
}

virtual_server 10.0.0.123 80 {
    delay_loop 6
    lb_algo rr
    lb_kind DR
    #persistence_timeout 30
    protocol TCP
    sorry_server 127.0.0.1 80
    real_server 10.0.0.102 80 {
        weight 1
        HTTP_GET {
            url {
                path /
                status_code 200
            }
            connect_timeout 3
            retry 3
            delay_before_retry 3
        }
    }
    real_server 10.0.0.103 80 {
        weight 1
        HTTP_GET {
            url {
                path /
                status_code 200
            }
            connect_timeout 3
            retry 3
            delay_before_retry 3
        }
    }
}

测试

重启服务并测试

#可以看到通过keepalived的配置文件自动生成了LVS规则
[root@keepalived1 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.0.0.100:80 rr
  -> 10.0.0.38:80                 Route   1      0          0         
  -> 10.0.0.48:80                 Route   1      0          0    

[root@keepalived2 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.0.0.100:80 rr
  -> 10.0.0.38:80                 Route   1      0          0         
  -> 10.0.0.48:80                 Route   1      0          0

实现单主单VIP的 LVS-DR 模式2

ka1配置

心跳检测报文

root@rs1:~# tail -f /var/log/nginx/access.log
...
10.0.0.101 - - [15/Apr/2022:19:26:20 +0800] "GET / HTTP/1.0" 200 4 "-" "KeepAliveClient"
10.0.0.100 - - [15/Apr/2022:19:26:20 +0800] "GET / HTTP/1.0" 200 4 "-" "KeepAliveClient"
...

实现双主多VIP的 LVS-DR 模式

Nginx+keepalived

Nginx+keepalived(单主模式)

利用keepalived检测脚本的方式实现对nginx的状态监测从而实现故障切换

环境准备

IP	VIP	service	hostname
10.0.0.18	10.0.0.100	Keepalived + nginx(proxy)	Keepalived1
10.0.0.28	10.0.0.100	Keepalived + nginx(proxy)	Keepalived2
10.0.0.38	NULL	nginx(web)	web1.xiangzheng.org
10.0.0.48	NULL	nginx(web)	web2.xiangzheng.org

10.0.0.18 配置

准备nginx

[root@keepalived1 ~]# vim /etc/nginx/nginx.conf
...
http {
...
    upstream nginx {
        server 10.0.0.38:80 weight=1 fail_timeout=15s max_fails=3;
        server 10.0.0.48:80 weight=1 fail_timeout=15s max_fails=3;
    }
...
    server {
...
        location / {
        proxy_pass http://nginx;
        }
...

准备keepalived

#定义子配置文件
[root@keepalived1 ~]# cat /etc/keepalived/keepalived.conf 
global_defs {
   router_id KA1
}

include /etc/keepalived/conf.d/*.conf

---------------------------------------------------------------------------

#准备子配置文件
[root@keepalived1 ~]# vim /etc/keepalived/conf.d/virtual_vip.conf
vrrp_script check_nginx { #定义脚本
       script "/etc/keepalived/check_nginx.sh"
       interval 2
       weight -30
       fall 2
       rise 2
}

vrrp_instance WEB1 {
    state MASTER
    interface eth0
    virtual_router_id 66
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 666
    }

    virtual_ipaddress {
        10.0.0.100/24 dev eth0 label eth0:0
    }

    unicast_src_ip 10.0.0.18
    unicast_peer {
    10.0.0.28
    }

    track_script { #引用脚本
    	check_nginx
    }
}

准备检测脚本

**0信号说明：**对进程进行健康性检查(健康则$?结果为0，反之则为1)，但结果并不严谨因为进程只要存在它就认为进程是健康的如：进程已经成为僵尸态而$?的返回值依旧为0

[root@keepalived1 ~]# vim /etc/keepalived/check_nginx.sh
#!/bin/bash
killall -0 nginx || systemctl restart nginx.service

[root@keepalived2 ~]# chmod +x /etc/keepalived/check_nginx.sh

10.0.0.28配置

准备nginx

[root@keepalived2 ~]# vim /etc/nginx/nginx.conf
...
http {
...
    upstream nginx {
        server 10.0.0.38:80 weight=1 fail_timeout=15s max_fails=3;
        server 10.0.0.48:80 weight=1 fail_timeout=15s max_fails=3;
    }
...
    server {
...
        location / {
        proxy_pass http://nginx;
        }
...

准备keepalived

#定义子配置文件
[root@keepalived2 ~]# cat /etc/keepalived/keepalived.conf 
global_defs {
   router_id KA2
}

include /etc/keepalived/conf.d/*.conf

----------------------------------------------------------------------------

#准备子配置文件
[root@keepalived2 ~]# vim /etc/keepalived/conf.d/virtual_vip.conf
vrrp_script check_nginx {
       script "/etc/keepalived/check_nginx.sh"
       interval 2
       weight -30
       fall 2
       rise 2
}

vrrp_instance WEB1 {
    state BACKUP
    interface eth0
    virtual_router_id 66
    priority 80
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 666
    }

    virtual_ipaddress {
        10.0.0.100/24 dev eth0 label eth0:0
    }

    unicast_src_ip 10.0.0.28
    unicast_peer {
    10.0.0.18
    }

    track_script {
    	check_nginx
    }
}

准备检测脚本

**0信号说明：**对进程进行健康性检查(健康则$?结果为0，反之则为1)，但结果并不严谨因为进程只要存在它就认为进程是健康的如：进程已经成为僵尸态而$?的返回值依旧为0

[root@keepalived2 ~]# vim /etc/keepalived/check_nginx.sh
#!/bin/bash
killall -0 nginx || systemctl restart nginx.service

[root@keepalived2 ~]# chmod +x /etc/keepalived/check_nginx.sh

10.0.0.38配置

web页面准备过程省略

[root@client ~]# curl 10.0.0.38
10.0.0.38 web page

10.0.0.48配置

web页面准备过程省略

[root@client ~]# curl 10.0.0.48
10.0.0.48 web page

测试

#因为keepalived1优先级高，所以目前VIP在keepalived1上
[root@keepalived1 ~]# hostname -I
10.0.0.18 10.0.0.100 
[root@keepalived2 ~]# hostname -I
10.0.0.18

#模拟keepalived1上的nginx故障
[root@keepalived1 ~]# killall nginx 

#观察VIP飘动情况，可以看到VIP并没有进行切换，这是为什么呢？这是因为在定义脚本的时候配置了只要监测nginx进程不在 那么就立刻执行nginx重启，所以既然nginx重启了 进程恢复了 那么VIP就不会进行飘动了
[root@keepalived1 ~]# hostname -I
10.0.0.18 10.0.0.100 
[root@keepalived2 ~]# hostname -I
10.0.0.18

#将脚本修改成故障不进行重启，只进行检测
[root@keepalived1 ~]# vim /etc/keepalived/check_nginx.sh
#!/bin/bash
killall -0 nginx

#再次模拟keepalived1上的nginx故障
[root@keepalived1 ~]# killall nginx

#观察VIP飘动情况，因为没有定义检测进程失效后执行服务重启的操作，所以脚本执行失败后就会将优先级-30，100-30=70，70的优先级没有备份节点的80优先级高，所以自然VIP就飘到了keepalived2上
[root@keepalived1 ~]# hostname -I
10.0.0.18 
[root@keepalived2 ~]# hostname -I
10.0.0.28 10.0.0.100

#恢复nginx
[root@keepalived1 ~]# systemctl start nginx

#观察VIP飘动情况，因为使用的是抢占式VIP，所以服务恢复后通过脚本检测发现nginx服务恢复正常，所以又将keepalived1标记位可用 然后将优先级恢复为原先的100，keepalived1的100优先级比keepalived2的80高 自然VIP就飘到keepalived1上了
[root@keepalived1 ~]# hostname -I
10.0.0.18 10.0.0.100 
[root@keepalived1 ~]# hostname -I
10.0.0.18

状态检测和故障切换

前言

keepalived虽然内置了配合LVS实现高可用的配置文件，但对其他服务并没有相应的配置，但这没关系，keepalived还支持通过script来实现对服务状态的检测和故障切换

原理

在global配置段外定义脚本或者shell命令，然后再virtual_server配置段中调用脚本，最后根据脚本的执行结果(成功或失败) 来增加或减少当前的节点的优先级，因为keepalived是根据优先级来判断是否为主节点所以当优先级低时则就成为备份节点从而将VIP飘动到优先级高的主节点上实现了对其他服务的高可用

范例：利用脚本实现主从切换

环境准备

IP	VIP	service	hostname
10.0.0.18	10.0.0.100	Keepalived	Keepalived1
10.0.0.28	10.0.0.100	Keepalived	Keepalived2

Keepalived1配置

#定义子配置文件
[root@keepalived1 ~]# cat /etc/keepalived/keepalived.conf 
global_defs {
   router_id KA1
}

include /etc/keepalived/conf.d/*.conf

----------------------------------------------------------------------------

#添加子配置文件
[root@keepalived1 ~]# cat /etc/keepalived/conf.d/virtual_vip.conf 
vrrp_script check_test { #脚本名称
       script "/etc/keepalived/down.sh" #调用的脚本
       interval 2
       weight -30 #当执行失败后本节点权重+-30 也就等于减30了
       fall 2 #连续几次执行脚本失败就认为失败，然后执行权重-30
       rise 2
}

vrrp_instance WEB1 { #定义的虚拟路由器实例
    state MASTER
    interface eth0
    virtual_router_id 66
    priority 100 #权重100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 666
    }

    virtual_ipaddress {
        10.0.0.100/24 dev eth0 label eth0:0
    }

    unicast_src_ip 10.0.0.18
    unicast_peer {
    10.0.0.28
    }

    track_script {
    	check_test #调用脚本
    }
}

Keepalived2配置

#定义子配置文件
[root@keepalived2 ~]# cat /etc/keepalived/keepalived.conf 
global_defs {
   router_id KA2
}

include /etc/keepalived/conf.d/*.conf

----------------------------------------------------------------------------

#添加子配置文件
[root@keepalived2 ~]# cat /etc/keepalived/conf.d/virtual_vip.conf 
vrrp_script check_test {
       script "/etc/keepalived/down.sh"
       interval 2
       weight -30
       fall 2
       rise 2
}

vrrp_instance WEB1 {
    state BACKUP
    interface eth0
    virtual_router_id 66
    priority 80 #权重80
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 666
    }

    virtual_ipaddress {
        10.0.0.100/24 dev eth0 label eth0:0
    }

    unicast_src_ip 10.0.0.28
    unicast_peer {
    10.0.0.18
    }

    track_script {
    	check_test
    }
}

测试脚本

注意：修改检测脚本无需重启 keepalived 即可立即生效

[root@keepalived2 ~]# cat /etc/keepalived/down.sh
#!/bin/bash
[ ! -f /etc/keepalived/down ] #文件不存在则为真$?0 存在则为假$?非0

测试

重启keepalived后进行测试

#因为keepalived1的优先级高 所以目前VIP在 keepalived1上
[root@keepalived1 ~]# hostname -I
10.0.0.18 10.0.0.100 
[root@keepalived2 ~]# hostname -I
10.0.0.28

#抓包查看
[root@web1 ~]# tcpdump -nn src host 10.0.0.18 and dst host 10.0.0.28
07:36:43.287823 IP 10.0.0.18 > 10.0.0.28: VRRPv2, Advertisement, vrid 66, prio 100, authtype simple, intvl 1s, length 20

#创建测试文件
[root@keepalived1 ~]# touch /etc/keepalived/down

#观察日志，可以看到通过监测脚本发现结果为假 所以进而执行减权重
Feb 21 00:18:58 keepalived1 Keepalived_vrrp[9509]: Script `check_test` now returning 1
Feb 21 00:19:00 keepalived1 Keepalived_vrrp[9509]: VRRP_Script(check_test) failed (exited with status 1)
Feb 21 00:19:00 keepalived1 Keepalived_vrrp[9509]: (WEB1) Changing effective priority from 100 to 70
Feb 21 00:19:04 keepalived1 Keepalived_vrrp[9509]: (WEB1) Master received advert from 10.0.0.28 with higher priority 80, ours 70
Feb 21 00:19:04 keepalived1 Keepalived_vrrp[9509]: (WEB1) Entering BACKUP STATE
Feb 21 00:19:04 keepalived1 Keepalived_vrrp[9509]: (WEB1) removing VIPs.

#观察VIP飘动情况
[root@keepalived1 ~]# hostname -I
10.0.0.18 
[root@keepalived2 ~]# hostname -I
10.0.0.28 10.0.0.100

#抓包查看，注意这个时候应该反方向抓包了，因为使用的是单播模式
[root@web1 ~]# tcpdump -nn src host 10.0.0.28 and dst host 10.0.0.18
08:19:13.013447 IP 10.0.0.28 > 10.0.0.18: VRRPv2, Advertisement, vrid 66, prio 80, authtype simple, intvl 1s, length 20

#删除测试文件
[root@keepalived1 ~]# rm -f /etc/keepalived/down

#观察日志，可以看到检测脚本发现执行结果为0 进而恢复权重 因此夺回VIP
Feb 21 00:22:01 keepalived1 Keepalived_vrrp[9509]: Script `check_test` now returning 0
Feb 21 00:22:03 keepalived1 Keepalived_vrrp[9509]: VRRP_Script(check_test) succeeded
Feb 21 00:22:03 keepalived1 Keepalived_vrrp[9509]: (WEB1) Changing effective priority from 70 to 100
Feb 21 00:22:03 keepalived1 Keepalived_vrrp[9509]: (WEB1) received lower priority (80) advert from 10.0.0.28 - discarding
Feb 21 00:22:04 keepalived1 Keepalived_vrrp[9509]: (WEB1) received lower priority (80) advert from 10.0.0.28 - discarding
Feb 21 00:22:05 keepalived1 Keepalived_vrrp[9509]: (WEB1) received lower priority (80) advert from 10.0.0.28 - discarding
Feb 21 00:22:06 keepalived1 Keepalived_vrrp[9509]: (WEB1) Receive advertisement timeout
Feb 21 00:22:06 keepalived1 Keepalived_vrrp[9509]: (WEB1) Entering MASTER STATE
Feb 21 00:22:06 keepalived1 Keepalived_vrrp[9509]: (WEB1) setting VIPs.
Feb 21 00:22:06 keepalived1 Keepalived_vrrp[9509]: Sending gratuitous ARP on eth0 for 10.0.0.100
Feb 21 00:22:06 keepalived1 Keepalived_vrrp[9509]: (WEB1) Sending/queueing gratuitous ARPs on eth0 for 10.0.0.100
Feb 21 00:22:06 keepalived1 Keepalived_vrrp[9509]: Sending gratuitous ARP on eth0 for 10.0.0.100

#观察VIP飘动情况
[root@keepalived1 ~]# hostname -I
10.0.0.18 10.0.0.100
[root@keepalived2 ~]# hostname -I
10.0.0.28

#抓包查看
[root@web1 ~]# tcpdump -nn src host 10.0.0.18 and dst host 10.0.0.28
08:23:58.380622 IP 10.0.0.18 > 10.0.0.28: VRRPv2, Advertisement, vrid 66, prio 100, authtype simple, intvl 1s, length 20

故障检测脚本最佳实践

前言

keepalived 支持脚本来进行故障检测从而实现故障切换和恢复那么脚本应该怎么编写呢？我们可以从以下几个角度来判断service是否正常工作
如：进程是否存活、主页面是否能成功访问并返回200状态码、服务提供的状态信息展示等多方维度来进行判断
检测失败后，还可以配合重启服务等操作来实现自行恢复，如果能恢复那么会减少节点切换来带的时间消耗
下面举几个例子可以作为参考：

通用进程检测

方案一

**0信号说明：**对进程进行健康性检查(健康则$?结果为0，反之则为1)，但结果并不严谨因为进程只要存在它就认为进程是健康的如：进程已经成为僵尸态而$?的返回值依旧为0

#kilall默认未安装，此工具来自psmisc包
killall -0 process_name  \
> ||
> systemctl restart process_name

方案二

ps aux | grep process_name | grep -v grep &> /dev/null \
> ||
> systemctl restart process_name

方案二

systemctl is-active nginx.service &> /dev/null \
> ||
> systemctl restart process_name

MySQL健康性检测

方案一

mysqladmin -uroot -p123456 ping &> /dev/null \
> ||
> systemctl restart mysql

方案二

mysql -uroot -p123456 -e 'status' &> /dev/null \
> ||
> systemctl restart mysql

MySQL主从复制检测

[root@keepalived1 ~]# vim /etc/keepalived/check_mysql.sh
#!/bin/bash

slave_is="$(mysql -uroot -p123456 -e 'show slave status\G' | grep "Slave_.*_Running:" | awk '{print $2}')"

if [ "${slave_is[0]}" = "Yes" -a "${slave_is[1]}" = "Yes"];then
    exit 0
else
    exit 1
fi

Haproxy健康性检查

[root@haproxy-ka1 ~]# vim /etc/keepalived/check_haproxy.sh
ps aux | grep process_name | grep -v grep &> /dev/null \
||
systemctl is-active nginx.service &> /dev/null \
||
systemctl restart haproxy

故障检测脚本避坑指南

这是一起在使用 haproxy+keepalived 时遇到的坑
来自：https://github.com/kubernetes/kubeadm/blob/main/docs/ha-considerations.md
先看官方提供的脚本：

#!/bin/sh

errorExit() {
    echo "*** $*" 1>&2
    exit 1
}

curl --silent --max-time 2 --insecure https://localhost:${APISERVER_DEST_PORT}/ -o /dev/null || errorExit "Error GET https://localhost:${APISERVER_DEST_PORT}/"
if ip addr | grep -q ${APISERVER_VIP}; then
    curl --silent --max-time 2 --insecure https://${APISERVER_VIP}:${APISERVER_DEST_PORT}/ -o /dev/null || errorExit "Error GET https://${APISERVER_VIP}:${APISERVER_DEST_PORT}/"
fi

因为 keepalived 不需要标准输出，只需要 $? 的值是 0 还是非 0，所以脚本可以简化成这样：
当然还需要添加所需的变量

#!/bin/sh

APISERVER_VIP='10.0.0.123'
APISERVER_DEST_PORT='6443'

#判断一
curl --silent --max-time 2 --insecure https://localhost:${APISERVER_DEST_PORT}/ -o /dev/null

#判断二
if ip addr | grep -q ${APISERVER_VIP}; then
    curl --silent --max-time 2 --insecure https://${APISERVER_VIP}:${APISERVER_DEST_PORT}/ -o /dev/null
fi

判断一是没问题的，只要可以成功通过本机的 haproxy 访问 apiserver，那么 $? 的返回值就会为0
判断二是有问题的，假设 MSTER 节点的 haproxy 无法访问，判断一 $? 值将返回非0，这时会进行减权重操作，进而 VIP 会飘到 BACKUP上。但这时因为 VIP的6443 端口又可以正常访问了，所以MASTER 脚本的判断二又可以判断成功，进而 MSTER 节点又会将权重增加，这时 VIP 又会飘到 MASTER 节点上，但 MASTER 节点的 haproxy 已经无法访问，所以权重又会减少，VIP 又会飘到 BACKUP 上。所以最终导致的结果就是 VIP 在 MASTER 和 BACKUP 节点上不断来回飘
正确写法：（有待完善）

#!/bin/bash
#
#********************************************************************
#Author:	     	xiangzheng
#QQ: 			    767483070
#Date: 		     	2022-07-03
#FileName：		    1.sh
#URL: 		    	https://www.xiangzheng.vip
#Email: 		    rootroot25@163.com
#Description：		The test script
#Copyright (C): 	2022 All rights reserved
#********************************************************************

APISERVER_DEST_PORT='6443'

curl --silent --max-time 2 --insecure https://localhost:${APISERVER_DEST_PORT}/ -o /dev/null

总结

他人写的脚本慎用！还是借鉴或参考比较好

Keepalived

LVS+keepalived

前言

virtual_server 的文件结构

virtual_server 的配置说明

virtual_server 应用层监测

范例：

virtual_server TCP监测

范例：

实现单主单VIP的 LVS-DR 模式

部署前言

环境

安装ipvsadm工具

准备web服务和页面

将RS绑定VIP

keepalived1配置

keepalived2配置

测试

实现单主单VIP的 LVS-DR 模式2

ka1配置

心跳检测报文

实现双主多VIP的 LVS-DR 模式

Nginx+keepalived

Nginx+keepalived(单主模式)

环境准备

10.0.0.18 配置

准备nginx

准备keepalived

准备检测脚本

10.0.0.28配置

准备nginx

准备keepalived

准备检测脚本

10.0.0.38配置

10.0.0.48配置

测试

状态检测和故障切换

前言

原理

相关配置

定义脚本

调用脚本

范例：利用脚本实现主从切换

环境准备

Keepalived1配置

Keepalived2配置

测试脚本

测试

故障检测脚本最佳实践

前言

通用进程检测

方案一

方案二

方案二

MySQL健康性检测

方案一

方案二

MySQL主从复制检测

Haproxy健康性检查

故障检测脚本避坑指南

总结