Keepalived源码安装与实践

1、环境准备

2、下载源码包

• keepalived官网：https://www.keepalived.org/download.html
• 主备服务器都要执行以下操作

[[email protected]_balance-1 ~]# cd /opt/soft
[r[email protected]_balance-1 soft]# wget https://www.keepalived.org/software/keepalived-2.0.20.tar.gz

• 先安装一些依赖程序，解决以下问题
  • 问题1：*** WARNING – this build will not support IPVS with IPv6. Please install libnl/libnl-3 dev libraries to support IPv6 with IPVS.
  • 问题2：configure: error: libnfnetlink headers missing
  • 问题3：configure: error:
    • !!! OpenSSL is not properly installed on your system. !!!
    • !!! Can not include OpenSSL headers files.            !!!

• 解决方案

[[email protected]_balance-1 soft]# yum -y install libnl libnl-devel libnfnetlink-devel openssl-devel

• 解决以上问题，再进行编译安装

[[email protected]_balance-1 soft]# tar -zxvf keepalived-2.0.20.tar.gz
[[email protected]_balance-1 soft]# cd keepalived-2.0.20/
[[email protected]_balance-1 keepalived-2.0.20]# ./configure --prefix=/usr/local/keepalived

[[email protected]_balance-1 keepalived-2.0.20]# make && make install

3、配置文件详解

! Configuration File for keepalived

global_defs {
     notification_email {
     # email 接收方
     [email protected]
     [email protected]
     [email protected]
   }
   # email 发送方
   notification_email_from [email protected]
   # 邮件服务器, smtp 协议
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id app2
   vrrp_skip_check_adv_addr
   
   # 使用 unicast_src_ip 需要注释 vrrp_strict，而且也可以进行 ping 测试
   #vrrp_strict 
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}

# vrrp实例
vrrp_instance VI_1 {
    # 指定 keepalived 的角色，MASTER 表示此主机是主服务器，BACKUP 表示此主机是备用服务器
    state MASTER

    # 指定网卡
    interface ens33

    # 虚拟路由标识，这个标识是一个数字，同一个vrrp实例使用唯一的标识。
    # 即同一vrrp_instance下，MASTER和BACKUP必须是一致的
    virtual_router_id 51

    # 定义优先级，数字越大，优先级越高（0-255）。
    # 在同一个vrrp_instance下，MASTER 的优先级必须大于 BACKUP 的优先级
    priority 100

    # 设定 MASTER 与 BACKUP 负载均衡器之间同步检查的时间间隔，单位是秒
    advert_int 1

    # 如果两节点的上联交换机禁用了组播，则采用 vrrp 单播通告的方式
    unicast_src_ip 192.168.66.11
    unicast_peer {
        192.168.66.12
    }

    # 设置验证类型和密码
    authentication {
        #设置验证类型，主要有PASS和AH两种
        auth_type PASS
        #设置验证密码，在同一个vrrp_instance下，MASTER与BACKUP必须使用相同的密码才能正常通信
        auth_pass 1111
    }

    #设置虚拟IP地址，可以设置多个虚拟IP地址，每行一个
    virtual_ipaddress {
        # 虚拟 IP
        192.168.66.100/24  // 如果两个nginx的ip分别是192.168.66.11,12，则此处的虚拟ip跟它俩同一个网段即可 24代表3个255的子网掩码
    }
}

# 虚拟服务器端口配置（以下配置根据实际情况配置）
virtual_server 192.168.66.100 80 {
    delay_loop 6
    lb_algo rr
    lb_kind NAT
    persistence_timeout 50
    protocol TCP

    real_server 192.168.66.11 80 {
        weight 1
    }
}

4、修改配置文件

[[email protected]_balance-1 keepalived-2.0.20]# cd /usr/local/keepalived/
[[email protected]_balance-1 keepalived]# cd etc/keepalived/

• 先备份一下

[[email protected]_balance-1 keepalived]# cp keepalived.conf{,_bak}
[[email protected]_balance-1 keepalived]# ls
keepalived.conf  keepalived.conf_bak  samples

• 主修改的内容

[[email protected]_balance-1 keepalived]# vim keepalived.conf
! Configuration File for keepalived

global_defs {
   notification_email {
     [email protected]
     [email protected]
     [email protected]
   }
   notification_email_from [email protected]
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id load_balance-1     //唯一值，一般为所在服务器的主机名
   vrrp_skip_check_adv_addr
   #vrrp_strict                 //注释该参数，如果开启了防火墙一定要注释
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}

vrrp_instance VI_1 {
    state MASTER            //主为MASTER,备为BACKUP
    interface ens33         //绑定网卡,此处为网卡名称
    virtual_router_id 51    //主备Id必须一致,默认值51
    priority 100                //优先级，主必须大于备
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
       192.168.66.100/24      //虚拟IP地址，跟真实网卡同一网段即可，必须主备一致
    }
}

• 备修改的内容

[[email protected]_balance-2 keepalived-2.0.20]# cd /usr/local/keepalived/
[[email protected]_balance-2 keepalived]# cd etc/keepalived/
[[email protected]_balance-2 keepalived]# cp keepalived.conf{,_bak}
[[email protected]_balance-2 keepalived]# ls
keepalived.conf  keepalived.conf_bak  samples

[[email protected]_balance-2 keepalived]# vim keepalived.conf
! Configuration File for keepalived

global_defs {
   notification_email {
     [email protected]
     [email protected]
     [email protected]
   }
   notification_email_from [email protected]
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id load_balance-2   //唯一值，一般为所在服务器的主机名
   vrrp_skip_check_adv_addr
   #vrrp_strict              //注释该参数，如果开启了防火墙一定要注释
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}

vrrp_instance VI_1 {
    state BACKUP             //备为BACKUP
    interface ens33          //绑定的网卡名称
    virtual_router_id 51     //与主ID保持一致
    priority 90              //备的优先级低于主
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.66.100/24     //虚拟IP地址，与主保持一致
    }
}

5、配置防火墙

• 防火墙放通VRRP组包发送
• Keepalived使用vrrp组播，默认地址是224.0.0.18，因此要配置防火墙放过
• 主备服务器都需要执行

[[email protected]_balance-1 ~]# firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 0 --destination 224.0.0.18 --protocol vrrp -j ACCEPT
[[email protected]_balance-1 ~]# firewall-cmd --direct --permanent --add-rule ipv4 filter OUTPUT 0 --destination 224.0.0.18 --protocol vrrp -j ACCEPT

# 注意下面的网卡更改为自己服务器的真实网卡名称(更改前一定要确认自己服务器的网卡名称)
[[email protected]_balance-1 ~]# firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 0  --in-interface ens33 --destination 224.0.0.18 --protocol vrrp -j ACCEPT

[[email protected]_balance-1 ~]# firewall-cmd --direct --permanent --add-rule ipv4 filter OUTPUT 0  --out-interface ens33 --destination 224.0.0.18 --protocol vrrp -j ACCEPT

[[email protected]_balance-1 ~]# firewall-cmd --reload
success

• 如果要删除策略
  • 将策略中的--add-rule 改成 --remove-rule ，执行下就删除了
• 关闭SELINUX

[[email protected]_balance-1 ~]# sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/sysconfig/selinux

[[email protected]_balance-1 ~]# setenforce 0
[[email protected]_balance-1 ~]# getenforce 
Permissive

6、监控Nginx心跳

• 当Nginx服务宕机则keeplived的vip地址自动漂移至另外节点
• 主备都需要添加

[[email protected]_balance-1 ~]# cd /script/
[[email protected]_balance-1 script]# vim check_nginx.sh
#!/bin/bash
network=`ifconfig ens33 |grep -w inet |awk '{print $2}'` &>/dev/null
nginx=`ps -C nginx --noheader|wc -l` &>/dev/null
if [ "$nginx" -eq 0 ];then
    #这里调用消息通知脚本，并配置需要发送的消息
    /script/qywx_nginx.py "主机：$(hostname)--主机IP：$network：Nginx服务心跳检测异常,请登入服务器查看;VIP已切换,请注意"
    systemctl stop keepalived.service
fi

• 主备服务器都确保安装了pip和request模块

#安装pip和request库
[[email protected]_balance-1 ~]# curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py   # 下载安装脚本

[[email protected]_balance-1 ~]# python get-pip.py
[[email protected]_balance-1 ~]# pip install requests

#这里我通过企业微信机器人的方式将Nginx健康检查的状态推送至企业微信
[[email protected]_balance-1 ~]# cd /script/
[[email protected]_balance-1 script]# vim qywx_nginx.py
#!/usr/bin/python
# -*- coding: utf-8 -*-
import requests
import json
import sys
import os

headers = {'Content-Type': 'application/json;charset=utf-8'}
api_url = "企业微信机器人的webhook地址"
def msg(text):
    json_text= {
     "msgtype": "text",
        "text": {
            "content": text
        },
    }
    print requests.post(api_url,json.dumps(json_text),headers=headers).content

if __name__ == '__main__':
    text = sys.argv[1]
    msg(text)

[[email protected]_balance-1 script]# chmod +x check_nginx.sh qywx_nginx.py

• 将脚本传给备机

[[email protected]_balance-1 script]# scp check_nginx.sh [email protected]:/script/
[[email protected]_balance-1 script]# scp qywx_nginx.py [email protected]:/script/

#注意：备机将这2个脚本同样需要赋予执行权限

[[email protected]_balance-1 ~]# cd /usr/local/keepalived/etc/keepalived/

[[email protected]_balance-1 keepalived]# vim keepalived.conf
! Configuration File for keepalived

global_defs {
   notification_email {
     [email protected]
     [email protected]
     [email protected]
   }
   notification_email_from [email protected]
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id load_balance-1		#每台主机该值必须唯一
   vrrp_skip_check_adv_addr
   # vrrp_strict        //注释该参数
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}

vrrp_script check_nginx {
    script "/bin/bash /script/check_nginx.sh"	#上面步骤中创建的Nginx健康检查脚本路径
    interval 2			#每隔2秒执行一次脚本
}

vrrp_instance VI_1 {
    state MASTER
    interface ens33
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
       192.168.66.100/24
    }
    track_script {   #调用脚本
       check_nginx   #这里的名称必须与vrrp_scrip写的保证一致
    }
}

• 将配置文件传给备机，并进行相应的修改

[[email protected]_balance-1 keepalived]#  scp keepalived.conf [email protected]:/usr/local/keepalived/etc/keepalived/

[[email protected]_balance-2 ~]# cd /usr/local/keepalived/etc/keepalived/
[[email protected]_balance-2 keepalived]# vim keepalived.conf
! Configuration File for keepalived

global_defs {
   notification_email {
     [email protected]
     [email protected]
     [email protected]
   }
   notification_email_from [email protected]
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id load_balance-2
   vrrp_skip_check_adv_addr
   # vrrp_strict         //注释该参数
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}

#注意此处添加配合Nginx服务做健康检查，如果没有Nginx则无需添加
vrrp_script check_nginx {
    script "/bin/bash /script/check_nginx.sh"	#Nginx健康检查脚本路径
    interval 2			#每隔2秒执行一次脚本
}

vrrp_instance VI_1 {
state BACKUP
    interface ens33
    virtual_router_id 51
    priority 90
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
	192.168.66.100/24
    }
    track_script {   #调用脚本
       check_nginx   #这里的名称必须与vrrp_scrip写的保证一致
    }
}

7、启动服务

• keepalived默认会读取/etc/keepalived/keepalived.conf配置文件
• 主备服务器上都需要执行

[[email protected]_balance-1 ~]# mkdir /etc/keepalived/
[[email protected]_balance-1 ~]# cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/

[[email protected]_balance-2 ~]# mkdir /etc/keepalived/
[[email protected]_balance-2 ~]# cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/

• 主备服务节点启动服务

[[email protected]_balance-1 ~]# systemctl start keepalived
[[email protected]_balance-1 ~]# systemctl enable keepalived
[[email protected]_balance-1 ~]# systemctl status keepalived

8、查看虚拟IP情况

[[email protected]_balance-1 ~]# ip a | grep ens33
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    inet 192.168.66.11/24 brd 192.168.66.255 scope global noprefixroute ens33
inet 192.168.66.100/24 scope global secondary ens33

• 可通过手动停止nginx服务，来触发Nginx健康脚本；然后观察VIP是否漂移至BACKUP节点；
• 当MASTER节点没有配置不抢占模式后，手动启动MASTER节点keepalived，检查VIP是否漂移到了MASTER节点；
• 当MASTER节点配置了不抢占(nopreempt )模式后，查看VIP是否还会保留在BACKUP节点。

9、问题总结

• 启动keepalived服务后，发现主备的ens33网卡都出现vip
• 通过tcpdump抓包发现，2台主机都在同时向224.0.0.18发送消息

• 备机在一定的时间内未收到主节点发出的VRRP广播报文，从而备一直以为主挂掉了。
• 关闭SELINUX(这个安全性太高，关掉)
• 关闭firewalld(生产环境放开，测试环境可关闭)
• Keepalived使用vrrp组播，默认地址是224.0.0.18，因此要配置防火墙放过。

[[email protected]_balance-1 ~]# firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 0 --destination 224.0.0.18 --protocol vrrp -j ACCEPT
[[email protected]_balance-1 ~]# firewall-cmd --direct --permanent --add-rule ipv4 filter OUTPUT 0 --destination 224.0.0.18 --protocol vrrp -j ACCEPT

[[email protected]oad_balance-1 ~]# firewall-cmd --reload
success

• 关闭SELINUX

[[email protected]_balance-1 ~]# sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/sysconfig/selinux

[[email protected]_balance-1 ~]# setenforce 0
[[email protected]_balance-1 ~]# getenforce 
Permissive

• 配置好keepalived无法启动，无错误日志，就是查看keepalived状态一直是没有启动
• 查看日志/var/log/masseges

• 原因：查看是否配置了Nginx健康检查脚本 (或其他服务)
• 解决方案：如果配置了Nginx健康检查(或其他服务的)，那么就检查服务器上是否存在对应的服务，不然keepalived会定期检查，如果没有这个服务则直接触发服务健康检查脚本，自动停止keepalived服务，所以我出现无任务错误日志，就是keepalived起不来的假象。

问题描述：

keepalived服务运行正常，日志无报错，但是访问VIP地址无法访问，ping不通；但是关闭防火墙后居然就可以了

问题原因：

1、防火墙问题，网上好多文章教程都是关闭防火墙后搭建keppalived，所以不会有这些问题；但是线上环境肯定是开启了防火墙的，所以就会有各种各样的问题

2、 配置文件中开启了vrrp_stric，未注释掉(该参数默认是开启的)

解决方案：

• 添加以下2条防火墙策略规则

# 注意下面的网卡更改为自己服务器的真实网卡名称(更改前一定要确认自己服务器的网卡名称)
[[email protected]_balance-1 ~]# firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 0  --in-interface ens33 --destination 224.0.0.18 --protocol vrrp -j ACCEPT

[[email protected]_balance-1 ~]# firewall-cmd --direct --permanent --add-rule ipv4 filter OUTPUT 0  --out-interface ens33 --destination 224.0.0.18 --protocol vrrp -j ACCEPT

[[email protected]_balance-1 ~]# firewall-cmd --reload
success

• 修改配置文件注释掉vrrp_stric(主备服务都需要注释掉)

[[email protected]_balance-1 ~]# cd /etc/keepalived
[[email protected]_balance-1 keepalived]# vim keepalived.conf
! Configuration File for keepalived

global_defs {
   notification_email {
     [email protected]
     [email protected]
     [email protected]
   }
   notification_email_from [email protected]
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id load_balance-1
   vrrp_skip_check_adv_addr
   #vrrp_strict		              //注释该参数，如果开启了防火墙一定要注释
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}
...