雲計算 | Ansible部署OpenStack平台

Ansible部署OpenStack平台部署教程！

好久不見,最近還好嗎?

...

👀關於雲計算哪些事嘻嘻👀.

• 💪 各种服务器集群技术.
• 💪 并且这是免费的.

🔧准备.

• 🔧 3台计算机,或3台虚拟机.
• 🔧 操作系统Debian10.
• 🔧 安装类型,全部最小安装.

博客中的实验环境.

• 物理系统Arch Linux
• 虚拟化平台KVM
• 虚拟机操作系统Debian10

博客中虚拟机IP及主机名配置.

• MASTER ⇒ 主机名:MASTER ⇒ IP:10.43.0.29/22
• BACKUP0 ⇒ 主机名:BACKUP0 ⇒ IP:10.43.0.58/22
• BACKUP1 ⇒ 主机名:BACKUP1 ⇒ IP:10.43.0.31/22

修改镜像源.

#--最小安装的Debian10默认不能使用https源,需要安装 apt-transport-https ca-certificates
cat << "EOF" >/etc/apt/sources.list
deb http://mirrors.tuna.tsinghua.edu.cn/debian/ buster main contrib non-free
deb-src http://mirrors.tuna.tsinghua.edu.cn/debian/ buster main contrib non-free
deb http://mirrors.tuna.tsinghua.edu.cn/debian/ buster-updates main contrib non-free
deb-src http://mirrors.tuna.tsinghua.edu.cn/debian/ buster-updates main contrib non-free
deb http://mirrors.tuna.tsinghua.edu.cn/debian/ buster-backports main contrib non-free
deb-src http://mirrors.tuna.tsinghua.edu.cn/debian/ buster-backports main contrib non-free
deb http://mirrors.tuna.tsinghua.edu.cn/debian-security buster/updates main contrib non-free
deb-src http://mirrors.tuna.tsinghua.edu.cn/debian-security buster/updates main contrib non-free
EOF

apt update && \
apt -y install \
apt-transport-https \
ca-certificates     #建立本地软件包缓存列表,以及支持https源.

sed 's/http/https/g' /etc/apt/sources.list  #修改软件源为https.

apt-get update && \
apt-get -y install \
bash-completion \
vim \
wget \
build-essential  #建立软件包缓存列表,安装基本工具,编译环境;

构建Keepalived

下载并解压Keepalived源代码.

wget https://www.keepalived.org/software/keepalived-2.2.7.tar.gz    #下载Keepalived源代码.

#--解压Keepalived压缩包.
for i in *gz; do
    tar -xvf ${i}
done

配置Keepalived源代码.

./configure \
--prefix=/usr/local/keepalived/ \
--disable-fwmark \
--sbindir=/usr/local/sbin/ \
--bindir=/usr/local/bin/ \
--with-systemdsystemunitdir=/etc/systemd/system/

#--disable-fwmark 加此项关闭生成防火墙策略

安装Keepalived依赖包.

apt-get update && \
apt-get -y install \
libsnmp-dev \
libnl-route-3-dev \
libnfnetlink-dev \
libipset-dev \
iptables-dev \
libsnmp-dev \
libnl-genl-3-dev

编译并安装Keepalived.

make V=99 -j $(nproc) && \
make install >install.log

修改Keepalivedsystemd服务.

• 默认安装好的Keepalivedsystemd服务无法正常使用,需要修改哈~

[Unit]
Description=LVS and VRRP High Availability Monitor
After=network-online.target syslog.target
Wants=network-online.target
Documentation=man:keepalived(8)
Documentation=man:keepalived.conf(5)
Documentation=man:genhash(1)
Documentation=https://keepalived.org

[Service]
Type=simple
PIDFile=/run/keepalived.pid
KillMode=process
EnvironmentFile=-/usr/local/keepalived/etc/sysconfig/keepalived
ExecStart=/usr/local/sbin/keepalived -f /usr/local/keepalived/etc/keepalived/keepalived.conf $KEEPALIVED_OPTIONS
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target

配置Keepalived.

配置MASTER.

• MASTER IP:10.43.0.29/22

! Configuration File for keepalived
#--Keepalived主,MASTER
global_defs {
    router_id Nginx            #路由Id,全局唯一;
}
vrrp_script check_nginx {
    script "/usr/local/keepalived/etc/keepalived/check_nginx.sh"     #服务检查脚本;
    interval 2              #检查频率/秒;
    weight -5               #proiority减5;
    fall 3                  #失败3次;
    rise 2
}
vrrp_instance VI_1 {
    state MASTER                        #主从区分;
    interface enp1s0                    #通过ens33网口发送广播;
    mcast_src_ip 10.43.0.29             #心跳源IP,本机IP地址;
    virtual_router_id 51                #虚拟机路由编号,主从需一致;
    priority 150                        #优先级,权重;
    advert_int 1                        #心跳间隔1秒;
    #密钥认证;
    authentication {
        auth_type PASS
        auth_pass keep520
    }
    #VIP Conf;
    virtual_ipaddress {
        10.43.1.25/22
    }
    #引用脚本;
    track_script {
        check_nginx
    }
}

配置BACKUP0.

• BACKUP0 IP:10.43.0.58/22

! Configuration File for keepalived
#--Keepalived备,BACKUP0
global_defs {
    router_id Nginx
}
vrrp_script check_nginx {
    script "/usr/local/keepalived/etc/keepalived/check_nginx.sh"     #服务检查脚本;
    interval 2              #检查频率/秒;
    weight -5               #proiority减5;
    fall 3                  #失败3次;
    rise 2
}
vrrp_instance VI_1 {
    state BACKUP                        #主从区分;
    interface enp1s0                    #通过ens33网口发送广播;
    mcast_src_ip 10.43.0.58             #心跳源IP;
    virtual_router_id 51                #虚拟机路由编号,主从需一致;
    priority 140                        #优先级;
    advert_int 1                        #心跳间隔1秒;
    #密钥认证;
    authentication {
        auth_type PASS
        auth_pass keep520
    }
    #VIP Conf;
    virtual_ipaddress {
        10.43.1.25/22
    }
    #引用脚本;
    track_script {
        check_nginx
    }
}

配置BACKUP1.

• BACKUP1 IP:10.43.0.31/22

! Configuration File for keepalived
#--Keepalived备,BACKUP1
global_defs {
    router_id Nginx
}
vrrp_script check_nginx {
    script "/usr/local/keepalived/etc/keepalived/check_nginx.sh"     #服务检查脚本;
    interval 2              #检查频率/秒;
    weight -5               #proiority减5;
    fall 3                  #失败3次;
    rise 2
}
vrrp_instance VI_1 {
    state BACKUP                        #主从区分;
    interface enp1s0                    #通过ens33网口发送广播;
    mcast_src_ip 10.43.0.31             #心跳源IP;
    virtual_router_id 51                #虚拟机路由编号,主从需一致;
    priority 130                        #优先级;
    advert_int 1                        #心跳间隔1秒;
    #密钥认证;
    authentication {
        auth_type PASS
        auth_pass keep520
    }
    #VIP Conf;
    virtual_ipaddress {
        10.43.1.25
    }
    #引用脚本;
    track_script {
        check_nginx
    }
}

Nginx服务状态检查脚本.

#!/bin/bash
counter=$(ps -C nginx --no-heading |wc -l)
if [ "${counter}" = "0" ]; then
    systemctl start nginx.service
    sleep 2
    counter=$(ps -C nginx --no-heading|wc -l)
    if [ "${counter}" = "0" ]; then
        systemctl stop keepalived.service
    fi
fi

测试HA集群.

3台机器全部开启Keepalived服务(正常情况下使用MASTER).

关掉MASTER服务器,Keepalived服务(BACKUP0,替代MASTER).

关掉MASTER和BACKUP0服务器,Keepalived服务(BACKUP1将替代MASTER).

Keepalived配置文件解析.

Keepalived简介.

Keepalived.是集群管理中保证集群高可用的一个服务软件,它的作用是:检测web服务器的状态,如果有一台web服务器宕机,或出现工作故障,Keepalived将检测到,并将有故障的服务器,从集群里踢除,当web服务器工作状态正常之后,自动将web服务器加入到集群当中,解决了静态路由的单点故障问题.

Keepalived工作原理.

Keepalived是以VRRP协议为实现基础的,VRRP全称(Virtual Router Redundancy Protocol),即虚拟路由冗余协议.虚拟路由冗余协议,可以认为是实现路由器高可用的协议,也就是说N太提供相同功能的路由器组成一个路由器组,这个组里面有一个master,和多个backup,master上面有一个对外提供服务的VIP,master不断向backup发送心跳消息,告诉backup自己还活着,当backup收不到心跳信息时就认为master已经宕机啦,这时就根据VRRP的优先级,来选举一个backup当master,从而保证服务高可用.

Keepalived重要模块.

• core ⇒ Keepalived核心,负责主进程的启动,维护,以及全局配置文件的加载及解析.
• check ⇒ 负责健康检查,包括常见的各种检查方式.
• Vrrp ⇒ 负责实现Vrrp协议.

Keepalived配置文件.

Keepalived只有一个配置文件keepalived.conf,里面主要包括以下几个配置区域,分别是global_defs,static_ipaddress,static_routes,vrrp_script,vrrp_instance和virtual_server.

1, global_defs区域

• 👉主要是配置故障发生时的通知对象以及机器标识

global_defs {
    notification_email {
        a@abc.com
        b@abc.com
        ...
    }
    notification_email_from alert@abc.com
    smtp_server smtp.abc.com
    smtp_connect_timeout 30
    enable_traps
    router_id host163
}

• notification_email 故障发生时给谁发邮件通知.
• notification_email_from 通知邮件从哪个地址发出.
• smpt_server 通知邮件的smtp地址.
• smtp_connect_timeout 连接smtp服务器的超时时间.
• enable_traps 开启SNMP陷阱(Simple Network Management Protocol).
• router_id标识本节点的字条串,通常为hostname,但不一定非得是hostname.故障发生时,邮件通知会用到.

2,static_ipaddress和static_routes区域.

tatic_ipaddress和static_routes区域配置的是是本节点的IP和路由信息,如果你的机器上已经配置了IP和路由,那么这两个区域可以不用配置.
其实,一般情况下你的机器都会有IP地址和路由信息的,因此没必要再在这两个区域配置.

static_ipaddress {
    10.210.214.163/24 brd 10.210.214.255 dev eth0
    ...
}
static_routes {
    10.0.0.0/8 via 10.210.214.1 dev eth0
    ...
}

• 👉以上上分别表示启动/关闭keepalived时在本机执行的如下命令:

  ip addr add 10.210.214.163/24 brd 10.210.214.255 dev eth0
  ip route add 10.0.0.0/8 via 10.210.214.1 dev eth0
  ip addr del 10.210.214.163/24 brd 10.210.214.255 dev eth0
  ip route del 10.0.0.0/8 via 10.210.214.1 dev eth0

• 👉注意! 请忽略这两个区域,因为我坚信你的机器肯定已经配置了IP和路由.

3,vrrp_script区域.

• 👉用来做健康检查的,当时检查失败时会将vrrp_instance的priority减少相应的值.

  vrrp_script chk_http_port {
    script "</dev/tcp/127.0.0.1/80"
    interval 1
    weight -10
  }

• 👉以上意思是如果script中的指令执行失败,那么相应的vrrp_instance的优先级会减少10个点.

4,vrrp_instance和vrrp_sync_group区域.

• 👉vrrp_instance用来定义对外提供服务的``VIP```区域及其相关属性.
• 👉vrrp_rsync_group用来定义vrrp_intance组,使得这个组内成员动作一致,举个例子来说明一下其功能:
• 👉两个vrrp_instance同属于一个vrrp_rsync_group,那么其中一个vrrp_instance发生故障切换时,另一个vrrp_instance也会跟着切换(即使这个instance没有发生故障).

vrrp_sync_group VG_1 {
    group {
        inside_network   # name of vrrp_instance (below)
        outside_network  # One for each moveable IP.
        ...
    }
    notify_master /path/to_master.sh
    notify_backup /path/to_backup.sh
    notify_fault "/path/fault.sh VG_1"
    notify /path/notify.sh
    smtp_alert
}
vrrp_instance VI_1 {
    state MASTER
    interface eth0
    use_vmac <VMAC_INTERFACE>
    dont_track_primary
    track_interface {
        eth0
        eth1
    }
    mcast_src_ip <IPADDR>
    lvs_sync_daemon_interface eth1
    garp_master_delay 10
    virtual_router_id 1
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 12345678
    }
    virtual_ipaddress {
        10.210.214.253/24 brd 10.210.214.255 dev eth0
        192.168.1.11/24 brd 192.168.1.255 dev eth1
    }
    virtual_routes {
        172.16.0.0/12 via 10.210.214.1
        192.168.1.0/24 via 192.168.1.1 dev eth1
        default via 202.102.152.1
    }
    track_script {
        chk_http_port
    }
    nopreempt
    preempt_delay 300
    debug
    notify_master <STRING>|<QUOTED-STRING>
    notify_backup <STRING>|<QUOTED-STRING>
    notify_fault <STRING>|<QUOTED-STRING>
    notify <STRING>|<QUOTED-STRING>
    smtp_alert
}

• 👉 notify_master/backup/fault 分别表示切换为主/备/出错时所执行的脚本.
• 👉 notify 表示任何一状态切换时都会调用该脚本,并且该脚本在以上三个脚本执行完成之后进行调用,keepalived会自动传递三个参数($1="GROUP"|"INSTANCE",$2=name of group or instance,$3=target state of transition(MASTER/BACKUP/FAULT)).
• 👉 smtp_alert 表示是否开启邮件通知(用全局区域的邮件设置来发通知).
• 👉 state 可以是MASTER或BACKUP,不过当其他节点keepalived启动时会将priority比较大的节点选举为MASTER，因此该项其实没有实质用途.
• 👉 interface 节点固有IP(非VIP)的网卡,用来发VRRP包.
• 👉 use_vmac 是否使用VRRP的虚拟MAC地址.
• 👉 dont_track_primary 忽略VRRP网卡错误.(默认未设置)
• 👉 track_interface 监控以下网卡,如果任何一个不通就会切换到FAULT状态.(可选项)
• 👉 mcast_src_ip 修改vrrp组播包的源地址,默认源地址为master的IP.(由于是组播,因此即使修改了源地址,该master还是能收到回应的)
• 👉 lvs_sync_daemon_interface 绑定lvs syncd的网卡.
• 👉garp_master_delay 当切为主状态后多久更新ARP缓存，默认5秒.
• 👉 virtual_router_id 取值在0-255之间,用来区分多个instance的VRRP组播.
• 👉 注意: 同一网段中virtual_router_id的值不能重复,否则会出错,相关错误信息如下.

Keepalived_vrrp[27120]: ip address associated with VRID not present in received packet :
one or more VIP associated with VRID mismatch actual MASTER advert
bogus VRRP packet received on eth1 !!!
receive an invalid ip number count associated with VRID!
VRRP_Instance(xxx) ignoring received advertisment...

• 👉可以用这条命令来查看该网络中所存在的vrid:tcpdump -nn -i any net 224.0.0.0/8

• 👉priority 用来选举master的,要成为master,那么这个选项的值最好高于其他机器50个点,该项取值范围是1-255(在此范围之外会被识别成默认值100).

• 👉advert_int 发VRRP包的时间间隔,即多久进行一次master选举(可以认为是健康查检时间间隔).

• 👉authentication 认证区域,认证类型有PASS和HA(IPSEC),推荐使用PASS(密码只识别前8位).

• 👉virtual_ipaddress vip,不解释了.

• 👉virtual_routes 虚拟路由,当IP漂过来之后需要添加的路由信息.

• 👉virtual_ipaddress_excluded 发送的VRRP包里不包含的IP地址,为减少回应VRRP包的个数.在网卡上绑定的IP地址比较多的时候用.

• 👉nopreempt 允许一个priority比较低的节点作为master,即使有priority更高的节点启动.

• 👉首先nopreemt必须在state为BACKUP的节点上才生效(因为是BACKUP节点决定是否来成为MASTER的),其次要实现类似于关闭auto failback的功能需要将所有节点的state都设置为BACKUP,或者将master节点的priority设置的比BACKUP低.我个人推荐使用将所有节点的state都设置成BACKUP并且都加上nopreempt选项,这样就完成了关于autofailback功能,当想手动将某节点切换为MASTER时只需去掉该节点的nopreempt选项并且将priority改的比其他节点大,然后重新加载配置文件即可(等MASTER切过来之后再将配置文件改回去再reload一下).

• 👉当使用track_script时可以不用加nopreempt,只需要加上preempt_delay 5，这里的间隔时间要大于vrrp_script中定义的时长.

• 👉preempt_delay master启动多久之后进行接管资源(VIP/Route信息等),并提是没有nopreempt选项.

5, virtual_server_group和virtual_server区域.

virtual_server_group一般在超大型的LVS中用到,一般LVS用不过这东西,因此不多说.

virtual_server IP Port {
    delay_loop <INT>
    lb_algo rr|wrr|lc|wlc|lblc|sh|dh
    lb_kind NAT|DR|TUN
    persistence_timeout <INT>
    persistence_granularity <NETMASK>
    protocol TCP
    ha_suspend
    virtualhost <STRING>
    alpha
    omega
    quorum <INT>
    hysteresis <INT>
    quorum_up <STRING>|<QUOTED-STRING>
    quorum_down <STRING>|<QUOTED-STRING>
    sorry_server <IPADDR> <PORT>
    real_server <IPADDR> <PORT> {
        weight <INT>
        inhibit_on_failure
        notify_up <STRING>|<QUOTED-STRING>
        notify_down <STRING>|<QUOTED-STRING>
        # HTTP_GET|SSL_GET|TCP_CHECK|SMTP_CHECK|MISC_CHECK
        HTTP_GET|SSL_GET {
            url {
                path <STRING>
                # Digest computed with genhash
                digest <STRING>
                status_code <INT>
            }
            connect_port <PORT>
            connect_timeout <INT>
            nb_get_retry <INT>
            delay_before_retry <INT>
        }
    }
}

• 👉delay_loop 延迟轮询时间(单位秒).
• 👉lb_algo 后端调试算法(load balancing algorithm).
• 👉lb_kind LVS调度类型NAT/DR/TUN.
• 👉virtualhost 用来给HTTP_GET和SSL_GET配置请求header的.
• 👉sorry_server 当所有real server宕掉时，sorry server顶替.
• 👉real_server 真正提供服务的服务器.

  weight 权重.

• 👉notify_up/down 当real server宕掉或启动时执行的脚本.
  健康检查的方式，N多种方式.
• 👉path 请求real serserver上的路径.
• 👉digest/status_code 分别表示用genhash算出的结果和http状态码.
• 👉connect_port 健康检查，如果端口通则认为服务器正常.
• 👉connect_timeout,nb_get_retry,delay_before_retry分别表示超时时长,重试次数,下次重试的时间延迟。

其他选项暂时不作说明.