Apollo生产环境部署实践

一、Apollo简介

Apollo（阿波罗）是携程框架部门研发的开源配置管理中心，能够集中化管理应用不同环境、不同集群的配置，配置修改后能够实时推送到应用端，并且具备规范的权限、流程治理等特性。

1. application (应用)
2. environment (环境)
3. cluster (集群)
4. namespace (命名空间)

项目地址：https://github.com/ctripcorp/apollo

二、Apollo架构

• Portal部署在生产环境的机房，通过它来直接管理FAT、UAT、PRO等环境的配置
• Meta Server、Config Service和Admin Service在每个环境都单独部署，使用独立的数据库
• Meta Server、Config Service和Admin Service在生产环境部署在两个机房，实现双活
• Meta Server和Config Service部署在同一个JVM进程内，Admin Service部署在同一台服务器的另一个JVM进程内

三、Apollo部署

• Apollo服务端共需要两个数据库：ApolloPortalDB和ApolloConfigDB；
• ApolloPortalDB只需要在生产环境部署一个即可，而ApolloConfigDB需要在每个环境部署一套，如fat、uat和pro分别部署3套ApolloConfigDB。

[[email protected] ~]# hostnamectl set-hostname apollo-portal
[[email protected] ~]# hostnamectl set-hostname apollo-dev
[[email protected] ~]# hostnamectl set-hostname apollo-fat
[[email protected] ~]# hostnamectl set-hostname apollo-uat
[[email protected] ~]# hostnamectl set-hostname apollo-pro

[[email protected] ~]# vi /etc/hosts
192.168.66.210   apollo-portal
192.168.66.211   apollo-dev
192.168.66.212   apollo-fat
192.168.66.212   apollo-uat
192.168.66.212   apollo-pro

• portal节点实现ssh免密登入其他节点

[[email protected] ~]# ssh-keygen -t rsa
[[email protected] ~]# ssh-copy-id [email protected]
[[email protected] ~]# ssh-copy-id [email protected]
[[email protected] ~]# ssh-copy-id [email protected]
[[email protected] ~]# ssh-copy-id [email protected]
[[email protected] ~]# ssh-copy-id [email protected]

• hosts文件传给各节点,实现通过主机名解析

[[email protected] ~]# for i in 192.168.66.{211..214}; do echo ">>> $i";scp /etc/hosts [email protected]$i:/etc/; done

[[email protected] ~]# ssh-copy-id [email protected]
[[email protected] ~]# ssh-copy-id [email protected]
[[email protected] ~]# ssh-copy-id [email protected]
[[email protected] ~]# ssh-copy-id [email protected]

[[email protected] ~]# for i in 192.168.66.{210..214}; do echo ">>> $i";ssh [email protected]$i "sed -ri 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config"; done

[[email protected] ~]# for i in 192.168.66.{210..214}; do echo ">>> $i";ssh [email protected]$i "setenforce 0 && getenforce"; done

[[email protected] ~]# for i in 192.168.66.{210..214};do echo ">>> $i";ssh [email protected]$i "yum -y install gcc gcc-c++ libaio make cmake zlib-devel openssl-devel pcre pcre-devel wget git curl lynx lftp mailx mutt rsync ntp net-tools vim lrzsz screen sysstat yum-plugin-security yum-utils createrepo bash-completion zip unzip bzip2 tree tmpwatch pinfo man-pages lshw pciutils gdisk system-storage-manager git  gdbm-devel sqlite-devel chrony";done

[[email protected] ~]# for i in 192.168.66.{210..214};do echo ">>> $i";ssh [email protected]$i "yum install -y epel-release";done

• portal节点与互联网时间服务器进行同步，其他节点则于node1节点进行时间同步

[[email protected] ~]# vim /etc/chrony.conf

• 启动服务

[[email protected] ~]# systemctl start chronyd.service 
[[email protected] ~]# systemctl enable chronyd.service
[[email protected] ~]# systemctl status chronyd.service

#防火墙放通ntp服务端口：123
[[email protected] ~]# firewall-cmd --zone=public --permanent --add-port=123/udp
[[email protected] ~]# firewall-cmd --reload

• 检查主节点时间同步信息

[[email protected] ~]# timedatectl

• 其他节点关闭ntpd服务，我们这里使用chronyd服务

[[email protected] ~]# for i in 192.168.66.{211..214};do echo ">>> $i";ssh [email protected]$i "systemctl stop ntpd && systemctl disable ntpd && systemctl status ntpd";done

• 登入各个节点服务器进行手动修改chronyd

~]# vim /etc/chrony.conf
server 192.168.66.210 iburst

• 另外2个节点启动服务，在portal节点操作

[[email protected] ~]# for i in 192.168.66.{211..214};do echo ">>> $i";ssh [email protected]$i "systemctl restart chronyd.service && systemctl enable chronyd.service && systemctl status chronyd.service";done

• 检查时间同步状态 ^*表示已经同步

[[email protected] ~]# for i in 192.168.66.{211..214};do echo ">>> $i";ssh [email protected]$i "chronyc sources ";done

• 所有节点运行以下命令进行安装，安装包可自行去oracle官网下载

rpm -ivh jdk-8u231-linux-x64.rpm

• 部署方式RPM包安装，所需MySQL包文件如下：

MySQL官网地址：https://downloads.mysql.com/archives/community/

• mysql-community-server
• mysql-community-client
• mysql-community-common
• mysql-community-libs
• mysql-community-devel
• mysql-community-libs-compat

• 安装服务，所有机器执行

#卸载掉系统默认的mariadb
[[email protected] ~]# rpm -qa|grep mariadb|xargs rpm -e --nodeps

[[email protected] ~]# mkdir -p /opt/soft && cd /opt/soft
[[email protected] soft]# yum install mysql-community-*

• 启动MySQL服务(所有节点都要启动)

[[email protected] ~]# systemctl start mysqld
[[email protected] ~]# systemctl enable mysqld
[[email protected] ~]# systemctl status mysqld

• 重置ROOT密码(所有节点都需要执行)

[[email protected] ~]# cat /var/log/mysqld.log |grep password
[[email protected] ~]# mysql -uroot -p
Enter password:

#修改密码策略
mysql> set global validate_password_policy=0;

mysql> set global validate_password_mixed_case_count=0;

mysql> set global validate_password_number_count=3;

mysql> set global validate_password_special_char_count=0;

mysql> set global validate_password_length=3;

mysql> ALTER USER 'root'@'localhost' IDENTIFIED BY 'dqz123456';

mysql> flush privileges;

• 通过克隆Apollo项目，然后将项目中自带的SQL文件导入到数据库中

注意：

       1、ApolloPortalDB只需要在Applo-potal节点的数据库上创建；存储路径apollo\scripts\sql\apolloportaldb.sql

       2、ApolloConfigDB需要在DEV FAT UAT PRO 环境执行；存储路径apollo\scripts\sql\apolloconfigdb.sql

• 下载代码；如果GitHub拉取代码很慢的话可以使用gitee
• DEV/FAT/UAT/PRO环境需要下载apollo-adminservice和apollo-configservice
• potal节点需要下载apollo-portal
• 把下载的包统一放在/opt/soft目录下，并解压至/usr/local目录下

（1）下载代码并解压

[[email protected] ~]# mkdir -p /opt/soft/
[[email protected] ~]# mkdir /usr/local/apollo-portal
[[email protected] ~]# cd /opt/soft/
[[email protected] soft]# unzip apollo-portal-1.6.2-github.zip -d /usr/local/apollo-portal/

（2）导入ApolloPortalDB

• 注意：只需要在Applo-portal节点数据库执行
• 官方SQL文件地址

[[email protected] ~]# mysql -uroot -p
Enter password:

mysql> source /usr/local/apollo/scripts/sql/apolloportaldb.sql

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| ApolloPortalDB     |
| mysql              |
| performance_schema |
| sys                |
+--------------------+
5 rows in set (0.00 sec)

#导入成功后，可通过以下语句检查
mysql> select `Id`, `Key`, `Value`, `Comment` from `ApolloPortalDB`.`ServerConfig` limit 1;
+----+--------------------+-------+--------------------------+
| Id | Key                | Value | Comment                  |
+----+--------------------+-------+--------------------------+
|  1 | apollo.portal.envs | dev   | 可支持的环境列表         |
+----+--------------------+-------+--------------------------+
1 row in set (0.00 sec)

（3）创建读写权限用户

• Portal数据库创建一个有读写ApolloPortalDB库权限的用户

mysql> CREATE USER 'portal'@'%' IDENTIFIED BY 'dqz123456';
mysql> grant all privileges on ApolloPortalDB.* to [email protected]'%';
mysql> flush privileges;

（4）修改数据库连接信息

• 这里改为portal服务数据库IP地址；由于我将应用和数据库放在了同一台，所以我这里直接添加127.0.0.1地址即可
• 注意：数据库用户需要有ApolloPortalDB库的读写权限

[[email protected] soft]# cd /usr/local/apollo-portal/config/
[[email protected] config]# vim application-github.properties

（5）配置apollo-portal的meta service信息

[[email protected] config]# vim apollo-env.properties

（6）数据库数据修改

• 修改ApolloPortalDB库中UserRole，在apollo.portal.envs中添加对应的环境参数

（7）放通防火墙端口

[[email protected] ~]# firewall-cmd --permanent --zone=public --add-port=8070/tcp

[[email protected] ~]# firewall-cmd --reload

（1）解压程序文件

• 将configservic和adminservic文件上传到DEV/FAT/UAT/PRD服务器中并解压

[[email protected] ~]# mkdir /opt/soft
[[email protected] ~]# mkdir /usr/local/apollo-configservic
[[email protected] ~]# mkdir /usr/local/apollo-adminservic
[[email protected] ~]# cd /opt/soft/

[[email protected] soft]# unzip apollo-adminservice-1.6.2-github.zip -d /usr/local/apollo-adminservic/

[[email protected] soft]# unzip apollo-configservice-1.6.2-github.zip -d /usr/local/apollo-configservic/

（2）导入ApolloConfigDB

• 注意：DEV/FAT/UAT/PRO环境对应的数据库都需要执行
• 官方SQL文件地址

mysql> source /usr/local/apollo/scripts/sql/apolloconfigdb.sql
mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| ApolloConfigDB     |
| mysql              |
| performance_schema |
| sys                |
+--------------------+
5 rows in set (0.00 sec)

#导入成功后通过以下语句检查
mysql> select `Id`, `Key`, `Value`, `Comment` from `ApolloConfigDB`.`ServerConfig` limit 1;
+----+--------------------+-------------------------------+------------------------------------------------------+
| Id | Key                | Value                         | Comment                                              |
+----+--------------------+-------------------------------+------------------------------------------------------+
|  1 | eureka.service.url | http://localhost:8080/eureka/ | Eureka服务Url，多个service以英文逗号分隔             |
+----+--------------------+-------------------------------+------------------------------------------------------+
1 row in set (0.00 sec)

（3）创建读写权限用户

• DEV/FAT/UAT/PRO环境的数据库创建一个有读写ApolloConfigDB库权限的用户

mysql> CREATE USER 'apollo'@'%' IDENTIFIED BY 'dqz123456';
mysql> grant all privileges on ApolloConfigDB.* to [email protected]'%';
mysql> flush privileges;

（4）修改数据库连接信息

• 注意：同环境的配置apollo-configservice和apollo-adminservice数据库配置一致
• 注：由于在每个环境都有部署，所以对不同的环境config-service和admin-service需要配置对应环境的数据库参数；由于我这里数据库和应用都在同一台机器上，所以我这里数据库地址填写127.0.0.1即可

cd /usr/local/apollo-configservic/config/
vim application-github.properties

cd /usr/local/apollo-adminservic/config/
vim application-github.properties

• 注意：启动顺序依次是 apollo-configservice > apollo-adminservice > apollo-portal
• 默认日志存储路径/opt/logs/xxx
• 放通端口：apollo-configservice：8080；apollo-adminservice：8090；apollo-portal：8070
• 启动操作在Portal节点通过ssh远程连接操作
• 启动命令：scripts/startup.sh
• 关闭命令：scripts/shutdown.sh

#放通各环境8080、8090监听端口
[[email protected] ~]# for i in 192.168.66.{211..214}; do echo ">>> $i";ssh [email protected]$i "firewall-cmd --permanent --zone=public --add-port=8080/tcp && firewall-cmd --permanent --zone=public --add-port=8090/tcp && firewall-cmd --reload"; done

#启动各节点服务
[[email protected] ~]# for i in 192.168.66.{211..214}; do echo ">>> $i";ssh [email protected]$i "/usr/local/apollo-configservic/scripts/startup.sh && /usr/local/apollo-adminservic/scripts/startup.sh"; done


#最后启动apollo-portal
[[email protected] ~]# cd /usr/local/apollo-portal/scripts/
[[email protected] scripts]# ./startup.sh

[[email protected] ~]# cd /usr/local/apollo-configservic/scripts/
[[email protected] scripts]# ./startup.sh

• 如果启动失败，请查看日志；日志存储位置/opt/logs

• 确保四套环境的状态都正常