滴滴开源监控系统Nightingale安装使用

一、项目介绍

Nightingale 是一套衍生自 Open-Falcon 的互联网监控解决方案，融入了部分滴滴生产环境的最佳实践，灵活易用，稳定可靠

• 告警引擎重构为推拉结合模式，通过推模式保证大部分策略判断的效率，通过拉模式支持了nodata告警，去除原来的nodata组件，简化系统部署难度
• 引入了服务树，对endpoint进行层级管理，去除原来扁平的HostGroup，同时干掉告警模板，告警策略直接与服务树节点绑定，大幅提升灵活度和易用性
• 干掉原来的基于数据库的索引库，改成内存模式，单独抽出一个index模块处理索引，避免了原来MySQL单表达到亿级的尴尬局面，索引基于内存之后效率也大幅提升
• 存储模块Graph，引入facebook的Gorilla，即内存TSDB，近期几个小时的数据默认存内存，大幅提升数据查询效率，硬盘存储方式仍然使用rrdtool
• 告警引擎judge模块通过心跳机制做到了故障自动摘除，再也不用担心单个judge挂掉导致部分策略失效的问题，index模块也是采用类似方式保证可用性
• 客户端中内置了日志匹配指标抽取能力，web页面上支持了日志匹配规则的配置，同时也支持读取目标机器特定目录下的配置文件的方式，让业务指标监控更为易用
• 将portal(falcon-plus中的api)、uic、dashboard、hbs、alarm合并为一个模块：monapi，简化了系统整体部署难度，原来的部分模块间调用变成进程内方法调用，稳定性更好

二、部署安装

• 由于夜莺监控平台采用Go编写，所以需要首先安装Go环境，数据存储方面需要安装MySQL、Redis；前端页面展示需要安装Nginx
• 注：下面安装所以的程序包文件我都存储在/opt/soft目录下

• Go地址：https://golang.org/dl/

[[email protected] ~]# mkdir -p /opt/soft/
[[email protected] ~]# cd /opt/soft/
[[email protected] soft]# tar -zxvf go1.14.4.linux-amd64.tar.gz -C /usr/local/

#添加到系统环境变量中
[[email protected] ~]# vim /etc/profile
export GOROOT=/usr/local/go
export GOPATH=/usr/local/gopath
export PATH=$GOROOT/bin:$GOPATH/bin:$PATH:$HOME/bin 

[[email protected] ~]# source /etc/profile
[[email protected] ~]# go version
go version go1.14.4 linux/amd64

• 安装方式RPM包，版本1.18
• 下载地址：http://nginx.org/packages/rhel/7/x86_64/RPMS/

[[email protected] ~]# cd /opt/soft/
[[email protected] soft]# rpm -ivh nginx-1.18.0-1.el7.ngx.x86_64.rpm

[[email protected] ~]# systemctl start nginx
[[email protected] ~]# systemctl enable nginx
[[email protected] ~]# firewall-cmd --permanent --zone=public --add-port=80/tcp
[[email protected] ~]# firewall-cmd --reload

• 我这块采用RPM包的方式安装MySQL，MySQL版本为5.7.30
• 下载地址：
  • 【官网地址】https://dev.mysql.com/downloads/mysql/
  • 【清华源】https://mirrors.tuna.tsinghua.edu.cn/mysql/yum/

[[email protected] ~]# cd /opt/soft/
[[email protected] soft]# wget http://mirror.centos.org/centos/6/os/x86_64/Packages/libaio-0.3.107-10.el6.x86_64.rpm

[[email protected] soft]# wget https://mirrors.tuna.tsinghua.edu.cn/mysql/yum/mysql-5.7-community-el7-x86_64/mysql-community-common-5.7.30-1.el7.x86_64.rpm 

[[email protected] soft]# wget https://mirrors.tuna.tsinghua.edu.cn/mysql/yum/mysql-5.7-community-el7-x86_64/mysql-community-libs-5.7.30-1.el7.x86_64.rpm

[[email protected] soft]# wget https://mirrors.tuna.tsinghua.edu.cn/mysql/yum/mysql-5.7-community-el7-x86_64/mysql-community-libs-compat-5.7.30-1.el7.x86_64.rpm

[[email protected] soft]# wget https://mirrors.tuna.tsinghua.edu.cn/mysql/yum/mysql-5.7-community-el7-x86_64/mysql-community-devel-5.7.30-1.el7.x86_64.rpm

[[email protected] soft]# wget https://mirrors.tuna.tsinghua.edu.cn/mysql/yum/mysql-5.7-community-el7-x86_64/mysql-community-client-5.7.30-1.el7.x86_64.rpm

[[email protected] soft]# wget https://mirrors.tuna.tsinghua.edu.cn/mysql/yum/mysql-5.7-community-el7-x86_64/mysql-community-server-5.7.30-1.el7.x86_64.rpm

[[email protected] soft]# rpm -ivh libaio-0.3.107-10.el6.x86_64.rpm
[[email protected] soft]# yum -y install mysql-community-*

[[email protected] soft]# systemctl start mysqld
[[email protected] soft]# systemctl enable mysqld

#重置ROOT密码
[[email protected] ~]# cat /var/log/mysqld.log |grep password
[[email protected] ~]# mysql -uroot -p
Enter password:

mysql> set global validate_password_policy=0;
mysql> set global validate_password_number_count=0; 
mysql> set global validate_password_special_char_count=0; 
mysql> ALTER USER 'root'@'localhost' IDENTIFIED BY 'dqz123.';

#将下载的包存储在/opt/soft目录下
[[email protected] ~]# cd /opt/soft
[[email protected] soft]# wget http://download.redis.io/releases/redis-5.0.6.tar.gz

[[email protected] soft]# yum groupinstall 'Development Tools'
[[email protected] soft]# tar -xf redis-5.0.6.tar.gz -C /usr/local/

[[email protected] soft]# yum -y install gcc make
[[email protected] soft]# cd /usr/local/
[[email protected] local]# mv redis-5.0.6 redis
[[email protected] local]# cd redis
#编译
[[email protected] redis]# make

#安装
[[email protected] redis]# make install PREFIX=/usr/local/redis

#拷贝执行文件到/usr/local/bin下
[[email protected] ~]# cd /usr/local/redis/bin
[[email protected] bin]# cp * /usr/local/bin/

[[email protected] ~]# vim /usr/local/redis/redis.conf

• 创建数据目录和日志存储目录

[[email protected] redis]# mkdir -p /data/redis/logs

[[email protected] ~]# vim /usr/lib/systemd/system/redis.service

[Unit]
Description=Redis persistent key-value database
After=network.target

[Service]
ExecStart=/usr/local/redis/bin/redis-server /usr/local/redis/redis.conf --supervised systemd
ExecStop=/bin/kill -s QUIT $MAINPID
ExecRepload=/bin/kill -s HUP $MAINPID
Type=notify
User=root
Group=root
RuntimeDirectory=redis
RuntimeDirectoryMode=0755

[Install]
WantedBy=multi-user.target

[[email protected] ~]# systemctl daemon-reload
[[email protected] ~]# systemctl start redis
[[email protected] ~]# systemctl enable redis
[[email protected] ~]# systemctl status redis
[[email protected] ~]# ps -aux | grep redis
[[email protected] ~]# lsof -i:6379

#安装git
yum install git -y

#创建编译目录
[[email protected] ~]# source /etc/profile
[[email protected] ~]# mkdir -p $GOPATH/src/github.com/didi
[[email protected] ~]# ls $GOPATH
src

#克隆代码
[[email protected] ~]# cd $GOPATH/src/github.com/didi
[[email protected] didi]# git clone https://github.com/didi/nightingale.git

#执行编译
[[email protected] didi]# cd nightingale
[[email protected] nightingale]# ./control build
[[email protected] nightingale]# ./control pack

[[email protected] ~]# cd /usr/local/gopath/src/github.com/didi/nightingale/

[[email protected] nightingale]# mysql -uroot -p < sql/n9e_hbs.sql
[[email protected] nightingale]# mysql -uroot -p < sql/n9e_mon.sql
[[email protected] nightingale]# mysql -uroot -p < sql/n9e_uic.sql

[[email protected] nightingale]# mysql -uroot -p
mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| n9e_hbs            |
| n9e_mon            |
| n9e_uic            |
| performance_schema |
| sys                |
+--------------------+
7 rows in set (0.00 sec)

[[email protected] ~]# cd /usr/local/gopath/src/github.com/didi/nightingale/

[[email protected] nightingale]# vim etc/mysql.yml

• 修改Redis密码，默认配置文件为空。而我这里添加了Redis认证密码，所以需要修改夜莺的配置文件

[[email protected] nightingale]# vim etc/monapi.yml

[[email protected] nightingale]# vim etc/judge.yml

• 注：另外在etc/address.yml下可以看到各个模块监听的端口，如果与本地其他服务端口冲突了，就需要手工修改一下

• 在克隆的夜莺项目的代码下有关于Nginx的配置文件，我们需要拷贝至默认Nginx的配置目录下进行替换和修改

[[email protected] ~]# mv /etc/nginx/nginx.conf{,_bak}
[[email protected] ~]# cp /usr/local/gopath/src/github.com/didi/nightingale/etc/nginx.conf /etc/nginx/

[[email protected] ~]# vim /etc/nginx/nginx.conf

[[email protected] ~]# nginx -t
[[email protected] ~]# nginx -s reload

[[email protected] ~]# cd /usr/local/gopath/src/github.com/didi/nightingale/
[[email protected] nightingale]# ./control start all
[[email protected] nightingale]# ./control status

• 注：一共6个模块均需启动

三、登入使用

• 默认访问账号密码：root/root