Teuthology架构

哈哈哈123asdf
789 阅读3分钟

Teuthology架构


自动化测试框架流程大致如下：



软件组成

Teuthology 测试框架所使用到的软件组成如下：


  • Jenkins 持续集成工具
  • teuthology Ceph 测试套件
  • shaman 查询提供软件包的 chacra 节点，水平扩展和调度 chacra 节点
  • chacra 提供不同架构二进制包或文件的管理 REST API 接口


Teuthology部署


部署paddles/pulpito node


安装依赖


yum install git python-dev python-virtualenv postgresql postgresql-contrib postgresql-server-dev-all supervisor gcc epel-release


配置postgresql数据库


初始化数据库


postgresql-setup initdb

systemctl start postgresql //启动数据库

systemctl enable postgresql


postgresql安装并启动后，会自动生成postgres用户


su - postgres //进入该用户

-bash-4.2$ psql //进入数据库

postgres=# \password postgres //更改用户postgres的密码


//按照输出的提示输入密码即可


创建数据库实例paddles、shaman


postgres=#create database paddles; // 创建名为paddles的数据库

postgres=#create database shaman; // 创建名为shaman的数据库

postgres=#create database chacra; // 创建名为chacra的数据库


安装配置paddles


创建paddles用户，并克隆paddles项目


useradd -m paddles -g root -G root //新建用户 paddles

su - paddles //切换到paddles用户下

git clone https://github.com/ceph/paddles.git //克隆该项目

cd paddles

virtualenv ./virtualenv //创建python的沙盒环境

source ./virtualenv/bin/activate //进入沙盒环境


修改配置文件，paddles中有配置文件的例子，只需要复制一份并修改即可。

cp config.py.in config.py


需要修改的内容

job_log_href_templ= 'http://qa-proxy.ceph.com/teuthology/{run_name}/{job_id}/teuthology.log' # 将qa-proxy.ceph.com/teuthology改成自己对应日志服务器的地址

...

sqlalchemy = {

# You may use SQLite for testing

'url': ' postgresql://postgres:password@localhost/paddles',

# When you set up PostreSQL, it will look more like:

#'url': 'postgresql+psycopg2://USER:PASSWORD@DB_HOST/DB_NAME',

'echo': True,

'echo_pool': True,

'pool_recycle': 3600,

'encoding': 'utf-8'

}


修改/var/lib/pgsql/data/pg_hba.conf,解决初始化paddles环境时的报错Ident authentication failed。

# "local" is for Unix domain socket connections only

local all all trust

# IPv4 local connections:

host all all 127.0.0.1/32 md5

# IPv6 local connections:

host all all ::1/128 trust


修改完后，保存并退出。然后安装相关依赖，初始化环境。

$pip install -r requirements.txt //安装相关依赖

$python setup.py develop // 初始化环境

$pecan populate config.py// 在postgresql创建paddles需要的表


配置数据迁移工具

$ cp alembic.ini.in alembic.ini

$ vi alembic.ini

sqlalchemy.url = postgresql://postgres:password@localhost/paddles

$alembic stamp head//触发迁移工具生效


通过supervisord部署paddles服务

#cat /etc/supervisord.d/paddles.ini

[program:paddles]

user=paddles

environment=HOME="/home/paddles", USER="paddles"

directory=/home/paddles/paddles

command=/home/paddles/paddles/virtualenv/bin/gunicorn_pecan -c gunicorn_config.py config.py

autostart=true

autorestart=true

redirect_stderr=true

stdout_logfile = /home/paddles/paddles.out.log

stderr_logfile = /home/paddles/paddles.err.log


安装配置pulpito


创建pulpito用户，切换到pulpito用户进行部署

#useradd -m pulpito -g root -G root

#su - pulpito

$git clone https://github.com/ceph/pulpito.git

$cd pulpito

$virtualenv ./virtualenv


修改配置文件，paddles中有配置文件的例子，只需要复制一份并修改即可。

$cp config.py.in prod.py

$vi prod.py


主要的修改部分如下：

………

paddles_address = 'http://192.168.122.34:8080' #实际的paddles服务地址

………

'debug': False,


修改完后，保存并退出。然后安装相关依赖。

$source ./virtualenv/bin/activate

$pip install -r requirements.txt //安装相关依赖


通过supervisord部署pulpito服务

#cat /etc/supervisord.d/pulpito.ini

[program:pulpito]

user=pulpito

directory=/home/pulpito/pulpito

command=/home/pulpito/pulpito/virtualenv/bin/python run.py

environment=HOME="/home/pulpito",USER="pulpito"

autostart=true

autorestart=true

redirect_stderr=true

stdout_logfile = /home/pulpito/pulpito.out.log

stderr_logfile = /home/pulpito/pulpito.err.log

启动supervisor

sudo supervisord -c /etc/supervisord.conf

sudo supervisorctl reload

sudo supervisorctl start all//启动

sudo supervisorctl status//查看状态，可以看到paddles 和pulpito的进程启动了

检测服务


在浏览器或者终端，访问 http://{ip}:8080/和http://{ip}:8081/就可以了，看到如下界面：


部署shaman

shaman的部署方式分为两种一种是手动部署，另一种通过ansible，在此采用的是手动部署。


安装相关依赖

yum install epel-release gcc gcc-c++ glibc-devel make ncurses-devel openssl-devel autoconf java-1.8.0-openjdk-devel git wget wxBase.x86_64 erlang


安装RabbitMQ

wget https://www.rabbitmq.com/releases/rabbitmq-server/v3.6.1/rabbitmq-server-3.6.1-1.noarch.rpm

rpm --import https://www.rabbitmq.com/rabbitmq-signing-key-public.asc

yum install rabbitmq-server-3.6.1-1.noarch.rpm

systemctl start rabbitmq-server #启动rabbitmq-server

systemctl enable rabbitmq-server #设置rabbitmq-server开机自启动


部署shaman


创建shaman用户，并克隆shaman项目，然后创建沙盒环境。

useradd -m shaman -g root -G root //新建用户shaman

su - shaman //切换到shaman用户下

git clone https://github.com/ceph/shaman.git //克隆该项目

cd shaman

virtualenv ./virtualenv //创建python的沙盒环境

source ./virtualenv/bin/activate //进入沙盒环境


安装一些在requirements.txt中没有的软件依赖

pip install --upgrade setuptools

pip install --upgrade pip

pip install "cherrypy>=3.8.0,<9.0.0"

pip install wsgiserver


修改配置文件config/dev.py。

$vim config/dev.py

...

],

'debug': False,

}

...

sqlalchemy_w = {

#'url': 'sqlite:///dev.db',

'url': 'postgresql://postgres:password@localhost/shaman',

'echo': True,

'echo_pool': True,

'pool_recycle': 3600,

'encoding': 'utf-8'

}

sqlalchemy_ro = {

#'url': 'sqlite:///dev.db',

'url': 'postgresql://postgres:password@localhost/shaman',

'echo': True,

'echo_pool': True,

'pool_recycle': 3600,

'encoding': 'utf-8'

}

....


修改config/run.py，需要修改的内容为，其中服务IP根据具体改动：

simpleapp_wsgi_app = deploy('config/dev.py')

...

server = wsgiserver.CherryPyWSGIServer(('0.0.0.0', 8082), application, server_name='simpleapp')


初始化环境

$pip install -r requirements.txt //安装相关依赖

$python setup.py develop // 初始化环境

$pecan populate config/dev.py// 在postgresql创建shaman需要的表


配置数据迁移工具

$ cp alembic.ini.in alembic.ini

$ vi alembic.ini

sqlalchemy.url = postgresql://postgres:password@localhost/shaman

$alembic stamp head//触发迁移工具生效


配置supervisord


通过supervisord部署shaman服务

#cat /etc/supervisord.d/shaman.ini

[program:shaman]

user=shaman

directory=/home/shaman/shaman

command=/home/shaman/shaman/virtualenv/bin/python config/run.py

environment=HOME="/home/shaman",USER="shaman"

autostart=true

autorestart=true

redirect_stderr=true

stdout_logfile = /home/shaman/shaman.out.log

stderr_logfile = /home/shaman/shaman.err.log


启动shaman服务

sudo supervisorctl reread && sudo supervisorctl update shaman && sudo supervisorctl start shaman


检测服务

在浏览器或者终端，访问 http://{ip}:8082/就可以了，看到如下界面：


部署chacra

chacra的部署较为复杂，采取的是ansible的方式，但是官方提供的部署文件是针对ubuntu系统的，需要进行改动。


修改playbooks相关文件


修改deploy/playbooks/examples/deploy_test.yml中对应参数

development_server: true

api_user: "admin"

api_key: "secret"

graphite_api_key: '1234-asdf-1234'

callback_url: "http://{ip}:8082/api/repos/" #shaman的地址

callback_user: "admin"

callback_key: "secret"

health_ping_url: "http://{ip}:8082/api/nodes/" #shaman的地址


修改deploy/playbooks/roles/common/tasks/letsencrypt.yml

#删除下面几行

- name: install software-properties-common

apt:

name: software-properties-common

state: latest

update_cache: yes

become: true

- name: add certbot PPA

apt_repository:

repo: "ppa:certbot/certbot"

become: true


修改deploy/playbooks/roles/common/tasks/rabbitmq.yml

#删除下面几行

- name: add the rabbitmq official repo

become: true

apt_repository:

repo: "deb http://www.rabbitmq.com/debian/ testing main"

state: present

- name: add the rabbitmq public key

become: true

apt_key:

url: "https://www.rabbitmq.com/rabbitmq-release-signing-key.asc"

state: present

- name: update the apt cache

become: true

apt:

update_cache: true


修改deploy/playbooks/roles/common/tasks/systemd.yml

# 将state由running改为started

service: name=chacra state=started enabled=yes

service: name=chacra-celery state=started enabled=yes

service: name=chacra-celerybeat state=started enabled=yes


修改deploy/playbooks/roles/common/templates/systemd/chacra-celery.service.j2

# 由于%I在systemd中有特殊的含义，所有在此将%I改为%%I

ExecStart={{ app_home }}/bin/celery multi start 5 -Q:1,2 poll_repos,celery -Q:3-5 build_repos -A async --logfile=/var/log/celery/%n%%I.log

ExecReload={{ app_home }}/bin/celery multi restart 5 -Q:1,2 poll_repos,celery -Q:3-5 build_repos -A async --logfile=/var/log/celery/%n%%I.log


部署chacra


使用ansible-playbook部署ansible服务


检查服务

在浏览器或者终端，访问 http://{ip}:8083/就可以了，看到如下界面：


Teuthology Node


Teuthology部署中官方建议为两个角色scheduler和worker。


创建/etc/teuthology.yaml


内容如下：

lab_domain: didichuxing.com #testnode的域名后缀

archive_base: /home/teuthworker/archive # 存储日志信息的地方

automated_scheduling: False

reserve_machines: 0

queue_host: 127.0.0.1

queue_port: 11300

lock_server: 'http://{ip}:8080' # paddles的地址

results_server: 'http://{ip}:8080' # paddles的地址

results_ui_server: 'http://{ip}:8081' # pulpito的地址

#results_sending_email: 'teuthology',

results_timeout: 43200

# Verify the packagee signatures

check_package_signatures: false

ceph_git_base_url: 'git@git.xiaojukeji.com:chishui/' # gitlab地址

ceph_git_url: 'git@git.xiaojukeji.com:chishui/deph.git'

ceph_qa_suite_git_url: 'git@git.xiaojukeji.com:chishui/deph.git'

use_conserver: False

conserver_master: 'conserver.front.sepia.ceph.com'

conserver_port: 3109

max_job_time: 259200 # 3 days

baseurl_template: 'http://{host}/{proj}-{pkg_type}-{dist}-{arch}-{flavor}/{uri}'

use_shaman: True

shaman_host: '{ip}:8082'

src_base_path: /home/teuthworker/src

teuthology_path: /home/teuthworker/src/teuthology_master

suite_verify_ceph_hash: False

suite_allow_missing_packages: False


安装依赖


yum -y install git python-devel python-pip python-virtualenv libevent-devel libvirt-python beanstalkd


启动beanstackd队列：

systemctl start beanstalkd

systemctl enable beanstalkd # 配置开机自启动


部署teuthology


创建teuthology与teuthworker两个用户：

useradd -m teuthology -g root -G root

useradd -m teuthworker -g root -G root


分别给两个账号授予passwordless sudo access权限：

useradd ubuntu

vim /etc/sudoers

#添加下面两行

teuthology ALL=(ALL) NOPASSWD: ALL

teuthworker ALL=(ALL) NOPASSWD: ALL


scheduler

su - teuthology

mkdir ~/src

git clone https://github.com/ceph/teuthology.git src/teuthology_master

pushd src/teuthology_master/

./bootstrap

popd


worker

su - teuthworker

mkdir ~/src

git clone https://github.com/ceph/teuthology.git src/teuthology_master

pushd src/teuthology_master/

./bootstrap

popd

mkdir ~/bin

wget -O ~/bin/worker_start https://raw.githubusercontent.com/ceph/teuthology/master/docs/_static/worker_start.sh

echo 'PATH="$HOME/src/teuthology_master/virtualenv/bin:$PATH"' >> ~/.profile

source ~/.profile

mkdir -p ~/archive/worker_logs

worker_start magna 1


使用worker_start来启动一个plana的工作进程

worker_start plana 1


配置teuthology和teuthworker用户与test nodes间的免密登录

ssh-copy-id -i ~/.ssh/id_rsa.pub ubuntu@{machine_type}00x.example.com


Test Nodes


添加ubuntu用户

在每个test node上创建ubuntu用户，并为ubuntu用户授予passwordless sudo access权限。

useradd ubuntu

passwd ubuntu

New password: <密码>

etype new password: <密码>

vim /etc/sudoers

#添加下面一行

ubuntu All=(ALL) NOPASSWD:ALL


配置ssh

#vim /etc/ssh/sshd_config

//更改为下面行

#HostKey /etc/ssh/ssh_host_key

# HostKeys for protocol version 2

HostKey /home/ubuntu/.ssh/id_rsa

#HostKey /etc/ssh/ssh_host_rsa_key

#HostKey /etc/ssh/ssh_host_dsa_key

#HostKey /etc/ssh/ssh_host_ecdsa_key

#HostKey /etc/ssh/ssh_host_ed25519_key

......

PermitRootLogin yes

......

PubkeyAuthentication yes

......

PasswordAuthentication yes

......

#systemctl restart sshd


安装依赖

yum install -y redhat-lsb yum-priorities genisoimage wget


submitting nodes


在paddles中注册test nodes

wget https://raw.githubusercontent.com/ceph/teuthology/master/docs/_static/create_nodes.py


修改create_nodes.py

paddles_url = 'http://{ip}:8080/nodes/' # paddles的地址

machine_type = 'plana'

lab_domain = 'example.com'

# Don't change the user. It won't work at this time.

user = 'ubuntu'

# We are populating 'plana001' -> 'plana003'

machine_index_range = range(1, 4) # 创建了'plana001' -> 'plana003' 3台机器

......


日志服务器部署 添加nginx服务配置

server {

listen 80;

server_name {ip};

default_type text/plain;

access_log /var/log/nginx/log-access.log;

error_log /var/log/nginx/log-error.log;

location / {

autoindex on;

alias /home/teuthworker/archive/;

}

}


重启nginx服务

systemctl restart nginx


修改paddles配置中的job_log_href_templ的值为：

job_log_href_templ= 'http://{ip}/{run_name}/{job_id}/teuthology.log'

作者：路锦博


avatar
文章
阅读
粉丝