NGINX Controller的目的是通过多云环境来管理NGINX Plus服务器。它为用户提供了用户界面来创建和配置NGINX Plus实例，同时还为它们提供了监控和警报功能。
安装前准备
安装相关依赖包
#jq如果安装不上的话直接下载二进制文件
yum install curl jq gettext awk bash getent grep gzip less openssl sed tar coreutils yum-plugin-versionlock
#安装Docker和kubernetes相关依赖
yum install util-linux coreutils iproute2 iptables socat ebtables ethtool conntrack
安装PostgreSQL
安装yum仓库
yum install https://download.postgresql.org/pub/repos/yum/reporpms/EL-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm -y
安装PostgreSQL客户端
yum install postgresql95 -y
安装PostgreSQL服务端
yum install postgresql95-server -y
初始化数据库
/usr/pgsql-9.5/bin/postgresql95-setup initdb
修改监听地址
默认情况下PostgreSQL只监听localhost，修改为监听本机所有IP
echo "listen_addresses='*'" >> /var/lib/pgsql/9.5/data/postgresql.conf
允许远程主机连接
0.0.0.0/0表允许所有远程主机连接。
echo "host all all 0.0.0.0/0 md5" >> /var/lib/pgsql/9.5/data/pg_hba.conf
配置随机启动
systemctl enable postgresql-9.5
启动PostgreSQL
systemctl start postgresql-9.5
为Nginx Controller创建PostgreSQL账号
切换用户
sudo -i -u postgres
启动PostgreSQL客户端
psql
启动成功后会进入下面界面
postgres=#
创建Nginx Controller使用的用户
CREATE USER naas WITH PASSWORD '123456';
允许该用户创建数据库
ALTER USER naas CREATEDB;
退出PostgreSQL客户端
\q
exit
安装Nginx Controller
#解压controller-installer-3.8.0.tar.gz安装包后执行安装脚本
[root@nginx-controller controller-installer]# ./install.sh
Install logs will be stored in /var/log/nginx-controller/nginx-controller-install.log
--- This script will install the NGINX Controller system ---
1. Checking for existing installation...
2. Checking required ports... OK
3. Attempting to detect your Operating System... Found core
4. Checking for required tools: grep basename comm sort head openssl dirname cat tee rev sed ps envsubst awk mkdir jq id less tar numfmt gunzip getent base64 yum-plugin-versionlock. All found.
#启动Docker
5. Checking Docker version...
Docker version 18.09.4, build d14af54
We recommend setting native.cgroupdriver to systemd for Docker.
WARNING! Docker configuration does not seem to have log rotation enabled. We recommend enabling log rotation for docker containers.
For steps to enable log rotation follow this link: https://success.docker.com/article/how-to-setup-log-rotation-post-installation
#配置PostgreSQL数据库登录信息
6. Database configuration
Provide the database hostname: 192.168.1.107
Provide the database port (for example, 5432): 5432
Provide the database username: naas
Provide the database password:
Repeat password:
7. Analytics Database configuration
Provide time series DB volume type [local, nfs, aws]: local
#安装Kubernetes
8. Checking Kubernetes...
Running Kubeadm with swap on is not recommended.
See: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
Will still attempt to install by ignoring swap errors.
Loading required packages: kubectl
Checking for required packages: coreutils util-linux iptables socat ebtables ethtool iproute conntrack-tools . All found.
Installing k8s components. This can take up to 4m0s
Loading required packages: kubeadm kubernetes-cni cri-tools kubelet
Loaded image: k8s.gcr.io/kube-proxy:v1.15.5
Loaded image: k8s.gcr.io/pause:3.1
Loaded image: k8s.gcr.io/etcd:3.3.10
Loaded image: k8s.gcr.io/coredns:1.3.1
Loaded image: quay.io/coreos/flannel:v0.11.0-amd64
Loaded image: k8s.gcr.io/kube-apiserver:v1.15.5
Loaded image: k8s.gcr.io/kube-controller-manager:v1.15.5
Loaded image: k8s.gcr.io/kube-scheduler:v1.15.5
[WARNING Service-Docker]: docker service is not enabled, please run 'systemctl enable docker.service'
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING Hostname]: hostname "nginx-controller" could not be reached
[WARNING Hostname]: hostname "nginx-controller": lookup nginx-controller on 114.114.114.114:53: read udp 192.168.1.107:58230->114.114.114.114:53: i/o timeout
Completed k8s components installation
9. Checking resource requirements...
Warning: Available disk space on node nginx-controller: 54GB. The Controller needs at least 80GB of disk space to work effectively.
In order to avoid performance issues, consider installing the Controller with the recommended specifications.
10. End User License Agreement
......
#同意License Agreement
Do you accept this End User License Agreement [y/n]? y
#下载Docker镜像
11. Loading Docker images...
Loaded image: nats/nats-streaming:0.0.5
Loaded image: controller-analytics/analytics-mgr:0.156.4-2170130.release-3-8
Loaded image: controller-analytics/catalogs:0.156.4-2170130.release-3-8
Loaded image: controller-analytics/db-consumer:0.156.4-2170130.release-3-8
Loaded image: controller-analytics/db-rollup-job:0.156.4-2170130.release-3-8
Loaded image: controller-analytics/metrics:0.156.4-2170130.release-3-8
Loaded image: controller-backend/controller-prod:3.8.0-2166009
Loaded image: vault-startup/vault-startup:0.3.0
Loaded image: vault:1.2.2
Loaded image: certificate-container/certificate-container:0.5.6
Loaded image: controller-analytics/clickhouse-secured-migrations:0.156.4-2170130.release-3-8
Loaded image: controller-analytics/events:0.156.4-2170130.release-3-8
Loaded image: controller-analytics/forwarder-manager:0.156.4-2170130.release-3-8
Loaded image: controller-backend/controller-openapi-engine:3.8.0-2166009
Loaded image: controller-infra/platform-mgr:2.20.7
Loaded image: nats/stan-proxy:0.0.29
Loaded image: controller-backend/controller-cron:3.8.0-2166009
Loaded image: controller-backend/controller-init:3.8.0-2166009
Loaded image: controller-backend/controller-nginx-init:3.8.0-2166009
Loaded image: controller-infra/secrets:0.1.6
Loaded image: controller-installer/apigw:3.8.0-2182317
Loaded image: nats/nats:0.0.3
Loaded image: controller-data-plane/declarative-ext-api:0.48.1-2151936.release-3-8
Loaded image: controller-infra/cloud-mgr:0.38.6
Loaded image: nats/tls-proxy:0.0.2
#设置邮箱
12. SMTP settings
Provide the SMTP host: smtp.qq.com
Provide the SMTP port: 465
Use SMTP authentication? [y/n]: y
Provide the SMTP username: 971460221@qq.com
Provide the SMTP password:
Repeat password:
Use TLS for SMTP communication? [y/n]: y
Provide a do-not-reply email address: 971460221@qq.com
#设置管理员账户
13. Admin user configuration
The FQDN, for example, controller.mycompany.com, will be used to access NGINX Controller in the browser as https://{FQDN}.
Provide the FQDN for your Controller: 192.168.1.107
Provide the organization name: chengzw
Provide the admin's first name: zhiwei
Provide the admin's last name: cheng
Provide the admin's email address: 971460221@qq.com
Provide the admin's password. Passwords must be 8 to 64 characters, and must include letters and digits:
Repeat password:
#生成自签证书
14. Checking HTTPS certificates...
An SSL/TLS cert and key location for HTTPS was not supplied by the --apigw-cert and --apigw-key flags
to install.sh invocation or through the CTR_APIGW_CERT and CTR_APIGW_KEY environment vars. There is no cert/key pair
at /opt/nginx-controller/certs/controller/server.crt (and server.key) either.
This certificate is required to establish a TLS connection between the NGINX Controller
and your web browser or agents.
If you choose not to generate a self-signed certificate,
you will be prompted to provide the path to your certificate and key files.
WARNING: Generating a self-signed certificate is not recommended for production systems.
Would you like to generate a self-signed certificate now? [y/n]? y
Generating a 4096 bit RSA private key
...................................................................................................................................++
...............................................................................................................................................++
writing new private key to '/root/controller-installer/files/k8s/base/certs/controller/server.key'
-----
15. Generating password and session salts... OK.
16. Running database initialization task...
NGINX Controller database has been initialized.
17. Starting up NGINX Controller stack...
NGINX Controller services are ready.
OK, everything went just fine!
Thank you for installing NGINX Controller.
You can find your installation in /opt/nginx-controller.
You can find the install log file in /var/log/nginx-controller/nginx-controller-install.log.
Access the system using your web browser at https://192.168.1.107.
Documentation is available at https://192.168.1.107/docs/.
安装完成后可以看到是以kubernetes的方式部署的
[root@nginx-controller ~]# kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-5c98db65d4-fkmmd 1/1 Running 0 3h7m
kube-system coredns-5c98db65d4-xmjwh 1/1 Running 0 3h7m
kube-system etcd-nginx-controller 1/1 Running 0 3h7m
kube-system kube-apiserver-nginx-controller 1/1 Running 1 3h7m
kube-system kube-controller-manager-nginx-controller 1/1 Running 1 3h7m
kube-system kube-flannel-ds-amd64-wc7r8 1/1 Running 0 3h7m
kube-system kube-proxy-xcdkq 1/1 Running 0 3h7m
kube-system kube-scheduler-nginx-controller 1/1 Running 0 3h7m
nginx-controller analytics-mgr-84f9f76468-fmgx7 1/1 Running 0 3h
nginx-controller apigw-69c47868f6-4z55x 1/1 Running 0 3h
nginx-controller apimgmt-866487957f-r86qk 2/2 Running 0 3h
nginx-controller catalogs-69789765b8-xgq7p 1/1 Running 0 3h
nginx-controller clickhouse-0 1/1 Running 0 3h
nginx-controller cloud-mgr-647db55d55-5sdqr 1/1 Running 2 3h
nginx-controller coreapi-6c69c644b9-mxsqj 1/1 Running 0 3h
nginx-controller cron-7cb4f78d54-9j4wx 1/1 Running 0 3h
nginx-controller db-consumer-7d74d9b86d-xlv66 1/1 Running 0 3h
nginx-controller db-rollup-job-1600319400-58jqt 0/1 Completed 0 13m
nginx-controller db-rollup-job-1600319700-q5wbb 0/1 Completed 0 8m7s
nginx-controller db-rollup-job-1600320000-7chbh 0/1 Completed 0 3m5s
nginx-controller declarative-ext-api-6d49bf997-jhlzq 1/1 Running 1 3h
nginx-controller events-8656fb6899-5p7b2 1/1 Running 0 3h
nginx-controller forwarder-manager-65c8f6f6fc-bs8vc 1/1 Running 0 3h
nginx-controller metrics-58cbfcf4dc-rssdw 1/1 Running 0 3h
nginx-controller nats-5b96489869-rmbp2 2/2 Running 0 3h
nginx-controller nats-streaming-6664f6675f-sk44x 1/1 Running 2 3h
nginx-controller nats-streaming-worker-6c5c876579-jtpdn 1/1 Running 2 3h
nginx-controller nats-worker-766f7c6888-strc5 1/1 Running 0 3h
nginx-controller platform-mgr-7568878ddd-qzrml 1/1 Running 0 3h
nginx-controller receiver-55b9d65c77-jfqvj 1/1 Running 0 3h
nginx-controller secrets-svc-8d545cb66-v57ff 2/2 Running 0 3h
nginx-controller stan-proxy-6b987b5f45-bd2dr 1/1 Running 1 3h
添加Nginx实例
首先创建一个locaiton，用于给实例分组
然后创建实例，选择添加一个已存在的实例
在Nginx上运行上面生成的安装Nginx Controller Agent命令
curl -k -sS -L https://192.168.1.107:8443/1.4/install/controller/ > install.sh && \ API_KEY='e21dcd9adb359d829c15ddb63a5cf08f' sh ./install.sh -i nginx-plus1 -l nginx-shanghai
#后面安装包都输y
之后可以看到Nginx成功注册到Nginx Controller上
部署一个Application
创建environment
nvironment是一个逻辑容器用于将app，gateway，certificate关联到一个域中。
创建gateway（相当于创建server块中的server_name的配置）
在gateway中可以定义一组ingress URIs（hostname）然后关联到Application Components。 例如：gateway的hostname定义的是controller.example.com，那么app components定义了两个路径/path1和/path2，这样nginx实例中的配置文件就会有controller.example.com/path1和controller.example.com/path2两个配置了。
instance是下发配置的nginx实例，URIs就是server配置块里的server_name（虚拟主机）
创建app
app是一组app components的集合
创建App components(相当于创建server块中的location配置)
填写App components名字
选择前面创建的gateway
创建两个URIs，相当于是locaiton
创建Workload Group，相当于创建upstream，上游tomcat服务器
其余配置可以保持默认，点击提交完成配置。
配置下发完成后status状态会变成绿色
在Nginx-Plus-1实例上查看下发的nginx配置
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log notice;
pid /var/run/nginx.pid;
load_module modules/ngx_http_f5_metrics_module.so;
events {
worker_connections 1024;
}
http {
f5_metrics on;
f5_metrics_server unix:/tmp/avr-socket.sr;
types {
text/html html htm shtml;
text/css css;
text/xml xml;
image/gif gif;
image/jpeg jpeg jpg;
application/javascript js;
application/atom+xml atom;
application/rss+xml rss;
text/mathml mml;
text/plain txt;
text/vnd.sun.j2me.app-descriptor jad;
text/vnd.wap.wml wml;
text/x-component htc;
image/png png;
image/svg+xml svg svgz;
image/tiff tif tiff;
image/vnd.wap.wbmp wbmp;
image/webp webp;
image/x-icon ico;
image/x-jng jng;
image/x-ms-bmp bmp;
application/font-woff woff;
application/java-archive jar war ear;
application/json json;
application/mac-binhex40 hqx;
application/msword doc;
application/pdf pdf;
application/postscript ps eps ai;
application/rtf rtf;
application/vnd.apple.mpegurl m3u8;
application/vnd.google-earth.kml+xml kml;
application/vnd.google-earth.kmz kmz;
application/vnd.ms-excel xls;
application/vnd.ms-fontobject eot;
application/vnd.ms-powerpoint ppt;
application/vnd.oasis.opendocument.graphics odg;
application/vnd.oasis.opendocument.presentation odp;
application/vnd.oasis.opendocument.spreadsheet ods;
application/vnd.oasis.opendocument.text odt;
application/vnd.openxmlformats-officedocument.presentationml.presentation pptx;
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet xlsx;
application/vnd.openxmlformats-officedocument.wordprocessingml.document docx;
application/vnd.wap.wmlc wmlc;
application/x-7z-compressed 7z;
application/x-cocoa cco;
application/x-java-archive-diff jardiff;
application/x-java-jnlp-file jnlp;
application/x-makeself run;
application/x-perl pl pm;
application/x-pilot prc pdb;
application/x-rar-compressed rar;
application/x-redhat-package-manager rpm;
application/x-sea sea;
application/x-shockwave-flash swf;
application/x-stuffit sit;
application/x-tcl tcl tk;
application/x-x509-ca-cert der pem crt;
application/x-xpinstall xpi;
application/xhtml+xml xhtml;
application/xspf+xml xspf;
application/zip zip;
application/octet-stream bin exe dll;
application/octet-stream deb;
application/octet-stream dmg;
application/octet-stream iso img;
application/octet-stream msi msp msm;
audio/midi mid midi kar;
audio/mpeg mp3;
audio/ogg ogg;
audio/x-m4a m4a;
audio/x-realaudio ra;
video/3gpp 3gpp 3gp;
video/mp2t ts;
video/mp4 mp4;
video/mpeg mpeg mpg;
video/quicktime mov;
video/webm webm;
video/x-flv flv;
video/x-m4v m4v;
video/x-mng mng;
video/x-ms-asf asx asf;
video/x-ms-wmv wmv;
video/x-msvideo avi;
}
default_type application/octet-stream;
log_format controller_recommended_log_format '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$http_x_forwarded_for" "$host" sn="$server_name" rt=$request_time ua="$upstream_addr" us="$upstream_status" ut="$upstream_response_time" ul="$upstream_response_length" cs="$upstream_cache_status" pa="$published_api_name" ';
map $host $published_api_name {
default -;
}
access_log /var/log/nginx/access.log controller_recommended_log_format;
error_log /var/log/nginx/error.log;
sendfile on;
keepalive_timeout 65;
server_tokens off;
server {
server_name cr7controller.example.com; #gateway中指定的hostname
listen 80;
status_zone server_99aa3cfe57515f541b3466aec884648e;
set $f5_gateway cr7-gateway;
f5_metrics_marker gateway $f5_gateway;
set $f5_environment cr7-environment;
f5_metrics_marker environment $f5_environment;
location /path1 { #app components中定义的URIs
set $f5_app cr7-app;
f5_metrics_marker app $f5_app;
set $f5_component cr7-path;
f5_metrics_marker component $f5_component;
set $f5_published_api '';
f5_metrics_marker published_api $f5_published_api;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header Host $host;
proxy_set_header Connection '';
proxy_http_version 1.1;
proxy_pass http://cr7-path_http_12dda1d0-13b4-46de-b55e-0b64d90fa321;
}
location /path2 { #app components中定义的URIs
set $f5_app cr7-app;
f5_metrics_marker app $f5_app;
set $f5_component cr7-path;
f5_metrics_marker component $f5_component;
set $f5_published_api '';
f5_metrics_marker published_api $f5_published_api;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header Host $host;
proxy_set_header Connection '';
proxy_http_version 1.1;
proxy_pass http://cr7-path_http_12dda1d0-13b4-46de-b55e-0b64d90fa321;
}
location / {
set $f5_app '';
set $f5_component '';
set $f5_published_api '';
return 404;
}
}
upstream cr7-path_http_12dda1d0-13b4-46de-b55e-0b64d90fa321 { #app components中定义的workload groups
zone cr7-path_http_12dda1d0-13b4-46de-b55e-0b64d90fa321 64k;
server 192.168.1.225:8080;
server 192.168.1.226:8080;
keepalive 64;
keepalive_requests 100;
keepalive_timeout 60s;
} #server-block-marker
server {
server_name 127.0.0.1;
listen 127.0.0.1:49151;
access_log off;
f5_metrics off;
location /api {
api;
}
}
}
客户端验证
默认的负载均衡策略是轮询的，所以会在上游的两台tomcat之间轮询访问。
❯ curl cr7controller.example.com/path1/index.html
this is path1 from tomcat1
❯ curl cr7controller.example.com/path1/index.html
this is path1 from tomcat2
❯ curl cr7controller.example.com/path2/index.html
this is path2 from tomcat1
❯ curl cr7controller.example.com/path2/index.html
this is path2 from tomcat2
