部署环境准备
安装centos7系统后，只包含了很少的工具包。
首先给yum换源
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
查看ip地址
ipaddr或者ifconfig
ifconfig的运行需要net-tools工具包
yum install -y net-tools
读入u盘需要ntfs-3g包,换源后的yum可以直接下载
yum install -y ntfs-3g
挂载u盘
fdisk -l
mount -t ntfs-3g /dev/sdb1 /mnt
另外一些常用命令
tar -czvf ...tar.gz . #压缩当前文件夹
tar -xzvf ...tar.gz /home/... #解压某个文件到/home/...
ps -ef |grep xxx #查找xxx的运行状态，包含进程号
kill pid #杀死某个进程
curl www.baidu.com #访问baidu网站，一般会返回首页html，也可用于代替ping命令
cp -r #复制文件夹及其全部子文件
chmod -R xxx #给某文件夹及其全部子文件赋权限
系统部署阶段
本次项目运行在centos7系统上，用到的镜像包有
- mysql.tar :项目中用于部分数据持久化，如登录用户，已爬取的新闻
- kgtwitter.tar :项目中用于爬取twitter中敏感人物组织事件信息
- neo4j.tar :项目中用于持久化非结构化数据，如人物组织事件等信息以及他们之间的关系
- tomcat.tar :部署javaweb项目，解压war包，并运行
- kgqa.tar :问答系统
- scu_crwal.tar :爬虫模块
- kgextract.tar ：在某些新闻网站爬取敏感人物组织事件信息，另一个爬虫模块
可惜，这些镜像包都不是我写的
不过，可以查看这些镜像的制作步骤
docker history mysql:5.6.31
给大家看一下部署目录
| 路径 | 作用 |
|---|---|
| firstStep.sh和secondStep.sh | 为了方便部署，把部署需要做的准备工作都写在了这里 |
| iamges | 存放着各大镜像包 |
| kgabs | 存放着启动kgextreact容器的启动文件 |
| Kg_Auto | 存放了启动scu_crwal的启动文件 |
| kg_mysql | 存放了.sql格式的数据库文件 |
| KGMultiMedia | 存放了项目需要的多媒体文件 |
| kgqa | 存放着启动kgqa容器的启动文件 |
| neoData | 存放着neo4j的数据库文件 |
| webapps | tomcat的工作目录，放着已经打好的工程war包。 |
firstStep.sh
echo "installing docker"
sudo yum install -y yum-utils device-mapper-persistent-data lvm2
wget -O /etc/yum.repos.d/docker-ce.repo https://download.docker.com/linux/centos/docker-ce.repo
sudo sed -i 's+download.docker.com+mirrors.tuna.tsinghua.edu.cn/docker-ce+' /etc/yum.repos.d/docker-ce.repo
yum makecache fast
yum install -y docker-ce
service docker start
systemctl enable docker #设置开机自启动
echo "installation finished"
cur_path=`pwd` #保存当前目录
#导入镜像包，导出用docker save > 路径
echo "starting import"
docker load < ${cur_path}/images/mysql.tar
echo "kgmysql.tar imported!!!"
docker load < ${cur_path}/images/kgtwitter.tar
echo "kgtwitter.tar imported!!!"
docker load < ${cur_path}/images/neo4j.tar
echo "neo4j.tar imported!!!"
docker load < ${cur_path}/images/tomcat.tar
echo "tomcat.tar imported!!!"
docker load < ${cur_path}/images/kgqa.tar
echo "kgqa.tar imported!!!"
docker load < ${cur_path}/images/scu_crawl.tar
echo "scu_crwal.tar imported!!!"
docker load < ${cur_path}/images/kgextract.tar
echo "kgextract.tar imported!!!"
#这一小部分用于导入sql数据，具体方式是根据mysql的镜像包导入后重新生成一个新的镜像包
#以后开启的就是带有数据的镜像包
cd ./kg_mysql/
./start.sh
cd ${cur_path}/
#创建docker容器互通的网络，其实可以不用指定ip，用名字来互相访问
echo "starting create network"
docker network create --subnet 192.193.0.0/24 --gateway 192.193.0.1 kgnet
echo "creating network finished,gateway: 192.193.0.1 "
#启动容器
echo "starting container kgMysql..."
docker run -d -p 3306:3306 --name kgMysql --net kgnet --ip 192.193.0.2 --restart always kg_mysql
echo "kgMysql started successfully, starting kgNeo4j..."
#启动neo4j需要赋权限为最大，不然容易报错
chmod 777 -R ${cur_path}/neoData/
chmod 777 -R ${cur_path}/KGMultiMedia/
#挂载必带--priviledged=true,这里每次换新机器部署就出错，一定要带上，而且要带在-v命令后
docker run -d -p 7474:7474 -p 7687:7687 -v ${cur_path}/neoData/:/var/lib/neo4j/data/databases/ --net kgnet --ip 192.193.0.3 --privileged=true --name kgNeo4j --restart always neo4j:3.4
echo "kgNeo4j started successfully, please replace the password of neo4j by http://localhost:7474/browser/"
#为了不影响tomcat的运行，必须先改好密码，程序中已经写好了
secondStep.sh
cur_path=`pwd`
echo "starting kgabs container..."
docker run -itd -p 8081:8081 --net kgnet --ip 192.193.0.4 --name kgabs -v ${cur_path}/kgabs/:/home --privileged=true --restart always kgextract /bin/bash /home/start_crawl.sh
echo "kgabs started sucessfully, starting container kgtwitter..."
docker run -d -p 8083:8083 --name kgtwitter --net kgnet --ip 192.193.0.5 --restart always kg_twitter
echo "kgtwitter started sucessfully, starting container kgqa..."
docker run -d -v ${cur_path}/kgqa/package/:/code/package/ --privileged=true -p 8086:8086 --net kgnet --ip 192.193.0.6 --name kgqa --restart always kgqa:1.0
echo "kgqa started sucessfully, starting container tomcat..."
docker run -d -p 8080:8080 --name tomcat --net kgnet --ip 192.193.0.7 --restart always -v ${cur_path}/webapps:/usr/local/tomcat/webapps -v ${cur_path}/KGMultiMedia/:/KGMultiMedia/ --privileged=true tomcat:9.0.30-jdk8-adoptopenjdk-hotspot
docker run --net host -v ${cur_path}/Kg_Auto/:/home --privileged=true --restart always --name craw_install -itd scu_crawl /bin/bash /home/package/start_crawl.sh
echo "deployment finished !"
部署完成
系统测试阶段
docker ps查看各个容器运行状况
全部运行成功是最理想的状态，总有一些容器运行失败，这时我们需要查看日志，有以下几种方法
docker logs 容器id
此种方法有时候日志并不能及时打印
docker run -it [options] 容器id
此种方法在运行时查看日志，也就是有交互的运行容器，如果加入bash代表运行并进入容器命令行，此种方法能及时查看容器日志，但退出容器会关闭
docker attach --sig-proxy 容器id
此种方法能及时查看日志，退出时并不会关闭容器
可以根据日志内容进行debug
查看网络命令
docker network ls
查看某个网络的细节内容
docekr inspect kgnet
如下：
[
{
"Name": "kgnet",
"Id": "234cc76dc6e454e8c40f14b7548d16f9765849533bb6b32d16661d4ee58af87f",
"Created": "2020-08-01T22:23:19.630335841+08:00",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": {},
"Config": [
{
"Subnet": "192.193.0.0/24",
"Gateway": "192.193.0.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"007d607f7fce02f3c6e3f6fd928b32d9028f5168f0e7420a78931ba996f6c376": {
"Name": "kgMysql",
"EndpointID": "19ecee27dc8031c5e89ce9ecfc80d2fb0dae0c15ca1f6d2b451a846b04787f4f",
"MacAddress": "02:42:c0:c1:00:02",
"IPv4Address": "192.193.0.2/24",
"IPv6Address": ""
},
"3973e6d5c19f1bbc7596a34552f0b53117d088bb20e8f6eb78ac613c4a6124b7": {
"Name": "kgtwitter",
"EndpointID": "62df6633215b4cdfe933af9092c937e85a9d20b8779c4cea0eab967955f12495",
"MacAddress": "02:42:c0:c1:00:05",
"IPv4Address": "192.193.0.5/24",
"IPv6Address": ""
},
"699e38334cf94efd0bae04ff78c0caef29e66075938a72fbb26dacf0314fc530": {
"Name": "kgNeo4j",
"EndpointID": "2e16db834cc0ddb23812086b72c8095056f30ae5d8c3d7087a6a99b31fda16ef",
"MacAddress": "02:42:c0:c1:00:03",
"IPv4Address": "192.193.0.3/24",
"IPv6Address": ""
},
"7213cc6945745c872769b55516c53510c4646aee613f22e234816ccb73b1a00f": {
"Name": "tomcat",
"EndpointID": "331bafa45ef236d5d9763941ec83eaa752f09ba7c13c49f73eef2ff16c39b634",
"MacAddress": "02:42:c0:c1:00:07",
"IPv4Address": "192.193.0.7/24",
"IPv6Address": ""
},
"a83b3e3c5b080c865e90ce4509a3514f736818bc0e47ea837e7606d0d3a8e64a": {
"Name": "kgabs",
"EndpointID": "6135f3028354afa8b76c6dd694d3ad6057fb8dbb450f5d76e414ce1f7f0ad5ad",
"MacAddress": "02:42:c0:c1:00:04",
"IPv4Address": "192.193.0.4/24",
"IPv6Address": ""
},
"b1cf119c0100f80eedc6636372f85c2136500dc6a6593de6079a1bc26198e249": {
"Name": "kgqa",
"EndpointID": "3a99c594b557d7269388144c3597056ee92b08290bd384426cb7b2f25fc580ac",
"MacAddress": "02:42:c0:c1:00:06",
"IPv4Address": "192.193.0.6/24",
"IPv6Address": ""
}
},
"Options": {},
"Labels": {}
}
]
可以看到以上六个开启的容器都在这个“局域网”之下，他们是互联的，我甚至可以在tomcat容器内部去ping mysql容器
还可以查看容器的细节
docker inspect tomcat
[root@502Server kgOld]# docker inspect tomcat
[
{
"Id": "7213cc6945745c872769b55516c53510c4646aee613f22e234816ccb73b1a00f",
"Created": "2020-08-01T14:24:09.028871075Z",
"Path": "catalina.sh",
"Args": [
"run"
],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 23940,
"ExitCode": 0,
"Error": "",
"StartedAt": "2020-08-05T07:32:08.075131497Z",
"FinishedAt": "2020-08-05T07:32:01.518944135Z"
},
"Image": "sha256:a5a21a07a43992eece885cb577751ae00824b602801e8988fc1a43a0d9bce9d7",
"ResolvConfPath": "/var/lib/docker/containers/7213cc6945745c872769b55516c53510c4646aee613f22e234816ccb73b1a00f/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/7213cc6945745c872769b55516c53510c4646aee613f22e234816ccb73b1a00f/hostname",
"HostsPath": "/var/lib/docker/containers/7213cc6945745c872769b55516c53510c4646aee613f22e234816ccb73b1a00f/hosts",
"LogPath": "/var/lib/docker/containers/7213cc6945745c872769b55516c53510c4646aee613f22e234816ccb73b1a00f/7213cc6945745c872769b55516c53510c4646aee613f22e234816ccb73b1a00f-json.log",
"Name": "/tomcat",
"RestartCount": 0,
"Driver": "overlay2",
"Platform": "linux",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "",
"ExecIDs": null,
"HostConfig": {
"Binds": [
"/home/liang/Downloads/kglatest/kgOld/webapps:/usr/local/tomcat/webapps",
"/home/liang/Downloads/kglatest/kgOld/KGMultiMedia/:/KGMultiMedia/"
],
"ContainerIDFile": "",
"LogConfig": {
"Type": "json-file",
"Config": {}
},
"NetworkMode": "kgnet",
"PortBindings": {
"8080/tcp": [
{
"HostIp": "",
"HostPort": "8080"
}
]
},
"RestartPolicy": {
"Name": "always",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"CapAdd": null,
"CapDrop": null,
"Capabilities": null,
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": null,
"GroupAdd": null,
"IpcMode": "private",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": true,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": [
"label=disable"
],
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "runc",
"ConsoleSize": [
0,
0
],
"Isolation": "",
"CpuShares": 0,
"Memory": 0,
"NanoCpus": 0,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": [],
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": [],
"DeviceCgroupRules": null,
"DeviceRequests": null,
"KernelMemory": 0,
"KernelMemoryTCP": 0,
"MemoryReservation": 0,
"MemorySwap": 0,
"MemorySwappiness": null,
"OomKillDisable": false,
"PidsLimit": null,
"Ulimits": null,
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0,
"MaskedPaths": null,
"ReadonlyPaths": null
},
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/9ec7257bafedc0cb23f26b5ea3183cb91ed223c5f0ecbfd634b286b4b17ea81d-init/diff:/var/lib/docker/overlay2/1847e3c0bf8c2dcb9af53aa886b3d4e8a88b0e4438d1bc90c2ab3cfb5587917a/diff:/var/lib/docker/overlay2/47a0f8fa94d8e5f66fc33af6d925f2754bf473eef00871edc0290c3343625ade/diff:/var/lib/docker/overlay2/6c364eeca868c0b72b301dc3692f5fef40fd12ed880a71ce221bf140c20dcbc0/diff:/var/lib/docker/overlay2/e98f521e8683d5ede9533c1fc6b3cd8dfff5a7422fdf35a85ea9a0dda98eaf11/diff:/var/lib/docker/overlay2/46a58762146162d02ce99b7af032925b35a919efc52e93493399729aaf53a57e/diff:/var/lib/docker/overlay2/3a195192e60d8c089290475e15d69ca4a0706f76c6462d302a731ab49f3891b0/diff:/var/lib/docker/overlay2/77a6673707aa2dba34be2ce13167e35409eada0833c7c73fc54a9a2a324c7282/diff:/var/lib/docker/overlay2/33008aa4612d97102c6a018ac2cb41dbfc63d3bf6a753f33ec39fabd28771145/diff:/var/lib/docker/overlay2/e4204b2c473b43901f7d69f71bde1d57bbb8cd13ed5b747b06b1f6eae4a98ea4/diff",
"MergedDir": "/var/lib/docker/overlay2/9ec7257bafedc0cb23f26b5ea3183cb91ed223c5f0ecbfd634b286b4b17ea81d/merged",
"UpperDir": "/var/lib/docker/overlay2/9ec7257bafedc0cb23f26b5ea3183cb91ed223c5f0ecbfd634b286b4b17ea81d/diff",
"WorkDir": "/var/lib/docker/overlay2/9ec7257bafedc0cb23f26b5ea3183cb91ed223c5f0ecbfd634b286b4b17ea81d/work"
},
"Name": "overlay2"
},
"Mounts": [
{
"Type": "bind",
"Source": "/home/liang/Downloads/kglatest/kgOld/KGMultiMedia",
"Destination": "/KGMultiMedia",
"Mode": "",
"RW": true,
"Propagation": "rprivate"
},
{
"Type": "bind",
"Source": "/home/liang/Downloads/kglatest/kgOld/webapps",
"Destination": "/usr/local/tomcat/webapps",
"Mode": "",
"RW": true,
"Propagation": "rprivate"
}
],
"Config": {
"Hostname": "7213cc694574",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"8080/tcp": {}
},
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/tomcat/bin:/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"LANG=en_US.UTF-8",
"LANGUAGE=en_US:en",
"LC_ALL=en_US.UTF-8",
"JAVA_VERSION=jdk8u232-b09",
"JAVA_HOME=/opt/java/openjdk",
"CATALINA_HOME=/usr/local/tomcat",
"TOMCAT_NATIVE_LIBDIR=/usr/local/tomcat/native-jni-lib",
"LD_LIBRARY_PATH=/usr/local/tomcat/native-jni-lib",
"GPG_KEYS=05AB33110949707C93A279E3D3EFE6B686867BA6 07E48665A34DCAFAE522E5E6266191C37C037D42 47309207D818FFD8DCD3F83F1931D684307A10A5 541FBE7D8F78B25E055DDEE13C370389288584E7 61B832AC2F1C5A90F0F9B00A1C506407564C17A3 79F7026C690BAA50B92CD8B66A3AD3F4F22C4FED 9BA44C2621385CB966EBA586F72C284D731FABEE A27677289986DB50844682F8ACB77FC2E86E29AC A9C5DF4D22E99998D9875A5110C01C5A2F6059E7 DCFD35E0BF8CA7344752DE8B6FB21E8933C60243 F3A04C595DB5B6A5F1ECA43E3B7BBB100D811BBE F7DA48BB64BCB84ECBA7EE6935CD23C10D498E23",
"TOMCAT_MAJOR=9",
"TOMCAT_VERSION=9.0.30",
"TOMCAT_SHA512=40bf428a534816317e5e8732b679e6d73f6ed977aab4ac5e2b9eaf1eb087eaa68a18b0fd3da8c39ca4b96f5b6fd919dec84517abba0d5e797285ade934a92990"
],
"Cmd": [
"catalina.sh",
"run"
],
"Image": "tomcat:9.0.30-jdk8-adoptopenjdk-hotspot",
"Volumes": null,
"WorkingDir": "/usr/local/tomcat",
"Entrypoint": null,
"OnBuild": null,
"Labels": {}
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "53d2677a4ab57f683a502f744a3a8906031ec4915b405b307319d32ba26dd471",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {
"8080/tcp": [
{
"HostIp": "0.0.0.0",
"HostPort": "8080"
}
]
},
"SandboxKey": "/var/run/docker/netns/53d2677a4ab5",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {
"kgnet": {
"IPAMConfig": {
"IPv4Address": "192.193.0.7"
},
"Links": null,
"Aliases": [
"7213cc694574"
],
"NetworkID": "234cc76dc6e454e8c40f14b7548d16f9765849533bb6b32d16661d4ee58af87f",
"EndpointID": "331bafa45ef236d5d9763941ec83eaa752f09ba7c13c49f73eef2ff16c39b634",
"Gateway": "192.193.0.1",
"IPAddress": "192.193.0.7",
"IPPrefixLen": 24,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:c0:c1:00:07",
"DriverOpts": null
}
}
}
}
]
可以看到这个容器的各种配置。 后面会出关于dockerfile的制作操作，因为又有项目要在linux上部署了。
