Linux系统健康巡检 systemd 服务与定时任务系统健康巡检 systemd 服务与定时任务。每日自动检查负载、内

系统健康巡检 systemd 服务与定时任务

1. 文档信息

项目	内容
任务名称	系统健康巡检服务 health-check
适用系统	使用 systemd 的 Linux 发行版（Rocky/Alma/CentOS、Ubuntu/Debian 等）
执行权限	root 或具备 sudo 权限的用户
产出	巡检脚本 + systemd service + systemd timer
实际作用	每日自动检查负载、内存、磁盘、inode、大文件、已删除未释放文件、关键服务、失败单元，输出 WARN/CRIT 报告到 journald

2. 任务目标与验收标准

2.1 目标

实现一个每日自动运行的系统健康巡检服务，满足：

使用 systemd service 执行巡检脚本，Type=oneshot。
使用 systemd timer 每天定时触发，支持关机错过补跑。
巡检结果输出到 journald，用 journalctl 统一查看。
覆盖磁盘满排查链路：分区使用率、inode、du 下钻、大文件、/var/log 大日志、lsof +L1。
脚本退出码有语义：0 正常，1 有 WARN，2 有 CRIT。
service 以 root 运行但收紧能力集，保持只读诊断。

2.2 验收标准

完成后应满足：

systemctl list-timers | grep health-check 能看到下次触发时间。
sudo systemctl start health-check.service 能手动执行成功。
journalctl -u health-check.service 能看到完整巡检报告。
制造 600MB 大文件后，报告“Top Files”段能显示该文件。
制造“已删除但被进程持有”的文件后，报告“Deleted Files”段能显示。
重启后 timer 仍生效：systemctl is-enabled health-check.timer 输出 enabled。
systemctl show health-check.service -p ProtectSystem -p CapabilityBoundingSet 显示沙箱配置生效。

3. 架构与文件清单

text

/usr/local/bin/health-check.sh             巡检脚本
/etc/systemd/system/health-check.service   oneshot 服务单元
/etc/systemd/system/health-check.timer     每日定时器单元

执行关系：

text

health-check.timer  --每天 08:00-->  health-check.service  -->  health-check.sh
                                                              |
                                                              v
                                                         journald 日志

4. 前置检查

确认系统使用 systemd：

bash

ps -p 1 -o comm=

应输出 systemd。

确认 root 权限：

bash

sudo -v

确认关键命令存在：

bash

command -v df du find awk sort head systemctl journalctl lsof bc numfmt

如果有缺失，进入下一步安装依赖。

5. 步骤一：安装依赖

Debian / Ubuntu

bash

sudo apt update
sudo apt install -y lsof bc coreutils findutils procps util-linux

RHEL / Rocky / Alma / CentOS

bash

sudo dnf install -y lsof bc coreutils findutils procps-ng util-linux

说明：

lsof：检测已删除未释放文件
bc：浮点数比较
coreutils：numfmt、df、du 等
findutils：find
procps / procps-ng：ps、top 等
util-linux：常用系统工具

6. 步骤二：创建巡检脚本

写入 /usr/local/bin/health-check.sh：

bash

sudo tee /usr/local/bin/health-check.sh >/dev/null <<'EOF'
#!/usr/bin/env bash
set -uo pipefail

# ===== 阈值 =====
DISK_WARN=80
DISK_CRIT=90
INODE_WARN=80
INODE_CRIT=90
MEM_WARN=80
MEM_CRIT=90
LOAD_WARN=4.0
LOAD_CRIT=8.0

# ===== 磁盘满排查参数 =====
BIGFILE_SIZE_MB=500
BIGFILE_TOPN=5
DU_TOPN=8
LOG_BIGFILE_SIZE_MB=50
DELETED_TOPN=5

# 大文件扫描目录，避免全盘 find
BIGFILE_SCAN_DIRS="/var /tmp /home /data /opt"

# 排除的伪文件系统类型
EXCLUDE_FSTYPES="tmpfs|devtmpfs|proc|sysfs|cgroup|cgroup2|overlay|squashfs|nsfs|ramfs"

warnings=0
criticals=0
report=""

add_line() { report+="$1"$'\n'; }

mark() {
    local level="$1" msg="$2"
    if [[ "$level" == "CRIT" ]]; then
        criticals=$((criticals + 1))
    elif [[ "$level" == "WARN" ]]; then
        warnings=$((warnings + 1))
    fi
    add_line "[$level] $msg"
}

human() {
    numfmt --to=iec "$1" 2>/dev/null || echo "$1"
}

add_line "========== Health Check $(date '+%Y-%m-%d %H:%M:%S') =========="
add_line "Hostname: $(hostname)"
add_line "Uptime:   $(uptime -p 2>/dev/null || uptime)"
add_line ""

# ---------- 1. 负载 ----------
load1=$(awk '{print $1}' /proc/loadavg)
if (( $(echo "$load1 >= $LOAD_CRIT" | bc -l) )); then
    mark CRIT "Load(1m): ${load1} (threshold ${LOAD_CRIT})"
elif (( $(echo "$load1 >= $LOAD_WARN" | bc -l) )); then
    mark WARN "Load(1m): ${load1} (threshold ${LOAD_WARN})"
else
    add_line "[OK]   Load(1m): ${load1}"
fi

# ---------- 2. 内存 ----------
mem_total=$(awk '/MemTotal/ {print $2}' /proc/meminfo)
mem_avail=$(awk '/MemAvailable/ {print $2}' /proc/meminfo)
mem_used_pct=$(awk -v t="$mem_total" -v a="$mem_avail" 'BEGIN {printf "%.1f", (t-a)/t*100}')
if (( $(echo "$mem_used_pct >= $MEM_CRIT" | bc -l) )); then
    mark CRIT "Memory used: ${mem_used_pct}% (threshold ${MEM_CRIT}%)"
elif (( $(echo "$mem_used_pct >= $MEM_WARN" | bc -l) )); then
    mark WARN "Memory used: ${mem_used_pct}% (threshold ${MEM_WARN}%)"
else
    add_line "[OK]   Memory used: ${mem_used_pct}%"
fi

# ---------- 3. 所有本地分区使用率 ----------
add_line ""
add_line "--- Disk Usage (per filesystem) ---"

full_mounts=()

while read -r fs type size used avail pcent mount; do
    [[ -z "${pcent:-}" ]] && continue
    if (( pcent >= DISK_CRIT )); then
        mark CRIT "Disk ${mount} (${type}) used: ${pcent}% (threshold ${DISK_CRIT}%)"
        full_mounts+=("$mount")
    elif (( pcent >= DISK_WARN )); then
        mark WARN "Disk ${mount} (${type}) used: ${pcent}% (threshold ${DISK_WARN}%)"
        full_mounts+=("$mount")
    else
        add_line "[OK]   Disk ${mount} (${type}) used: ${pcent}%"
    fi
done < <(df -PT 2>/dev/null | awk -v re="$EXCLUDE_FSTYPES" '
    NR>1 && $2 !~ re {
        gsub(/%/, "", $6);
        print $1, $2, $3, $4, $5, $6, $7
    }')

# ---------- 4. inode 使用率 ----------
add_line ""
add_line "--- Inode Usage (per filesystem) ---"

while read -r fs type inodes iused ifree ipcent mount; do
    [[ -z "${ipcent:-}" ]] && continue
    if (( ipcent >= INODE_CRIT )); then
        mark CRIT "Inode ${mount} (${type}) used: ${ipcent}% (threshold ${INODE_CRIT}%)"
    elif (( ipcent >= INODE_WARN )); then
        mark WARN "Inode ${mount} (${type}) used: ${ipcent}% (threshold ${INODE_WARN}%)"
    else
        add_line "[OK]   Inode ${mount} (${type}) used: ${ipcent}%"
    fi
done < <(df -iPT 2>/dev/null | awk -v re="$EXCLUDE_FSTYPES" '
    NR>1 && $2 !~ re {
        gsub(/%/, "", $6);
        print $1, $2, $3, $4, $5, $6, $7
    }')

# ---------- 5. 超阈值分区 du 下钻 ----------
if (( ${#full_mounts[@]} > 0 )); then
    add_line ""
    add_line "--- Top Directories on Full Filesystems ---"
    for mnt in "${full_mounts[@]}"; do
        add_line "  [$mnt]"
        while read -r kbytes path; do
            add_line "       $(human $((kbytes * 1024)))  $path"
        done < <(du -x --max-depth=1 "$mnt" 2>/dev/null | sort -rn | head -n "$DU_TOPN")
    done
fi

# ---------- 6. 大文件 Top N ----------
add_line ""
add_line "--- Top ${BIGFILE_TOPN} Files > ${BIGFILE_SIZE_MB}MB (scanned dirs) ---"

bigfile_found=0
while IFS=$'\t' read -r size path; do
    [[ -z "${path:-}" ]] && continue
    bigfile_found=1
    add_line "       $(human "$size")  $path"
done < <(
    for dir in $BIGFILE_SCAN_DIRS; do
        [[ -d "$dir" ]] || continue
        find "$dir" -xdev -type f -size +${BIGFILE_SIZE_MB}M \
            -printf '%s\t%p\n' 2>/dev/null
    done | sort -rn | head -n "$BIGFILE_TOPN"
)
(( bigfile_found == 0 )) && add_line "       (none)"

# ---------- 7. /var/log 大日志 Top ----------
add_line ""
add_line "--- Top 5 Log Files > ${LOG_BIGFILE_SIZE_MB}MB in /var/log ---"

log_found=0
while IFS=$'\t' read -r size path; do
    [[ -z "${path:-}" ]] && continue
    log_found=1
    add_line "       $(human "$size")  $path"
done < <(
    find /var/log -xdev -type f -size +${LOG_BIGFILE_SIZE_MB}M \
        -printf '%s\t%p\n' 2>/dev/null | sort -rn | head -n 5
)
(( log_found == 0 )) && add_line "       (none)"

# ---------- 8. 已删除但未释放的文件 ----------
add_line ""
add_line "--- Deleted Files Still Held Open (Top ${DELETED_TOPN}) ---"

deleted_found=0
while IFS=$'\t' read -r size comm pid name; do
    [[ -z "${name:-}" ]] && continue
    deleted_found=1
    add_line "       $(human "$size")  PID=$pid  $comm  $name"
done < <(
    lsof +L1 2>/dev/null \
        | awk 'NR>1 && $7 ~ /^[0-9]+$/ {
            name=$9;
            for (i=10; i<=NF; i++) name=name" "$i;
            print $7"\t"$1"\t"$2"\t"name
        }' \
        | sort -rn | head -n "$DELETED_TOPN"
)
(( deleted_found == 0 )) && add_line "       (none)"

# ---------- 9. 关键服务 ----------
add_line ""
add_line "--- Services ---"

SERVICES_TO_CHECK="sshd chronyd"
for svc in $SERVICES_TO_CHECK; do
    if ! systemctl list-unit-files --type=service 2>/dev/null \
            | awk '{print $1}' | grep -qx "${svc}.service"; then
        add_line "[SKIP] Service ${svc} not installed"
        continue
    fi
    state=$(systemctl is-active "$svc" 2>/dev/null || true)
    if [[ "$state" == "active" ]]; then
        add_line "[OK]   Service ${svc}: active"
    else
        mark CRIT "Service ${svc}: ${state:-unknown}"
    fi
done

# ---------- 10. 失败单元 ----------
add_line ""
add_line "--- Failed systemd Units ---"

failed_units=$(systemctl list-units --state=failed --no-legend --no-pager 2>/dev/null | awk '{print $1}')
if [[ -n "$failed_units" ]]; then
    mark CRIT "Failed units detected:"
    while IFS= read -r unit; do
        [[ -n "$unit" ]] && add_line "       - $unit"
    done <<< "$failed_units"
else
    add_line "[OK]   No failed systemd units"
fi

# ---------- 汇总 ----------
add_line ""
add_line "========== Summary: WARN=${warnings}, CRIT=${criticals} =========="

printf '%s\n' "$report"

if (( criticals > 0 )); then
    exit 2
elif (( warnings > 0 )); then
    exit 1
else
    exit 0
fi
EOF

设置权限：

bash

sudo chmod 755 /usr/local/bin/health-check.sh

手动执行验证：

bash

sudo /usr/local/bin/health-check.sh
echo "exit=$?"

7. 步骤三：创建 service 单元

写入 /etc/systemd/system/health-check.service：

bash

sudo tee /etc/systemd/system/health-check.service >/dev/null <<'EOF'
[Unit]
Description=System Health Check & Report
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/health-check.sh

# 巡检需要跨进程读取 /proc 和 lsof 信息，用 root 但收紧能力集
User=root
Group=root
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=read-only
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
ProtectClock=yes
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
CapabilityBoundingSet=CAP_SYS_PTRACE CAP_DAC_READ_SEARCH
EOF

说明：

配置	作用
`Type=oneshot`	脚本执行完退出，适合巡检任务
`User=root`	`lsof +L1` 跨进程读取需要 root 或 `CAP_SYS_PTRACE`
`CapabilityBoundingSet=CAP_SYS_PTRACE CAP_DAC_READ_SEARCH`	只保留读取进程和文件所需最小能力
`ProtectSystem=strict`	整个文件系统只读，巡检只读不写
`ProtectHome=read-only`	`/home` 只读可见，用于扫描大文件
`NoNewPrivileges=yes`	禁止通过 SUID 等方式提权

注意：这里没有启用 PrivateTmp=yes，因为脚本需要扫描宿主机 /tmp 下的大文件。若启用私有 /tmp，服务看到的是独立临时目录，会漏掉宿主 /tmp。

8. 步骤四：创建 timer 单元

写入 /etc/systemd/system/health-check.timer：

bash

sudo tee /etc/systemd/system/health-check.timer >/dev/null <<'EOF'
[Unit]
Description=Daily Health Check Timer

[Timer]
OnCalendar=*-*-* 08:00:00
Persistent=true
RandomizedDelaySec=120
AccuracySec=1min

[Install]
WantedBy=timers.target
EOF

说明：

OnCalendar=*-*-* 08:00:00：每天 08:00 触发，可按需修改。
Persistent=true：关机错过时间后，开机补跑一次。
RandomizedDelaySec=120：随机延迟最多 120 秒，避免多机同时触发。
AccuracySec=1min：允许 1 分钟精度，降低系统唤醒频率。

9. 步骤五：部署与启用

bash

sudo systemctl daemon-reload
sudo systemctl enable --now health-check.timer

查看 timer：

bash

systemctl list-timers --all | grep health-check
systemctl status health-check.timer

手动触发一次 service：

bash

sudo systemctl start health-check.service

查看报告：

bash

journalctl -u health-check.service --no-pager --since today

查看 service 状态和退出码：

bash

systemctl status health-check.service

10. 步骤六：模拟测试

10.1 测试大文件告警

bash

sudo dd if=/dev/zero of=/var/tmp/test-big.img bs=1M count=600 status=none
sudo systemctl start health-check.service
journalctl -u health-check.service --no-pager --since "1 min ago" | grep -A6 "Top 5 Files"
sudo rm -f /var/tmp/test-big.img

应看到类似：

text

600M  /var/tmp/test-big.img

10.2 测试已删除未释放文件

bash

sudo dd if=/dev/zero of=/var/tmp/deleted-test.img bs=1M count=200 status=none
sudo bash -c 'exec 3>/var/tmp/deleted-test.img; rm -f /var/tmp/deleted-test.img; sleep 300' &
sudo systemctl start health-check.service
journalctl -u health-check.service --no-pager --since "1 min ago" | grep -A6 "Deleted Files"
sudo pkill -f "sleep 300"

应看到 deleted-test.img 出现在“Deleted Files Still Held Open”段。

10.3 测试阈值告警

临时修改脚本：

bash

sudo sed -i 's/^DISK_WARN=80/DISK_WARN=1/' /usr/local/bin/health-check.sh
sudo systemctl start health-check.service
systemctl status health-check.service

应看到 [WARN] Disk / used: ...，且 service 状态为 failed (Result: exit-code)，退出码为 1。

恢复：

bash

sudo sed -i 's/^DISK_WARN=1/DISK_WARN=80/' /usr/local/bin/health-check.sh

11. 日常运维命令

bash

# 查看下次触发时间
systemctl list-timers --all | grep health-check

# 查看 timer 状态
systemctl status health-check.timer

# 手动执行巡检
sudo systemctl start health-check.service

# 查看巡检报告
journalctl -u health-check.service --no-pager --since today

# 实时跟踪巡检日志
journalctl -u health-check.service -f

# 查看 service 沙箱配置
systemctl show health-check.service -p ProtectSystem -p ProtectHome -p CapabilityBoundingSet -p NoNewPrivileges

# 查看是否开机启用
systemctl is-enabled health-check.timer

12. 常见问题排查

12.1 `Permission denied`

bash

sudo chmod 755 /usr/local/bin/health-check.sh

12.2 `lsof: command not found`

安装 lsof：

bash

sudo apt install -y lsof        # Debian/Ubuntu
sudo dnf install -y lsof        # RHEL 系

12.3 `bc: command not found`

安装 bc：

bash

sudo apt install -y bc
sudo dnf install -y bc

12.4 timer 不触发

bash

sudo systemctl daemon-reload
sudo systemctl enable --now health-check.timer
systemctl list-timers --all | grep health-check

12.5 报告里看不到 `/tmp` 大文件

确认 service 中没有启用 PrivateTmp=yes。本方案有意不启用，以便扫描宿主机 /tmp。

12.6 `ProtectHome=read-only` 不支持

较老 systemd 可能不支持，改成：

ini

ProtectHome=no

然后：

bash

sudo systemctl daemon-reload

12.7 service 显示 failed 但报告正常

检查退出码：

bash

systemctl show health-check.service -p ExecMainStatus

ExecMainStatus=1 表示有 WARN，2 表示有 CRIT，0 表示全部正常。这是预期行为，便于监控接入。

13. 回滚与清理

bash

sudo systemctl stop health-check.timer
sudo systemctl disable health-check.timer
sudo systemctl stop health-check.service

sudo rm -f /etc/systemd/system/health-check.timer
sudo rm -f /etc/systemd/system/health-check.service
sudo rm -f /usr/local/bin/health-check.sh

sudo systemctl daemon-reload
sudo systemctl reset-failed

14. 验收记录模板

验收项	命令	结果
脚本可执行	`sudo /usr/local/bin/health-check.sh; echo $?`
service 手动执行	`sudo systemctl start health-check.service`
报告可查看	`journalctl -u health-check.service --since today`
timer 已启用	`systemctl is-enabled health-check.timer`
下次触发时间	`systemctl list-timers	grep health-check`
大文件告警	模拟 600MB 文件
删除未释放告警	`lsof +L1` 模拟
沙箱配置	`systemctl show health-check.service -p ProtectSystem -p CapabilityBoundingSet`

15. 交付说明

写一个 systemd 服务 + 定时任务管理自建小服务。

一个可每日自动运行的系统健康巡检脚本。
一个带安全加固的 oneshot systemd service。
一个支持关机补跑的 systemd timer。
一套覆盖磁盘满排查链路的巡检逻辑。
可复现的模拟测试和验收记录。

Linux系统健康巡检 systemd 服务与定时任务

系统健康巡检 systemd 服务与定时任务

1. 文档信息

2. 任务目标与验收标准

2.1 目标

2.2 验收标准

3. 架构与文件清单

4. 前置检查

5. 步骤一：安装依赖

Debian / Ubuntu

RHEL / Rocky / Alma / CentOS

6. 步骤二：创建巡检脚本

7. 步骤三：创建 service 单元

8. 步骤四：创建 timer 单元

9. 步骤五：部署与启用

10. 步骤六：模拟测试

10.1 测试大文件告警

10.2 测试已删除未释放文件

10.3 测试阈值告警

11. 日常运维命令

12. 常见问题排查

12.1 Permission denied

12.2 lsof: command not found

12.3 bc: command not found

12.4 timer 不触发

12.5 报告里看不到 /tmp 大文件

12.6 ProtectHome=read-only 不支持

12.7 service 显示 failed 但报告正常

13. 回滚与清理

14. 验收记录模板

15. 交付说明

12.1 `Permission denied`

12.2 `lsof: command not found`

12.3 `bc: command not found`

12.5 报告里看不到 `/tmp` 大文件

12.6 `ProtectHome=read-only` 不支持