💖💖作者：计算机编程小央姐 💙💙个人简介：曾长期从事计算机专业培训教学，本人也热爱上课教学，语言擅长Java、微信小程序、Python、Golang、安卓Android等，开发项目包括大数据、深度学习、网站、小程序、安卓、算法。平常会做一些项目定制化开发、代码讲解、答辩教学、文档编写、也懂一些降重方面的技巧。平常喜欢分享一些自己开发中遇到的问题的解决办法，也喜欢交流技术，大家有技术代码这一块的问题可以问我！ 💛💛想说的话：感谢大家的关注与支持！ 💜💜
💕💕文末获取源码
@TOC
中国水污染监测数据可视化分析系统-系统功能介绍
基于大数据的中国水污染监测数据可视化分析系统是一个集成Hadoop分布式存储、Spark大数据处理、Python数据分析和Vue前端展示的综合性平台。该系统通过HDFS分布式文件系统存储海量水质监测数据，运用Spark SQL和Spark Core进行大规模数据清洗、转换和分析处理，结合Pandas和NumPy进行深度数据挖掘，最终通过Vue+ElementUI+Echarts技术栈实现直观的数据可视化展示。系统涵盖全国各省份水质时空分布分析、核心污染指标深度剖析、污染成因探索和综合评价四大核心模块，能够处理包含化学需氧量、氨氮、总磷、总氮、重金属等多维度水质指标的大规模监测数据，通过K-Means聚类分析识别城市污染模式，利用主成分分析挖掘影响水质的关键因子，生成水质指数地理热力图和污染等级分布统计，为水环境管理和污染防治提供科学的数据支撑和决策参考。
中国水污染监测数据可视化分析系统-系统技术介绍
大数据框架：Hadoop+Spark（本次没用Hive，支持定制） 开发语言：Python+Java（两个版本都支持） 后端框架：Django+Spring Boot(Spring+SpringMVC+Mybatis)（两个版本都支持） 前端：Vue+ElementUI+Echarts+HTML+CSS+JavaScript+jQuery 详细技术点：Hadoop、HDFS、Spark、Spark SQL、Pandas、NumPy 数据库：MySQL。
中国水污染监测数据可视化分析系统-系统背景意义
随着我国工业化和城镇化进程的快速发展，水环境污染问题日益突出，水质监测数据呈现出海量、多维、复杂的特点，传统的数据处理方式已无法满足大规模水质数据分析的需求。全国范围内分布着数以万计的水质监测站点，每天产生包含化学需氧量、氨氮、总磷、总氮、重金属等数十项指标的监测数据，这些数据蕴含着丰富的水环境变化规律和污染成因信息。然而，由于数据量庞大、处理复杂，现有的水质数据分析多停留在单点统计或小范围分析层面，缺乏从时空维度进行大规模关联分析的技术手段。大数据技术的兴起为解决这一问题提供了新的思路，Hadoop生态系统能够有效存储和管理海量水质监测数据，Spark框架具备强大的分布式数据处理能力，结合现代可视化技术，能够实现对全国水污染状况的全面监测和深度分析。
从技术层面来看，本课题将大数据处理技术应用于环境监测领域，探索了Hadoop+Spark架构在水质数据分析中的实际应用场景，通过整合分布式存储、并行计算和数据挖掘技术，为大规模环境数据处理提供了一种可行的技术方案。从实际应用角度分析，系统能够帮助更好地掌握全国水质状况，通过时空分布分析识别污染热点区域，通过污染指标关联分析找出关键污染因子，为制定针对性治理措施提供数据支撑。对于学术研究而言，本课题将传统的环境科学与现代信息技术相结合，为水环境研究提供了新的数据分析视角和方法。从社会价值来说，虽然作为毕业设计在规模和深度上存在一定局限，但系统展示的数据分析思路和技术架构，对于推动环境信息化建设具有一定的参考价值，也体现了新一代计算机专业学生运用所学知识服务社会发展的实践能力。
中国水污染监测数据可视化分析系统-系统演示视频
中国水污染监测数据可视化分析系统-系统演示图片
中国水污染监测数据可视化分析系统-系统部分代码
from pyspark.sql import SparkSession
from pyspark.sql.functions import *
from pyspark.ml.clustering import KMeans
from pyspark.ml.feature import VectorAssembler, StandardScaler
from pyspark.ml.stat import Correlation
import pandas as pd
import numpy as np
spark = SparkSession.builder.appName("WaterPollutionAnalysis").config("spark.sql.adaptive.enabled", "true").config("spark.sql.adaptive.coalescePartitions.enabled", "true").getOrCreate()
def water_quality_spatial_analysis():
df = spark.read.option("header", "true").option("inferSchema", "true").csv("hdfs://localhost:9000/water_data/china_water_quality.csv")
df.createOrReplaceTempView("water_quality")
province_quality = spark.sql("""
SELECT Province,
AVG(Water_Quality_Index) as avg_quality_index,
COUNT(*) as monitoring_points,
STDDEV(Water_Quality_Index) as quality_stability,
MAX(Water_Quality_Index) as max_pollution,
MIN(Water_Quality_Index) as min_pollution
FROM water_quality
GROUP BY Province
ORDER BY avg_quality_index DESC
""")
pollution_level_dist = spark.sql("""
SELECT Province, Pollution_Level, COUNT(*) as count,
ROUND(COUNT(*) * 100.0 / SUM(COUNT(*)) OVER (PARTITION BY Province), 2) as percentage
FROM water_quality
GROUP BY Province, Pollution_Level
ORDER BY Province, count DESC
""")
monthly_trend = spark.sql("""
SELECT MONTH(Date) as month,
AVG(Water_Quality_Index) as avg_quality,
COUNT(*) as sample_count,
AVG(COD_mg_L) as avg_cod,
AVG(Ammonia_N_mg_L) as avg_ammonia
FROM water_quality
GROUP BY MONTH(Date)
ORDER BY month
""")
hotspot_analysis = spark.sql("""
SELECT Province, City, Longitude, Latitude,
AVG(Water_Quality_Index) as pollution_intensity,
COUNT(CASE WHEN Water_Quality_Index > 80 THEN 1 END) as high_pollution_days,
AVG(COD_mg_L + Ammonia_N_mg_L + Total_Phosphorus_mg_L) as composite_pollution
FROM water_quality
GROUP BY Province, City, Longitude, Latitude
HAVING pollution_intensity > 60
ORDER BY pollution_intensity DESC
""")
result_dict = {
'province_ranking': province_quality.collect(),
'pollution_distribution': pollution_level_dist.collect(),
'monthly_trends': monthly_trend.collect(),
'pollution_hotspots': hotspot_analysis.collect()
}
province_quality.coalesce(1).write.mode("overwrite").option("header", "true").csv("hdfs://localhost:9000/results/province_analysis")
return result_dict
def pollution_indicator_correlation_analysis():
df = spark.read.option("header", "true").option("inferSchema", "true").csv("hdfs://localhost:9000/water_data/china_water_quality.csv")
df.createOrReplaceTempView("water_quality")
pollutant_stats = spark.sql("""
SELECT
AVG(COD_mg_L) as avg_cod,
AVG(Ammonia_N_mg_L) as avg_ammonia,
AVG(Total_Phosphorus_mg_L) as avg_phosphorus,
AVG(Total_Nitrogen_mg_L) as avg_nitrogen,
AVG(Heavy_Metals_Pb_ug_L) as avg_lead,
AVG(Heavy_Metals_Cd_ug_L) as avg_cadmium,
AVG(Heavy_Metals_Hg_ug_L) as avg_mercury,
STDDEV(COD_mg_L) as std_cod,
STDDEV(Ammonia_N_mg_L) as std_ammonia,
PERCENTILE_APPROX(COD_mg_L, 0.95) as cod_95th,
PERCENTILE_APPROX(Ammonia_N_mg_L, 0.95) as ammonia_95th
FROM water_quality
""")
heavy_metal_risk = spark.sql("""
SELECT Province,
AVG(Heavy_Metals_Pb_ug_L) as avg_lead,
MAX(Heavy_Metals_Pb_ug_L) as max_lead,
AVG(Heavy_Metals_Cd_ug_L) as avg_cadmium,
MAX(Heavy_Metals_Cd_ug_L) as max_cadmium,
AVG(Heavy_Metals_Hg_ug_L) as avg_mercury,
MAX(Heavy_Metals_Hg_ug_L) as max_mercury,
COUNT(CASE WHEN Heavy_Metals_Pb_ug_L > 10 THEN 1 END) as lead_exceed_count,
COUNT(CASE WHEN Heavy_Metals_Cd_ug_L > 5 THEN 1 END) as cadmium_exceed_count,
COUNT(CASE WHEN Heavy_Metals_Hg_ug_L > 1 THEN 1 END) as mercury_exceed_count
FROM water_quality
GROUP BY Province
ORDER BY avg_lead DESC
""")
eutrophication_risk = spark.sql("""
SELECT Province,
AVG(Total_Phosphorus_mg_L) as avg_tp,
AVG(Total_Nitrogen_mg_L) as avg_tn,
AVG(Total_Phosphorus_mg_L + Total_Nitrogen_mg_L) as nutrient_load,
COUNT(CASE WHEN Total_Phosphorus_mg_L > 0.2 AND Total_Nitrogen_mg_L > 2.0 THEN 1 END) as high_risk_points,
ROUND(AVG(Total_Phosphorus_mg_L + Total_Nitrogen_mg_L) / COUNT(*), 4) as risk_density
FROM water_quality
GROUP BY Province
HAVING nutrient_load > 1.0
ORDER BY nutrient_load DESC
""")
numeric_cols = ['COD_mg_L', 'Ammonia_N_mg_L', 'Total_Phosphorus_mg_L', 'Total_Nitrogen_mg_L', 'Water_Quality_Index']
assembler = VectorAssembler(inputCols=numeric_cols, outputCol="features")
correlation_data = assembler.transform(df.select(*numeric_cols)).select("features")
correlation_matrix = Correlation.corr(correlation_data, "features", "pearson").head()[0]
correlation_array = correlation_matrix.toArray()
correlation_results = []
for i in range(len(numeric_cols)):
for j in range(i+1, len(numeric_cols)):
correlation_results.append({
'indicator1': numeric_cols[i],
'indicator2': numeric_cols[j],
'correlation': float(correlation_array[i][j])
})
pollutant_stats.coalesce(1).write.mode("overwrite").option("header", "true").csv("hdfs://localhost:9000/results/pollutant_statistics")
heavy_metal_risk.coalesce(1).write.mode("overwrite").option("header", "true").csv("hdfs://localhost:9000/results/heavy_metal_risk")
return {
'pollutant_statistics': pollutant_stats.collect(),
'heavy_metal_risk': heavy_metal_risk.collect(),
'eutrophication_risk': eutrophication_risk.collect(),
'correlation_analysis': correlation_results
}
def city_pollution_clustering_analysis():
df = spark.read.option("header", "true").option("inferSchema", "true").csv("hdfs://localhost:9000/water_data/china_water_quality.csv")
df.createOrReplaceTempView("water_quality")
city_features = spark.sql("""
SELECT City,
AVG(COD_mg_L) as avg_cod,
AVG(Ammonia_N_mg_L) as avg_ammonia,
AVG(Total_Phosphorus_mg_L) as avg_phosphorus,
AVG(Total_Nitrogen_mg_L) as avg_nitrogen,
AVG(Heavy_Metals_Pb_ug_L) as avg_lead,
AVG(pH) as avg_ph,
AVG(Water_Quality_Index) as avg_quality_index,
STDDEV(Water_Quality_Index) as quality_volatility,
COUNT(*) as monitoring_frequency
FROM water_quality
GROUP BY City
HAVING COUNT(*) >= 10
""")
feature_cols = ['avg_cod', 'avg_ammonia', 'avg_phosphorus', 'avg_nitrogen', 'avg_lead', 'avg_ph', 'avg_quality_index', 'quality_volatility']
assembler = VectorAssembler(inputCols=feature_cols, outputCol="features")
scaler = StandardScaler(inputCol="features", outputCol="scaled_features", withStd=True, withMean=True)
city_vector = assembler.transform(city_features)
scaler_model = scaler.fit(city_vector)
city_scaled = scaler_model.transform(city_vector)
kmeans = KMeans(k=5, seed=42, featuresCol="scaled_features", predictionCol="cluster")
kmeans_model = kmeans.fit(city_scaled)
city_clustered = kmeans_model.transform(city_scaled)
cluster_analysis = city_clustered.groupBy("cluster").agg(
count("City").alias("city_count"),
avg("avg_cod").alias("cluster_avg_cod"),
avg("avg_ammonia").alias("cluster_avg_ammonia"),
avg("avg_phosphorus").alias("cluster_avg_phosphorus"),
avg("avg_quality_index").alias("cluster_avg_quality"),
collect_list("City").alias("cities_in_cluster")
).orderBy("cluster")
pollution_spike_analysis = spark.sql("""
SELECT Province, City, Date,
COUNT(CASE WHEN Remarks LIKE '%High pollution spike detected%' THEN 1 END) as spike_events,
AVG(Water_Quality_Index) as avg_spike_intensity,
MAX(Water_Quality_Index) as max_spike_value
FROM water_quality
WHERE Remarks LIKE '%High pollution spike detected%'
GROUP BY Province, City, Date
ORDER BY spike_events DESC, avg_spike_intensity DESC
LIMIT 50
""")
city_stability_ranking = spark.sql("""
SELECT City,
AVG(Water_Quality_Index) as avg_quality,
STDDEV(Water_Quality_Index) as stability_score,
COUNT(*) as data_points,
ROUND(STDDEV(Water_Quality_Index) / AVG(Water_Quality_Index), 4) as coefficient_variation
FROM water_quality
GROUP BY City
HAVING COUNT(*) >= 20
ORDER BY stability_score ASC
""")
city_clustered.select("City", "cluster", "avg_quality_index", "quality_volatility").coalesce(1).write.mode("overwrite").option("header", "true").csv("hdfs://localhost:9000/results/city_clusters")
cluster_analysis.coalesce(1).write.mode("overwrite").option("header", "true").csv("hdfs://localhost:9000/results/cluster_profiles")
return {
'city_clusters': city_clustered.select("City", "cluster", "avg_quality_index", "quality_volatility").collect(),
'cluster_profiles': cluster_analysis.collect(),
'pollution_spikes': pollution_spike_analysis.collect(),
'stability_ranking': city_stability_ranking.collect(),
'cluster_centers': kmeans_model.clusterCenters()
}
中国水污染监测数据可视化分析系统-结语
💟💟如果大家有任何疑虑，欢迎在下方位置详细交流。
