💖💖作者：计算机编程小央姐 💙💙个人简介：曾长期从事计算机专业培训教学，本人也热爱上课教学，语言擅长Java、微信小程序、Python、Golang、安卓Android等，开发项目包括大数据、深度学习、网站、小程序、安卓、算法。平常会做一些项目定制化开发、代码讲解、答辩教学、文档编写、也懂一些降重方面的技巧。平常喜欢分享一些自己开发中遇到的问题的解决办法，也喜欢交流技术，大家有技术代码这一块的问题可以问我！ 💛💛想说的话：感谢大家的关注与支持！ 💜💜
💕💕文末获取源码
@TOC
基于Hadoop+Spark的压力检测数据分析系统-系统功能介绍
基于Hadoop+Spark的压力检测数据分析系统是一个专门针对个体压力状态进行智能分析的大数据应用平台。该系统采用Hadoop分布式存储架构和Spark内存计算引擎，能够高效处理海量的压力监测数据，包括心理量表评分、生理指标监测、行为模式记录等多维度数据源。系统通过Python和Java双语言支持，提供Django和Spring Boot两套后端解决方案，前端采用Vue+ElementUI+Echarts技术栈实现数据的可视化展示。核心功能涵盖压力水平分布统计、人格特质与压力关系分析、生活行为模式与压力关联性研究、生理指标与压力状态映射等多个分析维度。系统利用Spark SQL进行复杂查询分析，结合Pandas和NumPy进行数据预处理和统计计算，通过HDFS实现数据的可靠存储，最终将分析结果通过直观的图表形式呈现给用户，为压力管理和健康监测提供科学依据和决策支持。
基于Hadoop+Spark的压力检测数据分析系统-系统技术介绍
大数据框架：Hadoop+Spark（本次没用Hive，支持定制） 开发语言：Python+Java（两个版本都支持） 后端框架：Django+Spring Boot(Spring+SpringMVC+Mybatis)（两个版本都支持） 前端：Vue+ElementUI+Echarts+HTML+CSS+JavaScript+jQuery 详细技术点：Hadoop、HDFS、Spark、Spark SQL、Pandas、NumPy 数据库：MySQL
基于Hadoop+Spark的压力检测数据分析系统-系统背景意义
现代社会节奏加快，个体面临的压力源日益多样化和复杂化，传统的压力评估方法主要依赖主观问卷调查和专业人员的经验判断，存在评估周期长、主观性强、难以捕捉压力变化动态特征等局限性。随着物联网设备和智能穿戴设备的普及，能够持续收集个体的生理指标、行为数据和心理状态信息，这些数据蕴含着丰富的压力状态特征。同时，大数据技术的成熟为处理和分析这些海量、多源、异构的压力相关数据提供了技术支撑。Hadoop生态系统能够有效存储和管理大规模数据，Spark计算引擎能够快速处理复杂的数据分析任务，这为构建智能化的压力检测系统奠定了技术基础。基于这样的技术背景和现实需求，开发一个能够综合分析多维度数据并准确识别压力状态的大数据系统具有重要的现实意义。 本课题的研究具有多重实际意义和价值。从技术层面来看，该系统验证了大数据技术在健康监测领域的应用可行性，为相关技术的推广应用提供了具体的实践案例，有助于推动大数据技术在医疗健康领域的深入发展。从应用层面来看，系统能够为个人用户提供客观的压力状态评估，帮助用户更好地了解自身压力水平和变化趋势，为制定个性化的压力管理策略提供数据支持。对于医疗机构和心理健康服务提供者而言，系统提供的多维度分析结果能够辅助专业人员进行更准确的压力评估和干预决策。从社会层面来看，该系统有助于提升公众对压力管理的认知和重视程度，为构建更加健康的社会环境贡献力量。从学术研究角度来看，系统探索了人格特质、生活行为、生理指标与压力状态之间的关联关系，为相关领域的研究提供了新的思路和方法。尽管作为毕业设计项目，该系统在规模和功能上相对有限，但其展现的技术实现路径和分析思路具有一定的参考价值。
基于Hadoop+Spark的压力检测数据分析系统-系统演示视频
基于Hadoop+Spark的压力检测数据分析系统-系统演示图片
基于Hadoop+Spark的压力检测数据分析系统-系统部分代码
from pyspark.sql import SparkSession
from pyspark.sql.functions import *
from pyspark.sql.types import *
import pandas as pd
import numpy as np
from sklearn.cluster import KMeans
from scipy.stats import pearsonr
import json
spark = SparkSession.builder.appName("PressureDetectionAnalysis").config("spark.sql.adaptive.enabled", "true").config("spark.sql.adaptive.coalescePartitions.enabled", "true").getOrCreate()
def pressure_level_distribution_analysis():
pressure_df = spark.read.csv("hdfs://localhost:9000/pressure_data/stress_detection.csv", header=True, inferSchema=True)
pressure_df.createOrReplaceTempView("pressure_data")
distribution_result = spark.sql("""
SELECT
CASE
WHEN PSS_score BETWEEN 10 AND 20 THEN '低压力'
WHEN PSS_score BETWEEN 21 AND 30 THEN '中压力'
WHEN PSS_score BETWEEN 31 AND 40 THEN '高压力'
ELSE '异常'
END as pressure_level,
COUNT(*) as count,
ROUND(COUNT(*) * 100.0 / (SELECT COUNT(*) FROM pressure_data), 2) as percentage
FROM pressure_data
WHERE PSS_score IS NOT NULL
GROUP BY
CASE
WHEN PSS_score BETWEEN 10 AND 20 THEN '低压力'
WHEN PSS_score BETWEEN 21 AND 30 THEN '中压力'
WHEN PSS_score BETWEEN 31 AND 40 THEN '高压力'
ELSE '异常'
END
ORDER BY count DESC
""")
time_trend_result = spark.sql("""
SELECT
day,
AVG(PSS_score) as avg_pressure,
COUNT(DISTINCT participant_id) as participant_count,
STDDEV(PSS_score) as pressure_std
FROM pressure_data
WHERE PSS_score IS NOT NULL AND day IS NOT NULL
GROUP BY day
ORDER BY day
""")
participant_comparison = spark.sql("""
SELECT
participant_id,
AVG(PSS_score) as avg_pressure,
MAX(PSS_score) as max_pressure,
MIN(PSS_score) as min_pressure,
COUNT(*) as record_count,
SUM(CASE WHEN PSS_score > 30 THEN 1 ELSE 0 END) as high_pressure_days
FROM pressure_data
WHERE PSS_score IS NOT NULL
GROUP BY participant_id
HAVING COUNT(*) >= 5
ORDER BY avg_pressure DESC
""")
distribution_pandas = distribution_result.toPandas()
time_trend_pandas = time_trend_result.toPandas()
participant_pandas = participant_comparison.toPandas()
analysis_result = {
'distribution': distribution_pandas.to_dict('records'),
'time_trend': time_trend_pandas.to_dict('records'),
'participant_analysis': participant_pandas.to_dict('records'),
'total_records': pressure_df.count(),
'avg_overall_pressure': pressure_df.agg({'PSS_score': 'avg'}).collect()[0][0]
}
return analysis_result
def personality_pressure_correlation_analysis():
personality_df = spark.read.csv("hdfs://localhost:9000/pressure_data/stress_detection.csv", header=True, inferSchema=True)
personality_df.createOrReplaceTempView("personality_data")
correlation_result = spark.sql("""
SELECT
participant_id,
AVG(PSS_score) as avg_pressure,
AVG(Openness) as avg_openness,
AVG(Conscientiousness) as avg_conscientiousness,
AVG(Extraversion) as avg_extraversion,
AVG(Agreeableness) as avg_agreeableness,
AVG(Neuroticism) as avg_neuroticism
FROM personality_data
WHERE PSS_score IS NOT NULL AND Openness IS NOT NULL
GROUP BY participant_id
HAVING COUNT(*) >= 3
""")
correlation_pandas = correlation_result.toPandas()
personality_traits = ['avg_openness', 'avg_conscientiousness', 'avg_extraversion', 'avg_agreeableness', 'avg_neuroticism']
correlation_coefficients = {}
for trait in personality_traits:
if len(correlation_pandas) > 10:
corr_coef, p_value = pearsonr(correlation_pandas['avg_pressure'], correlation_pandas[trait])
correlation_coefficients[trait] = {
'correlation': round(corr_coef, 4),
'p_value': round(p_value, 4),
'significance': 'significant' if p_value < 0.05 else 'not_significant'
}
neuroticism_analysis = spark.sql("""
SELECT
CASE
WHEN Neuroticism <= 2.5 THEN '低神经质'
WHEN Neuroticism <= 3.5 THEN '中神经质'
ELSE '高神经质'
END as neuroticism_level,
AVG(PSS_score) as avg_pressure,
COUNT(DISTINCT participant_id) as participant_count,
STDDEV(PSS_score) as pressure_std
FROM personality_data
WHERE Neuroticism IS NOT NULL AND PSS_score IS NOT NULL
GROUP BY
CASE
WHEN Neuroticism <= 2.5 THEN '低神经质'
WHEN Neuroticism <= 3.5 THEN '中神经质'
ELSE '高神经质'
END
ORDER BY avg_pressure DESC
""")
extraversion_analysis = spark.sql("""
SELECT
CASE
WHEN Extraversion <= 2.5 THEN '内向型'
WHEN Extraversion <= 3.5 THEN '中间型'
ELSE '外向型'
END as extraversion_type,
AVG(PSS_score) as avg_pressure,
COUNT(DISTINCT participant_id) as participant_count,
MAX(PSS_score) as max_pressure,
MIN(PSS_score) as min_pressure
FROM personality_data
WHERE Extraversion IS NOT NULL AND PSS_score IS NOT NULL
GROUP BY
CASE
WHEN Extraversion <= 2.5 THEN '内向型'
WHEN Extraversion <= 3.5 THEN '中间型'
ELSE '外向型'
END
ORDER BY avg_pressure
""")
personality_result = {
'correlation_coefficients': correlation_coefficients,
'neuroticism_analysis': neuroticism_analysis.toPandas().to_dict('records'),
'extraversion_analysis': extraversion_analysis.toPandas().to_dict('records'),
'sample_size': len(correlation_pandas)
}
return personality_result
def lifestyle_behavior_cluster_analysis():
lifestyle_df = spark.read.csv("hdfs://localhost:9000/pressure_data/stress_detection.csv", header=True, inferSchema=True)
lifestyle_df.createOrReplaceTempView("lifestyle_data")
behavior_features = spark.sql("""
SELECT
participant_id,
AVG(sleep_duration) as avg_sleep_duration,
AVG(PSQI_score) as avg_sleep_quality,
AVG(screen_on_time) as avg_screen_time,
AVG(num_calls) as avg_calls,
AVG(call_duration) as avg_call_duration,
AVG(mobility_radius) as avg_mobility_radius,
AVG(mobility_distance) as avg_mobility_distance,
AVG(PSS_score) as avg_pressure
FROM lifestyle_data
WHERE sleep_duration IS NOT NULL AND screen_on_time IS NOT NULL
AND mobility_radius IS NOT NULL AND PSS_score IS NOT NULL
GROUP BY participant_id
HAVING COUNT(*) >= 5
""")
behavior_pandas = behavior_features.toPandas()
if len(behavior_pandas) > 10:
feature_columns = ['avg_sleep_duration', 'avg_sleep_quality', 'avg_screen_time',
'avg_calls', 'avg_call_duration', 'avg_mobility_radius', 'avg_mobility_distance']
feature_data = behavior_pandas[feature_columns].fillna(behavior_pandas[feature_columns].mean())
feature_normalized = (feature_data - feature_data.mean()) / feature_data.std()
kmeans = KMeans(n_clusters=4, random_state=42, n_init=10)
cluster_labels = kmeans.fit_predict(feature_normalized)
behavior_pandas['cluster'] = cluster_labels
cluster_analysis = behavior_pandas.groupby('cluster').agg({
'avg_pressure': ['mean', 'std', 'count'],
'avg_sleep_duration': 'mean',
'avg_screen_time': 'mean',
'avg_mobility_distance': 'mean'
}).round(3)
sleep_pressure_correlation = spark.sql("""
SELECT
CASE
WHEN sleep_duration < 6 THEN '睡眠不足'
WHEN sleep_duration BETWEEN 6 AND 8 THEN '睡眠正常'
ELSE '睡眠充足'
END as sleep_category,
AVG(PSS_score) as avg_pressure,
COUNT(*) as record_count,
AVG(PSQI_score) as avg_sleep_quality
FROM lifestyle_data
WHERE sleep_duration IS NOT NULL AND PSS_score IS NOT NULL
GROUP BY
CASE
WHEN sleep_duration < 6 THEN '睡眠不足'
WHEN sleep_duration BETWEEN 6 AND 8 THEN '睡眠正常'
ELSE '睡眠充足'
END
ORDER BY avg_pressure DESC
""")
screen_behavior_analysis = spark.sql("""
SELECT
CASE
WHEN screen_on_time < 4 THEN '低使用'
WHEN screen_on_time BETWEEN 4 AND 8 THEN '中等使用'
ELSE '高使用'
END as screen_usage_level,
AVG(PSS_score) as avg_pressure,
AVG(num_calls) as avg_daily_calls,
AVG(num_sms) as avg_daily_sms,
COUNT(DISTINCT participant_id) as participant_count
FROM lifestyle_data
WHERE screen_on_time IS NOT NULL AND PSS_score IS NOT NULL
GROUP BY
CASE
WHEN screen_on_time < 4 THEN '低使用'
WHEN screen_on_time BETWEEN 4 AND 8 THEN '中等使用'
ELSE '高使用'
END
ORDER BY avg_pressure
""")
lifestyle_result = {
'cluster_analysis': cluster_analysis.to_dict() if len(behavior_pandas) > 10 else {},
'sleep_pressure_correlation': sleep_pressure_correlation.toPandas().to_dict('records'),
'screen_behavior_analysis': screen_behavior_analysis.toPandas().to_dict('records'),
'total_participants': len(behavior_pandas)
}
return lifestyle_result
基于Hadoop+Spark的压力检测数据分析系统-结语
💟💟如果大家有任何疑虑，欢迎在下方位置详细交流。
