CS 7280 数据库管理以及大数据分析

CS 7280 Special Topics in Database Management Spring 2024
Project 3: Big Data Analytics
Objectives:

1. Understanding Hadoop Ecosystem and Data Analytics
2. Become familiar with MapReduce programming and Spark
3. Gain experience with research on big data and data analytics
   This will be a group project (by 2 students) for one semester. The main purpose of this
   project is to become familiar with Big Data platform, including Hadoop system,
   MapReduce programming, and cloud based big data solutions (e.g., Google Big Query).
   You need to follow the instruction to conduct the project.
   Phase 1 (15%): Selecting Data Set - Due: March 27, 2024 (Wed)
   • Each student researches on any data that you are interested in, and collect the
   information about the data.
   • Find any characteristics of the data you select, and describe why you are
   interested in
   • If possible, prepare 34 sample data, which can be either real data or manipulated
   one.
   • Make 2 3 pages of Powerpoint file as a report
   • Submit the PPT file to Canvas
   o PPT, PPTX or PDF file format ONLY
   Phase 2 (15%): Defining Problems – Due: April 3, 2024 (Wed)
   • In this 2nd phase, you are going to research on the following topics based on the
   data you selected in Phase 1:

• What you can analyze using the selected data in terms of Hadoop HDFS with
  Spark, and Google Big Query using GCP.
  o 1 Spark
  o 1 Google Big Query using GCP
• How you can collect the data at least 1GB. That means your data MUST be
  uploaded to HDFS using VM in Phase 4-5.
  • Make 2~ 3 pages of Powerpoint file as a report
  • Submit the PPT file to Canvas
  o PPT, PPTX or PDF file format ONLY
  Phase 3 (20%): Preparing Proposal – Due: April 3, 2024 (Wed)
  • Prepare a proposal using a MS word template: A proposal template can be found
  at Canvas
  o DOC, DOCX or PDF file format ONLY
  • Prepare and submit 5~10 pages of Powerpoint file for presentation
  o PPT, PPTX or PDF file format ONLY
  • Then, submit 10 minutes presentation video to Canvas
  o Submit a link such as YouTube,代 写 CS 7280 Special Topics in Databaseor record your presentation using Canvas
  • In your proposal, you need to consider how to prepare the final deliverable of
  following outputs

1. Write-up
2. Source code
3. Data set
4. Poster
   ** Note that this is a plan to prepare 1 ~ 4 above. NOT implementation right now.
   • Then, submit your proposal to Canvas
   • Prepare for 5 mins presentation for your proposal (submit PPT file also)
   Phase 4 (25%): Implementation – Due: April 10, 2024 (Wed)
5. Preparing Data and Upload to HDFS. You can use variety of ways to prepare your
   data set including:

• Use API provided by each website, such as Facebook API, Twitter API and
  Flickr API
• Use benchmarking data sets, such as
  o UCI data set: archive.ics.uci.edu/ml/datasets…
  o Wikipedia database: en.wikipedia.org/wiki/Databa…
• Government database
  o US Census data:
  factfinder.census.gov/faces/nav/j…
  o NOAA weather data: www.ncdc.noaa.gov/cdo-web/
• Implement Data collection program using Web query
• Synthesized data set
• Use googling

2. You data set MUST have at least 100,000 instances (or rows)
3. Upload your data set into HDFS (VM)
4. Implement Spark or Big Query

• You can use PySpark or any Steaming with other program language such as
  Python.
  o 1 Spark, or
  o 1 Big Query

5. Submit your source code to Canvas and download link for your data set

• All source files should be compressed with TAR (e.g., tar cvf XXX.tar) on
  VM (JAR, TAR or ZIP file format ONLY)
• For the dataset, you can upload it to Google Drive (or any Web hard) and then
  send a link when you submit your source

6. Then, submit 10 minutes demo video to Canvas

• Submit a link such as YouTube, or record your presentation using Canvas
  Phase 5 (25%): Presentation of Project – Due: April 17, 2024 (Wed) before class.

1. Writing-up (at least 4 pages with IEEE format). You must use IEEE format.
   o DOC, DOCX or PDF file format ONLY
2. Poster (36 x 24 inches Powerpoint file). You can use one of templates provided
   on Canvas.
   o PPT, PPTX or PDF file format ONLY
3. Submit your paper and poster to Canvas
4. Make 8 ~ 10 pages of Powerpoint file and submit to Canvas
   o PPT, PPTX or PDF file format ONLY
5. Then, prepare 8 minutes final presentation on April 27, 2022 (Wednesday)
   Submission
   You will submit your program using Canvas. If you have any trouble to use blackboard,
   you can contact TA or instructor.
   Grading
   15 Phase 1
   15 Phase 2
   20 Phase 3
   25 Phase 4
   25 Phase 5
   Bonus +20 for high quality writing-up that can be submitted to either conference
   WX：codinghelp