Week 2: Collecting Data, Sourcing Data and Big Data

Additional Videos
Exploring the Role of AI in Optimizing the Data Pipeline
A Bite of Me
Nicholas Felton: A Quantified Life
The House that Spied on Me
Darknet (The Most Connected Human)
CH4

Tools
Data Miner
Tabula
Python Beautiful Soup

Guest Speaker
Olivia Robinson, Data Director at Color of Change. Watch video below.

This week we will be jumping into the world of data and looking at what is openly available to us. We will also be talking about all the different types of data (there are lots) and all the different formatting types it could possibly be saved as. We will also be talking about some great tools to collect your own data. Finally we will be demystifying what “Big Data” actually is.

Lecture Slides

Homework

  1. Find an interesting open data set. Present it to the class in the class Slack channel giving your reasons why you chose it and what interesting data points you noticed in it. This can be a website or open data set, lets avoid APIs for now.
  2. Watch the video of the Week 3 lecture and come with questions next week.
  3. Reading: Chapter “Why Visualize: From Information to Wisdom”, PagesĀ 5-19
  4. Start collecting your own personal data that you can use in your final project. You must have at least three different data sets to work with. Some examples could be location data, exercise data, type of food you eat, who you interact with, what time you wake up and go to sleep, your mood for the day, etc.

    Here are a few tools to help you do this:
    Gyroscope, best main tool.
    Reporter App simple diary app created by Nicholas Felton.
    Daytum track your daily events on this website.
    Apple Data, download all the data Apple has on you.
    Facebook Data, download some of the data Facebook has on you.
    iPhone Health App track steps, calories burned, goals, etc.
    And of course devices like the Apple Watch, FitBit, Nike Fuelband, etc.

Week 1: Introduction to Data: Source, Scrub, Analyze and Visualization (Data Pipeline)

This week we will be going over what to expect in the class over the course of the semester. We will also be talking about the data pipeline from sourcing and collecting to scrubbing and cleaning data to analysis then finally visualizing for presentation. We also will cover a brief history of data and data collection as well as look at a few inspiring visualizations.

Lecture Slides

Homework

  1. Sign up for Slack and make sure you are added to the Slack class channel.
  2. Watch the Week 2 lecture video and come to Slack class with questions/comments next week.
  3. Buy the textbook: The Functional Art, by Alberto Cairo