2.4 Using Programs with Data

What you need to know

• Programs can process data to discover information and generate new knowledge — automating what would be impossible manually at scale.
• Data cleaning is a necessary, real step: making data uniform (same date formats, consistent capitalization), removing duplicates, and handling missing or invalid values, all without changing the meaning of the data.
• Programs can transform data — reformat, combine, or derive new values — to make analysis possible (e.g., converting temperatures to one unit, computing an average per row).
• Visualization (charts, graphs, maps) communicates information in a dataset in a way that's much easier to interpret than the raw table.
• Extracting information requires asking a question first, then choosing which data, transformations, and tools answer it. Not every question can be answered by the data available.
• Tools include spreadsheets, purpose-built programs, and code; the process is the same regardless of tool.
• The knowledge gained from data can raise new questions, driving another round of collection and analysis (iteration again).

Worked example

You export a class survey to a spreadsheet. Some students typed "yes", others "Yes", others "Y". A program that normalizes all of these to yes is cleaning. Adding a column that converts "hours per week" from text like "3h" to the number 3 is transforming. Plotting hours-studied against grade is visualizing. Only after all three can you ask whether studying correlates with grades — and remember 2.3's warning about what a correlation does and doesn't show.

Going deeper

The nuance, edge cases, and connections that turn a 3 into a 5.

• This topic exists because the CED wants you to understand that programs, not people, do the work on large data. A human can't compute the average of ten million values; a five-line loop can. The value of computing is that it makes analysis of large datasets possible at all.
• Cleaning in the CED means making data uniform without changing its meaning: consistent capitalization, consistent date formats, consistent units, removing exact duplicates, deciding how to handle blanks. The "without changing meaning" clause is what separates cleaning from manipulation.
• Transformation derives new values from existing ones: converting Celsius to Fahrenheit, computing a per-student average from a list of scores, combining first and last name into one field. The original data is preserved; new columns are added.
• Filtering selects a subset by condition (only seniors, only scores above 80). Sorting orders by a field. Aggregating combines many records into a summary (count, sum, average, max). Most analyses chain these.
• Visualization choices matter: a bar chart for categories, a line chart for change over time, a scatter plot for relationships between two variables. The wrong chart type hides the pattern.
• The CED says data analysis can raise new questions, driving new collection. That iterative loop is the same idea as iterative development — investigate, analyze, refine, repeat.

Mistakes that cost points

• Calling manipulation "cleaning." If an option describes changing values to produce a desired conclusion ("adjusting outliers upward to raise the average"), that's not cleaning — it's falsification. Cleaning never changes meaning.
• Skipping cleaning and analyzing anyway. Questions that describe inconsistent data ("NY" and "New York") and ask what to do first want cleaning, not analysis.
• Choosing a visualization that hides the pattern. A pie chart for change over time, a line chart for unrelated categories. Match chart to question.

Practice questions

Written in the style of the real exam. Try each one before revealing the answer.