Lecture 12 · Pandas and Exploratory Data Analysis

COMP 521 · 30-minute introduction

Pandas and exploratory data analysis

From labelled observations to inspectable evidence

Anna Rosen · San Diego State University

A Python library for labelled data

What is Pandas?

What it provides

DataFrame: a table with row and column labels.
Series: one labelled sequence, such as a column.
Tools to read files, select records, handle missing values, and summarize data.

Where it fits

NumPy computes with arrays. Pandas adds table labels and column types. Matplotlib draws the plots.

The usual import

import pandas as pd

pd is an alias for the library, not a dataset.

Choose a representation for the task

Compare the data containers

Container	How you address data	What it organizes
Python list	Positions	A sequence of objects
Python dictionary	Keys	Key–value associations
NumPy array	Positions in a shape	An array with one dtype
Pandas Series	Index labels or positions	One labelled sequence with one dtype
Pandas DataFrame	Row/column labels or positions	A table with one dtype per column

A dictionary can supply a table’s columns. A DataFrame adds a shared row index and table operations; each column is a Series.

Synthetic teaching data

A dictionary of columns becomes a DataFrame

Dictionary keys name the columns

columns = {
    'sample': ['A', 'B', 'C', 'D', 'E'],
    'exposure_s': [1., 2., 2.5, np.nan, 4.],
    'signal': [2., 3., 3., 4., 18.],
    'quality': ['ok', 'ok', 'ok', 'ok', 'flagged']}

Give the table its row labels

row_labels = ['r10', 'r11', 'r12',
              'r13', 'r14']
toy = pd.DataFrame(columns,
                   index=row_labels)
print(toy.to_string())

Each list supplies one column. Entries at the same position form one row.

These equal-length lists describe five synthetic exposures. Signal uses arbitrary units; exposure uses seconds.

What the representation preserves

Inspect the records

    sample  exposure_s  signal  quality
r10      A         1.0     2.0       ok
r11      B         2.0     3.0       ok
r12      C         2.5     3.0       ok
r13      D         NaN     4.0       ok
r14      E         4.0    18.0  flagged

Sample D lacks an exposure. Sample E carries a flag. Keep both facts visible.

toy.shape is (5, 4). Inspect toy.index, toy.columns, and toy.dtypes. Types do not supply units.

The brackets specify the result

One column is a Series

A column and a subtable

print(type(toy['signal']).__name__)
print(type(toy[['signal']]).__name__)
print(toy[['sample', 'signal']].to_string())

Expression	Result
`toy['signal']`	A labelled Series
`toy[['sample', 'signal']]`	A two-column DataFrame

The inner list names the columns to keep.

Addressing a record

Labels and positions are different

.loc uses labels; .iloc uses positions

print(toy.loc['r10', 'signal'])
print(toy.iloc[0]['signal'])
row = toy.loc['r12', ['sample', 'signal']]
print(row.to_string())

2.0
2.0
sample      C
signal    3.0

The first row’s label is r10, while its position is 0.

Keep quantities paired

Columns support row-wise arithmetic

Signal divided by its own exposure

rate = toy['signal'] / toy['exposure_s']
print(rate.to_string())

r10    2.0
r11    1.5
r12    1.2
r13    NaN
r14    4.5

Pandas matches row labels before dividing. These columns come from the same table. The result uses arbitrary signal units per second.

Preserve missingness

Missing is a state, not a measured zero

isna() marks missing cells True. .sum() counts those marks down each column: True contributes 1; False contributes 0.

Find and count missing entries

print(toy.isna().sum().to_string())
missing = toy['exposure_s'].isna()
fields = ['sample', 'exposure_s']
unknown = toy.loc[missing, fields]
print(unknown.to_string())

sample        0
exposure_s    1
signal        0
quality       0
    sample  exposure_s
r13      D         NaN

D’s rate is missing because its exposure is unknown. Replacing the exposure with zero would invent a value and change the calculation.

A boolean mask

A comparison gives one decision per row

A named condition

is_ok = toy['quality'].eq('ok')
print(is_ok.to_string())

r10     True
r11     True
r12     True
r13     True
r14    False

.eq('ok') compares each value with 'ok'. The result is a boolean Series with the table’s row labels. True means keep that row for this rule.

Combine conditions with & (both), | (either), and ~ (reverse). Put comparisons in parentheses.

Selection preserves the rule

Read `.loc[rows, columns]`

Select records and variables

toy.loc[is_ok, ['sample', 'exposure_s', 'signal']]

Before the comma: a row label, list of labels, or mask.
After the comma: the columns to retain.
Build the mask from the same table’s labelled columns.

In-class reasoning

Explain the selection before running it

is_ok marks the rows with an “ok” quality flag. Compare these two selections:

Two row-selection rules

toy.loc[is_ok]
toy.loc[is_ok & toy['exposure_s'].notna()]

Predict

Which row labels will each expression retain? Explain the difference.

Explain

The first keeps r10–r13. The second keeps r10–r12. Selection does not renumber labels or fill missing values.

Describe what is available

Start EDA with counts and ranges

Descriptive summaries from the same table

fields = ['exposure_s', 'signal']
summary = toy[fields].describe()
stats = ['count', 'mean', '50%',
         'min', 'max']
shown = summary.loc[stats].round(3)
print(shown.to_string())

       exposure_s  signal
count       4.000     5.0
mean        2.375     6.0
50%         2.250     3.0
min         1.000     2.0
max         4.000    18.0

count reports available values in each column. 50% is the median. Different columns can summarize different numbers of observations.

Missing values change the available count

The denominator matters

The four known exposures are 1, 2, 2.5, and 4 seconds.

Predict

Is their mean $9.5/4$ or $9.5/5$ ? What would the other denominator imply?

Explain

\bar t=\frac{1+2+2.5+4}{4}=2.375\ {\rm s}.

Dividing by five would effectively count the missing exposure as zero. The available-value mean does not establish that missingness is harmless.

EDA · one variable

Plot a distribution, not just one number

Histogram of five synthetic signals. Four values lie between zero and five, one near eighteen. The mean is six and the median three. — A bin is an interval of signal values. Its height counts stored values in that interval.

EDA · two variables

A relationship requires paired records

notna() checks each cell. .all(axis=1) checks across columns within one row; both fields must be known.

Select rows with both variables

xy = ['exposure_s', 'signal']
paired = toy[xy].notna().all(axis=1)
pairs = toy.loc[paired]
print(pairs.to_string())

    sample  exposure_s  signal  quality
r10      A         1.0     2.0       ok
r11      B         2.0     3.0       ok
r12      C         2.5     3.0       ok
r14      E         4.0    18.0  flagged

For this plot, retain rows with exposure and signal. Do not remove missing values separately from the two columns and then pair positions.

One published parameter set per planet

The CSV has a source contract

The saved NASA Planetary Systems (ps) query retains one default published parameter set per planet, with names and source references.

Read your saved Project 4 file

raw = pd.read_csv(
    'project4-nasa-exoplanets.csv',
    comment='#')

CSV is a text format with a header and comma-separated records. read_csv returns a DataFrame. Blank fields usually become missing values; check inferred types.

Save the CSV beside your own notebook. Keep the query and retrieval date.

Source before selection

Inspect what was read

(437, 15)
     pl_name  pl_masse  pl_rade
TRAPPIST-1 h     0.326    0.755
TRAPPIST-1 d     0.388    0.788
    KOI-55 b     0.440    0.759
   L 98-59 b     0.460    0.837

raw.shape gives the dimensions. raw.head(4) previews the four rows shown here; raw.head() defaults to five. Inspect column types with raw.dtypes.

This snapshot has 437 rows and 15 columns. A later live query may differ.

Make selection reproducible

Apply the rule and report the counts

Preserve raw; select planets

keep = (finite & central & in_window
        & signed_errors
        & positive_lower)
planets = raw.loc[keep].copy()
print('Input:', len(raw))
print('Selected:', len(planets))
print('Excluded:', len(raw)-len(planets))

Input: 437
Selected: 371
Excluded: 66

The Project 4 brief supplies the comparison rule: finite central estimates, the mass window, signed uncertainty offsets, and positive lower endpoints.

This snapshot retains 371 of 437 rows. Failed conditions can overlap; their excluded counts should not be added independently.

EDA · selected radii

Explore one variable

Histogram of radii in Earth units for the 371 selected NASA rows, with counts per equal-width bin. — The shape depends on the selected table, available measurements, and binning.

An estimate includes precision information

Read the relationship with its uncertainties

Logarithmic NASA mass–radius plot with asymmetric uncertainty bars, Earth marked, and TRAPPIST-1 e and Kepler-51 b highlighted. — The supplied offsets follow their source studies; a common confidence level is not established here.

Bridge Pandas to NumPy and Matplotlib

Reference recipe: preserve both uncertainties

Lower row, then upper row

mass_errors = np.vstack((
    -planets['pl_masseerr2'].to_numpy(),
     planets['pl_masseerr1'].to_numpy()))
radius_errors = np.vstack((
    -planets['pl_radeerr2'].to_numpy(),
     planets['pl_radeerr1'].to_numpy()))
print(mass_errors.shape, radius_errors.shape)

Two rows, $N$ planets

\begin{pmatrix} \text{lower lengths}\\ \text{upper lengths} \end{pmatrix}

For Kepler-51 b, one column is $\begin{pmatrix}0.8\\1.5\end{pmatrix}$ Earth masses.

.to_numpy() extracts the values. np.vstack stacks them as rows.

One column per selected planet. Every array comes from the same selected rows in the same order.

Install, load, and explore

Your turn: create your own notebook

In your course terminal

conda activate comp521
conda install -c conda-forge pandas

Create a blank Jupyter notebook beside the saved NASA CSV.
Import Pandas and load the CSV with pd.read_csv.
Inspect the table, choose columns, and try selections, summaries, and plots.
Discuss one result and one question with a neighbor.

Download the CSV and read the setup instructions. This is class exploration, with no new submission.

Inspect the source