Jupyter notebooks

Code, results and notes in one document

A Jupyter notebook is a document made of cells. A code cell holds Python you can run; its output (numbers, tables, charts) appears right below it. A markdown cell holds formatted text for explanations, equations and headings. A background process called the kernel runs the code and remembers every variable between cells. Notebooks are ideal for exploring data and explaining an analysis, which is why they are everywhere in quantitative research.

# Inside your project's virtual environment
python -m pip install jupyterlab

# Start it; a browser tab opens
jupyter lab

# Keyboard shortcuts you will use all day
#   Shift+Enter   run the cell and move on
#   Esc then A/B  insert a cell above/below
#   Esc then M/Y  turn a cell into markdown/code

The kernel remembers, and that is the danger

Cells can be run in any order and any number of times, and the kernel keeps the latest value of every variable. A notebook can therefore look correct on screen while depending on a cell you deleted, or on running cells out of order, and nobody else (or you tomorrow) can reproduce it. The execution counter next to each code cell ([3], [7]) shows the real order. Before you trust or share a notebook, use Restart Kernel and Run All Cells: if it fails or the numbers change, the notebook was not reproducible.

# Cell [1]
fee = 0.001

# Cell [2]
cost = 10000 * fee
print(cost)          # 10.0

# Cell [3]: you edit this to fee = 0.002 but never re-run cell [2]
fee = 0.002
print(cost)          # still 10.0: the notebook now says fee=0.002 but shows the old cost

A notebook is just JSON

An .ipynb file is a JSON document with a cells list. Each cell has a cell_type ("code", "markdown" or "raw") and a source (a list of lines or one string); code cells also store their outputs and execution_count. Because it is plain JSON, you can read and edit notebooks with the json module, which is how tools convert them to scripts or strip their outputs. This also explains why notebooks make noisy Git diffs: outputs and counters change on every run.

import json

notebook = {
    "nbformat": 4, "nbformat_minor": 5, "metadata": {},
    "cells": [
        {"cell_type": "markdown", "metadata": {}, "source": ["# Fee study\n", "Does the fee matter?"]},
        {"cell_type": "code", "metadata": {}, "execution_count": 2,
         "source": ["fee = 0.001\n", "print(10000 * fee)"],
         "outputs": [{"output_type": "stream", "name": "stdout", "text": ["10.0\n"]}]},
    ],
}

# What the file looks like, and how to walk it
print(len(json.dumps(notebook, indent=1).splitlines()) > 10)     # True
for cell in notebook["cells"]:
    print(cell["cell_type"], "".join(cell["source"]).splitlines()[0])
# markdown # Fee study
# code fee = 0.001

Habits for tidy, trustworthy notebooks

Keep notebooks short and linear: top to bottom, imports and settings in the first cell, one idea per section with a markdown heading. Move reusable logic into functions in a normal .py module (see packages) and import it, so it can be tested; the notebook then just calls and plots. Restart and run all before every commit. Under Git, clear outputs first (a tool such as nbstripout can do this automatically) or you will commit megabytes of charts and unreadable diffs. Never leave API keys in a cell or an output.

# Keep notebooks tidy under Git
python -m pip install nbstripout
nbstripout --install           # strips outputs automatically on every commit

# Run a notebook top to bottom from the terminal (a reproducibility check)
jupyter nbconvert --to notebook --execute study.ipynb --output checked.ipynb

# Turn a notebook into a plain script
jupyter nbconvert --to script study.ipynb

• Cells run in any order and the kernel remembers: Restart and Run All before you trust or share.
• An .ipynb is JSON: cells, each with cell_type, source, and for code outputs and execution_count.
• Keep logic in modules, clear outputs before committing, keep secrets out.

Next up: reproducible research workflows, putting seeds, environments, versions and data checks together so a result can be rebuilt by anyone.

Lab · what's graded

• Pull the code out of a notebook with code_cells25 pts
• Count the kinds of cells with count_cells25 pts
• Strip the outputs before committing with clear_outputs25 pts
• Turn a notebook into a script with to_script25 pts