Reproducible research workflows

Same inputs, same answer

A result is reproducible when you, a colleague or a reviewer can run the same project later and get the same numbers. A backtest that cannot be rebuilt cannot be audited, debugged or trusted. Results drift for a handful of reasons: random numbers, different library versions, data that changed or was edited by hand, settings scattered through the code, and hidden state such as a notebook run out of order. A reproducible workflow removes each source of drift and records what was used.

import random

# Same seed, same "random" numbers
a = random.Random(42)
b = random.Random(42)
print([round(a.random(), 3) for _ in range(3)])   # [0.639, 0.025, 0.275]
print([round(b.random(), 3) for _ in range(3)])   # [0.639, 0.025, 0.275]

# A different seed gives a different but equally repeatable sequence
print(round(random.Random(7).random(), 3))        # 0.324

Seeds, and which generator owns them

Simulations and bootstraps use random numbers, which are really a deterministic sequence started from a seed. Fix the seed and the run repeats exactly. random.seed(42) seeds the one global generator that every part of the program shares, so any other code that draws numbers shifts your sequence. It is safer to create your own generator, rng = random.Random(42) (or numpy.random.default_rng(42)), pass it around and draw from it. Then your results depend only on your seed, and your seeding does not disturb anyone else's randomness.

import random

# The global generator is shared: seeding it changes everyone's numbers
random.seed(1)
first = random.random()
random.seed(1)
print(random.random() == first)        # True: global seeding repeats...

# ...but a private generator leaves the global one untouched
random.seed(1)
before = random.getstate()
mine = random.Random(99)
mine.random()
print(random.getstate() == before)     # True: your draws did not disturb it

Fingerprints for data and settings

How do you know the data file is the one you used last month? Compute a checksum: a hash such as SHA-256 turns any file into a short fixed-length string that changes if even one byte does. hashlib.sha256(data).hexdigest() gives it; read big files in chunks. Settings deserve the same treatment: serialise the configuration with sorted keys (json.dumps(config, sort_keys=True)), hash that text, and the fingerprint is identical for two configurations that mean the same thing, whatever the key order, and different as soon as a value changes.

import hashlib
import json

# A checksum of bytes
print(hashlib.sha256(b"AAPL,184.25").hexdigest()[:12])   # 06ea4aa73ec1 (12 characters of 64)

# A fingerprint of settings, independent of key order
def fingerprint(config):
    text = json.dumps(config, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(text.encode("utf-8")).hexdigest()[:12]

one = {"fee": 0.001, "window": 20}
two = {"window": 20, "fee": 0.001}
print(fingerprint(one) == fingerprint(two))              # True
print(fingerprint(one) == fingerprint({"fee": 0.002, "window": 20}))  # False

Write down what you ran

Every run should leave a small manifest next to its results: the seed, the configuration fingerprint, the Python and library versions, the checksum of each input file and the Git commit of the code. With that record you can say exactly what produced a number and rebuild it later. Combine it with the earlier habits: an environment pinned in requirements.txt, code in Git, settings in one file, one command that runs everything from raw data to final tables, and no manual edits to data in between.

import json
import platform

# A manifest records what a run used
manifest = {
    "seed": 42,
    "config_fingerprint": "a1b2c3d4e5f6",
    "python": platform.python_version().split(".")[0] + ".x",
    "inputs": {"prices.csv": "9f86d081884c"},
}
print(json.dumps(manifest, indent=2, sort_keys=True))
# {
#   "config_fingerprint": "a1b2c3d4e5f6",
#   "inputs": {
#     "prices.csv": "9f86d081884c"
#   },
#   "python": "3.x",
#   "seed": 42
# }

• Use a private random.Random(seed) so results depend only on your seed.
• Fingerprint data (a file checksum) and settings (sorted JSON, then SHA-256).
• Save a manifest with each run; pinned environment + Git commit + one command to rebuild.

Next up: classes and objects, bundling data and the functions that work on it into one tidy unit.

Lab · what's graded

• Generate repeatable random returns with seeded_returns25 pts
• Fingerprint a data file with file_checksum25 pts
• Fingerprint a configuration with config_fingerprint25 pts
• Record what a run used with run_manifest25 pts