Separate processes, separate memory
Threads cannot speed up CPU-bound Python because of the GIL, but processes can: each one is a full Python interpreter with its own GIL, and the operating system runs them on different cores. The multiprocessing module and the friendlier concurrent.futures.ProcessPoolExecutor start a pool of worker processes and give each a share of the work. A backtest over a thousand parameter sets, or a Monte Carlo with millions of paths, can use all eight cores of a laptop and finish in roughly an eighth of the time.
# A pool of processes: each gets a slice of the work, computed on its own core
from concurrent.futures import ProcessPoolExecutor
def sum_of_squares(bounds):
start, stop = bounds
return sum(i * i for i in range(start, stop))
if __name__ == "__main__": # required on Windows and macOS, where workers re-import this file
bounds = [(0, 250_000), (250_000, 500_000), (500_000, 750_000), (750_000, 1_000_000)]
with ProcessPoolExecutor(max_workers=4) as pool:
partials = list(pool.map(sum_of_squares, bounds))
print(sum(partials)) # 333332833333500000
print(sum(partials) == sum(i * i for i in range(1_000_000))) # True the same answer as one processWhat crosses the border: copies, pickling, chunks
Processes do not share variables. Whatever you send to a worker, the function and its arguments, is pickled (serialised to bytes), copied across and rebuilt, and the result travels back the same way. So only picklable things can cross: top-level functions and plain data yes, lambdas, nested functions, locks and generators no, which fails with a PicklingError or AttributeError. Copying has a cost, so send little and compute a lot: give each worker a pair of indexes (start, stop) instead of a copy of the data when it can load or generate it itself, hand out a few big chunks rather than a million tiny tasks, and return small results such as a sum, not a whole array.
# Index pairs instead of data, and a check that something can be sent to another process
import pickle
def chunk_bounds(n, parts):
size, extra = divmod(n, parts)
bounds, start = [], 0
for index in range(parts):
stop = start + size + (1 if index < extra else 0)
bounds.append((start, stop))
start = stop
return bounds
def is_picklable(obj):
try:
pickle.dumps(obj)
except (pickle.PicklingError, AttributeError, TypeError):
return False
return True
print(chunk_bounds(1_000_000, 4)) # [(0, 250000), (250000, 500000), (500000, 750000), (750000, 1000000)]
print(is_picklable({"AAPL": 184.25}), is_picklable(lambda x: x)) # True FalseIndependent random streams and combining partial results
Two details decide whether a parallel Monte Carlo is right. First, random seeds: a worker that starts with the same seed as another produces the same "random" numbers, and giving workers seed + 0, seed + 1, ... makes streams that can overlap or correlate. np.random.SeedSequence(seed).spawn(n) derives n statistically independent child seeds from one parent, so the whole run is still reproducible from a single number. Second, combining: each worker returns a small summary and you merge them. Counts add, means combine as a count-weighted average, and the variance of the whole must include the spread between the workers' means, so you cannot just average the variances.
# Independent seeds for the workers, and merging their partial statistics exactly
import numpy as np
def spawn_seeds(seed, n):
return [int(child.generate_state(1)[0]) for child in np.random.SeedSequence(seed).spawn(n)]
def combine_mean_var(partials):
count = sum(n for n, _, _ in partials)
mean = sum(n * m for n, m, _ in partials) / count
variance = sum(n * (v + (m - mean) ** 2) for n, m, v in partials) / count
return count, mean, variance
seeds = spawn_seeds(7, 4)
print(len(set(seeds)), spawn_seeds(7, 4) == seeds) # 4 True four different seeds, reproducible from one number
data = np.random.default_rng(seeds[0]).normal(0.0, 1.0, size=1_000)
partials = [(len(p), p.mean(), p.var()) for p in np.array_split(data, 4)]
count, mean, variance = combine_mean_var(partials)
print(count, np.isclose(mean, data.mean()), np.isclose(variance, data.var())) # 1000 True True# A parallel Monte Carlo: every worker has its own seed, and the run is still exactly reproducible
from concurrent.futures import ProcessPoolExecutor
import numpy as np
def simulate(job):
seed, paths = job
z = np.random.default_rng(seed).standard_normal(paths)
payoff = np.maximum(100.0 * np.exp(-0.5 * 0.2 ** 2 + 0.2 * z) - 100.0, 0.0) # at-the-money call, 20% volatility
return paths, payoff.mean(), payoff.var()
if __name__ == "__main__":
seeds = [int(child.generate_state(1)[0]) for child in np.random.SeedSequence(2024).spawn(4)]
jobs = [(seed, 50_000) for seed in seeds]
with ProcessPoolExecutor(max_workers=4) as pool:
partials = list(pool.map(simulate, jobs))
count = sum(n for n, _, _ in partials)
price = sum(n * mean for n, mean, _ in partials) / count
print(count, round(price, 1)) # 200000 7.9 close to the exact Black-Scholes value of about 7.97
print(partials == [simulate(job) for job in jobs]) # True the same seeds give the same numbers in one processnp.load(..., mmap_mode="r") or multiprocessing.shared_memory) instead of each holding a copy. And always guard the entry point with if __name__ == "__main__":, or on Windows and macOS each worker will try to start its own pool.- Processes have their own GIL and memory, so CPU-bound Python can use every core; a pool (
ProcessPoolExecutor) hands out the work. - Arguments and results are pickled and copied: send index pairs and small results, in a few big chunks; lambdas and nested functions cannot be sent.
- Give each worker an independent seed with
SeedSequence.spawn, merge partial statistics exactly, guard the entry point, and measure the real speed-up.
Next up: SQL databases, storing and querying market data with tables and queries.
Lab · what's graded
- Cut the range into slices with
chunk_bounds25 pts - Merge partial statistics with
combine_mean_var25 pts - Give each worker its own seed with
spawn_seeds25 pts - Count the cost of going parallel with
parallel_speedup25 pts