Calling C++ with pybind11

When Python needs C++

Much of the trading world's serious code is C++: matching engines, market-data handlers, pricing libraries tuned over fifteen years. You rarely want to rewrite it, and you rarely want to give up Python for research. pybind11 is the bridge: a header-only C++ library in which you write a few lines saying "expose this C++ function to Python under this name", compile them into a shared library, and import it from Python like any module. Arguments and results are converted for you, so a Python list of floats becomes a std::vector<double> and back.

# A C++ function and the pybind11 file that exposes it to Python (shown here as text)
cpp = """\
#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
#include <vector>

namespace py = pybind11;

double mean(const std::vector<double>& x) {
    double total = 0.0;
    for (double v : x) total += v;
    return x.empty() ? 0.0 : total / x.size();
}

PYBIND11_MODULE(fastmath, m) {
    m.def("mean", &mean, "Mean of a list of floats");
}
"""
lines = cpp.splitlines()
print(lines[-2].strip())                             # m.def("mean", &mean, "Mean of a list of floats");
print(len(lines))                                    # 15

Types across the bridge

Each Python type has a C++ counterpart: int is int64_t, float is double, str is std::string, a list of floats is std::vector<double>; anything else can be held as a generic py::object. Including <pybind11/stl.h> makes the conversions of standard containers automatic. Pass large values by const reference (const std::vector<double>& x) so the C++ function does not copy them, and small values like numbers by value. Conversion of a Python list to a vector still copies once per call, so for big arrays use py::array_t<double>, which views a NumPy array's memory directly with no copy.

# Choosing the C++ type of each argument; big values are passed by const reference
CPP_TYPES = {"int": "int64_t", "float": "double", "str": "std::string", "list[float]": "std::vector<double>"}

def cpp_type(py_type):
    return CPP_TYPES.get(py_type, "py::object")

def cpp_signature(name, params, returns):
    result = "void" if returns is None else cpp_type(returns)
    arguments = []
    for param, py_type in params:
        ctype = cpp_type(py_type)
        arguments.append(f"const {ctype}& {param}" if ctype.startswith("std::") else f"{ctype} {param}")
    return f"{result} {name}({', '.join(arguments)})"

print(cpp_signature("mean", [("x", "list[float]"), ("n", "int")], "float"))      # double mean(const std::vector<double>& x, int64_t n)
print(cpp_signature("label", [("symbol", "str")], "str"))                       # std::string label(const std::string& symbol)

Building, GIL and ownership

Build with CMake (pybind11_add_module) or with pybind11.setup_helpers in a setup.py, and the result is a .so or .pyd file you import. Two things to know. First, while C++ code runs, Python's Global Interpreter Lock stays held unless you release it with py::gil_scoped_release; a long C++ computation that does not touch Python objects should release it so other Python threads can run. Second, ownership: if C++ returns a reference or pointer into its own memory, Python must not outlive it (py::return_value_policy controls this), and a crash with no Python traceback usually means a dangling pointer. Test the bound functions from Python against a pure-Python version, exactly as in the previous labs.

# Binding lines and the module file, generated from a table of functions
BACKSLASH = chr(92)

def binding_line(py_name, cpp_name, doc):
    escaped = doc.replace(BACKSLASH, BACKSLASH + BACKSLASH).replace('"', BACKSLASH + '"')
    return f'm.def("{py_name}", &{cpp_name}, "{escaped}");'

def module_source(module, functions):
    lines = ["#include <pybind11/pybind11.h>", "#include <pybind11/stl.h>", "", "namespace py = pybind11;", ""]
    lines.append(f"PYBIND11_MODULE({module}, m) {{")
    for py_name, cpp_name, doc in functions:
        lines.append("    " + binding_line(py_name, cpp_name, doc))
    lines.append("}")
    return "\n".join(lines) + "\n"

source = module_source("pricing", [("mean", "mean_impl", "Mean of x"), ("price", "bs_price", 'Black-Scholes "call" price')])
print(source.splitlines()[-3])                       #     m.def("mean", &mean_impl, "Mean of x");
print(source.splitlines()[-2])                       #     m.def("price", &bs_price, "Black-Scholes \"call\" price");

# From Python the built module is used like any other; this stand-in has the same interface
import types

def mean(x):
    """Mean of a list of floats"""
    return sum(x) / len(x) if x else 0.0

fastmath = types.SimpleNamespace(mean=mean)          # stands in for `import fastmath` after the build
print(fastmath.mean([100.0, 102.0, 101.0]))          # 101.0
print(fastmath.mean.__doc__)                         # Mean of a list of floats   the doc string given to m.def
print(fastmath.mean([]))                             # 0.0   what the C++ code returns for an empty vector

• pybind11 exposes C++ functions to Python with PYBIND11_MODULE and m.def(...) lines; the result is an importable .so/.pyd module.
• Types map int→int64_t, float→double, str→std::string, lists→std::vector; pass big values by const reference and NumPy arrays as py::array_t.
• Release the GIL for long pure-C++ work, mind who owns returned memory, validate inputs at the boundary, and test against a pure-Python version.

Next up: threads and the GIL, what Python threads can and cannot do in parallel.

Lab · what's graded

• Map Python types to C++ with cpp_type25 pts
• Declare a function with cpp_signature25 pts
• Expose a function with binding_line25 pts
• Assemble the module with module_source25 pts