Python and C/C++
Python is usually described as an interpreted language. When we run a Python program using CPython, the source code is compiled into Python bytecode (it can be considered as some form of low-level instructions), which is then executed by CPython’s virtual machine.
However, not every function called from Python must be implemented in Python. CPython provides the Python C API, which allows native C code to interact with the Python runtime.
Using this API, developers can write functions in C, compile them into native machine code, and expose them as modules that can be imported by Python programs.
The Python C API is a collection of C functions, macros, types, and data structures provided by CPython.
A C extension normally begins by including:
#include <Python.h>
This header provides access to functionality such as:
For example, a Python integer is represented in C using a pointer to a Python object:
PyObject *number = PyLong_FromLong(42);
Most values handled by CPython—including integers, strings, lists, functions, classes, and modules—are represented through some form of PyObject.
When C code is written as a Python extension, it is compiled into a native shared library.
The exact file format depends on the operating system:
Linux: mymodule.cpython-313-x86_64-linux-gnu.so
macOS: mymodule.cpython-313-darwin.so
Windows: mymodule.pyd
Although the output is a shared library containing native machine code, Python can import it like an ordinary module:
import mymodule
result = mymodule.add(2, 3)
The compiled extension is not converted into Python bytecode. Its implementation remains native machine code.
Suppose Python executes:
import mymodule
The import system searches for a compatible Python file or compiled extension module named mymodule.
After locating and loading the shared library, CPython looks for a specially named initialization function. This initialization function is implemented by the extension module and compiled into shared library:
PyInit_mymodule
A typical initialization function looks like this:
PyMODINIT_FUNC PyInit_mymodule(void) {
return PyModule_Create(&module_definition);
}
The name of the initialization function must correspond to the imported module name:
import mymodule
↓
PyInit_mymodule
This exported function acts as the entry point through which CPython initializes the extension module. After calling it, it returns a pointer to the Python module object PyObject *.
CPython does not scan the C source code to discover functions automatically.
Instead, every C function that should be available from Python must be explicitly registered in a PyMethodDef table.
For example:
static PyMethodDef methods[] = {
{"add", add, METH_VARARGS, "Add two integers."},
{NULL, NULL, 0, NULL}
};
The first entry connects the Python-visible name "add" to the compiled C function add.
Conceptually, each entry contains:
Python name "add"
C function add
Calling convention METH_VARARGS
Documentation "Add two integers."
The actual C function might be implemented as follows:
static PyObject *add(PyObject *self, PyObject *args) {
int a;
int b;
if (!PyArg_ParseTuple(args, "ii", &a, &b)) {
return NULL;
}
return PyLong_FromLong(a + b);
}
Here:
PyArg_ParseTuple converts Python arguments into C integers;PyLong_FromLong converts the result back into a Python integer;NULL indicates that a Python exception occurred.The method table is then attached to the module definition:
static struct PyModuleDef module_definition = {
PyModuleDef_HEAD_INIT,
"mymodule",
"An example C extension module",
-1,
methods
};
When PyModule_Create processes this definition during the initialization of the module, CPython creates Python-callable wrapper objects for the registered C functions and adds them to the module namespace.
Conceptually, it performs something similar to:
mymodule.__dict__["add"] = wrapper_for_compiled_C_function
From Python, the result appears as a built-in function:
>>> mymodule.add
<built-in function add>
Consider the following Python code:
result = mymodule.add(2, 3)
The call involves both Python bytecode and native C code.
First, CPython interprets the Python operations needed to:
mymodule;add attribute;The add attribute is a Python wrapper object containing a pointer to the compiled C function.
Once CPython reaches the call, it invokes that function pointer directly. The processor then executes the native machine instructions produced by the C compiler.
The execution flow is therefore:
Python source code
↓
Python bytecode
↓
CPython evaluates the function call
↓
CPython finds a registered C function wrapper
↓
CPython calls the native function pointer
↓
The CPU executes compiled C machine code
↓
The C function returns a PyObject *
↓
CPython resumes executing Python bytecode
The body of the C function is not translated into Python virtual-machine instructions.
Instead, control temporarily leaves CPython’s bytecode evaluation loop and enters already-compiled native code.
A normal Python function follows this path:
Python source
↓
Python bytecode
↓
CPython interpreter
↓
CPU
A C extension function follows this path:
C source
↓
C compiler
↓
Native machine code in a shared library
↓
CPU
Python still controls the outer function call, but the function implementation itself is executed natively.
The C compiler translates this loop into optimized machine instructions, avoiding Python bytecode interpretation for every iteration.
C extensions can improve performance because they can avoid overhead of processing the Python objects and use native machine instructions or numeric types. It can also use optimized C or C++ libraries.
However, calling C from Python is not free.
Crossing the Python–C boundary may require:
Case study
For the Python loop:
total += i
CPython executes virtual-machine instructions conceptually like:
LOAD_FAST total
LOAD_FAST i
BINARY_OP +
STORE_FAST total
Each VM instruction must be fetched and interpreted by CPython. BINARY_OP also operates on PyObject values, so CPython must handle Python integer semantics, create the result object, and update reference counts.
The equivalent C statement:
total += i;
may compile to a native CPU instruction such as:
add total, i
Therefore:
Python:
VM instruction dispatch
+ PyObject handling
+ dynamic type and reference-count management
C:
direct native arithmetic on machine integers
The Python loop is slower because every iteration involves both bytecode interpretation and Python object management, while the compiled C loop directly executes native machine instructions.
A C extension executes native code, but it normally still holds CPython’s Global Interpreter Lock, or GIL.
Therefore, native execution does not automatically allow multiple Python threads to execute Python code simultaneously.
A long-running C operation can release the GIL when it does not need to interact with Python objects:
Py_BEGIN_ALLOW_THREADS
/* Native computation that does not access Python objects */
Py_END_ALLOW_THREADS
While the GIL is released, other Python threads may run.
However, the extension generally must reacquire the GIL before manipulating Python objects through the C API.
The Python C API supports two opposite forms of integration.
A Python program imports and calls compiled C code:
Python program
↓
Compiled C extension
This produces an importable .so or .pyd module.
A C or C++ application starts a Python interpreter:
C/C++ application
↓
Embedded Python interpreter
For example:
#include <Python.h>
int main(void) {
Py_Initialize();
PyRun_SimpleString("print('Hello from embedded Python')");
Py_Finalize();
return 0;
}
This is normally compiled into an ordinary executable rather than an importable Python module.
The Python C API acts as a bridge between CPython and native C code.
A C extension is compiled into a native shared library that Python can import as a module. CPython identifies the module through its PyInit_<module_name> initialization function and identifies its functions through explicitly registered PyMethodDef entries.
When Python calls one of these functions, CPython does not convert the C implementation into Python bytecode. Instead, it invokes the registered function pointer, and the processor executes the already-compiled native machine code directly.
A useful mental model is:
Python handles the call, compiuled C code performs the native computation, and the Python C API translates objects and control between the two environments.
The Python C API is a low-level interface written in C. Although it can also be used from C++, working with it directly requires developers to manually manage details such as:
PyObject * values;For C++ projects, a commonly used higher-level alternative is pybind11. It is sometimes informally referred to as “pybind,” but the library’s actual name is pybind11.
pybind11 allows developers to expose C++ functions and classes to Python using ordinary C++ syntax. Internally, it still communicates with CPython through the Python C API.
The relationship can be represented as:
Python program
↓
pybind11-generated binding layer
↓
Python C API
↓
Compiled C++ implementation
Therefore, pybind11 does not replace the Python C API at the runtime level. Instead, it provides a more convenient C++ abstraction over it.
A function exposed using the raw Python C API might look like this:
#include <Python.h>
static PyObject *add(PyObject *self, PyObject *args) {
int a;
int b;
if (!PyArg_ParseTuple(args, "ii", &a, &b)) {
return NULL;
}
return PyLong_FromLong(a + b);
}
static PyMethodDef methods[] = {
{"add", add, METH_VARARGS, "Add two integers."},
{NULL, NULL, 0, NULL}
};
The developer must manually convert the Python arguments into C values and convert the result back into a Python object.
The equivalent extension using pybind11 is much shorter:
#include <pybind11/pybind11.h>
int add(int a, int b) {
return a + b;
}
PYBIND11_MODULE(mymodule, module) {
module.def("add", &add, "Add two integers.");
}
The function itself is an ordinary C++ function:
int add(int a, int b);
The following statement registers it as a Python-visible function:
module.def("add", &add);
pybind11 automatically generates the binding code needed to:
Python can then import and call the compiled extension normally:
import mymodule
print(mymodule.add(2, 3))
PYBIND11_MODULE Do?The following macro defines the extension module:
PYBIND11_MODULE(mymodule, module) {
module.def("add", &add);
}
It performs a role similar to the following module initialization function in the raw Python C API:
PyMODINIT_FUNC PyInit_mymodule(void) {
return PyModule_Create(&module_definition);
}
In both cases, the compiled shared library exports an initialization entry point that CPython can find when it executes:
import mymodule
pybind11 generates much of this initialization and registration machinery automatically.
One major advantage of pybind11 is that it can expose C++ classes more naturally.
Consider the following class:
class Vector {
public:
Vector(double x, double y) : x(x), y(y) {}
double length() const {
return std::sqrt(x * x + y * y);
}
private:
double x;
double y;
};
It can be bound to Python as follows:
#include <cmath>
#include <pybind11/pybind11.h>
namespace py = pybind11;
PYBIND11_MODULE(geometry, module) {
py::class_<Vector>(module, "Vector")
.def(py::init<double, double>())
.def("length", &Vector::length);
}
Python can then use the C++ class as though it were a normal Python class:
from geometry import Vector
vector = Vector(3, 4)
print(vector.length()) # 5.0
Behind the scenes, the Python Vector object wraps an instance of the compiled C++ class. Calls such as vector.length() cross the Python–C++ boundary and invoke the native C++ method.
Implementing the same class directly through the Python C API would require substantially more code for:
pybind11 handles much of this boilerplate through C++ templates and RAII.
Code bound with pybind11 is still compiled into native machine code.
The generated result is the same general kind of extension module produced using the raw Python C API:
Linux: mymodule.cpython-313-x86_64-linux-gnu.so
macOS: mymodule.cpython-313-darwin.so
Windows: mymodule.pyd
When Python calls a pybind11 function, the execution flow is approximately:
Python bytecode evaluates mymodule.add(2, 3)
↓
CPython finds the extension function
↓
pybind11 converts Python objects into C++ values
↓
The CPU executes the compiled C++ function
↓
pybind11 converts the C++ result into a Python object
↓
CPython resumes executing Python bytecode
Just like a direct C extension, the C++ implementation is not translated into Python bytecode.
| Aspect | Python C API | pybind11 |
|---|---|---|
| Primary language | C | C++ |
| Abstraction level | Low-level | Higher-level |
| Python object handling | Mostly manual | Wrapped in C++ types |
| Argument conversion | Manual parsing | Usually automatic |
| Reference counting | Explicit in many cases | Largely managed through RAII |
| Exception conversion | Manual | Often automatic |
| Exposing C++ classes | Possible but verbose | Directly supported |
| Function overloads | Manual dispatch | Supported naturally |
| Compiled output | .so or .pyd | .so or .pyd |
| Underlying CPython interface | Python C API | Python C API |
The distinction can be summarized as:
The Python C API is the low-level protocol through which native code communicates with CPython, while pybind11 is a C++ convenience layer that generates and manages much of that protocol automatically.
Python remains the public-facing interface, pybind11 provides the conversion and binding layer, and the underlying C++ code performs the native computation.
This learning note was developed from my studies and discussions with GPT. The final content was independently reviewed, technically verified, organized, and edited by the author.