Raise of Rust in Python: A Perfect Synergy
There is an ever-growing number of Python tools written in Rust, as well as an increasing number of Python libraries either entirely or partially implemented in Rust.
Slow Snake, Fast Crab
Python is the most popular language. It’s commonly used for a variety of applications, ranging from tiny system tools to complex web frameworks. It’s also the lingua franca of data engineering and machine learning. However, Python is a dynamic language, and as such, it is extremely slow.
On the other hand, Rust is an extremely fast, statically typed, compiled language. It’s almost as fast as pure C, but Rust is a modern high-level language with syntax and features inspired by Ruby, as well as modern C++, C#, and several functional languages, including relatively modern ones like Haskell.
"So why not just write everything in Rust then? Well, for one, most organizations just aren’t ready for that. Rust is relatively new, and while the number of Rustaceans is growing, they’re vastly outnumbered by Pythonistas. The language being relatively new in the big leagues means that the selection of third-party tools and support is more limited. Thus, for now, the most viable strategy for most organizations is to use Python as the “glue” while gradually rewriting performance-critical paths in Rust.
Python Tools Written in Rust
For a while, Rust has been a popular language for reimplementing popular CLI tools and making them better. Tools like bat: a cat w syntax highlighting, delta: a syntax-highlighting diff, are notable examples. The logical next step was to use Rust to improve Python CLI tools. Some of the more notable ones are:
Ruff
Ruff is an extremely fast Python linter, replacement for Flake8. In more recent versions, it can also perform code formatting, thus replacing tools such as AutoPEP8, isort, or Black.
Even though it’s pure Rust, it can be installed using pip. Also, unlike flake8, Ruff supports modern Python pyproject.toml.
Rye
Rye isn’t really a replacement for any existing tool. It’s more of a one-stop-shop for Python project management, combining the functionality of PyEnv, venv, Poetry, and Build/Twine. It lets you isolate environments for a Python project, manage its dependencies, build, and publish the package.
uv
uv is more or less a drop-in replacement for pip and pip-tools. Besides being extremely fast, it has also recently added useful features, such as time restrictions using --exclude-newer.
Mypy in Rust?
Unfortunately not a thing yet. Given how slow mypy is it would be a great addition to the tools mentioned above.
Maturin
Maturin was formerly part of PyO3 but is now a standalone tool that lets you publish pure Rust crates as PyPI packages with minimal effort.
PyO3: a Bridge Between Rust and Python
PyO3 (read: Pythonium Trioxide) are Rust bindings for Python. Its most important function is enabling the creation of native Python extension modules. It also works in the opposite direction, allowing the calling of Python code from Rust and even creating fully embedded Python code.
Using PyO3 to Create a Python Module
A minimal example of a Python module implemented in Rust:
use pyo3::prelude::*;
/// Formats the sum of two numbers as string.
#[pyfunction]
fn sum_as_string(a: usize, b: usize) -> PyResult<String> {
Ok((a + b).to_string())
}
/// A Python module implemented in Rust. The name of this function must match
/// the `lib.name` setting in the `Cargo.toml`, else Python will not be able to
/// import the module.
#[pymodule]
fn string_sum(m: &Bound<'_, PyModule>) -> PyResult<()> {
m.add_function(wrap_pyfunction!(sum_as_string, m)?)?;
Ok(())
}After building, a string_sum.so file is created, which can then be used as a Python module directly:
>>> import string_sum
>>> string_sum.sum_as_string(5, 20)
'25'Using PyO3 to Access Python from within Rust
An example which uses standard library modules, but you can access your own modules the same way:
use pyo3::prelude::*;
use pyo3::types::IntoPyDict;
fn main() -> PyResult<()> {
Python::with_gil(|py| {
let sys = py.import_bound("sys")?;
let version: String = sys.getattr("version")?.extract()?;
let locals = [("os", py.import_bound("os")?)].into_py_dict_bound(py);
let code = "os.getenv('USER') or os.getenv('USERNAME') or 'Unknown'";
let user: String = py.eval_bound(code, None, Some(&locals))?.extract()?;
println!("Hello {}, I'm Python {}", user, version);
Ok(())
})
}Using PyO3 to Create Fully Embedded Python Module
A slightly more advanced example creates a Python module that is fully embedded within Rust code:
use pyo3::{prelude::*, types::IntoPyDict};
Python::with_gil(|py| {
let activators = PyModule::from_code_bound(
py,
r#"
def relu(x):
"""see https://en.wikipedia.org/wiki/Rectifier_(neural_networks)"""
return max(0.0, x)
def leaky_relu(x, slope=0.01):
return x if x >= 0 else x * slope
"#,
"activators.py",
"activators",
)?;
let relu_result: f64 = activators.getattr("relu")?.call1((-1.0,))?.extract()?;
assert_eq!(relu_result, 0.0);
let kwargs = [("slope", 0.2)].into_py_dict_bound(py);
let lrelu_result: f64 = activators
.getattr("leaky_relu")?
.call((-1.0,), Some(&kwargs))?
.extract()?;
assert_eq!(lrelu_result, -0.2);
})
Polars: Pandas, but in Rust
Polars is a very practical example of what can be achieved by implementing a performance-critical Python library in Rust. Polars is a blazing fast alternative to Pandas and NumPy, up to 30x faster than the former. Yet it’s easily installed with just pip install polars, and its interface is fully Pythonic. Polars achieves similar performance to NumPy; however, NumPy is a combination of code written in C, C++, Cython, and Fortran — hardly a thing you’d look forward to maintain.
The Future
I expect the adoption of Rust in the Python community to grow. The niche of optimizing highly performance-critical code was traditionally occupied by C, C++, and Cython, all of which are significantly harder for Pythonistas to work with. Due to its interoperability with Python and substantial performance benefits, Rust is already being adopted by data-heavy organizations, such as those dealing with data science or artificial intelligence.