Cryptography relies on a core principle: secrecy of key material. This is what Hardware Security Modules (HSMs) offer: isolation of cryptographic keys and of operations using those keys, like digital signature. HSMs are computers dedicated to cryptographic key lifecycle operations: generation, storage, and usage.

Some HSM models also support the integration of custom software, to run business logic or integrate cryptographic schemes not supported by the HSM. Such custom code is typically written in the C language, prone to memory corruption bugs such as buffer overflows. This class of bugs is the root cause behind most security flaws; about 70% according to a 2019 Microsoft survey. In an HSM, software security flaws could have disastrous consequences.

This is where the Rust language brings tremendous value. Rust is by design memory safe, which (almost) eliminates the risk of memory corruption bugs, such as use-after-free and buffer overflows. Integrating Rust software in an HSM would thus drastically reduce the attack surface caused by memory corruption, without compromising performance.

In this article, we will explore the feasibility of integrating Rust components in an HSM, discuss the motivations behind this project as well as the constraints inherent to this environment, and finally demonstrate how to build and run Rust code in a Thalès ProtectServer 3 HSM.

Supporting material is published as open-source code at https://github.com/taurushq-io/FM-Rust-PoC.

Custom in-HSM code: why and how

The HSM’s firmware provides the vendor’s software stack, including a stripped Linux, bootloader components, and software libraries, including cryptographic logic. The vendor firmware also defines the PKCS#11 interface of the HSM, a standardized cryptographic API also known as Cryptoki (for Cryptographic Token Interface) maintained by Oasis. Cryptoki is a C-based API used by client applications to interact with the HSM.

To integrate custom code, Thalès defined the concept of functionality module (FM). An FM can extend the HSM’s capabilities beyond the standard Cryptoki functions. FM code can also implement various hardening mechanisms, as Taurus presented at Black Hat 2024. For example:

• Support for blockchain-specific signatures, hash functions, and key derivation schemes.

• Enforcement of the HSM secure configuration, such as PKCS#11 key attributes.

• Attack surface reduction, by patching PKCS#11 functions.

• Implementation of an extended role-based access control model.

• Integration of business logic, in our case a transaction policy engine.

To create an FM, Thalès provides an SDK and guidelines to create custom code and compile C code for the HSM platform. An FM thus relies on the headers, libraries, and API of the SDK. The vendor firmware manages the integration of the FM into the HSM’s and its interaction with HSM components like the Cryptoki API and the low-level storage (SMFS).

The diagram below shows how messages and Cryptoki function calls are handled from the host to the HSM.

The custom code of an FM runs within the same secure and isolated environment as the built-in components. Nevertheless, the FM also increases the attack surface, as memory corruption bugs could potentially be exploited to compromise the HSM code’s integrity and the confidentiality of its memory.

Why Rust

Rust is often introduced as the emerging system programming language. Organizations such as Microsoft, Google, and even the Linux kernel (not without resistance) integrated some Rust code, to get memory safety without a performance penalty. Other popular memory-safe languages come with a performance cost:

• Interpreted languages like Java and Python are slowed down by the virtual machine execution and garbage collector (GC) approach to (safe) memory management.

• The Go language runs a GC for memory management.

• The Swift language uses reference counting, with notable runtime overhead.

And none of those languages is designed to be a system language like Rust is.

Indeed, GC heavily impacts performance as it checks used memory pointers at runtime. Most GCs periodically pause the running program to collect the unused memory, leading to non-deterministic execution time, a major drawback for systems programming. They also choose which memory to free by scanning the heap. This costs CPU cycles and affects the runtime performance.

Rust uses a different approach to memory management, with two key concepts: ownership and borrow checking, ideas that can be traced back at least to Girard’s 1987 linear logic article. Both of these tools run (most of the time) at compile time. Programs performance is predictable and undegraded. Although data leaks are theoretically possible in safe Rust, they are difficult to trigger and unlikely to happen by accident.

Rust also benefits from a rich toolchain, thanks to the cargo suite and associated utilities. Rust can be cross-compiled for various operating systems and processor architectures, thanks to its LLVM-based back-end. This will prove useful for our HSM integration.

Integrating Rust in an HSM

We’ll describe how to run Rust in a Thalès ProtectServer 3 (PSE3). This approach won’t be suitable for other HSMs: most do not support custom code.

Approach

Instead of adapting the Thalès toolchain to process Rust code rather than C, we will create a static library from Rust code and link it to the functional module C code. The PSE3 HSM contains a PowerPC 476 core, a 32-bit CPU operating in big-endian convention (while most modern processors are little-endian). It runs a minimal Linux-based operating system. The Rust logic must therefore be generated for that platform, using the cross-compilation capabilities of the Rust toolchain.

From an FM entirely written in C, the most sensitive and complex operations may thus be replaced by Rust code.

Rust code requirements

The HSM, although considered PC-based, is an embedded device. To run reliably, the Rust program must not make assumptions about the underlying systems and its libraries. It must be run on what Rust calls a no_std environment, where the Rust binary will only load its minimal core components. As explained in The Embedded Rust Book, in a no_std environment, the runtime sets up the stack overflow protection and spawns the main thread before invoking the main function.

To run in such an environment, the Rust library code (in its file lib.rs) must include the no_std attribute and must define a panic handler function, typically as follows:

The last essential step for building Rust code compatible with C is disabling name mangling. The no_mangle attribute must be used on any items that will be used by the C program, especially functions.

Cross-compilation

When it comes to cross-compiling, a convenient tool is cross. It uses the same command-line interface as cargo. Cross compiles code for another platform (OS, CPU architecture). It sets up all the necessary environment in Docker containers. Given the specifications of the PSE3, the required target is powerpc-unknown-linux-gnu.

Example implementation

As a minimal proof of concept, we extend one of the sample functional modules with a static library compiled from Rust, then we run it in a PSE3 HSM.

Creating the static library

We write a trivial library re-implementing basic arithmetic operations (addition, multiplication, exponentiation) in Rust. This way we only check if pure Rust code can be executed without using extern crates:

Notice that no_mangle is considered “unsafe” here. This is because if two symbols have the same name without mangling there will be collisions. Here, “unsafe” highlights the safety requirements rather than a real unsafe code.

The configuration file (Cargo.toml) of the library is as follows:

The library may now be built. After having added the target with rustup, we cross-compiled using cross as explained. The resulting file is libadd_ppc.a that we linked to an FM as explained below.

As promised, cross is convenient as it uses the same command-line interface as cargo. From the root of the project, after having added the PowerPC target with rustup, the only command to write is the build command with the release tag and the specified target.

Building, signing and uploading the FM

To make the proof-of-concept easily reproducible, we adapt the xorsign FM example from the Thalès SDK. It returns a single output from its simple program, and is easy to modify. The original output is an eight-byte signature. We will replace it with an output from the Rust library.

First, we copy the static library built from Rust to the dependencies directory of the FM. In the C file, we declare the Rust functions with the keyword extern. We can then call the function, for example pow() below:

We then built and deployed the FM following the same steps as for a pure C FM: generation of a .fm file, signature, and upload to the HSM. This requires the SDK and a connection to the HSM.

We verify that the pow() function behaves as intended, by sending a request to the FM and reading the result in the response: 0x0000020, or 16 = 2⁵.

Our code is available in https://github.com/taurushq-io/FM-Rust-PoC, to reproduce this proof of concept.

Taking it further

The methods shown above allow a developer to integrate Rust in its patched or custom functions, but we cannot use the Cryptoki functions with this setup. The Rust code is completely unaware of the C environment wrapping it. Let’s introduce foreign function interfaces (FFIs). They are mechanisms allowing Rust code to interact with other programming languages using their routines. Hence, calling Cryptoki from Rust is possible. The only requirement is having the types and functions definitions written in the source directory of the static library. Obviously, as Cryptoki has big header files we need a tool to translate them into Rust definitions.

Too lazy for binding

Generating FFI automatically from C headers is what the bindgen crate is made for. It is a build-dependency, which means it is executed once at compile time, and requires specifying a build.rs file in the root of your Rust project. This file must search for the target header file and translate all the definitions from that file into a bindings.rs file in the source directory. Then, bindings.rs acts as a header file for the Rust modules.

See build.rs in our source code for details.

Dynamic allocations

The no_std context enforces developers to rely entirely on stack allocations as there is no heap allocator in the core library. This could restrict the usage of Rust without the capability to dynamically allocate data structures such as vectors or boxes. Hopefully, as mentioned before, the PSE3 has a partial glibc library which offers dynamic allocation functions. It is possible to define a custom global allocator in Rust no_std and use it for heap allocation.

The libc crate is an FFI binding to the C system library and as malloc and free are defined in the HSM we can use them in safe Rust. Offering all the possibilities of dynamic allocations within the Rust language.

Rust Cryptoki

Having all the definitions from the Cryptoki header, it is possible to call PKCS#11 functions from Rust. This allows having an, almost, full FM logic from a Rust library. We can now define custom functions entirely in Rust or even patch other functions. We will demonstrate how we used these definitions to patch a signing function by refactoring our last example.

Making the firmware speak Rust

FM samples from the SDK use Cprov tables to map the original functions with the patched ones and warning the firmware to use the right version.

In the same source file as the arithmetic functions previously written, we rewrote the signing function from the sample FM as a Rust version. This code contains calls to Cryptoki functions and hence executes C code from the firmware.

On the FM source, we overwrote the Cprov table and replaced the C signing function with the Rust one. The FM will call directly our Rust code when the original signing function from Cryptoki is called.

Again, we built, signed, and deployed the FM on the HSM from the container used earlier. Running the test at this stage produced a signature in the same format as the original version provided by the SDK.

It validated that our Rust Cryptoki function worked as intended and that it was feasible to integrate Rust in the HSM environment efficiently while maintaining compatibility with the existing C API.

Unsafe Rust?

However, it would have been too easy if we could just call C functions from Rust and forget about all memory issues. Introducing C functions in Rust code implies (most of the time) raw pointers and in fact, raw pointers in Rust are considered “unsafe”, but not in the same manner as the no_mangle attribute seen before. It is truly unsafe, because raw pointers cannot be checked at compile time by the borrow checker and therefore, they may lead to memory issues. This seems to break everything we’ve built until now.

Our case is not an exception. A lot of Cryptoki functions use context or session variables, themselves raw pointers to structures. Every call to such a function must be contained in an unsafe block, as well as pointer dereferencing and other pointer-related operations.

Unsafe blocks mark areas where the compiler cannot promise correctness. The developer has to provide this guarantee and be especially careful within this region of code.

Hence, the integration of such blocks does not make Rust irrelevant. Imagine using Rust for only some parts of your FM, where the logic is particularly complex and the data is extremely sensitive. You have two main choices when it comes to handling unsafe blocks.

1. Keep it as it is and use unsafe blocks as shown before.

2. Wrap unsafe logic in a safe Rust function which performs pointer validations and exposes a safe interface.

Is it really a solution? Well, the way unsafe Rust code is written emphasizes the dangerous parts. Once the pointers are dereferenced or validated in unsafe code you can use perfectly safe Rust and don’t worry about memory safety anymore. It is not a solution; it is a tool. Furthermore, Miri is a compiler tool that can be used to detect undefined behavior in unsafe code, making it even safer.

Conclusion

Rust is, on paper, a great solution to handle critical environment software. In this article we showed a simple way to integrate Rust in PSE3 HSMs, via a static library. Such Rust components offer security guarantees that C code cannot fulfill by design. It is even possible to have completely safe code if not using C FFIs, but the applications of such code in an FM are restricted and we must, at some point, rely on FFIs.

That’s where unsafe code appears. As we saw, Rust suffers from the same issues as C, but these unsafe blocks put a scope on the dangerous code and help maintainers focus on sensitive parts.

It is in Taurus' interest to reduce the private-key compromise risks. Narrowing the attack surface and the insecure memory codebase would be a solution to avoid resulting loss of funds. Knowing where things could go wrong, typically in unsafe blocks, is a notable maintenance shortcut.

Finally, while Rust demonstrates strong potential for improving safety and reliability, replacing C might be challenging in the short term as it is deeply integrated in the HSMs. However, integrating Rust as a complement to C within the HSM was demonstrated as feasible. Other vendors such as Securosys allow direct integration of Java through Docker containers or even Go integration with the USB Armory and the TamaGo framework from F-Secure. Exploring these paths could increase security in a system already remarkably efficient.