Matt Bolitho


Rust's Null Pointer Optimisation and NonZero<T>

Rust has two awesome types, std::ptr::NonNull<T> and std::num::NonZero<T>, which allow you to encode constraints on values without changing the underlying representation. Namely, in these cases, that a pointer is never null or that a numeric type never has a value of 0, respectively.

Why is this cool? It lets us use types expressively to perform some optimisations, which is a win for performance, readability, and correctness.

Null Pointer Optimisation

Starting with the optimisation side, a key feature enabled by such types is Rust’s null pointer optimisation. This allows certain types to reuse values that can appear in a type’s binary representation, but are not considered semantically valid, for other representations.

In Rust, Option<T> is an enum with two variants: Some(T) (the value if it exists) or None (no value). The size of an instance of Option<T> is usually sizeof(T) + 1. The +1 here is a byte of extra storage to track whether an instance of Option<T> contains a value or not. Clearly, if we assume all possible bit patterns of T’s binary representation are valid instances of T, then we cannot do anything clever - we must add the extra byte for tracking.

However, if we know that the all 0 bit pattern is never used, then we can do something clever! We can transmute between the all 0 bit pattern of our non-zero types to Option::<T>::None (because T is guaranteed to not be 0 in safe Rust), which allows us to elide the extra storage requirement for variant tracking.

NonZero<T>

Let’s look at some code that illustrates the null pointer optimisation. We’ll only consider using Option<NonZero<T>>; it should be fairly apparent how it generally applies to NonNull<T> as well.

Let’s compare the sizes of Option instances when used with both u8 and NonZeroU8 inner types.

 1use std::{mem::size_of, num::NonZeroU8};
 2
 3fn main() {
 4    println!("sizeof(u8)                = {} byte", size_of::<u8>());
 5    println!("sizeof(Option<u8>)        = {} bytes", size_of::<Option<u8>>());
 6    println!("sizeof(NonZeroU8)         = {} byte", size_of::<NonZeroU8>());
 7    println!("sizeof(Option<NonZeroU8>) = {} byte", size_of::<Option<NonZeroU8>>());
 8}
 9
10// Output:
11// sizeof(u8)                = 1 byte
12// sizeof(Option<u8>)        = 2 bytes
13// sizeof(NonZeroU8)         = 1 byte
14// sizeof(Option<NonZeroU8>) = 1 byte

As previously discussed, we incur a single byte of overhead for storing whether the value is None or Some. With Option<NonZeroU8>, reusing the all 0 bit pattern for None is allowing the Rust compiler to elide this storage overhead.

Ok this sounds great in theory, but what are the consequences of applying this approach more widely throughout a codebase?

Firstly, we are not using trivial types anymore, which may have some ramifications for API consumers. Naturally, they would need to construct values of NonZeroU8 (for example) from their existing u8 values, which can make consumption of our APIs less ergonomic.

This also has non-zero runtime cost in the general case. The conversion of a u8 to NonZeroU8 is clearly fallible when the input is 0, so we must return a Result when we construct instances of such types at runtime from the underlying integral type. Naturally, this implies a runtime check of the returned Result. It is worth noting that NonZero<T>::new() is a const fn, so there should be no cost to convert compile time constants to NonZero<T> variants.

1let one = 1u8;
2let nonzero_one = NonZeroU8::new(one).unwrap();
3
4let zero = 0u8;
5let error = NonZeroU8::new(zero); // Will be Err(...)

Sounds great! Can I use this for my custom types?

Not easily, at the time of writing at least.

Support for NonZero* types is implemented in the compiler via the trait ZeroablePrimitive. You’re most likely to find this by reading the “Note on generic usage” section from the NonZero<T> docs, which states:

NonZero<T> can only be used with some standard library primitive types (such as u8, i32, etc.). The type parameter T must implement the internal trait ZeroablePrimitive, which is currently permanently unstable and cannot be implemented by users. Therefore, you cannot use NonZero<T> with your own types, nor can you implement traits for all NonZero<T>, only for concrete types.

Accessing ZeroablePrimitive is technically possible behind the unstable feature gate nonzero_internals, but let me save you a click. At the time of writing that link says:

This feature has no tracking issue, and is therefore likely internal to the compiler, not being intended for general use.

So could we use this feature in our own code if we really wanted to? Sure, if we were inclined to rely on a “not intended for general use” feature or fork the Rust compiler… But that is not a satisfying conclusion, so let’s keep digging!

What is this magic unstable implementation detail?

Just because we can’t easily make our own non-zero types doesn’t mean we can’t learn how this feature works under the hood. Because, as we just discussed, the implementation of null pointer optimisation for NonZero* types is an unstable implementation detail, we will look specifically at the implementation details in Rust 1.96.1 - the latest stable toolchain at the time of writing. I’ll also snip some extraneous text out of code samples for brevity.

Intuitively, we can imagine that we need to communicate the use of the all 0 bit pattern to not be a valid instance of a given type. Perhaps we can generalise this further to multiple invalid values or even ranges of invalid values.

With that in mind, let’s start by reading nonzero.rs from the tip of the unstable feature’s iceberg: ZeroablePrimitive.

1pub unsafe trait ZeroablePrimitive: Sized + Copy + private::Sealed {
2    /// A type like `Self` but with a niche that includes zero.
3    type NonZeroInner: Sized + Copy;
4}

Ok, so that’s just a wrapper around an inner type (which should be u8 in the case of NonZeroU8), but what is this “niche” that’s being described? It turns out the Rust compiler has a notion of so-called “pattern types”, which are types that carry a validity range. From this, we can guess that we should find a way to tie the NonZero part of the type to an integral type with the same size and alignment, whilst somehow communicating that the value can never be 0. The Rust compiler does it like this:

1#[repr(transparent)]
2#[rustc_nonnull_optimization_guaranteed]
3pub struct NonZero<T: ZeroablePrimitive>(T::NonZeroInner);

A strongly-typed wrapper over the zero-able inner type, which uses #[repr(transparent)] to match the size and alignment of the underlying integral type. The niche_types.rs contains the validity range definitions, implemented by a proc macro that expands the pattern type definition:

1define_valid_range_type! {
2    pub struct NonZeroU8Inner(u8 is 1..);
3    pub struct NonZeroU16Inner(u16 is 1..);
4    // etc.
5
6    pub struct NonZeroI8Inner(i8 is ..0 | 1..);
7    pub struct NonZeroI16Inner(i16 is ..0 | 1..);
8    // etc.
9}

The Rust compiler can use this range validity information all the way through the compilation pipeline. Specifically in the case of null pointer optimisation, when Rust solves for the layout of an enum, it is able to see the spare all 0 bit pattern and reserve it for the None variant. Code generation can exploit this too - we can potentially apply LLVM metadata such as nonnull to values so that LLVM’s optimiser can also make the same assumptions.

If you want to dig further into how the Rust compiler implements the above, I’d look at the files named layout.rs in both the rustc_abi and rustc_ty_utils crates.

Why is this worth thinking about?

If we can’t apply these pattern types effectively in our user code, what good is any of this to most Rust users?

Correctness in software has become a big topic this decade. I don’t believe that no one cared about correctness before, but software has had and continues to have a significant impact on society. We’ve seen calls for increased use of memory safe languages (mostly due to defects that can have real-world consequences) and an industry-wide move to generate swathes of code with LLMs at hitherto unseen frequencies, with potentially unknown consequences.

Programming language toolchains are reacting to this accordingly. Rust has the borrow checker to prevent some memory/temporal/concurrency bugs at compile time, the Fil-C project is researching garbage collected C, C++26 has introduced contracts, to name but a few examples.

Actually, if we hone in on the last example, C++26 contracts, let’s look at an example for safe division…

1int divide(int dividend, int divisor) pre(divisor != 0)
2{
3  return dividend / divisor;
4}

Ah! If we squint a bit, that could look a little like…

1fn divide(dividend: i32, divisor: core::num::NonZeroI32) -> i32 {
2  dividend / divisor.get()
3}

Now OK, the two code snippets are of course not the same, implementation-wise.

I just think it’s cool to see the orthogonal approaches of contract expressions and constrained types being used to achieve improved correctness in a similar way. Whilst it is arguable that NonZero types are not strictly aimed at correctness (but rather layout optimisation), it’s easy to see how we could apply constrained types to correctness more generally. As languages push different features aimed at improving correctness, I wonder how many approaches we will see and to what extent they will cross-pollinate.

Conclusion

We can use strong, expressive types not only to prove correctness in our programs but for some code generation wins!

I wonder if we will see idioms like pattern types becoming more user-focussed customisation points in future as we quest for safer, more performant software.

#Rust