Removes some unsafety and reduces the number of `usize` -> `ptr` transmutes which might be helpful for CHERI-like targets in the future.