Preserve unused pointer to address casts Fixes #97421. cc `@RalfJung`
For more information about how rustc works, see the rustc dev guide.