Skip to main content

core/sync/
atomic.rs

1//! Atomic types
2//!
3//! Atomic types provide primitive shared-memory communication between
4//! threads, and are the building blocks of other concurrent
5//! types.
6//!
7//! This module defines atomic versions of a select number of primitive
8//! types, including [`AtomicBool`], [`AtomicIsize`], [`AtomicUsize`],
9//! [`AtomicI8`], [`AtomicU16`], etc.
10//! Atomic types present operations that, when used correctly, synchronize
11//! updates between threads.
12//!
13//! Atomic variables are safe to share between threads (they implement [`Sync`])
14//! but they do not themselves provide the mechanism for sharing and follow the
15//! [threading model](../../../std/thread/index.html#the-threading-model) of Rust.
16//! The most common way to share an atomic variable is to put it into an [`Arc`][arc] (an
17//! atomically-reference-counted shared pointer).
18//!
19//! [arc]: ../../../std/sync/struct.Arc.html
20//!
21//! Atomic types may be stored in static variables, initialized using
22//! the constant initializers like [`AtomicBool::new`]. Atomic statics
23//! are often used for lazy global initialization.
24//!
25//! ## Memory model for atomic accesses
26//!
27//! Rust atomics currently follow the same rules as [C++20 atomics][cpp], specifically the rules
28//! from the [`intro.races`][cpp-intro.races] section, without the "consume" memory ordering. Since
29//! C++ uses an object-based memory model whereas Rust is access-based, a bit of translation work
30//! has to be done to apply the C++ rules to Rust: whenever C++ talks about "the value of an
31//! object", we understand that to mean the resulting bytes obtained when doing a read. When the C++
32//! standard talks about "the value of an atomic object", this refers to the result of doing an
33//! atomic load (via the operations provided in this module). A "modification of an atomic object"
34//! refers to an atomic store.
35//!
36//! The end result is *almost* equivalent to saying that creating a *shared reference* to one of the
37//! Rust atomic types corresponds to creating an `atomic_ref` in C++, with the `atomic_ref` being
38//! destroyed when the lifetime of the shared reference ends. The main difference is that Rust
39//! permits concurrent atomic and non-atomic reads to the same memory as those cause no issue in the
40//! C++ memory model, they are just forbidden in C++ because memory is partitioned into "atomic
41//! objects" and "non-atomic objects" (with `atomic_ref` temporarily converting a non-atomic object
42//! into an atomic object).
43//!
44//! The most important aspect of this model is that *data races* are undefined behavior. A data race
45//! is defined as conflicting non-synchronized accesses where at least one of the accesses is
46//! non-atomic. Here, accesses are *conflicting* if they affect overlapping regions of memory and at
47//! least one of them is a write. (A `compare_exchange` or `compare_exchange_weak` that does not
48//! succeed is not considered a write.) They are *non-synchronized* if neither of them
49//! *happens-before* the other, according to the happens-before order of the memory model.
50//!
51//! The other possible cause of undefined behavior in the memory model are mixed-size accesses: Rust
52//! inherits the C++ limitation that non-synchronized conflicting atomic accesses may not partially
53//! overlap. In other words, every pair of non-synchronized atomic accesses must be either disjoint,
54//! access the exact same memory (including using the same access size), or both be reads.
55//!
56//! Each atomic access takes an [`Ordering`] which defines how the operation interacts with the
57//! happens-before order. These orderings behave the same as the corresponding [C++20 atomic
58//! orderings][cpp_memory_order]. For more information, see the [nomicon].
59//!
60//! [cpp]: https://en.cppreference.com/w/cpp/atomic
61//! [cpp-intro.races]: https://timsong-cpp.github.io/cppwp/n4868/intro.multithread#intro.races
62//! [cpp_memory_order]: https://en.cppreference.com/w/cpp/atomic/memory_order
63//! [nomicon]: ../../../nomicon/atomics.html
64//!
65//! ```rust,no_run undefined_behavior
66//! use std::sync::atomic::{AtomicU16, AtomicU8, Ordering};
67//! use std::mem::transmute;
68//! use std::thread;
69//!
70//! let atomic = AtomicU16::new(0);
71//!
72//! thread::scope(|s| {
73//!     // This is UB: conflicting non-synchronized accesses, at least one of which is non-atomic.
74//!     s.spawn(|| atomic.store(1, Ordering::Relaxed)); // atomic store
75//!     s.spawn(|| unsafe { atomic.as_ptr().write(2) }); // non-atomic write
76//! });
77//!
78//! thread::scope(|s| {
79//!     // This is fine: the accesses do not conflict (as none of them performs any modification).
80//!     // In C++ this would be disallowed since creating an `atomic_ref` precludes
81//!     // further non-atomic accesses, but Rust does not have that limitation.
82//!     s.spawn(|| atomic.load(Ordering::Relaxed)); // atomic load
83//!     s.spawn(|| unsafe { atomic.as_ptr().read() }); // non-atomic read
84//! });
85//!
86//! thread::scope(|s| {
87//!     // This is fine: `join` synchronizes the code in a way such that the atomic
88//!     // store happens-before the non-atomic write.
89//!     let handle = s.spawn(|| atomic.store(1, Ordering::Relaxed)); // atomic store
90//!     handle.join().expect("thread won't panic"); // synchronize
91//!     s.spawn(|| unsafe { atomic.as_ptr().write(2) }); // non-atomic write
92//! });
93//!
94//! thread::scope(|s| {
95//!     // This is UB: non-synchronized conflicting differently-sized atomic accesses.
96//!     s.spawn(|| atomic.store(1, Ordering::Relaxed));
97//!     s.spawn(|| unsafe {
98//!         let differently_sized = transmute::<&AtomicU16, &AtomicU8>(&atomic);
99//!         differently_sized.store(2, Ordering::Relaxed);
100//!     });
101//! });
102//!
103//! thread::scope(|s| {
104//!     // This is fine: `join` synchronizes the code in a way such that
105//!     // the 1-byte store happens-before the 2-byte store.
106//!     let handle = s.spawn(|| atomic.store(1, Ordering::Relaxed));
107//!     handle.join().expect("thread won't panic");
108//!     s.spawn(|| unsafe {
109//!         let differently_sized = transmute::<&AtomicU16, &AtomicU8>(&atomic);
110//!         differently_sized.store(2, Ordering::Relaxed);
111//!     });
112//! });
113//! ```
114//!
115//! # Portability
116//!
117//! All atomic types in this module are guaranteed to be [lock-free] if they're
118//! available. This means they don't internally acquire a global mutex. Atomic
119//! types and operations are not guaranteed to be wait-free. This means that
120//! operations like `fetch_or` may be implemented with a compare-and-swap loop.
121//!
122//! Atomic operations may be implemented at the instruction layer with
123//! larger-size atomics. For example some platforms use 4-byte atomic
124//! instructions to implement `AtomicI8`. Note that this emulation should not
125//! have an impact on correctness of code, it's just something to be aware of.
126//!
127//! The atomic types in this module might not be available on all platforms. The
128//! atomic types here are all widely available, however, and can generally be
129//! relied upon existing. Some notable exceptions are:
130//!
131//! * PowerPC and MIPS platforms with 32-bit pointers do not have `AtomicU64` or
132//!   `AtomicI64` types.
133//! * Legacy ARM platforms like ARMv4T and ARMv5TE have very limited hardware
134//!   support for atomics. The bare-metal targets disable this module
135//!   entirely, but the Linux targets [use the kernel] to assist (which comes
136//!   with a performance penalty). It's not until ARMv6K onwards that ARM CPUs
137//!   have support for load/store and Compare and Swap (CAS) atomics in hardware.
138//! * ARMv6-M targets (`thumbv6m-*`) only provide `load` and `store` operations, and do
139//!   not support Compare and Swap (CAS) operations, such as `swap`,
140//!   `fetch_add`, etc. Full CAS support is available on ARMv7-M and ARMv8-M
141//!   (`thumbv7m-*`, `thumbv7em*`, `thumbv8m.base-*` and `thumbv8m.main-*`).
142//!
143//! [use the kernel]: https://www.kernel.org/doc/Documentation/arm/kernel_user_helpers.txt
144//!
145//! Note that future platforms may be added that also do not have support for
146//! some atomic operations. Maximally portable code will want to be careful
147//! about which atomic types are used. `AtomicUsize` and `AtomicIsize` are
148//! generally the most portable, but even then they're not available everywhere.
149//! For reference, the `std` library requires `AtomicBool`s and pointer-sized atomics, although
150//! `core` does not.
151//!
152//! The `#[cfg(target_has_atomic)]` attribute can be used to conditionally
153//! compile based on the target's supported bit widths. It is a key-value
154//! option set for each supported size, with values "8", "16", "32", "64",
155//! "128", and "ptr" for pointer-sized atomics.
156//!
157//! [lock-free]: https://en.wikipedia.org/wiki/Non-blocking_algorithm
158//!
159//! # Atomic accesses to read-only memory
160//!
161//! In general, *all* atomic accesses on read-only memory are undefined behavior. For instance, attempting
162//! to do a `compare_exchange` that will definitely fail (making it conceptually a read-only
163//! operation) can still cause a segmentation fault if the underlying memory page is mapped read-only. Since
164//! atomic `load`s might be implemented using compare-exchange operations, even a `load` can fault
165//! on read-only memory.
166//!
167//! For the purpose of this section, "read-only memory" is defined as memory that is read-only in
168//! the underlying target, i.e., the pages are mapped with a read-only flag and any attempt to write
169//! will cause a page fault. In particular, an `&u128` reference that points to memory that is
170//! read-write mapped is *not* considered to point to "read-only memory". In Rust, almost all memory
171//! is read-write; the only exceptions are memory created by `const` items or `static` items without
172//! interior mutability, and memory that was specifically marked as read-only by the operating
173//! system via platform-specific APIs.
174//!
175//! As an exception from the general rule stated above, "sufficiently small" atomic loads with
176//! `Ordering::Relaxed` are implemented in a way that works on read-only memory, and are hence not
177//! undefined behavior. The exact size limit for what makes a load "sufficiently small" varies
178//! depending on the target:
179//!
180//! | `target_arch` | Size limit |
181//! |---------------|---------|
182//! | `x86`, `arm`, `loongarch32`, `mips`, `mips32r6`, `powerpc`, `riscv32`, `sparc`, `hexagon` | 4 bytes |
183//! | `x86_64`, `aarch64`, `loongarch64`, `mips64`, `mips64r6`, `powerpc64`, `riscv64`, `sparc64`, `s390x` | 8 bytes |
184//!
185//! Atomics loads that are larger than this limit as well as atomic loads with ordering other
186//! than `Relaxed`, as well as *all* atomic loads on targets not listed in the table, might still be
187//! read-only under certain conditions, but that is not a stable guarantee and should not be relied
188//! upon.
189//!
190//! If you need to do an acquire load on read-only memory, you can do a relaxed load followed by an
191//! acquire fence instead.
192//!
193//! # Examples
194//!
195//! A simple spinlock:
196//!
197//! ```ignore-wasm
198//! use std::sync::Arc;
199//! use std::sync::atomic::{AtomicUsize, Ordering};
200//! use std::{hint, thread};
201//!
202//! fn main() {
203//!     let spinlock = Arc::new(AtomicUsize::new(1));
204//!
205//!     let spinlock_clone = Arc::clone(&spinlock);
206//!
207//!     let thread = thread::spawn(move || {
208//!         spinlock_clone.store(0, Ordering::Release);
209//!     });
210//!
211//!     // Wait for the other thread to release the lock
212//!     while spinlock.load(Ordering::Acquire) != 0 {
213//!         hint::spin_loop();
214//!     }
215//!
216//!     if let Err(panic) = thread.join() {
217//!         println!("Thread had an error: {panic:?}");
218//!     }
219//! }
220//! ```
221//!
222//! Keep a global count of live threads:
223//!
224//! ```
225//! use std::sync::atomic::{AtomicUsize, Ordering};
226//!
227//! static GLOBAL_THREAD_COUNT: AtomicUsize = AtomicUsize::new(0);
228//!
229//! // Note that Relaxed ordering doesn't synchronize anything
230//! // except the global thread counter itself.
231//! let old_thread_count = GLOBAL_THREAD_COUNT.fetch_add(1, Ordering::Relaxed);
232//! // Note that this number may not be true at the moment of printing
233//! // because some other thread may have changed static value already.
234//! println!("live threads: {}", old_thread_count + 1);
235//! ```
236
237#![stable(feature = "rust1", since = "1.0.0")]
238#![cfg_attr(not(target_has_atomic_load_store = "8"), allow(dead_code))]
239#![cfg_attr(not(target_has_atomic_load_store = "8"), allow(unused_imports))]
240// Clippy complains about the pattern of "safe function calling unsafe function taking pointers".
241// This happens with AtomicPtr intrinsics but is fine, as the pointers clippy is concerned about
242// are just normal values that get loaded/stored, but not dereferenced.
243#![allow(clippy::not_unsafe_ptr_arg_deref)]
244
245use self::Ordering::*;
246use crate::cell::UnsafeCell;
247use crate::hint::spin_loop;
248use crate::intrinsics::AtomicOrdering as AO;
249use crate::mem::transmute;
250use crate::{fmt, intrinsics};
251
252#[unstable(
253    feature = "atomic_internals",
254    reason = "implementation detail which may disappear or be replaced at any time",
255    issue = "none"
256)]
257#[expect(missing_debug_implementations)]
258mod private {
259    #[cfg(target_has_atomic_load_store = "8")]
260    #[repr(C, align(1))]
261    #[ferrocene::prevalidated]
262    pub struct Align1<T>(T);
263    #[cfg(target_has_atomic_load_store = "16")]
264    #[repr(C, align(2))]
265    #[ferrocene::prevalidated]
266    pub struct Align2<T>(T);
267    #[cfg(target_has_atomic_load_store = "32")]
268    #[repr(C, align(4))]
269    #[ferrocene::prevalidated]
270    pub struct Align4<T>(T);
271    #[cfg(target_has_atomic_load_store = "64")]
272    #[repr(C, align(8))]
273    #[ferrocene::prevalidated]
274    pub struct Align8<T>(T);
275    #[cfg(any(target_has_atomic_load_store = "128", doc))]
276    #[repr(C, align(16))]
277    #[ferrocene::prevalidated]
278    pub struct Align16<T>(T);
279}
280
281/// A marker trait for primitive types which can be modified atomically.
282///
283/// This is an implementation detail for <code>[Atomic]\<T></code> which may disappear or be replaced at any time.
284//
285// # Safety
286//
287// Types implementing this trait must be primitives that can be modified atomically.
288//
289// The associated `Self::Storage` type must have the same size, but may have fewer validity
290// invariants or a higher alignment requirement than `Self`.
291#[unstable(
292    feature = "atomic_internals",
293    reason = "implementation detail which may disappear or be replaced at any time",
294    issue = "none"
295)]
296pub impl(self) unsafe trait AtomicPrimitive: Sized + Copy {
297    /// Temporary implementation detail.
298    type Storage: Sized;
299}
300
301macro impl_atomic_primitive {
302    (
303        @impl [$($T:ident)?] $Primitive:ty as $Storage:ident<$Operand:ty>,
304        $cfg:meta
305    ) => {
306        #[unstable(
307            feature = "atomic_internals",
308            reason = "implementation detail which may disappear or be replaced at any time",
309            issue = "none"
310        )]
311        #[cfg($cfg)]
312        unsafe impl $(<$T>)? AtomicPrimitive for $Primitive {
313            type Storage = private::$Storage<$Operand>;
314        }
315    },
316
317    (
318        [$($T:ident)?] $Primitive:ty as $Storage:ident<$Operand:ty>,
319        size($size:literal)
320    ) => {
321        impl_atomic_primitive!(
322            @impl [$($T)?] $Primitive as $Storage<$Operand>,
323            target_has_atomic_load_store = $size
324        );
325    },
326
327    (
328        [$($T:ident)?] $Primitive:ty as $Storage:ident<$Operand:ty>,
329        size($size:literal),
330        doc
331    ) => {
332        impl_atomic_primitive!(
333            @impl [$($T)?] $Primitive as $Storage<$Operand>,
334            any(target_has_atomic_load_store = $size, doc)
335        );
336    },
337}
338
339impl_atomic_primitive!([] bool as Align1<u8>, size("8"));
340impl_atomic_primitive!([] i8 as Align1<i8>, size("8"));
341impl_atomic_primitive!([] u8 as Align1<u8>, size("8"));
342impl_atomic_primitive!([] i16 as Align2<i16>, size("16"));
343impl_atomic_primitive!([] u16 as Align2<u16>, size("16"));
344impl_atomic_primitive!([] i32 as Align4<i32>, size("32"));
345impl_atomic_primitive!([] u32 as Align4<u32>, size("32"));
346impl_atomic_primitive!([] i64 as Align8<i64>, size("64"));
347impl_atomic_primitive!([] u64 as Align8<u64>, size("64"));
348impl_atomic_primitive!([] i128 as Align16<i128>, size("128"), doc);
349impl_atomic_primitive!([] u128 as Align16<u128>, size("128"), doc);
350
351#[cfg(target_pointer_width = "16")]
352impl_atomic_primitive!([] isize as Align2<isize>, size("ptr"));
353#[cfg(target_pointer_width = "32")]
354impl_atomic_primitive!([] isize as Align4<isize>, size("ptr"));
355#[cfg(target_pointer_width = "64")]
356impl_atomic_primitive!([] isize as Align8<isize>, size("ptr"));
357
358#[cfg(target_pointer_width = "16")]
359impl_atomic_primitive!([] usize as Align2<usize>, size("ptr"));
360#[cfg(target_pointer_width = "32")]
361impl_atomic_primitive!([] usize as Align4<usize>, size("ptr"));
362#[cfg(target_pointer_width = "64")]
363impl_atomic_primitive!([] usize as Align8<usize>, size("ptr"));
364
365#[cfg(target_pointer_width = "16")]
366impl_atomic_primitive!([T] *mut T as Align2<*mut T>, size("ptr"));
367#[cfg(target_pointer_width = "32")]
368impl_atomic_primitive!([T] *mut T as Align4<*mut T>, size("ptr"));
369#[cfg(target_pointer_width = "64")]
370impl_atomic_primitive!([T] *mut T as Align8<*mut T>, size("ptr"));
371
372/// A memory location which can be safely modified from multiple threads.
373///
374/// This has the same size and bit validity as the underlying type `T`. However,
375/// the alignment of this type is always equal to its size, even on targets where
376/// `T` has alignment less than its size.
377///
378/// For more about the differences between atomic types and non-atomic types as
379/// well as information about the portability of this type, please see the
380/// [module-level documentation].
381///
382/// **Note:** This type is only available on platforms that support atomic loads
383/// and stores of `T`.
384///
385/// [module-level documentation]: crate::sync::atomic
386#[unstable(feature = "generic_atomic", issue = "130539")]
387#[repr(C)]
388#[rustc_diagnostic_item = "Atomic"]
389#[ferrocene::prevalidated]
390pub struct Atomic<T: AtomicPrimitive> {
391    v: UnsafeCell<T::Storage>,
392}
393
394#[stable(feature = "rust1", since = "1.0.0")]
395unsafe impl<T: AtomicPrimitive> Send for Atomic<T> {}
396#[stable(feature = "rust1", since = "1.0.0")]
397unsafe impl<T: AtomicPrimitive> Sync for Atomic<T> {}
398
399// Some architectures don't have byte-sized atomics, which results in LLVM
400// emulating them using a LL/SC loop. However for AtomicBool we can take
401// advantage of the fact that it only ever contains 0 or 1 and use atomic OR/AND
402// instead, which LLVM can emulate using a larger atomic OR/AND operation.
403//
404// This list should only contain architectures which have word-sized atomic-or/
405// atomic-and instructions but don't natively support byte-sized atomics.
406#[cfg(target_has_atomic = "8")]
407const EMULATE_ATOMIC_BOOL: bool = cfg!(any(
408    target_arch = "riscv32",
409    target_arch = "riscv64",
410    target_arch = "loongarch32",
411    target_arch = "loongarch64"
412));
413
414/// A boolean type which can be safely shared between threads.
415///
416/// This type has the same size, alignment, and bit validity as a [`bool`].
417///
418/// **Note**: This type is only available on platforms that support atomic
419/// loads and stores of `u8`.
420#[cfg(target_has_atomic_load_store = "8")]
421#[stable(feature = "rust1", since = "1.0.0")]
422pub type AtomicBool = Atomic<bool>;
423
424#[cfg(target_has_atomic_load_store = "8")]
425#[stable(feature = "rust1", since = "1.0.0")]
426impl Default for AtomicBool {
427    /// Creates an `AtomicBool` initialized to `false`.
428    #[inline]
429    fn default() -> Self {
430        Self::new(false)
431    }
432}
433
434/// A raw pointer type which can be safely shared between threads.
435///
436/// This type has the same size and bit validity as a `*mut T`.
437///
438/// **Note**: This type is only available on platforms that support atomic
439/// loads and stores of pointers. Its size depends on the target pointer's size.
440#[cfg(target_has_atomic_load_store = "ptr")]
441#[stable(feature = "rust1", since = "1.0.0")]
442pub type AtomicPtr<T> = Atomic<*mut T>;
443
444#[cfg(target_has_atomic_load_store = "ptr")]
445#[stable(feature = "rust1", since = "1.0.0")]
446impl<T> Default for AtomicPtr<T> {
447    /// Creates a null `AtomicPtr<T>`.
448    fn default() -> AtomicPtr<T> {
449        AtomicPtr::new(crate::ptr::null_mut())
450    }
451}
452
453/// Atomic memory orderings
454///
455/// Memory orderings specify the way atomic operations synchronize memory.
456/// In its weakest [`Ordering::Relaxed`], only the memory directly touched by the
457/// operation is synchronized. On the other hand, a store-load pair of [`Ordering::SeqCst`]
458/// operations synchronize other memory while additionally preserving a total order of such
459/// operations across all threads.
460///
461/// Rust's memory orderings are [the same as those of
462/// C++20](https://en.cppreference.com/w/cpp/atomic/memory_order).
463///
464/// For more information see the [nomicon].
465///
466/// [nomicon]: ../../../nomicon/atomics.html
467#[stable(feature = "rust1", since = "1.0.0")]
468#[derive(Copy, Clone, Debug, Eq, PartialEq, Hash)]
469#[non_exhaustive]
470#[rustc_diagnostic_item = "Ordering"]
471#[ferrocene::prevalidated]
472pub enum Ordering {
473    /// No ordering constraints, only atomic operations.
474    ///
475    /// Corresponds to [`memory_order_relaxed`] in C++20.
476    ///
477    /// [`memory_order_relaxed`]: https://en.cppreference.com/w/cpp/atomic/memory_order#Relaxed_ordering
478    #[stable(feature = "rust1", since = "1.0.0")]
479    Relaxed,
480    /// When coupled with a store, all previous operations become ordered
481    /// before any load of this value with [`Acquire`] (or stronger) ordering.
482    /// In particular, all previous writes become visible to all threads
483    /// that perform an [`Acquire`] (or stronger) load of this value.
484    ///
485    /// Notice that using this ordering for an operation that combines loads
486    /// and stores leads to a [`Relaxed`] load operation!
487    ///
488    /// This ordering is only applicable for operations that can perform a store.
489    ///
490    /// Corresponds to [`memory_order_release`] in C++20.
491    ///
492    /// [`memory_order_release`]: https://en.cppreference.com/w/cpp/atomic/memory_order#Release-Acquire_ordering
493    #[stable(feature = "rust1", since = "1.0.0")]
494    Release,
495    /// When coupled with a load, if the loaded value was written by a store operation with
496    /// [`Release`] (or stronger) ordering, then all subsequent operations
497    /// become ordered after that store. In particular, all subsequent loads will see data
498    /// written before the store.
499    ///
500    /// Notice that using this ordering for an operation that combines loads
501    /// and stores leads to a [`Relaxed`] store operation!
502    ///
503    /// This ordering is only applicable for operations that can perform a load.
504    ///
505    /// Corresponds to [`memory_order_acquire`] in C++20.
506    ///
507    /// [`memory_order_acquire`]: https://en.cppreference.com/w/cpp/atomic/memory_order#Release-Acquire_ordering
508    #[stable(feature = "rust1", since = "1.0.0")]
509    Acquire,
510    /// Has the effects of both [`Acquire`] and [`Release`] together:
511    /// For loads it uses [`Acquire`] ordering. For stores it uses the [`Release`] ordering.
512    ///
513    /// Notice that in the case of `compare_and_swap`, it is possible that the operation ends up
514    /// not performing any store and hence it has just [`Acquire`] ordering. However,
515    /// `AcqRel` will never perform [`Relaxed`] accesses.
516    ///
517    /// This ordering is only applicable for operations that combine both loads and stores.
518    ///
519    /// Corresponds to [`memory_order_acq_rel`] in C++20.
520    ///
521    /// [`memory_order_acq_rel`]: https://en.cppreference.com/w/cpp/atomic/memory_order#Release-Acquire_ordering
522    #[stable(feature = "rust1", since = "1.0.0")]
523    AcqRel,
524    /// Like [`Acquire`]/[`Release`]/[`AcqRel`] (for load, store, and load-with-store
525    /// operations, respectively) with the additional guarantee that all threads see all
526    /// sequentially consistent operations in the same order.
527    ///
528    /// Corresponds to [`memory_order_seq_cst`] in C++20.
529    ///
530    /// [`memory_order_seq_cst`]: https://en.cppreference.com/w/cpp/atomic/memory_order#Sequentially-consistent_ordering
531    #[stable(feature = "rust1", since = "1.0.0")]
532    SeqCst,
533}
534
535/// An [`AtomicBool`] initialized to `false`.
536#[cfg(target_has_atomic_load_store = "8")]
537#[stable(feature = "rust1", since = "1.0.0")]
538#[deprecated(
539    since = "1.34.0",
540    note = "the `new` function is now preferred",
541    suggestion = "AtomicBool::new(false)"
542)]
543#[expect(clippy::declare_interior_mutable_const, reason = "legacy atomic initializer")]
544pub const ATOMIC_BOOL_INIT: AtomicBool = AtomicBool::new(false);
545
546#[cfg(target_has_atomic_load_store = "8")]
547impl AtomicBool {
548    /// Creates a new `AtomicBool`.
549    ///
550    /// # Examples
551    ///
552    /// ```
553    /// use std::sync::atomic::AtomicBool;
554    ///
555    /// let atomic_true = AtomicBool::new(true);
556    /// let atomic_false = AtomicBool::new(false);
557    /// ```
558    #[inline]
559    #[stable(feature = "rust1", since = "1.0.0")]
560    #[rustc_const_stable(feature = "const_atomic_new", since = "1.24.0")]
561    #[must_use]
562    #[ferrocene::prevalidated]
563    pub const fn new(v: bool) -> AtomicBool {
564        // SAFETY:
565        // `Atomic<T>` is essentially a transparent wrapper around `T`.
566        unsafe { transmute(v) }
567    }
568
569    /// Creates a new `AtomicBool` from a pointer.
570    ///
571    /// # Examples
572    ///
573    /// ```
574    /// use std::sync::atomic::{self, AtomicBool};
575    ///
576    /// // Get a pointer to an allocated value
577    /// let ptr: *mut bool = Box::into_raw(Box::new(false));
578    ///
579    /// assert!(ptr.cast::<AtomicBool>().is_aligned());
580    ///
581    /// {
582    ///     // Create an atomic view of the allocated value
583    ///     let atomic = unsafe { AtomicBool::from_ptr(ptr) };
584    ///
585    ///     // Use `atomic` for atomic operations, possibly share it with other threads
586    ///     atomic.store(true, atomic::Ordering::Relaxed);
587    /// }
588    ///
589    /// // It's ok to non-atomically access the value behind `ptr`,
590    /// // since the reference to the atomic ended its lifetime in the block above
591    /// assert_eq!(unsafe { *ptr }, true);
592    ///
593    /// // Deallocate the value
594    /// unsafe { drop(Box::from_raw(ptr)) }
595    /// ```
596    ///
597    /// # Safety
598    ///
599    /// * `ptr` must be aligned to `align_of::<AtomicBool>()` (note that this is always true, since
600    ///   `align_of::<AtomicBool>() == 1`).
601    /// * `ptr` must be [valid] for both reads and writes for the whole lifetime `'a`.
602    /// * You must adhere to the [Memory model for atomic accesses]. In particular, it is not
603    ///   allowed to mix conflicting atomic and non-atomic accesses, or atomic accesses of different
604    ///   sizes, without synchronization.
605    ///
606    /// [valid]: crate::ptr#safety
607    /// [Memory model for atomic accesses]: self#memory-model-for-atomic-accesses
608    #[inline]
609    #[stable(feature = "atomic_from_ptr", since = "1.75.0")]
610    #[rustc_const_stable(feature = "const_atomic_from_ptr", since = "1.84.0")]
611    pub const unsafe fn from_ptr<'a>(ptr: *mut bool) -> &'a AtomicBool {
612        // SAFETY: guaranteed by the caller
613        unsafe { &*ptr.cast() }
614    }
615
616    /// Creates a new pointer to `AtomicBool` from a pointer.
617    ///
618    /// This is useful if you want to do volatile atomic accesses, and thus avoid creating
619    /// a reference to the destination.
620    #[inline]
621    #[unstable(feature = "atomic_volatile", issue = "158947")]
622    pub const fn from_ptr_raw(ptr: *mut bool) -> *const AtomicBool {
623        ptr.cast_const().cast()
624    }
625
626    /// Returns a mutable reference to the underlying [`bool`].
627    ///
628    /// This is safe because the mutable reference guarantees that no other threads are
629    /// concurrently accessing the atomic data.
630    ///
631    /// # Examples
632    ///
633    /// ```
634    /// use std::sync::atomic::{AtomicBool, Ordering};
635    ///
636    /// let mut some_bool = AtomicBool::new(true);
637    /// assert_eq!(*some_bool.get_mut(), true);
638    /// *some_bool.get_mut() = false;
639    /// assert_eq!(some_bool.load(Ordering::SeqCst), false);
640    /// ```
641    #[inline]
642    #[stable(feature = "atomic_access", since = "1.15.0")]
643    #[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
644    pub const fn get_mut(&mut self) -> &mut bool {
645        // SAFETY: the mutable reference guarantees unique ownership.
646        unsafe { &mut *self.as_ptr() }
647    }
648
649    /// Gets atomic access to a `&mut bool`.
650    ///
651    /// # Examples
652    ///
653    /// ```
654    /// use std::sync::atomic::{AtomicBool, Ordering};
655    ///
656    /// let mut some_bool = true;
657    /// let a = AtomicBool::from_mut(&mut some_bool);
658    /// a.store(false, Ordering::Relaxed);
659    /// assert_eq!(some_bool, false);
660    /// ```
661    #[inline]
662    #[cfg(target_has_atomic_primitive_alignment = "8")]
663    #[stable(feature = "atomic_from_mut", since = "1.98.0")]
664    #[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
665    pub const fn from_mut(v: &mut bool) -> &mut Self {
666        // SAFETY: the mutable reference guarantees unique ownership, and
667        // alignment of both `bool` and `Self` is 1.
668        unsafe { &mut *(v as *mut bool as *mut Self) }
669    }
670
671    /// Gets non-atomic access to a `&mut [AtomicBool]` slice.
672    ///
673    /// This is safe because the mutable reference guarantees that no other threads are
674    /// concurrently accessing the atomic data.
675    ///
676    /// # Examples
677    ///
678    /// ```ignore-wasm
679    /// use std::sync::atomic::{AtomicBool, Ordering};
680    ///
681    /// let mut some_bools = [const { AtomicBool::new(false) }; 10];
682    ///
683    /// let view: &mut [bool] = AtomicBool::get_mut_slice(&mut some_bools);
684    /// assert_eq!(view, [false; 10]);
685    /// view[..5].copy_from_slice(&[true; 5]);
686    ///
687    /// std::thread::scope(|s| {
688    ///     for t in &some_bools[..5] {
689    ///         s.spawn(move || assert_eq!(t.load(Ordering::Relaxed), true));
690    ///     }
691    ///
692    ///     for f in &some_bools[5..] {
693    ///         s.spawn(move || assert_eq!(f.load(Ordering::Relaxed), false));
694    ///     }
695    /// });
696    /// ```
697    #[inline]
698    #[stable(feature = "atomic_from_mut", since = "1.98.0")]
699    #[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
700    pub const fn get_mut_slice(this: &mut [Self]) -> &mut [bool] {
701        // SAFETY: the mutable reference guarantees unique ownership.
702        unsafe { &mut *(this as *mut [Self] as *mut [bool]) }
703    }
704
705    /// Gets atomic access to a `&mut [bool]` slice.
706    ///
707    /// # Examples
708    ///
709    /// ```rust,ignore-wasm
710    /// use std::sync::atomic::{AtomicBool, Ordering};
711    ///
712    /// let mut some_bools = [false; 10];
713    /// let a = &*AtomicBool::from_mut_slice(&mut some_bools);
714    /// std::thread::scope(|s| {
715    ///     for i in 0..a.len() {
716    ///         s.spawn(move || a[i].store(true, Ordering::Relaxed));
717    ///     }
718    /// });
719    /// assert_eq!(some_bools, [true; 10]);
720    /// ```
721    #[inline]
722    #[cfg(target_has_atomic_primitive_alignment = "8")]
723    #[stable(feature = "atomic_from_mut", since = "1.98.0")]
724    #[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
725    pub const fn from_mut_slice(v: &mut [bool]) -> &mut [Self] {
726        // SAFETY: the mutable reference guarantees unique ownership, and
727        // alignment of both `bool` and `Self` is 1.
728        unsafe { &mut *(v as *mut [bool] as *mut [Self]) }
729    }
730
731    /// Consumes the atomic and returns the contained value.
732    ///
733    /// This is safe because passing `self` by value guarantees that no other threads are
734    /// concurrently accessing the atomic data.
735    ///
736    /// # Examples
737    ///
738    /// ```
739    /// use std::sync::atomic::AtomicBool;
740    ///
741    /// let some_bool = AtomicBool::new(true);
742    /// assert_eq!(some_bool.into_inner(), true);
743    /// ```
744    #[inline]
745    #[stable(feature = "atomic_access", since = "1.15.0")]
746    #[rustc_const_stable(feature = "const_atomic_into_inner", since = "1.79.0")]
747    pub const fn into_inner(self) -> bool {
748        // SAFETY:
749        // * `Atomic<T>` is essentially a transparent wrapper around `T`.
750        // * all operations on `Atomic<bool>` ensure that `T::Storage` remains
751        //   a valid `bool`.
752        unsafe { transmute(self) }
753    }
754
755    /// Loads a value from the bool.
756    ///
757    /// `load` takes an [`Ordering`] argument which describes the memory ordering
758    /// of this operation. Possible values are [`SeqCst`], [`Acquire`] and [`Relaxed`].
759    ///
760    /// # Panics
761    ///
762    /// Panics if `order` is [`Release`] or [`AcqRel`].
763    ///
764    /// # Examples
765    ///
766    /// ```
767    /// use std::sync::atomic::{AtomicBool, Ordering};
768    ///
769    /// let some_bool = AtomicBool::new(true);
770    ///
771    /// assert_eq!(some_bool.load(Ordering::Relaxed), true);
772    /// ```
773    #[inline]
774    #[stable(feature = "rust1", since = "1.0.0")]
775    #[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
776    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
777    #[ferrocene::prevalidated]
778    pub const fn load(&self, order: Ordering) -> bool {
779        // SAFETY: any data races are prevented by atomic intrinsics and the raw
780        // pointer passed in is valid because we got it from a reference.
781        unsafe {
782            atomic_load::<_, /* VOLATILE */ false>(self.v.get().cast::<u8>(), order) != 0
783        }
784    }
785
786    /// Perform a volatile atomic load from the bool.
787    ///
788    /// `load_volatile` takes an [`Ordering`] argument which describes the memory ordering
789    /// of this operation. Possible values are [`SeqCst`], [`Acquire`] and [`Relaxed`].
790    ///
791    #[doc = include_str!("./atomic_load_volatile.md")]
792    ///
793    /// # Safety
794    ///
795    /// Behavior is undefined if any of the following conditions are violated:
796    ///
797    /// * `self` must be [valid] for reads, or `self` must point to memory
798    ///   outside of all Rust allocations and reading from that memory must:
799    ///   - not trap, and
800    ///   - not cause any memory inside a Rust allocation to be modified.
801    ///
802    /// * Reading from `self` must produce a properly initialized value of type `bool`.
803    ///
804    /// [valid]: core::ptr#safety
805    ///
806    /// # Panics
807    ///
808    /// Panics if `order` is [`Release`] or [`AcqRel`].
809    #[inline]
810    #[unstable(feature = "atomic_volatile", issue = "158947")]
811    #[rustc_const_unstable(feature = "atomic_volatile", issue = "158947")]
812    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
813    pub const unsafe fn load_volatile(self: *const Self, order: Ordering) -> bool {
814        // SAFETY: follows from our own safety requirements.
815        unsafe {
816            atomic_load::<_, /* VOLATILE */ true>(self.cast::<u8>(), order) != 0
817        }
818    }
819
820    /// Stores a value into the bool.
821    ///
822    /// `store` takes an [`Ordering`] argument which describes the memory ordering
823    /// of this operation. Possible values are [`SeqCst`], [`Release`] and [`Relaxed`].
824    ///
825    /// # Panics
826    ///
827    /// Panics if `order` is [`Acquire`] or [`AcqRel`].
828    ///
829    /// # Examples
830    ///
831    /// ```
832    /// use std::sync::atomic::{AtomicBool, Ordering};
833    ///
834    /// let some_bool = AtomicBool::new(true);
835    ///
836    /// some_bool.store(false, Ordering::Relaxed);
837    /// assert_eq!(some_bool.load(Ordering::Relaxed), false);
838    /// ```
839    #[ferrocene::prevalidated]
840    #[inline]
841    #[stable(feature = "rust1", since = "1.0.0")]
842    #[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
843    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
844    #[lint::should_not_be_called_on_const_items]
845    pub const fn store(&self, val: bool, order: Ordering) {
846        // SAFETY: any data races are prevented by atomic intrinsics and the raw
847        // pointer passed in is valid because we got it from a reference.
848        unsafe {
849            atomic_store::<_, /* VOLATILE */ false>(self.v.get().cast::<u8>(), val as u8, order);
850        }
851    }
852
853    /// Performs a volatile atomic store into the bool.
854    ///
855    /// `store_volatile` takes an [`Ordering`] argument which describes the memory ordering
856    /// of this operation. Possible values are [`SeqCst`], [`Release`] and [`Relaxed`].
857    ///
858    #[doc = include_str!("./atomic_store_volatile.md")]
859    ///
860    /// # Safety
861    ///
862    /// Behavior is undefined if any of the following conditions are violated:
863    ///
864    /// * `self` must be either [valid] for writes, or `self` must point to memory
865    ///   outside of all Rust allocations and writing to that memory must:
866    ///   - not trap, and
867    ///   - not cause any memory inside a Rust allocation to be modified.
868    ///
869    /// [valid]: core::ptr#safety
870    ///
871    /// # Panics
872    ///
873    /// Panics if `order` is [`Acquire`] or [`AcqRel`].
874    #[inline]
875    #[unstable(feature = "atomic_volatile", issue = "158947")]
876    #[rustc_const_unstable(feature = "atomic_volatile", issue = "158947")]
877    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
878    #[lint::should_not_be_called_on_const_items]
879    pub const unsafe fn store_volatile(self: *const Self, val: bool, order: Ordering) {
880        // SAFETY: follows from our own safety requirements.
881        unsafe {
882            atomic_store::<_, /* VOLATILE */ true>(self.cast::<u8>().cast_mut(), val as u8, order);
883        }
884    }
885
886    /// Stores a value into the bool, returning the previous value.
887    ///
888    /// `swap` takes an [`Ordering`] argument which describes the memory ordering
889    /// of this operation. All ordering modes are possible. Note that using
890    /// [`Acquire`] makes the store part of this operation [`Relaxed`], and
891    /// using [`Release`] makes the load part [`Relaxed`].
892    ///
893    /// **Note:** This method is only available on platforms that support atomic
894    /// operations on `u8`.
895    ///
896    /// # Examples
897    ///
898    /// ```
899    /// use std::sync::atomic::{AtomicBool, Ordering};
900    ///
901    /// let some_bool = AtomicBool::new(true);
902    ///
903    /// assert_eq!(some_bool.swap(false, Ordering::Relaxed), true);
904    /// assert_eq!(some_bool.load(Ordering::Relaxed), false);
905    /// ```
906    #[ferrocene::prevalidated]
907    #[inline]
908    #[stable(feature = "rust1", since = "1.0.0")]
909    #[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
910    #[cfg(target_has_atomic = "8")]
911    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
912    #[lint::should_not_be_called_on_const_items]
913    pub const fn swap(&self, val: bool, order: Ordering) -> bool {
914        if EMULATE_ATOMIC_BOOL {
915            #[ferrocene::annotation(
916                "Cannot be covered as this code does not run in any of the platforms for which we track coverage"
917            )]
918            if val { self.fetch_or(true, order) } else { self.fetch_and(false, order) }
919        } else {
920            // SAFETY: data races are prevented by atomic intrinsics.
921            unsafe { atomic_swap(self.v.get().cast::<u8>(), val as u8, order) != 0 }
922        }
923    }
924
925    /// Stores a value into the [`bool`] if the current value is the same as the `current` value.
926    ///
927    /// The return value is always the previous value. If it is equal to `current`, then the value
928    /// was updated.
929    ///
930    /// `compare_and_swap` also takes an [`Ordering`] argument which describes the memory
931    /// ordering of this operation. Notice that even when using [`AcqRel`], the operation
932    /// might fail and hence just perform an `Acquire` load, but not have `Release` semantics.
933    /// Using [`Acquire`] makes the store part of this operation [`Relaxed`] if it
934    /// happens, and using [`Release`] makes the load part [`Relaxed`].
935    ///
936    /// **Note:** This method is only available on platforms that support atomic
937    /// operations on `u8`.
938    ///
939    /// # Migrating to `compare_exchange` and `compare_exchange_weak`
940    ///
941    /// `compare_and_swap` is equivalent to `compare_exchange` with the following mapping for
942    /// memory orderings:
943    ///
944    /// Original | Success | Failure
945    /// -------- | ------- | -------
946    /// Relaxed  | Relaxed | Relaxed
947    /// Acquire  | Acquire | Acquire
948    /// Release  | Release | Relaxed
949    /// AcqRel   | AcqRel  | Acquire
950    /// SeqCst   | SeqCst  | SeqCst
951    ///
952    /// `compare_and_swap` and `compare_exchange` also differ in their return type. You can use
953    /// `compare_exchange(...).unwrap_or_else(|x| x)` to recover the behavior of `compare_and_swap`,
954    /// but in most cases it is more idiomatic to check whether the return value is `Ok` or `Err`
955    /// rather than to infer success vs failure based on the value that was read.
956    ///
957    /// During migration, consider whether it makes sense to use `compare_exchange_weak` instead.
958    /// `compare_exchange_weak` is allowed to fail spuriously even when the comparison succeeds,
959    /// which allows the compiler to generate better assembly code when the compare and swap
960    /// is used in a loop.
961    ///
962    /// # Examples
963    ///
964    /// ```
965    /// use std::sync::atomic::{AtomicBool, Ordering};
966    ///
967    /// let some_bool = AtomicBool::new(true);
968    ///
969    /// assert_eq!(some_bool.compare_and_swap(true, false, Ordering::Relaxed), true);
970    /// assert_eq!(some_bool.load(Ordering::Relaxed), false);
971    ///
972    /// assert_eq!(some_bool.compare_and_swap(true, true, Ordering::Relaxed), false);
973    /// assert_eq!(some_bool.load(Ordering::Relaxed), false);
974    /// ```
975    #[inline]
976    #[stable(feature = "rust1", since = "1.0.0")]
977    #[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
978    #[deprecated(
979        since = "1.50.0",
980        note = "Use `compare_exchange` or `compare_exchange_weak` instead"
981    )]
982    #[cfg(target_has_atomic = "8")]
983    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
984    #[lint::should_not_be_called_on_const_items]
985    pub const fn compare_and_swap(&self, current: bool, new: bool, order: Ordering) -> bool {
986        match self.compare_exchange(current, new, order, strongest_failure_ordering(order)) {
987            Ok(x) => x,
988            Err(x) => x,
989        }
990    }
991
992    /// Stores a value into the [`bool`] if the current value is the same as the `current` value.
993    ///
994    /// The return value is a result indicating whether the new value was written and containing
995    /// the previous value. On success this value is guaranteed to be equal to `current`.
996    ///
997    /// `compare_exchange` takes two [`Ordering`] arguments to describe the memory
998    /// ordering of this operation. `success` describes the required ordering for the
999    /// read-modify-write operation that takes place if the comparison with `current` succeeds.
1000    /// `failure` describes the required ordering for the load operation that takes place when
1001    /// the comparison fails. Using [`Acquire`] as success ordering makes the store part
1002    /// of this operation [`Relaxed`], and using [`Release`] makes the successful load
1003    /// [`Relaxed`]. The failure ordering can only be [`SeqCst`], [`Acquire`] or [`Relaxed`].
1004    ///
1005    /// **Note:** This method is only available on platforms that support atomic
1006    /// operations on `u8`.
1007    ///
1008    /// # Examples
1009    ///
1010    /// ```
1011    /// use std::sync::atomic::{AtomicBool, Ordering};
1012    ///
1013    /// let some_bool = AtomicBool::new(true);
1014    ///
1015    /// assert_eq!(some_bool.compare_exchange(true,
1016    ///                                       false,
1017    ///                                       Ordering::Acquire,
1018    ///                                       Ordering::Relaxed),
1019    ///            Ok(true));
1020    /// assert_eq!(some_bool.load(Ordering::Relaxed), false);
1021    ///
1022    /// assert_eq!(some_bool.compare_exchange(true, true,
1023    ///                                       Ordering::SeqCst,
1024    ///                                       Ordering::Acquire),
1025    ///            Err(false));
1026    /// assert_eq!(some_bool.load(Ordering::Relaxed), false);
1027    /// ```
1028    ///
1029    /// # Considerations
1030    ///
1031    /// `compare_exchange` is a [compare-and-swap operation] and thus exhibits the usual downsides
1032    /// of CAS operations. In particular, a load of the value followed by a successful
1033    /// `compare_exchange` with the previous load *does not ensure* that other threads have not
1034    /// changed the value in the interim. This is usually important when the *equality* check in
1035    /// the `compare_exchange` is being used to check the *identity* of a value, but equality
1036    /// does not necessarily imply identity. In this case, `compare_exchange` can lead to the
1037    /// [ABA problem].
1038    ///
1039    /// [ABA Problem]: https://en.wikipedia.org/wiki/ABA_problem
1040    /// [compare-and-swap operation]: https://en.wikipedia.org/wiki/Compare-and-swap
1041    #[ferrocene::prevalidated]
1042    #[inline]
1043    #[stable(feature = "extended_compare_and_swap", since = "1.10.0")]
1044    #[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
1045    #[doc(alias = "compare_and_swap")]
1046    #[cfg(target_has_atomic = "8")]
1047    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
1048    #[lint::should_not_be_called_on_const_items]
1049    pub const fn compare_exchange(
1050        &self,
1051        current: bool,
1052        new: bool,
1053        success: Ordering,
1054        failure: Ordering,
1055    ) -> Result<bool, bool> {
1056        if EMULATE_ATOMIC_BOOL {
1057            #[ferrocene::annotation(
1058                "Cannot be covered as this code does not run in any of the platforms for which we track coverage"
1059            )]
1060            {
1061                // Pick the strongest ordering from success and failure.
1062                let order = match (success, failure) {
1063                    (SeqCst, _) => SeqCst,
1064                    (_, SeqCst) => SeqCst,
1065                    (AcqRel, _) => AcqRel,
1066                    (_, AcqRel) => {
1067                        panic!("there is no such thing as an acquire-release failure ordering")
1068                    }
1069                    (Release, Acquire) => AcqRel,
1070                    (Acquire, _) => Acquire,
1071                    (_, Acquire) => Acquire,
1072                    (Release, Relaxed) => Release,
1073                    (_, Release) => panic!("there is no such thing as a release failure ordering"),
1074                    (Relaxed, Relaxed) => Relaxed,
1075                };
1076                let old = if current == new {
1077                    // This is a no-op, but we still need to perform the operation
1078                    // for memory ordering reasons.
1079                    self.fetch_or(false, order)
1080                } else {
1081                    // This sets the value to the new one and returns the old one.
1082                    self.swap(new, order)
1083                };
1084                if old == current { Ok(old) } else { Err(old) }
1085            }
1086        } else {
1087            // SAFETY: data races are prevented by atomic intrinsics.
1088            match unsafe {
1089                atomic_compare_exchange(
1090                    self.v.get().cast::<u8>(),
1091                    current as u8,
1092                    new as u8,
1093                    success,
1094                    failure,
1095                )
1096            } {
1097                Ok(x) => Ok(x != 0),
1098                Err(x) => Err(x != 0),
1099            }
1100        }
1101    }
1102
1103    /// Stores a value into the [`bool`] if the current value is the same as the `current` value.
1104    ///
1105    /// Unlike [`AtomicBool::compare_exchange`], this function is allowed to spuriously fail even when the
1106    /// comparison succeeds, which can result in more efficient code on some platforms. The
1107    /// return value is a result indicating whether the new value was written and containing the
1108    /// previous value.
1109    ///
1110    /// `compare_exchange_weak` takes two [`Ordering`] arguments to describe the memory
1111    /// ordering of this operation. `success` describes the required ordering for the
1112    /// read-modify-write operation that takes place if the comparison with `current` succeeds.
1113    /// `failure` describes the required ordering for the load operation that takes place when
1114    /// the comparison fails. Using [`Acquire`] as success ordering makes the store part
1115    /// of this operation [`Relaxed`], and using [`Release`] makes the successful load
1116    /// [`Relaxed`]. The failure ordering can only be [`SeqCst`], [`Acquire`] or [`Relaxed`].
1117    ///
1118    /// **Note:** This method is only available on platforms that support atomic
1119    /// operations on `u8`.
1120    ///
1121    /// # Examples
1122    ///
1123    /// ```
1124    /// use std::sync::atomic::{AtomicBool, Ordering};
1125    ///
1126    /// let val = AtomicBool::new(false);
1127    ///
1128    /// let new = true;
1129    /// let mut old = val.load(Ordering::Relaxed);
1130    /// loop {
1131    ///     match val.compare_exchange_weak(old, new, Ordering::SeqCst, Ordering::Relaxed) {
1132    ///         Ok(_) => break,
1133    ///         Err(x) => old = x,
1134    ///     }
1135    /// }
1136    /// ```
1137    ///
1138    /// # Considerations
1139    ///
1140    /// `compare_exchange` is a [compare-and-swap operation] and thus exhibits the usual downsides
1141    /// of CAS operations. In particular, a load of the value followed by a successful
1142    /// `compare_exchange` with the previous load *does not ensure* that other threads have not
1143    /// changed the value in the interim. This is usually important when the *equality* check in
1144    /// the `compare_exchange` is being used to check the *identity* of a value, but equality
1145    /// does not necessarily imply identity. In this case, `compare_exchange` can lead to the
1146    /// [ABA problem].
1147    ///
1148    /// [ABA Problem]: https://en.wikipedia.org/wiki/ABA_problem
1149    /// [compare-and-swap operation]: https://en.wikipedia.org/wiki/Compare-and-swap
1150    #[inline]
1151    #[stable(feature = "extended_compare_and_swap", since = "1.10.0")]
1152    #[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
1153    #[doc(alias = "compare_and_swap")]
1154    #[cfg(target_has_atomic = "8")]
1155    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
1156    #[lint::should_not_be_called_on_const_items]
1157    pub const fn compare_exchange_weak(
1158        &self,
1159        current: bool,
1160        new: bool,
1161        success: Ordering,
1162        failure: Ordering,
1163    ) -> Result<bool, bool> {
1164        if EMULATE_ATOMIC_BOOL {
1165            return self.compare_exchange(current, new, success, failure);
1166        }
1167
1168        // SAFETY: data races are prevented by atomic intrinsics.
1169        match unsafe {
1170            atomic_compare_exchange_weak(
1171                self.v.get().cast::<u8>(),
1172                current as u8,
1173                new as u8,
1174                success,
1175                failure,
1176            )
1177        } {
1178            Ok(x) => Ok(x != 0),
1179            Err(x) => Err(x != 0),
1180        }
1181    }
1182
1183    /// Logical "and" with a boolean value.
1184    ///
1185    /// Performs a logical "and" operation on the current value and the argument `val`, and sets
1186    /// the new value to the result.
1187    ///
1188    /// Returns the previous value.
1189    ///
1190    /// `fetch_and` takes an [`Ordering`] argument which describes the memory ordering
1191    /// of this operation. All ordering modes are possible. Note that using
1192    /// [`Acquire`] makes the store part of this operation [`Relaxed`], and
1193    /// using [`Release`] makes the load part [`Relaxed`].
1194    ///
1195    /// **Note:** This method is only available on platforms that support atomic
1196    /// operations on `u8`.
1197    ///
1198    /// # Examples
1199    ///
1200    /// ```
1201    /// use std::sync::atomic::{AtomicBool, Ordering};
1202    ///
1203    /// let foo = AtomicBool::new(true);
1204    /// assert_eq!(foo.fetch_and(false, Ordering::SeqCst), true);
1205    /// assert_eq!(foo.load(Ordering::SeqCst), false);
1206    ///
1207    /// let foo = AtomicBool::new(true);
1208    /// assert_eq!(foo.fetch_and(true, Ordering::SeqCst), true);
1209    /// assert_eq!(foo.load(Ordering::SeqCst), true);
1210    ///
1211    /// let foo = AtomicBool::new(false);
1212    /// assert_eq!(foo.fetch_and(false, Ordering::SeqCst), false);
1213    /// assert_eq!(foo.load(Ordering::SeqCst), false);
1214    /// ```
1215    #[ferrocene::prevalidated]
1216    #[inline]
1217    #[stable(feature = "rust1", since = "1.0.0")]
1218    #[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
1219    #[cfg(target_has_atomic = "8")]
1220    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
1221    #[lint::should_not_be_called_on_const_items]
1222    pub const fn fetch_and(&self, val: bool, order: Ordering) -> bool {
1223        // SAFETY: data races are prevented by atomic intrinsics.
1224        unsafe { atomic_and(self.v.get().cast::<u8>(), val as u8, order) != 0 }
1225    }
1226
1227    /// Logical "nand" with a boolean value.
1228    ///
1229    /// Performs a logical "nand" operation on the current value and the argument `val`, and sets
1230    /// the new value to the result.
1231    ///
1232    /// Returns the previous value.
1233    ///
1234    /// `fetch_nand` takes an [`Ordering`] argument which describes the memory ordering
1235    /// of this operation. All ordering modes are possible. Note that using
1236    /// [`Acquire`] makes the store part of this operation [`Relaxed`], and
1237    /// using [`Release`] makes the load part [`Relaxed`].
1238    ///
1239    /// **Note:** This method is only available on platforms that support atomic
1240    /// operations on `u8`.
1241    ///
1242    /// # Examples
1243    ///
1244    /// ```
1245    /// use std::sync::atomic::{AtomicBool, Ordering};
1246    ///
1247    /// let foo = AtomicBool::new(true);
1248    /// assert_eq!(foo.fetch_nand(false, Ordering::SeqCst), true);
1249    /// assert_eq!(foo.load(Ordering::SeqCst), true);
1250    ///
1251    /// let foo = AtomicBool::new(true);
1252    /// assert_eq!(foo.fetch_nand(true, Ordering::SeqCst), true);
1253    /// assert_eq!(foo.load(Ordering::SeqCst) as usize, 0);
1254    /// assert_eq!(foo.load(Ordering::SeqCst), false);
1255    ///
1256    /// let foo = AtomicBool::new(false);
1257    /// assert_eq!(foo.fetch_nand(false, Ordering::SeqCst), false);
1258    /// assert_eq!(foo.load(Ordering::SeqCst), true);
1259    /// ```
1260    #[inline]
1261    #[stable(feature = "rust1", since = "1.0.0")]
1262    #[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
1263    #[cfg(target_has_atomic = "8")]
1264    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
1265    #[lint::should_not_be_called_on_const_items]
1266    pub const fn fetch_nand(&self, val: bool, order: Ordering) -> bool {
1267        // We can't use atomic_nand here because it can result in a bool with
1268        // an invalid value. This happens because the atomic operation is done
1269        // with an 8-bit integer internally, which would set the upper 7 bits.
1270        // So we just use fetch_xor or swap instead.
1271        if val {
1272            // !(x & true) == !x
1273            // We must invert the bool.
1274            self.fetch_xor(true, order)
1275        } else {
1276            // !(x & false) == true
1277            // We must set the bool to true.
1278            self.swap(true, order)
1279        }
1280    }
1281
1282    /// Logical "or" with a boolean value.
1283    ///
1284    /// Performs a logical "or" operation on the current value and the argument `val`, and sets the
1285    /// new value to the result.
1286    ///
1287    /// Returns the previous value.
1288    ///
1289    /// `fetch_or` takes an [`Ordering`] argument which describes the memory ordering
1290    /// of this operation. All ordering modes are possible. Note that using
1291    /// [`Acquire`] makes the store part of this operation [`Relaxed`], and
1292    /// using [`Release`] makes the load part [`Relaxed`].
1293    ///
1294    /// **Note:** This method is only available on platforms that support atomic
1295    /// operations on `u8`.
1296    ///
1297    /// # Examples
1298    ///
1299    /// ```
1300    /// use std::sync::atomic::{AtomicBool, Ordering};
1301    ///
1302    /// let foo = AtomicBool::new(true);
1303    /// assert_eq!(foo.fetch_or(false, Ordering::SeqCst), true);
1304    /// assert_eq!(foo.load(Ordering::SeqCst), true);
1305    ///
1306    /// let foo = AtomicBool::new(false);
1307    /// assert_eq!(foo.fetch_or(true, Ordering::SeqCst), false);
1308    /// assert_eq!(foo.load(Ordering::SeqCst), true);
1309    ///
1310    /// let foo = AtomicBool::new(false);
1311    /// assert_eq!(foo.fetch_or(false, Ordering::SeqCst), false);
1312    /// assert_eq!(foo.load(Ordering::SeqCst), false);
1313    /// ```
1314    #[ferrocene::prevalidated]
1315    #[inline]
1316    #[stable(feature = "rust1", since = "1.0.0")]
1317    #[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
1318    #[cfg(target_has_atomic = "8")]
1319    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
1320    #[lint::should_not_be_called_on_const_items]
1321    pub const fn fetch_or(&self, val: bool, order: Ordering) -> bool {
1322        // SAFETY: data races are prevented by atomic intrinsics.
1323        unsafe { atomic_or(self.v.get().cast::<u8>(), val as u8, order) != 0 }
1324    }
1325
1326    /// Logical "xor" with a boolean value.
1327    ///
1328    /// Performs a logical "xor" operation on the current value and the argument `val`, and sets
1329    /// the new value to the result.
1330    ///
1331    /// Returns the previous value.
1332    ///
1333    /// `fetch_xor` takes an [`Ordering`] argument which describes the memory ordering
1334    /// of this operation. All ordering modes are possible. Note that using
1335    /// [`Acquire`] makes the store part of this operation [`Relaxed`], and
1336    /// using [`Release`] makes the load part [`Relaxed`].
1337    ///
1338    /// **Note:** This method is only available on platforms that support atomic
1339    /// operations on `u8`.
1340    ///
1341    /// # Examples
1342    ///
1343    /// ```
1344    /// use std::sync::atomic::{AtomicBool, Ordering};
1345    ///
1346    /// let foo = AtomicBool::new(true);
1347    /// assert_eq!(foo.fetch_xor(false, Ordering::SeqCst), true);
1348    /// assert_eq!(foo.load(Ordering::SeqCst), true);
1349    ///
1350    /// let foo = AtomicBool::new(true);
1351    /// assert_eq!(foo.fetch_xor(true, Ordering::SeqCst), true);
1352    /// assert_eq!(foo.load(Ordering::SeqCst), false);
1353    ///
1354    /// let foo = AtomicBool::new(false);
1355    /// assert_eq!(foo.fetch_xor(false, Ordering::SeqCst), false);
1356    /// assert_eq!(foo.load(Ordering::SeqCst), false);
1357    /// ```
1358    #[inline]
1359    #[stable(feature = "rust1", since = "1.0.0")]
1360    #[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
1361    #[cfg(target_has_atomic = "8")]
1362    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
1363    #[lint::should_not_be_called_on_const_items]
1364    pub const fn fetch_xor(&self, val: bool, order: Ordering) -> bool {
1365        // SAFETY: data races are prevented by atomic intrinsics.
1366        unsafe { atomic_xor(self.v.get().cast::<u8>(), val as u8, order) != 0 }
1367    }
1368
1369    /// Logical "not" with a boolean value.
1370    ///
1371    /// Performs a logical "not" operation on the current value, and sets
1372    /// the new value to the result.
1373    ///
1374    /// Returns the previous value.
1375    ///
1376    /// `fetch_not` takes an [`Ordering`] argument which describes the memory ordering
1377    /// of this operation. All ordering modes are possible. Note that using
1378    /// [`Acquire`] makes the store part of this operation [`Relaxed`], and
1379    /// using [`Release`] makes the load part [`Relaxed`].
1380    ///
1381    /// **Note:** This method is only available on platforms that support atomic
1382    /// operations on `u8`.
1383    ///
1384    /// # Examples
1385    ///
1386    /// ```
1387    /// use std::sync::atomic::{AtomicBool, Ordering};
1388    ///
1389    /// let foo = AtomicBool::new(true);
1390    /// assert_eq!(foo.fetch_not(Ordering::SeqCst), true);
1391    /// assert_eq!(foo.load(Ordering::SeqCst), false);
1392    ///
1393    /// let foo = AtomicBool::new(false);
1394    /// assert_eq!(foo.fetch_not(Ordering::SeqCst), false);
1395    /// assert_eq!(foo.load(Ordering::SeqCst), true);
1396    /// ```
1397    #[inline]
1398    #[stable(feature = "atomic_bool_fetch_not", since = "1.81.0")]
1399    #[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
1400    #[cfg(target_has_atomic = "8")]
1401    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
1402    #[lint::should_not_be_called_on_const_items]
1403    pub const fn fetch_not(&self, order: Ordering) -> bool {
1404        self.fetch_xor(true, order)
1405    }
1406
1407    /// Returns a mutable pointer to the underlying [`bool`].
1408    ///
1409    /// Doing non-atomic reads and writes on the resulting boolean can be a data race.
1410    /// This method is mostly useful for FFI, where the function signature may use
1411    /// `*mut bool` instead of `&AtomicBool`.
1412    ///
1413    /// Returning an `*mut` pointer from a shared reference to this atomic is safe because the
1414    /// atomic types work with interior mutability. All modifications of an atomic change the value
1415    /// through a shared reference, and can do so safely as long as they use atomic operations. Any
1416    /// use of the returned raw pointer requires an `unsafe` block and still has to uphold the
1417    /// requirements of the [memory model].
1418    ///
1419    /// # Examples
1420    ///
1421    /// ```ignore (extern-declaration)
1422    /// # fn main() {
1423    /// use std::sync::atomic::AtomicBool;
1424    ///
1425    /// extern "C" {
1426    ///     fn my_atomic_op(arg: *mut bool);
1427    /// }
1428    ///
1429    /// let mut atomic = AtomicBool::new(true);
1430    /// unsafe {
1431    ///     my_atomic_op(atomic.as_ptr());
1432    /// }
1433    /// # }
1434    /// ```
1435    ///
1436    /// [memory model]: self#memory-model-for-atomic-accesses
1437    #[inline]
1438    #[stable(feature = "atomic_as_ptr", since = "1.70.0")]
1439    #[rustc_const_stable(feature = "atomic_as_ptr", since = "1.70.0")]
1440    #[lint::never_returns_null_ptr]
1441    #[lint::should_not_be_called_on_const_items]
1442    pub const fn as_ptr(&self) -> *mut bool {
1443        self.v.get().cast()
1444    }
1445
1446    /// An alias for [`AtomicBool::try_update`].
1447    #[inline]
1448    #[stable(feature = "atomic_fetch_update", since = "1.53.0")]
1449    #[cfg(target_has_atomic = "8")]
1450    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
1451    #[lint::should_not_be_called_on_const_items]
1452    #[deprecated(
1453        since = "1.99.0",
1454        note = "renamed to `try_update` for consistency",
1455        suggestion = "try_update"
1456    )]
1457    pub fn fetch_update<F>(
1458        &self,
1459        set_order: Ordering,
1460        fetch_order: Ordering,
1461        f: F,
1462    ) -> Result<bool, bool>
1463    where
1464        F: FnMut(bool) -> Option<bool>,
1465    {
1466        self.try_update(set_order, fetch_order, f)
1467    }
1468
1469    /// Fetches the value, and applies a function to it that returns an optional
1470    /// new value. Returns a `Result` of `Ok(previous_value)` if the function
1471    /// returned `Some(_)`, else `Err(previous_value)`.
1472    ///
1473    /// See also: [`update`](`AtomicBool::update`).
1474    ///
1475    /// Note: This may call the function multiple times if the value has been
1476    /// changed from other threads in the meantime, as long as the function
1477    /// returns `Some(_)`, but the function will have been applied only once to
1478    /// the stored value.
1479    ///
1480    /// `try_update` takes two [`Ordering`] arguments to describe the memory
1481    /// ordering of this operation. The first describes the required ordering for
1482    /// when the operation finally succeeds while the second describes the
1483    /// required ordering for loads. These correspond to the success and failure
1484    /// orderings of [`AtomicBool::compare_exchange`] respectively.
1485    ///
1486    /// Using [`Acquire`] as success ordering makes the store part of this
1487    /// operation [`Relaxed`], and using [`Release`] makes the final successful
1488    /// load [`Relaxed`]. The (failed) load ordering can only be [`SeqCst`],
1489    /// [`Acquire`] or [`Relaxed`].
1490    ///
1491    /// **Note:** This method is only available on platforms that support atomic
1492    /// operations on `u8`.
1493    ///
1494    /// # Considerations
1495    ///
1496    /// This method is not magic; it is not provided by the hardware, and does not act like a
1497    /// critical section or mutex.
1498    ///
1499    /// It is implemented on top of an atomic [compare-and-swap operation], and thus is subject to
1500    /// the usual drawbacks of CAS operations. In particular, be careful of the [ABA problem].
1501    ///
1502    /// [ABA Problem]: https://en.wikipedia.org/wiki/ABA_problem
1503    /// [compare-and-swap operation]: https://en.wikipedia.org/wiki/Compare-and-swap
1504    ///
1505    /// # Examples
1506    ///
1507    /// ```rust
1508    /// use std::sync::atomic::{AtomicBool, Ordering};
1509    ///
1510    /// let x = AtomicBool::new(false);
1511    /// assert_eq!(x.try_update(Ordering::SeqCst, Ordering::SeqCst, |_| None), Err(false));
1512    /// assert_eq!(x.try_update(Ordering::SeqCst, Ordering::SeqCst, |x| Some(!x)), Ok(false));
1513    /// assert_eq!(x.try_update(Ordering::SeqCst, Ordering::SeqCst, |x| Some(!x)), Ok(true));
1514    /// assert_eq!(x.load(Ordering::SeqCst), false);
1515    /// ```
1516    #[inline]
1517    #[stable(feature = "atomic_try_update", since = "1.95.0")]
1518    #[cfg(target_has_atomic = "8")]
1519    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
1520    #[lint::should_not_be_called_on_const_items]
1521    pub fn try_update(
1522        &self,
1523        set_order: Ordering,
1524        fetch_order: Ordering,
1525        mut f: impl FnMut(bool) -> Option<bool>,
1526    ) -> Result<bool, bool> {
1527        let mut prev = self.load(fetch_order);
1528        while let Some(next) = f(prev) {
1529            match self.compare_exchange_weak(prev, next, set_order, fetch_order) {
1530                x @ Ok(_) => return x,
1531                Err(next_prev) => prev = next_prev,
1532            }
1533        }
1534        Err(prev)
1535    }
1536
1537    /// Fetches the value, and applies a function to it that returns a new value.
1538    /// The new value is stored and the old value is returned.
1539    ///
1540    /// See also: [`try_update`](`AtomicBool::try_update`).
1541    ///
1542    /// Note: This may call the function multiple times if the value has been changed from other threads in
1543    /// the meantime, but the function will have been applied only once to the stored value.
1544    ///
1545    /// `update` takes two [`Ordering`] arguments to describe the memory
1546    /// ordering of this operation. The first describes the required ordering for
1547    /// when the operation finally succeeds while the second describes the
1548    /// required ordering for loads. These correspond to the success and failure
1549    /// orderings of [`AtomicBool::compare_exchange`] respectively.
1550    ///
1551    /// Using [`Acquire`] as success ordering makes the store part
1552    /// of this operation [`Relaxed`], and using [`Release`] makes the final successful load
1553    /// [`Relaxed`]. The (failed) load ordering can only be [`SeqCst`], [`Acquire`] or [`Relaxed`].
1554    ///
1555    /// **Note:** This method is only available on platforms that support atomic operations on `u8`.
1556    ///
1557    /// # Considerations
1558    ///
1559    /// This method is not magic; it is not provided by the hardware, and does not act like a
1560    /// critical section or mutex.
1561    ///
1562    /// It is implemented on top of an atomic [compare-and-swap operation], and thus is subject to
1563    /// the usual drawbacks of CAS operations. In particular, be careful of the [ABA problem].
1564    ///
1565    /// [ABA Problem]: https://en.wikipedia.org/wiki/ABA_problem
1566    /// [compare-and-swap operation]: https://en.wikipedia.org/wiki/Compare-and-swap
1567    ///
1568    /// # Examples
1569    ///
1570    /// ```rust
1571    ///
1572    /// use std::sync::atomic::{AtomicBool, Ordering};
1573    ///
1574    /// let x = AtomicBool::new(false);
1575    /// assert_eq!(x.update(Ordering::SeqCst, Ordering::SeqCst, |x| !x), false);
1576    /// assert_eq!(x.update(Ordering::SeqCst, Ordering::SeqCst, |x| !x), true);
1577    /// assert_eq!(x.load(Ordering::SeqCst), false);
1578    /// ```
1579    #[inline]
1580    #[stable(feature = "atomic_try_update", since = "1.95.0")]
1581    #[cfg(target_has_atomic = "8")]
1582    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
1583    #[lint::should_not_be_called_on_const_items]
1584    pub fn update(
1585        &self,
1586        set_order: Ordering,
1587        fetch_order: Ordering,
1588        mut f: impl FnMut(bool) -> bool,
1589    ) -> bool {
1590        let mut prev = self.load(fetch_order);
1591        loop {
1592            match self.compare_exchange_weak(prev, f(prev), set_order, fetch_order) {
1593                Ok(x) => break x,
1594                Err(next_prev) => prev = next_prev,
1595            }
1596        }
1597    }
1598}
1599
1600#[cfg(target_has_atomic_load_store = "ptr")]
1601impl<T> AtomicPtr<T> {
1602    /// Creates a new `AtomicPtr`.
1603    ///
1604    /// # Examples
1605    ///
1606    /// ```
1607    /// use std::sync::atomic::AtomicPtr;
1608    ///
1609    /// let ptr = &mut 5;
1610    /// let atomic_ptr = AtomicPtr::new(ptr);
1611    /// ```
1612    #[inline]
1613    #[stable(feature = "rust1", since = "1.0.0")]
1614    #[rustc_const_stable(feature = "const_atomic_new", since = "1.24.0")]
1615    pub const fn new(p: *mut T) -> AtomicPtr<T> {
1616        // SAFETY:
1617        // `Atomic<T>` is essentially a transparent wrapper around `T`.
1618        unsafe { transmute(p) }
1619    }
1620
1621    /// Creates a new `AtomicPtr` from a pointer.
1622    ///
1623    /// # Examples
1624    ///
1625    /// ```
1626    /// use std::sync::atomic::{self, AtomicPtr};
1627    ///
1628    /// // Get a pointer to an allocated value
1629    /// let ptr: *mut *mut u8 = Box::into_raw(Box::new(std::ptr::null_mut()));
1630    ///
1631    /// assert!(ptr.cast::<AtomicPtr<u8>>().is_aligned());
1632    ///
1633    /// {
1634    ///     // Create an atomic view of the allocated value
1635    ///     let atomic = unsafe { AtomicPtr::from_ptr(ptr) };
1636    ///
1637    ///     // Use `atomic` for atomic operations, possibly share it with other threads
1638    ///     atomic.store(std::ptr::NonNull::dangling().as_ptr(), atomic::Ordering::Relaxed);
1639    /// }
1640    ///
1641    /// // It's ok to non-atomically access the value behind `ptr`,
1642    /// // since the reference to the atomic ended its lifetime in the block above
1643    /// assert!(!unsafe { *ptr }.is_null());
1644    ///
1645    /// // Deallocate the value
1646    /// unsafe { drop(Box::from_raw(ptr)) }
1647    /// ```
1648    ///
1649    /// # Safety
1650    ///
1651    /// * `ptr` must be aligned to `align_of::<AtomicPtr<T>>()` (note that on some platforms this
1652    ///   can be bigger than `align_of::<*mut T>()`).
1653    /// * `ptr` must be [valid] for both reads and writes for the whole lifetime `'a`.
1654    /// * You must adhere to the [Memory model for atomic accesses]. In particular, it is not
1655    ///   allowed to mix conflicting atomic and non-atomic accesses, or atomic accesses of different
1656    ///   sizes, without synchronization.
1657    ///
1658    /// [valid]: crate::ptr#safety
1659    /// [Memory model for atomic accesses]: self#memory-model-for-atomic-accesses
1660    #[inline]
1661    #[stable(feature = "atomic_from_ptr", since = "1.75.0")]
1662    #[rustc_const_stable(feature = "const_atomic_from_ptr", since = "1.84.0")]
1663    pub const unsafe fn from_ptr<'a>(ptr: *mut *mut T) -> &'a AtomicPtr<T> {
1664        // SAFETY: guaranteed by the caller
1665        unsafe { &*ptr.cast() }
1666    }
1667
1668    /// Creates a new pointer to `AtomicPtr` from a pointer.
1669    ///
1670    /// This is useful if you want to do volatile atomic accesses, and thus avoid creating
1671    /// a reference to the destination.
1672    #[inline]
1673    #[unstable(feature = "atomic_volatile", issue = "158947")]
1674    pub const fn from_ptr_raw(ptr: *mut *mut T) -> *const AtomicPtr<T> {
1675        ptr.cast_const().cast()
1676    }
1677
1678    /// Creates a new `AtomicPtr` initialized with a null pointer.
1679    ///
1680    /// # Examples
1681    ///
1682    /// ```
1683    /// #![feature(atomic_ptr_null)]
1684    /// use std::sync::atomic::{AtomicPtr, Ordering};
1685    ///
1686    /// let atomic_ptr = AtomicPtr::<()>::null();
1687    /// assert!(atomic_ptr.load(Ordering::Relaxed).is_null());
1688    /// ```
1689    #[inline]
1690    #[must_use]
1691    #[unstable(feature = "atomic_ptr_null", issue = "150733")]
1692    pub const fn null() -> AtomicPtr<T> {
1693        AtomicPtr::new(crate::ptr::null_mut())
1694    }
1695
1696    /// Returns a mutable reference to the underlying pointer.
1697    ///
1698    /// This is safe because the mutable reference guarantees that no other threads are
1699    /// concurrently accessing the atomic data.
1700    ///
1701    /// # Examples
1702    ///
1703    /// ```
1704    /// use std::sync::atomic::{AtomicPtr, Ordering};
1705    ///
1706    /// let mut data = 10;
1707    /// let mut atomic_ptr = AtomicPtr::new(&mut data);
1708    /// let mut other_data = 5;
1709    /// *atomic_ptr.get_mut() = &mut other_data;
1710    /// assert_eq!(unsafe { *atomic_ptr.load(Ordering::SeqCst) }, 5);
1711    /// ```
1712    #[inline]
1713    #[stable(feature = "atomic_access", since = "1.15.0")]
1714    #[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
1715    pub const fn get_mut(&mut self) -> &mut *mut T {
1716        // SAFETY:
1717        // `Atomic<T>` is essentially a transparent wrapper around `T`.
1718        unsafe { &mut *self.as_ptr() }
1719    }
1720
1721    /// Gets atomic access to a pointer.
1722    ///
1723    /// **Note:** This function is only available on targets where `AtomicPtr<T>` has the same alignment as `*const T`
1724    ///
1725    /// # Examples
1726    ///
1727    /// ```
1728    /// use std::sync::atomic::{AtomicPtr, Ordering};
1729    ///
1730    /// let mut data = 123;
1731    /// let mut some_ptr = &mut data as *mut i32;
1732    /// let a = AtomicPtr::from_mut(&mut some_ptr);
1733    /// let mut other_data = 456;
1734    /// a.store(&mut other_data, Ordering::Relaxed);
1735    /// assert_eq!(unsafe { *some_ptr }, 456);
1736    /// ```
1737    #[inline]
1738    #[cfg(target_has_atomic_primitive_alignment = "ptr")]
1739    #[stable(feature = "atomic_from_mut", since = "1.98.0")]
1740    #[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
1741    pub const fn from_mut(v: &mut *mut T) -> &mut Self {
1742        let [] = [(); align_of::<AtomicPtr<()>>() - align_of::<*mut ()>()];
1743        // SAFETY:
1744        //  - the mutable reference guarantees unique ownership.
1745        //  - the alignment of `*mut T` and `Self` is the same on all platforms
1746        //    supported by rust, as verified above.
1747        unsafe { &mut *(v as *mut *mut T as *mut Self) }
1748    }
1749
1750    /// Gets non-atomic access to a `&mut [AtomicPtr]` slice.
1751    ///
1752    /// This is safe because the mutable reference guarantees that no other threads are
1753    /// concurrently accessing the atomic data.
1754    ///
1755    /// # Examples
1756    ///
1757    /// ```ignore-wasm
1758    /// use std::ptr::null_mut;
1759    /// use std::sync::atomic::{AtomicPtr, Ordering};
1760    ///
1761    /// let mut some_ptrs = [const { AtomicPtr::new(null_mut::<String>()) }; 10];
1762    ///
1763    /// let view: &mut [*mut String] = AtomicPtr::get_mut_slice(&mut some_ptrs);
1764    /// assert_eq!(view, [null_mut::<String>(); 10]);
1765    /// view
1766    ///     .iter_mut()
1767    ///     .enumerate()
1768    ///     .for_each(|(i, ptr)| *ptr = Box::into_raw(Box::new(format!("iteration#{i}"))));
1769    ///
1770    /// std::thread::scope(|s| {
1771    ///     for ptr in &some_ptrs {
1772    ///         s.spawn(move || {
1773    ///             let ptr = ptr.load(Ordering::Relaxed);
1774    ///             assert!(!ptr.is_null());
1775    ///
1776    ///             let name = unsafe { Box::from_raw(ptr) };
1777    ///             println!("Hello, {name}!");
1778    ///         });
1779    ///     }
1780    /// });
1781    /// ```
1782    #[inline]
1783    #[stable(feature = "atomic_from_mut", since = "1.98.0")]
1784    #[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
1785    pub const fn get_mut_slice(this: &mut [Self]) -> &mut [*mut T] {
1786        // SAFETY: the mutable reference guarantees unique ownership.
1787        unsafe { &mut *(this as *mut [Self] as *mut [*mut T]) }
1788    }
1789
1790    /// Gets atomic access to a slice of pointers.
1791    ///
1792    /// **Note:** This function is only available on targets where `AtomicPtr<T>` has the same alignment as `*const T`
1793    ///
1794    /// # Examples
1795    ///
1796    /// ```ignore-wasm
1797    /// use std::ptr::null_mut;
1798    /// use std::sync::atomic::{AtomicPtr, Ordering};
1799    ///
1800    /// let mut some_ptrs = [null_mut::<String>(); 10];
1801    /// let a = &*AtomicPtr::from_mut_slice(&mut some_ptrs);
1802    /// std::thread::scope(|s| {
1803    ///     for i in 0..a.len() {
1804    ///         s.spawn(move || {
1805    ///             let name = Box::new(format!("thread{i}"));
1806    ///             a[i].store(Box::into_raw(name), Ordering::Relaxed);
1807    ///         });
1808    ///     }
1809    /// });
1810    /// for p in some_ptrs {
1811    ///     assert!(!p.is_null());
1812    ///     let name = unsafe { Box::from_raw(p) };
1813    ///     println!("Hello, {name}!");
1814    /// }
1815    /// ```
1816    #[inline]
1817    #[cfg(target_has_atomic_primitive_alignment = "ptr")]
1818    #[stable(feature = "atomic_from_mut", since = "1.98.0")]
1819    #[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
1820    pub const fn from_mut_slice(v: &mut [*mut T]) -> &mut [Self] {
1821        // SAFETY:
1822        //  - the mutable reference guarantees unique ownership.
1823        //  - the alignment of `*mut T` and `Self` is the same on all platforms
1824        //    supported by rust, as verified above.
1825        unsafe { &mut *(v as *mut [*mut T] as *mut [Self]) }
1826    }
1827
1828    /// Consumes the atomic and returns the contained value.
1829    ///
1830    /// This is safe because passing `self` by value guarantees that no other threads are
1831    /// concurrently accessing the atomic data.
1832    ///
1833    /// # Examples
1834    ///
1835    /// ```
1836    /// use std::sync::atomic::AtomicPtr;
1837    ///
1838    /// let mut data = 5;
1839    /// let atomic_ptr = AtomicPtr::new(&mut data);
1840    /// assert_eq!(unsafe { *atomic_ptr.into_inner() }, 5);
1841    /// ```
1842    #[inline]
1843    #[stable(feature = "atomic_access", since = "1.15.0")]
1844    #[rustc_const_stable(feature = "const_atomic_into_inner", since = "1.79.0")]
1845    pub const fn into_inner(self) -> *mut T {
1846        // SAFETY:
1847        // `Atomic<T>` is essentially a transparent wrapper around `T`.
1848        unsafe { transmute(self) }
1849    }
1850
1851    /// Loads a value from the pointer.
1852    ///
1853    /// `load` takes an [`Ordering`] argument which describes the memory ordering
1854    /// of this operation. Possible values are [`SeqCst`], [`Acquire`] and [`Relaxed`].
1855    ///
1856    /// # Panics
1857    ///
1858    /// Panics if `order` is [`Release`] or [`AcqRel`].
1859    ///
1860    /// # Examples
1861    ///
1862    /// ```
1863    /// use std::sync::atomic::{AtomicPtr, Ordering};
1864    ///
1865    /// let ptr = &mut 5;
1866    /// let some_ptr = AtomicPtr::new(ptr);
1867    ///
1868    /// let value = some_ptr.load(Ordering::Relaxed);
1869    /// ```
1870    #[inline]
1871    #[stable(feature = "rust1", since = "1.0.0")]
1872    #[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
1873    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
1874    pub const fn load(&self, order: Ordering) -> *mut T {
1875        // SAFETY: data races are prevented by atomic intrinsics.
1876        unsafe {
1877            atomic_load::<_, /* VOLATILE */ false>(self.as_ptr(), order)
1878        }
1879    }
1880
1881    /// Perform a volatile atomic load from the pointer.
1882    ///
1883    /// `load_volatile` takes an [`Ordering`] argument which describes the memory ordering
1884    /// of this operation. Possible values are [`SeqCst`], [`Acquire`] and [`Relaxed`].
1885    ///
1886    #[doc = include_str!("./atomic_load_volatile.md")]
1887    ///
1888    /// # Safety
1889    ///
1890    /// Behavior is undefined if any of the following conditions are violated:
1891    ///
1892    /// * `self` must be [valid] for reads, or `self` must point to memory
1893    ///   outside of all Rust allocations and reading from that memory must:
1894    ///   - not trap, and
1895    ///   - not cause any memory inside a Rust allocation to be modified.
1896    ///
1897    /// * `self` must be aligned to `align_of::<AtomicPtr<T>>()` (note that on some platforms this
1898    ///   can be bigger than `align_of::<*mut T>()`).
1899    ///
1900    /// * Reading from `self` must produce a properly initialized value of type `*mut T`.
1901    ///
1902    /// [valid]: core::ptr#safety
1903    ///
1904    /// # Panics
1905    ///
1906    /// Panics if `order` is [`Release`] or [`AcqRel`].
1907    ///
1908    /// # Examples
1909    ///
1910    /// Assuming an MMIO region at `MMIO_ADDR` that belongs to a device with direct memory
1911    /// access, we may receive a buffer in shared memory from that device as follows:
1912    ///
1913    /// ```rust,no_run
1914    /// #![feature(atomic_volatile)]
1915    /// use std::sync::atomic::{fence, AtomicPtr, Ordering};
1916    /// use std::ptr;
1917    ///
1918    /// const MMIO_ADDR: *mut *mut u8 = ptr::without_provenance_mut(0xCAF0);
1919    /// let atomic_ptr = AtomicPtr::<u8>::from_ptr_raw(MMIO_ADDR);
1920    ///
1921    /// // Spin until we see a non-zero value.
1922    /// let buf = 'buf: loop {
1923    ///     let buf = unsafe { atomic_ptr.load_volatile(Ordering::Relaxed) };
1924    ///     if !buf.is_null() {
1925    ///         break 'buf buf;
1926    ///     }
1927    /// };
1928    /// // Synchronize with the store whose value we just read.
1929    /// // Note: a standard acquire fence may not be sufficient to synchronize with DMA devices.
1930    /// // Depending on your target, you may have to use inline assembly to emit a special fence.
1931    /// fence(Ordering::Acquire);
1932    ///
1933    /// // Now process the data in `buf`.
1934    /// ```
1935    #[inline]
1936    #[unstable(feature = "atomic_volatile", issue = "158947")]
1937    #[rustc_const_unstable(feature = "atomic_volatile", issue = "158947")]
1938    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
1939    pub const unsafe fn load_volatile(self: *const Self, order: Ordering) -> *mut T {
1940        // SAFETY: follows from our own safety requirements.
1941        unsafe {
1942            atomic_load::<_, /* VOLATILE */ true>(self.cast::<*mut T>(), order)
1943        }
1944    }
1945
1946    /// Stores a value into the pointer.
1947    ///
1948    /// `store` takes an [`Ordering`] argument which describes the memory ordering
1949    /// of this operation. Possible values are [`SeqCst`], [`Release`] and [`Relaxed`].
1950    ///
1951    /// # Panics
1952    ///
1953    /// Panics if `order` is [`Acquire`] or [`AcqRel`].
1954    ///
1955    /// # Examples
1956    ///
1957    /// ```
1958    /// use std::sync::atomic::{AtomicPtr, Ordering};
1959    ///
1960    /// let ptr = &mut 5;
1961    /// let some_ptr = AtomicPtr::new(ptr);
1962    ///
1963    /// let other_ptr = &mut 10;
1964    ///
1965    /// some_ptr.store(other_ptr, Ordering::Relaxed);
1966    /// ```
1967    #[inline]
1968    #[stable(feature = "rust1", since = "1.0.0")]
1969    #[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
1970    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
1971    #[lint::should_not_be_called_on_const_items]
1972    pub const fn store(&self, ptr: *mut T, order: Ordering) {
1973        // SAFETY: data races are prevented by atomic intrinsics.
1974        unsafe {
1975            atomic_store::<_, /* VOLATILE */ false>(self.as_ptr(), ptr, order);
1976        }
1977    }
1978
1979    /// Performs a volatile atomic store into the pointer.
1980    ///
1981    /// `store_volatile` takes an [`Ordering`] argument which describes the memory ordering
1982    /// of this operation. Possible values are [`SeqCst`], [`Release`] and [`Relaxed`].
1983    ///
1984    #[doc = include_str!("./atomic_store_volatile.md")]
1985    ///
1986    /// # Safety
1987    ///
1988    /// Behavior is undefined if any of the following conditions are violated:
1989    ///
1990    /// * `self` must be either [valid] for writes, or `self` must point to memory
1991    ///   outside of all Rust allocations and writing to that memory must:
1992    ///   - not trap, and
1993    ///   - not cause any memory inside a Rust allocation to be modified.
1994    ///
1995    /// * `self` must be aligned to `align_of::<AtomicPtr<T>>()` (note that on some platforms this
1996    ///   can be bigger than `align_of::<*mut T>()`).
1997    ///
1998    /// [valid]: core::ptr#safety
1999    ///
2000    /// # Panics
2001    ///
2002    /// Panics if `order` is [`Acquire`] or [`AcqRel`].
2003    ///
2004    /// # Examples
2005    ///
2006    /// Assuming an MMIO region at `MMIO_ADDR` that belongs to a device with direct memory
2007    /// access, we may submit a buffer in shared memory to that device as follows:
2008    ///
2009    /// ```rust,no_run
2010    /// #![feature(atomic_volatile)]
2011    /// use std::sync::atomic::{fence, AtomicPtr, Ordering};
2012    /// use std::ptr;
2013    ///
2014    /// const MMIO_ADDR: *mut *mut u8 = ptr::without_provenance_mut(0xCAF0);
2015    /// let atomic_ptr = AtomicPtr::<u8>::from_ptr_raw(MMIO_ADDR);
2016    ///
2017    /// // Prepare some data for the DMA device.
2018    /// # fn get_dma_buffer() -> *mut u8 { panic!() }
2019    /// let buf = get_dma_buffer();
2020    ///
2021    /// // Ensure the other side can synchronize with the store we do below.
2022    /// // Note: a standard release fence may not be sufficient to synchronize with DMA devices.
2023    /// // Depending on your target, you may have to use inline assembly to emit a special fence.
2024    /// fence(Ordering::Release);
2025    ///
2026    /// unsafe { atomic_ptr.store_volatile(buf, Ordering::Relaxed) };
2027    /// ```
2028    #[inline]
2029    #[unstable(feature = "atomic_volatile", issue = "158947")]
2030    #[rustc_const_unstable(feature = "atomic_volatile", issue = "158947")]
2031    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
2032    #[lint::should_not_be_called_on_const_items]
2033    pub const unsafe fn store_volatile(self: *const Self, ptr: *mut T, order: Ordering) {
2034        // SAFETY: follows from our own safety requirements.
2035        unsafe {
2036            atomic_store::<_, /* VOLATILE */ true>(self.cast::<*mut T>().cast_mut(), ptr, order);
2037        }
2038    }
2039
2040    /// Stores a value into the pointer, returning the previous value.
2041    ///
2042    /// `swap` takes an [`Ordering`] argument which describes the memory ordering
2043    /// of this operation. All ordering modes are possible. Note that using
2044    /// [`Acquire`] makes the store part of this operation [`Relaxed`], and
2045    /// using [`Release`] makes the load part [`Relaxed`].
2046    ///
2047    /// **Note:** This method is only available on platforms that support atomic
2048    /// operations on pointers.
2049    ///
2050    /// # Examples
2051    ///
2052    /// ```
2053    /// use std::sync::atomic::{AtomicPtr, Ordering};
2054    ///
2055    /// let ptr = &mut 5;
2056    /// let some_ptr = AtomicPtr::new(ptr);
2057    ///
2058    /// let other_ptr = &mut 10;
2059    ///
2060    /// let value = some_ptr.swap(other_ptr, Ordering::Relaxed);
2061    /// ```
2062    #[inline]
2063    #[stable(feature = "rust1", since = "1.0.0")]
2064    #[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
2065    #[cfg(target_has_atomic = "ptr")]
2066    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
2067    #[lint::should_not_be_called_on_const_items]
2068    pub const fn swap(&self, ptr: *mut T, order: Ordering) -> *mut T {
2069        // SAFETY: data races are prevented by atomic intrinsics.
2070        unsafe { atomic_swap(self.as_ptr(), ptr, order) }
2071    }
2072
2073    /// Stores a value into the pointer if the current value is the same as the `current` value.
2074    ///
2075    /// The return value is always the previous value. If it is equal to `current`, then the value
2076    /// was updated.
2077    ///
2078    /// `compare_and_swap` also takes an [`Ordering`] argument which describes the memory
2079    /// ordering of this operation. Notice that even when using [`AcqRel`], the operation
2080    /// might fail and hence just perform an `Acquire` load, but not have `Release` semantics.
2081    /// Using [`Acquire`] makes the store part of this operation [`Relaxed`] if it
2082    /// happens, and using [`Release`] makes the load part [`Relaxed`].
2083    ///
2084    /// **Note:** This method is only available on platforms that support atomic
2085    /// operations on pointers.
2086    ///
2087    /// # Migrating to `compare_exchange` and `compare_exchange_weak`
2088    ///
2089    /// `compare_and_swap` is equivalent to `compare_exchange` with the following mapping for
2090    /// memory orderings:
2091    ///
2092    /// Original | Success | Failure
2093    /// -------- | ------- | -------
2094    /// Relaxed  | Relaxed | Relaxed
2095    /// Acquire  | Acquire | Acquire
2096    /// Release  | Release | Relaxed
2097    /// AcqRel   | AcqRel  | Acquire
2098    /// SeqCst   | SeqCst  | SeqCst
2099    ///
2100    /// `compare_and_swap` and `compare_exchange` also differ in their return type. You can use
2101    /// `compare_exchange(...).unwrap_or_else(|x| x)` to recover the behavior of `compare_and_swap`,
2102    /// but in most cases it is more idiomatic to check whether the return value is `Ok` or `Err`
2103    /// rather than to infer success vs failure based on the value that was read.
2104    ///
2105    /// During migration, consider whether it makes sense to use `compare_exchange_weak` instead.
2106    /// `compare_exchange_weak` is allowed to fail spuriously even when the comparison succeeds,
2107    /// which allows the compiler to generate better assembly code when the compare and swap
2108    /// is used in a loop.
2109    ///
2110    /// # Examples
2111    ///
2112    /// ```
2113    /// use std::sync::atomic::{AtomicPtr, Ordering};
2114    ///
2115    /// let ptr = &mut 5;
2116    /// let some_ptr = AtomicPtr::new(ptr);
2117    ///
2118    /// let other_ptr = &mut 10;
2119    ///
2120    /// let value = some_ptr.compare_and_swap(ptr, other_ptr, Ordering::Relaxed);
2121    /// ```
2122    #[inline]
2123    #[stable(feature = "rust1", since = "1.0.0")]
2124    #[deprecated(
2125        since = "1.50.0",
2126        note = "Use `compare_exchange` or `compare_exchange_weak` instead"
2127    )]
2128    #[cfg(target_has_atomic = "ptr")]
2129    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
2130    #[lint::should_not_be_called_on_const_items]
2131    pub fn compare_and_swap(&self, current: *mut T, new: *mut T, order: Ordering) -> *mut T {
2132        match self.compare_exchange(current, new, order, strongest_failure_ordering(order)) {
2133            Ok(x) => x,
2134            Err(x) => x,
2135        }
2136    }
2137
2138    /// Stores a value into the pointer if the current value is the same as the `current` value.
2139    ///
2140    /// The return value is a result indicating whether the new value was written and containing
2141    /// the previous value. On success this value is guaranteed to be equal to `current`.
2142    ///
2143    /// `compare_exchange` takes two [`Ordering`] arguments to describe the memory
2144    /// ordering of this operation. `success` describes the required ordering for the
2145    /// read-modify-write operation that takes place if the comparison with `current` succeeds.
2146    /// `failure` describes the required ordering for the load operation that takes place when
2147    /// the comparison fails. Using [`Acquire`] as success ordering makes the store part
2148    /// of this operation [`Relaxed`], and using [`Release`] makes the successful load
2149    /// [`Relaxed`]. The failure ordering can only be [`SeqCst`], [`Acquire`] or [`Relaxed`].
2150    ///
2151    /// **Note:** This method is only available on platforms that support atomic
2152    /// operations on pointers.
2153    ///
2154    /// # Examples
2155    ///
2156    /// ```
2157    /// use std::sync::atomic::{AtomicPtr, Ordering};
2158    ///
2159    /// let ptr = &mut 5;
2160    /// let some_ptr = AtomicPtr::new(ptr);
2161    ///
2162    /// let other_ptr = &mut 10;
2163    ///
2164    /// let value = some_ptr.compare_exchange(ptr, other_ptr,
2165    ///                                       Ordering::SeqCst, Ordering::Relaxed);
2166    /// ```
2167    ///
2168    /// # Considerations
2169    ///
2170    /// `compare_exchange` is a [compare-and-swap operation] and thus exhibits the usual downsides
2171    /// of CAS operations. In particular, a load of the value followed by a successful
2172    /// `compare_exchange` with the previous load *does not ensure* that other threads have not
2173    /// changed the value in the interim. This is usually important when the *equality* check in
2174    /// the `compare_exchange` is being used to check the *identity* of a value, but equality
2175    /// does not necessarily imply identity. This is a particularly common case for pointers, as
2176    /// a pointer holding the same address does not imply that the same object exists at that
2177    /// address! In this case, `compare_exchange` can lead to the [ABA problem].
2178    ///
2179    /// [ABA Problem]: https://en.wikipedia.org/wiki/ABA_problem
2180    /// [compare-and-swap operation]: https://en.wikipedia.org/wiki/Compare-and-swap
2181    #[inline]
2182    #[stable(feature = "extended_compare_and_swap", since = "1.10.0")]
2183    #[cfg(target_has_atomic = "ptr")]
2184    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
2185    #[lint::should_not_be_called_on_const_items]
2186    pub fn compare_exchange(
2187        &self,
2188        current: *mut T,
2189        new: *mut T,
2190        success: Ordering,
2191        failure: Ordering,
2192    ) -> Result<*mut T, *mut T> {
2193        // SAFETY: data races are prevented by atomic intrinsics.
2194        unsafe { atomic_compare_exchange(self.as_ptr(), current, new, success, failure) }
2195    }
2196
2197    /// Stores a value into the pointer if the current value is the same as the `current` value.
2198    ///
2199    /// Unlike [`AtomicPtr::compare_exchange`], this function is allowed to spuriously fail even when the
2200    /// comparison succeeds, which can result in more efficient code on some platforms. The
2201    /// return value is a result indicating whether the new value was written and containing the
2202    /// previous value.
2203    ///
2204    /// `compare_exchange_weak` takes two [`Ordering`] arguments to describe the memory
2205    /// ordering of this operation. `success` describes the required ordering for the
2206    /// read-modify-write operation that takes place if the comparison with `current` succeeds.
2207    /// `failure` describes the required ordering for the load operation that takes place when
2208    /// the comparison fails. Using [`Acquire`] as success ordering makes the store part
2209    /// of this operation [`Relaxed`], and using [`Release`] makes the successful load
2210    /// [`Relaxed`]. The failure ordering can only be [`SeqCst`], [`Acquire`] or [`Relaxed`].
2211    ///
2212    /// **Note:** This method is only available on platforms that support atomic
2213    /// operations on pointers.
2214    ///
2215    /// # Examples
2216    ///
2217    /// ```
2218    /// use std::sync::atomic::{AtomicPtr, Ordering};
2219    ///
2220    /// let some_ptr = AtomicPtr::new(&mut 5);
2221    ///
2222    /// let new = &mut 10;
2223    /// let mut old = some_ptr.load(Ordering::Relaxed);
2224    /// loop {
2225    ///     match some_ptr.compare_exchange_weak(old, new, Ordering::SeqCst, Ordering::Relaxed) {
2226    ///         Ok(_) => break,
2227    ///         Err(x) => old = x,
2228    ///     }
2229    /// }
2230    /// ```
2231    ///
2232    /// # Considerations
2233    ///
2234    /// `compare_exchange` is a [compare-and-swap operation] and thus exhibits the usual downsides
2235    /// of CAS operations. In particular, a load of the value followed by a successful
2236    /// `compare_exchange` with the previous load *does not ensure* that other threads have not
2237    /// changed the value in the interim. This is usually important when the *equality* check in
2238    /// the `compare_exchange` is being used to check the *identity* of a value, but equality
2239    /// does not necessarily imply identity. This is a particularly common case for pointers, as
2240    /// a pointer holding the same address does not imply that the same object exists at that
2241    /// address! In this case, `compare_exchange` can lead to the [ABA problem].
2242    ///
2243    /// [ABA Problem]: https://en.wikipedia.org/wiki/ABA_problem
2244    /// [compare-and-swap operation]: https://en.wikipedia.org/wiki/Compare-and-swap
2245    #[inline]
2246    #[stable(feature = "extended_compare_and_swap", since = "1.10.0")]
2247    #[cfg(target_has_atomic = "ptr")]
2248    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
2249    #[lint::should_not_be_called_on_const_items]
2250    pub fn compare_exchange_weak(
2251        &self,
2252        current: *mut T,
2253        new: *mut T,
2254        success: Ordering,
2255        failure: Ordering,
2256    ) -> Result<*mut T, *mut T> {
2257        // SAFETY: This intrinsic is unsafe because it operates on a raw pointer
2258        // but we know for sure that the pointer is valid (we just got it from
2259        // an `UnsafeCell` that we have by reference) and the atomic operation
2260        // itself allows us to safely mutate the `UnsafeCell` contents.
2261        unsafe { atomic_compare_exchange_weak(self.as_ptr(), current, new, success, failure) }
2262    }
2263
2264    /// An alias for [`AtomicPtr::try_update`].
2265    #[inline]
2266    #[stable(feature = "atomic_fetch_update", since = "1.53.0")]
2267    #[cfg(target_has_atomic = "ptr")]
2268    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
2269    #[lint::should_not_be_called_on_const_items]
2270    #[deprecated(
2271        since = "1.99.0",
2272        note = "renamed to `try_update` for consistency",
2273        suggestion = "try_update"
2274    )]
2275    pub fn fetch_update<F>(
2276        &self,
2277        set_order: Ordering,
2278        fetch_order: Ordering,
2279        f: F,
2280    ) -> Result<*mut T, *mut T>
2281    where
2282        F: FnMut(*mut T) -> Option<*mut T>,
2283    {
2284        self.try_update(set_order, fetch_order, f)
2285    }
2286    /// Fetches the value, and applies a function to it that returns an optional
2287    /// new value. Returns a `Result` of `Ok(previous_value)` if the function
2288    /// returned `Some(_)`, else `Err(previous_value)`.
2289    ///
2290    /// See also: [`update`](`AtomicPtr::update`).
2291    ///
2292    /// Note: This may call the function multiple times if the value has been
2293    /// changed from other threads in the meantime, as long as the function
2294    /// returns `Some(_)`, but the function will have been applied only once to
2295    /// the stored value.
2296    ///
2297    /// `try_update` takes two [`Ordering`] arguments to describe the memory
2298    /// ordering of this operation. The first describes the required ordering for
2299    /// when the operation finally succeeds while the second describes the
2300    /// required ordering for loads. These correspond to the success and failure
2301    /// orderings of [`AtomicPtr::compare_exchange`] respectively.
2302    ///
2303    /// Using [`Acquire`] as success ordering makes the store part of this
2304    /// operation [`Relaxed`], and using [`Release`] makes the final successful
2305    /// load [`Relaxed`]. The (failed) load ordering can only be [`SeqCst`],
2306    /// [`Acquire`] or [`Relaxed`].
2307    ///
2308    /// **Note:** This method is only available on platforms that support atomic
2309    /// operations on pointers.
2310    ///
2311    /// # Considerations
2312    ///
2313    /// This method is not magic; it is not provided by the hardware, and does not act like a
2314    /// critical section or mutex.
2315    ///
2316    /// It is implemented on top of an atomic [compare-and-swap operation], and thus is subject to
2317    /// the usual drawbacks of CAS operations. In particular, be careful of the [ABA problem],
2318    /// which is a particularly common pitfall for pointers!
2319    ///
2320    /// [ABA Problem]: https://en.wikipedia.org/wiki/ABA_problem
2321    /// [compare-and-swap operation]: https://en.wikipedia.org/wiki/Compare-and-swap
2322    ///
2323    /// # Examples
2324    ///
2325    /// ```rust
2326    /// use std::sync::atomic::{AtomicPtr, Ordering};
2327    ///
2328    /// let ptr: *mut _ = &mut 5;
2329    /// let some_ptr = AtomicPtr::new(ptr);
2330    ///
2331    /// let new: *mut _ = &mut 10;
2332    /// assert_eq!(some_ptr.try_update(Ordering::SeqCst, Ordering::SeqCst, |_| None), Err(ptr));
2333    /// let result = some_ptr.try_update(Ordering::SeqCst, Ordering::SeqCst, |x| {
2334    ///     if x == ptr {
2335    ///         Some(new)
2336    ///     } else {
2337    ///         None
2338    ///     }
2339    /// });
2340    /// assert_eq!(result, Ok(ptr));
2341    /// assert_eq!(some_ptr.load(Ordering::SeqCst), new);
2342    /// ```
2343    #[inline]
2344    #[stable(feature = "atomic_try_update", since = "1.95.0")]
2345    #[cfg(target_has_atomic = "ptr")]
2346    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
2347    #[lint::should_not_be_called_on_const_items]
2348    pub fn try_update(
2349        &self,
2350        set_order: Ordering,
2351        fetch_order: Ordering,
2352        mut f: impl FnMut(*mut T) -> Option<*mut T>,
2353    ) -> Result<*mut T, *mut T> {
2354        let mut prev = self.load(fetch_order);
2355        while let Some(next) = f(prev) {
2356            match self.compare_exchange_weak(prev, next, set_order, fetch_order) {
2357                x @ Ok(_) => return x,
2358                Err(next_prev) => prev = next_prev,
2359            }
2360        }
2361        Err(prev)
2362    }
2363
2364    /// Fetches the value, and applies a function to it that returns a new value.
2365    /// The new value is stored and the old value is returned.
2366    ///
2367    /// See also: [`try_update`](`AtomicPtr::try_update`).
2368    ///
2369    /// Note: This may call the function multiple times if the value has been changed from other threads in
2370    /// the meantime, but the function will have been applied only once to the stored value.
2371    ///
2372    /// `update` takes two [`Ordering`] arguments to describe the memory
2373    /// ordering of this operation. The first describes the required ordering for
2374    /// when the operation finally succeeds while the second describes the
2375    /// required ordering for loads. These correspond to the success and failure
2376    /// orderings of [`AtomicPtr::compare_exchange`] respectively.
2377    ///
2378    /// Using [`Acquire`] as success ordering makes the store part
2379    /// of this operation [`Relaxed`], and using [`Release`] makes the final successful load
2380    /// [`Relaxed`]. The (failed) load ordering can only be [`SeqCst`], [`Acquire`] or [`Relaxed`].
2381    ///
2382    /// **Note:** This method is only available on platforms that support atomic
2383    /// operations on pointers.
2384    ///
2385    /// # Considerations
2386    ///
2387    /// This method is not magic; it is not provided by the hardware, and does not act like a
2388    /// critical section or mutex.
2389    ///
2390    /// It is implemented on top of an atomic [compare-and-swap operation], and thus is subject to
2391    /// the usual drawbacks of CAS operations. In particular, be careful of the [ABA problem],
2392    /// which is a particularly common pitfall for pointers!
2393    ///
2394    /// [ABA Problem]: https://en.wikipedia.org/wiki/ABA_problem
2395    /// [compare-and-swap operation]: https://en.wikipedia.org/wiki/Compare-and-swap
2396    ///
2397    /// # Examples
2398    ///
2399    /// ```rust
2400    ///
2401    /// use std::sync::atomic::{AtomicPtr, Ordering};
2402    ///
2403    /// let ptr: *mut _ = &mut 5;
2404    /// let some_ptr = AtomicPtr::new(ptr);
2405    ///
2406    /// let new: *mut _ = &mut 10;
2407    /// let result = some_ptr.update(Ordering::SeqCst, Ordering::SeqCst, |_| new);
2408    /// assert_eq!(result, ptr);
2409    /// assert_eq!(some_ptr.load(Ordering::SeqCst), new);
2410    /// ```
2411    #[inline]
2412    #[stable(feature = "atomic_try_update", since = "1.95.0")]
2413    #[cfg(target_has_atomic = "ptr")]
2414    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
2415    #[lint::should_not_be_called_on_const_items]
2416    pub fn update(
2417        &self,
2418        set_order: Ordering,
2419        fetch_order: Ordering,
2420        mut f: impl FnMut(*mut T) -> *mut T,
2421    ) -> *mut T {
2422        let mut prev = self.load(fetch_order);
2423        loop {
2424            match self.compare_exchange_weak(prev, f(prev), set_order, fetch_order) {
2425                Ok(x) => break x,
2426                Err(next_prev) => prev = next_prev,
2427            }
2428        }
2429    }
2430
2431    /// Offsets the pointer's address by adding `val` (in units of `T`),
2432    /// returning the previous pointer.
2433    ///
2434    /// This is equivalent to using [`wrapping_add`] to atomically perform the
2435    /// equivalent of `ptr = ptr.wrapping_add(val);`.
2436    ///
2437    /// This method operates in units of `T`, which means that it cannot be used
2438    /// to offset the pointer by an amount which is not a multiple of
2439    /// `size_of::<T>()`. This can sometimes be inconvenient, as you may want to
2440    /// work with a deliberately misaligned pointer. In such cases, you may use
2441    /// the [`fetch_byte_add`](Self::fetch_byte_add) method instead.
2442    ///
2443    /// `fetch_ptr_add` takes an [`Ordering`] argument which describes the
2444    /// memory ordering of this operation. All ordering modes are possible. Note
2445    /// that using [`Acquire`] makes the store part of this operation
2446    /// [`Relaxed`], and using [`Release`] makes the load part [`Relaxed`].
2447    ///
2448    /// **Note**: This method is only available on platforms that support atomic
2449    /// operations on [`AtomicPtr`].
2450    ///
2451    /// [`wrapping_add`]: pointer::wrapping_add
2452    ///
2453    /// # Examples
2454    ///
2455    /// ```
2456    /// use core::sync::atomic::{AtomicPtr, Ordering};
2457    ///
2458    /// let atom = AtomicPtr::<i64>::new(core::ptr::null_mut());
2459    /// assert_eq!(atom.fetch_ptr_add(1, Ordering::Relaxed).addr(), 0);
2460    /// // Note: units of `size_of::<i64>()`.
2461    /// assert_eq!(atom.load(Ordering::Relaxed).addr(), 8);
2462    /// ```
2463    #[inline]
2464    #[cfg(target_has_atomic = "ptr")]
2465    #[stable(feature = "strict_provenance_atomic_ptr", since = "1.91.0")]
2466    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
2467    #[lint::should_not_be_called_on_const_items]
2468    pub fn fetch_ptr_add(&self, val: usize, order: Ordering) -> *mut T {
2469        self.fetch_byte_add(val.wrapping_mul(size_of::<T>()), order)
2470    }
2471
2472    /// Offsets the pointer's address by subtracting `val` (in units of `T`),
2473    /// returning the previous pointer.
2474    ///
2475    /// This is equivalent to using [`wrapping_sub`] to atomically perform the
2476    /// equivalent of `ptr = ptr.wrapping_sub(val);`.
2477    ///
2478    /// This method operates in units of `T`, which means that it cannot be used
2479    /// to offset the pointer by an amount which is not a multiple of
2480    /// `size_of::<T>()`. This can sometimes be inconvenient, as you may want to
2481    /// work with a deliberately misaligned pointer. In such cases, you may use
2482    /// the [`fetch_byte_sub`](Self::fetch_byte_sub) method instead.
2483    ///
2484    /// `fetch_ptr_sub` takes an [`Ordering`] argument which describes the memory
2485    /// ordering of this operation. All ordering modes are possible. Note that
2486    /// using [`Acquire`] makes the store part of this operation [`Relaxed`],
2487    /// and using [`Release`] makes the load part [`Relaxed`].
2488    ///
2489    /// **Note**: This method is only available on platforms that support atomic
2490    /// operations on [`AtomicPtr`].
2491    ///
2492    /// [`wrapping_sub`]: pointer::wrapping_sub
2493    ///
2494    /// # Examples
2495    ///
2496    /// ```
2497    /// use core::sync::atomic::{AtomicPtr, Ordering};
2498    ///
2499    /// let array = [1i32, 2i32];
2500    /// let atom = AtomicPtr::new(array.as_ptr().wrapping_add(1) as *mut _);
2501    ///
2502    /// assert!(core::ptr::eq(
2503    ///     atom.fetch_ptr_sub(1, Ordering::Relaxed),
2504    ///     &array[1],
2505    /// ));
2506    /// assert!(core::ptr::eq(atom.load(Ordering::Relaxed), &array[0]));
2507    /// ```
2508    #[inline]
2509    #[cfg(target_has_atomic = "ptr")]
2510    #[stable(feature = "strict_provenance_atomic_ptr", since = "1.91.0")]
2511    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
2512    #[lint::should_not_be_called_on_const_items]
2513    pub fn fetch_ptr_sub(&self, val: usize, order: Ordering) -> *mut T {
2514        self.fetch_byte_sub(val.wrapping_mul(size_of::<T>()), order)
2515    }
2516
2517    /// Offsets the pointer's address by adding `val` *bytes*, returning the
2518    /// previous pointer.
2519    ///
2520    /// This is equivalent to using [`wrapping_byte_add`] to atomically
2521    /// perform `ptr = ptr.wrapping_byte_add(val)`.
2522    ///
2523    /// `fetch_byte_add` takes an [`Ordering`] argument which describes the
2524    /// memory ordering of this operation. All ordering modes are possible. Note
2525    /// that using [`Acquire`] makes the store part of this operation
2526    /// [`Relaxed`], and using [`Release`] makes the load part [`Relaxed`].
2527    ///
2528    /// **Note**: This method is only available on platforms that support atomic
2529    /// operations on [`AtomicPtr`].
2530    ///
2531    /// [`wrapping_byte_add`]: pointer::wrapping_byte_add
2532    ///
2533    /// # Examples
2534    ///
2535    /// ```
2536    /// use core::sync::atomic::{AtomicPtr, Ordering};
2537    ///
2538    /// let atom = AtomicPtr::<i64>::new(core::ptr::null_mut());
2539    /// assert_eq!(atom.fetch_byte_add(1, Ordering::Relaxed).addr(), 0);
2540    /// // Note: in units of bytes, not `size_of::<i64>()`.
2541    /// assert_eq!(atom.load(Ordering::Relaxed).addr(), 1);
2542    /// ```
2543    #[inline]
2544    #[cfg(target_has_atomic = "ptr")]
2545    #[stable(feature = "strict_provenance_atomic_ptr", since = "1.91.0")]
2546    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
2547    #[lint::should_not_be_called_on_const_items]
2548    pub fn fetch_byte_add(&self, val: usize, order: Ordering) -> *mut T {
2549        // SAFETY: data races are prevented by atomic intrinsics.
2550        unsafe { atomic_add(self.as_ptr(), val, order).cast() }
2551    }
2552
2553    /// Offsets the pointer's address by subtracting `val` *bytes*, returning the
2554    /// previous pointer.
2555    ///
2556    /// This is equivalent to using [`wrapping_byte_sub`] to atomically
2557    /// perform `ptr = ptr.wrapping_byte_sub(val)`.
2558    ///
2559    /// `fetch_byte_sub` takes an [`Ordering`] argument which describes the
2560    /// memory ordering of this operation. All ordering modes are possible. Note
2561    /// that using [`Acquire`] makes the store part of this operation
2562    /// [`Relaxed`], and using [`Release`] makes the load part [`Relaxed`].
2563    ///
2564    /// **Note**: This method is only available on platforms that support atomic
2565    /// operations on [`AtomicPtr`].
2566    ///
2567    /// [`wrapping_byte_sub`]: pointer::wrapping_byte_sub
2568    ///
2569    /// # Examples
2570    ///
2571    /// ```
2572    /// use core::sync::atomic::{AtomicPtr, Ordering};
2573    ///
2574    /// let mut arr = [0i64, 1];
2575    /// let atom = AtomicPtr::<i64>::new(&raw mut arr[1]);
2576    /// assert_eq!(atom.fetch_byte_sub(8, Ordering::Relaxed).addr(), (&raw const arr[1]).addr());
2577    /// assert_eq!(atom.load(Ordering::Relaxed).addr(), (&raw const arr[0]).addr());
2578    /// ```
2579    #[inline]
2580    #[cfg(target_has_atomic = "ptr")]
2581    #[stable(feature = "strict_provenance_atomic_ptr", since = "1.91.0")]
2582    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
2583    #[lint::should_not_be_called_on_const_items]
2584    pub fn fetch_byte_sub(&self, val: usize, order: Ordering) -> *mut T {
2585        // SAFETY: data races are prevented by atomic intrinsics.
2586        unsafe { atomic_sub(self.as_ptr(), val, order).cast() }
2587    }
2588
2589    /// Performs a bitwise "or" operation on the address of the current pointer,
2590    /// and the argument `val`, and stores a pointer with provenance of the
2591    /// current pointer and the resulting address.
2592    ///
2593    /// This is equivalent to using [`map_addr`] to atomically perform
2594    /// `ptr = ptr.map_addr(|a| a | val)`. This can be used in tagged
2595    /// pointer schemes to atomically set tag bits.
2596    ///
2597    /// **Caveat**: This operation returns the previous value. To compute the
2598    /// stored value without losing provenance, you may use [`map_addr`]. For
2599    /// example: `a.fetch_or(val).map_addr(|a| a | val)`.
2600    ///
2601    /// `fetch_or` takes an [`Ordering`] argument which describes the memory
2602    /// ordering of this operation. All ordering modes are possible. Note that
2603    /// using [`Acquire`] makes the store part of this operation [`Relaxed`],
2604    /// and using [`Release`] makes the load part [`Relaxed`].
2605    ///
2606    /// **Note**: This method is only available on platforms that support atomic
2607    /// operations on [`AtomicPtr`].
2608    ///
2609    /// This API and its claimed semantics are part of the Strict Provenance
2610    /// experiment, see the [module documentation for `ptr`][crate::ptr] for
2611    /// details.
2612    ///
2613    /// [`map_addr`]: pointer::map_addr
2614    ///
2615    /// # Examples
2616    ///
2617    /// ```
2618    /// use core::sync::atomic::{AtomicPtr, Ordering};
2619    ///
2620    /// let pointer = &mut 3i64 as *mut i64;
2621    ///
2622    /// let atom = AtomicPtr::<i64>::new(pointer);
2623    /// // Tag the bottom bit of the pointer.
2624    /// assert_eq!(atom.fetch_or(1, Ordering::Relaxed).addr() & 1, 0);
2625    /// // Extract and untag.
2626    /// let tagged = atom.load(Ordering::Relaxed);
2627    /// assert_eq!(tagged.addr() & 1, 1);
2628    /// assert_eq!(tagged.map_addr(|p| p & !1), pointer);
2629    /// ```
2630    #[inline]
2631    #[cfg(target_has_atomic = "ptr")]
2632    #[stable(feature = "strict_provenance_atomic_ptr", since = "1.91.0")]
2633    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
2634    #[lint::should_not_be_called_on_const_items]
2635    pub fn fetch_or(&self, val: usize, order: Ordering) -> *mut T {
2636        // SAFETY: data races are prevented by atomic intrinsics.
2637        unsafe { atomic_or(self.as_ptr(), val, order).cast() }
2638    }
2639
2640    /// Performs a bitwise "and" operation on the address of the current
2641    /// pointer, and the argument `val`, and stores a pointer with provenance of
2642    /// the current pointer and the resulting address.
2643    ///
2644    /// This is equivalent to using [`map_addr`] to atomically perform
2645    /// `ptr = ptr.map_addr(|a| a & val)`. This can be used in tagged
2646    /// pointer schemes to atomically unset tag bits.
2647    ///
2648    /// **Caveat**: This operation returns the previous value. To compute the
2649    /// stored value without losing provenance, you may use [`map_addr`]. For
2650    /// example: `a.fetch_and(val).map_addr(|a| a & val)`.
2651    ///
2652    /// `fetch_and` takes an [`Ordering`] argument which describes the memory
2653    /// ordering of this operation. All ordering modes are possible. Note that
2654    /// using [`Acquire`] makes the store part of this operation [`Relaxed`],
2655    /// and using [`Release`] makes the load part [`Relaxed`].
2656    ///
2657    /// **Note**: This method is only available on platforms that support atomic
2658    /// operations on [`AtomicPtr`].
2659    ///
2660    /// This API and its claimed semantics are part of the Strict Provenance
2661    /// experiment, see the [module documentation for `ptr`][crate::ptr] for
2662    /// details.
2663    ///
2664    /// [`map_addr`]: pointer::map_addr
2665    ///
2666    /// # Examples
2667    ///
2668    /// ```
2669    /// use core::sync::atomic::{AtomicPtr, Ordering};
2670    ///
2671    /// let pointer = &mut 3i64 as *mut i64;
2672    /// // A tagged pointer
2673    /// let atom = AtomicPtr::<i64>::new(pointer.map_addr(|a| a | 1));
2674    /// assert_eq!(atom.fetch_or(1, Ordering::Relaxed).addr() & 1, 1);
2675    /// // Untag, and extract the previously tagged pointer.
2676    /// let untagged = atom.fetch_and(!1, Ordering::Relaxed)
2677    ///     .map_addr(|a| a & !1);
2678    /// assert_eq!(untagged, pointer);
2679    /// ```
2680    #[inline]
2681    #[cfg(target_has_atomic = "ptr")]
2682    #[stable(feature = "strict_provenance_atomic_ptr", since = "1.91.0")]
2683    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
2684    #[lint::should_not_be_called_on_const_items]
2685    pub fn fetch_and(&self, val: usize, order: Ordering) -> *mut T {
2686        // SAFETY: data races are prevented by atomic intrinsics.
2687        unsafe { atomic_and(self.as_ptr(), val, order).cast() }
2688    }
2689
2690    /// Performs a bitwise "xor" operation on the address of the current
2691    /// pointer, and the argument `val`, and stores a pointer with provenance of
2692    /// the current pointer and the resulting address.
2693    ///
2694    /// This is equivalent to using [`map_addr`] to atomically perform
2695    /// `ptr = ptr.map_addr(|a| a ^ val)`. This can be used in tagged
2696    /// pointer schemes to atomically toggle tag bits.
2697    ///
2698    /// **Caveat**: This operation returns the previous value. To compute the
2699    /// stored value without losing provenance, you may use [`map_addr`]. For
2700    /// example: `a.fetch_xor(val).map_addr(|a| a ^ val)`.
2701    ///
2702    /// `fetch_xor` takes an [`Ordering`] argument which describes the memory
2703    /// ordering of this operation. All ordering modes are possible. Note that
2704    /// using [`Acquire`] makes the store part of this operation [`Relaxed`],
2705    /// and using [`Release`] makes the load part [`Relaxed`].
2706    ///
2707    /// **Note**: This method is only available on platforms that support atomic
2708    /// operations on [`AtomicPtr`].
2709    ///
2710    /// This API and its claimed semantics are part of the Strict Provenance
2711    /// experiment, see the [module documentation for `ptr`][crate::ptr] for
2712    /// details.
2713    ///
2714    /// [`map_addr`]: pointer::map_addr
2715    ///
2716    /// # Examples
2717    ///
2718    /// ```
2719    /// use core::sync::atomic::{AtomicPtr, Ordering};
2720    ///
2721    /// let pointer = &mut 3i64 as *mut i64;
2722    /// let atom = AtomicPtr::<i64>::new(pointer);
2723    ///
2724    /// // Toggle a tag bit on the pointer.
2725    /// atom.fetch_xor(1, Ordering::Relaxed);
2726    /// assert_eq!(atom.load(Ordering::Relaxed).addr() & 1, 1);
2727    /// ```
2728    #[inline]
2729    #[cfg(target_has_atomic = "ptr")]
2730    #[stable(feature = "strict_provenance_atomic_ptr", since = "1.91.0")]
2731    #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
2732    #[lint::should_not_be_called_on_const_items]
2733    pub fn fetch_xor(&self, val: usize, order: Ordering) -> *mut T {
2734        // SAFETY: data races are prevented by atomic intrinsics.
2735        unsafe { atomic_xor(self.as_ptr(), val, order).cast() }
2736    }
2737
2738    /// Returns a mutable pointer to the underlying pointer.
2739    ///
2740    /// Doing non-atomic reads and writes on the resulting pointer can be a data race.
2741    /// This method is mostly useful for FFI, where the function signature may use
2742    /// `*mut *mut T` instead of `&AtomicPtr<T>`.
2743    ///
2744    /// Returning an `*mut` pointer from a shared reference to this atomic is safe because the
2745    /// atomic types work with interior mutability. All modifications of an atomic change the value
2746    /// through a shared reference, and can do so safely as long as they use atomic operations. Any
2747    /// use of the returned raw pointer requires an `unsafe` block and still has to uphold the
2748    /// requirements of the [memory model].
2749    ///
2750    /// # Examples
2751    ///
2752    /// ```ignore (extern-declaration)
2753    /// use std::sync::atomic::AtomicPtr;
2754    ///
2755    /// extern "C" {
2756    ///     fn my_atomic_op(arg: *mut *mut u32);
2757    /// }
2758    ///
2759    /// let mut value = 17;
2760    /// let atomic = AtomicPtr::new(&mut value);
2761    ///
2762    /// // SAFETY: Safe as long as `my_atomic_op` is atomic.
2763    /// unsafe {
2764    ///     my_atomic_op(atomic.as_ptr());
2765    /// }
2766    /// ```
2767    ///
2768    /// [memory model]: self#memory-model-for-atomic-accesses
2769    #[inline]
2770    #[stable(feature = "atomic_as_ptr", since = "1.70.0")]
2771    #[rustc_const_stable(feature = "atomic_as_ptr", since = "1.70.0")]
2772    #[lint::never_returns_null_ptr]
2773    pub const fn as_ptr(&self) -> *mut *mut T {
2774        self.v.get().cast()
2775    }
2776}
2777
2778#[cfg(target_has_atomic_load_store = "8")]
2779#[stable(feature = "atomic_bool_from", since = "1.24.0")]
2780#[rustc_const_unstable(feature = "const_convert", issue = "143773")]
2781const impl From<bool> for AtomicBool {
2782    /// Converts a `bool` into an `AtomicBool`.
2783    ///
2784    /// # Examples
2785    ///
2786    /// ```
2787    /// use std::sync::atomic::AtomicBool;
2788    /// let atomic_bool = AtomicBool::from(true);
2789    /// assert_eq!(format!("{atomic_bool:?}"), "true")
2790    /// ```
2791    #[inline]
2792    fn from(b: bool) -> Self {
2793        Self::new(b)
2794    }
2795}
2796
2797#[cfg(target_has_atomic_load_store = "ptr")]
2798#[stable(feature = "atomic_from", since = "1.23.0")]
2799#[rustc_const_unstable(feature = "const_convert", issue = "143773")]
2800const impl<T> From<*mut T> for AtomicPtr<T> {
2801    /// Converts a `*mut T` into an `AtomicPtr<T>`.
2802    #[inline]
2803    fn from(p: *mut T) -> Self {
2804        Self::new(p)
2805    }
2806}
2807
2808#[allow(unused_macros)] // This macro ends up being unused on some architectures.
2809macro_rules! if_8_bit {
2810    (u8, $( yes = [$($yes:tt)*], )? $( no = [$($no:tt)*], )? ) => { concat!("", $($($yes)*)?) };
2811    (i8, $( yes = [$($yes:tt)*], )? $( no = [$($no:tt)*], )? ) => { concat!("", $($($yes)*)?) };
2812    ($_:ident, $( yes = [$($yes:tt)*], )? $( no = [$($no:tt)*], )? ) => { concat!("", $($($no)*)?) };
2813}
2814
2815#[cfg(target_has_atomic_load_store)]
2816macro_rules! atomic_int {
2817    ($cfg_base:meta,
2818     $cfg_cas:meta,
2819     $cfg_align:meta,
2820     $stable:meta,
2821     $stable_cxchg:meta,
2822     $stable_debug:meta,
2823     $stable_access:meta,
2824     $stable_from:meta,
2825     $stable_nand:meta,
2826     $const_stable_new:meta,
2827     $const_stable_into_inner:meta,
2828     $s_int_type:literal,
2829     $extra_feature:expr,
2830     $min_fn:ident, $max_fn:ident,
2831     $align:expr,
2832     $int_type:ident $atomic_type:ident) => {
2833        /// An integer type which can be safely shared between threads.
2834        ///
2835        /// This type has the same
2836        #[doc = if_8_bit!(
2837            $int_type,
2838            yes = ["size, alignment, and bit validity"],
2839            no = ["size and bit validity"],
2840        )]
2841        /// as the underlying integer type, [`
2842        #[doc = $s_int_type]
2843        /// `].
2844        #[doc = if_8_bit! {
2845            $int_type,
2846            no = [
2847                "However, the alignment of this type is always equal to its ",
2848                "size, even on targets where [`", $s_int_type, "`] has a ",
2849                "lesser alignment."
2850            ],
2851        }]
2852        ///
2853        /// For more about the differences between atomic types and
2854        /// non-atomic types as well as information about the portability of
2855        /// this type, please see the [module-level documentation].
2856        ///
2857        /// **Note:** This type is only available on platforms that support
2858        /// atomic loads and stores of [`
2859        #[doc = $s_int_type]
2860        /// `].
2861        ///
2862        /// [module-level documentation]: crate::sync::atomic
2863        #[$stable]
2864        pub type $atomic_type = Atomic<$int_type>;
2865
2866        #[$stable]
2867        impl Default for $atomic_type {
2868            #[inline]
2869            #[ferrocene::prevalidated]
2870            fn default() -> Self {
2871                Self::new(Default::default())
2872            }
2873        }
2874
2875        #[$stable_from]
2876        #[rustc_const_unstable(feature = "const_convert", issue = "143773")]
2877        const impl From<$int_type> for $atomic_type {
2878            #[doc = concat!("Converts an `", stringify!($int_type), "` into an `", stringify!($atomic_type), "`.")]
2879            #[inline]
2880            #[ferrocene::prevalidated]
2881            fn from(v: $int_type) -> Self { Self::new(v) }
2882        }
2883
2884        #[$stable_debug]
2885        impl fmt::Debug for $atomic_type {
2886            #[ferrocene::prevalidated]
2887            fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
2888                fmt::Debug::fmt(&self.load(Ordering::Relaxed), f)
2889            }
2890        }
2891
2892        impl $atomic_type {
2893            /// Creates a new atomic integer.
2894            ///
2895            /// # Examples
2896            ///
2897            #[cfg_attr($cfg_base, doc = "```")]
2898            #[cfg_attr(not($cfg_base), doc = "```compile_fail")]
2899            #[doc = concat!($extra_feature, "use std::sync::atomic::", stringify!($atomic_type), ";")]
2900            ///
2901            #[doc = concat!("let atomic_forty_two = ", stringify!($atomic_type), "::new(42);")]
2902            /// ```
2903            #[inline]
2904            #[$stable]
2905            #[$const_stable_new]
2906            #[must_use]
2907            #[ferrocene::prevalidated]
2908            pub const fn new(v: $int_type) -> Self {
2909                // SAFETY:
2910                // `Atomic<T>` is essentially a transparent wrapper around `T`.
2911                unsafe { transmute(v) }
2912            }
2913
2914            /// Creates a new reference to an atomic integer from a pointer.
2915            ///
2916            /// # Examples
2917            ///
2918            #[cfg_attr($cfg_base, doc = "```rust")]
2919            #[cfg_attr(not($cfg_base), doc = "```rust,compile_fail")]
2920            #[doc = concat!($extra_feature, "use std::sync::atomic::{self, ", stringify!($atomic_type), "};")]
2921            ///
2922            /// // Get a pointer to an allocated value
2923            #[doc = concat!("let ptr: *mut ", stringify!($int_type), " = Box::into_raw(Box::new(0));")]
2924            ///
2925            #[doc = concat!("assert!(ptr.cast::<", stringify!($atomic_type), ">().is_aligned());")]
2926            ///
2927            /// {
2928            ///     // Create an atomic view of the allocated value
2929            // SAFETY: this is a doc comment, tidy, it can't hurt you (also guaranteed by the construction of `ptr` and the assert above)
2930            #[doc = concat!("    let atomic = unsafe {", stringify!($atomic_type), "::from_ptr(ptr) };")]
2931            ///
2932            ///     // Use `atomic` for atomic operations, possibly share it with other threads
2933            ///     atomic.store(1, atomic::Ordering::Relaxed);
2934            /// }
2935            ///
2936            /// // It's ok to non-atomically access the value behind `ptr`,
2937            /// // since the reference to the atomic ended its lifetime in the block above
2938            /// assert_eq!(unsafe { *ptr }, 1);
2939            ///
2940            /// // Deallocate the value
2941            /// unsafe { drop(Box::from_raw(ptr)) }
2942            /// ```
2943            ///
2944            /// # Safety
2945            ///
2946            /// * `ptr` must be aligned to
2947            #[doc = concat!("  `align_of::<", stringify!($atomic_type), ">()`")]
2948            #[doc = if_8_bit!{
2949                $int_type,
2950                yes = [
2951                    "  (note that this is always true, since `align_of::<",
2952                    stringify!($atomic_type), ">() == 1`)."
2953                ],
2954                no = [
2955                    "  (note that on some platforms this can be bigger than `align_of::<",
2956                    stringify!($int_type), ">()`)."
2957                ],
2958            }]
2959            /// * `ptr` must be [valid] for both reads and writes for the whole lifetime `'a`.
2960            /// * You must adhere to the [Memory model for atomic accesses]. In particular, it is not
2961            ///   allowed to mix conflicting atomic and non-atomic accesses, or atomic accesses of different
2962            ///   sizes, without synchronization.
2963            ///
2964            /// [valid]: crate::ptr#safety
2965            /// [Memory model for atomic accesses]: self#memory-model-for-atomic-accesses
2966            #[inline]
2967            #[stable(feature = "atomic_from_ptr", since = "1.75.0")]
2968            #[rustc_const_stable(feature = "const_atomic_from_ptr", since = "1.84.0")]
2969            #[ferrocene::prevalidated]
2970            pub const unsafe fn from_ptr<'a>(ptr: *mut $int_type) -> &'a $atomic_type {
2971                // SAFETY: guaranteed by the caller
2972                unsafe { &*ptr.cast() }
2973            }
2974
2975            /// Creates a new pointer to an atomic integer from a pointer.
2976            ///
2977            /// This is useful if you want to do volatile atomic accesses, and thus avoid creating
2978            /// a reference to the destination.
2979            #[inline]
2980            #[unstable(feature = "atomic_volatile", issue = "158947")]
2981            pub const fn from_ptr_raw(ptr: *mut $int_type) -> *const $atomic_type {
2982                ptr.cast_const().cast()
2983            }
2984
2985            /// Returns a mutable reference to the underlying integer.
2986            ///
2987            /// This is safe because the mutable reference guarantees that no other threads are
2988            /// concurrently accessing the atomic data.
2989            ///
2990            /// # Examples
2991            ///
2992            #[cfg_attr($cfg_base, doc = "```")]
2993            #[cfg_attr(not($cfg_base), doc = "```compile_fail")]
2994            #[doc = concat!($extra_feature, "use std::sync::atomic::{", stringify!($atomic_type), ", Ordering};")]
2995            ///
2996            #[doc = concat!("let mut some_var = ", stringify!($atomic_type), "::new(10);")]
2997            /// assert_eq!(*some_var.get_mut(), 10);
2998            /// *some_var.get_mut() = 5;
2999            /// assert_eq!(some_var.load(Ordering::SeqCst), 5);
3000            /// ```
3001            #[inline]
3002            #[$stable_access]
3003            #[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
3004            #[ferrocene::prevalidated]
3005            pub const fn get_mut(&mut self) -> &mut $int_type {
3006                // SAFETY:
3007                // `Atomic<T>` is essentially a transparent wrapper around `T`.
3008                unsafe { &mut *self.as_ptr() }
3009            }
3010
3011            #[doc = concat!("Get atomic access to a `&mut ", stringify!($int_type), "`.")]
3012            ///
3013            #[doc = if_8_bit! {
3014                $int_type,
3015                no = [
3016                    "**Note:** This function is only available on targets where `",
3017                    stringify!($atomic_type), "` has the same alignment as `", stringify!($int_type), "`."
3018                ],
3019            }]
3020            ///
3021            /// # Examples
3022            ///
3023            #[cfg_attr($cfg_align, doc = "```rust")]
3024            #[cfg_attr(not($cfg_align), doc = "```rust,compile_fail")]
3025            #[doc = concat!($extra_feature, "use std::sync::atomic::{", stringify!($atomic_type), ", Ordering};")]
3026            ///
3027            /// let mut some_int = 123;
3028            #[doc = concat!("let a = ", stringify!($atomic_type), "::from_mut(&mut some_int);")]
3029            /// a.store(100, Ordering::Relaxed);
3030            /// assert_eq!(some_int, 100);
3031            /// ```
3032            ///
3033            #[ferrocene::prevalidated]
3034            #[inline]
3035            #[cfg(any($cfg_align, doc))]
3036            #[stable(feature = "atomic_from_mut", since = "1.98.0")]
3037            #[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
3038            pub const fn from_mut(v: &mut $int_type) -> &mut Self {
3039                let [] = [(); align_of::<Self>() - align_of::<$int_type>()];
3040                // SAFETY:
3041                //  - the mutable reference guarantees unique ownership.
3042                //  - the alignment of `$int_type` and `Self` is the
3043                //    same, as promised by $cfg_align and verified above.
3044                unsafe { &mut *(v as *mut $int_type as *mut Self) }
3045            }
3046
3047            #[doc = concat!("Get non-atomic access to a `&mut [", stringify!($atomic_type), "]` slice")]
3048            ///
3049            /// This is safe because the mutable reference guarantees that no other threads are
3050            /// concurrently accessing the atomic data.
3051            ///
3052            /// # Examples
3053            ///
3054            #[cfg_attr($cfg_base, doc = "```ignore-wasm")]
3055            #[cfg_attr(not($cfg_base), doc = "```ignore-wasm,compile_fail")]
3056            #[doc = concat!($extra_feature, "use std::sync::atomic::{", stringify!($atomic_type), ", Ordering};")]
3057            ///
3058            #[doc = concat!("let mut some_ints = [const { ", stringify!($atomic_type), "::new(0) }; 10];")]
3059            ///
3060            #[doc = concat!("let view: &mut [", stringify!($int_type), "] = ", stringify!($atomic_type), "::get_mut_slice(&mut some_ints);")]
3061            /// assert_eq!(view, [0; 10]);
3062            /// view
3063            ///     .iter_mut()
3064            ///     .enumerate()
3065            ///     .for_each(|(idx, int)| *int = idx as _);
3066            ///
3067            /// std::thread::scope(|s| {
3068            ///     some_ints
3069            ///         .iter()
3070            ///         .enumerate()
3071            ///         .for_each(|(idx, int)| {
3072            ///             s.spawn(move || assert_eq!(int.load(Ordering::Relaxed), idx as _));
3073            ///         })
3074            /// });
3075            /// ```
3076            #[ferrocene::prevalidated]
3077            #[inline]
3078            #[stable(feature = "atomic_from_mut", since = "1.98.0")]
3079            #[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
3080            pub const fn get_mut_slice(this: &mut [Self]) -> &mut [$int_type] {
3081                // SAFETY: the mutable reference guarantees unique ownership.
3082                unsafe { &mut *(this as *mut [Self] as *mut [$int_type]) }
3083            }
3084
3085            #[doc = concat!("Get atomic access to a `&mut [", stringify!($int_type), "]` slice.")]
3086            ///
3087            #[doc = if_8_bit! {
3088                $int_type,
3089                no = [
3090                    "**Note:** This function is only available on targets where `",
3091                    stringify!($atomic_type), "` has the same alignment as `", stringify!($int_type), "`."
3092                ],
3093            }]
3094            ///
3095            /// # Examples
3096            ///
3097            #[cfg_attr($cfg_align, doc = "```ignore-wasm")]
3098            #[cfg_attr(not($cfg_align), doc = "```ignore-wasm,compile_fail")]
3099            #[doc = concat!($extra_feature, "use std::sync::atomic::{", stringify!($atomic_type), ", Ordering};")]
3100            ///
3101            /// let mut some_ints = [0; 10];
3102            #[doc = concat!("let a = &*", stringify!($atomic_type), "::from_mut_slice(&mut some_ints);")]
3103            /// std::thread::scope(|s| {
3104            ///     for i in 0..a.len() {
3105            ///         s.spawn(move || a[i].store(i as _, Ordering::Relaxed));
3106            ///     }
3107            /// });
3108            /// for (i, n) in some_ints.into_iter().enumerate() {
3109            ///     assert_eq!(i, n as usize);
3110            /// }
3111            /// ```
3112            #[ferrocene::prevalidated]
3113            #[inline]
3114            #[cfg(any($cfg_align, doc))]
3115            #[stable(feature = "atomic_from_mut", since = "1.98.0")]
3116            #[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
3117            pub const fn from_mut_slice(v: &mut [$int_type]) -> &mut [Self] {
3118                let [] = [(); align_of::<Self>() - align_of::<$int_type>()];
3119                // SAFETY:
3120                //  - the mutable reference guarantees unique ownership.
3121                //  - the alignment of `$int_type` and `Self` is the
3122                //    same, as promised by $cfg_align and verified above.
3123                unsafe { &mut *(v as *mut [$int_type] as *mut [Self]) }
3124            }
3125
3126            /// Consumes the atomic and returns the contained value.
3127            ///
3128            /// This is safe because passing `self` by value guarantees that no other threads are
3129            /// concurrently accessing the atomic data.
3130            ///
3131            /// # Examples
3132            ///
3133            #[cfg_attr($cfg_base, doc = "```")]
3134            #[cfg_attr(not($cfg_base), doc = "```compile_fail")]
3135            #[doc = concat!($extra_feature, "use std::sync::atomic::", stringify!($atomic_type), ";")]
3136            ///
3137            #[doc = concat!("let some_var = ", stringify!($atomic_type), "::new(5);")]
3138            /// assert_eq!(some_var.into_inner(), 5);
3139            /// ```
3140            #[inline]
3141            #[$stable_access]
3142            #[$const_stable_into_inner]
3143            #[ferrocene::prevalidated]
3144            pub const fn into_inner(self) -> $int_type {
3145                // SAFETY:
3146                // `Atomic<T>` is essentially a transparent wrapper around `T`.
3147                unsafe { transmute(self) }
3148            }
3149
3150            /// Loads a value from the atomic integer.
3151            ///
3152            /// `load` takes an [`Ordering`] argument which describes the memory ordering of this operation.
3153            /// Possible values are [`SeqCst`], [`Acquire`] and [`Relaxed`].
3154            ///
3155            /// # Panics
3156            ///
3157            /// Panics if `order` is [`Release`] or [`AcqRel`].
3158            ///
3159            /// # Examples
3160            ///
3161            #[cfg_attr($cfg_base, doc = "```")]
3162            #[cfg_attr(not($cfg_base), doc = "```compile_fail")]
3163            #[doc = concat!($extra_feature, "use std::sync::atomic::{", stringify!($atomic_type), ", Ordering};")]
3164            ///
3165            #[doc = concat!("let some_var = ", stringify!($atomic_type), "::new(5);")]
3166            ///
3167            /// assert_eq!(some_var.load(Ordering::Relaxed), 5);
3168            /// ```
3169            #[inline]
3170            #[$stable]
3171            #[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
3172            #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
3173            #[ferrocene::prevalidated]
3174            pub const fn load(&self, order: Ordering) -> $int_type {
3175                // SAFETY: data races are prevented by atomic intrinsics.
3176                unsafe { atomic_load::<_, /* VOLATILE */ false>(self.as_ptr(), order) }
3177            }
3178
3179            /// Perform a volatile load from the atomic integer.
3180            ///
3181            /// `load_volatile` takes an [`Ordering`] argument which describes the memory ordering
3182            /// of this operation. Possible values are [`SeqCst`], [`Acquire`] and [`Relaxed`].
3183            ///
3184            #[doc = include_str!("./atomic_load_volatile.md")]
3185            ///
3186            /// # Safety
3187            ///
3188            /// Behavior is undefined if any of the following conditions are violated:
3189            ///
3190            /// * `self` must be [valid] for reads, or `self` must point to memory
3191            ///   outside of all Rust allocations and reading from that memory must:
3192            ///   - not trap, and
3193            ///   - not cause any memory inside a Rust allocation to be modified.
3194            ///
3195            /// * `self` must be aligned to
3196            #[doc = concat!("  `align_of::<", stringify!($atomic_type), ">()`")]
3197            #[doc = if_8_bit!{
3198                $int_type,
3199                yes = [
3200                    "  (note that this is always true, since `align_of::<",
3201                    stringify!($atomic_type), ">() == 1`)."
3202                ],
3203                no = [
3204                    "  (note that on some platforms this can be bigger than `align_of::<",
3205                    stringify!($int_type), ">()`)."
3206                ],
3207            }]
3208            ///
3209            /// * Reading from `self` must produce a properly initialized value of the underlying
3210            ///   integer type.
3211            ///
3212            /// [valid]: core::ptr#safety
3213            ///
3214            /// # Panics
3215            ///
3216            /// Panics if `order` is [`Release`] or [`AcqRel`].
3217            #[inline]
3218            #[unstable(feature = "atomic_volatile", issue = "158947")]
3219            #[rustc_const_unstable(feature = "atomic_volatile", issue = "158947")]
3220            #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
3221            pub const unsafe fn load_volatile(self: *const Self, order: Ordering) -> $int_type {
3222                // SAFETY: follows from our own safety requirements.
3223                unsafe {
3224                    atomic_load::<_, /* VOLATILE */ true>(self.cast::<$int_type>(), order)
3225                }
3226            }
3227
3228            /// Stores a value into the atomic integer.
3229            ///
3230            /// `store` takes an [`Ordering`] argument which describes the memory ordering of this operation.
3231            ///  Possible values are [`SeqCst`], [`Release`] and [`Relaxed`].
3232            ///
3233            /// # Panics
3234            ///
3235            /// Panics if `order` is [`Acquire`] or [`AcqRel`].
3236            ///
3237            /// # Examples
3238            ///
3239            #[cfg_attr($cfg_base, doc = "```")]
3240            #[cfg_attr(not($cfg_base), doc = "```compile_fail")]
3241            #[doc = concat!($extra_feature, "use std::sync::atomic::{", stringify!($atomic_type), ", Ordering};")]
3242            ///
3243            #[doc = concat!("let some_var = ", stringify!($atomic_type), "::new(5);")]
3244            ///
3245            /// some_var.store(10, Ordering::Relaxed);
3246            /// assert_eq!(some_var.load(Ordering::Relaxed), 10);
3247            /// ```
3248            #[ferrocene::prevalidated]
3249            #[inline]
3250            #[$stable]
3251            #[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
3252            #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
3253            #[lint::should_not_be_called_on_const_items]
3254            pub const fn store(&self, val: $int_type, order: Ordering) {
3255                // SAFETY: data races are prevented by atomic intrinsics.
3256                unsafe { atomic_store::<_, /* VOLATILE */ false>(self.as_ptr(), val, order); }
3257            }
3258
3259            /// Performs a volatile store into the atomic integer.
3260            ///
3261            /// `store_volatile` takes an [`Ordering`] argument which describes the memory ordering
3262            /// of this operation. Possible values are [`SeqCst`], [`Release`] and [`Relaxed`].
3263            ///
3264            #[doc = include_str!("./atomic_store_volatile.md")]
3265            ///
3266            /// # Safety
3267            ///
3268            /// Behavior is undefined if any of the following conditions are violated:
3269            ///
3270            /// * `self` must be either [valid] for writes, or `self` must point to memory
3271            ///   outside of all Rust allocations and writing to that memory must:
3272            ///   - not trap, and
3273            ///   - not cause any memory inside a Rust allocation to be modified.
3274            ///
3275            /// * `self` must be aligned to
3276            #[doc = concat!("  `align_of::<", stringify!($atomic_type), ">()`")]
3277            #[doc = if_8_bit!{
3278                $int_type,
3279                yes = [
3280                    "  (note that this is always true, since `align_of::<",
3281                    stringify!($atomic_type), ">() == 1`)."
3282                ],
3283                no = [
3284                    "  (note that on some platforms this can be bigger than `align_of::<",
3285                    stringify!($int_type), ">()`)."
3286                ],
3287            }]
3288            ///
3289            /// [valid]: core::ptr#safety
3290            ///
3291            /// # Panics
3292            ///
3293            /// Panics if `order` is [`Acquire`] or [`AcqRel`].
3294            #[inline]
3295            #[unstable(feature = "atomic_volatile", issue = "158947")]
3296            #[rustc_const_unstable(feature = "atomic_volatile", issue = "158947")]
3297            #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
3298            #[lint::should_not_be_called_on_const_items]
3299            pub const unsafe fn store_volatile(self: *const Self, val: $int_type, order: Ordering) {
3300                // SAFETY: follows from our own safety requirements.
3301                unsafe {
3302                    atomic_store::<_, /* VOLATILE */ true>(self.cast::<$int_type>().cast_mut(), val, order);
3303                }
3304            }
3305
3306            /// Stores a value into the atomic integer, returning the previous value.
3307            ///
3308            /// `swap` takes an [`Ordering`] argument which describes the memory ordering
3309            /// of this operation. All ordering modes are possible. Note that using
3310            /// [`Acquire`] makes the store part of this operation [`Relaxed`], and
3311            /// using [`Release`] makes the load part [`Relaxed`].
3312            ///
3313            /// **Note**: This method is only available on platforms that support atomic operations on
3314            #[doc = concat!("[`", $s_int_type, "`].")]
3315            ///
3316            /// # Examples
3317            ///
3318            #[cfg_attr($cfg_cas, doc = "```")]
3319            #[cfg_attr(not($cfg_cas), doc = "```compile_fail")]
3320            #[doc = concat!($extra_feature, "use std::sync::atomic::{", stringify!($atomic_type), ", Ordering};")]
3321            ///
3322            #[doc = concat!("let some_var = ", stringify!($atomic_type), "::new(5);")]
3323            ///
3324            /// assert_eq!(some_var.swap(10, Ordering::Relaxed), 5);
3325            /// ```
3326            #[ferrocene::prevalidated]
3327            #[inline]
3328            #[$stable]
3329            #[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
3330            #[cfg(any($cfg_cas, doc))]
3331            #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
3332            #[lint::should_not_be_called_on_const_items]
3333            pub const fn swap(&self, val: $int_type, order: Ordering) -> $int_type {
3334                // SAFETY: data races are prevented by atomic intrinsics.
3335                unsafe { atomic_swap(self.as_ptr(), val, order) }
3336            }
3337
3338            /// Stores a value into the atomic integer if the current value is the same as
3339            /// the `current` value.
3340            ///
3341            /// The return value is always the previous value. If it is equal to `current`, then the
3342            /// value was updated.
3343            ///
3344            /// `compare_and_swap` also takes an [`Ordering`] argument which describes the memory
3345            /// ordering of this operation. Notice that even when using [`AcqRel`], the operation
3346            /// might fail and hence just perform an `Acquire` load, but not have `Release` semantics.
3347            /// Using [`Acquire`] makes the store part of this operation [`Relaxed`] if it
3348            /// happens, and using [`Release`] makes the load part [`Relaxed`].
3349            ///
3350            /// **Note**: This method is only available on platforms that support atomic operations on
3351            #[doc = concat!("[`", $s_int_type, "`].")]
3352            ///
3353            /// # Migrating to `compare_exchange` and `compare_exchange_weak`
3354            ///
3355            /// `compare_and_swap` is equivalent to `compare_exchange` with the following mapping for
3356            /// memory orderings:
3357            ///
3358            /// Original | Success | Failure
3359            /// -------- | ------- | -------
3360            /// Relaxed  | Relaxed | Relaxed
3361            /// Acquire  | Acquire | Acquire
3362            /// Release  | Release | Relaxed
3363            /// AcqRel   | AcqRel  | Acquire
3364            /// SeqCst   | SeqCst  | SeqCst
3365            ///
3366            /// `compare_and_swap` and `compare_exchange` also differ in their return type. You can use
3367            /// `compare_exchange(...).unwrap_or_else(|x| x)` to recover the behavior of `compare_and_swap`,
3368            /// but in most cases it is more idiomatic to check whether the return value is `Ok` or `Err`
3369            /// rather than to infer success vs failure based on the value that was read.
3370            ///
3371            /// During migration, consider whether it makes sense to use `compare_exchange_weak` instead.
3372            /// `compare_exchange_weak` is allowed to fail spuriously even when the comparison succeeds,
3373            /// which allows the compiler to generate better assembly code when the compare and swap
3374            /// is used in a loop.
3375            ///
3376            /// # Examples
3377            ///
3378            #[cfg_attr($cfg_cas, doc = "```")]
3379            #[cfg_attr(not($cfg_cas), doc = "```compile_fail")]
3380            #[doc = concat!($extra_feature, "use std::sync::atomic::{", stringify!($atomic_type), ", Ordering};")]
3381            ///
3382            #[doc = concat!("let some_var = ", stringify!($atomic_type), "::new(5);")]
3383            ///
3384            /// assert_eq!(some_var.compare_and_swap(5, 10, Ordering::Relaxed), 5);
3385            /// assert_eq!(some_var.load(Ordering::Relaxed), 10);
3386            ///
3387            /// assert_eq!(some_var.compare_and_swap(6, 12, Ordering::Relaxed), 10);
3388            /// assert_eq!(some_var.load(Ordering::Relaxed), 10);
3389            /// ```
3390            #[ferrocene::prevalidated]
3391            #[inline]
3392            #[$stable]
3393            #[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
3394            #[deprecated(
3395                since = "1.50.0",
3396                note = "Use `compare_exchange` or `compare_exchange_weak` instead")
3397            ]
3398            #[cfg(any($cfg_cas, doc))]
3399            #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
3400            #[lint::should_not_be_called_on_const_items]
3401            pub const fn compare_and_swap(&self,
3402                                    current: $int_type,
3403                                    new: $int_type,
3404                                    order: Ordering) -> $int_type {
3405                match self.compare_exchange(current,
3406                                            new,
3407                                            order,
3408                                            strongest_failure_ordering(order)) {
3409                    Ok(x) => x,
3410                    Err(x) => x,
3411                }
3412            }
3413
3414            /// Stores a value into the atomic integer if the current value is the same as
3415            /// the `current` value.
3416            ///
3417            /// The return value is a result indicating whether the new value was written and
3418            /// containing the previous value. On success this value is guaranteed to be equal to
3419            /// `current`.
3420            ///
3421            /// `compare_exchange` takes two [`Ordering`] arguments to describe the memory
3422            /// ordering of this operation. `success` describes the required ordering for the
3423            /// read-modify-write operation that takes place if the comparison with `current` succeeds.
3424            /// `failure` describes the required ordering for the load operation that takes place when
3425            /// the comparison fails. Using [`Acquire`] as success ordering makes the store part
3426            /// of this operation [`Relaxed`], and using [`Release`] makes the successful load
3427            /// [`Relaxed`]. The failure ordering can only be [`SeqCst`], [`Acquire`] or [`Relaxed`].
3428            ///
3429            /// **Note**: This method is only available on platforms that support atomic operations on
3430            #[doc = concat!("[`", $s_int_type, "`].")]
3431            ///
3432            /// # Examples
3433            ///
3434            #[cfg_attr($cfg_cas, doc = "```")]
3435            #[cfg_attr(not($cfg_cas), doc = "```compile_fail")]
3436            #[doc = concat!($extra_feature, "use std::sync::atomic::{", stringify!($atomic_type), ", Ordering};")]
3437            ///
3438            #[doc = concat!("let some_var = ", stringify!($atomic_type), "::new(5);")]
3439            ///
3440            /// assert_eq!(some_var.compare_exchange(5, 10,
3441            ///                                      Ordering::Acquire,
3442            ///                                      Ordering::Relaxed),
3443            ///            Ok(5));
3444            /// assert_eq!(some_var.load(Ordering::Relaxed), 10);
3445            ///
3446            /// assert_eq!(some_var.compare_exchange(6, 12,
3447            ///                                      Ordering::SeqCst,
3448            ///                                      Ordering::Acquire),
3449            ///            Err(10));
3450            /// assert_eq!(some_var.load(Ordering::Relaxed), 10);
3451            /// ```
3452            ///
3453            /// # Considerations
3454            ///
3455            /// `compare_exchange` is a [compare-and-swap operation] and thus exhibits the usual downsides
3456            /// of CAS operations. In particular, a load of the value followed by a successful
3457            /// `compare_exchange` with the previous load *does not ensure* that other threads have not
3458            /// changed the value in the interim! This is usually important when the *equality* check in
3459            /// the `compare_exchange` is being used to check the *identity* of a value, but equality
3460            /// does not necessarily imply identity. This is a particularly common case for pointers, as
3461            /// a pointer holding the same address does not imply that the same object exists at that
3462            /// address! In this case, `compare_exchange` can lead to the [ABA problem].
3463            ///
3464            /// [ABA Problem]: https://en.wikipedia.org/wiki/ABA_problem
3465            /// [compare-and-swap operation]: https://en.wikipedia.org/wiki/Compare-and-swap
3466            #[ferrocene::prevalidated]
3467            #[inline]
3468            #[$stable_cxchg]
3469            #[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
3470            #[cfg(any($cfg_cas, doc))]
3471            #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
3472            #[lint::should_not_be_called_on_const_items]
3473            pub const fn compare_exchange(&self,
3474                                    current: $int_type,
3475                                    new: $int_type,
3476                                    success: Ordering,
3477                                    failure: Ordering) -> Result<$int_type, $int_type> {
3478                // SAFETY: data races are prevented by atomic intrinsics.
3479                unsafe { atomic_compare_exchange(self.as_ptr(), current, new, success, failure) }
3480            }
3481
3482            /// Stores a value into the atomic integer if the current value is the same as
3483            /// the `current` value.
3484            ///
3485            #[doc = concat!("Unlike [`", stringify!($atomic_type), "::compare_exchange`],")]
3486            /// this function is allowed to spuriously fail even
3487            /// when the comparison succeeds, which can result in more efficient code on some
3488            /// platforms. The return value is a result indicating whether the new value was
3489            /// written and containing the previous value.
3490            ///
3491            /// `compare_exchange_weak` takes two [`Ordering`] arguments to describe the memory
3492            /// ordering of this operation. `success` describes the required ordering for the
3493            /// read-modify-write operation that takes place if the comparison with `current` succeeds.
3494            /// `failure` describes the required ordering for the load operation that takes place when
3495            /// the comparison fails. Using [`Acquire`] as success ordering makes the store part
3496            /// of this operation [`Relaxed`], and using [`Release`] makes the successful load
3497            /// [`Relaxed`]. The failure ordering can only be [`SeqCst`], [`Acquire`] or [`Relaxed`].
3498            ///
3499            /// **Note**: This method is only available on platforms that support atomic operations on
3500            #[doc = concat!("[`", $s_int_type, "`].")]
3501            ///
3502            /// # Examples
3503            ///
3504            #[cfg_attr($cfg_cas, doc = "```")]
3505            #[cfg_attr(not($cfg_cas), doc = "```compile_fail")]
3506            #[doc = concat!($extra_feature, "use std::sync::atomic::{", stringify!($atomic_type), ", Ordering};")]
3507            ///
3508            #[doc = concat!("let val = ", stringify!($atomic_type), "::new(4);")]
3509            ///
3510            /// let mut old = val.load(Ordering::Relaxed);
3511            /// loop {
3512            ///     let new = old * 2;
3513            ///     match val.compare_exchange_weak(old, new, Ordering::SeqCst, Ordering::Relaxed) {
3514            ///         Ok(_) => break,
3515            ///         Err(x) => old = x,
3516            ///     }
3517            /// }
3518            /// ```
3519            ///
3520            /// # Considerations
3521            ///
3522            /// `compare_exchange` is a [compare-and-swap operation] and thus exhibits the usual downsides
3523            /// of CAS operations. In particular, a load of the value followed by a successful
3524            /// `compare_exchange` with the previous load *does not ensure* that other threads have not
3525            /// changed the value in the interim. This is usually important when the *equality* check in
3526            /// the `compare_exchange` is being used to check the *identity* of a value, but equality
3527            /// does not necessarily imply identity. This is a particularly common case for pointers, as
3528            /// a pointer holding the same address does not imply that the same object exists at that
3529            /// address! In this case, `compare_exchange` can lead to the [ABA problem].
3530            ///
3531            /// [ABA Problem]: https://en.wikipedia.org/wiki/ABA_problem
3532            /// [compare-and-swap operation]: https://en.wikipedia.org/wiki/Compare-and-swap
3533            #[ferrocene::prevalidated]
3534            #[inline]
3535            #[$stable_cxchg]
3536            #[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
3537            #[cfg(any($cfg_cas, doc))]
3538            #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
3539            #[lint::should_not_be_called_on_const_items]
3540            pub const fn compare_exchange_weak(&self,
3541                                         current: $int_type,
3542                                         new: $int_type,
3543                                         success: Ordering,
3544                                         failure: Ordering) -> Result<$int_type, $int_type> {
3545                // SAFETY: data races are prevented by atomic intrinsics.
3546                unsafe {
3547                    atomic_compare_exchange_weak(self.as_ptr(), current, new, success, failure)
3548                }
3549            }
3550
3551            /// Adds to the current value, returning the previous value.
3552            ///
3553            /// This operation wraps around on overflow.
3554            ///
3555            /// `fetch_add` takes an [`Ordering`] argument which describes the memory ordering
3556            /// of this operation. All ordering modes are possible. Note that using
3557            /// [`Acquire`] makes the store part of this operation [`Relaxed`], and
3558            /// using [`Release`] makes the load part [`Relaxed`].
3559            ///
3560            /// **Note**: This method is only available on platforms that support atomic operations on
3561            #[doc = concat!("[`", $s_int_type, "`].")]
3562            ///
3563            /// # Examples
3564            ///
3565            #[cfg_attr($cfg_cas, doc = "```")]
3566            #[cfg_attr(not($cfg_cas), doc = "```compile_fail")]
3567            #[doc = concat!($extra_feature, "use std::sync::atomic::{", stringify!($atomic_type), ", Ordering};")]
3568            ///
3569            #[doc = concat!("let foo = ", stringify!($atomic_type), "::new(0);")]
3570            /// assert_eq!(foo.fetch_add(10, Ordering::SeqCst), 0);
3571            /// assert_eq!(foo.load(Ordering::SeqCst), 10);
3572            /// ```
3573            #[ferrocene::prevalidated]
3574            #[inline]
3575            #[$stable]
3576            #[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
3577            #[cfg(any($cfg_cas, doc))]
3578            #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
3579            #[lint::should_not_be_called_on_const_items]
3580            pub const fn fetch_add(&self, val: $int_type, order: Ordering) -> $int_type {
3581                // SAFETY: data races are prevented by atomic intrinsics.
3582                unsafe { atomic_add(self.as_ptr(), val, order) }
3583            }
3584
3585            /// Subtracts from the current value, returning the previous value.
3586            ///
3587            /// This operation wraps around on overflow.
3588            ///
3589            /// `fetch_sub` takes an [`Ordering`] argument which describes the memory ordering
3590            /// of this operation. All ordering modes are possible. Note that using
3591            /// [`Acquire`] makes the store part of this operation [`Relaxed`], and
3592            /// using [`Release`] makes the load part [`Relaxed`].
3593            ///
3594            /// **Note**: This method is only available on platforms that support atomic operations on
3595            #[doc = concat!("[`", $s_int_type, "`].")]
3596            ///
3597            /// # Examples
3598            ///
3599            #[cfg_attr($cfg_cas, doc = "```")]
3600            #[cfg_attr(not($cfg_cas), doc = "```compile_fail")]
3601            #[doc = concat!($extra_feature, "use std::sync::atomic::{", stringify!($atomic_type), ", Ordering};")]
3602            ///
3603            #[doc = concat!("let foo = ", stringify!($atomic_type), "::new(20);")]
3604            /// assert_eq!(foo.fetch_sub(10, Ordering::SeqCst), 20);
3605            /// assert_eq!(foo.load(Ordering::SeqCst), 10);
3606            /// ```
3607            #[ferrocene::prevalidated]
3608            #[inline]
3609            #[$stable]
3610            #[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
3611            #[cfg(any($cfg_cas, doc))]
3612            #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
3613            #[lint::should_not_be_called_on_const_items]
3614            pub const fn fetch_sub(&self, val: $int_type, order: Ordering) -> $int_type {
3615                // SAFETY: data races are prevented by atomic intrinsics.
3616                unsafe { atomic_sub(self.as_ptr(), val, order) }
3617            }
3618
3619            /// Bitwise "and" with the current value.
3620            ///
3621            /// Performs a bitwise "and" operation on the current value and the argument `val`, and
3622            /// sets the new value to the result.
3623            ///
3624            /// Returns the previous value.
3625            ///
3626            /// `fetch_and` takes an [`Ordering`] argument which describes the memory ordering
3627            /// of this operation. All ordering modes are possible. Note that using
3628            /// [`Acquire`] makes the store part of this operation [`Relaxed`], and
3629            /// using [`Release`] makes the load part [`Relaxed`].
3630            ///
3631            /// **Note**: This method is only available on platforms that support atomic operations on
3632            #[doc = concat!("[`", $s_int_type, "`].")]
3633            ///
3634            /// # Examples
3635            ///
3636            #[cfg_attr($cfg_cas, doc = "```")]
3637            #[cfg_attr(not($cfg_cas), doc = "```compile_fail")]
3638            #[doc = concat!($extra_feature, "use std::sync::atomic::{", stringify!($atomic_type), ", Ordering};")]
3639            ///
3640            #[doc = concat!("let foo = ", stringify!($atomic_type), "::new(0b101101);")]
3641            /// assert_eq!(foo.fetch_and(0b110011, Ordering::SeqCst), 0b101101);
3642            /// assert_eq!(foo.load(Ordering::SeqCst), 0b100001);
3643            /// ```
3644            #[ferrocene::prevalidated]
3645            #[inline]
3646            #[$stable]
3647            #[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
3648            #[cfg(any($cfg_cas, doc))]
3649            #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
3650            #[lint::should_not_be_called_on_const_items]
3651            pub const fn fetch_and(&self, val: $int_type, order: Ordering) -> $int_type {
3652                // SAFETY: data races are prevented by atomic intrinsics.
3653                unsafe { atomic_and(self.as_ptr(), val, order) }
3654            }
3655
3656            /// Bitwise "nand" with the current value.
3657            ///
3658            /// Performs a bitwise "nand" operation on the current value and the argument `val`, and
3659            /// sets the new value to the result.
3660            ///
3661            /// Returns the previous value.
3662            ///
3663            /// `fetch_nand` takes an [`Ordering`] argument which describes the memory ordering
3664            /// of this operation. All ordering modes are possible. Note that using
3665            /// [`Acquire`] makes the store part of this operation [`Relaxed`], and
3666            /// using [`Release`] makes the load part [`Relaxed`].
3667            ///
3668            /// **Note**: This method is only available on platforms that support atomic operations on
3669            #[doc = concat!("[`", $s_int_type, "`].")]
3670            ///
3671            /// # Examples
3672            ///
3673            #[cfg_attr($cfg_cas, doc = "```")]
3674            #[cfg_attr(not($cfg_cas), doc = "```compile_fail")]
3675            #[doc = concat!($extra_feature, "use std::sync::atomic::{", stringify!($atomic_type), ", Ordering};")]
3676            ///
3677            #[doc = concat!("let foo = ", stringify!($atomic_type), "::new(0x13);")]
3678            /// assert_eq!(foo.fetch_nand(0x31, Ordering::SeqCst), 0x13);
3679            /// assert_eq!(foo.load(Ordering::SeqCst), !(0x13 & 0x31));
3680            /// ```
3681            #[ferrocene::prevalidated]
3682            #[inline]
3683            #[$stable_nand]
3684            #[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
3685            #[cfg(any($cfg_cas, doc))]
3686            #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
3687            #[lint::should_not_be_called_on_const_items]
3688            pub const fn fetch_nand(&self, val: $int_type, order: Ordering) -> $int_type {
3689                // SAFETY: data races are prevented by atomic intrinsics.
3690                unsafe { atomic_nand(self.as_ptr(), val, order) }
3691            }
3692
3693            /// Bitwise "or" with the current value.
3694            ///
3695            /// Performs a bitwise "or" operation on the current value and the argument `val`, and
3696            /// sets the new value to the result.
3697            ///
3698            /// Returns the previous value.
3699            ///
3700            /// `fetch_or` takes an [`Ordering`] argument which describes the memory ordering
3701            /// of this operation. All ordering modes are possible. Note that using
3702            /// [`Acquire`] makes the store part of this operation [`Relaxed`], and
3703            /// using [`Release`] makes the load part [`Relaxed`].
3704            ///
3705            /// **Note**: This method is only available on platforms that support atomic operations on
3706            #[doc = concat!("[`", $s_int_type, "`].")]
3707            ///
3708            /// # Examples
3709            ///
3710            #[cfg_attr($cfg_cas, doc = "```")]
3711            #[cfg_attr(not($cfg_cas), doc = "```compile_fail")]
3712            #[doc = concat!($extra_feature, "use std::sync::atomic::{", stringify!($atomic_type), ", Ordering};")]
3713            ///
3714            #[doc = concat!("let foo = ", stringify!($atomic_type), "::new(0b101101);")]
3715            /// assert_eq!(foo.fetch_or(0b110011, Ordering::SeqCst), 0b101101);
3716            /// assert_eq!(foo.load(Ordering::SeqCst), 0b111111);
3717            /// ```
3718            #[ferrocene::prevalidated]
3719            #[inline]
3720            #[$stable]
3721            #[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
3722            #[cfg(any($cfg_cas, doc))]
3723            #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
3724            #[lint::should_not_be_called_on_const_items]
3725            pub const fn fetch_or(&self, val: $int_type, order: Ordering) -> $int_type {
3726                // SAFETY: data races are prevented by atomic intrinsics.
3727                unsafe { atomic_or(self.as_ptr(), val, order) }
3728            }
3729
3730            /// Bitwise "xor" with the current value.
3731            ///
3732            /// Performs a bitwise "xor" operation on the current value and the argument `val`, and
3733            /// sets the new value to the result.
3734            ///
3735            /// Returns the previous value.
3736            ///
3737            /// `fetch_xor` takes an [`Ordering`] argument which describes the memory ordering
3738            /// of this operation. All ordering modes are possible. Note that using
3739            /// [`Acquire`] makes the store part of this operation [`Relaxed`], and
3740            /// using [`Release`] makes the load part [`Relaxed`].
3741            ///
3742            /// **Note**: This method is only available on platforms that support atomic operations on
3743            #[doc = concat!("[`", $s_int_type, "`].")]
3744            ///
3745            /// # Examples
3746            ///
3747            #[cfg_attr($cfg_cas, doc = "```")]
3748            #[cfg_attr(not($cfg_cas), doc = "```compile_fail")]
3749            #[doc = concat!($extra_feature, "use std::sync::atomic::{", stringify!($atomic_type), ", Ordering};")]
3750            ///
3751            #[doc = concat!("let foo = ", stringify!($atomic_type), "::new(0b101101);")]
3752            /// assert_eq!(foo.fetch_xor(0b110011, Ordering::SeqCst), 0b101101);
3753            /// assert_eq!(foo.load(Ordering::SeqCst), 0b011110);
3754            /// ```
3755            #[ferrocene::prevalidated]
3756            #[inline]
3757            #[$stable]
3758            #[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
3759            #[cfg(any($cfg_cas, doc))]
3760            #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
3761            #[lint::should_not_be_called_on_const_items]
3762            pub const fn fetch_xor(&self, val: $int_type, order: Ordering) -> $int_type {
3763                // SAFETY: data races are prevented by atomic intrinsics.
3764                unsafe { atomic_xor(self.as_ptr(), val, order) }
3765            }
3766
3767            /// An alias for
3768            #[doc = concat!("[`", stringify!($atomic_type), "::try_update`]")]
3769            /// .
3770            #[inline]
3771            #[stable(feature = "no_more_cas", since = "1.45.0")]
3772            #[cfg(any($cfg_cas, doc))]
3773            #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
3774            #[lint::should_not_be_called_on_const_items]
3775            #[deprecated(
3776                since = "1.99.0",
3777                note = "renamed to `try_update` for consistency",
3778                suggestion = "try_update"
3779            )]
3780            pub fn fetch_update<F>(&self,
3781                                   set_order: Ordering,
3782                                   fetch_order: Ordering,
3783                                   f: F) -> Result<$int_type, $int_type>
3784            where F: FnMut($int_type) -> Option<$int_type> {
3785                self.try_update(set_order, fetch_order, f)
3786            }
3787
3788            /// Fetches the value, and applies a function to it that returns an optional
3789            /// new value. Returns a `Result` of `Ok(previous_value)` if the function returned `Some(_)`, else
3790            /// `Err(previous_value)`.
3791            ///
3792            #[doc = concat!("See also: [`update`](`", stringify!($atomic_type), "::update`).")]
3793            ///
3794            /// Note: This may call the function multiple times if the value has been changed from other threads in
3795            /// the meantime, as long as the function returns `Some(_)`, but the function will have been applied
3796            /// only once to the stored value.
3797            ///
3798            /// `try_update` takes two [`Ordering`] arguments to describe the memory ordering of this operation.
3799            /// The first describes the required ordering for when the operation finally succeeds while the second
3800            /// describes the required ordering for loads. These correspond to the success and failure orderings of
3801            #[doc = concat!("[`", stringify!($atomic_type), "::compare_exchange`]")]
3802            /// respectively.
3803            ///
3804            /// Using [`Acquire`] as success ordering makes the store part
3805            /// of this operation [`Relaxed`], and using [`Release`] makes the final successful load
3806            /// [`Relaxed`]. The (failed) load ordering can only be [`SeqCst`], [`Acquire`] or [`Relaxed`].
3807            ///
3808            /// **Note**: This method is only available on platforms that support atomic operations on
3809            #[doc = concat!("[`", $s_int_type, "`].")]
3810            ///
3811            /// # Considerations
3812            ///
3813            /// This method is not magic; it is not provided by the hardware, and does not act like a
3814            /// critical section or mutex.
3815            ///
3816            /// It is implemented on top of an atomic [compare-and-swap operation], and thus is subject to
3817            /// the usual drawbacks of CAS operations. In particular, be careful of the [ABA problem]
3818            /// if this atomic integer is an index or more generally if knowledge of only the *bitwise value*
3819            /// of the atomic is not in and of itself sufficient to ensure any required preconditions.
3820            ///
3821            /// [ABA Problem]: https://en.wikipedia.org/wiki/ABA_problem
3822            /// [compare-and-swap operation]: https://en.wikipedia.org/wiki/Compare-and-swap
3823            ///
3824            /// # Examples
3825            ///
3826            #[cfg_attr($cfg_cas, doc = "```rust")]
3827            #[cfg_attr(not($cfg_cas), doc = "```rust,compile_fail")]
3828            #[doc = concat!($extra_feature, "use std::sync::atomic::{", stringify!($atomic_type), ", Ordering};")]
3829            ///
3830            #[doc = concat!("let x = ", stringify!($atomic_type), "::new(7);")]
3831            /// assert_eq!(x.try_update(Ordering::SeqCst, Ordering::SeqCst, |_| None), Err(7));
3832            /// assert_eq!(x.try_update(Ordering::SeqCst, Ordering::SeqCst, |x| Some(x + 1)), Ok(7));
3833            /// assert_eq!(x.try_update(Ordering::SeqCst, Ordering::SeqCst, |x| Some(x + 1)), Ok(8));
3834            /// assert_eq!(x.load(Ordering::SeqCst), 9);
3835            /// ```
3836            #[ferrocene::prevalidated]
3837            #[inline]
3838            #[stable(feature = "atomic_try_update", since = "1.95.0")]
3839            #[cfg(any($cfg_cas, doc))]
3840            #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
3841            #[lint::should_not_be_called_on_const_items]
3842            pub fn try_update(
3843                &self,
3844                set_order: Ordering,
3845                fetch_order: Ordering,
3846                mut f: impl FnMut($int_type) -> Option<$int_type>,
3847            ) -> Result<$int_type, $int_type> {
3848                let mut prev = self.load(fetch_order);
3849                while let Some(next) = f(prev) {
3850                    match self.compare_exchange_weak(prev, next, set_order, fetch_order) {
3851                        x @ Ok(_) => return x,
3852                        Err(next_prev) => prev = next_prev
3853                    }
3854                }
3855                Err(prev)
3856            }
3857
3858            /// Fetches the value, and applies a function to it that returns a new value.
3859            /// The new value is stored and the old value is returned.
3860            ///
3861            #[doc = concat!("See also: [`try_update`](`", stringify!($atomic_type), "::try_update`).")]
3862            ///
3863            /// Note: This may call the function multiple times if the value has been changed from other threads in
3864            /// the meantime, but the function will have been applied only once to the stored value.
3865            ///
3866            /// `update` takes two [`Ordering`] arguments to describe the memory ordering of this operation.
3867            /// The first describes the required ordering for when the operation finally succeeds while the second
3868            /// describes the required ordering for loads. These correspond to the success and failure orderings of
3869            #[doc = concat!("[`", stringify!($atomic_type), "::compare_exchange`]")]
3870            /// respectively.
3871            ///
3872            /// Using [`Acquire`] as success ordering makes the store part
3873            /// of this operation [`Relaxed`], and using [`Release`] makes the final successful load
3874            /// [`Relaxed`]. The (failed) load ordering can only be [`SeqCst`], [`Acquire`] or [`Relaxed`].
3875            ///
3876            /// **Note**: This method is only available on platforms that support atomic operations on
3877            #[doc = concat!("[`", $s_int_type, "`].")]
3878            ///
3879            /// # Considerations
3880            ///
3881            /// [CAS operation]: https://en.wikipedia.org/wiki/Compare-and-swap
3882            /// This method is not magic; it is not provided by the hardware, and does not act like a
3883            /// critical section or mutex.
3884            ///
3885            /// It is implemented on top of an atomic [compare-and-swap operation], and thus is subject to
3886            /// the usual drawbacks of CAS operations. In particular, be careful of the [ABA problem]
3887            /// if this atomic integer is an index or more generally if knowledge of only the *bitwise value*
3888            /// of the atomic is not in and of itself sufficient to ensure any required preconditions.
3889            ///
3890            /// [ABA Problem]: https://en.wikipedia.org/wiki/ABA_problem
3891            /// [compare-and-swap operation]: https://en.wikipedia.org/wiki/Compare-and-swap
3892            ///
3893            /// # Examples
3894            ///
3895            #[cfg_attr($cfg_cas, doc = "```rust")]
3896            #[cfg_attr(not($cfg_cas), doc = "```rust,compile_fail")]
3897            #[doc = concat!($extra_feature, "use std::sync::atomic::{", stringify!($atomic_type), ", Ordering};")]
3898            ///
3899            #[doc = concat!("let x = ", stringify!($atomic_type), "::new(7);")]
3900            /// assert_eq!(x.update(Ordering::SeqCst, Ordering::SeqCst, |x| x + 1), 7);
3901            /// assert_eq!(x.update(Ordering::SeqCst, Ordering::SeqCst, |x| x + 1), 8);
3902            /// assert_eq!(x.load(Ordering::SeqCst), 9);
3903            /// ```
3904            #[ferrocene::prevalidated]
3905            #[inline]
3906            #[stable(feature = "atomic_try_update", since = "1.95.0")]
3907            #[cfg(any($cfg_cas, doc))]
3908            #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
3909            #[lint::should_not_be_called_on_const_items]
3910            pub fn update(
3911                &self,
3912                set_order: Ordering,
3913                fetch_order: Ordering,
3914                mut f: impl FnMut($int_type) -> $int_type,
3915            ) -> $int_type {
3916                let mut prev = self.load(fetch_order);
3917                loop {
3918                    match self.compare_exchange_weak(prev, f(prev), set_order, fetch_order) {
3919                        Ok(x) => break x,
3920                        Err(next_prev) => prev = next_prev,
3921                    }
3922                }
3923            }
3924
3925            /// Maximum with the current value.
3926            ///
3927            /// Finds the maximum of the current value and the argument `val`, and
3928            /// sets the new value to the result.
3929            ///
3930            /// Returns the previous value.
3931            ///
3932            /// `fetch_max` takes an [`Ordering`] argument which describes the memory ordering
3933            /// of this operation. All ordering modes are possible. Note that using
3934            /// [`Acquire`] makes the store part of this operation [`Relaxed`], and
3935            /// using [`Release`] makes the load part [`Relaxed`].
3936            ///
3937            /// **Note**: This method is only available on platforms that support atomic operations on
3938            #[doc = concat!("[`", $s_int_type, "`].")]
3939            ///
3940            /// # Examples
3941            ///
3942            #[cfg_attr($cfg_cas, doc = "```")]
3943            #[cfg_attr(not($cfg_cas), doc = "```compile_fail")]
3944            #[doc = concat!($extra_feature, "use std::sync::atomic::{", stringify!($atomic_type), ", Ordering};")]
3945            ///
3946            #[doc = concat!("let foo = ", stringify!($atomic_type), "::new(23);")]
3947            /// assert_eq!(foo.fetch_max(42, Ordering::SeqCst), 23);
3948            /// assert_eq!(foo.load(Ordering::SeqCst), 42);
3949            /// ```
3950            ///
3951            /// If you want to obtain the maximum value in one step, you can use the following:
3952            ///
3953            #[cfg_attr($cfg_cas, doc = "```")]
3954            #[cfg_attr(not($cfg_cas), doc = "```compile_fail")]
3955            #[doc = concat!($extra_feature, "use std::sync::atomic::{", stringify!($atomic_type), ", Ordering};")]
3956            ///
3957            #[doc = concat!("let foo = ", stringify!($atomic_type), "::new(23);")]
3958            /// let bar = 42;
3959            /// let max_foo = foo.fetch_max(bar, Ordering::SeqCst).max(bar);
3960            /// assert!(max_foo == 42);
3961            /// ```
3962            #[ferrocene::prevalidated]
3963            #[inline]
3964            #[stable(feature = "atomic_min_max", since = "1.45.0")]
3965            #[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
3966            #[cfg(any($cfg_cas, doc))]
3967            #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
3968            #[lint::should_not_be_called_on_const_items]
3969            pub const fn fetch_max(&self, val: $int_type, order: Ordering) -> $int_type {
3970                // SAFETY: data races are prevented by atomic intrinsics.
3971                unsafe { $max_fn(self.as_ptr(), val, order) }
3972            }
3973
3974            /// Minimum with the current value.
3975            ///
3976            /// Finds the minimum of the current value and the argument `val`, and
3977            /// sets the new value to the result.
3978            ///
3979            /// Returns the previous value.
3980            ///
3981            /// `fetch_min` takes an [`Ordering`] argument which describes the memory ordering
3982            /// of this operation. All ordering modes are possible. Note that using
3983            /// [`Acquire`] makes the store part of this operation [`Relaxed`], and
3984            /// using [`Release`] makes the load part [`Relaxed`].
3985            ///
3986            /// **Note**: This method is only available on platforms that support atomic operations on
3987            #[doc = concat!("[`", $s_int_type, "`].")]
3988            ///
3989            /// # Examples
3990            ///
3991            #[cfg_attr($cfg_cas, doc = "```")]
3992            #[cfg_attr(not($cfg_cas), doc = "```compile_fail")]
3993            #[doc = concat!($extra_feature, "use std::sync::atomic::{", stringify!($atomic_type), ", Ordering};")]
3994            ///
3995            #[doc = concat!("let foo = ", stringify!($atomic_type), "::new(23);")]
3996            /// assert_eq!(foo.fetch_min(42, Ordering::Relaxed), 23);
3997            /// assert_eq!(foo.load(Ordering::Relaxed), 23);
3998            /// assert_eq!(foo.fetch_min(22, Ordering::Relaxed), 23);
3999            /// assert_eq!(foo.load(Ordering::Relaxed), 22);
4000            /// ```
4001            ///
4002            /// If you want to obtain the minimum value in one step, you can use the following:
4003            ///
4004            #[cfg_attr($cfg_cas, doc = "```")]
4005            #[cfg_attr(not($cfg_cas), doc = "```compile_fail")]
4006            #[doc = concat!($extra_feature, "use std::sync::atomic::{", stringify!($atomic_type), ", Ordering};")]
4007            ///
4008            #[doc = concat!("let foo = ", stringify!($atomic_type), "::new(23);")]
4009            /// let bar = 12;
4010            /// let min_foo = foo.fetch_min(bar, Ordering::SeqCst).min(bar);
4011            /// assert_eq!(min_foo, 12);
4012            /// ```
4013            #[ferrocene::prevalidated]
4014            #[inline]
4015            #[stable(feature = "atomic_min_max", since = "1.45.0")]
4016            #[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
4017            #[cfg(any($cfg_cas, doc))]
4018            #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
4019            #[lint::should_not_be_called_on_const_items]
4020            pub const fn fetch_min(&self, val: $int_type, order: Ordering) -> $int_type {
4021                // SAFETY: data races are prevented by atomic intrinsics.
4022                unsafe { $min_fn(self.as_ptr(), val, order) }
4023            }
4024
4025            /// Returns a mutable pointer to the underlying integer.
4026            ///
4027            /// Doing non-atomic reads and writes on the resulting integer can be a data race.
4028            /// This method is mostly useful for FFI, where the function signature may use
4029            #[doc = concat!("`*mut ", stringify!($int_type), "` instead of `&", stringify!($atomic_type), "`.")]
4030            ///
4031            /// Returning an `*mut` pointer from a shared reference to this atomic is safe because the
4032            /// atomic types work with interior mutability. All modifications of an atomic change the value
4033            /// through a shared reference, and can do so safely as long as they use atomic operations. Any
4034            /// use of the returned raw pointer requires an `unsafe` block and still has to uphold the
4035            /// requirements of the [memory model].
4036            ///
4037            /// # Examples
4038            ///
4039            /// ```ignore (extern-declaration)
4040            /// # fn main() {
4041            #[doc = concat!($extra_feature, "use std::sync::atomic::", stringify!($atomic_type), ";")]
4042            ///
4043            /// extern "C" {
4044            #[doc = concat!("    fn my_atomic_op(arg: *mut ", stringify!($int_type), ");")]
4045            /// }
4046            ///
4047            #[doc = concat!("let atomic = ", stringify!($atomic_type), "::new(1);")]
4048            ///
4049            /// // SAFETY: Safe as long as `my_atomic_op` is atomic.
4050            /// unsafe {
4051            ///     my_atomic_op(atomic.as_ptr());
4052            /// }
4053            /// # }
4054            /// ```
4055            ///
4056            /// [memory model]: self#memory-model-for-atomic-accesses
4057            #[ferrocene::prevalidated]
4058            #[inline]
4059            #[stable(feature = "atomic_as_ptr", since = "1.70.0")]
4060            #[rustc_const_stable(feature = "atomic_as_ptr", since = "1.70.0")]
4061            #[lint::never_returns_null_ptr]
4062            pub const fn as_ptr(&self) -> *mut $int_type {
4063                self.v.get().cast()
4064            }
4065        }
4066    }
4067}
4068
4069#[cfg(target_has_atomic_load_store = "8")]
4070atomic_int! {
4071    target_has_atomic_load_store = "8",
4072    target_has_atomic = "8",
4073    target_has_atomic_primitive_alignment = "8",
4074    stable(feature = "integer_atomics_stable", since = "1.34.0"),
4075    stable(feature = "integer_atomics_stable", since = "1.34.0"),
4076    stable(feature = "integer_atomics_stable", since = "1.34.0"),
4077    stable(feature = "integer_atomics_stable", since = "1.34.0"),
4078    stable(feature = "integer_atomics_stable", since = "1.34.0"),
4079    stable(feature = "integer_atomics_stable", since = "1.34.0"),
4080    rustc_const_stable(feature = "const_integer_atomics", since = "1.34.0"),
4081    rustc_const_stable(feature = "const_atomic_into_inner", since = "1.79.0"),
4082    "i8",
4083    "",
4084    atomic_min, atomic_max,
4085    1,
4086    i8 AtomicI8
4087}
4088#[cfg(target_has_atomic_load_store = "8")]
4089atomic_int! {
4090    target_has_atomic_load_store = "8",
4091    target_has_atomic = "8",
4092    target_has_atomic_primitive_alignment = "8",
4093    stable(feature = "integer_atomics_stable", since = "1.34.0"),
4094    stable(feature = "integer_atomics_stable", since = "1.34.0"),
4095    stable(feature = "integer_atomics_stable", since = "1.34.0"),
4096    stable(feature = "integer_atomics_stable", since = "1.34.0"),
4097    stable(feature = "integer_atomics_stable", since = "1.34.0"),
4098    stable(feature = "integer_atomics_stable", since = "1.34.0"),
4099    rustc_const_stable(feature = "const_integer_atomics", since = "1.34.0"),
4100    rustc_const_stable(feature = "const_atomic_into_inner", since = "1.79.0"),
4101    "u8",
4102    "",
4103    atomic_umin, atomic_umax,
4104    1,
4105    u8 AtomicU8
4106}
4107#[cfg(target_has_atomic_load_store = "16")]
4108atomic_int! {
4109    target_has_atomic_load_store = "16",
4110    target_has_atomic = "16",
4111    target_has_atomic_primitive_alignment = "16",
4112    stable(feature = "integer_atomics_stable", since = "1.34.0"),
4113    stable(feature = "integer_atomics_stable", since = "1.34.0"),
4114    stable(feature = "integer_atomics_stable", since = "1.34.0"),
4115    stable(feature = "integer_atomics_stable", since = "1.34.0"),
4116    stable(feature = "integer_atomics_stable", since = "1.34.0"),
4117    stable(feature = "integer_atomics_stable", since = "1.34.0"),
4118    rustc_const_stable(feature = "const_integer_atomics", since = "1.34.0"),
4119    rustc_const_stable(feature = "const_atomic_into_inner", since = "1.79.0"),
4120    "i16",
4121    "",
4122    atomic_min, atomic_max,
4123    2,
4124    i16 AtomicI16
4125}
4126#[cfg(target_has_atomic_load_store = "16")]
4127atomic_int! {
4128    target_has_atomic_load_store = "16",
4129    target_has_atomic = "16",
4130    target_has_atomic_primitive_alignment = "16",
4131    stable(feature = "integer_atomics_stable", since = "1.34.0"),
4132    stable(feature = "integer_atomics_stable", since = "1.34.0"),
4133    stable(feature = "integer_atomics_stable", since = "1.34.0"),
4134    stable(feature = "integer_atomics_stable", since = "1.34.0"),
4135    stable(feature = "integer_atomics_stable", since = "1.34.0"),
4136    stable(feature = "integer_atomics_stable", since = "1.34.0"),
4137    rustc_const_stable(feature = "const_integer_atomics", since = "1.34.0"),
4138    rustc_const_stable(feature = "const_atomic_into_inner", since = "1.79.0"),
4139    "u16",
4140    "",
4141    atomic_umin, atomic_umax,
4142    2,
4143    u16 AtomicU16
4144}
4145#[cfg(target_has_atomic_load_store = "32")]
4146atomic_int! {
4147    target_has_atomic_load_store = "32",
4148    target_has_atomic = "32",
4149    target_has_atomic_primitive_alignment = "32",
4150    stable(feature = "integer_atomics_stable", since = "1.34.0"),
4151    stable(feature = "integer_atomics_stable", since = "1.34.0"),
4152    stable(feature = "integer_atomics_stable", since = "1.34.0"),
4153    stable(feature = "integer_atomics_stable", since = "1.34.0"),
4154    stable(feature = "integer_atomics_stable", since = "1.34.0"),
4155    stable(feature = "integer_atomics_stable", since = "1.34.0"),
4156    rustc_const_stable(feature = "const_integer_atomics", since = "1.34.0"),
4157    rustc_const_stable(feature = "const_atomic_into_inner", since = "1.79.0"),
4158    "i32",
4159    "",
4160    atomic_min, atomic_max,
4161    4,
4162    i32 AtomicI32
4163}
4164#[cfg(target_has_atomic_load_store = "32")]
4165atomic_int! {
4166    target_has_atomic_load_store = "32",
4167    target_has_atomic = "32",
4168    target_has_atomic_primitive_alignment = "32",
4169    stable(feature = "integer_atomics_stable", since = "1.34.0"),
4170    stable(feature = "integer_atomics_stable", since = "1.34.0"),
4171    stable(feature = "integer_atomics_stable", since = "1.34.0"),
4172    stable(feature = "integer_atomics_stable", since = "1.34.0"),
4173    stable(feature = "integer_atomics_stable", since = "1.34.0"),
4174    stable(feature = "integer_atomics_stable", since = "1.34.0"),
4175    rustc_const_stable(feature = "const_integer_atomics", since = "1.34.0"),
4176    rustc_const_stable(feature = "const_atomic_into_inner", since = "1.79.0"),
4177    "u32",
4178    "",
4179    atomic_umin, atomic_umax,
4180    4,
4181    u32 AtomicU32
4182}
4183#[cfg(target_has_atomic_load_store = "64")]
4184atomic_int! {
4185    target_has_atomic_load_store = "64",
4186    target_has_atomic = "64",
4187    target_has_atomic_primitive_alignment = "64",
4188    stable(feature = "integer_atomics_stable", since = "1.34.0"),
4189    stable(feature = "integer_atomics_stable", since = "1.34.0"),
4190    stable(feature = "integer_atomics_stable", since = "1.34.0"),
4191    stable(feature = "integer_atomics_stable", since = "1.34.0"),
4192    stable(feature = "integer_atomics_stable", since = "1.34.0"),
4193    stable(feature = "integer_atomics_stable", since = "1.34.0"),
4194    rustc_const_stable(feature = "const_integer_atomics", since = "1.34.0"),
4195    rustc_const_stable(feature = "const_atomic_into_inner", since = "1.79.0"),
4196    "i64",
4197    "",
4198    atomic_min, atomic_max,
4199    8,
4200    i64 AtomicI64
4201}
4202#[cfg(target_has_atomic_load_store = "64")]
4203atomic_int! {
4204    target_has_atomic_load_store = "64",
4205    target_has_atomic = "64",
4206    target_has_atomic_primitive_alignment = "64",
4207    stable(feature = "integer_atomics_stable", since = "1.34.0"),
4208    stable(feature = "integer_atomics_stable", since = "1.34.0"),
4209    stable(feature = "integer_atomics_stable", since = "1.34.0"),
4210    stable(feature = "integer_atomics_stable", since = "1.34.0"),
4211    stable(feature = "integer_atomics_stable", since = "1.34.0"),
4212    stable(feature = "integer_atomics_stable", since = "1.34.0"),
4213    rustc_const_stable(feature = "const_integer_atomics", since = "1.34.0"),
4214    rustc_const_stable(feature = "const_atomic_into_inner", since = "1.79.0"),
4215    "u64",
4216    "",
4217    atomic_umin, atomic_umax,
4218    8,
4219    u64 AtomicU64
4220}
4221#[cfg(any(target_has_atomic_load_store = "128", doc))]
4222atomic_int! {
4223    target_has_atomic_load_store = "128",
4224    target_has_atomic = "128",
4225    target_has_atomic_primitive_alignment = "128",
4226    unstable(feature = "integer_atomics", issue = "99069"),
4227    unstable(feature = "integer_atomics", issue = "99069"),
4228    unstable(feature = "integer_atomics", issue = "99069"),
4229    unstable(feature = "integer_atomics", issue = "99069"),
4230    unstable(feature = "integer_atomics", issue = "99069"),
4231    unstable(feature = "integer_atomics", issue = "99069"),
4232    rustc_const_unstable(feature = "integer_atomics", issue = "99069"),
4233    rustc_const_unstable(feature = "integer_atomics", issue = "99069"),
4234    "i128",
4235    "#![feature(integer_atomics)]\n\n",
4236    atomic_min, atomic_max,
4237    16,
4238    i128 AtomicI128
4239}
4240#[cfg(any(target_has_atomic_load_store = "128", doc))]
4241atomic_int! {
4242    target_has_atomic_load_store = "128",
4243    target_has_atomic = "128",
4244    target_has_atomic_primitive_alignment = "128",
4245    unstable(feature = "integer_atomics", issue = "99069"),
4246    unstable(feature = "integer_atomics", issue = "99069"),
4247    unstable(feature = "integer_atomics", issue = "99069"),
4248    unstable(feature = "integer_atomics", issue = "99069"),
4249    unstable(feature = "integer_atomics", issue = "99069"),
4250    unstable(feature = "integer_atomics", issue = "99069"),
4251    rustc_const_unstable(feature = "integer_atomics", issue = "99069"),
4252    rustc_const_unstable(feature = "integer_atomics", issue = "99069"),
4253    "u128",
4254    "#![feature(integer_atomics)]\n\n",
4255    atomic_umin, atomic_umax,
4256    16,
4257    u128 AtomicU128
4258}
4259
4260#[cfg(target_has_atomic_load_store = "ptr")]
4261macro_rules! atomic_int_ptr_sized {
4262    ( $($target_pointer_width:literal $align:literal)* ) => { $(
4263        #[cfg(target_pointer_width = $target_pointer_width)]
4264        atomic_int! {
4265            target_has_atomic_load_store = "ptr",
4266            target_has_atomic = "ptr",
4267            target_has_atomic_primitive_alignment = "ptr",
4268            stable(feature = "rust1", since = "1.0.0"),
4269            stable(feature = "extended_compare_and_swap", since = "1.10.0"),
4270            stable(feature = "atomic_debug", since = "1.3.0"),
4271            stable(feature = "atomic_access", since = "1.15.0"),
4272            stable(feature = "atomic_from", since = "1.23.0"),
4273            stable(feature = "atomic_nand", since = "1.27.0"),
4274            rustc_const_stable(feature = "const_ptr_sized_atomics", since = "1.24.0"),
4275            rustc_const_stable(feature = "const_atomic_into_inner", since = "1.79.0"),
4276            "isize",
4277            "",
4278            atomic_min, atomic_max,
4279            $align,
4280            isize AtomicIsize
4281        }
4282        #[cfg(target_pointer_width = $target_pointer_width)]
4283        atomic_int! {
4284            target_has_atomic_load_store = "ptr",
4285            target_has_atomic = "ptr",
4286            target_has_atomic_primitive_alignment = "ptr",
4287            stable(feature = "rust1", since = "1.0.0"),
4288            stable(feature = "extended_compare_and_swap", since = "1.10.0"),
4289            stable(feature = "atomic_debug", since = "1.3.0"),
4290            stable(feature = "atomic_access", since = "1.15.0"),
4291            stable(feature = "atomic_from", since = "1.23.0"),
4292            stable(feature = "atomic_nand", since = "1.27.0"),
4293            rustc_const_stable(feature = "const_ptr_sized_atomics", since = "1.24.0"),
4294            rustc_const_stable(feature = "const_atomic_into_inner", since = "1.79.0"),
4295            "usize",
4296            "",
4297            atomic_umin, atomic_umax,
4298            $align,
4299            usize AtomicUsize
4300        }
4301
4302        /// An [`AtomicIsize`] initialized to `0`.
4303        #[cfg(target_pointer_width = $target_pointer_width)]
4304        #[stable(feature = "rust1", since = "1.0.0")]
4305        #[deprecated(
4306            since = "1.34.0",
4307            note = "the `new` function is now preferred",
4308            suggestion = "AtomicIsize::new(0)",
4309        )]
4310        #[expect(clippy::declare_interior_mutable_const, reason = "legacy atomic initializer")]
4311        pub const ATOMIC_ISIZE_INIT: AtomicIsize = AtomicIsize::new(0);
4312
4313        /// An [`AtomicUsize`] initialized to `0`.
4314        #[cfg(target_pointer_width = $target_pointer_width)]
4315        #[stable(feature = "rust1", since = "1.0.0")]
4316        #[deprecated(
4317            since = "1.34.0",
4318            note = "the `new` function is now preferred",
4319            suggestion = "AtomicUsize::new(0)",
4320        )]
4321        #[expect(clippy::declare_interior_mutable_const, reason = "legacy atomic initializer")]
4322        pub const ATOMIC_USIZE_INIT: AtomicUsize = AtomicUsize::new(0);
4323    )* };
4324}
4325
4326#[cfg(target_has_atomic_load_store = "ptr")]
4327atomic_int_ptr_sized! {
4328    "16" 2
4329    "32" 4
4330    "64" 8
4331}
4332
4333#[inline]
4334#[cfg(target_has_atomic)]
4335#[ferrocene::prevalidated]
4336const fn strongest_failure_ordering(order: Ordering) -> Ordering {
4337    match order {
4338        Release => Relaxed,
4339        Relaxed => Relaxed,
4340        SeqCst => SeqCst,
4341        Acquire => Acquire,
4342        AcqRel => Acquire,
4343    }
4344}
4345
4346#[ferrocene::prevalidated]
4347#[inline]
4348#[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
4349#[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
4350const unsafe fn atomic_store<T: Copy, const VOLATILE: bool>(dst: *mut T, val: T, order: Ordering) {
4351    // SAFETY: the caller must uphold the safety contract for `atomic_store`.
4352    unsafe {
4353        match order {
4354            Relaxed => intrinsics::atomic_store::<T, { AO::Relaxed }, VOLATILE>(dst, val),
4355            Release => intrinsics::atomic_store::<T, { AO::Release }, VOLATILE>(dst, val),
4356            SeqCst => intrinsics::atomic_store::<T, { AO::SeqCst }, VOLATILE>(dst, val),
4357            Acquire => panic!("there is no such thing as an acquire store"),
4358            AcqRel => panic!("there is no such thing as an acquire-release store"),
4359        }
4360    }
4361}
4362
4363#[ferrocene::prevalidated]
4364#[inline]
4365#[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
4366#[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
4367const unsafe fn atomic_load<T: Copy, const VOLATILE: bool>(dst: *const T, order: Ordering) -> T {
4368    // SAFETY: the caller must uphold the safety contract for `atomic_load`.
4369    unsafe {
4370        match order {
4371            Relaxed => intrinsics::atomic_load::<T, { AO::Relaxed }, VOLATILE>(dst),
4372            Acquire => intrinsics::atomic_load::<T, { AO::Acquire }, VOLATILE>(dst),
4373            SeqCst => intrinsics::atomic_load::<T, { AO::SeqCst }, VOLATILE>(dst),
4374            Release => panic!("there is no such thing as a release load"),
4375            AcqRel => panic!("there is no such thing as an acquire-release load"),
4376        }
4377    }
4378}
4379
4380#[inline]
4381#[cfg(target_has_atomic)]
4382#[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
4383#[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
4384#[ferrocene::prevalidated]
4385const unsafe fn atomic_swap<T: Copy>(dst: *mut T, val: T, order: Ordering) -> T {
4386    // SAFETY: the caller must uphold the safety contract for `atomic_swap`.
4387    unsafe {
4388        match order {
4389            Relaxed => intrinsics::atomic_xchg::<T, { AO::Relaxed }>(dst, val),
4390            Acquire => intrinsics::atomic_xchg::<T, { AO::Acquire }>(dst, val),
4391            Release => intrinsics::atomic_xchg::<T, { AO::Release }>(dst, val),
4392            AcqRel => intrinsics::atomic_xchg::<T, { AO::AcqRel }>(dst, val),
4393            SeqCst => intrinsics::atomic_xchg::<T, { AO::SeqCst }>(dst, val),
4394        }
4395    }
4396}
4397
4398/// Returns the previous value (like __sync_fetch_and_add).
4399#[inline]
4400#[cfg(target_has_atomic)]
4401#[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
4402#[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
4403#[ferrocene::prevalidated]
4404const unsafe fn atomic_add<T: Copy, U: Copy>(dst: *mut T, val: U, order: Ordering) -> T {
4405    // SAFETY: the caller must uphold the safety contract for `atomic_add`.
4406    unsafe {
4407        match order {
4408            Relaxed => intrinsics::atomic_xadd::<T, U, { AO::Relaxed }>(dst, val),
4409            Acquire => intrinsics::atomic_xadd::<T, U, { AO::Acquire }>(dst, val),
4410            Release => intrinsics::atomic_xadd::<T, U, { AO::Release }>(dst, val),
4411            AcqRel => intrinsics::atomic_xadd::<T, U, { AO::AcqRel }>(dst, val),
4412            SeqCst => intrinsics::atomic_xadd::<T, U, { AO::SeqCst }>(dst, val),
4413        }
4414    }
4415}
4416
4417/// Returns the previous value (like __sync_fetch_and_sub).
4418#[inline]
4419#[cfg(target_has_atomic)]
4420#[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
4421#[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
4422#[ferrocene::prevalidated]
4423const unsafe fn atomic_sub<T: Copy, U: Copy>(dst: *mut T, val: U, order: Ordering) -> T {
4424    // SAFETY: the caller must uphold the safety contract for `atomic_sub`.
4425    unsafe {
4426        match order {
4427            Relaxed => intrinsics::atomic_xsub::<T, U, { AO::Relaxed }>(dst, val),
4428            Acquire => intrinsics::atomic_xsub::<T, U, { AO::Acquire }>(dst, val),
4429            Release => intrinsics::atomic_xsub::<T, U, { AO::Release }>(dst, val),
4430            AcqRel => intrinsics::atomic_xsub::<T, U, { AO::AcqRel }>(dst, val),
4431            SeqCst => intrinsics::atomic_xsub::<T, U, { AO::SeqCst }>(dst, val),
4432        }
4433    }
4434}
4435
4436/// Publicly exposed for stdarch; nobody else should use this.
4437#[inline]
4438#[cfg(target_has_atomic)]
4439#[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
4440#[unstable(feature = "core_intrinsics", issue = "none")]
4441#[doc(hidden)]
4442#[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
4443#[ferrocene::prevalidated]
4444pub const unsafe fn atomic_compare_exchange<T: Copy>(
4445    dst: *mut T,
4446    old: T,
4447    new: T,
4448    success: Ordering,
4449    failure: Ordering,
4450) -> Result<T, T> {
4451    // SAFETY: the caller must uphold the safety contract for `atomic_compare_exchange`.
4452    let (val, ok) = unsafe {
4453        match (success, failure) {
4454            (Relaxed, Relaxed) => {
4455                intrinsics::atomic_cxchg::<T, { AO::Relaxed }, { AO::Relaxed }>(dst, old, new)
4456            }
4457            (Relaxed, Acquire) => {
4458                intrinsics::atomic_cxchg::<T, { AO::Relaxed }, { AO::Acquire }>(dst, old, new)
4459            }
4460            (Relaxed, SeqCst) => {
4461                intrinsics::atomic_cxchg::<T, { AO::Relaxed }, { AO::SeqCst }>(dst, old, new)
4462            }
4463            (Acquire, Relaxed) => {
4464                intrinsics::atomic_cxchg::<T, { AO::Acquire }, { AO::Relaxed }>(dst, old, new)
4465            }
4466            (Acquire, Acquire) => {
4467                intrinsics::atomic_cxchg::<T, { AO::Acquire }, { AO::Acquire }>(dst, old, new)
4468            }
4469            (Acquire, SeqCst) => {
4470                intrinsics::atomic_cxchg::<T, { AO::Acquire }, { AO::SeqCst }>(dst, old, new)
4471            }
4472            (Release, Relaxed) => {
4473                intrinsics::atomic_cxchg::<T, { AO::Release }, { AO::Relaxed }>(dst, old, new)
4474            }
4475            (Release, Acquire) => {
4476                intrinsics::atomic_cxchg::<T, { AO::Release }, { AO::Acquire }>(dst, old, new)
4477            }
4478            (Release, SeqCst) => {
4479                intrinsics::atomic_cxchg::<T, { AO::Release }, { AO::SeqCst }>(dst, old, new)
4480            }
4481            (AcqRel, Relaxed) => {
4482                intrinsics::atomic_cxchg::<T, { AO::AcqRel }, { AO::Relaxed }>(dst, old, new)
4483            }
4484            (AcqRel, Acquire) => {
4485                intrinsics::atomic_cxchg::<T, { AO::AcqRel }, { AO::Acquire }>(dst, old, new)
4486            }
4487            (AcqRel, SeqCst) => {
4488                intrinsics::atomic_cxchg::<T, { AO::AcqRel }, { AO::SeqCst }>(dst, old, new)
4489            }
4490            (SeqCst, Relaxed) => {
4491                intrinsics::atomic_cxchg::<T, { AO::SeqCst }, { AO::Relaxed }>(dst, old, new)
4492            }
4493            (SeqCst, Acquire) => {
4494                intrinsics::atomic_cxchg::<T, { AO::SeqCst }, { AO::Acquire }>(dst, old, new)
4495            }
4496            (SeqCst, SeqCst) => {
4497                intrinsics::atomic_cxchg::<T, { AO::SeqCst }, { AO::SeqCst }>(dst, old, new)
4498            }
4499            (_, AcqRel) => panic!("there is no such thing as an acquire-release failure ordering"),
4500            (_, Release) => panic!("there is no such thing as a release failure ordering"),
4501        }
4502    };
4503    if ok { Ok(val) } else { Err(val) }
4504}
4505
4506#[inline]
4507#[cfg(target_has_atomic)]
4508#[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
4509#[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
4510#[ferrocene::prevalidated]
4511const unsafe fn atomic_compare_exchange_weak<T: Copy>(
4512    dst: *mut T,
4513    old: T,
4514    new: T,
4515    success: Ordering,
4516    failure: Ordering,
4517) -> Result<T, T> {
4518    // SAFETY: the caller must uphold the safety contract for `atomic_compare_exchange_weak`.
4519    let (val, ok) = unsafe {
4520        match (success, failure) {
4521            (Relaxed, Relaxed) => {
4522                intrinsics::atomic_cxchgweak::<T, { AO::Relaxed }, { AO::Relaxed }>(dst, old, new)
4523            }
4524            (Relaxed, Acquire) => {
4525                intrinsics::atomic_cxchgweak::<T, { AO::Relaxed }, { AO::Acquire }>(dst, old, new)
4526            }
4527            (Relaxed, SeqCst) => {
4528                intrinsics::atomic_cxchgweak::<T, { AO::Relaxed }, { AO::SeqCst }>(dst, old, new)
4529            }
4530            (Acquire, Relaxed) => {
4531                intrinsics::atomic_cxchgweak::<T, { AO::Acquire }, { AO::Relaxed }>(dst, old, new)
4532            }
4533            (Acquire, Acquire) => {
4534                intrinsics::atomic_cxchgweak::<T, { AO::Acquire }, { AO::Acquire }>(dst, old, new)
4535            }
4536            (Acquire, SeqCst) => {
4537                intrinsics::atomic_cxchgweak::<T, { AO::Acquire }, { AO::SeqCst }>(dst, old, new)
4538            }
4539            (Release, Relaxed) => {
4540                intrinsics::atomic_cxchgweak::<T, { AO::Release }, { AO::Relaxed }>(dst, old, new)
4541            }
4542            (Release, Acquire) => {
4543                intrinsics::atomic_cxchgweak::<T, { AO::Release }, { AO::Acquire }>(dst, old, new)
4544            }
4545            (Release, SeqCst) => {
4546                intrinsics::atomic_cxchgweak::<T, { AO::Release }, { AO::SeqCst }>(dst, old, new)
4547            }
4548            (AcqRel, Relaxed) => {
4549                intrinsics::atomic_cxchgweak::<T, { AO::AcqRel }, { AO::Relaxed }>(dst, old, new)
4550            }
4551            (AcqRel, Acquire) => {
4552                intrinsics::atomic_cxchgweak::<T, { AO::AcqRel }, { AO::Acquire }>(dst, old, new)
4553            }
4554            (AcqRel, SeqCst) => {
4555                intrinsics::atomic_cxchgweak::<T, { AO::AcqRel }, { AO::SeqCst }>(dst, old, new)
4556            }
4557            (SeqCst, Relaxed) => {
4558                intrinsics::atomic_cxchgweak::<T, { AO::SeqCst }, { AO::Relaxed }>(dst, old, new)
4559            }
4560            (SeqCst, Acquire) => {
4561                intrinsics::atomic_cxchgweak::<T, { AO::SeqCst }, { AO::Acquire }>(dst, old, new)
4562            }
4563            (SeqCst, SeqCst) => {
4564                intrinsics::atomic_cxchgweak::<T, { AO::SeqCst }, { AO::SeqCst }>(dst, old, new)
4565            }
4566            (_, AcqRel) => panic!("there is no such thing as an acquire-release failure ordering"),
4567            (_, Release) => panic!("there is no such thing as a release failure ordering"),
4568        }
4569    };
4570    if ok { Ok(val) } else { Err(val) }
4571}
4572
4573#[inline]
4574#[cfg(target_has_atomic)]
4575#[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
4576#[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
4577#[ferrocene::prevalidated]
4578const unsafe fn atomic_and<T: Copy, U: Copy>(dst: *mut T, val: U, order: Ordering) -> T {
4579    // SAFETY: the caller must uphold the safety contract for `atomic_and`
4580    unsafe {
4581        match order {
4582            Relaxed => intrinsics::atomic_and::<T, U, { AO::Relaxed }>(dst, val),
4583            Acquire => intrinsics::atomic_and::<T, U, { AO::Acquire }>(dst, val),
4584            Release => intrinsics::atomic_and::<T, U, { AO::Release }>(dst, val),
4585            AcqRel => intrinsics::atomic_and::<T, U, { AO::AcqRel }>(dst, val),
4586            SeqCst => intrinsics::atomic_and::<T, U, { AO::SeqCst }>(dst, val),
4587        }
4588    }
4589}
4590
4591#[inline]
4592#[cfg(target_has_atomic)]
4593#[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
4594#[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
4595#[ferrocene::prevalidated]
4596const unsafe fn atomic_nand<T: Copy, U: Copy>(dst: *mut T, val: U, order: Ordering) -> T {
4597    // SAFETY: the caller must uphold the safety contract for `atomic_nand`
4598    unsafe {
4599        match order {
4600            Relaxed => intrinsics::atomic_nand::<T, U, { AO::Relaxed }>(dst, val),
4601            Acquire => intrinsics::atomic_nand::<T, U, { AO::Acquire }>(dst, val),
4602            Release => intrinsics::atomic_nand::<T, U, { AO::Release }>(dst, val),
4603            AcqRel => intrinsics::atomic_nand::<T, U, { AO::AcqRel }>(dst, val),
4604            SeqCst => intrinsics::atomic_nand::<T, U, { AO::SeqCst }>(dst, val),
4605        }
4606    }
4607}
4608
4609#[inline]
4610#[cfg(target_has_atomic)]
4611#[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
4612#[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
4613#[ferrocene::prevalidated]
4614const unsafe fn atomic_or<T: Copy, U: Copy>(dst: *mut T, val: U, order: Ordering) -> T {
4615    // SAFETY: the caller must uphold the safety contract for `atomic_or`
4616    unsafe {
4617        match order {
4618            SeqCst => intrinsics::atomic_or::<T, U, { AO::SeqCst }>(dst, val),
4619            Acquire => intrinsics::atomic_or::<T, U, { AO::Acquire }>(dst, val),
4620            Release => intrinsics::atomic_or::<T, U, { AO::Release }>(dst, val),
4621            AcqRel => intrinsics::atomic_or::<T, U, { AO::AcqRel }>(dst, val),
4622            Relaxed => intrinsics::atomic_or::<T, U, { AO::Relaxed }>(dst, val),
4623        }
4624    }
4625}
4626
4627#[inline]
4628#[cfg(target_has_atomic)]
4629#[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
4630#[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
4631#[ferrocene::prevalidated]
4632const unsafe fn atomic_xor<T: Copy, U: Copy>(dst: *mut T, val: U, order: Ordering) -> T {
4633    // SAFETY: the caller must uphold the safety contract for `atomic_xor`
4634    unsafe {
4635        match order {
4636            SeqCst => intrinsics::atomic_xor::<T, U, { AO::SeqCst }>(dst, val),
4637            Acquire => intrinsics::atomic_xor::<T, U, { AO::Acquire }>(dst, val),
4638            Release => intrinsics::atomic_xor::<T, U, { AO::Release }>(dst, val),
4639            AcqRel => intrinsics::atomic_xor::<T, U, { AO::AcqRel }>(dst, val),
4640            Relaxed => intrinsics::atomic_xor::<T, U, { AO::Relaxed }>(dst, val),
4641        }
4642    }
4643}
4644
4645/// Updates `*dst` to the max value of `val` and the old value (signed comparison)
4646#[inline]
4647#[cfg(target_has_atomic)]
4648#[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
4649#[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
4650#[ferrocene::prevalidated]
4651const unsafe fn atomic_max<T: Copy>(dst: *mut T, val: T, order: Ordering) -> T {
4652    // SAFETY: the caller must uphold the safety contract for `atomic_max`
4653    unsafe {
4654        match order {
4655            Relaxed => intrinsics::atomic_max::<T, { AO::Relaxed }>(dst, val),
4656            Acquire => intrinsics::atomic_max::<T, { AO::Acquire }>(dst, val),
4657            Release => intrinsics::atomic_max::<T, { AO::Release }>(dst, val),
4658            AcqRel => intrinsics::atomic_max::<T, { AO::AcqRel }>(dst, val),
4659            SeqCst => intrinsics::atomic_max::<T, { AO::SeqCst }>(dst, val),
4660        }
4661    }
4662}
4663
4664/// Updates `*dst` to the min value of `val` and the old value (signed comparison)
4665#[inline]
4666#[cfg(target_has_atomic)]
4667#[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
4668#[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
4669#[ferrocene::prevalidated]
4670const unsafe fn atomic_min<T: Copy>(dst: *mut T, val: T, order: Ordering) -> T {
4671    // SAFETY: the caller must uphold the safety contract for `atomic_min`
4672    unsafe {
4673        match order {
4674            Relaxed => intrinsics::atomic_min::<T, { AO::Relaxed }>(dst, val),
4675            Acquire => intrinsics::atomic_min::<T, { AO::Acquire }>(dst, val),
4676            Release => intrinsics::atomic_min::<T, { AO::Release }>(dst, val),
4677            AcqRel => intrinsics::atomic_min::<T, { AO::AcqRel }>(dst, val),
4678            SeqCst => intrinsics::atomic_min::<T, { AO::SeqCst }>(dst, val),
4679        }
4680    }
4681}
4682
4683/// Updates `*dst` to the max value of `val` and the old value (unsigned comparison)
4684#[inline]
4685#[cfg(target_has_atomic)]
4686#[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
4687#[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
4688#[ferrocene::prevalidated]
4689const unsafe fn atomic_umax<T: Copy>(dst: *mut T, val: T, order: Ordering) -> T {
4690    // SAFETY: the caller must uphold the safety contract for `atomic_umax`
4691    unsafe {
4692        match order {
4693            Relaxed => intrinsics::atomic_umax::<T, { AO::Relaxed }>(dst, val),
4694            Acquire => intrinsics::atomic_umax::<T, { AO::Acquire }>(dst, val),
4695            Release => intrinsics::atomic_umax::<T, { AO::Release }>(dst, val),
4696            AcqRel => intrinsics::atomic_umax::<T, { AO::AcqRel }>(dst, val),
4697            SeqCst => intrinsics::atomic_umax::<T, { AO::SeqCst }>(dst, val),
4698        }
4699    }
4700}
4701
4702/// Updates `*dst` to the min value of `val` and the old value (unsigned comparison)
4703#[inline]
4704#[cfg(target_has_atomic)]
4705#[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
4706#[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
4707#[ferrocene::prevalidated]
4708const unsafe fn atomic_umin<T: Copy>(dst: *mut T, val: T, order: Ordering) -> T {
4709    // SAFETY: the caller must uphold the safety contract for `atomic_umin`
4710    unsafe {
4711        match order {
4712            Relaxed => intrinsics::atomic_umin::<T, { AO::Relaxed }>(dst, val),
4713            Acquire => intrinsics::atomic_umin::<T, { AO::Acquire }>(dst, val),
4714            Release => intrinsics::atomic_umin::<T, { AO::Release }>(dst, val),
4715            AcqRel => intrinsics::atomic_umin::<T, { AO::AcqRel }>(dst, val),
4716            SeqCst => intrinsics::atomic_umin::<T, { AO::SeqCst }>(dst, val),
4717        }
4718    }
4719}
4720
4721/// An atomic fence.
4722///
4723/// Fences create synchronization between themselves and atomic operations or fences in other
4724/// threads. It can be helpful to think of a fence as preventing the compiler and CPU from
4725/// reordering certain types of memory operations around it, but that is a simplified model which
4726/// fails to capture some of the nuances.
4727///
4728/// There are 3 different ways to use an atomic fence:
4729///
4730/// - atomic - fence synchronization: an atomic operation with (at least) [`Release`] ordering
4731///   semantics synchronizes with a fence with (at least) [`Acquire`] ordering semantics.
4732/// - fence - atomic synchronization: a fence with (at least) [`Release`] ordering semantics
4733///   synchronizes with an atomic operation with (at least) [`Acquire`] ordering semantics.
4734/// - fence - fence synchronization: a fence with (at least) [`Release`] ordering semantics
4735///   synchronizes with a fence with (at least) [`Acquire`] ordering semantics.
4736///
4737/// These 3 ways complement the regular, fence-less, atomic - atomic synchronization.
4738///
4739/// ## Atomic - Fence
4740///
4741/// An atomic operation on one thread will synchronize with a fence on another thread when:
4742///
4743/// -   on thread 1:
4744///     -   an atomic operation 'X' with (at least) [`Release`] ordering semantics on some atomic
4745///         object 'm',
4746///
4747/// -   is paired on thread 2 with:
4748///     -   an atomic read 'Y' with any order on 'm',
4749///     -   followed by a fence 'B' with (at least) [`Acquire`] ordering semantics.
4750///
4751/// This provides a happens-before dependence between X and B.
4752///
4753/// ```text
4754///     Thread 1                                          Thread 2
4755///
4756/// m.store(3, Release); X ---------
4757///                                |
4758///                                |
4759///                                -------------> Y  if m.load(Relaxed) == 3 {
4760///                                               B      fence(Acquire);
4761///                                                      ...
4762///                                                  }
4763/// ```
4764///
4765/// ## Fence - Atomic
4766///
4767/// A fence on one thread will synchronize with an atomic operation on another thread when:
4768///
4769/// -   on thread:
4770///     -   a fence 'A' with (at least) [`Release`] ordering semantics,
4771///     -   followed by an atomic write 'X' with any ordering on some atomic object 'm',
4772///
4773/// -   is paired on thread 2 with:
4774///     -   an atomic operation 'Y' with (at least) [`Acquire`] ordering semantics.
4775///
4776/// This provides a happens-before dependence between A and Y.
4777///
4778/// ```text
4779///     Thread 1                                          Thread 2
4780///
4781/// fence(Release);      A
4782/// m.store(3, Relaxed); X ---------
4783///                                |
4784///                                |
4785///                                -------------> Y  if m.load(Acquire) == 3 {
4786///                                                      ...
4787///                                                  }
4788/// ```
4789///
4790/// ## Fence - Fence
4791///
4792/// A fence on one thread will synchronize with a fence on another thread when:
4793///
4794/// -   on thread 1:
4795///     -   a fence 'A' which has (at least) [`Release`] ordering semantics,
4796///     -   followed by an atomic write 'X' with any ordering on some atomic object 'm',
4797///
4798/// -   is paired on thread 2 with:
4799///     -   an atomic read 'Y' with any ordering on 'm',
4800///     -   followed by a fence 'B' with (at least) [`Acquire`] ordering semantics.
4801///
4802/// This provides a happens-before dependence between A and B.
4803///
4804/// ```text
4805///     Thread 1                                          Thread 2
4806///
4807/// fence(Release);      A --------------
4808/// m.store(3, Relaxed); X ---------    |
4809///                                |    |
4810///                                |    |
4811///                                -------------> Y  if m.load(Relaxed) == 3 {
4812///                                     |-------> B      fence(Acquire);
4813///                                                      ...
4814///                                                  }
4815/// ```
4816///
4817/// ## Mandatory Atomic
4818///
4819/// Note that in the examples above, it is crucial that the access to `m` are atomic. Fences cannot
4820/// be used to establish synchronization between non-atomic accesses in different threads. However,
4821/// thanks to the happens-before relationship, any non-atomic access that happen-before the atomic
4822/// operation or fence with (at least) [`Release`] ordering semantics are now also properly
4823/// synchronized with any non-atomic accesses that happen-after the atomic operation or fence with
4824/// (at least) [`Acquire`] ordering semantics.
4825///
4826/// ## Memory Ordering
4827///
4828/// A fence which has [`SeqCst`] ordering, in addition to having both [`Acquire`] and [`Release`]
4829/// semantics, participates in the global program order of the other [`SeqCst`] operations and/or
4830/// fences.
4831///
4832/// Accepts [`Acquire`], [`Release`], [`AcqRel`] and [`SeqCst`] orderings.
4833///
4834/// # Panics
4835///
4836/// Panics if `order` is [`Relaxed`].
4837///
4838/// # Examples
4839///
4840/// ```
4841/// use std::sync::atomic::AtomicBool;
4842/// use std::sync::atomic::fence;
4843/// use std::sync::atomic::Ordering;
4844///
4845/// // A mutual exclusion primitive based on spinlock.
4846/// pub struct Mutex {
4847///     flag: AtomicBool,
4848/// }
4849///
4850/// impl Mutex {
4851///     pub fn new() -> Mutex {
4852///         Mutex {
4853///             flag: AtomicBool::new(false),
4854///         }
4855///     }
4856///
4857///     pub fn lock(&self) {
4858///         // Wait until the old value is `false`.
4859///         while self
4860///             .flag
4861///             .compare_exchange_weak(false, true, Ordering::Relaxed, Ordering::Relaxed)
4862///             .is_err()
4863///         {}
4864///         // This fence synchronizes-with store in `unlock`.
4865///         fence(Ordering::Acquire);
4866///     }
4867///
4868///     pub fn unlock(&self) {
4869///         self.flag.store(false, Ordering::Release);
4870///     }
4871/// }
4872/// ```
4873#[inline]
4874#[stable(feature = "rust1", since = "1.0.0")]
4875#[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
4876#[rustc_diagnostic_item = "fence"]
4877#[doc(alias = "atomic_thread_fence")]
4878#[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
4879#[ferrocene::prevalidated]
4880pub const fn fence(order: Ordering) {
4881    // SAFETY: using an atomic fence is safe.
4882    unsafe {
4883        match order {
4884            Acquire => intrinsics::atomic_fence::<{ AO::Acquire }>(),
4885            Release => intrinsics::atomic_fence::<{ AO::Release }>(),
4886            AcqRel => intrinsics::atomic_fence::<{ AO::AcqRel }>(),
4887            SeqCst => intrinsics::atomic_fence::<{ AO::SeqCst }>(),
4888            Relaxed => panic!("there is no such thing as a relaxed fence"),
4889        }
4890    }
4891}
4892
4893/// An atomic fence for synchronization within a single thread.
4894///
4895/// Like [`fence`], this function establishes synchronization with other atomic operations and
4896/// fences. However, unlike [`fence`], `compiler_fence` only establishes synchronization with
4897/// operations *in the same thread*. This may at first sound rather useless, since code within a
4898/// thread is typically already totally ordered and does not need any further synchronization.
4899/// However, there are cases where code can run on the same thread without being synchronized:
4900/// - The most common case is that of a *signal handler*: a signal handler runs in the same thread
4901///   as the code it interrupted, but it is not synchronized with that code. `compiler_fence`
4902///   can be used to establish synchronization between a thread and its signal handler, the same way
4903///   that `fence` can be used to establish synchronization across threads.
4904/// - Similar situations can arise in embedded programming with interrupt handlers, or in custom
4905///   implementations of preemptive green threads. In general, `compiler_fence` can establish
4906///   synchronization with code that is guaranteed to run on the same hardware CPU.
4907///
4908/// See [`fence`] for how a fence can be used to achieve synchronization. Note that just like
4909/// [`fence`], synchronization still requires atomic operations to be used in both threads -- it is
4910/// not possible to perform synchronization entirely with fences and non-atomic operations.
4911///
4912/// `compiler_fence` does not emit any machine code. However, note that `compiler_fence` is also
4913/// *not* a "compiler barrier". It can be helpful to think of a `compiler_fence` as preventing the
4914/// compiler from reordering certain types of memory operations around it, but that is a simplified
4915/// model which fails to capture some of the nuances. The only actual guarantee made by
4916/// `compiler_fence` is establishing synchronization with signal handlers and similar kinds of code,
4917/// under the rules described in the [`fence`] documentation.
4918///
4919/// `compiler_fence` corresponds to [`atomic_signal_fence`] in C and C++.
4920///
4921/// [`atomic_signal_fence`]: https://en.cppreference.com/w/cpp/atomic/atomic_signal_fence
4922///
4923/// # Panics
4924///
4925/// Panics if `order` is [`Relaxed`].
4926///
4927/// # Examples
4928///
4929/// Without the two `compiler_fence` calls, the read of `IMPORTANT_VARIABLE` in `signal_handler`
4930/// is *undefined behavior* due to a data race, despite everything happening in a single thread.
4931/// This is because the signal handler is considered to run concurrently with its associated
4932/// thread, and explicit synchronization is required to pass data between a thread and its
4933/// signal handler. The code below uses two `compiler_fence` calls to establish the usual
4934/// release-acquire synchronization pattern (see [`fence`] for an image).
4935///
4936/// ```
4937/// use std::sync::atomic::AtomicBool;
4938/// use std::sync::atomic::Ordering;
4939/// use std::sync::atomic::compiler_fence;
4940///
4941/// static mut IMPORTANT_VARIABLE: usize = 0;
4942/// static IS_READY: AtomicBool = AtomicBool::new(false);
4943///
4944/// fn main() {
4945///     unsafe { IMPORTANT_VARIABLE = 42 };
4946///     // Marks earlier writes as being released with future relaxed stores.
4947///     compiler_fence(Ordering::Release);
4948///     IS_READY.store(true, Ordering::Relaxed);
4949/// }
4950///
4951/// fn signal_handler() {
4952///     if IS_READY.load(Ordering::Relaxed) {
4953///         // Acquires writes that were released with relaxed stores that we read from.
4954///         compiler_fence(Ordering::Acquire);
4955///         assert_eq!(unsafe { IMPORTANT_VARIABLE }, 42);
4956///     }
4957/// }
4958/// ```
4959#[inline]
4960#[stable(feature = "compiler_fences", since = "1.21.0")]
4961#[rustc_const_unstable(feature = "const_atomic", issue = "160078")]
4962#[rustc_diagnostic_item = "compiler_fence"]
4963#[doc(alias = "atomic_signal_fence")]
4964#[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
4965#[ferrocene::prevalidated]
4966pub const fn compiler_fence(order: Ordering) {
4967    // SAFETY: using an atomic fence is safe.
4968    unsafe {
4969        match order {
4970            Acquire => intrinsics::atomic_singlethreadfence::<{ AO::Acquire }>(),
4971            Release => intrinsics::atomic_singlethreadfence::<{ AO::Release }>(),
4972            AcqRel => intrinsics::atomic_singlethreadfence::<{ AO::AcqRel }>(),
4973            SeqCst => intrinsics::atomic_singlethreadfence::<{ AO::SeqCst }>(),
4974            Relaxed => panic!("there is no such thing as a relaxed fence"),
4975        }
4976    }
4977}
4978
4979#[cfg(target_has_atomic_load_store = "8")]
4980#[stable(feature = "atomic_debug", since = "1.3.0")]
4981impl fmt::Debug for AtomicBool {
4982    #[ferrocene::prevalidated]
4983    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
4984        fmt::Debug::fmt(&self.load(Ordering::Relaxed), f)
4985    }
4986}
4987
4988#[cfg(target_has_atomic_load_store = "ptr")]
4989#[stable(feature = "atomic_debug", since = "1.3.0")]
4990impl<T> fmt::Debug for AtomicPtr<T> {
4991    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
4992        fmt::Debug::fmt(&self.load(Ordering::Relaxed), f)
4993    }
4994}
4995
4996#[cfg(target_has_atomic_load_store = "ptr")]
4997#[stable(feature = "atomic_pointer", since = "1.24.0")]
4998impl<T> fmt::Pointer for AtomicPtr<T> {
4999    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
5000        fmt::Pointer::fmt(&self.load(Ordering::Relaxed), f)
5001    }
5002}
5003
5004/// Signals the processor that it is inside a busy-wait spin-loop ("spin lock").
5005///
5006/// This function is deprecated in favor of [`hint::spin_loop`].
5007///
5008/// [`hint::spin_loop`]: crate::hint::spin_loop
5009#[inline]
5010#[stable(feature = "spin_loop_hint", since = "1.24.0")]
5011#[deprecated(since = "1.51.0", note = "use hint::spin_loop instead")]
5012pub fn spin_loop_hint() {
5013    spin_loop()
5014}