You're probably looking for the documentation around the Linux kernel's [memory barriers](https://www.kernel.org/doc/Documentation/memory-barriers.txt)
For Linux from a high level perspective I think How Linux Works from no starch is a really good book: https://www.nostarch.com/howlinuxworks2
From a programming perspective The Linux Programming Interface: https://www.nostarch.com/tlpi
While i haven't read it yet I've seen good reviews of Advanced Programming in the Unix Environment 3rd edition. It has some applications to Mac as well since Mac is BSD based.