3D Math Primer for Graphics and Game Development is a great book for this. Really covers the stuff you need to know and more importantly ways to think about these subjects.
Some of the things they teach would have saved me immense amounts of time back in the day working on early game engines.
https://www.amazon.com/Math-Primer-Graphics-Game-Development-ebook/dp/B008KZU548/
I've found the 3D Math Primer for Graphics and Game Development to be great. It doesn't really cover the OpenGL API (for that I recommend https://learnopengl.com/), but it does provide a nice background on all the math behind 3D graphics (since learnopengl uses the GLM math library).
http://www.songho.ca/opengl/index.html also has some great tutorials on the math behind OpenGL transformations (although it doesn't use "modern" OpenGL, but that doesn't really change the math involved).