There might be other options, but Multiple View Geometry in Computer Vision was a classic when I was a student.
There are some problem classes that are much more efficiently solved by classical (as in: without neural nets) CV. Even neural nets use linear algebra a lot, and although it's not exactly the same as for 2D/3D transformations you might want to get acquainted with the math behind all that.
I'd like to recomend several things, that greatly inspired me and made some order in my UAV-related photogrammetry understanding:
Came at this from the opposite direction - needed to write projection calibration algorithms; this incredibly useful book supplied the linalg algorithms: https://www.amazon.co.uk/Multiple-Geometry-Computer-Vision-Second/dp/0521540518/
Yep, you've got it! Pick up Multiple View Geometry if you really want to get your hands dirty.