This 4 year old reddit thread sums up what I'm talking about. There just seemed to be a running theme of geometric prisms, ethereal backgrounds and checkerboards. You could say it's vaporwave, but I don't think that fits it exactly.
But someone did call it "mathstetics" and I kinda like that.
These are good examples as well.
https://www.amazon.co.uk/Geometry-University-Chicago-Mathematics-Project/dp/067345956X
https://www.amazon.com/Cengage-Advantage-Books-Intermediate-Applications/dp/1133103731
Maybe I'm crazy, but I think these could be vaguely related enough to be called something specific.