Yes, abstract algebra will be useful for some parts. So will number theory.
You should definitely have some idea about algorithms and time/space complexity.
An introductory book like Katz - Lindell will be pretty self contained and it will be easy to understand having a math degree.
The ideal way would be to learn some discrete mathematics, logic/proofs skills, and then read a proofs-based textbook like Katz & Lindell's Introduction to Modern Cryptography. Other good resources: Dan Boneh's Cryptography course at Stanford, Bruce Schneier, Handbook of Applied Cryptography, and Matt Green's Cryptography course at Johns Hopkins.
You can probably get started in cryptography without getting a math degree first, but I'll try to give you some intuition as to where the math comes in. I mention this because it would be helpful for letting you decide what math/theory is worth learning.
To oversimplify, cryptography creates puzzles/math problems that have special properties.
Example: encryption. An encryption algorithm takes your message and your secret encryption key, and uses that to create a custom "puzzle" (ciphertext). The puzzle is designed so that the solution to the puzzle is just the original message! The puzzle is also designed so that it's extraordinarily difficult to solve the puzzle, unless you also have a "shortcut" (secret decryption key).
Indeed, this is a secure way to send your message (even if someone is spying on you): It's hard for an eavesdropper to decipher the message, because they don't have the secret decryption key. They don't have the decryption key, so the puzzle (ciphertext) is extremely hard to solve! But it's easy for the receiver to read the message, because they do have the secret decryption key. In that case, the puzzle is easy to solve.
You might ask: "Well that sounds nice, but how on earth could you design a puzzle like that?!?"
To give some intuition about how this sort of thing might work, think about a maze. It's very difficult to find your way out of a maze. But if you have a map, then it's very easy. As far as we can tell*, there are some puzzles/math problems with "shortcuts" like that. So we make cryptography using those puzzles!
Cryptography is all about making puzzles that embed information somehow (e.g. the message). Cryptography also quantifies how difficult it is to solve those puzzles. Example: Suppose you're stuck in a maze without a map again. Cryptography would use statistics/probability to give a ballpark estimate of how long you'll stumble around before you find your way out. Now consider the case where you do have a map. How much easier is it?
So there it is, that's where the math comes in. I encourage you to learn the math/theory, but I hope that gives you an intuitive start on the "why" behind crypto. (I personally find that to be very helpful when I'm trying to learn about a new subject).
* Note: I'm assuming P != NP
Good beginner/intermediate texts on cryptography/cryptology:
From a mathematical perspective: An Introduction to Mathematical Cryptography by Silverman et al.
From a provable security perspective (probably the most important to both academia and industry): Introduction to Modern Cryptography (new 2nd ed.) by Katz & Lindell
Serge Vaudenay's A Classical Introduction to Cryptography (it's an in between of the above 2 books).
Christoph Paar's Understanding Cryptography with a video course.
Lastly, a really fantastic all around book on network security (including crypto) would be: Network Security 2nd Ed. by Kaufman and Perlman. It is a little old though but still relevant. Also has great analysis of real world protocols such as IPsec (IKE, ISAKMP), Kerberos, SSL/TLS, S/MIME, etc.
I would actually recommend starting from the last text by Perlman & Kaufman and then selecting from the above 4 books.
I used this book in an intro crypto class. It's proof based and also relatively self-contained, so you might find it useful.
I have no clue what you're talking about but if you want proof it's from this book: