There's a field called symbolic computation that deals with this sort of thing. Assuming that you have a minimal mathematical background (calculus, differential equations) and have studied algorithms, then I would recommend an introductory book like this one to you. If you have just started to learn how to program and have not studied algorithms, do not buy this book now. Instead, you'll need to gain a good grasp of C++ (a language that is already difficult to learn how to use properly), and then read a basic algorithms book. It would help if you know how to write mathematical proofs, and have some experience with discrete math.
If you want to read more advanced material in the field, then I anticipate that you will need a much more rigorous mathematical background (linear algebra, abstract algebra, analysis, etc.).