I know how you are feeling and I am sorry, I am going through the same thing...abandonment is a cumulative injury and when we go through it again all of the past pains come along with it. It IS a type of trauma. This book has literally been saving my life and can give far better advice than I could. Good luck on your journey, we will both be better people for this I really believe that!
The Journey from Abandonment to Healing: Revised and Updated: Surviving Through and Recovering from the Five Stages That Accompany the Loss of Love https://www.amazon.com/dp/B00G3L1BMG/ref=cm_sw_r_apan_29SXYMJD2PBXWHR1628M
What a heartbreaking story, OP. Have you received any counseling for your childhood abandonment issues? Your father unintentionally abandoned you, and it sounds like your mother emotionally abandoned/neglected you. These things can dramatically impact a child and cause issues well into adulthood. It can make you feel like you're never good enough to be accepted/loved, and that may be why those judgmental comments feel so gut-wrenching to you.
I recommend this book if you want something to read that may help with some of your feelings.
Edit: Try not to feel guilty about your financial situation. I doubt any of the people who are envious of your circumstances would trade having their parents around during their childhood for money.