You are not doomed! You met a girl that you genuinely connected with. Even if it doesn't work out with this one, chances are there will be another great girl out there for you. I know that sounds cliche, but I really believe it.
If you're looking for a good read, I recommend this book. The author is a dating coach that has been there and knows what he's talking about!
20-something female here. I don't think your situation is a red flag, but I'm sure there are women out there that have had negative experiences with partners and may have a hard time looking past this - those women aren't a match for you, and that's okay.
No matter how much experience you have, dating sucks most of the time. Meeting someone that you want to connect with and having that instant attraction is great, and there are countless tales of great dating experiences from people from all walks of life. But insecurities are really hard to overcome, and make it difficult to confidently put your best self out there in the dating world.
I recommend this book. He's a dating coach with tons of experience. He also has a podcast.
I truly wish you the best of luck!!