I recommend you read Alone at Dawn , the story of John Chapman, a Combat Controller and Medal of Honor recipient.
What happened to Chapman is unique, and isn't likely to happen to you.
It's not the biggest or strongest or fastest guys that make it through SOF training. It's the guys who stand arm in arm with their teammates, no matter what challenges come their way, and just don't know how to quit.
If that's you, and you still want to walk that path with your brothers in arms, then don't let anyone tell you what you can't do.
A book came out last year about this:
Maybe Par or Joakim will read it at one point!