Hey there! The thing is that it's intended as a second factor. This means that if either your password is stolen, or your key is stolen, then they still don't have access to your account - and you have time to either change your password or cancel your second factor.
If physical safety is a concern, then I'd perhaps suggest that in terms of security then the best next step you can take to improve your security is to use a password manager if you aren't already. By using a password manager, you can use autogenerated passwords which are much stronger, and you don't have to remember them either.
Personally I use Keepass, but it can be somewhat complicated to setup and sync. For ease of use others in my family use Firefox + Firefox Lockwise, and I've heard good things about Bitwarden - though I haven't tried it myself.
I mean, unless they are risk of the being crushed (like at the bottom of a bag with a lot of weight), it should be alright
​
If you want to be ultra-sure, you could get cheap dust covers on Amazon for 0.25€/piece, Like these (sorry for the link Amazon France, it's just to give you the idea)