So wear a rainbow T-shirt. I just looked on amazon and there are a ton that don’t have anything associated with pride. They just look like rainbows.
You’re work events are that casual that you’d wear rainbows? If you’re a man, and people are that intolerant they’ll see rainbows as feminine and you wouldn’t be “allowed” to wear it whether it was a symbol of pride or not. So either way the rainbow is not the issue, it’s the bigots.
Here are some....