I'm a southerner, and although I've never had a confederate flag on anything I've ever owned, and I don't condone it being flown in any form by something sanctioned by the government, but I don't view it as a racist symbol because it's just isn't racist from where I come from. When I see it, my first thought is southern, not racist. I've known plenty of southern black people who have flown the flag themselves. People will call me ignorant and tell me that I don't know what the flag stands for, and my response is typically that they are ignorant because that isn't what that symbol means to people anymore.
Flags are symbols, and they mean what they mean to the person who is looking at the symbol. You can buy a Japanese naval flag on Amazon, and if you put it in your house most people wouldn't just automatically assume that you hate America. If my great grandfather saw that flag in your house he would probably spit on you though.
Conversely, a swastika symbol historically means 'good'. What if all the sudden people started telling you that swastikas meant good and was a good symbol to use and there was nothing wrong with it, and that the symbol can't change just because some Nazi's used it. Historically, the rebel flag was a racist symbol, but now it has more changed to symbolize southern culture. Contrary to popular belief, overt racism isn't accepted in the vast majority of the south, and racism isn't part of the southern culture anymore; at least for the younger generations.
Amazon New 3x5 Japanese Battle Flag Japan Naval Ensign Flags https://www.amazon.com/dp/B000FAI8S6?ref=yo_pop_ma_swf
And my waifu is kuroneko from oreimo.