Let him go. He's not into you, even if for reasons that you might not consider legitimate. Take his response as an indicator of the way he perceives himself and read http://www.amazon.com/The-Choke-Artist-Confessions-Underachiever/dp/0446573450 for a different person's unrelated experiences which might nevertheless help give you some perspective. If you truly like him as a person not just a love interest and there's a mutual possibility for a friendship and he's not treating you badly, be his friend. Be honest and upfront about the crush and don't expect anything out of the situation.
Or, you could go put your energy into people who aren't concerned about your race and maybe save yourself some pain.