Big problem is that no one seems to realize, as Nick Thomas did (very good read that addresses this very topic), that you cannot throw out anecdotal information.
It must be brought into the body of study along with all of the empirical or quantitative information or evidence. Sad that "science" either hasn't figured out how to do it or refuses to understand that it can be done.
Another extremely good read for you to perhaps understand what's going on in lucid dreaming (among many other things), is this book by Cater.