True, many academic scholars doubt the truth of the virgin birth for this very reason. However, a lot of the biblical sources are the earliest ones we have to go by. Even if they're biased, they're still something. Questioning is good, but if you end up questioning absolutely everything, you wind up like the Jesus Seminar, with nothing of real substance one way or another.
Paul's letters are the earliest source that we have, stemming from around the 50s (Jesus died sometime around 30 CE). Mark is generally considered the earliest Gospel (although there is a growing minority camp arguing for priority of Matthew), and then Matthew and Luke-Acts, followed by John, most of these written after the destruction of the Temple in 70 AD. Our earliest source is Paul, who didn't know Jesus, at least not in the same way the disciples would have.
A great book on the subject, which it sounds like you may be interested in, of what we can know about Jesus from proof and archaeology and other historical records is Jesus Outside the New Testament by Van Voorst.