Replacing them (or dealing with them in some other way) will have an effect.
Not replacing them (or dealing with them some other way) will also have an effect. :)
You want to do the thing that is mostly like to lead to unbiased estimates of the population parameters, whilst also giving you small standard errors.
If it's age that's missing, and older people are less likely to report their age, then if you don't deal with the missing data in some way you are going to have incorrect age estimates (and other things). That doesn't mean that there is a way to deal with them though.
If people don't give their age randomly, then you don't have as much of a problem - you'll get an unbiased estimate of age.
There is a large literature, and several books, on dealing with missing data (amazon.com/Applied-Missing-Analysis-Methodology-Sciences/dp/1606236393/ is my favorite). But it depends how far you want to go.