According to this book, time travel to the past is probably possible, but only under extremely restrictive circumstances which would only allow you to travel as far back as the physical conditions allowing for time travel have existed. It's technically possible for them to arise naturally, but even an intergalactic supercivilization might not be able to contrive time travel to the past which is practical enough to do anything particularly useful with, at least if your goal is what we usually think of as "travel to the past."
Even if the models described here are wrong though, it could still be that time travel to the past is technically possible, but impractical. If the lower limit on how much energy it takes to send a certain mass back in time is cosmically high- like a tenth of the universe's mass-energy to send a kilogram of mass back a year- we'd probably never expect to see colonization of the past even if time travel to the past is technically achievable.