I discuss that a bi more in the new book, and David Albert looks at it carefully in his book:
http://www.amazon.com/Time-Chance-David-Z-Albert/dp/0674011325/
The main point is that we can only successfully correlate conditions in the current universe (such as "I remember liking that show") to what conditions actually were ("I was watching that show, liking it") because entropy used to be lower. Otherwise, the most likely explanation would be "the impression that I liked the show just randomly fluctuated into existence."
Some of my favorites:
David Albert's <em>Time and Chance</em>
David Albert's <em>Quantum Mechanics and Experience</em>
Tim Maudlin's <em>The Metaphysics Within Physics</em>
Huw Price's <em>Time's Arrow and Archimedes' Point</em>
Tim Maudlin's <em>Quantum Non-Locality and Relativity</em>
Hans Reichenbach's <em>The Direction of Time</em>
Hans Reichenbach's <em>The Philosophy of Space and Time</em>
John Bell's <em>Speakable and Unspeakable in Quantum Mechanics</em>
Harvey Brown's <em>Philosophical Foundations of Quantum Field Theory</em>