The canonical text (which by now is outdated, but that's OK if you're just starting with game theory) is Fudenberg and Tirole's <em>Game Theory</em> (1991). The book isn't lacking for rigor; they dive right into fixed point theorems and the like in the first chapter.
I'm not sure abandoning credible precommitments is really a strong strategy in general, though.
This is inherently a larger game-theoretic problem.
Sure, sometimes you commit to a position that you'd rather not or would be disadvantageous in general, but a utility-maximiser that's always greedy in respect to the most immediate choice (so no long-term strategy for iterated games) gets absolutely annihilated by a longer-term thinker.
Often winning games relies on credibility, and credibility relies on following through on commitments. Tit-for-tat is a powerful prisoner's dilemma strategy because it precommits to defecting against defectors. Conversely, defect-o-bot is not a big winner in the iterated game.
Given that Schelling Fences are just a specific case of precommitment, I don't think they're 'neurological pacifism'. Game theory is more complex and agent-dependent than that.
edit: To be honest OP, I suspect you either haven't studied game theory (which is no slight to you - most people haven't), or you're just poorly communicating any understanding you have. The problem most people have with your logic is that it seems to follow a chain that goes like this:
(1) Schelling Fences are a form of neurological pacifism or extreme defensiveness compared to other strategies.
(2) Neurological pacifism is a losing strategy in a large iterated game like the one you've sort of laid out for 'thought-space'.
(3) Not only are Schelling Fences losing strategies (as sometimes you may not have a winning strategy - not all games are fair!), they are heavily suboptimal losing strategies and should hence be abandoned.
The problem is that you assume (1) without proving it, and then fail to prove any link between (1) and (2), and also fail to prove that (2) proceeds from (3).
So people get lost at (1), as you haven't proven it, and sort of move on bemused through (2) and (3) trying to figure out when the proof is going to arrive.
When the average poster is saying "I wonder what heck this has to do with Schelling fences", they're just confused because the logical structure they expected wasn't there.
I'd honestly suggesting going through one of the texts on the field, digesting it, and coming back and hitting these ideas again in a few weeks or months once the information has percolated through your brain. If you've got a bit of a mathematical background, Fundenberg and Tirole is pretty much the way to go. If you'd prefer something online for free, Kockesen and Ok have their text available on their uni website.
I'm like 95% confident that's correct.
I would check out this pdf of Fudenberg and Tirole's book and see if it meets your needs. It was basically my go-to text in Grad School. It would definitely suffice for a upper-division or grad level course.
https://homepage.univie.ac.at/Mariya.Teteryatnikova/WS2011/FT.pdf
Note that this link triggers a download of the following book: https://www.amazon.com/Game-Theory-Press-Drew-Fudenberg/dp/0262061414
i'm not a game theorist but, if you wanna get serious about it i hear Fudenberg and Tirole is a pretty classic textbook on the subject.
https://www.amazon.com/Game-Theory-Press-Drew-Fudenberg/dp/0262061414
i also did a quick perusal of this more recent open source textbook and it looks quite readable... and more importantly, free.
http://faculty.econ.ucdavis.edu/faculty/bonanno/PDF/GT_book.pdf
I agree that O&R is a good reference book, and I'd also recommend Fudenberg & Tirole for a more verbose (but still precise) introduction.