Is it this: I haven’t read the whole series so I don’t know if the main character goes back to his own time in each book. He does travel through a portal (I think in the first book it’s a hill in a bog) and end up on an island during the blitz where he finds a mansion full of kids who have powers. The rest of the series, I believe, is them trying to find other “time bubbles”.
http://www.amazon.com/Miss-Peregrines-Home-Peculiar-Children/dp/1594744769
http://www.amazon.com/Amulet-Samarkand-Bartimaeus-Trilogy-Book/dp/0786852550
First two off the top of my head. Both of the main characters are guys and they are both fantasy-ish. Good luck.