I recently had a sort of similar experience, except mine went South way more than yours did.
Someone recommended to me the book, The Legal Handbook for Photographers. I bought it, and can't recommend it enough. If you're concerned about what you're taking pictures of, and how people will react, that book is money very well spent.
And yes, you have every right to stand on the sidewalk and take pictures of peoples' houses in the US.
Definitely a good resource: https://www.amazon.com/Legal-Handbook-Photographers-Liabilities-Selling/dp/1682032868
Read this book, it might help.
https://www.amazon.com/Legal-Handbook-Photographers-Liabilities-Selling/dp/1682032868/