We have "Baby Banz" hearing protection for each of our kids. They wear them at drum corps shows and anytime they're in close proximity to the drumline I teach.
They work very well, and the kids like them: http://www.amazon.com/gp/product/B002SW3EZ6
Is your toddler a tablet/phone obsessed child like most toddlers are? Could you bribe him with a screen? I feel like this might be a really opportune time to screw the AAP recommendations, throw some toddler-sized headphones (here are some noise-cancelling ones from Amazon.com) on your kid, and let him go to town with Paw-Patrol and Bubble Guppies on the iPad.
I've taken my (now) 6yo to preseason games for the last two years, starting when she was 4. I wouldn't take her to regular season games due to noise/crowd. I wouldn't take her before that because of concern about dealing with possible potty accidents and, well, 3yos are a bit crazier than 4yos.
Note that with the new bag restriction it would be even more difficult to bring a kid who is not potty-trained.
I have a set of headphones for if/when the game is loud and bring along a variety of games/toys/books/whatever so that she will be entertained even if the action on the field isn't that compelling (especially since we should up a half hour before kickoff).
Sure they can be used inappropriately, but as long as the parents aren't right in front of the speakers and stay at an appropriate distance the kids will be fine. The Baby Banz hearing protectors are specifically made for children to wear at concerts, races or other loud events. Just because you have children doesn't mean you can't continue to go to things that you enjoy, it just means you have to take extra precautions.
BabyBanz are worth the price - around $30 on Amazon.
You sound like you want to bring them, so do it
I'd recommend getting ear muffs for at least the baby, maybe also for your 2 year old
We use them at baseball games
Not sure if these are the ones we bought, but it's the same brand and we got them from amazon
https://www.amazon.ca/Baby-Banz-Hearing-Protector-Earmuffs/dp/B002SW3EZ6