This app was mentioned in 11 comments, with an average of 1.55 upvotes
That is not a feature build into Android. There are 2 ways to do it though. As you already mentioned, there are a few 3rd party launchers that support doing this. You can also do it via a widget. There are widgets that mimic an icon but with the unread counts like this one:
https://play.google.com/store/apps/details?id=co.uk.cornwall_solutions.notifyer&hl=en
I did just find this.
https://play.google.com/store/apps/details?id=co.uk.cornwall_solutions.notifyer&hl=en
In saying that, I have a watch on its way which kinda relies on Google Now launcher as far as I know
Notifyer with Nova launcher: https://play.google.com/store/apps/details?id=co.uk.cornwall_solutions.notifyer
I downloaded an app called notifyer unread count. Its works well but obviously not ideal. Link to the app is below.
https://play.google.com/store/apps/details?id=co.uk.cornwall_solutions.notifyer
Notifyer should let you do this on any launcher. A little bit of work to setup but it works well.