If you are looking specifically at the history of the penal colonies, try Robert Hughes' The Fatal Shore. I read it some time ago and enjoyed it quite a bit.
N.B. Hughes is an Aussie and no fan of the British.
As a Canadian, I find Australian history to be very interesting.
The Fatal Shore by Robert Hughes is a great read... up to it's final couple of chapters on the 20th Century, when Australian history becomes pretty similar to any of the other Dominion's.
Oh, no, no, no. I recommend to you a close read of The Fatal Shore, by Robert Hughes, which will get you better understanding of how it wasn't ever a "nation of inmates", as you say.
That won't cover the period where the world was falling in love with Australia, though, and it flirted closely with being the best of all of the other Big Experiments up into the '90s (with some extremely notable and regrettable exceptions). I don't know what reading to recommend for that, unfortunately.
Robert Hughes' The Fatal Shore will answer your question in excruciating detail. Mr. Hughes was perhaps the 20th century's most highly accomplished art critic. He was also a native Australian. The book is a very good non-academic history of Australia's settlement by England. At the risk of stating the obvious, the criminals who were sent to Australia had very bleak existences to put it mildly. Note that most of the criminals were sent to Australia for petty offenses.
The first small convict flotilla of 11 ships found its way into Sydney Harbor in the humid summer (January in those parts) of 1788. America had had an oblique hand both in this first tenuous settlement and in the development of all the other vast Australian outdoor gulags. For the American colonies, which had once taken Britain's prisoners, often assigning them to farmers along the Eastern seaboard, were now independent and refused to receive Britain's exported criminality.
You can read about Australia's beginnings as a penal colony and the effects that that had on the development of the country here:
After WW2 the Greeks chartered planes to send young teens out to Oz as 'domestic servants'. You can imagine how well that went for them.
Australia has an appalling history of human right abuses and while I love my Aussie family members none of them have a damn fucking clue about anything.
This is an excellent book if anyone gives a damn https://www.amazon.com/Fatal-Shore-Epic-Australias-Founding/dp/0394753666