I don't know what PHP libraries are best, but one option you have is to use Markdown, which lets you do basic formatting and embed code snippets. Then you can simply copy/paste the markdown.
Another option you have, if it's just a blog, is to use a static site generator. I'm not sure what PHP-based options are best (I use Jekyll, which is written in Ruby, but if I'm being honest I don't actually interact with the Ruby part of it at all), but basically the way it works is that you write locally (in markdown files), then use the generator to create HTML. Then you can push that to your server.
Sculpin seems like an option, as does Phrozn.
This is a little easier, in many cases, than trying to create a whole blog (including rich text editing) from scratch.
There are lots of different static site generators, and there are also pre-existing blogs (Wordpress isn't terrible for code samples, but it's not amazing, and I've heard good things about Silvrback, but haven't ever used it).
But if you're still committed to creating something from scratch, finding a markdown parser so you can write in markdown locally and then simply use those posts (instead of having to worry about splitting your work between multiple editors), copy-pasted in directly, will likely be a reasonable solution.
It's not one I have much experience with, but a number of colleagues have used Silvrback (https://www.silvrback.com/), which is ad-free and supports MathJax. But, it is subscription-based, charging about $4/month or $35/year (USD), so WordPress as you describe it would be cost competitive.
If you want to avoid costs, you could go with Google's Blogger platform and configure it to read LaTeX. See an example here: (https://www.codecogs.com/latex/integration/blogger/install.php). Most people who stick with blogging however will often migrate to WordPress which offers additional features, so keep that in mind while making your choices.
After you've gotten several posts up, please share a link to your site - I'm always looking for that elusive combination of good writing and good math!
As mentioned below, Ghost is really good and getting better every day.
I would also recommend the lesser-known Silvrback. I really enjoy their minimalist themes and some of the baked in features they offer.