No problem, my friend!
You're going to have trouble boring 1/8" holes with a 1/8" end mill, you need some room to ramp into the stock so the bit isn't plunging directly down, that's not how they're meant to cut.
I would invest in a smaller single-flute end mill (2mm is a good size, still rigid enough to be a little aggressive in aluminum) and do helical bores for the holes: try using a 0.002" pitch and a 30IPM feed rate for the boring operation, vacuuming out chips more or less continuously. This way, your bit is constantly stepping over 60% instead of plunging right into the stock.
This is a good variety pack:
https://www.amazon.com/HOZLY-3-175mm-Carbide-Milling-Engraving/dp/B07F33HP75/ref=mp_s_a_1_10