Check out Auphonic. It can increase LUFS to -19 for mono and -16 for stereo. Run your raw (quiet) recording through that and compare the results with what Audition is doing for you.
Auphonic gives you 2 hours of free processing time per month. So maybe only use five minutes of your show to test.
Good guide here: https://www.buzzsprout.com/blog/auphonic-for-podcasters
Ignore the guy's advice about not adding music. I think that's out of date but otherwise the guide is good.
Here's a quick pass at it, but as /u/adobeaudition said, the source material is going to make it tough to improve.
You need to use EQ to reduce the low frequencies to get rid of the "boom" and "muddiness" in the sound (due to the loud volume and the proximity of the camera).
So reading up on the RockMix5, it seems as though it only sends the master mix over the USB. Which is why you are seeing it as a single source. If you are wanting to see each mic as a different track you will need to purchase something that is a 4 channel audio interface. I highly recommend the Behringer UMC404HD for beginners, then you can route each input to a separate track in Audition.
OK, I thought you were using ASIO4ALL. I would reccomend using it, as it is a lot easier to configure that the UMC ASIO. http://www.asio4all.org/ Just install the program, then ASIO4ALL should show up as an ASIO device. Then follow the above instructions
Another solution, although you would have to pay for it is Audio Hijack from Rogue Amoeba. Probably the best piece of software there is for this kind of thing: https://rogueamoeba.com/audiohijack/
Udemy.com has excellent Audition courses and other Adobe apps too.
All are around USD12 in the sales that come around every 3 days or so.
30 day refund no quibble policy
Mike Russell has free stuff and is excellent.
I just found this app that allows you to make custom shortcuts like the stream deck. I just downloaded it a few days ago to use with premiere and am still playing with it touch portal
You need-
1/8” (3.5mm) iPhone headphone jack adapter. Example
A cable where one end is a normal 1/8” headphone jack, and it splits into two different ends. You can get one that splits into two mono 1/4” (6.35mm) jacks, which is your best bet. Example
I use one that splits into 2 RCA jacks, and I attach 1/4” adapters to each. I already had that stuff, so I just snap it together for the few times I need to do this.
Always remember that there at least a trillion ways to get audio from any point A to any point B, none of it being particularly special. As long as your jacks match the locations they need to go in and out of, there’s a way to make it work.
You can get a headphone amp. Here’s an inexpensive one that behringer makes
Behringer Microamp HA400 Ultra-Compact 4-Channel Stereo Headphone Amplifier,Silver https://www.amazon.com/dp/B000KIPT30/ref=cm_sw_r_cp_api_glc_fabc_KUt6Fb9EKQVE3
Just keep in mind these are 1/4” ports not 1/8” like you have on average headphones. You can get converters but get decent ones because I’ve had a bad experience with a cheap one.
>2013 Mac Pro 5,1
I kinda glanced over this at first thinking it was a trashcan. The USB ports on the 5,1's are USB 2.0 which could very well be the issue. One solution would be a USB PCI-E card such as this one.