Have you tried sleeping on a soft and flat pillow? I have a down pillow that isn't too lofty (could be higher imo) and it doesn't cause this issue but is still supportive enough when I ball it up to sleep in my side. This is the Amazon link.
Popular alternatives to
DOWNLITE Extra Soft Down Pi...
Coop Home Goods Original Loft Pillow Queen Size Bed Pillows for Sleeping - Adjustable Cross Cut Memory Foam Pillows - Medium Firm Back, Stomach and Side Sleeper Pillow - CertiPUR-US/GREENGUARD Gold
YourFacePillow Beauty Pillow - Anti Wrinkle & Anti Aging Back Sleeping Pillow - Wrinkle Prevention Pillow to Sleep on Back - Memory Foam Beauty Sleep Pillow to Keep Head Straight (Standard)
Beckham Hotel Collection Bed Pillows for Sleeping - Queen Size, Set of 2 - Soft Allergy Friendly, Cooling, Luxury Gel Pillow for Back, Stomach or Side Sleepers