The difference in amplifier quality between what's in a receiver and a dedicated power amplifier is night and day. I won't get in to all of the differences, there's dedicated subreddits for audiophiles.
At one iteration of my setup I attempted sort of what you are trying to do.
The Chromecast is terrible for streaming audio.
I had a device to extract the digital audio from my Chromecast ultra which I fed to my separate DAC through an optical connection.
I compared streaming to the Chromecast vs connecting the DAC using USB to my phone. The Chromecast was definitely worse. The USB connection directly was far better.
I now am using a raspberry pi connected to my dac through usb and stream to it using Moode, and bubbleupnp on my phone.. If you're interested you can check some info about it here.
http://moodeaudio.org/forum/showthread.php?tid=2199
It's a finicky setup and not for everyone. There are much easier alternatives though they don't come cheap.
Sorry if it sounded like I was talking down to you, I'm just trying to prevent others from making the same mistakes I already have! Let me know if you have any questions!
I'm running the iQaudio DAC+ with a Pi2 and MoodeAudio - http://moodeaudio.org - I'd recommend this over Volumio and RuneAudio since their development seems to have stagnated.
I wouldn't recommend using USB DACs with the Raspberry Pi.
Woah, sorry for the deluge of comments. I kept getting an error on mobile saying "something went wrong." Here's link that talks about multiroom using moOde
There are a bunch of how to's out there if you google "DIY Raspberry Pi DAC". I ended up drawing from a bunch of different sources as I went along. I built two systems:
For the first, I just hooked up a Raspberry Pi 3 b+ with a HiFiBerry DAC+ Pro so I could stream audio to my vintage AR receiver. I tried a couple different OS distributions before settling on Moode. The four main options are Rune Audio, Volumio, Moode Audio, and Max2Play. Moode seemed to provide a lot of flexibility while still being easy to use.
For the second, I started with a Pi Zero W, but later upgraded to the Pi 3 b+. Since my main objective was to get rid of the whole receiver + CD player + tape player system in my wife's studio, I went with the HiFiBerry AMP2. I later added an enclosure, a rotary encoder for volume control, an OLED display, and a physical power switch. this thread provided the best info on how to setup the OLED display (got to play around with Docker), and this thread provided some guidance for the rotary encoder. I setup the rotary encoder push button to do a clean OS shutdown. It isn't supper elegant, but it does allow you to avoid SD card corruption when the hard power switch is toggled.
Be prepared to do some debugging and you may need to learn a bit about Linux if you don't already have some experience with it, but it is a fun project and there are a lot of resources out there to help.
I might also recommend Moode Audio - a similar project.
I switched to it from Volumio because its web GUI keeps in sync with MPD. I don't know if volumio fixed this, but they broke this feature with an update so I had to leave the project.
And if you like Volumio, take a look at the other forks like RuneAudio and Moode Audio Player. The Moode dev has opted to start charging but he's doing decent work with it. They each have some advantages and disadvantages over each other, in terms of features and hardware support.