You remind me of the hidden hand thread that gets posted here from time to time.
The idea that pain/struggle is necessary for growth feels like a jagged pill to swallow. But then you can look at a baby learning to walk and it doesn't seem cruel, just normal, even cute. So why would the spirit/consciousness be so different?
Are you familiar with the concept of wetiko at all? I am currently reading this https://www.amazon.com/Dispelling-Wetiko-Breaking-Curse-Evil/dp/1583945482 and it also reminds me of some of your comments. The idea is that the self identity as separate from God is the wetiko/devil and it plays out in our treatment of others when we see both ourselves and them as separate from God and therefore unconnected from each other.
The book alludes to breaking from binary and dualistic thinking/seeing as the cure, but I haven't finished it yet.
I don't know if I'm ready for breaking through yet but the message came and went too fast for me to catch it so I am guessing maybe not haha