It sounds to me like you need to do a shit tonne more research on dwarfism before you embark on writing a world for us. I would recommend this book: <strong>https://www.amazon.com/Lives-Dwarfs-Journey-Curiosity-Liberation/dp/0813535484</strong>
Around 80% of people with dwarfism have no family history of it and are born to average height people. You'd need to come up with a pretty convoluted story to justify how a society of us came to exist and how it is maintained as many LPs have normal height kids.
A useful thought exercise might be this:
If your child was/is born with dwarfism, how would that child come to end up in this dwarf society? Would you be compelled to abandon it to strangers in the city? If so, why? What are the implications of these actions? Also, why does this child grow up identifying more with these strangers than with you, it's family?