This is not a pleasant problem to deal with, and I will point you to the literature I used when dealing with ICC.
First of all, don't bother with Cronbach's Alpha. It is baffling to me how often that metric is used, despite that - at least I couldn't find these - there are no mathematical derivations what it is measuring, how to construct confidence intervals with it, measuring significance, etc. In fact, Cronbach has never dared refer to that metric with his name on it (referring it to as just "alpha"), and he published an article posthumously stating
> I no longer regard the alpha formula as the most appropriate way to examine most data.
suggesting generalizability theory, or G-theory, as an alternative.
You should read this paper on G-theory by Shavelson and Webb as a brief introduction, and then get yourself a copy of Brennan's text. My recollection is you'll need to read into the reliability coefficient and other such quantities which are in Brennan's text for what you seek.
Significance tests will have to be done via bootstrapping, and Brennan discusses these in detail.