There are other Frozen comics that have been published.
Here's everything so far from Dark Horse Comics (probably the main publisher for Frozen comics right now), and you already have some of them. 4 graphic novels were also published, and they are compiled in the Library Edition. There are also 2 comic adaptations, but only one of them adapted both movies while the other only adapted the first movie.
Joe Books used to publish Frozen comics, but they're out of business. Here are the issues for the comic line, and there was a single annual issue. There was also 2 collections of short comics: Hearts Full of Sunshine and Travel Arendelle. However, at least some of the stories in those 2 collections have been republished in the comics collection from Dark Horse Comics. Joe Books also published a comic adaptation of the first movie, but that already got republished in one of the Dark Horse Comics adaptations, specifically the comic that adapted both movies.
There are also books that show concept art of the movies. And there's also A Sister More Like Me which is illustrated by one of the artists from the movie. Dangerous Secrets and Conceal: Don't Feel might also interest you, though I personally haven't read them yet.
I don't know about magazines, but here's a list of Frozen comics that have been published.
Here's everything so far from Dark Horse Comics (probably the main publisher for Frozen comics right now).
Joe Books used to publish Frozen comics, but they're out of business. Here are the issues for the comic line, and there was a single annual issue. There was also 2 collections of short comics: Hearts Full of Sunshine and Travel Arendelle. However, at least some of the stories in those 2 collections have been republished in the comics collection from Dark Horse Comics.
At least some of those stories you're looking for are in Travel Arendelle, but I don't know about the others.
>Are these books considered canon?
There are two answers to this. One is that they're canon unless Frozen 2 contradicts them. The second (and more likely answer to me) is probably not because I don't know if the filmmakers have read the books.
>Is there a comprehensive guide or list of all Frozen-related books?
There are plenty of books according to this site, but I don't know if that's everything.
>Have you read any of the books? What did you think of them?
I've picked up a few comics, and while I do like the artwork and some of the stories are fun, nothing too big happens in them. However, I have not read the Dark Horse Comics publications, and those stories seem a little bigger than Hearts Full of Sunshine and Travel Arendelle. I have also read The Art of Frozen, and it's definitely worth a pickup for the concept art at the very least. Also, I like the cover of that book. There's also A Sister More Like Me, and I'd also recommend that for the artwork and the different proses for the sisters. One of the storyboard artists did the artwork for the book, and the book shows other sides of them (particularly their childhoods), but I don't know if the book is canon or not.