That would be a shame. Library of America publishes a kickass collection. I am big on a book having the right proportions, feel of the paper, printing and such, and for some reason my tome of Lovecraft is just perfect.
Allow me to suggest a non-Kindle alternative. The Library of America edition. It's a non-profit publisher, so it's a surprisingly high-quality book (binding, paper, etc.) at a reasonable price. Plus, it has a huge number of his stories, and includes all of his most famous ones. It also has a good amount of interesting end material; notes, biography, etc. I read it last year, and loved it.
Do the stories need to be read in any particular order? This was suggested to me, and I think I'll check it out once I'm done the book I'm currently reading.
This one is my favorite. But, if cost is a concern, note that there are links to a free PDF/e-book collection in the sidebar of this sub.
Great books of great short stories:
Tales by HP Lovecraft
Stories by Anton Chekhov
The Complete Short Stories by Mark Twain
The Complete Stories: Vol 1 by Isaac Asimov
The Collected Tales by Nikolai Gogol
Paycheck and Other Classic Stories by Philip K Dick
The Martian Chronicles by Ray Bradbury
The Illustrated Man by Ray Bradbury
I'd start with "Shadow Over Innsmouth", then move onto "The Colour out of Space", and then hit up "The Dunwich Horror"
the library of america's volume is excellent. its 22 works include almost everything worth reading that he wrote.