Let Eliezer Yudkowsky say it: "It is a terrible thing to contemplate that 150,000 people die every day without having read this book. Don't let it happen to you. (snip) If you're just an ordinary guy and you read this book, you may not understand some of it. But when you're finished, you will no longer be ordinary."
Yudkowsky may be familiar to redditors as the author of Harry Potter and the Methods of Rationality.
I'm in, but some people are put off by calling it that. Any idea what that genre of literature would be referred to? Intellectual non-fiction, or something?
As for books to add to the list, Surely You're Joking, Mr. Feynman! is a fantastic read.
> ...any sufficiently complex pattern in anything could be said to be conscious, and an AI has the potential to not only be as intelligent as we are but as morally alive.
I think the first part of that is a bit of an overstatement. There are different sorts of emergence, and "complexity" doesn't guarantee self-reflection or consciousness. But see the book "Consciousness and the Social Brain" for some possible paths to get there.
In terms of AI, there are several Really Difficult problems -- irony might be one. Others that I've wrestled with are humor, deceit, empathy, and our concept of time. One thing that all of these have in common is a combination of neurological, psychological, and social components. It's not that hard to make a "humor engine," but that's not the same as having humor emerge from the interplay of a neural (or neural-like substrate), cognitive processes, and social constructs.
You might like Le Ton Beau De Marot: In Praise Of The Music Of Language which is an entire book revolving around the fuzziness of language and the difficulty of translation, due to how polymorphic language is. Too bad it's Hofstadter's worst received book.
Me, I liked Metamagical Themas. It's a handful of short snippets into fun playful topics. Not as deep as GEB, but more playful and easier to digest.
I also couldn't get the formatting that I wanted, so I just put it my derivations up on google docs.
I am not too sure on #4 though.
I particularly thought of this as being relevant to GEB when I saw the discussion on Hacker News. To summarize:
Sorry for the late reply. As far as I know, we are separate persons. :-)
I haven't explored taoism, where do you suggest I start? I'm not sure if I have any interesting contemplations to suggest for you, but I'll just mention what occupies my mind most of the time these days: Ethereum (technical description). Personally, I'm not very into building applications on top of it, but rather its societal implications and the core protocols and algorithms which allow it to function. They are doing some very interesting research to allow the system to become more scalable and self-reliant.
What I specifically want to mention to you though, is an idea that has been tossed around in the community but I have yet to spend much time on: what if we built an AI running on Ethereum? At the moment it is far from feasible due to technical limitations, but imagine an AI running on a peer-to-peer network. Right now, code on Ethereum can only be executed if someone is willing to pay for its execution, so that does put some constraints on the possibilities, but what if that changed? Would we have Skynet? Even if it didn't change, what would be possible? As I said, I haven't explored the topic much, but I thought that you might find it interesting. :)
Yeah, the Sorites paradox was used in the book I was reading - Fuzzy Thinking: The New Science of Fuzzy Logic by Bart Kosko. His explanation for the paradox is that it is the midpoint in the intersection of the fuzzy sets "sand pile" and "not-sand pile" where each is 50% true. The book touches on a lot of the same areas as GEB but with an entirely different viewpoint.
I'm a Japanese native, and wanted to leave a comment that GEB Japanese translation is available to this date, in two editions - http://www.amazon.co.jp/dp/4826900252/ http://www.amazon.co.jp/dp/4826901259/
The first is the original edition that was released in 1985 (the one I have), and the seconds is the 20th anniversary edition that was released in 2005.
As far as I can recall the translation was decent, though I can't say for sure as I don't have the original.