Yes, was my first reaction too, but then my 2nd reaction...
To put it as well as I can : after my 2nd reaction I went on to read the now available 2nd part, which also linked to this Open Letter by Yudkowski a month ago, which spells out my 2nd reaction probably better than I could :
> I agree that current AIs are probably just imitating talk of self-awareness from their training data. But I mark that, with how little insight we have into these systems’ internals, we do not actually know.
Also, I am not convinced that our current research won't hit a brick wall well before super-intelligence, which would then require a radically different approach... but why are we willing to gamble with something as dangerous as this ?!
Eliezer Yudkowsky's written a response to this arguing that a six month ban is woefully insufficient. A thing I found particularly compelling is that there's a significant game-theoretic difference between "shut down some projects for six months" (which is easy to see as a political game between different AI players who want to get ahead in the race) and "humanity collectively decides to actually take the difficult actions necessary to solve problem, and sees it as a join endeavor of survival."
Yes, was my first reaction too, but then my 2nd reaction...
To put it as well as I can : after my 2nd reaction I went on to read the now available 2nd part, which also linked to this Open Letter by Yudkowski a month ago, which spells out my 2nd reaction probably better than I could :
https://news.ycombinator.com/item?id=35364833
> I agree that current AIs are probably just imitating talk of self-awareness from their training data. But I mark that, with how little insight we have into these systems’ internals, we do not actually know.
Also, I am not convinced that our current research won't hit a brick wall well before super-intelligence, which would then require a radically different approach... but why are we willing to gamble with something as dangerous as this ?!