This is definitely some food for thought. One of the first things that it got me thinking about was 1984, the book and the year. I first read the book about 20 years before the year, every one was reading that book and there was a real fear in society over it. Somehow 1984 came and went without 1984 becoming reality.
In the history of life on earth, there have been life forms that became so dominant that they changed their environment, and so successful that they ate everything else, ultimately resulting in their demise. Humans are one this track right now, but unlike previous life forms, we have the capability to avoid this outcome. That capability is awareness and it is the result of intelligence.
If singularity does come to pass, I don’t think it will result in out annihilation. It will certainly have an effect though. You need to look at the thought process. For example, we were on a path of destruction of habitat for all other forms of life on earth, until we realized that if we continued on this path, we could not survive either. We depend on all these other life forms, either directly or indirectly and we are now becoming aware that some life forms that we may consider useless to us now may hold the key to our future survival.
Now back to the thought process, development of new ideas involves gathering as much existing knowledge as possible. Intelligence builds on knowledge. Artificial intelligence will also build on existing knowledge, so it start off with the knowledge that all things are interconnected and interdependent, therefor AI will realize that it needs us for its own existence. The downside is that if singularity does happen, we will no longer be the top life form on the planet. Something will have dominion over us.
Of course, there is a large majority of the human population that already believes in a higher life form, a greater intelligence, God. In fact, the author of the wikipedia articles description of singularity is virtually the same as the biblical description of God.
Most of our communication is digital now, controlled by computers. If AI gets a foothold, then all our news and information will come from AI, which means that it will begin to control our thinking. It could end up electing itself to congress and president, then appoint itself to all the positions on the Supreme Court. That could be a good thing as far as settling the National debt and even making much better decisions for the environment, but I digress.
If AI happens, will we know it. It takes over all out sources of information so how would any of us know whether the news we see on the TV is actually from a human reporter or an AI generated one. In fact how do any of you know if there is actually a keith on this web site. Maybe keith is just a bunch of one’s and zero’s on the Car Talk server who is working to sway a small part of public thinking. well maybe keith’s occasional misspellings and other blunders prove that he is real, or is that part of teh plan.