Researchers at the University of Illinois at Chicago recently IQ-tested an AI system to see how intelligent it really is. Results showed that it's about as smart as the average 4-year-old —with some important caveats. Indeed, some very, very important caveats.
You can imagine my surprise — and subsequent eye-rolling — when a press release appeared in my inbox yesterday with the title, "Computer smart as a 4-year old." I immediately dismissed it as nonsense without even reading it (we're still about 25 to 50 years from creating an AI that's truly as "smart" as a human child), but many media outlets are reporting on it so I felt obliged to take a closer look.
Here's the lowdown: UIC researchers put the ConceptNet 4 — an AI system developed at MIT — through the verbal portions of the the Weschsler Preschool and Primary Scale of Intelligence Test, a standard IQ assessment for young children. Results showed that it has the average IQ of a young child, but the scores were a bit wonky in different portions of the test.
From the release:
“If a child had scores that varied this much, it might be a symptom that something was wrong,” said Robert Sloan, professor and head of computer science at UIC, and lead author on the study.
Sloan said ConceptNet 4 did very well on a test of vocabulary and on a test of its ability to recognize similarities.
“But ConceptNet 4 did dramatically worse than average on comprehension—the ‘why’ questions,” he said.
One of the hardest problems in building an artificial intelligence, Sloan said, is devising a computer program that can make sound and prudent judgment based on a simple perception of the situation or facts–the dictionary definition of commonsense.
Commonsense has eluded AI engineers because it requires both a very large collection of facts and what Sloan calls implicit facts–things so obvious that we don’t know we know them. A computer may know the temperature at which water freezes, but we know that ice is cold.
And this is exactly why the AI is not nearly as smart as a 4-year old. It's just a glorified calculator at this point — crunching numbers, running scripts, and making probability assessments. What it's not doing are all those things that make a 4-year-old so brilliant: living in an environment and learning from experience.
What's more, the AI is not embodied, nor does it have the biological inclinations that drive human tendencies. It's also important to remember that a four-year-old's brain is in full-on developmental mode; it's a work in progress that's being forged by experience. Intelligence is not something that's constructed, it's something that develops over time.
Sometimes I get the feeling that AI developers simply want to create an end-product AI and say, 'voila, here's an intelligent entity right out of the box.' But that's not how intelligence comes about, and that's not how it works — at least not in the human sense of the term.
Additionally, an IQ test, or even a Turing Test, is a severely limited way of assessing intelligence, whether it be for a human or artificial intelligence. Neither measures for self-awareness, which is an indelible component of intelligence. And until machine consciousness is developed (which is not a sure thing), an AI will not be emulating human intelligence. It'll only be simulating it. It'll only appear to be smart, but it's just pretending.
Which is not to say that ConceptNet 4 isn't performing some clearly amazing functions, many of which jibe well with the verbal portion of the IQ test — an element that's obviously related to human intelligence. But as stated, it's just the tip of the ice-berg when compared to the robustness, depth, and adaptability of human general intelligence.
Thankfully, and despite the oversold title of the press release, the researchers are aware of this.
“We’re still very far from programs with commonsense — AI that can answer comprehension questions with the skill of a child of 8,” noted Sloan. They hope to see AI developers focus on these "hard spots."
Sloan will present his report on July 17 at the U.S. Artificial Intelligence Conference in Bellevue, Wash.