I think we disagree on two big things, and the rest is fluff.
First, I still don't think it's fair to characterize the questions ("which is more likely" and "which has more M&Ms") as "nonsensical and purely logical." I wouldn't even call the questions atypical. We contend with simple questions like this all the time in society. The intent behind both questions is unambiguous, and there are no reasonable alternative interpretations, unless you assume the question asker is simply mistaken.
Second, we're on different pages about what the experiment is evaluating. I don't claim that the people who answer incorrectly are less capable of rationality, nor that they don't understand probability. What I believe is that they are less likely to recognize when it's best to apply their logical toolset. As a result, they're more likely to use intuition to answer questions that are best answered via careful analysis. This appears exactly like failing to understand the question itself.
To wit: The younger kids used the length of the line to answer "which has more M&Ms", but the older kids immediately recognized that it's better to use a more in-depth analytical tool: counting. I hypothesize that the younger kids are simply not as good (yet) at knowing when to apply this tool. Intuition is easier and less effortful, thus it's the default unless we make ourselves think harder. This pattern extends to adults, too. How many people think we should be tougher on crime because it "feels" like there is more violence today than ever before, yet don't even consider that they need to look at some actual numbers to justify this conclusion?
It's specifically a question of "which is more likely, A or A^B" that is weird. The typical "which is more likely" has non-overlapping categories. It doesn't even need to be a mistake. If I ask whether you want a sandwich or a burger, it's obvious that 'sandwich' actually means 'non-burger sandwich'.
I'm claiming that they are not using intuition instead of analysis. Or at least, you can't tell that from their answers. There are logical reasons to interpret the question as non-overlapping sets. They could be performing a very careful analysis and still pick the second option.
I only disagree with your last sentence. There's no way to perform a careful analysis of option A vs option A+B while maintaining the assumption that they are non-overlapping sets. Simply reading the answers proves that assumption wrong. Thus, it seems way more likely that the explanation is a lack of thinking/analysis... i.e. intuition. People are answering the question they expect instead of the question that's in front of them, because that's easier: http://lesswrong.com/lw/9l3/the_substitution_principle
But you are right, we can't know that from their answers alone. There are numerous possible reasons for picking the wrong answer, so further experimentation is required if we really want to know why.
First, I still don't think it's fair to characterize the questions ("which is more likely" and "which has more M&Ms") as "nonsensical and purely logical." I wouldn't even call the questions atypical. We contend with simple questions like this all the time in society. The intent behind both questions is unambiguous, and there are no reasonable alternative interpretations, unless you assume the question asker is simply mistaken.
Second, we're on different pages about what the experiment is evaluating. I don't claim that the people who answer incorrectly are less capable of rationality, nor that they don't understand probability. What I believe is that they are less likely to recognize when it's best to apply their logical toolset. As a result, they're more likely to use intuition to answer questions that are best answered via careful analysis. This appears exactly like failing to understand the question itself.
To wit: The younger kids used the length of the line to answer "which has more M&Ms", but the older kids immediately recognized that it's better to use a more in-depth analytical tool: counting. I hypothesize that the younger kids are simply not as good (yet) at knowing when to apply this tool. Intuition is easier and less effortful, thus it's the default unless we make ourselves think harder. This pattern extends to adults, too. How many people think we should be tougher on crime because it "feels" like there is more violence today than ever before, yet don't even consider that they need to look at some actual numbers to justify this conclusion?