I’ve spent a life trying to develop tasks, that are good tasks, that actually measure what we think they measure. When I think about it, I think the one common factor going through anything that I’ve ever tried to do, I would call and other people have called the “task impurity problem.” No task measures only what you think it measures. It doesn’t measure one thing — in cognitive and linguistic, in our kind of research — it measures a whole bunch of things. Trying to find out what all of the contributions of all of those cognitive and disparate linguistic contributions to that task, even when you narrow it down to almost a trivial task, it’s huge.

What is a “good task”?

A good task is a task that measures what you want it to measure. But it’s hard to know what that is. It’s a good task if it measures the thing that you’re measuring — that you think you’re measuring — without unknown contributions to that task.

An example might be a simple sentence comprehension task. There’s a task I’ve worked on for 40 years. It seems impossible that we don’t understand it yet, but we don’t understand it. It’s this simple: When you give people a set of objects and ask them to touch them or manipulate them. And you just say, “Touch the red circle.” What cognitive and linguistic mechanisms and constraints does that impose on the system? Then you take it to the next step: What are the neuromechanisms involved? What are the lesions that cause impairments to different parts of that? It’s enormously complex. It involves visual processing, non-linguistic and linguistic and right hemisphere and left hemisphere and words and sentences and syntax — that sentence isn’t loaded with syntax, but the other tasks like it are. It’s just huge.

So trying to narrow that down, sometimes with imaging help — I don’t that’s proven all that helpful, maybe. And by studying different populations with that task. Who has trouble with it? In what ways? Once we can do that, then we know that’s a good task. If it is good for this group, and not this group. Or it reveals something specific about some population that you’re interested in. Or some cognitive or linguistic process that you’re interested in.

I didn’t answer the question, “What’s a bad task?” But it’s a bad task if you don’t know what it measures. That’s the only bad task.

With so many unknowns, how can you measure anything?

You can become paralyzed by what you don’t know. That’s a common doctoral student problem.

So we can’t be paralyzed by what we don’t know, but we always have to be concerned about it. Clinicians probably aren’t that concerned about it, because they just have to do the work. When I am faced with doing actual real clinical work, that for somebody’s life it matters what I do with them — for experiments, the pressure is off compared to what clinicians have to do. So clinicians just accept more unknowns. You have to. That’s not just true for speech and language pathology that’s true for … neurosurgeons. What they don’t know is mind boggling, given that they’re digging around in your brain. But they know enough to be pretty good a lot of the time. So, you can’t be paralyzed by it, that’s all.

We don’t know enough to do anything. But we know enough to do something.

Further Reading: The Task Impurity Problem

Burgess, P. W. (1997). Theory and methodology in executive function research. In Rabbitt P. (Ed.), Methodology of frontal and executive function. Hove, UK: Psychology Press. 81-116.
“many (if not most) executive tasks tap a range of processes incidental to their main purpose, and that this measurement error (probably more accurately referred to as ‘task impurity’ cf. Weiskrantz, 1992) if one is using the tasks as measures of executive processing-is likely to be greater than in non-executive tasks.”

Miyake, A., Friedman, N. P., Emerson, M. J., Witzki, A. H., Howerter, A. & Wager, T. (2000). Theory and methodology in executive function research. Cognitive Psychology, 41, 49–100.
“this issue highlights the so-called task impurity problem, particularly vexing issue in studies of executive functions (Burgess, 1997; Phillips, 1997).. Because executive functions necessarily manifest themselves by operating on other cognitive processes, any executive task strongly implicates other cognitive processes that are not directly relevant to the target executive function. For these reasons, a low score on a single executive test does not necessarily mean inefficient or impaired executive functioning. Similarly, low zero-order correlations or multiple separable factors may also not be due to dissociable executive functions (Miyake & Shah, 1999).”