What I’m going to talk to you about today is sort of an overview giving you a feel for how it is that the brain interacts with the ear when we’re in complex settings especially, because that’s really where you see the effects of the brain more clearly. And I should say I started out as a psychophysicist, what I was interested in is how sound is localized in space, and over time I realized that one of the main things — that knowing where things are is a good thing, but we don’t really use our sense of sound if we’re sighted to do that without a lot of detail. We really rely on the sense of sight for that, but nonetheless if you went to the poster session or go to a poster session today and try to listen with the only one ear you’ll have trouble, and I guess I challenge you actually to go into one of the poster sessions and plug one ear and try and converse, because you’ll find that having two ears and having a spatial sense of sound really is very helpful in exactly those kinds of settings. So I became very interested in understanding that and decided as I went along that just doing psychophysics and computation wasn’t enough, and so what you’ll see today is a blend of a kind of an overview and some demonstrations of the effects that I study, but then also some brain data kind of showing us how it is that we use our brain when we’re listening.

So it is pretty amazing when you think about it, how well we communicate in noisy settings. And if you go into a crowded cocktail party, because that is sort of the stereotypical example is the cocktail party, one of the things that’s kind of interesting is all of the sound that is in that environment adds up acoustically before it ever enters your ear. And so there may be some very interesting interactions going on at some cocktail party and there’s some things that are less interesting, because if you’ve ever seen me talk about this I always do use this slide, although I do update it a little bit depending on the circumstances. But it’s kind of funny because invariably I seem to be the person stuck talking to this guy, who is not the most interesting person in the world. He’s a little self-involved, he’s not maybe the most fun person in the cocktail party. But one of the beautiful things about it is I can stand there and nod politely at the right times while actually throwing my attention elsewhere, and paying attention to the very interesting things that I might be more excited to hear about. And that’s pretty neat. If we try to build a machine to do that, we fail. We’re not nearly as good at this as any good listener with healthy hearing. And it really involves the brain very, very directly. So I’m very interested in studying those processes and understanding how it is that we can filter out sounds.

One of the things that’s kind of interesting is if you think about how psychophysics and psychoacoustics has evolved. We started by worrying about whether things were audible. And certainly if you can’t hear that something is there, that’s a problem. You’re not going to be able to do much with that if you can’t hear it. But one of the things that’s fun is in a lot of settings, it’s not it’s not actually – this is even better. It’s not that my son Nick is not visible somewhere in the scene, it’s that there’s a whole bunch of interesting things going on in the scene, and to find him in that scene you have to interrogate each of the faces, one after the other. Now it’s kind of interesting you, literally do interrogate each face one after the other because your brain organizes that visual scene into faces. It doesn’t investigate each nose, it investigates each face until it finds the right one. So it’s not that he’s an invisible, the analog of inaudible, but rather that there’s clutter. There’s other things vying for attention, and you can’t make sense of him until you look at him specifically. And that’s really what attention is all about.

Now vision is actually — we’re catching up, but we traditionally, the studies of attention actually started in audition at in the turn of the mid-last century. But vision took over because vision is a bigger field and they made much, much greater advances, faster advances in understanding the neuroscience underlying attention. We started to catch up but we’re still a little bit behind. And one of the things that’s really fun is if you look in vision, you can actually show that if I direct attention like to a particular location in space — this is my other son Will — the effect on the brain is literally to turn down the representation of other things so that his face is the thing that gets through and is analyzed in detail. Now I kind of hinted that I’ve been using these slides for a while. I just I continue to because it’s getting funnier and funnier because my son Nick is about to graduate from college, and Will is now a sophomore. So I really have been working on this for quite a while and have been looking at it from all different angles. And like I said I started out looking at it as a behavioral issue, but then I brought in techniques from neuroscience to study how is the brain engaging in these tasks.

So it’s early morning everybody stretch. Okay, because I’m about to make you do some work. I’m going to demonstrate — hopefully these demos work — I’m going to demonstrate for you the effects of attention and how good you are at it. And what I’m going to do is play a mixture — and you can see this mixture of two voices, it’s clearly two voices, right? That’s a joke you can laugh. No it’s not. This is time versus energy, and you can see there’s stuff going on. But this is two voices, but they’re mixed together, and you’re going to be able to attend to one if you focus attention on the right features. So I’m going to play this for you, and you’re gonna listen for the sentence that starts “her shaky” and to make it easier, these are sentences that are used to do speech testing and they’re over articulated, clear speech so it should be really easy. And I’m going to ask for people to raise their hands afterwards about what you hear so try to focus on the demonstration.

How many people felt like they heard the sentence that started “her shaky” — you want me to play it again? Of course I could play it again, I could play it an infinite number times — this is not a normal cocktail party. These are hyper articulated speech sounds. I will play it again. Listen again — you’re probably not going to be able to follow the right speaker because these are identical talkers talking nonsense sentences from the same speakers with no spatial cues. There’s no linguistic, I guess there’s no linguistic or semantic cues that differentiate the two streams and so they mix together. But listen again, and listen to what you can get out. You’ll hear words.

Despite the fact that it’s a mixture of the same voice, with the same pitch range, from the same direction, how many people have heard words? Everybody could hear words, and that’s one of the things our brains are immensely good at: pulling out structure from a mixture of sound. So you’re pulling out the words, the syllables the words formed together naturally because of the continuity and time and frequency and expectations about what words look like. But because there’s nothing to allow you to latch on to the right words through time that connect them and differentiate one sentence from another, you can’t follow one sentence against the other. Here are the two sentences just for completeness:

Her shaky increases will leap on their quarrel

Really no predictability, that’s part of the reason this example works. They’re nonsense sentences. Here’s the other.

Their greedy pull ends at the carpet

And if I put them together, you can follow it if you kind of know, if you’re primed a little bit. But the mixture cannot be pulled apart, so it really is more like that picture of blue, it really is like a picture where the organization of the different words or the different messages is to unclear for your brain to separate the sounds. And in order to focus attention on one thing, you have to separate it from everything else first. That’s a necessary piece of this puzzle. Now I don’t mean to imply that segregating sound sources is a process that happens, and then attention happens. Especially in noisy settings where it’s just on the edge of what you can do, you may start to sense that there’s something you want to pay attention to, focus on it that will help it get segregated by starting to suppress other things, and it will build up through time. You can actually see this in behavior, that people through time get more and more precise at this focus of attention, if it’s not obvious at the very beginning of something. So if I’m trying to read this mess of words it’s really quite hard. And I don’t know — many of you are more clinically minded than I am. I sit in my lab all day, but everybody that I know who works with people with hearing impairment, it’s really this problem of segregation that they tend to describe. They’ll tend to describe being at a restaurant and hearing, when they put in their hearing aid, that all it does is make the clinking sound of the dishes nearby louder. That’s really a failure of the process of segregation and attention, they’re unable to filter out the unimportant stuff because they don’t have a representation that supports the detail that you need to segregate and then select out from a mixture the right thing. So really, this is not the normal situation. For a healthy ear, most people are not talking at the same time as they’re identical twin, nonsense sentences from the same direction. Normally in the real world, everything is independent of everything else. And so if one speaker is talking and making some sense, somebody nearby is talking, but there’s features in the sound or in the visual image that differentiate the words or the messages. So when you look at this scene versus this scene it’s completely different. You can make out words when they’re a mush, but it’s effortful, it takes time, it’s not natural. You see it as a blob that you then have to really analyze.

That’s very different than when this the words are differentiated by the feature of color, and its gray scale actually works too if you’re colorblind, I was able to get that to work. But if you look at this scene, you now interrogate each word very naturally and the meaning comes very quickly compared to when they’re all mixed together. So it’s not that you can’t get any information out of a scene that is mixed that you can’t segregate naturally. It’s just that it’s slower, less precise, and guess what if you’re trying to keep up in a conversation that extra time that it takes to pull stuff out of the mixture, kills you. You cannot keep up with the conversation. And I myself, I’m a middle-aged person about you know, when I was 45-ish I started to really notice myself that I was having more trouble in complex settings, and it’s annoying because I’m kind of a smart alec and I like to make smart ass remarks. But now sometimes if it’s too noisy I can’t be quick enough because I’m still processing this, and just a split-second late and the joke is gone. it’s so disappointing.

So it’s real problem, and I don’t have overt hearing loss. I just have normal middle-aged hearing — and we can talk about that at the end — but the process takes time now that it didn’t used to, and that a healthy young ear doesn’t need in order to make this kind of decision about what to suppress and what not to.

So now, I kind of played with you, I gave you an example, got you all excited — you’re going to tell me and I was going to show you how good you are at listening. Hopefully the acoustics in this room are not so bad that this example will actually work for those of you who are not my age or older, or who have hearing impairments at least. But what I’m going to do now is play an example that really you should be able to segregate. Now I gave this example — two days ago I gave a talk and I used this example with a roomful of psychologists, and I asked a question and they all raised their hands, but I it may not be that way here because your backgrounds are very different. So how many people here have heard of the seven plus or minus two problem. About half. That did not remove everybody, like everybody. It’s a memory thing. Seven plus or minus two is kind of the number of items you can keep in working memory, quickly, you know without memorizing it overtly. So if I’ll play something you can rehearse it and keep it in mind for a short time and then recite it back. And guess what a seven digit phone number just is at the limit of what you can do. So I’m about to test your short-term working memory, and I’m going to play you a seven digit phone number. It’s spoken in a mixture, but it’s segregable. You should be able to hear it and isolate it and focus attention on it so that you can tell me the number. And it’s a male voice, its metallic sounding, it’s been processed to be metallic sounding — it’s actually, for reasons I won’t go into, for an experiment. But it works really well for this demo because it’s really very distinct and it has a monotone so it doesn’t change pitch. It’s really easy to grab and latch onto it. And so is everyone ready, because I really am going to call on someone so and if you want to write the number down if you’re worried you’ll forget it, you can write it down in real time. I don’t mind. This is not really about the memory part, it’s just it happens to be what the limit of what you can do usually. Is everyone ready hopefully I’ll click on the right thing, here we go. [mixture of voices] How many people got the phone number? Good, a majority. Of those who have your hand up, how many got all seven digits? Let’s do it this way — of those who got all seven, did you — keep your hand up I’m a teacher of those who have your hands up: How many of you (without speaking out loud because you don’t want to ruin it for everybody) how many of you can tell me any of the content of the other speaker that was in the mixture. That’s pretty common, that’s pretty common. There’s one person in the audience of about a hundred two days ago who could do it. And sometimes that’s about right. Now, if you’ve heard me play the example before, you probably could because you knew I was going to do this mean awful trick. I really made you focus attention. I challenged you. I said this is about limit of what you’re going to be able to do. And you’re overachievers, you’re like I’m gonna get this. You really focused. This is not an ordinary audience exactly. And I was depending on that. But I also played phone digits, a set of phone digits. Can anybody guess why I used numbers rather than a sentence? Less context. Because if I’m speaking and making some sense then you don’t need every word to know, you still get the gist, which is really what communication is about. But with a phone number, you’re at the bar, you meet somebody nice, you missed one digit — sorry. There’s no predictability. So that really focuses you and makes you maintain attention throughout, which is why you were so successful. So nobody could tell me any of the content. Again without saying it out loud, can anybody tell me anything. Did you get anything about the quality of the other thing in the mixture? I absolutely made you concentrate that’s the point. When you are successful at concentrating, you will suppress things. So just for fun, if anyone wants to get in contact with my very dear colleague Steve Coburn he’s a wonderful man, very very knowledgeable about speech and hearing and binaural hearing — just 617 before those digits you’re set. The other thing in the mixture, though, didn’t get processed. Now if I had asked you more right after I played the mixture, even if you hadn’t successfully pulled out the content, some of you would have been able to say it with a woman. Because I’ve done this since my kids were tiny. I’ve used this example because it’s engineered to be a good example. I’m going to play the same mixture, but now in exactly the same mixture just listen for me. It’s a little quieter so I’ll play it because some of you may have hearing impairment.

It’s hard to understand two things at once

So that’s the thing, when you focus when you concentrate, when it’s easy to segregate things, focusing on one suppresses the other. It’s a competition. And we’ve done some studies where we literally try to get people to listen to two things at once, and really there’s not evidence for you listening simultaneously attentively to two things at once. There’s evidence that you multiplex, that you switch between them rapidly in order to make sense of things. Which can work in normal conversation because there is so much predictability, especially when you’ve been married 29 years and you’re trying to watch the TV and your husband is talking. I know what he’s going to say before he even says it. I don’t have to hear every word. I just know from the way he moves what he’s about to say.

So anyhow, predictability is a huge feature that we use, and it’s when things get hard that predictability becomes less helpful, because it’s not as redundant because we’re slower and we’re really trying to pull this information out. But one of the things that’s fun is, when you focus successfully — which isn’t always — but when you do, you will suppress the other thing if it’s segregated. And you you can see it in the brain. This is where I think it gets really fun.

So there’s been a lot of studies looking at the responses of the brain to sound. And one of the things you can do is put a cap on the head and measure voltages, and Elyse I’m sure will tell us much more about this, but you can put a cap on the head and measure voltages on the scalp. So this is Inyong Choi and this is Scott Bressler. This is Inyong when he was a postdoc now he’s a professor at University of Iowa, but he did some studies with us when he was a postdoc looking at exactly this, and got me in the lab too, which is fun. And I’m you can actually see, whenever there’s an onset of a sound it actually causes synchronous activity in a lot of neurons in the brain. And when that happens it will end up — those the currents that are being generated by the brain are aligned in the right way so it aligns and produces a strong total current so that they don’t cancel. You’ll end up seeing energy on the scalp when you measure it on the scalp. Any onset of a sound can lead to a very strong, very stereotypical response. And you can see it on the scalp simply by measuring voltages. So we played a game where we created an attentive task. We played really simple melodies, there are three melodies. They were different in timbre like different musical instruments. One came from the front, one came from the left, one came from the right. But the game we played is that the each of these melodies had a different number of notes. Which meant that the onset of each of these melodies was isolated in time, so you could actually see from the the different time points which brain response you were looking at. It was either a response to this set of tones or this set of tones or this set of tones. Which is really nice because EEG, electroencephalography this measurement of voltages on the scalp, is actually very precise in time so you can actually by looking at time know which signal you’re seeing. So the melody on the left was four notes. And each of the melodies either went from a low note to a high note and back down which we called zigzagging. It could have been high to low to high and still be zigzagging. Or it started high and went low. Or it started low and went high. But to make sure that we made people listen to the end if it was a zigzag melody, the last zig or I guess it’s the last zag happened on the last pair of notes. So you had to pay attention through the whole thing to know whether it was zigzagging up or down. So just a way of forcing people to pay attention. And what I’m going to do on this here is just plot all of the voltages from all the places on the scalp for exactly the same mixture of sounds — that mixture of sounds from the left right and center — but up on the top, I’m going to plot it when you’re attending to the left stream, which has four notes kind of shown here. And on the bottom I’m going to plot it when you’re attending to the right stream. But it’s exactly the same mixture of sounds and they look pretty similar. Until you look at the details because after every onset that you’re attending, there’s this stereotypical negative positive negative response. The differences in the magnitude come from differences in place on the scalp and reflect the fact that sound evokes — that sensory response tends to evoke strongest responses at the top of the head for reasons I can explain or I’m sure Elyse could explain better. But if you look at the right, it’s similar, but the timing of the events is different because they’re aligned to the onset of the attended stream, not to be ignored stream. So literally you can look at the scalp and tell what a person is listening to. Inyong actually took this a step farther and was able to predict for a single three-second trial like this, which of the streams the person had attended based on knowledge of what their brain did on all the other trials at better than chance performance, for every single listener. So one single trial, if you know what the structure of the signals are, you can tell which one the brain is responding to. Which is pretty neat. And the other thing is there’s a large individual variation — in this kind of test you can get big variability in how well people do, and it turns out the variability is related to how well you modulate the response in the brain. So this is to the same sound — on average there’s a big response when you’re listening here at this time point to the blue, but not when you’re listening to the red. So if I look at the change in the brain response, the same signals but it’s big here and small here that’s a a modulation of the brain response. I can measure how big that is on the individual listener, and that turns out to predict how well they do on the task which suggests that there’s actually differences in how effectively you can focus attention that you can see in the brain signal.

So this is the data. This is the change, the modulation of that brain response, versus how well people did on the attention task. For 11 listeners, and you can see there’s a pretty strong relationship. It turns out to relate to other features in the brain signals but this is pretty neat. That how effective you are at suppressing the unattended sound relates to how well you can do the analytic task of answering what the sound was that you’re attending. It’s also interesting because Scott Bressler just published a study, and I think I didn’t even update the slides because that was published last week it finally came out — looking at a bunch of veterans who’ve been through blast. And it turns out they aren’t very good at these kinds of tasks at all. And we studied these veterans who had been blast exposed because there’s lots of evidence that they have lots of cognitive problems. One thing that’s kind of hard to tease out is, they may have some hearing damage too because they’re veterans, they’ve been exposed to lots of noise. The blast exposure itself could cause damage to the cochlea. We did some control experiments to convince ourselves we don’t think that there’s damage to the cochlea or to the sensory apparatus that explains the differences that I’m about to show you. We think it really is a central deficit having to do with the blast wave that goes through the brain. When the blast wave goes through the brain it’s this physical wavefront that passes through the brain. The brain is not one single monolithic thing, it’s got different material it’s got skull and air and and soft tissue, so that blast wave travels at different speeds depending on the medium, which means that as you’re going through the brain you’re to get faster and slower parts of the wave which is going to cause shearing forces on the on the brain and that may damage long-range connections in the brain which could lead to the kind of cognitive executive function deficits that people often have. I know that I missed him because I was not here during it, but Eric Galen was here earlier in the week talking he has a whole session on this. So we had this example that we did with a bunch of veterans who are blast exposed to see if there were cortical deficits in this population. We did exactly the same kind of task, it was three different streams. They actually had pitches that were very separate so this is pitch and this is time, and again they were doing exactly what I just showed you attending left or right to these different melodies. This is a a plot of the control subjects it’s the same data I just showed you. But now what I’ve done is I plotted on top of each other the responses in the brain on average across this group for exactly the same stimuli. But when they were attending to the red onsets which are kind of vertical lines and red, when they’re attending to the blue onsets or when they were told not to respond. So a third of the trials we actually — the screen would come up and say listen left, or it would say listen right, or it would say don’t do anything. And every time anybody got a right answer we give them a penny. Which on the trials that worked. The passive trials meant they couldn’t answer anything or they didn’t get their penny. It’s amazing what a penny will do because these are really kind of boring experiments, but a penny per trial is enough to keep people on task. And they really focus, they want that penny. So these are exactly the same stimuli it’s exactly the same inputs to the brain. The brain is changing what it’s doing, and so after a red onset you see a strong negative response when they’re attending to the red but not when they’re ignoring everything. Or not when they’re paying attention to the blue stream. Conversely blue is bigger than red, red is bigger than blue, the attempt that there’s actually that third stream from the center they never attended actually has onsets that never elicit big responses, blue bigger than red, red bigger than blue. So they’re doing exactly what I showed you before, it’s just a different way of plotting the data. Here’s the same data in a group of mild traumatic brain injury patients. They don’t show the same modulation. In fact the ERPs are noisier to begin, with but there’s no modulation to speak of. Some of the listeners got a little bit, most didn’t get much. Here’s the behavior — again this is the data that I didn’t show you, but it’s the data from that beginning, that first set of control subjects with Inyong’s study. This is how often they got the right answer for this particular version of the task. This was an easy, easy version of the task for normal subjects. They were around ninety five percent correct on average. The worst one was at about 85%, you can see the individual subjects in circles. This is the percentage of the time on those third of the trial — this is the percentage of the time that they failed to respond in the little time window that they were allowed to respond in, on the two-thirds of the trials where they had to say what the sounds were. And they had very low likelihood of not responding. And this is that the percentage of the time when they were told don’t respond or you’re not getting your penny, and they incorrectly made some responses regardless of what it was. And most of the subjects were quite low. There’s one dude who — I think he’s ADHD — I know there’s one guy who just couldn’t control himself. I don’t know. But in general he’s clearly an outlier. Here are the data behaviorally for the blast exposed veterans. The best subject is still barely at the 50th percent — I mean the 25th quartile the worst subject is actually below chance level which is one-third, although chance performance here because of the number of trials we have, this is statistically not different than chance. But you can see most of them, the mean is sitting around 62 percent. These people cannot do this task. Now it’s not that they couldn’t do is the melody going up down or zigzag. Before we even take this data we test them on just that task by itself. Here’s a melody, is it going up, down or zigzagging, no problem. You put that in a case where there’s another melody at a completely different frequency range from a different direction, and they fail. They fall apart, but they don’t just follow apart there. They’re actually also unlikely to answer — of course that could be because they’re not sure, they’re trying to work it out and then time’s up. But they even are worse at withholding responses. That’s an executive function problem. They cannot withhold the response when they’re supposed to withhold a response. So they clearly have executive function problems here. And like I said we did some tests, it’s in the paper we convinced ourselves there are some that there they don’t have — as a group — statistically they’re hearing isn’t really different than the normal group. That’s not because they’re hearing is great, it’s because the normal control group of young adults who are college age have huge variation in their hearing ability, even when they have normal thresholds, and again we’ll come back to this at the end. So our blast-exposed veterans look a lot like the worst of our subjects who come in every day, statistically the two groups are not different because it’s not a large number in either group, but the vets — the hearing difference doesn’t explain this because even the worst of our normal controls is far, far better than this group. So that’s kind of the first part.

So what did we talk about? The big picture, the big take-homes: auditory attention allows us to communicate if you can segregate the sources — if you can separate them. And if you can do that the cortical responses in your brain are actually modulated, as long as the processing that allows you to do that executive control is working.

So how many of you — we are going to change gears a little bit — how many of you know what a neuron is most. people know what a neuron is. How many of you, would you say that a neuron spiking, firing of neurons is that where information lies in the brain? Can we kind of agree on that? Most people if you ask them — I mean I can believe that the information is in the spike patterns of the neurons, right, there’s information going on and there’s electrical activity. But one of the things about it is in order for those neurons to work well they need to talk to each other. The traumatic brain injury, that blast exposure is causing damage not necessarily even to the neurons, it’s causing damage perhaps to the connections in the brain. This is an image — I love this image — of all of the white matter. The connections between parts of the brain colored to make it clearer, and you can see these big sheaths, it’s like wires truly like wires connecting different regions of the brain that have different function. And the thing is, a neuron maybe spiking and doing the right thing, but if its message can’t connect to the other parts of the brain that need to know what its computing, it doesn’t matter so the interconnected networks in the brain are really, really critical for you to hear in a complex setting where you need executive function, where you need the front part of your brain to help you out. And it’s those connections from parts of the brain that are making decisions about what’s important right now that are sending modulation back to other parts of the brain to suppress the things that are unimportant.

So we did some studies using MEG. MEG is a lot like EEG, it’s measuring electrical activity, but it’s doing it by measuring the magnetic fields generated by those currents in the brain. And it it gives you a slightly better look at some parts of the brain, but it has this property of having very fine time resolution just like EEG. So this is a study that KC Lee who was a postdoc at the time, and Hari Bharadwaj who was a postdoc at the time –also they’re both former students, both were still working with me as postdocs, now they’re both professors. Hari just started at Purdue. But we did an MEG study and it was a really simple study to try and understand what is it in the brain that allows you to focus on different aspects of sound. So we had a cue that was a visual dot, and it came on one second before the sound that you’re about to hear. And you’re supposed to fixate on that and hold your eyes there. And then at time zero a pair of digits came on, just one pair of digits. One was high-pitched one was low-pitched. One was from the left and was one was from the right. So you could listen to those digits, you were supposed to listen to one of the digits, and then respond. But the cue you told you how to listen. You could listen for the source on the left. Or you could listen for the source on the right. You could listen for the source that was low pitched, or you could listen to the source that was high-pitched. So there are two ways of doing exactly the same thing — one paying attention to a spatial feature and one paying attention to a more acoustic feature, the pitch.

What I’m going to show you is a movie of the brain, and parts of the brain that were more active statistically than average on a pitch trial. And the first movie — so this is a blown-up side cross section of the brain this is facing actually for you it’s like this — this is the back of the brain, the front of the brain. The back of the brain is where visual inputs come in and the first thing that happens is we give a visual cue that tells people what to do. And that happens at minus 1000 milliseconds so this is a little time stamp back here. So the first little snippet of movie is just that. Just you’ll see activity in the visual cortex. Boom little bit of activity — this is statistically thresholded to throw out a whole ton of noise. But statistically there’s a little bit of activity because the visual cue just came on and told you what to listen for. The next piece of movie gets a little more interesting. Most of the movie is at negative time before the sound you’re about to pay attention to is on. But it’s starting after that little blip of activity — you now know what to listen for. And what’s cool is you’ll see activity in the brain that’s significantly more than average activity in different regions in preparation for the sound that’s about to happen. First you’ll see an area which is the frontal eye field area, it’s an area associated with moving the eyes, which is really closely associated with attending to things at different places in space and is thought to be part of the visuospatial attention network. But guess what it’s not just a visual spatial attention network — because this is a sound, and you’re not moving your eyes, you have fixate your eyes, yet you’ll see activity in the frontal eye fields part of the visuospatial attention network, because you’re listening spatially. After that happens, you’ll start to see a little bit of activity in the more frontal regions of the brain. This is a latest evolutionarily speaking part of the brain, and it’s the part that makes decisions, and it’s the part that controls what you’re going to do. Around just after that little bit of activity starts, suddenly you’re finally up to the time when the sound comes on, and then you’ll see areas in auditory cortex come on, and when the sound actually starts you’ll see all of these areas talking to each other. So here we go. So you see the end of the visual cue the frontal eye fields are engaged, sound still not on, there’s this activity boom, sound comes on and the whole brain talks to each other this whole network is talking to it each other. This is what happens in a spatial trial. Tou engage this spatial network. If I just kind of look at that spatial region and look at the activity over time, this is what happens to that region — just the energy in that region when I’m paying attention to space. And this is for the exact same trials — or not the same trials, exact same statistically stimulus, but when you’re paying attention to pitch there’s more activity in this area, when you’re paying attention to space then when you’re paying attention to pitch.

If I look, however, at a different region of the brain, an area that’s associated with auditory processing, that activity is actually bigger when I’m paying attention to pitch than when I’m attending to space. And in fact in both sides this is when the sound turns on, this is the evoked response from the sound both areas so a little bit of activity that you can’t localize with MEG perfectly but that’s much bigger in the auditory areas. But all of this stuff is happening before the sound happens. Because the brain knows what it wants to do, and it’s preparing to shut down the stuff that’s inconsistent that’s about to come in.

So I’ve shown you EEG, I’ve shown you MEG, I also collaborate with David Sommers a professor who does vision work. And he’s an fMRI guy. Now fMRI is a way of looking at brain activity that doesn’t have good temporal resolution, but it allows you to see things with great spatial precision, what parts of the brain are engaged. And this was work that we did with Sam Michalka, who just took on a job as a professor, and Abby Noyce a current postdoc with us. And we decided to compare what — we have this suspicion that the visuospatial attention network is not really a visuospatial attention network. So we did tasks where we compared attending to sound versus vision.

So in the first experiment what we did is we had sound and we had visual inputs, and I won’t go into the details but it was kind of comparable tasks in the two domains but both sound and vision are on all the time and you’re either attending to the sound or either attending to the visual inputs. This is data for three different subjects and what we’ve done is color-coded areas of the brain that were more active when you’re paying attention to the sound, there in warm colors, or more attention to — more on when you’re attending to vision, in the cold colors. So the cold colors — visual areas, unsurprisingly in all the subjects in the back of the brain. But if you look carefully in the frontal regions of the brain there’s a blue region and then another blue region. There’s a blue region and another blue region, across all subjects across both sides of the brain. Now people know that these areas are involved in attention, but one of the things that’s kind of funny is most of the studies that have looked at multi-sensory attention have kind of said, well all of these areas up in here are kind of engaged whether you’re paying attention to sound or vision. Part of the problem is if you look at how these places align across subjects, it’s pretty poor. It’s very hard to get them to align because different people’s brains anatomically are different, so it’s hard to get them to align. What we did is actually said, okay, we’ve got four individual subjects this really clear pattern of blue and blue, and if you look inter-digitated sound gets the auditory areas and yellow and yellow. And everybody shows a similar pattern, but their individual areas are different from one to another. If we use those areas defined for that person, can we get interesting results out? And the answer is obviously yes or I wouldn’t be telling you.

So one thing you can do is say, ok I’ve got these regions of interest, these ROIs. I’m going to define them for the individual subject based on whether they were more turned on for it for vision or audition. And I can also see sensory areas that are visual and auditory. If I look at those and define those I can actually plot they energy as a function of time in fMRI, and it’s smeared out it’s nothing like the precision that I just showed you for EEG or MEG, but you can still get this time course. And then you can ask a very silly game or ask a silly question — if I just put you in the magnet and don’t ask you to do anything special, what parts of the brain tend to turn on together? If this time course is on at the same time as some other part of the brain statistically at the same time, that’s called being connected. It kind of suggests those parts of the brain tends to work together. When I was an engineer when I first kind of got into this field I thought it was the silliest thing in the world you’re putting people in a magnet and measuring this brain activity, not telling them what to do. What a waste of money and time. It’s a lot of money. It’s five hundred dollars for me per hour to get this person to sit there. Why are we doing this? But the answer really kinda cute: It’s like a big data problem you’re saying I don’t know what’s going to happen, its statistical and on average, if I do this enough, I’m going to get actually interesting answers out. So what you do is you look and see what parts of the brain are correlated more than chance.

And we had a hypothesis — we hypothesized that if I look at those visual areas in the frontal regions they should be connected to visual regions, and not connected to auditory regions. And if I look at the visual sensory areas they should be strongly connected to the visual frontal control areas, but not to the auditory. Conversely auditory sensory regions where the sound is first represented should be talking to those little orange dots that are auditory, but not to the visual ones.

So that’s our hypothesis and here are the data. If you look at the little blue areas and ask are they connected to this blue region the answer is yes, bilaterally, both sides of the brain. But if I look at the orange areas they’re not. If I look at the auditory regions they’re connected, they are strongly correlated with activity in the auditory sensory regions, but these two areas are not strongly connected to the visual regions. So it really looks like these areas are kind of specialized, sort of.

But there’s this one little twist. And that one little twist is actually kind of interesting. This is the frontal eye field, this network that includes these two things is really what people have called the visuospatial attention network, and also includes areas in the back of the brain up here in the parietal cortex. This is the area that the MEG study I just showed you said was involved in a spatial auditory attention task, yet if I just emit resting and just doing random stuff it doesn’t connect to the auditory regions.

But it turns out you can see connectivity to that region in a task that has spatial demands that’s auditory. So if I do an auditory task and try and make you do a spatial –an auditory input and try to make you do a spatial task we predict, we thought going in that that might recruit auditory regions might recruit the visuospatial attention network. If I did an auditory task that was temporal, it shouldn’t recruit the visual network. We also wondered if the converse might be true too. Sound is really — auditory inputs are really processed with great temporal precision, but not visual inputs. If I do a visual task and it has high temporal demands it might recruit the auditory network even though a visuospatial task doesn’t need to recruit that network. And again I’m showing you the data, it’s published it’s out there, we got what we expected. If I do a visual task whether its spatial or temporal, if I look at those two regions in the brain at the front of the brain that are kind of visual, they’re on, they’re turned on. But if I look at the auditory regions in this control area of the brain that are kind of auditory, if I do a visuospatial task the visuospatial task doesn’t recruit those areas, but the visual temporal task does. So sound is encoded in a network that’s kind of processing temporal information, and if I do a visual task that’s got high temporal demands that grabs that network.

Let’s look at the auditory areas. The auditory areas are on when I do an auditory task whether it’s spatial or temporal. If I look at the visual areas they’re on also in the spatial tasks and a little bit in the temporal task it turns out, that may be that’s something that we’re still pursuing. But what’s interesting is they are more recruited, they’re more strongly recruited, when the task is an auditory spatial task. It’s grabbing this visuospatial network. It’s not just a visuospatial network it’s a network that is spatial, and vision is always spatial — it’s kind of the way we think about it.

So second quick summary: focusing attention cause preparatory activity. Before you know the sounds you are going to attend to, you know what they’re likely to be, and you’re going to process things differently based on what you want to hear. And that starts to happen before the sound turns on. Once you know what to listen for. If you see your husband’s coming from over here and there’s a lot of sound over here, you prepare to shut that off and pay attention over here. No, I didn’t say that.

The brain networks are biased to represent different kinds of information, temporal versus spatial, but that actually means that the temporal tasks — the temporal network is always kind of addressing auditory information and similarly the spatial network always encoding visual information. That’s kind of the inherent kinds of information that those two senses encode to begin with. But you can show recruitment of the other network even if the input modality is auditory it will recruit the spatial network if the task demands are right.

So, the last little piece I want to talk about, I kind of hinted at, which may be most relevant to some of you who do hearing stuff for people with hearing impairment, which is looking at how it is that things can fail if the sensory part of the apparatus is damaged. One of the things I like to think about is when I go to get my hearing screened, they ask me: do you detect that something is there? And if I say yes, they say great. Which is not really what it takes to make sense of a scene. If I went to a vision specialist and he said I was fine just because I can detect that blob, I would not be particularly happy to go out and drive.

So one of the things that’s kind of interesting is the most overt forms of hearing loss really elevate hearing thresholds, but there’s more and more evidence that you can have normal hearing thresholds, yet still have differences in sensation that are really physiological. And the first hit we had of this — Dorea Ruggles was a student in my lab and wanted to study the effects of reverberation on hearing. And I was at the time about 45, and I was starting to have trouble in settings where there was reverberation and crowded settings. So we started to look at normal hearing listeners, and we thought we were going to study an age effect. And what I now call them is listeners with normal hearing thresholds because I don’t think normal hearing is a really good descriptor for people who happen to all have normal hearing thresholds.

So the test is a lot like the kinds of things I just showed you. People were under headphones. We simulated sources from three different directions, each of these three directions had a stream of numbers and your job was to listen to the ones in the center, this is kind of time. So the right answer would be 9 2 6 1. The separation in space that we simulated is quite small — it was 15 degrees. Which isn’t very much at all, it’s about what the limits of a good listener can do. Which was on purpose.

And when we tested people, we tested a bunch of people with different ages in the initial study. What you’ll see is really we’re kind of disappointed because there wasn’t much effect of age to see. What we did see — which was shocking — were huge variance in ability. On this particular task we’re doing, chance turned out to be thirty-three percent. That’s because when people got the numbers wrong, it’s not that they just reported a random number one out of nine — they reported one of the three numbers, just the wrong one. They could hear the words. They all had normal hearing, we expected them to be able to hear the words. When they failed it’s because they failed at using these really fine spatial cues to choose the right one. They failed in selecting the right one. Not failing to hear, but failing to select the correct word. So chance performance given the variation, we expect statistically anything above this grey stipple area is above chance.

Everybody’s above chance. Some people are terrible. Some of our worst listeners are pretty young. Some of our best listeners are pretty — I said that backward — some of worst listeners are pretty young — this guy. Some of our best listeners are pretty old. So age didn’t turn out to be a factor. We actually got some extra older listeners in. Still not a factor. It turns out they were better than our initial older listeners. And notice these aren’t really aged listeners, these are middle-aged listeners. Our oldest was 54, I believe.

What we then did is simulated reverberant energy, and every single person took a hit. Every single person, their ability went down. Now some people are in chance levels.

And then we simulated — this is not a classroom. We made a classroom whose walls were like a bathroom wall so very very echoic. Every single person got worse again. Now the only listeners who are above chance, it turns out two much younger listeners, and two of the listeners are kind of the 30 age range.

It didn’t turn out that age was a factor. if I just looked at age, I couldn’t tell you whether it was a good listener or a bad listener just based on age. But if I told you which of the older listeners it was. Or if I told you the age of a listener and I know how well they did in the echoic case, the older you were the worst the reverberation made you. The hit that you took was bigger. So the younger listeners were more robust. They didn’t go down as much as the older listeners. So age was not a factor, but the older listeners were hurt more by reverberation suggesting that there is something going on with aging.

But the individual variation even among the young listeners was what really shocked us. These are all young listeners with normal hearing yet they vary enormously.

It turns out there’s some interesting work going on down the street from us looking at animal models, and looking at noise exposure. And it turns out you can get a loss of nerve fibers without an effect of threshold detection. This is now known as hidden hearing loss which is a cute term. More correctly it’s auditory — or cochlear snynaptopathy. What really happens is the synapses in the ear, due to noise exposure, are damaged. When they die off the nerves they enervate die and then you have fewer nerves coming out of the ear to convey information to the brain. So it’s a really very peripheral, in my mind, sensory loss. But it’s not peripheral in the sense that the cochlea actually is functioning perfectly. The amplification in the cochlea is fine, and you can show this.

It also turns out that neuropathy occurs with aging even if you don’t have noise exposure. If you have noise it ends up speeding up what looks like a normal aging process. Which kind of starts to make sense if you think about the data I just showed you — older listeners are more hurt by reverberation. They have a little less robustness. They have fewer nerve fibers. Some of our younger listeners probably were exposed more to noise, etc.

So Hari Bharadwaj again came in and looked at what happened for this supra-threshold hearing. So we’re not testing things at threshold and seeing whether you can tell that there’s something there or not. we’re testing things that are clearly there and audible and asking can you make sense of them and analyze them and use selective attention well.

So we got a whole bunch of listeners in — all younger because we knew already we would see great variation even younger listeners. They had normal thresholds — within 15 dB of normal hearing up to 8 kilohertz. We didn’t test higher, they might have some high frequency loss above that. But they also had normal cochlear function. We did a whole bunch of tests, and it’s in the paper that’s now out, looking at the cochlear function. Their cochleas were intact, the amplifiers intact. It has all the right properties. What small differences were there in their cochlear function didn’t correlate with any of the data I’m about to show you. Which suggests there is something that correlates.

And what we did is measured again using just voltages on the scalp how well the sensory apparatus is encoding fine timing information. You do this exactly like you do when you’re going to measure the brain, as you normally think of it that kind of cortex neocortex. But if if I look at high frequency sounds and put in a sound like this.

The DA syllable. Courtesy Nina Kraus who has done a ton of work with these sorts of experiments. You play a DA syllable. For our purposes, the nice thing about the DA syllable is it has a nice pitch. This is a hundred hertz pitch. That hundred hertz is so rapid that the cortex doesn’t fire that fast. It can’t follow that fast a rate. But lower parts of the brain, the subcortical portions of the brain can fire at a hundred hertz can entrain to the hundred hertz stimulus. So if I just filter out the low-frequency stuff and look at just the high-frequency stuff, instead of looking at cortical responses like everything I’ve shown you so far, you actually look at subcortical early parts of the brain. So you can look and, say, play the same sound over and over and over and just average it and you can get out that. Which is a noisy thing, but you can hear the pitch in it, right?

Well you can hear that pitch. It’s in there. It’s noisy/ This is a track just made by one listener sitting for many many trials and we average the heck out of a bunch of trials in response to this DA syllable and you see this response.

You can then come up with a metric for how strong is the subcortical portion of the brain encoding the hundred hertz information. And that metric turns out to be very predictive and very variable across subjects. Again for the sake of time I’m not going to go into the details but you can look at a measure of that of how strong is that response and up in this graph is more fragile, so better listeners are down here. And then you can look at how well do they do on an attention task, and there’s a negative correlation. It’s quite strong. The people who are weaker in their brain response do poorer on this attention task. The people who have the strongest brain response are the best on this attention task — tests just like I’ve shown you already.

But it turns out almost anything you do, it seems, that’s really probing really fine timing information in the brain. These listeners all — these measures are correlate. The envelope following responses — that measure I just talked about in vague terms — that measure of the subcortical ability to encode this hundred hertz sound. That correlates with selective attention ability I just showed you that. It also correlates with how well you can pull out fine amplitude modulation in terms of a sound. It also correlates with how well you can detect a little frequency modulation of a sound. It also correlates with how well you can — what your discrimination threshold is for interaural time differences of sound. So all of these things are features that really rely on fine spectra temporal timing. And there’s something that you might think might go wrong if you had fewer nerves coding things. Every single auditory nerve is a little bit random. The normal ear has many different nerve fibers encoding almost the same information, but with different randomness on them. And the brain takes them and puts them together to get rid of some of that noise, to get better temporal coding. If I have fewer auditory nerves I can’t get rid of as much noise. I can’t average it out the same way. So it’s these kinds of things where these problems show up and interestingly that turns out to really affect selective attention ability in a crowded setting. That’s where you need to be able to pull apart sounds you need to have a really fine representation. When it gets muddy it’s almost like that picture of all blue. I can no longer segregate the sound so I can no longer effectively select the right sound. I can’t segregate them or select them. And in fact the very features I might want to use to select them may not be as clear, like the location cues may not be as clear.

So these are two examples. This is the modulation threshold and an amplitude modulation threshold. And here it is I don’t even read — this is versus the brainstem response strength, and you can you can see a nice correlation people who have a stronger in this case — excuse me, a weaker brainstem response on the top. This is how big does it have to be for you to detect it. And if it’s big, it’s bad. And similarly how big is the modulation have to be for you to detect it, and big is bad. And so again there’s this nice correlation. Here’s that same behavioral AM detection threshold versus interval time difference threshold. And again there’s this nice correlation.

So listeners who are all young adults, who all have normal cochlear function — we would say our normal young adults — differ in their physiology and that correlates with their behavioral ability. So this is, I think quite real, and it’s always the things that it shows up in are tasks that require coding of fine spectrotemporal features. It turns out also it was sort of a post-hoc thing after we started this study we added a little questionnaire and ask people to self-report their noise exposure. There was actually a relationship. The people who reported by their own self-report more noise exposure tended to be the worst subjects, and it was statistically related.

Now there’s been some other studies since then including one very large scale study out of England that doesn’t find a relationship between self-report and these sorts of measures, but in talking to the guy who did that Chris Plack, he’s a great researcher. The one thing he hasn’t looked at — so he doesn’t see a relationship between self-reported noise exposure and the measures I just talked about — but what he hasn’t looked at is whether there are self consistent differences. So self-report is pretty unreliable, I’m surprised that we found it in fact. And it also may not capture everything that matters like genetic predisposition, and you know all sorts of other things. So self-report may not be really a good reliable measure, and even noise exposure may not be the only thing that matters here. But what we see and what Chris hasn’t yet looked for, is we see individual differences that are very, very consistent that show up in the physiology objectively, and that show up in behavior in all sorts of ways.

So I think I will end here and the last piece of the talk, the thing you should take home, is that there are big differences amongst hearers who we think of as normal, with normal hearing thresholds. and normal cochlear function that are different in their sensory abilities. That also has an enormous impact when it comes to hearing in noisy environments because you’re feeding that central system a representation that’s not rich enough to make sense of. And we think that this is a form of cochlear synaptopathy, although showing that that’s what it is in humans is basically impossible directly because we cannot do a basic kind of study to look at that. But with that I’ll stop and take questions if there’s time. Thank you.