The following is a transcript of the presentation video, edited for clarity.

We’re back to the point where you have gotten that review back. And so we’re going to consider sort of two different scenarios. One is the good score where it’s going to move on immediately to counsel, and you think you have a good shot at getting funded. And the other is a poor score or not discussed. And this is where you’re going to immediately move into resubmission mode.

Holly Storkel:

So now going on to resubmission, here’s a quote from the Ogden and Goldberg book. As you can see, no matter how difficult it may be to shrug off an unfavorable summary statement, “shrug it off you must and proceed to a well-written positive improved revision.”

And then there’s a couple of other points here. There’s nothing to gain and everything to lose by contesting a comment so you don’t want to call the reviewer stupid and so on. In your official response, you can do whatever you’d like in your own head. And then the goal of submitting a proposal is to get the funds to support your research. So it’s not the time to have major theoretical debates or, you know, discuss controversies and so forth. To some extent, you need to just capitulate, and do what you can to make your resubmission competitive.

First Steps and Plan of Attack

Holly Storkel:

So the way you want to approach this is to deconstruct the summary statement. Go through that summary statement and pull out the comments enumerating them point by point. Start to put them into thematic groups because some of the reviewers will have similar kinds of comments. And you don’t want to just look at the weaknesses. You do want to look at the strengths too. So you want to make sure that you are seeing what really resonated with the reviewers and make sure that you continue to highlight those aspects of the work. Don’t cut the aim they loved. Make sure you keep that one. But maybe you want to cut the aim that they all hated. So, you know, be thinking about that.

It’s also really good to get another opinion on this to make sure that you’re really reading the reviews in the right way. And that you’re understanding what the comments are. There is a little bit of an art form to deciphering what a reviewer’s trying to tell you. They don’t always tell it to you in a real straightforward way. So it can be very helpful to have someone else who’s had a lot of experience reading summary statements go through your summary statement, and see if what you’re pulling out of it is what they’re pulling out of it as well. Anything to add, Bill?

William Yost:

Just to emphasize something that we’ve talked about many times in the past. You’re a salesman. You’re trying to sell something. And you’re not going to sell something by telling the customer he’s an idiot. You want this. And you have to convince them of that. You have to be their best buddy.

Holly Storkel:

It’s also good to talk to your program officer. They were likely in the room when your application was discussed. They might have notes that they can share with you. Sometimes there’s something that happened at the table that’s, you know, really brought everybody’s scores down. Your program officer will know that. And again, it might not be so clear in their critiques where or when that happened. But your program officer can help give you some insights as to why maybe something went the way it did or why maybe the score doesn’t seem to match the critiques or something along those lines.

So once you have that, you need to come up with your plan of attack. You’ve got two tasks. One is to do this introduction to the resubmission that sort of encapsulates your response. And the other is to actually change the proposal where appropriate.

So we’re going to focus now in the rest of this talk pretty much on the response to reviewers. And assume that you can use the other principles we’ve talked about in grant writing to actually write the resubmission and change that. But you’re going to go through all of the same sorts of things that you do when you write the first version of the grant to write your resubmission. You still want to share it with others, get feedback, all of those kinds of things.

Response to Critique: Introduction

Holly Storkel:

Typically, your response to critique is going to start with an introduction. Now, this response to critique is typically one page. There’s some mechanisms where you might get a little more than that. But usually it’s one page for R03s and R01s. So you’ve got very limited space to respond to the critiques. But usually people will start with some kind of introduction. Here’s one example.

“We thank the reviewers for their careful reading of our initial submission.” You always thank the reviewers, even though you don’t like them and aren’t gratuitous.

Most people will say, and sometimes there’s some controversy here about doing this, but “the reviewers noted a number of strengths.” Sometimes people in this group will say, well, you’re wasting space noting the strengths. But others of us like to note the strengths so that you’re starting out on a positive note. And sort of saying, other people liked these things. You should probably like them too. And so here there’s an enumeration of the things that the reviewers liked. And you can just read that on your own.

And then, “We have retained these strengths and carefully addressed the weaknesses noted by the reviewers.” And so then, they’re going to go into now more a enumerated list of what those things are. They’re also noting here that the major changes are summarized below. So you don’t have to summarize every single change you made because, again, you only have one page. So what you want to do here is try to pick out the major things that you think really drove the scores away from the pay line. And that’s what you want to be addressing here are those major issues. You want to address minor issues too, but that’s just going to be in the revision of your grant in the longer text. You don’t need to highlight those.

William Yost:

Back to the do I list the strengths and take up space. That has a large answer in who is going to review that revision. It may not be — it’s some probability that no one who read your first grant will read your revision. NIH will try desperately to avoid that, but it may be unavoidable. So, yes the person who read your first grant probably, but maybe not, will remember who you are — remember the grant. And the strengths may not be necessary. But you can’t trust that that’s going to happen. So I definitely believe that you as succinctly, but as strongly as you possibly can, say what the previous people liked. Because otherwise, that message may never get to the people who are going to look at your revision.

Holly Storkel:

And again, if you’re writing that letter to counsel, after you’ve done that kind of general advocacy paragraph, this would be the next paragraph in that letter. That again might be highlighting what the reviewers liked. And then you’re going to give a response to some of the weaknesses.

So here’s another one. Same kind of thing. “Overall the reviewers were quite excited about this proposal,” so going into that. “The PI thanks the reviewers for their helpful comments.” And then, “the revised proposal contains” — and now this is one where they have done some significant preliminary studies. So you see they’ve added eight preliminary studies. So if you do — if you have collected additional pilot data and so on, that’s good to highlight in this introduction. And then they also enumerate some of the significant revisions to the different sections as well.

Here’s another example. “Uniform enthusiasm was expressed about” — and now they kind of go by the review criteria here — “the investigative team, the innovation, and the environment. At the same time, questions were raised about the approach. And the reviewers were split on the significance.” So here’s an example of the same kind of thing, but now aligning it a little more by sections and by those sub-scores or criterion scores.

You’ll also note that on most of these introductions, two examples, they’ve also noted in some way of how changes are denoted in the text. So one common thing is to put a vertical line in the margin where you have significant revision. You don’t usually line, you know, every single minor thing like where you had to cut words just to get into the page limit because you made some other major change. But if there are, you know, certain large paragraphs of text where you’ve made changes, you might use this vertical margin. Others will do some other font changes, which I don’t know like. Where they italicize all of the changes or underline all of the changes in the font. I personally find that distracting, but you do see that as well.

Response to Specific Critiques: Examples

Okay, so then after you’ve had that general introduction, now you’re going to go into specific, again, major concerns that the reviewers raised and how you went about addressing those. We’ve tried to kind of group them into some different sorts of themes.

Holly Storkel:

So here’s one that has to do with the investigative team where someone had recommended a statistical consultant. So the first line here is straight from what was written in the review. And then the response is what’s below that. And it’s basically that a statistician was added and then a little bit more detail. Not just, oh, we stuck a statistician on here because you wanted one. But now what are they doing? How have they added to the analysis approach? You’ll also note too that it points you to where in the text you’re going to see that elaboration. So it’s pointing you to BC2.4, whatever. That’s a particular header in the grant that the reviewer could then go to, to see the details of how the analysis approach was clarified or changed.

William Yost:

Yeah, and C 2 means critique .2, second critiquer.

Holly Storkel:

Now here’s a critique about aims. The original critique was it wasn’t clear how the experiments proposed for aim three relate to the theme of aim three. So basically the reviewer didn’t like aim three. And so then the response was to agree. And then to go ahead and delete that aim.

William Yost:

I think this is a particularly good way to respond. You just don’t want to say, “I was an idiot the first time. You’re right.” I think you need to find a way to say that, I still don’t think we made a big mistake, but I understand where you’re coming from. And I think to make things better, we’re going to move on and do something somewhat different. So I do think you don’t — as one of our rules we’ll see in a minute, you know, you just don’t say yes, yes, yes, yes, yes to the reviewer. You show that you really thought this through. And you have reasons for doing what you’re doing.

Holly Storkel:

Here’s one on significance. So the reviewer says it’s not clear that some of the literature that was cited can be applied to children. And so now here’s the response, that there’s studies of general learning in children that support the applicability of the concepts to this group. And then there’s also some feasibility data that further demonstrate the applicability of these concepts to this particular group. So trying to make that link more explicit for the reviewer.

William Yost:

And throughout all this the earlier discussion we have about not being vague now really comes into play.

Holly Storkel:

This is another feasibility/preliminary study kind of critique. It would be helpful if there were pilot data for such and such. And so now the revised proposal provides new pilot data. And it shows the technique is a feasible approach. So again, you always want to not just say, oh, we added pilot data. But go forward and be specific about what that shows or how that really meets the reviewer’s concern.

Here’s some issues with approach. There’s no discussion of statistical power. So we have that come up in a couple of reviews yesterday. And so here’s a response to that.

Now this is a good example of where maybe the reviewer didn’t exactly know what they were doing. And rather than the grant writer saying, “well, you’re an idiot,” they come back with kind of a tutorial in helping the reviewer understand the difficulty of what they’re asking the applicant to do. So they say that procedures for estimating power in this particular area are not established. And they go on to show, it’s not the same as just computing a power on a T test. This is a more sophisticated analysis technique. And so then they do say though how they have tried to figure out power. But then they move on to say, “but more convincing is our ability to detect multiple converging effects using the proposed experimental procedure.” So they give this tutorial to the critiquer. And they do try to do something to meet the concern and meet it in several different kinds of ways.

Here’s another issue with the approach. The generalizability to different levels of socioeconomic status is not considered. So they had some concerns here. And so then the response — and again, this is sort of the same kind of thing where maybe you don’t really know the literature here. But instead of saying that “you don’t know the literature,” I’m going to just tell you what you need to know. So even though we typically find no differences in SES between preschool age X and Y, we plan our analytical models to take SES into consideration. So, you know, here it’s no problem to add in the SES information and see if it contributes anything. We don’t really expect it does. And you’re kind of silly for thinking that maybe. But we just won’t say that. We’ll let it go.

Another approach issue. Concerns or confusion about the dependent variable. And this could either be what it is or how it’s measured or why there’s several different ones. What each one contributes. And so here’s the response. This was a case where they weren’t sure maybe what dependent variable went with what sort of hypothesis or why you need two. So this is clarified — use of accuracy versus reaction time depends basically whether you’re getting ceiling level performance or not. And again, the feasibility data demonstrates our ability to detect differences. So if they’re worried about how this dependent variable is going to operate in your particular population, having those feasibility data will help to deal with that concern very effectively.

Okay, so that’s it for some of the examples. Of course, there’s many more, but you get kind of the flavor of it that you’re not going into long detail in this introduction to resubmission. You just don’t have time. You want to succinctly summarize some of the major issues. And then succinctly summarize your approach or the way you handle those major issues. And then you’ll do a more detailed revision, obviously in the body of the grant.

So here’s some things though to keep in mind as you’re writing this response to review. You need to watch your language. There’s nothing to gain and everything to lose by confrontation. So again, calling the reviewer stupid, or suggesting that they’re ignorant of important scientific concepts and these types of things isn’t going to get you anywhere. You need to take the blame for the mistakes.

So for example, here’s one here on the slide. The PI didn’t state the material clearly in an accessible to review manner, whatever. Or even not say that at all. Just, you know, provide the information needed. Give the tutorial information so that the reviewer understands why you’re doing things in a particular way in your area. Adversarial language is self-defeating. Just let it go.

Revising the Proposal

Holly Storkel:

Okay, so the revision should be substantive. Even if your score was pretty close to the pay line, don’t just think, oh, I’m going to change a couple of things and that’ll be good. You know, really look at those summary statements and, you know, really take to heart the reviewer’s concerns. And make a good effort to revise everything.

Some sources of the material for the revision. We see now the summary statement. How you’re going to take those summary statement comments into account. But you need to think beyond the summary statements.
So one might be your own preliminary data. Maybe you’ve run some studies in the meantime. And you’ve realized somethings not going to work. Well, even though the reviewer didn’t question whether something was going to work, if you know it’s not going to work, you need to change it. So take into account whatever has gone on in your lab since that initial revision. And make sure that you incorporate whatever new knowledge you’ve come up with into that revision.

Likewise, there might be recent findings from others, and so you’re going to need to incorporate whatever articles have come out since you’ve submitted this application the first time. If you’ve added on colleagues or collaborators or if they have pursued their work in terms of other grant applications or other publications, whatever they’ve found out from doing that, that needs to be incorporated as well.

You might need to rethink your methods. You might need to refine or clarify various aspects of your aim, significance, innovation, or approach. And again, try to highlight your revisions, the major revisions, in the margin in some way.

And now here we kind of say what not to include. So, yes, you want to be comprehensive in your revision, but you want to be careful. So include new revised materials to satisfy the reviewer’s concerns. That’s good. But don’t include things that really aren’t relevant to the reviewer’s concerns. So, don’t go off on some sort of tangent or open up new criticisms. You might need to delete some things. But really think about the things that you’re adding. Anything new is open up to new critique. Did you want to add?

William Yost:

Be somewhat careful about adding your preliminary data or pilot data feasibility material. Be sure it really is supportive of what you’re proposing. Quite often, you know, you rush off, you do some pilot work in order to deal with the criticism. And it’s not real clear what’s going on. Look hard at that before you put it in because you may only be amplifying the reviewer’s concern.

Holly Storkel:

Don’t simply do what the reviewer suggests. So we have said, you know, kind of capitulate to the reviewer, but, you know, be careful with that. And I think Lisa Cunningham gave a great example the other day of how she responded to the reviewer’s concern. They wanted her to do a particular technique. But it was going to require a gajillion mice to do that. So if she hadn’t really thought it through and had just said, “Oh, yeah, fine. We’ll do technique X. Stick it in.” But hadn’t really thought it through and thought about the implications of that she would have actually made a problem for herself. Because if she had done that and then gone through the next stage of review, someone familiar with that technique would have likely said you’re going to need a gajillion mice for that. That’s just not feasible.

So do think through what the reviewers are telling you to do. Their suggestion, that they came up with when they were tired and cranky and had thought about it for only 30 seconds, may not be an actually good suggestion. And then don’t dismiss the reviewer’s concern because their solution was not very good or wasn’t optimal. You know, you need to think about, well, why are they suggesting that? What is really the underlying concern? And think about the best way, the optimal way, to address that concern. So don’t just do what they tell you. But really think through if that’s a good solution to whatever the concern is.

All right, get to work. A revision’s only going to receive an improved score if it has been significantly rewritten. You must spend a nontrivial amount of time writing and rewriting grants. It’s not any faster when you’re revising it than the first time you wrote it. You still have all of those same kinds of steps. Don’t short change it. You still need to do that internal review before the resubmission. You can’t just, you know, wait forever and then the week before the deadline suddenly bang out the revision. That’s not going to work. So you really need to set aside enough time to get the job done and get it done right. You don’t have a lot of time to wallow in self-pity and leave those summary statements in the drawer forever. You can leave them in the drawer for a couple of days, but then you need to pull them out and really get to work and make your plan B.

Your Proposal Isn’t Funded: Now What?

Holly Storkel:

Okay, so now what happens? This is maybe a little out of date. We had in here what happens if your second application is not funded. And now we do know that you can put that back in as a new submission. But I still think you really want to think about whether that’s in your best interest. And this is going to be another case where it’s good to talk to your program officer about that. To talk to your mentors and really think through if it’s going to be in your best interest to go in with a new submission. And even if you decide not to go in with a new submission to NIH, I mean, you still need to look at these summary statements and think about what it is that’s resonating with people in the field. What is not resonating with people in the field? And how you might change your approach to these things?

William Yost:

I would say in the next year, given this new redefinition of resubmission after an A1, it’s not that you should talk to your program officer, you have to talk to your program officer. This is new. It’s going to be interpreted. They’re going to be the frontline of the interpretation. You’re going to have to talk to them to figure out what to do next.

Holly Storkel:

And even if you go to a different agency, you might go back to the same reviewers. So it’s never good to ignore your reviewers. They’re going to pop up somewhere else. They’re going to pop up in your manuscript reviews. So, you know, be thinking about what they’re telling you. And really if one person in the field has this opinion, they’re probably not alone. So even if you don’t get that exact same reviewer, you’re probably going to get someone who has sort of similar theoretical background and emphases and so on. So you really can never ignore your summary statement. It is telling you something. And you’ve just got to figure out what that message is and how you can move forward with that particular message.

William Yost:

And if you get funded, read your summary statement. Study it, okay? Unless it’s your last grant then that might be it. But, you’re going to do this again and again and again and again. And this is a huge educational experience. Use it.

Holly Storkel:

And also, I mean, even when you’re funded, those reviewers might still have some good ideas that are going to help you out as you move this work forward.

Audience Comment:

Sometimes you don’t realize how smart they were until you’re two years into your study. And then you realize that. And this is the truth. They’re smart people. You know that part where the reviewer can respond to you and it doesn’t go into the reviews? Remember we saw that yesterday? Comments directly from the reviewer to you. If they do that for you, I think often they’re really saying, you know, if you did this thing, I would really look favorably on here. Or you should really go read this article. Or you should try this technique. I’d pay special attention to that. It’s kind of like a gift to you from that reviewer.

Holly Storkel:

And there’s some options on the slide here for what you might consider if your revision isn’t funded. You might consider reconceptualizing the grant and going to a different agency. You might do some preliminary work with internal funding or free labor like student projects. I mean, maybe you really just needed some more preliminary data and some more publications to set the foundation for what you’re doing. And maybe it’ll turn out that you have to abandon this particular project or this particular approach to your grand challenge question. But it doesn’t mean that you abandon your grand challenge question or you abandon your mission statement. You’re just abandoning this little piece of it so keep that in perspective.

So lots of people encounter the worst case scenario. You’re not alone. This is true at any time. We’ve already kind of talked about it. There’s many, many of us in the room, probably all of us really, have experienced some form of rejection with grant writing, if not a lot of rejection in grant writing. So, you’re not alone. This should not be something where you think should I leave my career. Is this really what I’m meant to do? You know, this person doesn’t like my projects so should I just throw the whole thing out the door? The answer is most likely no. But really don’t let it derail you in that way. We all come up with these roadblocks. But if you really are happy with the things you’re doing, if you’re really invested in the kinds of questions that you’re asking, you know, this is really just a small roadblock on your path to that. So don’t abandon the path just for that.

And here’s some parting words from Ed Conture. Which we should have mentioned him at the beginning. He was one of the people who had worked on this talk previously. And a lot of his notes and advice are part of this. But he says, “Above all listen to the summary statement. It may not be singing your song, but it is telling you a story. And you should be listening. There will be a test, namely, your next submission and review.”

William Yost:

Let me just take a second and talk about nondiscussed proposals. First of all, that word is appropriate. That means that your proposal was not discussed by the study section. It doesn’t mean it was not reviewed. The primary, secondary, tertiary reviewers all reviewed it. They submitted their reviews. They’re up there. What you will not see in your summary statement, if you were not discussed, is you will not see a panel summary or a study section summary because they didn’t discuss it. What you will see is the individual critiques — strengths and weaknesses of each one of the criteria of the review process. Now, those critiques have no benefit from discussion. So if one of the reviewers was off base and would have come back on base because of a discussion, that’s not going to happen. So the variability may be greater in terms of the accuracy of what you’re reading. So it becomes much more difficult. And your program officer isn’t going to be able to help to the same extent. There was no discussion. He or she can’t tell you, well, really and truly what the real path here you need to follow — because it didn’t happen. They don’t know that. Not that you shouldn’t talk to them and get more insights, but their ability to help you is now going to be significantly limited.

So it’s tough to read the summary statement, to read these three critiques, look at the strength and weaknesses and figure out what the hell am I supposed to do, okay? First of all, should I just give up? I didn’t get discussed. No. Do not give up. I gave the example the other day, and I’m not kidding. Someone did not get discussed. I helped this person — very difficult to figure out why. You could see the numbers, but the wording didn’t seem to match the numbers. And yet, this person got the second highest rated proposal the very next time she went in. What happened in this case? Well, the message was “something’s wrong.” Somehow at least one reviewer — you could tell by the score and maybe somewhat of another by this score — just didn’t like what you did. So it was sort of out of my area so we both knew the obvious. Get people in your area to read it, and redo it. And she did a pretty significant rewrite without really changing the basic experiments or data analysis or anything because that wasn’t clear that’s what she had to do. She just reconceptualized it. She used more positive valences. She did lots of things and boom.

So don’t read that summary statement and say I don’t know what to do. You have to change it. You have to go back and figure out how to get your message across in a better way. And you’ve given it your best shot. So you’re best opportunity to get new insights on how to do that is other people, a mentor or colleague, whatever — colleagues.

Audience Comment:

If you don’t have the benefit of a panel discussion, a summary of the discussion, what do you respond to? What if your critiques are vastly different?

William Yost:

Well, you can do a similar sort of thing. You hopefully could find that there were some consistencies among the strengths listed among the three reviewers. And say it appears that the three reviewers liked blah, blah, blah. It appears as if there was an issue about my productivity. That seemed to come up. And here’s what I’ve done about that. That’s always a really tough one. You’re not going to go out and publish ten articles in the time between the first submission and the second submission. So if your productivity is low, you’ve got an uphill battle. So the first message is don’t let your productivity go low. That’s the obvious input. But maybe that’s just where you’re at so you have to respond. If you’ve presented papers — anything that you can do with your productivity you can do.

Holly Storkel:

So the other thing though that I would add to that you do get your criterion scores, so that should help you. So if you see you got one on significance, and seven on approach, that’s telling you what drove your score down, that the approach was really the thing that sunk your application for that particular reviewer. So that’s going to tell you that that’s a major area. So you can use criterion scores, even if you’re not discussed, to sort of help figure out what did they like. Did they like the investigative team? Did they like the significance? But they didn’t like the approach.
And the other thing is reviewers should be giving you somewhat clear feedback on what they felt were major weaknesses versus minor weaknesses. So you’ll still get their critique that shows the strengths that the reviewer saw and the weaknesses that they saw. And you want and try and reconcile those strengths and weaknesses with whatever their criterion score was. So it should be in the way that they’ve written it should give you a little bit of a clue of whether they felt that a particular weakness was minor or more major. And so you need to try to kind of read between the lines on that to figure out what was the thing that really drove that seven on approach. It’s probably not every single weakness that’s listed. It’s probably three or four of them. And you need to try and figure out which ones.

Audience Comment:

I just want to point out that people who don’t get discussed, you shouldn’t feel like you’re missing out on this great big font of information. It’s a paragraph, four to five sentences that is going to touch on the highlights and low lights. It’s not even going to cover everything that’s in those individual reviews. So don’t feel like you have nothing to go on because you’re missing those four to five sentences. You’re not.

Audience Comment:

A few situations for you to talk about. So, you know, common situations would be when there are factual errors in the review. I mean, flat out factual, provable errors. The case where the reviewer’s scores don’t match the impact score at all. Which people, I think, often encounter. And then the case where the reviews are so terse, that you really don’t get a lot to respond to. And so you have these impact scores so, you know, where you have, you know, PI “good.” Or strengths “good,” weaknesses “no concerns.” PI — seven. I mean, that happens, right? And then, I think there are cases when the PI should give up on the particular idea. So I think it’s worth identifying what those are. So those three things.

William Yost:

I worked with Ed on doing this the first time. The second time I went back and looked at my career: 85% of the submissions I’ve made in grants got turned down at least once. Eight out of ten grants that I have submitted got turned down. And I’ve never not been funded. I’ve never not been funded by more than one agency. I don’t give up. I’m just going to not let them beat me down. Now every once in a while, I have a grant at the NSF that was a bomb. I actually got it. It didn’t work. I abandoned it quickly. So it’s not like I’m just stupid stubborn. Okay, I hope I’m smart stubborn.

So here’s an incident. A reviewer says — I’ll make it trivial, but the point is well made. “Nowhere in the proposal could I find what the level of the stimuli were going to be.” That’s really important. And you say, my god, it’s right in the general methods section. I can see it. It’s right there. You idiot. It’s on line fifteen. No, no, no. I’m sorry that I did not make it clear in the proposal as to the level. We did mention it in the general section. We have now tried to reiterate it at the key points in the proposal. It was 75 DBSBL — you idiot — okay? So, yeah, you find the common ground there.

The literature does not support your condescension. What? You know, since Helmholtz people have been pointing this out. You know, perhaps our literature review was not thorough enough given the page limitations. That’s a good copout, okay? We now have included and want to recognize the following literature. Unfortunately, it’s longer than the two pages that I could — no. But, you know, yeah, you know — but there is some substantial articles I suggest. For instance you read the reviews by blah, blah, blah or the chapter or the books by blah, blah, blah, whatever. You know, I’m exaggerating. But you need to find this midground between you’re the bad guy. You’re the one that made the mistake. It wasn’t clear. Whether you believe that or not, that’s I think the approach you should take. And then provide the information. And point out that actually you did in fact know what you were talking about or did present the information.

Holly Storkel:

And I think another thing you can do too is you might want to talk to your program officer about some of these issues as well. So again, it might depend on exactly what your score was. I mean, if it was not discussed, you know, any of these sort of, you know, inaccuracies or kind of bad reviewer behavior, you know, might not help you in any way. But it could help you if you were more on the bubble or something. If you were able to have a conversation with your program officer about, maybe this terse review. Or it looks like the person barely read the grant and yet gave it a score. That might do something for you because they would have some power to pull it up for counsel to look at it.

Audience Comment:

Before we move on, I would just want to add to that. That in your one page response — if a reviewer wants substantially more information about a particular area and you’ve got a six page grant and as Mario said, you use that whole six pages the first time. I often put in what I took out of the grant in order to put the more stuff in. And that way the new set of reviewers doesn’t say conversely, oh, you now didn’t pay enough attention to this. And so I’ll say in order to provide this additional information, I’ve reduced the section on blah, blah, blah.

William Yost:

Wow, that is so important. I can’t tell you the number of times in which I have had people tell me that they took out experiment one in order to meet the reviewers only to find on the revision that the next reviewer said you can’t do this without experiment one. I mean, basically. So letting them know why it wasn’t there is a great idea.

So a classic example of not enough information in a nondiscussed proposal, that I see all the time. In fact, I used the term once that I sometimes think the strengths and weaknesses are pull down menus. It’s not enough detail, period. How many times have you experienced people seen that? Now, the approach in experiment 3C did not provide enough detail. What detail? Now that could have come up in the discussion and might have been highlighted. And a reviewer might have gone back and changed and added more information. But if you don’t have the discussion, you’re not going to get that. All I can tell you in that case is there’s something in that experiment that a really knowledgeable person didn’t find. So you got to pour over that and try to figure out what that was. It’s really tough. There might be something in the overall context of where they’re coming from with the strength or weaknesses that point you into what that detail might have been. But it is tough. You do have a challenge.

Holly Storkel:

Other responses from folks about some of the questions that Chris threw out? How about when to abandon a project? So that was one of those I don’t think we’ve talked about at all. How do you know when you should just stop with that particular version of a project? Anybody have experiences?

William Yost:

Well, I mean, there’s some big clues. You know, if you’re getting two out of the three people in weaknesses just saying that I really do not think this experiment can advance the field. I really do not think it does what the investigator says it’s going to do. And you don’t find a real argument against that, well, these are smart people. These are experienced people. These are people who’ve reviewed lots of grants and everything. They’re probably not unique. That’s a clue at that point. Because lack of enthusiasm drives a huge amount of this. We’ve been talking on and on and on again about trying to get the reviewers enthusiastic. And if you don’t, that might be a sign to move on.

A positive way to move on is when you know, this particular approach doesn’t seem to be flying, but I have another way. I have another way to get at this. I didn’t think it was the best way, but I’m going to go out and throw it out there with my colleagues, my mentor, whomever, and see. So you’re staying with a theme. You’re going forward. That would have been particularly good in the old days when you couldn’t resubmit an A1. But I still think it’s applicable now.

Audience Comment:

I wanted to respond to Chris’ comment about these terse reviews — and especially about the investigator. So if you have a comment like “good” and then the investigator is rated a seven or a six or something like that, how to interpret that, I think, is that that’s a good investigator. And remember we heard that NIH is not interested in funding good research. They’re interested in funding excellent research. So in my experience if you see a comment like good, that probably means just that. It’s good. It’s not excellent. And that could reflect things like record of publication. It could reflect things like the quality of the institutional support. So I think it gets to things like publication record, quality, intellectual environment that you’re operating in. I think at that point, that’s a good discussion to have with your program officer. And to think about if I want to move this idea forward and I want to move my research program forward, how can I move that statement of good to a statement of excellent. So that’s how I would interpret that kind of comment. Final comments from Holly or Bill?

This was really, really helpful. Thank you so much for your expertise.