The following is a digested version of the presentation video transcript.
Impact Score
The impact score: The reviewers provide an overall impact score that reflects their whole assessment of the value of the grant. It reflects their assessment of the likelihood for the project to exert a sustained, powerful influence on the research field(s) involved, in consideration of the following five scored review criteria, and additional review criteria (e.g., new investigator). An application does not need to be strong in all categories to be judged likely to have major scientific impact. As Elena pointed out, it’s not an average of all those scores.
A lot of times, in my experience, it’s the Approach that pulls something down. Reviewers recognize that the candidate is a good one, the mentor is a good one, or the principal investigator has a track record, but there are just things wrong with the Approach and that pulls the overall value of the work as it’s presented down.
Common Weaknesses: Significance
Some common weaknesses that may show up in your final summary statement for the Significance. Very important: the grant might lack or have a weak theoretical framework. You need something to pull the whole Approach together
Hypothesis-driven aims are very important. As is the motivation. Sometimes you end up saying, “Why are they doing this? What is the motivation?” That could be tied to a specific aim or to the entire approach.
Integration is really hard to master. You start with a big idea, and you’ve got several specific aims that come out of that, but they don’t seem to be tied together. They might not individually be motivated, or they might not be tied together well.
And sometimes you do get grants that aren’t tied well to human health. And that’s important especially for the National Institutes of Health.
Common Weaknesses: Environment
The environment. There are a couple of things that could go wrong there. The institutional commitment could be weak. It’s important for you to make sure that’s in place. For example, if you get this grant, are you going to have a release from your responsibilities. This is really important for the people writing early grants, showing that you’re not going to be stuck teaching a lot. It might even require a letter from your chair or something that you’re going to have released time to do this work.
Poor infrastructure. This is not your fault, this is where you are. To take care of something like that, you might establish a collaboration with somebody from a stronger institution. We have to get jobs somewhere. Sometimes we don’t get jobs in institutions that have everything going for them. If you want to establish a research program, you might have to do it by collaborating with someone else.
The community itself is weak. There’s a lack of a scientific community to provide consultation. This is particularly important if you’re in an area that’s new, so you don’t have a lot of resources out there in terms of empirical and theoretical support.
Common Weaknesses: Investigator
The investigator. One of the difficulties when you get very busy is keeping up with your publications. This is important at every stage of your career. When you’re just starting out, you have to have a couple of publications out there. Make sure that you’re the first author on some of them, if that’s possible. It depends on what stage you’re at. As you go along, too, you have to keep up with publications. You can’t just do posters at conferences and never write them up.
One reason is that people forget what you’re doing. It’s very important to get your work out there and published. That takes time, so you have to make time for that.
For postdocs — for any training grants — the mentor is evaluated as much as the trainee, so that’s very important to consider. I don’t think that was talked about too much. I did review training grants for a while, and it’s very important to have a mentor that is involved. But not overly involved — there’s a balance.
The PI is not trained well, at least specifically to the area that they are applying to or attempting to do a project in. If you want to go into another area, or even if you’re just starting out in your career, you need to make sure you have collaborators or mentors for whatever stage you’re at that have that expertise.
And if you do have collaborators, it’s very important that you have a plan for collaboration, for communicating with each other. In this day and age, that’s not hard. Even if you’re cross-country, you might one or two trips a year to meet face-to-face, but you’ve got all the technology to have online meetings and so forth. You build those in, and that’s what reviewers like to hear.
The team itself that you assemble has to also be accomplished in the area that you’re trying to do research in.
Common Weaknesses: Innovation
Innovation. Techniques may not be current or appropriate.
The grant may be too incremental — so, it’s not very innovative it’s just another step. Sometimes the next step has to be accomplished, but you can’t claim that it’s innovative.
It’s just not interesting. This does happen. I get a grant once in a while, and I’m just going, I’m just not excited by this.” But that’s why there’s three people to review — because it might be exciting to someone else.
Does it advance an important scientific area? That’s important. Maybe you won’t change the world, but you do need to move your field forward.
Common Weaknesses: Approach
Being overly ambitious. This one is big, especially at the beginning. But I think it stays with you because everybody has so much they want to do. You should look at the grant mechanism and the funding. Obviously, if you’re doing an R03, you’re not going to propose something that’s going to take five years. You have to think about how long is it going to take, how many people do I really need, and how much funding can I get from that. It’s a very common mistake that young investigators make — they propose way too much for the time and budget that they have.
Someone said that they are in love with their sentences — I’m in love with whole pages. I keep a file, I put them somewhere else. “I’m going to use this later.” I say, but I never do.
Feasibility of sample size, population, techniques not demonstrated. This is another problem.
And sample size is not supported by a power analysis. This is particularly hard in my field, because I work with people with aphasia and you just don’t get 50 people with aphasia just like that. But we work with that somehow. And look for reasonable effect sizes.
Example Reviewer Comments
Significance Score of 2
Here are some examples of each section and the scores they got. Remember that it’s from 1 to 9.
Here are the sort of comments associated with a 2 in Significance. “This proposal’s topic has significant implications for the nation’s health.” This is the comment of the reviewer. “The PI plans to explore two promising mechanisms implicated in the cause of X and, as such, findings from this project could meaningfully advance our understanding of X.”
That’s the kind of comment you hope to get, and that’s kind of the goal you should have in terms of establishing significance.
Weaknesses — “The Specific Aims section does not provide sufficient rationale for the study of X, its importance for the nation’s health…” and so on.
Sometimes when I read grants, I can see the significance, but the grant writer has not put it in there. So you say something like that so hopefully they understand that they didn’t miss it, but they didn’t write it down.
Investigator Score of 1
Investigator. “The investigator has been productive in recent years.” And the publications are relevant to the project you are applying for grant funds to support.
A strong research team is in place. That’s a plus. They investigator knows the expertise they need and they’ve gone out and found it. “Prominent world-class consultants.”
The investigator — if this is a re-write — has been responsive to reviews. There’s actually a section where you can comment specifically whether the person has been responsive to reviews. And reviewers have different opinions about that.
And in this case, this is a 1, so no apparent weaknesses.
Innovation Score of 3
Innovation. The project is innovative in that it proposes to study the pathogenesis of X.
And this is a weakness, that although the PI makes apparent his or her expertise and ability to use this approach there are concerns whether the findings will go beyond versus essentially confirm the findings.
That’s not very attractive just confirmatory research.
Approach Score of 2
The Approach. There’s only one strength here. Usually you will see many more examples. Those of you who have gotten feedback know sometimes you get — they switched from paragraph format to bullets, but some people like me have trouble with bullets and each of my bullets is a small paragraph.
The PI has seemingly improved his/her approach making it more suitable for addressing his/her hypothesis and research questions.
But the weakness: The format of the Significance section makes it quite difficult to appreciate the contribution these studies make to the application.
That’s important. You don’t have as much room for the Significance section, but you still somehow have to concisely make it clear that this work is really relevant. And it’s not easy.
Environment Score of 1
Environment, personnel and consultants are excellent. No weaknesses apparent.
So, further comments for the PI — this is a good place fore the reviewer to say, although I’ve made a lot of seemingly negative comments, I like what you’re doing and I encourage you to come back and follow what we recommend. That happens.
For new investigators, often reviewers will use this section if their writing is a little off track with what the section is supposed to include. You might get a little direction on what to focus on. Or if the biosketch isn’t in compliance, I’ll say, double-check the instructions for the biosketch or something like that. Often that section is used for stuff that isn’t scored, but would make their grant better.
The Discussion
During the discussion, the preliminary impact scores are given.
The primary reviewer speaks, followed by the secondary and tertiary. Then it’s discussed.
Final Scores
Final scores are given. We’ve discussed all that.
Though, I should mention the concept of “scoring outside the range” when we talk about final scores. Sometimes the panel chair says, “Gosh, there’s an awful lot of negative comments, yet your scores are 2 or 3. What’s going on?” And then other people on the panel will say, “Yes. This sounds like a 5 or 6 to me.”
For example, if three people give scores of 2, 3, and 4 — so you have a range from 2 to 4 — typically everybody would just vote within that range. But if there’s a strong feeling from members of the panel that scores should be lower than that, it’s called, “scoring outside the range.”
So the Chair or the SRO would say, “If anybody is voting outside the range, please acknowledge this and send me an email as to why you are doing it.” That doesn’t happen a lot, but it happens more than you might think.
This is a really fair system. I know you sometimes get back these summary statements that are frightful. But the process that goes on to come up with these is by and large a very fair one. And the reviewers are invested, as we’ve already said. They spend a lot of time going through the grant. And the discussions are sincere and well-intended. We’re all in your court.
So after the discussion is all over with the scoring, they discuss Human Subjects — this is important, to discuss the minority representation and the woman representation. Sometimes you do have issues where people give you the demographics of the state of Massachusetts, but they’re really working in Boston. So you just say, “That doesn’t make sense.” Things like that come up now and then. So you’re allowed to comment on those. But that’s done after the scientific review is completed.
After the final scores come, they get multiplied by 10, and your final score is some average of all those scores. So the range of scores in the final application is from 10 to 90.
Grants Not Discussed
If the grant is not discussed, you have that little squiggle there instead of a number. Which is kind of sad. But you do get the written reviews.
And, again, people go back and after the discussion they can review their reviews. They often do. They have several days before the SRO takes all the reviews and turns them into summary statements. So the reviewers have a chance to benefit from the discussion themselves and change things where they’ve said something too extreme, or not extreme enough.
Other Categories
There are some non-scored categories. Not recommended for review. That’s happened occasionally. If the application has no seemingly scientific merit at all.
Deferred for lack of sufficient information.
Abstention is rare.
Conflict — during the review process, people who are in conflict are asked to leave the room.
Summary Statement
In the summary statement, you get a face page with the overall impact score. A summary of the discussion. Someone writes an abstract, then there are three critiques covering all the categories that we’ve mentioned. And then a statement about human subjects.
After the review
After the review, the overall impact score is released and appears on the eRA website. You have your own page by that time. And it’s exciting to go in there and see how progress is happening with your grant. Sometimes it’s exciting. Sometimes it’s really scary.
Many are probably already familiar with this, but all the scores get put into a pool. R01s are not, for example, mixed up with R03s, so there’s some subdivision.
Then you get a percentile rank. And this is the real important one that is tied to the payline. This varies depending on how many proposals are submitted, how good they are, how calibrated the study sections are. These all figure into it. If there are a lot of 1s to 3s, then even if you got a 3, for example, it’s not as good as the thousands of 2s and 1s that are there.
The Summary Statement is released and appears on your Commons site.
And then what happens? You have to wait to get the review score. The scores come very quickly now. You used to have to wait — but now it’s within days, and you can even look ahead of time before you get that email saying your scores are there.
For those of you who are interested, here are links to the two websites with more information about the Grant Review process:
