This article appeared in the SCC Liaison Volume 26, Number 3, August 2012. It was prepared by Duncan Murdoch and Dave Stephens (members of NSERC Evaluation Group for Mathematics and Statistics) and Louis-Paul Rivest (Chair of the SSC Research Committee) in consultation with NSERC.
The last four years have seen important changes to the way in which Discovery Grant (DG) applications to the Discovery Grant Program of NSERC are evaluated. A conference evaluation model is now used by NSERC. Mathematics and Statistics have been grouped in a single Evaluation Group (EG-1508). NSERC reinforced its guidelines for the eligibility of applications related to health and social sciences, referred in this article as the ‘NSE rule’. The article of Dave Stephens on pages 52-55 of the August 2011 issue of Liaison, abbreviated Stephens (2011), discussed the evaluation of Statistics applications in this new environment. However, subsequent to the publication of that article, the NSE rule (see section 2(b) below) was strengthened in the Peer Review Manual for 2011-2012.
This article revisits some of these issues in the light of the results of the 2012 competition. It has a question and answer format and touches important aspects of the evaluation of Statistics applications to the NSERC Discovery Grant Program. The first section deals specifically with first time applicants while the second section is more general.
1. EARLY CAREER RESEARCHERS (ECR)
Under which conditions is an applicant classified as an ECR? Is it done by NSERC staff prior to the competition? Can an applicant be classified as an ECR for several years?
There are two main categories of applicants to NSERC Discovery Grants (DG) program: Early Career Researchers (ECR) or Established Researchers (ER). Upon receipt of DG applications, applicants are classified as ECR or ER by NSERC staff according to the information provided in the Forms 100 and 101. New applicants who are applying for the first time and applicants who have not received an award in their previous attempts are considered as ECR if they meet the following two conditions: (1) they are within two years of the start date of their first eligible position at the university and (2) they have no prior academic or non-academic independent research experience. All other applicants are considered as ER. With respect to the first condition, the 2-year window refers to two years from July of the year of the application deadline. For example, applicants who have not been successful in their previous DG submissions or applicants who will submit their first DG application in November 2012 will be considered as ECR if their first eligible position started on or after July 1, 2010.
How is the evaluation of an ECR different from that of an experienced researcher (ER)?
As per NSERC guidelines on the review of DG applications (Peer review manual: http://www.nserc-crsng.gc.ca/_doc/Reviewers-Examinateurs/CompleteManual-ManualEvalComplet_eng.pdf), all applications are evaluated against the same expectations for the three selection criteria. The assessment of Contributions to Training of Highly Qualified Personnel (HQP) includes both the record (past) and the plans for future training for ECRs and for ERs. It is recognized that the training record of ECRs can be limited compared to that of ERs. In light of this, ECRs are not rated as Insufficient solely due to not having a training record. At the same time, it is unacceptable for an ER not to have a training record. Both ECRs and ERs must present a comprehensive training plan for the future. In the assessment of Excellence of the Researcher, ECRs are assessed on the same terms as ERs and reviewers consider impact and contributions over a six year window. The quality and impact of the evidence presented for similarly rated researchers should be comparable, regardless of career stage. It is understood that most ECRs will not have a full six years of experience as independent researchers. However, Evaluation Group members rely on their knowledge and experience to compare quality and impact for ERs and ECRs. For most ECRs, the independent researcher clock can be thought to start at completion of PhD. ECRs who continue to collaborate with previous supervisors, or who carry out research as part of a group, should clearly define their contributions to the collaborative work. In the assessment of Merit of the proposal, ERs and ECRs are treated identically. If appropriate, ECRs can point out connections with their past work to demonstrate the feasibility of their proposed research.
Some recent PhDs might not have any accepted publication when they submit their first application. Can they still be funded?
Yes. In our discipline, it is still more common for publications only to be accepted after the PhD is completed. Therefore ECRs who have only recently completed their PhD will most likely have no accepted papers. In this case, assessment is usually predominantly based on the quality of the four contributions submitted, and a general assessment of the applicant’s Excellence from other information included in the application (such as the information given in the Five Most Important Contributions section).
Do you have any tips to help an ECR to write a successful application?
When you write your first grant application is usually not the right time to do drastic changes in your research program. Your knowledge of the proposed research area will be assessed when evaluating your excellence as a researcher and the merit of your proposal. Thus writing a project in a new area might be risky. Mentoring is a key to success and seeking comments and suggestions from colleagues with a successful NSERC track record is highly advisable. NSERC staff can also be contacted for information or clarifications on instructions and policies related to Discovery Grants applications.
APPLYING FOR A DISCOVERY GRANT
How is NSERC using this form? Does my research project need to be finalized when this form is submitted?
A proposal is assigned to an Evaluation Group on the basis of Form 180. External reviewers for an application are selected using the info on Form 180. If an application is multi-disciplinary, this has to be emphasized in this form as well. If appropriate the major field of application of the research, besides Statistics, must be identified through the selection of research codes from the most appropriate Evaluation Group. Thus the research project does not need to be finalized however some key decisions for the evaluation of an application are made on the basis of Form 180 so it must be an accurate summary of the forthcoming complete application.
The NSE RULE:
When did NSERC reinforce its guidelines on eligibility of applications related to health and social sciences research? How has this changed the way in which statistics applications are evaluated?
NSE is an abbreviation for natural science and engineering. As part of its mandate, NSERC supports research programs addressing research challenges, and clearly advancing knowledge and training in the natural sciences and engineering (NSE). While the mandate has remained the same over time, NSERC reinforced its guidelines for the eligibility of applications related to health and social sciences, with the implementation of the revised peer review system (i.e., three years ago). The NSE rule means that NSERC only supports research and training in the natural sciences and in engineering. Thus a research program in the health or the social sciences, without any NSE connection, will be rejected from the DG competition. All areas of Statistics are covered by the NSE, as defined by NSERC, see http://www.nserc-crsng.gc.ca/Professors-Professeurs/Grants-Subs/DGPList-PSDListe_eng.asp#1508. Thus the development of a new statistical methodology is in the NSE, irrespective of the potential areas of application. The impact of this research in statistics and in other NSE subject matter areas contributes to the excellence of an applicant and of his proposal. If the impact of the research is on non-NSE areas, it can still be highlighted to demonstrate the significance of the larger research program (see Stephens (2011)). However, the ratings for the three criteria will be based on the NSE-related research and training. The NSE issue mainly arises with collaborative research, e.g. the innovative application of known statistical techniques that lead to advances in a subject matter area. The NSE rule means that collaborative research is valued as long as it involves the application of statistics in the natural sciences or in engineering. Researchers should outline how their proposal impacts and contributes to areas outside of Statistics. In practice, the importance of collaborative research is challenging to demonstrate; the case that it is truly innovative and in the NSE scope has to be made in the application.
NSE RULE APPLICATION:
Does the NSE rule apply to the three criteria used to evaluate an application (Excellence of the Researcher, Merit of the Proposal, and the Training of HQP)?
Yes it does. Stephens (2011) provides a detailed discussion of how the NSE rule plays a role for the first two criteria. For HQP this means that funds cannot be requested to support graduate students working outside the natural sciences and engineering. This would for instance imply that support of medical students learning biostatistics would normally not be admissible. This does not rule out students doing biostatistics. Indeed the NSERC URL given above has subjects such as quantitative methods in health sciences, epidemiological methods, and statistical methods for health policy research within the NSE. Thus while basic training in biostatistics is not admissible, working on the statistical methodology used in the health sciences is in scope.
What are the points that statisticians need to emphasize when preparing the HQP training component of the project?
If appropriate the context of your own university and the way it influences the training that you are providing can be explained. Programs of several types are now available in Statistics, especially at the Master level, and it is important to specify how the students under your supervision are trained. Are they writing a research thesis? Are they involved in short term data analysis? The role of co-supervisors, if any, has to be laid out clearly. Training in a multi-disciplinary environment involving researchers in subject matter areas is viewed positively. It is to be noted that while the ‘proposed training under the DG plan must be in the NSE domain, HQP training in other domains (e.g., heath, social sciences) may be considered as part of the demonstrated commitment of the applicant to training and the quality of the training environment, particularly when there have been opportunities for training synergy or interdisciplinary training’ (PRM, Section 220.127.116.11.1). Applicants must clearly explain the research context and training environment when appropriate. The scientific challenges involved have to be presented; routine data management is not in the NSE scope. The training plan provided in the application should be a concrete description of the involvement of trainees in the proposed research program including the specific project(s) or aspects of the overall program that they will work on and the objectives of their research. How will the work that students do under your supervision contribute to their future success?
What changes are planned for the 2012-2013 competition?
A change for the coming competition is that deadlines will be strictly enforced. We are unaware of any changes to the evaluation procedures. However, applicants are strongly urged to read and pay close attention to the 2012-2013 Peer Review Manual when it is published on the NSERC web site. If any important changes do appear, we will attempt to point them out via the d-ssc mailing list.
Where can I get additional information on the preparation of my application?
The Discovery Grants Information Center (http://www.nserc-crsng.gc.ca/Professors-Professeurs/DGIC-CISD_eng.asp) comprises links to important documents that should be consulted in the preparation of a Discovery Grants application:
On-line Services – which includes the on-line application system and guidelines for completing forms 180, 101 and 100
List of Evaluation Groups and Research Topics:
Discovery Grants Program – which provides a description of the program and selection criteria
Evaluation indicators – which provides the main elements assessed for each of the three selection criteria
Peer Review Manual – which provides detail on the peer review process
Programs Selection Committees – which provides the names of Evaluation Group members
FAQ: Discovery Grants Competition
Contact NSERC Staff.
This article was prepared by: