Consistency of Peer Review Results and Analysis of Influencing Factors——An Empirical Study Based on the Evaluation of National Key R&D Projects
Zhang Jinqiannan1, Ma Zongwen2,3
1. International Science and Technology Cooperation Center, MOST, Beijing 100036, China; 2. National Science and Technology Infrastructure Center, Beijing 100038, China; 3. China Science and Technology Exchange Center, Beijing 100045, China
Abstract：Peer review is a common means used by countries to select and fund scientific and technological projects.Four batches of review data of the National Key R&D Program were selected as samples,including 2919 scoring data of 288 projects by 375 peer reviewers.The consistency of peer review and its influencing factors were quantitatively analyzed with the methods of Cronbach α coefficient,binary coefficient,and coefficient of variation.The main results showed that the consistency of peer review results was generally high,the number of peer experts,the review method,the number of projects in the group and the differences of projects all affected the consistency of the review,and the average consistency of the review was increased by 2.25% for each additional expert in the defense review.For a review with 5 experts,the consistency of the review results in the defense review is improved by about 10% averagely compared to the communication review.The defense review for the same batch of projects was increased by about 15% compared with the communication review.With the decrease of the number of projects in the group or the increase of differences between the projects in the group,the consistency of the review results increases; the consistency of the review results of the next round of review or project approval is higher than the average result of the whole group of projects,and the improvement effect is more obvious in the defense review.Finally,this paper summarizes the influence of organization on the consistency of the review and discusses the strategies to improve the consistency of peer review.
张金倩楠, 马宗文. 同行评议结果一致性及影响因素分析——基于国家重点研发计划项目评审的实证研究[J]. 中国科技论坛, 2025(5): 21-29.
Zhang Jinqiannan, Ma Zongwen. Consistency of Peer Review Results and Analysis of Influencing Factors——An Empirical Study Based on the Evaluation of National Key R&D Projects. , 2025(5): 21-29.
[1] LI D,AGHA L.Big names or big ideas:Do peer-review panels select the best science proposals?[J].Science,2015,348(6233):434-8. [2] 财政部,科技部.财政部、科技部关于印发《国家重点研发计划资金管理办法》的通知[J].中华人民共和国国务院公报,2021(35):74-82. [3] DEMICHELI V,PIETRANTONJ C D.Peer review for improving the quality of grant applications[J].Cochrane Database of Systematic Reviews,2003,18(2):MR000003. [4] COLE S,SIMON G.Chance and consensus in peer review[J].Science,1981,214(4523):881-886. [5] PIER E L,BRAUER M,FILUT A,et al.Low agreement among reviewers evaluating the same NIH grant applications[J].Proceedings of the National Academy of Sciences,2018,115(12):2952-2957. [6] JERRIM J,VRIES R D.Are peer-reviews of grant proposals reliable? An analysis of Economic and Social Research Council(ESRC)funding applications[J].The Social Science Journal,2023,60(1):91-109. [7] JONATHAN S,FRAMPTON G K,KAREN P,et al.Peer review of health research funding proposals:A systematic map and systematic review of innovations for effectiveness and efficiency[J].Plos One,2018,13(5):e0196914. [8] VALLÉE-TOURANGEAU G,WHEELOCK A,VANDREVALA T,et al.Peer reviewers' dilemmas:A qualitative exploration of decisional conflict in the evaluation of grant applications in the medical humanities and social sciences[J].Humanities and Social Sciences Communications,2022,9(1):s41599-022-01050-6. [9] GALLO S A,SULLIVAN J H,GLISSON S R.The influence of peer reviewer expertise on the evaluation of research funding applications[J].Plos One,2016,11(10):e0165147. [10] REINHART M.Peer review of grant applications in biology and medicine.Reliability,fairness,and validity[J].Scientometrics,2009,81(3):789-809. [11] 江虎军,徐岩英,朱蔚彤,等.同行评议制度的公正性与局限性[J].中国科学基金,2019,33(4):403-406. [12] 陈光,陈凯华,龚旭,等.优化科学基金同行评议机制的思考[J].中国科学院院刊,2021,36(12):1427-1433. [13] 贺祖斌.高等教育评价的元评价及其量化分析模型[J].教育科学,2001,17(3):56-58. [14] 汪建,王裴裴,丁俊.科技项目专家评审的元评价综合模型研究[J].科研管理,2020,41(2):183-192. [15] 罗军,陈之瑶,莎薇,等.科技项目评审专家工作质量元评价体系及应用研究:以广东省重点领域研发项目为例[J].科技管理研究,2021,41(13):65-70. [16] 张立军,彭浩.科技成果评价的信度分析及模型优化[J].管理现代化,2016,36(6):92-94. [17] 蔡文学,温旖旎,郑顺潮,等.基于偏差度权重的科技项目评审专家绩效评价方法研究[J].科技管理研究,2012,32(15):75-78,83. [18] 杨健安,刘传斌,余乐安.基于相对逆序数的评审专家可信度评价方法[J].统计与决策,2022,38(3):184-188. [19] 李强,孟宪飞.科技奖励评选中同行评议的信度问题研究[J].科学学研究,2023,41(11):1948-1957,1966. [20] 赵亚娟.专家群评价结果可信度分析与检验[J].中国科学技术大学学报,2016,46(2):165-172. [21] 余红梅,罗艳虹,萨建,等.组内相关系数及其软件实现[J].中国卫生统计,2011,28(5):497-500. [22] 徐林生,王执铨,戴跃伟.评审专家可信度评价模型及应用[J].南京理工大学学报(自然科学版),2010,34(1):30-33. [23] 陈伟.科学评估中不可忽略的要素:评估专家的作用、信度及素质[J].中国高等教育评估,2003(1):52-54.
2025, 
