Political scientists and economists claim to understand the mechanisms of economic sanctions as a tool for foreign policy and assert to have provided convincing statistical evidence for their theories. In this contribution we argue that their theories and evidence are significantly influenced by publication bias. What does this mean for our understanding of the history of economic sanctions? Further what are the implications for the future application of the sanction instrument?
Economic sanctions have become a much more important instrument in the international arena since the 1990s. Indeed, 1990 is a watershed year. It is the year that marks the fall of the Soviet Union and thereby the end of the superpower conflict that complicated United Nations security sanctions. It is the year of the sanctions against Saddam Hussain’s Iraq. Sanctions that stood out, because they were imposed within a week, truly multilateral (the first case, for example, in which Switzerland participated), and covered both trade and financial flows. Also, the imposition could and was monitored closely by means of a blockade by the navy. The sanctions against Iraq are the show case where all economic conditions for success were met. And, yet they failed.
It may of course have been the case that the goal of the sanctions was impossible to meet because giving in would mean the end (and actually the death) of the regime. So, research continued to investigate what factors are important for sanctions to be successful. We have investigated three determinants of sanctions’ success by means of a meta-analysis of thirty-six studies that were published in the period between 1985 – 2018 (most of these studies appeared in peer-reviewed journals). These determinants are trade linkage (economic sanctions do not make much sense if the sanction target and sanction sender do not trade), sanction duration (sanctions probably need to be quick and unexpected to have a maximum impact) and prior relations between sender and target (sanctions may work better against friends than foes).
Based on this, the first conclusion is that the research findings are not converging. This is illustrated in Figure 1 indicating the consensus of the literature until the turn of the millennium. It shows that the trade linkage is a determinant of sanction success, but after say 2005 the literature increasingly disagrees. We find the same pattern for both sanctions’ duration and prior relations.
Figure 1 Reported coefficients for trade linkage in 32 studies (a positive coefficient means: the study finds that more trade linkage is positively correlated with sanction success)
In our analysis, we focussed on publication bias as a potential source of the heterogeneity of the research. Publication bias may be introduced by the science publication industry. Editors and referees force scientists to look for significance and authors in a publish-or-perish-environment may be tempted to report the regression with the significant coefficient although “the other million of regressions” that they run were insignificant. Authors may also self-select because they are, for example, convinced of the need to use sanctions and therefore, want to show that sanctions work. Or they are ideologically inclined to argue against limiting international trade flows. We use meta-analysis to measure the extent of bias with respect to the findings for trade linkage, duration and prior relations. Consistently, we find a significant bias, so strong that the underlying average effect actually is zero.
Our findings fit in the so-called replication crisis that is a general and disturbing trend in science. Scientifically, this is disturbing because it hurts the credibility and reliability of our knowledge production in the hearth.
In the end, it poses a clear challenge for policy makers inclined to design evidence-based policies and an imperative question then becomes – what evidence to consider?
This contribution is based on a research project for MA students at the Institute of Social Studies for which Alemayehu Rita, Gabriela Benalcazar Jativa and Patrick Kimararungu received the Award for the best Research Paper Project 2018. It is the second article in a series related to the 19th Jan Tinbergen European Peace Science Conference that was hosted by the ISS from June 24th to 26th June 2019. Read the first article here.
Image Credit: IsaacMao on Flickr
About the authors:
Peter van Bergeijk (www.petervanbergeijk.org) is Professor of International Economics and Macroeconomics at the ISS.
Binyam Afewerk Demena is a Postdoctoral Researcher at the International Institute of Social Studies (ISS), Erasmus University Rotterdam (EUR). His PhD and MA in Development Economics from the ISS, EUR. His research interests relate to primary empirical research and meta-analysis in development economics, international economics, fishery economics, health economics and other related issues. He has published articles in Applied Economics, Journal of Economic Surveys, Journal of International Trade and Economic Development, and Third World Quarterly among others.
Alemayehu Sisay Reta is a research assistant at the ISS. His research and professional experiences are in the areas of Development, Economics, Economic sanction, Project Feasibility Studies, Business and Economic analysis, Monitoring and Impact Evaluation and Program Development. He has an MA graduate in Economics of Development, 2018, International Institute of Social Studies of Erasmus University Rotterdam.
Gabriela Benalcazar Jativa is an MA graduate in Economics of Development, 2018, International Institute of Social Studies of Erasmus University Rotterdam. She specializes in quality assurance but she is also interested in researching issues concerning local development and meta-analysis.
Patrick Bitandaro Kimararungu is an MA graduate in Economics of Development, 2018, International Institute of Social Studies of Erasmus University Rotterdam. Patrick’s MA thesis focused on Meta-Analysis on Economic Sanctions. His interests lie in the continuation of economic policy research especially in developing countries.
