Tag Archives fieldwork

‘I cannot understand your question’: challenges and opportunities of including persons with disabilities in participatory evaluation

By Posted on
Colourful Hands

Participatory evaluation has been praised for engaging vulnerable groups such as persons with disabilities (PwD). However, the inclusion of this group can be challenging and even self-defeating if carried out incorrectly. Despite the challenges, evaluators and researchers can follow some strategies to make the evaluation process with PwD as inclusive as possible.

Disability and participatory methods

For a long time, persons with disabilities (PwDs) were socially ostracized and confined to special schools and health centers. Growing pressure from disability rights organizations made possible a shift from an individual and biological view of disability towards a social and inclusive model that focuses on the interaction between individual impairments and social and environmental barriers (Shakespeare, 2006). Since then, international progress has been made to recognize the right of PwDs as full and contributing members of society; the formation of the 2006 UN Convention on the Rights of Persons with Disabilities is an example of a step in the right direction on this front.

In previous decades there has been a shift in research and evaluation methodologies in academia as well. Criticism of the ineffectiveness of the positivist paradigm to include vulnerable groups in research has led to the rise of participatory approaches in which PwDs and other marginalized groups play an important role in shaping research agendas and outcomes (Parry et al., 2001). The alternative bottom-up methodologies became known for challenging power relations and giving voice to marginalized groups, including PwDs (Chambers, 1994).

As a result, participatory methods have been crucial for engaging PwDs in more active roles in the processes of monitoring and evaluation (M&E) and not only as simple research subjects. For instance, many evaluations now involve PwDs organizations in the role of advisers where they can choose data collection instruments (Robinson et al., 2014) and use their expertise to interpret results and provide feedback (Olshanska et al., 2016). Increased participation has been praised for improving the validity and general outcomes of the evaluations (Brandon, 1998).

The challenges of inclusion

Despite recent achievements, many challenges lie ahead for greater inclusive participation of PwDs in program evaluations. One of the most overlooked aspects is the design of inclusive evaluation instruments (surveys, focus groups): evaluators tend to regard PwDs as a homogeneous group. Therefore, the instruments fail to take into consideration the diversity of disability, especially in terms of communication styles.

This creates an under-representation of the least advantaged within the target group. A study of 31 peer-reviewed articles in ten top-ranking evaluation journals shows that people with intellectual and development disabilities were less likely to participate in evaluation processes than people with any other type of disability (Jacobson et al., 2012). Even if they do participate, their answers in most of the cases might be biased or incomplete (Ware, 2004) since they communicate differently than their peers or experience psychological barriers such as low self-esteem.

Conducting evaluation activities in venues with physical barriers or far from the beneficiaries’ houses can hinder the participation of people with a physical disability. Therefore, ineffective M&E planning and instruments could not only bias the results, but also could end up creating negative unintended consequences such as exclusion and disempowerment. However, even if considering the linguistic and cognitive heterogeneity, what are the best alternatives to engage PwDs in participatory evaluation processes? Is inclusive participatory evaluation more time consuming?

Lessons learned: How to overcome the obstacles?

From my experience working with women with disabilities in Nicaragua[1], when it comes to disability, there are no one-size-fits-all solutions. Nonetheless, there are low-cost alternatives that can improve the overall level of participation. Here are some things to keep in mind:

Learn about your target group. An overview of the type of disability and some social variables is crucial to balance participants in focus groups, disaggregate data by categories, and prepare in advance for special requirements (e.g. the use of a sign interpreter, ramps for wheelchairs). It is also key to better understand power dynamics within the group. For instance, women face more discrimination than men, even if they have the same disability.

Be flexible. PwDs have different limitations, but also different sets of skills. Take advantage of the preferred method of communication and be open about the methodology.  For instance, photographs have proven to be effective to communicate with participants with physical, hearing or development disabilities (Jurkowski, 2008). This is an example of an alternative that requires small adjustments and can be easily triangulated with other methods.

Listen. When in doubt, ask the participants what methodology makes them feel more comfortable. Participation is also about listening and learning from others, and PwDs hold the key to understanding what suits them best.

Create capacities. Strengthen the M&E capacity of disability organizations. This will help to develop the organizations and build and share bi-directional knowledge. As a development practitioner, also invest some time educating yourself more about disability. For instance, learn some basic sign language to integrate yourself with people with hearing disabilities.

Be aware of trade-offs. Programs face time constraints, and full participation is not always feasible. Identify the phase of the evaluation that can be participatory and that can also have the most benefits for the participants. In the context of the COVID-19 pandemic, it is important to consider that digital tools might not be accessible to some PwDs. Therefore, outcome should be interpreted taking into account the selection bias.

PwDs are one of the most vulnerable groups according to the World Report on Disability; they experience higher rates of poverty and are more likely to be unemployed (World Health Organization, 2011). Thus, PwDs should have the opportunity to have a voice in the evaluation of programs and policies that impact their lives and communities.

References:

Brandon, P. R. (1998). Stakeholder participation for the purpose of helping ensure evaluation validity: Bridging the gap between collaborative and non-collaborative evaluations. American Journal of Evaluation, 19, 325–337.

Chambers, R. (1994). Participatory rural appraisal (PRA): Challenges, potentials and paradigm. World development, 22(10), 1437-1454.

Jacobson, M. R., Azzam, T., & Baez, J. G. (2013). The nature and frequency of inclusion of people with disabilities in program evaluation. American Journal of Evaluation, 34(1), 23-44.

Jurkowski, J. M. (2008). Photovoice as participatory action research tool for engaging people with intellectual disabilities in research and program development. Intellectual and developmental disabilities, 46(1), 1-11.

Olshanska, Z., van Doorn, J., & van Veen, S. C. (2016). My Story My Rights: how individual stories of people with disabilities can contribute to knowledge development for UNCRPD monitoring. Knowledge Management for Development Journal, 11(2), 43-62.

Parry, O., Gnich, W., & Platt, S. (2001). Principles in practice: reflections on a ‘postpositivist’ approach to evaluation research. Health Education Research, 16(2), 215-226.

Robinson, S., Fisher, K. R., & Strike, R. (2014). Participatory and inclusive approaches to disability program evaluation. Australian Social Work, 67(4), 495-508.

Shakespeare, T. (2006). The social model of disability. In L. J. Davis (Ed.), The disability studies reader (2nd ed., pp. 197–204). New York: Routledge.

Ware, J. (2004). Ascertaining the views of people with profound and multiple learning developmental disabilities. British Journal of Learning Disabilities, 32, 175–179.

World Health Organization. (2011). World report on disability. Malta: World Health Organization.

[1] The author worked as M&E officer in a project of empowerment of women with disability in Nicaragua from 2018 to 2019.

About the author:

Gersán Vásquez GutiérrezGersán Vásquez Gutiérrez is an economist and holds a master’s degree in governance and development. He works as an M&E officer in a regional irregular migration prevention program in Nicaragua. His main areas of interest are impact evaluation, migration, and local development.

 

Featured

COVID-19 | Remote research in times of COVID-19: considerations, techniques, and risks by Rodrigo Mena and Dorothea Hilhorst

By Posted on

The current COVID-19 pandemic is preventing many scholars and students, especially those in the social sciences, from visiting identified research sites and interacting with the groups or actors important for their research. Many researchers now plan to shift to forms of remote research where data are gathered without meeting research participants in person. While COVID-19 compels this trend, even before the pandemic scholars have had to conduct remote research when fieldwork is considered risky or difficult, for example in high-conflict or remote contexts. Our research of the interaction of disasters and conflict in Afghanistan and Yemen shows what to keep in mind when conducting remote research.

Remote research refers to research where the principal researcher is not engaging in face-to-face data gathering processes ‘on the ground’. This means that other people can gather data on behalf of the researcher in research locations, or that interviews with research participants are conducted by phone or using the Internet. Whereas quantitative research often uses enumerators to survey, qualitative research usually relies on face-to-face interviews or focus-group discussions that now need to be organized and conducted from a distance. Research shows that it is indeed possible to talk to participants using interfaces like telephones or social media platforms and to obtain rich and qualitative data through these, mostly internet-based, forms of communication[1]. However, the use of technology also needs to be approached with caution and in a reflective manner, as discussed in another blog.

Fundamentals and ethics for sound research still apply

No matter how hard one tries, remote research creates additional challenges, and some research questions beg so much nuance and depth that they better not be considered in remote research. Data gathered by means of remote research is also difficult to triangulate and validate, as a multitude of data sources not considered at the onset of the data collection process may present themselves in the field. Researchers may also come closer to understanding complex dynamics when immersed in the communities they are studying. Otherwise, many other routes can be explored to validate data. Think newspaper articles, GIS or satellite images, secondary sources, consulting other researchers familiar with the area, among others.

Research ethics can also be complicated when research is conducted remotely. Whether data are collected through video-based conversations or by using a third person to conduct the interview, it is important to consider whether informed consent genuinely has been obtained and how confidentiality can be guaranteed. In case of sensitive issues, face-to-face interaction allows one to read participants’ body language to detect whether the interview creates discomfort. It also allows researchers to build a trust relationship with research participants. How can researchers make sure that enough checks and balances support remote interviewing processes to avoid interviews creating anxiety or discomfort?

Finally, we need to think about how to convey the message that the research is in the interest of the research participants. Without the engagement and personal attention of a real encounter, will participants feel that they benefit from the research? Researchers often seek to ‘leave something behind’—stories, information, advice, or perhaps volunteer work for a community or NGO—to ‘give back’ to the research participants. Remote research requires questioning ways in which to move beyond the mere extraction of information that so clearly signals the asymmetric power relations between researchers and researched actors.

Some do’s and don’ts

When these complicated questions have been addressed, the question remains how to do remote research. Here are some pointers that we developed out of our experiences of researching the interaction of disasters and high-intensity conflict in Afghanistan and Yemen:

  • Some research questions cannot be addressed remotely, hence, the research design and questions needed to be adapted for remote research.
  • Ethical board approval is just as important for remote research as it is for fieldwork and cannot be skipped.
  • In order to enrich and triangulate findings, we need to be innovative. For our research, interviews were also conducted with people that recently migrated from the areas of interest to a place where they could be reached physically. Similarly, aid workers active in the area were interviewed during stop-overs at airports.
  • In order to create a broad and in-depth range of data, a multiplicity of methods besides interviews were used. These included digital surveys, the analysis of photographs taken by the participant and voice messages from participants describing places and situations, and many other creative options.
  • To remotely understand the context, relevant news, and everyday life in research areas, talking to people who know the area and reading the news about those places were key. This information allowed for better interviews and better data analysis.
  • Just like in normal interviews, body language is important for creating trust and diminishing anxiety. Sitting too close to your camera can make your presence intimidating, whereas keeping some distance and not filling the screen allows the participants to see your hand movements and background. Participants will see everything, also when you stop being attentive because you want to check some information on your phone, for example. It is therefore important to be mindful of your actions and to try to remain focused and engaged.
  • Rules for asking questions, such as using active language, asking questions one by one, trying to phrase questions and reword them in understandable language, apply even more in remote research.

Remote research is possible, but as students and researchers have to adapt to remote research, so do universities, research institutions, supervisors, and donors. Budget lines for travel may be reduced, but it may be important to provide funds for better computers, webcams, and video-based solutions.

Remote research can also be seen as an opportunity to do research differently, especially in an era where the need for travel must constantly be weighed up against the harm of adding to emissions related to climate change. We can now think of expanding the geographies of our research and reaching people in regions and places that were not considered possible before. For many students and researchers with limited budgets, it also can be a means to reduce the costs of research. However, as mentioned before, all these benefits and the use of remote research need to be weighed against adverse risks.

Which other relevant considerations would you like to share? Please feel free to leave a comment with tips, tricks or concerns.

[1] Bolt N and Tulathimutte T (2016) Remote Research. Rosenfeld Media. Available from: http://rosenfeldmedia.com/books/remote-research/#faq (accessed 1 November 2016). No page.

This article is part of a series about the coronavirus crisis. Read all articles of this series here.

About the authors:

R. Mena (2019)Rodrigo (Rod) Mena is a socio-environmental researcher and AiO-PhD at the International Institute of Social Studies of the Erasmus University Rotterdam. His current research project focuses on disaster response and humanitarian aid governance in places affected by high-intensity conflict, with South Sudan, Afghanistan and Yemen as main cases. He has experience conducting fieldwork and researching in conflict and disaster zones from in Africa, Latin America, Europe, Oceania and Asia. Twitter: @romenaf

Foto kleiner formaatDorothea Hilhorst is Professor of Humanitarian Aid and Reconstruction at the International Institute of Social Studies of Erasmus University Rotterdam. She is a regular author for Bliss. Read all her posts here.