Participatory evaluation has been praised for engaging vulnerable groups such as persons with disabilities (PwD). However, the inclusion of this group can be challenging and even self-defeating if carried out incorrectly. Despite the challenges, evaluators and researchers can follow some strategies to make the evaluation process with PwD as inclusive as possible.
Disability and participatory methods
For a long time, persons with disabilities (PwDs) were socially ostracized and confined to special schools and health centers. Growing pressure from disability rights organizations made possible a shift from an individual and biological view of disability towards a social and inclusive model that focuses on the interaction between individual impairments and social and environmental barriers (Shakespeare, 2006). Since then, international progress has been made to recognize the right of PwDs as full and contributing members of society; the formation of the 2006 UN Convention on the Rights of Persons with Disabilities is an example of a step in the right direction on this front.
In previous decades there has been a shift in research and evaluation methodologies in academia as well. Criticism of the ineffectiveness of the positivist paradigm to include vulnerable groups in research has led to the rise of participatory approaches in which PwDs and other marginalized groups play an important role in shaping research agendas and outcomes (Parry et al., 2001). The alternative bottom-up methodologies became known for challenging power relations and giving voice to marginalized groups, including PwDs (Chambers, 1994).
As a result, participatory methods have been crucial for engaging PwDs in more active roles in the processes of monitoring and evaluation (M&E) and not only as simple research subjects. For instance, many evaluations now involve PwDs organizations in the role of advisers where they can choose data collection instruments (Robinson et al., 2014) and use their expertise to interpret results and provide feedback (Olshanska et al., 2016). Increased participation has been praised for improving the validity and general outcomes of the evaluations (Brandon, 1998).
The challenges of inclusion
Despite recent achievements, many challenges lie ahead for greater inclusive participation of PwDs in program evaluations. One of the most overlooked aspects is the design of inclusive evaluation instruments (surveys, focus groups): evaluators tend to regard PwDs as a homogeneous group. Therefore, the instruments fail to take into consideration the diversity of disability, especially in terms of communication styles.
This creates an under-representation of the least advantaged within the target group. A study of 31 peer-reviewed articles in ten top-ranking evaluation journals shows that people with intellectual and development disabilities were less likely to participate in evaluation processes than people with any other type of disability (Jacobson et al., 2012). Even if they do participate, their answers in most of the cases might be biased or incomplete (Ware, 2004) since they communicate differently than their peers or experience psychological barriers such as low self-esteem.
Conducting evaluation activities in venues with physical barriers or far from the beneficiaries’ houses can hinder the participation of people with a physical disability. Therefore, ineffective M&E planning and instruments could not only bias the results, but also could end up creating negative unintended consequences such as exclusion and disempowerment. However, even if considering the linguistic and cognitive heterogeneity, what are the best alternatives to engage PwDs in participatory evaluation processes? Is inclusive participatory evaluation more time consuming?
Lessons learned: How to overcome the obstacles?
From my experience working with women with disabilities in Nicaragua[1], when it comes to disability, there are no one-size-fits-all solutions. Nonetheless, there are low-cost alternatives that can improve the overall level of participation. Here are some things to keep in mind:
Learn about your target group. An overview of the type of disability and some social variables is crucial to balance participants in focus groups, disaggregate data by categories, and prepare in advance for special requirements (e.g. the use of a sign interpreter, ramps for wheelchairs). It is also key to better understand power dynamics within the group. For instance, women face more discrimination than men, even if they have the same disability.
Be flexible. PwDs have different limitations, but also different sets of skills. Take advantage of the preferred method of communication and be open about the methodology. For instance, photographs have proven to be effective to communicate with participants with physical, hearing or development disabilities (Jurkowski, 2008). This is an example of an alternative that requires small adjustments and can be easily triangulated with other methods.
Listen. When in doubt, ask the participants what methodology makes them feel more comfortable. Participation is also about listening and learning from others, and PwDs hold the key to understanding what suits them best.
Create capacities. Strengthen the M&E capacity of disability organizations. This will help to develop the organizations and build and share bi-directional knowledge. As a development practitioner, also invest some time educating yourself more about disability. For instance, learn some basic sign language to integrate yourself with people with hearing disabilities.
Be aware of trade-offs. Programs face time constraints, and full participation is not always feasible. Identify the phase of the evaluation that can be participatory and that can also have the most benefits for the participants. In the context of the COVID-19 pandemic, it is important to consider that digital tools might not be accessible to some PwDs. Therefore, outcome should be interpreted taking into account the selection bias.
PwDs are one of the most vulnerable groups according to the World Report on Disability; they experience higher rates of poverty and are more likely to be unemployed (World Health Organization, 2011). Thus, PwDs should have the opportunity to have a voice in the evaluation of programs and policies that impact their lives and communities.
References:
Brandon, P. R. (1998). Stakeholder participation for the purpose of helping ensure evaluation validity: Bridging the gap between collaborative and non-collaborative evaluations. American Journal of Evaluation, 19, 325–337.
Chambers, R. (1994). Participatory rural appraisal (PRA): Challenges, potentials and paradigm. World development, 22(10), 1437-1454.
Jacobson, M. R., Azzam, T., & Baez, J. G. (2013). The nature and frequency of inclusion of people with disabilities in program evaluation. American Journal of Evaluation, 34(1), 23-44.
Jurkowski, J. M. (2008). Photovoice as participatory action research tool for engaging people with intellectual disabilities in research and program development. Intellectual and developmental disabilities, 46(1), 1-11.
Olshanska, Z., van Doorn, J., & van Veen, S. C. (2016). My Story My Rights: how individual stories of people with disabilities can contribute to knowledge development for UNCRPD monitoring. Knowledge Management for Development Journal, 11(2), 43-62.
Parry, O., Gnich, W., & Platt, S. (2001). Principles in practice: reflections on a ‘postpositivist’ approach to evaluation research. Health Education Research, 16(2), 215-226.
Robinson, S., Fisher, K. R., & Strike, R. (2014). Participatory and inclusive approaches to disability program evaluation. Australian Social Work, 67(4), 495-508.
Shakespeare, T. (2006). The social model of disability. In L. J. Davis (Ed.), The disability studies reader (2nd ed., pp. 197–204). New York: Routledge.
Ware, J. (2004). Ascertaining the views of people with profound and multiple learning developmental disabilities. British Journal of Learning Disabilities, 32, 175–179.
World Health Organization. (2011). World report on disability. Malta: World Health Organization.
[1] The author worked as M&E officer in a project of empowerment of women with disability in Nicaragua from 2018 to 2019.
About the author:
Gersán Vásquez Gutiérrez is an economist and holds a master’s degree in governance and development. He works as an M&E officer in a regional irregular migration prevention program in Nicaragua. His main areas of interest are impact evaluation, migration, and local development.
What do you think?