By Anika Hannan, monitoring and evaluation specialist at ICF International.
Monitoring progress toward achieving global goals such as improving gender equality and women’s empowerment requires developing instruments that measure these constructs. For an abstract concept like ‘women’s empowerment’, any survey instrument must be well understood by both participants and researchers. A team from the Gender, Agriculture, and Assets Project, Phase 2 (GAAP2) project endeavored to improve the validity of women’s empowerment survey instruments through cognitive interviewing.
GAAP2 launched the project-level Women’s Empowerment in Agriculture Index (pro-WEAI) in April 2018 to measure women’s empowerment over the course of an agricultural development project. Pro-WEAI is a composite index that shows the level of empowerment of the women surveyed both as a group, and compared with men in their households. It includes modules for adapting the index to focus on certain aspects, such as health and nutrition, or intra-household relationships.
To assess whether participants could understand the pro-WEAI survey questions, researchers at the International Food Policy Research Institute (IFPRI) conducted cognitive interviewing. Cognitive interviewing is a qualitative method assessing whether participants understand survey questions as the researchers intended. It can identify potential sources of error and provide insight to participants’ interpretations. It is administered by adding qualitative follow-up questions and probes to standard (pro-WEAI) survey questions. Enumerators then note the responses as well as observations and non-verbal communication.
For this study, enumerators needed training on cognitive interviewing as they were not familiar with it. Cognitive interviewing was conducted on the pro-WEAI health and nutrition module and intra-household relationships module in two upazilas (sub-units of districts) in rural Bangladesh: Sitakunda (Chittagong division) and Aditmari (Rangpur division).
Getting enumerators up to speed from training to data collection
The blog author and senior researchers from DATA (a local data collection team in Bangladesh) familiar with data collection instruments gave a six-day training to six DATA field enumerators. The training introduced pro-WEAI and cognitive interviewing methodologies. It proved critical as the six field enumerators were very well versed in quantitative data collection but unaccustomed with qualitative data collection.
Trainees thoroughly discussed the format and content of each question from the cognitive interview guide and piloted the questions with colleagues. They were initially hesitant about following up with (cognitive testing) probes and recording potentially lengthy responses by hand. However, over the course of the training and throughout data collection, they became more comfortable and confident with the structure and methodology of cognitive interviewing.
In addition to collecting participant responses, enumerators were invited to record interesting observations or non-verbal communication that added meaningful context to the survey questions. For example, if a woman responded she understood a pro-WEAI question, but only after the question was repeated multiple times and reworded differently from the written survey question, it implied that she hadn’t understood the question initially.
During data collection, it turned out that enumerators had differing interpretations of what was ‘meaningful’ to notate and recorded observations that were not consistent or clear. Upon analysis, these additional observations could not be used to draw accurate conclusions about the context of the pro-WEAI responses.
The cognitive interviewing guide became lengthy as questions were added during its development. Some participants had difficulty understanding the cognitive questions and needed those questions to be repeated several times. Some surveys took up to a burdensome four hours! The limited yet ambitious data collection schedule demanded some adjustments: for example, instead of conducting a full interview in one sitting, interviews were divided across two days. This meant shorter data collection sessions, participants no longer getting distracted by other responsibilities (i.e. household chores, child care, etc.), and fewer breaks during the interview. Enumerators welcomed this change and participants became more engaged.
- Dedicating time to review and pilot the cognitive interview questions is helpful for adjusting survey questions before administering them in the field. Ideally this should be done with a few members of the population of interest. During training, enumerators piloted the cognitive interviewing guide (with DATA colleagues) and streamlined the questions daily, to avoid repetition, facilitate understanding, and improve the quality of the responses. Field administration revealed that some participants still found cognitive interviewing questions difficult to understand. A few community members could have perhaps provided better feedback to adjust the cognitive interview guide than DATA colleagues.
- Trainings in cognitive interviewing, especially with enumerators unfamiliar with this method, are critical to ensure data collection progresses efficiently and timely. Enumerators became more familiar with the cognitive interviewing methodology during the six-day training and even more so during the field data collection. Trained enumerators can more effectively probe and elicit responses from hesitant or uncomfortable participants. They seek to uncover why those questions might be perceived as sensitive or personal, and how the question could be improved to prompt more robust responses. Further training, including examples of potentially noteworthy observations (i.e. a participant asking for a question to be repeated multiple times, or avoiding eye contact during an uncomfortable question), may improve the quality of data recorded to collect robust information to improve the pro-WEAI modules.
- When translating the pro-WEAI survey and cognitive interviewing guides from English to other languages and cultural contexts, it is important to review and ensure that the original meaning is preserved. The meaning of the question (and of the response translated back into English) can be misinterpreted because of the language context. This is particularly important for cognitive interviewing and surveys researching abstract concepts such as women’s empowerment. Responses to open-ended cognitive interviewing questions partly reflect the specific cultural context of the participants’ experiences. To ensure that survey instruments are capturing accurate data and are valid in making cross-cultural comparisons, it is imperative that participants understand the researchers’ questions in their own contexts and that their experiences are captured accurately and translated reliably back to English. This adds to the richness of the data collected and helps develop strong instruments for measuring women’s empowerment. Thus, translations must be reviewed, and instruments must be piloted among participants who are familiar and knowledgeable with the population of interest.
Hannan, A., Heckert, J., James‐Hawkins, L., & Yount, K. M. (2019). Cognitive interviewing to improve women’s empowerment questions in surveys: Application to the health and nutrition and intrahousehold relationships modules for the project‐level Women’s Empowerment in Agriculture Index. Maternal & child nutrition, e12871. https://onlinelibrary.wiley.com/doi/full/10.1111/mcn.12871
Hannan A. 2018. How can you tell if your survey questions work? Using cognitive testing to improve measurement of women’s empowerment in nutrition and health. http://a4nh.cgiar.org/2018/04/16/how-can-you-tell-if-your-survey-questions-work/
Johnson KB and Diego-Rosell P. 2015. Assessing the cognitive validity of the Women’s Empowerment in Agriculture Index instrument in the Haiti Multi-Sectoral Baseline Survey. Survey Practice 8(3).
Martinez E, Seymour G. 2018. Introducing pro-WEAI: A tool for measuring women’s empowerment in agricultural development projects. http://gaap.ifpri.info/2018/04/27/introducing-pro-weai-a-tool-for-measuring-womens-empowerment-in-agricultural-development-projects/
Malapit, H., Quisumbing, A., Meinzen-Dick, R., Seymour, G., Martinez, E. M., Heckert, J., … & Team, S. (2019). Development of the project-level Women’s Empowerment in Agriculture Index (pro-WEAI). World Development, 122, 675-692. https://www.sciencedirect.com/science/article/pii/S0305750X19301706
Malapit HJ, Sproule K, Kovarik C, Meinzen-Dick RS, et al. 2014. Measuring progress toward empowerment: Women’s empowerment in agriculture index: Baseline report. Washington, DC: International Food Policy Research Institute.
Willis GB and Miller K. 2011. Cross-cultural cognitive interviewing: Seeking comparability and enhancing understanding. Field Methods 23(4): 331-341.
About the author
Anika Hannan undertook the majority of this project as an MPH candidate at Emory University’s Rollins School of Public Health. She is currently a monitoring and evaluation specialist at ICF with experience in women’s empowerment, maternal and child health, food security, nutrition, behavior change, and community health. Her skills include monitoring and evaluation, developing data collection tools, data analysis, report writing and editing, academic research, and project coordination. She looks forward to continuing her career and engagement in gender related research and interventions.