Mixed methods – a practical guide for the gold standard of evaluation research
Theresa Lorenz 1
Mixed-method evaluations combine qualitative and quantitative research methods, offering a deep and comprehensive analysis of complex social phenomena. This approach is widely regarded as the gold standard in evaluation research. However, effectively integrating these methods can be challenging. One common issue is the tendency to treat the findings from each method separately (e.g., in distinct publications), rather than fully integrating them. This separation can hinder a holistic understanding of the evaluation results, limiting the potential of a mixed-methods approach. In this paper, we provide an overview of various mixed-methods evaluation designs and aim to mitigate some of the challenges faced by researchers employing this approach. Our goal is to equip evaluation researchers with the knowledge and tools needed to incorporate mixed methods into their methodological toolkit, enabling them to apply this approach effectively.
Mixed methods combine qualitative and quantitative research methods to investigate social phenomena. This approach is regarded as the gold standard in evaluation research for many research questions because the blend of qualitative and quantitative research methods can provide a deep and comprehensive understanding of the impacts and processes of financial education initiatives. Consequently, mixed methods may enable more robust and reliable evaluations. However, despite these advantages, mixed methods are not often used due to the additional challenges they present.
The idea of combining different methods in social and evaluation research can be traced back almost 200 years (Bazeley, 2018). Back then, researchers often employed what we would now recognize as mixed or multi-mode methods, although these terms were not explicitly used back then. Fundamental conflicts between qualitative and quantitative methods and differing views on the nature of reality and knowledge (ontology and epistemology) were also not particularly pronounced during this era (Maxwell, 2016).
It was only later, in the 1970s and 1980s, that strong conflicts between qualitative and quantitative methods emerged (Bazeley, 2018). What became known as the “paradigm wars” led to a distinct divide between qualitative and quantitative methods. Qualitative research became closely aligned with the interpretivism paradigm, characterized by inductive reasoning and interpretative techniques. In contrast, quantitative researchers typically adhered to a positivist paradigm, relying on deductive reasoning, experimental methods, and statistical analysis (Bazeley, 2018).
The late 20th and early 21st centuries finally marked the formal recognition of mixed-methods research as a distinct methodology. This shift was driven by increasing dissatisfaction with the limitations imposed by the paradigm wars and a growing acknowledgment that neither quantitative nor qualitative approaches were sufficient on their own to comprehensively address complex social phenomena (Creswell and Clark, 2011). A new paradigm (called pragmatic paradigm) emerged as a response, advocating for the selection of methods and paradigms best suited to answer the research question at hand (Dawadi et al., 2021). While this can be seen as an effort to resolve the paradigm wars, the strict separation of methods that resulted from the paradigm wars has remained a considerable challenge in the application of mixed-methods research.
The historical divide between qualitative and quantitative research methods has led to a high degree of specialization among qualitative and quantitative researchers with limited communication among them. This presents significant challenges in mixed-methods research and is one reason why, despite being perceived as the gold standard of evaluation, mixed methods are not frequently applied, and their full potential is often not realized. Indeed, it is common practice of mixed-methods research that researchers do not work together, which gives rise to the risk of finding diverging results that are then presented in separate papers rather than a combined mixed-methods publication. The lack of formal education in mixed-methods research compounds these issues. The required skills include understanding design options, all methods used in the study, potential integration strategies, and ways to maintain focus without overextending the analysis.
Furthermore, there is a lack of guidelines that demonstrate how to practically integrate results. Despite some existing guidelines for publishing mixed-methods papers (Palinkas et al., 2019), there is also a lack of standardized, established protocols comparable to those in other research areas - like standardized protocols for randomized controlled trials or standardized protocols for structured literature reviews (Bazeley, 2018). This gap is problematic because diverse disciplinary traditions can reinforce the issue of poor integration, thus not fully leveraging the potential of mixed methods.
Lastly, mixed methods often require additional efforts and greater resources than more traditional evaluation approaches. This requires management commitment to provide the needed resources and time, and to protect these from budget and time pressures (Bamberger, 2012).
With this paper, we aim to reduce the burden on evaluation researchers who are considering mixed methods as an approach, allowing them to make this approach part of their methodological toolkit available for application when appropriate. The remainder of this paper is, thus, structured as follows: In the first chapter, we examine the utility of mixed methods in evaluation studies, highlighting their unique benefits and the prerequisites for their successful implementation. Subsequently, we delve into the core design features of mixed methods and common mixed-methods evaluation designs. The second chapter provides practical guidelines and strategies for integrating data collected in mixed-methods studies.
1 Mixed-methods designs
1.1 Defining mixed methods
To this day, mixed-methods research lacks a universally accepted definition (Schoonenboom, 2023). Bazeley (2018) addresses this issue by referring to a synthesized definition derived from 19 different descriptions of mixed methods proposed by Johnson et al. (2007). Serving as the foundation for this document, this concise definition describes mixed methods as follows:
“The type of research in which a researcher or team of researchers combines elements of qualitative and quantitative research approaches (e.g., use of qualitative and quantitative viewpoints, data collection, analysis, inference techniques) for the breadth and depth of understanding and collaboration.” (Johnson et al., 2007, p. 123).
As highlighted by the definition above, a fundamental prerequisite for a study to qualify as a mixed-methods study is some degree of integration — the process of combining different research methods to achieve a common research goal. In contrast, studies that employ both qualitative and quantitative elements but with minimal integration are typically categorized as multi-method or combined studies (Bazeley, 2018; Creswell and Clark, 2017). 2
Indeed, Bazeley (2018) emphasizes that a crucial component of a mixed-methods study is “a ‘conversation’ between the different sources and/or methods used (...) within the analysis, continuing to the presentation of results and discussion of those results” (Bazeley, 2018, p. 7). The extent of “conversation” required to distinguish between mixed-methods and multi-method studies, however, remains undefined. In evaluation contexts, the use of qualitative and quantitative approaches can vary, being either applied at specific stages (e.g. only collecting quantitative and qualitative data) or integrated throughout the entire process by additionally interacting analysis and findings. The ambiguity lies in the extent of integration that needs to occur in some or all phases for a study to qualify as a mixed-methods study (Bureau of Policy & Learning, 2013; Palinkas et al., 2019).
Two examples illustrating the ambiguity between mixed-methods and multi-method studies
For instance, qualitative methods used in planning stages of an evaluation, like focus groups for hypothesis generation or think-aloud techniques for survey item interpretation, often fall into “mixed-methods light” or “multi-method” categories if the approaches are not further integrated in analysis and interpretation. Such approaches thus primarily support the development of quantitative instruments without achieving thorough integration. However, the same study would be qualified as a mixed-methods study by most researchers if the analysis integrates both qualitative and quantitative data by comparing and synthesizing findings to draw richer, more nuanced conclusions. In such an integrated approach, the final interpretation reflects a more cohesive narrative, showing how qualitative insights explain or expand upon the quantitative results and vice versa.
Similarly, using statistical census data to contextualize qualitative findings during analysis only partially combines both methods. Full integration might involve follow-up interviews exploring insights from the census data, promoting dynamic interaction between the data sources. Moreover, the presentation of findings should intertwine results from both methods, each adding context and depth to the other.
1.2 Why mixed methods?
Overall, mixed methods in evaluation research are highlighted as being particularly useful for providing a more thorough, nuanced and reliable understanding of complex research topics. More specifically, the advantages can be summarized as follows (Bamberger, 2012; Bazeley, 2018; Bureau of Policy & Learning, 2013; Dawadi et al., 2021; Frechtling and Sharp, 1997):
Comprehensive understanding: Quantitative data provides robust numerical insights, while qualitative data adds rich context, depth and explanation. This integrated approach effectively compensates for the limitations of each method individually, tapping into the nuanced complexities of social environments and human behavior to paint a more holistic picture which allows a better generalization of findings (internal and external validity). In short, a mixed-methods approach “enables researchers to answer questions with sufficient depth and breadth” (Dawadi et al., 2021, p. 27).
Validation and cross-verification: Mixed methods allow for the results from one method to be validated or challenged by the other. This cross-verification enhances the reliability and validity of research findings.
Flexibility and adaptability: A mixed-methods approach is adaptable to various research settings and questions. It allows researchers to adapt their research design based on the evolving nature of the program studied.
Richer data and perspectives: Mixed-methods research captures a wider range of perspectives and data types, which can lead to more nuanced, detailed as well as unexpected and sometimes also contradictory findings. This is particularly valuable in evaluation research where understanding different stakeholders’ perspectives is crucial.
1.3 Prerequisites for applying mixed methods
Mixed methods have a number of advantages, but there are also some prerequisites to consider when applying this method, as detailed by Bamberger (2012), Bazeley (2018), Creswell and Clark (2017), and Dawadi et al. (2021).
First, mixed methods require a well-suited research question: As discussed earlier, mixed-methods approaches offer a more comprehensive perspective in various evaluation scenarios. However, there are specific research questions that benefit significantly from the integration of qualitative and quantitative methods. This is, for instance, the case when the evaluation aims to address both processes and impact. In such cases, qualitative methods excel in explaining the processes, while quantitative methods are better suited for evaluating the impact. In general, mixed methods are effective when complex interventions are investigated.
Second, mixed-methods approaches can be cost- and time-intensive. As Bamberger (2012) notes, this requires commitment to provide the needed resources and time, and to protect these from budget and time pressures. Given that researchers often specialize in their respective methods, time should also be allocated for them to understand each other’s methodologies and cultivate mutual respect and trust. The more integrated the mixed-methods approach, the more time, particularly during the planning and analysis stages of evaluation, needs to be dedicated.
Third, success in mixed-methods research requires skills in both qualitative and quantitative methods. Researchers often specialize in one or the other, making it essential for the team to collectively possess all the necessary skills. While not essential, having team members from different disciplines can be an additional benefit, enhancing the research with diverse perspectives. Each team member should, however, have at least a basic familiarity with all the methods applied in the evaluation. In addition, researchers need the skills to adequately combine different methods.
Lastly, mixed methods require openness and respect toward different perspectives that are often grounded in different quantitative and qualitative philosophical assumptions. This is crucial as diverse viewpoints can significantly enrich the quality of research, but these diverging approaches are often the reason why the full potential of mixed-methods approaches are not utilized. In a similar vein, it is critical to equally value both qualitative and quantitative methods even though methods may have different weighting in terms of how intensely they are used in the evaluation.
1.4 Main evaluation design features of mixed-methods studies
In this chapter, we present evaluation design aspects specific to mixed-methods studies. Aspects relevant to evaluation studies in general will be thoroughly discussed in additional parts of the evaluation series. Following the guidelines of Bamberger (2012) and Creswell and Clark (2017) three aspects need to be considered when applying a mixed-methods study, namely: (1) the integration stage, (2) timing of data collection, and (3) weighting of methods.
1.4.1 Stage of integration
Integrating qualitative and quantitative methods can happen at single, some or all stages of the evaluation cycle. Full integration across stages often yields the richest insights. Integration strategies may vary at each stage. For illustration purposes, the following are examples of integration at the planning, conducting, analysis and communication stage of an evaluation during the research/evaluation process:
At the planning stage, integration occurs at the design level - i.e. applying a mixed-methods evaluation design. For instance, this could involve planning a randomized controlled trial to quantitatively assess the outcomes of a financial literacy program together with qualitative interviews to understand program mechanisms. 3
At the stage of conducting the evaluation, integration might involve collecting quantitative and qualitative data, like a quantitative survey with qualitative interview insights.
At the level of analysis, integration involves blending quantitative statistical data with qualitative data, offering a richer, more nuanced understanding through validation, contrasting themes or cases. When communicating outcomes, synthesizing findings from integrated analyses is crucial for drawing comprehensive conclusions and effectively conveying the results.
Typically, mixed method approaches involve the combination of methods during several phases, where integration at one stage is often a precondition for integration at the subsequent stage. Naturally, integrating qualitative and quantitative data during analysis is impossible if only quantitative data have been collected. However, in many studies integration does not extend beyond the data collection phase as results from different data sources are not combined but rather presented in separate qualitative and quantitative reports (Bazeley, 2018). To illustrate this, Bazeley (2018) refers to a review by Niglas (2004), noting that of 1,156 articles from educational journals, only 145 used a combined approach, and even fewer fully exploit the potential of integration. He concludes that “while the baseline is to apply complementary methods, greater levels of integration are both possible and desirable” (Bazeley, 2018, p.117). For more details on integration see chapter 2.
1.4.2 Timing of data collection
In terms of timing, there are two ways of combining data collection in mixed methods – either sequentially or concurrently (also called a parallel or convergent designs) (Alele and Malau-Aduli, 2023; Bazeley, 2018). In a parallel design, qualitative and quantitative data are collected at the same time, whereas in the sequential design, one type of data source (qualitative or quantitative) is collected first, followed by the other data type.
It is worth mentioning that timing as a design option is not only relevant for data collection but also for the analysis of various data sources. For instance, quantitative and qualitative data might be collected concurrently, yet their analysis could be sequential (Palinkas et al., 2019). Despite this fact, the primary focus of mixed-methods design features often revolves around the timing of data collection, which is the main emphasis of this section. The discussion on the timing of data analysis will be further elaborated in chapter 2.
1.4.3 Methodological weighting
The dominant method is the one that is given relative priority in answering the research question. This can either be a dominantly quantitative design, a dominantly qualitative design or a balanced design with equal weight for each methodical approach (Dawadi et al., 2021). This distinction matters because researchers are typically specialized in either qualitative or quantitative or methods. Thus, they may hold different expectations regarding the involvement of “their” methodical approach (Bamberger, 2012).
A balanced design in evaluation may be one giving both process and impact evaluation equal weight, using qualitative methods to explore processes and quantitative methods investigating impacts. Another typical approach for a balanced design is the use of qualitative and quantitative methods for the same unit of analysis (such as participants of an intervention) (Bamberger, 2012).
1.5 Design combinations
The presented design options can be combined in various ways (Palinkas et al., 2019) and adaptions and variations throughout the research process are common as new insights or requirements might emerge and mixed methods are particularly suited to such dynamic scenarios (Palinkas et al., 2019). This section summarizes common examples for combining design options based on Alele and MalauAduli (2023), Bazeley (2018), and Creswell and Clark (2017).
To systematically illustrate design options and their combinations, mixed-methods research often relies on the following notation (Morse, 1991): Concerning timing, arrows indicate a sequential design. For example, an arrow from the first to the second data collection method represents the sequence of data collection (data collection method 1 p data collection method 2). A parallel design, on the other hand, is represented by a plus sign (data collection method a + data collection method b). With regards to weighting, it is also common practice to write the dominant method in bold, i.e. QUAN for a dominantly quantitative design and QUAL for a dominantly qualitative design. “QUAL p quan”, for example, reflects a sequential design where more weight is given to the qualitative method. For a summary on mixed-methods design options notation see Table 1.
| Timing | Weighting | ||
|---|---|---|---|
| Sequential data collection | → | Dominantly qualitative design | QUAL, quan |
| Parallel data collection | + | Dominantly quantitative design | QUAN, qual |
| Source: OeNB. | |||
1.5.1 Convergent parallel design (QUAN + QUAL)
In this design, qualitative and quantitative data are collected and analyzed separately, with results then being compared, expanded, contrasted or validated. Finally, the goal is to reach coherent, valid conclusions about the research question studied. The main objective of this design is to obtain a complete understanding of the research problem. Typically, equal weight is given to qualitative and quantitative methods.
Example 1: Convergent parallel design
In a financial literacy evaluation, this might involve conducting surveys to gather quantitative data on participants’ financial competencies (impact evaluation) while simultaneously conducting interviews or focus groups for an in-depth understanding of their experiences and perceptions of the program (process evaluation). After analyzing both sets of data independently, the findings are compared and integrated. This approach allows for a comprehensive understanding of the effectiveness of the financial literacy program, combining quantitative data on financial competencies with nuanced insights into the participants’ experiences and perceptions.
While this design can lead to nuanced and comprehensive results, it is time intensive and places the greatest demands on the mixed-method skills of the researchers. They might face challenges for integration when the findings from qualitative and quantitative research diverge. In such cases, researchers are advised to reanalyze the data to reconcile these discrepancies. On the other hand, these divergent findings may also reveal aspects that may otherwise be overlooked. Some guidelines on exploring dissonance and divergence can be found in section 2.4.
1.5.2 Explanatory sequential design (QUAN → qual)
This combination starts with quantitative data collection and analysis, followed by qualitative data. The qualitative findings are used to gain more in-depth information and help to explain and interpret the quantitative results. For example, if new questions arise from quantitative results they can be investigated in the subsequent qualitative phase. Also, outliers or unexpected results can be further explored. This design is usually dominantly quantitative.
Example 2: Explanatory sequential design
A financial literacy program might first be evaluated using a quantitative approach, such as surveys to gather quantitative data on participants’ financial competencies (impact evaluation). The results from this phase could highlight areas needing deeper exploration. The subsequent qualitative phase (e.g. interviews or focus groups) would aim to understand the reasons behind the quantitative findings, such as why certain financial concepts were not well understood by low-performing participants or how participants felt about the program. The initial quantitative findings guide the qualitative research, making the quantitative phase the starting point and foundation for further exploration of unexpected or highly relevant results.
1.5.3 Exploratory sequential design (QUAL → quan)
This design begins with qualitative data collection and analysis, with the findings guiding the design and development of quantitative data collection. First, this design is useful for development and validation of research instruments (e.g. a survey). Second, it can be used to explore a research topic which lacks theoretical grounds or if hypotheses are not known. Third, it can be applied to test whether qualitative results also apply to a wider population. This design is usually dominantly qualitative.
Example 3: Exploratory sequential design
Such an approach may start with a broad qualitative exploration, perhaps through detailed interviews and/or focus groups with participants to understand their experiences and challenges with financial literacy. The insights gained would then shape the subsequent quantitative phase, such as designing a small survey to measure the prevalence of these challenges among a larger group. However, the qualitative findings are at the center of the design.
1.5.4 Embedded Design
The embedded design is characterized by integrating a smaller component of one type of data (qualitative or quantitative) within a larger framework of the other data type. The nested component plays a supportive and supplementary role. Data collection and analysis in this design can be sequential or concurrent, occurring before, during or after the primary method is implemented. This design is particularly effective when a research question demands a combination of both quantitative and qualitative data to provide context and depth. The primary focus is on one method, often quantitative, with the aim of understanding how the additional data source enhances the understanding of the research question.
Differentiation between an embedded design and other mixed-methods variations is not straightforward. Creswell and Clark (2017) point out that the key distinction lies in the standalone value of the data sources. In an embedded design, the value of the secondary, nested component is largely dependent on the context provided by the primary data source. Conversely, in other mixed-methods designs, the secondary data sources frequently possess enough standalone value to be meaningful even without the primary method.
Example 4: Embedded Design
In a financial literacy program evaluation, the primary focus might be a quantitative survey assessing participants’ financial competencies. Simultaneously, a smaller qualitative component, such as interviews with a subset of participants is conducted. These interviews are designed to provide deeper insight into specific aspects of the quantitative findings on how the program’s delivery affected participants’ learning. The qualitative insights are used to add depth and context to the quantitative findings, enhancing the overall understanding of the program’s impact.
Each design has its specific strengths and is chosen based on the research objectives. The convergent parallel design is suitable for studies where a comprehensive understanding from both the qualitative and the quantitative perspective is needed simultaneously as data points are more likely to be comparable if they are collected at the same point in time. Sequential designs are preferred when the findings from one method are essential to inform or complement the other method. An embedded design is beneficial when a comprehensive understanding of the research topic is needed, but there is less time or there are fewer resources than in a convergent parallel design to devote equal emphasis to both qualitative and quantitative methods (Alele and Malau-Aduli, 2023; Creswell and Clark, 2017). For more details on common mixed-method designs see Creswell and Clark (2017).
The design options presented offer potential approaches for mixed-methods evaluation. But it is important to remember that these are no rigid formulas. The key to effective research is to flexibly combine and adapt elements of typical mixed-methods frameworks, always focusing on the overarching goal of addressing your research question effectively.
| Design | Description |
Timing of data
collection |
Dominant
method |
Primary objective |
|---|---|---|---|---|
|
Convergent parallel
design (QUAN + qual) |
Quantitative and qualitative
data are collected and analyzed separately; findings are then compared or validated. |
Concurrent | Equal |
Comprehensive
understanding of the research problem |
|
Explanatory sequential
design (QUAN → qual) |
Starts with quantitative data
collection and analysis, followed by qualitative approach to explore findings further. |
Quantitative first,
qualitative second |
Quantitative |
In-depth explanation
of quantitative results |
|
Exploratory sequential
design (QUAL → quan) |
Begins with qualitative data
collection and analysis to guide the development of quantitative instruments. |
Qualitative first,
quantitative second |
Qualitative |
Qualitative exploration
guides quantitative research |
| Embedded Design |
Integrates a smaller component
of one type of data within a larger framework of the other type. |
Flexible (concurrent
or sequential) |
Primary
method (usually quantitative) |
Enhance primary
method’s understanding |
| Source: OeNB. | ||||
A big challenge in mixed-methods research is choosing the right design while maintaining focus. Each stage of evaluation offers numerous possibilities for combining mixed-methods design features, leading to a complex decision-making process. There is no universal recipe for selecting and applying mixed methods which makes it hard to give universally applicable guidelines. While integration often falls short, in cases where researcher intend to integrate, they often risk losing focus (Dawadi et al., 2021).
To navigate this challenge, early and careful consideration of the optimal design can be effective. This strategy refers to the thoughtful combination of qualitative and quantitative methods which begins at the very start of the planning phase. The initial focus should be on defining the overarching goals and specific research questions, which will inform the selection of design features. Key considerations for these features, including their respective benefits and limitations, are presented above. Prior to data collection, researchers should clearly define the purpose of each data sources they intend to use for the evaluation. Additionally, they should develop an initial plan outlining how and when these data sources will be combined. This proactive planning ensures that each component of the research is aligned with the overall objectives. Implementing a theoretical framework can also be beneficial, offering a guide for the entire research process. However, it’s important to remember that while these strategies provide structure, they should not be rigid. Maintaining a degree of flexibility is crucial to foster exploration and adaptability as research unfolds (Bazeley, 2018).
2 Mixed-methods integrative data analysis
This chapter underscores the vital role of integration in mixed-methods research, extending beyond the initial phases of planning and execution to the crucial stages of analysis and communication. This is where the true potential of mixed methods is realized. However, challenges often lead to separate analysis and reporting of qualitative and quantitative data. Addressing this gap, we introduce three advanced techniques for combining data: the triangulation protocol, “following a thread” and the mixed-methods matrix. Building upon the work of O’Cathain et al. (2010), who proposed these three types of integration techniques, we expand on recent advancements.
The techniques will be illustrated by the following mixed-methods financial literacy evaluation example, which will be applied throughout this chapter:
Financial literacy evaluation example
In this example, the evaluation aims to assess the effectiveness of a voluntary financial literacy program in improving secondary school students’ knowledge, skills and attitudes toward personal financial management.
The quantitative component of the evaluation is an impact evaluation where students complete surveys before and after the program to measure changes in their skills. Also, the curriculum and educational materials have been reviewed by the researchers to evaluate content coverage. Statistical analysis of survey data were used to identify significant changes in knowledge and behavior. Analysis of attendance records proved information about engagement levels. Content analysis of program materials and curriculum gave information on alignment with learning objectives.
The qualitative component contains in-depth interviews with a selection of students to gather insights about their experiences and perceived impact of the program. Focus groups are used to explore students’ views on the program’s content and delivery. Analysis of interview and focus group transcripts were used to derive common themes, insights and student perceptions.
2.1 Triangulation protocol
This technique was initially developed by Farmer et al. (2006) and was later adapted by O’Cathain et al. (2010) and Bazeley (2018). Triangulation is a term that has been used in different ways. It was originally meant to test the convergence of results as a quality control tool by applying more than one method in parallel. However, in mixed-methods research it has been proven that results from different methods do not necessarily give more valid results but rather provide a more complete and extended picture with often diverging results as well. Hence, finding convergence is only one element of a triangulation protocol (Bazeley, 2018).
The aim of the triangulation protocol is to compare and explain qualitative and quantitative results to gain a complete picture about the research topic to eventually develop meta-themes across the different data sources. The triangulation protocol entails several key steps:
Preparation: Separately analyze qualitative and quantitative datasets using an analytical method suitable for the data source to identify shared concepts and themes relevant to the research questions.
Convergence matrix: Organize themes/concepts in a matrix in accordance with the data sources and their respective results in a so-called convergence matrix. The matrix’s rows represent the identified themes/concepts, while columns display various data sources and their findings, like qualitative quotes and quantitative frequencies. An additional column for data convergence indicates agreement, partial agreement, silence (a theme appearing in some, but not all, data sources) or dissonance (differing meanings or prominence across data sources) between the different data sources for one theme.
It is important to mention that dissonance across data sources is not a sign that something went wrong but is rather perceived as vital to get a better understanding of the research question. We will provide some strategies on how to deal with dissonance across data sources at the end of this chapter. Similarly, “silence” as described above might be expected as not every method is best suited to deal with every theme. However, unexpected silence should be similarly treated as dissonance across data sources and can contribute to a better understanding of the research topic. Researchers are also invited to creatively adapt and expand the convergence matrix for their particular purpose.
Collaborative analysis: If the triangulation protocol is conducted by multiple researchers, it’s important to compare their individual assessments to identify any differences and to clarify interpretations of the findings. Farmer et al. (2006) advocate for an approach in which at least two researchers are involved in the triangulation process.
Synthesis: Synthesize a cohesive interpretation and description of each theme, considering the broader context of the research.
Applying the financial literacy evaluation example
After the qualitative and quantitative data have been primarily analyzed separately, the application of the convergence matrix may look as follows:
| Themes |
Quantitative data
(frequencies) |
Qualitative data
(quotes) |
Program material
analysis |
Data convergence |
|---|---|---|---|---|
|
Understanding of
financial concepts |
75% improvement
in test scores |
“I feel more confident
about managing money.” |
Curriculum covers
key financial concepts. |
Agreement |
| Application of skills |
60% reported using
budgeting tools |
“I started tracking my
expenses regularly.” |
Emphasis on practical
budgeting tools. |
Agreement |
| Attitude toward saving |
70% reported that
they had started to save more |
“I’m trying to save, but
it’s not always easy to stick to the plan.” |
Materials encourage
saving but respondents acknowledge challenges. |
Partial agreement |
| Real-world relevance |
Not specifically
measured |
“Wish there were
more reallife examples.” |
Mostly theoretical
content. |
Silence |
|
Engagement in the
program |
High attendance
rates (above 85%) |
“Many sessions were
uninteresting and too technical |
Materials are dense
and heavily theoretical. |
Dissonance |
| Source: OeNB. | ||||
In cases as illustrated by the last theme, where data shows divergence, we recommend specific techniques to explore the reasons behind this variance. For instance, further investigations revealed that students’ high attendance rates were not solely due to engagement with the program but were also influenced by external factors, such as monetary incentives provided by their parents.
Once the results of the convergence matrix have been discussed and a uniform interpretation has been reached by the collaborating researchers, the final step involves synthesizing the findings from the convergence matrix, which can be presented as follows:
Both the quantitative and qualitative data indicate an improvement in participant’s understanding of financial concepts, which is supported by the comprehensive coverage of financial concepts in the program materials. Regarding the application of skills, there is a notable agreement between participants using budgeting tools (quantitative) and their qualitative reflections on adopting new financial habits. There is a general alignment of improved saving attitudes and behaviors, but nuances in the qualitative data and program materials suggest that the implementation of these savings practices also comes with the challenge of actually sticking to long-term goals. The lack of real-world application is noted qualitatively, but not directly measured in the quantitative data or addressed in the program materials. True engagement in the program could be improved as student feedback (qualitative) and the nature of the program materials suggest a lack of true engagement or interest in the program.
2.2 Following a thread
Initially developed by Moran-Ellis et al. (2004) and further taken up by O’Cathain et al. (2010) the “following a thread” technique offers a narrative and iterative approach to integrative data analysis. It focuses on how a single theme or issue, identified within one set of data, is explored across various datasets. This method allows for a deep and nuanced understanding of research findings and facilitates a focused exploration of specific issues across different datasets (O’Cathain et al., 2010). It provides a framework that enables both deductive (testing existing theories) and inductive (generating new theories) analysis (Dupin & Borglin, 2020).
Though less standardized than the triangulation protocol, “following a thread” may entail the following steps:
Preparation: Separately analyze qualitative and quantitative datasets using an analytical method suitable for the data source. From this initial analysis, derive concepts and themes and particular questions that require further exploration. This can be an unexpected discovery or topics that appear promising for further exploring this topic through the additionally available data sources.
Follow the thread: Then select this particular question or theme from one data source as a “thread” to explore it across other datasets. This can be a back and forth movement across datasets with the goal to iteratively interweave the findings from different datasets. This is an iterative analysis process, where initial findings guide hypothesis formation, which is then examined using other data sources. Finally, the findings from the thread lead to a pattern of findings that can also be intertwined with findings from other threads.
However, a detailed description of how to practically apply and utilize these frameworks remains insufficiently explored, as highlighted in a systematic review by Dupin and Borglin (2020).
Applying the financial literacy evaluation example
After separately analyzing the qualitative and quantitative data, the quantitative results showed a significant improvement in survey scores related to budgeting and savings, indicating a positive change in financial skills following the completion of the program. Consequently, the first theme, “understanding of financial concepts,” was identified and traced through the qualitative data. Interviews revealed that several students began using budgeting apps or saving a portion of their allowance due to the program. Focus groups also indicated that many students found the interactive components of the program helpful. Both findings likely contributed to the positive impact on financial skills after the program.
However, some students expressed a desire for more real-life financial scenarios, leading to the identification of the second theme: “real-life scenarios are more accepted.” Attendance records showed higher attendance rates for practical financial management sessions, which included hands-on activities such as creating personal budgets and learning about financial tools. Revisiting the survey results indicated that students with higher attendance rates performed best. Therefore, the second theme is also supported by various data sources.
Intertwining both threads, we can conclude that interactive teaching methods were well-received and translated into real-life application for some students. Attendance patterns corroborate that engaging methods boost participation and lead to better learning outcomes.
2.3 Mixed-methods matrix – also called metamatrix or joint displays
This technique was notably developed and refined by Bazeley (2018) and O’Cathain et al. (2010) and has various forms. Its primary objective is to summarize and contrast trends and patterns across multiple cases or data points. While similar to the triangulation protocol, its focus shifts toward individual cases rather than overarching concepts or themes. A distinctive feature of some mixed-methods research can be the presence of qualitative and quantitative data for the same case (also called unit of analysis). Cases can be individuals, organizations, longitudinal datapoints, groups, geographical areas etc. that share some characteristics. This approach enables a very granular integration of qualitative and quantitative data sources. Cases are the unit of analysis in a mixed-methods matrix - also known as a joint display or metamatrix. Importantly, cases in the mixed-methods matrix serve as a means to illustrate derived themes but the case itself is not the main objective of the study. 4
Illustration of cases as the unit of analysis
For instance, a group of students who have completed a survey might also participate in detailed interviews, resulting in a specific group of cases where there is both survey data and interview transcripts available. In this example, students represent the case.
In a longitudinal study the date of data collection can be a case to investigate changes over time throughout different data sources. Different data sources can then be matched for the same case and date becomes the unit of analysis.
The technique of the mixed-methods matrix may include the following steps:
Preparation: Even if the analysis level is individual cases, initial separate analysis of qualitative and quantitative data can also be useful for this technique. This can be coding, calculations, selecting relevant variables etc. to identify relevant cases for the research questions. Indeed, rather than examining every case available in the sample, focusing on a subset of cases that presents particularly unique or surprising results can be more insightful.
Matching: Construct a matrix where each row represents a different case, identified by an identifier. Columns correspond to various variables or themes, encompassing both quantitative data (like scores or demographics) and qualitative data (such as interview themes or quotes). Each matrix cell then contains the data relevant to that case and variable or theme.
Synthesis: Additional methods can be employed to further synthesize the data, integrating matched data into coherent themes or findings. A further extension of this approach is the pillar integration method by Johnson et al. (2019), which provides a more detailed systematization of application and interpretation of results. However, this advanced methodology extends beyond the scope of this discussion.
Applying the financial literacy evaluation example
After the qualitative and quantitative data have been primarily analyzed separately and cases have been matched, the application of the mixed-methods matrix may look as follows:
| Case number | Score improvement | Quoted statement | Summary of individual changes |
|---|---|---|---|
| Highest improvement in financial literacy | |||
| Student A | 60% → 85% |
“Saving regularly made me more
confident about my financial future.” |
Demonstrated the highest uptake
of financial skills and concepts. |
| Student B | 55% → 90% |
“Understanding taxes and
investments has opened up new opportunities for me.” |
Showed significant engagement and
application of complex financial topics. |
| Lowest improvement in financial literacy | |||
| Student C |
No change, score
remained at 75% |
“I get the concepts in theory,
but it’s hard to apply them.” |
Minimal change indicates a struggle
to apply knowledge practically. |
| Student D | 75% → 73% |
“The program was informative,
but I haven’t changed much about how I handle money.” |
Small regression suggests a
disconnect between knowledge and behavior. |
| Source: OeNB. | |||
Synthesizing the results, we can conclude that students with the highest improvement demonstrated a proactive approach to applying learned financial concepts, resulting in improved financial behaviors and decision-making. Students with the lowest improvement understood the financial concepts but did not implement substantial behavioral changes, indicating a need for more practical approaches.
2.4 Exploring dissonance and divergence
In the previous discussion, we have seen that mixed-methods research sometimes uncovers different, and potentially conflicting, results. This is particularly salient in studies of complex social phenomena, where such divergent findings are not just common but can be highly informative. Uprichard and Dawney (2019) capture the importance of contrasting findings in social research with the following analogy:
“After all, we tend to assume that one method depicts one part or aspect of the object of study and if another method presents a different part or aspect, then the methods have together shown different parts or aspects of the same thing. But what if one method captures the ‘ear of the elephant’ and another method captures the ‘tail of a mouse’? Mixed methods, very successfully, capture multiple aspects of multiple parts that are entangled together instead of revealing some singular ‘thing’.” Uprichard and Dawney (2019, p. 22)
In this vein, the primary goal in mixed-methods research should not be to force convergence of findings. Instead, divergent results should be viewed as gateways to new perspectives and deeper understanding. When discrepancies arise, it is advisable to delve deeper by returning to the data or conducting additional analysis to uncover the reasons for these differences. Bazeley (2018) suggests several potential sources of divergence that could be further investigated in the data:
Sampling differences: Differences can be related to sampling, when qualitative and quantitative data do not investigate the same target population and/or a subgroup. For instance, a subsample from a larger survey may yield different results due to its unique characteristics. Additionally, qualitative and quantitative methods usually have different sampling objectives. While qualitative research often seeks diverse representation, quantitative methods aim for a more generalized, representative view.
Difference in topics: If quantitative and qualitative methods do not measure the same topic, they are likely to lead to diverging results. Also, different interviewers may have treated a topic differently (for example more or less in depth). Even if the same topic is covered in the same depth, the format how a topic has been presented in the qualitative and quantitative method can affect the results. For example, if sensitive questions are presented in a face-to-face setting participants can be more likely to give a socially expected answer than in a more anonymous online setting.
Theoretical bias: Consider whether there is a biased theoretical framework guiding the research. It can be advisable to underpin mixed methods with a common conceptual framework (more details on this see Bazeley (2018)), yet some degree of flexibility is required to revise theories if the findings don’t align with the underlying framework.
Participant variability: Different backgrounds and interpretations of survey or interview questions among participants can yield varied results. Such variability is not necessarily negative; it can be beneficial if properly understood and managed. But researchers must need to understand what mechanisms lead to different interpretations - like reactions to sensitive questions - that influence participant responses. Reanalyzing data by specific characteristics, like sociodemographic variables, are likely to contribute to a better understanding of different interpretations.
Researcher perspective: Researchers’ philosophical beliefs and/or varying perception of the research purpose can lead to different interpretations. In this case, strategic integration (Bazeley, 2018) would be advisable to avoid unnecessary dissonance already from the beginning of the project. This would involve developing a thorough understanding of how each data source will contribute to the overall research question. All data are collected and analyzed with a clear purpose, subject to the broader research objective.
3 Summary and conclusion
In response to the limited use of mixed methods, this paper aims to illuminate opportunities and potential obstacles of mixed-methods research, provide a starting point to evaluators interested in pursuing mixed-methods approaches, and consequently promote their adoption in evaluations of financial education programs.
The first chapter explains why the use of mixed methods is advisable and presents the key design features and common designs of mixed-methods evaluations. Even when a mixed-methods approach is chosen, the full potential of the collected data is often not realized due to a lack of integration between qualitative and quantitative data and results. This issue arises partly because effective integration requires additional effort and partly due to the lack of practical guidelines on how to achieve integration.
Therefore, the second chapter introduces three techniques suggested in the literature: triangulation protocol, “following a thread”, and mixed-methods matrix. These techniques are demonstrated through their application in a financial education case study. While “following a thread” offers flexibility and depth for exploring particular themes, the triangulation protocol enhances validity and reliability through cross-verification, and the mixed-methods matrix excels in case-by-case analysis.
With mixed-methods approaches, evaluations can combine the best of both worlds - quantitative and qualitative research methods - by integrating approaches of data collection and analysis, allowing researchers to capture both numerical data and rich contextual insights.
4 Glossary
- Evaluation research: Research aimed at assessing the processes, outcomes or impact of a particular program, intervention or policy.
- Focus groups: A qualitative research method involving a small group of participants who engage in guided discussions facilitated by a moderator, aimed at exploring their perceptions, opinions and experiences regarding a specific topic.
- Impact evaluation: A type of evaluation research focused on assessing the effects, outcomes or impacts of a program, intervention or policy. It aims to determine the extent to which desired changes have occurred and the attribution of these changes to the program or intervention.
- Integration: The process of combining different research methods to achieve a common research goal, essential in mixed-methods research.
- Interviews: A qualitative research method where a researcher addresses questions to participants individually, aiming to gather in-depth information, insights and perspectives on a particular topic or phenomenon.
- Mixed-methods design: A research approach that combines qualitative and quantitative methods to gain a comprehensive understanding of a research topic.
- Multi-method design: A study that employs both qualitative and quantitative elements without significant integration between them.
- Process evaluation: A type of evaluation research focused on understanding the implementation, delivery and mechanisms of a program, intervention or policy. It examines how and why the program works (or not), the fidelity of implementation, and the contextual factors influencing outcomes.
- Qualitative research: Research focused on understanding human behavior and experiences through non-numeric data such as in-depth interviews or focus groups.
- Quantitative research: Research focused on numerical data and statistical analysis to understand patterns and relationships in data.
5 References
Alele, F., and B. Malau-Aduli. 2023. An introduction to research methods for undergraduate health profession students. James Cook University. Townsville. QLD. Australia.
Bamberger, M. 2012. Introduction to mixed methods in impact evaluation. In: Impact evaluation notes 3(3). 1–38.
Bazeley, P. 2018. Integrating analyses in mixed methods research. SAGE Publications Ltd. https://doi. org/10.4135/9781526417190
Bureau of Policy, P., & Learning. (2013). Technical note: Conducting mixed method evaluations. https://2017-2020.usaid.gov/sites/default/files/documents/1870/Mixed_Methods_Evaluations_ Technical_Note.pdf
Creswell, J. W. and V. P. Clark. 2017. Designing and conducting mixed methods research. SAGE Publications Ltd.
Creswell, J. W. and V. P. Clark. 2011. Mixed methods research. SAGE Publications Ltd.
Dawadi, S., S. Shrestha and R. A. Giri. 2021. Mixed-methods research: A discussion on its types, challenges, and criticisms. In: Journal of Practical Studies in Education 2(2). 25–36.
Dupin, C. M. and G. Borglin. 2020. Usability and application of a data integration technique (following the thread) for multi-and mixed methods research: A systematic review. In: International Journal of Nursing Studies 108. 103608.
Farmer, T., K. Robinson, S. J. Elliott and J. Eyles. 2006. Developing and implementing a triangulation protocol for qualitative health research. In: Qualitative health research, 16(3). 377–394.
Frechtling, J. A. and L. M. Sharp. 1997. User-friendly handbook for mixed method evaluations. Washington, DC: Division of Research. Evaluation and Communication, National Science Foundation.
Johnson, R. B., A. J. Onwuegbuzie and L. A. Turner. 2007. Toward a definition of mixed methods research. In: Journal of mixed methods research 1(2). 112–133.
Johnson, R. E., A. L. Grove and A. Clarke. 2019. Pillar integration process: A joint display technique to integrate data in mixed methods research. In: Journal of Mixed Methods Research 13(3). 301– 320.
Maxwell, J. A. 2016. Expanding the history and range of mixed methods research. In: Journal of mixed methods research 10(1). 12–27.
Moran-Ellis, J., V. Alexander, A. Cronin, M. Dickinson, J. Fielding, J. Sleney and H. Thomas. 2004. Following a thread–An approach to integrating multi-method data sets. In: ESRC Research Methods Programme. Methods Festival Conference. Oxford.
Morse, J. M. 1991. Approaches to qualitative-quantitative methodological triangulation. In: Nursing research 40(2). 120–123.
Niglas, K. (2004). The Combined Use of Qualitative and Quantitative Methods in Educational Research. Tallinn Pedagogical University Press. Tallinn.
O’Cathain, A., E. Murphy and J. Nicholl. (2010). Three techniques for integrating data in mixed methods studies. Bmj. 341.
Palinkas, L. A., S. J. Mendon and A. B. Hamilton. 2019. Innovations in mixed methods evaluations. In: Annual review of public health 40. 423–442.
Schoonenboom, J. (2023). The fundamental difference between qualitative and quantitative data in mixed methods research. In: Forum Qualitative Sozialforschung/Forum: Qualitative Social Research 24(1).
Uprichard, E. and L. Dawney. 2019. Data diffraction: Challenging data integration in mixed methods research. In: Journal of mixed methods research 13(1). 19–32.
1 Oesterreichische Nationalbank, financial literacy and culture division, theresa.lorenz@oenb.at. Opinions expressed by the author of studies do not necessarily reflect the official viewpoint of the OeNB, the or the Eurosystem. The author would like to thank Maximilian Zieser (OeNB), Katharina Felbermayr (OeNB) and Sandra Mauser (OeNB) for helpful comments and valuable suggestions. This paper is part of an evaluation series by the Oesterreichische Nationalbank and the Bank of Greece. The series will provide an overview on financial literacy in policy and research, a literature review addressing the effectiveness of financial education programs and guidelines for designing, implementing and analyzing comprehensive evaluation studies.
2 Studies that are purely qualitative or pure quantitative are called mono-method studies (Niglas, 2004).
3 Quantitative methods, like a randomized controlled trial and qualitative methods like in-depth interviews are described in detail in additional parts of the evaluation series.
4 This is a main difference to the case study method where a few cases are the main objective of analysis (Bazeley, 2018).