By Dun Grover, Director of Monitoring, Evaluation and Learning, Transforming Market Systems Activity (TMS) Honduras, ACDI/VOCA
In this blog, Dun highlights some important learning from his project’s efforts to apply more rigorous monitoring methods when tracking systemic change. A key point is that quantitative methods are valuable when tracking systemic change, but they have to be understood in the context of complexity. As Dun explains, there has been pushback on the use of quantitative methods when trying to monitor and learn about systemic change. While there is a valid argument against quantitative methods, the concern is more narrow and related to how many practitioners perceive such methods as providing absolute answers. For example, traditional approaches have applied such methods to come up with absolute yes and no answers related to project attribution. While we have learned that such ways of thinking are not valid from a systems thinking perspective, we also have to recognize that quantitative methods are important when applied properly. The blog lays out important considerations and insights into how to ensure quantitative methods add value when trying to gain insights into whether and how systems change.
In Cape Town, I had the opportunity to share some of my experiences working with the Honduras Transforming Market Systems Activity and our market system diagnostic – more specifically, how we were attempting to measure several aspects of market systems, including resilience. One of the points that sparked dialogue at our Symposium session was the feasibility and usefulness of quantitative methods (versus qualitative ones) to measure market systems and their attributes.
Since then, we have completed the market systems diagnostic. I encourage you to check out the dashboards and whitepaper here at http://cohep.com/sistemasdemercado/. Now that this process is complete, I’ve had the chance to reflect on the discussion from Cape Town. In addition to the interesting results which you can read about in the link, here are some of the realizations I’ve had:
Quantifying helps inform management decision making, even if those measures are less precise. I think the challenge most people bump into when attempting to quantify complex systems is that is impossible to do so with the same level of precision that we can, say, measure yields. USAID’s definition for precision states that “data should have a sufficient level of detail to permit management decision-making.” It turns out that we can measure the number of shocks experienced and the pace of recovery of enterprises across a sample of enterprises in an industry within an acceptable margin of error. Though these may be proxies for systems-level change, they are very meaningful measures that inform adaptive decision-making for the project and its stakeholders. Data analysis brought to light several hidden features of Honduran market systems, such as the impact of certain shocks of enterprise performance, that have shifted activities and generated insights that have prompted qualitative inquiry.
You should focus on the key variables and their directionality of change – you don’t need to explain it all. In statistics, R squared is the percent of variance explained by the model. 0 percent indicates the model explains none of the variance and 100 percent indicates the model explains all the variance. In modeling our data, we found relatively low R squares (note: this is not surprising for social sciences fields). Despite this, we also found 17 variables that were statistically predictive of systems performance results. These discoveries are potential levers for systems change. Although these quantitative models might not tell you whether pulling these levers will result in an 11%, 32.5% or even a 500% increase in system ‘performance’, we do have a stronger evidence base to inform our decisions around which levers to pull in which directions. Further, we know with a level of confidence that when we do, the system will materially change to produce more of the results we want now and likely into the future.
How to cross-validate to avoid overfitting your models in order to reduce errors in your predictions. In developing a statistical model, you may develop a model that fits your data perfectly but doesn’t fit in the real world and leads you to make errors in your predictions. This statistical phenomenon is called overfitting. In ‘real-life’ overfitting is akin to when we try to generalize experiences from one situation to another and mistakenly apply variables that don’t belong in our understanding. In statistical analysis it is a standard practice to use validation methods to detect and remedy such errors. One of the methods we are applying to avoid overfitting is cross-validation. To do this, we are facilitating workshops with a set of enterprises to validate the measures, identify ones which may have been mistakenly included, and, further, to identify variables which we missed that we should try to measure the next year.
Quantitative reasoning is integral to constructing knowledge of systems. A core feature of systems that doesn’t change is that our knowledge and understanding of systems must always change. Quantitative reasoning is a process and way of thinking that helps us construct knowledge about systems. Quantitative reasoning involves the collection and reinterpretation of data and subsequent revisions to models and theories based on new lines of evidence. In our market system diagnostic, we intend to adapt, drop, or replace indicators on an annual basis that do not prove statistically or materially predictive of target outcomes. The purpose of this process is to continue to improve the precision and fit of our measurement methods to Honduran market systems. Further, by engaging academia and the private sector in this process, we are strengthening a systems mindset oriented towards exploration and discovery among local stakeholders in constructing collective knowledge of Honduran market systems.
We welcome you to contribute to this process of learning and adaptation. If you have recommendations of variables to include in the 2019 diagnostic or methodologies to model the data, please send them to email@example.com or firstname.lastname@example.org. Please include any evidence and sources as to why this contribution can help us to better explain Honduras’s market system performance.
Co-Authored by Anoushka Boodhna, Consultant, EcoVentures International &
Devi Ramkissoon, Acting Division Chief, USAID
In this post, Devi and Anoushka start an important evidence-based discussion about why diversity and inclusion are central to addressing complex challenges. Central to the rationale for taking a market systems approach is that market systems are complex systems, which means that they are dynamic, evolving systems that are influenced by many factors all at the same time. Because there are many factors influencing a market system at one time, traditional approaches that assumed away all the factors except for one technically solvable factor have not worked. As donors have become more comfortable with recognizing that development is complex, they have also had to realize that the process for addressing complex challenges is different to traditional approaches. Specifically, complex challenges need teams that can engage in a learning and creative problem-solving process that emerges of overtime. Organizationally, best practice for such challenges includes teams that are diverse and inclusive of as many perspectives as possible. Diversity and inclusivity are not only a nice thing to have, but are necessary characteristics of teams that are effective at generating results in complex environments.
In this blog series, we examine the ways in which the concepts of diversity and inclusion apply to market systems programming. We hold that diversity in complex adaptive systems is one signal of a healthy market system. To this end, embedding these principles in programming will contribute to successful market systems activities whose results will ultimately be more sustainable in the long run.
The concepts of diversity and inclusion are increasingly becoming integrated into workforce management across the public, private, and non-profit sectors. According to George Washington University (GWU), the term “diversity” is commonly used to describe “individual differences (e.g., life experiences, learning and working styles, personality types) and group/social differences (e.g., race, socio-economic status, class, gender, sexual orientation, country of origin, ability, intellectual traditions and perspectives, as well as cultural, political, religious, and other affiliations) that can be engaged to achieve excellence in teaching, learning, research, scholarship, and administrative and support services.” GWU defines inclusion as “the active, intentional, and ongoing engagement with diversity -- in people, in the curriculum, in the co-curriculum, and in communities (e.g., intellectual, social, cultural, geographical) with which individuals might connect.” Diversity and Inclusion (D&I) are now seen as a crucial aspect of increasing productivity for organizations, no matter what their goal.
Diversity is an important characteristic of a healthy market system. In particular, there are three ‘capacities’ in market systems that are bolstered or enhanced as a system becomes more diverse:
Within international development programming, D&I is integral when working in market systems and navigating complexity. A diverse and inclusive team is more likely to bring about the variation needed to catalyze market system change through individual and social differences in perspectives, viewpoints, and ideas. When aligned around core principles such as poverty alleviation, they are better able to support market actors to explore different pathways for business growth and inclusion as well as generate innovation where needed. A diverse and inclusive team is also more likely to support market actors to navigate risk in constantly changing environments by thinking ‘out-of-the-box’ when observing signals, making adaptations, and finding creative ways to solve problems.
From program design through implementation, learning, and adaptive management, and monitoring and evaluation, diversity in staffing and perspectives can be invaluable to the market system change. For example, by cultivating an inclusive, diverse team and hiring staff with diverse backgrounds, one program in Kenya was able to advance its goals. Specifically, the program hired a marketing expert to support inputs distribution strategies for smallholder farmers. Not only did she fit a diversity profile as a young Kenyan woman, she also was from the private sector and introduced new cross-cutting interventions in ICT and media, which became a new space in MSD and agriculture. She also linked the program to a young, creative, tech startup scene in Nairobi. This is just one of the ways that programming responds positively to applying D&I concepts to MSD.
In Part 2 of the Diversity and Inclusion Blog Post Series, we’ll dig deeper to explore how D&I plays out at some of the key junctures of MSD program design and implementation.
The MSDHub Blog Series is authored by respected implementers and donors of market systems projects globally.