By Jonathan Mitchell, the Oxford Policy Management portfolio leader for financial and private sector development, and project director for the Decision Support Unit of the DFID private sector development programme in the Democratic Republic of the Congo. Note that the views and opinions expressed in this article are those of the author and do not necessarily represent those of his employers, donor organisations, or the programmes he works with. Jonathan’s blog post highlights some important points and insights surrounding the question of the effectiveness of market systems thinking approaches. In addition to these relevant insights from Jonathan, I would add that the whole discussion of evidence also has to recognize what is it that we want to learn or evaluate. Jonathan touches on this when he points out that market systems projects have different objectives from more traditional project approaches. Market systems approaches also frame the challenge very differently from traditional expert-driven technical fix approaches in the understanding that real and durable change has to come from within the local system, which has important implications on issues like attribution. Maybe the most important takeaway from Jonathan’s blog post that the development industry has to struggle with is the separation of the donor political economies’ realities with honest, objective debate and learning about the role and effectiveness of international development. ![]() In Hans Christian Andersen’s story The Emperor’s New Clothes, the monarch parades before his subjects in his new regalia and no one dares point out that he’s naked. It takes a child in the crowd to call out the charade. Looking at barriers to effective learning at the Market Systems Symposium recently, we took inspiration from Andersen’s story. Cultural and structural obstacles that frustrate meaningful learning develop over time in any field. Every one of us involved in delivering MSD programmes needs to put ourselves in the place of the child in the crowd from time to time. No, we do not think that the market systems approach is a charade. But the evidence base for MSD approach effectiveness is not strong enough. We can undoubtedly do better, but only if we begin to ‘call out’ some of the entrenched issues that are hiding in plain sight. Evidence matters because the market systems approach is increasingly visible in donor-funded private sector development, and also becoming significant in non-traditional social development sectors such as water and sanitation, health and education. In our discussions we asked ourselves two key questions:
Do we have robust impact evidence from which to learn? On the face of it, the suggestion that there is a lack of evidence of impact about the effectiveness of market systems approaches is absurd. Since the first explicitly market systems project, the FinMark Trust launched by DFID in South Africa in 2001, the BEAM Exchange Evidence Map has collected over 150 evidence documents on market systems interventions across 41 countries. Each of these documents reflects a large volume of monitoring data. The problem is not with the quantity of evidence, but rather with its quality. Few of these analyses meet the minimum thresholds of evaluation rigour. BEAM’s Evidence Review in 2019 reported clear signs of publication bias, and we cannot learn from mistakes that we hide. In addition, most of the evidence was commissioned by implementation teams – so not strictly independent. Few of the evidence base documents are ex-post impact evaluations, which is the most reliable way to assess the overall performance of market systems programmes. By 2019 only 14 percent of the Evidence Map comprised impact evaluations and external reviews. So it appears not much has changed since Ruffer and Wach’s review of M4P programme evaluations in 2013 reported that ‘evaluations reviewed here are generally weak’ in terms of considering systemic changes, data quality, triangulation practices, use of theories of change and consistency of units. Clearly there is a problem here. But do market systems projects perform any differently in this regard to other similarly complex development sectors? Part of the problem is obviously structural to the aid sector generally. The need to show that taxpayers’ money is being spent effectively generally does not sit well alongside a nuanced reporting of a complicated picture. In addition, the recipients of aid do not generally complain about poor services. However, we should recognise that monitoring the impact of a market systems programme is harder than, say, building a school with aid funds. Market systems projects deliver, in a tangible sense, very little beyond diagnosis, facilitation and monitoring. Results are delivered (or not) by entrepreneurs adopting business innovations or public officials changing regulations, over whom the project has limited direct control. Getting accurate impact evidence from market systems projects is even harder than for other types of aid projects. Improving accuracy is partly, therefore, a technical issue of applying counterfactuals to evaluations; taking the attribution of results more seriously; and undertaking ex-post evaluations. It also is related to a commitment to "serious monitoring" (internal, longitudinal etc.) and a decade of effort and experience has gone into developing the DCED’s Standard for Results Measurement which includes independent auditing of programmes’ results measurement systems. However, in my view, the root of the problem is located in incentive frameworks created by the political economy of aid. Under pressure from donors to report rapid and extremely high impact-level results, combined with light touch donor management of monitoring systems, project teams are incentivised to generate an optimistic view of their interventions. This tendency is only sharpened when payment by results modalities are used - where consultants’ payments are contingent upon the achievement of specific high-level results being achieved. The ICAI review of DFID’s private sector development work in 2014 gave an amber-red (meaning ‘performs relatively poorly’) score for its assessment of impact, linking this explicitly to the pressure to demonstrate results against measurable targets, rather than systemic change and broader growth and poverty reduction. In short, everyone is incentivised to pretend that the Emperor is wearing beautiful clothes. I do not think this situation is inevitable. We need inspirational people to create the space and environment where development practitioners are incentivised to tell the truth about the results of their interventions and to report failures as well as successes. This is not an easy task, but it is vital and it is possible How can we learn better? Assuming an environment is created that will generate sufficiently robust evidence to support learning, the question emerges – how can we learn better? We think this requires action at the cultural and institutional level. Even though humans are biologically wired to learn, institutional learning in development cooperation seems to be fragmented and owned by individuals. Many stakeholders recognise the importance of building the culture of learning but struggle to put it into practice. Happily we already have a pretty good idea from experienced MSD programme managers, about how to build high-performing teams with strong learning cultures. Donors also have a role in either promoting or inhibiting the learning culture. In general, the donor approach has been to out-source learning to consultancy firms and platforms run by external entities such as the donor-financed BEAM Exchange, DCED or MarketLinks. These online platforms perform a useful function in that they are a repository of evidence and can synthesise and evaluate this evidence with a degree of critical oversight. However, institutionally, we need to evolve from scattered independent evaluations and ad-hoc research about market systems topics into a robust and recognised field of learning that attracts independent researchers from different backgrounds. From this viewpoint our current repositories of knowledge and evidence are not ideal. Instead we should be looking to create an enabling environment for serious learning around market systems. First, market systems practitioners (implementing organisations) should make effective their demand for better evidence and knowledge by being prepared to pay for it. In this way the online platforms can create a sustainable revenue stream which is independent from the pressures that come from donor funding. Second, the distance between market system practitioner and academic worlds should be reduced. Market systems thinking has its conceptual roots in a respectable and currently vibrant academic critique of neo-liberal economics, drawing upon behavioural and evolutionary economics, and the science of complex adaptive systems. (See for example Cunningham & Jenal on systems change, or Raworth’s work on doughnut economics). There is a window of opportunity for the market systems world to establish links with academic institutions in both donor and recipient countries in order to establish the latter as repositories of market system thinking and application. For practitioners this solution offers an institutionalisation of knowledge which the private sector cannot replace. For academic institutions, engaging with market systems practitioners will yield a rich palate of empirical case studies and the promise of funding from a new source. In conclusion, we need to be honest with ourselves that while the Emperor has new clothes, they still look a bit threadbare at present. We all have a role to play in taking learning more seriously. Donors should engage with politicians to nudge the incentive frameworks that they are creating away from impact-level to outcome-level results. Implementing agencies should value their ‘results’ not just as a way to demonstrate the effectiveness of a specific aid programme but as a valuable input to a broader learning process. And academics should recognise that the MSD approach presents their institutions with an important opportunity to apply some of the most innovative thinking in a relevant and meaningful context.
0 Comments
By Dun Grover, Director of Monitoring, Evaluation and Learning, Transforming Market Systems Activity (TMS) Honduras, ACDI/VOCA In this blog, Dun highlights some important learning from his project’s efforts to apply more rigorous monitoring methods when tracking systemic change. A key point is that quantitative methods are valuable when tracking systemic change, but they have to be understood in the context of complexity. As Dun explains, there has been pushback on the use of quantitative methods when trying to monitor and learn about systemic change. While there is a valid argument against quantitative methods, the concern is more narrow and related to how many practitioners perceive such methods as providing absolute answers. For example, traditional approaches have applied such methods to come up with absolute yes and no answers related to project attribution. While we have learned that such ways of thinking are not valid from a systems thinking perspective, we also have to recognize that quantitative methods are important when applied properly. The blog lays out important considerations and insights into how to ensure quantitative methods add value when trying to gain insights into whether and how systems change. ![]() In Cape Town, I had the opportunity to share some of my experiences working with the Honduras Transforming Market Systems Activity and our market system diagnostic – more specifically, how we were attempting to measure several aspects of market systems, including resilience. One of the points that sparked dialogue at our Symposium session was the feasibility and usefulness of quantitative methods (versus qualitative ones) to measure market systems and their attributes. Since then, we have completed the market systems diagnostic. I encourage you to check out the dashboards and whitepaper here at http://cohep.com/sistemasdemercado/. Now that this process is complete, I’ve had the chance to reflect on the discussion from Cape Town. In addition to the interesting results which you can read about in the link, here are some of the realizations I’ve had: Quantifying helps inform management decision making, even if those measures are less precise. I think the challenge most people bump into when attempting to quantify complex systems is that is impossible to do so with the same level of precision that we can, say, measure yields. USAID’s definition for precision states that “data should have a sufficient level of detail to permit management decision-making.” It turns out that we can measure the number of shocks experienced and the pace of recovery of enterprises across a sample of enterprises in an industry within an acceptable margin of error. Though these may be proxies for systems-level change, they are very meaningful measures that inform adaptive decision-making for the project and its stakeholders. Data analysis brought to light several hidden features of Honduran market systems, such as the impact of certain shocks of enterprise performance, that have shifted activities and generated insights that have prompted qualitative inquiry. You should focus on the key variables and their directionality of change – you don’t need to explain it all. In statistics, R squared is the percent of variance explained by the model. 0 percent indicates the model explains none of the variance and 100 percent indicates the model explains all the variance. In modeling our data, we found relatively low R squares (note: this is not surprising for social sciences fields). Despite this, we also found 17 variables that were statistically predictive of systems performance results. These discoveries are potential levers for systems change. Although these quantitative models might not tell you whether pulling these levers will result in an 11%, 32.5% or even a 500% increase in system ‘performance’, we do have a stronger evidence base to inform our decisions around which levers to pull in which directions. Further, we know with a level of confidence that when we do, the system will materially change to produce more of the results we want now and likely into the future. How to cross-validate to avoid overfitting your models in order to reduce errors in your predictions. In developing a statistical model, you may develop a model that fits your data perfectly but doesn’t fit in the real world and leads you to make errors in your predictions. This statistical phenomenon is called overfitting. In ‘real-life’ overfitting is akin to when we try to generalize experiences from one situation to another and mistakenly apply variables that don’t belong in our understanding. In statistical analysis it is a standard practice to use validation methods to detect and remedy such errors. One of the methods we are applying to avoid overfitting is cross-validation. To do this, we are facilitating workshops with a set of enterprises to validate the measures, identify ones which may have been mistakenly included, and, further, to identify variables which we missed that we should try to measure the next year. Quantitative reasoning is integral to constructing knowledge of systems. A core feature of systems that doesn’t change is that our knowledge and understanding of systems must always change. Quantitative reasoning is a process and way of thinking that helps us construct knowledge about systems. Quantitative reasoning involves the collection and reinterpretation of data and subsequent revisions to models and theories based on new lines of evidence. In our market system diagnostic, we intend to adapt, drop, or replace indicators on an annual basis that do not prove statistically or materially predictive of target outcomes. The purpose of this process is to continue to improve the precision and fit of our measurement methods to Honduran market systems. Further, by engaging academia and the private sector in this process, we are strengthening a systems mindset oriented towards exploration and discovery among local stakeholders in constructing collective knowledge of Honduran market systems. We welcome you to contribute to this process of learning and adaptation. If you have recommendations of variables to include in the 2019 diagnostic or methodologies to model the data, please send them to politica.economica@cohep.com or dgrover@acdivoca.org. Please include any evidence and sources as to why this contribution can help us to better explain Honduras’s market system performance. Growth and Value Addition, Not Profit Motive, are Key to Market Systems Becoming More Inclusive7/23/2018 By: Mike Field, Senior Technical Advisor, EcoVentures International ![]() When working on private sector development it is often cited without question that profit motives are good. The thinking goes that a firm that makes a profit would have to make their customers happy in order to sell to them and as a result generate that profit. While this thinking on its surface makes sense, it is founded on a set of assumptions that when seen through various systemic lenses is quite flawed. Let’s start with a definition of profit. As a banker once stated in a training, profit is an opinion based on accounting rules. It is not a universal thing, but an evolved definition related to accounting and tax requirements. So in most cases, when people say profit they really mean gross margin or what is left of revenues once you subtract out costs. Even if we leave aside the detailed accounting definitions and only frame the discussion around a very general idea of profit meaning what a business person gets to take home, then we are still left with a range of assumptions related to how markets function. Understanding how markets function is where systems lenses have proven very valuable in uncovering important misguided assumptions about profit. Going back to the original statement that profit motives are good, the assumption is that markets and market forces universally reward business that add value through customer satisfaction and that companies that cheat customers are sanctioned effectively. This assumption in how markets reward and sanction firm behavior is at the core of the concerns that good systems thinking highlights. More specifically, in most countries where there is widespread poverty, market systems are biased in favor of more extractive behaviors like rent seeking, mis-information, fake products, adulteration, etc.. – i.e., tactics and behaviors that business people can do to extract revenues without providing value in return. In many cases such biases can become rooted in social systems resulting in a process of institutionalization where norms emerge reinforcing the behavior. For example, if we take a case of an ag input provider that sees an opportunity to begin selling fake seeds, how would the market sanction that input firm if:
The multiple important mechanisms required in a market to sanction poor behavior relative to value addition are not present so sanctioning is not effective. In this scenario, it is likely that the farmers would not be able to recognize that the seed seller is cheating them, and since competitors or third party organizations like media or consumer protection NGOs are not organized to advocate on behalf of the farmer and against the seed seller, it is unlikely for the seed seller to be sanctioned. More likely, the seed seller would be rewarded for cheating the farmers by gaining more revenues with lower costs. Further it would be rational for the seed seller in such market conditioned to think that maximizing profits would mean cheating more. In this case above, the profitability of the shop is not an indicator of adding value as the shop is specifically reducing value in the system – it is extracting value from the system not only in real financial terms in that it is taking financial capital in exchange of something that has no productivity value, but also because it is reducing the efficacy of social capital by further eroding trust. The systemic feedback via the lack of sanctions and rewards from having immediate higher margins would also reinforce that kind of behavior to be normalized, which is exactly what is observed in many developing country contexts. The result is that firms that are profit driven in systems that reward extractive business practices would, as a normal course of business, engage in behaviors (i.e., selling/buying tactics) that capture value at the expense of their customers/suppliers. Given that in international development where many if not most market systems are biased in favor of extractive business practices, profit is especially a poor indicator of positive change at firm or market system level. Even further, the greater the profit orientation of firms within such a system often suggests that the underlying incentives pushing extractive behaviors is deeply rooted in the system and will be difficult to change. If most firms are aggressively applying tactics to extract value from their suppliers, staff, and customers then that would be an indication of a system’s strong bias in favor of markets that have self-organized to capture resources (i.e., and not add value). It is important to understand that such biases are mostly likely related to a risk management strategy that accumulates resources as a means to deal with shocks/stresses. The table below provides specific firm and systemic patterns that indicate an extractive or a value addition orientation of a market system. A good market systems project will focus its efforts on catalyzing firms to see the value in behaving in ways that align with a value addition orientation. They would also try to catalyze systemic changes that also align with a value addition orientation. Simply assuming profit motive is good further a myth in how market work and why they are not providing appropriate benefits to a wide enough set of citizens. Systems thinking is critical to understanding the orientation of market systems and how to shift that orientation to being more value added. A good market systems project will focus its efforts on catalyzing firms to see the value in behaving in ways that align with a value addition orientation. They would also try to catalyze systemic changes that also align with a value addition orientation. Simply assuming profit motive is good further a myth in how market work and why they are not providing appropriate benefits to a wide enough set of citizens. Systems thinking is critical to understanding the orientation of market systems and how to shift that orientation to being more value added.
|
AuthorsThe MSDHub Blog Series is authored by respected implementers and donors of market systems projects globally. Archives
December 2020
Categories
All
Highlights |