One of the great challenges of the kind of work I and others are doing is figuring out how to measure our progress. We often have fundors involved who want to know what our metrics are, but more importantly our communities themselves need to know if their efforts are effective. All to often our work is sidelined when someone says where’s your strategic plan? Many of us don’t believe in strategic planning. We say show me a plan that was actually followed and that was helpful. But, still, we don’t have much that we work with instead. Sure, we follow our intuition and we’re good at using conversation and other tools. But it feels insufficient.
And I am so tired of hearing if you don’t know where you’re going, you’ll never get there . What does it mean to work with making the path by walking on it? How do we get collective reassurance that we’re headed in the right direction and not just spinning our wheels? One insight I had a few years ago was that in our new ways of working, we measure from the inside out. Rather than having a fixed set of reference points (goals and objectives) constructed in advance, we’re actually called on to learn how to watch what is actually happening in order to understand the direction that is emerging from within our system and then we have to talk about what that direction means.
One methodology I’ve come across that works in this way is something called Most Significant Change. The link will lead you to a version of the user’s guide from a couple of years ago. I have some problems with the methodology in that it uses a somewhat elitist process to refine data — but I think it has a promising, simple, fresh start. You ask people to notice what is significant about changes and then look for patterns and trends in their responses.
Someone recently forwarded an interesting article from a 2006 NonProfit Quarterly Evaluation for the way we Work. Michael Quinn Patter, the author, begins by saying The very possibility articulated in the idea of making a major difference in the world ought to incorporate a commitment to not only bring about significant social change, but also think deeply about, evaluate, and learn from social innovation as the idea and process develops. However, because evaluation typically carries connotations of narrowly measuring predetermined outcomes achieved through a linear cause-effect intervention, we want to operationalize evaluative thinking in support of social innovation through an approach we call developmental evaluation.
I’ve also been collecting interesting pieces of this and that for the last several years while this question has been bubbling in me including:
- Learning for Social Change
- A Framework for Learning and Results in Community
- Organic Evaluation at Santerpol Roulant
- Introduction to Outcome Asset Impact Model
And many more. I’ve been collecting these for some time and finally have an opportunity to read and think about them. I’m very curious. What are others here finding helpful in terms of Evaluation and Measurement? What are you doing? What kinds of resources and documentation are helpful? What can you share here?
17 thoughts on “Evaluations and Outcomes: Working with Emergent Processes”
Bob, good to see you moving with new topics, questions and ideas.
From my recent dialogues (e.g. with Maria and Sarah) I can tell you that it will be long way to bridge the two perspectives, which we have been discussion in “how to align a business (investors) perspective with emergent “projects”. It first ended to keep those two still seperate as the span of the bridge would be too big, means invest in things with return and sponsor frames for emergent processes. One idea to combine both of it is to make the investor part of the process (producer/contributor) rather a sole consumer. A second one is to dig into the nature of money. Nobody told me, where it comes from, which energy it holds, most just get enoyed of it as a cumbersome mean or get freaked for it. Is there the need for a new form of currency (I do not mean just a local currency,…). Something new that helps exchange trust.
Beside this, I guess noone on the path would even post the question about the plan, etc. So the answer is only needed to bridge the two worlds. Within the emergent model you would ask a different question, e.g. how to find the resources needed for the situation, for the path ahead, rather fulfililling investors interests who are not really part of the process yet.
Best regards from cold Munich,
Hi Mark, thanks for stopping in! A couple of thoughts.
First, with anything, the more perspectives involved in formulating something, the more likely it is to resonate with people holding those perspectives. A friend of mine from Saybrook Institute, John Adams, uses the phrase “getting their fingerprints on it.” When people have their fingerprints on something, it can be theirs!
There are several people doing really excellent work around alternative monetary systems. “The Future of Money” by Bernard Lietaer even has a German edition: “Das Geld der Zukunft.” Thomas Greco’s “The End of Money and the Future of Civilization” is also excellent. “The Ecology of Money” (Schumacher Briefing, 4) by Richard J. Douthwaite and some of his other work is also very good.
Back to evaluation and measurement. I don’t know. Perhaps sometimes we just make it to complicated. Perhaps what’s needed is that in each effort, we have to pay attention here and explicitly name and describe the processes we will use to evaluate our progress. Again — fingerprints. In most of the systems I work in there is explicit clarity around intention (or, it would not be that hard to make it explicit). Operating from that field of intention, perhaps what is needed can be as direct as following a simplified version of the “U” Process: check-in with others about what they are thinking and feeling (sensing), check-in with those most involved (talk about it), have a conversation about whether or not things are going in the direction intended. Perhaps doing this on a regular basis — sensing and conversation — and making it clear that that’s the process is enough. I often think we just make things too complicated.
Looking forward to more conversation about this!
I like the idea to use the “intention” as base-line for measuring as this is the source for the new paradigm and way of working. This might work well together with the MSC suggested, using storytelling on the change people feel that has been made and seen. From my own donor experience I think this is than a good and intention rich feedback to a donor, feeling the impact of his/her donation. It’s though like buying a newspaper. I give money and get good stories back 🙂
Talking about investors, I’m not sure, if this fully works out as the investor (at least most of them today) would expect kind of a financial or materialistic return. A good story would not be enough. They also wanna benchmark one project to another to see where the investment brings more money,… With friends we developed a “return on donation” approach, where we try to measure the increase of local income assuming that whatever measures a project uses this is a visible effect at the end. Together and alongside with this you may use stories, etc. to evaluate the “how” you achieve this increase.
I agree with your “fingerprints perspective” but do not see the real involvement of a money-oriented investor. It is of course different if I contribute with my investment to buy some e.g. “learning center” land and have the right to pass my holidays there. But most investors I know, just wanna get a pecific return, even in sustainable investments, it’s just a different calculation on risk and re-payment stability, but I don’t see their real fingerprint in a sense that this would contribute to the path people take based on their intention. Yes, there’s kind of interaction but the influence given from this kind of fingerprint I do not see healthy to the intention-based process. For me it feels like a alarm clock kicking you out of the dreams and picture floating in your mind. I agree that the money perspective is important for a project to be considered, but the way we are doing this today I do feel of having a different energy, think of all the cumbersome discussions on fundraising as a duty to survive. Why don’t we find ways where this is part of the intentional-process and include it somehow in the flow and make these discussions much easier. However. Just some thoughts…
A college classmate and friend of mine had an very interesting perspective on this: Perhaps to “claim a framework of our own” we would be well served to abandon altogether the notion of goals (and metrics) in favor of the idea of direction (and reevaluation). If life is a journey, the real question must not be whether we have arrived, but where we are headed. The former is defined in terms of success and failure; the latter is all about choices. This seems very close to what you are saying, Mark. I want to think more about this, but in some ways both of you are pointing towards how the very nature and structure of measurements in one paradigm/worldview are both invisible and irrelevant in another.
Hello Good Men
Good to find you this morning. Sarah and I are driving to Greece and we are in Italy – just getting ready to set to Ancona to catch the ferry. I am dropping in right now because I wanted to offer the perspective that I found really interesting in our conversations with Mark. It is the perspective of shifting from customer or investor to participant. From my experience – when you participate – your perspective of evaluation is completely different. It is not a ‘white glove’ or ‘out of body’ way – but a personal, embodied action that is about moving one forward or about evolving to the next level. So the return can include a financial return – but you have been part of making this happen – not just receive it – and a of course the deeper return is the next level. The currency of exchange is broader – it includes belonging, learning, etc. as well as perhaps monetary return. And – what I speak of here – participant investors who ‘gives and gets’ – are still not easy to find. There is still a path to walk to create more fully in the manifest this form of investor. Much love to you both
Oh and I just wanted to add. We at Axladitsa are deeply involved in harvesting from the past three years – this is our form of evaluation – and we are beginning to use an action research approach – where identifying intention as the aspect that ‘pulls one into a direction’ and includes what ‘we’ as participant/researchers have been impacted on is a part of. I know that in action research – they have also been working with evaluation – from the perspective of having been part of setting intention and also living it.
Hi friends, great to see this conversation emerge – timely! Was recently hosting a reflection circle with some core investors in an initiative of mine and was honestly asking the same questions – about how they measure ‘return’ and what they have learned from investing in us so far and what they saw the ‘return on engagement’ to be. What I found interesting was the discernment between measuring the ‘past’ i.e. what has already happened, and measuring the progression towards an emerging future i.e. how what they are investing in is becoming more manifest, visible. Part of the measure they named was also the impact it has had on them personally in making shifts in their own lives, beliefs, etc about what is possible based on a different experience with us, based on being part of an emerging way of being in practice, in real life, that they weren’t sure they would see in their lifetime. Thanks for the links here, I like the most significant change approach and fine that corroboration of a story across a diverse and extended community is a useful indicator as well that a given ‘change’ has meaning.
Very interesting friends.
Measuring the progression towards an emerging future. I’m not sure if that quite works for me, Tatiana. Maybe it is a language thing. Perhaps consciously viewing and describing to others the path we are making as we walk towards that emerging future. I think this is what my old college friend was trying to say — perhaps the very construct of measurement is an entirely limiting one and we need a new construct.
I’m sure, Maria, that the great attention you have been giving to harvesting is a key part of this — it is a continual noticing. And I am sure that language which describes the ways in which we are connected — participants — is much more useful than the language which shows how we are separated — investors.
To me, part of what you say here is the key, Tatiana: Part of the measure they named was also the impact it has had on them personally in making shifts in their own lives, beliefs, etc about what is possible based on a different experience with us, based on being part of an emerging way of being in practice, in real life, that they weren’t sure they would see in their lifetime. I think this is near the essence. We’re looking for ways which help people see how critical it is for them as participants, from inside their systems to notice where they are and where they are going. One of the things I got clarity on a couple of years ago was that we needed to be looking for indicators which arise in the progress of our work rather than trying to determine them in advance. And part of this, of course, is the whole thing of emerging outcomes rather than pre-set ones. Stimulating thinking here.
Well, Robert, this is a “fine” topic you’ve picked out I must say. 🙂 Since I live in the area that is home to the Bill & Melinda Gates Foundation, which is driven by a desire to be able to measure the effects of their investments in the projects they have chosen, I have a somewhat reluctant, yet appreciative, interest in this topic.
I think our current metrics are more “technical” in nature and therefore not up to the task of effectively measuring our current complex/adaptive social evolution. Because if this assumption, I think my tendency has been to avoid the issue of measurement rather than engage in the work of helping to create/determine the kinds of metrics that let me (and others) know that the work I am doing is making a different that’s makes a difference.
I like your reference to the Theory U framework. Using that, I find myself wondering what is it that I/we pay attention to in the “rapid prototyping” stage that helps us discern what next steps to take following each experiment? It seems that this would be a good example of what to attend to/measure within our evolutionary process.
This of course raises the equally challenging question about what, if anything, is done with the data gathered in through the measurement process. What ensures that it is used, rather than places on a shelf to gather dust, and if used, is used in service of the common good rather than narrow/selfish gain?
I hope to be attending a presentation at the Foundation next week that is supposed to be exploring some of these questions, so good timing, Sir! :
To be continued…
Good to see this dialogue evolving with differnt kinds of directions, questions and thoughts.
One thought was around “semantics” and language, where I thought that we need long discussions just to make sure that we have the same understanding of something. May be we wanna try some of the emerging tools at MIT for dialogue mapping: http://globalsensemaking.wik.is/About_GSm/Existing_Sm_Tools/Collaboratorium
Another thought was around “representation”. With imersion and emergent processes we deal with intentions, feelings and emotions, which have to be shared by representing them in some way – making them explicit in words, pictures, gestures, etc. in order for others to catch ’em and relate to them. The same is with measurements. they are just abstract representations with the wish to encapsulate the whole truth about an impact, etc.
The tricky thing is that they loose perspective the more abstract, mathematical we make them. So, we might think of attaching them a “degree of abstraction” or a tag with a rout cause in order to track back from which intention they come from.
Is there a communication scheme for “intentions” yet ? Otherwise our measurement can only be done by feeling the intention and impact, which brings us back to the “participnt” approach. You can only measure, if you feel yourself.
“The water is 15 degree”. It’s 2 degree below average. But this mean that it’s too cold for you ? Do we need to measure just to communicate/justify because we do not trust each other ?
Hi Dan and Mark,
Cool autumn afternoon in Spokane.
So there is something in both of your entries about trust and purpose. It highlights the difference between evaluation and measurement that is used to prove one thing or another and indicators which are created to learn our way forward. Perhaps this is akin to what you are saying about representation and language, Mark.
Rapid prototyping. Hmm. View from the inside. View from the outside.
My mind jumps around.
>>I remember back in the seventies or eighties there was a project in the Northwest Region called the Flexible Intergovernmental Grant (FIG). A regional commission worked on behalf of the region and acted as a translator. The states in the region had things they wanted to accomplish. The feds had money. The FIG pooled the federal money so the states could do what was needed and reported the results back to the fed using their own metrics.
>>Then there is the story of the National Disabilities Services in Australia where NDS likewise acts as a buffer – figuring out what government needs to see in order to justify support and extracting those measurements, without producing those measurements being a primary focus in the work done
>>Or the unusual schools where students get very high marks on the various state competency tests because, in part, the schools ignore them and concentrate on engendering curiosity and creativity.
In these times of transformation it sounds like we need to walk in two worlds — carrying a constant mindfulness of the difference. On the one hand, we do need the assessments that help us discern right direction. Rapid Prototyping. On the other we need to make visible the progress being made with markers which are understandable by others — fundors, investors, and many times to the people who are transforming the systems themselves.
I think it is important to look at the promises made by the evaluator and the expectations of the stakeholders. If the intent is for emergence, then formative evaluation from the inside out makes sense. In cases where the intent of the investment in people and time is for specific results it is useful to have at least a dotted line to changes in workplace performance and organizational results.
In a recent evaluation of Open Space Technology (OST) training at a large health board, I used the hard edge of results based management to look at what is usually a “soft” subject. Everyone learned a lot about what could be expected from OST to inform future work.
Yup, I agree Hal.
In many ways this issue begins back at a point where a commitment is made to work with emergence (follow the energy of YES, make the path by walking it, etc), rather than working towards pre-determined outcomes. Sometimes, of course, the work towards specific outcomes is what’s needed and possible. But the means chosen for evaluation/measurement/learning needs to line up with the strategic approach. And, yes, I think that working with emergence is a strategic choice. Not sure I knew that before writing here today.
Would you be willing to post your OST report, Hal? You can should show the link in a post to where it can be downloaded (or send it to me and I’ll get it up here)
Re: the futility of “strategic planning” and the efficacy of “strategic design”
Strategic planning is usually neither, said Henry Mintzberg. And I agree. Most plans are merely tactical programs with no design framework.
With a strategic design structure, you set a clear, compelling vision of desired results, ground that vision in the current reality of the result (which is dynamic and changing) and then take action, exploring, experimenting, learning and making up the plan as you go.
This is how the act of “creating” almost anything unfolds. It allows one to commit to desire outcomes, accept and work with a changing reality and take action that honours both the vision and the reality.
It also allows for openness and emergence. Actions emerge from the creative tension between vision and reality, and often the learning that results causes you to expand or refine your vision, getting closer and closer to what you truly want to create through a series of experiments. Create and adjust…, create and adjust… and see what emerges along the way.
The difference between strategic planning and strategic design and creating is like the difference between taking the train and travelling by sailboat.
On a train, you know where you want to go, where you’re starting and all the stops in between. Works great, so long as the ground under the tracks remains firm and unchanging. Disastrous is the ground shifts!
On a sailboat, you know where you’re starting and where you want to end up, but you make up the route as you go — taking into account winds, tides, currents, storms, shipping and other circumstances. If you get blown off course, you note your new reality, recommit to your desired destination, and set a new course.
Strategic design is a far more powerful structure than strategic planning.
What I am starting to be more clear about, however, is the need to embrace both spirit and strategy. We need both. We need to be grounded in spirit and, sometimes we need strategic thinking and action to move forward. I agree completely with you in terms of the pitfalls of traditional strategic planning, Bruce. AND, I wonder what’s really required to work well with guidance and spirit? I once had a colleague say to me, with a bit of irritation “yeah, you’re intuitively strategic..” How do those two blend?
In the creative tension that arises between vision and current reality in strategic design/creating there is room for rational actions and intuitive actions. Indeed, creative tension forms a container – a possibility space – in which it is possible to integrate intuition and rationality.
And, of course, many visions of desired arise out of intuition, out of a gut feel for what you want to create. A kind of conception that is honed and focused over time, and with experience, into a clear and compelling vision.
What if we imagine that the process of planning for/responding to the future is a dance? What if the “partners” are the discipline of strategic planning and the imagination of emergent design? Perhaps then the question becomes more about the dancers’ artistic response to/expression of the music being played. And the natural tensions contained in the partners is in service of a robust relationship to the music, which is the formula of a dance. 🙂