July 18, 2016

My forthcoming book: Learn to speak Evalian”

I have been working for a while now on a book provisionally entitled Learn to speak Evalian”. This little post is just a place-holder with information about the book; I will update it when there is more to say.

I intend to publish it open-access, i.e. it should be primarily available online free of charge, perhaps with on-demand printing too. There will be a beta version and comments will be very welcome. It may even stay in a kind of permanent beta phase as I update and add to it.

If you are interested, drop me a line (steve AT pogol.net) and I will let you know when the first version is ready.

For the really curious, there is an alpha version in pdf and html.

The book is not meant as an introduction to monitoring and evaluation (M&E)1. It will make most sense to people who have some practical experience and who are familiar with the theoretical outline of more than one well-known approach to these topics such as the logical framework approach (LFA) or Outcome Mapping.

The book might be of interest for social scientists in general but I am writing in particular for monitoring and evaluation (“M&E”) professionals - people who are responsible for monitoring and evaluating the success of and processes within many different kinds of projects and programmes, for example in international development, education and so on.

What’s Evalian?

The book is disguised, in a light-hearted way, as an introduction to Evalian”, the fictional language of a fictional country, Evalia. Evalian is a language very like English in which M&E concepts can be more easily expressed than in English. In particular Evalians are more precise in the way they express Variables” and their possible (factual and counterfactual) Values”, and the way in which these can be linked up into more or less deterministic Mechanisms” which they describe with Theories” about those Mechanisms. So concepts like Effectiveness” and Impact” are quite easy to define and use in Evalian. What’s more, many of the key ideas of a wide range of evaluation approaches from Outcome Mapping to RCTs can be quite neatly expressed in Evalian.

As a bonus, well-formed Evalian can be pasted into the Theory Maker web app to produce corresponding diagrams.

Many of the key ideas are based on the theory of causal networks set out in Pearl (2000)2

So, this book will teach you to speak Evalian, the fictional language of a fictional country, Evalia.

Why a fictional language?

Because the Evalians manage to communicate with one another about things without the misunderstandings which plague English and (I assume) the other languages which M&E staff use today. They can recite a Theory of Change or a project proposal as if it was an amusing story or an inspiring poem, and no-one argues about what it means or the words it contains. They often conduct Randomised Controlled Trials, almost for fun, but they also love Outcome Mapping and many other approaches to evaluation; and they see no contradiction in this.

So, learning Evalian” helps us to introduce some words, concepts and conventions which we can use to think and write about projects and programmes from an evaluation perspective. Using these conventions, we can have a fairly standard way to communicate how we think projects work and what effects can be attributed to them.

So we will define some slightly new way of making causal statements, as well as some key words like Variable, Value of a Variable, and so on, for talking about these statements. We will do it in a way which is (I hope) compatible with most approaches in social science textbooks, but which is a bit different from them too:

  • Evalians, although they are skilled at mathematics, are very comfortable with fuzzy and vague formulations and are very good at drawing precise conclusions from vague premises; they do not feel they are being sloppy when they don’t express a Variable in terms of numbers
  • Evalians are particularly good at describing Differences between factual and counterfactual states of affairs; the concept of Difference” is very important in evaluation.

Preview of first section

One of the most convenient things about Evalian is that it is very like English, with just a few tweaks. When Evalians are speaking about evaluation matters - variables, efficiency, outcomes, etc - they use a special kind of intonation which they capture in writing by using a special grey background like this:

This sentence is in Evalian.

For everything else, Evalians just use English. They do use quite a lot of Evalian though, because they like to pass time in the long evenings talking about things like attribution and contribution, bless them.

Written Evalian should be pretty easy for non-native speakers to read and write. However, it can be difficult for non-native speakers to speak.

For example, this piece of Evalian:

Teacher skills
 Teacher presence on training course

says something like Teacher presence on training course contributes to Teacher skills”. The second line is indented by one space, and Evalians have a special kind of intonation for this which is difficult for non-native speakers to hear or say themselves.

In Chapter xx, we discuss a bit more how to speak Evalian. Right now, let’s concentrate on learning the written language. As unfortunately many Evalians are hard of hearing, they also use diagrams which exactly mimic statements in Evalian. We are grateful to them, because non-Evalians too often find diagrams easier to understand (especially with longer texts).

Another convenient thing about Evalian is that we English speakers can just use bits of it without learning the whole language. Hopefully this will lead, at least piecemeal, to better understanding between English speakers when speaking about things that matter in evaluation. So just as you can sparkle at a party (sometimes) by throwing in a bit of French, you can make a mark with your M&E colleagues with even just a few phrases of Evalian.

Theorymaker.info, a website which understands Evalian

The book is a companion to Theory Maker, at theorymaker.info, a (free and open-source) web app for quickly sketching out many different kinds of Theory of Change and other kinds of diagram like project theories and evaluation plans relevant to M&E.

Theory Maker speaks Evalian! At the Theory Maker website, you will see a text window on the left which you can type into. If you type in Evalian, a corresponding diagram is produced on the right.

Below most of the Evalian phrases in the book, you will see a diagram, the same one you would get at theorymaker.info if you typed it into the text window. In the book, you will see an edit” button below each diagram. If you are reading the electronic version of this book, when you click the button you will be taken to the website where the text will be conveniently pasted in for you. There, can play with the text to see how it works, and perhaps adapt it to your own needs.


  1. Sometimes M&E” is used to mean a relatively low-level function, namely the mere monitoring of projects and programmes, contrasted with the more illustrious discipline of evaluation. I make no such distinction and refer to both as M&E”.

  2. Causality: models, reasoning and inference. Cambridge Univ Press.

evalian book Evalian
May 18, 2016

Everything should be evidence-based - if only the evidence would make up its mind already

This question came up on an Evaluation mailing list and was forwarded to non other than Andrew Gelman, and for our purposes it can be boiled down to: what do we really know about financial motivation in organisations? Do we know enough to be able to say something like in this case use this reward system, it will bring you optimal results”?

Reading through the illustrious answers it seems clear that there are dozens of different theories and variations of theories, each with some research evidence, but nowhere anything like an evidence-backed consensus.

Policy, project and programme design, …. everything should be evidence-based - if only the evidence would make up its mind already.

Motivation and reward are comparatively easy to conceptualise and research. If we can’t get consensus here, how are we going to get consensus with hundreds of harder real-world problems like how much should cash programmes after a natural disaster target women? - single women? how much? family size? conditionality?”

How can we construct Theories of Change if there is such a lack of consensus about what leads to what?

twitter evaluation Theories of Change social research
May 16, 2016

Using tags on blot.im to get a custom rss feed e.g. for contributing to r-bloggers.

On my old blog, any posts tagged with R appeared in special rss feed with the tag R, which was picked up by r-bloggers. On this shiny new blog powered by blot.im, that doesn’t happen automatically. So if you are using blot and want to contribute to r-bloggers, follow along with what I did, thanks to a tip from the Maker of blot:

  1. Created a new feed adapting these instructions:
  2. In the design for my template, added a new view” which I decided to name rfeed.xml (.xml to ensure the proper content-type is used). As URL I specified /rfeed.rss.
  3. Pasted in the code; note the two mentions of tagged.R. (I had to post the code on github because it gets eaten by blot.) I tweaked the example given to include the body of the post as well.

Now as I tagged this post with R, it should appear in the feed. I could also have created a tag folder called [R]’ to hold such posts but it is enough just to tag posts with the R tag.

R socialmedia nerdvana



This blog by Steve Powell is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License, syndicated on r-bloggers and powered by Blot.
Privacy Policy
.