Thursday, June 30, 2022

Using ParEvo to conduct thought experiments


I have just had an interesting conversation with a funding agency who have been developing some criteria to: (a)  help speed up the approval and release of funding in humanitarian emergencies, but (b) at same time minimising risk of poor use of those funds.

They think these criteria are useful but are not entirely sure whether those seeking funding will agree.  So they are exploring ways of testing out their applicability through a wider consultation process.

One way doing this, which we have been discussing, involves the use of ParEvo.org. The plan is that a group of participants representing potential grantees will develop a set of storylines which starts off with a particular organisation seeking funding for a particular humanitarian emergency. Then a branching structure of possible subsequent storyline developments will be articulated through the usual ParEvo process

After those storylines been developed there will be an evaluation phase, as is common practice now with most ParEvo exercises.  At this point the participants will be asked two generic types of questions ( and variations on these), as described below:

1.  Which of the criteria in the current framework would be most likely to help avoid or mitigate the problems seen in storyline X? (Answer=Description & Explanation) 

  • and if the answer is none, are there any other criteria that could be included in the framework that might have helped?

2.  Which of the storylines in the current exercise would have most benefited by criteria X in the current framework, in the sense of problems described there would have been avoided or mitigated. (Answer=Description & Explanation) 

  • and if the answer is none, does this suggest that the criteria is irrelevant and could be removed?
Postscript: One interesting thing about this type of thought experiment is that the theory (the proposed funding criteria) and the possible realities that they may be applied to (where the theories may or may not work there as expected) are constructed by different parties who are independent from each other.  This is not usually the case with thought experiments, and could be seen as a positive variation.

Stay tuned for if and when this idea flies, then soars or crashes


Courtesy https://xkcd.com/

For more on thought experiments, see Armchair science



Friday, June 17, 2022

Alternative futures as "search strategies"




When you read the phrase "search strategy' this may bring to mind what you need when you are doing a literature search on the Internet.  Or you may be thinking about different forms of supervised machine learning, which involve different types of search strategies.  For example in my Excel-based EvalC3 prediction modelling app there are four different search strategies that users can choose from, to help find the most accurate predictive model describing what combinations of attributes are the best predictor of a particular outcome.  Or you may have heard of James March, an organisational theorist who in 1981 wrote a paper called 'A model of adaptive organizational search ' where he talks about how organisations find the right new technologies to develop and explore.This is probably the closest thing to the type of search process that I'm describing below.

Right now I am in the process of helping some other consultants design a ParEvo exercise, in which recipients of research grants from the same foundation will collaboratively develop a number of alternative storylines describing how their efforts to ensure the uptake and use of the research findings takes place (and sometimes fails to take place) over the coming three years.  Because these are descriptions of possible futures they are inherently a form of fiction.  But please note they are not an attempt at "predicting" fiction.  Rather, they are more like a form of 'preparedness enabling ' fiction.

As part of the planning process for this exercise we have had to articulate our expectations of what will come out of this exercise, in terms of possible desirable benefits for both the participants and the foundation.  In other words the beginnings of a Theory of Change, which needs to be supplemented by details of how the exercise will be best be run in this particular instance, and thus hopefully deliver these results.

When thinking about reasonable expectations for this exercise I came up with the following possibilities, which are now under discussion:

1 Participants will hear different interpretations and views of 
  1. What other participants mean when they use the term "research uptake ' 
  2. What successful, and unsuccessful, research uptake looks like in its various forms, to various participants
  3. How the process of research uptake can be facilitated, and inhibited, by a range of factors – some within researchers control and some beyond their control.
2.  This experience may then inform how each of the participants proceed with their own work on facilitating research uptake

3. The storylines that are generated by the end of the exercise will provide the participants and the XXXX trust with a flexible set of expectations against which actual progress with research uptake can be compared at a later date.

So, my current thinking is that what we have here is a description of a particular kind of search strategy where both the objectives worth pursuing, and the means of achieving them, are both being explored at the same time, at least within the ParEvo exercise.  Though other things will also be happening after the exercise, hopefully involving some use of the ideas generated during exercise (see possibility 2)

There is also another facet of the idea of search strategies which needs to be mentioned here.  When search is used in a machine learning context it is always accompanied by an evaluation function which determines whether the search continues or comes to a stop because the best possibility has now been identified (a stopping rule, I think is the term involved).  So, in the three possibilities listed above the last one describes the possibility of an evaluation function.  Exactly how it will work needs more thinking, but I think it will be along the lines of asking participants in the prior exercise to identify the extent to which their experience in the interim period has fitted any of the storylines that were developed earlier, and in what ways it has and has not, and why so in both cases.  Stay tuned...