Saturday, May 18, 2019

Evaluating innovation...

Earlier this week I sat in on a very interesting UKES 2019 Evaluation Conference presentation "Evaluating grand challenges and innovation" by Clarissa Poulson and Katherine May (of IPE Tripleline).

The difficulty of measuring and evaluating innovation reminded me with similar issues I struggled with many decades ago when doing the Honours year of my Psychology degree, at ANU. I had a substantial essay to write on the measurement of creativity! My faint memory of this paper is that I did not make much progress on the topic.

But while listening to this weeks presentation I through there were some ideas that could be usefully borrowed from work I am currently doing on the evaluation and analysis of scenario planning exercises. I made a presentation on that work in this year's UKES conference (PowerPoint here).

In that presentation, I explained how participants' contributions to scenarios developed in the form of storylines could be analyzed in terms of diversity. More specifically, three dimensions of diversity, as conceptualised by Stirling (1998):

  • Variety: Numbers of types of things 
  • Balance: Numbers of cases of each type 
  • Disparity: Degree of difference between each type 
Disparity seemed to be the hardest to measure, but there are measures used within the field of Social Network Analysis (SNA) that can help. In SNA distance between actors or other kinds of nodes in a network, is measured in terms of "degree", i.e. the number of links between any two nodes of interest. There are various forms of distance measure but one simple one is "Closeness", which is the average distance between a node in a network and all other nodes in that network. The inverse of closeness is distance. 

One aspect of innovation is newness, as in difference from previous products or service of its type. While inventors can usually explain how a product is different from other products, measuring that difference seems more challenging. 

But one way forward, which occurred to me earlier this week, would be to ask the inventor/owner of an innovation to identify what other product, in a particular population of products, their product was most similar to. All other unnamed products would be, by definition, more different. Repeating this question for all owners of the products in the population would generate an "adjacency matrix", where a cell value (1 , 0) tells us whether a row item is seen as most similar to a column item (1) or not (0). Such a matrix can be visualised (with available software) as a network structure, and closeness values can be calculated for all nodes in that network. Some nodes will be less close to other nodes, than others. That measure is a measure of their difference or "disparity"

The advantage of a network visualisation is that as well as identifying how different a given product was, it would also be possible to identify any clusters of products that were more similar to each other but more different from others.

Here is a simulated example. The blue nodes are the products. Larger blue nodes are, on average, more distant i.e. more different, from all the other nodes