Evaluation recent issues

Editorial

Stern, E. — Thu, 01 Oct 2009 03:40:22 PDT

Evaluating Service Organization Models: The Relevance and Methodological Challenges of a Configurational Approach

Thu, 01 Oct 2009 03:40:22 PDT

Based on the example of the evaluation of service organization models, this article shows how a configurational approach overcomes the limits of traditional methods which for the most part have studied the individual components of various models considered independently of one another. These traditional methods have led to results (observed effects) that are difficult to interpret. The configurational approach, in contrast, is based on the hypothesis that effects are associated with a set of internally coherent model features that form various configurations. These configurations, like their effects, are context-dependent. We explore the theoretical basis of the configuration approach in order to emphasize its relevance, and discuss the methodological challenges inherent in the application of this approach through an in-depth analysis of the scientific literature. We also propose methodological solutions to these challenges. We illustrate from an example how a configurational approach has been used to evaluate primary care models. Finally, we begin a discussion on the implications of this new evaluation approach for the scientific and decision-making communities.

Synthesizing Legislative Evaluations: Putting the Pieces Together

Klein Haarhuis, C.M., Niemeijer, E. — Thu, 01 Oct 2009 03:40:22 PDT

Laws are important vehicles for policy. They are generally complex and involve various interventions. Despite their pervasive presence and numerous evaluation studies, laws have not been a topic of meta-analytical interest among social scientists. As a result, we lack an overall picture of the type(s) of interventions involved in laws, of the chains of events these are expected to set off and of the outcomes. Nor do we know much about the contents and quality of legislative evaluations. In a first attempt to address this gap, we developed a method to review legislative evaluations inspired by the sociology of law and by the realist synthesis approach. The article presents the results of a review of 75 legislative evaluation reports, which were completed between 1998 and 2005 in the Netherlands. These reports address laws in a variety of policy domains, from justice to health care and from social affairs to education. First, we subjected the reports to a methodological quality assessment procedure, after which 59 reports of acceptable quality remained for inclusion in a synthesis. Among other things, our method revealed that laws very often contain (new) public management interventions directed at executive bodies, rather than regulatory or economic interventions directed at citizens and businesses. The chains of events to be activated by these interventions remain vague. Nevertheless, we were able to come up with a preliminary typology of these programme ‘mechanisms’, which can be used as input for future research. Our method also demonstrated that contexts affected these chains of events in a variety of ways.

Stakeholder Mapping as an Assessment Framework for Policy Implementation

Mehrizi, M. H. R., Ghasemzadeh, F., Molas-Gallart, J. — Thu, 01 Oct 2009 03:40:22 PDT

In this article we develop a ‘Stakeholder Mapping’ approach to ex-ante policy evaluation. The proposed framework helps to assess policy implementation activities by applying several tests to evaluate the completeness, non-redundancy, proper assignment and internal consistency of a policy design. We illustrate the method by applying it to the analysis of e-commerce policy in Iran. We conclude by assessing the strengths and weaknesses of the approach.

Using the Objective Hermeneutics Method in Policy Evaluation

Mann, S., Schweiger, J. — Thu, 01 Oct 2009 03:40:22 PDT

At the outset of a policy evaluation, it is often necessary to gather data from the few personnel charged with responsibility for the policy in order to better understand the policy and its objectives. The objective hermeneutics method provides a tool for deriving the maximum benefit from several core text sequences of such interviews. Two examples of interviews with administrative officials are provided: the first on the subject of investment subsidies for winegrowers and the second concerning subsidies for regional marketing initiatives. These are used to illustrate that the objective hermeneutics method reveals important thought structures and patterns of justification underlying policy design, which may prove highly useful in subsequent stages of policy evaluation.

The European Union's Drive towards Public Policy Evaluation: The Case of Spain

Vinas, V. — Thu, 01 Oct 2009 03:40:22 PDT

Spain’s entry into the European Union (EU), in 1986, represented a great challenge in several areas, including the evaluation of public policy. In 1988 Structural Fund regulations were amended to include mandatory evaluation of European structural policies by both the Commission and the member states, and the 1999 reform strengthened evaluation requirements. This contribution discusses the major changes introduced in the Spanish evaluation landscape due to Spain’s entry into the EU. It provides a description and analysis of a case that has elicited great interest among countries that have entered the EU in the 21st century and are trying to learn from the experience of Spain and other countries.

Evaluation Networking in Romania

Mihalache, R. — Thu, 01 Oct 2009 03:40:22 PDT

In the context of a growing evaluation community in Romania, networking in evaluation tends to be not only a topic for analysis and research, but mainly a practical tool for reaching the generic community of evaluation practitioners. Furthermore, this is a process requiring resources (time, dedicated people, information and funding) which are scarce. Therefore the better documented the process is, the higher the chances of achieving results. The process of networking in evaluation is of course influenced by the context in which it is taking place, mainly characterized by confusion around what evaluation really is and how it works and by confusion around what would be the roles and the benefits of networking in evaluation. In spite of these challenges there are some opportunities to justify this analysis, and probably the most relevant one is the interest of the members and stakeholders who realize the potential for the ‘soft’ but highly relevant contribution of networking to professional development in evaluation. In this context, the central authorities in Romania wanted to better understand networking in evaluation in order to identify how this process might be stimulated in the future, and the how Romanian Evaluation Association — EvalRom — came about.

Book Review: Ann Crabbe and Pieter Leroy, The Handbook of Environmental Policy Evaluation: London and Sterling, VA: Earthscan, 2008

Vedung, E. — Thu, 01 Oct 2009 03:40:22 PDT

Cairo 2009: Conference on Impact Evaluation in International Development

Thu, 01 Oct 2009 03:40:22 PDT

French-language abstracts

Thu, 01 Oct 2009 03:40:22 PDT

Acknowledgement of Referees

Thu, 01 Oct 2009 03:40:22 PDT

Editorial

Stern, E. — Thu, 25 Jun 2009 07:34:05 PDT

Conceptual, Theoretical and Practical Issues in Measuring the Benefits of Public Participation

Burton, P. — Thu, 25 Jun 2009 07:34:05 PDT

Among parliamentary democracies there is a widespread belief that above and beyond the occasional opportunity to vote, citizens should be allowed to participate in decisions that affect them. Governments at all levels are now going further and supporting more active forms of citizenship in which various decision processes are open to more public participation. While this principle may be widely accepted, the practice has remained remarkably free from empirical scrutiny. For something that is held to deliver a myriad of benefits, we still know little of the extent to which these are in fact delivered. This article addresses this gap by developing a framework for conducting more robust empirical scrutiny of participatory exercises. It does so at three levels: first by proposing a conceptual clarification of the perceived benefits of greater participation, second by considering some of the methodological challenges in designing more robust evaluative studies and finally by reviewing measures that might be used in practice to quantify benefits.

Exploring Stakeholder Engagement in Impact Evaluation Planning in Educational Development Work

Hart, D., Diercks-O'Brien, G., Powell, A. — Thu, 25 Jun 2009 07:34:05 PDT

This article presents a case study of engaging stakeholders in the early stages of an impact evaluation of educational development work in a UK university. The rationale for undertaking participative impact evaluation is outlined in relation to the national and local context. The aim is to contribute to wider knowledge about appropriate methodology to lead to a better understanding of change processes in learning and teaching. We outline how stakeholder engagement in evaluation in this context has been influenced by the Aspen Institute's `Theories of Change' approach, and how we interpreted and applied it in the context of a grant scheme for educational development work. Our experience is discussed in relation to previous learning about the application of the approach, in particular in the health sector. This has highlighted implications for future work, not least that it should be informed by a more appropriate theoretical framework for exploring the complexity of evaluation in this context.

How Legitimate and Justified are Judgments in Program Evaluation?

Hurteau, M., Houle, S., Mongiat, S. — Thu, 25 Jun 2009 07:34:05 PDT

The main function of program evaluations is to describe programs in order to generate judgments of value. To be considered credible, judgments should be both legitimate and justified. The research presented in this article posed the following question: do program evaluation practitioners generate legitimate and justified judgments? A meta-analysis of 40 program evaluation reports was carried out, which found that only 50 percent of the reports generated judgments. While these judgments seemed legitimate, they were rarely justified. However, the elements required to support legitimate and justified judgments were present in the reports in similar proportions, whether the reports generated a judgment or not.

Critical Connections between Participatory Evaluation, Organizational Learning and Intentional Change in Pluralistic Organizations

Suarez-Herrera, J. C., Springett, J., Kagan, C. — Thu, 25 Jun 2009 07:34:05 PDT

The current debate around the emergence of participatory approaches in evaluation practice suggests that participatory evaluation may be considered an organizational learning praxis, one which facilitates the development of a holistic process of intentional change. Through critical reflection on how participatory evaluation has been conceptualized, this article offers an overview of some of the contextual challenges encountered when using participatory evaluation to enable the creation of learning environments. Given the pluralistic nature of modern organizations and some contextual constraints, evaluators appear to have largely developed a more instrumental type of learning, which may, paradoxically, result in a significant source of resistance to intentional change. This article proposes a process of capacity building for evaluative research (CBER). This process offers a collaborative way of overcoming unforeseen resistance to intentional change by overcoming the challenges found in the relationship between participatory evaluation and organizational learning. The article concludes by suggesting some epistemological and organizational issues that evaluators should take into account when enabling the implementation of a process of CBER in pluralistic organizations.

Approaches to Participation in Evaluation: Some Conditions for Implementation

Plottu, B., Plottu, E. — Thu, 25 Jun 2009 07:34:05 PDT

Approaches to participation in evaluation rely on the principle of active participation by major stakeholders, including less organized groups, as fundamental to good evaluation practice. This process offers a number of advantages but implementation requires certain prerequisites. The goal of our article is to weigh up the advantages of participation and to examine the conditions necessary for approaches to participation in evaluation to achieve their objectives.

American Evaluation Association Conference: Context and Evaluation: Orlando, Florida Conference: 11--14 November 2009 Workshops: 9--11 & 15 November 2009

Thu, 25 Jun 2009 07:34:05 PDT

French-language abstracts

Thu, 25 Jun 2009 07:34:05 PDT

Editorial

Stern, E. — Thu, 09 Apr 2009 04:05:22 PDT

The Evidence Movement: The Development and Consequences of Methodologies in Review Practices

Hansen, H. F., Rieper, O. — Thu, 09 Apr 2009 04:05:22 PDT

The evidence movement and the idea of systematic reviews, defined as summaries of the results of already existing evaluation and research projects, have gained considerable support in recent years as many international as well as national evidence-producing organizations have been established. This article analyses how the idea is practised in the areas of health, social welfare and education and shows that evidence-producing organizations work differently. Some subscribe to the hierarchy of evidence, others to a typology of evidence. The consequences of these variations are discussed.

Using Logic Analysis to Evaluate Knowledge Transfer Initiatives: The Case of the Research Collective on the Organization of Primary Care Services

Brousselle, A., Contandriopoulos, D., Lemire, M. — Thu, 09 Apr 2009 04:05:22 PDT

Models that shift more responsibility onto researchers for the process of incorporating research results into decision-making have greatly gained in popularity during the past two decades. This shift has created a new area of research to identify the best ways to transfer academic results into the organizational and political arenas. However, evaluating the utilization of information coming out of a knowledge transfer (KT) initiative remains an enormous challenge. This article demonstrates how logic analysis has proven to be a useful evaluation method to assess the utilization potential of KT initiatives. We present the case of the evaluation of the Research Collective on the Organization of Primary Care Services, an innovative experiment in knowledge synthesis and transfer. The conclusions focus not only on the utilization potential of results coming out of the Research Collective, but also on the theoretical framework used, in order to facilitate its application to the evaluation of other knowledge transfer initiatives.

Evaluating the Quality of Italian Local Vocational Training Systems: Towards a Sustainable and Shared Self-Assessment Tool

Di Battista, G., Palomba, L., Vergani, A. — Thu, 09 Apr 2009 04:05:22 PDT

In Europe, the aim of achieving a knowledge-based society makes the quality of training systems a topic of crucial importance. Italy is also interested in working out strategies and tools for improving its training supply. To this end the Ministry of Labour asked a public research institute, the Institute for the Development of Workers' Vocational Training (ISFOL), to design a self-assessment tool (SAT) for evaluating the quality of regional training systems. The article tracks the logical and chronological course undertaken to design the SAT, bearing in mind the current conditions of Italian public authorities responsible for the programming, management and evaluation of vocational education and training policies. The SAT methodological choices and its design are described, as well as its application by all Italian central and northern regions. The article concludes by mapping out ways in which the SAT can be used and pointing out some still unresolved issues.

Network Evaluation from the Everyday Life Perspective: A Tool for Capacity-Building and Voice

Horelli, L. — Thu, 09 Apr 2009 04:05:22 PDT

Despite the rich literature on networks, publications on the evaluation of networks are scarce. The aim of the article is to present the core concepts of network evaluation in the context of local and regional development, and a case study in Finnish North-Karelia. It is argued that network evaluation from the everyday life perspective (NEELP) is a special case. It requires an integrative design and the building of a collective monitoring and self-evaluation system with a variety of enabling and traditional assessment tools. It contributes to the empowerment and capacity-building of individuals and groups of people who are involved in the co-creation of their contexts, while it also strives to recognize the complex systemic aspects of the environment.

Comparing Gross and Net Employment Outcomes from Structural Fund Programmes: Issues from the Mid-Term Evaluations of the Objective 1 Programme in West Wales and the Valleys

Munday, M., Williams, G. — Thu, 09 Apr 2009 04:05:22 PDT

This article examines the relationship between the gross and net employment outcomes from the largest EU structural funding programme in the UK, the Objective 1 West Wales and the Valleys programme. The article demonstrates the practical difficulties of moving from gross to net outputs. It is suggested that these difficulties are not always explicitly considered in EU programme evaluations. One result is that, in the UK at least, previous evaluations of the relationship between gross and net employment outcomes may have overestimated the additionality of the European structural funding element.

News from the Community: The EPOS International Conference: Sustainable Development in Policy Assessment: Methods, Challenges and Policy Impacts

Thu, 09 Apr 2009 04:05:22 PDT

French-language abstracts

Thu, 09 Apr 2009 04:05:22 PDT

Editorial

Stern, E. — Fri, 09 Jan 2009 06:39:23 PST

Realistic Evaluation and Case Studies: Stretching the Potential

K{oelig}nig, G. — Fri, 09 Jan 2009 06:39:23 PST

Embedded in the critical realist tradition, this article aims to explore the potentialities of the case study for evaluation purposes when complexity and specificity are moderate.Three objectives are pursued. First, it is stated that the focus of the evaluation effort is not necessarily the programme itself, but can be one premise upon which the programme is built. Second, taking into account Donald Campbell's contributions, it is asserted that, under favourable conditions, not only deterministic propositions, but also probabilistic ones can be rejected as highly improbable via case studies. Third, making use of the Context—Mechanism—Outcome model proposed by Pawson and Tilley, this article demonstrates the capacity of a critical case study to sustain theory building.

Desperately Seeking Regulatory Impact Assessments: Diary of a Reflective Researcher

Radaelli, C. M. — Fri, 09 Jan 2009 06:39:23 PST

This article provides some reflective thoughts on current research on regulatory impact assessment (RIA) in Europe.The narrative draws on the author's fieldwork diaries. It starts from the methodological problems of identifying and analysing regulatory impact assessment in the UK, Denmark, the Netherlands and Sweden.This leads to more fundamental theoretical questions about research design, the relationship between supply and demand of research and, ultimately, the aims of academic research in this area.The conclusions draw lessons for the emerging research agenda on better regulation.The main problem is not whether `real' RIAs exist or not, since this would be equivalent to trying to fit the amoeba-like RIA `substance' into our abstract`forms'.The interesting problems are: what are the functions of different approaches to assessment, how are they used and how can they improve governance?

Testing Methodological Guidance on the Conduct of Narrative Synthesis in Systematic Reviews: Effectiveness of Interventions to Promote Smoke Alarm Ownership and Function

Fri, 09 Jan 2009 06:39:23 PST

The objective was to assess the impact of new guidance on the conduct of narrative synthesis in systematic reviews of effectiveness, by means of a blinded comparison of guidance-led narrative synthesis against a meta-analysis of the same study data.The conclusions of the two syntheses were broadly similar. However, differences between the approaches meant that conclusions about the impact of moderators of effect appeared stronger when derived from the meta-analysis, whereas implications for future research appeared more extensive when derived from the narrative synthesis. These findings emphasize that a rigorously conducted narrative synthesis can add meaning and value to the findings of meta-analysis.The guidance framework provided a useful vehicle for structuring a narrative synthesis and increasing transparency and rigour of the process.While there may be risks with overinterpretation of study data, the framework, tools and techniques described in the guidance appear to increase the transparency and reproducibility of narrative synthesis.

Cooperation Networks and Innovation: A Complex Systems Perspective to the Analysis and Evaluation of a Regional Innovation Policy Programme

Russo, M., Rossi, F. — Fri, 09 Jan 2009 06:39:23 PST

Recent developments in innovation theory and policy have led policy makers to assign particular importance to supporting networks of cooperation among heterogeneous economic actors, especially in production systems composed of small and medium enterprises. Such innovative policies call for parallel innovations in policy analysis, monitoring and assessment. Our analysis of a policy experiment aimed at supporting innovation networks in the Italian region of Tuscany intends to address some issues connected with the design, monitoring and evaluation of such interventions. Combining tools from ethnographic research and social networks analysis, the article explores the structural elements of the policy programme, its macroscopic impact on the regional innovation system and the success of individual networks in attaining their specific objectives.This innovative approach allows us to derive some general methodological suggestions for the design and evaluation of similar programmes.

Developing a Culture of Evaluation in the Irish Public Sector: The Case of Education

McNamara, G., O'Hara, J., Boyle, R., Sullivan, C. — Fri, 09 Jan 2009 06:39:23 PST

This article is a case study of the emergence of an evaluation culture in the public sector and particularly in education in Ireland. It suggests that the emergence of this culture was strongly influenced by external bodies, particularly the EU and, to a lesser but significant degree, the OECD. It is further argued that the continuation of systematic evaluation is still dependent on external forces, since a commitment to evaluation as a tool of governance has not taken hold among key policy-makers in Ireland. However it is postulated that, notwithstanding its arguably insecure foundations, evaluation practice has moved beyond the confines of externally funded EU programmes, which saw its first introduction into Ireland. In recent years a broad quality assurance agenda within the public service and to an extent beyond has emerged.The article concludes by making the point that an evaluation culture in a particular country is hugely contextualized and influenced by the constraints of existing ideologies and relationships between different interest groups.Thus, in Ireland, in line with the corporatist and partnership-driven approaches to economic policy and industrial relations which have been dominant in recent decades, the form of evaluation which has emerged is consensual, collaborative and negotiated.

Evaluating Development Learning: The World Bank Experience

Lockheed, M. E. — Fri, 09 Jan 2009 06:39:23 PST

Evidence-based evaluation of adult learning (training and capacity development) programs is essential for guiding investment decisions regarding such programs. But standard practice for evaluation of training and development programs relies on participants' self-reports regarding the quality, effectiveness, and impact of such training. More rigorous evaluation methods — including reliable measures of learning; evaluation designs that include multiple methods of data collection, scientific sampling, and randomized control trials (RCTs); and multivariate analyses — are rarely found in the training and development literature, which compromises the utility of these evaluations.This submission presents methods for increasing the rigor of evaluation of adult learning events, describes how these methods were put to use in one organization, and — drawing on these evaluations — identifies the features of effective programs.

News from the Community

Fri, 09 Jan 2009 06:39:23 PST

French-language Abstracts

Fri, 09 Jan 2009 06:39:23 PST