Panama: Evaluating the Attribution and Contribution of UNICEF LAC in high/middle income countries

Organization: UN Children’s Fund
Country: Panama
Closing date: 01 Jun 2018

UNICEF LAC Regional Office

Terms of Reference

Evaluating the Attribution and Contribution of UNICEF LAC in high/middle income countries

Level of experience: Junior [ ] 1-3yrs Medium [ ] 4-10yrs Senior [ X ] more than 10yrs

Location: LACRO, Panama

Language(s) required: Fluency in Spanish and English.

Closing date for applying to this consultancy: May 31, 2018

Estimated start date of this consultancy: 13 June 2018 Estimated end date: 13 December 2018

Background and Rationale

  • The majority of countries where UNICEF LAC functions are high-middle income countries. Country offices have strategically designed their country programmes focused on upstream work, often through a combination of policy and programme advocacy as well as modelling interventions and contributing to pilot projects to be scaled up, targeting institutional and behavior changes in societies towards better results for children. Country programmes are typically designed to create the most appropriate enabling environment, through supporting adequate child focused policies and programmes, and promoting legal frameworks laws in line with international standards, and improving learning and capacity of government institutions. Piloting and modelling are implemented in the field to create evidence and engage local partners to identify solutions for children to be replicated or scaled up. In addition, UNICEF is frequently invited to provide top-expert advice on child-related policies without a local or demonstrative project. For example, on social protection, child-focused budgeting, child protection legislation, systems strengthening, etc. This type of efforts is complex and highly demanding in terms of time, investment and capacity, considering that it usually requires an inter-sectoral approach to address key systemic barriers that impede the full realization of children’s rights. This remains a challenge to demonstrate results that can be directly attributed to UNICEF and measure the actual contribution of UNICEF to high level results.

  • Results for children is the main ingredient for the definition of UNICEF country programmes. For policy work, the time span of five years of a country programme is usually not enough to demonstrate results at the level that it is required. Implementing policy work is a long-term investment to achieve results at large scale and often the influence of different factors may overshadow the casual relationship between the actual achieved changes and the changes in the life of children.

  • A critical but frequently unanswered question remains – are these programmes actually achieving changes in children’s lives? Regular monitoring is often focused on reporting indicators at the output level, which aim at reducing bottlenecks and barriers, such as generation of evidence policy reports, or elaboration of a toolkits and guidance documents for policy makers, that are impeding institutional or behavioral changes. At the outcome level, results reflect contributions to strengthening systems, and results reports are focused on institutional changes, such as monitoring system in place, or a draft law/policy. Results also include the generation of models, which are the systematic representation of lessons learned and recommendations to scale up validated pilot projects to be used by government officials for national or sub-national programs design and implementation. Whether or not these outputs and outcomes are changing the lives of children as intended by the UNICEF strategic plan is not evident through the current monitoring and evaluation mechanisms in practice. Outputs (attribution) and outcomes (contribution) are part of a theory of change that in the long run is expected to change the lives of children which will eventually translate into children survive and thrive, all children learning, all children protected from violence and exploitation, a fair chance for all children, and all children live in secure environments.

    The Role of Evaluation

  • In line with the UNICEF’s Evaluation Policy (2013), the evaluation function in UNICEF LAC aims at helping to improve its performance and results, by supporting organizational learning and accountability.

  • UNICEF LAC seeks to enhance the strategic role of Evaluations to understanding the context in which advocacy, policy advice, and models and scaled up pilots can reach results at scale. Policy advocacy is present at all levels.

  • In the last two years, UNICEF LAC evaluations have been mainly focused on assessing interventions at the outcome and output level. Although, some evaluations had an indication of impact results, evaluation analysis of long term and sustained results has been restricted. There is limited evidence that can validate the hypothesis (theory of change) that the activities, outputs and outcomes of a UNICEF LAC intervention is reaching (or contributing to) long term and sustained changes that can have an impact. Even when evaluations have assessed national policies and programmes (jointly with Government counterparts), the contribution of UNICEF to impact level results has been hardly assessed.

  • Evaluating the attribution of specific interventions and its contribution of UNICEF at the impact level is of high importance, as UNICEF is accountable to bring results at scale and contribute to high level results described in the new UNICEF Strategic Plan, and ultimately to the Sustainable Development Goals. In the context of high/middle income economies, the contribution of UNICEF may even be more “diluted” leaving the question of the value-added role of the organization in these contexts. Evaluation, through adequate evaluation methods, can help clarifying this role, making the causal inferences clear, even for advocacy strategies, and identifying the linkages all the way from activity to output to outcome to impact (which in many cases is not linear as it is portrayed) – and therefore offering a response to the question are these programmes actually achieving changes in children’s lives?, What are the conditions that caused reaching those changes?.

  • Considering the nature of the upstream work in the Latin America and the Caribbean region in middle/high income countries, the evaluation methods package UNICEF LAC is looking for is threefold: i) to assess the attribution of the effect of pilot / demonstrative projects; ii) to assess the contribution of those pilots / demonstrative projects to scaled up interventions; and iii) to assess policy advocacy and technical assistance at high levels.

  • Evaluation methods such as those based on causal inference of Theory of Change (theory-based approaches to evaluation) can show the complete spectrum of how a single advocacy intervention and/or a combination intervention that includes advocacy, technical advice, and implementation pilots is making a difference in children’s lives – where other stakeholders are also contributing. Specific evaluation methods can assess the effects of different upstream components a combination of often non-linear actions directly attributable to UNICEF and others pitched as contribution as part of a broad spectrum of a policy advocacy work, where UNICEF is one in many other stakeholders.

  • As part of the results-based management approach to design, implement and report results, UNICEF requires programmes to use a “theory of change” to present the broad vision of what results will be achieved and how they will be achieved. The theory of change articulates the theories and assumptions that underpin the anticipated change process through which inputs and activities are converted to or lead to identify results at the output, outcome and impact level (Results-Based Management Handbook, UNICEF, 2017). To design a proposed methodology for this specific assignment, theory-based approached to evaluation, which use Theory of Change to assess with evidence whether and how an intervention contributed to observed results, would be prioritized.
  • Purpose

    11. UNICEF LAC aims at positioning evaluations as a critical tool to assess the contribution /attribution of UNICEF work to results at scale, especially in the context of high/middle income transitioning economies. UNICEF LAC is looking to improve organizational learning through an evaluation framework that can assess if an intervention has produced or influenced high level results and determine the role of UNICEF in these contexts.

    12. The evaluation framework intends to provide a guide for country offices and the LAC regional office (or other regional offices) to design, implement, and report results for children resulting from UNICEF LAC interventions, including one or a combination of these interventions: i) evidence generation through modelling and implementation of pilot projects, ii) policy advocacy of bring models to scale; and iii) Policy advocacy / technical assistance upstream work.

    13. The evaluation framework will include a technical evaluation note that includes a proposed methodology to assess the attribution/contribution of UNICEF. The proposed methodology will then be tested in a country office – in an area of work related to the LAC Regional Priorities. Specifically, one country office evaluation will be used as case study to test the proposed methodological framework to assess the attribution/contribution of UNICEF in a specific context. The case study will help be used to fine-tune the technical note.

    14. In addition, the framework will also include practical tools for applying the proposed framework. Practical tools will include sample ToRs/inception report components, checklist for a rapid evaluability assessment for applying the proposed methodology, and checklist to assess the technical soundness of the evaluation reports. Additionally, the evaluation framework will include costing the application of this methodology in the UNICEF LAC context and the identification of financial options/arrangements to fund these type of evaluations, considering financial limitations in the UNICEF LAC context

    Objective

    15. The objective of this assignment is to prepare a framework to de used to evaluate the attribution/contribution of UNICEF work in middle and high income countries in Latin America and the Caribbean Region. The framework will have a strong methodological focus, but at the same time realistic approach to work of UNICEF in the region.

    16. The evaluation framework will include:

  • Evaluation Technical Note: A proposed evaluation methodology for upstream work, including i) assessing pilot/demonstrative projects; ii) advocacy work to bring models up to scale; and iii) advocacy / technical assistance upstream work. The note will go beyond impact evaluations approaches to assess attribution/ contribution to impact level results: When to apply the methodology, what are the conditions needed to apply the methodology. The framework will be refined after its application in a country case study.
  • A case study for testing: Based on the theoretical, the proposed methodology will be tested/piloted in one country office evaluation. The case study will assess the three parts the evaluation framework (assessing pilot/demonstrative projects; advocacy work to bring models up to scale; and advocacy / technical assistance upstream work). The country office evaluation will be selected according to the country evaluation plan, capacity to participate in this process, and available resources. The topic of the evaluation will need to be one directly linked to one selected regional priorities. The evaluation will assess the contribution of UNICEF in a specific context. Lessons learned from this experience will allow the proposed methodology to be refined.
  • Practical tools: Sample ToRs/inception report components, components to be considered in the inception report checklist for a rapid evaluability assessment for applying the proposed methodology, and checklist to assess the technical soundness of the evaluation reports.
  • Costing: The work will include standard costing of the application of this methodology in the LAC context, within and outside the organization.
  • Management arrangements

    17. The assignment will be managed by the UNICEF LAC Regional Office, in coordination with the Office of Evaluation, and the strong involvement of one LAC country office participating in the case study. The final product will be a co-creation effort, and will be shared within the region and at the global level.

    18. This framework will be built collaboratively and consultatively with organizations and institutional bodies with demonstrated expertise on this topic, and strengthening collaboration with organizations and governments with similar work. UNICEF will partner with development organizations (e.g World Bank, Inter-American Development Bank, DFID), academic institutions (e.g. University of California-Berkeley), and governments with strong evaluation national capacity (e.g. Mexico), that can share evaluation frameworks used to assess upstream work for better results on the ground.

    19. A reference group will be established to act as a sounding board to the elaboration of this framework. The Reference Group is expected to provide inputs to the inception and draft reports. The reference group will be made of UNICEF LAC staff, staff at the Evaluation Office, and external Evaluation experts/advisor.

    Time Frame

    20. This work is expected to be implemented in six months.

    Payment

    21. The anticipated budget will be distributed as follows:

    Evaluation Technical Note (draft and final versions)

    30%

    Case study- Country Office evaluation (evaluation and lessons learned)

    40%

    Practical tools

    15%

    Costing

    15%

    Total

    100%

    Approval of Products

    22. The steering committee, led by the Regional M&E Specialist and Monitoring Specialist will review products delivered by the Consultant and will approve their final version. The consultant or team of consultants will inform the M&E Specialist and Monitoring Specialist of any remarks on the product, within five (5) working days after the product has been delivered.

    23. Products will be reviewed by the Reference Group. The Reference Group will provide feedback within two (2) weeks of receiving the deliverable. The consultant will then edit the deliverable as needed and will provide a revised version within one (1) week after receiving the feedback.

    Qualifications

    24. The required consultant or team of consultants (a team of two three people can be considered) will be a qualified professional with the following characteristics:

  • Excellent and proved knowledge of Evaluation methods and approaches, with especial focus on Theory-based evaluation methods
  • Previous experience with Evaluation Quality Assurance systems
  • At least 10 years of proven professional experience in designing and conducting evaluations
  • Excellent analytical and writing skills in Spanish and English
  • Excellent interpersonal skills with ability to work in a multi-cultural environment
  • Post-graduate University degree in subject related matters
  • Working knowledge in the LAC region
  • Familiarity with UNICEF programmes in middle and high income countries
  • Additional considerations

    26. UNICEF reserves the right to withhold all or a portion of payment if performance is unsatisfactory, if outputs are incomplete, not delivered of for failure to meet deadlines

    26. All materials developed will remain the copyright of UNICEF and UNICEF will be free to adapt and modify them in the future.

    To Apply

    27. Individual consultant or a small team of consultants (two or three) can apply for this work.

    28. Applicants will submit: i) a technical proposal, and ii) an economic proposal, as follows:

    A technical proposal should include:

    A CV and/or P11 for he individual or each of the team members

    Profile of the consultant or the team

    A brief technical proposal (5 pages max.) describing methods that will be considered for the work, methodology to complete each of the components of the assignment, timeline, and workplan.

    b. An economic proposal including a total fee to conduct this consultancy.

    29. Incomplete proposals (technical and economic) will not be considered.

    How to apply:

    UNICEF is committed to diversity and inclusion within its workforce, and encourages qualified female and male candidates from all national, religious and ethnic backgrounds, including persons living with disabilities, to apply to become a part of our organization. To apply, click on the following link http://www.unicef.org/about/employ/?job=512977

    Read more here:: RELIEFWEB CAREERS INT