Evaluators – AIMS, Mathematical Sciences for Climate Resilience Program

African Institute for Mathematical Sciences (AIMS) Mathematical Sciences for Climate Resilience (MS4CR) Program

Mid-Term Evaluation Terms of Reference

1. INTRODUCTION

The African Institute for Mathematical Sciences (AIMS) was established in 2003 in Cape Town, South Africa. Since then, it has expanded across the continent and established six centres of excellence for mathematical sciences in South Africa, Senegal, Cameroon, Ghana, Tanzania and Rwanda. AIMS is a pan-African network for the advancement of STEM, offering post-graduate training, research, and public outreach & engagement. The AIMS Master’s in Mathematical Sciences program is at the core of AIMS – the program trains hundreds of talented young Africans each year and prepares them for leadership positions in education, research, government and the private sector. Research is also an integral part of AIMS with three AIMS Research Centres across the network and a growing AIMS Research Chair program. International researchers partner with local researchers and students at AIMS centres to pursue innovations in STEM and their application to society.

The AIMS Mathematical Sciences for Climate Change Resilience (MS4CR) Program is a six-year (2017-2023) CAD 20 million initiative funded by Canada’s International Development Research Centre (IDRC). The key objectives of the MS4CR program are to increase the contribution of African mathematical scientists in finding solutions to climate change-related challenges in Africa through training, internships, and research; and to support the consolidation of AIMS’ operations across the continent through strengthening existing AIMS centres. These will be achieved through five complementary interventions:

1) Master’s level training for 88 AIMS students with a focus on climate science;

2) Climate science research and grants for 3 research chairs, supported by 50 postgraduate researchers in addition to 16 small research grants;

3) Climate science fellowships for 20 women and funding to enable young women and girls to access opportunities such as STEM camps to learn about mathematics and climate science;

4) Internships for 100 students or alumni in the field of climate change; and

5) Consolidation of the AIMS network.

In keeping with the terms of the grant agreement, AIMS, in collaboration with IDRC, is commissioning an external and independent mid-term evaluation of the AIMS MS4CR program.

The mid-term evaluation is both an important learning and accountability exercise for AIMS to assess progress in the implementation of the MS4CR program, identify signs of early impact, note where results have been impeded, and adjust the delivery of the program going forward to ensure the intended objectives are met.

2. EVALUATION OBJECTIVES

The mid-term evaluation will be guided by the OECD DAC Criteria for Evaluating Development Assistance. The key objectives of the evaluation are to:

• Assess the progress made towards achieving the Program’s objectives and intended outcomes;
• Recommend where adjustments are needed for the next phase of implementation to ensure the realization of the expected results; and
• Consider the broader context in which the Program was implemented to identify the key factors that may have been barriers or facilitators and should be considered in the next phase of the program.

3. EVALUATION SCOPE

The evaluation will focus exclusively on the MS4CR program, from its inception in 2017 to present, within the broader context of the AIMS network, including training, research, internships, and operations supported by IDRC. Geographically, this will focus on AIMS Rwanda and the AIMS Global Secretariat, both in Kigali, Rwanda and phone or Skype interviews with other AIMS Centres in Senegal, Ghana, South Africa and Cameroon. The Evaluation Team is expected to survey AIMS Ms4CR students (current and alumni) to assess and document the perceptions and experiences of the program’s direct beneficiaries.

It is expected that the evaluation will focus on the immediate outcome results and will comment on the extent to which the MS4CR program is progressing towards higher-level outcomes. To measure this progress, the evaluation will be guided by the MS4CR program’s Performance Measurement Framework. The evaluation will also examine any changes in the context in which the program was implemented and the impact this has had on program implementation.

The mid-term evaluation will seek to capture information from a wide range of stakeholders, including:

• AIMS Executive Team, Senior Management, Academic Directors and relevant staff across the network;
• AIMS MS4CR lecturers and tutors;
• AIMS MS4CR students, interns, and alumni;
• External partners in the private sector, government, and education;
• AIMS Research staff;
• Employers of AIMS MS4CR interns and graduates; and
• IDRC and GAC staff.

4. EVALUATION METHODOLOGY AND APPROACH

The evaluation methodology should take a mixed methods approach (e.g. combining quantitative, qualitative, and participatory tools) to allow for the triangulation of information and assurance of accuracy. A gender equality lens should be applied to all analysis. The evaluation envisages a combination of desk review of all relevant program documents; field visits to AIMS Rwanda and AIMS Secretariat; key informant interviews with select staff, partners, and donors; and focus group discussions with students, lecturers, tutors, and researchers; and survey of AIMS alumni.

The proposed methodology should be presented in the Inception Report. It should include an evaluation matrix to demonstrate how each evaluation question will be answered (i.e. data source, collection tools, and method of analysis). The Inception Report will also outline the evaluation team’s overall work plan, the division of responsibilities between team members, and a risk assessment with mitigation solutions.

While this is broadly speaking the approach envisaged by AIMS, applicants are encouraged to be innovative and creative in their proposal and draw from their own expertise and experience, where relevant.

5. OVERSIGHT OF THE EVALUATION

The evaluation will be overseen by AIMS, led by the Director of Monitoring & Evaluation and the M&E Division. An Internal Advisory Committee, comprised of the Director of M&E, Vice President and Chief Programs Officer, IDRC’s Senior Program Officer in M&E, and IDRC’s Program Officer for the MS4CR, will be responsible for reviewing and approving all deliverables related to evaluation.

6. KEY EVALUATION QUESTIONS

Key questions have been developed to help guide the evaluation. They will be further detailed and refined during the inception phase:

a) Relevance

· To what extent is the MS4CR program at AIMS relevant to the development priorities of Africa, specifically in terms of higher education and research in climate change?

· Are the activities and outputs of the program consistent with the overall goal and objectives?

· Are the activities and outputs of the program consistent with the intended outcomes and impacts?

b) Effectiveness

· Have program activities been implemented as planned? Have they rolled out successfully and as intended? Is the delivery of the program on track?

· How appropriate are the implementation modalities?

· How effectively is the program managed?

· Do any activities need to be adapted to a changing context or emerging opportunities?

· To what extent are the program’s objectives likely to be achieved by the end-date of the program?

· How adequate are the M&E mechanisms in place? Are the indicators reflective of the objectives of the program? Are they being tracked and monitored effectively?

· What are the early signs, if any, that immediate and higher-level outcome results have been achieved?

· What are the key factors influencing the achievement or non-achievement of the objectives?

c) Efficiency

· Is the program being implemented in the most cost efficient means possible?

· Are the objectives likely to be achieved on time and within the resources allocated?

· How adequate are the available resources to deliver the program (human, financial etc.)

4. EVALUATION PHASES

4.1 Inception phase

· Kick-off Skype call with Evaluation Team and AIMS

· Desk review of relevant program documents

· Inception meeting at AIMS Secretariat in Kigali, Rwanda with AIMS staff

· Inception Report including proposed approach, methodology and data collection tools and outline for the final report

· Logistical arrangements for field visits

4.2 Data Collection

· Field visit to AIMS Rwanda and AIMS Secretariat, both in Kigali, Rwanda. The visit will include: Interviews and/or focus groups with select AIMS students, staff, lecturers, tutors, researchers, employers of interns, and key partners; and participatory observation of the AIMS academic model and 24/7 learning environment

· Survey of AIMS alumni from the MS4CR-funded climate science stream

· Interviews with IDRC and GAC staff

4.3 Analysis and Reporting

· Synthesis, analysis and triangulation of findings

· Follow-up interviews where necessary

· Submission of draft report to AIMS for review, consideration and feedback. The outline for the report will be discussed and agreed upon at the Inception Meeting.

4.4 Learning Workshop

· Evaluation team will be invited back to the AIMS Secretariat in Kigali, Rwanda to present their findings and proposed recommendations to AIMS staff. This will provide an opportunity for AIMS to review, discuss, and validate the findings and begin to prioritise the relevant recommendations.

4.5 Final Report

· Based on the feedback and insight gathered at the Learning Workshop, the Evaluation Team will revise the draft report and submit a final report of the mid-term evaluation.

5. PROPOSED TIMELINE

The following is a proposed timeline for undertaking the mid-term evaluation, to be discussed, reviewed and finalised with the selected evaluation team during the inception phase.

6. EVALUATION TEAM COMPOSITION AND SELECTION CRITERIA

To ensure the independence of the evaluation and the credibility of the findings, a team of external consultants, identified through an open and competitive selection process, will conduct the mid-term evaluation. The selected evaluation team must have experience in undertaking evaluations in the context of higher education in Africa. Preference will be given to those teams who have practical experience in STEM and particularly the field of climate change

The Team Leader must have extensive and broad experience in the design and execution of evaluations for international development, with excellent analytical, team management, and communication skills (verbal and written). He/she should be fluent in English and French. The Team Leader should be supported by one or two other evaluators with subject matter expertise in higher education, STEM and/or climate change. They must have strong and practiced skills in data collection, data analysis and report writing.