Annual Outcome Study - Terms of Reference

Foundations for Health and Empowerment (F4HE) Annual Outcome Study:

The Aga Khan Foundation (AKF), an agency of the Aga Khan Development Network (AKDN), is a private, non-denominational, international development agency established in 1967. It seeks sustainable solutions to long-term problems of poverty, hunger, illiteracy, and ill-health with special emphasis on the needs of rural communities in mountainous and other resource poor regions, primarily in Asia, Africa and the Middle East. Its main areas of focus are education, health, rural development, environment and strengthening civil society. 

The Aga Khan Foundation (Pakistan) [AKF (P)] is seeking a high caliber professional consultant to conduct field work in Gilgit Baltistan and Chitral. This is a short-term consultancy for 8-12 weeks (October 2024 to November 2024). The beginning date will be mutually agreed upon between the consultant and AKF (P).

1. Background of the Project: 

Foundation for Health and Empowerment (F4HE) is a multi-year initiative being implemented in selected areas in Afghanistan, Kyrgyzstan, Pakistan, and Tajikistan with funding from the Global Affairs Canada and the Aga Khan Foundation Canada. F4HE was developed to strengthen delivery of quality, gender-responsive, and inclusive health, early childhood development, and other sustainable development services; reduce gender and social barriers to utilization and uptake of health, early childhood development, and other sustainable development services and practices in select areas of Asia by women and girls, adolescents, men, and boys; and enhance engagement of stakeholders in gender sensitive and evidence-based development issues and programming in target geographies. This five-year initiative has been developed in the context of AKDN’s long-term presence, indigenous recognition, and extensive familiarity in the region. This project seeks to coordinate, align, expand and enrich the current activities of the various AKDN agencies in the region, in close collaboration with the local, provincial and national governments. 

F4HE reach and serve rural and marginalized communities in project geographies where sexual, reproductive, maternal, neonatal, child and adolescent health (SRMNCAH) and nutrition indicators are particularly poor and where non-communicable diseases (NCDs) like breast and cervical cancer and mental health issues are rapidly increasing. And, through ECD Programming it enables parents, caregivers, and health care providers to support and promote child development holistically – socially, emotionally, cognitively, and physically. The integrated programming approach is aligned with the health and education sector policies and strategies of Pakistan.

AKF Pakistan is now commissioning an annual outcome assessment of the project to determine if F4HE is indeed contributing to enhanced access to high-quality and gender responsive health and early childhood development services in designated healthcare facilities and schools. This assessment aims to measure improvement in service delivery within the healthcare sector and quality of the teaching and learning environment offered by teachers in comparison to the project's initial benchmarks and relevant existing information. The valuable insights and recommendations obtained from this assessment will be instrumental in strengthening program interventions and delivering higher-quality services to the targeted communities. 

2. Study Objectives

The overarching aim of the annual outcome assessment is to gauge the project's performance against the predefined essential immediate and intermediate outcome indicators outlined in the project PMF.

It aims to achieve three specific objectives:  

• Assess annual progress against the project's outcome-level indicators under the Foundations for Health (F4H) and Foundations for Children (F4C) components of the project.
• Identify successes and challenges, informing data-driven decisions on project adaptations. 
• Measure performance of relevant Government officials and departments in delivering sustainable health and ECD services.

Scope of Study

• The annual outcome assessment will cover targeted areas in 12 districts in two provinces where F4HE activities are implemented-ten districts in Gilgit-Baltistan (GB) province, Ghizer, Gilgit, Hunza, Nagar, Astore, Diamer, Skardu, Kharmang, Ghanchee, Shigar and two districts in Khyber Pukhtunkhwa (KPK) province, Lower and Upper Chitral district. 
• The F4HE annual outcome assessment has two components: Health Facility Assessment (HFA) and Education/ECD Facility Assessment (Schools and ECDRC). This assessment is designed to compare baseline and annual assessment findings, evaluating project activities’ effectiveness in achieving the targeted annual goals.

Study Framework

Data Collection Tools and Methodology

To ensure standardization, baseline methodology and tools will be adopted for annual evaluation. The list of indicators, data collection tools and guidelines for analysis and reporting is already available and will be provided to the consultant. Specifically, the following customized tools will be employed to assess related components of the F4HE project.

• Health Facility Assessment (HFA)
• Teach ECE
• ECD Facility Client Exit Interview
• ECD Enrolment Data Collection Tool
• Government Performance Index

Health Component Assessment Method:

During the baseline study, a total of 39 health facilities were assessed within the project intervention districts. Also, a total of 90 healthcare providers (healthcare staff) were assessed for the purpose of determining project baseline information. The annual assessment plan will include all facilities assessed during baseline. If a facility assessed at baseline is not accessible, replacement facilities would be randomly selected to complete the sample. Along with this, 2 health workers per Health Facility or number reached at baseline (whichever is greater) will be selected using a random sampling technique (if more than 2 staff are present) for assessment on KAP module of the tool. Due consideration will be made to ensure appropriate gender representation.

Further, a total of 609 clients from health facilities were evaluated during the baseline study, with 15 clients assessed at each facility. Ideally, the annual assessment will aim to include a comparable number of clients, or the number available at that time, utilizing a random sampling method if there are more clients present at the time of evaluation. Due consideration will be made to ensure gender representation.

ECD Component Assessment Method:

During the baseline study, a total of 33 Early Childhood Development (ECD) facilities (Phase I schools) were assessed in the project intervention districts. In the third year of implementing the project, additional ECD facilities(schools) were introduced into the project's population framework, marked as Phase II for ECD intervention. In light of this, the annual assessment will include the evaluation of 33 ECD facilities from Phase I which were assessed during baseline and Y3 annual outcome assessment and an additional 66 ECD facilities assessed during Y3 annual outcome assessment from Phase II. If a facility is inaccessible in either phase geography, replacement will be randomly selected from same district out of the available list of targeted ECD facilities. 

On the other hand, a total of 66 ECD teachers assessed during baseline. The annual assessment plan will include ECD teachers from both Phase I and Phase II schools, specifically involving two teachers per ECD facility, which represents a twofold increase from the baseline assessment. Moreover, during the baseline, a total of 170 ECD clients were evaluated, with an average of 5 clients assessed at each ECD facility. The annual assessment plan will include ECD clients from both Phase I and Phase II schools, with a particular focus on assessing 5 clients per ECD facility. This indicates a doubling of the number compared to the baseline assessment. 

Government Institutions Performance Assessment Method:

For the government institutions targeted for support under F4HE, the standard government performance index will be administered in all selected institutions. A total of 8 representatives from the Education and Health department entities assessed at baseline will be assessed for the annual outcome assessment using the GPI tool.

Project Indicators

The annual outcome assessment will include quantitative evaluation method to assess the project targeted indicators. Project indicators to be measured under the study are listed in table 1 below.

Table 1: Project Indicators to be included in Annual Outcome Evaluation:

Indicator Tool

Health Facility Assessment

% of health institutions and facilities that are gender responsive and adolescent/child responsive as per standards (by facility type (gender-responsive and adolescent friendly), and geography) Health Facility Assessment (HFA)

Data source: Supported health facilities

% of AKF supported health facilities that implement quality assurance (by geography) Health Facility Assessment (HFA)- Data source: Supported health facilities

% of AKF-supported health facilities with effective management systems (by geography) Health Facility Assessment (HFA)-Data source: Supported health facilities

% of supported health facilities with at least 3 modern family planning contraception methods on the day of the assessment (by geography) Health Facility Assessment (HFA)-Data source: Supported health facilities

% of clients who are satisfied with their access to health services, including SRHR and family planning (by gender, age, geography) Client exit interview-Data source: Client at interview at supported health facility

% of health workers in project geographies with appropriate knowledge, attitudes and practices related to gender-responsive and respectful health service delivery (by gender, geography) KAP - Data source: Supported health facilities staff

ECD Facility Assessment

% of AKF-supported ECD centers meeting the minimum quality and secure learning environment standards (by geography) Teach ECE tool-Data 

source: Supported ECD centers

Level of client satisfaction with the quality of AKF-supported early learning and ECD services (by gender) ECD client (parents/caregivers) exit interview

% of supported ECD staff (government or community health workers, pre-school teachers and others) with improved knowledge, attitudes and practices regarding ECD (by gender, geography) Teach ECE Classroom Observations and Educator’s survey

% of education facilities with contextually relevant and age-appropriate teaching and learning resources Teach ECE tool

Data source: Supported ECD facilities

Government Institutions Performance Assessment 

% of AKF-supported government agencies with improved performance (by geography) GPI tool- Data source supported government agencies assessed

Specific Tasks of Consultancy

The consultant will be responsible for all aspects of work related to enumerator recruitment and training, data collection, data entry, field reporting (including electronic data capture and submission, data quality assurance and developing the draft and final report. The consultant/firm will be responsible for all aspects of work related to survey implementation.

The specific tasks of the consultant include:

• The Consultant will share a detailed inception report with AKF (P) after a two-day inception meeting. The report will outline steps for the study process, reviewing and adapting/developing instruments etc. The inception report for the study will include the following sections: 
• Feasibility of the ToRs and any critical issues or questions.
• Key consideration and approach
• Proposed methodology
• Detailed implementation plan
• Development of enumerator guidelines and protocols for data collection and enumerators training
• Data collection protocols
• Monitoring and Quality Assurance Mechanism
• Plan for data entry and analysis including list of tables required.
• Upon receiving feedback from AKFP and AKF Regional Team, the consultant will incorporate the feedback into the inception report before commencing of the field work, 
• Recruitment of all field staff (locally in Chitral and GB) involved in the data collection including field enumerators, and supervisors. The enumerators for the HFs should be females and must have previous experience in data collection in HFs.
• Pilot testing all data collection tools.
• Provision of raw data during data collection for spot checking (soft copies), as well as a complete and cleaned data set upon conclusion of data collection and entry, as per the schedule.
• The consultant will present a draft study report to AKF (P) for review and feedback within the agreed timeframe. S/he will present a final version of the report (including recommendations where appropriate) to AKF (P) within the specified time frame.  The consultant will submit a well-organized and well-formatted report on template provided by AKF in the English language in print and electronic version.
• The consultant will coordinate with AKFP and will lead a workshop focused on Data Interpretation and Dissemination. This workshop will involve both internal and external stakeholders and will encompass activities like data visualization and trend analysis, comparing the data with other secondary sources like the Demographic Health Survey (DHS), Multi Integrate Cluster Survey (MICS) and project baseline.
• The consultant must not share any of the data sheets, data sets, and assessment results with any third party.

6. Ethical Consideration 

• Written informed consent will be obtained from all survey participants, including the health care providers, schools head teachers, ECD teachers, parents and or any other participant of the survey. Participants will be asked to provide informed consent following a verbal description of the risks, benefits and procedures.
• Participant’s identities will only be known to the Surveyors. On the questionnaire used for noting survey data, only the participant number will be noted, so no personal identifying information will be available to anyone.
• The consultant/consulting firm will abide by all AKF safeguarding protocols throughout the assessment exercise at all levels.

Outputs and Suggested Timeframe

• The period of the contract will be for two months, beginning on a date agreed upon between the consultant and the Aga Khan Foundation Pakistan. The assignment is expected to be done between October to November 2024.

Selection Criteria 

The Consultant should have:

• At least a master’s degree in a related field.
• A proven record of at least 10 years of experience in the development field out of which at least 10 years has been gained in the relevant area (as required in ToRs).
• Prior experience in data collection as a part of baseline, annual assessment, endline studies on programs and projects focusing on health and ECD.
• Extensive experience in assessing preschool or early learning programs or including a team member with such expertise.
• Experience in the leadership and management of development programmes will be considered an asset.
• Credible records and references of previous consulting experience in Pakistan and with AKDN.
• Fluency in the English language (written & spoken), knowledge of local languages will be preferred. 

Application Procedure

The selection of the consultant will be carried out in accordance with the procedures of the AKF Pakistan. Qualified and interested parties are asked to submit the following:

• Letter of interest, including the names and contact information of three previous clients who can be contacted regarding relevant experience,
• Detailed technical proposal clearly demonstrating a thorough understanding of this request for proposals and including the following:

Consultant profile

• Demonstrated previous experience in quantitative evaluation methods and other qualifications outlined in this ToRs.
• A proposed timeframe detailing activity and a schedule/work plan (including a Gantt chart).
• Team composition and level of effort of each proposed team member.
• Curriculum Vitae(s) of all proposed staff outlining relevant experience (annexed to technical proposal).
• The financial proposal shall include lump sum cost with a breakdown for fee, travel and number of working days.