Regional Evaluation Specialists for Mid-term Evaluation of Paper Trail for Better Governance III project

Posted on

Balkan Investigative Regional Reporting Network – BIRN Hub, is opening a call for a Regional evaluation specialists for mid-term evaluation of Paper Trail for Better Governance III project

Balkan Investigative Regional Reporting Network – BIRN Hub, is opening a call for the following position:

Regional evaluation specialists for mid-term evaluation of Paper Trail for Better Governance III project

  1. Project context and background 

Balkan Investigative Reporting Network (BIRN), is the leading Western Balkans independent non-governmental organisation focusing on media development, investigative reporting and watchdog monitoring of public institutions. The Balkan Investigative Reporting Network (BIRN) was established in 2004 and is a network of non-governmental organisations promoting freedom of speech, human rights and democratic values in South-Eastern and Central Europe. BIRN is active in Albania, Bosnia and Herzegovina, Kosovo, North Macedonia, Romania and Serbia as well as editorially present in Greece, Bulgaria, Croatia, Moldova, Montenegro, Ukraine, the Czech Republic, Poland, Slovakia, Hungary and Slovenia.

BIRN Kosovo is the largest member entity with 45 staff members. Founded in 2005, BIRN’s main goal is to provide information to citizens and contribute to the democratic transition process in the country, while simultaneously promoting accountability, rule of law and policy reform through professional investigative journalism.

In October 2013, BIRN Kosovo and BIRN HUB partnered in what became the first phase of the Paper Trail to Better Governance which is currently in the third phase of implementation. The main purpose of all phases of implementation was pushing for better governance, transparency, accountability and better responsiveness of public institutions to freedom of information in the Western Balkans.

Specifically, BIRN Kosovo in partnership with BIRN HUB received funding of EUR 650,000.00 from the Austrian Development Agency for a three year project to build on the impact, achievements and the lessons learnt from two phases of BIRN’s “A Paper Trail to Better Governance” project.

The project is  contributing to and to promoting rule of law, accountability and transparency, exposing wrongdoings of governments, public and private companies, as well as individuals, enabling the public across the region to make informed decisions.

 

Area of activity: Rule of law, good governance and transparency and accountability
Project duration: 1 February 2020 – 31 January 2023
Location of implementation: South-Eastern Europe/Danube Region: Albania, Bosnia and Herzegovina, Kosovo, North Macedonia, Montenegro, and Serbia.
Implementing partners: Balkan Investigative Regional Reporting Network – BIRN HUB
Target groups: 100 Men and women investigative journalists, 25 young journalists, 200 researchers, 10 media outlets, and the general public (at least 3 million people exposed to published material resulting from investigative reporting, monitoring of public institutions compliance with legislation and annual review reports published);
Funded by:             Austrian Development Agency

 

In particular, the Paper Trail to Better Governance III project on a cross-regional action which responds to the European Commission’s (EC’s) membership conditions, aiming to strengthen the rule of law, more accountable governance and judiciary in the Western Balkan (WB) region. The project is expected to directly contribute to SDG targets 16, 6, and 16.10, through advocating for a better functioning public sector that abides by the principles of transparency and accountability to its citizens.  Overall, the project is expected to reach about 100 Men and women investigative journalists, 25 young journalists, 200 researchers, 10 media outlets, and at least 3 million people who will be exposed to published material resulting from investigative reporting, monitoring of public institutions compliance with legislation and annual review reports published.

 

BIRN had in 2016 and again in 2018 commissioned external evaluations which assessed efficiency of implementation and overall success, which looked into the projects’ relevance, efficiency, effectiveness, impact, sustainability and get recommendations for possible continuation of the project.

Both evaluation reports found that BIRN fulfilled, and in some cases exceeded, expectations against the five evaluation criteria. As such, evaluation reports found that BIRN had achieved more than planned project outcomes that were born out in a cost-effective manner; numerous positive changes towards achievement of the overall objective, influence on wider impact reached through the project effects and a solid sustainability potential. Both evaluation reports provided BIRN and ADA with valuable recommendations serving to improve the efficiency of the project which were taken into account in all phases of the project. Recommendations pertained to stakeholder involvement, gender considerations and more flexibility in financial administration and investment of funds in more face to face team meetings and ad-hoc emerging investigations/stories, introducing  induction training for interns, and investing more into the development of short video production.

BIRN ensured to take recommendations provided into account in preparation of Phase II and Phase III of the project and made sure that the evaluation reports served as guidance in future interventions.

 

  1. Purpose and Objectives of the Mid-term Evaluation

The main purpose of this mid-term evaluation is to undertake an independent and comprehensive assessment of the progress and results achieved  by the “A Paper Trail to Better Governance” to date, through an analysis of effectiveness of implementation and impact of the project. In particular, the following are the objectives of the evaluation:

 

  • Assess the implementation of the project to date, identifying factors (either positive or negative) affecting project implementation of the project. If deemed necessary, suggest potential revisions to the expected level of achievement of the objectives and remedial actions needed to achieve the initially set objectives;
  • Analyse the project’s implementation strategies and actions from the effectiveness perspective in achieving the project expected results as per the approved logical framework;
  • Analyse the institutional set-up, partnership capacity for project implementation and coordination amongst partners; ;
  • Identify lessons and potential good practices for the key stakeholders;
  • Provide strategic recommendations for the different key stakeholders in improving project implementation and achievement of the set objectives.

 

The final evaluation will be presented to the donor agency and organisational management.

 

Finally, the evaluation should provide evidence on the potential for future continuation and/or up-scaling of the initiative.

 

  1. Scope of the evaluation

 

The mid-term evaluation will cover the following time period of implementation:

 

  • February 2020 until 30 March 2022. The evaluation will analyse the whole scope of the project and its results framework, which comprises all elements of the intervention logic. All project documents and materials will be made available to the Evaluation Team.

 

The Evaluation Team is expected to apply a participatory approach to ensure that the BIRN project team is closely involved in the entire evaluation, so that achievements as well as challenges are openly and transparently discussed. The evaluation is expected to focus on the Effectiveness and Impact Criteria per the OECD/DAC criteria.

 

The findings of this evaluation are also expected to serve the Project to steer interventions towards reaching the expected results and if necessary make prompt adaptations.

 

The Evaluation Team is expected to assess:

 

  • The approach of the project in terms of implementation arrangements (management structure, operational strategies, monitoring system and procedures);
  • Identify any (internal or external constraints), to achieving progress;
  • Identify challenges and obstacles faced during the implementation of the project;

 

The primary location of implementation of the activities occurred in Kosovo and in Bosnia and Herzegovina (where BIRN Kosovo and BIRN HUB are physically registered and located). Considering that the project has a multi-national and cross-regional team, project stories, thus journalists and researchers are spread out across the Western Balkans.

For this purpose, the Evaluation Team is expected to conduct interviews and its research primarily in Kosovo and Bosnia and Herzegovina and online through covering the entire Western Balkans.

 

While BIRN has made efforts to include previous recommendations on gender mainstreaming generated from the external evaluations commissioned, the Evaluation Team is expected to integrate and pay particular attention to gender equality and social inclusion, in line with the logical framework of the project. The Evaluation Team is also expected to pay attention to issues related to poverty reduction and environmental and climate resilience approaches.

 

  1. Evaluation Questions

 

The following evaluation DAC criteria are to be used; related evaluation questions are proposed for the evaluation process. From the 6 DAC criteria, the Evaluation Team will look into effectiveness and impact and sustainability.

 

The following are the Evaluation Questions that the Evaluation Team needs to provide answers for:

 

Effectiveness of implementation: Evaluation of project set-up and steering:

  1. To what extent has the project contributed to the overall objective by March 2022?
  2. Has it achieved its purpose, expected results and desired outputs?
  3. Does the project have any effect on the functioning of other media in Kosovo and the region in terms of other media outlets willing to send their employees on trainings/workshops organised by BIRN; are these trained journalists encouraged to implement skills they learned during BIRN trainings in the media outlets where they work)?
  4. To what extent has the project empowered target groups, in particular journalists and youth?
  5. What were the major factors influencing the achievement or non-achievement of the objectives (positive and negative factors);

 

 

Impact criteria:

 

  1. Have there been any changes that may have resulted from the project implementation?
  2. Did the project have any concrete impact on the functioning of judiciary/politics/society/other media (as this is the third phase of the project, potential results achieved in the long term since the beginning and up to 2022 may be provided as examples);
  3. To what extent the project has been able to adapt and/or mitigate the effects of external factors in order to adapt to changing circumstances and situations and therefore increase positive and decrease negative impact to the society and sector
  4.  What real difference has the activity made to the beneficiaries and how many people have been affected?

 

  1. Design and Approach

 

The evaluation process will be carried out through various phases, each serving the purpose of collecting data and responding to the evaluation questions under Chapter 4.

 

Specifically, the following steps must be taken:

 

– Contracting and Kick-off meeting: A kick-off meeting occurs with the purpose of clarifying any outstanding issues in the ToR and assignment of roles. Then, the contract is signed between BIRN and the selected bidder. With the signature of the contract, BIRN provides all project documentation and relevant annexes as well as reports submitted to ADA until this reporting period. In case any changes to the ToR due to changed circumstances on the ground are needed, all changes need to be proposed and approved at this stage.

– Desk Research/Project documentation analysis: After obtaining all the project documentation, the Evaluation Team  analyses all necessary project documents; re-construct and analyses the intervention logic/programme theory. The Evaluation Team then analyses and interprets all the existing data.

– Briefing with the team: An introductory meeting with the project team (hybrid meeting as the management team is multi-national) as well as individual meetings with editorial and other team members, focusing on management procedures and project set-up, coordination as well as resources at the disposal of the project.

– Inception Report: The inception report will serve as a roadmap for the evaluation that will ensure shared understanding between the Evaluation Team, BIRN team and the ADA programme/focal point in regards to finalised work-plan, deliverables and timelines. The Inception Report also needs to further detail the evaluation design and present the data collection and analysis methods, tools to be used and identify potential risks and limitations along with adequate mitigation strategies. The Inception Report needs to be in line with the proposed structure in Annex 5 Quality Checklist for Inception Report (IR)Guidelines for Project and Programme Evaluations.

 

The Evaluation Team is also expected to draft Evaluation Matrix. which is the first task and the integral part when it comes to preparing the inception report. When developing the evaluation matrix, the Evaluation Team should follow the following elements:

  1. Evaluation criteria
  2. Evaluation questions
  3. Indicators
  4. Sources
  5. Methods for data collection

 

The Evaluation Matrix must be in line with Annex 7 Template for Evaluation Matrix, p.51.

 

– Field visits/Hybrid meetings: Field visits/physical meetings need to take place in Kosovo or Bosnia and Herzegovina whereas meetings with other actors may take place online.  They will serve the purpose of conducting in-depth individual interviews as well as focus group discussions using semi-structured interviews where applicable.

– Debriefing/Presentation: Presentation of key findings (feedback workshop) with BIRN team and ADA will take place to present key findings and recommendations. After the debriefing meeting, the draft report will be shared with BIRN.

– Submission of the draft Final Report. After the draft is received, BIRN and ADA will provide feedback and comments to the report.

 

Proposed methods:

 

For the different phases, the Evaluation Team  is expected to collect the data and information through different methods such as:

  • Desk research and analysis of documents;
  • Structured interviews and semi-structured interviews through face-to face or ZOOM/Skype meetings;
  • At least one group discussion, preferably with youth who attended the internship programme.

 

All data collected needs to be disaggregated by sex, ethnicity and where feasible also in terms of disability. It is expected that the Evaluation Team will present concrete recommendations which are addressed to the specific stakeholders. In addition, all research questions should be addressed paying attention to gender, social and environmental issues.

 

The Evaluation Team needs to base its work and methods in line with the Guidelines for Project and Programme Evaluations developed by the Austrian Development Agency and OECD DAC norms need to be considered throughout the entire evaluation process (found at: https://www.entwicklung.at/fileadmin/user_upload/Dokumente/Evaluierung/Evaluierungs_Leitfaeden/Guidelines_for_Programme_and_Project_Evaluations_ADA_2020.pdf)

 

Important to note: as most of the project interventions are still ongoing, it is early to expect definite results at the level of systemic change, and even more so on impact, yet orientation towards impact can be measured.

 

Expected outputs and deliverables/Report(s)

 

Inception Report

 

  • Inception report (approx. 10-15 pages) containing appropriate methodology to be applied during the mid-term evaluation, as well as the work plan to be used during the course of the assignment is drafted, submitted, and approved by BIRN. Report must be in line with the Annex 5: Quality Checklist for Inception Report (IR), p.46 and 47;

 

Final Report

 

 

All reports need to be written in English. The executive summary should summarise key findings and recommendations (three pages maximum) and needs to be submitted as part of the final draft report.

 

 

  1. Workplan

37 working days are currently estimated for this assignment. The mid-term evaluation is foreseen to start in the beginning of May and to be completed by June 30th, 2022. The first draft report must be submitted by June 20th whereas the final report will be completed by July 6th 2022.

 

 

Activity Expert 1 / Team Leader Expert 2 / Team member Total days per activity
Desk research phase and submission of inception report 4 3 7
Field visit/data collection 7 7 14
Data analysis and drafting the report 4 3 7
Debriefing and finalisation of the report 5 4 10
Total 20 17 37

 

The written feedback from BIRN Kosovo, partners and the donors is planned to be given within a maximum of two weeks after submission of the reports.

 

  1. Evaluation Management Arrangements

 

BIRN Project Manager will be the focal point for the Evaluation Team in all communication, obtaining documents and any assistance required in reaching out to stakeholders for any arrangements required.

 

The Evaluation Team will be fully independent from the project management and organisation management of BIRN Kosovo and BIRN HUB.

The BIRN management team will at all times uphold the independence, impartiality and ethics of the Evaluation team in line with the evaluation guiding principles and in line with BIRN Code of Ethics.

 

 

  1. Requirements for the Company/Team of Evaluators

 

BIRN is looking for a team of two evaluators or a company applying with at least two experts, namely a Team Leader and a Team Member. At least one team member must be a woman.

BIRN will not consider the applications of any candidates who have been engaged in any stages of design or implementation of the project.

 

The interested company/candidates, must possess the following experience and qualifications:

 

  • Experience in conducting evaluations, preferably covering the good governance, rule of law or media sector;
  • Experience in previous regional projects evaluations is preferred;
  • Excellent communication skills in English, while knowledge or Albanian or Serbo-Croatian languages of at least one team member is a must;
  • Sound communication and presentation skills;
  • Submit a professional profile of the evaluating team/company (CVs of all individuals included in the consultancy team);
  • At least 7 years overall professional experience and Master degree in relevant field as in International Development, Management, Law, Political Sciences or Media similar social sciences;
  • At least one of the experts should have previous experience or background in project covering gender mainstreaming and/or environmental protection.
Additionally, the following skills are expected from:

 

Team Leader/Senior Expert:

General experience and skills:

  • At least 7 years overall professional experience and Master degree in relevant field as in International Development, Management, Law, Political Sciences or Media similar social sciences;
  • At least 5 years of experience in working with regional/international development projects and evaluating according to the OECD/DAC criteria;
  • Experience in at least 3 evaluations in the position of Team leader/Senior Expert in project evaluations, with preference given to previous experience in the Western Balkans;
  • Experience of programme formulation, monitoring and evaluation and qualitative data collection and analysis;
  • Experience in compiling professional, reader-friendly and concise reports;
  • Fluency in English language in both written and verbal communication;
  • Good knowledge of Serbo-Croatian or Albanian languages;

 

Team Member

General experience and skills

  • At least 7 years of professional experience and studies in Law, Political Sciences, International Development or Media Sector;
  • Previous experience in rule of law or media projects;
  • Experience in qualitative data collection, analysis and interpretation;
  • Excellent understanding of results-based evaluations and impact evaluations;
  • Fluency in English language in both written and verbal communication;
  • Good knowledge of Serbo-Croatian or Albanian languages;

 

  1. Specifications for the Submission of Offers Application process

 

Qualified and interested candidates are asked to submit an electronic copy of their expression of interest/proposal by May 10th  2022 at 23:59pm, with the subject REF: “Paper Trail for Better Governance III – Evaluation” to ivana.nikolic@birnnetwork.org and albulena.sadiku@birn.eu.com.

 

Applicants must submit:

Technical Proposal:

  • Suitability proof/document:
    • For companies a profile of suitability for the assignment in line with previous work experience, qualifications etc., also containing short biographies of experts and their CV’s attached. (max 5 pages);
    • For individuals a profile of the team with their previous experiences matching the requirements of this ToR and detailed CV;
  • Composition and division of work between team members, clearly assigning the Team Leader and Team expert, ensuring that one member is a woman;
  • Methodology for carrying out the Evaluation and methods to be used;
  • Draft work plan and suggested timetable within the proposed days in this ToR;

Financial Offer:

Financial offer including budget broken down per deliverables and individual daily fees.

The proposed amounts must be all-inclusive, thus accommodation and transportation costs need to be included in the offer, if applicable.

Evaluation of Offers:

The Technical Proposal will be evaluated with 70% whereas the financial one will be evaluated at 30%. The financial offer will be evaluated by using the best value for money approach (combined scoring method).

When using this weighted scoring method, the award of the contract should be made to the company or Evaluation Team, whose offer has been evaluated and determined as:

 

  • Responsive/compliant/acceptable, and
  • Having received the highest score out of a predetermined set of weighted technical and financial criteria as explained below:

 

  • Technical Criteria weight; [70 points]
  • Financial Criteria weight; [30 points]

 

 

Criteria Weight Max. Point
Technical evaluation: 70% 70
●      Relevant professional

experience in line with Evaluation Requirements;

25% 25
●      Quality of prepared Methodology and methods to be used in line with the Evaluation Requirements 35% 35
●      Team Composition, gender and previous experience in thematic and geographical areas as required in the ToR 10% 10
Financial offer: 30% 30

 

 

Only candidates obtaining a minimum of 49 points would be considered for the Financial Evaluation.

 

BIRN has a maximum budget of EUR 15,000 for carrying out the required Mid-Term Evaluation.

 

  1. ANNEXES

 

The Evaluation Team should take note of the following documents, comprising annexes to this ToR:

Annex 1 – Short Project Summary

Annex 2 – Paper Trail Investigations – Focus Page with investigations produced by the project

Annex 3 – BIRD Source Database

Annex 4 – ADA Guidelines for programme and project evaluations

Annex 5 – ADA Results Assessment Form (RAF)