Mid-term evaluation of PAs 1, 2 and 3 of the Good Governance OP, 2014-2020, in Bulgaria
Междинна оценка на напредъка по изпълнението на приоритетни оси 1, 2 и 3 на ОПДУ
Global Advisers AD, Tel: +359 2 980 50 55, E-mail: s.tsanova@globaladvisers.eu
This is the interim evaluation of the “Good Governance” OP.
To assess progress in implementing PA1 “Administrative Service Delivery and E-Government”, PA2 “Effective and Professional Governance in Partnership with Civil Society and Business”, and PA 3 “A Transparent and Efficient Judiciary System”.
Analysis of monitoring data.Interviews and focus groups with stakeholders.Survey of a sample of beneficiaries of PAs 2 and 3.
The three PAs risk that funding available is not fully used.For PA1, there is the risk of not achieving the 2023 targets for two performance indicators, O1-2 and O1-8 and some result indicators.For PA2, two performance indicators (O2-3 and O2-4) and one result indicator (R2-3) have not been reported and no new procedures are envisaged for achieving their targets.For PA3, there is a low risk of not achieving performance and result indicators.Overall, people consider the administrative burden to be too high.The changing evolving regulatory framework makes it necessary sometimes to re-adjust measures.
Some data for performance and result indicators are not reported.
Re-allocate funding to avoid it not being used.Revise metadata for performance indicators.
OP for the implementation of YEI in mainland France and outermost regions; 12 ROPs 2014-2020
​​Lot 1 (core evaluation): Quadrant Conseil and KPMG; Lot 2 (national counterfactual analysis); Edater and Amnyos; Lot 5 (experimental projects): Itinere. Lot 6 (Synthesis): Orseu​
The final YEI evaluation is a pooled impact evaluation of the national OP for YEI in mainland France and outermost regions and the 12 regional OPs, combining theory-based and counterfactual impact evaluations and covering the 2014-2017 period. The national OP focuses on employment, social inclusion and early school leaving, while the regional OPs on training. The evaluation process produced several reports. This summary covers the final report, the report on the counterfactual approach used for the national evaluation and the report on experimental and innovative projects.
To assess the measures on employment, social inclusion, education and training supported by YEI, as well as the YEI contribution to these measures and to explain why, how and under what conditions YEI has contributed to the changes expected.
- Surveys of participants 6 months after exit. - Theory of change (contribution analysis). - CIE of effect on the NEET and unemployment rate using a difference-in-difference analysis (comparing regions with and without YEI support, as well as people under 26 and those aged 26 and over in eligible areas) and quasi-panel approach. - Interviews with stakeholders. - Case-studies . Data come from the YEI monitoring system, the Labour Force Survey, the Public Employment Service (PES), the National Statistical Office and Eurostat.
- Most MAs used YEI to support existing youth employment policy. - YEI mostly financed existing projects (by strengthening activities, increasing the number of participants or focusing on NEETs). - Most participants were low-skilled young people already known by the PES. YEI struggled to reach ‘invisible’ or inactive non-registered NEETs. - Many traditional beneficiaries did not reply to the calls for projects because of difficulties in reaching NEETs (small amounts devoted to identifying them, many projects already financed by ESF, most YEI projects targeted all NEETs instead of a specific sub-group, requirement to prove NEET status, distrust of institutions, bad previous experience). The short duration of YEI projects implied that little time was left to find a lasting solution to these problems. - The YEI programme met its target, between 2014 and 2017, over 360,000 young people participated in an operation funded by YEI. At its peak in 2015, it supported one-fifth of the NEETS population in eligible areas. - 31% of participants were in employment immediately after exit, 14% in training and 55% were still NEETs. 6 months after exit, about 50% were in employment and 13-19% in training. - Job offers were mainly full-time with an average wage slightly below the minimum; they broadly matched the qualification levels of the participants and most of them found the job interesting. - Critical factors for a positive immediate outcome were the duration of the measure, the unemployment spell, and age (after 6 months, it’s the level of education). For innovative projects, they were a link with the enterprise and work experience, individualised guidance, and participation in collective activities. - While there was a reduction in long-term unemployment, the national counterfactual analysis does not find a clear overall net effect on employment. In some cases, an increase in precarious or part-time employment was observed among participants. - A significant net positive effect was however found regarding maintaining youth in education and increasing access to training.
- The variety of monitoring systems makes it difficult to aggregate data. - Difficulty of linking the results of the ‘contribution analysis’ and the CIE (e.g. the former analysed the global impact of YEI, the latter the effect of specific measures). - Data available (only up to 2016) do not allow individual monitoring of participants over a reasonable period of time, which explains why ‘quasi-panels’ were used.
- Maintain a NEETs-based approach instead of having a general policy focusing on all young people. - Concentrate support on the most vulnerable groups and those far from the labour market. - Increase communication of information on the approach. - Carry out detailed analysis on the situation of NEETs to better understand their characteristics and needs. - Strengthen partnerships to better identify NEETs. - Promote integrated policies. - Eliminate the requirement to justify NEET status for those furthest from the labour market and reduce the number of common indicators. - Support innovation and experimental projects. - Examine sustainability of positive outcomes.
Evaluation of the implementation and impact of the Environment OP, 2014-2020, in Bulgaria
Оценка на въздействието на ОПОС 2014-2020 г. и на приноса на подкрепата от ЕСИФ за постигането на специфичните цели по всеки приоритет
https://www.ecorys.com/; https://bim-bg.com/en/
The evaluation examines the effectiveness and impact of the OP, as well as the efficiency of the implementation and the sustainability of the results achieved.
To determine the contribution of the OP to the promotion of the environmental and resource efficiency, in relation to water, waste, biodiversity, risk of floods and landslides, and air quality.
CBA. Counterfactual analysis with administrative data from 44 municipalities on landfilled amounts of municipal waste. Comparative analysis. Analysis of monitoring data (Umis). Analysis of the intervention logic. Analysis of official statistics (from NSI, Eurostat and other official sources). 4 online surveys of beneficiaries (102 municipalities, 9 water supply and sewerage operators, 19 government bodies, 21 non-profit legal and non-legal entities). 26 interviews with municipalities, government bodies, non-profit legal and non-legal entities.
The measures led to a positive trend in reducing landfilled municipal waste and increasing recycled waste. However, the counterfactual study shows no significant net effect in terms of disposed quantity of municipal waste per capita. The measures contributed to improving the monitoring of water resources and water management. Projects helped improve flood risk management in the municipalities supported, but the contribution to the reduction of air pollution by fine particulate matter and to biodiversity conservation is limited, also because of projects being not completed. There is a multiplier effect given by the resources added by beneficiaries for investments in water supply and sanitation infrastructure. The use of financial instruments seems to have had limited effects.
Many projects were still ongoing at the time of the evaluation. Data have limited availability and quality (e.g., pre-intervention data and data in waste).
Start the ready-to-implement measures of the 2021-2027 OP and prioritise those directly contributing to the objectives. Assess beneficiary readiness and help potential beneficiaries by presenting them a list of possible projects. Present EU-funded measures in a more attractive way and expand support through financial instruments to cover additional needs. Ensure additional data collection to analyse the achievements of the OP comprehensively.
Evaluation of the implementation, results and impact of Priority Axis 1 of the Human Resource Development OP, 2007-2013 (ESF)
Оценка на изпълнението, резултатите и въздействието на схеми по директно предоставяне на БФП по приоритетна ос 1 на Оперативна програма „Развитие на човешките ресурси” 2007-2013
Balkan Institute for Labour and Social Policy
Balkan Institute for Labour and Social Policy, Boulevard "Tsarigradsko shose" 24, building 22, 1113 g.k. Iztok, Sofia, Bulgaria
The evaluation assesses five schemes funded by the Human Resource Development OP 2007-2013, aimed at supporting young people and the disadvantaged through job subsidies and training.
To assess the impact of interventions and to provide guidance for programming for the next period.
Analysis of monitoring data from the Employment Agency;8 focus groups;cost-effectiveness analysis;12 interviews with relevant experts and stakeholders;5 case studies;survey of beneficiaries and employers (1,358 and 1,790 respondents).
70% or more of participants (depending on the scheme) believe the intervention was an effective response to labour demand and 60% report that the ESF helped them find a job.At time of the survey (in 2016), between 42% and 75% of participants were in employment.Only 21% of people with disabilities were working at the moment of the analysis. According to the evaluation the low performance is due to the fact that schemes are not specifically focused on people with disabilities.The "Youth employment with permanent inclusion", "Youth Employment service" and "First job" are the best in terms of the sustainability of results: 25%, 27% and 22% of participants, respectively, being still with the same employer.Employers consider that workplace training is more useful than classroom training.The measures are efficient except for one scheme. The scheme for young people has the lowest cost per person who finds employment.
The sample examined was smaller than expected since many participants could not be contacted.
There is a need to:give more attention to the most vulnerable groups;focus on improving the quality of the workforce;use integrated services to support the most vulnerable groups;develop a system of regulations to encourage on-the-job training.
Mid-term evaluation of the ESF+ Education Programme, 2021-2027, in Bulgaria
Вътрешна оценка (assessment) на напредъка на подкрепата на Програма „Образование“ 2021 – 2027 г. за целите на междинния преглед на програмата
Institute for Public Administration and European Integration
The evaluation examines the implementation and execution of the ESF+ Education Programme in Bulgaria which aims to support reforms in the education system. The programme was approved in 2022 and implementation of certain projects commenced in 2023, and others in 2024. The cut-off date of the monitoring information used in the evaluation was 31-12-2024.
To assess the progress of implementation and effectiveness of educational measures and support decision-making on the allocation of the remaining financial resources and inform programme adjustments where needed.
Analysis of monitoring data and statistical office indicators.Analysis of administrative data.Analysis of expert assessments (such as the World Bank).Desk reviews of project documents.Stakeholder feedback (number and type of stakeholders consulted not specified).
Output indicators are much lower than the milestones because the implementation of the programme was delayed.Significant delays in the negotiations with the European Commission were due to multiple factors, such as the political crisis in Bulgaria in 2021 and the Russian invasion of Ukraine in 2022.The main challenges to successfully implement the programme are: regional disparities in the distribution of funding, insufficient capacity among beneficiaries to manage the procedures and administrative requirements, staff shortages, and incomplete use of monitoring systems for tracking implementation progress.The programme’s focus on equity, skills and innovation is relevant given the current needs in Bulgaria.
Reduce procedural complexity by streamlining application and reporting requirements to ease participation, especially for smaller stakeholders.Use data-driven approaches to better align funding with regional needs and disadvantaged groups.Improve the quality, timeliness, and use of monitoring data in programme management.Launch calls for proposals earlier in the programming period to prevent delays and provide guidance and capacity-building for beneficiaries and local education authorities.Strengthen dialogue with civil society, business and education sector representatives in programming and implementation.
​​Interim evaluation of measures supporting inclusive education under the Education Programme 2021-2027, in Bulgaria​
Изпълнение на проекти за приобщаващо образование в България: Изводи от оценката на “Силен старт” и “Успех за теб” - Междина тематична оценка 1 – приобщаващо образование (ОИСР)
​​Christa Rawkins, Zhasmin Avetisyan, Julian Rojas​
The report focuses on two inclusive education projects – “Bright Start” (2024-2029) for pre-school education and “Success for You” (2023-2028) for primary to upper secondary – supporting the personal development of all learners. The evaluation was carried out between February and December 2025, monitoring data go up to October 2025.
To assess the initial implementation of the two projects, to provide recommendations to improve implementation, and to analyse the effectiveness, relevance, coherence, efficiency, sustainability, and EU added value of the measures.
Desk research on project documents.Analysis of statistical and monitoring data.Theory of change, applied to both projects and mainly used to address questions on effectiveness.Survey of regional project teams in all 28 regional education districts (sent to 118 people, 84% response rate).108 semi-structured interviews with national, regional and institutional project teams and final recipients.Site visits to participating kindergartens and schools.
The projects are relevant overall with respect to the country’s needs, with some limitations. Despite the need for inclusion during adolescence in Bulgaria, support declines as students age. Measures to increase the professional capacity of teachers are not adequately funded and have a narrow focus, reducing the relevance of activities.The projects align with international good practices, showing a high level of external coherence with the broader policy landscape, delivering interventions at a much larger scale than national programmes.Experienced national and regional project teams responsible for implementation effectively ensure that activities are carried out as intended. The uptake of activities aimed at students is high, but implementation is slower for activities that target staff and parents, particularly under “Success for You”.The projects are prompting behavioural changes in kindergartens and schools, including improved diagnosis of learner needs and the implementation of structured, small-group support.Some activities are not always understood by those implementing them. This includes activities with parents and support for gifted learners or those whose first language is not Bulgarian, as well as general support under “Success for You”.Communication between project teams at different levels (national, regional, school) is clear, timely and responsive, supporting efficient implementation. However, the time dedicated by project teams to reporting and control reduces the time for implementation of project activities.Recruitment efforts and implementation support are not targeted enough to proactively reach those that do not apply.Sustainability beyond project duration remains uncertain.The collection of output and outcome data has improved, but this is not yet being used systematically to enhance inclusive practice.
Purposive sampling was adopted to select individuals and organisations to participate in data collection, meaning they are not nationally representative. Participation was voluntary, raising the risk of selection bias. The availability of outcome data is limited at this stage of implementation.
Enhance measures to strengthen professional capacity for inclusive education, by expanding the scope, format and quality of training under the project.Improve activity guidance by better defining relevant practical approaches to parental engagement, supporting gifted learners and those whose first language is not Bulgarian.Strengthen monitoring and evaluation processes to facilitate more learning across the system.Develop targeted efforts to increase project participation and enhance implementation in needy kindergartens and schools.
Evaluation of the impact of ESF measures on the quality, availability, and accessibility of health services under the Wielkopolskie OP, 2014-2020
Wplyw wsparcia EFS w ramach WRPO 2014+ na jakosc i dostepnosc swiadczen zdrowotnych na terenie wojewodztwa wielkopolskiego
Agnieszka Śnieżek, Beata Dec, Sebastian Pałka, Marta Cichowicz-Major, Wojciech Pieniążek, Adam Grajek, Aleksandra Kamińska, Michał Marciniak, Paweł Penszko
The evaluation examines the implementation of the ESF support dedicated to the quality and accessibility of health services in Wielkopolskie, with particular attention to Regional Health Programmes (RHP).
To assess the effects of ESF support for health service access and quality in Wielkopolskie, with particular attention to Regional Health Programmes.
Desk research.Monitoring and statistical data.Survey of participants, medical staff, and legal guardians in RHP projects (3,406 respondents).7 interviews with MA/IB and related institutional actors.30 interviews with beneficiaries.3 case studies.3 group interviews with representatives of MA/IB, beneficiaries and medical experts.Implementation workshop.
Demand and levels of achievement varied substantially across the 16 programmes.Participation was territorially uneven, with stronger concentration in larger centres and weaker uptake in peripheral areas.Programme quality and relevance were strengthened by multi-stage programme design, Steering Committee review and selection criteria promoting accessibility, organisational quality, and evidence-based delivery.The programmes generated health, educational, social, and system-level effects, including improved access to diagnostics, treatment and rehabilitation that were otherwise difficult or impossible to obtain in the standard system.Some programmes are characterised by incomplete reporting, over-reliance on product indicators and, in some cases, subjective indicators without adequate target values or post-completion reporting.Accessibility improved through the removal of geographical, organisational, and financial barriers, including free services and, in some cases, transport support or support for dependants.
The evaluation was constrained by time lag, staff turnover of MA/IB, which reduced respondent availability, and uneven reporting, that limited the comparability of results.
Organise service delivery for working-age adults and older people as close as possible to the place of residence.Prioritise mobile delivery models for calls for one-off preventive interventions, local service points, and cooperation with local actors to maximise accessibility.Introduce a uniform, mandatory, final reporting template for projects/RHPs, with consistent product and result indicators and clearly defined baseline and target values from the planning stage.Combine patient-reported outcomes with objective clinical measures wherever feasible, to improve reliability and comparability.Centralise public information on regional and local health programmes and develop the patient portal.
Mid-term evaluation of the implementation of the Baden-Württemberg ERDF programme, 2021–2027
Bewertung der Umsetzung des EFRE-Programms Baden-Württemberg 2021 – 2027 Halbzeitbewertung
Oliver Rohde, Julian Emmler, Karin Hummel
DLR Projektträger, www.projektträger.dlr.de
The purpose of the evaluation is to analyse the extent to which the programme is achieving its objectives by examining the output and outcome indicators set for each specific objective.
To assess the state of implementation of the programme, the extent to which its objectives are likely to be achieved and the factors affecting this, and to provide an outlook on the future development of the measures.
Desk analysis. Analysis of official statistical data. Reconstruction of the logic of the interventions. Analysis of output and result indicators. Interview to the representatives of the Managing Authorities.
By 31 December 2024, 74.8% of the programme budget had been committed and 8.9% disbursed, well above the German average (33.3% and 6% respectively). As for RSO1.1, which concerns expansion of research and Innovation capacities, 90% of the budget is approved (EUR 81.7 million) and most indicators already met or exceed programme targets, such as 785 researchers supported (262% of target) and 20 patent applications presented (500%). Progress is slower for RSO1.3, on SMEs’ innovation uptake, with 32% of funds approved (EUR 5m) and indicators still below targets (e.g. 27 SMEs supported vs 83 planned). Although 6,110 SMEs invested in skills for smart specialisation (63% of the target), only 33% of the RSO1.4 budget supporting skills development (EUR 2.6m) has been approved and most indicators remain below target. Measures on energy efficiency and greenhouse gas reduction are very advanced, with 87% of funds approved (EUR 48m) and several indicators already above the target (e.g. companies cooperating with R&D institutions at 167% of the target). Circular economy and resource efficiency implementation is moderate, with 60% of funds approved (EUR 24m) and mixed indicator performance (e.g. 21 firms cooperating with R&D institutions vs 7 target; but prototypes are only 29% of the target). Administrative and regulatory issues caused delays in specific projects, notably in the hydrogen sector, where approval procedures for electrolysers were initially unclear.
Some indicators suffer from inconsistent definitions and certain outcomes may be underreported because of classification issues (e.g. bioeconomy and circular-economy investments were counted only if projects received a specific “bioeconomy bonus” in the database. However, some projects that clearly contribute to the bioeconomy or circular economy did not receive that bonus).
Accelerate pending calls and approvals, particularly under RSO 1.3, 1.4 and 2.6, to ensure timely achievement of targets. Strengthen incentives for SME investment, especially in innovation and resource efficiency, given current economic uncertainty. Maintain flexibility in programme management, allowing reallocations between measures where demand is weak. Improve indicator definitions and data collection for some indicators (e.g. dissemination actions) to ensure consistent monitoring.
Evaluation of the implementation of the Sud-Provence Alpes-Côte d’Azur ERDF-ESF+-JTF Programme, 2021-2027
Évaluation de la mise en œuvre du programme FEDER-FSE+-FTJ Provence-Alpes-Côte d'Azur et Massif des Alpes pour la période 2021-2027
Sud-Provence Alpes-Côte d’Azur ERDF/ESF+/JTF 2021-2027
http://www.teriteo.fr ; https://www.astereuropeconseil.fr
The evaluation examines the mid-term implementation of the programme. It covers the implementation period from 2022 to 2024.
To assess the implementation of the programme and provide recommendations for revisions, given the changes in the context.
Monitoring data analysis (Synergie and Viziaprog database)Analysis of statistical office data.Interviews with the OP and project leaders and various stakeholders.9 workshops on implementation with the monitoring committee and the MA departments.
The launch of the programme was delayed due to the COVID-19 pandemic, which led to a late adoption of the 2021–2027 regulatory framework. However, the OP had anticipated launching certain calls for projects as early as the beginning of 2022 on established SOs, allowing for a catch-up effect.For PA 1 “Innovation potential and regional business development”, the launch was delayed, but implementation is now underway. However, several challenges were encountered, such as the processing certain of applications or the human resources required.For PA 2, “Preserving the region's resources by supporting the environmental and energy transition,” and PA 4, “Developing skills,” only some SOs have been activated.The implementation of PA 3, “Promoting sustainable, low-carbon urban mobility,” has not yet begun. However, the budget allocated to this priority may prove insufficient.PA 5, “Integrated territorial development,” is generally performing very well. Notably, programming under the new rural component (OS 5.2) has reached 80% of the allocation.PA 6, “Alpine Massif,” is making satisfactory progress at the midpoint.PA 7, “Just Transition,” has taken longer time to launch than other PAs, as this is a new topic and approach in the region.
No limitations are reported, but methodological precautions regarding the analysis of follow-up data are provided.
Implement the “annual procedure review” in order to develop an action plan for procedure optimisation.Implement risk management measures and integrate them into the procedures with a focus on proportionality.Structure and strengthen the partnership between the European Funds division and the operational divisions.Strengthen and standardise collaborative efforts regarding the preparation of technical reviews and monitoring of operations.Improve communication about the programme.Align the allocated resources and management decisions with the ambition programme.
Ex-post evaluation of the 2007-2013 communication plan
Evaluation ex-post du Plan de Communication plurifonds 2007-2013
NOP France FSE; OP Franche-Comté ERDF 2007-2013
Impact evaluation of urban and transport measures in Occitanie supported under the Languedoc-Roussillon and Midi-Pyrénées et Garonne OPs, 2014-2020
Évaluation de l'impact du FEDER sur l'urbain et le transport en Occitanie
ROP Languedoc-Roussillon 2014-2020; ROP Midi-Pyrénées et Garonne 2014-2020
The evaluation examines the ERDF Fund on energy transition (especially in priority neighbourhoods - QPV) and the development of more environmentally friendly transport networks. Its scope covers the 2014-2020 implementation period.
To assess the implementation, the effectiveness and impact of measures for urban development and transport, and the coherence of the ERDF Fund. To provide recommendations for the 2021-2027 period and post-2027 strategy.
TBE approach.1 inception interview.Analysis of monitoring data (Synergie database and the latest implementation report available).3 interviews with pilots (instructors and managers of the former Midi-Pyrénées region) and partners.16 interviews with project leaders, analysed in the form of 6 case studies of supported projects and 3 beneficiary portraits.
In Languedoc-Roussillon, the choice of an Integrated Territorial Approach (ITA) was based on voluntary applications and multi-thematic local strategies. In Midi-Pyrénées, the intervention was based on city contracts, providing access to funding for eligible projects. This difference in approach has had significant consequences on the mobilisation of territories.In total, 82 projects were financed (EUR 34.8 million ERDF, average programming rate of 93%), enabling the redevelopment of public spaces, the creation/renovation of sport, cultural, educational and health facilities, with a significant impact on quality of life and social cohesion for the inhabitants of QPVs.The role of the ERDF has not been primarily that of trigger, but rather of accelerator and amplifier of projects, facilitating their implementation, increasing their ambitions.The arrangements for using European funds have proved to be broadly complementary to territorial strategies, such as: State-Region Planning Contracts (CPER), the “Occitanie 2040” Regional Plan for Land Use, Sustainable Development, and Territorial Equality (SRADDET).However, synchronisation issues were observed, particularly at the beginning of the period with the late signature of city contracts, and at the end because of differences in the calendars of ERDF and the National Agency for Urban Redevelopment.
The long time elapsed from the start of the programming period limited the information retrievable. The merging of the two OPs causes difficulty in contacting pilots and partners (3 possible interviews out of the 5 to 10 originally planned).
Support the sustainability of funding for the development of soft mobility and the revitalisation of disadvantaged neighbourhoods in the post-2027 period.Strengthen the coordination of public funding and foster a cross-financing approach for the post-2027 period.Capitalise good coaching and animation practices from 2014-2020 for the end of the 2021-2027 programming period.Establish an appropriate monitoring system to measure the impact at programme level.Optimise the timing of evaluations.
Evaluation of the Interregional Pyrénées programme, 2014-2020
Analyse qualitative du programme opérationnel interrégional Pyrennées 2014-2020
Interregional programme Pyrénées 2014-2020
The evaluation examines the implementation and effects of the measures supporting environmental and heritage protection and economic development.
To assess the effectiveness, efficiency and external coherence of the programme. To provide recommendations for the future programme.
Analysis of monitoring data and statistical office data.Document analysis.2 inception interviews.6 interviews with intermediate bodies.8 case studies and 4 beneficiary profiles based on 16 interviews with beneficiaries.Benchmarking with three other French interregional programmes.
Priority 1 (“economic development”) has funded individual projects in the timber sector, networking initiatives and events related with the “mountain economy” and tourist accommodation.The support to individual projects has contributed to the modernisation of small craft businesses, their viability and the preservation of jobs. Networking activities and the support to the organisation of specific events contribute to the promotion of the Pyrénées as a tourist destination and the skills development of professionals in the tourism and timber sectors.Priority (“heritage and tourism”) has funded a diversity of individual projects in the tourism sector.The support contributed to the diversification of tourism services, the emergence of year-round destinations and the modernisation of accommodation facilities. It also produced positive effects on the number of tourists.There is good level of coordination with regional OPs covering the area and the Interreg programme between France, Spain and Andorra.
Foster the partnership with local authorities.Prioritise the support for projects relating to environmental and heritage preservation and the diversification of tourism servicesImprove the clarity of the support provided for the intermediary bodiesConduct more regular surveys of beneficiaries.Develop tools to monitor the number of tourists.
​​Evaluation of the project selection system used under the European Funds for Małopolska, 2021-2027​
Ewaluacja systemu wyboru i oceny projektów oraz kryteriów wyboru projektów w ramach programu Fundusze Europejskie dla Małopolski 2021-2027
Jan Szczucki, Maciej Gajewski, Robert Kubajek, Wiktor Mrozowski, Tomasz Schimanek, Justyna Witkowska
Policy & Action Group Uniconsult; www.pag-uniconsult.pl, biuro@pag-uniconsult.pl
The subject of the evaluation is the system used for selecting project proposals under the programme.
To assess the validity of the criteria and project selection system, and the consistency with the objectives of the programme.
Analysis of monitoring data and statistical office data.Desk analysis of relevant documents.7 interviews with implementing institutions, 10 with experts and 8 with representatives of consulting firms.CAWI/CATI survey of 719 beneficiaries, 380 unsuccessful applicants and 226 project evaluators.2 workshops.
The project assessment and selection system is objective and transparent, and ensures equal access for all applicants.The criteria are overall clear, easy to understand and properly aligned with the programme's objectives.The length of the selection process is the problem most frequently mentioned by those surveyed.The beneficiaries found the criteria relating to horizontal issues, such as sustainable development and equal opportunities, the most difficult to interpret.
Organise more training sessions and informational activities on horizontal principles.Organise additional training sessions for staff of the institutions involved in the selection process, covering the areas addressed by the call for proposals.Organise informational meetings with experts to standardise the application of the evaluation criteria.
Mid-term evaluation of Priority 2 “A digital Region” under the Central Region Programme, 2021-2027
Evaluarea interventiilor de digitalizare in beneficiul cetatenilor, companiilor si guvernului
Andreea Savin, Adina Dragoman, Alina David
Civitta, www.civitta.com; Lideea, www.lideea.eu
The report is a mid-term evaluation of digitalisation interventions under the programme. It assesses the implementation progress of the interventions for SMEs and local public administrations. Data used go up to January 2026.
To assess the state of implementation, the relevance, effectiveness, efficiency and coherence of measures on digitalisation. To assess the contribution to the digital transformation of the regional economy and the local public administration. To support decision-making by producing evidence-based conclusions and operational recommendations.
Document analysis.Analysis of monitoring data and statistical office data (National statistical Institute, EUROSTAT, DESI, EIB, etc.).TBE approach used for the two specific objectives evaluated (SO1.2 and SO1.4).Interviews with MA staff, 7 funding applicants, 2 digitalisation consultants and 2 local public administration representatives.Three thematic workshops with SME beneficiaries, potential local public administration beneficiaries and consultants.Panel of experts.Online survey (on Survey Monkey platform) of 40 beneficiaries with contracted projects and under implementation.Online survey to all 339 applicants. 96 answers received, response rate 28%.11 case studies of beneficiaries (7 SMEs and 4 local public administrations) to deepen evidence on implementation patterns, benefits and constraints.
Overall, the interventions are strategically relevant and well aligned with EU, national and regional digital strategies. They respond to persistent needs linked to the low digital maturity of SMEs, fragmented public-sector digital processes and territorial disparities.The main results observed are in SMEs, where integrated digital solutions are already associated with improvements in operational efficiency, productivity and the quality of internal processes. By contrast, for local public administrations the expected effects remain largely potential, as no projects under the dedicated SO 1.2 call had yet been contracted.Progress is strongest where beneficiaries have sufficient internal capacity, effective organisational leadership and good collaboration with IT providers and consultants.The main barriers identified are lengthy assessment and contracting procedures, administrative rigidity, and shortages of specialised human resources.Effects on efficiency seem to be positive but they are provisional. SMEs largely perceive benefits as outweighing costs, especially where software, equipment, staff training and process redesign are combined.Demand for funding was high, especially from local public administrations. Coherence with other digitalisation initiatives is moderate and mostly operational.Horizontal principles are generally incorporated in projects, but they are often treated more as compliance conditions than as substantive design features, especially in SMEs.
Most projects were still at an early stage of implementation. For measures covering local public administrations, no projects had yet been contracted at the time of the assessment. The report also notes a lower response rate to surveys from local public administrations and the consequent need to rely on triangulation of survey, interview and case-study evidence.
Maintain administrative rigour while allowing justified technological flexibility, notably through recognition of equivalent digital solutions and clearer technical guidance.Increase transparency and predictability in project assessment and contracting, including more regular status information and faster handling of recurring technical issues.Provide differentiated support according to beneficiaries' digital maturity, both during current implementation and in the design of future calls.Facilitate peer learning and exchange of experience between beneficiaries, including practical examples of projects with visible operational results.For the post-2027 period, introduce a digital maturity screening tool, differentiate calls by maturity level and test guidance requirements earlier to improve clarity and consistency.
​​Evaluation of measures supporting the internationalisation of SMEs under the Piemonte ERDF OP, 2014-2020​
I progetti integrati di filiera nel POR FESR 2014-2020. Azione III.3b.4.1
https://www.ires.piemonte.it; piazza@ires.piemonte.it
The report focuses on the features and results of the measures supporting the internationalisation of SMEs funded under the OP. Targeted SMEs operate in aerospace, automotive, mechatronics, clean tech, healthcare, textile, food, and clothing/design sectors. Data go up to April 2023.
To assess the implementation and the results of the measure, focusing on the features of beneficiary firms, with a view to providing policy recommendations.
Desk research on programme documents.Analysis of statistical office data and monitoring data.Interviews with 8 beneficiary enterprises, with representatives of the regional administration and with “Ceipiemonte”, the regional entity dedicated to the internationalization of firms and responsible for implementation.
The intervention provides for a discount on participation costs in networking activities (fairs, business conventions, meetings with potential buyers) and internationalization support projects. Eligibility requirements are consistent with the regional S3.Overall, 1,279 regional firms participated in the projects (between 2017 and 2022). More than half of beneficiaries are located in the province of Torino, and about 80% of firms have participated in more than one edition of the measure. Activities in the mechatronics, automotive and food sectors had the highest participation rates, while those in textile had the lowest. About half of beneficiaries participated in more than one activity.Enterprises were satisfied with the support received. The measure may prove to be particularly beneficial to smaller firms in the long term, especially if their activities can be framed as part of a wider regional productive system.The measure reinforced interactions among firms and has the potential to foster collaborations and synergies.Certain sectors – such as aerospace – are difficult to access for new players, and national regulations can pose a barrier to the entry of Italian firms into foreign markets.Smaller firms find it difficult to reconcile ordinary workload and participation in the projects.Without the funding, some beneficiaries would not have been able to participate in the projects, would have delayed participation or reduced their involvement.
Keep supporting the entry of new players in sectors and/or foreign markets which are difficult to access.Provide support for firms to balance ordinary workload and project participation.Tailor the initiatives to firm size.
Evaluation of measures on education and career guidance, under the Piemonte ESF OP, 2014-2020
Valutazione di Obiettivo Orientamento Piemonte - I colloqui individuali di orientamento - Rapporto 2022
IRES Piemonte, https://www.ires.piemonte.it/
The report is the 2022 annual evaluation of the regional measures on educational and career guidance under the OP. Data used go up to December 2022.
To assess the coverage of the target population, the delivery modes, and the ability of lifelong educational and vocational guidance to adapt to students' needs. To assess the contribution to school-to-work transition and to the reduction of dropout rates.
Document analysis.Literature review.Analysis of statistical and monitoring data.Five semi-structured interviews with guidance counsellors operating in the four territories covered.Online survey to capture the expectations, experience, and choices of young people aged 16+ who used individual services in 2021/22 (61 responded out of 761 contacted - response rate 8.4%).
Delivery and participation increased as compared to 2020/21 (+17% measures; +8% participants), which shows the programme’s capacity to operate at scale after the pandemic period.Project delivery shifted back to face-to-face; remote delivery dropped to marginal levels and is treated as a complement rather than a substitute for relationship-based support.Use of individual interviews increased strongly. These are used in two ways: as preventive transition support when linked to school; and as more intensive intervention for vulnerable youth when accessed via employment/welfare/dispersion channels (especially in the metropolitan area).User feedback (16+) suggests that interviews help clarify options (including redirecting jobseekers towards study/training), but the pathways are often perceived as too short (too few interviews).
Only a subset of eligible participants had usable contact details, and the survey response rate is low. Also, there is a geographical representativeness problem in the survey, since most of the respondents live in the province of Torino.
Increase proximity/continuity between schools and guidance centres in the metropolitan area.Strengthen the flexibility of individual interviews so they can adapt (in intensity/structure) to different user needs and to the local territorial context.Involve upper-secondary schools more in the activation of individual pathways and promote a shared “reporting/signalling” system for students in difficulty before they drop out.For early school leavers aged 17–18 without a qualification, use traineeships that do not require a qualification, to keep them engaged until they can enter adult training (18+).Support and invest in the relational/network dimension to make the orientation programme a stable territorial reference point for “permanent” guidance across life stages.
Final evaluation report of the Piemonte ERDF OP, 2014-2020
Rapporto di valutazione conclusivo degli Assi del POR FESR 2014-2020 del Piemonte
This report is the final evaluation of the OP. Data used go up to June 2024.
To assess how the programme performed across the main policy areas: research and innovation, digitalisation, SME competitiveness, transition to a low-carbon economy, sustainable development, and the reprogramming carried out during COVID-19. To examine financial and procedural progress, and to check the concrete outputs and immediate effects of the projects. To identify the contribution of the programme to the result indicators, using evaluation questions linked to the logic of the programme.
Document analysisAnalysis of previous evaluation thematic reports and studies on the programmeAnalysis of statistical and monitoring dataAnalysis of administrative data at firm levelAnalysis of the programme intervention logic
The programme helped strengthen R&D and innovation, with several spending-related indicators exceeding targets and strong contributions from the private sector and large innovative firms. However, systemic collaboration between firms and external research actors remained below the target (31.2% vs 45%).Digitalisation of services progressed particularly well, with 125 digital applications activated against a target of 30, but broadband coverage was delayed due to bureaucratic and infrastructure bottlenecks and COVID-19 disruption.Support for SME competitiveness and internationalisation performed well; measures for export promotion and integrated supply-chain projects reached far more firms than initially expected.The uptake of access-to-credit instruments was very successful, as 15,827 firms were supported against a target of 4,881, but SMEs still face broader structural problems for credit access.Urban and cultural interventions were mostly effective; however, implementation was territorially uneven, as some cities did not carry out the planned interventions, which reduced the overall regional impact.The programme was heavily reprogrammed, with substantial resources shifted and used in response to COVID-19.
Some inaccuracy in the monitoring system might have led to underestimation of target achievements.
Continue strengthening collaboration between firms and the research system, and provide stronger support to knowledge-intensive firms, so that the innovation gains become more structural and consistent with the Smart Specialisation Strategy.In the 2021–2027 cycle, monitor complex projects more closely and use more efficient administrative procedures for calls, so that implementation and spending can be managed more efficiently.Revise the monitoring system and make greater use of integrated monitoring and statistical sources to track skills, employment, and wider programme effects more credibly.Reinforce access-to-credit measures, including stronger public guarantees and measures that reduce credit rationing.Improve the local implementation capacity and coordination in urban, cultural, and energy interventions, and distribute interventions more evenly across the territory.
Evaluation of measures on early career education under the Piemonte ESF OP, 2014–2020
Valutazione di Obiettivo Orientamento Piemonte - Diffusione e uso del kit regionale di orientamento precoce - Rapporto 2023
The report evaluates the implementation of the Early Career Education toolkit and other measures developed to promote early awareness of professions and educational pathways in primary schools. Data used go up to June 2023.
To assess the implementation of the Regional Guidance System in Piemonte, with particular focus on the Early Career Education measures. More specifically, to examine how orientation activities are implemented in the region, to monitor the participation of adolescents and young people, and to assess the dissemination and practical use of the regional early guidance toolkit developed under the programme.
Analysis of monitoring data on orientation activities carried out in the 2022/2023 school year.Analysis of statistical office data.Desk analysis of the Early Career Education toolkit and the activities included in it.Five structured interviews with regional guidance coordinators and teachers who used the toolkit.Review of programme documentation and regional guidance policies.
During the 2022/2023 school year, approximately 94,000 people participated in career orientation activities.The majority of participants were adolescents aged 11–15, reflecting the programme’s strategic focus on early guidance interventions.The Metropolitan area of Torino accounts for the largest share of programme participants (around 46%).The Early Career Education toolkit represents an innovative instrument designed to introduce career exploration activities in primary schools through interactive exercises and group work.Teachers reported that the toolkit helps broaden children’s awareness of possible career paths and educational choices and facilitates classroom engagement.
Limited number of interviews with teachers and regional actors. Limited information on behavioural or educational outcomes.
Integrate orientation activities within school curricula, linking them to subjects such as civic education or standard teaching programmes.Introduce the early guidance toolkit from primary school starting at the age of 8, and orientation activities to preschool for 5-year-olds where possible.Strengthen cooperation between schools, training institutions, and regional actors by consolidating local networks and multi-level governance.Include the continuous development and widespread dissemination of innovative tools in the regional guidance system.Enhance the use of visual channels, such as videos, as narrative stimuli for children.Provide permanent workshops and training for teachers dedicated to the use of the early guidance toolkit.
Impact evaluation of digital and competitiveness measures in Occitanie under the Languedoc-Roussillon and Midi-Pyrénées et Garonne OPs, 2014-2020
Évaluation de l'impact du FEDER sur le numérique et la compétitivité des entreprises en Occitanie
http://www.teriteo.fr; http://www.taranconsulting.com
The evaluation assesses the measures for the digital development and the competitiveness of enterprises under the OPs. Data used go up to November 2024.
To assess the relevance, coherence, effectiveness and implementation of digital and competitiveness measures - specifically focusing on the FOSTER TPE-PME financial instrument, Contrats de Croissance, Pass Occitanie, and digital development schemes such as THD (High-Speed Broadband) and FTTH (Fiber To The Home). To identify lessons for future cohesion policy programmes.
Review of programme, strategy, and previous evaluation documents.Use of Theory of Change.Analysis of statistical office data.Analysis of monitoring data (extracted from the Synergie information system as of May/June 2024).Analysis of secondary data from regional economic observatories.Interviews: 5 scoping interviews, 11 interviews with pilots/partners, and 22 interviews with project promoters.Two online surveys: 73 responses out of 198 contacts for TO3 projects (37% response rate) and 946 completed responses for FOSTER TPE-PME beneficiaries (20% response rate).Use of qualitative counterfactual approach in the two surveys to assess the added value of European funding by analysing beneficiaries' statements regarding what would have occurred in the absence of the grant.8 case studies (project focus) and 4 beneficiary profiles/portraits (storytelling articles).
ERDF support was broadly relevant to regional needs, as 88% of beneficiaries would not have implemented their projects without EU funding.Projects supporting high-speed broadband infrastructure and digital uses effectively reduced the territorial divide through the expansion of THD and FTTH networks in rural areas; furthermore, REACT-EU strongly accelerated e-education and the digitalisation of public services in response to the pandemic, but territorial disparities remain.Grants for competitiveness and the FOSTER financial instrument strengthened SME investment capacity, innovation, and access to finance for over 4,700 enterprises. They also supported tourist accommodation upgrading and the modernisation of production tools.In general, positive effects of the measures are reported on working conditions, productivity, turnover, employment, and the leverage of private finance.A heavy administrative burden was reported by smaller businesses.
Limitations concern the availability of indicator data and the difficulty of measuring long-term impacts.
Continue supporting digital infrastructure and digital uses, with stronger targeting of remaining territorial disparities.Maintain and strengthen a diversified mix of grants and financial instruments, especially for SME access to finance and different stages of firm development.Increase support for advanced digital uses and innovation (AI, cybersecurity, data and e-education) within a responsible and resilient regional digital strategy.Improve coordination between ERDF interventions and regional economic development strategies, while simplifying support for beneficiaries, tailored to different contexts.
Meta-analysis and evaluation of the effects of regional health programmes under ESF projects in 2014-2020, in Poland
Metaanaliza i ocena Regionalnych Programów Zdrowotnych współfinansowanych ze środków EFS w okresie programowania 2014- 2020
All ROPs 2014-2020 except ROP Podkarpacie Voivodeship 2014-2020
The evaluation examines Regional Health Programmes (RHP) co-financed by cohesion policy funds in 2014–2020, focusing on their design, implementation, monitoring and reported effects across regions.
To conduct a meta-analysis and assess the implementation and effects of Regional Health Programmes funded in the 2014–2020 period.
Desk research, including previous evaluations.Statistical and monitoring data.Survey of 13 Monitoring Committee members.55 online workshops with representatives of institutions involved in design, implementation, and evaluation of RHPs.17 online expert panels.
A total of 92 RHPs were implemented, averaging 6 per region, but with strong regional variation, as Podkarpackie did not implement any RHP.The most common thematic areas were prevention of infectious, cardiovascular and oncological diseases.Designing RHPs was difficult for regional authorities because of demanding epidemiological justification, indicator-setting requirements, and rigid guidelines, which often delayed programme launch.Implementation difficulties were most visible in cases for which the availability of specialists was limited or programmes addressed narrow target groups.Monitoring was assessed moderately positively, but many implementers focused more on formal reporting than on actual effects on health, and satisfaction tools were not standardised across programmes.Reported effects were stronger in cardiovascular, oncological and rehabilitation programmes, in terms of earlier diagnosis, improved fitness/quality of life and higher health awareness.Only individual RHPs achieved all planned targets, and no region achieved all targets across all programmes.
The report mentions uneven regional evaluation coverage, pending evaluations in some regions, limited use of one regional evaluation due to irregularities, and monitoring/comparability issues.
Continue preventive support in future programming periods, given the scale of unmet health needs and the positive reported effects of many RHPs.Broaden the scope of programmes, especially towards mental health and environmental health, which are less covered.Ensure that the support from the Agency for Health Technology Assessment and guidance on the Tariff System provide more flexibility for regional adaptation, while preserving compliance with current medical standards.Test pilot innovative solutions before wider rollout.Involve more actively primary healthcare professionals in programme promotion and design more attractive information campaigns for residents.Standardise monitoring and reporting through more coherent indicators and, ideally, a central IT platform enabling ongoing tracking and exchange between regions.
Interim evaluation of the 'Data analysis support for the evaluation and management of the Czech education system ' project (DATA), under the Johannes Amos Comenius ESF+ OP, 2021-2027, in the Czech Republic
Evaluace individuálního projektu systémového: Datově-analytická podpora pro hodnocení a řízení vzdělávací soustavy ČR
Johannes Amos Comenius CZ ERDF/ESF+ 2021-2027
This first interim report evaluates the project DATA carried out under the OP. The project aims to strengthen evidence-based governance of the Czech education system through surveys, secondary analyses, a shared education database, an analytical unit and communication activities. The report covers implementation from March 2023 to August 2025.
To assess the risks to implementation and the achievement of objectives of the project to improve management and evaluation of the education system. To assess the quality and user relevance of the database; and the extent to which the tools created are supporting decision-making in education policy.
Document analysis including the project charter, implementation reports, risk registers, internal evaluation outputs and project materialsAnalysis of statistical data office and monitoring dataQuestionnaire surveys of 44 primary users (Ministry officials involved in decision-making) and 115 secondary users (staff from implementation bodies and education institutions)Nine interviews of people involved in project administration, project management, the steering committee, the internal evaluator, the MA, and key activities coordinators and staff.Two participatory workshops with 11 primary users and 10 secondary users.
Project implementation is broadly on schedule, the main risk is recruitment and retention of specialist data and IT staff because public-sector salary ceilings are not competitive.Good project governance and internal evaluation have improved coordination and adaptive management, while cooperation with other projects and interventions is functioning well.There are high levels of awareness and use among primary users inside the Ministry, but this is more limited awareness for secondary users, especially schools and municipalities.The database is only partly developed at this stage; users value easy search, transparency and accuracy, but find the concept and structure of the data base insufficiently clear.The DATA project is helping to systematise data work and strengthen analytical culture inside the Ministry, but wider practical use outside the Ministry is still limited.The project is useful to mitigate several data-management problems, but cannot by itself resolve broader organisational, data-sharing and legislative constraints.
The database is still under construction, so the related findings are provisional. Evidence on socially excluded localities and structurally affected regions is weak because response rates of municipalities were low.
Stabilise specialist staffing through a structured retention and recruitment plan, flexible contracts and capacity building activities.Strengthen targeted communication to less engaged secondary users, especially schools, municipalities and part of the academic sector, and publish outputs with a clearer timetable.Clearly explain the architecture of the database and ensure user-friendly public access outside the Ministry.Continue strengthening project coordination and address broader data-sharing, standardisation and legislative barriers that lie beyond the project’s direct remit.
Evaluation of the long-term ESF result indicator 'Number of people in a better labour market situation 6 months after leaving the programme' under the Łódzkie OP, 2014-2020 (4th measurement)
Szacowanie wskaźnika rezultatu długoterminowego EFS pn. „Liczba osób znajdujących się w lepszej sytuacji na rynku pracy sześć miesięcy po opuszczeniu programu” w ramach RPO WŁ 2014–2020 (pomiar 4)
Ewelina Dąbrowska–Nowak, Malwina Malczewska, Kamila Stefaniak, Aneta Gierowska, Wiktoria Świat, Maria Kotkiewicz, Justyna Romanowska–Zaparta, Joanna Syrda, Marek Skrzyński, Olga Korczyńska, Aleksandra Płaczek, Małgorzata Walczak–Gomuła
ASM Research Solutions Strategy, www.asmresearch.pl
The evaluation is the 4th measurement of the ESF long-term result indicator concerning the improvement of labour market outcomes of project participants under the OP (the first and second measurements were conducted in 2019, and the third measurement in 2022).
To assess the impact of support provided from ESF funds under the OP on changes in the situation of project participants who were employed at the time of entering the project, primarily by estimating the value of the indicator: “Number of programme participants that have improved their job market status within six months after termination of the programme”.
Desk research (programme and strategic documents).Analysis of monitoring data.Analysis of statistical office data (only on population).CATI/CAWI survey of 4,608 project participants and 372 beneficiaries.36 telephone in-depth interviews with project participants.
The value of the long-term result indicator was 20% for all the IPs (in previous measurement waves, the value of the indicator ranged from 17.0% to 17.6%) and it was mainly due to promotion and pay rise.The highest value, equal to 35.1%, was achieved for IP 9i (comprehensive support including counselling, training, internships, and social assistance, fostering long-lasting improvement of the professional situation of people at risk of exclusion).The lowest value of 13.3% was reported for IP 8vi (preventive health care).The usefulness of the support was enhanced by the durability of the results: almost all the participants managed to hold down their full-time employment and stable workplace.The effectiveness of the measures was higher for those who benefited from more than one form of support.5.9% of individuals who improved their situation on the labour market attributed this change largely to the project.Factors impacting on the value of the long-term result indicator included: type of provided support, number of forms of provided support, social-demographic factors (age and education level) and personal motivations of participants.
The evaluation results reveal an overrepresentation of individuals with certain characteristics, resulting from the absence of a requirement to ensure sample representativeness with respect to specific socio-demographic categories, such as age, level of education, or disability status.
Subsidise child-care institutions for children under the age of 3 and extend the scope of professional activation measures for parents in order to facilitate their return to work.Implement solutions aimed at increasing the number of support forms available for project participants and their integrated use.Prolong the duration of the support for IPs with the lowest values of the indicator and too limited time to conduct the activity, e.g. IP 8vi.
Evaluation of financial instruments supporting access to credit for SMEs under the ERDF Piemonte OP, 2014-2020
Gli strumenti finanziari di accesso al credito, POR FESR 2014-2020. Fondo MPMI e Sezione speciale Piemonte Fondo di Garanzia. (Azioni III.3C.1.1 e IIID.6.1)
https://www.ires.piemonte.it/ piazza@ires.piemonte.it
This report evaluates two financial measures under the OP, designed to improve credit access for businesses: the Guarantee Fund enhances public guarantees on SME loans to provide vital liquidity, while the MPMI Fund is a revolving fund offering subsidised loans for investments in innovation, environmental sustainability, and workplace safety. Data used go up to the end of 2022.
To assess the results of the measures, the implementation process and the characteristics of beneficiary firms. To assess impact of the measures on induced investments in research, innovation, and energy efficiency. To identify strengths and weaknesses emerged during the administrative phases to provide policy guidance for the 2021-2027 programming cycle.
Analysis of official statistics and monitoring data. Analysis of relevant documents, including the programming documents and related regional and national legislative acts. Analysis of administrative data (from the Aida database) concerning balance sheet data, such as turnover, value added, and productivity for 4,496 beneficiary companies. Interviews conducted with officials from the Monitoring, Evaluation, and Control units as well as the Economic Development Promotion and Credit Access unit within the Regional Directorate for System Competitiveness. Analysis of secondary data (Bank of Italy, etc.).
The Guarantee Fund proved to be an excellent practice due to its streamlined and standardised management, successfully providing immediate liquidity to over 15,000 businesses—predominantly micro-enterprises with more fragile economic profiles and lower credit ratings—particularly during the peak of the pandemic crisis in 2021. The measure effectively reached firms with lower turnover and value-added than the regional average, specifically targeting those with significant liquidity needs during the emergency. The MPMI Fund was characterised by greater selectivity and procedural complexity. The instrument supported approximately 138 larger companies concentrated in the manufacturing sector, though it had lower resource absorption capacity due to competition from national measures and initially low interest rates. Despite these problems, the fund remained operational as a revolving instrument and saw a significant increase in the value of approved funding during 2021.
The pandemic and the national measures (such as the "Liquidity Decree") fundamentally altered the nature of the Guarantee Fund, making it difficult to assess the original objectives planned before the emergency. Furthermore, detailed balance sheet information from the Aida database was only traceable for approximately 30% of beneficiary companies. The small sample size of the MPMI Fund also led to high statistical variance. Finally, the qualitative analysis was primarily restricted to interviews with internal regional managers and administrative stakeholders rather than a broader group of external beneficiaries.
Build on the success of the Guarantee Fund to ensure that financial tools use national-standardised processes that minimise bureaucratic burdens for both banks and SMEs. Coordinate regional and national instruments to avoid overlaps. Reduce administrative complexity by eliminating double assessments where possible, moving towards a single-entry evaluation process. Broaden sectoral eligibility to better support the diverse regional economic fabric. Continue the support for micro-enterprises with lower credit ratings and weaker economic profiles that face significant barriers to accessing traditional bank credit. Enhance monitoring and data integration by linking regional monitoring systems with national databases to improve data traceability.
Evaluation of the implementation of the Good Governance ESF OP, 2014-2020
„Оценка на изпълнението на ОПДУ в периода 2015-2017 г. ипроучвания, подпомагащи процеса по планиране събирането наданни за всички оценки след 2017 г., включени в Плана за оценка наОПДУ”. Първи доклад за оценка (финален): Оценка на изпълнението на ОПДУ
Ecorys South East Europe LTD; DnD Consulting; Bulgarian Consultancy Organisation
ecoryssee@ecorys.com; www.d-d.be; www.bco.bg
This evaluation assesses the implementation of the Good Governance OP 2014-2020 co-financed by ESF in Bulgaria.
To assess the results so far achieved by the OP and to make recommendations for the implementation of the OP in the period 2018-2020.
Desk research and data analysis.Focus groups.Interviews with Managing Agency and beneficiaries.Online survey of managers and project coordinators.Case studies.
The OP’s objectives are still relevant for Bulgaria’s socio-economic situation.The targets for some result indicators are not consistent with the European Commission guidelines for the 2014-2020 programming period.The implementation of the OP has been delayed for various reasons, but especially for a lack of an agreement among key stakeholders on how to achieve the objectives of administrative and judicial reform and the introduction of e-governance.Certified expenditure in March 2018 was below the expected milestones: only 7% for PA1, 9% for PA2 and 13% for PA3 and there is a risk of not achieving targets for 2018 for the first two.The electronic submission of proposals and the information system for management and monitoring have reduced the administrative burden, but accounting costs remain heavy because of the large volume of supporting documents required.The OP does not explicitly include measures for social innovation or consider this to be a horizontal principle.A problem for advertising the OP is the lack of a recognisable and easily accessible website.
There is a need to:change the targets for some indicators;organise meetings with beneficiaries to clarify the implementation rules and legislative changes;monitor the difficulties of some in preparing quality project proposals and managing projects;alleviate the unnecessary administrative burden on cost accounting;set up sub-committees for key aspects, such as for e-governance, administrative capacity and justice;organise large-scale events to launch projects for administrative judicial and e-Government reform.
Evaluation of measures supporting the socio-economic integration of marginalised communities under the Human Resources Development OP, 2014-2020, in Bulgaria
Оценка на ефективността, ефикасността и въздействието на мерките за социално-икономическа интеграция на маргинализирани общности с фокус върху ромите
OP Human Resources Development 2014-2020
Vladislav Georgiev; Kamelia Petkova; Stefan Atanasov; Irena Vladimirova
Prime Consulting Ltd - vladislav@primebg.eu
The evaluation examines measures supporting the social inclusion of marginalised communities, especially Roma. Data go up to 31 July 2021.
To assess the progress in implementing the measures for social inclusion, as well as the main results, impact, effectiveness and efficiency up to July 2021.
Analysis of documents.Analysis of monitoring and statistical data.Analysis of administrative data (unemployment registers).Interviews with stakeholders.Survey of beneficiaries (45% response rate).DiD counterfactual analysis.
The implementation of integrated projects has been favoured since target groups usually have several disadvantages and need a tailored and long-lasting approach.Projects are most effective when stakeholders and representatives of marginalised communities are involved in the planning, implementation and monitoring phases, to avoid lack of motivation among potential participants.According to the counterfactual analysis, 12 months after exiting the project, the employment rate of participants who were unemployed previously was 14.5 pp higher than for a control group, though the employment rate of participants previously inactive was lower than for a control group.No effects on access to healthcare are evident from the analysis, but they are positive both for school attendance of children and preparedness for school.Projects are generally efficient, with unit costs per participant being lower than planned, except for projects where representatives from the Roma community have been involved to support participation.Projects had little effect on tackling stereotypes, but they increased the self-confidence of participants.
Results from the counterfactual analysis should be regarded with caution due to the lack of data needed to set up a reliable control group.
Increase the involvement of Roma cultural mediators to support participation in projects.Track the use of financial resources better in relation to the results achieved, through ad hoc documentation.Identify indicators monitoring the ex-post condition of participants over the long-term, and not only 4 weeks after exiting projects.
Evaluation of the effectiveness, efficiency and impact of measures for unemployed and inactive under PA 1 of the Human Resources Development OP, 2014 – 2020, in Bulgaria
Оценка на ефективността, ефикасността и въздействието на мерките за подкрепа на безработни и неактивни лица по приоритетна ос 1 на ОП РЧР 2014 – 2020
Irena Vladimirova, Nedyalka Dimitrova, Todor Todorov, Boyka Boynuzova, Nikolina Stoyanova
Ecorys South East Europe EOOD, Tel.: (+359 2) 8155 684, E-mail: EcorysSEE@ecorys.com
The evaluation examines 9 measures on training and subsidised employment under PA1 of the OP up to December 2020.
To assess the implementation, effectiveness, efficiency and impact of 9 measures for training and subsidising the employment of the unemployed and inactive.
Monitoring data.Statistical data.Interviews with MA and implementing bodies.Online survey of beneficiaries (14% response rate, out of 854); telephone survey of participants (98% out of 1,122) and a control group (72% out of 1,122) based on Employment Agency data; survey of participants in the “Promoting entrepreneurship” measure (4% out of 474). Counterfactual analysis (PSM and DID based on Employment Agency data and survey).
OP measures are more attractive than similar national measures because of a greater variety of activities and higher remuneration for participants.The result indicators for the acquisition of qualifications show little progress, because of a lack of interest among employers and the high qualification requirements to access courses.The implementation procedures are flexible in adapting to the needs of the target group and the means of coordination with beneficiaries have ensured timely implementation.Deficiencies in monitoring procedures affect the accuracy of the data collected.59% of unemployed participants were employed 6 months after and 70% 12 months after. The effectiveness of measures is positively correlated with the level of education and disability condition of participants.The net employment effects after 12 months are positive for all target groups - for unemployed over 54, +27% pp, low-educated, +32 pp and disabled, +45%, though the positive trend in labour market conditions may be responsible. All the measures are cost-efficient, mainly because of their complexity, duration and territorial scope.
The low response rate to the survey of beneficiaries and participants in the “promoting entrepreneurship” measure means that the results may not be representative. The control group is smaller than planned because of deficiencies in microdata and refusals to respond. Findings of the counterfactual analysis need to be interpreted with caution because of data limitations.
Ensure greater coordination between the Human Resources Development OP and the Science and Education for Smart Growth OP.Define measures at the regional level given the specifics of socio-economic development and the labour market in regions.Improve the quality of microdata collected by introducing procedures for verifying completeness and correctness of data.
Evaluation of the community-led local development approach (CLLD) supported by the OP Environment 2014-2020
Резюме от Оценката на прилагането на ОПОС 2014-2020 чрез подхода Водено от общностите местно развитие (ВОМР)
Ecostil association (including Ecorys South-east Europe and BIM Consulting)
Ms Mariana Stefanova, E-mail: mariana.stefanova@ecorys.com; ecoryssee@ecorys.com; Tel: +359 28155680; Fax: +359 29816032 and E-mail: office@bim-bg.com; Tel: +359 29872482; Fax: +359 29816032
Progress report on PAs 1,2,3,4 and 6 of the Environment OP, 2014-2020, in Bulgaria
Резюме на доклад за оценка на напредъка по изпълнението на приоритетни оси 1,2,3,4 и 6
Ecorys South East Europe - BIM consulting ltd.
Ecorys, www.ecorys.com/global - BIM, www.bim-bg.com
The evaluation examines the implementation of the Environment OP 2014-2020 in Bulgaria.
To assess the support under the OP in respect of water, waste, biodiversity, flooding and landslides and technical assistance.
Monitoring data.Interviews with stakeholders.Surveys of beneficiaries and potential beneficiaries.Survey of MA staff.Focus group with MA.
The ERDF support is appropriate for the different areas of intervention, though there are problems with the CLLD approach for nature conservation measures.Progress in spending and achieving targets and results is limited, especially because of a lack of technical expertise in most municipalities, delays in settling up managing bodies.Application procedures and monitoring systems are overall fine, but beneficiaries have difficulty in ensuring completeness of project documentation and in cooperating with the Ministry of Environment.Despite some interest in using financial instruments, problems arise from lack of experience in their application and “competition” with grants.The horizontal principles are well integrated in the OP’s implementation and ex ante conditionalities have been met.
Include the state enterprise in flood prevention as a potential beneficiary of support given the lack of interest of municipalities.Refine the strategy in respect of waste management, improving information campaigns and the use of financial instruments.Speed up the implementation of projects, provide information and guidance to applicants to avoid non-achievement of targets.Provide specific training on financial instruments to both MA staff and beneficiaries and through exchange of experience.
Evaluation of innovative transnational projects under the Human Resource Development OP, 2014-2020, in Bulgaria
Оценка на иновативни транснационални проекти по процедури по Оперативна програма „Развитие на човешките ресурси“ 2014 – 2020
Mariana Stefanova, Magdalena Ivanova, Camelia Petkova, Raina Timcheva, Eleonora Ivanova
MB Assist Ltd, linkedin.com/in/mariana-stefanova-85710632
The evaluation aims to assess the effectiveness, performance and sustainability of projects supporting the transfer and introduction of social innovation, good practices, peer-to-peer approaches through transnational cooperation. Data go up to June 2023.
To build a conceptual model of theory-based approach on innovative practices and models, to examine whether, why and how the intervention leads to the achievement of expected or observed outcomes.
TBE approach. Contribution analysis. Documents review. Interviews. Case studies. Comparative analysis. Stakeholder analysis. Analysis of good practices. Experts’ assessment. Analysis of monitoring data.
Most of the practices aim at improving (“modernising”) existing practices, while others focus on filling in gaps and deficits (“recovery”). A smaller number of innovations have the potential to reform the labour market or social services (‘transformation’). Over 320 social innovations have been supported and almost 4,700 persons benefited from these innovations and/or contributed to them. The sustainability of social innovations introduced is a key challenge for beneficiaries. There are four main factors in this respect: 1. financial security; 2. specific equipment purchased; 3. specific product created for use by members of associations and companies and 4. clear benefits for beneficiaries. Some of the products require regular updating and/or expansion of the content to guarantee the effects.
The scope of the evaluation of results carried out by beneficiaries was not clearly understood by them. Additional indicators have been formulated by the evaluators to compensate for those not capable of capturing the effects of the measures.
Extend the implementation period of projects on social innovations. Introduce a mechanism to track the sustainability of implemented social innovation, which includes clear responsibilities of all parties. Make sure an evaluation in a predefined format is carried out by the beneficiary within one year after the implementation.
Analysis of the implementation of the Administrative capacity ESF OP, 2007-2013
Анализ на изпълнението на Оперативна програма „Административен капацитет“ и Оперативна програма „Техническа помощ“ във връзка с изготвянето на окончателните доклади по двете програми
The evaluation examines the interventions funded by the Administrative Capacity OP 2007-2013 and attempts to identify success and failure factors in its implementation and examples of good practice, in order to make recommendations for 2014-2020.
To assess the results achieved and the extent to which needs were met and to describe the main problems encountered in implementation of projects.
Online survey with MA, beneficiaries, NGOs and other stakeholders;focus groups with MA and NGOs;data analysis;desk analysis of completed projects.
The measures financed by the OP meet the current needs of public authorities;the most active beneficiaries are municipalities;the targets for the result indicators have been achieved;electronic services to improve public authority performance have been developed;the OP led to an improvement in human resource management, an increase in staff qualifications and the development of a human resource assessment system;the OP helped develop a model of budgeting in the Supreme Judicial Council, to improve the Unified Information System for Combating Crime and to establish a network of judges;the main objective of the technical assistance priority, to absorb the ESF financing available, was achieved.
Additional investments are necessary to improve human resource management and develop public-private partnerships;more attention and importance should be given to training interventions;indicators relating to horizontal objectives should be taken into account.
Assessment of the effectiveness, efficiency and impact of measures for promoting the social economy under the Human Resources Development OP, 2007-2013 and 2014-2020, in Bulgaria
Оценка на ефективността, ефикасността и въздействието на мерките за насърчаване на социалната икономика, финансирани в рамките на ОП РЧР 2007 – 2013 и ОП РЧР 2014 - 2020
OP Human Resources Development; OP Human Resources Development 2014-2020
Ms Kamelia Kaloyanova, Dr. Alexander Naydenov, Dr. Maria Prohaska, Ms Petya Yakimova, Ms Milica Doychinova, Ms Silvia Todorova, Ms Yana Spasova, Ms Ruska Boyadzhieva
Five Consult EOOD - Тel: +359 2 418 6459, Е-mail: info@fiveconsult.eu, Website: https://dsmsconsulting.wixsite.com/fiveconsult
The evaluation examines ESF and YEI support for the social economy financed by the OP, including through the Risk Sharing Micro Finance Facility.
To assess the contribution of ESF and YEI support, including through the Risk-Sharing Micro Finance Facility, to the viability of social enterprises and to raising the social status of beneficiaries.
Analysis of monitoring data.Analysis of data from the Commercial Register, Register of Non-Profit Legal Entities, the Register of Social Enterprises and official statistics.Focus groups and interviews with stakeholders and beneficiaries.Case studies.
81 new social enterprises were created in 2007-2013 (54% of the target) and 7,789 people started working in the social economy (target 2,000).Targets for the number of inactive/unemployed supported for 2014-2020 have been exceeded (2,399 as against 1,640).Schemes supporting social enterprises with the lowest average costs have the worst results (“A chance for all” and “Compassion”).In both periods, employment levels increased, especially in small enterprises with less than 10 employees.The sustainability of projects is questionable and it seems unlikely that enterprises would continue them without funding.In 2014-2020, 63% of beneficiaries were employed immediately after completing programmes and 57% a year after.At end-September 2020, 210 micro-credit contracts had been signed. But interest in micro-credit is generally lacking as social enterprises rely mainly on grants, which involved only a small co-financing rate.
Lack of sufficiently reliable statistical information over time on the enterprises supported at municipal level after the end of the intervention, which makes it hard to assess the sustainability of projects.
Consider investing in larger projects to increase their effectiveness and ensure sustainability.Design separate support schemes for start-ups and existing companies, as the needs are often different.Adopt a balanced combination of investments supporting the building of a business model and the development of the enterprise after the funding period to increase sustainability.
Evaluation of the YEI within OP Human Resources Development 2014-2020 (ESF+YEI)
Оценка на мерките по Инициативата за младежка заетост в рамките на приоритетна ос 1 на Оперативна програма „Развитие на човешките ресурси” 2014–2020
DZZD Sigma Metrics – Sigma Hat – Global Metrics Ltd.
Sigma Hat, Acad. Stefan MladeNov Str. Sofia, Bulgaria, www.sigma-hat.com - Global Metric Ltd., 42 Alabin Str., Sofia 1000, Bulgaria, globalmetrics.eu/en/, office@globalmetrics.eu
The evaluation is concerned with the YEI element of the OP Human Resources Development 2014-2020. This is the final report.
The objectives of the evaluation is to assess the effectiveness, efficiency and impact of the YEI implemented through three schemes: “Youth Employment”, “Active” and “Training and employment for young people”.
The evaluation adopted qualitative and quantitative methods:desk research;analysis of relevant policy documents and studies/researches;analysis of statistical data;focus groups with participants and/or potential participants;in-depth interviews with stakeholders and employers;survey to participants to collect quantitative data.The data used are: data from interviews and focus groups; secondary data from studies, researches and statistical databases, such as the National Statistical Institute, Eurostat, Ministry of Labour, Ministry of Finance.
The main findings from the evaluation can be summarised as follows:The Youth unemployment rate in Bulgaria is higher than the EU average and has become a structural problem. Participants were selected according to non-discrimination and gender equality principles. The interventions implemented comply with the EU 2020 strategy.There were no particular difficulties in implementing the initiatives and the measures were timely put in place thanks to the pro-active approach of the stakeholders such as the social partners and the vocational training centres.The interventions were “classic” in nature and consisted of supporting internships, training, subsidised employment and professional qualification. They were tailored to the needs of the labour market.The results of the interventions were positive. The best results in terms of satisfaction of stakeholders and participants were registered in vocational training and practical training in schools. The measures could however been better designed to tackle early school-leaving.The interventions were found effective in supporting employment (45% of participants had a job after leaving the scheme) and tackling unemployment.The Ministry of Finance estimated (by using the SIBILA 2.0 impact assessment model) that by 2023 the YEI will have helped to increase the employment rate of the target group by 10.9% and to decrease the unemployment rate by 4.6%.
The main policy implications are:The interventions should be monitored on a regular basis to ensure that the scheme addresses the needs of the different target groups.The information system should be fully functional and capable of delivering updated data for analysing developments on the labour market and for ensuring that the interventions address the peculiarities of participants.The administrative burden for applicants should be reduced.The matching between demand and offer should be improved.Early school-leavers and people with low education should be better covered by the interventions.
Effects of the Human Resource Development OP 2007–2013 in Bulgaria (ESF)
„Изследване на ефекта от изпълнението на мерки по ключови области от Оперативна програма „Развитие на човешките ресурси“ 2007 - 2013“
Mariana Stefanova (Ecorys Southeast Europe)
Ecorys Southeast Europe, boulevard "Cherni vrah" 1, Sofia 1421, Bulgaria, Mariana.stefanova@ecorys.com
The evaluation assesses the effects of the 2007-2013 Human resource development OP in key priority areas and the degree of attainment of the targets set by the programme.
To verify the contribution of the programme in key areas and the degree of attainment of objectives, to review the programming process and to identify successful measures.
Desk analysis of monitoring data and other documents;interviews;on-line survey of beneficiaries.
The design of the OP was in line with the main EU and national strategies.Overall the intervention logic was satisfactory.The system of indicators was well designed, but there were differences in interpretation and measurement by MAs and beneficiaries.The targets set were often too low or too high because of insufficient information and analysis during the planning phase; in addition, the allocation of funds to priority areas was heavily unbalanced.Concentration of resources was insufficient to achieve the desired results.The financial management of the OP was satisfactory since the absorption rate was high.Some measures to create employment lacked sustainability, as jobs were lost rapidly after they came to an end.The young and unemployed over 55 were insufficiently targeted.The effects of measures aimed at improving the adaptability of those in employment were limited.The measures to improve access to education, especially those focused on vulnerable children, achieved good results.
There is need to:undertaken thorough research before programming to fine tune the quantitative targets;better balance resources within and between priority areas;better organise participation of stakeholders in setting selection criteria.
Evaluation of measures supporting people employed under PA1 of the Human Resources Development ESF/YEI OP, 2014-2020, in Bulgaria
Icon Оценка на ефективността, ефикасността и въздействието на мерките за подкрепа на заети лица по приоритетна ос 1 на Оперативна програма „Развитие на човешките ресурси“ 2014 – 2020
Asenka Hristova, Stoyan Baev, Lyubomir Dimitrov, Nedyalka Dimitrova-Vasileva, Nataliya Panayotova
Industry Watch Group Bulgaria, hristova@iwatchbulgaria.com
The evaluation covers 10 measures to improve work organisation and working conditions of employees. Data go up to December 2020.
To assess the effectiveness, efficiency and impact of 10 measures supporting employees up to end-2020.
Desk research.Analysis of monitoring data.Interviews with stakeholders.4 surveys: project beneficiaries (121 respondents – 14% response rate), participants (1,151 respondents – 79% response rate), firms supported (6 replies) and non-supported (5 replies).Counterfactual analysis (DiD).
The measures supported are coherent with OP priorities and national strategic goals and relevant to the needs of the target groups.Most measures seem efficient, as their average unit costs are lower than planned.But skill needs arising from technological changes and the transition to a green economy do not seem to be addressed.There is no significant reduction in occupational injuries in the firms supported compared to others.The activities supported, including the provision of sporting and recreational facilities, have had a limited effect on labour productivity.The counterfactual analysis shows that training programmes initiated by employers increased the monthly income of participants, while those selected by employees did not.Indicator targets have generally been met.The main challenges are reaching older workers, identifying the real needs of employers and employees, and a lack of motivation to participate because of difficulties in combining work and family life, the length of the training, logistical issues or poor career guidance services).
A very small number of replies were received from enterprises. There were difficulties in designing the control group for the counterfactual analysis.
Target the specific needs of individuals more accurately.Establish a reliable system for checking the demand for and supply of skills across sectors.Provide continuous and flexible support for updating the skills of employees (in particular technical and digital ones).Provide comprehensive advisory and guidance services on the needs and opportunities for skills development.
Evaluation of REACT-EU measures under the Human Resources Development OP, 2014-2020, in Bulgaria
Оценка на процедури, финансирани по REACT-EU по Оперативна програма „Развитие на човешките ресурси“ 2014-2020
​​Radostina Tsvetanova, Nedyalka Dimitrova-Vasileva, Hristova, Irena Vladimirova, Rosselina Stoeva, Kristina Popdoncheva, Kostadin Antonov, Stefan Kamenski​
The evaluation examines the implementation of three REACT-EU funded measures under the “Human Resources Development” OP in Bulgaria. The “Save me” and “Save me+” interventions aimed to maintain employment and provide supports for employers, workers and self-employed people in the context of the COVID-19 pandemic crisis. The “Solidarity” procedure aimed to integrate displaced persons from Ukraine by providing employment, housing and support for independent living. The cut-off date of the data and information used was October 2024.
To assess the implementation, effectiveness, impact, coherence and added value of the measures supporting employment and the integration of migrants from Ukraine.
Analysis of monitoring data (financial data, microdata on participants, technical reports).Analysis of statistical data from Eurostat, the National Statistical Institute, the National Social Security Institute (NSSI) and the Employment Agency.Analysis of administrative data from the Employment Agency and the NSSI.Analysis of other secondary data form the Organisations involved in the measures (documents, guidelines, analysis).Interviews with representatives from the Managing Authority, the Ministry of Labour and Social Policy, the Employment Agency, the National Social Security Institute, the Social Assistance Agency, the State Agency for Refugees, the UNHCR Representation in Bulgaria, Caritas Bulgaria, Council of Refugee Women in Bulgaria and Aid for Ukraine Foundation.Interviews with 28 employers who participated in the three measures as well as with five who applied but were not approved.Online surveys of employees of the “Save Me” measure (9% response rate: 101 full answers for 1,109 participants).Online survey of workers, employees and self-employed persons of “Save me+” measure (9% response rate: 21 full responses for 229 persons).Online survey of Ukrainian citizens who participated in “Solidarity” measure (6% response rate: 30 answers for 490 real email addresses).
“Save me” and “Save me+” helped preserve jobs, mitigate household income loss, reduce pandemic-related stress and motivate workers to re-join the labour market. 91% of “Save me” and 94% of “Save me+” participants were in employment at the end of the programme.The measures were considered a successful example of a rapid response to crisis, and the EU added value consisted mainly in covering segments of the labour market that were not supported before.The “Solidarity” project had a positive short-term impact on the acquisition of labour skills but failed to ensure sustainable integration into the labour market in the long term: of those who remained in Bulgaria and for whom information was available in the NSSI, only 18.9% were employed 6 months later, but this increased to 27% after 12 months.Lack of provision of language training and hourly childcare services within the “Solidarity” project was problematic for the integration of Ukrainian migrants.Other problems included declining interest from employers and the target group, difficult recognition of qualifications, mainly low-skilled jobs, complicated access to banking services and administrative burden for employers in applying and reporting.The implementation of the procedure was further hampered by insufficient coordination between different institutions.
Reluctance to participate in consultations by representatives of employers' and trade unions. Low participation rate of surveys of target groups, because of invalid email addresses and the unwillingness of participants to fill in the questionnaire. Finally, poor quality of the information collected from some respondents, which limited the scope for analysing certain aspects of implementation.
Expand the linguistic support that target groups (foreign nationals) can receive, especially in the form of language courses.Define monitoring systems for programmes on employment, which can regularly report the satisfaction of participants and employers and analyse the correspondence between qualifications and employment.Use alternative communication channels to promote the measures.
Evaluation of the progress of the Transport and Infrastructure OP, 2014-2020, in Bulgaria
Оценка на напредъка по Оперативна програма „Транспорт и транспортна инфраструктура“ 2014-2020 Г. и принос към Стратегията на ЕС
OP Transport and transport infrastructure 2014-2020
Elana (investment@elana.net; info@elana.net); Intel Advisers (office@inteladvisers.com)
The evaluation examines the progress of the OP, as at end-2019.
To assess the financial advancement of the OP, the obstacles and factors affecting absorption of funds, the results so far achieved and the contribution to the Europe 2020 strategy.
Monitoring data.Desk analysis of documents.In-depth interviews and focus groups.Survey of stakeholders.
At the end of 2019, the certified expenditures amounted to just under 30% of the budget; 56 projects were supported, 7 of them major projects.The implementation of the OP is slow and there is the risk of not complying with the N+3 rule.The least progress is on PAs 1 and 2, on rail and road infrastructure, and PA 4 on transport management systems.Delays on PA 1 are due to funding being concentrated on two main projects, one of which involves a high degree of risk of default; for PA 2, they are due to difficulties on the Krupnik–Kresna stretch of the Struma Motorway.The main general obstacles to implementation are: the long duration of selection procedures, the prolonged tender appeals, the complex conciliation procedures and the low degree of project readiness.Targets were reached for only two of the 12 output indicators and PAs 1 and 4 did not achieve the 2018 milestones.
Identify additional and alternative projects under PAs 1, 2 and 4.Strengthen monitoring and control activities.Carry out a detailed analysis of the status of all projects.Consider introducing legislative changes to the implementation of major infrastructure projects.
First Assessment Report, Evaluation of the implementation of OPAC for the period January - December 2014
Първи доклад, Oценка на изпълнението на ОПАК за периода януари - декември 2014 г.
Ecorys South East Europe, Sofia, bul. Cherni Vrah 1, tel: (+359 2) 8155 684, email: mariana.stefaNova@ecorys.bg; irena.vladimirova@ecorys.bg
Impact evaluation of the Transport OP 2007-2013, in Bulgaria
Оценка на въздействието и ефектите от изпълнението на Оперативна програма „Транспорт“ 2007 - 2013 г.
Ms Raliza Ganeva; Mr Daniel Bogdanov; Ms Julia Spiridonova; Ms Teodora Pavlova; Ms Kristina Tsvetanska; Ms Velina Savcheva; Ms Silvia Todorova; Ms Desislava Asparuhova; Ms Rosalia Kirilova
The evaluation examines the effects of the implementation of the OP.
To assess the effects of projects implemented under the transport OP, 2007-2013, their effectiveness, efficiency, relevance and coherence.
Monitoring data.Official statistics.Administrative data (regional business statistics).Interviews with stakeholders.Survey of stakeholders.12 case studies.
Almost all the financial and physical indicator targets were achieved and the OP contributed to sustainable development objectives.The OP is estimated to have had a positive net impact on employment, wages, GDP, exports and investment.Results were achieved at lower average costs than the EU averages.The main benefits were savings in travel time, a reduced number of accidents, and positive environmental effects.Effects from rail investment were relatively small because traffic levels were lower than expected.The main problems in implementing the OP came from lack of experience in managing infrastructure projects and public procurement procedures, lack of beneficiary co-financing and unforeseen work.
Ensure systematic risk assessment of projects and the administrative capacity of MA staff and beneficiaries.Speed up rail projects linked to the TEN-T network to increase European connectivity and interoperability.Improve complementarity with other EU-funded programmes.Ensure availability of data on beneficiaries to carry out future impact evaluations.
Second evaluation of the implementation and impact of the Environment OP, 2014-2020, in Bulgaria
Bтори доклад по наблюдението и контрола на въздействието върху околната среда в резултат от прилагането на оп „околна среда 2014-2020 г." (16.06.2018 - 30.06.2021 г.)
Directorate General “Environment Operational Programme” - Ministry of Environment and Water
http://ope.moew.government.bg/ - ope@moew.government.bg
The evaluation examines progress in implementing the OP and its impact up to 30 June 2021.
To assess implementation and the impact of measures for environmental protection, adaptation to climate change and sustainable use of natural resources up to end-June 2021.
The implementation of the programme is proceeding at the expected pace and funding committed is close to 100%, except for PA3 Natura 2000 and biodiversity.All PAs are making a substantial contribution to improving the environment and the use of natural resources.The OP supports green investment projects that improve people’s quality of life, create new jobs, and promote an ecological culture.No adverse effects on the environment and human health have been identified as a result of the implementation of the OP.
Some indicators could not be quantified since data are missing.
Provide guidelines for applicants that include requirements and criteria for project proposals for environmental protection and human health.
Environmental compliance of the implementation of the Environment OP, 2014-2020
Доклад по наблюдението и контрола на въздействието върху околната среда в резултат от прилагането на оперативна програма „Околна среда 2014-2020 г.”
The evaluation assesses the extent of environmental compliance in the implementation of the Environment OP 2014-2020 in Bulgaria (between June 2015 and June 2018).
To assess how far the OP is compliant with the Environmental Assessment Opinion No 15-9/2014 produced by the Environment and Water Ministry.
Desk review of documentation and relevant information.
Most of the projects under the OP, which include those involving waste and water management, Natura 2000 and biodiversity, prevention and management of risks of floods and landslides and improvements in air quality, are in an early stage of implementation and most had not started in mid-June 2018.The criteria for project selection are generally in line with the Environmental Assessment Opinion No 15-9/2014.
During the implementation of the OP, no adverse effects on the environment were envisaged, so there is no need to take any remedial measures.
Evaluation of the effectiveness and efficiency of PA5 ‘Improvement of the ambient air quality’ of the Environment OP, 2014-2020, in Bulgaria
Резюме на доклад за оценка на ефективността и ефикасността на изпълнението на ПО 5 "Подобряване на качеството на атмосферния въздух" на ОПОС 2014- 2020 г.
ECORYS, www.ecorys.com/bg/bulgaria; BIM Consulting, www.bim-bg.com
The evaluation examines PA5 ‘Improvement of the ambient air quality’ of the Environment OP.
To assess efficiency in implementing PA5, the effectiveness in achieving results, relevance of support and the application of financial instruments.
Monitoring data.Documentation.Interviews with stakeholders.Survey of beneficiaries and potential beneficiaries (response rate: 75%).Focus group with experts, municipalities and the Association of Ecologists.
The OP strategy addresses needs as regards air quality, though not all measures are expected to directly affect pursuit of the objectives.The milestones at end-2018 have been achieved. As at January 2020, no progress was reported for performance and result indicators.The financial absorption rate is only 32%.The funding agreed significantly exceeds the PA5 budget, which requires an amendment of the OP.Results are not yet observable, in part because the two measures expected to have immediate effects on air quality are still at the initial phase of implementation.Factors impeding implementation include the lack of previous experience in the MA, the delayed launch and difficulties in implementing projects, and support for measures that do not directly affect air quality.80% of survey respondents had difficulties in preparing projects.86% of project proposals submitted are approved, usually in a shorter time than foreseen.The OP does not envisage using financial instruments for PA5.
Launch activities to prioritise measures at the municipal level and fund them under the next OP.Seek synergies with other OPs and funding opportunities and examine the applicability of financial instruments.Plan public information campaigns.Finance measures showing a clear effect on improving air quality at the application stage.Consider implementing competitive selection procedures to ensure that effective measures are financed.Consider the possibility of introducing a co-financing requirement for beneficiaries under certain procedures.Consider extending potential beneficiaries to non-public institutions.
Evaluation of projects implemented under the Science and Education for Smart Growth OP, 2014-2020, in Bulgaria
Оценка на изпълнението на действащите проекти на МОН като конкретен бенефициент - 2017 г.
OP Science and Education for Smart Growth 2014-2020
The evaluation assesses the implementation of measures under TO10 supported by the Science and Education for Smart Growth OP, 2014-2020 in Bulgaria.
To assess the effectiveness and efficiency of measures on education.
Desk analysis.Analysis of monitoring data.Interviews with beneficiaries of projects.
Main issues identified in the implementation of projects supported are:Some projects lack synchronisation of activities which led to the duplication of activities and costs.Lack of clear rules to implement some projects.There is a strong concern about the sustainability of the project ‘System for Career Guidance in School Education’. If the project is not continued, this will put at risk the relevant performance indicators and results set out in the OP (which may lead to financial corrections due to non-compliance with EU rules).In several cases, project objectives were not linked to specific indicators and this create problems in the subsequent monitoring of the implementation.The mechanism to calculate the implementation indicator of ‘Students Participating in OP activities in support of vocational education in areas of priority for the economy’ is not clear and this causes inaccurate reporting.There are delays in reporting the pace of implementation.In many projects, activities were not linked to specific objectives and this brought to an inefficient use of resources; in other cases, the project costs were not adequately explained.
Avoid overlapping of activities between projects.Develop clear criteria on how to plan activities to be supported.Link projects objectives to relevant indicators.Define clear rules on the monitoring of indicators.Clearly define the costs of projects.
Evaluation of YEI measures under priority axis 1 of the Human resources development OP 2014-2020 in Bulgaria (ESF+YEI) – final report
This report examines the impact of four YEI measures: ‘Active’, ‘Ready for work’, ‘Youth employment’ and ‘Training and employment for young people’ in Bulgaria in 2014-2020. The first two focus on inactive young people, the latter two on unemployed. Three are largely traditional, while ‘Active’ involves a new approach to identifying, activating and motivating young people.
To assess the effectiveness, efficiency and impact of the measures co-funded by YEI as well as their consistency with the YG, the Europe 2020 Strategy objectives and the national employment strategy.
Desk research and analysis of documents.Analysis of monitoring data and data from the unemployment register and the National Statistical Institute.In-depth interviews and focus groups with participants, employers and other stakeholders.Surveys of participants and employers.Statistical analysis.SIBILA 2.0 macroeconomic model developed by the Ministry of Finance.Counterfactual impact analysis (using propensity score matching and difference in difference).
The measures are coherent with the objectives of Europe 2020, the National Reform Programme, the national Employment strategy and the national plan for YG, are suitable for addressing the needs of young NEETs, and are compliant with the horizontal principles of equality and non-discrimination.By end-2018, over 52,000 people aged 15-29 participated in the four schemes.The target values are expected to be met or exceeded by the end of the period.The main difficulties encountered are a large administrative burden for employers, difficulty of attracting participants, working with the most vulnerable and low wages.After participation, 32% of young people were in employment or training. Employment rates are particularly high for the ‘Youth employment’ (75%) and ‘Training and employment’ (63%) measures.The net effect of YEI on the share of young people unemployed is negligible (0.3% - except for young people with higher education 2.2%). And YEI participants were 8% more likely to be working at end-2018 than members of the control group. The chances of being in employment 6 months after exit increase by 37% (and by 49% after 12 months).The deadweight effect is estimated at 2%, the substitution effect at 13% and the displacement effect at 12%.The average annual cost of participation (EUR 1,260) is below the average wage (EUR 7,540 in 2017). The majority of participants were paid the statutory minimum wage by employers (less than EUR 240). Most of the employers surveyed reported the measure costs them the same as they benefited, though the number of employers for whom the measure costs more than the benefit it brought was larger than the number for whom it costs less. ‘Ready to work’ is estimated to be the most cost efficient measure (EUR 150 per person).Without YEI, nearly 4,000 young people would be out of the labour market, 9,000 would not have received further vocational training, 19,000 would not have increased their qualification through traineeships, and some 26,500 would not have been in employment.Nearly three-quarters of participants received a job offer after the end of the programme. The quality of the offers was high in terms of job characteristics and compliance with the situation of young people but the wages were too low.63% of job offers involved permanent contracts, 95% of them full-time, and the large majority a salary equivalent to the minimum wage.In 2020, the number employed in the economy is estimated to grow by 1.8% as a result of YEI and the unemployment rate to decline by 0.9 of a percentage point. These positive net effects are estimated to continue up to 2023.
Follow-up data on participants after their exit are not available for all participants.
There is a need to:further study pay levels in active labour market measures as well as job quality and sustainability;continue developing an efficient monitoring system;continue implementing active labour market policies targeted at young people (in particular for young mothers and young disabled);improve the quality of education and cooperation between schools and enterprises;address the problem of the quality of skills by providing training at the workplace.
Evaluation of the effectiveness, efficiency and impact of measures under the Good Governance OP, 2014-2020, in Bulgaria - 2023 report
Оценка на ефективността, ефикасността и въздействието от инвестициите, извършени по Оперативна програма „Добро управление“ - 2023 г.
The evaluation aims to assess impact, effectiveness and efficiency of interventions under the OP. Monitoring data used go up to the beginning of 2023.
To assess impact, effectiveness, and efficiency of measures promoting the development of e-management, administrative reforms and training in public administrations, reforms and e-services in the area of justice, civic participation, and the development of the management and control system of ESI Funds.
Analysis of official statistics, monitoring and administrative data.Findings from previous evaluations.Standardised survey.Focus groups and interviews. Cost-effectiveness analysis.Analysis of the intervention logic.
Measures on administration accountability and judicial reform:All result indicators have reached at least 85% of the target, and some of the targets have been exceeded.Standardisation of costs emerged as a critical driver of efficiency.ESF support enabled the promotion of a legal aid system and alternative dispute resolution methods.The OP fostered partnerships between civil society and public institutions.Measures on the management of ESIF funds:Result indicators exceeded the target values.Implementation of e-services has enhanced transparency and accessibility, and reduced users’ administrative burden.EU investments were crucial, as the ESF was the only source of funding for these measures.
Uncompleted projects are not covered by the evaluation, though these are a minority.
Support civil society organisations in monitoring government policies and services, and promote associations between NGOs.Modernise public administration and enhance civil society organizations’ capacity.Introduce mechanisms of reciprocal feedback between NGOs and administrations.Increase exchange of experience among institutions managing EU funds and improve coordination mechanisms.Increase administrative capacity of applicants and absorption of EU funds in the 2021-2027 period.Support upskilling and retention of already trained staff.
Analysis of the concept of demand support for SME and employees in the Śląskie region
Analiza koncepcji systemu popytowego wsparcia MSP i pracowników w województwie śląskim w ramach RPO WSL 2014-2020
Ul. Spławie 53, 61-312 Poznań, Tel: 61 622 92 06, email: biuro@re-source.pl
Evaluation of the "Competitiveness of the Bulgarian Economy” ERDF OP, 2007-2013
Последваща оценка на изпълнението, резултатите и въздействието на Оперативна Програма „Развитие на конкурентоспособността на българската икономика 2007-2013 г.“ - Окончателен доклад
OP Development of the Competitiveness of the Bulgarian Economy
BIM Consulting OODSofia 1000, 23 William Gladstone Str., Floor 1, ap. 3tel: (+359 2) 987 24 82e-mail: office@bim-bg.com
The evaluation assesses the appropriateness of the objectives of the OP "Development of the Competitiveness of the Bulgarian Economy" 2007-2013, the effectiveness and efficiency of the programme and its impact on the national economy. It also gives recommendations for the 2014-2020 programming period.
To analyse the coherence between needs and the programme strategy.To evaluate the effectiveness and efficiency of the OP.To assess the impact of the OP on socio-economic development.To evaluate the sustainability of interventions and their contribution to horizontal policy objectives.To identify best practice in the implementation of the OP.
Analysis of national and EU documents, OP and JEREMIE annual implementation reports, databases (national statistics, Eurostat, commercial register, monitoring data). Focus groups, in-depth interviews, surveys.
The objectives and activities of the programme properly reflect the needs of the Bulgarian economy.Benefits of the creation of partnerships between the business and scientific communities are not sufficiently emphasised and there is an overall low administrative and financial capacity among target groups.There was high over-achievement of targets due to their unrealistic nature.The system for measuring, monitoring and reporting the effectiveness of financial instruments under the JEREMIE initiative and their impact on the economy at micro and macro level is not adequate.The OP has contributed to increasing employment and improving the quality of the labour force.
Responsiveness of the beneficiaries of the program in online surveys, focus groups and deep interviews is not high. National statistics data on indicators linked to the impact of the OP are not always available.
Innovation indicators for evaluating operations should be defined and measured.Partnerships between research centres and enterprises should be simulated.It is necessary to establish procedures for the collection and validation of data.Contracts for the provision of financing through financial instruments (JEREMIE) should include the reporting of relevant monitoring indicators.
Ex-ante study on the use of financial instruments under the Human Resources Development ESF+ Programme, 2021-2027, in Bulgaria
Human Resources Development BG ESF+ 2021-2027
The report is an ex-ante study of the use of financial instruments (FIs) under the Human Resources Development Programme (HRDP) 2021-2027, to support the MA’s decision on the use of loans and equity instruments alongside grants. Data used go up to November 2024.
To identify key market failures, to draw lessons from the previous programming period and to propose a strategy for the use of financial instruments for 2021-2027.
Document analysis.Analysis of monitoring data from the previous programming period.Analysis of statistical data from the National Statistical Institute and the Ministry of Finance.Analysis of secondary data from the National Bank of Bulgaria and the Association of Banks.Analysis of previous studies on the financial and labour market.
Despite recent crises, demand for FIs-based support remains strong, especially from social enterprises, start-ups and vulnerable groups.The experience of the previous 2014-2020 OP shows that FIs products are increasingly recognised and attractive, particularly for people and enterprises that are not bankable.There is a persistent finance gap for SMEs, especially micro and start-ups, which rely on own/family money and struggle to get bank loans because of weak collateral, no-credit history and limited financial skills.Microcredit is undersupplied and demand for equity financing is estimated to be nearly three times the available and planned instruments.Vulnerable groups and young people face double constraints: they lack both finance and mentoring/skills, and existing EU schemes (EaSI, InvestEU, etc.) are too narrow.
The effect of the financial instruments used in the previous programming period is only measured in the short-term, since long-term effects need more time to materialise.
Continue and scale up the use of financial instruments in 2021–2027, with a mix of debt and equity based on market demand and programme progress.Design all debt and equity instruments so they can include a grant top-up (up to 40% for loans and 25% for equity), to boost take-up, sustainability and simplify management.Guarantee demarcation and complementarity with other measures.Attract private investors by using risk-sharing microloans and early-stage equity funds with clear leverage targets.Include non-financial support (mentoring, training, networks, legal/accounting support, product validation) as an eligible grant component.Exploit existing institutional experience and keep learning from previous lessons.
Evaluation of the achievement of the objectives of the Transport and Transport Infrastructure OP, 2014-2020, in Bulgaria
Оценка на приноса на подкрепата от европейските структурни и инвестиционни фондове за постигането на целите по всеки приоритет на ОПТТИ 2014-2020
Ecorys SEE, +359 2 8155 684, ecorysSEE@ecorys.com
This evaluation examines the impact of the projects financed under the OP on the completion and modernisation of the major national and pan-European transport infrastructure. Data go up to 31 December 2021.
To monitor the overall achievement of the OP as regards the completion and modernisation of the major national and pan-European transport network up to end-2021.
Documentation review.Analysis of official statistics and monitoring data.Interviews with stakeholders and beneficiaries.Face-to-face interviews with 300 passengers.Theory of change (ToC) testing.Analysis of data from the macrosimulation model (SIBILA 2.0).CIE.6 case studies (three on the OP examined and three on the previous transport OP).
The OP shows good internal coherence and a good integration with other public investment.Achievement of targets for railway projects is at risk, because of lengthy procedures for land appropriation and appeals to the results of procurement procedures.Road projects are either completed or at an advanced stage and connectivity to the TEN-T has been significantly increased.Metro extension projects have achieved or exceeded targets, though the fall in demand after Covid-19 pandemic reduced the number of users.The OP has made a positive contribution to improving accessibility, reducing travel time and mitigating the adverse effects of transport on the environment.The OP has resulted in an estimated increase in GDP of 0.7% compared to a scenario without the investment concerned and an increase in employment and wages.The most significant macroeconomic impact is estimated to come from road and metro projects.
Quantitative data are not always complete. Covid-19 affected the implementation of the OP and the quantitative data collected after 2020 might be biased, especially as regards the effectiveness and impact of the OP.
Streamline bureaucracy and increase institutional capacity building.Systematically collect the data needed to assess the effects and benefits of the projects implemented.Set up an inter-institutional team to better identify strategic projects of national interest.
Evaluation of the deinstitutionalization measures for children and young people supported under PA2 of the Human Resources Development OP, 2014-2020, in Bulgaria
Оценка на ефективността, ефикасността и въздействието на мерките за деинституционализация, финансирани в рамките на приоритетна ос 2 на ОП РЧР 2014-2020
Asenka Hristova, Pavlina Mladenova, Irena Vladimirova, Nedyalka Dimitrova-Vasileva
Ecorys, ecoryssee@ecorys.com, +359 2 981 6032
The evaluation examines 5 measures supported under PA2 ‘Reducing poverty and promoting social inclusion’ aimed at deinstitutionalisation. Data used are up to June 2022.
To assess the effectiveness, efficiency and impact of 5 measures supported under PA2 (Reducing poverty and social exclusion).
• Desk review.• Analysis of monitoring and statistical data.• Evidence from previous evaluations.• Interviews with the MA and other public bodies involved in implementation and social workers, psychologists, speech-language pathologists, and medical experts.• Five focus groups with beneficiaries.• Online survey of beneficiaries (response rate of 34% out of 138 participants).
• A large part of activities carried out were related to need assessment or individual support plans, involving parents, where possible.• Support has improved access to social, integrated health and social services and early childhood development services. Positive results have also been achieved in terms of improving parental capacity, parental behaviour, the social skills of children and young people, their physical development and health, and attitudes to participation in education.• Municipalities demonstrated a good capacity to manage the various social services in the community and to interact with nurseries, health and educational institutions, local authorities and NGOs.• The main problems for municipalities have been the availability of specialists to work at the local level, staff turnover, pay levels, status and working conditions and management skills for administrating social services.• For all measures, the cost per participant was lower than planned and contracted, but most have a lower cost effectiveness than expected.• COVID-related restrictions adversely affected the implementation of activities.
Measures analysed are different in nature, types of beneficiary and eligible activities. For the ‘Early childhood development’ measure which targets both children and their families, no data are collected on the latter. As of end-June 2022, only one measure was completed and this made difficult to assess impact.
• Target specialists and staff working in residential care to develop their expertise.• Develop a formal system for tracking the progress of children after leaving early childhood development care.• Assess characteristics, motivational factors, priorities and needs of young people leaving care more specifically in order to develop focused measures.• Develop support for municipalities to carry out activities for increasing the visibility of children with disabilities in the local community in a systematic way and establish long-term partnership relationships with cultural, education and sports institutions at local level.• Raise awareness among potential beneficiaries of having good quality data.• Add a procedure-specific indicator on activities focused on the families of the children and young people included in projects.
Strategic evaluation of the Bulgarian Centres of Competence and Centres of Excellence financed under the Science and Education for Smart Growth OP, 2014-2020, in Bulgaria
Стратегическа оценка на българските центрове за компетентност и центрове за върхови постижения и препоръки за тяхното развитие
Andonova E., Barrada M., Bole D., Gulda K., Hudson J., Jimenez V., Karanikic P., Kaymaktchiyski S., Kokorotsikos P., Koycheva C., Mladenov T., Rizzuto C., Shvarova O., Taylor S., Teernstra F.
JRC’s Centre of Competence on Technology Transfer (CCTT), Elena Andonova, Policy Officer, I.4, JRCEC-CC-TT@ec.europa.eu
This is a strategic assessment of the Bulgarian Centres of Competence (CoCs) and Centres of Excellence (CoEs) and their contribution to the development of RDI in Bulgaria.
To assess the functioning of 14 CoCs and CoEs financed by the OP and how they can support the development of applied sciences in the country.
Desk analysis of relevant literature and policy documents.Case studies based on surveys and interviews of CoCs and CoEs.
Bulgaria’s RDI system has a number of structural shortcomings, among them low levels of public and private investment, weak science-business links and inefficient governance.CoCs and CoEs are strongly focused on research but less on Technology Transfer (TT) and Knowledge Transfer (KT), making them less competitive in the market, and collaboration between universities and business is still little developed.The organisation of the Bulgarian Academy of Sciences (BAS) needs to be revised, so that it acts as a central hub for involving CoCs and CoEs in projects.The use of State aid is sometimes avoided or wrongly applied because of lack of understanding of EU rules and of the difference between economic and non-economic activities.
Devise a clear development strategy to make CoCs and CoEs competitive in the EU market.Put in place a coordinated and centralised government policy for capacity building and TT.Make CoCs and CoEs independent legal entities to support their smooth and sustainable development.Mainstream knowledge among research bodies on how to correctly apply State aid rules to RDI.
Evaluation of the implementation of the Good Governance ESF OP in 2015-2017 - Second Report
„Оценка на изпълнението на ОПДУ в периода 2015-2017 г. ипроучвания, подпомагащи процеса по планиране събирането наданни за всички оценки след 2017 г., включени в Плана за оценка наОПДУ”. Втори доклад за оценка (финален): Анализ на данните, които ще са
The report considers the data that will be needed for the evaluation post-2017 included in the Evaluation Plan and to calculate the common long-term result indicators and assesses whether the information system is able to provide these data.
To identify data needs for calculating the long-term indicators and to carry out the evaluations and the measures to protect personal data and to assess the advantages and disadvantages of different data collection methods.
Desk research and data analysis.Meetings and interviews.Case studies.
The data from the monitoring system are in most cases not adequate for the calculation of the long-term indicators, or for providing information to conduct surveys of participants.At the moment only the National Institute of Justice is able to provide data for individual participants in electronic form for evaluations use.There will, therefore, be serious difficulties in carrying out the planned counterfactual evaluation.In the Implementation Manual for the Good governance OP, data collection procedures for common performance and result indicators are not described.The “Microdata_ESF" Excel file is used in all programmes financed by the ESF, but much of the information it contains is irrelevant, creating an unnecessary administrative burden for both beneficiaries and the MA.
There is a need to:improve the collection of data for the evaluation;apply data protection rules and give evaluators anonymised data;drop the counterfactual analysis for Evaluation no.7 in the evaluation plan;improve the guidelines for e-reporting;produce detailed documentation describing the data collection process;collect data on an annual basis, ensuring the collection of information that can be used for both management purposes and for monitoring.
Interim evaluation of measures supporting vocational education under the Education Programme, 2021-2027, in Bulgaria
Проектът „Модернизиране на професионалното образование и обучение“ в България: Изводи от ранния етап на изпълнение (разработено от ОИСР)
Anett John, Zhasmin Avetisyan, Christa Rawkins
The report focuses on four key project activities aimed at enhancing the responsiveness of vocational pathways to evolving labour market needs by adapting VET to labour market dynamics. The evaluation was carried out between March and December 2025, monitoring data go up to October 2025.
To assess the implementation of a project supporting both system-level reforms, such as the revision of the qualifications and curricular framework for VET (Activities 1 and 2), and school-level innovations, regarding practitioner teachers and expansion of student placement (Activities 5 and 8), with the aim of strengthening the labour market relevance of vocational pathways.
Analysis of project documentation. Analysis of monitoring and statistical office data. Reconstruction of the logic of the interventions. Online survey of all 218 Sectoral Skills Council (SSC) members (51% response rate). Semi-structured interviews with the Ministry of Education and Science (MES), the National Agency for VET (NAVET) and the representatives from business associations and trade unions. Focus groups with SSC members. Purposive sampling used in the interviews and focus groups carried out, to select participants directly involved in project activities. Site visits to three VET schools.
The creation of SSCs responds directly to a major challenge in Bulgaria’s VET system by bringing together VET professionals and industry practitioners on a permanent basis at sectoral level. SSC participation relies largely on voluntary engagement, which results in limited involvement, especially from industry practitioners. Contractual constraints and administrative barriers also led to the underrepresentation of industry practitioners in some SSCs, with academics and other profiles often being overrepresented. Although almost two-thirds (63%) of SSCs members are willing to remain engaged for the next revision cycle, the long-term contracting model remains unclear. The revised List of Professions includes broader vocational categories than its predecessor. The simultaneous update of 150 State Educational Standards and national examination programmes is comprehensive, but some expert teams did not sufficiently integrate VET professionals and industry practitioners. Public procurement for Standards drafting teams was lengthy (up to six months) and deterred VET practitioners and industry experts. Institutional capacity was stretched (4 NAVET coordinators for 150 standards; 8 MES coordinators for 218 SSC members). Student placement strengthens school–employer links and work-based learning, but their funding model raises sustainability concerns. The project is delivering system-level reforms at a scale and pace unmatched by previous national efforts.
Purposive sampling and voluntary participation mean the data are not nationally representative, raising the risk of selection bias. The analysis relies mainly on qualitative methods, as outcome data are not yet available. In addition, several activities were still ongoing, meaning findings reflect mostly early lessons.
Institutionalise SSCs through a clear legal framework. Review the design of future State Educational Standards revisions: broaden industry practitioner participation, extend timelines, and involve SSCs earlier. Establish a staged review of teacher competence profiles and develop a comprehensive implementation plan covering initial and continuous teacher development. Provide clearer guidance to schools on good practice for collaboration with external practitioners. Explore sustainable options for the 240-hour student placement, including new financing models and a broader pool of host organisations. Strengthen the monitoring system to capture intermediate outcomes and longer-term effects, not just activity-level progress.
Evaluation of the project selection system under the European Funds for Dolny Śląsk, 2021-2027
Ewaluacja bieżąca kryteriów i systemu wyboru projektów FE dla Dolnego Śląska 2021-2027
Agnieszka Śnieżek, Beata Dec, Marta Cichowicz-Major, Karolina Drozdowicz, Adam Grajek, Aleksandra Kamińska, Michał Marciniak, Sebastian Pałka, Wojciech Pieniążek, Marcin Pierzchała, Paweł Pijanowski, Wojciech Onyśków
The evaluation examines the project selection system and project-selection criteria used in the programme, covering both competitive and non-competitive calls across all programme priorities and funds.
To assess whether the project selection system and selection criteria of the programme comply with Article 73 of the Common Provisions Regulation and the national project-selection guidelines, to support the effective and efficient programme implementation.
Desk research.Analysis of statistical and monitoring data.Survey of 358 beneficiaries and 439 applicants.3 interviews with MA representatives, 4 with IB representatives, 2 with members of the Monitoring Committee, 5 with Project Evaluation Committee staff, 6 with external experts and 7 with ITI representatives.46 interviews with beneficiaries and unsuccessful applicants.Expert panel.Linguistic and semiotic analysis.Benchmarking of 15 regional programmes.
The call schedule is formally well regulated and broadly aligned with programme logic, but its operational usefulness is limited.The project selection and appraisal system is assessed as compliant with higher-level requirements on transparency, impartiality, equal treatment, accessibility and horizontal principles.Applicants and beneficiaries perceive the selection process as transparent, predictable, and well organised. The main complaints concern the long appraisal times and heavy formal burdens in competitive calls.Institutional capacity is assessed positively in operational terms, but the system is constrained by the excessive bureaucratic documentation, limited staffing, and limited specialist expertise.The ability to apply successfully depends heavily on applicants’ internal capacity.The criteria are generally well aligned with programme objectives and help steer support towards strategic, territorial, environmental and quality-related goals.Some criteria are too declarative, some documentation requirements are burdensome, and financial-analysis requirements are excessive for selected types of projects.
Difficulties in recruiting interviewees, as some planned interviews with unsuccessful applicants could not be completed and were replaced with beneficiary interviews.
Simplify the national template for the call schedule.Add thematic search and filtering functions to the calls webpage so that users can identify relevant calls by objective or project area rather than by number or title.Reduce financial-analysis requirements for project types where this creates a disproportionate burden, especially for non-revenue-generating projects and lower-value investments.Prepare detailed guidance on financial-analysis requirements and simplifications for the 2027+ period.Allow applicants to correct missing or unsigned declaration-based documents for selected “zero-one” criteria in competitive calls, where this would prevent unnecessary rejection of eligible applications.
Impact evaluation of measures on higher technological education (ITS Academies) supported under the Piemonte ESF OP, 2014-2020
Gli effetti occupazionali della formazione tecnologica superiore Gli ITS Academy del Piemonte
The report is an impact evaluation of ITS Academies (Post-secondary Technical Institute system) in Piemonte, financed under the OP. It focuses on graduates from biennial courses started in 2019 and 2020 and completed in 2021 and 2022. Labour-market outcomes are observed up to December 2024.
To assess whether obtaining an ITS Academy diploma improves employment chances compared with two alternative pathways: entering job search after upper secondary school or completing a related three-year university degree. It also examines whether any advantage extends to stable employment and whether effects vary by field, gender, territory and prior work status.
Document analysis of the programme.Literature review and analysis of statistical data.Counterfactual impact evaluation using Coarsened Exact Matching (CEM). The study tracks a treated group of 883 ITS graduates, compared with a control group from two pools: active jobseekers and bachelor's degree graduates in related fields.Analysis of administrative and monitoring data on ITS participants linked to regional labour-market records (COB) and jobseekers’ declarations of immediate availability for work (DID).
As compared to upper-secondary graduates who immediately searched for work, ITS graduates experience an expected lock-in effect during the study but record a +5.9 percentage-point employment effect two years after completion.As compared to graduates from related bachelor's degrees, ITS graduates have substantially higher employment rates both one year (+23.2 pp) and two years (+19.1 pp) after graduation.ITS does not increase permanent-employment rates relative to immediate jobseekers after two years, but it does so relative to bachelor's graduates (+5.0 pp after one year and +7.7 pp after two years).Effects are heterogeneous: compared with jobseekers they are strongest in high-tech pathways and in metropolitan Torino; compared with bachelor's graduates they are larger for traditional fields, for women (in the medium term), and for provinces other than Torino.Effects also tend to be larger for previously inactive participants than for those already unemployed.
Only part of the treated group could be matched (363 of 883 in the jobseeker comparison and 242 of 883 in the graduate comparison); subgroup estimates are based on small numbers and should be interpreted cautiously; and follow-up is limited to roughly two years after obtaining the qualification.
Preserve the strong work-oriented character of ITS pathways and treat ITS as complementary to university pathways and employment services rather than as a substitute.Recognise the different role of ITS in metropolitan and non-metropolitan labour markets and support people with little or no prior work experience.
Final evaluation of the Sêr-Cymru II programme supported under the West Wales and the Valleys and East Wales ERDF OPs, 2014-2020
This is the final evaluation of the ERDF measures of the Sêr Cymru II programme, which aimed to increase research capacity in STEM subjects in Higher Education Institutions (HEIs), by attracting world class researchers and providing targeted support. The evaluation covers the implementation period from November 2016 to June 2023.
To review the strategic fit of the programme, assess the achievement of the objectives of the business plan, the effectiveness of management, delivery and monitoring processes. To assess the success in meeting the objective set for Cross Cutting Themes of the Welsh European Funding Office (WEFO), the impact of the programme on research quality and capacity in Wales and provide recommendations for future programmes.
Document reviews (recent policies published by the Welsh and UK Governments, recent trends in research income).Analysis of monitoring data and progress reports.2 surveys of grant recipients (response rate > 40%).Semi-structured interviews with 30 stakeholders (4 Welsh Government officers, 4 WEFO officers, 6 evaluation panel members and 16 HEI stakeholders).Semi-structures interviews with 32 grant recipients.Counterfactual analysis of the impact on research income, research publications and citations.Theory-based approach for assessing impact on the personal and career development of researchers.
The programme funded a wide range of projects, many of which were not clearly aligned with existing research groups or building on recognised strengths of the research sector.Research income in subject areas that received support has grown at a faster rate than similar institutions and the UK average for the same areas.The programme had a significant, positive additional impact on the number of research publications by supported fellows (+1.2 publications per researcher per annum, compared to a control group).There is some evidence that the programme increased the scope of subject areas covered by research fellows (indicating increased interdisciplinary working).There is mixed evidence about the impact of the programme on the quality of research.The programme had only a modest impact on research capacity. Although 180 new researchers were recruited, only 74 are still employed by their host institution as of December 2023 (a retention rate of 41%).
A robust counterfactual analysis could not be undertaken on HEIs because of a lack of available datasets. Instead, a benchmarking analysis was done which compares the income performance of Wales’s HEIs with similar universities. Monitoring data regarding the number of research fellows taking up permanent roles in their host institution are inaccurate. There is uncertainty about the total number of partners cooperating in research projects.
Continue the Sêr Cymru programme and provide clear, long-term vision and strategy, to avoid uncertainty for HEIs.Focus investments on internationally excellent research which builds on Wales’s strengths and addresses national priorities.Ensure that industry is engaged in determining priorities.Provide more flexibility of funding and allow universities to leverage other funding.Scrutinise rigorously outcomes and progress of the programme and streamline reporting and claims processes.Work with universities and heads of department to avoid the loss of high-quality researchers in the long run.Ensure early career researchers are supported and integrated into existing research teams.
Evaluation of the Centres for Independent Living (KSL) supported under the Nordrhein-Westfalen ESF OP, 2014-2020, and ESF+/JTF Programme, 2021-2027
Evaluation der Kompetenzzentren Selbstbestimmt Leben (KSL). Eine empirische Untersuchung auf Basis von einer Stakeholderbefragung und Dokumentenanalysen.
OP Nordrhein-Westfalen ESF 2014-2020; Nordrhein-Westfalen ESF+/JTF 2021-2027
Arthur Wawrzonkowski, Esra Eichener, Georg Worthmann
G.I.B. - Gestaltung, Innovation und Beratung in der Arbeits- und Sozialpolitik, www.gib.nrw.de
This is a survey report concerning the “Competence Centres for Self-Determined Living” (KSL) funded under the ESF, ESF+ and by Nordrhein-Westfalen. The study began in 2025 and the report was published in 2026.
To assess the perception and use of KSL by relevant stakeholders, operating in the area of social inclusion. To identify potential for improvement.
3 experts interviews.Document analysis: KSL factual report for 2024 and a previous evaluation by the Bochum Centre for Disability Studies (2016-2020).Survey of various stakeholders (overall response rate of 39,8%): municipal representatives on inclusion and disability and advisory council (31,5%), consultants of the Supplementary Independent Participation Counselling (EUTB - 40,9%), stakeholders in regional associations (42,6%), and other actors (43,3%).
KSL was known to most of the stakeholders surveyed, and most of them used its services, such as publications, networking and events.The satisfaction of the stakeholders surveyed is high and the work of the KSL has been assessed as effective in all aspects examined.Most of the stakeholders surveyed have KSL services in their own government district.While respondents from EUTB are extremely satisfied with the work of KSL, it is slightly less the case for the inclusion and disability officers and the advisory council. The actors of regional associations assessed the added value for their own practical work lower than other groups.
Evaluation of the Transport development programme, 2021-2027, in Lithuania
2022–2030 metų plėtros programos valdytojos Lietuvos Respublikos susisiekimo ministerijos susisiekimo plėtros programos pažangos priemonių investicijų įgyvendinimo ir rodiklių pažangos vertinimas
EU Funds’ Investments LT CF/ERDF/ESF+/JTF 2021-2027
Andrius Jaržemskis, Ieva Girdvainienė, Jevgenij Siniov, Aurelijus Jaruševičius
Smart Continent Management Institute; www.smartcontinent.com, lt@smartcontinent.com
The report evaluates the implementation of Lithuania's 2022-2030 Transport Development Programme managed by the Ministry of Transport and Communications. It covers 10 of the 13 planned measures, including road, rail and air connections, traffic safety, infrastructure-use efficiency, 5G and next-generation networks, alternative fuels and sustainable mobility. Evidence covers documents and monitoring data up to 1 July 2025 and only projects with signed contracts by that date.
To assess whether measures are coherent with each other and with other national interventions. To monitor progress towards objectives and the efficiency. To identify future priority areas and refine the investment-allocation model for the next funding period.
Analysis of programme document and other related programmes.Analysis of monitoring and statistical data.Analysis of intervention logic of the measures to trace their effects.Survey of project applicants and promoters to assess whether the planned measures met the needs of the target groups.Over 20 interviews with MAs and implementing social and economic partners (i.e. unions and associations), applicants and project promotersCost-benefit and cost-effectiveness/proportionality analysis based on budget absorption, unit costs and indicator-to-funding ratios.One focus-group with eleven representatives of different transport sectors on future priorities and investment allocation.
The indicator system is not fully adequate as many activities have no result indicators and two have no indicator at all.Some indicators measure outputs rather than results, and the National Progress Plan and the Transport Development Programme are not fully aligned.Measures are aligned with national objectives and do not significantly overlap with other national interventions.Progress is uneven. Some indicators are far above interim or final targets, notably alternative-fuels infrastructure and cycling-infrastructure users.At least six indicators risk underachievement, including TEN-T road-network improvement, air passengers, the share of electric cars and traffic-safety.Budget use varies from 7% to 120% across measures. Most measures show an indicator-to-funding ratio above 1, but air transport and 5G/communications show limited efficiency.The current investment model is considered fragmented, dependent on the annual budget and insufficient for long-term maintenance and modernisation.
Actual project results were not yet available. The assessment is based mainly on signed contracts, planned indicator values and committed funding, not completed outputs or observed outcomes. Some national projects could not be fully included because of data gaps.
Strengthen coordination and consistency between the Transport Development Programme and the National Progress Plan.Review the result-indicator system and link each activity with an appropriate result indicator.Re-establish clear intervention logic and objective criteria for assessing impact on greenhouse-gas and sustainable-mobility.Increase programme flexibility by combining EU funding with national, private and other financial mechanisms.Revise funding requirements to better reflect target-user needs.Integrate sustainable mobility and traffic-safety measures, and combine electrification with demand management, modal shift, public-transport attractiveness and rail competitiveness.
​​Evaluation of measures on career guidance targeting young people under the Piemonte ESF+ Programme, 2021-2027​
Valutazione di Obiettivo Orientamento Piemonte. Parlarne presto e con le imprese - Rapporto 2025
IRES Piemonte; https://www.ires.piemonte.it/
The report focuses on the implementation of measures on career guidance targeting young people between 8 and 24 years old, funded under the programme. Monitoring data on participants refer to the period September 2024 – August 2025; interviews were carried out in October – November 2025.
To assess the level of implementation of career guidance measures, the participation rates of beneficiaries, with a focus on 13-year-old students, and the functioning of visits to firms of students aged 8-15.
Desk research on programme documents.Analysis of statistical and monitoring data.9 Interviews with guidance coaches, primary and secondary school teachers and tutors from hosting firms.
Analysed measures include three types of guidance activities targeting young people: introductory guidance services, strengthening of career management skills, and support to school-school and school-to-work transitions.During 2024/2025, 142,800 (+23% compared to previous year) people were involved in the 14,750 activities (+27% compared to previous year) offered by the measures.Among participants, 77% are children and teenagers between 8-15 years old; 23% are teenagers and youngsters between 16-24 years old.Overall, the majority (63.6%) of beneficiaries participated in activities aimed at strengthening career management skills.Overall, 80% of those aged 13 of the region have participated in the measure (about 30,300 children), with the highest rates in the provinces of Cuneo and Vercelli.Visits to firms are most effective when they are co-planned between schools, guidance coaches and firms, and when they include follow-up work.Students have shown involvement and curiosity with respect to visits to firms. The feedback on their engagement is positive overall.Difficulties in organising visits to firms mainly depend on the rigidity of school hours, on firm safety protocols, and lack of adequate public transport between school and firms.
Define more clearly roles and responsibilities of guidance coaches throughout the various stages of the visits to firms.Provide for exchange of best practices between guidance coaches.Increase awareness of the objectives of the visits to firms in both schools and enterprises.Include activities that provide for some type of return to the host firm.Prepare students for the visits and for the follow-up work, to maximise their effectiveness.Keep monitoring early guidance activities, collecting feedback from all the actors involved.
Effects of support for young people under Knowledge Education Growth OP 2014-2020 in Poland
Badanie efektów wsparcia zrealizowanego na rzecz osób młodych w ramach Programu Operacyjnego Wiedza Edukacja Rozwój - III RAPORT WSKAŹNIKOWY
Iga Magda Ph.D, Jan Baran, Wojciech Hardy, Henryk Kalinowski
Instytut Badań Strukturalnych Wiśniowa 40B lok.8, 02-520 Warszawa, Polska, Tel: 22 629 33 82, e-mail: ibs@ibs.org.pl - Imapp Sp. z o. o. Puławska 145, 02-715 Warszawa, Tel: + 48 22 390 53 04, imapp@imapp.pl - IQS Sp z. o.o. ul. Mangalia 2A, 02-758 Warszaw
The evaluation assesses the effectiveness of the YEI initiatives to assist young NEETS under the Knowledge Education Development OP in Poland.
To assess the effectiveness of the support to young NEETs provided under the OP.
Survey.Analysis of indicators.Econometric regression model.
27% of participants were in education 6 months after finishing the programme.The chances of finding employment depend on age and education level – 69% of those aged 25-29 doing so and 76% for those with tertiary education.13% of participants were self-employed 6 months after finishing the programme.60% of participants were employed 6 months after finishing the programme.Those participating in projects of local labour offices were more likely to find employment than others.
There is a need to provide support to more people with low education.
Evaluation of the A40 Llanddewi Velfrey to Redstone Cross highway operation under the West Wales and the Valleys ERDF OP, 2014-2020
Formative Evaluation of ERDF Funded A40 Road Operation
OP West Wales and The Valleys ERDF 2014-2020
The evaluation examines progress on two linked schemes (Llanddewi Velfrey to Penblewin and Penblewin to Redstone Cross), which together form the A40 highway operation in Mid and South West Wales funded under the programme. The highway is part of the E30 TEN-T route extending from Ireland to Russia. Data cover the period up to October 2023.
​​To assess whether the measure on the A40 highway has achieved the targets set and to provide recommendations for future, similar interventions.
Desk-based review. Analysis of programme monitoring data. Analysis of geospatial (Ordnance Survey) data for map production. Analysis of ONS 2021 Census data. Analysis of TrafficMaster data for baseline values and INRIX GPS-based data for ongoing measurement. 48 semi-structured interviews to members of the project delivery teams (22), Welsh Government representatives (6), Local authority representative (9), Community and business organisations (11). Reconstruction of the Logic of intervention.
According to 2021 Census data, almost two thirds (61%) of workers aged 16–74 in the region commuted by car, van, motorcycle, scooter or moped (around 170,278 individuals). In comparison, 7% (20,840 individuals) travelled on foot or by bike, while only 2% used public transport (train, bus, minibus or coach). Despite a policy shift towards low-carbon priorities, the A40 Llanddewi Velfrey to Redstone Cross operation remains strategically aligned with Welsh Government strategic objectives to improve regional connectivity and wellbeing. Project management arrangements are seen as effective and working well, and the monitoring and reporting systems are robust, with strong coordination across actors. All cross-cutting theme indicators have been achieved or exceeded, due to a wide range of actions related to sustainability, community benefits and active travel. Progress on output targets is very slow, with 1 km of footway/cycleway completed and 4.5 km of road upgraded so far. Most of the outcomes (journey duration; reduction in collisions; noise pollution; traffic reduction..) cannot yet be identified. A 3–8% increase in average speed along the A40 has been recorded against a baseline of 58 kph as defined in the business plan. In terms of value for money, the scheme shows small returns (benefit to cost ratio: 0.13–0.27), but it is still considered by stakeholders to deliver public value, particularly in terms of safety and quality of life.
The impact on air quality cannot be robustly assessed due to the absence of monitoring data in the project area and the use of a distant monitoring site, which prevents reliable attribution of any observed changes to the measure.
Appoint a dedicated Public Liaison Officer to strengthen community engagement and deliver local benefits. Explicitly embed cross-cutting themes in contracts to ensure early commitment and effective delivery. Organise collaborative workshops in the early stages of projects to foster coordination and open communication among delivery teams.
Evaluation of the support for care and education implemented under the Podlaskie OP, 2014–2020
Ocena wpływu wsparcia RPOWP 2014-2020 na edukację opiekuńczo-wychowawczą w województwie podlaskim – etap 3
Anna Czauż, Edyta Dąbrowska, Maciej Muczyński
Urząd Marszałkowski Województwa Podlaskiego, Departament Zarządzania Funduszami i Programami; www.podlaskie.eu
The evaluation is an internal assessment conducted by the Evaluation Unit of the MA. It was conducted to fulfil the obligation to estimate the value of the long-term result indicators in the field of care and upbringing education. Data go up until end-2024.
To estimate the value of the long-term result indicators: 1) number of childcare places for children up to the age of three, which are in operation 2 years after receiving ESF funding, 2) number of pre-school education places in operation 2 years after receiving ESF funding.
Desk research (programme and strategic documents).Analysis of monitoring and statistical data.Literature review of evaluation reports.CATI/CAWI survey of 113 beneficiaries.
The number of childcare places for children up to the age of three which are in operation 2 years after receiving ESF funding amounted to 1,091, compared with 1,448 places created in total by projects completed by end-2022. This represents a sustainability rate of 75.3%. The achieved value exceeded the planned target (70%).The number of pre-school education places in operation 2 years after receiving ESF funding amounted to 2,282, compared with 2,518 places created in total by projects completed by end-2022. This represents a sustainability rate of 90.6%. The achieved value exceeded the planned target (82%).
​​Evaluation of individual continuing and lifelong learning measures supported under the Piemonte ESF+ Programme, 2021-2027​
La formazione individuale continua e permanente in Piemonte: il punto di vista degli enti di formazione
The report is an evaluation of early implementation of individual continuing and lifelong training measure financed under the programme. The measure supports employed adults through a catalogue of courses and a training vouchers. The monitoring data used go up to November 2025.
To assess how the regional Guidance Act and Notice on individual continuing and lifelong training have been translated into operational practice by training providers. To examine the organisation and implementation of the course catalogue and training vouchers, to identify operational and administrative bottlenecks, reconstruct the regional programmer's perspective, and provide a first descriptive assessment of uptake, offer and participants’ profile.
Document analysis of the regional Guidance Act, Notice and related implementation frameworkReview of literature and secondary sources on continuing training in Italy and EuropeAnalysis of statistical office data.Analysis of monitoring data.Analysis of other secondary data (INAPP - national institute for public policy analysis).One in-depth interview conducted in June 2025 with the regional officials responsible for the measureFour remote semi-structured focus groups with 15 training agencies out of 104 in the catalogue selected for heterogeneity by size, territorial location and weight of continuing training
Monitoring data show substantial early uptake: by November 2025 the catalogue included 1,481 courses from 104 agencies across 238 sites; 13,720 training vouchers had been activated, 93% on individual initiative, and more than 42% had already reached the reporting stage.The measure mainly reaches relatively educated employed adults: 58.3% of recipients are women, 65.7% are aged 30-54, 35.6% hold a degree or postgraduate qualification, and only 10.9% have low education levels.Training providers see the measure as highly relevant and fair because public co-financing allows workers with low income to access training; the shift from voucher to training voucher is generally viewed positively.Long intervals between proposal, authorisation and course start limit the demand; providers report losing interested participants during the lag and consider the 15% cap on participant replacement too restrictive.The standard cost of EUR 12.75 per hour per participant is seen as barely adequate for basic classroom training and not sufficient for specialist or laboratory activities; delayed regional reimbursement also creates cash-flow pressure for providers.
Mid-term Evaluation of the national ERDF and ESF+ Programmes 2021–2027
Midtvejsevaluering af Danmarks nationale programmer for EU’s Regionalfond og Socialfond Plus 2021-2027
Education and Skills DK ESF+ 2021-2027; Smart green and digital transition of enterprises DK ERDF 2021-2027
Pluss leadership: post@pluss.dkIris Group: irisgroup@irisgroup.dk
Interim evaluation of the Transport Programme 2021-2027
Evaluarea intermediara a Programului Transport 2021-2027
Managing Authority for the Transport Programme (Ministry of Transport and Infrastructure)
EvalEvaluation of the implementation of the Interreg Saxony–Czech Republic 2021–2027
Evaluierungen während des Programmplanungszeitraumes 2021-2027 für das Programm Interreg Sachsen – Tschechien 2021-2027. Umsetzungsanalyse des Programmes 2021-2027. Endbericht.
Interreg VI-A Germany/Saxony-Czechia 2021-2027
Erich Dallhammer, Arndt Münch, Wolfgang Neugebauer, Chien-Hui Hsiung, Roland Gaugitsch
Evaluation of the Business Support Services Wales (BSSW) programme under the West Wales and the Valleys and East Wales ERDF OPs, 2014-2020
Evaluation of Business Support Services Wales
OP West Wales and The Valleys ERDF 2014-2020; OP East Wales ERDF 2014-2020
Oliver Allies, Michael Pang, Benjamin Reinertz, Prabhat Vaze
https://www.wavehill.com, oliver.allies@wavehill.com
This final evaluation assesses the Business Support Services Wales (BSSW) programme delivered between 2016 and April 2023 under the OPs. It covers Business Wales Entrepreneurship Support, SME Support, the Accelerated Growth Programme, Superfast Broadband Exploitation (SFBE) and, where relevant, Enterprise Hubs, and examines both implementation and impact on supported businesses. Data used goes up mid-2023.
To assess the performance, effectiveness and impact of the programme to improve entrepreneurship and business competitiveness, implemented under the ERDF OPs.
Document review to reconstruct programme rationale and performance.Analysis of administrative and monitoring data (MI) on beneficiaries and support patterns.Use of national statistical data (ONS Business Structure Database).Use of secondary data, mainly previous evaluations and relevant reports.Survey of beneficiaries (1,014 responses from firms supported from 2019 onwards, 15% response rate).25 scoping interviews with stakeholders. Those who could provide an informed, strategic perspective on the provision or who had been involved in the delivery of the support services associated with the operations.CIE to explore firm- level employment and turnover impact in participant businesses. The CIE combines PSM with DiD using firm-level data to estimate impact on employment, turnover and other business outcomes.
The Business Wales programme is well established and widely recognised, functioning effectively as a “one-stop shop” for business support.The programme responds to structural weaknesses of the Welsh economy (e.g. low productivity and entrepreneurial activity), and shows high accessibility and satisfaction from beneficiaries and stakeholders.Partnerships and referral mechanisms have improved over time, enhancing outreach and engagement, especially for underrepresented groups. Enterprise Hubs also play an important role in reaching new and previously underserved entrepreneurs.The programme has strengthened linkages with growth-oriented support instruments (e.g. Development Bank of Wales, SMART schemes).The CIE indicates positive effects on firm-level outcomes (notably employment and turnover), although the robustness of results varies and is more limited where reliance on self-reported data is higher.The CIE finds significant positive impact of the SME Support programme (around 47% additionality on employment growth over three years), while SFBE-supported firms underperform relative to the control group. Productivity effects are initially negative but converge over time with those of the control group.Overall, the intervention appears effective in improving access to business support and strengthening the SME ecosystem.
The evaluation is limited by data quality issues, including incomplete firm identifiers and sample bias, which constrain the counterfactual analysis. As a result, part of the impact assessment relies on self-reported data, which reduce the robustness of causal inference.
Maintain and further develop the “one-stop shop” model, which has proven effective in simplifying access to support.Strengthen the integration of support instruments, as fragmentation remains an issue despite improvements.Enhance outreach to underrepresented groups, building on the success of Enterprise Hubs.Improve data systems and monitoring, particularly firm identifiers and longitudinal tracking, to enable more robust impact evaluation.Consolidate links between general support and high-growth programmes.
​​Mid-term evaluation of the Smart Specialisation Strategy (S3) under the Castilla-La Mancha ERDF Programme, 2021-2027​
Informe de evaluación intermedia de la Estrategia Regional de Especialización Inteligente 2021-2027 de Castilla-La Mancha
Agencia de Investigación e Innovación de Castilla-La Mancha (INNOCAM)
The report is a mid-term evaluation of the Regional Smart Specialisation Strategy (S3) of Castilla-La Mancha and assesses the region’s progress in research, development and innovation (R&D&I), expenditure implementation, the performance of the S3 indicator system. The evaluation covers the implementation period 2021-2024. Monitoring data go up to end-2024 whereas qualitative evidence is collected up to early 2025.
To assess the progress of the S3, analyse the implementation of projects and expenditure, its alignment with current European innovation trends and regional needs, and the governance model. To identify areas for improvement for the remainder of the programming period and to propose governance and policy adjustments for 2025-2027.
Desk review of the S3, related regional plans, and relevant national and European innovation policy documents.Analysis of official statistical data.Analysis of monitoring data (S3 monitoring indicators) for the period 2021-2024.Mapping of 127 initiatives linked to the S3 and comparison of planned and executed expenditure by challenge and strategic line.Four semi-structured interviews with key management actors (regional government and university representatives) involved in implementing the S3.Online survey of 41 actors from the regional research, development and innovation ecosystem.One contrast workshop with 30 participants (representatives from the public sector, universities or other training institutions, beneficiary firms and business associations) and evidence from two follow-up meetings of Axis 3 'Knowledge' of the Pact for Economic Reactivation (19 and 25 participants).
127 initiatives have been planned or launched, 92.1% had been implemented and 22% are already completed, with activity concentrated in 2022-2023.Executed expenditure for 2021-2024 amounted to EUR 3.2 billion, 22.6% less than planned (EUR 4.2 billion).Indicator progress is mixed. Those concerning “Talent and excellence in R&D&I” perform best, while those on “Strengthening the R&D&I system” are below the target, especially on patents and Horizon Europe return.Across the priority sectors, the number of projects financed by the regional government and the investment mobilised generally exceed the 2024 targets. Aeronautics is the only sector meeting or exceeding all sector-level targets.Implementation and expenditure are heavily concentrated in “energy transition and circular economy” but energy‑consumption and emissions‑reduction targets have not been met.The areas of demographic challenge and social inclusion show the lowest implementation level, with no directly linked initiatives and no budget execution during 2021-2024.There are persistent bottlenecks in talent attraction and retention, weak internationalisation, limited uptake of innovative technologies, and underdeveloped governance mechanisms.
For part of the implementation analysis, the latest monitoring information available was only up to 2022, with updated period data expected in 2025.
Strengthen governance through a formal S3 monitoring committee led by Castilla-La Mancha Research and Innovation Agency (INNOCAM), with clearer roles and regular follow-up.Create a digital monitoring platform and a more operational set of management indicators.Reinforce public-private collaboration instruments, including innovation hubs, sandboxes, living labs and public procurement of innovation.Improve dissemination of the S3 and the participation of quadruple helix actors in implementation and feedback loops.Strengthen measures for the attraction and retention of talents, including industrial doctorates, research chairs, specialised training, also outside provincial capitals.Develop more targeted instruments for underserved areas, especially demographic challenges, social inclusion and parts of the energy-transition agenda.
Strategic evaluation for the mid-term review of the North-East Regional Programme 2021–2027, in Romania
Evaluarea midterm a intervențiilor Programului Regional Nord-Est în noul context socio-economic și al Recomandărilor Specifice de Țară din 2024
Sorin Grigorescu, Izabella Grădinescu, Adina Sescioreanu, Mihaela Filimon, Oana Ciobotaru, Raluca Bradu, Roxana Cazanegrea
ADR Nord-Est (North-East Regional Development Agency), www.adrnordest.ro/en
This is a strategic evaluation preparing for the mid-term review of the programme. It assesses the implementation of the programme across several key sectors: competitiveness and innovation, digitalisation, energy efficiency, biodiversity and urban mobility, road connectivity, educational infrastructure, integrated urban development, tourism and culture, and technical assistance. Data used go up to April 2025.
To measure the effectiveness of the interventions and their coherence with EU recommendations, in order to facilitate the decision-making process of current implementation and post-2027 programming.
Desk review of the programme, its theory of change, the 2024 Country Report and Country Specific Recommendations (CSRs), the 9th Cohesion Report and other EU policy documents.Analysis of statistical and administrative data from INS, Eurostat, ESPON, JRC-DG REGIO, line ministries, listefirme.ro and other sources.Analysis of monitoring data.Survey of stakeholders and beneficiaries: 1,224 invitations, 1,159 received, 189 responses, including 62 partial responses; response rate 16.31%.Survey of Monitoring Committee members and observers: 60 invitations, 46 received, 21 responses; response rate 45.65%.Interviews with MA and the North-East Regional Development Agency.
The programme remains highly relevant and broadly aligned with the 2024 Country Specific Recommendations, especially for competitiveness, innovation, digitalisation, energy efficiency, sustainable urban mobility, education and administrative capacity.The region faces development- and talent-trap risks: ageing, migration of young and skilled workers, low tertiary development and territorial disparities affect competitiveness and the twin transition.Economic and productivity gaps are substantial: in 2022 GDP per capita in PPS was EUR 16,300, around 46% of the EU average, and productivity remained around 45% of the EU average.Innovation and digitalisation weaknesses persist: R&D expenditure is 0.26% of regional GDP, employment in high-tech sectors is 1.5% of total employment, and the number of innovative firms in 2022 (177) fell by 50% as compared to 2020.Stakeholders generally consider measures on SMEs, digitalisation and financial instruments highly relevant in relation to the barriers faced by SMEs such as skilled labour shortages and difficult access to credit.Green-transition and social needs remain strong, including energy poverty, slow residential decarbonisation, urban green-space gaps, limited clean/local public transport, rural-urban education disparities and weak small-urban attractiveness.Progress of the OP was delayed by its late adoption, management and control-system set-up, state aid/de minimis schemes, administrative burdens, limited market capacity and delays in innovation and circular economy calls.
The evaluation is limited by the early stage of programme implementation and survey results that cannot be generalised due to stakeholders' partial awareness of the project calls.
Maintain the current programme design, including objectives, support forms, financial instruments, implementation mechanisms and capacity-building measures.Accelerate implementation by strengthening the MA and beneficiary capacity, simplifying procedures, improving communication and increasing contracting and payment levels.Adjust allocations where absorption is weak and preserve interventions that can absorb funding in a timely manner.Introduce or strengthen interventions for business infrastructure, STEP pilot projects, New European Bauhaus investments under Priority 7, and social housing or housing facing energy poverty.For the post-2027 period, increase flexibility, simplify implementation, empower beneficiaries, and strengthen coordination across funds and policies.
Interim evaluation of the Smart Growth, Digitalisation and Financial Instruments Programme, 2021-2027, in Romania
Evaluarea Intermediară a Programului Creștere Inteligentă, Digitalizare și Instrumente Financiare 2021-2027
Programme Evaluation and Financial Instruments Service (MIPE)
The report is a mid-term evaluation of the implementation of the programme. It covers all four programme priorities: RDI, digitalisation of public administration and SMEs, digital transformation in the cultural sector, and Strategic Technologies for Europe Platform (STEP). The implementation period covered goes from December 2022 to 31 May 2025. Data used go up to 31 May 2025.
To assess the mid-term progress of the programme, to identify implementation bottlenecks, and to evaluate the programme’s effectiveness, relevance, coherence and its innovative framework.
Document analysis: review of programme documents, applicant guides, implementation documents, and relevant national and EU strategies. Analysis of statistical office data. Analysis of monitoring data: analysis of data from the Single Management Information System 2021 (SMIS2021) and the Structural Fund Communication system (SFC2021). Survey of 350 beneficiaries and potential beneficiaries (112 responses received, 32% response rate). Survey of representatives of the MA and Intermediate Bodies, focusing on implementation, administrative barriers, absorption issues and procedural challenges. Use of the Theory of Change (ToC) to assess whether the intervention logic remains coherent and whether early implementation results are consistent with expected causal pathways.
The programme is strategically relevant and coherent, with strong alignment between programme priorities, national strategies and the EU framework. Implementation is progressing, but unevenly, with stronger progress in some priorities and delays in launching or contracting under several calls. Financial absorption remains low, with actual payments reaching only 2.44% of the total budget as of 31 May 2025. Administrative complexity is a major bottleneck, especially due to complex applicant guides, extensive documentation requirements and constraints in the SMIS2021. Pre-defined projects and the European Digital Innovation Hubs (EDIHs) are identified as positive practices, supporting faster implementation of digitalisation services. The programme shows good adaptive capacity, especially through the inclusion of Strategic Technologies for Europe Platform (STEP) priorities, but several effects remain potential because implementation is still at an early stage.
The main limitations are related to the early stage of implementation, as many calls and projects were launched late or were still under evaluation.
Accelerate implementation, especially by reducing delays in launching calls, project assessment and contracting. Simplify administrative procedures, including applicant guides, documentation requirements and project assessment. Improve the functioning of monitoring systems, especially the SMIS2021, to support more reliable and timely tracking. Strengthen support to beneficiaries, particularly SMEs and applicants facing difficulties in accessing funding. Maintain strategic adaptability, especially by aligning future calls with technological change, European priorities and the STEP.
​​Evaluation of measures on regional development co-financed under the Programme for EU Funds’ Investments, 2021-2027, in Lithuania​
2022–2030 m. Regionų plėtros programos įgyvendinimo tarpinio (pažangos) vertinimas
Jonas Jatkauskas, Inga Bartkevičiūtė, Aušra Šilinskienė, Kasparas Vasiliauskas
The report focuses on the measures on economic development, social cohesion and environmental sustainability of the 2022-2030 Regional Development Plan, co-funded under the Programme, which aims at fostering regional development and reducing territorial imbalances. Data go up to March 2026.
To assess the adequacy, relevance and expected impact of the measures on economic development, cohesion and environmental sustainability, and to provide recommendations regarding future investment for regional development.
Desk research on programme documents, relevant literature, and national and regional legislation.Analysis of statistical and monitoring data.Survey of representatives from Regional Development Councils (10 replies, 100% response rate); survey of socio-economic partners of Regional Development Councils (46 replies, 38% response rate); survey of officials from municipal administrations (46 replies, 77% response rate).17 semi-structured interviews with officials from national, regional and municipal administrations.Macro-economic modelling using the HERLIT equilibrium model.
Measures are suitable for addressing individual municipal challenges, but some needs are given more attention than others, such as education, social services, tourism and waste collection.The implementation of the measures is sometimes hindered by the preconditions set for their funding, which in some cases are not met. This means that the full investment potential available cannot always be used.Regional Development Plans do not contain overarching analyses of territorial issues; rather, they identify the root causes only for problems for which investment is planned.Contribution to reducing regional disparities is mainly expected in the areas of zero-emission mobility, drinking water supply and wastewater management.Weaker regions are assigned a higher volume of funding and, according to macro-modelling, this should result in a greater impact on nominal GDP (at least 3.5% per year between 2025-2028), employment (at least 0.3% in the same period) and productivity (at least 1.1% in the same period).However, this will not accelerate convergence towards the national average in the short and medium terms, and the largest regional economies (Vilnius and Kaunas) will continue to account for 2/3 of the national GDP.Overarching challenges – demographic, geopolitical and climate change related – will require systematic action, crisis management and resilience building on a wider scale.After 2021-2027, investment should focus on the optimization of service provision and adaptation to change, giving regions more discretion to decide how to achieve objectives.The multi-level governance system for planning regional investment is not particularly effective, due to limited coordination between national and regional/municipal authorities, resulting in sub-optimal planning and, sometimes, in regions and municipalities readjusting plans developed at the national level.
Loosen the requirements for funding, which in some cases are too rigid and limit the possibility of implementing the measures.Make sure that the socio-economic analysis of the regional situation contained in the next Regional Development Plan concerns all the relevant issues and their root causes.Define clearly measurable indicators and achievable objectives for the measures in the next programming period, with the support of regional administrations.Ensure better coordination between national and regional/municipal administrations in the development of regional plans and set up specific objectivesDiversify investment for economic growth to foster economic convergence between the regions.
Evaluation of measures on continuing training (education cheques) financed under the Bayern ESF OP, 2014-2020. Final report
Evaluation des Bayerischen Bildungsschecks (Aktion 4b). Abschlussbericht
Institute of Social Research and Social Policy (ISG), feldens@isg-institut.de
The evaluation examines the implementation and effectiveness of Education Cheques (2019-2022) which aimed to support the continuing vocational training of employees to adapt their skills to the digital transformation of their work.
To identify implementation challenges and failures in achieving targets and assess the effectiveness of the funding.
Analysis of funding-related documents and scientific literature.Analysis of monitoring data, official statistics and administrative data (Eurostat, International Institute of Empirical Social Economics, Kantar/Initiative…).2 interviews with stakeholders involved in the planning and implementation of the education cheque funding.14 interviews with training initiators (WBI) and continuing education providers.Standardised survey of continuing education participants: 353 responses, 40% response rate.Standardised survey of continuing education providers (including cost-benefit ratio): 24 answers, 53% response rate.
The financial and material objectives of the support were largely missed (only 1,212 cheques issued vs a target of 6,000), namely because of excessive rules and requirements and the complexity of the overall procedure. Ultimately, only 808 people benefited from the measures.Employees and companies had limited awareness of the measures and the support for providers was not sufficiently attractive, because of poorly comprehensible and manageable framework conditions.Rural areas were often left out by the measure.The introduction of a new support instrument normally requires more than two years to be effective.Employees who benefitted from a training were highly satisfied with the content, organization and implementation of the courses.Cheque-supported training contributed positively to participant’s ability to work and adapt their skills.
Design the measure in a more open, flexible and non-bureaucratic way to improve its uptake.
Ex-post evaluation of measures promoting tourism under PA 4 of the Puglia ERDF OP, 2007-2013
Servizio di valutazione ex-post relativa alla programmazione regionale 2007-2013 in materia di turismo
IZI - https://www.izi.it/ - mail@izi.it; CLES s.r.l. - https://cleseconomia.com/ - cles@cleseconomia.com; LaSER s.r.l. - http://www.laserconsulting.biz/ - laser@laserconsulting.it
The evaluation examines the interventions on tourism supported by the OP in 2007-2013.
To assess the effectiveness, impact and additionality of the measures concerned.
Desk review.Analysis of statistical and monitoring data.Interviews.CAWI survey of 22,500 of tourism providers (14% response rate).Three case studies.Two focus groups.
Support was mainly focused on infrastructure and measures to enhance tourism, for communication and promotion.Infrastructure projects improved the harbours in Vieste, Otranto, Castro and Fasano; parking areas in Castellaneta and Carovigno; and urban regeneration in Palmariggi and Fasano. However, these projects were not adequately integrated into the local development planning.Foreign tourists increased by 90% from 2013 to 2019, and Italians by 20%. Tourists are mainly concentrated in the summer period although the number arriving between March/June and September/October increased by 35-40%.Main channels of promotions are online websites for accommodation and social networks and phone for service providers.Main weaknesses in the region are lack of transport and infrastructure.The role of ‘Pugliapromozione’ as the main body to promote the attractiveness of the region for tourists is assessed positively, as is the strengthening of the regional network of tourist information.Cooperation between providers is not developed enough and the complementarity with other measures is inadequate.
Ensure continuity and coherence with local planning.Implement measures which take account of different aspects of local development, e.g. transport, education, and entrepreneurship.
​​Evaluation of the measure “Training in situation” for persons with disabilities under the Piemonte ESF+ programme, 2021-2027​
Formazione in situazione: analisi di attuazione
The report analyses the implementation of the “Training in situation” pathways for persons with disabilities, an experimental and place-and-train measure that combines work placement, on-the-job learning and tutoring to support social and labour-market inclusion. The report covers three annual calls from 2021-2022 to 2023-2024, involving 208 participants in 34 courses, for a total expenditure of EUR 1.24 million. Data used go up to early 2025.
To examine how the training for disabled people has been implemented in practice, how users are involved in training and work-insertion projects, and which organisational factors facilitate or hinder pathway delivery and labour-market insertion. To assess the immediate outcomes, and to identify, through case analysis, the factors associated with success or failure in the pathways.
Literature review and analysis of statistical data on the relation between disability, poverty and work.Document analysisAnalysis of monitoring data.In-depth interviews conducted between February and April 2024 with 10 implementing bodies and 2 business-representation organisations, involving 24 operators, coordinators or managers.Case-based qualitative analysis of 20 participation histories (10 successful and 10 unsuccessful).
The measure is widely regarded by implementing actors as an innovative and promising place-and-train model, more closely connected to real work contexts than traditional training courses.88% of the overall 208 final recipients were classified as suitable at the end of the pathway, and 59% moved to a traineeship or some employment formula after the project.The main implementation bottleneck is the identification and involvement of companies, especially firms under hiring obligations. Employers are seen as reluctant to recruit persons with disabilities, and company lists provided by employment centres are described as incomplete or outdated.Cooperation with employment centres is important but uneven across territories. More active support is needed on governance, company screening and first contact with firms.Positive outcomes depend on strong tutor support, a good match between participants and tasks, prior knowledge of the participant, and a supportive relational environment involving services and families.
Occupational outcomes cannot be intended as the causal effect of the project. Pathway outcomes depend on highly specific individual, family, organisational and territorial factors, which limits the generalisation of results.
Strengthen the role of employment centres in territorial governance and in the identification and first engagement of firms, especially those subject to hiring obligations.Improve information and communication for employers, including more visible engagement with ‘virtuous’ firms and wider awareness-raising on labour-market inclusion of persons with disabilities.Mainstream the experimental pathways in the 2025-2028 period and continue integration with active labour market policies financed through the Regional Fund for the Employment of Persons with Disabilities.Address mobility barriers affecting participants, including through coordination with transport and welfare authorities.
Evaluation of the outcomes and impact of the West Wales and the Valleys and East Wales ESF OPs, 2014-2020
ESF Participant Survey 2015-2022: analytical report
Lorna Adams, Rhys Davies, Guido Miani, Sam Selner
IFF Research Ltd, www.iffresearch.com; WISERD (Cardiff University), www.wiserd.ac.uk
The evaluation assesses the outcomes and long-term impact of the activities supported under the OPs for different target groups through a survey of beneficiaries. It covers interventions aimed at supporting labour market participation for unemployed and economically inactive individuals and progression in employment and skills development. Data used covers participants leaving ESF-supported projects between 2015 and 2022.
​​​To assess the effectiveness and the impact of the interventions to enhance employability and to help develop skills and knowledge, implemented under the ESF OPs.
Analysis of monitoring data. Proportionate stratified sampling of ESF leavers based on the distribution of background characteristics of participants. CATI telephone interviews with participants supplemented with online questionnaires to 23,228 ESF participants, across two survey waves (2018–2019 and 2022–2023). Counterfactual analysis, using administrative and survey data and Propensity Score Matching to compare ESF participants with matched APS (Annual Population Survey, run by the Office for National Statistics).
66% of the respondents previously unemployed or economically inactive, but who were in employment by the time of the survey, reported that ESF had helped them obtain their current job, including 14% who said they got the job directly because of the training. Over a third of respondents (36%) reported that, six months after completing the projects, they were in jobs requiring skills and competences higher than those they had before. A quarter (25%) said their role required higher qualifications, while 41% reported increased levels of responsibility. Participation of long-term unemployed and economically inactive people was associated to an additional 7 percentage points of those in employment, as compared to the control group, 12 months after the projects (29% for the ESF group compared to 22% for the APS group). Men in long-term unemployment and economic inactivity achieved a 4 p.p. in employment more than the control group. The same result was achieved by participants with educational attainment at or below NQF Level 2. ESF participants with work-limiting ill-health conditions were 4 p.p. more in employment than the APS group. NEETs participants were 16 p.p more into employment, education or training compared to the control group, while the outcome was +13 p.p. for those at risk of poverty. ESF respondents achieved higher qualification attainment, with the share of participants at or below NQF Level 2 falling from 45% before ESF to 36% after participation. When only unemployed individuals are considered, the control group shows higher employment rates than ESF participants.
Participants from some ESF activities could not be included, particularly those under 16 years old or especially vulnerable groups. The counterfactual analysis is subject to several limitations related to data quality and comparability. The ESF Participant Survey collected less detailed information than the Annual Population Survey (APS), which constrained the precision of statistical matching. In addition, matching techniques could only control for observable characteristics and could not account for unobserved factors. The analysis was also limited to outcomes measured 12 months after participation and could not distinguish the specific effects of different components of ESF support. Another key limitation is the APS applies stricter ILO criteria related to job search activity and availability for work, meaning that APS respondents classified as unemployed may be more job-ready than ESF participants, limiting the comparability of employment outcomes.
Strengthen tailored support for participants facing multiple disadvantages. Prioritise qualification attainment and skills development. Improve retention strategies for vulnerable participants. Combine employment support with broader labour market and regional policies. Increase the use of administrative data for future evaluations.
The Smart Specialisation Strategy in Piemonte: elements for monitoring and evaluation as of 2018
La Strategia di specializzazione intelligente del Piemonte
The report monitors the progress of RIS3 in Piemonte and ongoing technological and socio-economic trends which should be taken into account in the remainder of the programming period, as well as in programming the next one. It is a study paving the way for a future evaluation.
To assess the extent to which OP initiatives, aimed at achieving the RIS3 goals, are leading to the expected results.
Qualitative and quantitative analysis of projects;analysis of monitoring data;analysis of macroeconomic data.
The regional policy for research and innovation co-financed by the OP has contributed to:promoting RTDI investment for technology transfer and collaborative research in most of the priority areas of the RIS3;facilitating research that can be readily exploited industrially and commercially;promoting integration between initiatives for innovation and those for development of competences;increasing complementarity and coherence with national initiatives;supporting consolidation of the regional innovation ecosystem and facilitating transition from traditional manufacturing specialisations towards transversal industrial ones, which require a capacity to develop and utilise technologies and digital means that can be taken up by multiple sectors;strengthening multinational corporation ties with Piemonte.
This is a study preliminary to carrying out an evaluation of results and effects.
There is need to:continue to facilitate the transition of the Piemonte innovation system towards Industry 4.0;address the relatively low capacity in the region to commercialise research outputs.
Evaluation of the territorial instruments used under the European Funds for Pomorze programme, 2021-2027
Wstępna ocena efektów wykorzystania instrumentów terytorialnych w ramach programu Fundusze Europejskie dla Pomorza 2021-2027
Adam Rybkowski, Paulina Sęk, Beata Belica, Cezary Gołębiowski, Anna Górecka Ojdana, Maciej Huculak, Roksana Kołodziejczyk, Monika Sochaczewska
The evaluation examines the early implementation of territorial instruments under the programme: Integrated Territorial Investments (ITI), Community-Led Local Development (CLLD), municipal revitalisation programmes and Integrated Territorial Agreements (ITA).
To assess the effects of territorial instruments on the implementation system, their capacity to generate high added-value projects, the durability of territorial cooperation and to identify lessons for the current and future programming periods.
Desk research.Statistical and monitoring data.10 interviews with MA and IB.4 Focus groups with ITI offices, LAGs, ITAs representatives and municipal revitalisation coordinators.Survey of 116 local government units and LAGs involved in territorial instruments.Benchmarking of 3 regional programmes (Zachodniopomorskie, Małopolskie and Kujawsko-Pomorskie).8 case studies.1 panel with experts on territorial instruments.
In ITI, most partnerships use inter-municipal agreements, which offer flexibility and low costs but create risks for institutional durability.ITI has helped local governments shift from project-based to strategic and functional planning, but cooperation outside the area remains largely pragmatic and dependent on access to dedicated EU funding.CLLD is the strongest participatory instrument: preparation of local development strategies (LDS) was orderly and cooperation improved efficiency, alignment to needs, absorption capacity and dialogue, although it rarely extends beyond the LDS framework.Municipal revitalisation programmes are assessed as useful and effective, but their implementation capacity depends strongly on the size and institutional resources of municipalities.Across all instruments, staffing capacity is a major bottleneck.Programme implementation arrangements are generally well-valued, especially communication with the Managing Authority, training and digital tools.However, non-competitive selection in ITI/revitalisation programmes tends to reduce project ambition, while CLLD procedures can overburden smaller applicants.
The early stage of implementation prevents a full assessment of effectiveness and efficiency for some instruments.
Maintain non-competitive selection for complex projects, but link it to clearer milestones, readiness checks and simplified pre-investment procedures.Replace the competitive ITA pathway with a negotiated or non-competitive bundle-based model, focused on coherent groups of projects drawn from territorial strategies.Provide funding for administrative capacity for ITI offices, LAGs and local revitalisation coordinators, including stable staffing, advisory support and practical training.Strengthen participation beyond formal consultation, using deliberative tools, local workshops, design-thinking methods and obligatory feedback reports showing how comments were used.Concentrate revitalisation funding in the next period on fewer and clear priority areas, especially if the available allocation is reduced.Create regular regional learning mechanisms, including cross-LAG forums, expert panels, peer learning, competence support and incentives for inter-area projects and microgrants.
​​Ex post evaluation of the Śląskie OP, 2014-2020​
Ewaluacja ex post RPO województwa Śląskiego 2014-2020
The evaluation investigates the effects of the implementation of the OP at the level of investment priorities.
To assess the effects of programme implementation and to identify solutions to improve implementation in the 2027+ period.
Analysis of monitoring data and statistical office data.Desk analysis of relevant documents.Systematic review of evaluations.16 interviews with implementing institutions.12 focus groups with beneficiaries, 7 with experts, and 5 with representatives of local authorities.
Most of the programme’s result indicators were achieved. This was facilitated by the improvement in the overall socio-economic situation in the country and the region.The region achieved the target level of R&D expenditure as a share of GDP (1% in 2023).There was a significant increase in the share of innovative enterprises (from 14% in 2012 to 37.5% in 2022).54.85 km of new roads were constructed, 96.43 km of roads were upgraded, and 52.17 km of railway lines were modernised with funding from the programme.The Road Transport Accessibility Index increased to 63.38 in 2023, compared to 56.67 in 2013.Due to the significant improvement in the regional labour market, programme interventions in the area of employment showed a relatively high level of deadweight (around 50%).
Place emphasis on the energy and climate transition in 2028–2034, including energy diversification and support for people affected by energy poverty.Focus on modernising traditional sectors through automation and support for green and AI-related R&D.Place greater emphasis on social services for older people in response to population ageing.Develop dual-use infrastructure that serves civilian purposes and can be quickly used in emergencies for civil protection.
​​Mid-term evaluation of the Smart Specialisation Strategy under the Extremadura ERDF programme, 2021-2027​
Evaluación intermedia 2021-2024 de la RIS3 Extremadura 2027
The report is a mid-term evaluation of the RIS3 under the OP 2021-2027. It covers interventions implemented in 2021-2024. Data used go up to 2025.
To assess the relevance of the measures and coherence with EU, national and regional frameworks, their effectiveness, efficiency and sustainability. To assess governance, monitoring, participation and the ERDF enabling condition for good governance of smart specialisation. To update RIS3 for 2025-2027 and informing the VIII Regional R&D&I Plan.
Document review of programmes and regulatory and strategic framework documents.Analysis of monitoring data.Analysis of data from statistical sources, the RIS3 Observatory, the scorecard.22 in-depth interviews with key actors of the regional science, technology and innovation ecosystem.Two working groups with actors from the quadruple helix (public authorities, businesses, research and knowledge institutions, and civil society/citizens).Four surveys (119 valid responses) of companies, knowledge-generating bodies, intermediary organisations and citizens.
RIS3 remains broadly relevant and coherent with EU and regional frameworks, especially on smart specialisation, the green and digital transitions and R&D&I. But specialisation pattern needs an update to reflect challenges in terms on advanced digital technologies, energy storage, green hydrogen, raw materials, health technologies and under-represented sectors.580 projects have been launched, which mobilised EUR 130.7 million and involving 213 organisations. Agri-food is the most covered area, followed by green transition and digital transformation.The RIS3 impact indicator "Returns from participation in the EU Framework Programme"—which measures the capacity of local universities and businesses to win competitive European research funding—recorded EUR 16.7 million in 2023, comfortably exceeding the regional strategy's final target of EUR 3 million set for 2027. However, R&D intensity, business innovation expenditure, private R&D and the number of companies spending on innovation remain below the 2027 objectives.RIS3 has strengthened capacities, infrastructure, alliances and European cooperation, but its direct effects on SMEs, micro-enterprises and rural areas remain limited. The Entrepreneurial Discovery process (EDP) and collaborative platforms are in place, but business participation and visible follow-up are limited.Governance structures are adequate, but communication, transparency, annual reporting, administrative efficiency and the practical use of the monitoring system need improvement.
The report recognises that it is still early to measure structural impacts on production, employment and competitiveness. The implementation analysis is affected by limitations in project classification, incomplete thematic mapping and differences in project size, so the number of projects should not be interpreted as equivalent to investment intensity. Some indicators have no baseline or complete data. Evidence on effects by programme is not systematically disaggregated and relies partly on perceptions from interviews and surveys.
Simplify RIS3 management, moving towards a single processing platform, a stable calendar of calls and shorter resolution times.Strengthen specialised human resources in the administration and the System of Science, Technology and Innovation of Extremadura (SECTI), including stable calls for R&D&I talents and for digital and innovation-management skills.Extend innovation to SMEs, micro-enterprises and rural areas through technical support, smaller adapted grants, smart-territory actions and a clearer map of R&D&I agents and infrastructures.Refocus the EDP on a limited number of regional challenges and tractor projects, with clearer methodology, support and visible outputs.Improve governance and monitoring through an interdepartmental technical group, annual public reports, a more usable Observatory and a revised scorecard focused on verifiable results and impacts.
Evaluation of the implementation of measures for early childhood education under the Piemonte ESF OP, 2014-2020, and ESF+ Programme, 2021-2027
Il sostegno al prolungamento dell'orario nei servizi educativi per la prima infanzia: analisi di attuazione nell'anno educativo 2023/24
ROP Piemonte ESF 2014-2020, RP Piemonte ESF+2021-2027
The evaluation examines the measure for extending the opening hours in municipally owned nurseries and other early childhood education services. Monitoring data refers to three school years between 2021 and 2024 whereas interviews have been conducted between April and June 2025.
To assess how the measure was implemented, whether it responds to families’ work-life balance needs, and the constraints and challenges faced by municipalities. To provide recommendations for improving the continuity, effectiveness and territorial coverage of the intervention.
Document analysis.Analysis of statistical data from EUROSTAT, ISTAT and the regional statistical office.Analysis of indicators from monitoring data related to the intervention.23 interviews with 43 participants including regional officials, administrative staff of beneficiary and non-beneficiary municipalities, educators and pedagogical coordinators.
The measure is relevant because it addresses families’ need for more flexible childcare hours and supports work-life balance.Implementation spanned over three years, but participation remains uneven across municipalities and territories.Extended opening is mainly used in the afternoon, while early opening is less common.Municipalities and service providers generally value the measure positively, especially for working parents.Main implementation constraints concern staff recruitment, organisational sustainability, annual funding cycles and limited planning capacity.Smaller municipalities face greater difficulties in using the measure.The intervention improves flexibility in childcare provision, but it does not solve broader territorial inequalities in childcare access.
Give more support to smaller municipalities, which face greater organisational and financial difficulties.Plan additional measures to reduce territorial disparities in childcare provision, since the intervention improves flexibility but does not fully address unequal access across different areas.Improve monitoring of users and demand, especially to understand which families use extended hours and whether the measure reaches those most in need.Better integrate the measure with broader work-life balance and childcare policies, rather than only considering it as a short-term service extension.
Evaluation of the effects of apprenticeship measures under the Piemonte ESF+ programme, 2021-2027
Valutazione degli effetti dell’apprendistato per il diploma di istruzione secondaria superiore nella Regione Piemonte
Luisa Donato, Samuele Poy, Pasquale Cirillo
The report evaluates the apprenticeship measure for obtaining an upper secondary education diploma. It combines a descriptive analysis with a counterfactual assessment of employment effects completing the pathway in school years 2020-21 to 2022-23. Labour-market outcomes are observed up to December 2024.
To assess whether participation in apprenticeship for the upper secondary diploma improves the labour-market insertion of young people, particularly in the short and medium term after graduation. To document participation patterns, outcomes and enterprise involvement, and to draw policy lessons for strengthening the measure.
Document analysis of the programmeAnalysis of regional statistical data.Analysis of monitoring data: regional monitoring data on apprenticeship participants.Analysis of administrative data: Compulsory Communications labour-market data (COB), and jobseeker declarations of immediate availability for work (DID).Counterfactual impact evaluation using exact matching / Coarsened Exact Matching (CEM). The baseline treated group comprises 224 apprenticeship participants who completed the pathway in 2020-21, 2021-22 and 2022-23 and did not proceed to university, compared with matched controls drawn from 17,549 young people issuing a Declaration of Immediate Availability for work.
Participation rose from 91 students in 2020-21 to 159 in 2023-24 (+75%), completion was very high (94.1%), and around one third of those completing the pathway moved into either a permanent contract (11.3%) or a professional apprenticeship (23.0%).The measure nevertheless remains small relative to the upper secondary student population and is highly concentrated in the Metropolitan City of Torino (78.8% of participants), with strong concentration in technical and vocational institutes and predominant participation of men.The counterfactual analysis finds a positive employment effect. In the baseline estimates, employment is 71.9% for treated participants versus 19.1% for the control group after 6 months, 82.1% versus 53.3% after 12 months, and 81.2% versus 56.7% after 18 months.Effects are much larger where there is continuity with the host firm, i.e. for participants whose contract becomes a professional apprenticeship or continues on an open-ended basis.Positive effects are found for participants of both technical and vocational schools, but the effects are stronger for vocational institute participants in the first 12 months. Annual versus biennial duration of projects does not materially change effectiveness.
The causal interpretation of the estimated impact is limited by the potential presence of unobserved or uncontrolled differences between treated and control groups. Some analyses use small subsamples, especially heterogeneity and longer-term follow-up, limiting the robustness of the results.
Institutionalise and promote first-level apprenticeship more in technical and vocational schools, including beyond Torino.Simplify administrative procedures and provide targeted support to schools and tutors, especially for managing Individual Training Plans and assessment.Strengthen school-enterprise links and incentives for continuity with the host employer, since re-hiring or contract transformation appears central to effectiveness.Preserve the dual training logic of the measure and improve monitoring and data collection, including on costs, benefits and the experience of participants and firms.
Evaluation of the 'Escuelas Conectadas' programme to implement broadband in schools, under the Multiregional OP for Spain, 2014-2020
Evaluación de la contribución del Programa Escuelas Conectadas de RED.es
Instituto de Estudios Fiscales (IEF) , Ministerio de Hacienda, https://www.ief.es/
The evaluation examines the effectiveness of the « Escuelas Conectadas » programme aimed to provide public, non-university educational institutions with ultra-fast broadband Internet connectivity of at least 100 Mbps, as well as the supply, installation, configuration, deployment, and maintenance of a wireless system.
To assess the contribution of the programme to ICT adoption and use in Spanish public schools, in order to identify enabling factors and inform future investments in digital educational infrastructure.
Desk analysis, including review of previous evaluations. Reconstruction of the logic of intervention. Analysis of administrative data from the Public Employment Service (SEPE) at municipal level. Analysis of data from the National Statistics Institute (INE). Analysis of monitoring indicators. Online survey (2021–2022) of 16,987 recipients, including 3,355 schools' representatives, 12,291 teachers and 3,747 families. Factor analysis and reliability testing (Cronbach’s alpha) to validate the constructs measured through surveys. Contribution analysis with Structural Equation Models (SEM) with variables relating to infrastructure quality, ICT equipment and software use, teacher capacities, school characteristics and contextual factors. Model adequacy assessment through goodness-of-fit indices (χ², RMSEA, CFI, TLI and SRMR).
The interventions provided ultra-fast broadband (up to 1 Gbps) and internal Wi-Fi networks to around 14,000 educational centers, reaching approximately 3.6–4 millions of students in Spain. The programme has helped strengthen the focus on digital competences in schools with 94% of schools reporting increased demand for ICT training and 73.8% of teachers considering ICT increasingly important for teaching and learning processes. ICT-related constructs demonstrated high reliability across respondent groups, with Cronbach’s alpha values ranging from 0.88 to 0.97 for schools, teachers and families. Infrastructure quality emerged as the strongest explanatory factor for ICT adoption at school level (β = 0.81; p = 0.04), confirming broadband infrastructure as a key enabling condition for ICT uptake. Families associated ICT adoption with infrastructure quality (β = 0.28; p < 0.001), and ICT competences within families (β = 0.09; p < 0.001). Structural disadvantage at municipal level exerted a negative influence on ICT adoption (β = –0.73; p = 0.09), suggesting disadvantaged territorial contexts may constrain the effectiveness of the programme. Triangulation of findings with other evaluations indicate that infrastructure improvements effectively supported ICT adoption, but sustained educational transformation also depended on teachers’ capacities. Triangulated evidence confirmed that supported schools were more likely to developed innovative projects, especially where ICT funding was broader.
Continue investments in broadband infrastructure and connectivity. Prioritise disadvantaged schools and vulnerable territorial contexts to reduce digital inequalities. Strengthen digital training for teachers and pedagogical ICT integration. Promote complementary institutional support and leadership capacity to sustain educational transformation. Increase family involvement and digital literacy to maximise programme benefits.
Evaluation of the support for the adaptability of workers and businesses under the Łódzkie OP, 2014–2020
Ocena efektów wsparcia w zakresie adaptacyjności pracowników i przedsiębiorstw w ramach OPX RPO Województwa Łódzkiego 2014–2020
Maciej Mroczek, Karolina Szczepaniak, Jagoda Przybysz, Andrzej Kacprzak
Agencja Badawcza EDBAD Sp. z o.o., www.edbad.edu.pl
The evaluation was carried out from July to December 2025 and covered measures supporting the professional activation of people caring for children under the age of 3 (X.1), the development of employees' and entrepreneurs' competences and qualifications (X.2.1), outplacement activities (X.2.2) and health programmes supporting the professional activity of region's residents, especially those aged 50+ (X.3.1–X.3.3).
To examine the impact of support under Priority Axis X (Adaptability of employees and enterprises in the region) of the OP on increasing adaptability to change and competitiveness of SMEs.
Desk research (programme and strategic documents).Analysis of monitoring data.CATI/CAWI survey of 2,932 project participants.44 interviews with the recipients of support, beneficiaries and representatives of the MA.3 expert interviews.Case studies.
In relation to Measure X.1 – 'Return to the labour market of persons caring for children under the age of 3' outstanding result indicators were achieved, as 96.5% of participants returned to the labour market or remained employed.In the area of 'Development of employees and enterprises' the Entity Financing System (PSF) proved accurate. As part of the outplacement, high employment effectiveness was recorded and the durability of newly established businesses was high, as 85% of companies continued to operate after the support ended.Health programmes covered over 110,000 people, and participation rates in preventive screening tests exceeded the target levels by more than 230%. The entry into employment and continuation rates significantly exceeded the target values.
Promote projects that individualise participants' support paths by combining various components into a coherent activation model.Introduce simplified settlement rules (lump sums, unit costs) and uniform interpretations of the National Insurance Institution (ZUS) and VAT for activation projects.For health interventions, use qualitative and long-term indicators that better reflect the real impact on vocational activity.
Evaluation of the effectiveness of measures to promote social dialogue in the labour market, under the Knowledge Education Growth OP, 2014-2020, in Poland
Ocena efektywności wspierania dialogu społecznego w obszarze rynku pracy
The evaluation examines the impact of projects implemented under Measure 2.20 of the OP, which aims to strengthen the role of social partners in labour market governance. It covers training and analytical activities designed to support the development of skills and institutional capacity in the context of social dialogue and evidence-based policymaking.
To assess the support provided to social partners in relation to the governance of the labour market and its contribution to strengthening social dialogue and improving the quality of participation in the public policy process.
Analysis of monitoring and statistical data.Survey with 314 participants.25 interviews with beneficiaries and participants.3 focus group interviews.Expert panel.Delphi panel.2 case studies.
The intervention effectively addressed social partners' needs by bridging gaps in terms of organisational and analytical capacities. Training about negotiation, communication, and labour law directly enhanced participants' core competencies.The support boosted organisational professionalism and fostered evidence-based positions in social dialogue. Strong participant interest is reflected in the significantly exceeded targets and outputs of the project.The quality and application of analytical tools varied across organisations. Key challenges included uneven institutional capacities and limited resources.The measures’ success relied on integrating training with analytical work and aligning activities with the real-world decision-making process. Key drivers included practical relevance, high-quality trainers, and interactive course formats.
Beneficiaries were required to present the findings of their analysis at social dialogue forums. However, due to inconsistent reporting standards and pandemic-related restrictions, the data available was incomplete, which limited the possibility of conducting the evaluation as planned.
Define minimum methodological standards in the documentation of calls for projects and introduce mechanisms to engage specialised research entities.Define minimum requirements for the structure of research reports, including a mandatory “Conclusions and recommendations” section.Increase the use of practical and interactive training methods supporting applied skills.
Evaluation of the support for the development of R&D infrastructure in research institutions and enterprises, financed by Cohesion policy in Poland, in 2014–2020 and 2021–2027
Ewaluacja efektów wsparcia rozwoju infrastruktury B+R w jednostkach naukowych i przedsiębiorstwach w ramach polityki spójności 2014-2020 i 2021-2027
OP Smart Growth 2014-2020, EU Funds for Smart Economy 2021-2027, All ROPs 2014-2020, All Regional Programmes 2021-2027
Andrzej Dziurdzik, Tomasz Klimczak, Bartosz Ledzion, Marta Mackiewicz, Adam Miller
The evaluation assesses the impact of interventions for R&D infrastructure on the competitiveness and innovativeness of the Polish economy.
To assess the impact of R&D infrastructure projects implemented under the Regional OPs, the Smart Growth OP (2014–2020), and the Smart Economy Programme (2021–2027).
Analysis of monitoring data and statistical office data.Desk analysis of relevant documents and review of evaluations.35 interviews with implementing institutions, 4 with experts and 32 with beneficiaries.CAWI/CATI survey of beneficiaries: 292 enterprises and 600 scientists.Bibliometric analysis (Regression Discontinuity Design).5 case studies.Benchmarking.2 workshops.
Projects funded under the 2014–2020 programmes involved 29% of entities in Poland’s science sector and accounted for more than half of the overall R&D investment in the sector between 2017 and 2023.R&D funding helped increase the scale and quality of R&D in the science sector.According to 84% of researchers, the infrastructure they use is among the most advanced they have ever had access to.Researchers using the supported infrastructure produce publications with a citation impact more than twice the global average (Field-Weighted Citation Impact > 2). The increase in FWCI from 2.10 to 2.6 suggests that the infrastructure supports higher-impact research.For nearly half of the enterprises, the supported R&D infrastructure was their first investment of this kind and allowed them to start their own research activities.Thanks to the support, 79% of enterprises introduced new products or services, and 37% introduced process innovations.
Continue support for investments in R&D infrastructure in the science and business sectors in the 2028–2034 period.Concentrate national support on large, supra-regional projects and regional support on flexible tools for smaller entities and local R&D activities.
Mid-term evaluation of the Smart Specialisation Strategy of the Baleares ERDF Programme, 2021-2027
INFORME DE EVALUACIÓN INTERMEDIA DE LA ESTRATEGIA DE ESPECIALIZACIÓN INTELIGENTE DE LAS ISLAS BALEARES (RIS3) 2021-2024
The report is a mid-term evaluation of the RIS3, focusing on governance, monitoring, and implementation progress. The period analysed is 2021–2024, but the report uses data available up to September 2025. Some statistical indicators are more delayed and mainly refer to 2022–2024.
To assess the mid-term implementation of the RIS3 and to examine the functioning of governance, monitoring, and participation.
hod/DataAnalysis of monitoring data.Desk research: review of the RIS3 strategy, regional policy documents, relevant EU/national strategic frameworks, and policy papers.Stakeholder survey: questionnaire sent to 378 valid recipients; 49 responses received (response rate 13%).Qualitative interviews: 12 in-depth interviews with key regional innovation actors, including business clusters, Chambers of Commerce, regional government representatives, public agencies, and the University of Baleares.Analysis of official statistics (Instituto Nacional de Estadística).
Implementation is uneven: only part of the planned RIS3 initiatives is underway, with stronger progress in established areas and weaker progress in emerging sectors.Governance is the main weakness: formal RIS3 governance bodies have operated irregularly, with limited interdepartmental coordination.Monitoring remains limited: budget, output and result indicators are analysed, but data availability is incomplete, and the monitoring system has not worked continuously.Stakeholder awareness is low: many firms and actors of the innovation ecosystem have limited knowledge of the RIS3, its instruments, and its monitoring mechanisms.Innovation performance remains weak: the region still shows low R&D intensity and structural weaknesses in business innovation, despite some progress.Stakeholders call for clearer sectoral priorities, more stable innovation instruments, and stronger support for emerging sectors beyond tourism.
The analysis is limited by the early stage of implementation of several RIS3 measures and by incomplete implementation data. The report also notes that many statistical sources are available only with a 2–3-years delay.
Strengthen RIS3 leadership and governance, ensuring more stable coordination across departments and regular involvement of actors of the ecosystem.Improve communication and visibility, so that firms and stakeholders can better understand the RIS3, its instruments, and available support.Make the strategy more focused and vertical, with clearer sectoral priorities linked to regional strengths and diversification potential.Reinforce monitoring and evaluation, simplifying indicators, improving data availability, and ensuring regular reporting.Support innovation and diversification more consistently, especially through stable SME instruments, clusters, interregional cooperation, and pilot innovation missions.
​​Assessment of the socio-economic development strategy “Warmińsko-Mazurskie 2030"
Ewaluacja on-going strategii rozwoju społeczno-gospodarczego Warmińsko-Mazurskie 2030
ROP Warmińsko-Mazurskie Voivodeship 2014-2020; Warmia i Mazury ERDF/ESF+ 2021-2027
​​Beata Belica, Przemysław Bruhn, Cezary Gołębiowski, Anna Górecka-Ojdana, Roksana Kołodziejczyk, Paulina Kowalczyk, Dorota Kretkiewicz, Adam Rybkowski, Paulina Sęk, Weronika Zych​
The report examines the implementation, continued relevance and update needs of the regional socio-economic development strategy Warmińsko-Mazurskie 2030.
To assess the regional development policy in the context of the socio-economic changes occurred at regional, national and EU level, and in view of the post-2027 EU programming period.
Desk research.Analysis of statistical and monitoring data.Content analysis of 17 media/information channels.46 interviews with regional and institutional stakeholders and socio-economic partners.Survey of 102 people responsible for local government strategic planning and implementation.Survey of 8 operational objective coordinators of the monitoring system.9 case studies.Benchmarking of 4 regional development strategies (Podlaskie, Świętokrzyskie, Podkarpackie and Lubelskie).Focus group.
The Strategy’s vision remains broadly relevant and realistic, but it does not sufficiently reflect the importance of areas such as security, resilience, disinformation, critical infrastructure, energy, food and climate security.The main objective is assessed as effectively implemented: GDP per capita reached 71% of the national average (above the 2030 target) and the transport accessibility indicator is on track towards the target.Social cohesion indicators show progress, including the 76.9% activity rate target already exceeded, but the region still lags behind in terms of social assistance dependency, wages relative to the EU average and business density.Spatial cohesion is improving through major transport investments such as Via Baltica, Rail Baltica and S16, while local road and rail accessibility remains a key constraint for peripheral areas.The territorial approach is not well applied, because measures for the Areas of Strategic Intervention (ASI) cover almost the whole region, instead of limited territorial areas.Monitoring system generally works well, though there are a few technical issues and one indicator is not still not populated.
Revise the vision’s narrative to include security and resilience.Integrate the silver-economy into the Strategy, covering older workers’ activation, longer working lives, care services, medical prevention and workforce planning for care and health support.Simplify the territorial approach by reducing the regional ASI and concentrating interventions on clearly justified national or functionally coherent territorial categories.Strengthen the monitoring system by removing non-monitorable indicators, simplifying the reporting form and instructions, and introducing regular training for entities providing data.
​​Mid-term evaluation of the Smart Specialisation Strategy under the Canarias ERDF programme, 2021-2027​
Evaluación intermedia de la Estrategia de Especialización Inteligente Ampliada de Canarias 2021-2027
The report is a mid-term evaluation of the RIS3 2021-2027. It covers the first implementation period, from 2021 to 2025. The core operational data on instruments and financial deployment cover 2022-2024, while data used go up to 2025.
To assess the degree of maturity of the RIS3 after its initial implementation phase. To examine whether the management structures, participation mechanisms and actions deployed are adequate to support the expected transformation objectives. To contextualise the strategy in relation to recent European and national innovation and industrial policy trends. To assess governance and implementation, review the monitoring system and formulate recommendations for 2026-2027.
Review of RIS3 documents, monitoring reports, Technical Office reports, regional competitiveness report and European and national policy documents.Analysis of statistical office data from INE (Spanish National Statistics Institute), ISTAC (Statistics Institute of Canarias), and Eurostat such as the Regional Innovation Scoreboard (RIS) 2025 and the Regional Competitiveness Index (RCI 2.0).Analysis of administrative data from regional and national administrative registries specifically utilising the SICTI (Science, Technology, and Innovation Information System) database and the BDNS (National Subsidies Database).Analysis of monitoring data and RIS3 indicators.Online survey addressed to implementing actors with stronger representation of SMEs and knowledge centres.Contrast workshop with regional authorities managing function 46, namely research, innovation and information society, and with the main public R&D&I centres in the region.
The Extended RIS3 remains broadly relevant and coherent with EU and national policy orientations on innovation, digitalisation, talent and green transition. However, there is a functional gap with newer European priorities on strategic autonomy, industrial resilience, critical value chains and dual-use technologies.The governance model has improved compared to 2014-2020. The Technical Office, RIS3 Communities and RIS3 Platform have helped institutionalise the Entrepreneurial Discovery Process. However, sectoral co-leadership, feedback to participants, interdepartmental coordination and technical capacity still need to be strengthened.During 2022-2024, 92 instruments were mobilised, with a total amount of around EUR 165.2 million. The largest allocation concerned specialisation, with 21 instruments and EUR 77.2 million, followed by talent, territory, knowledge and internationalisation.Regional innovation indicators show progress, as the region moved into the Moderate Innovator category in the Regional Innovation Scoreboard 2025 and improved its European ranking. Research staff reached 4,529 in 2024, exceeding the 2027 target, and the private-sector share of R&D expenditure reached 29.54%.Some structural weaknesses remain. R&D expenditure over GDP stood at 0.52%, below the 0.75% target, partly because of strong GDP growth linked to tourism. Patents, innovation intensity, knowledge transfer and private technology-intensive investment remain limited.Participatory evidence shows moderate alignment between RIS3 priorities and sectoral needs, and more than 85% of survey respondents considered that a revision of priorities was needed. The RIS3 Communities are valued as networking spaces, but the RIS3 Platform is less appreciated.The main barriers identified were administrative bureaucracy and the insufficient adaptation of public financial instruments to business needs, both mentioned by 62.5% of respondents. Long resolution and payment times were mentioned by 41.67%.
Questionnaire results should be interpreted with caution because participation registered only a moderate response rate and the sample was heterogeneous, with stronger representation of SMEs and knowledge centres. Some indicators have recurrent data gaps, limited strategic value or measurement problems. The indicator on R&D expenditure over GDP is affected by strong regional GDP growth and may therefore distort the interpretation of R&D efforts.
Update the operational focus of the RIS3 to reflect the new European agenda on strategic autonomy, industrial resilience, critical value chains, strategic technologies and dual-use technologies.Strengthen the governance model through a more active Executive Committee, a stable technical coordination body, stronger interdepartmental coordination and a clearer role for the Technical Office.Make RIS3 Communities more result-oriented and link them more directly to financing instruments, prioritisation criteria and technical support for tractor projects.Simplify administrative procedures, reduce resolution and payment times, and improve the annual planning of calls.Rebalance the instrument mix towards scaling-up, productive capacity, technology-intensive private investment, public procurement of innovation and venture capital.Simplify the monitoring scorecard and use indicators that are verifiable, regularly updated and strategically meaningful.
Evaluation of the impact of support for business R&D&I under the Multiregional ERDF OP and the ROPs, 2014-20, in Spain
Evaluación del impacto de las ayudas a la I+D+I sobre la innovación en las empresas industriales españolas
Multi-regional ERDF OP 2014-2020, All ROPs 2014-2020
Instituto de Estudios Fiscales (IEF), Ministerio de Hacienda, https://www.ief.es/
The report presents the final evaluation of the impact of measures co-funded by the ERDF and supporting R&D&I in industrial companies over the period 2014-2020.
To assess the results achieved by the support for R&D&I in businesses. To provide policymakers with evidence on similar future policies and to support the 2021-2027 programming of Cohesion policy.
Analysis of statistical office data.Analysis of Business Strategies Survey (ESEE), managed by the SEPI Foundation, a public entity attached to the Ministry of Finance, a longitudinal panel that since 1990 collects detailed information on a representative sample of companies in the Spanish manufacturing sector. A restricted panel of 2,507 innovative companies was built, of which 479 have received public aid for R&D&I in at least one year in the period 2014-2021.Counterfactual impact evaluation based on a “difference in differences” approach.
For companies already investing in R&D, the grants generated actual additional expenditure on R&D in manufacturing companies supported compared with a scenario without support (increase by between 15% and 20%). This positive effect persisted after completion of the project.The grants spurred product innovation in innovative manufacturing companies that had not previously innovated in this area and increased the volume of product innovation (average increase of 17%-19%), compared with a scenario without support. This positive effect persisted after completion of the project.The grants also spurred process innovation, increasing the likelihood of introducing new production, logistics or distribution methods by between 10 and 15 percentage points, compared with a scenario without support. However, this effect was mostly short-term.By contrast, the grants had no effect on the number of patents registered, though this could also be linked to companies’ low propensity to protect their innovations.Concerning strategic business behaviour, the grants encouraged companies to be more open to external expertise, and they increased collaboration between companies and universities and technology centres (increased probability by between 26 and 29 percentage points compared with a scenario without support). They also contributed to strengthening the internal management of innovation.Overall, the additional effects on expenditure, product innovation and strategic business behaviour were particularly significant in less developed regions, in SMEs and in low-tech sectors, which helped to reduce disparities in the innovation ecosystem.
Results are limited to manufacturing companies because ESEE only includes industrial companies. ESEE is not a specific R&D&I survey and therefore lacks certain relevant information. Despite a good geographical coverage, ESEE includes fewer companies from less developed regions making the analysis for these areas less precise.
Maintain ERDF support for R&D&I activities of manufacturing companies.Strengthen the capacity to attract and retain specialised R&D staff.Support process innovation in SMEs and less developed regions.Promote new market-oriented innovation and its protection.Reinforce strategic innovation behaviour in SMEs.
Impact evaluation of the IPYME programme on SME Indebtedness under the SME Initiative OP, 2014-2020, in Spain
Evaluación de impacto del programa IPYME sobre la capacidad de endeudamiento de las empresas
Instituto de Estudios Fiscales (IEF), Ministerio de Hacienda,https://www.ief.es/
The evaluation examines the impact of the IPYME programme on the level of indebtedness of SMEs in Spain between 2013 and 2019. The IPYME programme, launched in 2015 and co-financed by the ERDF, aims at improving access to credit for European SMEs.
To analyse the evolution of the indebtedness of SMEs in Spain, with a focus on the role played by the ERDF through the IPYME programme. To generate relevant evidence and recommendations to promote the effective use of this type of policy instruments.
IPYME monitoring database of the European Investment Fund (EIF) 2014-2019.Data from the Tax Agency (AEAT) 2013-2019.Data from the Central Balance Sheet of the Bank of Spain.Contribution analysis of companies’ borrowing capacityCounterfactual impact evaluation based on the difference in differences approach
Companies involved in IPYME had debt ratios significantly higher than the national average, which suggests that the programme has succeeded in attracting companies that had severe difficulties accessing credit through standard financial channels.The IPYME programme has had positive and statistically significant effects on both short- and long-term debt levels. On average, receiving an IPYME loan implies an additional increase of €171,000 in short-term debt and of €156,000 in long-term debt, compared to a control group.Given that the average loan amount of IPYME is €65,000, the programme has led to loans to the private sector exceeding the amount initially anticipated, but also to companies being able to gradually increase their funding, above the levels achieved by similar non-beneficiary companies.The increase in debt attributable to IPYME has occurred without any significant changes in the debt-to-equity ratio. This means that the financial structure of the companies remained stable, which helps to mitigate the risk associated with higher levels of debt.
There are no limitations indicated in the report.
Define similar programmes to increase access to credit for SMEs, as long as this remains a public policy priority.Perform a more in-depth impact analysis on economic growth and employment.Use the available databases to improve the selection of beneficiaries based on company profiles.
Evaluation of the measure residential care voucher supported under the Programme Piemonte ESF+ 2021-2027
Buono Residenzialità: sostegno all’inserimento di persone non autosufficienti in strutture residenziali a carattere socio-sanitario o socio-assistenziale
Thematic evaluation on measures supporting the digitalisation of enterprises supported under the Programme Emilia-Romagna 2021-2027
Rapporto di valutazione tematica sugli interventi a sostegno della digitalizzazione delle imprese
Lattanzio KIBS S.p.A.; Centrale Valutativa s.r.l.
https://www.lattanziokibs.com/; https://www.centralevalutativa.it/
Evaluation of interventions on defense and security industry supported under the EU Funds' Investments Programme 2021-2027 in Lithuania
Ekonomikos ir inovacijų ministerijos valdymo sričių gynybos ir saugumo pramonės vertinimas
Jonas Jatkauskas, Inga Bartkevičiūtė, Monika Venčkauskaitė, Marijonas Mikaitis, Vytautas Šakalis / JSC „BGI Consulting“, The Baltic Institute of Advanced Technology
JSC „BGI Consulting“, 7 Aukštaičių Str., LT-11341 Vilnius, Lithuania, jonas@bgiconsulting.lt, www.bgiconsulting.lt - The Baltic Institute of Advanced Technology, info@bpti.lt, www.bpti.eu
Mid-term evaluation of the Programme Veneto ERDF 2021-2027
Valutazione del PR Veneto FESR 2021-2027 per riesame intermedio
Autorità di Gestione del Programma Regionale Veneto FESR 2021-2027- Direzione Programmazione Unitaria, in coordinamento con l'Area Politiche Economiche, Capitale umano e Programmazione Comunitaria.
Evaluation of measure supporting and training self-employment (business creation) under the OP Wallonie-Bruxelles ESF 2014-2020
Évaluation de la plus-value du Fonds social européen dans l'accompagnement et la formation à l'autocréation d'emploi (notamment par la création d'entreprise)
OP Wallonie-Bruxelles 2020.eu ESF 2014-2020
Joël Leroy, Gaëlle Hubert, Alexia Peyser
Evaluation of the Communication Strategy of the Programme Łódzkie 2021–2027
Ewaluacja Strategii komunikacji Programu Regionalnego Fundusze Europejskie dla Łódzkiego 2021-2027 oraz wybranych działań przewidzianych w Rocznym planie działań informacyjnych i promocyjnych na rok 2025 w ramach FEŁ2027
Sebastian Pałka, Adam Grajek, Karolina Drozdowicz, Aleksandra Kamińska, Michał Marciniak, Adam Mielczarek, Wojciech Pieniążek, Agnieszka Śnieżek, Marta Cichowicz-Major, Beata Dec
Evaluation of the measure 'Practice and School (PUSCH)' supported by the Programme Hessen 2021-2027
Europäischer Sozialfonds Plus in Hessen 2021 bis 2027. Praxis und Schule (PUSCH). Zwischenbericht der Programmevaluation.
ISG Institut für Sozialforschung und Gesellschaftspolitik GmbH, www.isg-institut.de
Survey of beneficiaries who completed training supoprted under the OP Wallonie-Bruxelles 2014-2020
Usages et effets de la participation des salariés à la formation continue Enquête exploratoire auprès de sortants de formations financées par le FSE en Wallonie et à Bruxelles
Bernard Conter, Muriel Fonder, Baptiste Féraud
Evaluation of interventions on technical assistance and REACT-EU under the Łódzkie OP, 2014–2020
Ocena wsparcia w ramach OP XII „Pomoc techniczna” i OP XIV „Pomoc techniczna REACT-EU” RPO Województwa Łódzkiego 2014-2020
Jolanta Kluba, Weronika Zych, Anna Górecka-Ojdana, Roksana Kołodziejczyk, Dorota Kretkiewicz
ECORYS Polska Sp. z o.o. , www.ecorys.pl
Evaluation of the implementation of the Wallonie-Bruxelles ESF OP 2014-2020
Rapport de synthèse des évaluations de la PROGRAMMATION FSE 2014-2020 WALLONIE-BRUXELLES 2020.EU
Evaluation of the meaure 'Fachkurse' supported by the Baden-Württemberg ESF+ 2021-2027
Evaluationsbericht zum ESF-Plus-Förderprogramm "Fachkurse"
Eva Roth und Anne-Marie Scholz (ISG GmbH)
Ulrike Arzbach (Referatsleiterin), ESF@sm.bwl.de
Evaluation of the Implementation of the Ipeiros Programme 2021–2027
1η Αξιολόγηση εφαρμογής του Προγράμματος Ήπειρος 2021-2027
Address: Athens, 54 Egialias Str., Marousi 151 25Phone: +30 210 0084000Email: remaco@remaco.grWebsite: https://remaco.gr/
Impact evaluation of the Vlaanderen ERDF OP 2014-2020
Impactevaluatie van het EFRO 14-20 programma voor Vlaanderen
First evaluation of the communication and publicity strategy of the Just Development Transition Programme 2021-2027
Παραδοτέο 2: Συνολική αποτίμηση, συμπεράσματα και προτάσεις βελτίωσης της Στρατηγικής Επικοινωνίας και Προβολής του ΠΔΑΜ
Just Development Transition EL JTF 2021-2027
Skoufa 75, Athina 106 80, Attiki, info@headway.gr, https://www.headway.gr/
Evaluation of the measure 'Competence Checks for Enterprises' under the recovery plan ‘Flemish Resilience' supported by the OP Vlaanderen ERDF 2014-2020
Evaluatiestudie van de werking ‘Competentiechecks voor ondernemingen’ Eindverslag
Prof. Dr. Simon Beausaert, Dr. Samantha Crans, Dr. Fatemeh Janesarvatan, Universiteit Maastricht and Prof. Dr. Eva Kyndt, Swinburne University of technology
Impact evaluation of the Kriti OP 2014–2020
Αξιολόγηση Επιπτώσεων του Επιχειρησιακού Προγράμματος Κρήτη 2014-2020
info@anelixisc.gr, https://anelixisc.gr/
Evaluation of the implementation of PO4 interventions under the Programme Central Region Romania 2021-2027
Evaluarea implementării intervențiilor OP 4 ]n educatie si turism sustenabil in Regiunea Centru
Civitta Strategy & Consulting office@civitta.com Lideea Development Actions srl office@lideea.eu
Evaluation of actions in the context of skills validation and recognition of prior learning supported under the OP Wallonie-Bruxelles ESF 2014-2020
Évaluation des actions Fonds social européen dans le cadre de la validation des compétences et de la valorisation des acquis de l’expérience
Evaluation on the calculation of long-term result indicators under the Programme Social Development ESF+ 2021-2027 in Poland
Badanie ewaluacyjne dotyczące wyliczenia wartości wskaźników rezultatu długoterminowego w ramach Europejskiego Funduszu Społecznego Plus
Piotr Fuchs Smart Research IDEA Instytut Sp. z o.o.
Mid-term evaluation of the Programme Brussels Capital Region ESF+ 2021-2027
Évaluation du Fonds Social Européen Plus (FSE+) Examen à mi-parcours
Ongoing evaluation of the Programme Brandenburg ERDF/JTF 2021–2027
Begleitende Evaluierung des Programms des Landes Brandenburg für den Europäischen Fonds für regionale Entwicklung 2021 bis 2027.
Dr. Jonathan Eberle, Prognos AG; Dr. Eva Dantas, Prognos AG; Nikolaus Bajor, Prognos AG; Prof. Dr. Dirk Czarnitzki, KU Leuven
Impact evaluation of the measure 'Outreach & Activation' supported under the OP Vlaanderen ESF 2014-2020
Impactevaluatie ESF-oproep Outreach & Activering. Resultaten arbeidsmarktsituatie
Evaluation of effects on productivity and employment of R&D incentives in projects with automation and digitalisation potential supported under the ROP Toscana 2014-2020
Gli effetti su produttività e occupazione degli incentivi alla R&S nei progetti con potenziale di automazione e digitalizzazione
Evaluation of the implementation of the Regional Health Programmes under the Łodzkie OP, 2014–2020
Ocena rezultatów i produktów wdrażania Regionalnych Programów Zdrowotnych w Regionalnym Programie Województwa Łódzkiego 2014-2020 - raport końcowy z badania
Malwina Malczewska, Ewelina Dąbrowska–Nowak, Kamila Stefaniak, Aneta Gierowska, Wiktoria Świat, dr Maria Kotkiewicz, Justyna Romanowska–Zapart, dr Joanna Syrda, Marek Skrzyński, Olga Korczyńska, Małgorzata Walczak–Gomuła
Evaluation of interventions on SME competitiveness under the PR North-West Romania ERDF 2021-2027
Evaluarea Competitivității IMM-urilor PR Nord-Vest
​​Intermediate evaluation of the Smart Specialisation Strategy of La Rioja ERDF programme, 2021 2027​
Evaluación intermedia de la vigente Estrategia de Especialización Inteligente (S3) de La Rioja 21-27
​​Evaluation of career guidance measures targeting young people, under the Piemonte ESF+ Programme, 2021-2027​
Le azioni di orientamento: il rapporto 2024
​​Evaluation of the implementation of the Madeira ERDF/ESF+ Programme, 2021-2027​
Avaliação de Fundos Europeus: Operacionalização do Madeira 2030
​​António Sampaio Ramos, Tatiana Alves, Ana Margarida Lopes, Alexandre Almeida, Américo Veloso Bento, Dulce Santana, Jorge Barbacena, Teresa Evaristo, Paulo Pedroso​
​​Evaluation of measures for reducing CO2 emissions under the Partnership Agreement, 2021-2027, in Romania​
Evaluarea OP2 Acordul de Parteneriat 2021-2027
Sustainable development RO CF/ERDF 2021-2027; Just Transition RO JTF 2021-2027; North-East Romania ERDF 2021-2027; South-East Romania ERDF 2021-2027; South Muntenia ERDF 2021-2027; Central Region Romania ERDF 2021-2027; South-West Oltenia ERDF 2021-2027;
Cristina Neacșu, François Levarlet, Rebeca Nistor, Tiberiu Nițu, Roxana Diaconu, Cristina Gligor
Evaluation of the implementation of the Interreg VI-A Germany/Brandenburg-Poland, 2021–2027
Evaluierungen des Kooperationsprogramms INTERREG VI A Brandenburg - Polen 2021-2027 im Rahmen der Europäischen Territorialen Zusammenarbeit (ETZ). Abschlussbericht – Los 1.
Interreg VI-A Germany/Brandenburg-Poland 2021-2027
Nicolas Niedrig, Beata Belica, Przemysław Bruhn, Cezary Gołębiowski, Roksana Kołodziejczyk, Piotr Popik, Weronika Zych
Spilett New Technologies, www.spilett.de; ECORYS Polska, www.ecorys.com
Evaluation of the implementation of cross-cutting principles under the Saarland ESF+ Programme, 2021–2027
Evaluationsbericht 2025. Sozialwissenschaftliche Begleitung des Programms des Saarlandes für den Europäischen Sozialfonds Plus (ESF+) in der Förderperiode 2021-2027
Christian Lenhart, Vanessa Kubek, Jenniver Asmussen
Ism (Institut für Sozialpädagogische Forschung Mainz e.V.), https://ism-mainz.de/
Evaluation of the Small Project Fund (SPF) under the Interreg VI-A Germany/Brandenburg-Poland, 2021-2027
Evaluierungen des Kooperationsprogramms INTERREG VI A Brandenburg - Polen 2021-2027 im Rahmen der Europäischen Territorialen Zusammenarbeit (ETZ). Abschlussbericht – Los 2.
Evaluation of the implementation of horizontal principles in transport projects under the European Funds for Infrastructure, Climate and Environment Programme, 2021–2027
Ocena wdrażania zasad horyzontalnych zapobiegania dyskryminacji i promowania równości płci w przedsięwzięciach podejmowanych w ramach Priorytetów transportowych (nr III, IV I V) FEnIKS 2021-2027
Infrastructure, Climate, Environment PL CF/ERDF 2021-2027
Centrum Unijnych Projektów Transportowych, www.cupt.gov.pl
Evaluation of the PIUS-Invest (Production-integrated environmental protection funding system) programme under the Hessen ERDF Programme, 2021–2027
Evaluation des Förderprogramms PIUS-Invest. Begleitende Evaluierung des Programms des Landes Hessen für den Europäischen Fonds für regionale Entwicklung (EFRE) im Förderzeitraum 2021 bis 2027. Endbericht 2025.
GEFRA, www.gefra-muenster.de; Joanneum Research, www.joanneum.at; Kovalis, www.kovalis.de
Evaluation of the measure 'Prison-to-Work' supported under the Piemonte ESF+ Programme, 2021-2027
Sportello Lavoro Carcere: analisi di attuazione
Gianfranco Pomatto, Niccolò Aimo, Luigi Nava
Analysis of critical issues relating to the implementation of the Programmes, in view of the post 2027 period, in Czech Republic
Evaluation of REACT-EU measures under Priority Axis 4 of the Hamburg ERDF OP, 2014-2020. Final Report
Endbericht. Prioritätsachse 4 – REACT-EU
GEFRA - Gesellschaft für Finanz und Regionalanalysen, IFS - Institut für Stadtforschung und Strukturpolitik, Kovalis
https://www.gefra-muenster.de, https://www.gefra-muenster.de/, https://kovalis.de/
The evaluation examines the effects of REACT-EU measures for research and innovation, healthcare, digitalisation and energy efficiency in companies. It covers the implementation period between 2021 and 2024.
To assess the effectiveness, efficiency and impact of the support for innovation, healthcare, digitalisation and energy efficiency, cofinanced by REACT-EU funding.
Analysis of monitoring data.Document analysis.4 case studies.Interviews with project managers.Expert interviews.
All planned projects had been successfully implemented and completed.Projects in the medical sector led to significant progress in drug development and personalised medicine. They resulted in a 30% reduction in screening duration and implementation of ten research projects and strengthened company’s ability to innovate.In the area of digitalisation, the 'FabLabs' measure created a basis for sustainable production, knowledge exchange and the further development of the Fab City approach in Hamburg. The digital infrastructure combines global collaboration with local added value, laying the foundations for a circular economy and decentralised production.The construction of the storage crane systems represents a technological innovation in container handling. The project will lead to energy efficiency gains (diesel consumption will be reduced by 77%) and CO2 savings (9,980 tonnes per year), corresponding to an increase in energy efficiency of 65%. In addition, further pollutant and noise reductions will be achieved.
Evaluation of the objectives and results of the Islas Baleares ERDF OP, 2014-2020
Informe de Evaluación de los objetivos/resultados del Programa Operativo FEDER 2014-2020 de Balears 2019
The evaluation examined the implementation progress of the Islas Baleares ERDF OP, 2014-2020.
To assess the progress of the OP and the achievement of performance framework targets, its impact as well as its contribution to the Europe 2020 strategy, horizontal aspects and climate change.
Monitoring data;desk analysis;in-depth interviews;online surveys;input-output econometric analysis.
PA2 (ICT), PA4 (Low carbon economy), PA6 (Protection of the environment) and PA10 (Education) have achieved the targets.PA1 (R&D) has not achieved the targets, mainly because of a lack of companies in the sector and implementation issues.PA3 (SME competitiveness) has not achieved the targets, because of delays in the activation of financial instruments.Overall, result indicators show a medium-to-high level of progress.The ERDF funding is estimated to increase GDP by EUR 102 million and create 884 new jobs. Long-term GDP growth is estimated to be 0.3% higher than without the ERDF.The OP is generally progressing well towards the Europe 2020 targets, though R&D expenditure is still low.
Shift funding from PA1 and PA3 to PA4 and from PA6 to PA10, improving the execution of PA1 through implementation of strategic projects.Increase MB’s awareness of the need for means to ensure gender equality.
Evaluation of systemic and conceptual projects of the Research, Development and Education OP, 2014-2020, in the Czech Republic - 3rd interim report
Evaluace systémových a koncepčních projektů z výzev PO 3 OP VVV - Evaluační okruh C, 3. Průběžná zpráva
OP Research, Development and Education 2014-2020
The evaluation examines the implementation of the “Inclusive and High-Quality Education in Territories with Socially Excluded Localities” (IHQE) project, funded under the Research, Development and Education OP 2014-2020.
To assess whether the principles of inclusive and high-quality education based on participation are applied in the municipalities supported under the "Inclusive and High-Quality Education in Territories with Socially Excluded Localities" project and whether conditions for sustainability are being created.
CAWI interviews with stakeholders.In-depth interviews with municipalities involved.Desk analysis.
Overall, the activities are progressing well, though the application process is complex and there is lack of funding, overburdening of schools and limited interest from target groups.Cooperation at local level is working well.Most of stakeholders believe that there is progress towards social inclusion, but some consider the duration of measures to be too short to produce significant effects.In some cases, there are coordination problems with other complementary projects on similar issues.
Assessment of the Sectoral Councils for Competences under the Poland ESF OP
Ocena funkcjonowania Sektorowych Rad ds. Kompetencji
Agnieszka Śnieżek, Zofia Bieńkowska, Paweł Penszko, Wojciech Pieniążek, Małgorzata Zub
Evalu Sp. z o.o. (former Agrotec Polska Sp. z o.o.), ul. Dzika 19/23 lok. 55, 00-172 Warszawa
The evaluation assesses the performance of the Sectoral Councils for Competences, established by the Polish Agency for Enterprise Development to counter the shortage of skilled workers under the ESF OP.
To assess the performance of the Sectoral Councils for Competences and to make suggestions for change to the selection system used.
Identification of the programme theory;case studies;interviews;analysis of indicators.
All Councils have implemented their plans and their personnel are working well;members were often not provided with full information about the activities of Council board meetings;the composition of the Council ensures fair representation of the different sectors;in a number of cases, key stakeholders were not interested in joining the Councils;a number of examples of good practice have been identified.
There is a need to:include information in Council communication activities on the Council objectives and the benefits;introduce a permanent system of collecting feedback;develop more opportunities for promoting Council activities.
Evaluation of the ‘Sekundarstufe I’ Initiative supported by the Brandenburg ESF OP 2014-2020
Evaluation der Umsetzung schulischer und außerschulischer Maßnahmen zur Entwicklung von Berufswahlkompetenz in der SEK I
The evaluation examines the INISEK I initiative supporting a large number of projects combining vocational and study orientation activities in Brandenburg schools.
To assess the implementation and effects of projects on vocational and study orientation with a particular focus on the contribution of ESF support to the development of career choice skills.
Desk analysis.Analysis of monitoring data.Three rounds of online surveys at 46 selected schools in school years 20017-18, 2018-19 and 2020-21 of: students (all students in the first two rounds and only those of 9th and 10th grade in the third round) using the Thüringen vocational orientation model (ThüBOM): response rate of 77% out of 12,639 students (1st round), 67% out of 12,669 (2nd round) and 68% out of 6,600 (3rd round)teachers (response rate 55% out of 1,594 (1st round), 41% out of 1,653 (2nd round) and 43% out of 1,616 3rd round)school coordinators: 96% (1st round), 87% (2nd round) and 72% (3rd round) out of a total of 46 for each round.Longitudinal analysis of survey data based on the deterministic linkage method, a multivariate and univariate analysis of variance.10 case studies (5 in each of the school years 2018-19 and 2020-21).
As of September 2020, 80% of the funding of EUR 30.4 million for the initiative was spent and targets set were largely met or exceeded.Around 80% of schools in Brandenburg implemented at least one project.The management structure was based on coordinators in each school and cooperation with companies and the Federal Employment Agency.Although an effort was made to standardise the structure in schools, the concept of vocational and career orientation still varied greatly and procedures for quality assurance and evaluating activities were often informal.40% of students had an idea of their further career in each of the three interviews, 40% were uncertain until they reached the 10th grade and 20% had not made a choice by then, half of whom made no choice throughout the survey.Activities that were particularly successful in helping students to make career choices were: high-quality input from teachers, exchange with parents and practical training in companies.Teachers considered vocational and study orientation an important topic for students but the way this was concretely applied varies across the different types of school, though most of them had the know-how to support students in their career selection.
Due to the COVID pandemic, the third round of survey originally planned between March and May 2020 was undertaken at the beginning of the 2020-21 school year. In the second and third rounds, 44 schools (rather than 46) participated because of technical problems and quarantine regulations.
Maintain the package of measures and further develop activities.Strengthen awareness of the use of vocational and study orientation.Raise awareness among teachers on the role of career and study orientation and support them.Support the role of parents further.Focus on students with special support needs.
Evaluation of the regional OPs, 2014-2020 – regional analysis and national synthesis
Temautvärdering Regionalfonden – en regional analys och nationell syntes
Linköpings universitet (University of Linköping)
The evaluation examines the 8 ERDF OP in Sweden in 2014-2020, which focus mainly on TO1 - strengthening research, technological development and innovation; TO2 - enhancing access to, and use and quality of, ICT; TO3 - enhancing the competitiveness of SMEs and TO4 - supporting the shift towards a low-carbon economy .
To assess the contribution of OP to smart, sustainable and inclusive growth in the 8 regions.
Desk research of programme documentation and literature review;in-depth interviews of key stakeholders;analysis of monitoring data;focus groups.
TO1 - projects are in line with the specific objectives, but these are not entirely clear and relevant. Project and company selection seems to be driven by their implementation capacity, which might result in certain sectors being underrepresented. Sustainability aspects of smart specialisation are not prominent role in the OPs.TO2 - broadband access is deemed important for businesses but it does not seem to have a significant impact on turnover and employment. Broadband investment appears to play an important role in creating a sense of inclusion in the development of society.TO3 - projects are in line with the specific objectives, but these are vaguely worded and incapable of guiding programme’s content, while projects are not sufficiently focused on the needs of business. Result indicators are inadequate as they are more based on increases in volume (e.g. increased turnover) rather than on increased competitiveness. The potential of programmes to have long-term effects on the competitiveness of SMEs is largely neglected.TO4 - programme logic is relevant and coherent and projects provide added value. Demand for funding has been low and projects do not seem to be linked with other energy instruments or national strategies. Projects seem to have incremental but not structural effects.There are no major systematic differences in implementation between regions. But there are differences in the time needed to initiate work and to achieve indicator targets, in the success rate in making contact with businesses and getting them involved and in adherence to the programme when selecting businesses.
For some TOs, assessment of results is uncertain as it is made at an early stage.
There is need:for the central government to take a more active role in steering and coordinating policies in close dialogue with regional development managers;for more clarity in the link between indicators and results and between results and long-term effects;for developing an experimental programme policy, involving new stakeholders and new innovative types of intervention while keeping a focus on results and long-term effects;for providing different support schemes aimed at different regional actors and project stakeholders. Support should be aimed at project development, improving sustainability, strengthening learning at all levels, increasing dissemination and having a strategic impact.
Evaluation of measures supporting the digital transformation of businesses financed, among other sources, by the Smart Growth OP, 2014-2020, the European Funds for Eastern Poland Programme, 2021-2027, and the European Funds for a Smart Economy Programme, 2021-2027, in Poland
Wsparcie transformacji cyfrowej przedsiębiorstw – wnioski na przyszłość
OP Smart Growth 2014-2020; Smart Economy PL ERDF 2021-2027; Eastern Poland ERDF 2021-2027
Bartosz Ledzion, Tomasz Klimczak, Adam Miller, Alicja Wolanin, Igor Lyubashenko
EGO – Evaluation for Government Organizations S.C., www.evaluation.pl; LBiE, www.lbie.eu; Fundeko, www.fundeko.pl
The evaluation examines the impact of digital transformation projects implemented by SMEs and supported by three EU programmes, against the background of the main technological trends in the field.
To assess the impact of measures to support the digital transformation of enterprises and draw suggestions for interventions in the period 2021-2027.
Desk analysis of relevant documents.Analysis of monitoring data and official statistics.4 interviews with implementing bodies.25 interviews with beneficiaries.CAWI survey of 402 beneficiaries.TBE approach.Workshop.
Half of the SMEs supported had achieved an impact in terms of strengthening the company's competitive position and a further 33% expected this to happen.For 37% of SMEs, the aid received has had a positive effect in terms of increased profitability and 42% expect this to happen.For 89% of SMEs, the project had brought benefits to their employees.
Launch three new instruments to support SMEs: creation of a network of AI centres, support for SMEs to adapt to new cybersecurity requirements and strengthening of digital skills in SMEs.Launch a call for proposals for start-ups to collaborate with large companies in the fields of communications and semiconductor technologies, edge computing and quantum computing.Promote the services offered by innovation centres to companies applying for the European Funds 2021-2027.
Evaluation of the “Yes I start up” project, on training for start-up businesses, under the Youth Employment OP, 2014-2020, in Italy
Silvia Ciampi, Cristina Lion, Vanessa Lupo, Katia Santomieri
The evaluation examines the implementation of measures supporting self-employment of young NEETs up to September 2020.
To assess the ability of the project to reach the target groups, the extent to which new businesses have been created thanks to the project up to September 2020 and their sustainability.
Monitoring data.Official statistics.Interviews with implementing bodies.Administrative data.4 case studies.
From June 2018 to September 2020, 333 training courses on self-employment were provided, involving 1,508 NEETs, mainly men and those with secondary education.The drop-out rate was 11%.586 participants in the trainings (39%) subsequently applied for funding to start a business, leading to the creation of 135 new firms (23% of applications for funding) mainly by men and those with low education.Up to September 2020, the survival rate of companies created was 97%, higher than for new companies that applied for funding and did not received it (83%).The training provided was important for the general empowerment of NEETs, even when they made no application for funding to start a business.The link between implementing bodies and the PES is weak and the timeframe for projects too tight.The administrative burden is heavy and limits participation of NEETs.
Strengthen post-training support through ad-hoc tutoring focussed on starting a business.Create networks of young entrepreneurs who participated in the project to develop peer learning opportunities.Plan further incentives to link new business ideas more to local needs and development plans.
Evaluation of Funding Action 9.1 “Training of Unemployed” under the Bayern ESF OP, 2014-2020 – 2nd Interim Report
Evaluation der Förderaktion 9.1 "Qualifizierung von Arbeitslosen" 2. Zwischenbericht
Dr. Jenny Bennett, Thorben Frie, Vera Maier, Katja Seidel, Hans Verbeek
Institut für Sozialforschung und Gesellschaftspolitik GmbH, Weinsbergstraße, 190 50825 Köln
The evaluation is the Second interim report examining training and social support measures to improve the employability of the unemployed and to support their integration into the labour market under the Bayern ESF OP.
To assess aspects of projects (characteristics of the target group, regional patterns of implementation, professional profiles, cooperation with enterprises for traineeships) and the effects on employability.
Analysis of monitoring data, programme documents, socio-economic statistical data and research studies on employability.Two interviews with those involved in programme planning and implementation.Survey of project partners (142; response rate 73%) and of traineeship enterprises (292; response rate 17%).9 interviews with traineeship enterprises.Three cases studies of projects.A counterfactual impact evaluation (the control group created from Federal Labour Office data).
The number of participants and projects has declined significantly over the programming period because of falling unemployment.68% of the output targets have been achieved which is on schedule.Most participants have multiple placement obstacles.Only half of participants remained to the end of the programme.Finding traineeships is difficult and the average duration is shorter than required.56% of the participants obtained a qualification (target 60%).Traineeships increase the employment opportunities of participants.Most of the project partners and traineeship enterprises report that further measures are necessary to integrate participants into the labour market.26% of participants were in employment immediately after participation and 36% 26 months after.The most important factors for successful integration into the labour market are intensive social-educational support, the practical relevance of training, and a positive group dynamic.The probability of integration into the labour market is improved by the measures but participants still need support from the welfare system, which indicates that earnings from employment are not enough to live on.
The results of the survey of traineeship enterprises are not representative. The control group in the counterfactual analysis may have participated in other funding measures.
Ensure projects have a very specific target group orientation.Lower entry requirements to admit more unemployed from abroad.Improve the regional network structure and cooperation between project partners and regional actors.Provide additional social educational support to relieve instructors during training.Organise projects in a way that makes dropping out less harmful.Examine the selection criteria for projects because of large differences in successful completions.Involve the Chamber of Industry and Commerce in identifying traineeship enterprises.
Evaluation of the effects on participants of projects under the England ESF OP, 2014-2020 – Survey Report 2016-2019
European Social Fund and Youth employment Initiative Leavers Survey Report 2016-2019
Lorna Adams, Mark Winterbotham, Andrew Skone James, Christabel Downing, Hamish Evans
IFF Research, https://www.iffresearch.com
The report examines the outcomes of work-related training programmes for participants 6 months after leaving, for 2016-2019.
To assess the effectiveness of ESF and YEI measures on the situation of participants in the period 2016-2019, six months after leaving.
Monitoring data from the Department for Work and Pensions (DWP).CATI survey of 19,769 participants (23% response rate).
Many participants in the ESF programmes reported improvements in soft-skills and increased motivation to find employment.54% of participants were employed after 6 months (32% before participation) and 45% of YEI participants (1% before). In the 6 months afterwards, half of YEI leavers were offered a job.Almost all those obtaining in-work support were still in employment after 6 months and almost all reported being more optimistic for the future.The number of participants claiming benefits fell by a quarter 6 months after entering the projects.A third of participants unemployed on entry became inactive 6 months after leaving, either going into education or training or being unable to work because of health issues.
The research cannot reliably say that the observed improvements can be attributed to ESF programmes.
Evaluation of the effects of revitalisation measures under the Świetokrzyskie OP, 2014-2020
Wpływ działań rewitalizacyjnych z perspektywy 2014-2020 na sytuację obszarów zdegradowanych w województwie świętokrzyskim
ROP Świętokrzyskie Voivodeship 2014-2020
Przemysław Bruhn, dr Natalia Dziarmakowska, Anna Górecka-Ojdana, dr Aleksandra Jadach-Sepioło, Katarzyna Lipczyńska, Adam Rybkowski, Paulina Sęk
ECORYS Polska Sp. z o.o., www.ecorys.pl;Instytut Rozwoju Miast i Regionów, www.irmir.pl
The evaluation sets out to examine the impact of revitalisation activities on the socio-economic situation of degraded areas in the region.
To assess the effect of measures on the socio-economic, physical and spatial development of urban areas, including old industrial areas, and their complementarity with those in the previous period and with other funds.
Literature review and analysis of statistical data.Interviews with programming and implementation bodies, beneficiaries and experts.Survey of 37 beneficiaries and 370 residents.3 case studies.Evaluation workshop.
Many municipalities have had difficulties in engaging the local community in revitalisation projects. Participation primarily occurred during the programming stage, but was limited subsequently. Municipalities struggled to designate degraded areas for revitalisation because of a lack of experience and data.The project selection criteria did not pose significant challenges for applicants, except for difficulties in finding partners and demonstrating job creation.Revitalisation projects faced unexpected challenges from external factors (the pandemic and increased prices of materials).
Encourage greater involvement of different stakeholder groups in revitalisation.Provide assistance and training to municipalities for conducting intra-urban analysis, selecting indicators, harmonising data, and designating degraded areas.Continue to prioritise comprehensive revitalisation activities, project partnerships and a two-stage project selection procedure to ensure successful implementation.Simplify criteria relating to partner involvement and the effect on job creation.Give preference to revitalisation projects in strategic intervention areas, particularly medium-sized cities in decline and strengthen the connection between projects and social activities.
Preparatory study on financial instruments for self-employment financed by the ESF+ for 2021-2027 in Poland
Ocena ex ante instrumentów finansowych w obszarze samozatrudnienia ze środków Europejskiego Funduszu Społecznego
Social Development PL ESF+ 2021-2027; All ROPs 2021-2027
Jan Szczucki, Maciej Gajewski, Henryk Kalinowski, Robert Kubajek, Anna Matejczuk-Rosa, Magdalena Ośka, Bogdan Pietrzak, Justyna Witkowska
Policy & Action Group Uniconsult Sp. z o.o., ul. Kierbedzia 4, 00-728 Warszawa, www.pag-uniconsult.pl; Imapp Consulting Sp. z o.o., ul. Puławska 145, 02-715 Warszawa, www. imapp.consulting.eu
The study identifies the financial needs of various categories of people for self-employment.
To assess the need for financial and non-financial support for self-employment in respect of various groups in particular labour market situations.
Desk research.41 interviews with implementing institutions, final recipients, financial and labour market institutions.2 case studies.CATI survey among self-employed (n=1,001).CAWI survey among final recipients (n=2,734).Creative workshop.
In most cases, subsidies go to people who have left their jobs to start their own business.The sustainability of companies supported is relatively high. After two years, 84% of companies are still active.The survival rate of enterprises is strongly affected by socio-demographic factors: enterprises established by men, 30-54 years old and highly educated people stay in business longer.Half of the companies would not have been set up without grants or loans, and 40% of them would have had to be delayed in the absence of financial assistance.The average demand for external funds was relatively small – on average, PLN 32,000, and only 5% of entrepreneurs needed funding above PLN 100,000.
Prepare and implement portfolio loan guarantee scheme for business start-ups.Provide pre-employment counselling to verify the feasibility of business ideas.Provide post start-up support, particularly individual counselling and mentoring.Reduce subsidies for people in a better situation on the labour market and direct them towards repayable support.
Evaluation of support for internationalisation under the Puglia ERDF OP, 2007-2013
Valutazione ex-post relativa alla programmazione regionale 2007- 2013 in materia di internazionalizzazione
The evaluation examines two measures (6.3.2 on support for events, exhibitions and other systemic activities and 6.3.3 on support for increasing presence in international markets) financed by the OP.
To analyse the effects of Measure 6.3.2 to support events, exhibitions and other systemic activities and Measure 6.3.3 to support increased presence in international markets and to identify the factors affecting the results.
Official statistics.7 interviews with those responsible for implementation. Mini case studies (in-depth interviews with enterprises).Survey of enterprises.Counterfactual analysis using administrative balance sheet and survey data.
The measures were in line with the needs of enterprises, though in some cases other measures (vouchers) seem to have been preferred.For Measure 6.3.3, critical issues were the obligation to set up a network of firms and to have a temporary project manager with strong experience in internationalisation.For Measure 6.3.2, administrative procedures were too complex.Firms supported under Measure 6.3.2 participated more in international exhibitions than others.For both measures, there is no evidence of firms supported being more innovative than others.The counterfactual analysis shows that the propensity to export was higher in companies supported than in the control group (by 11-13 pp) 10 years after the measures were initiated.
The measures were implemented several years before the evaluation, so it was not always possible to interview the relevant people. In addition, COVID-19 limited contact with enterprises and a second survey of them could not be carried out.
Evaluation of measures supporting informal carers under the Employment+ Programme, 2021-27, in the Czech Republic
Analýza projektů OPZ+ zaměřených na podporu neformálních pečujících: závěrečná, průběžná a vstupní zpráva
The evaluation analyses 26 projects supporting informal carers funded under the OP. Data used go up to October 2024.
To assess the effectiveness of the measures supporting informal carers and to identify the challenges encountered.
Analysis of documentation.Analysis of monitoring data.26 semi-structured interviews with all the beneficiaries.13 Focus groups with relevant stakeholders.
The support provided to informal carers is still very heterogenous in terms of type of support and target group.About one-fifth of the projects introduced an innovative supporting approach, while the rest primarily built upon past experiences.Most respondents were satisfied with the measures. Some were dissatisfied with restrictions on funding for medical procedures and therapy, while others raised issues related to administrative processes.Positive outcomes emerged in terms of increased accessibility of support services.The availability of flexible and part-time work is essential for carers.Home sharing is a promising model for supporting informal carers, but its sustainability depends on effective recruitment of the hosts, stable funding, and policy recognition.
Variability in service provision across regions complicated comparative assessment. External factors (flood) made face-to-face interviews more difficult.
Simplify administrative processes to reduce burden on service providers.Enhance regional coordination to ensure equitable access to services.Secure sustainable funding mechanisms to guarantee continuity of support programmes.
Evaluation of the absorption capacity of the Praha OP, 2014-2020 – Growth Pole of the Czech Republic
Ad hoc evaluace absorpční kapacity Operačního Programu Praha – Pól růstu ČR
EVASCO s.r.o. - marie.kaufmann@evasco.cz
The evaluation analyses the absorption capacity of the Praha OP.
To assess the absorption capacity of the different PAs of the OP and the implementation system and to identify the main challenges.
Desk analysis of relevant documents.40 interviews.Questionnaire survey (269 subjects from primary schools, 187 subjects from secondary schools, 177 subjects from NGOs and 22 entities from urban areas were contacted; response rate 10%, 43%, 7% and 32%).
PA1 (R&D): public awarding authorities are not adequately prepared for participating in EU Programmes. The milestone for 2018 could be at risk and simplification of administrative procedures and rules for applicants are needed to achieve them.PA2 (mobility and energy): the focus on technologically advanced measures (smart buildings), combined with high administrative costs of preparing and managing projects creates difficulties. Achieving milestones requires an acceleration of procedures for existing calls.PA3 (Social inclusion and poverty): the main barriers to absorption include the inappropriate timing of calls and disproportionate administrative requirements for infrastructure projects for social entrepreneurship compared to the relatively small budgets.PA4 (Education): preparing and implementing both ESF and ERDF projects involve heavy administrative costs , together with a risk of subsidy needed to be returned and the short time given to applicants for preparing projects.Other critical issues concern limited communication and information for applicants.
There are some weaknesses in the monitoring system.
Improve coordination between the MA and the Prague City departments involved in the OP’s implementation.Simplify and clarify rules for the preparation of project applications.Strengthen communication with potential applicants.
An update of an analysis concerning the possibility of using repayable financial instruments as part of Regional OP of the Małopolskie Province for the years 2014-2020
Aktualizacja analizy w zakresie możliwości zastosowania zwrotnych instrumentów finansowych w ramach Regionalnego Programu Operacyjnego Województwa Małopolskiego na lata 2014-2020.
Policy & Action Group Uniconsult - ImApp
Policy & Action Group Uniconsult Sp. z o.o., ul. Kierbedzia 4 , 00-728 Warszawa, e-mail: biuro@pag-uniconsult.pl - ImApp sp. z o.o. ul. Rejtana 15/25, 02-516 Warszawa, e-mail: imapp@imapp.pl
Evaluation of the communication strategy of the Interreg-IPA CBC Romania-Serbia 2014-2020
Evaluarea Strategiei de Comunicare Programul Interreg-IPA CBC Romania-Serbia
Alessandro Valenza, Rebeca Nistor, Roxana Diaconu, Cristina Cojoacă
ACZ Consulting SRL, Ștefan Velovan Str., no. 23A, Craiova, Dolj Conty, Romania, office@aczconsulting.ro, t33 SRL, Via Ctalafimi, no.1, Ancona, Italy, info@t33.it
Impact of Cohesion Policy 2007-2013 on information society development
Wpływ polityki spójności perspektywy finansowej 2007-2013 na rozwój społeczeństwa informacyjnego
OP Human Capital 2007-2013; OP Innovative Economy 2007-2013; OP Development of Eastern Poland 2007-2013; All ROPs 2007-2013
45-057 Opole ul. Ozimska 4/7, mail:office@openfield.pl, Tel:77 555 91 88
The evaluation is concerned with the activities undertaken under Cohesion Policy in the 2007-2013 period, focusing on the development of the information society in Poland (i.e. training to improve individual e-competence, activities to improve telecommunications infrastructure and to support e-entrepreneurship).
The objective of the evaluation is to assess the impact of EU programmes on the development of the information society.
The evaluation adopted qualitative and quantitative methods:desk research;analysis of relevant programming documents and past reports;analysis of monitoring data and published statistics (Eurostat);CATI and CAWI surveys to collect quantitative data;in-depth interviews with MAs, intermediate bodies and beneficiaries;case studies;expert panels.
Development of e-skills:about 8,000 residents in Eastern Poland participated to e-inclusion activities;nearly 100,000 households in difficult economic and social situations received basic ICT training;90,000 young people enrolled in ICT studies, 36,000 of whom were technical, natural science and mathematics graduates.Development of telecommunications infrastructure:around 9 million people gained access to the internet as a result of the telecommunications infrastructure installed (over 29,000 km of networks);563,000 final users gained access to the internet through the ‘last mile’ access networks.E-entrepreneurship:entrepreneurs supported assessed their situation on the market as being better than those who did not receive support;over 60% of the beneficiaries would not have been able to carry out their project without subsidies;data indicate that the effects of repayable and non-repayable financial instruments are similar.
The evaluation excluded those areas where, in addition to digital enhancement interventions, other measures were implemented.
The activities to develop e-skills should consider other fields of study and not concentrate only on some;planning of e-inclusion activities should ensure that support is comprehensive and includes individual counselling and should be undertaken in cooperation with local socio-economic actors. Account should be taken of local needs;measure to improve telecommunications infrastructure in the 2014-2020 period should be carried out under one Operational Programme only, so that implementation is properly coordinated;it is important to complement technological support with training activities so as to enable people to adapt to new working conditions;support for e-entrepreneurship (especially for new start-ups) should cover capital investment but also consultancy on implementing the proposed activity.
Evaluation of the Communication Plan of the OP Transport 2014-2020
Vyhodnocení plnění Komunikačního plánu Operačního programu Doprava v období 2019–2022
Evaluation of measures supporting energy transition under the Corse OP, 2014-2020
Evaluations d’impact du programme opérationnel FEDER-FSE Corse 2014-2020 Evaluation Transition Energetique (Pi04a, Pi04c)
http://www.teriteo.fr; https://www.astereuropeconseil.fr - Delphine Paumelle - astereurope@outlook.fr
The report examines measures for energy transition under IPs 4.1 and 4.3 of the OP up to 15 October 2020.
To assess the main results, impact, effectiveness, coherence and relevance of measures for energy transition up to October 2020.
Analysis of documentation.Analysis of statistical, monitoring and cartographic data.13 interviews with institutions and beneficiaries.4 case studies.
Funding committed and certified expenditure under IP 4.1 both amounted to 76% of financing available as at October 2020, with 4 projects being financed. Funding committed under IP 4.3 amounted to 70% of financing, but certified expenditure only to 6%, with 11 projects financed.Financing available is limited and cannot have a significant impact.The level of achievement of 2023 indicator targets is uneven and those under IP 4.1 will not be met.Measures under IP 4.1 have supported a reduction of 596 Teq in CO2 greenhouse emissions and an increase in renewable energy production (of 3.9 MW), which is limited because of the low production capacity of projects (especially solar energy ones).IP 4.3: projects have supported energy renovation of public buildings and a reduction in energy consumption of public lighting.A reduction of 3,517 Teq in CO2 greenhouse emissions from social housing was achieved.ITI projects under IP 4.3 make a very limited contribution to energy transition and in some cases overlap with measures financed by other funds.Simplification of administrative procedures is required by beneficiaries, along with the establishment of networks of stakeholders to support project implementation.Easier access to other funds such as those from CPER and ADEME reduces participation in ERDF tenders.
Reduce financial allocation to social housing and support other types of renewable energy projects.Reduce the administrative burden and adopt simplified cost options.Strengthen territorial stimulation of R&D and improvements in energy efficiency.
Evaluation of social housing projects under the Prague – Growth Pole OP, 2014-2020
Evaluace aktivit v rámci PO3 se zaměřením na sociální bydlení
Blanka Kissová, dr. Daniel Topinka, Helena Topinková, Pavel Proksch
SocioFactor, info@sociofactor.eu, www.sociofactor.eu
The evaluation examines social housing projects supported under Specific Objective 3.1 ‘Strengthened social infrastructure for integration, community services and prevention’ to assist people in need of housing and/or those who spend more than 40% of their income on housing.
To assess the implementation, results and effects of social housing projects to assist people in need.
Desk research (documentation, calls for projects, website data).Interviews with households, social workers, social housing operators, companies in charge of project preparation and administration, the housing commission and city councils.6 case studies.
The municipal districts have been the main applicants for social housing support, except for one case when it was a non-profit organisation.Because of the complexity of calls, most applicants hired companies to administer projects.The main issues during the preparation of construction work were the selection of contractors, the increasing prices of the materials, and longer project duration.The households accommodated have mostly been families with children in an unfavourable social situation, including single parents.According to interviews, moving to social housing improved people’s mental state, as well as in some cases their financial situation, and family and social relations and increased children’s interest in studying.Without EU support, the municipalities would not have been able to afford the reconstruction involved.
Interview respondents sometimes had issues in remembering the main problems during the construction phase and how they solved them, because of the elapse of time and because management had changed.
Support households in social housing through social work.Involve more non-profit organisations in social housing projects.Support cooperation with the neighbourhood and improve communication with the general public.Increase awareness of the existence of the Contact Point for Housing in Prague City Hall.Disseminate good practice examples.Look for other options during the period in social housing and starting points for moving households to standard housing.Provide support for first-time applicants.
Mid-term evaluation of the Smart Specialisation Strategy (S4) under the Navarra ERDF Programme, 2021-2027
Evaluación intermedia de la Estrategia de Especialización Inteligente de Navarra (S4) para el período 2021-2027
The evaluation aims to assess the implementation of Navarre’s Smart Specialisation Strategy for Sustainability (S4) from 2021 to 2024. Data used go up to November 2024.
To assess implementation progress of the S4 and the alignment with the EU enabling conditions for it, and to identify achievements and areas for improvement.
Document analysis.Analysis of statistical office data from National Institute of Statistics (INE), Eurostat, NASTAT (Navarre Community Statistical System).Online survey of 99 key stakeholders involved in the implementation of the S4 (public entities, research centres, universities, and private companies), with a response rate of 34.3 %.23 in-depth interviews with implementers of the S4.3 regionals case studies.8 thematic working groups with implementation stakeholders.SWOT and CAME analysis for each of the strategy’s thematic priority.Validation workshop with experts.
Six thematic areas and the cross-cutting green and digital transitions are central to the strategy. Industrial transition is focused on renewable energy (solar, wind, green hydrogen).The governance of S4 is participatory and improved compared to the previous period; however, it remains fragmented, and SMEs, local authorities, and civil society actors are underrepresented in governance bodies and thematic groups.The monitoring system is still in an early stage of development; some indicators are outdated and data are not systematically used for decision-making.Although there are numerous R&D projects in health, food, and digital sectors, there is limited applied research, low SME participation, and shortage of qualified professionals in key areas such as digitalisation, AI, and green technologies.Awareness of S4 is high among stakeholders but low among the general public.
Strengthen SME support and digital skills training to accelerate innovation and technology adoption across all sectorsEnhance inclusivity by ensuring direct involvement of SMEs, local authorities and civil society in decision-making and strategic planningUpdate and refine indicators to better reflect the current situation and improve their practical use in decision-makingIncrease funding for applied research and foster stronger public-private collaboration to address industrial and societal challengesRedesign the communication strategy to improve clarity, accessibility and public engagement, particularly among the general population
Ex-post evaluation of the Interreg V-A-Czech Republic-Poland. Task 1: evaluation of projects of the 2014-2020 period
Ex-post evaluace programu Interreg V-A Česká republika – Polsko : Úkol č. 1: Ex-post hodnocení projektů v období 2014–2020 (Závěrečná zpráva)
Interreg V-A-Czech Republic-Poland 2014-2020
Lubor Hruška, Andrea Hrušková, Ivana Foldynová, Joanna Kurowska-Pysz, Lucie Jungwiertová, Petr Proske, David Kuban
PROCES – Centrum pro rozvoj obcí a regionů, s.r.o., info@rozvoj-obce.cz, http://rozvoj-obce.cz/; EDREO s.r.o., lucie.jungwiertova@edreo.cz, https://www.edreo.cz/; Kreatus sp. z o. o, biuro@kreatus.eu, https://kreatus.eu
The evaluation examines 26 projects funded under 4 PAs of the OP.
To assess impact of selected projects of the OP as well as the contribution of the programme as a whole.
- Semi-structured interviews with beneficiaries, public administrations, the Joint Secretariat and the MA.- Focus group with representatives of the rescue services (measures for joint risk management), the educational establishments, employers, and cooperating institutions.- Survey of visitors of the Interreg area (600 replies).- Case study.- Analysis of monitoring data and official statistics.- Analysis of administrative data.
- Projects show a high level of achievement of their objectives, outputs, and planned impact. The latter is perceived by both project implementers and target groups.- The unintended effects and impacts are the creation of more significant partnerships, increased attendance at tourist attractions and the interest in good practice by entities outside the partnerships.- Deadweight of support is low, as most projects would not have been implemented without it.
They are not clearly indicated in the evaluation.
- Present examples of good practice, with special focus on the intervention logic, during the events of the Ministry of Regional Development/Joint Secretariat.- Consider for selected calls the introduction of an assessment of the capacity of applicants in the area of cross-border cooperation.- Ensure sustainability checks of the investments carried out under the projects.- Reduce the administrative burden on the beneficiaries.
Needs of beneficiaries in the regional OP Mazowieckie Voivodship post-2020
Określenie potrzeb interesariuszy Regionalnego Programu Operacyjnego Województwa Mazowieckiego post 2020
This study assesses the needs of potential beneficiaries of the regional OP Mazowieckie Voivodship in the post-2020 period.
To assess the needs of beneficiaries of the OP in the post-2020 period.
Desk Research;CATI;CAWI;focus groups;case studies;in-depth interviews;expert panel and workshop.
There is need:to provide more diverse measures to support innovation in of enterprises;for energy efficiency upgrades, energy-efficient street lighting and connections to gas networks;to improve the road network, especially as regards traffic safety and technical standards;to activate local communities and stimulate cooperation to promote social and economic development;to improve access to healthcare.
There are no major limitations reported.
There is a need in the next programming period: to promote projects:reducing emissions;building cycling paths;improving waste disposal and recycling;modernising roads and infrastructure.
Evaluation of progress of the Employment OP, 2014-2020, in the Czech Republic
Filip Kučera - Ministry of Labour and Social Affairs
filip.kucera@mpsv.cz - https://www.mpsv.cz/web/en
The evaluation examines progress in implementing the OP up to end-2018.
To assess the implementation of the OP, results achieved up to end-2018 and the contribution to the Partnership Agreement and the Europe 2020 Strategy.
Monitoring data.Interviews with MA.Survey of beneficiaries and final recipients.Evidence from previous evaluations.
As at end-2018, good progress is reported for measures on employment, including reducing gender disparities and CLLD.Progress was limited for measures on social inclusion, social enterprises, social innovation and effectiveness of public administration.For employment measures, costs per participant are lower than expected.Delays were mainly caused by low capacity of beneficiaries, the administrative and managerial complexity of large projects and changes in strategic decisions.
Evaluation of PA C of the Bund ESF OP 2014-2020
Zusammenfassende Bewertung der Prioritätsachse C
Lena Heitzenröder, Hans Verbeek, Christina Klingler, Melchior Decitre - ISG
ISG Institut für Sozialforschung und Gesellschaftspolitik GmbH, Hans Verbeek, Weinsbergstraße 190, 50825 Köln, GermanyTel.: +49 30 - 650 43 90, E-Mail: Verbeek@isg-institut.de
The evaluation is a summary of previous evaluations of PA C of the OP for continuing education and lifelong learning, digital and innovative learning and integration into training.
To assess the contribution of support for continuing education and lifelong learning, digital and innovative learning and integration into training to the OP objectives and the Europe 2020 strategy.
Methods used in previous evaluations are:Desk reviewSurvey of implementing bodiesAnalysis of monitoring dataCase study with interviews with project managers and external stakeholdersOnline survey of project managers and stakeholders (15% response rate, 11,446 people contacted)Focus group with the MA and Intermediate bodies.
Access to continuing education and lifelong learning is based on three programmes: ‘Bildungsprämie’ (vocational training for those with low income), ‘Quereinstieg’ (continuing education) and ‘Elternchance’ (training for parents to support the education of their children).Those with low education tend to invest less in continuing education than those with higher levels. For the ‘Bildungsprämie’, only 3% of participants had low education.66% of participants in the ‘Quereinstieg’ programme were trained as teachers, and 89% of them were in employment four weeks after the end of support.Continuing training has helped to improve the employment situation of people aged 18-54 than those under 27 or 55 and over.Support for older workers has helped maintain employability in old-age, strengthen occupational skills and give them more flexibility in the labour market.528 SMEs were supported under the ZdA programme (‘Zukunft der Arbeit’ for developing entrepreneurial innovation and improving work organisation based on enterprises’ needs) and the ‘Digitale Medien in der beruflichen Bildung’ (promoting media literacy and use of digital media in VET).The use of digital media has strengthened the motivation of trainees and employees and increased skills development.158,317 young people have participated in the ‘Jobstarter plus’ and ‘Berufseinstiegsbegleitung’ (BerEb) programmes, 47% with a migration background, 39% women.92% of participants were still in education or training 6 months after the end of support.Programmes focused mainly on strengthening the vocational competence of young people.
There is limited evidence on measures supporting digital and innovative processes.
Take into account potential barriers for accessing particular target disadvantaged groups.Ensure synergies between programmes focused on similar target groups and/or objectives.Maintain an intensive occupational orientation and continue support during the first months of training.Supplement training activities with mentoring.Focus vocational training on the low-skilled.
Evaluation of the Comunidad de Madrid ERDF OP, 2014-2020. 2018 Report
Informe de evaluación del Programa Operativo FEDER 2014-2020 de la Comunidad de Madrid
The evaluation examines progress in implementing the Madrid ERDF OP, 2014-2020, covering the 7 Axis and focuses on the period 2014-2018.
To assess the progress of the OP in terms of achievements of targets of the performance frameworks and result indicators, the contribution to the Europe 2020 strategy and compliance with the horizontal principles.
Desk analysis of programming and other relevant documents.Analysis of monitoring data and official statistics.Interviews with the MA and IBs (20 people from 8 different bodies).
5 out of 7 of the PAs meet the performance framework targets. The exceptions are s PA6 (TO6), which is carried out by local municipalities and which registers no financial progress and PA14 for developing financial instruments in RDI and which is not set up yet.7 of the 14 result indicators show good performance and for two an average one. The poor performance of 5 indicators is a result of socio-economic developments, delays in carrying out activities and the nature of some indicators which are not always adequate to register the results achieved.The OP’s objectives are in line with the Europe 2020 strategy. Horizontal aspects of partnership and multi-level governance, strengthening the capacity of the authorities to use the Funds and sustainable development are incorporated in the programming. The principle of gender equality has been incorporated, but it is not included among the selection criteria and measures directly related to this issue are not financed.
Transfer the Performance Reserve of PAs 6 and 14 which have not reached the milestones.Change the target values of some result indicator and the calculation method for the indicator of PA2, and replace the indicator PA9 - Shantytown nuclei existing in the region.
Evaluation of the implementation of horizontal principles and policies under the Hungarian OPs 2014-2020
Horizontális elvek és politikák érvényesülése
Balogh Attila, Cserneczky Tamás, Kiss Lilla, Rumi Edit, Torma Ferenc, Zima-Barna Zsuzsanna
AQUA Engine Kft., http://aquaengine.hu/; Collective-Intelligence Kft.
The evaluation examines the implementation of horizontal principles under the Hungarian OPs, more specifically equal opportunities and environmental protection.
To assess the incorporation of horizontal principles in the projects implemented and the measures applied to promote this.
Documentation and data analysis.Interviews with MAs.Expert workshops.Surveys.Identification of good practice examples.
Project implementers are not sufficiently aware of the essence of horizontal principles.Despite attempts to reduce the administrative burden on beneficiaries, it remains high.Expectations regarding the horizontal principles are sometimes disproportionate given the size of projects and tend to be described only vaguely.The guidelines for the implementation of horizontal principles were delayed and published after the calls for projects were out.The absence of proper monitoring and sanctions weakens the implementation of the horizontal principles.Because of insufficient staff and time constraints, there was a lack of central coordination in managing calls.MAs do not have a formal obligation to implement the horizontal principles.Horizontal indicators are only included in calls for which a horizontal principle was the objective.
Because of problems with them, there is no in-depth analysis of the indicators.
Make the central coordination take the lead in ensuring a more efficient implementation of horizontal principles.Apply a differentiated system of requirements to ensure more flexibility in project implementation, e.g. small projects being obliged to meet only minimum criteria.Ensure awareness-raising activities for project implementers and public authorities in general.Ensure timely and clear guidelines for the implementation of horizontal principles.Reduce the administrative burden further.Ensure sufficient staff at the central coordination level and establish a unit for horizontal objectives.Organise training and workshops to share experience and knowledge about horizontal principles between MAs and disseminate examples of good practice.Develop principles of accountability and sanctioning.Specify the obligations of MAs in the Rules of Procedure.Establish a central database of indicators.
Interim evaluation of the Communications Strategy of the Toscana ERDF OP 2014-2020
Valutazione intermedia Piano della Comunicazione FESR 2014/2020
Istituto Regionale Programmazione Economica Toscana (IRPET)
This evaluation assesses the performance of the Communications strategy implemented by the Toscana regional authority under the regional OP for 2014-2020.
To assess progress made in implementing the Communications strategy and to compare the expected results with the outturn.
There is reasonable awareness that the region receives funding from the EU.But there is still insufficient awareness of the OP itself.The enterprises surveyed were, however, aware of the possibility of applying for funding.
There are no policy recommendations included.
Evaluation of the progress of the Quality of Environment OP, 2014-2020, in Slovakia
Hodnotenie stavu implementácie OP KŽP z pohľadu plnenia pravidla N+3
The evaluation examines progress of the OP towards targets up to February 2022.
To assess the relevance and achievement of the objectives set up to February 2022, especially the measures put in place to achieve financial goals, and the risks of not meeting them.
PA2 (TO5) had the lowest absorption rate at the end of 2021 (only 24%), though there is a high potential for this to increase.Absorption capacity is positively affected by the quality of applications.The main obstacles to absorption are lengthy project appraisal procedures, the long duration of projects, the limited capacity of the MA and IBs, delays caused by COVID, and rising prices (especially for energy and building materials).
Shift funding from IPs with low financial absorption to IPs with high potential absorption capacity.
Evaluation of horizontal policies under the Podlaskie OP, 2014-2020
Ewaluacja wdrażania zasad horyzontalnych w ramach RPOWP 2014-2020
Krzysztof Gutta, Tomasz Gutta, Katarzyna Gutta
EU-CONSULT sp. z o.o., ul. Toruńska 18C lokal D, 80-747 Gdańsk
The study examines the implementation of horizontal principles at different stages of the implementation of the OP.
To assess the implementation of horizontal principles under the OP, as well as the relevance, usefulness, effectiveness, sustainability and efficiency of measures taken to implement them.
Desk research.CAWI/CATI survey with beneficiaries (642).CAWI/CATI survey with final recipients of support (723).Interviews with representatives of institutions, experts, beneficiaries, final recipients, stakeholders (11).Case studies.Panel of experts.
Horizontal principles are taken into account at all stages of implementing projects, but this is sometimes complicated, due to the nature of projects - applicants perceive it as an unnecessary formal requirement.Documents prepared at the national level (Agenda&Guidelines) are helpful in understanding how principles should be implemented. In case of LAGs (CLLD), the information flow needs to be improved.Stakeholders are very aware of the need to take account of horizontal principles.The administrative burden and costs relating to horizontal principles are small.Programme indicators do not always reveal the beneficial effect that projects have had on people exposed to discrimination.In the case of LAGs (CLLD), the information flow on horizontal principles needs to be improved.There are not enough training and information meetings on horizontal principles.
Use more detailed indicators, including long-term result ones, which enables changes in the situation of final recipients to be monitored.Use a more flexible approach to assessing projects, taking account of their specific features.Continue information and training activities and intensify those for LAGs.Continue cooperation with equal opportunity and non-discrimination organisations.Create catalogues of examples of good practice on horizontal principles and on the types of project which are neutral in their effect on horizontal principles.
Ex-ante study of the use of financial instruments in Bretagne in 2021-2027
Évaluation ex-ante portant sur la mobilisation des fonds européens 2021-2027 sous forme d’instruments financiers en Bretagne
This is a study of the availability of financial instruments in the region, the needs in different sectors and the extent of market failure, in preparation for 2021-2027.
To map the financing available in the region, to identify needs in the light of the COVID-19 crisis as well as longer term issues and the extent of market failure and to suggest potential strategies for using FIs.
Desk analysis of official statistics.Just under 100 interviews with relevant groups.7 thematic workshops.
There is a large number of financial instruments available for companies, covering all potential needs, though they are not always very visible.FIs to support innovation have increased in recent years. In 2004-2020 the Breizh Up Fund was created, though it was mainly taken up by firms in smaller and riskier parts of the economy. Funding for the energy renovation of social housing, public buildings and private housings mainly takes the form of national subsidies, though innovative schemes have been put in place to support the energy renovation of social housing.The funding available for renewable energy projects is satisfactory. There are many aid schemes at regional, national and EU levels and public banks are active in financing projects through specific funds for companies or loans for individuals.Several forms of market failure are present in the areas listed above.
Ensure consistency between FIs and the priorities set out in strategic documents.Pay attention to the complementarity of FIs funded from different sources.Focus EU funding on a limited number of interventions to ensure a critical mass.Involve regional bodies already managing financial instruments.
Evaluation of progress in achieving the objectives of the Environment OP, 2014-2020
Zhodnocení dosavadního pokroku při dosahování cílů OPŽP 2014-2020.
Lukaš Bumbálek, M. Hartlová, K. Šustrová et al.
Naviga 4, s.r.o. Pobřežní 249/46 186 00 Praha 8, bumbalek@naviga4.cz
The evaluation assesses progress in achieving the objectives of OP Environment, the relevance of the strategy and the achievement of specific objectives.
To assess the compliance of the OP with strategic documents, the intervention logic, the relevance of priorities and objectives and its effectiveness and efficiency.
Desk Research.CAWI.Interviews.Analysis of indicators.
The OP strategy is coherent with existing programmes and needs, though a strategy for combating droughts is missing.The schedule of calls is set up appropriately and corresponds to the OP intervention logic.There are absorption issues with respect to calls related to Priority Axis 4 and Priority Axis 5;The achievement of targets in 2018 for PA 4 and 5 appears to be highly unlikely.
There is a need to:Consider the integration of strategies to combat droughts.Include links to the national programmes of the Ministry of Agriculture under PA 1 and the PA 4.Include new links with the Rural Development Programme in respect of support for biodiversity.
Evaluation of interventions to support degraded areas under PA3 of the Metro OP, 2014-2020, in Italy
Report intermedio Indagine n. 2 – Analisi del processo di attuazione degli interventi di sostegno all’attivazione di nuovi servizi in aree degradate finanziati dall’Azione 3.3.1 dell’Asse 3 del PON Metro
IRS - https://www.irsonline.it/ - PTSCLAS
The evaluation analyses the implementation of measures to support the introduction of new services in degraded areas financed by the OP, 2014-2020. Data go up to October 2022.
To assess the level of implementation of projects for degraded areas and their management up to October 2022.
Desk analysisAnalysis of monitoring data.
Overall, the 122 projects launched have made it possible to provide services to over 132,000 people; 2,400 of them from disadvantaged groups.Bologna, Torino, Napoli and Bari have reinforced public-private networks through the creation of innovative services managed by Third Sector organisations.Other metropolitan cities have financed more "traditional" projects, with Third Sector organisations only being providers of services.
Evaluation in the Visegrad 4 countries plus 4 others: an overview of practices
Ewaluacja w krajach V4+4 - przegląd stosowanych praktyk
Karol Olejniczak, Tomasz Kupiec, Dominika Wojtowicz, Weronika Felcis
EGO – Evaluation for Government Organizations s.c., ul. Podmokła 53 B, 04-819 Warszawa, e-mail: evaluation@evaluation.pl
The study is aimed at assisting Evaluation Units to support decision makers by providing research-based knowledge.
To assess how Evaluation Units in 8 Central and Eastern countries undertake evaluations and to identify the positive and negative aspects.
Desk analysis of relevant documents;survey of 74 staff from evaluation units in Bulgaria, Croatia, the Czech Republic, Hungary, Poland, Romania, Slovakia and Slovenia;a survey of selected users of evaluation reports in each country;27 interviews with representatives of leading evaluation units, local evaluation experts or members of evaluation societies in each country;7 interviews with European Commission representatives.
There are significant differences between evaluation systems which are centralised in Hungary, Slovenia and Romania and decentralised in the other countries;dissemination of results is often a formal task and knowledge from evaluations is not accumulated and made sufficiently accessible;evaluation unit personnel reported being constrained by limited resources and low interest from top management;users of evaluation report learning most about processes and less about effects.
There is a need to:develop a “learning agenda” rather than a list of administrative requirements;co-design evaluations with users to tailor them to their needs;improve accessibility of data;strengthen dissemination through summaries and visuals outlining the main results and recommendations;limit the numbers of recommendations;use this material for training.
Impact evaluation of research and innovation measures supported under the Corse ERDF-ESF OP, 2014-2020
Evaluations d’impact du programme opérationnel FEDER-FSE Corse 2014-2020 - Evaluation Recherche et innovation (Pi01a, Pi01b)
Mathieu Sabourin - sabourin@teriteo.fr, contact@teriteo.fr
The evaluation examines the impact of measures on research and innovation. It covers the implementation period between 2014 and 2020.
To examine the impact of research and innovation projects on the structure of the R&D ecosystem in Corsica.
Analysis of monitoring and national statistical data.Interviews with institutional actors.Survey of project holders.8 case studies.
The measures focussed more on specific research projects, despite the need to finance the overall research ecosystem, especially in terms of infrastructure.Support for R&I projects positively contributed to the collaborations between researchers and initiated longer-term collaboration between public and private players.The timeframe of an EU programme is an obstacle to the implementation of complete innovation processes, as projects need more time to achieve technological readiness.Despite the leverage effect of support on the regional R&D expenditure being very significant, the measures are not sufficient to reverse the dynamics of the territory.
Evaluation of measures on urban development under the Centre-Val de Loire ERDF-ESF OP, 2014-2020
Evaluation d’impact Axe 5 – La solidarité avec les quartiers urbains les plus défavorisés
The evaluation examines the effect of measures to support urban development in deprived areas under the OP.
To assess the contribution of the measures to the creation of new SMEs and to improving energy efficiency in disadvantaged urban areas.
Interviews with MA, IBs, beneficiary social housing bodies. Analysis of monitoring data and official statistics. Survey of beneficiaries (response rate 14%). Workshop on recommendations.
The measure to stimulate the creation of new SMEs shows very little progress (only 1 project financed) because of problematic conditions in the eligible areas, the existence of similar projects having simpler rules and the limited finance provided by the ERDF. Projects on energy efficiency have contributed significantly to stabilising the energy costs of households in a context of high inflation and to improving the quality of accommodation. However, a rebound effect is also reported, as the reduction in the energy costs of housing has led to increased consumption.
The survey has limited validity because of uncertain information on the people contacted and self-selection bias due to access to e-mail and the language used in the survey.
Strengthen selection criteria to concentrate funding on projects with the largest higher impact on the environment and energy efficiency. Define awareness activities in respect of the households supported to improve energy consumption behaviour and monitor actual consumption. Increase support for beneficiaries and the visibility of ERDF funding.
Thematic evaluation of financial instruments supporting business investment in Sachsen-Anhalt in 2007-2013
Thematische Evaluation der einzelbetrieblichen kapitalorientierten Förderinstrumente: GRW gewerblich, KMU-Darlehensfonds und IBG Risikokapitalfonds II im Rahmen des EFRE Sachsen-Anhalt 2007-2013
Jacob Fittkau, Dr. Thorsten Lübbers, Marcus Neureiter, Philipp Till Schultz
Ramböll Management Consulting GmbH, Chilehaus C - Buchardstr. 13, 20095 Hamburg, Tel: +49 40 30 20 20-115, jacob.fittkau@ramboll.com
The evaluation assesses the financial instruments supporting business investments funded by the OP Sachsen-Anhalt ERDF in 2007-2013. Three measures are analysed in detail: IBG Risikokapitalfonds II, a venture capital fund; GRW (gewerblich) which provides investment grants; and KMU-Darlhensfonds which provide loans for SMEs.
To assess the relevance, effectiveness and impact of the measures supported.
Desk review;analysis of monitoring data;interviews with the Managing Authority (MA), the Investitionbank Sachsen-Anhalt (IB) and IGB;discussion with representatives of the Chambers of Industry, Business and Handcraft and Trade Union Confederation;survey of enterprises supported (25% response rate out of a total of 1,276).
Measures supported were generally in line with regional needs.IBG Risikokapitalfonds II (measure 1): there were difficulties in implementation; 135 investments were supported (around 50% more than the target), 358 new jobs were created (a third more than the target), 88 for women, and 502 safeguarded, 249 for women.GRW (gewerblich) (measure 2): implementation proceeded as expected; 846 projects were supported (5 times more than the target); 4,567 new jobs were created, 980 for women (in both cases a third less than the target) and 24,730 jobs safeguarded (two thirds of the target), 6,083 for women. The economic and financial crisis led enterprises to focus more on maintaining than creating jobs. KMU-Darlehensfonds (measure 3): implementation proceeded as planned with positive revolving effects; EUR 277 million of loans were paid (95% of the target); 16,592 jobs were safeguarded (nearly the double than the target set), of which 3,320 for women.The implementation procedure, including consultancy and cooperation with those managing the funds, was positively assessed by respondents (though the quality of information material was not considered adequate in measure 1).The support helped enterprises to improve their performance and innovation ability.Investment would not have taken place without ERDF support for two thirds of measure 1 respondents, less than half of measure 2 ones and 60% of measure 3 ones.
It was not possible to adequately assess impact and a counterfactual analysis was not carried out.
There is the need in the next programming period (2014-20) to:continue the support of these types of measures given the continuing need of financial support by enterprises;use more financial instruments such as loans and venture capital;improve the quality of information documents particularly for measure 1.
Evaluation of the measures for social inclusion under the Legality OP, 2014-2020, in Italy
Rapporto sul potenziamento dell’inclusione sociale
PTS, https://www.ptsclas.com/en/home-en/ - IRS, https://www.irsonline.it/ - MIPA, https://www.consorziomipa.it/
The evaluation examines the measures for social inclusion carried out under the OP, on inclusion of immigrants, redevelopment of properties confiscated from criminal organisations and support to the social economy and for tackling Covid-19.
To assess the preliminary evidence on the contribution of the measures to social inclusion, the social economy and the redevelopment of properties confiscated from criminal organisations.
Analysis of monitoring data and project documents.Interviews with MA.Surveys of beneficiaries.Case studies.
Measures have triggered a process that should result in increased social inclusion of migrants, because of good planning, partnership with local bodies and integration of different activities.Projects on redevelopment of confiscated properties have contributed to strengthening managerial competences of social enterprises, and awareness raising, partly through support offered to racket victims.Beneficiaries seem to have planned alternative means of ensuring financial sustainability of projects in the long-run, after EU funding ends.Targets relating to tackling COVID-19 have been met, though they were significantly underestimated.Projects helped increase administrative capacity, including through the development of new information systems, but project delays are significant.
There were difficulties in contacting project representatives and obtaining answers from all of them. In most cases the potential contribution of measures is assessed, as most projects are not completed.
Ensure the monitoring of management aspects of projects, in addition to administrative ones.
Operational Evaluation of the Melilla OP reprogramming
Programa Operativo FEDER 2014-2020 de Melilla Evaluación operativa por reprogramación técnica
The evaluation assesses the financial reprogramming of the Melilla OP for 2014-2020.
To assess whether the reprogramming is coherent with the initial OP strategy.
A combination of qualitative and quantitative methods based on documentation and data, mainly financial, from the Fondos2020 accounting system and intermediate bodies.
The main purpose of reprogramming was to include an increase in the OP budget of EUR 1.7 million resulting from technical revisions.Reprogramming increased financing for TO4, in particular, for the development of the Urban Mobility plan.Other adjustment made did not change the budgets for the respective TOs.Some output and outcome indicators were modified and associated targets changed.Reprogramming did not change the OP strategy or the intervention logic, which remains in line with the original OP objectives.
Pilot project - Contribution to the EU2020 - Climate Change and Energy Sustainability
Pilotný projekt – príspevok k EÚ 2020 – zmena klímy a energetická udržateľNosť
Vladimír Baláž, Tomáš Jeck, Richard Filčák, Daniel Škobla, Mario Rodiguez
Institute of Economic Research - Slovak Academy of Sciences, www.sav.sk - KPMG Slovakia and Štengl a.s., www.home.kpmg/sk/
Evaluation of interventions for improving education infrastructure in the Regional OP, 2014-2020
Evaluarea intervențiilor POR 2014-2020Îmbunătățirea infrastructurii educaționale
The evaluation examines PA10 of the Regional OP “Improving Educational Infrastructure” for investing in all areas of education and training, including lifelong learning. Projects started only in 2018 and none are finalised yet.
To assess the progress of the PA and the performance so far.
literature review; 9 case studies;8 focus group with beneficiaries;50 interviews with selected beneficiaries and RDAs;3 panels with education and public administration experts;analysis of administrative programme data.
Demand for funding for education infrastructure far exceeds the budget available.Legislative changes affected the implementation, such as in public procurement regulations, requirements as regards emergency situations and construction and the increase in the minimum wage.There is a risk that some investment may not be fully realised or may be delayed, due to the lack of capacity of beneficiaries in term of specialists in construction.Large centralised projects, which are the most advanced, can be considered effective.Students learning in renovated buildings, will benefit from the facilities and this will have an effect on school and university participation (helping to reduce drop-outs), and on learning outcomes.The sustainability of the projects is likely to be higher if they are part of a local development strategy and coherent with other investments.Because of a reduction in the number of school-age children, there is a risk of sustainability of the investment.There is a growing interest in investment in VET and continuing training.
The early stage of implementation of many projects greatly limited the analysis. In the absence of progress and monitoring reports, evaluation results were based mainly on estimates from contractual commitments and information from interviews and focus groups. The large number of evaluations that took place in parallel limited the availability of certain groups for the evaluation. The limited access to administrative data on education constrained the analysis.
carry out annual reviews of average costs for different categories of works to guide public procurement procedures;provide additional technical support for beneficiaries without sufficient institutional capacity;ensure stability of legislation or at least reduce the impact of legislative changes on the ongoing investment;focus more on interventions for vulnerable groups;include consideration of demographic changes in impact assessments;combine the development of educational infrastructure with other infrastructure projects, especially in developing neighbourhoods;focus future projects on the rehabilitation of secondary, high and vocational school infrastructure;simplify selection and contracting procedures, to reduce bureaucracy and improve the functioning of the management information system;ensure more flexibility in construction projects.
Evaluation Report of the Rheinland-Pfalz ESF OP, 2014-2020 – 2019/2020
Martin Angres, Jenniver Asmussen, Dr. Oliver Dick, Dr. Christian Lenhart, Lina Muhl, Dr. Dörte Schott, Dr. Michael Seligmann
Institut für Sozialpädagogische Forschung Mainz e.V., Augustinerstraße 64-66, 55116 Mainz, martin.angres@ism-mainz.de, Tel.: + 49 6131 - 906 18 - 10
The evaluation examines the implementation of the OP, especially the “Opening perspective” funding scheme under PA2 – IP 9.i. Data on implementation of the OP was collected in November–December 2020.
To assess the implementation and results of the “Opening perspective” funding scheme to provide training for the inactive and long-term unemployed under IP 9i and of other measures and the impact of the COVID pandemic on project implementation and participants on the basis of data collected in the last 2 months of 2020.
Analysis of documentation and previous evaluations.Analysis of monitoring data and official statistics.Survey of participants (220 respondents).
Adjustments introduced after COVID-19, such as the adaptation of funding guidelines and the switch to an on-line format, improved implementation.IP 8.v: on-line support and advisory services for SMEs and workers were activated to support adaptability of workers, which helped increase digitalisation of SMEs.IP 9.i: some of those disadvantaged were unable to access new digital means of support because of their low digital skills.The “Opening perspective” scheme meets the multidimensional needs of the target population. Most participants considered themselves better prepared for work as a result.IP 10.i: there is a mismatch between the training needs of young people and the training offered by companies, and COVID-19 is likely to worsen the problem.IP 10.i: a large number of disadvantaged young people involved in measures to reduce early-school-leaving have a lack of digital skills because of a lack of equipment.IP 10.iii: the implementation of measures was severely affected by the pandemic.IP 10.iv: the measures to improve the labour market relevance of the education system had no problem in adapting to working online as this was already an integral part of projects.
Take more account of the specific problems caused by the pandemic to those disadvantaged.Pay more attention to the new need to support young people willing to work.Consider the need to improve digital competences in education and training measures.
Evaluation of the projects selection system in the OP Development of Eastern Poland 2014-2020
Ewaluacja systemu wyboru projektów w Programie Operacyjnym Polska Wschodnia 2014-2020
OP Development of Eastern Poland 2014-2020
Policy & Action Group Uniconsult Sp. z o.o., ul. Kierbedzia 4 , 00-728 Warszawa, Tel: 022 256 39 00, e-mail: biuro@pag-uniconsult.pl - ImApp sp. z o.o. ul. Rejtana 15/25, 02-516 Warszawa, Tel: +48 22 390 53 04, e-mail: imapp@imapp.pl
Socio-economic analysis in preparation for the 2021-2027 programming period in Bremen
Begleitende Evaluation des Operationellem Programm 2014-2020. Sozioökonomische Veränderungen seit 2014 und Schlussvolgerungen für die neue Förderperiode
Birgit Nolte, Miguel Riviere, Susanne Gallenz, Marcus Neureiter
Ramboll Management Consulting GmbH, Chilehaus C – Burchardstraße 13, 20095 Hamburg, Birgit.Nolte@ramboll.com
The study examines the socio-economic context in the region in recent years. The results will be used for an impact assessment of the ERDF OP in the next programming period.
To assess the socio-economic situation in the region and to make recommendations for the 2021-2027 period.
Analysis of regional and national statistical data.
Bremen is a strong innovator at EU level, though private R&D expenditure is relatively low. The share of R&D personnel in total public sector employment is above the EU average.Despite the availability of high-performance broadband networks, digitalisation in SMEs poses some challenges.The number of SMEs in the region is declining and there is low investment and start-up intensity, as well as limited involvement of venture capital investors in technology-oriented companies.CO2 emissions per capita are relatively high Renovation of buildings, especially public non-residential ones, has the greatest energy saving potential.The consequences of climate change are noticeable. Air quality needs to be improved by reducing particulate matter.Green infrastructure plays an important role in preserving biodiversity.
Strengthen existing economic clusters and promote the development of innovative products, processes and services.Support knowledge transfer to industry, regional networking between SMEs and research institutions and the development of business-oriented R&D infrastructure.Increase support for business start-ups.Promote financial support for energy renovation of public buildings and identify energy saving potential in public infrastructure.Include environmentally friendly infrastructure in climate adaptation processes.
Ex-ante study of financial instruments (loans and guarantees) under the Basilicata ERDF/ESF+ Programme, 2021-2027
Valutazione ex-ante degli Strumenti finanziari. Prestiti e Garanzie
Regione Basilicata – Direzione Generale per la programmazione e la gestione delle risorse strumentali e finanziarie
The study analyses some of the financial instruments (loans and guarantees) proposed under the programme. The financial instruments (FIs) are planned to support SMEs, the green economy, sustainable urban mobility, the regional cultural heritage and territorial strategies.
To assess the relevance of the proposed financial instruments and their capacity to contribute to the programme’s objectives.
Desk research on (previous and current) programme documents, legal norms and on relevant studies on the theme. Analysis of statistical and monitoring data – the latter only used to summarise the level of financial achievement of the 2014-2020 OP.
The loans and guarantees planned under the programme are meant to facilitate access to credit for regional firms, which find it difficult to obtain liquidity due to their limited size.Overall, the total allocation planned under the programme amounts to EUR 79.6 mln for loans, and to EUR 22.6 mln for guarantees. The amount earmarked for loans and guarantees appears appropriate, except for RSO1.3, for which it could be reduced in favour of non-repayable grants, due to the increase in raw material prices and interest rates, the crisis of the automotive sector and the limited size of regional firms.The estimated leverage effect of loans is 3 and of guarantees is 6. All the FIs analysed are expected to stimulate proportionally higher investments than non-repayable grants.Financial instruments can be used together with grants or, with respect to loans only, with tax credit initiatives planned at regional or national level.Except for a few cases in which targets may be slightly adjusted (especially for RSO2.8 and RSO5.1) to reflect the trends of raw material prices, the targets set appear appropriate for all the Specific Objectives analysed.
Reduce the amount planned by the programme for RSO1.3 by EUR 15 mln for loans and 2.5 mln for guarantees.
Final Evaluation of the communication strategy of the Youth Employment OP 2014-2020
Evaluación Final de la Estrategia de Comunicación del Programa Operativo de Empleo Juvenil 2014-2020
Evaluation of the national Smart Specialisation strategy in Poland
Ewaluacja wsparcia w ramach PO IR w zakresie Krajowych Inteligentnych Specjalizacji
Jan Szczucki, M. Gajewski, P. Oborski, J. Witkowska
Policy & Action Group Uniconsult sp. z o.o. ul. Kierbedzia 4, 00-728 Warszawa, Tel.: 22 256 39 00, www.pag-uniconsult.pl, e-mail:biuro@pag-uniconsult.pl
This evaluation assesses the support provided under the Smart Growth ERDF OP 2014-2020 in respect of smart specialisation.
To assess the functioning of the OP’s implementation system and the projects’ selection system in relation to the National Smart Specialisation (Krajowe Inteligentne Specjalizacje – KIS) strategy.
Desk research.Data analysis.CAWI.Focus groups.Individual in-depth interviews.Panel of experts.
The Smart Specialisation Strategy (SSS) in Poland covers 5 thematic areas and 17 sub-categories.Projects supported under TO1 have to cover at least one of the SSS categories, while under TO3 compliance with SSS is not compulsory (though there is a premium in the selection process).So far, the funds have been concentrated mainly in ‘Healthy society’ and agri-food.Respondents state that the SSS is well defined and its categories enable a very wide range of enterprises to be supported.Beneficiaries consider the flexibility of the SSS to be a positive feature, though according to experts the areas covered should be reduced. The IT systems category, despite the high rate of applications, has the lowest success rate. The current description of this innovation areas does not cover most classic IT systems.
There is a need to:support directly specific sectors;improve communications between experts and members of the SSS working groups;consider modifications to the contents of the SSS areas;review the definition of the areas improving the quality of the description.
Evaluation of the Training Directive supported by the Thüringen OP
Evaluierung der Ausbildungsrichtlinie im Rahmen der Förderung durch den Europäischen Sozialfonds (ESF) im Freistaat Thürin gen in der Förderperiode 2014 bis 2020
Institut für Sozialökonomische Strukturanalysen (SÖSTRA GmbH)
The evaluation assesses the extent to which the Training Directive, supported by the ESF, was implemented in Thüringen. Under the Directive, SMEs have an opportunity to implement training in various ways - inter-company teaching instruction (Fördergegenstände Überbetriebliche Lehrunterweisung-ÜLU), supplementary courses (überbetriebliche Ergänzungslehrgänge -LG) and additional qualifications for digital competences (Zusatzqualifikationen zur Vermittlung digitaler Kompetenzen - ZQ).
To find out to what extent the Training Directive measures contributed to increasing participation in company-based training and how the same measures are implemented in other German regions.
Desk research.Analysis of monitoring system data and databases from the Bundesinstitut für Berufsbildung, the Heinz-Piest-Instituts and the Ministry of Economics and Energy.Meetings with experts and 12 interviews with Chambers of Commerce.Online surveys of companies (723 surveyed with a response rate of 56%), training associations and education service providers.Interviews with representatives of other German regions.
Up to 20 March 2019, 21,888 people received support under the Training Directive, 21,721 of these taking part in ÜLU (58%) and LG (42%) measures. The majority of participants were men (79%) under 25 (91%) and with a secondary education (61%). The low participation of women is in part explained by the highly technical and commercial nature of the ÜLU and LG measures.79% of participants in LG successfully completed the training compared to 63% of those participating in ÜLU.Comparison of inter-company apprenticeship in the German regions shows differences in the cost of individual basic level courses, varying from 30% in Hamburg, Niedersachsen and North Rhine-Westphalia to 60% in Hesse, Berlin and two-thirds in Brandenburg. In other cases, lump sums are charged.
These measures should be maintained in the future;more flexibility is desirable to better adapt the support to the specific needs of beneficiaries;the heterogeneity of beneficiaries should be taken more into account, by offering targeted measures, e.g. for weaker students;publicity means and communication of the measures supported needs to be improved;equal opportunities and reconciliation of work and family life should be taken more into account;the system of collecting data by directly transferring them into the monitoring system needs to be improved.an indicator to measure participants over 25 is needed.
Evaluation of road safety developments in Hungary in the 2007-2019 period
Közúti közlekedésbiztonsági fejlesztések értékelése
OP Transport 2007-2013; OP Integrated Transport 2014-2020
Főmterv Zrt – Collective Intelligence Kft.
The evaluation examines the implementation of measures for road safety in Hungary, funded under the Integrated Transport OP 2014-2020 and the Transport OP in 2007-2013.
To assess progress in implementing projects to develop local roads, their effectiveness and efficiency in 2007-2019.
Monitoring data.National statistics.Interviews with implementers.Surveys of recipients of support.Case studies.
Implemented projects have increased road safety. Fatal accidents of people under 14 have declined by 74% and, in 2018, the risk of death was a third of the level in 2002.However, targets are not expected to be met. Factors other than the projects supported (speed, seat belts,...) are also important in preventing targets being achieved.Improvements in main road junctions are welcomed by people living nearby.Advanced technological measures (mobile apps, variable signals,…) are not included in projects.
Continue to focus on road safety in 2021-2027.Improve coordination of programmes at regional and national level.Consider including advanced technological measures in road safety projects.Improve the system for collecting data on road safety, such as traffic flows and accident numbers.
Evaluation of measures under IP 9.v “Social economy” of the Human Resource Development, Education and Lifelong Learning OP, 2014-2020, in Greece
Παροχή υπηρεσιών συμβούλου για τη διεξαγωγή αξιολογήσεων του Σχεδίου Αξιολόγησης του ΕΠ-ΑΝΑΔΕΔΒΜ; Τμήμα 1: Μελέτη αξιολόγησης των παρεμβάσεων στο πλαίσιο της Κοινωνικής Οικονομίας και των λοιπών δράσεων της Επενδυτικής Προτεραιότητας 9.5
OP Human Resources Development, Education and Lifelong Learning 2014-2020
The report analyses the results of the OP in relation to IP 9v, which promotes the Social and Solidarity Economy (SSE). The main measures are the development and promotion of cooperative enterprises and the creation and strengthening of Support Centres (SC) for the social and solidarity economy.
To assess the correctness of the strategy, the achievement of the objectives, the adequacy of the measures and the way in which they were implemented.
Surveys. Interviews and focus groups. Analysis of monitoring data. Analysis of previous evaluations, studies and annual reports.
There have been serious delays in implementation because of poor planning, changing political priorities, lack of a culture of social entrepreneurship, the pandemic and complex public administration procedures. Support Centres were considered a successful measure that should be strengthened with staff training activities. The measures were difficult to access for several actors of the SSE ecosystem. Flexibility and updating of OP measures, as well as participatory planning and consultations with the SSE ecosystem are considered important factors.
The indicators are not always adequate. The measures were still ongoing at the time of the evaluation. There were difficulties in determining the survey sampling and the response rate is relatively low.
Strengthen Support Centres through partnerships between the different actors involved. Improve communication to better reach potential social entrepreneurs, financial institutions, and vulnerable groups. Strengthen the participation of organisations belonging to the social and solidarity economy by supporting systemic actions.
Evaluation of support for improving the quality of public social services under the Warmińsko-Mazurskie OP, 2014-2020
Ocena wpływu RPO WiM 2014-2020 na zwiększenie dostępności i poprawę jakości usług publicznych województwa warmińsko-mazurskiego
ROP Warmińsko-Mazurskie Voivodeship 2014-2020
The evaluation examines how the support from the OP has improved the quality and availability of social services.
To evaluate the existing and potential impact of the support on increasing the accessibility and quality of public social services in the region.
Desk research.Analysis of monitoring and statistical data.Interviews with representatives of institutions, experts evaluating projects and beneficiaries.Surveys of 115 local governments representatives.Surveys of 229 beneficiary entities and 156 non-supported entities.Counterfactual methods (DiD).Review of projects selected for funding.Cartographic analysis.4 case studies.Benchmarking.Advanced qualitative data analysis through digital tools.
Projects to improve healthcare infrastructure led to a significant increase in the quality and efficiency of healthcare services, especially in relation to the accessibility of specialised healthcare services.Projects substantially reduced social and spatial barriers to accessing social services, especially for socially excluded groups.The measures helped improve the skills and qualifications of students and the alignment of education with labour market needs.The strategic investments in infrastructure and equipment contributed to strengthening the healthcare system resilience to health crises.
Large differences in the number of supported and non-supported municipalities may have distorted the results of the survey and made it difficult to compare the results on the accessibility of social services.
Continue investing in healthcare infrastructure and equipment to further enhance service quality and accessibility.Expand and refine projects targeting social inclusion to further reduce barriers and improve access to services.Continue aligning the educational offer with market needs to address demographic changes and labour market requirements.
Evaluation of the design and implementation of the Kriti OP, 2014-2020
Αξιολόγηση του σχεδιασμού και της υλοποίησης του ΕΠ Κρήτη 2014-2020 / Προετοιμασία της Αναθεώρησης του ΕΠ - 1ο παραδοτέο
Aristotelous 11-15, 10432, Athens, Greece, http://www.lkn.gr/
The evaluation examines the implementation of the OP, 2014-2020, up to 30/9/2017.
To assess the effectiveness and efficiency of the OP, the achievement of the milestones, and the validity of the intervention logic and to make recommendations for revision.
- Desk research.- Analysis of monitoring and statistical data.
PA3 seems most effective, PA4 least, while PA1 is still to be started.PA2 is effective as regards specialisation, calls for proposal and contracts, while PA5 is effective as regards contracts.
Re-assess, add or remove some of the indicators used.Remove two Intervention Codes and reallocate their budget and add a new Intervention Code.
Evaluation of the "MOVE” initiatives for language training for young people, under the Veneto ESF OP, 2014-2020
Gli effetti del bando “Move for the Future” sui processi di transizione al lavoro
IRS, www.irsonline.it - PTSCLAS, www.ptsclas.com - GRETA, www.greta.it
The evaluation examines 4 measures for language training of young people to support the internationalisation of schools under the Veneto ESF OP.
To assess the relevance of the measures in relation to the need for language skills in the region and the results achieved.
Statistical analysis of monitoring and official statistical data (Eurobarometer, Eurostat, Istat, Veneto Lavoro and Miur).Desk analysis of research studies.
The investment in language education is strategic to the support of Italian companies, to encourage the professional development of young people and to create better job opportunities.Support for language training amounted to EUR 22.9 million and involved 8,800 students, 60% of them women. The areas with the most demand were those in which SMEs have a strong foreign involvement.
Promote greater territorial balance and involve more areas with little demand for support.Devote a minimum of hours to the teaching of technical language.Strengthen communication activities to disseminate the results more widely and ensure more visibility of ESF opportunities.
Social Scientific monitoring of the Saarland OP
Institut für Sozialpädagogische Forschung Mainz e.V.
This is the 2018 evaluation report on the implementation of the Saarland ESF OP 2014-2020, updating the 2017 report and focusing on three measures: “Education now” (ABJ); “Pre-employment measures”; “National Integration Support (LIB)”.
To examine the implementation and progress of the OP, in order to assess its effectiveness and identify any adjustments needed to meet its goals.
Analysis of relevant policy documents;interviews with funding recipients;interviews with intermediaries implementing the measures;analysis of monitoring data.
“Education now” (ABJ)The measure has been running since 1997 and is continuously adapting to the evolving socio-economic situation. Between 2014 and 2017, 1,700 people participated, 68% in training.Demand for support is high due to the rising number of migrants, for whom an ABJ Refugee module was created, though the 300 training places a year established is not enough to meet the needs of the labour market and the number will be increased.Participants can choose the training sector and parents, employment centres and schools cooperate to define the best pathway.“Pre-employment measures”In the 2014-2017 period, there were 4,967 participants, 38% of them women. 89% of participants have a low education (ISCED 1 or 2), so the approach needs to be tailored to their specific characteristics.45% of participants under 25 completed the programme as against a target of 40%.“National integration support” (LIB)The measure is targeted at long-term unemployed of 25 and over with a migrant background.The main challenge is to overcome the reluctance of companies to recruit migrant workers.
The report highlights the need for both the soft and hard skills of participants to be improved and for pathways to be tailored to individual needs.
Evaluation of the “Chance Zukunft” pilot project under the Nordrhein-Westfalen OP, 2014-2020
Statistische Analyse der IC3- und ICF-Daten im Modellprojekt „Chance Zukunft“
Landesbetrieb Information und Technik Nordrhein-Westfalen
Mauerstraße 51, 40476 Düsseldorf (DE), www.it.nrw
The evaluation assesses the effectiveness of the 'Chance Zukunft' pilot project under the Nordrhein-Westfalen ESF OP 2014-2020 aimed specifically at young people (under 30) with socio-economic and health problems to help them into education or employment.
To assess the effectiveness of the project through analysis of IC3 and ICF data.
Multivariate analysis;survey of participants with data collected every three months.Data used relate to IC3 (model developed by Peter Pantuček to measure level of social inclusion) and ICF (International Classification of Functioning of the World health Organisation), as well as ABBA data, collected by the Vocational Training Centres.
Most of the participants who abandoned the project in the first three months (15% of the total) had a very low social inclusion rate.Statistically significant improvements are evident in almost all areas of the IC3-data (education, labour market, health and medical care). Data for 20% of participants improved by the end of the measure. Improvements in education and the labour market were evident for over a third of participants.Improvements are evident in the ICF data throughout the project for 25% of participants with some difference between men and women, especially as regards self-confidence (for 30%), drive (28%) increase in professional capacity (33%), self-employment (26%), teamwork (26%) and taking responsibility (26%).Longer participation is associated with a higher chance of improvement in education, health and personal competences.
Since data for non-participants were not available, it was not possible to carry out a counterfactual analysis.
Evaluation of the project selection system and criteria under the European Funds for Eastern Poland, 2021-2027
Ewaluacja kryteriów i systemu wyboru projektów programu Fundusze Europejskie dla Polski Wschodniej 2021-2027
The evaluation assesses the project selection system defined for the Programme. Data used go up to November 2024.
To assess whether the selection criteria and the overall project selection system are adequate to achieve the programme's objectives and provide recommendations to optimise the selection process.
Use of Theory-Based approach for each action. Analysis of monitoring data. Quantitative surveys (CAWI): 540 responses collected from successful and unsuccessful applicants. 7 in-depth interviews with representatives of the MA and IB, 25 interviews with members of project assessment committees, 16 interviews with applicants. 5 Focus Groups with experts and representatives from committees and authorities. Analysis of relevant documents (Programme and call documentation, regulations, selection criteria and European legislation).
The selection system and criteria adopted were found to be transparent, detailed, adequate for the objectives of the programme, and sufficiently selective, enabling the choice of high-quality projects. Difficulties emerged in understanding and applying horizontal principles, such as sustainability and the "Do No Significant Harm (DNSH)" principle, both for beneficiaries and evaluators. Some discrepancies and inaccuracies in call documentation, such as definitions of "design process" or "green-blue infrastructure," created ambiguity in the requirements for beneficiaries. The IT system showed limitations, including the lack of functionality for direct communication with beneficiaries and restrictions in application forms (e.g., character limits), which required additional clarifications. The application process was simplified and the administrative burden on beneficiaries reduced, while the two-stage system for innovative projects improved the quality of applications. Some recommendations have already been implemented in subsequent calls, such as more detailed documentation, additional training on horizontal principles, and the simplification of application forms.
The evaluation was conducted shortly after the first calls, which limited the ability to collect comprehensive data on the implementation and results of projects.
Provide more detailed guidelines and training tools to facilitate the understanding and application of horizontal principles. Improve the definition of key concepts such as "design process" and "green-blue infrastructure" to avoid interpretative ambiguities and ensure consistency in the documentation. Optimise IT systems by introducing tools for direct communication with beneficiaries and simplifying access to project documentation. Adjust selection criteria to increase participation in calls and make the process more inclusive and appealing to a wider range of beneficiaries. Simplify forms and document requirements and strengthen training and capacity-building support for beneficiaries.
Evaluation of support for collaborative RTDI networks under the Veneto ERDF OP, 2014-2020
https://www.lattanziokibs.com - https://www.irisricerche.it/
The evaluation examines the effects of support for networks of enterprises for collaborative RTDI activities under the OP up to July 2021.
To assess the characteristics of the enterprises and of the projects supported and their effects up to July 2021.
Monitoring data.Official statistics.Interviews with MA.Survey of beneficiaries and non-beneficiaries for counterfactual analysis (PSM).
Most of the companies supported are small/micro sized, in manufacturing and professional, scientific activities.The MA is satisfied with the projects supported, for which quality networks have been created.Companies supported show a better performance in terms of employment and revenue growth than a control group, though a worse one in terms of innovation and RTDI employees.However, counterfactual analysis results are not statistically significant.
The time elapsed from the implementation of the measures is too little to fully capture their effects. The number of companies included in the counterfactual analysis is too small to obtain statistically significant results.
Plan in advance the use of official dataset for evaluation purposes and define the monitoring system taking evaluation into account.Promote an evaluation culture that includes companies in the discussion of the results achieved by measures.
Evaluation of achievements and impact of employment measures under the Human Resources OP, 2014-2020, in Slovakia
Hodnotenie plnenia stanovených cieľov a dopadov prioritnej osi Zamestnanosť
Jela Tvrdoňová, Marek Pihulič, Matej Smieško
Octigon, https://octigon.sk/en - Projektové služby, http://www.pseu.sk/01/index.php - RADELA, http://www.radela.sk
The evaluation examines the achievements of employment measures under IP 8.i (access to employment for job-seekers), 8.iv (gender equality in employment) and 8.vii (modernisation of labour market institutions) of the OP, up to end-2020.
To assess the achievement of the objectives of the measures, their results, and the causal links between them and the projects implemented.
Monitoring data.Official statistics.Interviews with MA.
Measures under IP8.i have been overall effective, partly contributing to long-term employability outcomes, despite the improved situation on the labour market.Measures have been more effective for job-seekers with disabilities, and less so for long-term, unskilled unemployed, and those aged 50+.Measures under IPs 8.iv made little contribution to the objectives, because of their limited use and effectiveness, while measures under IP 8.vii have been relatively effective.
Some of measures were difficult to assess because of their limited use.
Ensure that measures to improve employability are flexible enough to adapt to labour market needs.Strengthen measures for gender equality in employment.Develop the capacity of public employment services to provide an individual approach to job-seekers and to be able to build partnerships.The three priorities should be considered as different PAs in the future OP.
Evaluation of the SYPO project on the development of teachers’ competences financed by the Research, Development and Education OP, 2014-2020, in the Czech Republic - Progress Report II
Evaluace systémového projektu „SYPO“ - Průběžná zpráva 2
The evaluation examines the SYPO project supporting the professional development of teachers and school directors. The project started in January 2018 and it will be completed by June 2023. The analysis was conducted from August to December 2021.
To assess at the end of 2021 the implementation of the project, which lasts from January 2018 to June 2023, on the training of teachers and school directors, the output, the achievement of objectives, the obstacles to implementation and the unintended effects.
Analysis of relevant documents, previous evaluations and studies.Interviews with project representatives.Survey of 837 participants (average response rate: 27%).Focus group and interviews with participants in project activities.
The implementation of the project is in line with what was planned and 13 out of 16 objectives have been met.The national and regional groups on the Czech Language and Literature, Mathematics and its Applications, Informatics and ICT are assessed positively and meet the expectations of participants.According to respondents, these groups enable professional competences to be deepened and give the possibility for cooperation, the sharing of knowledge and experience, and new and innovative learning methods to be developed.76% of respondents report that webinars help to ensure the quality of education for teaching staff.The most serious risk affecting implementation is Covid-19.The most common form of collaboration of the project with others is participation in expert panels.The two possible unintended effects are the creation of teaching material for the professional public and training material for methodological cabinets of regional and local bodies.
Focus on the implementation of the Key Activities of the project on quality.Ensure the sustainability of activities after the end of the project.Support the networking of groups in areas where there is less interest and involvement.Strengthen cooperation between national and regional groups.Disseminate more information on project activities and material.
Evaluation of ESF measures supporting employability and the social economy in 2007-2013
A foglalkoztathatóság javításához és a szociális gazdasághoz kapcsolódó intézkedések ex post értékelése
The ex-post evaluation focuses on the measures included in the Hungarian Social Renewal Operational Programme (SROP) for improving employability and developing the social economy.The report provides an overview of national and EU labour market trends and of the EU target system for employment and related policies. The main focus is on assessing the efficiency and effectiveness of the OP through measuring the coherence of the implementation procedures with the objectives of the OP, the involvement of target groups identified by the strategy and the spatial distribution of interventions. The report also examines the use of resources, the extent of achievement of relevant targets, the time required for project selection and implementation and the sustainability of projects.
To assess the effectiveness and efficiency of the measures included in the Social Renewal Operational Programme (SROP) for improving employability and developing the social economy.
Programme monitoring data (EMIR) and national statistics.Personal interviews with officials, telephone interviews (with random sample of beneficiaries).Analysis of previous assessments and documents.Open databases (legal status, date of establishment, financial data of beneficiaries).
Overall the resources available were used and the targets achieved, especially as regards the objective of improving the employability of target groups.EU funds gradually replaced domestic resources undermining the long-term sustainability of development.At the beginning of the programme implementation, in most areas, there was no clear policy strategy to set long-term goals.Synergies between similar instruments and projects were not exploited.The lead time of the projects at the beginning of the period was very long but it was reduced by the end of the period.Measures supporting public employment for job seekers increased considerably and have overtaken active labor market measures.
Long-term sustainability of the projects should be encouraged after funding comes to an end (e.g. by tightening maintenance obligations and by properly designing maintenance plans).Domestic resources should be better integrated with EU resources in order to increase long-term sustainability.Longer project cycles including integrated approaches (e.g. including a mentoring programme) should be promoted.The use of simplified cost methods should be encouraged.
Analysis of measures for collaborative research under the Enterprise and Competitiveness OP, 2014-2020, in Italy
La ricerca collaborativa: analisi degli elementi di contesto ed attuativi delle politiche di sostegno
NOP Enterprise and Competitiveness 2014-2020
Daniele Del Guercio; Gloria Gasperini; Damiano Landolina; Giorgio Marini; Massimo Marnetto
The evaluation examines progress in implementing the measures on collaborative research under the OP.
To provide insights into collaborative research measures that could be used to define evaluative activities and future programming.
Monitoring data.Official statistics.11 case studies.
Most of the projects are carried out by groups of companies and a research organisation.Projects progressing more rapidly are those with a limited number of partners and only one academic institution.Projects are more targeted at product or service innovation than development of existing products or process innovation.Projects are also more concerned with design and development phases than exploitation of innovation.
Include multidimensional indicators on the impact of projects, such as the “Global Innovation Index” and the “Summary Innovation Index” in 2021-2027 calls.Diversify measures in terms of type of innovation (R&D, product innovation, process innovation, …).Strengthen the exploitation phase of projects and include it in project proposals.
Employment and Social Inclusion OP, 2014-2020 – performance analysis
Rapport d'analyse du suivi de la performance du PON FSE
NOP Employment and Social Inclusion ESF 2014-2020
The evaluation assesses the performance of the ESF OP 2014-2020 in France, in terms of the achievement of objectives as indicated by the indicators on the basis of data available at end-August 2016.
To assess the performance of the OP and the achievement of the objectives and to make recommendations for achieving the intermediate objectives.
Data analysis;desk analysis of documentation.
There are difficulties in reaching the target group participants and the main challenge is to include participants who can most benefit from ESF support;the indicators, in particular, those under Axes 1 and 2, are lagging behind the objectives;investment under priority 8.i accounts for more than three-quarters of the achievement of the targets for the two performance framework indicators;there is a significant risk of not achieving the objective because of the lack of completeness and quality of the data.
There is a lack of data of adequate quality.
There is a need to:increase short-term operational activities to advance the level of achievement of the framework indicators and encourage the participation of target groups;adapt the intervention logic of the programme and strengthen certain activities to achieve the objectives;increase data availability and make it as complete as possible.
Survey of participants in vocational training courses financed by YEI and ESF to calculate the employment result indicators in Centre-Val de Loire, 2014-2020
Enquête de suivi à 6 mois des participants aux formations professionnelles cofinancées par l'IEJ et le FSE
The study examines the situation of participants six months after leaving two sets of ESF and YEI-supported measures, one for those under 26 and the other for all ages, the courses being completed between July 2018 and June 2019.
To assess the employment situation of participants in ESF and YEI-financed training 6 months after exiting from the two sets of course and the quality of post-training job offers and employment.
Survey of 1382 participants, of which 725 were young and 657 were of all ages (response rate 49%).
The employment rate after six months was 48% for those under 26 and 41% for those of all ages.22% of participants over 54 were employed.Employment among women was less than for men in both cases.Young participants obtained better quality job offers than others.Men generally received better quality job offers than women.More qualified participants had better results than low qualified.
Comparison with surveys carried out in previous years is not always possible because the characteristics of participants in measures vary considerably from year to year, and some adjustment was required to ensure the representativeness of the sample.
Ex-ante study of the Lisbon Regional ERDF/ESF+ Programme, 2021-2027
Avaliação Ex Ante e Avaliação Ambiental Estratégica do Programa Operacional Regional de Lisboa 2021-2027
Augusto Medina, Isabel Morais, Hugo Magalhães, Leonel Ferreira, Sérgio Alves
Sociedade Portuguesa de Inovação, https://www.spi.pt/en/
The study aims to assess ex-ante the Regional Programme of Lisbon. Data used go up to September 2020.
To assess the relevance, coherence and complementarity of the measures, and the adequacy of financial, human, and organisational resources. To ensure the integration of environmental aspects into the programme and to establish foundation for monitoring and impact assessment.
Document analysis of policy and strategic documents.Analysis of monitoring data.Analysis of previous evaluation studies.10 stakeholder interviews, including national agencies and local authorities.4 online thematic workshops with experts on innovation, climate transition, inclusion, and territorial development.Use of Theory of Programme.
The programme is aligned with the needs identified, although some pieces of information date to 2019 and are not up-to-date.The measures are consistent with each other and with strategies at the regional and EU level, focusing on innovation, sustainability, and social inclusion.The programming theory is clear overall but needs improvement in relation to the clarity and robustness of the indicators selected.Financial and human resources are allocated coherently with the priorities, and the monitoring and evaluation system is adequate.Risks such as low participation or poor project implementation have been identified, and there is a need to strengthen the involvement of stakeholders.
Due to the restrictions imposed by the regulatory framework, it was difficult to provide complete and detailed justifications for all strategic decisions.
Improve diagnostic by using up-to-date data and ensuring consistency across all axes.Refine the Programme Theory by adding performance indicators to strengthen causal links.Enhance monitoring and evaluation by using clearer indicators and reliable data collection systems.Simplify project implementation through easier application procedures and better coordination.
Evaluation of measures to improve institutional capacity under the Effective Public Administration OP, 2014-2020, in Slovakia
Ongoing evaluation of the efficiency, material and financial efficiency and impacts of OP EVS in the area of strengthened institutional capacities and effective VS (PO1)
OP Effective Public Administration 2014-2020
consulting@centire.com; https://www.acceptaudit.sk/
This mid-term evaluation focuses on PA1 of the OP (‘Strengthening institutional capacity and effective public administration’). It complements a previous evaluation of all PAs published in November 2022. The period covered by this evaluation is 2017-2021.
To assess whether the specific objectives set under PA1 have been achieved focusing on the period 2017-2021.
Desk research.Survey of public institutions and NGOs (response rate: 45%).Analysis of monitoring data.Analysis of statistical data.Analysis of four national projects.
The implementation of measures aimed at optimising the regulatory environment led to the successful introduction of a functional system for assessing the impact of public policies.The OP supported the enforcement of new legislation regulating the functioning of the civil service and the development of the further education system for civil servants.The OP contributed to strengthening the capacity of analytical units in public administration.The OP contributed to improving public procurement in terms of transparency, preventing corruption, and green and social aspects.Continuous education of public administration personnel and paying appropriate salaries contributed to improving the quality of the services.
Implementation of measures supporting a more favourable business environment (through the streamlining of tax collection) had not started and some others were not completed at the time of the evaluation.
Continue supporting the development of e-services.Further develop the analytical capacity of public administration.Systematically support participation in the development, implementation and control of public policies.Ensure relevant remuneration for public administration officials in order to prevent them from moving to the private sector.Limit bureaucracy in the provision of services.Improve communication on results and best practice.
Ex-ante study on the use of financial instruments under the Scotland ERDF OP, 2014-2020
European Structural Investment Funds European Regional Development Fund 2014-2020 Ex Ante Assessment of Financial Instruments
The study examines the feasibility of using financial instruments (FIs) under the OP.
To assess market failures, consistency of FIs with other forms of support, leverage of FIs and expected results.
Official statistics.Monitoring data on Loan funds.Findings from previous evaluations and studies.
There is a potential financing gap of £470 mn (EUR 540 mn) annually in Scotland, with a possible 4,500 viable SMEs unable to secure finance.The financing gap in the Scottish risk capital market is estimated to be £35mn annually (EUR 40mn).FIs do not seem to be suitable to fund ICT projects.Further research is needed to establish the investment gap as regards the low carbon economy.
Allocate £40mn of ERDF funding to establish a dedicated SME Holding Fund which could then invest directly in a small number of debt and equity funds.
Evaluation of the impact of PA 3 "Reduction of CO2 Emissions" under the Niedersachsen ERDF/ESF OP, 2014-2020 - phase 2
Evaluationsbericht zur Prioritätsachse 3 Reduzierung der CO2-Emissionen Bericht zu Phase 2 – Wirkungsanalyse
Eul Clara, Von Mach Maria, Schwab Oliver
This report examines the results of the measures to reduce CO2 emissions up to September 2020 by improving energy efficiency and reducing energy consumption by public building and transport mobility.
To assess the preliminary effects of the measures to increase energy efficiency and the use of renewables and CO2 capture up to September 2020.
Desk analysis of official statistics and monitoring data.Interviews with beneficiaries and experts.Case studies.Theory-based approach (contribution analysis).
Up to September 2020, a total of 296 projects had been approved with total funding of EUR 282 million (81% of the budget).Energy efficiency measures account for two-thirds of over 62,000 tonnes of emissions saved a year.The total emissions saved amount to 1.6% of the emission reduction targeted.Support led to an average reduction in emissions of 368 tonnes per year per EUR 1 million with a range of between 258 and 946 depending on the measure.There are difficulties in involving companies in energy saving measures for corporate networks, while urban wastewater plants make a large contribution to reducing emissions.Implementation of the most innovative measures for heating is at risk; an indirect effects of the measures is a reduction in transport costs.In several cases the quality of planning has proved to be problematic, resulting in delays, additional costs and reduced effects. Personnel capacity was a bottleneck.
Only a small number of projects were completed, so the evaluation mainly assessed the potential impacts. Monitoring data cannot be easily interpreted.
Improve the monitoring system, especially by increasing the level of detail provided on interventions.Strengthen the measures on the wetting of peat bogs, rehabilitation of buildings and renewal of sewerage systems.
Evaluation of measures promoting research and innovation under the Competitiveness OP, 2014-2020, in Romania – 3rd report
Evaluarea intervențiilor PO Competitivitate în domeniul cercetării-dezvoltării-inovării (Raport 3)
Speranța Pîrciog, Adriana Ana-Maria Davidescu
Institutul Național de Cercetare Științifică în Domeniul Muncii și Protecției Sociale, https://new.incsmps.ro/en/homepage/; ERNST & YOUNG SRL
The evaluation focusses on the results and impact of projects implemented under the OP, which are aimed at research, technological development and innovation (RTDI) in support of economic competitiveness and business development. Monitoring data are updated as at 31 March 2023.
To assess the results and impact of projects to boost research and innovation, as well as their sustainability, and to draw lessons for the 2021-2027 programming period.
Analysis of relevant documents. Analysis of monitoring and statistical data. ToC elaboration and testing. Counterfactual impact evaluation (DID method). Surveys of beneficiaries. Case studies. Focus group. Cost-effectiveness analysis. Network analysis.
The main results of the projects supported are the creation of new products, the increase in the use of new technology and in the number of scientific articles and publications, and the improved collaboration between stakeholders of the RTDI sector. Measures to increase scientific capacity in the areas of smart specialisation and health resulted in a net positive effect on turnover, profit and average number of employees, in beneficiary companies and research institutes. The effects on profit are sustainable and are reported irrespective of the type of support provided. Projects had a positive short-term impact on the increase of private expenditure on RTDI one year after the intervention, but in the medium term (up to 4 years) the effect is negative. Measures to increase private investment in R&D did not led to any increase in the number of employees and their impact was visible only in the first year immediately after implementation. The contribution of the projects on increasing innovation capacity is limited and there is a risk that policy priorities are not aligned with the changing needs of the RTDI sector. The progress in the implementation of the OP is smooth and targets planned are likely to be achieved. According to the network analysis, there is no improvement over time in terms of effectiveness of knowledge transfer.
At the cut-off date, a large number of projects were still on-going and the programme indicators are only validated at the end of project implementation. The estimation of the effects in the medium term is only partly feasible.
Align the legislative and institutional framework with the objective of increasing scientific capacities and ensure better strategic programming and planning of financial resources. In the future, allocate more funds to the RTDI sector, also to ensure better access to Horizon 2020 funds. Reduce administrative burden and decrease timing necessary to complete tender procedures.
Evaluation of projects on innovative companies supported by the Ile-de-France ERDF-ESF OP, 2014-2020
Evaluation des projets soutenus par le Programme Opérationnel Régional Ile-de-France FEDER-FSE 2014/2020 sur la création et le développement des entreprises innovantes
OP Île-de-France et Seine ERDF-ESF 2014-2020
This is the final evaluation report on the measures for innovation in enterprises under PAs 6 and 7 (TOs 1, 2, 3) of the OP, up to end-2020.
To assess the results and impact of the projects supporting innovative enterprises up to the end of 2020.
Documentation analysis.Analysis of monitoring data.5 case studies.Interviews with enterprises and policy representatives.Survey of 44 project managers (21 respondents – 41%).Survey of a sample of beneficiaries of two financial instruments (FIs).
The projects are coherent with other regional policies promoting R&D, ICT and competitiveness of SMEs.Covid-19 outbreak caused several delays in the implementation of projects, and only 49 projects out of 174 have started.62% of funds are allocated to support of enterprises; 52% focus on digitalisation and are provided through three FIs (over 3,000 enterprises supported).The projects helped increase cooperation between SMEs and higher education institutes/research centres, expand innovation capacity and create innovative products.In start-ups, projects promoted new job opportunities and helped increase turnover (by over EUR 1 million after 3 years).
Results and impact presented are based on the perception of beneficiaries and cannot be generalised because of the small sample covered.
Ensure future support to R&D projects in enterprises, especially those coherent with other regional policies.Monitor project trajectories to assess their effectiveness in a timely way.
Evaluation of the impact of continuing vocational training supported by the Veneto ESF OP, 2014-2020
Istituto per la Ricerca Sociale, IRS; PTSCLAS; Gruppi di Ricerca Economica Teorica ed Applicata, GRETA
IRS, www.irsonline.it; PTSCLAS, www.ptsclas.it; GRETA, www.greta.it
The evaluation examines measures funded under IP 8.v of the Veneto ESF OP 2014-2020 and, specifically, six calls financing training for enterprises and workers.
To examine the characteristics of workers and enterprises supported and to assess the results and potential impact of the training measures concerned.
Desk and data analysis.Survey of enterprises (response rate: 5.3%).Macroeconomic model GREM-VE to estimate the impact.
27,219 workers were involved in measures, 64% of them men, half with a post-secondary degree.57% of the 8,062 enterprises supported were micro-sized, disproportionately in manufacturing.According to the macroeconomic model, in the scenario with the continuing vocational training, the employment rate would be 0.1% a year higher than without training, by 0.08% per year from 2019 to 2023. Value-added would be unchanged.The training courses concerned mostly Marketing and communication (48%) and Internationalisation (35%).Only 5% of respondents reported that the projects were not effective. 70% reported that they had positive results, mainly in terms of creating a better working environment (70% of cases) and reducing production costs (25% of cases).
It is underlined that a response rate of 10%, instead of the 5% reached, would have been more effective in terms of the representativeness of the survey.
There is a need to (according to respondents):reduce bureaucracy;make the training more practical;focus especially on the micro-enterprises;consider more the needs of the self-employed.There is also a need to collect more detailed data for evaluation.
Evaluation of measures contributing to protection against climate change under the Nordrhein-Westfalen ERDF OP 2014-2020
Evaluierung des OP EFRE NRW 2014-2020 - Beitrag der Klimaschutzwettbewerbe zur Senkung des Treibhausgasausstoßes
Dr. Stephan Heinrich, Mirjam Buck, Alexander Labinsky, Nora Lengreder, Christoph Thormeyer, Karsten Weinert, Paul Wendring
Prognos AG: St. Alban-Vorstadt 24, 4052 Basel; Tel.: +41 61 32 73 362, E-Mail: stephan.heinrich@prognos.com
The evaluation examines the implementation of 5 measures to reduce CO2 emissions (such as reducing energy consumption in SMEs, reducing energy use and increasing the use of renewables and developing new technologies) supported under PA3 of the OP in 2014-2017.
To assess the contribution of 5 measures (e.g. for reducing energy use and increasing the use of renewables) to reducing CO2 emissions, the innovative processes put in place, the success factors of the projects carried out and the extent of synergies between measures.
Desk review of documentation.Analysis of monitoring data.Interviews with experts.Online survey of beneficiaries (89 with 51% response rate).10 case studies.
Successes so far include the innovative nature of projects, the economic potential expected for SMEs supported and creation of synergies between projects.Innovative aspects include the development of new approaches to using renewables and the development of new technology.As of end-2017, indicators for energy saving and reduction in CO2 emissions were still below targets, though the number of companies supported was in line with the target.Because of the innovative nature of projects, savings are relatively limited and occur only in the medium-to-long-term.The two-stage procedure for assessing projects, though implemented successfully, caused some delay.There are difficulties in terms of the availability of the technology required on the market and encouraging the use of smart grids.
94% of projects were not completed at the time of the evaluation (end December 2017) making it more complicated to assess the achievement of targets and the effects.
Integrate climate protection measures under a single category.Ensure that results of pilot projects are disseminated so as to stimulate innovative projects and activities.Consider modifying the objectives and indicators of projects.
Evaluation of cross-cutting aspects of measures financed by the ESI Funds in Bolzano, 2014-2020
Rapporto di valutazione degli effetti trasversali delle misure finanziate grazie ai fondi SIE in riferimento agli ambiti strategici per l’Alto Adige - 2020 report
ROP PA Bolzano ESF 2014-2020; ROP PA Bolzano ERDF 2014-2020
The evaluation examines the contribution of the Bolzano OP to the regional development strategy.
To assess the contribution of the OP to the regional development strategy and other programmes.
Monitoring data.Interviews with MAs and relevant stakeholders.1 case study on a beneficiary of ERDF and ESF interventions.
The main contribution of the OP to the regional development strategy comes from ERDF measures on R&D, including e-government, energy efficiency and disaster resilience.Coordination between OPs is ensured by information exchange, participation in Monitoring Committees, and meetings between staff of different MAs.There are complementarities between the ERDF and EAFRD OPs but none between the ERDF and ESF at the measure level.
Continue measures for collaboration between R&D centres and business.Ensure broadband access in rural areas.Supplement ERDF measures on e-government with ESF interventions to develop digital competences.Continue measures for sustainable mobility.
Evaluation of the implementation of the Italy-France (Maritime) Interreg OP, 2014-2020
Valutazione dell’implementazione del programma Italia Francia Marittimo - Aggiornamento 2020
Interreg V-A-Italy-France (Maritime) 2014-2020
The evaluation examines the progress of the OP and implementation aspects.
To assess the progress of the OP up to 2019, the planning approach, management rules and the indicator system.
Monitoring data.Interviews with MA.Survey of beneficiaries (response rate around 30%).
As at end-2019, only PA1 (TO3) seems fairly advanced and the others lag behind.The administrative capacity of the MA is sufficient: improvements have been introduced in the information system and communication activities.According to beneficiaries, there are still problems with the information system, the duration of project selection procedures and reporting expenditure.The effects of measures seem more evident as regards innovation and the exchange of good practice, less so for labour market measures.
Ensure flexibility of simplified cost options.
Many faces of administration – evaluation of the Sustainable Growth and Jobs OP
Hallinnon monet kasvot. Kestävää kasvua ja työtä rakennerahasto-ohjelman toimeenpanon ja hallinnon arviointi.
OP Sustainable growth and jobs 2014-2020
Elina Auri (MDI), Timo Hirvonen (Spatia), Petri Kahila (Spatia) and Tommi Ranta (MDI)
Elina Auri, elina.auri@mdi.fi, Petri Kahila, petri.kahila@uef.fi
This is the first interim report on the implementation and management of the Finnish Sustainable growth and employment OP 2014-2020.
To examine the implementation of the OP and the management and implementation of projects.
Analysis of relevant policy documents.Online surveys of Public Authorities and beneficiaries of funding.Interviews with a sample of stakeholders.Case studies.
The implementation of the OP varies according to the regional management models, each of which has positive elements.The monitoring system is fed by local authorities, causing a lack of information or a delay in updating data.The MA and the local authorities implementing the OP can positively affect the achievement of goals if there is effective coordination between them.Interventions seem to be cost effective, despite differences between regions. The timing of disbursement of funding has halved compared to 2007-2013.All projects have taken account of horizontal principles, but it is sometimes hard to understand how they contribute to gender equality, sustainability or non-discrimination.
The evaluation uses data on projects from the monitoring system, which are updated with a 7-month delay.
To increase the quality of management and implementation of the interventions, it would be useful to set a unified model.Closer contact between the MA and beneficiaries would help to improve the quality of projects.Cross-regional projects should be increased to promote learning between regions and achieve scale economies.The monitoring system should be simplified and its indicators more easily understandable by stakeholders.Horizontal principles should be spelled out more clearly at central level and be recognisable by beneficiaries and local authorities.
Evaluation of support for SMEs under PA3 of the Toscana ERDF OP, 2014-2020
Rapporto di valutazione tematica “A.3. Analisi degli effetti degli interventi relativi alla promozione della competitività delle PMI realizzati prevalentemente nell'ambito dell’Asse 3”
Ecoter, http://ecoter.it/, info@ecoter.it
This evaluation examines initiatives (Action 3.3.2) for the promotion of tourism, implemented by the “Toscana Promozione Turistica” (TPT) agency over the period 2017 to 2019.
To assess the results of measures to promote tourism and their coherence with other ERDF measures on tourism, the regional policy framework and the Smart Specialisation Strategy (S3).
Desk review of relevant documents.Analysis of monitoring data.Interviews with stakeholders, mostly officials of the Regional Authority.
At the end of 2019 all the resources available (EUR 10 million) were allocated and 80% were spent.The measures are coherent with the policy framework and other ERDF measures supporting tourism.The initiatives seem suited to improving the sustainability of regional tourism, but not to achieving greater gender equality, increased use of clean energy and a reduction in emissions.
The interviews capture the opinions of stakeholders which are strongly influenced by the tourism crisis caused by Covid-19, though the measures examined were financed before the pandemic.
Continue financing the initiatives to promote tourism launched before the pandemic and which have demonstrated to work well.Make a greater effort, in 2021-2027, to support innovation in tourism to increase its resilience, energy efficiency and gender equality and to reduce emissions and seasonality.
Evaluation of the effects of projects on the low-carbon economy under the Łódzkie OP, 2014-2020
Ocena wpływu inwestycji podejmowanych w ramach OP IV "Gospodarka niskoemisyjna" RPO WŁ 2014-2020
Oliwia Mazan, Joanna Jaworska, Patrycja Klimek, Maciej Maj
The evaluation examines impact of support from the OP on the development of the low-carbon economy.
To assess impact of projects on improving energy efficiency and the state of the environment in the region.
Desk research.Survey of 364 beneficiaries.14 interviews with beneficiaries and bodies involved in implementation.8 case studies.Recommendation workshop.
Projects have contributed to achieving specific OP objectives, particularly for the thermal modernisation of public and residential buildings.The reduction of greenhouse gas emissions has been significant, especially through the use of renewables.Extensive application documentation has been a burden for applicants and the bodies involved in implementation.Destabilising socio-economic factors have adversely affected project implementation.
There were problems in completing the planned number of surveys.
Continue supporting investment to increase the energy efficiency of buildings.Maintain support for the development of renewables under the regional programme.Promote the use of financial instruments more.Simplify the procedures for applying for support and maintain a flexible approach to project implementation.
Evaluation of the Slovenia-Hungary Cross-Border Cooperation OP 2007–2013
Evaluation of the Operational Programme Cross-Border Cooperatio Slovenia-Hungary 2007–2013
Karin Žvokelj Jazbinšek, M.Sc., Raskó Géza, M.Sc., Matevž Premelč
MK projekt, družba za svetovanje in vodenje projektov, d. o. o., Rogaška cesta 25, 3240 Šmarje pri Jelšah
This evaluation assesses the Slovenia-Hungary OP 2014–2020, focussing on project implementation and financial indicators at project and OP level. Communication activities are also assessed.
To assess the results achieved by the OP and related indicators.To recommend improvements in the implementation of the Programme.
Desk analysis.Meetings and interviews with programme stakeholders.Statistical data analysis.Questionnaires to the OP bodies.Data from the Hungarian Central Statistical Office (HCSO), STADAT.Data from SI-STAT, AJPES, UMAR, DI (Direkcija RS za infrastrukturo), ARSO.
There was an improvement in the tourist ‘offer’ in the area. Infrastructure was developed along with joint marketing strategies. Coordination with cultural activities and the natural heritage was also strengthened.Small-scale restoration of the cultural heritage was undertaken, networks were set up between local media providers, cooperation between cultural institutions was established as well as between schools and joint events were organised.Preservation of biodiversity was supported, joint strategies for risk prevention were developed, public awareness of national parks was raised and plans were developed for the joint use of water reserves.Employment services were supported through training and research activities. Strategies for further development of the region were also formulated.
There were a large number of unfinished projects.Some of the relevant bodies (e.g. the Audit Authority) did not reply to the questionnaire.A number of interviews with beneficiaries could not be conducted as project managers or coordinators were no longer in the same job.Data cover only the period up to December 2014.
To define the next SI-HU CBC programme objectives so that they reflect the real needs of the cross-border region. Fewer and more focused objectives should be defined.To give greater emphasis to output and result indicators in respect of the events organised by the JTS in the preparation stage of projects.To include a table showing links between objectives and indicators in the Communication Plan. An effective system for monitoring the expenditure of funding and the cost efficiency of the activities carried out should be set up when the Plan is implemented.To improve cooperation between the various OP bodies. The JTS could be more effective in its work and develop innovative approaches and the means of facilitating the effective collection and processing of project data so as to avoid duplication of work.
Evaluation of ITIs for Urban development under the Sardegna ERDF and ESF OPs, 2014-2020
Valutazione investimenti urbani integrati
ROP Sardegna ESF 2014-2020; ROP Sardegna ERDF 2014-2020
The evaluation examines three ITIs (Cagliari, Sassari, and Olbia) in the period up to September 2021.
To identify what has worked and weaker aspects and how local bodies have dealt with the complexity of the ITIs and to make suggestions for the future.
Analysis of relevant documentation.Analysis of monitoring data.Focus groups with various people involved in the design and implementation of the ITIs.
As of 30 September 2021, all ITIs have serious delays in implementation, particularly in respect of ESF measures.Governance-related problematic aspects are: a too rigid top-down model, differences in the management procedures of the three public authorities involved and the heavy responsibility assigned to the IB.There is a lack of political commitment and clear institutional leadership.The participatory process did not work satisfactorily.Other difficulties concern the regulatory complexity of the ESF and ERDF, lack of experience of local authorities, tender procedures, and the complex articulation of ITI strategies.
Rethink some governance aspects, particularly the role of participatory procedures, the relationship between political and technical aspects and the role of the IB body.Identify the skills needed by local authorities.
Final evaluation of the Öresund-Kattegat-Skagerrak Interreg OP, 2014–2020
Utvärdering av Interreg Öresund-Kattegat-Skagerrak 2014–2020 - Slutrapport
Interreg V-A-Sweden-Denmark-Norway (Öresund-Kattegat-Skagerrak) 2014-2020
Ylva Grauers Berggren, Roe Langaas, Thomas Westerberg, Angelica Åström, Jennie Persson
Oxford Research AB, www.oxfordresearch.se, ylva.grauers@oxfordresearch.se, roe.langaas@oxfordresearch.se
The evaluation assesses the overall results and impact of 85 out of 88 projects funded under the Öresund-Kattegat-Skagerrak Interreg OP, 2014–2020.
To assess the overall level of achievement of the OP, the contribution to the Europe 2020 Strategy and to the Sustainable Development Goals (SDGs) of the UN 2030 Agenda.
Analysis of monitoring data. 903 semi-structured interviews with beneficiaries of 73 projects. Surveys of 137 beneficiaries to assess short and long-term project results (59% response rate). Document analysis of project applications, final reports, and external evaluations. Case studies of four projects (through focus group interviews and document analysis).
The projects achieved 88% of the OP's overall output indicator targets and 85% of the short-term results, with notable differences across the priority areas. Employment Axis exceeded its output targets to a huge extent, while the transport and innovation Axes did not reach the planned output objectives. According to the beneficiaries, the projects have largely achieved their results. The projects implemented are closely aligned with Europe 2020 priorities for smart, sustainable, and inclusive growth. The OP contributed to the SDGs (3, 7, 8, and 11 in particular), but there is a need to better integrate the 2030 Agenda into the programme design.
Some output indicators, particularly concerning employment, may not be fully reliable because of self-reported data and issues posed by the COVID-19. In addition, many projects did not fully report their expected results. Finally, surveys are subject to potential bias as project representatives selected the respondents.
Integrate the 2030 Agenda and SDGs into the programme's design and follow-up. Develop theories of change for the programme as a whole, and for each area of intervention. Ensure that the evaluation of the programme is carried out on an ongoing basis. Involve regional representatives and stakeholders in the validation of the results of the evaluation.
Second intermediate evaluation report on the Valle d'Aosta ERDF OP, 2014-2020
Secondo rapporto di valutazione intermedio
The evaluation examines the implementation of the OP 2014-2020.
To assess the level of the implementation of the OP up to end-2021.
Desk review.Analysis of monitoring data.Interviews with the MA.Surveys of supported enterprises under PA1 (21 replies, 48% response rate) and PA3 (11 replies, 46% response rate).
The OP strategy is still relevant to local needs and has been flexible enough to adapt to the changes caused by the COVID pandemic.Financial and physical implementation is proceeding smoothly; as of December 2021, payments were 83% of the budget and targets set have been largely achieved.The management and control system works, particularly because of the administrative simplification put in place, improvements to the monitoring system, a strengthening of personnel skills through training and an increase in their number. On PA1 (TO1), 62 companies were supported, mainly in respect of product innovation, and cooperation with research centres was reinforced.On PA2 (TO2), 7 agreements have been signed with operators for optical fibre concessions and the creation of a single regional data centre to promote e-services is on-going.On PA3 (TO3), start-up and innovative companies were mainly supported. Under the Cammino Balteo project, businesses were supported partly from PA2 and PA5 funding in order to expand and diversify the tourist services available and to support networking. On PA4 (TO4), the approach used to reduce energy use in public buildings, based on a feasibility analysis and dissemination of good practice, was positively assessed.
Create more synergies between ERDF and ESF to support an increase in qualified professionals and the transfer of knowledge and competences.Involve more public and private research centres.Improve the number of e-services provided by public authorities.Coordinate regional policies better to support local companies. Support the use of renewables more.Integrate measures for protections against earthquakes and energy efficiency.Integrate local support for culture and tourism.
Evaluation of unit costs relating to output indicators of the England ERDF OP, 2014 2020
National Evaluation of English ERDF Programme 2014-20 Unit Cost Analysis Paper
The evaluation examines the unit costs relating to the achievement of key outputs of the OP. Data goes up to July 2022.
To assess the unit costs of selected output indicators covering projects across the various priority axes of the OP.
Analysis of monitoring data (financial and physical progress). Analysis of the distribution of unit costs for all indicators considered. Sectoral and geographic breakdown analysis.
Research-intensive support and support for low-carbon innovation had the highest unit cost per output, while business start-up and digital technology measures had the lowest. The unit cost for an enterprise cooperating with a research entity was lower than in the previous ERDF programme, while the unit cost per enterprise supported to introduce new-to-market products was significantly higher. The unit cost analysis for the indicator on GHG reduction shows a wide variation, with a median cost of approximately EUR 3,923 across 137 projects. Unit costs of newly built or upgraded buildings also show a wide range, from approximately EUR 282 to EUR 8,783 per square metre built or renovated, with a median cost of EUR 1,485 across 13 projects.
Concurrent Evaluation of OP Brandenburg. Final report
Begleitende Evaluation des Operationellen Programms des Landes Brandenburg. Endbericht zur OP-Bewertung
ISW Institut für Strukturpolitik und Wirtschaftsförderung
ISW GmbH, Hoher Weg 3, 06120 Halle (Saale), Tel: +49-345521360, info@isw-institut.de
Evaluation of the proposals for Urban Innovation Projects under the Toscana ERDF OP, 2014-2020
Analisi valutativa delle proposte progettuali presentate a valere sulle procedure di selezione delle strategie per le aree urbane (PIU)
Chiara Agnoletti, Giuseppe Francesco Gori, Caterina Fusi
IRPET - Istituto Regionale Programmazione economica della Toscana, Via Pietro Dazzi 1, 50141 Firenze, Tel.: +39 55 459111
The evaluation concerns the territorial strategies of the Urban Innovation Projects (UIPs) co-funded under Axis 6. “Urban areas” of the Toscana OP 2014-2020.
To verify the effectiveness of the territorial strategies of the Urban Innovation Projects.
The evaluation uses qualitative and quantitative methods:Desk research (of OP documentation, call for tender and 9 selected UIPs);comparative analysis of UIPs;text mining;assessment of the financial sustainability of the projects co-financed (using a specific input-output multi-regional model developed by IRPET);analysis of the effects of UIPs;case study.The data used are mostly secondary data from OP documents, project proposals and the monitoring system.
The study shows a good level of consistency of the UIPs with respect to the OP objectives.UIPs financed interventions aimed at strengthening the cultural offer and participation methods to further the social inclusion and cohesion of the communities involved.The largest number of operations and amount of financial resources are cross-cutting and therefore involve interventions able to affect the quality of life of the population in general.Based on the analysis of well-being in the Municipality of Prato, the interventions supported are coherent with the needs assessed and mostly focused on social inclusion.
The notion of well-being adopted as a monitoring tool (covering 7 domains: the environment, quality of services, social relations, work, innovation, health and safety with a total of 23 indicators) and applied, on an experimental basis to the Municipality of Prato might be of general applicability.
Evaluation of the implementation of energy measures under the European Funds for Infrastructure, Climate, Environment Programme, 2021-2027, in Poland
Ocena programu FEnIKS 2021-2027 dla potrzeb przeglądu śródokresowego w obszarze energetyki: Priorytet I i II (cele szczegółowe 2.1, 2.2 i 2.3)
The evaluation examines the progress in implementing energy sector-related measures (specific objectives 2.1, 2.2 and 2.3) for the purpose of the mid-term review.
To assess the progress in implementing the measures relating to the energy sector, verify the relevance of identified challenges and provide recommendations for improvements.
Analysis of monitoring data and statistical data.13 interviews with implementing institutions and experts.9 interviews with beneficiaries.Survey of 24 project beneficiaries.9 case studies.Expert panel.
By the end of September 2024, 38% of available funds had been contracted and only 7% had been disbursed to beneficiaries.Due to the delay in programme implementation, none of the output indicators will reach their mid-term targets by the end of 2024.Interest in the ‘My Electricity’ programme is very high, so it is possible that the planned allocation will be insufficient.The need to move away from projects that use natural gas (for electricity and heat generation) to those supporting biogas and biomethane was recognised.There is a risk that beneficiaries will be less interested in financial instruments than in grants, which may limit the use of available funds.
Members of the project Appraisal Committee refused to take part in in-depth interviews (no reason given for this).
Adjust the mid-term indicator targets, given the delays in implementation.Move away from support for natural gas-based projects to greater support for other energy sources, including biomethane and biogas facilities.Increase the allocation of the ‘My Electricity’ programme and increasing the relative target values.Strengthen the flexibility of project implementation in order to respond effectively to dynamically changing challenges and needs.Take actions to raise awareness about the available financial instruments and monitor the applicants’ interest, to be able to adjust the allocation of funds.
Evaluation of the “Du kannst was!” project for job inclusion of vulnerable people in Salzburg, under the Employment Austria ESF OP, 2014-2020
Evaluierung der ESF-Umsetzung für Salzburg. Projektbericht "Du kannst was!"
Thomas Lankmayer; Manuela Hiesmair; Karl Niederberger
The evaluation examines the effects of the project up to the end of 2017.
To assess the implementation and effectiveness of the project, its contribution to equality and non-discrimination and the possibility of the measures continuing.
Monitoring data.Administrative data (from PES).Interview with project management.Focus group with trainers.Project documentation.
The main gross employment effects of the project are the increase in the labour force participation rate and in average earnings, while the employment rate is unchanged up to 2 years afterwards.Formal recognition of competences has helped to increase the self-esteem of participants and how they are seen.Success factors are: company commitment to the project, recognition of non-formal skills, the short duration of measures, homogeneity of the target group and group activities that strengthen motivation and support among participants.Critical aspects are: difficulties in reconciling work and family life of participants, limited participation of migrants and women (partly because of a lack of childcare facilities), and the restriction of the target group to the Salzburg area.
The sample analysed after the intervention is too small to capture the actual effects of the project. In addition, the effects observed are only gross ones and it is not possible to establish a causal link with the intervention.
Pay more attention to reaching all the target groups including through the collection of additional data on their background.Expand the territorial area for participants to be eligible.Adjust the range of training activities offered, on the basis of the subject area.Increase the exchange of information on the project between trainers and project managers.
Evaluation of the efficiency and effectiveness of the Attiki OP, 2014-2020. Year 2020
Αξιολόγηση της αποδοτικότητας και αποτελεσματικότητας του ΠΕΠ Αττικής ενόψει της αναθεώρησης του έτους 2020
The evaluation examines the progress in implementing the OP until 2020.
To assess the implementation of the OP in terms of its relevance, coherence, effectiveness, and efficiency and propose amendments for the future.
Analysis of statistical and monitoring data. Analysis of programme documents.
The level of implementation and effectiveness of the OP has significantly improved compared to 2018 (year of the previous evaluation), though progress is uneven across the Tos. Areas showing most progress are ICT, energy, transport, and education, though further investments are necessary. Progress is more limited in relation to measures on change adaptation, circular economy and social inclusion. Important bottlenecks are the willingness of enterprises to invest, delays in planning State aid measures, bureaucracy and, in particular in the area of environment/energy, the incomplete regulatory framework and the limited agreement with local communities.
Adjust the programming and implementation of the OP, to take account of delays in the absorption of financial allocations and the emergence of new needs (especially ICT and COVID-related adjustments).
Evaluation of regional investments under 17 ERDF OPs, 2014-2020, in Germany
Evaluation der EFRE-Programme in der Förderperiode 2014-2020 in Deutschland
GEFRA: Prof. Dr. Gerhard Untiedt (Project manager), Dr. Björn Alecke, Dr. Johannes Burmeister, Prof. Dr. Timo Mitze; Ramboll: Marnie von Fritsch, Carla Harnischfeger, Dr. Lukas Vonnahme; Ifo Dresden: Prof. Dr. Joachim Ragnitz; ZEW: Dr. Christian Rammer
GEFRA, https://www.gefra-muenster.de; Ramboll, www.ramboll.com; Ifo Institut, www.ifo.de; ZEW, www.zew.de
The evaluation examines impact and effectiveness of regional investments under 17 ERDF OPs. Data used go up to September 2024.
To assess the results and impact of ERDF OPs on employment, growth, innovation, competitiveness, climate and environmental protection, and regional development. To provide evidence-based recommendations for improving the design, implementation, monitoring and strategic orientation of programmes in 2021-2027.
Literature and document review, including 125 evaluations of federal states programmes for the 2014-2020 period.Analysis of monitoring data and statistical data.Contribution analysis.Analysis of administrative data from Business History Panel (BHP) and Mannheim Innovation Panel (MIP) for firm-level analyses.Counterfactual impact analysis (PSM+Difference in differences) – using administrative data.Statistical analysis at federal state and district level to assess regional effects.Macroeconomic analysis using the HERMIN model to simulate ERDF’s economy-wide impacts in East Germany.Six focus groups (a total of 29 participants) with federal states MAs.12 interviews with representatives responsible for programme management and implementation.
The ERDF benefited 26,585 firms and had a significant impact on economic growth and employment, and contributed to reducing regional disparities, especially in weaker regions such as East Germany.The funds have supported innovation and development in SMEs, with measurable impacts in terms of productivity and employment growth, particularly in the research and development sector.The counterfactual analysis based on BHP data shows that funded companies experienced on average 10% higher employment increase and 3.9% higher increase in labour productivity, compared to a control group. When only patent-active companies are considered, the effect on employment is slightly lower (8%).In particular, support for SME investment promotion led to an overall 11% higher increase in employment compared to non-supported firms, while the increase is 9.3% higher in the case of support for R&D promotion. However, the latter led to higher increases in R&D employment.The strongest effects were experienced by small firms and tech/knowledge-intensive start-ups.There was a significant disparity in funding between Eastern and Western Germany, with Eastern regions receiving considerably higher funding to reduce the gap with the richest areas.A large volume of funds was allocated to public infrastructure projects, with a particular focus on research, knowledge transfer and sustainability.Long-term positive effects are reported for regions with high funding intensity, particularly in Eastern Germany, where ERDF funding had lasting effects on GDP and employment growth.
Few evaluations were available in some areas (climate change, environmental and nature conservation, sustainable urban and regional development). There were gaps in firm-level information. Some applications of econometric models (e.g. convergence regression) are not robust due to the small sample size and the short observation period.
Allow regions to adjust programmes to local development strategies without lengthy EU approval processes.Improve effective stakeholder involvement from the outset and streamline administrative processes between federal and state levels.Reduce the high administrative burden of regional OPs.Reduce reporting requirements for SMEs and improve monitoring by standardising indicator systems.
Ex-ante study on the use of financial instruments under the Lubuskie ERDF/ESF+ programme, 2021-2027
Analiza ex-ante w zakresie możliwości zastosowania instrumentów finansowych w ramach Regionalnego Programu dla województwa lubuskiego na lata 2021-2027
The study assesses the demand for repayable financing in Lubuskie in the areas covered by the programme.
To provide recommendations for the preparation of the investment strategy, in particular in relation to the use of financial instruments.
Documental analysis.Statistical and monitoring data analysis.Survey of SMEs (n=384), housing associations (n=370) and local government units (n=76).40 interviews with potential recipients.12 interviews with financial intermediaries and implementing institutions.Expert panel.
The total value of the financial gap until 2030 in the areas analysed amounts to PLN 16.559 billion (EUR 3.5 billion).The highest financial gap was observed for environmental protection.81% of local governments and only 4% of entrepreneurs surveyed are aware of support from EU funds.Firms affected by the financial gap are mainly small, young firms with poorer financial standing.
Maintain flexibility in the implementation of support so that the offer of loans is attractive and competes effectively with commercial instruments.Launch financial instruments for the development of renewable energy and improving energy efficiency for SMEs.Maintain the ability to finance both capital and working capital expenditure.
Evaluation of PA4 "Sustainable Urban Development" of the Berlin ERDF OP, 2014-2020. Final Report
Evaluierung der Prioritätsachse 4 „Nachhaltige Stadtentwicklung“. Endbericht
IfS (Institut für Stadtforschung und Strukturpolitik), www.ifsberlin.de; schwab@ifsberlin.de
The evaluation examines the integrated measures supporting sustainable urban development under the OP. It covers the implementation period between 2014 and 2020.
To assess whether and how the integrated measures have an impact on the improvement of the quality of life, social inclusion and active participation of the population.
Analysis of monitoring data and official statistics.Analysis of social spatial data, project reports.Literature review.Two online surveys of local actors (administrations, associations, companies, educational and social institutions - 278 respondents in the two phases, 25% of the stakeholders contacted).39 interviews with administration officials, experts and beneficiaries.1 experts workshop.TBE - Impact model.
ERDF support helped the development of new educational infrastructures, libraries and networks.The measures enabled the development of skills of social actors and their networking.The measures had a positive influence in the sustainable development of the supported areas, in combination with other factors, such as the improvement of the housing and labour markets and the demographic growth.Cross-cutting objectives such as climate change or equality between men and women are taken into account in the implementation of the projects, but they are not specifically addressed.
Too limited time elapsed to assess the effects of individual projects, use of monitoring data to take into account the influence of other explanatory factors, limited validity of the interviews with stakeholders in comparison with statistical data.
Evaluation of the ESF Madrid OP, 2014-2020: Annual Report 2017
Informe de Evaluación del Programa Operativo FSE de la Comunidad de Madrid 2014-2020. Objetivos/Resultados del PO para el Informe Anual a remitir en 2017
This is the report on the evaluation of the ESF Madrid OP, focusing on the progress made in achieving objectives.
To assess the OP strategy, its implementation and whether the results are in line with expectations.
Documentation analysis;quantitative analysis of data;interviews with officials.Data used come from the monitoring system and interviews.
The strategy is in line with EU regulations.The outcome indicators for Axis 1 are not in line with targets.The outcome indicators for Axis 2 are more in line with targets, but progress in respect of performance indicators is limited and there is a high risk of not meeting them because of delays in implementation.Progress under Axis 3 is most satisfactory.
Delay in approving the OP has affected the likelihood of achieving targets. There are some problems in the monitoring data.
There is a need to:Identify at an early stage problems in respect of physical and financial performance;pay special attention to operations under Axes 1 and 2 for which progress is slow;continue to identify and develop specific activities in relation to the horizontal principles of equality between men and women and non-discrimination;identify better the contribution of activities to social innovation.
Second interim evaluation of the outcome indicator of PA 2 under the Germany-Netherlands Interreg OP, 2014-2020
Zweite Zwischevaluierung des Ergebnisindikators der Priorität 2: "Wahrnehmung der deutsch-niederländischen Grenze als Barriere"
Interreg V-A-Germany-The Netherlands (Deutschland-Nederland) 2014-2020
Lars Kleijne, Martin van der Velde, Radboud Universiteit Nijmegen
The evaluation examines the output indicator of PA2 “Perception of the Dutch-German border as a barrier” of the OP up to June 2021.
To assess the extent to which the border is perceived as a barrier for people and organisations.
Factor analysis and structural equation modelling (SEM).Analysis of monitoring indicators.Survey of people (1,162 responses), and survey of organisations (306 responses).Comparison of the results of the surveys with those in 2015 and 2018 on the same aspects.
Compared to 2015 and 2018, people’s perception of legal-administrative barriers has declined, while the overall general, economic, and socio-cultural barrier is considered to have increased in importance, especially for the low and medium-skilled.People who were unaware of INTERREG reported a lower barrier effect than those familiar with INTERREG.Changes in border perceptions are less for organisations than individuals. Although perceptions of the border as an economic barrier declined, for most other aspects, it increased.Public authorities generally experience a lower barrier effect than other organisations.Covid-19 has led to a stronger barrier effect for individuals.The result indicator “Perception of the border as a barrier” is below the target value for both individuals and organisations.
They are not indicated in the evaluation.
Evaluation of PA 7 - Social Innovation - under the Niedersachsen ERDF/ESF OP, 2014-2020
Abschlussbericht zur Evaluation der Prioritätsachse 7 „Soziale Innovation“
ISG Institut für Sozialforschung und Gesellschaftspolitik GmbH, Weinsbergstr. 190, 50825 Köln, www.isg-institut.de
The evaluation examines PA7 “Social innovation” (IP 8v and IP 9iv) measures, coordinated by three bodies, two for “The changing world of work”, one for “Social services of general interest”.
To assess the management, implementation, results and effects of the PA7 Social innovation measures, with data collected in 2018 on on-going projects and in 2020 when half of the projects were completed.
Desk analysis of programme and project documentation and the literature.Monitoring data.Statistical data.Interviews with programme managers and the Social Innovation steering committee (21 experts).2 surveys of project promoters: in 2018 on on-going project (48 out of 53 respondents), in 2020 on completed projects (17 out of 26 respondents).
The three bodies for social innovation provide regular assistance and support for coordination and communication between the public and private stakeholders involved.The implementation of projects is satisfactory; positive elements are the management procedures, the supporting bodies and the learning process initiated.EU cross-cutting issues are formally considered, but their mainstreaming is not pursued systematically.All 53 projects adopt a participatory approach, with active involvement of employers’ associations, companies, universities, youth offices and schools. Objectives relating to improved awareness, dialogue and new networks between regional stakeholders, access to information, and the implementation of a stable application process have been achieved.Projects have high transferability potential (training measures, services offered, IT interfaces) but there is little evidence of actual transfer. Transferability is affected by the quality of networks, the territorial relevance of projects, existing infrastructure and funding opportunities. There is high potential for projects to contribute to regional structural change, including the establishment of new forms of cooperation, the strengthening of labour markets in the region, the creation of cross-sectoral services and long-term organisational development.
At the time of reporting, only half of projects had been completed, so the long-term results and impact should be regarded as preliminary.
Simplify the application procedures and project selection process.Ensure a more differentiated assessment of innovation between the focal points and target regions.Analyse transfer and dissemination potential more systematically.Ensure stronger interregional exchange on funding and the creation of synergy effects.
Evaluation of the Modernisation of Vocational Education (MOV) project, under the Research, Development and Education OP, 2014-2020 - 3rd Interim Report
Evaluace individuálních systémových projektů podpořených z PO 3 OP VVV-II Část II: Evaluační okruh B – Evaluace projektu MOV, 3. průběžná zpráva
Palackého tř. 10, 612 00 Brno, https://evaluace.eu/en/
The evaluation is the 3rd interim report of the project, covering the period mid-2017 to mid-2020.
To assess implementation progress and the achievements of the project.
Interviews with project managers and OP representatives.8 case studies involving school directors, coordinators and teachers.Surveys of teachers (156 respondents out of 399), Czech Labour offices (162 respondents), personnel agencies (7 respondents out of 44), employers (20 respondents out of 63), research, educational and consultancy (REC) organisation (8 respondents out of 143).Expert panel.
The project is progressing as planned.The knowledge produced by the project has been shared with other schools and stakeholders.COVID-19 might affect the use of project output and reduce the interest of teachers.The project’s internal evaluation practices were appropriately interconnected with its activities and the findings were used in the follow-up of project implementation.Awareness of the National Register of Qualifications (NSK) varies a lot among stakeholders. Representatives of personnel agencies and the Labour Office were most aware, followed by employers and staff of REC organisations.Respondents positively assessed the usefulness of the NSK; barriers to its wider use are the low awareness of it and the cost for those interested in obtaining qualifications.The scope and the quality of the activities and outputs of the project are positively assessed by stakeholders, because they are usable and meet their needs.Cooperation with other projects enables effective transfer of the knowledge obtained and project outputs.
The pandemic hindered field research activities and reduced the response rate to the surveys. The current phase of the project does not enable objectives of single activities to be monitored.
Publicise the project’s benefits for schools and employers more extensively and improve the dissemination of outputs.Use project outputs in accordance with Strategy 2030, through the review of the VET programme.
Evaluation of support for the long-term unemployed under the Emilia Romagna ESF OP, 2014-2020. Final report
Efficacia delle politiche formative e delle politiche attive del lavoro per sostenere l’inserimento e il reinserimento nel mercato del lavoro contrastando la disoccupazione di lunga durata – Rapporto finale
Iris S.r.l., https://www.irisricerche.it/ - Performer S.r.l., https://performer.it/
The report is an update of a previous one on employment outcomes 6 months after participating in measures for the long-term unemployed under the OP. Data go up to December 2020.
To assess the performance of measures for increasing the employability of the long-term unemployed up to December 2020.
91% of beneficiaries report little or no difficulty in accessing the measures.24% of trainees were hired at the end of measures (men more than women).6 months after completing projects, 80% of traineeship participants were employed, 62% of those participating in lifelong learning and 60% of those in training to improve skills.14% of participants went into apprenticeships, 11% into self-employment.Two-thirds of workers earned less than EUR 1,200 a month and 29% less than EUR 800.
Evaluation of projects on integration and social inclusion of foreigners in the Czech Republic in 2014-2020
Evaluace projektů zaměřených na integraci a sociální začleňování cizinců
OP Research, Development and Education 2014-2020; OP Employment 2014-2020; OP Prague - Growth Pole 2014-2020
Konsorcium nevládních organizací pracujících s migranty
The evaluation examines the projects concerned supported under the three ESF OPs as well as by the Asylum, Migration and Integration Fund.
To collect information on integration projects financed by the EU funds, to assess the coherence, complementarity, results and impact of projects, the main issues involved, and to make recommendations for the next programming period.
Documentation and project data analysis.Analysis of monitoring data.69 interviews with beneficiaries, target groups, MAs, relevant ministries and experts.5 focus groups with NGOs.Expert groups and platforms.
In 2014-2020, 719 ESF-supported projects on the integration of foreigners have been carried out, 685 of them under the Prague – Growth Pole OP.The implementation of projects has been delayed.The EU is the main source of financing for integration projects.Some projects under the Employment OP have provided comprehensive support, in addition to mentoring or employment support, including the teaching of Czech and individual legal and social counselling. Projects under the Prague – Growth Pole OP enabled schools to supplement funding for teaching pupils with a different mother tongue, training teachers in this respect and employing bilingual teaching assistants.Because of the number of projects, in some schools, teachers and pupils felt overloaded.The administrative burden is considered high and communication with the MA lengthy.According to respondents, projects have led to structural changes, such as increasing the number of employees, branches, and the portfolio, although they are considered to be temporary.There is a limited use of project results, activities that influence public policy mainly concern support for pupils with a different mother tongue.
There is no specific indicator that monitors migrants, who are included in the same category as other minorities. It was not possible to obtain aggregate data on key project activities or target groups from the information system. Only four projects that support the integration of foreigners have been identified under the Research, Development and Education OP, and systemic projects are still ongoing, therefore, their results cannot yet be evaluated. It is not possible to compare the number of pupils with a different mother tongue supported under the two OPs.
Incorporate the integration of foreigners into strategic and programming documents for EU funding.Involve regional and local authorities, non-profit organisations and experts in the planning, implementation and evaluation of integration measures.Identify foreigners as the target group of integration measures and ensure their proper monitoring.Coordinate funding and programmes according to the Concept of integration of foreigners.Support the long-term sustainability of projects.Expand support for pupils and students with a different mother tongue and methodological support for educators.Support professional language and Czech language courses aimed at improving the situation of migrants in the labour market.Support projects aimed at preventing illegal employment and promote transparency.Improve the capacity of social services to increase their accessibility to foreigners.
Evaluation of the impact of large RTDI infrastructure on RTDI projects financed by Cohesion Policy in 2014-2020, in Spain
EVALUACIÓN DEL IMPACTO DE LAS GRANDES INSTALACIONES CIENTÍFICO-TÉCNICAS EN LOS RESULTADOS DE PROYECTOS DE INVESTIGACIÓN
Instituto de Estudios fiscales, https://www.ief.es/, informacion.ief@ief.hacienda.gob.es
The evaluation examines the effects of using large RTDI infrastructure by research projects co-financed by Cohesion policy in 2014-2020.
To assess whether there is a causal relationship between making use of large research infrastructure and the results of RTDI projects in terms of publications, patents, collaboration agreements, technology-based companies created, etc.
Analysis of monitoring data. Regression analysis using the instrumental variables method.
The use of large RTDI infrastructure had positive effects on the number of publications and on securing funding under the EU Horizon 2020 Programme. Large RTDI infrastructure also have a positive impact on the quality of the results of research projects and their ability to influence future research and innovations. No impact from using large research infrastructure is found in relation to doctoral theses, patents, technology-based companies created and collaboration agreements established.
Data available are uncertain and might bias the results. Presence of atypical values in the data limit the precision of the technique selected for the analysis. The effects on patents, companies created, collaboration agreements might be underestimated, because of the delay in updating the State Research Agency database, after project completion.
Ensure that different administrative levels continue to collaborate to guarantee the long-term sustainability of research infrastructures. Ensure that State, regional funding agencies and the authorities owning infrastructure implement adequate monitoring systems.
Assessment of lifelong learning measures financed by the ESF under the Pays de la Loire OP, 2014-2020
Impact du programme FEDER FSE sur la validation du parcours d’insertion professionnelle
The evaluation examines the two ESF-supported lifelong learning interventions under the OP:- Development of key competences (“J’acquiers ou je développe mes compétences clés” - JADCC);- Vocational training for prisoners (“Dispositif de formation professionnelle au sein des établissements pénitentiaires” - DFPS).
To assess the extent to which two measures ‘Development of key competences’ (“J’acquiers ou je développe mes compétences clés” - JADCC) and ‘Vocational training for prisoners’ (“Dispositif de formation professionnelle au sein des établissements pénitentiaires” - DFPS) reached the groups targeted and to assess their effects.
Analysis of monitoring data.Online survey of JADCC training institutions (response rate of 71%).Online survey of JADCC participants (response rate of 14%).Survey of DFPS participants (response rate of 14%).Interviews with stakeholders.2 workshops.
In December 2019, 23% of the budget planned for these 2 interventions had been paid and there were around 4,900 participants (85% of them participating in JADCC).JADCC:Participants generally match the group targeted (low-skilled jobseekers). 60% of the training providers interviewed reported that the intervention met the needs of participants.72% of the number of training hours planned for each participant were achieved (target 75%).After exit, 12% of participants were employed and 10% in training or studies (49% and 11% after 12 months). 85% of the training providers reported the competences of the participants to have increased and that JADCC helped to improve their capacity to learn and to confident.The individualised approach of the training seems its main advantage.DFPS:ESF activities helped to reach the target group.Around a third of participants achieved their planned training (target: 50%).
Because of the difficulty of reaching DFPS participants once they leave prison (response rate of 14% in 2014), data on the labour market situation after exit is limited and does not allow a robust analysis to be made.
JADCC:Complete the training provided by developing distance teaching.Strengthen the teaching of numeracy.DFPS:Improve the synergy between the various stakeholders and strengthen the link with those outside prison.
Evaluation of the Alsace, Lorraine and Champagne-Ardenne ESF/ERDF OPs, 2014-2020
Evaluation de mise en oeuvre des programmes opérationnels Alsace, Lorraine et Champagne-Ardenne 2014-2020- Note Révision cadre de performance CA-LO V2
This evaluation examines the performance framework of the Alsace, Lorraine and Champagne-Ardenne OPs 2014-2020.
To verify the appropriateness and progress in achieving the performance framework targets.
Analysis of the quality of the data and the indicators which need to be changed or revised.Analysis of data to verify progress and the level of achievement in meeting the targets for 2018 and to estimate the likely level of achievement by end-2018.
There are three main causes for the differences between actual achievement and the performance framework targets: reporting errors; underperformance of the indicator caused by delays in implementation and under- or over-estimation of the targets due to an incorrect method used for their calculation.Where targets were not achieved because of slow implementation of the programme, no action was taken to revise the target or accelerate implementation.
Revision of the indicators does not presume a change in policy but a change in the targets included in the performance framework.
Evaluation of progress towards the objectives of PA1 - Education of the Human Resources OP, 2014-2020, in Slovakia
Záverečná hodnotiaca správa Hodnotenie pokroku k naplneniu cieľov PO1 v rámci OP ĽZ
Octigon, info@octigon.sk; Projektové služby, office@pseu.sk; RADELA, radela@radela.sk
The evaluation examines the achievements of PA1 of the OP up to the end of 2019 and June 2020.
To assess the level of achievement of specific objectives of PA1 up to end-2019 in relation to the performance framework and the relevance of the interventions.
Analysis of monitoring data (TMS2014+).360 interviews with projects and 10 with Ministries.
Progress on IP10.i indicators is slow, but targets should be achieved.According to interviews/survey, inclusive teams for education work effectively and have helped establish a more inclusive model of education in schools.The National Project “Dual” under IP10iv is delayed and the number of participants who have completed activities on vocational training is relatively low.The number of participants in IP10.iii measures who have acquired or upgraded their qualifications exceeded the target, except for low-skilled ones.External factors in the form of legislative and socio-economic changes and the restrictions caused by Covid-19, which pose a risk of not achieving financial targets, have affected implementation.
The field phase had to be adapted to the constraints due to Covid-19. All surveys were carried out online.
Monitor delays and other adverse effects of Covid-19.Continue intensive and systematic support for applicants and beneficiaries.Speed up the public procurement process.Strengthen absorption capacity.
Evaluation of implementation of the France-Switzerland 2014-2020 Interreg OP, Interim Report
Evaluation de mise en oeuvre du programme de coopération territoriale européenne Interreg France Suisse 2014 2020
Interreg V-A-France-Switzerland 2014-2020
Edater, Tel: 04 67 02 29 02, contact@edater.com, www.edater.fr
The interim report examines the implementation of the France-Switzerland Interreg OP between 2014 and end-June 2018.
To assess the validity and strength of the strategy and progress in implementation.
Analysis of Annual Implementation Reports, the communication strategy, evaluations, and other relevant documents.Interviews with MA and Switzerland Regional Coordination body.Online survey of beneficiaries (40 responses from France, response rate 43%; 43 from Switzerland, response rate 44%).Analysis of monitoring and statistical data.
In general, the MA’s strategy is appreciated by beneficiaries.The strategy for attracting new types of beneficiary has been less effective than in 2007-2013.260 partners are involved in 81 projects, with only a third from the public sector.The selection procedure works satisfactorily, though some delays resulted from the reform of territorial authorities in France.Several indicators are not representative of the objectives of the OP.There is a significant delay in the achievement of the 2018 milestones.The OP is generally consistent with the French regional OPs and with the EU Strategy for the Alpine region and is in line with the socio-economic needs of the regions covered by the OP.
It was not possible to carry out a precise analysis of the performance framework, since some data are missing.
Evaluation of the “e-health file and online services” project under the Piemonte ERDF OP, 2014-2020
Fascicolo sanitario elettronico e servizi online (FSE SOL)
The evaluation examines the implementation and use of the e-health file, financed under the OP, up to September-2022.
To assess the implementation of the e-health file project and its use by medical staff and the general public up to September 2022, and to identify its strengths and weaknesses.
Analysis of monitoring data.Analysis of administrative data (national and regional agencies for digital services).Interviews with medical staff.Survey of medical doctors (476 responses).
As at September-2022, the implementation of the e-health file project was almost completed (97% of services activated).Only 20% of medical doctors report having used the e-health file, and official data on access seem to show an even smaller share. Piemonte is among the Italian regions with the lowest use.The main obstacles to its use are the limited interoperability between the file and the software already used, the reduced number of options and documents available, the unfriendly graphic interface and the general public’s lack of awareness of it.Only 6% of patients in the region for whom the e-health file was available have accessed it, as against 30% nationally.Some doctors are concerned about sharing all the information available on patients, as it might increase their legal responsibility to have a thorough knowledge of patient conditions.
The survey sample might have a self-selection bias. Some data on the use of the e-health file might be overestimates.
Promote communication activities, training and possibly the development of an App to facilitate access and use of the e-health file.Ensure the inter-operability of the various information systems used in the healthcare sectorDevelop communication activities to ensure there is permission from people to store and access personal data in the e-health file.
Evaluation of social centres financed under the Notio Aigaio OP, 2014-2020
Τεχνικός Σύμβουλος για την Αξιολόγηση της πορείας της λειτουργίας των Κοινωνικών Δομών του Επιχειρησιακού Προγράμματος Νοτίου Αιγαίου 2014-2020 που συγχρηματοδοτούνται από το Ευρωπαϊκό Κοινωνικό Ταμείο
28is Oktovriou (Patision) 125, GR-11257 Athens, info@eeogroup.gr, https://www.eeogroup.gr/
This is the final evaluation report on the implementation of social centres financed under TO9 of the OP. Monitoring data extend to December 2019 while surveys were carried out in 2020.
To assess the efficiency, relevance and coherence of the social centres financed by the OP (Community Centres, centres providing basic services, Day Care Centres for the Elderly (KIFI) and Day Care Centres for People with Disabilities (KDIF-Disabled)).
Survey of a sample of beneficiaries.Survey of coordinators and of the MA, the Social Welfare Department and the Regional Observatory.Monitoring data.
The interventions are in line with the needs of beneficiaries, especially those with disabilities and families at risk of extreme poverty, but the centres report that their activities are limited by understaffing and administrative requirements.The financial absorption rate ranges from 61% to 108%.The target for the “number of supported centres” indicator has been achieved, the “number of people supported by the centres” ranges from 38% of the target to 99.7%; but the “number of centres providing improved/expanded services” is well short of the target which probably will not be met.The monitoring system used to collect data on beneficiaries is inefficient and social centres do not comply with any standards or guidelines to record information.The efficiency of the measures differs. Community Centres and Centres for the Provision of Basic Services are the least efficient, while the majority of day-care centres for the elderly are more efficient than planned.Publicity plans have been poorly implemented and dissemination of information on the activities of the centres is insufficient.
Improve dissemination of information on Community Centres to increase the number of beneficiaries.Regularly update skills and competences of staff in the centres to provide better services.Upgrade the functionality of the monitoring system and set suitable standards for recording beneficiaries.
Analysis of the situation across the country in areas in which the ESF provides support in 2014-2020
Analiza sytuacji województw w obszarach oddziaływania Europejskiego Funduszu Społecznego w latach 2007-2014
Małgorzata Kruk, Beata Lewczuk, Hanna Lodczyk, Łukasz Mikulec, Ida Szczepocka, Marta Turczyn, Anna Żelazko-Kątna, Aleksandra Dmitruk
The evaluation examines the socioeconomic situation in the regions in the policy areas supported by the ESF in 2007-2013 (demographic issues, the labour market, social inclusion, adaptability of employees and enterprises, education and training, and healthcare).
To assess the socio-economic situation of the regions in selected policy areas that are most affected, or will be affected, by ESF measures in 2014-2020.
Desk research of official statistics, programming documents, monitoring data and evaluations;analysis of monitoring and statistical data.
The demographic issue of ageing is most evident in Opolskie, Świętokrzyskie and Łódzkie.Access to care for younger children is most limited in Świętokrzyskie, Warmian-Masurian and Lubelskie.Since 2012, unemployment in Poland has been lower than the EU average. The highest rate in 2014 was in Podkarpackie at 14%.In 2014, a quarter of the population were at risk of poverty or social exclusion, i.e. 9.5 million people.In 2014, only 14% of those aged 25-64 participated in any form of education, 1 p.p. less than in 2007.The largest number of economically inactive because of illness or disability was in Lubuskie, Lubelskie and Warmińsko-Mazurskie.
Support the employment of people of 50 over, particularly in Dolnośląskie, Zachodniopomorskie, Lubuskie, Mazowieckie, Śląskie, Warmińsko-Mazurskie, Kujawsko-Pomorskie and Podkarpackie.Favour projects on childcare in rural areas after verifying the demand for facilities for those of 3 or under.Ensure the possibility of continuing training of those of 50 and over, by rewarding those participating.Specify the minimum share of long-term unemployed who need to receive ESF support, especially in regions with the highest rates.Introduce a minimum level of employment to be reached by projects as a selection criterion.
Qualitative Survey of social innovation projects under the Employment OP, 2014-2020 - Phase 1
Kvalitativní šetření inovačních projektů PO 3 OP Z, Etapa 1
This is the first phase of the evaluation examining five projects, out of 39, funded by PA3 (Social Innovation and Transnational Cooperation) of the Employment OP.
To identify the success factors, how the 5 social innovation projects were implemented, their functioning and cooperation among those involved.
Semi-structured interviews and focus groups with projects.Text analysis using Atlas.ti.
The success factors include the level of cooperation between the implementation team and others involved and the number of innovative elements in the projects to make them interesting as well as the economic situation and the labour market situation in areas where the project is implemented.The projects succeeded in meeting the required indicators for evaluation.It is necessary after the projects come to an end to maintain support for them, especially for those involving disadvantaged groups (ex. prisoners, people with mental illness).There were no major changes in any of the projects during their implementation.
There were some complications in carrying out interviews because of the holiday period; unwillingness of some to be interviewed; and difficulties in reaching the right people.
No specific policy implications are drawn.
Evaluation of the contribution of ESI Funds, 2014-2020, to regional development (Tasks 5 and 6) in Plzeňský, in the Czech Republic
Výsledková evaluace přínosů Evropských fondů na regionální úrovni: Úkol 5 a 6 - Plzeňský kraj
OP Research, Development and Education 2014-2020; OP Employment 2014-2020; OP Transport 2014-2020; OP Environment 2014-2020; OP Enterprise and Innovation for Competitiveness 2014-2020; Integrated Regional OP 2014-2020
Ernst & Young, s.r.o., Na Florenci 2116/15, 110 00 Praha 1 – Nové Město, https://www.ey.com/en_cz
The evaluation examines the results and benefits of using EU funds in Plzeňský. Data used go up to 30/06/2022.
To evaluate outputs and results of the measures funded under the EU funds.
Analysis of official statistics and monitoring data. Case study. Focus group with Local Action Group operating in the region. Interviews with stakeholders. Survey of municipalities and citizens. Counterfactual analysis (DiD, PSM, regression analysis).
The measures co-funded by EU Funds are relevant and met output and result targets. A significant level of support from EU funds correlates with an increased number of science and research workers and significant reduction in social exclusion. Impact of ESI Funds on TOs 3, 5 and 8 are statistically insignificant. The projects led to the number of researchers in the region increasing by 534 (the largest increase for the Czech Republic). Projects helped increase the number of visitors of the region, promote the territory and improve cultural infrastructure. EU investments improved public transport efficiency and infrastructure, eco-friendly transportation and road networks. It is important noting that the impact analysis showed a weak correlation between EU funds and quality of transport. Local Action Group projects enhanced social services and community development. EU-funded projects helped align education with labor market needs and improve healthcare infrastructure.
Lack of granular data to estimate impact in relation to some TOs.
Promote polytechnic education by connecting secondary schools, local companies and other stakeholders. Implement integrated solutions through ITI to kick-start the new Borská Pole district. Enforce stricter conditions in social services, e.g. by limiting the employee use of the car to provide social services in less accessible areas. Improve the presentation of projects results, at both national and local levels, to increase citizen awareness.
Evaluation of Integrated Territorial Investments under the Île-de-France ERDF-ESF OP, 2014-2020
EDATER - Rouge Vif - Institut Paris Région
https://www.edater.fr/ - https://rougevif-territoires.fr/ - https://www.institutparisregion.fr/
The evaluation examines the implementation of integrated territorial investment (ITI) strategies under the OP. The initial report is supplemented with a subsequent report on 8 ITIs.
To assess the coherence and integration of ITIs with local policies in the region, their role in promoting partnerships and their territorial effects.
Analysis of monitoring data.Interviews with MA, ITI representatives (8 ITI out of 15) and staff.Survey of beneficiaries (92 out of 142, response rate 66%).
ITI are coherent with local policies, though the range of eligible “thematic areas” seems too wide.Progress in implementing ITI is adequate and targets should be reached.ITIs generally have had a strong leverage effect, e.g. in terms of participation in other EU projects and project management skills.Main weaknesses are that financing is limited to new projects and there are issues over the sustainability of projects, access to funding for small applicants, the requirement to provide personal data and the minimum amount required by projects.58% of beneficiaries report that they would have not applied for EU funding without support from ITI.
Ensure integration between EU funding and other national or local sources.Assign the status of ITI to a limited number of areas and concentrate ITI projects on a few specific objectives.Strengthen support for all the people involved in ITI at different levels and ensure continuing dialogue.Integrate all stakeholders better, e.g. by providing mentoring for small associations.
Evaluation of transport measures under the Transport, infrastructure, environment and sustainable development OP, 2014-2020, in Greece, to help prepare the new programme, 2021-2027
Αξιολόγηση του τομέα Υποδομών Μεταφορών του ΕΠ-ΥΜΕΠΕΡΑΑ (Παραδοτέο Α)
OP Transport, infrastructure, environment and sustainable development 2014-2020
Institute of Communication and Computer Systems (ICCS) – National Technical University of Athens (NTUA)
The evaluation examines the measures supporting road, rail, maritime, air and urban transport. It covers the implementation period between 2014 and 2023.
To assess whether the targets of transport measures have been achieved and the main factors behind this. To support the design of the 2021-2027 transport programme.
Analysis of monitoring data (management authority reports)Analysis of statistical data (Hellenic Statistical Authority, Hellenic Train…)
Six revisions of the OP have been introduced during the period to take into account the evolution of the socio-economic environment.In the road sector, the OP made a significant contribution to the main interregional regional feeder links.In the rail sector, the contribution of the OP to the Patra-Athina-Thessaloniki-Eidomeni-Proachonas (PATHEP) axis is much lower than expected.In the maritime transport sector, the contribution of the OP to the development and the interconnection of ports is limited.In the air transport sector, the contribution of the OP to the development of airports and the improvement of air navigation safety is lower than expected.In the urban sector, the OP contributed to the development of new metro lines in Athens and Thessaloniki.
The report includes a specific section on the design of the new transport programme, but it does not mention any real policy implication.
​​Evaluation of the implementation of the Grand-Est Programme, 2021-2027​
Evaluation à mi-parcours de la mise en œuvre du programme FEDER-FSE+-FTJ Grand Est et Massif des Vosges 2021-2027
The evaluation examines the implementation of actions contributing to the socioeconomic and environmental development of the Grand Est region. It covers the implementation period between 2021 and 2024.
​​To assess the programme implementation progress, the first results of the interventions, as well as the achievement of the objectives initially set and its contribution to national (energy transition, labour market, fight against poverty, etc.) and European priorities (the principles of the European Pillar of Social Rights, sustainable development and the fight against climate change). To provide suggestions for the remainder of the programming period.
Review of strategic documents.32 Interviews: 17 with members of the European Funds and Operational Directorates, 6 with regional partners, and 9 with beneficiaries of EU funds.Analysis of physical and financial monitoring data.Analysis of official statistics (INSEE).Workshops.Benchmark of other MAs and national actors.JTF Case Study.
The programme strategy is consistent with the current and future regional context and needs.Most of the projects are strongly aligned with the principles of the European Pillar of Social Rights (PA 4 and 5 in particular).The programme shows a good progress at 31/12/2024, with an overall programming rate of 41% for 769 projects (EUR 354 million programmed from ERDF), which is higher than the French and European average (30% for the ERDF and 35% for the ESF+).However, there are very heterogeneous situations depending on the specific/thematic objectives:More than half of the projects are financed under PO 1 (61% programming rate), in particular under SO 1.3, competitiveness of SMEs (250 projects), and SO 1.1, research and innovation (111 projects). Other policy areas, like ‘Culture/tourism’ (SO 4.6), ‘Urban development’ (SO 5.1), ‘Youth’ and ‘Training’ (SO 4.7) also show a relatively advanced state.On the contrary, PO2 has a low programming rate (22%): measures on SO 2.2, promoting renewable energy, and SO 2.4, climate change adaptation, are particularly delayed.For a few SOs, there is a significant overprogramming (over 200% in some cases).A large part of the measures supported under the 2021-2027 period are a continuation of those funded under the 2014-2020 period.
Readjust the project selection process, avoiding the ‘first come, first served’ approach that prevents innovative projects from emerging.Reinforce the collaboration between the European Funds Directorate and the operational directorates.Strengthen animation and communication activities around European funds and the regional programme, and more specifically on new and emerging themesConduct an in-depth economic analysis for major projects to adjust funding rates on the basis of the expected cash flow of beneficiaries.Improve the indicators’ system.
Evaluation of the indicator system and targets of the Technical assistance OP, 2014-2020
Evaluace indikátorové soustavy a jednotlivých definovaných cílů OPTP
Ernst & Young, s.r.o; Na Florenci 2116/15, 110 Praha 1, Czechia, www.ey.com/cz
The evaluation assesses the system of output and outcome indicators of the Technical Assistance OP 2014-2020 in the Czech Republic, which mainly finances activities to improve the management of the funds and training courses for Ministry of Regional Development and Ministry of Finance personnel.
To examines whether the indicators are consistent with the activities supported and the targets are realistic.
Data analysis.Interviews with funding recipients and the MA.Focus group with beneficiaries.A case study of a project.
The indicators covers most of the OP activities sufficiently, but there are several inconsistencies between the activities, indicators and targets.According to MA regulation, beneficiaries have to select at least one indicator among those defined in order to monitor their project. 78% of them selected only two indicators and 30% selected only one. Beneficiaries often chose indicators that can be easily controlled (e.g. “Purchase of material, goods and services”), that are not very informative.Some of the targets are being significantly overachieved, while for others, it is the reverse and for some of these, it is unlikely to meet the targets.
Data for some projects were not available and for these data from similar projects were used instead.
There is need to:drop some indicators that do not reflect the content of activities;interact more with beneficiaries to improve the selection of indicators;review targets;make a distinction between generic and specific indicators with beneficiaries needing to select at least one from each category;review the method of setting targets.
Mid-term evaluation of the ESF Comunitat Valenciana OP 2014-2020
Informe de Evaluación del Programa Operativo FSE de la Comunitat Valenciana 2014-2020. Período 2014-2016
The evaluation assesses the intervention logic of the ESF Comunitat Valenciana OP 2014-2020, as well as the results obtained under each axis in relation to the targets set.
To verify the results achieved at the end of 2016 by the axes of the OP.
Desk analysis;analysis of monitoring data;interviews with beneficiaries;on-line questionnaires to stakeholders.
The programme intervention logic has internal and external coherence and respects the horizontal principles. It does not need to be revised.The intervention logic of all three axes is coherent with needs, except for axis 9 which is considered too weak to achieve the changes aimed at.The results of the OP so far are below the targets set for the period examined.
There is a lack of monitoring data on result indicators and the available information is limited due to late implementation and problems with the information system.
There is a need to:update the socio-economic analysis to assess continuously the adequacy of the strategy;improve the monitoring system.
Evaluation of measures supporting access to employment and active inclusion under the ESF+ programme, 2021-2027, in the Netherlands
Evaluatie ESF+ 2021-2027 (arbeidsmarkttoeleiding en actieve inclusie): Definitief rapport procesevaluatie (eerste fase)
Bert-Jan Buiskool, Gert-Jan Lindeboom; Douwe Grijpstra, Thomas de Winter
Ockham, https://ockham-ips.nl; Panteia, https://panteia.nl
The evaluation examines four types of projects supported under Priority 1 of the ESF+ programme: access to employment, social inclusion, active inclusion – special needs education, active inclusion – custodial institutions agency.
To assess the implementation of projects for employment and active inclusion and draw lessons for the rest of the programming period as well as for future periods.
Analysis of programme documents and project applications. Interviews of key stakeholders (municipalities, schools, sector councils and representatives of the Custodial Institutions Agency). Group discussions with the Ministry of Social Affairs and Employment.
Applicants are generally positive about the application process (which hardly changed compared to the previous period), though the user-friendliness of the grant portal could be improved. Cooperation between stakeholders and the MA is assessed positively. However, the high turnover within the MA is perceived as problematic. No major implementation problems were observed; the measures are expected to achieve the intended number of participants. Respondents are largely positive about the use of simplified cost reporting (which was already in place previously). For projects supporting access to employment, the use of a fixed rate per case manager is an innovation (based on simplified cost options). Municipalities welcome the specific focus on social inclusion, as due to labour market shortages, only people more distant from the labour market are not able to access it. This also increases the importance of proposing a wider range of activities (such as debt counselling pathways). Special needs education: The current budget allocation is based on the number of unemployed per region and not on the number of pupils. Custodial Institutions Agency (DJI): In response to halving the budget dedicated to these projects compared to the previous period, a result-oriented approach was adopted where half of the budget is allocated based on actual participation and the other half on results achieved. The EU added value of ESF+ is widely recognised, in terms of size and scope of the projects financed.
Encourage flexibility in activities aimed at increasing social participation. Maintain stability across programming periods in terms of target groups and eligible activities. Maintain the possibility to justify costs through caseloads of case managers. Review the distribution key for special needs education and assess the consequences of using a different distribution key based on the number of students. Assess how result-oriented funding in DJI is implemented and how it encourages institutions to pursue this objective.
Evaluation of the support for the digitalisation of education financed by the ESF, 2014-2020, in Portugal
Avaliação do Contributo do Portugal 2020 para a Digitalização da Educação
OP Human Capital 2014-2020; ROP Norte 2014-2020; ROP Centro 2014-2020; ROP Alentejo 2014-2020; ROP Algarve 2014-2020
Sandra Primitivo, Ana Luísa Rodrigues, Ana Caetano, Joana Canada, Larissa Dowdney, Manuel Reis, Margarida Lourenço and Rui Faustino
EY-Parthenon, parthenon.ey.com, ernst.young@pt.ey.com
The evaluation examines the ESF measures in the field of digitalisation of education promoting equal access to education and improving teaching/learning processes. It covers the implementation period between 2020 and 2022.
To assess the effectiveness, efficiency, impact, sustainability, European added value, and relevance of the support for the digitalisation of education. To provide recommendations for the future programme.
Analysis of official statistics and monitoring data.Literature review.7 interviews and meetings with the MAs and the Directorate-General for Education and Schools.3 case studies of different schools.2 surveys – of the directors of schools (75% response rate) and of teachers (11% response rate).2 focus groups and 1 final workshop with key stakeholders (e.g. Directorate-General for Education and Science Statistics, Human Capital OP).
Digital School kits have been successfully delivered to teachers and students (approximatively 295,000 computers delivered to public school students). Around 90% of the students accepted to receive the kit and disadvantaged students show a higher rate of acceptance.The digital training to strengthen teachers' skills is considered by vast majority of teachers as useful for all the activities performed at school.Beyond the kit itself, several factors were decisive for its effective use in face-to-face teaching, such as the teachers’ interest in digital technology, proper access to the internet and a reliable technical assistance.The use of digital tools in classrooms is significantly higher than before the pandemic and very frequent overall, also because the average number of students per computer with internet went from 4.9 to 2.
The planned counterfactual analysis could not be carried out because of the impossibility to access specific personal data.
Rethink the distribution model of the Digital School kits, in order to ensure the sustainability of the universal access to digital teaching.Improve the adequacy of the equipment, in terms of quality, functionality and safety.Increase the technical assistance and computer support to students and teachers.Give continuity to and diversify the training for the digital skills of teachers.
Evaluation of measures supporting health diagnostics and prevention implemented under the Śląskie ESF OP, 2014-2020
Ewaluacja efektów wsparcia wybranych programów dotyczących diagnostyki i profilaktyki zdrowotnej realizowanych w ramach Regionalnego Programu Województwa Śląskiego 2014-2020
The evaluation examines the implementation of two supported programmes (‘Healthy Mother and Child’ and ‘Healthy Spine Programme’) on health diagnosis and prevention.
To assess the implementation of two regional health programmes and identify the reasons why another two were not implemented.
Desk research: programme and strategic documents, project documentation and websites.Analysis of monitoring data.5 interviews with representatives of institutions, 34 interviews with stakeholders involved in implementation.Survey of 23 beneficiaries and of 500 project participants.Group interview with MA.
A coherent information system about the availability and conditions of support is reported to be missing.Recruiting recipients and entities involved in the implementation of project activities is problematic.Administrative burdens for beneficiaries are heavy.There are problems with the monitoring system of the objectives and outcomes of the programmes.
Imperfections of the monitoring system made the evaluation difficult.
Better promote support offered under the programmes.Introduce a criterion on the adequacy of the method for recruiting the target group, rewarding non-standard methods.Use of simplified project settlement methods.Develop a more effective system for monitoring and evaluating progress and outcomes of the measures.
Evaluation of the “Inclusive Education Action Plan – APIV” projects, under the Research, Development and Education ESF OP, 2014-2020 – Third interim report
Evaluace individuálních systémových projektů podpořených z PO 3 OP VVV-II Část II: Evaluační okruh A – Evaluace projektů APIV 3.průběžná zpráva
Evaluation Advisory CE s.r.o., Palackého tř. 10, 612 00 Brno, https://evaluace.eu/en/contacts/
The evaluation examines the implementation of two projects for school managers: “Methodological support for Inclusive education Action Plan – APIV A” and “Support for inclusive education in educational practices - APIV B”, covering the period October 2019 to December 2020.
To assess progress in implementing two projects on “Methodological support for Inclusive education Action Plan – APIV A” and “Support for inclusive education in educational practices - APIV B” over the period October 2019 to December 2020 and their contribution to the Inclusive Education Action Plan.
Desk analysis of OP and project documentation and previous studies.Interviews with MA project administrators, project managers, internal evaluators, personnel of the national Support Education Centres, and representatives of target groups.Case studies of 5 beneficiary schools and 5 non-beneficiary schools.Participant observation in 2 expert panels.Questionnaire to 150 participants in 9 expert panels.
Nearly all project activities had to be implemented online because of Covid-19.Despite implementation difficulties, all key activities of the projects have the potential to achieve the objectives.The project teams have attempted to ensure sustainability of project outputs, e.g. through expanding the functionality of diagnostic tools and identifying follow-up activities.The continuing pandemic threatens to prevent the transfer of good practice, the realisation of public events and traineeships in schools.Support is perceived as beneficial by the schools involved, in particular, the practical training, the exchange of good practice and development of discussions on inclusive education.Cooperation between projects is not structured and consists of a mutual flow of information and data.The unintended effects of the measures are the dissemination of results and methodology beyond the projects themselves and the strengthening of relationships between education workers.
Project activities were extended and their content partially changed, because of Covid-19.
Reduce the number of expert panels on projects.Define a standardised procedure to adopt recommendations from expert panels on teaching methods.
Mid-term evaluation of regional ERDF OPs 2014-2020 in the Netherlands
MidTerm Evaluatie Uitvoering landsdelige EFRO-programma's 2014-2020
ROPs Noord, West, Zuid, Oost ERDF 2014-2020
Peter Bex; Michel Bloemheuvel; Joost Strijtveen; Lisanne Vis; Lisanne Vis; Mart Vreeswijk
SIRA consulting, SIRA consulting, www.siraconsulting.nl
The mid-term evaluation focuses on some implementation aspects of the four regional ERDF OPs, the effectiveness of the administrative cooperation agreements, the organisation adopted by each MA and the effectiveness of the communication strategies.
To assess the effectiveness of the cooperation arrangements between MAs in improving the quality of the implementation of the OPs and to make recommendations on implementation aspects for the next programming period.
desk analysis of relevant documents and evaluation studies;around 100 interviews with those involved in the implementation of the OPs, beneficiaries, grant advisors and member of expert committees;workshops with over 40 experts and stakeholders from public authorities and beneficiaries.
Because of the agreements, the MAs have adopted more uniform working methods, common assessment and ICT systems and more intensive knowledge sharing.However, the efficiency of the ICT system needs to be improved. And the agreements have not resulted in a reduction in the control and accountability burden on businesses.The effectiveness of communication in attracting new projects is good, but less so in targeting SMEs, which know little about the ERDF, which has a negative connotation for many of them.The effectiveness of communication in increasing the general public's awareness of the ERDF is limited. The visibility of innovation projects is lower than the large infrastructure or tourist projects in other Member States. The small budget for communications is a limiting factor.
improve the Cooperation agreements to achieve the objectives which have not yet been reached;standardise the procedures for control and accountability to reduce the administrative burden on businesses;promote the exchange of good practice between MAs;focus communication more on the results achieved by the projects funded;intensify communication targeted at SMEs;identify quantifiable communication targets.
Evaluation of the implementation of the Smart City concept in Visegrad countries in 2014-2020
Ewaluacja realizacji koncepcji Smart City w państwach Grupy Wyszehradzkiej
OP Technical Assistance 2014-2020; OP Infrastructure and Environment 2014-2020; OP Smart Growth 2014-2020
Michał Wolański, Anna Borowczak, Mirosław Czerliński, Paulina Kozłowska, Seweryn Krupnik, Bartosz Ledzion, Jakub Zawieska, Jakub Kaczorowski
Wolański ltd., ul. Stawki 8/7, 00-193 Warszawa; EGO – Evaluation for Government Organizations s.c., ul. Podmokła 53 B, 04-819 Warszawa; LB&E Sp. z o.o, Ul. Książenicka 98, 05-807 Owczarnia
The evaluation examines the practical application of the Smart City concept in public policies in the V4 countries, with a view to identifying transferable ‘good practice’.
To assess how the Smart City concept was implemented in key strategic programming documents in the V4 countries, to identify the examples of good practice and the factors affecting the projects concerned.
Desk research (strategic and programming documents, projects, literature).7 interviews.Case studies.Qualitative Comparative Analysis.
Smart city measures should be managed as IT projects - combining strategic vision and agility.There is a lack of holistic support for smart city initiatives and the role of strategic documents for stakeholders is unclear.Legal regulations limit the possibilities of introducing new installations, applications and digital services, and industry regulations often leave no room for innovative measures or require special adaptation.
Limitations relate to the multifaceted nature of the smart city concept, the subject of analysis adopted (general documents vs. specific projects), the comparative aspect of the analysis and interview recruitment in the Czech Republic.
Provide comprehensive support for smart city measures, including establishing a special unit for them at national and local level.Make funds more flexible and equalise their distribution.Make legal regulations more flexible, learning from other countries.Involve various stakeholders and develop general competences.Ensure central government has an important role in enforcing standards.Ensure compatibility between projects and city types.
Evaluation of the quality of regulation in 2007-2013 and 2014-2020
Értékelési jelentés a 2007-2013 időszak intézményrendszeri megoldásainak értékelése projekt. A szabályozás minőségének értékelése
This study assesses the administrative burden and extent of regulation as regards the use of EU funding.
To assess the administrative burden on beneficiaries resulting from the regulations relating to the use of EU funding.
Desk analysis of applications submitted, regulations, guidelines for beneficiaries and tender documentation.Analysis of a sample of calls for proposals from the Hungarian OP.
Over-regulation and the associated administrative burden adversely affect SMEs and small local authorities with limited capacity to apply for funding.The average time needed to prepare projects and payment requests has been significantly reduced over time.MAs tend to overregulate as a reaction to new challenges or issues and the lack of other means of controlling how funding is used.The EU funding management system disregards quality aspects, such as means of reducing the administrative burden or of simplifying regulations.
The use of e-procurement needs to be extended.Online Information Interfaces for applicants need to be improved.Quality control measures on elements such as the administrative burden and barriers to increasing the absorption rate need to be implemented.
Analysis of possible measures to strengthen R&D potential in Slovakia, especially under the Integrated Infrastructure OP, 2014-2020
Analýza možnosti posilnenia potenciálu výskumno-vývojovej infraštruktúry pre potreby rozvoja konceptu Industry 4.0
Visions Consulting, s.r.o. - Grant Thornton Advisory s.r.o.
https://www.visions.cc/ - https://www.grantthornton.sk/en/about-us/grant-thornton-slovakia
The evaluation examines the potential for strengthening research infrastructure in Slovakia under the OP, with a view to the next programming period.
To identify the technological potential and human capital in R&D and possible measures for strengthening the research infrastructure in the country, especially in respect of Industry 4.0.
Interviews with representatives of R&D organisations (22).Surveys of stakeholders.Network analysis.Review of Programme documentation.
The position of public R&D centres with respect to Industry 4.0 is sub-optimal and limits cooperation with businesses.The current technological equipment in public R&D centres is insufficient.Efficient and regular financing is necessary to develop emerging research trends.Human resources are not well remunerated, causing an outflow of students and researchers abroad.Cooperation between public and private R&D centres mostly occurs in advanced robotics and artificial intelligence.
Expand and regularly renew public technological infrastructure and make it available to companies.Provide adequate incentives (remuneration, job stabilisation, awards, scholarships,…) for working in the R&D sector and ensure business education for students.Ensure an adequate grant system, promoting project quality.Define measures to support commercialisation of research results.
Evaluation of measures for nature conservation under the Environmental and Energy Efficiency and the Competitive Central Hungary OPs, 2014-2020
1 2021. január 20. A természetvédelmi fejlesztések átfogó értékelése
OP Environmental and Energy Efficiency 2014-2020; OP Competitive Central-Hungary 2014-2020
TRENECON Ltd. Capital Square Office Building - 1133 Budapest, Váci út 76. 3rd floor Phone: +36 (1) 237 1450, email : info@trenecon.hu https://www.trenecon.hu/
The evaluation examines 6 calls financed under TO6 under the OPs.
To assess the results achieved by measures on nature conservation in 2014-2020 and to make recommendations for the next programming period.
Analysis of monitoring data.Interviews with the National Parks directorates, the MAs, policy experts and beneficiaries.Analysis of a sample of projects.Analysis of documentation of projects not primarily intended for nature conservation purposes.
In the 6 calls, 90% of available resources had been committed by end-2019. Though the increase in construction prices in recent years has adversely affected investment, only a few projects have failed.Projects mainly contribute to the objectives of "Conservation of protected, highly protected natural and near-natural habitats and species of community importance" and "Protection of inanimate natural values" through the reconstruction of habitats.Efforts to preserve biodiversity have potentially increased the adaptability of species and habitats to climate change, though this is not so clear for projects not directly on nature conservation.The output targets set have been achieved.There remains a shortage of professionals with planning experience and capable project managers.
Data on projects not directed primarily at conservation were insufficient or preliminary. Some indicators did not reflect the full content of the calls.
Provide training on nature conservation legislation to potential participants.Place more emphasis in calls on assessment and effective management of invasive species.Maintain current indicators on nature conservation and improve their use by clarifying the definitions and publishing guidance on calculation.Add indicators to better measure the achievement of nature conservation objectives.Standardise the methodology for recording data.
Evaluation of PA2 on competitiveness of SMEs of the Brandenburg ERDF OP, 2014-2020
Endbericht. PA 2: Stärkung der Wettbewerbsfähigkeit kleiner und mittlerer Unternehmen
MR Gesellschaft für Regionalberatung mbH, GEFRA – Gesellschaft für Finanz- und Regionalanalysen, IfS - Institut für Stadtforschung und Strukturpolitik GmbH
The evaluation analyses measures aimed at improving the competitiveness of SMEs financed under the OP. The measures concern: support for start-ups and young innovative businesses, and raising awareness of business transfers as a potential business successor (IP 3a); Strengthening the international competitiveness of SMEs through market development activities (IP 3b); Expanding the growth and innovation capabilities of Brandenburg SMEs (IP 3d). Monitoring data go up to 31 December 2022.
To assess the OP contribution to the Union's strategy for smart, sustainable, and inclusive growth as well as to identify best practices. To identify potential for optimisation of the measures.
TBIE.Analysis of official statistics and monitoring data.Questionnaires/surveys.Review of literature and previous thematic evaluations.
As at 31 December 2022, all the indicators, despite being below the set values, are moving towards the targets for 2023.IP 3.a: funding contributed in an innovative way to "smart specialisation" even though it has not been possible to fully achieve the objectives set for the end of 2022; a great need for consultancy for company succession is reported.IP 3.b: 674 projects were supported mainly for participation in fairs. The measure has reached a large number of SMEs and young companies that do not have yet the necessary structure to open up to foreign markets and it has helped increase company exports.Three financial instruments were used under IP 3.d: the BFB Early Phase and Growth Fund had a positive impact on R&D and helped create 514 new jobs; the Brandenburg Credit Mezzanine II, provided subordinated loans and helped develop new markets and business areas; the Microcredit Brandenburg contributed to the creation of 494 new jobs, but its impact has been very limited.
The exact quantification of the net effects of measures under IP 3b was not possible.
Continue focussing measures under IP 3d on the participation in trade fairs.Continue financing measures under IP 3d by using the financial instruments that led to positive results.Ensure that internationalisation measures in 2021-2027 continue to be an important part of Brandenburg's foreign trade strategy.
Evaluation of measures supporting employment under the Toscana ESF OP, 2014-2020, especially in respect of women
Rapporto di valutazione tematica dell’Asse A “Occupazione” sull’occupazione delle donne
The evaluation examines the effects on women of measures supporting employment under the OP.
To assess the implementation of employment support measures, especially for women, and their impact.
Analysis of official statistics and monitoring data.Literature review.Administrative data (hiring register and businesses register).Econometric analysis.Interviews with those responsible for the measures.Survey of women participants in the measures.
Measures for gender equality under the OP are especially focussed on work-life balance and incentives for female employment, though they do not seem to be sufficient.Placement rates of women participating in employment measures carried out before Covid are similar to those of men, while for measures implemented after Covid they are significantly lower than for men.The share of full-time time contracts is larger for women after participation in the measures, but it is still significantly lower than for men.Covid is likely to have reduced women’s participation in the labour market because of an increased need for family care and temporary contracts not being renewed.According to the women interviewed the main obstacles to accessing the labour market are maternity, the inflexibility of working hours, lack of child and elderly care, and cultural stereotyping that penalises the professionalism of women.
The evaluation examines the gross effect of the measures, not the net impact. Data do not include all kinds of employment contract. Employment rates might be underestimated because of missing administrative data on some participants.
Always include traineeships in employment measures and integrate them with other support for female employment, as they are the most effective means of helping women into work.Provide specific coaching services for women.Develop measures for reintegrating women into the labour market after child-birth.Define awareness raising activities for companies to support maternity and new organisational forms of working.
Evaluation of the use of financial instruments under the Niedersachsen ERDF/ESF OP, 2014-2020
Sonderuntersuchung 3 im Rahmen der Begleitevaluierung des Multifondsprogramms Niedersachsen. Finanzinstrumente im EFRE – Potenziale, Beschränkungen, Nutzungsmöglichkeiten.
Dr. Stefan Meyer (Kovalis), Dr. Björn Alecke (GEFRA)
meyer@kovalis.de, www.kovalis.de - GEFRA, Gesellschaft für Finanz- und Regionalanalysen GbR, info@gefra-muenster.de, www.gefra-muenster.de
The evaluation examines the implementation of financial instruments (FIs) under the OP, up to 31 December 2018. FIs support innovation and investments in start-ups and SMEs.
To assess the extent to which financial instruments, which are used in support of start-ups and SMEs, can support economic and regional policy, based on data up to end-2018.
Documentation analysis and literature review.Analysis of monitoring data.Interviews with experts.
FIs are expected to support SMEs with higher financing possibilities and greater possibilities of increasing profitability, but in Germany there is no clear evidence of the real impact of FIs.The use of FIs by start-ups and SMEs has increased compared to the previous programming period, but the planned innovation loan has not been introduced.The main FIs used are microcredit, equities, and subordinated loans.FIs supported show good financial performance.
Plan a combination of FIs with grants and non-repayable shares, to increase the incentive effect of the support.Adopt financial agreements to ensure the effective use of FIs.
Evaluation of the Rehabilitation and Education Programme for Cardiac Patients under the Łódzkie OP, 2014-2020
Ewaluacja Programu rehabilitacyjno-edukacyjnego dla pacjentów kardiologicznych z terenu województwa łódzkiego w ramach RPO WŁ 2014-2020
dr I. Franckiewicz-Olczak, dr A. Podkońska, K. Szczepaniak, M. Białuska, E. Pach, prof. dr habh. n.med. J. Drożdż
Biuro Badań Społecznych Question Mark Sp.z o.o., Al.. 1 maja 9 lok. 4, 90-717 Łódź, www.questionmark.pl
The evaluation examines the results of the programme.
To assess the effectiveness, efficiency, usefulness, and durability of the “Rehabilitation and Educational Programme for Cardiac Patients”.
Desk research.CATI with 424 participants.18 telephone in-depth interviews with patients.15 interviews with institutions participating in the implementation of the programme.Expert focus group interview.
28% of project participants were in employment upon completion of the programme (above the target); access to services has improved, but only one of the indicators on health conditions shows improvement.After three years, the majority of patients report benefiting from participation in the programme in terms of health and economic activity. Long-term results were achieved because of the motivation of patients to continue healthy behaviour.The quality of services provided are positively assessed by participants and experts.
There is a lack of data on the health condition of patients at entry to the programme and a lack of appropriate contextual information about their situation. Participation in the programme is too short to allow significant improvements in health.
Make support for patients more flexible and extend the duration of physical activity.Increase the number of physical activity sessions and the emphasis on physiotherapeutic care.Include services based on telephone rehabilitation/medical consultation.Introduce changes to the programme documentation to enable a realistic assessment to be made of the changes brought about by the programme and improve the consistency of the documentation submitted by beneficiaries.Modify measures and indicators by selecting parameters that better reflect the effectiveness of cardiac rehabilitation.
Evaluation of the effects of PA III (digital competences) of the Digital Poland OP, 2014-2020
Ewaluacja efektów realizacji III Osi priorytetowej (Kompetencje cyfrowe…) Programu Operacyjnego Polska Cyfrowa na lata 2014-2020
The evaluation examines the effects of support for the development of digital competences under the OP.
To assess the effects of support provided under PA3 in relation to the goals.
Monitoring and statistical data analysis.CAWI/CATI survey of 60 beneficiaries.CAWI/CATI survey of 65 project participants.10 interviews with beneficiaries.4 interviews with implementing bodies.2 expert panels.
The main strategic result targets have been achieved.Covid-19 adversely affected the implementation of projects (barriers to participation in trainings especially for the elderly, problems with deliveries, etc.).Education and information campaigns have helped to raise public awareness of the benefits of using digital technologies, including public e-services.There are significant unmet needs concerning digital competences and use of digital technologies.
Continue training activities on digital competences and take more account of the development of advanced skills in this regard.Promote to a wider extent projects taking account of the needs of people with disabilities.
Evaluation of the impact of measures on transport financed under the Łódzkie OP, 2014-2020, on the socio-economic situation of the region
Ocena wpływu inwestycji transportowych finansowanych w ramach OPIII RPO WŁ 2014-2020 na sytuację społeczno-gospodarczą województwa łódzkiego
Zdzisław Wolny, Wojciech Szymala, Aneta Stefaniak, Ewelina Kisiel
The evaluation examines transport investments implemented under the OP.
To assess impact of the measures to improve the transport system in terms of social, economic and environmental sustainable development.
Desk analysis of relevant documents.Analysis of monitoring data and official statistics.CAWI/CATI survey of 76 beneficiaries.CAPI survey of 400 rail passengers.13 interviews with implementing bodies.5 case studies.Expert panel.
The most significant effects were achieved by projects in rail transport, which helped achieve an increase of 1.4 million passengers in 2020.The main obstacle to the implementation of transport measures was COVID-19 pandemic, which led to delays in work schedules and to the reduction and temporary shutdown of public transport.Investments in the transport sector have led to unplanned effects, such as improved conditions for entrepreneurship and the increased visibility of beneficiaries and the Łódzkie region in Poland and Europe.
Promote investments in urban transportation by making use of adequate project selection criteria.Counteract transport exclusion in municipalities without railway connection by increasing the share of projects from small municipalities.Promote bicycle infrastructure projects.
Evaluation of the impact of cohesion policy, 2007-2013, 2014-2020, 2021-2027, (and other instruments) on the evolution of selected macroeconomic indicators at the national and regional level, in Poland
Ocena wpływu realizacji polityki spójności (i innych instrumentów) na kształtowanie się wybranych wskaźników makroekonomicznych na poziomie krajowym i regionalnym za pomocą modelu VESPA
Maciej Bukowski,Beata Belica, Przemysław Bruhn, Cezary Gołębiowski, Adam Rybkowski
EcorysPolska, www.ecorys.pl; WiseEuropa - Fundacja Warszawski Instytut Studiów Ekonomicznych i Europejskich, www.wise-europa.pl
The evaluation assesses the macroeconomic impact of the implementation of cohesion policy 2007-2013, 2014-2020, 2021-2027, in Poland.
To assess the impact of the implementation of cohesion policy 2007-2013, 2014-2020, 2021-2027 on the main macroeconomic indicators.
Analysis of monitoring and statistical data.Macro modelling (VESPA model).
Over the period 2004-2024, Poland's GDP has been on average 4% higher than it would have been without EU funding.Because of EU Funds, the rate of investment in fixed assets has been/will be higher by about 2.5-3.5 percentage points per year in the period 2011-2026.EU funds have been particularly important in accelerating the pace of convergence of Poland's poorest regions, especially in the east of the country.Over the period 2004-2023, the number of people in employment is on average almost 400,000 per year higher than in the scenario without EU funding.
Invest EU funding at a stable level of around 2-3% of GDP to stabilise its impact on the economy over the economic cycle.Rethink the direction of spending from pure infrastructure to the accumulation of physical and human capital, which may be more in line with the needs of Poland after 2029.
Operational Evaluation of the Castilla–La Mancha OP reprogramming
Evaluación Operativa por reprogramación del Programa Operativo FEDER 2014-2020 de Castilla – La Mancha
The evaluation assesses the financial reprogramming of the Castilla–La Mancha OP for 2014-2020.
The main purpose of reprogramming was to include an increase in the overall OP budget of EUR 62.3 million as a result of technical revisions.The reprogramming increased financing for TO1 ‘Research & Development & Innovation’ (+ EUR 2.6 million), TO2 “ICT” (+ EUR 5.4 million), TO3 ‘SMEs competitiveness’ (+ EUR 3.6 million), TO4 ‘Low-carbon economy’ (+ EUR 14.8 million), TO6 ‘Environment’ (+ EUR 7.9 million) and TO10 ‘Education’ (+ EUR 28 million).Other adjustment across SOs were made, affecting the overall budget for the respective TOs.Some output and outcome indicators were modified and associated targets changed.Introduction of Integrated Territorial Investment strategy for areas suffering socio-economic development problems and depopulation.The reprogramming slightly changed the initial OP strategy and intervention logic. However, the overall intervention remains pertinent, coherent and in line with the Europe 2020 Strategy.
First evaluation of the Greece-Italy Interreg OP, 2014-2020
Πρώτη αξιολόγηση της πορείας εφαρμογής και των επιπτώσεων του προγράμματος συνεργασίας Interreg V-A “Greece-Italy” (EL-IT) 2014-2020
Vasilissis Sofias Avenue 4, Athens 106 74, info@winconsulting.gr, https://winconsulting.gr/
The evaluation examines the implementation of the OP up to 31/12/2018.
To assess the progress of the OP and the level of achievement of objectives up to end-2018.
Desk research and literature review.Analysis of monitoring data and official statistics.Surveys of beneficiaries and MAs, Joint Secretariat, Monitoring Committee, National Authorities.
Milestones at 2018 have been achieved for output indicators, but financial progress was delayed, but 2023 targets should be met.Result indicator targets have been already achieved for all SOs except one.76% beneficiaries are satisfied with the communication strategy.Management procedures are working satisfactorily according to the MA and Joint Secretariat.
Focus more on innovation, competitiveness, and internationalisation.Address climate change effects that could produce tangible benefits for the regional economies.Improve quality and timing of the information made available on the website.
Evaluation of the criteria and project selection system of the Infrastructure & Environment OP 2014-2020 in Poland
Ewaluacja kryteriów i systemu wyboru projektów w ramach II osi POIiŚ 2014-2020
OP Infrastructure and Environment 2014-2020
Radomir Dyjak, Izabela Krok-Baściuk, Małgorzata Kachniarz, Agnieszka Puzio-Literska, Alicja Weremiuk
FUNDEKO Korbel, Krok-Baściuk Sp. J., ul. Baśniowa 3, lok. 311, 02-349 Warszawa, biuro@fundeko.pl, Tel: +48 22 658 03 70
The evaluation assesses the project selection criteria for the Polish Infrastructure and Environment OP 2014-2020.
To assess the project selection criteria and the selection system of Priority Axis (PA) II of the OP.
Desk research.Interviews with officials of the Managing Authority, implementing institutions, social partners and experts.CAWI survey of beneficiaries.22 case studies.Panel of experts.
Selection criteria and evaluation procedures are compatible with the OP’s objectives and horizontal principles.Selection criteria do not include all indicators defined in the OP.The majority of criteria are mostly formulated in a consistent and non-ambiguous way.Procedures in place to select projects are appropriate.The introduction of a second-level content-related assessment and the means of correcting are appreciated.Selection procedures are mostly generally conducted smoothly.
It is suggested to modify some criteria and to add others.
Ongoing evaluation of the Italy-France (Maritime) Interreg OP, 2014-2020 - 2021
Valutazione in itinere del Programma Interreg Italia-Francia Marittimo 2014-2020 - 2021
The evaluation analyses progress in implementing the OP, as at end-2020.
To assess advancement of the OP up to end-2020, the effects of Covid-19 on its implementation and the effects of the measures funded.
Monitoring data.Interviews with MA.Case studies.Survey of beneficiaries.
Covid-19 has affected the areas covered by the OP, especially the islands. The OP authorities have reacted promptly to this, reorganising some activities and implementing new ones, and changing the timing of measures.PA1 (SMEs) and PA2 (environment and risk prevention) are progressing satisfactorily, while the start of activities has been delayed for PA3 (transport) and PA4 (employment).Projects have significant effects on the areas concerned, especially on supply-chains and environmental threats, though the limited OP funding means that these are not evident at the macroeconomic level.Problems of bilingualism and first level control have been overcome, but those of MA staff turnover and the monitoring system remain.
Define different selection criteria for simple and integrated projects.Try to limit turnover of MA staff by reconsidering contractual conditions.Reconsider territorial strategic projects, which do not seem in line with the intervention logic of the new programming period.Identify further means of improving the monitoring system.
Assessment of progress in implementing innovation vouchers under priority axis 1 of regional OP Małopolskie Voivodship
Ocena przebiegu i efektów wdrożenia projektów wspartych w postaci bonów na innowacje w ramach 1 osi priorytetowej Regionalnego Programu Województwa Małopolskiego na lata 2014-2020
Maciej Maj, Justyna Wojtun, Piotr Bębenek
Openfield Sp. z o. o., ul. Ozimska 4/7, 45-057 Opole
The report assesses the implementation of innovation vouchers for a first wave of SMEs (November 2015-January 2016).
To assess the progress made in the implementation of innovation vouchers co-financed under the regional OP for Małopolskie.
The evaluation uses the following methods:Desk research.Individual in-depth interviews.CAPI surveys.
A total of 134 innovations are planned for the first wave, mostly relating to products.Most SMEs supported do not have an R&D department.Most of SMEs supported made use of services from research institutes and universities.The introduction of innovation usually takes from 2 to 6 months; but innovations were introduced only in 24% of SMEs examined.All SMEs availed themselves of the largest possible subsidy (i.e. 90% of eligible expenditure).The subsidy seems to be sufficient to undertake the planned investment for all SMEs surveyed.Most respondents reported that as a result of the innovation, their business situation had improved.
No limitations are indicated in the report.
The following recommendations are made:To monitor the effects of projects over the long-term.To better inform applicants that the subsidy only supports the development of innovation and not the implementation.
Analysis of territorial funding approaches in the Nordrhein-Westfalen ERDF OP, 2014-2020
Analyse territorialer Förderansätze im EFRE NRW
Prognos AGGoethestr. 8510623 Berlin | DeutschlandTel.: +49 30 5200 59-210Fax: +49 30 5200 59-201info@prognos.comwww.prognos.com
The evaluation examines the implementation of the Regio.NRW funding approach, which is aimed at improving regional competitiveness and innovation capacity and developing wider cooperation. Projects supported relate to innovation (PA1), SME support (PA2), climate protection (PA3) and urban development (PA4).
To assess the effectiveness of the Regio.NRW funding approach as compared with those adopted in other parts of Germany (namely ‘ITI-Westküste’ in Schleswig-Holstein, ‘RegioWIN’ in Baden-Wurttemberg, ‘Südniedersachsenprogramm’ in Niedersachsen and ‘EFRE-Schwerpunktegebiete’ in Bayern) in preparation for the 2021-2027 period.
Desk review.Contribution analysis based on 15 case studies (with 29 interviews) and 8 interviews with experts.Analysis of monitoring data.Final workshop with the MA and experts.Evidence from previous evaluation.
As of October 2019, 64 projects had been funded by Regio.NRW, mainly through grants.Regio.NRW provides a more effective coverage of the territorial context, mobilises regional actors and increases intra-regional cooperation, e.g. between chambers of commerce and universities and research institutes through joint committee meetings.Almost all projects would not have taken place, at least fully, without funding from Regio.NRW. Implementation based on a decentralised project management system has been successful.Beneficiaries report administrative procedures to be too burdensome, particularly because of the long process before project approval, mainly a result of the heterogeneity of projects supported; the size of beneficiaries, which requires much coordination; the heavy management burden falling on project coordinators; the difficulty in understanding programme documents; and lengthy procurement procedures.Despite differences in design, the territorial approaches in other regions produced activation and networking effects at the regional level, though they require an intensive capacity to plan and manage from the MA.
Consolidate further the networks built.Find financing solutions to continue many projects, particularly those involving network activities and advisory services.Maintain the same approach in 2021-2027 but making it more flexible and introduce a two-step selection procedure for innovative projects.
Review and assessment of social and professional inclusion actions with identification of effective and durable solutions possible to be implemented in ROP OV 2014-2020
Przegląd i ocena działań służących włączeniu społecznemu i zawodowemu oraz identyfikacja skutecznych i trwałych rozwiązań możliwych do wdrożenia w ramach RPO WO 2014-2020
Irena Wolińska, Sylwia Chacińska, Justyna Rybicka-Mróz
WYG PSDB sp. z o.o., Stowarzyszenie na Rzecz Rozwoju Rynku Pracy S-TO-S
Interim evaluation of the Provincia Autonoma di Trento ERDF OP, 2014-2020
Valutazione dell’esecuzione del PO FESR 2014-2020 della provincia autonoma di Trento
Piazza Sallustio 3, 00187 Roma, Tel: (+39) 06 77591884, E-mail: irsroma@irsonline.it
This is the interim evaluation of the Provincia Autonoma di Trento OP 2014-2020 financed by the ERDF.
To assess the implementation of the projects and activities planned and to provide a strategic reflection on the OP’s effectiveness, relevance and added-value.
The evaluation uses qualitative methods:Desk research of OP documentation and analysis of calls for tender;in-depth interviews relevant authorities;analysis of monitoring system data.Primary data come from in-depth interviews and secondary data come from OP documentation, the monitoring system and national statistics.
The rate of the OP’s implementation is relatively low.Delays are due to late approval of the EU regulations, lack of ownership of the authorities because of their low degree of involvement in the OP design phase, the diffused system of governance with the MA unable to play a decision-making and coordinating role and liquidity problems due to the Stability and Growth Pact.Planned interventions are coherent with needs.
The report is affected by the low rate of implementation and proposal for reprogramming.
Reprogramming of the OP should help to increase spending.The role of the MA in decision-making should be increased.The process of approving and financing projects should be speeded up by organising the work by project rather than by department and by appointing a single project manager.
Second evaluation of the Technical Assistance OP 2014-2020 in Romania
Al doilea raport de evaluare a Programului Operational Asistenta Tehnica 2014-2020
Michele Giove, Laura Trofin, Sofia Stoican, Vasilica Ciuca, Eva Militaru, Speranta Pirciog, Madalina Popescu, Liliana Lucaciu, Valentin Alexe, Alexandra Baciu, Andra-Corina Jurj, Maria Banu
ERNST & YOUNG, www.ey.com/en_ro - Institutul Național de Cercetare Științifică în Domeniul Muncii și Protecției Sociale (INCSMPS), www.incsmps.ro - QURES Quality Research and Support, www.qures.ro
This is the second evaluation report on the implementation of the OP. Data extend to end-2020.
To assess the contribution of the OP to strengthening the capacity of beneficiaries and ensuring the transparency of the European Funds.
Case studies.Analysis of administrative data.Survey of managing institutions, beneficiaries and trainees in horizontal activities.Two focus groups with stakeholders to validate results.
The effectiveness of the projects financed is limited, because of delays in implementation, especially because of Covid-19, and there is a risk of not achieving the 2023 targets.The capacity of beneficiaries to apply for funding and write good-quality proposals as well as to implement them has increased, and this is reflected by the higher success rate of projects.Working sessions and information events seem to have led to increased public awareness of the projects financed, though the National Information Centre does not play a fully effective role.Best practice with a view to 2021-2027 is linked to the streamlining of implementation procedures and to setting up transparent and effective procedures for selecting partners.
Limited level of implementation of the programme and small number of project managers available for interviews and for supporting the evaluators in data collection. In addition, some surveys have limited validity because of the low response rate.
Revise the communication strategy for projects financed by the National Information Centre with the help of MAs.Introduce intermediate targets for physical and financial progress to improve the monitoring of the OP.Increase training activities to support the sustainability of projects and resilience to changes in the institutional framework.
Evaluation of the governance system of the Legality OP, 2014-2020, in Italy - First part
"Rapporto sulla capacità di governance – parte 1"
https://www.ptsclas.com/ - info@ptsclas.com; https://www.irsonline.it/; https://www.consorziomipa.it/ - mipa@consorziomipa.it
The evaluation examines the adequacy and functioning of the governance of the OP. Monitoring data go up to 31 December 2021.
To assess the adequateness and functioning of the governance system of the OP up to end-2021 and to draw lessons for 2021-2027.
Interviews with MA.Interviews with regional representatives from Puglia and Campania signing the memorandum of understanding between the national and regional OPs.Analysis of monitoring data.Desk analysis of programme documentation.
At the end of 2021, payments received were only 46% of the funding available.The administrative capacity of beneficiaries is uneven across the country and causes delays, with smaller cities having to cope with staff shortages and unfamiliarity with administrative procedures and reporting requirements.Management and auditing staff have been burdened with control procedures, as often beneficiaries are asked to provide clarification of the information submitted and other documentation.The information system is not consistent with OP procedures and this has caused further delays.Delays have been reduced by the MA defining guidelines for implementation and territorial task-forces supporting beneficiaries.The 5 memoranda of understanding with regional OPs have helped strengthen inter-institutional dialogue and cooperation between national and regional stakeholders. In Campania and Puglia, this helped to plan the reuse of assets confiscated from the mafia to promote social inclusion and fight illegal employment in agriculture – so called “caporalato”.
Increase MA staff.Plan the timing of activities better.Simplify control procedures.Strengthen the role of the territorial task-forces.Ensure cooperation between national and regional MAs.
Evaluation of measures on Special Education under the Human Resource Development, Education and Lifelong Learning OP (ANADEDBM) and the Regional OPs, 2014-2020, in Greece
Παροχή υπηρεσιών Συμβούλου για την Αξιολόγηση δράσεων Ειδικής Αγωγής του Επιχειρησιακού Προγράμματος Ανάπτυξη Ανθρώπινου Δυναμικού, Εκπαίδευση & Διά Βίου Μάθηση (ΕΠ ΑΝΑΔΕΔΒΜ) και των Περιφερειακών Επιχειρησιακών Προγραμμάτων (ΠΕΠ)
OP Human Resources Development, Education and Lifelong Learning 2014-2020; All ROPs except OP Dytiki Makedonia 2014-2020
Eleftheriou Venizelou Avenue 2, 17676 Kallithea, Greece, icapcrif@icapcrif.com
The evaluation examines the measures on Special Education implemented under several OPs.
To assess three Special Education measures with the ultimate goal of using the findings to integrate them into the planning of subsequent implementation sessions of the programme and the next programming period.
Desk research.Analysis of statistical office data.Analysis of monitoring data.Surveys.Interviews.Focus groups.Other secondary data.
The three Special Education measures implemented have exceeded the targets planned.The budget for all three actions was spent as planned, but the projects were implemented at a lower cost than initially set, which indicates a high level of efficiency.Regarding relevance, the overall degree of all the objectives of the Actions amounts to 120% and is considered a very high level of relevance across all priorities.Most of participants consider that projects have had a positive impact and EU funding has led to increased awareness of recipients with special educational needs and to hiring of teachers for them.Moreover, participants think that the ESF measures contribute to improving the accessibility and social interaction of all students, and to addressing issues such as social exclusion, school failure and early school leaving.
Increase funding for special education measures and strengthen staff education and training.Improve collaboration between the various stakeholders involved, including the KEDASY (Interdisciplinary Assessment, Counselling and Support Centres), Mental Health Centres, Medical educational centres and define their role more clearly.Design and implement professional education programmes after high school.Evaluate the effects of the measures on the students participating in the projects.
Evaluation of 9 ERDF programmes, 2014-2020 (report 2)
Utvärdering av nio program. Regionala utvecklingsfonden, tematiskt mål 1. Delrapport 2 om målet: stärka forskning, teknisk utveckling och innovation
NOP Investments in growth and jobs ERDF 2014-2020; 8 ERDF ROPs 2014-2020
Johannes Henriksson, Ramboll Management Consulting AB
The report is the evaluation of the ERDF Thematic Objective 1 (TO1) for the 2014-2020, which is aimed at strengthening RTD and innovation.
To assess and describe the programmes’ results in relation to their goals.
Documentation analysis;theory-based evaluation;qualitative analysis;project management interviews.
The projects can be grouped into three main categories, where most of the funding has been allocated to complex projects which will show effects only after the end of the programming period.The programmes have the potential to contribute to the achievement of specific objectives and the result indicators, but the objectives are unclear and partly irrelevant.Few of the projects funded by the programmes relate to smart specialisation.
No particular limitations are indicated, but it was not possible to monitor the R&D costs of the companies supported.
There is a need to:introduce an updated approach to resource allocation, follow-up and learning between projects;develop pre-determined typologies of project types;develop relevant indicators for each type of project;clarify in the OP which project types are intended to be financed.
Evaluation of the "Pugliesi innovativi - PIN" measure for youth employment under the Puglia ERDF-ESF OP, 2014-2020
Intervento di valutazione in itinere relativo alle politiche giovanili iniziativa “pugliesi innovativi – PIN”
Fondazione Istituto Pugliese di Ricerche Economiche e Sociali, IPRES
The evaluation examines the “Pugliesi Innovativi – PIN” measure under IP 8i, which supports innovative start-up projects and accompanying services, and strengthening of skills of young people.
To analyse the characteristics of the measures implemented and of the participants and the preliminary results achieved.
Desk review of Programme documentation.Analysis of monitoring data.Analysis of statistical data.Questionnaire survey of participants (333 responses).
The measure mainly attracted high-educated young people rather than NEETs.The projects are mostly on social innovation and in small urban centres.The measure helped strengthen technical skills, but did not increase the sense of belonging to the local community, and gave rise to structured relationships with territorial bodies and companies other than project partners.According to many beneficiaries, funding is insufficient to start a business and there is a need to support the strengthening of relationships between PIN companies and local ones.
Take account of new youth needs that have arisen after the pandemic in designing youth policies and promote measures to strengthen skills together with accompanying measures.Rationalise the incentive schemes available for young people at regional level.
Evaluation of the effects of measures under TO3 of the Growth and Employment OP, 2014-2020, on the competitiveness of SMEs in Latvia
ES fondu 2014.–2020. gada plānošanas perioda darbības programmas “Izaugsme un nodarbinātība” 3.prioritārā virziena “Mazo un vidējo komersantu konkurētspēja” noslēguma izvērtējums
This evaluation examines the effects and impact of the support granted to SMEs under the OP. Monitoring data used go up to October 2023, but data on operations financed in 2022 and 2023 were only partly available.
To assess the effects and impact of the support for SMEs on the productivity and competitiveness of enterprises, turnover growth, profit and number of employees.
Desk analysis of relevant documents.Analysis of monitoring data and official statistics.Analysis of administrative data.Interviews with stakeholders.Case studies.Counterfactual analysis.Econometric analysis.
EU support did not always result in a competitive advantage for the enterprises supported.EU funding led to increased exports and capital productivity but less frequently to increased labour productivity. The qualification of the workforce remain challenging.The viability of new companies supported is higher than the national average; however, the funding was not sufficient to activate the business environment in the medium term, as the number of economically active companies remained constant over time.The overall leverage of financial instruments is higher than at the EU level, but the efficiency of guarantees is significantly lower.EU funds support did not necessarily reach high knowledge capacity sectors, whose potential remains untapped. Most represented sectors include food production, wood and cork manufacturing, and computer programming and consulting.The cascading implementation mechanism led to significant inefficiencies and data gaps.
Projects not completed at the time of the evaluation, low data quality and relevance.
Promote support to enterprises in the high and medium-high knowledge sectors.Ensure support for developing the qualifications and increasing productivity of the workforce, alongside support for investments.Improve efficiency of implementation, by reviewing the cascading mechanism.Improve monitoring mechanisms (i.e. data collection, review and further segment some of the indicators, review targets in line with the needs of the target groups, etc.)
Evaluation of the impact of the ESF on the results achieved in specific areas of the Human Capital OP, 2007-2013
Ocena stopnia realizacji celu głównego i celów szczegółowych PO KL oraz wpływu środków EFS na osiągnięte rezultaty w ramach poszczególnych obszarów interwencji Programu
Marzena Sochańska-Kawiecka, Ewa Kusideł, Edyta Makowska-Belta, Dominika Milczarek, Dorota Zielińska, Agnieszka Morysińska, Piotr Ziewiec
Laboratorium Badań Społecznych, ul. Okopowa 56/1, 01-04202-904 Warszawa, Tel: +48 22 300 14 59, e-mail: kontakt@laborato.pl
The evaluation was aimed at assessing the ESF impact on the results achieved in specific areas of the Human Capital OP (HCOP). This is the final report.
The main goal was to assess the effectiveness of interventions supported in different areas and the contribution of the ESF to the results achieved in these areas, as well as the impact on the socio-economic changes in Poland. In detail, the areas concerned are: the labour market; social integration; healthcare; education and training; adaptability of enterprises; good governance and capacity building.
The evaluation is based on the following methods:desk research;econometric models aimed at showing changes between 2008 and 2013 and at determining the impact of the interventions supported;case studies in four selected regions;in-depth interviews with representatives of implementing bodies;Computer Assisted Telephone Interviews (CATI) with project implementing organisations;Computer Assisted Personal Interviews (CAPI) with a representative sample of adult Polish citizens.
although employment declined by 7.1% on average between 2008 and 2013, it would have declined even further - by another 3.8 percentage points – if interventions funded by the HCOP had not occurred;the impact in the eastern regions was larger than in other regions. These received the largest amount of funding and had the worst initial situation;according to interviews, the best results were achieved in education and training, while the worst were in healthcare and the labour market;interventions for labour market and social integration were more needed, useful and effective than those in other areas;interventions increased regional convergence (mainly as regards labour market and social integration).
The lack of data concerning many indicators prevented a complete assessment of the results achieved.
The conclusions and recommendations focus on how to improve the design of interventions in terms of the quality of indicators, cooperation between the central government, local authorities and NGOs; and the involvement of individuals and companies in addition to the direct recipients of support. More detailed recommendations were also made on how to improve interventions in respect of the labour market, healthcare and education.
Impact evaluation of the Northern Ireland ESF OP, 2014-2020
An Impact Evaluation of the Northern Ireland European Social Fund Programme, 2014-20
The evaluation examines the Northern Ireland ESF OP up to end-2019.
To assess needs in the programme area, the progress made towards the achievement of targets for expenditure, output and results, and of the planned impact, the integration of cross-cutting themes and horizontal principles in the OP, the added-value and value for money.
Documentation analysis.40 interviews with beneficiaries/stakeholders.Surveys of programme participants.Case studies.Testing of the intervention logic.Cost-benefit analysis.
Participants exceeded the target for 2018 (173%), and are 86% of the 2023 target.51% of the total funding available (EUR 451 million) had been spent by end-2019.The N+3 target for 2019 was significantly exceeded.58% of participants are men.Upon completion of the projects under IPs 8.i and 8.ii, 10,088 participants found employment, while 7,173 entered education or training, in almost all cases exceeding the targets for 2023.Under IP 9.i, 976 participants with disabilities and 626 NEETs with Community Family Support found employment (respectively, 75% and 124% of the 2023 targets).Under IP 10.iv, upon completion of projects, 59% of participants achieved a full Level 2 framework, 63% achieved an NVQ2 Level, 57%, full Level 3 framework, and 60%, an NVQ3 Level, in all cases exceeding the 2023 targets.The current costs per participant are lower than those initially foreseen.Survey respondents report improving their career prospects, confidence and skills because of participation in the programme.Almost 90% of respondents under PAs 1 and 2 (Access to employment and Social inclusion) survey report that the programme helped them find employment and 74% that they would not have entered employment or education without support.81% of respondents to the Apprenticeships NI participant survey report that their employment situation had improved.According to beneficiaries, the flexibility of the OP allowed them to tailor support to the needs of participants.
Insufficient data as well as lack of access to relevant sources prevented a counterfactual analysis from being carried out as originally foreseen. The survey results are not representative. Analysis of wider benefits and social added-value does not take account of those who found employment 6 months after exiting programmes nor the impact on those who went into employment/training.
Maintain the successful principles of the OP in future schemes, while addressing labour market challenges.Insert an opt in/out clause into participant’s programme inductions confirming that the MA and beneficiary can share their name and contact details with third party organisations for the purposes of evaluation.Maintain a relationship-building approach with participants.Remove the age cap for apprenticeships.Broaden the spectrum of ‘hard outcomes’ and capture ‘soft’ skills.
Impact of measures for people with a migrant background under the Bund ESF OP, 2014-2020, in Germany
Themenzentrierte Studie: Wirkung von Maßnahmen auf Menschen mit Migrationshintergrund
ISG Institut für Sozialforschung und Gesellschaftspolitik GmbH, Weinsbergstraße 190, 50825 Köln, Tel.: +49 (0) 221 / 130 65 50, E-Mail: info@isg-institut.de
The evaluation examinees the cross-cutting issue of “equal opportunities and non-discrimination” in relation to people with a migrant background.
To examine the potential effects of the ESF-Federal programme and to make recommendations for the current and future programming periods.
Monitoring Data.2 Case studies.11 Interviews with experts and partners.Online survey of project promoters.Focus group.
The programme seems generally effective in promoting the social and labour market integration of those disadvantaged with a migrant background, though less so for older people and women.Support for the acquisition of a school qualification has beneficial effects for both younger and older people.There are few training opportunities in the IvAF: for some subgroups, especially refugees.Over two thirds of the 18,537 participants in the IvAF are employed or are in education or training. Women are more likely to be in school/vocational education (34%) than employed (27%), while for men the reverse is the case (29% and 40%, respectively).21% of those under 18 have qualifications and 64% have completed education. 48% of those aged 25-64- were in employment after taking part in the measures.Two months after leaving the programme, 39% of participants were in employment, 21% in training and 11% in general education.
The extent to which IvAF contributes to the social inclusion of the target group cannot be substantiated on the basis of the information available.
Develop and implement strategies for women refugees, involving families.Provide further support for training companies, especially in relation to apprentices with an uncertain prospect of remaining.Combine residence training courses with intercultural competence development.
Evaluation of the activities of Euro regions under the Bayern-Österreich Interreg V-A OP
Evaluation der Tätigkeiten der Euregios im Rahmen des INTERREG V-A Programms Österreich-Bayern 2014-2020
Interreg V-A-Austria-Germany/Bavaria (Bayern-Österreich) 2014-2020
Emamdeen Fohim; Dr. Roland Scherer; Dr. Kristina Zumbusch
Institut für Systemisches Management und Public Governance IMP-HSG of Universität St. Gallen; Tel.: +41712242525;
The evaluation assesses the roles and functions of the six “Euregions” in the implementation of the OP: Euregio Via Salina, Euregio Zug-spitze-Wetterstein-Karwendel (ZWK), Euregio Inntal, EuRegio Salzburg-Berchtesgadener Land-Traunstein, Inn-Salzach-EUREGIO, Euregio Bayerischer Wald-Böhmerwald-Mühlvier-tel.
To assess the activities and services provided by the six Euro regions and to make recommendations for their future role.
Desk analysis;30 interviews with representative of regions.
Only two of six regions (Euregio Salzburg and Inntal) are common cross-border regions. There are few cross-border structures (Euregio Presidency, Euregio General Assembly). Few regions have a common office and most rely on local decentralised contacts. Exchanges have usually a non-institutional character (thematic events, workshops, etc.). Nevertheless, the six regions are recognised as contact organisations for a wide range of cross-border issues and are perceived as "responsible" for cross-border cooperation. They support cross-border development, the implementation of large-scale projects and provide small project services.The regions communicate the activities of the programme and connect different actors to exchange ideas and explore ways to collaborate. But there is no systematic analysis of needs as regards cross-border development, partly because of limited capacity.Overall, the resources of regions are often inadequate, to implement large-scale projects but together with the regional coordination offices, they support major projects.Increasing focus on larger projects limits access to Interreg for many actors and the results of larger projects are less tangible for the population. Small projects make Interreg more tangible and visible. The complexity of small projects, however, is increasing and entailing more time and skills to provide support.
There is a need to:have a clear analysis of cross-border development issues and conditions in the sub-areas of each region;better match programme goals to the opportunities in individual regions;ensure more impact-oriented management with appropriate targets.
Evaluation of the impact on employability of vocational training supported by the Piemonte ESF OP, 2014-2020
L'effetto occupazionale della formazione professionale in piemonte - uno studio sui qualificati nel 2016
Luisa Donato, Maria Cristina Migliore, Samuele Poy
The evaluation assesses the effect on employability of vocational training provided to the unemployed and inactive in 2016 under the Piemonte ESF OP 2014-2020.
To assess the effects of the training measure supported on employability.
Counterfactual analysis based on propensity score matching (PSM) (treated and control groups made of 1,211 participants). Administrative data from the Monviso database and SILP Piemonte.
Due to the lock-in effect, the employability of the treated group is less than the control group during participation in training and also shortly after.12 months after the end of the training, the employment rate of the treated group was 4 percentage points (p.p.) higher than the control group (52% versus 48%) and increased to 8 p.p. after 18 months. The impact on employability is much the same for men and women, though on average men found a job around two months earlier than women.The effect on employability is larger for those over 40 (10 p.p. after 12 months), but this declines to 3 p.p. after 18 months. For those between 25 and 40, the positive effect increases over time but remains lower than for those over 40.The employment rate after participating was lower for long-tern unemployed.The effects of training are more evident for those courses relating to engineering.
Difficult to generalise the effects estimated.
Support for those who are looking for their first job after completing training should be strengthened.
Evaluation of advisory, development and networking measures supported by the Nordrhein-Westfalen OP, 2014-2020
Folgeinvestitionen im Nachgang zu Beratungs- und Aufschließungsmaßnahmen sowie von Vernetzungsaktivitäten, insbesondere von Unternehmen. Spezifische Evaluierung 1.8.
Dr. Jan-Philipp Kramer, Moritz Glettenberg
Prognos AG, Goethestr. 85, 10623 Berlin, jan.kramer@prognos.com, info@prognos.com , www.prognos.com
The evaluation examines consulting and networking measures under the OP, up to 30 June 2020.
To assess the effects of the measures to support SMEs up to June 2020, their contribution to the achievement of OP objectives, and the importance of following up of the investments.
Contribution analysis (validated, developed, and tested by experts).Analysis of existing literature.Analysis of monitoring data.4 explanatory interviews with Programme experts.Survey of 934 beneficiaries.27 interviews with beneficiaries.2 focus groups.
The added-value of the measures consists of the increased knowledge and competences of beneficiaries, easier access to finance and widening of company networking activity and investment strategies.According to the survey, induced follow-up investment by 221 firms supported can be estimated at EUR 28.5 million.A lack of access to funding as well as funding opportunities is identified as a key obstacle to investment.Measures generally contribute to both the OP and cross-cutting objectives.
Support a stronger internationalisation of networking activities.Strengthen monitoring and evaluation and further develop individual activities, including through continuous feedback from beneficiaries.
Evaluation of measures for increasing energy efficiency in the public sector and urban areas under the Thüringen ERDF OP, 2014-2020
Evaluierung der Maßnahmen zur Steigerung der Energieeffizienz und des Anteils erneuerbarer Energien in Kommunen, städtischen Quartieren und im Bereich der öffentlichen Hand im Rahmen des Operationellen Programms EFRE Thüringen 2014 bis 2020
Kienbaum Consultants International GmbH - https://www.kienbaum.com/en/
The evaluation examines the measures to improve energy efficiency in the public sector (IP 4c) and municipalities and urban areas (IP 4e) up to January 2021.
To assess the implementation of measures to improve energy efficiency in the public sector (IP 4c) and urban areas (IP 4e) and their impact up to January 2021.
Monitoring data.Interviews with the MA and IBs.Interviews with beneficiaries.Case studies.
Overall, progress in implementing the measures has been good, but a lack of management structure caused initial delays and Covid-19 has led to further delays at a later stage.The administrative burden is considered heavy by beneficiaries, especially in terms of the personnel tied up.The eCohesion system for managing the application process and cooperation from the MA and IBs are welcomed by beneficiaries.Not all targets are expected to be met by IP 4c measures, while those for IP 4e should be by 2023. Targets for result indicators for CO2 emissions under of IP 4c were set too low and were already achieved before the measures were taken.Energy efficiency measures in the public sector are expected to save around 6,860 tonnes of CO2 equivalent a year.
The approach had to be more qualitative-oriented because of the heterogeneity of measures assessed, the fact that projects were only partly completed and the unavailability of some data.
Establish management structures for 2021-2027 quickly.Increase the attractiveness of ERDF funding, to ensure that it is absorbed and targets achieved. Set ambitious but realistic targets and so far as possible implement metrological or balance sheet monitoring for each measure.Maintain and further develop the eCohesion system.Strengthen the competences of municipalities to manage ERDF financing.
Evaluation of the Saarland ESF OP, 2014-2020. Year 2021
Dr. Christian Lenhart, Dr. Vanessa Kubek
ISM - Institut für Sozialpädagogische Forschung Mainz e.V, https://www.ism-mainz.de/
The evaluation examines how the cross-cutting objectives “gender equality”, “equal opportunities”, “non-discrimination” and “ecological sustainability” were implemented under the OP, between 2014 and 2020.
To assess the implementation of cross-cutting objectives of the OP and identify good practices for the 2021-2027 funding period.
Analysis of projects documentation, including their assessments.
On “Gender Equality”, two projects of the OP have systematically focused on women as a target group and have made substantial contributions in the area of equal opportunities.Most projects of the OP make an explicit contribution to the objective of “Equal opportunities and non-discrimination”. Projects have supported the social and professional integration of people who are highly disadvantaged and at risk of exclusion.Most projects also contribute to the sustainability objective, which has been increasingly taken into account over time.
Organise information events about cross-cutting objectives of the ESF to improve future project proposals.Give a wider scope to the topic of cross-cutting objectives in the external presentation of ESF funding.Update the guiding criteria regarding cross-cutting objectives for assessing the applications of funding.
Summary of evaluations on growth and competitiveness in Sweden under ERDF OPs, 2014-2020
Ger företagsstödjande strukturfondsinsatser tillväxt och ökad konkurrenskraft i små och medelstora företag?
All regional ERDF OPs 2014-2020; NOP Investments in growth and jobs ERDF 2014-2020
Agency for Growth Policy Evaluations and Analysis, www.tillvaxtanalys.se, +46104474451, torbjorn.danell@tillvaxtanalys.se
This final report summarises evidence collected in three previous interim reports focusing on the assessment of eight regional and one national OP in support of SMEs. Data used go up to the end of 2020.
To evaluate the effectiveness of the measures, as well as their short-, medium- and long-term impact at regional level on competitiveness of SMEs.
Evidence from previous evaluations, that applied the following methods: Documentation analysisAnalysis of monitoring dataAnalysis of microdata on companiesSurvey for 16 regional (858 enterprises, 572 respondents) and 5 national selected projects (104 enterprises; response rate 34%)13 interviews with entrepreneursCase studiesCounterfactual analysis (D-in-D with multiple time periods weighted by PSM)Cost-benefit analysis.
ERDF interventions have been targeted at enterprises with higher productivity and larger proportion of younger, female, highly-skilled entrepreneurs and employees.Some 50% of SMEs supported report that knowledge improvement and concrete changes occurred in the company to a great or very great extent.Companies supported increased their net turnover by SEK 121,900, value-added by SEK 57,000 (EUR 10,250) and employees by 1% compared to control group.In metropolitan areas SMEs supported increased their employees by only 1% compared to other areas.The cost of support is on average almost SEK 61,000 per company. Measures are not cost-effective (around SEK 549,000 (EUR 50,000) per new job created).
The data available have a lag of around 2 years and do not cover a large number of projects that were still to be completed.
In the future, define measures capable of improving regional performance potential.Provide project implementers with guidelines, concrete examples, and an ex-ante evaluation on how to plan and implement projects.
Estimation of the value of the long-term result indicator 'Number of people with an improved labour market situation six months after leaving the programme', for measures under the Kujawsko-Pomorskie OP, 2014-2020
Oszacowanie wartości wskaźnika rezultatu długoterminowego EFS Liczba osób znajdujących się w lepszej sytuacji na rynku pracy sześć miesięcy po opuszczeniu programu w ramach RPO WK-P 2014-2020
ROP Kujawsko-Pomorskie Voivodeship 2014-2020
This is the fourth measurement of the value of the indicator 'Number of people with an improved labour market situation six months after leaving the programme’ and is focused on project participants who completed their participation in the measures between 1 January 2021 and 31 December 2023.
To measure the achievement of targets of the indicator ‘Number of people in a better labour market situation six months after leaving the programme,’ in relation to measures on employment, social inclusion and training funded by the OP.
Analysis of monitoring and statistical data.Meta-analysis of results concerning the measurement of the indicator.CAWI/CATI survey of 5,470 participants in the measures.
The overall gross value of the indicator reached 13.4%, while the net value – 6.9% (compared to a target value of 30%).The highest value of the indicator was achieved for employment measures for low-skilled people.The most effective forms of support in the context of the indicator were job placement, which led to the largest improvement in the professional situation of participants (34.6%), followed by training and courses for retraining (29.1%).Project participants rated high the quality of the support provided.
Limited trust of survey respondents. Incorrect contact details of respondents. Technical problems and errors in filling in the questionnaires.
Expand effective forms of support by promoting and developing job placement, organising courses and training that best support employment.Expand childcare, reimbursement of care costs, internships, and apprenticeships, which were highly rated by participants.Ensure flexible forms of support (e.g. online training) and work closely with employers to combine education with professional practice.Consider stress management, well-being, and work-life balance to help participants, especially women, cope with new challenges.
Evaluation of measures on renewable energies and energy efficiency under PA3 of the Schleswig-Holstein ERDF OP, 2014-2020
Begleitende Evaluierung des Operationellen Programms EFRE Schleswig-Holstein 2014-2020 (OP EFRE) Wirkungsevaluierung der Prioritätsachse 3
The evaluation examines the measures on renewable energies and energy efficiency under PA3 of the OP. Data used goes up to mid-2022.
To assess the impact of the measures on renewable energies and energy efficiency financed under the OP.
Analysis of relevant documentation. Analysis of monitoring and statistical data. Interviews with beneficiaries/target groups and MA. Case studies. Long-term assessment (2-3 years) through the Innovation biographies approach (document analysis, narrative interviews with projects’ leader, design and test of impact models, analysis of the contribution, semi-structured interviews with key players).
PA3 measures focused on the production and storage of renewable energies, and the improvement of energy efficiency in companies and public buildings. They contributed to sustainable growth objectives of the Europe 2020 strategy, including the reduction of CO2 emissions. S.O. 6 “Development of smart infrastructures for the optimal integration and use of renewable energy” (3.1.1): the projects led to an increase in the use of renewable energy and the production of renewable energy exceeded the target. S.O. 7 “Reduction of CO2 emissions from enterprises” (3.2.1): the estimated annual reduction in greenhouse gas emissions is 89.909 tonnes of CO2 equivalents. S.O. 8 “Increase the energy efficiency of public infrastructures” (3.3.1/3.3.3): the total annual decrease in final energy consumption in the public sector is 1,642,073 kWh/year. This results in an estimated annual decrease in GHG emissions of 1,260.25 tonnes (CO2 equivalents). The interaction with other instruments concerning climate protection and energy transition is considered as positive.
Make more efforts in improving project development and awareness raising to obtain the required innovation levels. Include support for monitoring and implementation along with support for energy saving investments, to motivate more companies to make investments. Ensure a quick start of funding in the 2012-2027 period. Continue promoting energy optimisation of public infrastructure and innovative heating networks.
Implementation of the Communication Strategy of ERDF Nordrhein-Westfalen 2014-2020 OP
Umsetzung der Kommunikationsstrategie des OP EFRE NRW. Spezifische Evaluierung 1.7
Evaluation of implementation of the territorial dimension in the Czech Republic, 2014-2020
Procesní hodnocení implementace územní dimenze
The evaluation examines the territorial dimension in the Czech OPs.
To assess whether strategic and methodological issues relating to the territorial dimension are adequately taken into account in OPs.
Desk analysis of documentsInterviews with Ministry of Regional Development, OPs and regional authorities (27)Survey of members of Regional permanent conferences (596 contacted, 17% response rate)Participation in territorial meetings
The concept of the territorial dimension differs between the EU Funds and the National Document on Territorial dimension.The advantage of taking account of the territorial dimension is the possibility of concentrating different resources in the same region, the disadvantage is that, in some areas, projects are not of high quality.The National Permanent Conference is intended to coordinate the implementation of territorial approaches but in reality it often has only a formal character. Its usefulness is mostly in communicating and exchanging information.Regional Permanent Conferences (RPC) are important as a means of identifying strategic territorial issues and needs, but there is no feedback from MAs on RPC requests and proposals.Regional action plans are potentially important to help regions increase their absorption capacity, but in practice they are merely “lists of projects”.
Identify better in a new version of the national document on the territorial dimension the scope and the objectives, through a participatory approach.Regional Permanent Conferences should become a real tool for integrating strategies and projects, and not only for the ESI Funds.Clarify the responsibilities and roles of members of NPCs for the next programming period and involve representatives of neighbouring regions.Boost the means of transferring experience, such as through round tables and mutual participation.
Interim evaluation of progress in implementing horizontal principles in the 2014-2020 OPs
Priebežné hodnotenie pokroku implementácie HP UR a HP RMŽaND na úrovni programov - záverečná správa
All OPs 2014-2020 except OP Quality of Environment 2014-2020
The evaluation is the first interim report on progress in implementing the Sustainable Development Horizontal Principle (HP SD) and Equality of Men and Women and Non-Discrimination Horizontal Principle (HP EMW) in ESIF programmes for 2014- 2020.
To assess progress in implementing HP SD and HP EMW as well as the means created for doing this and for ensuring they are applied.
The evaluation uses qualitative methods only in the form of desk research, analysis of relevant documents and data from in-depth interviews.
The HPs are in practice being implemented without major problems.the range of measurable indicators that are being monitored is significantly larger than necessary (e.g. OP Human Resources has 144 indicators).Indicators of horizontal principles are often indicators at the output level, reflecting in particular the intervention logic of programmes, so they do not directly measure the achievement of objectives in this regard.For the Sustainable Growth HP, there is no strategy at national level as yet and it is important to develop one to provide a framework for sustainable development in the country.
The limited state of the programme implementation significantly restricted the evaluation.
There is a need, for all HPs, to:Reduce the number of indicators and collect only relevant data.Streamline project monitoring and selection indicators that measure the main effects of horizontal principles (the final impact of ESIF programmes may be monitored through context indicators).Strengthen the administrative and professional capacity of staff involved in monitoring.And for HP SD, to:Finalise the strategic reference framework for its implementation.Further develop a methodology for monitoring the contribution of individual OPs to the HP.Focus on 10 thematic indicators to assess its implementation.Improve information exchange between OPs which may be complementary and identify potential areas of conflict.
Evaluation of the implementation of the Wallonie-Bruxelles ESF OP, 2014-2020
Évaluation de la mise en oeuvre du programme opérationnel
The evaluation examines the implementation of OP up to the end of 2021.
To analyse and assess the implementation of the OP and the results and effects of the measures up to end-2021.
Review of relevant documentation and review of evaluations.Interviews with relevant institutions.Focus groups with those involved in PAs 1, 3, and 4.Case studies of YEI projects.Survey of trainees who received ESF support for self-employment in 2018 (n=400).A survey of those employed who completed training in 2018.A quantitative survey of all operators under PA4.Analysis of monitoring data and indicators (ESF Agency).
The OP strategy is still relevant in relation to the territorial needs and most of the operational objectives have been achieved.Under PA 1, 17% of participants were in employment, including in self-employment, at the end of projects.Under PA 2, almost 20% of participants obtained a qualification and 5% of them were in employment.Under PA 3, 17% of participants were in employment after projects finished.Under PA 4, 36% of participants were in employment, obtained a qualification or were in education after projects ended.Surveys suggest that the measures led to higher employment rates than the indicators show.Participants are highly satisfied with vocational training measures.The OP has strong internal coherence, despite the geographical diversity of the region, and complements regional and other EU policies.Measures helped operators introduce new working methods and improve the quality of services and encouraged new partnerships and pooling of resources and means of achieving the objectives.
Aggregated data per project have been used as there are no data for individual participants. Monitoring data are limited only to indicators developed to monitor the performance of the OP.
Develop complementary indicators (e.g. a qualitative barometer on social innovation and indicators of positive exits).Focus more on measures for individual follow-up of trainees (or companies).Support partnerships through animation activities.Encourage a flexible approach for reaching targets.Improve digital skills.Increase participation of young women.
Mid-term evaluation of the implementation of the European Funds for Social Development Programme, 2021-2027, in Poland
Ocena śródokresowa realizacji Programu Fundusze Europejskie dla Rozwoju Społecznego 2021-2027
Joanna Jaworska, Patrycja Klimek, Oliwia Mazan, Małgorzata Teuber, Justyna Wojtun
The evaluation is the mid-term review of the programme and examines its implementation up to December 2024.
To assess the implementation system of the programme and the solutions adopted. To identify possible reprogramming needs, structural changes and identification of barriers to effective implementation.
Desk research (programme documents, reports, legal acts).Analysis of monitoring data and statistical office data.39 interviews with representatives of institutions implementing the programme.CAWI/CATI survey of 140 beneficiaries.FGI with experts and programme stakeholders.
The use of CST2021 (Central Data Communication System 2021) causes problems for employees of institutions providing support and beneficiaries. It contains new functionalities compared to the 2014-2020 period, and it has the potential to improve processes, but users need more time to adapt to it.The multi-stage project assessment has proven problematic, as it significantly prolongs the time required to sign project agreements from the end of the call for proposals, especially in the case of competitive calls.Most of the selection criteria in the programme fulfil their objectives, but there are ambiguities in the application of horizontal criteria, and the requirements are excessively detailed and complex, which leads to interpretative difficulties for both applicants and the experts assessing applications.
Low number of projects financed for some of the measures analysed, which limits the sample to analyse.
Introduce systematic monitoring of the training on the information systems and the level of knowledge acquired, in order to adapt training programmes to the real needs of users.Develop training materials for users of the CST system, in simple and understandable language.Simplify the process of project evaluation and selection by reducing the number of selection stages to speed up the implementation of the programme.
The effectiveness of the implementation of the OP Smart Growth (Module I, stage I) in Poland
Ocena skuteczności wdrażania PO IR (Moduł I, Etap I)
EGO – Evaluation for Government Organizations s.c., ul. Podmokła 53 B, 04-819 Warszawa, e-mail: evaluation@evaluation.pl, LB&E Sp. z o.o. 05-807 OwczarniaKsiążenicka 98, e-mail: kontakt@lbie.eu, MCM Institute Poland Sp. z o.o., Al. Niepodległości 80, 02
The evaluation assesses the support provided by the National Centre for Research and Development (NCRD) within the OP Smart Growth.
To compare the support provided by the NCRD with the R&D support provided by other institutions and to identify possible improvements.
Desk Research;interviews with NCRD officers, companies and scientific institutions;CATI questionnaires to companies;panel of experts;workshop with NCRD representatives.
The NCRD is generally considered to be more efficient than regional OP MAs by beneficiaries;companies interested in R&D support prefer to submit their applications to the NCRD rather than regional OP;neither the NCRD nor the regional allow for the purchase of necessary equipment to carry out research;apart from minor exceptions, there is lack of support for projects at national level.
There is a need toexpand the current support provided by the NCRD;include purchase of equipment in eligible costs;improve management skills at regional level through implementation of good practice and staff training.
Evaluation of measures for older workers in Lubelskie under the Human Capital OP, 2007-2013
Ocena działań na rzecz wydłużenia aktywności zawodowej osób starszych w kontekście niekorzystnej sytuacji demograficznej w województwie lubelskim
This evaluation focuses on the policies to support older workers and the elderly in Lubelskie Voivodeship, as implemented within the OP Human Capital framework.
The objectives of the evaluation are:to assess the demographic context in Lubelskie Voivodeship, with a focus on older people.To assess the main needs, risks and barriers to the labour market for older people.To assess the effectiveness and efficiency of the support provided to the elderly under the regional component of OP HC.To analyse the measures aimed at keeping older people in employment.
Desk research.CATI analysis targeting projects participants.Interviews with companies that employ older workers and with beneficiaries.Difference-in difference counterfactual analysis.
The share of people aged 50+ in Lubelskie Province is steadily increasing but their economic activity rate is low.Employers tend to believe that the 50+ age group lacks important attributes, especially women.The counterfactual analysis shows that support for the unemployed aged 50+ had a net positive effect on employment, though there was a relatively large deadweight effect.
Action to improve the qualifications of those aged 50+ should take account of common problems that affect them (e.g. a relatively low level of education).Plans related to health policy programmes should take more account of the specific needs of those of 50+.Measures should be taken to overcome barriers to training of those of 50+, which hinder their employment in the private sector.
Impact evaluation of the Zachodniopomorskie OP, 2014-2020, on social inclusion
Ocena wpływu RPO WZ 2014-2020 w zakresie włączenia społecznego w regionie
ROP Zachodniopomorskie Voivodeship 2014-2020
Stanisław Bienias, Piotr Fuchs, Katarzyna Hermann-Pawłowska, Sabina Kasoń, Maciej Kolczyński, Zofia Rutkowska, Paulina Skórska, Elżbieta Stosio-Sielach, Piotr Stronkowski
Piotr Fuchs Smart Research, ul. Wojskowa 5/73, 60-792 Poznań - IDEA Instytut Sp. z o.o., Wierzbica 57B, Gmina Serock 05-140, www.ideainstytut.eu
The evaluation examines social policy, existing issues and social potential in the region, the impact of the OP on social inclusion, and makes recommendations for the future.
To assess the socio-economic situation in the region, the effectiveness, efficiency and impact of the OP on social inclusion and to make recommendations.
Analysis of statistical and monitoring data.Analysis of documentation, other studies and reports.Interviews with experts, NGOs, local government units and social service providers (29), regional institutions (4), project promoters (21) and project participants (24).Surveys of local government units (response rate 88%), 128 social service providers, 393 residents of the region, beneficiaries (97 responses out of 171; in respect of revitalisation, 28 responses out of 53) and project participants (1,000 responses).Expert panel and workshops for developing and clarifying recommendations.Case studies.
92% of ESF output indicator targets had been achieved on average by Q1 2021.During the OP implementation, the allocation to active inclusion (IP 9.i) had been reduced by 59%, while allocation to social services (IP 9.iv) had increased by almost four times.Support was not sufficiently targeted at areas with the greatest needs.Few local authorities apply for EU funds, mainly because of capacity issues and project implementation requirements.37% of participants previously unemployed were employed after the programme under IP 9.i, and 21% of participants under IP 9.v.The support led to 30% of jobs being created in social enterprises (IP 9.v) by June 2018.The great majority of project participants state that the projects helped them to gain or improve professional skills, improve well-being, strengthen social ties, cope with everyday problems better and have a more independent life.According to 77% of participants in projects under IPs 9.i and 9.v, their situation would have been worse had they not participated, but 75% of those participating in projects under IP 9.iv stated that their lives would not have changed.Support for families (IP 9.iv) helped participants to better fulfil their role as parents, although it had limited effect on improving their financial situation.The projects have improved access to social services and allowed them to broaden their range.
Because of the specificity of the target group, it was not possible to carry out a counterfactual analysis. Indicators for health services are not evaluated. Because of the limited data at district level, there were difficulties in measuring the territorial distribution of participants. The indicator on the percentage of participants in employment after the programme understates the actual situation. The effects of projects on reducing the risk of children being placed in foster care could not be assessed, since only 9% of respondents with families reported that they were facing such a risk.
Use ESF resources to finance activities with a higher risk of not being implemented.Reduce the allocations to IP 9.i.Promote cross-sectoral partnership projects.Intensify support for counties in organising family foster care.Provide expert support for municipalities and districts in designing and promoting social services and in developing information systems on the support available.Provide support for local authorities in developing personnel skills.Maintain the financial allocation for revitalisation measures.Incentivise participation (other than for local government units) in revitalisation programmes.
Synthesis report on the results of the impact assessment of the Regional OP, 2007-2013
Raport de Sinteză privind rezultatele evaluării impactului intervenţiilor POR
Eugen Perianu, Alina Boşoi, Cristina Mocanu
OTP Consulting, Bd-ul Dacia, nr.83, etaj 1, sector2, Bucureşti, www.otpconsulting.ro; AAM Management Information Consulting, Str. Eremia Grigo-rescu, No 20, Sector 1, Bucureşti, www.aam.hu; HETFA Research Institute Ltd. H-1051 Budapesta str. Október 6. nr
The report summarises the evaluation of four Priority Axes of the Regional ROP, 2007-2013 in Romania: PA 1, “Support for the sustainable development of cities – urban growth poles”; PA 3, “Social Infrastructure improvement”; PA 4,“Strengthening the regional and local business environment”; and PA 5, “Sustainable development and promotion of tourism”.
To disseminate the results of the evaluation carried out on interventions supported under the Regional OP.
A mix of qualitative and quantitative methods were used in the evaluation, including:questionnaire surveys;analysis of documentation;cluster analysis;regression analysis;in-depth interviews;case study;benchmarking;expert panel;SWOT and PEST analysis;Counterfactual analysis (using propensity score matching and difference in difference).
PA 1: projects created new short-term jobs in the construction of new roads and other urban infrastructure.PA 3 projects contributed to increasing health services and their quality, the number of patients being treated and their degree of satisfaction with the healthcare facilities. They also contributed to upgrading social services, increasing the number of users and improving equipment in social centres. In addition, they improved emergency response equipment, reduced the intervention time and increased the population covered. Moreover, they increased the quality of pre-university and university education and study conditions.PA 4: projects improved the quality of the business’ premises and created new jobs, led to the decontamination of 113 ha of land, though the construction of business incubators on the areas concerned is delayed. Support for micro-enterprises resulting in more jobs being created and business capacity being increased.PA 5: projects helped to increase the number of visitors to cultural heritage sites, the number of hotel rooms and accommodation units, tourist arrivals, overnight stays and occupancy rates. While the number of people employed and business turnover increased, profits did not. In addition, support for increasing tourist potential improved the national brand strategy was improved but this was not reflected in the increased foreign tourist numbers.
There were difficulties collecting information on the treatment and the control groups, differences in the unit of measurement between the monitoring indicators and the indicators reported by beneficiaries and limited availability of result indicators.
There is a need to:simplify public procurement procedures and develop the means of supporting beneficiaries, provide specialised training for applicants to improve their technical, planning, financial and organisational capacity;support future evaluations by improving data on measures supported, typology of target groups and beneficiaries and define and monitor complementary indicators to better quantify the impact of each area of support.deepen and improve the evaluation process through better availability of data on both beneficiaries and non-beneficiaries, to carry out an evaluability study before an evaluation so as to assess the kinds of information available, guide the evaluation process and help select evaluation methods.
Evaluation of the implementation of the Azores Programme, 2021-2027
Avaliação de Fundos Europeus: Operacionalização do Programa Açores 2030
Sérgio Barroso, Luís Carvalho, Ana Caetano, Carla Figueiredo, Martim Carvalho, João Guimarães, Manuel Reis
Centro de Estudos e Desenvolvimento Regional e Urbano (CEDRU), https://cedru.com/contactos/, geral@cedru.com
The evaluation examines the measures supporting sustainable development under the Programme. It covers the implementation period between 2022 and 2024.
To assess the implementation, effectiveness, relevance, and the operational efficiency of the programme. To suggest potential adjustments to achieve the objectives of the programme.
Analysis of monitoring data.Document analysis.5 interviews with the MA and intermediate bodies.Survey of beneficiaries (21.9 % response rate).2 focus group discussions, one with the MA and intermediate bodies, and one with beneficiaries.
The programme faced significant delays due to systemic constraints: a one-year gap between the approval of the Partnership Agreement (July 2022) and the first calls (August 2023), difficulties in adapting the Information System, and competition with the Recovery and Resilience Plan.As a result, the implementation rate as of 31 May 2025 was only 7.3%, although this was higher than the national average (3.0%).Administrative procedures and technical capacity show three main weaknesses: tight deadlines (48% of beneficiaries consider them to be highly restrictive), bureaucratic complexity (44.4% report obstacles) and lack of qualified human resources. These issues mainly affect smaller and less experienced beneficiaries.Demand for funding varies significantly between areas of intervention. Areas such as scientific research and entrepreneurship recorded demand exceeding 300% of the allocation, while others, such as "young investors", recorded only 10%.The Funds Desk emerges as a significant innovation in the administrative architecture of PT2030, providing a unified platform that simplifies beneficiaries' access to the various funding programmes.The realistic scenario projects an implementation rate of 80% by 2029, provided that corrective measures are implemented across three areas: administrative management, technical capacity of beneficiaries, and coordination among financing instruments.
Speed up the assessment of applications, simplify procedures, and ensure greater IT robustness to avoid delays.Modernise digital infrastructure and administrative procedures and develop differentiated communication strategies.Adapt management criteria and practices to regional specificities.Institutionalise adaptive learning mechanisms, create databases of lessons learned, and promote regular meetings between the MA and intermediate bodies.Improve the monitoring system to capture structural impacts, harmonise European and regional metrics, and ensure adequate follow-up mechanisms.
Impact evaluation of Smart Specialisation Strategies implemented under 13 ROPs, 2014-2020, in Greece
Αξιολόγηση Επιπτώσεων των Στρατηγικών Έξυπνης Εξειδίκευσης (RIS3) των ΠΕΠ 2014-2020
Yannis Tolias, Michalis Metaxas, Christos Chiou, Apostolos Apostolidis, Theodora Historiou
Innovatia systems: Dodekanisou 22, 54626, Thessaloniki, tel: +302310567442, http://serial.innovatiasystems.eu/01/index.php/el/
The evaluation investigates the impact of 13 Regional OPs on the relative Strategies for Smart Specialisation (RIS3) in the period 2014-2020. The data used go up to July 2023.
To assess the implementation of Smart Specialisation Strategies and their impact.
ToC (for each policy instrument) to assess the expected impact. Analysis of monitoring data. Analysis of official statistics. Structured interviews with the thirteen MAs and/or the RIS3 Management and Monitoring Bodies. Online survey of beneficiaries. Analysis of documents (OPs, RIS3 strategies and action plans, evaluations of the ROPs and Call for proposals). Validation seminar.
As of July 2023, seven OPs had achieved 90% or more of their targets for end-2023 for more than half of the result indicators. The implementation of RIS3 did not meet the expectations of front-loaded and flexible support to beneficiaries. The average period from the approval of the RIS3 and the first relevant Calls was 3.5 years. Compared to other national and international sources financing RIS3s, the OPs had rather limited funds, which are not able to generate significant impact. The existence of competitive calls funded by other programmes might have created internal competition between funding sources, particularly for TO1. More than 60% of beneficiaries intend to apply for calls for similar measures in the 2021-2027 period. There is an over-concentration of resources in the capital cities of six out of the thirteen Greek regions, which serve as hubs for innovation potential. In the remaining regions, there are multiple growth poles. The measures have the potential to significantly improve research infrastructure, digital capacities of businesses and promote technological collaborations and the employment of qualified professionals. However, there is a risk of concentration of resources in already advantaged areas.
It was not possible to assess the impact of the measures, as most of them have not yet been completed at the time of the evaluation.
Improve coordination and cooperation between sectoral and regional programmes and exploit synergies to achieve economies of scale. Make the policy mix more responsive to the needs of different territories at both national and regional levels. Monitor RIS3 at a higher level of detail, beyond the indicator system established in the Common Provisions Regulation, by using both publicly available data and the results of field surveys. Define ex-ante outcome targets, direction and magnitude of impact of the RIS3, to improve its evaluation.
Evaluation of the relevance, efficiency and effectiveness of the Notio Aigaio OP, 2014-2020
Αξιολόγηση συνάφειας, αποδοτικότητας και αποτελεσματικότητας του Επιχειρησιακού Προγράμματος Νοτίου Αιγαίου 2014-2020
remaco@remaco.gr; https://remaco.gr/en/home-en/
The evaluation examines the implementation of the OP until 25/09/2023.
To assess the performance of the implemented measures, the degree of achievement of the planned objectives, and the effectiveness and efficiency of the OP. To assess the contribution to the Europe 2020 strategy.
 Desk research. Analysis of monitoring and statistical data.
 The Programme has already achieved its financial targets. Financial absorption is higher for measures strengthening competitiveness and innovation, regional cohesion and sustainable development. Measures on social inclusion, human resource development and basic infrastructure show lower progress, though they seem to have a sufficient level of commitments to cover project withdrawals or delays.
Reprogramme the allocation of funding between the various PAs and IPs in order to ensure the largest absorption of funds.
Final Evaluation of the Sustainable Transport Programmes, 2007-2013
A fenntartható közlekedésfejlesztés értékelése. Ex-post értékelési jelentés a 2007-2013-as időszakról
György Kukely, László Dulicz, Tamás Jancsó, Regina Kapitány, Balázs Orosz, Andrea Rácz, Zsolt Zábrádi
The evaluation assesses the sustainability (environmental, economic and social) of transport projects carried out in Hungary in the 2007- 2013 period. It is focused mostly on urban and suburban projects but extended transport networks are also covered.
The evaluation has multiple objectives: to examine the extent to which sustainability criteria have been applied and the factors and directives which have most affected developments as well as to assess the results of the projects carried out.
Desk-based review of background documentation and data from national and local development plans and projects.Case studies.
Sustainable development of the transport network was evident in the nationwide strategy plans.Most of the projects related to railways on the TEN-T and urban-suburban lines.25% of the projects were carried out in an environmentally-friendly way, the development of public transport and cycle paths was a prominent feature.Retaining passenger numbers is not only improving the quality of service but awareness-raising and extolling the virtues of environmental friendliness are also important.The number of well-prepared transport projects increased significantly over the 2007-2013 period, but there were still a number of problems concerning the quality of preparation and quality assurance.In the selection of projects, financial factors were decisive over and above social factors and a short-term approach was wide-spread.There was insufficient effort to optimise costs.Intelligent transport systems contributed to improving environmental sustainability.The projects carried out alleviated pressure on the environment.Increasing Energy efficiency was not fully successful because of road improvement that led to more frequent use of cars.
The evaluation does not mention any limitations.
When formulating development strategies, transport should be treated as a means of creating a more competitive economy rather than as a goals in itself.Urban strategy should treat together cover spatial planning, pricing, public transport and energy supply for environment-friendly vehicles.Sustainability, competitiveness, energy efficiency of public transport and awareness-raising should be emphasised more.To improve the financial sustainability of public transport, operating costs should be reduced.Intelligent transport systems need to be used more in the next programming period.There should be greater emphasis on improving communication between service providers.Financial sustainability needs to be considered when preparing projects not when carrying them out.Maintenance of infrastructure and vehicles needs to be accorded more attention.State and local sources have to be involved in financing transport investment as well as EU funding.
Evaluation of monitoring indicators and verification of the intervention logic of the Integrated Regional ERDF OP, 2014-2020
Evaluace nastavení indikátorové soustavy IROP, prověření intervenční logiky a vyhodnocení informačních zdrojů a datové báze IROP
The report analyses the system of indicators, validates the intervention logic and assesses the information sources for the Integrated Regional OP in the Czech Republic, aimed at improving public administration and public services.
To examine the adequacy of the indicator system for monitoring and effective OP management and evaluation, the availability and reliability of data and the result indicators and to assess the intervention logic.
Desk analysis and review of existing documentation and literature.Meetings with managers in charge of the indicator system.On-line survey and structured phone interviews with experts on specific objectives.Pilot survey to test proposed evaluation designs for two specific objectives (SO 2.1 and SO 3.1).
The OP intervention logic is not detailed enough and should be improved to be a useful basis for designing and carrying out impact evaluations.Overall, the intervention logic, the indicator system and the information sources are adequate.However, the indicator system has a number of deficiencies that may constrain assessment of the OP: it is focused on outputs and on monitoring of progress rather than on measuring outcomes and impacts; there is no measure of the quality of the services financed by the OP and of impact on target groups; the target values are inadequately defined or not defined at all; some indicators are difficult to measure; some of the macro indicators are dependent on exogenous factors rather than reflecting the impact of the OP.Some indicators are problematic since they can be affected by external factors.The data sources which are currently used are incomplete and of poor quality. They often do not provide data for the required frequency or the level of detail.
The number of completed projects is small and the timing of the pilot surveys did not facilitate data collection.
There is a need to:use, in the next programming period, methodological guidelines for the design and monitoring of indicators;organise multi-round validation sessions on the intervention logic of specific objectives;revise the system of indicators and set up a regular reporting system;incorporate measurement tools in project design and focus more on the measurement of impact on wellbeing and of the quality of services supported;use multiple and high quality secondary and primary data sources to enable triangulation of results.
Evaluation of financial instruments funded by the ERDF in 2007-2013
Pénzügyi eszközök értékelése (2007-2013)
OP Economic Development 2007-2013; OP Central Hungary 2007-2013
The report examines the level of demand for financial instrument, and the enterprises using them and assesses their overall performance during the 2007-2013 period. It also compares EU-supported instruments (JEREMIE) with similar instruments financed from domestic sources.
To assess the use of the financial instruments in Hungary during the 2007-2013 period.
Analysis of fund reporting documents and programme monitoring documents (both EU and national financial instruments).National statistics data.
Many national financing instruments are available (along with the EU JEREMIE initiative) with some overlap between them;regulations (as regards state aid and the Structural Funds) are the most important criteria for choosing the instrument to use;the goal of developing the domestic capital market was successfully achieved by JEREMIE funds and other domestic funds. Because of these a domestic venture capital industry has emerged;milestones should be set binding the target companies and encouraging them to achieving agreed results;most support went to the distributive trade, car repair, professional, scientific and technical activities as well as to manufacturing, which is in line with the sectoral division of domestic businesses.
Businesses need a kind of resource map that will guide them to the appropriate funding instrument. There is a need for mentoring institutions so that they can help businesses to choose the most suitable financial instrument;financial instruments will be more attractive if they were less constrained administratively than non-refundable grants;combined products should be fostered because they are considered to be more attractive to businesses.
Evaluation of PA 2 "Economic productivity" of the Berlin ERDF OP, 2014-2020 - Interim report
Evaluierung der PA 2 "Produktivität der Wirtschaft" - Zwischenbericht
Kovalis, Am Wall 174, 28195 Bremen, meyer@kovalis.de - IfS - Institut für Strukturforschung und Stadtpolitik GmbH, Lützowstr. 93, 10785 Berlin, schwab@ifsberlin.de
The evaluation examines PA2 of the OP aimed at increasing productivity.
To assess how and to what extent productivity improved as a result of the measures implemented. The report is part of a broader evaluation and it sets out the theory of change which will be then tested in a subsequent exercise.
Statistical and monitoring data analysisLiterature reviewInterviews with actors and projects managers.
By end 2019, 94% of available resources were allocated and 70% spent.Funding mainly goes to business start-ups and investment in SMEs. It takes the form predominantly of loans.1,108 projects have been approved and 1,224 start-ups and SMEs have been supported.The knowledge economy (industry and knowledge-intensive services) were supported in nearly half of the projects implemented.Without ERDF support, SMEs and start-ups involving high level of innovation would have found it difficult to obtain loans.
First report on the communication strategy of the ESF OP, 2014-2020
Primo Rapporto tematico sulla Strategia di comunicazione del PO FSE 2014-2020
Social innovation – Reducing training drop-outs in elderly care in the Bayern ESF OP, 2014-2020
Soziale Innovation - Verringerung von Ausbildungsabbrüchen in der Altenpflege
Christian Loschelder, Jenny Bennett, Franziska Dachner - Institut für Sozialforschung und Gesellschaftspolitik GmbH, Weinsbergerstraße 190, 50825 Köln
The evaluation examines projects financed by the Bayern ESF OP aiming at reducing training drop-outs through socio-educational assistance for people with special needs.
To identify the problems which arise in the training for the care of the elderly and the measures implemented and to assess whether these measures reduce drop-outs from training.
Statistical office data.Monitoring data.Case studies of two selected projects (several interviews each).
Staff shortages and a heavy work load mean a lack of practical instruction and support and insufficient time for trainees to cope with the amount of learning required. Trainees also have to cope with financial problems, low education level and balancing family and career responsibilities (most of the trainees are women).There is a need for an additional socio-education supervisor.The measure led to a reduction of the training drop-out rate and to an improvement of the quality of the training as a result of the socio-education support provided. Closer cooperation between the socio-education professionals, the teachers and practical trainers is called for.Given the short duration of the project, it is questionable how far it contributed to the achievement of the programme target.
The case studies are not representative given the small number.
Improve the awareness of the project among potential trainees.Support fewer vocational schools in order to intensify the support.
Evaluation of the implementation of the YEI and ESF under the Knowledge, Education and Growth OP, 2014-2020
Badanie efektów wsparcia zrealizowanego na rzecz osób młodych w ramach Programu Operacyjnego Wiedza Edukacja Rozwój
Instytut Badań Strukturalnych - Imapp Sp. z o. o. - IQS Sp z. o.o.
Instytut Badań Strukturalnych, Rejtana 15/28, 02-516 Warszawa, ibs@ibs.org.pl -Imapp, Rejtana 15/25, 02-516 Warszawa, imapp@imapp.pl -IQS Sp z. o.o., ul. Francuska 37, 03-905 Warszawa, kontakt@grupaiqs.pl
The evaluation is concerned with assessing the support provided to young people under the Operational Programme Knowledge, Education, and Development (OP KED), which is one of the main measures for implementing the Youth Guarantee (YG) in Poland using the funds provided by the European Social Fund (EFS) and the Youth Employment Initiative (YEI).
The aim is to assess the support for young people implemented under the OP KED.
The evaluation is based on the following research methods:Desk research: literature and database of support recipients (21,204 persons);computer-assisted telephone interviews with project participants (1,150 CATIs);computer-assisted web interviews with project coordinators (422 CAWI interviews);in-depth interviews with support coordinators at the local and central level (10 IDIs).
The support has been implemented in a suitable and effective manner:The composition of the project participants was in line with that of the target groups, however there was evidence of ‘creaming’ in the PLO projects (selecting participants with more chances on the labour market);the projects failed to reach people with disabilities or those from ethnic minorities effectively;forms of support were in line with the needs of the participants, but assessment of these needs (PLO) was often neglected or conducted as a mere formality; the projects implemented by the VLO were targeted at a younger population with more severe problems and the support offered was more comprehensive compared to PLO projects;due to a lack of a counterfactual evaluation the real impact of the project was not measured. The evaluation provides information on the proportion of project participants in employment 3 months, 3-6 months and over 6 months after participating in the measure. When the study was conducted, 67% of the project participants were in employment (87% in the case of the PLO and 35% in the case of the VLO).
Due to the delay in launching the programme, some schemes were not evaluated;lack of a control group to assess the real impact of the intervention;the cost incurred per project participant (cost-efficiency) cannot be precisely calculated on the basis of the data available. A detailed study of the cost-efficiency of support will be carried out in 2018.
The risk of ‘creaming’ should be minimised; stronger emphasis in the PLO projects on an authentic, in-depth assessment of needs and the provision of comprehensive support;the timetable of the projects should match the situation of the target groups, particularly as regards the school year. This would require a revision to the traditional timing of projects (started in the spring and finished in the autumn). The PLOs should have an opportunity to implement projects lasting longer than a year.
Impact evaluation of measures under PA9 “Education, skills, and lifelong learning” of the Competitiveness and Cohesion OP, 2014-2020, in Croatia
Vrednovanje učinka Operativnog programa Konkurentnost i kohezija 2014. – 2020. Prioritetna os 9: Obrazovanje, vještine i cjeloživotno učenje (Grupa 8)
OP Competitiveness and Cohesion 2014-2020
Equinox Consulting kft, E: info@equinox.huUrbanex, E: ivana.katuric@urbanex.hr
This is the final evaluation report focusing on measures under PA9 of the OP, up to 31 December 2021.
To assess the main results, impact, effectiveness and efficiency of the measures financed by the OP under PA9, as well as their contribution to the national objectives for early school leaving and people completing higher education.
Analysis of relevant documents.Analysis of monitoring and statistical data.Interviews and surveys of stakeholders.Case study.Theory of Change (ToC) design and testing.
151 primary and secondary schools were provided with ICT equipment, with 49,795 teachers and 74,254 students using it during classes. As a result, 91% of secondary schools achieved an e-enabled digital maturity level, exceeding the 75% target.2,196 places were created in new dormitories for disadvantaged students, accommodating 5,195 students in total.The average cost of creating one extra bed for a disadvantaged student is EUR 32,800 while the average cost of keeping one bed is EUR 7,500. Compared to data for USA and Hungary, the funds spent are cost-effective.The expected impact according to the ToC is generally confirmed, though some interventions are still on-going.
There is a lack of monitoring data, inconsistent names of indicators, different sources of data, non-performant indicators, and missing values in the database which are used to assess the effects. Some projects were still on-going, and their effects cannot be fully assessed.
Ensure that the management and control system play an incentive and support role in facilitating the revision of project plans and securing additional financial resources for projects at risk of disruption.Strengthen the capacity (especially on the part of the management and control system) and competence (on the part of the final beneficiaries) to make public procurement procedures and their control more effective and to reduce irregularities. Ensure more consistency between ERDF and ESF financed measures to enhance their effectiveness.
Evaluation of family harmonisation and professional life measures in Greece – Interim report on the results concerning childcare services
Παραδοτέο Δ: «Ενδιάμεσο παραδοτέο της αξιολόγησης των αποτελεσμάτων της δράσης ως προς τις δομές παροχής υπηρεσιών φροντίδας και φύλαξης παιδιών» στο πλαίσιο του έργου με τίτλο: «Αξιολόγηση των δράσεων εναρμόνισης οικογενειακής και επαγγελματικής ζωής στο
OP Human Resources Development, Education and Lifelong Learning 2014-2020; All ROPs 2014-2020; OP Development of Human Resources 2007-2013
Spyrou Merkouri 38, Athens 116 34, tel. 2107298192, www.cmtprooptiki.gr
This is an interim report of a wider evaluation, “Evaluation of family harmonisation and professional life measures in the Human Resources Development, Education and Lifelong Learning OP (HRDELL OP) and the regional OPs" in Greece, focusing on the results as regards care and assistance for children.
To assess the results of the harmonisation of work and family life with regard to care and assistance services for children.
Data analysis;individual interviews;online questionnaire.
In total 1,356 providers of childcare participated in the three periods (2014-15; 2015-16; 2016-17) of the OP.Over half of them were concentrated in Central Macedonia and Attica.The vast majority (80%) of places are in nursery schools and children day care centres.93% of respondents reported that the regulation in 2016-2017 (Voucher system) were better than previous ones.89% of respondents were satisfied with the changes in funding procedures.91% of providers reported that the programme helped to maintain jobs and 77% that unemployed women were helped to improve their personal skills and know-how.
There is a need to:simplify the application procedures and extend the time allowed for applications;improve the matching between vouchers and places available;provide active employment measures.
Ex post evaluation of the Umbria ERDF OP, 2007-2013
Lattanzio monitoring & evaluation – Iris – Isri
This is the ex post evaluation of the Umbria ERDF OP 2007-2013 which examines its achievements.
To assess the achievements of the OP in terms of projects financed and the results obtained in respect of innovation, the environment, energy efficiency and governance and the impact of R&D investments.
CAWI surveys of beneficiaries.CATI surveys of implementing bodies.Interviews with MA.Monitoring data.
Support under PA1 (Innovation and knowledge economy) had significant effects on employment and innovation but little impact on turnover.Beneficiaries would have carried out innovation projects without support but later.Performance was good for PA3 (Energy efficiency) output indicators, but poor for result indicators.Support had little effect on energy costs and production of energy from renewables.Complex urban programmes successfully combined public and private financing, but there were coordination problems.The governance system was fragmented and there was a lack of coordination with local authorities involved in the OP.
A counterfactual analysis was not possible because of difficulties in identifying a control group, because of the impossibility of excluding companies that benefited from other subsidies. The response rate to the CAWI survey (25 companies out of 223) was very low.
Design calls for proposal which increases the involvement of small companies to reduce deadweight.Improve programming of private investment for urban regeneration.Create specific structures for the different managing phases of the OP together with a coordination unit.
Evaluation of employment interventions under the Human Capital OP, 2014-2020, in Romania
Implementarea Planului de Evaluare a Programului Operațional Capital Uman 2014-2020. Evaluarea intervențiilor în domeniul ocuparii.
Laura Trofin, Speranța Pîrciog, Mădălina Popescu, Irina Lonean, Vasile Strat, Ana Bulai, Michaela Stoichici, Mihaela Iorgulescu-Aioanei, Daniela Pănica, Adelina Alexe, Dan Duțu, Eva Militaru, Ana-Maria Zamfir, Cristina Mocanu, Monica Mihaela Maer Matei, Maria Denisa Vasilescu, Amalia Cristescu, Cristina Lincaru, Larisa Stănilă, Anamaria Beatrice Aldea, Anamaria Năstasă, Adriana Davidescu
Ernst&Young, www.ey.com/en_ro; National Institute for Scientific Research in the Field of Labour and Social Protection, www.incsmps.ro; QURES Quality Research and Support, www.qures.ro
This evaluation examines the employment interventions under the OP up to September 2020, covering employment of NEETs, support to enterprises in urban areas, adaptation of the PES structure to labour market needs, adaptation of companies and improvement of knowledge and skills of employees in sectors with competitive potential and in smart specialisation areas.
To assess the effectiveness and impact of employment measures, unintended and spill-over effects, the sustainability of effects, mechanisms that facilitated or prevented the effects, the extent to which the objectives could have been better achieved, economic benefits or opportunity costs, as well as the contribution to GDP and employment.
Theory-based assessment.Counterfactual impact analysis.Documentation and literature review.Analysis of statistical and monitoring data.Analysis of National Agency for Employment administrative data.Survey of target and control groups.Interviews.Focus group.Expert panels.Case studies.Spatial analysis (GIS).Macro modelling (input-output model).
IP 8.ii:The OP has increased the enrolment of young NEETs in the PES by 35%.It has provided training to 2% of NEETs registered in the PES, and increased participation of young people in education and training by only 1%.There is no visible effect on the employment of young NEETs.There seems to have been a positive impact on young Roma, in terms of both skills acquired and employment.IP 8.i:The OP increased the chances of long-term unemployed being employed 6 months after the intervention by 12% and Roma by 16%.The interventions increased the chances of the unemployed and inactive completing vocational training by 10% and by 17% for unemployed with low educational attainment.The income of the unemployed and inactive increased by 25%. IP 8.iii:48% of beneficiaries would not have started their business without support.Companies supported have a 10% higher chance of survival than others.The interventions increased employment in firms supported, especially those in services.The interventions increased turnover and net profits of start-ups supported.Most respondents consider that their activity has been affected by the COVID-19 pandemic.IP 8.vii:The OP made a limited contribution to adapting PES structures to labour market needs, since only one project on this is being implemented.IP 8.v:Training courses helped to improve the skills of managers and entrepreneurs and the management of human resources, and to introduce innovative, productive and “green” work organisation models.IP 10.iii:The low effectiveness of training measures is in part affected by the late launch of calls, the relatively long project evaluation, selection and contracting procedures and delays caused by the COVID-19.Macroeconomic effects:The direct contribution of OP investment to GDP is estimated at 0.8% and at 36,110 to employment.
At the time of the evaluation, interventions were at an intermediate or even earlier stage. The counterfactual analysis was hindered by the small number of projects completed and other data limitations. The gross effects of COVID-19 could not be assessed. The input-output analysis was restricted to 2016-2018.
Allow applicants to propose activities for active NEETs registered at the PES.Make the youth mobilisation activities eligible and promote them.Continue existing measures and provide additional ones to improve skills and increase employment.Diversify the training programmes offered, adapting them better to the profiles and interests of target groups.Modernise governance of continuing vocational training.Continue financing interventions developing entrepreneurial skills and introduce new programmes for the development of established businesses.Give more importance to the development of digital skills.Simplify project approval, reporting and monitoring procedures.Revise and simplify the rules on classification of expenditure.Make the implementation procedures more flexible.Regularly carry out strategic analyses of target group needs.
Evaluation of the implementation of the Bourgogne and Franche-Comté OPs, 2014-2020
Evaluation de mise en oeuvre des PO FEDER-FSE Bourgogne & Franche Comté Massif du Jura
ROP Bourgogne 2014-2020; ROP Franche-Comté et Jura 2014-2020
The evaluation assesses the implementation of the Bourgogne and Franche-Comté-Jura Mountains ERDF-ESF OPs 2014-2020.
To analyse the implementation and identify ways of improving the chances of achieving the targets and implementation procedures.
Analysis of financial data and indicators;desk analysis of programming documents;interviews with programme officers;simulations of expenditure scenarios.
Franche-Comté OPThe ‘ICT’ and ‘urban development’ priorities have been the most problematic to implement. At the end of 2016 only one project under the former was underway and for the latter, only 4% of the budget had been spent and only three projects were underway.Under the training priority, almost 50% of the resources have been committed, while the figure was higher for ‘supporting SMEs’ priority, ‘housing and energy’ and tourism.Bourgogne OPImplementation was slow for ‘Smart growth’ and ‘energy' and the ‘ICT’ priority was delayed. For ‘urban development’, only 3% of resources had been spent.The rate of implementation of the ‘biodiversity’ priority was relatively satisfactory with 41 projects being underway.
Financial data were only available up to end-2016.
The number of ERDF calls for projects needs to be increased.Project management procedures need to be revised.The number of programming documents which cause an additional workload for the MA should be reduced.
Evaluation of the Communication Plan of the Umbria ERDF OP, 2014-2020
Valutazione del Piano di Comunicazione del POR FESR 2014-2020
Socio-economic analysis in preparation for the Schleswig-Holstein ERDF Programme, 2021-2027
Sozioökonomische Analyse und SWOT Analyse im Rahmen der Erstellung des OP EFRE Schleswig-Holstein 2021-2027
The evaluation examines the socio-economic situation in Schleswig-Holstein as a basis for preparing the ERDF OP for the 2021-2027 programming period.
To assess developments in the region and to identify strengths and weaknesses to prepare for the future OP.
Official statistics.Documentation analysis.
Schleswig-Holstein has a strong economy but there is further potential to improve its attractiveness as a business location.Venture capital companies and business angels are little developed and the share of equity investment in GDP is below the national average.Schleswig-Holstein is a “strong innovator” according to the EU Scoreboard, but innovation-related cooperation between SMEs is low and so is the share of population with tertiary education.Greenhouse gas emissions per inhabitant continue to decline and are below the national average.The attractiveness of the region for tourists is high, though there are wide internal differences.
Intensify start-up support and transfer activities and push forward digital transformation in the economy and society.Improve (international) growth opportunities for companies and ensure the availability of capital for start-ups.Address increased risks from natural hazards due to climate change.
Evaluation of PA1 “RTDI” of the Lombardia ERDF OP, 2014-2020
Focus tematico Asse I - Rafforzare la ricerca, lo sviluppo tecnologico e l’innovazione
Fondazione Brodolini, www.fondazionebrodolini.it - MBS, www.mbsconsulting.com - IZI, www.izi.it
The evaluation presents the findings of a survey of beneficiaries of two calls under IP 1.2: “Research and Innovation HUB” and “Agreement for RTDI”.
To assess the validity of the design and the EU added value of the measures carried out under two calls under IP 1.2: “Research and Innovation HUB” and “Agreement for RTDI” up to the end of 2020.
Survey of enterprises supported (119 out of 278).
The design of measures is suitable for fostering relationships between enterprises, universities and research centres and for involving SMEs in research activities.The presence of research centres in partnerships encourages SMEs to carry out research activities.The involvement of large enterprises is appreciated by smaller enterprises because they bring skills which enable the commercialisation of project results to be maximised.Deadweight is high, as only 14% of projects would not have been undertaken without the funding provided.
Mid-term evaluation of the My First Job service, 2014-2020
Tegevuse „Minu esimene töökoht“ vahehindamine
OP for Cohesion Policy Funding 2014-2020
Merit Tatar, Maarja Käger, Maarja Vollmer, Kats Kivistik, Anastasia PertšjoNok
Balti Uuringute Instituut Lai 30 51005 Tartu, www.ibs.ee
The mid-term evaluation focuses on the implementation of the "My First Job" initiative and aims at assessing the relevance, performance and efficiency of the scheme. My first job aims at supporting the (re)entry of young people into the labour market through wage subsidies and training benefits. The analysis is mostly based on interviews and surveys focussing on young unemployed, unemployed adults with children under 8, employers and NEETs.
Assess the implementation of the “my first job” scheme and provide recommendations to improve its performance.
Semi-structured interviews, questionnaires, focus group discussions, telephone interviews;monitoring data and national data (Unemployment Insurance Fund and Statistics Estonia);analysis of previous sectorial studies and other countries' experience.
500 young people and 452 employers have used or are using the service, which is significantly less than predicted.66% of young people who used the service were working for their original employer 6 months into the service, after 12 months the number dropped to 47% and after 18 months, to 26%. However, about half of those who dropped out of the service had returned to employment within 18 months.Only 6% of young people who used the service participated in training, mainly because of the lack of knowledge of employers and employees.Conditions for participating in the service should be re-evaluated. For example, other studies reveal that for an increasing number of young university graduates, the requirement of no higher or vocational education might not be relevant.NEETs and unemployed parents with a higher education require help in entering employment and should be included in the service.
The evaluation sets out the following recommendations:Shorten the period of unemployment as a criterion - continuous unemployment should not be the only criterion;take into account NEET status or inactivity;remove the condition of excluding those with higher or vocational education from the service or allow exclusion only in justified cases;focus more attention on raising awareness of the possibilities of training;make 16-year-olds also eligible for the service and take into account the needs and possibilities of parents with small children;foster closer cooperation between institutions implementing the service.
Evaluation of support for self-employment and business creation under the Lombardia ESF OP, 2014-2020
Una analisi valutativa della dul autoimprenditorialità e dei percorsi di avvio di imprese culturali e creative sulla creazione e sullo sviluppo di nuove imprese
IRS, irsmi@irsonline.it - COGEA, consulenza@cogea.it
The evaluation examines two measures supporting business creations: the “DUL entrepreneurship” voucher for training and employment services (2016-2018) and the “Imprese creative e culturali” call for training in cultural and creative sectors (2020-2021). Monitoring data for “DUL entrepreneurship” are up to 05/11/2018, while those for “Imprese creative e culturali” are up to 31/12/2020.
To assess the implementation and the results achieved by the “DUL entrepreneurship” voucher for training and employment services and the “Imprese creative e culturali” call for training in cultural and creative activities and to make recommendations for the next OP.
Desk analysis of relevant documentation.Interviews with regional representatives.Analysis of monitoring data.Survey of beneficiaries of the “DUL entrepreneurship” scheme (14% response rate from 2,649 beneficiaries).Focus group with beneficiaries of the “Imprese creative e culturali” call.
Over 50% of the projects from the “DUL entrepreneurship” call were successfully completed by November 2018, which mostly related to self-employment rather than business creation.Over 50% of respondents considered the services offered under the “DUL entrepreneurship” call to be useful.The survival rate of newly created firms is high and turnover grew for the majority of respondents. The employment rate of participants is also high 2-3 years after.Participants in the “Imprese creative e culturali” call were positive about it despite the difficulties caused by COVID-19 and the lack of guidelines.Six projects were funded, 153 beneficiaries involved and 33 new enterprises created.
Some categories of beneficiary are underrepresented in the survey. For the “Imprese culturali e creative” call, the information available did not enable the outcome to be assessed in terms of the survival rate of companies, employment and turnover after 2-3 years.
Strengthen and develop the “DUL entrepreneurship” scheme.Improve the “Imprese creative e culturali” call in terms of continuity, flexibility, and integration with other regional policies and provide progressive support.
Evaluation of PA7 ”Diversifying local economies by promoting tourism” of the Regional OP 2014-2020
Evaluarea intervențiilor POR 2014-2020Dezvoltarea durabilă a turismului prin sprijinirea forței de muncă, sporirea accesibilității și dezvoltarea resurselor naturale și culturale
The evaluation examines the implementation and results of PA 7 of the Regional OP 2014-2020 on diversifying local economies and creating jobs by supporting a range of measures including investment in leisure infrastructure, rehabilitation of natural assets and marketing.
To assess progress in implementing the measures under the PA, the results so far and the validity of the intervention logic.
literature review; survey of RDAs;survey of beneficiaries (Administrative Territorial Units);6 case studies;focus group with RDAs;interviews with selected beneficiaries and stakeholders;panel of experts;administrative programme data.
The objectives are expected to be reached, though no projects have yet been completed.RDAs consider the quality of applications generally satisfactory but there are differences between regions.Public procurement procedures create delays.Projects funded are mainly aimed at improving accessibility and community spaces and at developing recreational facilities.Marketing plans (required as part of applications) are of low quality.There could by a more strategic approach to projects supported by better analysing the tourism market.
Projects were still underway, there was limited involvement of the panel of experts in validating the methodology for case study selection and developing tourism scenarios and low participation in the focus group.
use common tool for analysing and interpreting tourism context, such as an index of tourist attractiveness;focus tourism strategies better on market segments;address tourism needs through a dedicated PA;update the national policy framework for tourism, starting from the lessons learned and the results obtained from the Master Plan for tourism 2007-2026, involving participation of business associations as well as local authorities;revise output and result indicators;ensure the ex-post evaluation takes account of the potential scenarios for tourism development of localities and the interaction with other PAs and financing from other sources;strengthen the help-desk function in the RDAs, especially on public procurement procedures.
Evaluation of effects of PA I (access to broadband) under the Digital Poland OP, 2014-2020
Ewaluacja efektów realizacji I Osi priorytetowej (Dostęp do szerokopasmowego internetu) Programu Operacyjnego Polska Cyfrowa na lata 2014-2020
The evaluation examines the progress of measures to eliminate territorial differences in access to broadband under the OP.
To assess the achievement of the objectives set and the preliminary effects of support under PA1 aimed at eliminating regional differences in access to broadband.
Analysis of monitoring and statistical data.Documental analysis.CAWI/CATI survey of 169 beneficiaries.11 interviews with beneficiaries.4 interviews with implementing bodies.2 expert panels.
OP targets for broadband have not been achieved and a large number of projects are still in progress, though they are all likely to be achieved by 2023.Covid-19 adversely affected the implementation of projects (difficulties in contacting institutions, limited access to material and equipment, etc.).82% of beneficiaries surveyed report high sustainability of the results because of the high demand for broadband and the quality of projects implemented.
Continue advisory support for beneficiaries.Consider the possibility of recouping project costs if the market situation allows.Expand the range of eligible costs.
Mid-term evaluation of the Lubuskie OP, 2014-2020
Ewaluacja mid-term dotycząca postępu rzeczowego i finansowego Regionalnego Programu Operacyjnego – Lubuskie 2020 dla potrzeb przeglądu śródokresowego, w tym realizacji zapisów ram i rezerwy wykonania
Openfield Sp z .o.o., ul. Ozimska 4/7, 45-057 Opole, office@openfield.pl
This is the mid-term evaluation of the progress in implementing the Lubuskie OP, 2014-2020.
To assess the progress in the pursuit of the objectives of the OP and the Europe 2020 Strategy.
Desk research of legislation, EU and national guidelines, strategic and programme documents, monitoring data, public statistics, evaluation reports and other studies.41 in-depth Interviews with MA, IB, experts and monitoring committee members.CAWI with experts to assess project proposals.CATI with beneficiaries.Semi-structured interviews with unsuccessful applicants.Econometric and statistical analysis including forecasts of spending and individual indicators.PESTEL analysis of the context.Benchmarking with other OPs.Panel of experts.TBE.
There are potential threats in achieving some OP objectives.But all planned (mid-term) performance framework objectives were achieved as at the end of 2018.More funding has been allocated to entrepreneurship (EFRR), thermo-modernisation, low emissions, health and education infrastructure, digital development.There is a risk of the planned allocation not being used in waste economy, flood protection, activation of the unemployed and social integration. The greatest risk of not achieving targets or not fully using the allocation concerns IP: 8i, 9i, 6a and 5b because of administrative changes and delays, low interest in waste recycling, changes in the regional socio-economic situation.
There was not enough data to clearly specify the amount of resource which should be re-allocated.
Carefully assess the performance of PAs 5-8 (IPs: 8i, 9i, 6a and 5b) at the end of 2019 and consider reallocating funding and adjusting targets.
Analysis of human resources and working conditions in authorities responsible for the implementation of the European Funds in 2014-2020 in Poland
Badanie zasobów kadrowych i warunków pracy w administracji rządowej odpowiedzialnej za wdrażanie Funduszy Europejskich
Study of the human resources and working conditions in the authorities administering the EU funds.
To determine the conditions and measures for ensuring a stable and professional administration staff in the authorities responsible for the EU Funds.
Survey of 1,613 employees.Interviews with 18 managers of different institutions.4 focus groups with employees.Expert panel.Design Thinking workshop.Case studies.
A salary rise is needed in public authorities, including for those administering the EU Funds area.The authorities should develop a system for determining wage proposals for staff.The effectiveness of the HR policy can be improved by better management of resources including at overall level.
Finance salaries based on analysis of workloads.Build a positive atmosphere in teams, including through integration activities.Conduct career development talks with personnel.Introduce non-wage benefits for staff.
Evaluation of measures for workplace integration under the ESF+ Employment Programme, 2021-2027, in Czech Republic
Evaluace pilotního ověření integračního pracovního místa
Ministry of Labour and Social Affairs, www.mpsv.cz
The evaluation examines the Active Employment Policy (AEP) instrument which aims to help jobseekers with specific needs and barriers to enter the labour market by providing them with an “integration job” (IPM). This job combines the creation of job opportunity and support for integration. This pilot programme ran between January-December 2024.
To assess the functionality and effectiveness of the measure for labour market integration and provide recommendations for possible adjustment, given its potential systemic introduction into the employment services of the Labour Office of the Czech Republic.
Analysis of monitoring data.Analysis of project documentation.Participatory observation during the IPM piloting implementation meetings.45 Semi-structured interviews with 3 representatives of employers, 7 providers of psychosocial support, 3 representatives from the Labour Office of the Czech Republic, 8 pilot site coordinators, 7 employers and 15 employees.Input and output questionnaire filled out by key psychosocial support staff regarding employees.4 focus groups with a total of 30 employees.
As IPM places high demands on employers and provides only a 50% wage contribution, it was necessary to promote awareness-raising of employers about employing disadvantaged people.At the end of the 6-months pilot project, 16% of employees had fully implemented their individual plan, 44% had fulfilled it by about two thirds, 21% by half, 14% by one quarter and 5% had no implementation.Employees who were not motivated to change their situation often misunderstood and rejected the psychosocial support and integrations activities part of IPM. Further, the IPM tool is not very effective for those whose only barrier to entering the labour market is age.As the IPM’s duration of 6 months was insufficient to ensure a positive impact and the stable integration of employees into the labour market, this was extended for the following phase of the project.The cooperation of all relevant actors at local levels is essential for the success of the project. This creates multidisciplinary support for the individual participant.Coordinators are an essential link for transferring information and experience between employers and the Labour Office.Insufficient absorption capacity of social services was problematic in several pilot sites and can be a significant risk when implementing IPM across the board.
Respondents’ approaches to recording responses varied considerably and the responses were often not realistic. Respondents did not have sufficient information about the life situation of participants after a month of cooperation. In some instances, employees themselves filled in the survey.
Ensure IPM jobs last 12 months with the possibility of extension for a further year or two.Consistently explain to candidates the nature of IPM and that psychosocial support is a key part of it.Select employees based on their motivation to change their unfavourable situation.Strengthen the methodological preparation of the territory where IPM is introduced, notably by ensuring that local employment networks are in place.
Second Evaluation of the Youth Employment Initiative 2014-2020
2h Έκθεση Αξιολογησησ Των Παρεμβασεων Τησ Πρωτοβουλιασ Για Την Απασχοληση Των Νεων (ΠΑΝ) Του Ε.Π. ΑΝΑΔ ΕΔΒΜ 2014-2020
Thetidos 10, 11528 Athens, www.premiumc.gr, Tel.: +302107213919
Evaluation of YEI which is being implemented in all Greek regions under Priority Axis 3 (Facilitating Access to Youth Employment to 29 years) of OP Human Resource Development, Education and Lifelong Learning 2014-2020. The interventions supported consist of work experience, education, mentoring, training in the private sector and certification of professional qualifications.
To assess the effectiveness, efficiency and impact of the YEI up to mid-2018, as well as to provide data for the long-term result indicators to feed the AIR due in June 2019.
The evaluation is based on:Desk research (incl. OP, AIR, national legislation, external studies).Quantitative research (questionnaires sent to participants on quality of job offers and progress in continuing education, finding sustainable and decent jobs or moving into apprenticeships or quality traineeships).Qualitative surveys (focus group discussions and interviews with various stakeholders).Data sources: monitoring data, EU Labour Force Survey, National Statistical Office.
The values of the output indicators amount to 40% of the targets.65% of participants were satisfied with the YEI programme.The completion rate is high (96%).45% of participants are high school or university graduates.Only 18% of participants received a job offer.Around 80% of participants accepted the job offer.47% of participants were highly satisfied with the job offer.44% of job offers involved fixed-term contracts and 43% of non-fixed-term offers were part-time.Just over half of the job offers involved a salary of EUR 300-600 a month (and 23% less than EUR 300).Unit costs are higher compared to those originally budgeted. The most efficient intervention concerns work experience. The highest unit cost is for University/ Technology Education Institute graduates.
The quality of YEI actions can be further improved to increase the chances of successful and sustainable placement.
Evaluation of interventions in transport supported by the Large Infrastructure OP, 2014-2020, in Romania
Evaluarea interventiilor POIM in domeniul transportului
Civitta Strategy & Consulting, Archidata, NTSN Connect, Structural Consulting Group
Civitta - www.civitta.com; Archidata – www.archidata.it, NTSN Connect - www.ntsn.ro
The evaluation examines investment supported under PA 1 (improving mobility through the development of the TEN-T and metro networks) and PA 2 (Development of multimodal, quality, sustainable and efficient transport system) of the OP up to the end of 2020.
To assess the implementation and impact of 11 specific objectives under PA1 (improving mobility through the development of the TEN-T and metro networks) and PA 2 (Development of multimodal, quality, sustainable and efficient transport system of the OP up to end-2020.
Desk analysis.Analysis of monitoring and statistical data.Interviews with the MA and the beneficiaries.Workshops.Focus groups of stakeholders.GIS maps.Seven surveys of target groups.Case studies.
Up to end-2020, 121 projects were contracted, of which 19 are major projects. Those not yet implemented will be continued under the 2021-2027 programming period.Projects were prioritised in terms of maturity and absorption capacity rather than needs and accordingly, there is not necessarily a balanced regional coverage.40% of projects have not yet been started or only just, while 25% are more than 90% completed.Factors limiting the implementation of projects are: long approval and authorisation procedures often due to the limited skills of IBs.PA1 comprises 32 road investment projects (11 in the core TEN-T network and 21 in the comprehensive TEN-T network), 4 rail projects and 7 shipping ones.For roads, there has been little impact on network-wide mobility. There were some negative effects from the construction of motorways such as discomfort, blocking access to property, and increases in accidents because of insufficient local consultation and congestion.For rail, investment has helped to reduce travel time on the TEN-T core network and Pan-European Corridor IV, but there has been no major systemic impact at national and regional level. Positive effects for passengers have not yet occurred because obsolete rolling stock does not allow designed speeds to be obtained. Projects do not seem sustainable because of insufficient resources allocated for maintenance.For waterways, projects have been limited, but they have increased goods transported and reduced pollution.For urban mobility, support allowed work on the M4 (Parc Bazilescu — Străulești), M5 (section of the Râul Doamnei - Eroilor (PS Opera), inclusive Valea Ialomiței) and M6 (1 Mai — Otopeni) to be continued.There was a decrease in the share of public transport because of a sharp increase in car use but also because of an insufficient development of inter-modal links.The modernisation and development of infrastructure at Bacău International Airport increased connectivity in the eastern region before the onset of the COVID-19 pandemic.
The level of project implementation does not allow impact to be properly assessed. There is a lack of data on the Danube Delta ITI project to assess effects on biodiversity and the environment and data are missing for the National Transport Model. The response rates to the surveys of public transport passengers and freight transport were low and data on commercial companies in relation to the metro network were not available.
Carry out a feasibility study of the national transport model every four years and propose a technical assistance project for planning future TEN-T investment, the necessary staff, the financing needed and implementation.Prioritise and accelerate major projects so that the TEN-T network can be completed by 2030.Prioritise the completion of railway projects by implementing and purchasing new rolling stock.Develop balanced territorial plans for investment in road infrastructure as a precondition for reducing regional disparities.Ensure sustainability of projects funding the maintenance of infrastructure in the future.Complete the process of tariff integration of public transport in Bucharest, and design complementary park & ride projects for each metro bus.Improve cooperation between Metrorex and local authorities and integrate available databases, e.g. by developing a GIS database for the capital.Ensure that at least one mode of transport (road/rail) per region is prioritised in 2021-2027 and involve more local authorities in the decision-making process.Accelerate progress in developing intermodal transport.Increase traffic safety.
Mid-term evaluation of business and innovation policy measures in 2007-2013
EASi ettevõtlus- ja inNovatsioonipoliitika meetmete vahehindamine
Civitta Estonia AS, Lõõtsa 8, 11415 Tallinn, tel +372 646 448 8, info@civitta.ee
The evaluation assesses the efficiency and effect of schemes implemented by Enterprise Estonia (EAS) to support the development of the business environment, business start-ups, innovation and exports. Measures managed by EAS were financed by the ERDF in 2007-2013 and its activities will continue in 2014-2020.The mid-term evaluation focuses on 2007-2013 and compares the performance of enterprises supported in terms of employment, sales, exports, and value-added with non-supported ones. Based on the findings, recommendations are made for EAS initiatives in 2014-2020.
To answer the following evaluation questions:Have the EAS funds been allocated to enterprises that have used them for activities that they would have been incapable of carrying out under market conditions?How much has the performance of enterprises supported improved?What have been the benefits of EAS support measures in different regions?
Analysis of enterprise economic indicators;online survey and semi-structured interviews;counterfactual analysis using propensity score matching.
The largest share of financial support was allocated to medium-low-tech manufacturing industries.The number of knowledge-intensive service sector enterprises showed the most growth and the largest number of start-ups was in this sector.Overall the sales revenue of micro, small and large enterprises grew faster than that of companies in the comparison group.Receipt of at least one type of support has a positive impact on the number of employees, the return on sales, total labour costs and value added per employee (for the latter, especially in micro and large companies and in Rapla, Lääne-Viru and Võru counties).Measures supporting the entry into new markets also facilitated the development of new products. 11% of beneficiaries investing in product development would have been able to undertake the investment without support from EAS.
There is a need to:Provide support to companies in the application and implementation of processes;reduce the deadweight involved in projects supported and encourage repayable subsidy schemes;strengthen the counselling programme for start-up companies;develop a comprehensive R&D and product development support programme and link support for investment to improving the organisation of production.
Evaluation of the ESF long-term result indicators under the Pomorskie OP, 2014-2020. Third edition
Oszacowanie wartości wskaźników rezultatu długoterminowego Europejskiego Funduszu Społecznego w ramach Regionalnego Programu Operacyjnego Województwa Pomorskiego na lata 2014-2020 – trzecia edycja
Malwina Pietrzyk, Ewelina Baryła-Zapała, Ewelina Dąbrowska-Nowak, Aneta Gierowska
ASM Research Solutions Strategy , asmresearch.com.pl
The evaluation estimates the values of ESF long-term result indicators of the OP.
To measure the value of two long-term result indicators of the OP: “Participants in a better situation on the labour market six months after leaving the Programme” and “Number of pre-school education places that exist two years after obtaining funding from the ESF”.
Analysis of monitoring data and official statistics.CATI/CAWI survey of 2,692 participants in the measures.
The estimated value for the gross long-term result indicator ‘Participants in a better situation on the labour market six months after leaving the Programme’ is 26.9%. The value of the long-term net result indicator is 8.6%.Project participants reported that the support received contributed to positive changes in their situation on the labour market.For the indicator ‘Number of pre-school places existing two years after obtaining funding from the ESF’ is 100%.
Continue the measures under the 2021-2027 OP.
Evaluation of the Islas Canarias ESF OP, 2014-2020
Evaluación de los objetivos/resultados del PO FSE 2014-2020 de Canarias para el informe anual de 2018 a remitir en 2019 y de cumplimiento del marco de rendimiento
cdi Consulting, cdiconsultoria.com, info@cdiconsultoria.com
The evaluation examines the implementation and impact of the Islas Canarias ESF OP 2014-2020, covering all the PAs. A counterfactual analysis focuses on the “integrated programmes for working inclusion” (IP 9.i) and two surveys of participants are on “Career counselling – employment mentor” (IP 8.i) and “Information and business advice” (IP 8.iii).
To assess the logic and strategy of the OP, the management and implementation, the achievements and impact.
Desk analysis of documentation.Analysis of monitoring and statistical data.Interviews with managing and implementing bodies.Telephone questionnaire survey of participants in measures under IPs 8.i (90 respondents out of 7,075) and 8.iii (90 out of 7,908).Counterfactual impact analysis (propensity score matching).
The OP strategy remains valid and is in line with socio-economic needs.Coordination between management bodies works smoothly and internal operational procedures work well, though there are areas for improvement in IT applications.There were delays in approving the programme, the designation of the IB and approval of regulatory rules. Budget constraints have reduced the staff available to an inadequate number.Result indicators (CR04, Participants in employment upon leaving; CR03, Participants gaining a qualification upon leaving) show satisfactory performance, particularly measures under IPs 10.iv, 9.i and 8.iii. The milestones of the performance framework have been met, except for PA2 where the financial target has not been achieved.The CIE of “integrated programmes for working inclusion” (IP 9.i) shows a net placement rate of 23%.Participants in “Career counselling – employment mentor” (IP 8.i) and “Information and business advice” (IP 8.iii) report that the measures have not improved their employability.The OP has contributed to the objectives of the Europe 2020 strategy, especially those for employment and skills, as well as to the European Pillar of Social Rights objectives.
The results of the CIE must be regarded with caution, since it was based on a limited number of variables and excludes some which could have affected the results.
approve the rules for the management and monitoring of the OP early;prepare manuals for simpler and more practical procedures;carry out training on computer applications for management bodies;periodically monitor the performance and result indicators to verify that the 2023 targets can be achieved;continue promoting the development of “integrated programmes for working inclusion” (IP 9.i), since they have positive net impacts;continue to promote training and improvements in qualifications, given the direct relationship between a higher level of training and greater chances of accessing employment (though this is not demonstrated in the report).
Evaluation of energy efficiency measures financed under several OPs, 2014-2020, in Spain
Evaluación del Proceso de Implementación de inversiones en eficiencia energética. FEDER 2014-2020
Multiregional OP for Spain ERDF 2014-20; All ERDF ROPs 2014-2020
Institute for Fiscal Studies (IEF), Ministry of Finance; https://www.ief.es/informacion.ief@ief.hacienda.gob.es
The evaluation examines energy efficiency measures for businesses, the building sector and public buildings, financed under various OPs. Data goes up to June 2023.
To evaluate the progress of the measures on energy efficiency in terms of financial implementation and results achieved, with respect to 2023 targets.
Analysis of monitoring data (result and financial indicators). Analysis of relevant documentation. Analysis of official statistical data. In-depth interviews. Surveys of all Intermediate Bodies responsible for the implementation of the OPs. Design of ToC.
Financial progress is low, as the certified expenditure for energy efficiency measures is 64.5% of the target for 2023, though there are considerable differences between regions. The management, monitoring and control of the measures is problematic, also because of the limited human resources of MAs and IBs, and difficulties relating to their training. Homeowners have problems in accessing the measures, because of financial reasons and the difficult compliance with the requirements. External factors, such as COVID-19, the war in Ukraine and high inflation caused significant delays in the implementation and execution of projects. In terms of result indicators, half of them have a very low level of compliance with respect to the targets set.
Limitations relate to the difficulties of assessing heterogeneous measures and very diverse types of beneficiaries.
Reduce the administrative burden that slows down the implementation of the measures. Intensify awareness-raising campaigns on energy efficiency, and provide specific information and guidance to support participation in the measures. Increase funding and improve the planning and organisation of available resources. Enhance information systems to speed up the management of support from the start of the programmes. Train those responsible for the management of European funds.
Evaluation of projects on sustainable urban development under the Nordrhein-Westfalen ERDF OP, 2014-2020
Verzahnung der EU-Fonds zur Stärkung des Präventionsziels im Rahmen der nachhaltigen Stadt- und Quartiersentwicklung
Neitzel, Michael (InWIS); Elbe, Sebastian (SPRINT), Prognos AG
michael.neitzel@inwis.de, elbe@sprintconsult.de
The evaluation examines the achievement of the objectives of PA4 of the OP on urban development.
To assess the integration of the ERDF with other funds, the results achieved from the “Strong Neighbourhood - Strong People” (SQsM) call on social, economic, and environmental aspects up to June 2020, and whether or not the integration between funds brings added-value.
Theory-based approach (“Contribution analysis”).Analysis of relevant documentation and literature.Analysis of monitoring and financial data.Interviews with those managing EU funds.Survey of 29 applicants.12 case studies (of projects).Focus group with MA and projects supported by the SQsM call.Workshop with several actors.
Under the SQsM call, cooperation between bodies with little or no experience of development projects is considered successful.Integration of ERDF with other funds involves a heavy organisational burden at both a programme and project level, coordination being more difficult at OP level. There is still need to improve the design of integrated interventions funded by different EU Funds.The measures have the potential to improve the quality of life and health conditions in the urban areas supported and to make cities and neighbourhoods more attractive.Case studies (and the survey) show that the measures contributed to achieving the objectives of equal opportunities, non-discrimination and sustainability.
The early stage of implementation of projects limited assessment of the effects.
Specify better the strategy to integrate the various funds.Define common rules on the budgetary procedures.Continue to use the INTERMAG Social City institution as the main body in the selection of integrated urban development strategies, but consider establishing subcommittees on different strategies.Provide applicants with tutorials on application procedures and consider organising forums and networks for the exchange of good practice on it.
Evaluation of environmental and risk prevention measures under the Île-de-France OP, 2014-2020
Evaluation de l’axe inter régional dédié à l’axe Seine du Programme opérationnel régional d’Île-de-France 2014-2020
http://www.teriteo.fr – Raphaël Botti, botti@teriteo.fr; https://www.cereg.com
This is the final evaluation report on the implementation of measures for preserving the ecosystem of the Seine basin and increasing the security of the surrounding areas under the OP, up to 18 January 2022.
To assess the level of implementation, results and impact of the measures implemented up to January 2023 and their efficiency, relevance and coherence.
Documentation and monitoring data analysis.Analysis of statistical data from relevant environmental bodies.Interviews with institutional stakeholders.15 interviews with a sample of beneficiaries.6 case studies.
Most of financial and output indicators have either been be met or are likely to.Measures are generally relevant for the needs of beneficiaries and coherent with national policies.Measures are efficient and, according to the cost-effectiveness analysis carried out on IP 5b measures, the added-value generated is EUR 3 for each 1 EUR spent.The impact of the measures is estimated to be positive, though this needs to be verified in the future.According to beneficiaries, funding allocated to the measures is too small to support the structural costs involved.
Since several projects (especially those under IP 6d) are still ongoing, the impact estimated is only potential. Monitoring data used relate to 2020.
Raise awareness of the importance of the ecological continuity of the Seine basin.Estimate indicator targets against more realistic benchmarks.Reduce the administrative burden for beneficiaries.
Evaluation of the implementation of Priority Axis 1 "Strengthening the economy through the application of research and innovation" under the Competitiveness and Cohesion OP, 2014-2020, in Croatia
Završno izvješće Vrednovanje učinka Operativnog programa Konkurentnost i kohezija 2014.-2020. PO1 Vrednovanje učinka „Jačanje gospodarstva primjenom istraživanja i inovacija“
The evaluation examines the implementation of PA 1 "Strengthening the economy through the application of research and innovation" of the OP.Data used go up to December 2021.
To assess the effects of measures for research and innovation in relation to the targets set, and draw recommendations for improving the implementation of the existing and future programmes.
- Desk analysis of relevant documentation.- Analysis of monitoring and statistical data.- 11 interviews with project implementers.- Case studies on CEKOM projects.- Two focus groups with entrepreneurs.- Survey of beneficiaries (21% response rate).- ToC testing.- Counterfactual analysis of projects under IP 1.2 (DiD).
- 33% of the targets set have already been achieved and it is expected that, by 2023, targets will be fully achieved.- Selection and awarding procedures have been lengthy for most complex projects, especially under IP 1.1.- In terms of efficiency, costs are considered adequate to the results achieved.- According to the counterfactual analysis, development expenditure was significantly higher among beneficiaries than the control group, and kept increasing between 2017 and 2021, especially as a consequence of the measure for the development of new products and services resulting from R & D activities.- The number of employed people in the control group registered a 40% increase in 2021 compared to 2017, while this was 24% in beneficiary SMEs.- Questionnaires and interviews report major changes after the implementation of projects, such as staff turnover in beneficiary SMEs.
Only 69 projects (17%) out of a total of 397 were completed at the time of the evaluation. It is not possible to objectively determine the achievement of large-scale effects at this point in time and establish causal inference.
- Maintain a participatory approach, involving ministries responsible for innovation policies, representatives of the scientific-research private and public sectors, as well as business associations and non-governmental associations.- Ensure the synergy between investments in human capital, infrastructure, development of new technologies, products, services and processes, transfer of knowledge and strengthening of cooperation between the business and scientific sectors.- Align the calls for proposals with the needs and life cycle of innovations.- With respect to award procedures, pay more attention to assessing the maturity, scope, readiness and sustainability of the project, as well as the implementation capacities of beneficiaries.
Evaluation of EU-funded infrastructure in Wales, 2007-2013: Phase Two
Evaluation of EU Funded Infrastructure: Phase Two
OP West Wales and the Valleys ERDF Objective Convergence 2007-2013; OP East Wales ERDF Objective Competitiveness and Employment 2007-2013
Oliver Chapman and Neil Evans, Hatch Regeneris
The evaluation examines EU-funded investment in infrastructure in education, RTDII, tourism, transport and other areas in 2000-2006 and 2007-2013, focusing on the use and maintenance of new and improved infrastructure.
To identify how EU funding has been used to invest in infrastructure, its scale and spatial distribution, how it is being used and maintained and cluster areas which have received significant investment.
Analysis of documentation.Analysis of indicators and statistical data.Interviews with project officer.50 case studies.
Over £800 million of EU funding was invested in infrastructure with a total value of £1.8 billion, widely distributed across Wales, but with particular concentrations in Swansea, Neath Port Talbot, Gwynedd and Pembrokeshire.The majority of new business premises funded are occupied and have attracted firms in high value sectors and businesses are using the enhanced digital infrastructure.ERDF investment in research infrastructure has led to an increase in the quantity and quality of research and the number of collaborations with business.The ERDF has been used to restore or develop some of Wales’s most popular visitor attractions.Investment in the rail network has led to a large increase in rail use on certain lines, as evidenced by the increase in entries and exits at stations which have received investment.Investments in new access roads has helped to unlock substantial new development, contributing to regeneration in Swansea, Port Talbot, Ebbw Vale and other places.There were also failures: several sites which received infrastructure investment have not yet been developed.Investment in incubation facilities through the Technium programme failed to achieve most of its targets.There is mixed and inconclusive evidence about the results of investment in tourism infrastructure.As regards maintenance, the EU-funded infrastructure is still in a good condition and there are a number of examples of where further investment has been made.
There were some data limitations (e.g. no data on the take-up of broadband by different groups).
Evaluation of projects for secondary schools, higher vocational schools and school accommodation facilities under the Johannes Amos Comenius Programme 2021-2027, in Czechia
Vyhodnocení projektů SŠ, VOŠ a DM se zjednodušeným vykazováním
Evaluation and Analysis Department, Ministry of Education, Youth and Sports, Managing Authority
The evaluation examines the implementation of projects in secondary schools, higher vocational schools, youth home facilities, under the OP. Monitoring data go up to September 2025, while questionnaire data were collected in October 2025.
To evaluate the implementation of projects focused on personnel support, innovative education, and cooperation with parents, and which applied simplified reporting.
Analysis of monitoring data.Analysis of two online questionnaires submitted to all beneficiaries.Fieldwork conducted on a sample of 28 institutions (16 secondary schools, 6 higher vocational schools, and 6 dormitories), consisting of: 18 online interviews with project coordinators and staff.10 on-site visits to observe project implementation and collect qualitative data on effectiveness and challenges faced.Use of generative AI tools for qualitative data synthesis from interviews.
Most respondents found the application and project administration process user-friendly, with external assistance further improving the overall satisfaction.Schools and accommodation facilities viewed the projects as highly beneficial for learner support, particularly for students at risk, and the professional development of staff.Initial monitoring issues and difficulties in recruiting specialist roles such as school psychologists and assistants were noted.Dormitories and school accommodation facilities reported specific challenges concerning the tracking of student attendance for activities, especially in larger institutions.Staff training and innovative educational measures, particularly in technology, foreign language teaching, and cooperation with industry experts through workshops and internships, were the most common and successful activities.Institutions expressed significant concerns about the sustainability of the projects after the funding, for example in terms of retention of specialised staff.
Update the report based on a more advanced state of implementation or finalisation of projects.Share information from the current report with relevant departments of the Ministry of Education.
Effects of Measure 7.6 of the Lubuskie OP on the social economy
Ocena efektów wsparcia udzielonego w ramach Działania 7.6 Wsparcie dla OWES i ROPS we wzmacnianiu sektora ekonomii społecznej
The evaluation assesses support for the social economy in the region as part of Measure 7.6 and Investment Priority 9.v of the Lubuskie OP 2014-2020 and some of the effects.
To assess support for the social economy in the Lubuskie Voivodeship, the scope of activities of particular OWES (Social Economy Support Centres) and their effectiveness, usefulness and efficiency.
Desk research;in-depth interviews;three focus groups;CAWI surveys with 300 respondents;case study;mystery customer examination;panel of experts.
Measure 7.6 of the OP created a new social economy support system, which reduced the negative impact of local barriers.The support system is generally functioning as intended and the OWES are fulfilling their tasks.The services provided by OWES are in line with the needs and expectations of social economy bodies in the region.Most of those supported have acquired new competencies or improved existing ones and have found a job.
To enhance business promotion and improve the promotion of support to potential beneficiaries;to further promote grants and loan funds aimed at raising capital.
Ex post evaluation of the Enterprise and Innovation OP in the Czech Republic, 2007-2013
Ex post hodnocení Operačního programu Podnikání a inovace 2007 – 2013
Lubor Hruška, Ondřej Dvouletý, Vladimír Sodomka, Ivana Foldynová, Ivana Blažková, Jiří Svobodník, Petr Proske, Oto Potluka, Andrea Hrušková, Martin Lukeš, David Kubáň
PROCES - www.rozvoj-obce.cz, e-mail: info@rozvoj-obce.cz - Vysoká škola ekonomická v Praze - www.kpo.vse.cz, E-mail: ondrej.dvoulety@vse.cz - Akses, spol. s r.o., www.akses.cz, W-mail: sodomka@akses.cz
This is the final report of the ex-post evaluation of the Enterprise and Innovation OP, 2007-2013, which financed 15 sub-programmes on innovation (3 funded by financial instruments (FIs), 12 by grants).
To assess the results in relation to planned objectives and the impact of the OP and to identify obstacles and unintended effects.
Desk analysis of documents and monitoring data.Identification of programme theories and discussion with expert panel.Survey of supported and non-supported enterprises (412 and 50, respectively).Counterfactual analysis.In-depth interviews (35 with enterprises supported and 15 with non-supported).Focus groups.Case studies.
The objectives of the OP were consistent with those of the Czech Republic's national policy and with the needs of the target groups.All the indicator targets were achieved except for new firm creation.According to the CIE, the interventions did not have a statistically significant impact on the competitiveness of the firms supported 2 years after the end of the programme, though there were positive effects on fixed assets.Firms receiving support through both FIs and grants reported an increase in their market share but not in export performance.Most firms supported reported that the funding concerned enabled them to increase the quality of products or services or to introduce new ones.Positive unintended effects were reported to outweigh negative ones. Administrative complexity, the short time for preparing projects, too length of approval procedures and changes in regulations from one call to another deterred applying for funding.
It was not possible to obtain data for several enterprises supported.
Reduce the administrative burden and increase transparency of and simplify the application process.Reduce the time taken by the project assessment process.Take account of regional disparities in the assessment process.Set up FIs more in line with the objectives of the programme.Avoid overlaps between the different forms of support.Improve the data system for evaluation (by cooperating with the Czech Statistical Office and the Social Security Administration and obtaining the consent of applicants for using data for evaluation and monitoring purposes).
Assessment of the impact of ESF interventions on enterprises and employees in the Dolnośląskie Voivodeship, 2014-2020
Ocena wpływu wsparcia EFS na poprawę sytuacji pracowników i przedsiębiorstw województwa dolnośląskiego
Krzysztof Gutta, Michał Dominik, Agata Płate
EU-CONSULT Sp. z o.o., ul. Toruńska 18C, lokal D, 80-747 Gdańsk
The evaluation examines the impact of ESF interventions on the labour market situation of participants who were employed beforehand and on enterprises, whose employees participated in training or benefited from other services.
To assess the impact of the ESF measures on enterprises and employees in the Dolnośląskie Voivodeship in 2014-2020.
Desk research of programme documents, national and EC guidelines; quantitative survey through CAWI and CATI of project participants and management staff of firms supported by the Demand Financing System;interviews with experts;panel discussion with MA and IBs.
11% of participants were in a better labour market situation 6 months after completing the programme, though lower for those with ISCED 1 and 2 education and those below 24.30% of project participants received an increase in pay as a result of their participation, more for those of 50 and older and those with ISCED 2 education.46% of project participants did not notice any change in their labour market situation, more for those aged 24-50 and with ISCED 5-8 education.For employers, the main problem in using support is the difficulty of navigating the Development Services Database.The large number of employees coming from abroad should be supported by updating and matching their qualifications to the needs of the regional labour market.
Although no limitations are directly indicated, the data suggest that the small number of participants in some IPs was a limitation.
Take action to increase the intensity and effectiveness of projects targeting those in a disadvantaged labour market situation, i.e. those with low education, little professional experience or from disadvantaged groups.Continue support in the next period, targeted at young people with low education.Change the form of support for those with higher education to mainly career counselling.Create a more intuitive and user-friendly Development Services Database.Direct support to those coming from abroad so that they can better integrate into society.
Final evaluation of the “Building Better Opportunities” scheme, financed under the England ESF OP, 2014-2020
Building Better Opportunities Evaluation: Final Evaluation Report
Kath McKenna, Kate Smith, Lilly Monk, George Horton, Gabriela Freitas, Polly Clowes, Lucy Newman
The evaluation draws on data collected between 2016 and 2023, focused on assessing the implementation of the 'Building Better Opportunities Programme (BBO)', funded by the ESF and National Lottery Community Funds. The BBO programme aimed to enhance the employability of the most disadvantaged, using a decentralised approach, with 39 Local Enterprise Partnerships involved in producing project outlines to guide the programme's implementation at local level. The goal of the National evaluation final report is to provide a summary view of the programme's achievements, linking them back to the initial aims.
To assess what the programme achieved in terms of participants engagement and results, the features of the delivery model used perceived to be particularly important, and how organisations engaged in the programme, as well as the benefits from participation in the programme.
Qualitative analysis of previously collected data to address specific evaluation questions and explore different analytical issues.
The programme reached 181,522 people and made a strong contribution to national targets by reaching many of the groups identified as most at risk of exclusion. 34% of participants obtained employment. 84% of grant holders surveyed agreed that it improved their capacity to deliver. Employer engagement was a strong feature of the programme and facilitated job outcomes. The programme helped organisations to secure funding for employment support from other sources (such as Local Enterprise Partnerships or local authorities). Specialist partners played a vital role in supporting participants with disabilities in BBO projects, ensuring tailored support and accessibility, while volunteering emerged as crucial for their employment progression. Specialist partners played a vital role in supporting participants with disabilities in BBO projects, ensuring tailored support and accessibility to jobs.
Reduce the burden of ESF administrative requirements for voluntary, community and social enterprises participating. Reduce uncertainty around future funding to make planning continuation activities easier. Take account of the importance of accessibility in work placements.
Evaluation of the contribution of ESI Funds to regional development in Jihomoravský in 2014-2020 (Tasks 5 and 6)
Výsledková evaluace přínosů Evropských fondů na regionální úrovni: Úkol 5 a 6 - Jihomoravský kraj
All OPs 2014-2020 except OP Prague - Growth Pole 2014-2020
This interim report examines the contribution of ESI Funds to the regional development in Jihomoravský (south Moravia), up to 31 December 2021.
To assess the contribution of ESI funding to regional development, in terms of results and impact up to end-2021, as well as the added-value of CLLD and ITI instruments.
Analysis of monitoring and official statistical data.Analysis of administrative data.Case study of the Šlapanice District.Interviews with 12 stakeholders.Survey of citizens and municipalities (60).Counterfactual analysis (DID method).
ESI funding primarily supported R&D in Brno, especially the South Moravian Innovation Centre (JIC) and the INTEMAC innovation centre, and led to the employment of 186 new researchers.Cooperation between SMEs and research centres has increased, but the number of projects is too limited to have a strong impact on firm competitiveness.The counterfactual analysis shows a significant correlation between the intensity of support to municipalities and the increase in the number of researchers employed, road safety, and the employment rate, but not for waste management.The counterfactual analysis on measures under TO3, TO5 and TO9 produced results which are either inconclusive or not significant.CLLD projects and LAGs were able to meet the needs of smaller municipalities.ITI projects were aimed at connecting the most peripheral municipalities of the region with Brno.
The impact of the ESI Funds on R&D might be overestimated. The counterfactual analysis on TO7 cannot be considered fully reliable.
Evaluation of REACT-EU measures under the Competitiveness and Internationalisation OP, the Madeira OP, and the Azores OP, 2014-2020, in Portugal
Avaliação de Fundos Europeus: REACT-EU – Assistência à recuperação para a Coesão e os Territórios da Europa
ROPs Madeira, Azores 2014-2020; OP Competitiveness and Internationalisation 2014-2020
Sandra Primitivo, Rui Faustino, Pedro Casimiro, Ana Caetano, Manuel Reis, Luísa Silveira, João Maria Guimarães, João Vieira
EY – Parthenon, https://www.ey.com/en_pt/services/strategy/parthenon
The report is an evaluation of how REACT-EU funding was used in Portugal under one national and two regional programmes. It focuses on the interventions under three priority lines: (1) support for business survival and stabilisation, (2) strengthening social responses (health and education measures), and (3) support for job retention and creation. Data used go up to September 2024.
To assess performance and effects of REACT-EU measures, to assess impacts where feasible, and to generate evidence to inform future policy/programme design.
Use of ToC to review the REACT-EU intervention logic. Analysis of statistical and monitoring data. Analysis of administrative data (Instituto do Emprego e Formação Profissional). Analysis of secondary data (Serviço Nacional de Saúde). 12 interviews with MAs, implementing bodies and other relevant stakeholders. 5 focus groups with main stakeholders (MAs, business associations, trade unions, health and infrastructure administrators, tourisms representatives, bodies promoting social inclusion and training).Business measures: Online survey of business promoters (713 surveyed beneficiaries). Counterfactual analysis using the difference-in-differences (DiD) methodology focusing on three main variables: Volume of Business, Gross Value Added and Employment.Health and education measures: 3 case studies with in-depth analysis of selected projects.Job measures: Contribution analysis. Document and literature review.
The funding proved decisive in ensuring immediate liquidity for firms, especially for micro and small enterprises in the tourism and culture sectors. Beneficiary companies demonstrated significantly higher survival rates compared to those that did not receive incentives. The funds protected turnover and gross value added during the peak of the crisis, while post-pandemic growth aligned with the market average. The resources allowed the National Health Service (SNS) to react swiftly through the bulk procurement of vaccines, tests, and protective equipment. Education measures, such as the "Escola+" plan, limited the damage caused by lockdowns by providing pedagogical and digital support, and helped reduce the risk of early school leaving. Furthermore, support for solidarity organisations and food banks ensured continuous assistance for the most vulnerable groups. Employment programmes, such as ATIVAR.PT, favoured the placement of young people and the long-term unemployed through internships and hiring incentives. A positive rate of conversion from internships into permanent contracts was recorded, highlighting a structural impact on the market. Moreover, professional reskilling courses were found to be consistent with the new digitalisation requirements of the production system. The flexibility of the REACT EU instrument allowed for a much faster flow of capital into the real economy compared to traditional structural funds. The projects accelerated the digital transformation of businesses and public services and served as a bridge between the acute phase of the pandemic and long-term economic recovery.
The evaluation is carried out shortly after the implementation deadline, so there are some limitations in data collection of information systems, especially inconsistencies across sources, requiring triangulation with issuing entities.For education measures, an assessment of the net impact is not possible, as the distinction between supported and non-supported students is not feasible.
Evolve business support policies by shifting from aid focused on mere survival and immediate liquidity towards instruments that promote structural competitiveness and the long-term growth of SMEs. Establish permanent flexible funding channels for the national health system, ensuring that investments in digitalisation and telemedicine initiated during the emergency are fully integrated into ordinary services. Maintain the use of hiring incentives linked to training pathways, refine selection criteria to minimise the deadweight effect and concentrate resources where the added value is highest. Adopt the procedural flexibility of the REACT-EU model as a standard for managing future cohesion funds, simplifying bureaucracy and implementing real-time monitoring systems to allow for timely policy adjustments.
Evaluation of changes in development needs in the Czech Republic-Poland Interreg OP, 2014-2020 (2018-2019)
Evaluace plnění priorit a specifických cílů a ověření změn rozvojových priorit programu Interreg V-A Česká republika –Polsko: Ověření změn rozvojových potřeb programu (2018-2019)
KPMG Česká republika, https://home.kpmg/cz, kpmg@kpmg.cz;Naviga4, https://www.naviga4.cz/, naviga4@naviga4.cz
The evaluation examines how the OP investments can address the main problems in the cross-border area (related evaluations – CZE110, CZE112).
To assess whether the OP tackles the main problems in the programme area, what opportunities can be developed through financial intervention and how.
Analysis of statistical and administrative data.Documentation analysis.Interviews with those responsible for the programme strategy and target groups.SWOT analysis.
The OP is aimed at tackling demographic, unemployment, education, transport and tourism issues in the cross-border area.The OP contributes to cross-border cooperation, particularly in respect of crises and emergency situations.At end-January 2019, 112 projects had been implemented.Czech-Polish cooperation is still limited because of lack of knowledge of the business environment and of interest in the business and employment situation on the other side of the border.Cooperation between primary and secondary schools is still limited.
Promote tourism in the programme area by supporting the implementation of destination management and marketing principles.Create conditions for intensive cooperation between schools and businesses, with an emphasis on Industry 4.0, and make technical education more attractive.Promote innovation and applied research.
Evaluation of the impact on employment and the regional labour market of the Świętokrzyskie ESF OP, 2014-2020
Ocena wpływu Regionalnego Programu Województwa Świętokrzyskiego 2014-2020 na zatrudnienie i regionalny rynek pracy
K. Gutta, dr M. Jażdżewska-Gutta, K. Gutta, T. Gutta, M. Dominik
EU-CONSULT sp. z o.o., ul. Toruńska 18C, lokal D, 80-747 Gdańsk, www.eu-consult.pl
The study examines the interventions to increase employment in the region and formulates proposals for the new programming period, based on current experience.
To assess the effects of the measures undertaken under the OP and to identify measures for the new programming period.
Desk research.Interviews with self-government representatives (12) and self-governments where no projects on childcare for children up to 3 have been implemented (8).Surveys of beneficiaries (110) and final recipients (2679).Evaluation workshop.
IP 8ivMost projects are aimed at helping women into employment and developing their professional careers.Projects are highly concentrated in Kielce and its functional area.Projects have created childcare places in communes where they were missing and allowed parents to return to employment.The return to a permanent job or taking up employment has increased family incomes.Some communes did not apply for support because of lacking the co-financing.IP 8iThe scope of support has been tailored to needs. Grants for setting up businesses were rated as most useful.Projects made it possible for participants to obtain employment, improve their qualifications, open their own businesses.The control on employers offering traineeships was insufficient and salaries, much of which often went on commuting, were too low.IP 8iii, IP 8vSome issues for starting businesses were missing from the training, such as tax and social insurance obligations, accounting, legal and administrative issues, marketing, acquiring and retaining clients, and financing investment.Measures financed through non-competitive procurement procedures have been more effective because of the greater experience of the institutions concerned.A significant number of enterprises supported have ceased operation mainly because of not being able to make sufficient profit.
8ivSupport projects for creating and maintaining childcare places for children up to 3.Extend the scope of support by introducing incentives for employers and creating creches at workplaces.8i, 8iii, 8vSupport projects on training to open businesses.Extend the training issues to tax and social insurance obligations, accounting, legal and administrative issues and financing investment.Increase the flexibility of support in terms of the duration of traineeships, etc.Assess the needs of businesses in the region as regards the COVID-19 pandemic.Introduce additional incentives for beneficiaries willing to implement measures in line with smart specialisation in the region.
Evaluation of the impact of projects aimed at the protection of sea coasts affected by erosion under the Infrastructure and Environment OP, 2007-2013
Efekty projektów zabezpieczania brzegów morskich zagrożonych erozją – perspektywa Programu Infrastruktura i Środowisko 2007-2013
OP Infrastructure and Environment 2007-2013
BLUEHILL SP Z O O Ludwika Nabielaka 6/5 00-743 Warszawa - Quality Watch Sp. z o. o., Ul. Stępińska 22/30, 00-739 Warszawa
The evaluation assesses the impact of projects aimed at the protection of sea coasts affected by erosion under the OP Infrastructure and Environment (OP I&E) 2007-2013.
The study is aimed at assessing the impact of projects funded under measure 2.1 of the OP I&E 2007-2013. It also includes an ex ante evaluation of this in the OP I&E 2014-2020 which is aimed at protecting the seashore.
The study used a number of qualitative and quantitative methods:desk research;case studies of each of the 8 projects implemented under Measure 2.1;CATI survey of indirect beneficiaries (i.e. enterprises, local authorities, non-governmental organisations) and correlation analysis of its results;individual interviews with institutions involved in the implementation of the OP;DELPHI analysis by a panel of experts.
38.6 km of seashore have been protected, 77% of the target value. The main reason for not achieving the target was due to problems encountered during project selection. The beneficiaries were also not fully prepared for implementing projects, in particular they had no experience in managing EU funded projects;the structures constructed are sustainable and protect the shore from erosion. The vast majority of them are fully functional and undamaged except for a few cases of faults in Kopań Lake and the Oksywie area;main outcomes: stabilisation of the shoreline, protection of infrastructure on the shore, reductions in the risk of floods and increased tourism.
The number of replies to the CATI survey from local authorities were less than originally planned but the evaluators considered that this did not affect the results.
The following recommendations were made for the future implementation of similar projects in the 2014-2020 period:determine a hierarchy of projects to be implemented;organise meetings and training with beneficiaries to improve skills on project implementation;involve experts in assessment of the proposal;improve the monitoring system, defining clear indicators, increasing the frequency of monitoring and using the data for regular assessment of implementation;include the “impact on the landscape” among the selection criteria.
Evaluation of the implementation system of the Kujawsko-Pomorskie OP, 2014-2020
Ewaluacja podsumowująca system realizacji RPO WK-P na lata 2014-2020
Pracownia Badań i Doradztwa “Re-Source” Korczyński Sarapata sp.j., ul. Spławie 53; 61-312 Poznań, biuro@re-source.pl
The evaluation examines the implementation system of the Kujawsko-Pomorskie OP so as to identify improvements in the current programming period and ways in which it could be better designed in the next programming period.
To assess the effectiveness and efficiency of the OP’s implementation system and of specific measures.
Desk research of strategic, programming and implementation documents, legislation, data on calls for proposals, guidelines, evaluation and expert reports and other studies.53 interviews with different implementing bodies.CAWI survey of beneficiaries and applicants.Benchmarking.Workshop.
The implementation system is generally considered to be effective and efficient.The MA, however, is relatively fragmented which makes communication and decision-making longer.The tasks entrusted to the IB in the next period should be reconsidered.The largest staff shortages are in the implementation units of the MA.Digitalising the application and project assessment process should be a priority in the next programming period.The system complies with the horizontal principles and security of personal data.
Assess the need for specific changes in the institutional structure of the implementation system so as to decide the shape of this in the next period.Limit the negative effects of introducing changes in the institutional structure.Reduce the fragmentation of the implementation system for the next period.Involve key units of the Marshall's Office more in implementation.Streamline the financing of cooperation between categories of stakeholder in the implementation system in the future.Identify the needs of personnel in the MA and IB, ensure the availability of equipment and provide training.Increase the digitalisation of internal and external procedures in the MA and IB.Improve information and promotion activities, recruitment, payments, monitoring, control, and evaluation.Improve financial instruments, territorial measures and the functioning of the IB.
Evaluation of Action 4 'Qualification of employees' of the Bayern ESF OP 2014-2020 (1st interim report)
Evaluation der Förderaktion 4: Qualifizierung von Erwebstätigen - 1. Zwischenbericht
Institut für Sozialforschung und Gesellschaftsppolitik GmbH - Weinsbergstraße 190, 50825 Köln, www.isg-institut.de
The evaluation examines the provision of training for those employed under the Bayern OP to strengthen the adaptability and that of companies to future changes. The report is part of a series of interim evaluation on measures financed by the OP, summarised in DEE53.
To assess the implementation of the training measure and its first effects.
Analysis of administrative and monitoring data and relevant documentation.4 case studies of project managers and participants.Online survey of participants (325 covered out of 759 contacted, 43% response rate).Interview with a person involved in organising and implementing the funding.
Implementation of the measure is slow.Results exceed the target but output lags behind.Satisfaction with the training provided is high among both employees and employers.Up to 2017, 87% of participants had obtained a qualification, exceeding the target.Participants include only small numbers of elderly and low-skilled, possibly because of inadequate content or competing funding.SMEs could be better represented as they are the main target group.
Provide more suitable training for elderly and low-skilled employees.Take action to increase the participation of SMEs.
Evaluation of pilot projects to alleviate economic and labour market difficulties of SMEs due to COVID-19, in Hungary
A KKV-k COVID-járvány miatti gazdasági és munkaerőpiaci nehézségeinek enyhítését szolgáló pilot projektek értékelése
OP Economic Development and Innovation 2014-2020; OP Competitive Central-Hungary 2014-2020
The evaluation examines two calls under the Economic Development and Innovation OP and the Competitive Central Hungary OP to support the adaptation of SMEs to challenges due to Covid-19. Data are updated to 2019 and 2020.
To examine the characteristics of applicants and beneficiaries as regards two calls to support SMEs in adapting to the challenges caused by the Covid pandemic and the relevance of the calls and to make recommendations for future similar calls.
Analysis of monitoring data from the Prime Minister’s Office and companies database (cross-table and descriptive statistics analysis).Online survey of beneficiaries (508 responses, 12% response rate).Interviews with Ministries involved in launching the calls.
5,427 project proposals were received and 80% were agreed, mainly with companies from the central region.From a simple comparison based on data from the Development Policy Database and Information System it seems that applicant companies were able to generate significantly higher turnover and added value compared to the non-applicants.The selection criteria clearly favoured stronger medium-sized enterprises. Construction enterprises received a larger share of support than their share of national value-added.For the 50% of respondents, the call only partly met their needs, partly because the amount of support available was not sufficient. Overall beneficiaries are satisfied with the conditional reimbursable grants.Most respondents reported that the introduction of the market survey increased the administrative burden on applying and considered that it would be more effective if it avoided pricing abuses.Overall, the institutional system was able to manage smoothly the very large number of project proposals received.
Support target groups (including micro-enterprises) through specific calls or by setting a limiting framework.Reduce the amount of support available (below 50 %) and increase the performance requirements for the reversal of aid to reduce over-demand for funding.Make the performance measurement criteria more challenging for beneficiaries and take more account of ex-ante data analysis.Apply higher support rates to priority target groups.Introduce limitations for construction companies under calls for general technological development.
Evaluation of the 'Master & back' measure financed by the Sardegna ESF OP, 2007-2013
Secondo rapporto di valutazione del programma Master & back
The evaluation assesses the Programme "Master & Back", funded by the Regional ESF OP Sardegna for the programming period 2007-2013.
The evaluation assesses the implementation and results achieved by the "Master and back" Programme, funded by the Regional ESF OP for Sardegna for 2007-2013 programming period and aimed at, first, encouraging access to post-graduate education (PhD and Master) at national and international universities (Advanced Training action); and, secondly, supporting the return of young people to Sardegna and helping them integrate into the labour market (Comeback action).
The main methods used are:desk research;CAWI (Computer-assisted web interview) survey with beneficiaries;CATI (Computer-assisted telephone interview) survey with organisations involved in the programme;analysis of monitoring, statistical and survey data.The data used come mainly from the monitoring system, the regional Agency for Labour, Almalaurea, ISTAT and EUROSTAT and from surveys.
A year after the end of the programme, 61% of participants were employed, 29% unemployed and the rest inactive;88% of participants who found employment found jobs in services;there were indirect effects on those supported in terms of opening of their cultural horizons and strengthening of their sense of responsibility;a year after the end of the programme 63% of those completing the Comeback action were living outside Sardegna because of a lack of job opportunities for them in the region;there was a high level of satisfaction of participants in the Advanced Training action (87.5%), though 52% of participants reported that they would have undertaken the training without regional support;84% of participants in the Comeback action considered the action relevant given their professional background.
The results of the CATI survey were not reported as those responding were too few (43% of the total questionnaires sent).
support the integrated use of ESF and ERDF funds in order to combine interventions aimed at increasing human capital with those aimed at supporting SMEs and innovation;give more freedom to participants to choose their education path;consider the possibility of using financial instruments, such as microcredit and venture capital, to support entrepreneurship.
Evaluation of the interventions financed under TOs 1, 2 and 3 by the Auvergne and Rhône-Alpes ERDF OPs, 2014-2020
Évaluation de l’impact des fonds européens – Lot 3 : Recherche, innovation et Développement économique, PO Auvergne et PO Rhône-Alpes
ROP Auvergne 2014-2020; ROP Rhône Alpes 2014-2020
Chloé Bureau, Yann Cadiou, Aurélien Fichet de Clairfontaine, Guillaume Dion, PatrickEparvier, Karine Lanoix, Anne-Gaëlle Muths, Noémie Peycelon, Léonor Rivoire, FrancieSadeski, Noémie Peycelon, Léonor Rivoire, Elisabeth Zaparucha
Technopolis - https://www.technopolis-group.com
The evaluation examines the effects of the measures funded under the TO 1 (Research and innovation), 2 (Digital) and 3 (Business competitiveness). The evaluation is based on data up to September 2019.
To assess the effects of the measures, the success factors and the possibility for replicating them and to make recommendations for 2021-2027.
Desk analysis of relevant documentsAnalysis of monitoring data and data on patentsNetwork analysisCounterfactual analysisInterviews and focus groupsSurvey of project beneficiaries under TO2 (response rate 15%)Case studies.
ERDF projects under TO1 (R&I) have provided an opportunity for young researchers to advance their careers, for international research collaboration by the University of Clermont Ferrand, and for participation in international trade fairs.Although some TO1 projects have covered issues present in other PAs (e.g. healthcare), synergies have not been strategically planned.Without ERDF investment under TO2, rural areas would not have been covered, though the coverage of population is limited.ERDF strategies in both regions are in line with strategic objectives of regional policies, but projects have been carried out separately.In Rhône-Alpes projects to improve digitalisation in firms have worked satisfactorily and are in line with company needs.TO3 projects in Auvergne in tourism helped to increase the number of visitors and led to companies financed having a higher growth of turnover, value-added and employment than non-financed ones; which was not the case in Rhône-Alpes.In Rhône-Alpes, projects on social innovation led to the introduction of new management practices, improved support for work force skills and led to an increase in partnerships between companies.
The majority of projects were still on-going at the time of evaluation. Moreover, in the analysis of TO1 there was a high-rate of non-response to interview requests and the cancellation of a planned focus group. Bibliometric impact analysis could not be carried out. For TO2, the survey covered only part of the measures implemented (those on enterprises). Counterfactual analysis of financial instruments under TO3 could not be undertaken.
Devise a strategy for improving ERDF support to networks between research institutions and enterprises under TO1.Focus on a limited number of strategic TO1 areas.Implement more digital projects under TO2 within a regional strategic framework and improve synergies.Ensure more dissemination of results under TO2.Improve the monitoring of financial instruments under TO3.Continue support of tourism in the next period and not only at the main sites.Maintain collaborative measures for enterprises.Reduce the complexity of implementation rules in respect of social innovation.
Evaluation of support for business investment under the Opolskie OP, 2014-2020
Efekty wsparcia w zakresie przygotowania terenów inwestycyjnych w województwie opolskim
K. Heffner, M. Foltyn-Zarychta, M. Tomecki, K. Koryzma, R. Wieczorek
The evaluation examines how support for the development of investment areas under the OP has contributed to changing the socio-economic situation in the region.
To determine whether the measures increased the attractiveness of the investment areas and the investment potential of the region, as well as to suggest changes in investment strategies.
Desk research.Analysis of monitoring and statistical data.Survey of 51 municipalities.Survey of 44 companies.3 focus interviews with beneficiaries, representatives of institutions and municipalities.5 interviews with the most active municipalities in terms of investment support.Evaluation workshop.
It is not easy to determine which features of an investment area are most important for investors.The quality of most of the investment areas in the region is average.The road system connecting investment areas requires modernisation.Municipalities are not prepared to provide services for investors and they offer small investment incentives.Businesses are more willing to invest if the local workforce has the desired qualifications.
Go beyond standard schemes of investment attractiveness rankings and build a knowledge base on investment areas, based on the results of qualitative research and information from sources other than public statistics.Put greater emphasis on promoting investment areas.Improve the quality of local roads connecting investment areas, as well as links to the TEN-T network.Provide training for officials on investor service, use of investment incentives and promotion of investment areas.Provide training to improve the qualifications of the local work force, in line with the needs of potential investors.
Evaluation of the effectiveness, efficiency and implementation of the Sterea Ellada OP, 2014-2020 - 1st deliverable
Σύμβουλος Αξιολόγησης αποτελεσματικότητας, αποδοτικότητας και εφαρμογής του Ε.Π. Στερεά Ελλάδα 2014-2020 (έτη αναφοράς 2014-2016) - 1ο παραδοτέο
The evaluation examines the implementation of the OP up to 23/10/2017.
To assess the effectiveness and efficiency of the OP, the achievement of outputs and results and the validity of the intervention logic up to October 2017 and to make recommendations.
- Desk research.- Analysis of monitoring data.
The OP is relatively effective in terms of specialisation and calls for proposal but not as regards projects selected, contracts and expenditure.The degree of achievement of output indicators is very low.PAs 1, 2 and 8 have not yet been activated.The very limited number of projects completed makes it difficult to draw conclusions on the overall efficiency of the OP.
Accelerate the implementation of the OP.Increase the resources for PAs 5, 6 and 9.Eliminate intervention codes 115 and 120.
Evaluation of the low-carbon economy priority under the Zuid-Nederland OP, 2014-2020
Evaluatie prioriteit Koolstofarme Economie Operationeel Programma Zuid
Andreas Ligtvoet, Annemieke van Barneveld-Biesma, Veerle Bastiaanssen, Alexander Buitenhuis, Chiel Scholten, Geert van der Veen
Technopolis group, www.technopolis-group.com
This is a mid-term evaluation of priority 4F (‘Stimulating innovation coupled with the smart roll-out of low-carbon technologies and instruments focusing on the built environment’) under the Zuid-Nederland OP, intended to identify possible areas for improvement.
To assess the intervention logic, relevance and effectiveness of priority 4F under the OP.
focus groups (7-9 people from SMEs, municipalities, provinces and other organisations);in-depth interviews (with MA, accepted and rejected projects, and experts);literature review;case studies of selected projects.
The intervention logic of the 4F priority is sound.Municipalities participate in only 5 of the 20 projects, while SMEs participate in 90%.For the majority of projects, indicators have not yet been achieved.North Brabant received a larger share of funds (because it co-financed some projects itself), while Zeeland received most funds per head.
It is difficult to estimate the actual impact of projects on energy saving, renewable energy and CO2 reduction. The data reported on indicators have no quality check. Energy saving is not monitored.
maintain the focus on the built environment;adjust the OP to the policy in the Climate Agreement;oblige several municipalities to take part in projects;improve communication on the calls, specify the grant conditions more clearly;improve programme monitoring by checking the data quality and include the energy savings indicator;promote the implementation of tested measures.
Analysis of the Implementation System of the Regional OP, 2014-2020 – Final Report
Evaluarea intervențiilor POR 2014 -2020 Analiza Sistemului de implementare POR
Lattanzio Advisory Spa and Lattanzio Monitoring & Evaluation Srl; Via Cimarosa, 4, 20144 Milano – ITALY
The evaluation examines the management and implementation system of the Regional OP.
To assess the capacity of institutions involved in the implementation and management of EU and the extent to which the OP contributes to strengthening the capacity of regional and local bodies to manage projects.
Analysis of relevant documents.Analysis of monitoring data.3 case studies.A national focus group.A panel of experts.19 interviews with technical and management staff.
The national regulations on regional development are clear but some national coordination structures mentioned do not exist and there are no inter-territorial or public private partnerships with a role in regional development.The regulations are clear and they enable responsibilities for regional development policy to be identified.The Local Action Group and the Urban Authority would assume more responsibility for measures if a formal institutional framework was established and adequate financial resources provided.The contribution of sectoral ministries to regional development policy is not sufficiently defined.The resources allocated to the MA are not in line with needs, though for regional or Intermediate bodies, human resources are adequate.Compared to the previous period, there is a significant increase in the tasks carried out by the RDAs.The OP has significantly contributed to the development of the strategic planning capacity of potential beneficiaries and of the capacity of decision-makers to identify and select priority projects.The contribution of the OP to administrative decentralisation processes is limited and indirect.
The legislation is constantly changing which complicated data collection and documentation analysis.
Strengthen the coordination function of government policies.Increase the responsibility of sectoral ministries for regional development policy.Introduce legislative changes to ensure better links between central and local authorities and to strengthen the role of RDAs in the programming, monitoring and evaluation of regional development policy.Reduce the complexity of the OP and maintain the management system in the next programming period.Review the role of legal departments, public procurement and county offices so that they help implement the OP in a more systematic way.
How do EU-15 Member States benefit from Cohesion Policy in 2007-2013 in the Visegrad 4 countries?
Jak państwa UE-15 korzystają z realizacji polityki spójNości w krajach Grupy Wyszehradzkiej?
koordynacja: Andrzej Regulskia, utorzy raportu: Marek Antosiewicz, Piotr Bartkiewicz, Anna Matejczuk, Henryk KaliNowski, Magdalena OśkaAndrzej Regulski, Julian Zawistowski, współpraca:Marek Antosiewicz, Marcin Buliński, Krzysztof Czauderna, Beata Dombi,
IMAPP sp. z o.o.ul. Rejtana 15/2502-516 Warszawaimapp@imapp.plwww.imapp.plInstytut Badań Strukturalnychul. Rejtana 15/2802-516 Warszawa, Polskaibs@ibs.org.plwww.ibs.org.p
The evaluation investigates positive spill-over effects from 2007-2013 cohesion policy interventions in the Visegrad group (VG), which ultimately benefit EU15 countries.
To assess the positive spillover effects for EU15 countries from the implementation of cohesion policy in the VG countries.
The evaluation is based on the following methods:a simulation model of the impact of cohesion policy the VG economies, supplemented by secondary data – in particular input-output tables and data on international trade in goods and services by sector;desk analysis of additional data from Eurostat, OECD and European Commission and data on public procurement;case studies;CAWI survey.
Indirect benefits (estimated at EUR 76.9 billion): Cohesion policy interventions triggered an increase of consumption and investment in goods and services in the VG, resulting in increased EU15 exports to these countries;direct benefits (estimated at EUR 11.7 billion): the benefits of direct involvement of EU15 companies as contractors/suppliers in respect of Cohesion policy interventions;the direct benefits of capital (estimated at EUR 8.0 billion): the benefits of local joint ventures in which the largest share of capital is controlled by EU15 shareholders.Additional sector-based spill-over effects through qualitative analysis and surveys:interventions relating to entrepreneurship and innovation development benefit enterprises with the majority of capital from EU15 countries;interventions relating to transport infrastructure enhance territorial accessibility with benefits for all companies operating in Central Europe and VG-based enterprises with the majority of capital from EU15 countries;interventions relating to higher education improve VG universities and research centres, which in turn contribute to scientific cooperation with centres in the EU15;interventions relating to energy and environmental protection improve the quality of air and water and contribute to the implementation of EU energy policy and security.
Relatively low survey response rate in Poland and Czech Republic on human capital projects.
No explicit policy recommendations are identified.
Evaluation of the National Centre for Electronic Information Resources – CzechELib financed by the Research, Development and Education OP, 2014-2020, in the Czech Republic – Final Report
Evaluace systémového projektu „Národní centrum pro elektronické informační zdroje – CzechELib“ podpořeného z PO 1 OP VVV, Závěrečná zpráva za rok 2021
Ernst & Young, s.r.o., https://www.ey.com/en_cz, Na Florenci 2116/15, 110 00 Praha 1
The evaluation examines the “CzechELib” project, aimed at streamlining research management at national level and creating a motivating environment for research. The analysis goes up to April 2022.
To evaluate the implementation of the project and results up to April 2022 and to provide feedback and recommendations.
Desk analysis of documents.Monitoring data.Survey of institutions (155 contacts, 59% response rate).Survey of university students (36 complete answers).Interviews with representatives of member institutions and other key stakeholders.Interviews with representatives of public authorities in respect of RTDI, publishers and foreign organisations.Three meetings with representatives of the CzechELib team.
The project has met the expectations of the stakeholders and achieved the goals set.Financial gains and time savings are the key benefits of acquiring Electronic Information Resources (EIR) according to member institutions.The project has made a major contribution to the centralization of EIR provisioning.Other significant outcome are an increase in transparency and cost-effectiveness, and financial savings and the creation of an EIR community are the main unexpected effects.In terms of international ties, the project is a success, as the team concerned has become a member of three foreign platforms.The CzechELib project has enabled a greater centralisation and simplification of communication with foreign publishers.The transition to the follow-up National Centre for Information Support for Research, Development, and Innovation (NCIP) project has been smooth.
Carry out an internal evaluation of the subsequent NCIP project.Consider current issues in the communication strategy, especially information on the follow-up NCIP project.Maintain relevant online activities, as member institutions would like to continue activities online or in a hybrid form.Keep the existing team on the follow-up NCIP project.Continue community building, which is a positive, unexpected effect of the project.
Evaluation of the progress of the Spain-Portugal Interreg OP, 2014-2020
Evaluación preliminar de impacto del Programa de Cooperación Interreg V-A España-Portugal POCTEP 2014-2020 con un seguimiento de los indicadores de resultado
Interreg V-A-Spain-Portugal (POETCP) 2014-2020
The evaluation examines progress and the preliminary impact of the OP, as at end-2018.
To assess the achievement of indicator targets and the preliminary impact of measures up to end-2018.
Monitoring data.Official statistics.Interviews with MA.Survey of beneficiaries.
At end-2018, financial progress of the OP was still limited. Progress was facilitated by the use of simplified cost options and a suitable management guide, and hindered by lengthy project approval times.The performance framework targets has been met for all PAs.Most of result indicators show good progress and only a few of them have poor performance.
ERDF output and result indicators are not fully adequate to assess the impact of Interreg measures.
Select result indicators more in line with the objectives of the OP and revise some of the targets for 2023.Increase financial allocation to PA 1 (RTDI) and PA 3 (SMEs competitiveness), as their progress is particularly satisfactory.
Evaluation of the impact on sustainability and quality of employment of support under the Warmińsko-Mazurskie OP, 2014-2020
Ocena wpływu Regionalnego Programu Województwa Warmińsko-Mazurskiego 2014-2020 na trwałość i jakość zatrudnienia w województwie warmińsko mazurskim
The evaluation examines the labour market situation of participants in ESF measures in TO8.
Assessment of the impact of the projects implemented under TO8 on the sustainability and quality of employment.
Analysis of monitoring and statistical data.Review of projects application forms.Interviews with MA and project providers.Survey of projects participants and non-participants.Case studies.Benchmarking.
The OP has made a significant contribution to improving the labour market situation of project participants.The demographic crisis is one of the main barriers for labour market development.Support for the unemployed and inactive is effective.Information activities are not sufficiently developed.
Continue the support for the unemployed and inactive in 2021-2027.Intensify information and promotion activities addressed to target groups.
Possibilities of implementing ESCO instruments under OP Infrastructure and Environment (I&E) in Poland in 2014-2020
Możliwości wdrażania instrumentu ESCO w ramach I priorytetu Programu Infrastruktura i Środowisko 2014-2020
The evaluation assesses the possibility of using ESCO instruments (investment in energy efficiency) in the Infrastructure and Environment OP in 2014-2020.
To identify opportunities to implement projects using ESCO instruments under the Infrastructure and Environment OP.
Desk Research;CAWI;case study;panel of experts.
Potential beneficiaries are not familiar with ESCO instruments and have insufficient knowledge of energy efficiency;there is lack of knowledge about ESCO instruments in terms of off-balance sheet costs;funds available at OP level may be used to implement pilot projects using ESCO instruments, in order to standardise formal and legal arrangements.
develop standard practices to improve accounting;develop information material such as examples of good practice for ESCO projects;standardise regulatory documents.
Study on the Partnership Agreement, 2014-2020, of the Czech Republic. Summary report on the theories of change for all thematic objectives
Výsledková tematická evaluace Dohody o partnerství pro období 2014-2020: SOUHRNNÁ ZPRÁVA ZA TEORIE ZMĚNY PRO VŠECHNY TC (TC1-TC11)
Ernst & Young, s.r.o., Naviga Advisory and Evaluation
Ernst & Young, s.r.o., Na Florenci 2116/15, 110 00 Praha 1 – Nové Město, https://www.ey.com/en_ro
This is part of a more comprehensive study on the Partnership Agreement and focuses on the Theories of Change (ToC) adopted for each Thematic Objective (TO), covering ERDF, ESF, YEI and CF.
To describe all the ToCs defined and the expected causal chains between needs, measures and outputs.
Analysis of policy documents.Interviews with experts on the ToCs elaborated.Analysis of statistical data on the socio-economic context.
TO1: main expected effects of the measures are improvement of cooperation between firms and research centres to support business competitiveness and internationalisation.TO2: main expected effect is the impact of ICT development on the productivity of the economy, an increase of the value-added of SMEs and accordingly GDP growth.TO3: main expected effects are growing SMEs capable of stimulating economic growth, including through their expansion abroad.TO4: expected effects are a reduction in dependency on fuel and raw material imports, a shift towards a balanced energy mix and strengthening of energy infrastructure.TO5: main expected effects are improved flood protection, increased adaptation to climate change in urban areas, and reduced environmental risks.TO6: main expected effects are improved efficiency of waste management, reduced air pollution, increased protection of water resources and more efficient wastewater management.TO7: main expected effect is a more reliable and efficient transport system.TO8: main expected effect is the provision of more targeted active labour market policies, leading eventually to GDP growth.TO9: main expected effects are increased employability of vulnerable people and better access to social housing and healthcare facilities, leading to a larger workforce and GDP growth.TO10: main effects are increased quality of education infrastructure, education pathways developing key competences for the labour market and increased participation in higher education.TO11: main expected effects are increased quality of territorial investment, increased quality of services provided by public authorities and increased transparency of administrative data.
The study does not make an assessment of causal links but only analyses their initial setting at the beginning of 2014-2020.
Evaluation of support for adaptation to climate change, 2014-2020, in Hungary
A klímaváltozás hatásaihoz való alkalmazkodás támogatása érdekében megvalósult fejlesztések és központi intézkedések értékelése
OP Environmental and Energy Efficiency 2014-2020; OP Territorial and settlement development 2014-2020
The evaluation examines the measures for adapting to climate change funded under TO5 of the Environment and Energy Efficiency OP and the Territorial and Settlement OP and the contribution of Hungarian OP PAs supporting infrastructure to climate change adaptation as a horizontal objective.
To assess the adequacy of the institutional framework relating to the OPs, whether the projects funded meet the objectives on climate set out in the OPs and how the OPs contribute to the pursuit of Europe 2020 objectives, the EU adaptation strategy and the National Climate Change Strategy.
Analysis of documentation.Interviews with MA officials.Statistical analysis of monitoring data.Analysis of official statistics from Szechenyi.gov.hu.
Projects contribute to the objectives of Europe 2020, EU adaption strategy and the National Climate Change Strategy 2’, relating rural development, agriculture and disaster prevention and management.Most of the calls are on track to meet the output targets in the OPs.Climate change adaptation calls are considered generally efficient in financial terms as projects are expected to reach output targets at half the cost initially estimated.The authorities’ lack of technical knowledge resulted in delays in the development and implementation of projects.
The vast majority of projects analysed were submitted before the publication of the national Guidelines on Climate Change, which prevents a full assessment of how climate action has been treated as a horizontal requirement.
Strengthen coherence of climate action across OPs, for example by establishing a coordinating authority.Focus more on soft measures for climate change adaptation.Establish a comprehensive climate policy monitoring framework, e.g. identifying a more appropriate set of indicators to measure the outcomes of activities.Provide a clearer and more coherent methodology to help municipalities in planning adaption and mitigation activities.Strengthen integration of horizontal principles in the selection process.
Investment under action 1.3 of the Development of Eastern Poland OP, 2007-2013
Efekty schematu Wsparcie na przygotowanie terenów inwestycyjnych w ramach działania I.3 PO RPW
OP Development of Eastern Poland 2007-2013
Andrzej Jędrzejowski, Jacek Pokorski, Zuzanna Popis, Dariusz Szymańczak, Katarzyna Józefowicz, Arkadiusz Dewódzki
The report assesses the implementation and effects of the Polski Wschodniej Development Operational Programme (RPW OP) for the 2007-2013 period aimed at improving the attractiveness of particular areas for investment. Interventions are focused on investment in infrastructure (roads, water supply, sewerage systems and energy) in industrial parks and innovative services. The report provides evidence on the amount of new investment, employment growth and project implementation.
To assess the level of investment and employment resulting from the implementation of action 1.3 of the RDW OP (Support for the preparation of investment areas) and the main features of the projects supported.
Desk analysis of monitoring data and national and regional statistical data;CAWI Survey of beneficiaries and entrepreneurs/investors.
The total value of projects was 257.5 million and 343.8 hectares of land were targeted;infrastructure was considered by investors the most important factor to attract them to the area;investments carried out up to then had led to the creation of 1,547 FTE jobs;total actual and planned investment amounted to EUR 345 million.
The CAWI survey of entrepreneurs/investors had a low response rate (33 out of 71). A significant proportion of investors had not yet started their businesses at the time of the evaluation and so effects of interventions could not be assessed.
Similar future interventions should include selection criteria which take due account of the needs of potential investors and regional specialisation. The existence of road network development plans should be included in the selection criteria for projects.
Evaluation of the effects of Cohesion funds, under all national and regional OPs, 2014-2020, in Poland
Synteza efektów polityki spójności 2014-2020
Policy & Action Group Uniconsult; IDEA Instytut; Wolański; Fundeko
www.pag-uniconsult.pl; www.ideaorg.eu; www.michalwolanski.pl; www.fundeko.pl
The evaluation aims to synthetise the effects of the implementation of all national and regional OPs in Poland. Data used go up to June 2025.
To measure the programmes’ impact on smart, sustainable and inclusive growth, as well as on GDP, jobs, investment and regional disparities. To update intervention logics and assess target achievements for each thematic objective. To identify successes and gaps and recommend improvements for post-2027 cohesion policy.
Analysis of monitoring and statistical data.Validation of theory of change of the programmes.Meta-analysis of ex-post, intermediate and thematic evaluations of the programmes.Six expert panels to summarise and validate results from the evaluation reports.Forty in-depth interviews with MAs and Intermediate Bodies.Review of national and European studies.Three workshops focusing on identifying challenges and formulating recommendations for future cohesion policy.
Macroeconomic impact: EU funds boosted GDP by +4% and created 340,000–480,000 jobs per year, while also helping the economy tackle crises like COVID-19 and the war in Ukraine.Smart growth (TO1-TO3): cohesion policy improved R&D, digitalisation and SME competitiveness, but innovation and private investment did not meet the targets set.Sustainable development (TO4–TO7): EU funds led to the achievement of significant progress in energy efficiency and climate resilience, but the objectives on transport and the circular economy are not fully met.Inclusive growth (TO8–TO11): employment rates rose and social inclusion improved, but challenges remain in relation to demographic changes and the sustainability of adult education outcomes.Institutional and regional impact: cohesion policy helped reduce regional disparities and strengthen cooperation in cross-border areas. However, systemic issues about institutional capacity and sustainability persist.
Some of the ex-post and thematic evaluations used were completed too early, which made them partially outdated. Changes in measurement methods for certain indicators (e.g., broadband access) have caused issues in evaluating target achievements. The assessment of impact of digital skills projects were based on outdated data due to rapid changes in the sector of digitalisation and innovation.
Make the delivery system more flexible to respond effectively to unforeseen crises, like the COVID-19 pandemic, and geopolitical disruptions.Increase the selectivity of support measures to focus on key sectors such as economic competitiveness and innovation, targeting areas with the highest potential for sustainable growth.Focus on talent development, especially for highly skilled workers whose competences are crucial for the regional economy.Strengthen the institutional capacity of public administrations and stakeholders to ensure the effective management and long-term sustainability of EU funds.Continue the support to disadvantaged regions, particularly Eastern Poland and rural areas.
Evaluation of specific objectives 2.1 and 2.2 of the Enterprise and Innovation for Competitiveness OP in the Czech Republic, 2014-2020
Výsledková evaluace specifických cílů 2.1 a 2.2 OP PIK
OP Enterprise and Innovation for Competitiveness 2014-2020
The evaluation examines two specific objectives of the Enterprise and Innovation for Competitiveness OP, 2014-2020: SO 2.1 to support the competitiveness of starts-ups and development of SMEs and SO 2.2 to support the internationalisation of SMEs.
To assess the achievements of the objectives and the impact in terms of the performance of the firms supported and to identify unintended effects.
Desk analysis of documents.Analysis of monitoring data.Interviews with those responsible for implementing the OP.Survey of beneficiaries.Case studies.Counterfactual analysis.
SO 2.1: beneficiaries reported support up to the end of 2018 improved their competitiveness as a result of innovations, the increased quality and efficiency of production and the introduction of new products.CIE shows positive results in terms of added-value and sales, but not on other aspects of performance.There have been positive effects on employment (3,1 jobs on average being created per firm).SO 2.2: supported enterprises have had significantly higher export growth than the national average.Beneficiaries report that their possibility of expanding their business abroad has increased, but the measures were only one among many factors which contributed to this.The estimated effects of support are statistically insignificant.There seems to be no positive effect on employment.Unexpected effects are reported as regards employee development, reduction of adverse environmental effects, the need to reorganise production and follow-up investment, which needs to be financed from the firms’ own resources.Both SOs are relevant in relation to needs and the national economic problems identified.For most projects a relatively high level of innovation was achieved, but for almost a third of projects, innovation is rated as low.
The counterfactual analysis faced a significant number of constraints: only a small share of projects were completed at the time of the analysis and administrative data were not available for many cases.
Maintain and further develop measures to support the internationalisation of SMEs.Simplify project selection criteria and reduce the administrative burden.Focus support for SMEs on digital transformation in 2021-2027.Focus support for start-ups and SMEs development in 2021-2027 on automation, robotics and IT technologies.Consider linking the monitoring system to external databases.
Impact evaluation of projects on company creation supported by the Ile-de-France ERDF-ESF OP, 2014-2020
Evaluation d’impacts des projets ayant un cofinancement FEDER / FSE du PO d’île-de-France 2014-2020 - Lot 2 - Employabilité - Axe 3 OS 4 – Création reprise d’entreprises
Marine Mounier, Béatrice Minet, Elise Crovella, Carole Beaugendre
Itinere Conseil, https://www.itinere-conseil.com, contact@itinere-conseil.com
This is the final evaluation report on measures implemented under PA3 of the OP. Measures analysed cover IP 8.iii. Monitoring data go up to January 2021.
To assess the effectiveness and efficiency of the IP 8.iii measures under PA3 and their impact on employment and business creation.
Documentation analysis.20 interviews with beneficiaries, project managers and IBs.4 surveys 6 months after interventions ended of 6,521 participants from 2014 to 2017.Analysis of monitoring data.
The measures are generally coherent with the main regional policies and meet the needs of the beneficiaries, especially with respect to the promotion of female entrepreneurship. However, support was uneven across the region, with Paris and Seine-Saint-Denis being the most supported.The administrative burden and changes in the rules for the management, implementation and monitoring of projects are the main constraints for beneficiaries.Measures financed are not cost-effective, since the average cost per participant are generally higher than planned.3 years after support, three-fourth of the newly created or recovered enterprises still existed.52% of participants were employed when leaving the measures, against 35% when entering. Men and those aged 54+ are most likely to be employed after participation.
Strengthen the support for specific target groups, such as women and micro and young entrepreneurs.Support the creation of networks of entrepreneurs.Make the co-financing rules clearer.
Evaluation of support for internationalisation, organisational change and the development of competences of university personnel under the Knowledge, Education, Development, 2014-2020
Ewaluacja interwencji wspierających umiędzynarodowienie, zmiany organizacyjne i rozwój kompetencji kadr uczelni
ECORYS Polska Sp. z o.o., ul. Solec 38 lok. 105, 00-394 Warszawa, ecorys@ecorys.pl
The evaluation examines the measures in higher education to increase internalisation and improve of the teaching and management skills of personnel.
To assess the effectiveness, usefulness and potential sustainability of measures supporting internationalisation, organisational change and the development of competences of university personnel and to update the underlying assumptions of Measures 3.3 and 3.4 of PA3 of the OP.
Desk analysis.Interviews with implementing bodies, main beneficiaries, higher education institutes.Survey of project teams and participants.Focus group with projects participants.Benchmarking with other countries.Case studies.Workshop.
The internalisation of higher education institutes has increased largely because of the wide range of support.The publishing activity of scientists is the least developing area of internationalisation.Only a few universities have developed internationalisation strategies.Most university personnel supported report improvements in their teaching and management competences.The most frequently reported needs concern the further development of social, digital, coaching, mentoring, tutoring, multilingual competences and the ability to use innovative didactic methods in teaching.The coronavirus pandemic has increased the need for digital competences relating to e-learning.Developing competences should be tailored to the needs of target groups and the range of forms of support should not be limited.The competences acquired by university personnel are not always put into practice.
Continue support for internationalisation because without support, many universities will not be able to develop it.Strengthen the system of support for scientists to promote international research and cooperation.Continue support for improving the teaching and management competences of university personnel.Support online universities.Define recipients of support, activities and forms of support more broadly in programme documents.Continue to support universities to implement the "Constitution for Science" Act.Increase the emphasis on the sustainability of the effects of the support provided.
Final evaluation of all Slovenian OPs over the period 2007-2013
KončNo vredNotenje izvajanja Operativnih programov programskega obdobja 2007-2013 v Sloveniji
dr. Maja Bučar (Evaluation Team Leader), Maja Kranjc, Matevž Premelč, mag. Bojan Radej
The purpose of the evaluation is to provide a comprehensive overview of the achievements of Cohesion Policy in the 2007-2013 programming period in Slovenia.
The evaluation focuses on the results achieved in terms of sustainability, ability to create synergies between programmes and the experience gained in management and implementation. Special attention is given to three selected policy areas: tourist infrastructure, social entrepreneurship and human resource development.
Desk review.Survey of beneficiaries and Managing Authorities.Interview with selected beneficiaries.Focus group.Analysis of monitoring data.
Administrative burdens are identified as the main cause of bottlenecks that hindered the implementation of policy measures and led to delays, failure to achieve targets and the slow absorption of funds.Administrative difficulties were mainly linked to inadequate organisation, unclear implementation manuals and lack of competent staff.In tourism, new products and services were created and this is reflected in the growth in visitor numbers and revenue.In social entrepreneurship, 14 new activities were supported which continued in operation 1 to 4 years after co-financing came to an end and almost a third of new jobs generated were still in existence a year after the end of projects.Human resource development has been intensified and this has raised awareness of the importance of continuing education and training of employees.The cooperation of enterprises also continued after projects came to an end.
Create a training programme system in OP management based on lessons learned from previous experiences.Ensure that more visibility is given to Slovenian tourism which is growing.Disseminate the results of projects supporting social entrepreneurship.Raise awareness among potential consumers of the products and activities of social enterprises.
Evaluation of Action A 1.1 – Integration of long term unemployed under the Baden-Württemberg OP, 2014-2020
Evaluationsbericht zum spezifischen Ziel A1.1
ISG - Institut für Sozialforschung und Gesellschaftspolitik, Katrin Hunger und Kai Sattler
This is an evaluation of 4 projects undertaken for the ’Integration of long-term unemployed’ objective under the Baden-Württemberg ESF OP for 2014-2020S aimed at providing sustainable jobs for long-term unemployed.
To evaluate the measures implemented under the Baden-Württemberg ESF OP to find sustainable employed for the long-term unemployed.
Interviews with project coordinators and participants.Case studies.Online survey.
Project participants are relatively satisfied with the support received.The majority of participants had difficulties in establishing successful contact with potential employers. Counselling and support had a limited effect on this.Except for one project, cooperation between project coordinators and job-centres proved to be difficult, partly due to priority being given to other employment initiatives managed at federal level. In addition, job-centres tend to support their own clients rather than those from other projects.
Only a limited number of enterprises agreed to participate in the interviews.
Contacts with potential employers can be managed by a company or a consultant but they need necessary skills.The issue of competing federal and regional employment initiatives needs to be raised at the Federal level so as to identify possible solutions.
Evaluation of social innovation measures in the Knowledge, Education, Growth OP, 2014-2020, in Poland. Third thematic report
Ewaluacja innowacji społecznych w ramach Programu Wiedza Edukacja Rozwój - III raport tematyczny
EGO-Evaluation for Government Organizations s.c., www.evaluation.pl; Ośrodek Ewaluacji Sp. z o.o., www.ewaluacja.eu
This is the third report on the ‘Evaluation of social innovation measures in the Knowledge, Education, Development OP'. It identifies the implementation barriers for social innovators and sets out recommendations for the present and future programming periods to overcome them.
To identify the reasons for delays arising for innovative projects under Measure 4.1 and to indicate possible ways of preventing them from happening.
Desk research.6 interviews.Focus group.CAWI with 34 project coordinators.Bulletin Board Discussion.Workshop with managing institution.
The most frequently reported problem is the need to apply the principle of competition when selecting experts.Another problem is the low limit put on indirect costs.The long time between the submission of applications and the conclusion of agreements is a further problem.Replies to calls are deterred by the demanding nature of innovative projects, the difficulty of implementing them and a lack of clear benefits.Grant recipients have a serious problem with financing the administrative costs involved in handling the grant.
The regulations changed during the implementation period which could have reduced the barriers but respondents might not be aware of them.
Allow the possibility of launching innovative projects in the form of pilot programmes.Expand the scope of information activities and information channels in respect of applications.Introduce more flexible rules for implementing innovative projects, increase the limit on indirect costs and withdraw the need to apply the principle of competition when selecting experts.Provide incubators with an updated database of innovation.
Evaluation of community-led local development (CLLD) supported by the Podlaskie OP 2014-2020
Ocena systemu wdrażania i wstępny pomiar rezultatów instrumentu Rozwój Lokalny Kierowany przez Społeczność w województwie podlaskim
Agnieszka Śnieżek, Sebastian Pałka, Cezary Przybył, Aleksandra Kamińska, Paulina Kowalczyk, Michał Marciniak, Adam Grajek, Beata Tomczak, Magdalena Głomska
EVALU Sp. z o.o., ul. Dzika 19/23 lok. 55, 00-172 Warszawa, e-mail: evalu@evalu.pl
The evaluation examines the implementation of the CLLD instrument supported by the Podlaskie OP 2014-2020 and the preliminary effects produced.
To assess the implementation and relevance of CLLD strategies adopted in the Podlaskie region.
Analysis of documentation and literature review;48 in-depth interviews with experts, selected departments of the Marshal's Office, offices and/or Boards of selected Local Action Groups (LAGs), members of LAG Councils and selected beneficiaries;CAWI with beneficiaries;SWOT analysis;workshops.
The CLLD approach was based on specific needs of the region and the use of multi-funding allowed the design of a broad spectrum of possible types of project.The participatory way of preparing local development strategies (LDS) was a key success factor. The structure of the local strategy based on specific indicators on the problems identified ensured the implementation of a results-based LDS and compliance with regulations and procedures.Mechanisms to create LAGs and their composition did not cause any friction with the CLLD implementation. The organisational structure of the Groups is favourably assessed, though there are certain factors limiting the implementation of LDS.The implementation of CLLD projects with participation of the LAGs encouraged the submission of applications from those not yet in receipt of EU funds, the local nature of the project and support of LAG offices providing an element of security for applicants.The selection procedures and criteria are considered to be appropriate but the application process to be moderately difficult.Communication activities are considered to be effective.The division of tasks and responsibilities and cooperation between units implementing CLLD works quite well.
There was a low willingness to participate in interviews by entrepreneurs who were part of the LAG Council.
Maintain the CLLD as a multi-fund formula in the next programming period;align some CLLD regulations with other Cohesion Policy Funds to ensure better coordination;use a uniform IT system for all Funds;simplify the rules for implementing LAG operations.
Evaluation of the Building Better Opportunities programme under the England ESF OP, 2014-2020. Year 2020
Evaluation of the Building Better Opportunities Programme
This is the 2020 evaluation on the implementation of the Building Better Opportunities (BBO) programme. The programme has financed 132 projects aimed at supporting employment among those farthest from the labour market, as well as fostering social interaction between isolated people. Data used go up to June 2019.
To assess the main results achieved and draw policy recommendations for the future.
Analysis of documents.Analysis of monitoring data.Survey of 200 participants and 256 grant holders.Interviews with 14 grant holders.
BBO projects have engaged 89,969 participants (49% disabled) up to June 2019 and 35,292 are those who completed the projects.According to the surveys, the one-to-one support offered by BBO projects, either through a mentor or a coach, was successful in addressing social isolation. 74% of respondents increased their self-confidence and 46% improved their wellbeing.BBO projects helped participants in overcoming isolation also by developing digital skills, which support interaction through virtual means.Young people have been reached mainly through social fairs, to make them aware of the activities available. Support has been mainly provided to tackle anxiety and mental health, especially after Covid-19.Volunteering activities have been an effective means for inactive or far from the labour market participants to gain practical skills. After exiting projects, 14% of participants were employed in permanent jobs.Several projects have been extended until 2022, in order to ensure the continuity of the activities implemented, though less funding was provided and project partnerships had to be limited.Cost-effectiveness analysis found that for each 1£ invested, between 1.40£ and 6.35£ of social value has been created.
Nord-Pas de Calais OP – Evaluation of the Youth Employment Initiative in 2015
Évaluation de l’impact du programme européen « Initiative pour l’emploi des jeunes » en 2015 - Nord-Pas-de-Calais
This evaluation assesses progress in implementing the YEI in Nord-Pas-de-Calais in 2015, complementing the national YEI evaluation 2015.
To assess the implementation of the programme, its progress towards achieving its targets and coherence.
Desk research of OP documentation;analysis of data from the monitoring system and Eurostat, the National Institute of Statistics and Economic Studies (INSEE) and the Ministry of Labour (DARES) on employment policy and youth employment;interviews with 11 representatives of the MA, beneficiary institutions and other key stakeholders.
EUR 90.2 million of YEI and ESF has been allocated in Nord-Pas-de-Calais (EUR 44.6 million from the regional OP and EUR 45.6 million from the regional component of the national OP). This corresponds to around EUR 1,200 for each NEET identified in 2014 (74,000);91% of the resources available from the regional OP have been allocated as compared with 87% from the national OP;by the end of 2015, around 13,000 people had started national OP activities and 1,000 regional OP ones;out of a 54% sample of participants supported by the national OP, 57% were employed or had started training, while 16% had withdrawn from the programme.
There is a need to:finalise the allocation of funds by investing in activities that maximise expected impact and best meet employment challenges;capitalise on the implementation of common calls for proposal to encourage the implementation of coordinated approaches between operators;collect more detailed data through the monitoring system (e.g. the types of activity undertaken by each beneficiary) in light of the counterfactual assessment foreseen in 2018;consider data and application sharing between national and regional OPs.
Evaluation of the effects of ITI implementation under the Malopolskie OP, 2014-2020
Ewaluacja efektów wdrażania instrumentów rozwoju terytorialnego w postaci Zintegrowanej Inwestycji Terytorialnej (ZIT) w ramach RPO Województwa Małopolskiego na lata 2014-2020 - etap I.
EVALU Sp. z o.o. ul. Dzika 19/23 lok. 55 00-172 Warszawa, e-mail: evalu@evalu.pl
The evaluation examines the implementation of the Malopolskie OP in terms of efficiency and effectiveness and indicates its strengths and weaknesses.
To assess the implementation of the ITI in Malopolskie, its rate of progress and effects, project selection, communication activities and cooperation between partners.
Desk research of strategic and programming documents, draft regulations for 2021-2027, evaluation and expert reports and other studies.Analysis of monitoring data and public statistics.Interviews with institutions involved in ITI implementation and with potential beneficiaries.CAWI and CATI with beneficiaries and the ITI association.Case studies.Workshop.
The ITI Strategy targets will largely be achieved by the end of 2023.The effects of ITI can be seen in the transition to a low-carbon economy, increased access to main drainage and selective waste collection.ITI has led to improvements in the quality of healthcare and social services, more pre-school places, and the better adaptation of the VET system to the needs of the labour market.It has also led to a deepening of cooperation in planning and coordinating the development of the functional area.Frequent changes on the list of projects included in the ITI Strategy and lack of interest in calls are problematic.
Replace the list of projects included in the detailed description of PAs and the ITI Strategy with a description of the general directions of support and main project activities.Allocate funding only between IPs.Involve NGOs and companies more as partners at the programming stage and as project partners in the next programming period and broaden the involvement of poviats in ITI implementation.Shift funding for ITI around more flexibly.
Ex-post evaluation of the benefits obtained by EU member states as a result of the implementation of Cohesion policy in Poland
Ewaluacja ex post korzyści uzyskiwanych przez państwa członkowskie UE w wyniku realizacji polityki spójności w Polsce
Jakub Adamski, Zbigniew Mogiła, Wiktor Mrozowski, Dominika Sikora-Kruszka, Maciej Ziomek
EGO S.C., www.evaluation.pl; LB&E Sp. z o.o., www.lbie.eu
The evaluation assesses the macroeconomic and microeconomic benefits that EU countries have obtained as a result of the cohesion policy measures implemented in Poland in 2007-2030.
To assess the impact of the implementation of cohesion policy in Poland on the economies of other EU Member States, particularly the EU-14.
Analysis of monitoring data and official statistics.Macro modelling.CAWI survey of 12,274 beneficiaries.
The total estimated remuneration attributed to non-Polish contractor companies amounts to EUR 21 bn (PLN 89.9 billion), representing approximately 15.2% (10.2% attributed to contractors from EU Member States) of the total projects value in Poland.The total value of additional exports from EU countries other than Poland, stimulated by the implementation of the cohesion policy in Poland, is estimated to be EUR 227 bn in constant 2015 prices over the analysed period 2007-2030.The total impact of the cohesion policy implemented in Poland on the GDP of EU Member States between 2007 and 2030 is estimated to be EUR 118 bn at 2015 prices.
Evaluation of transport measures financed by Cohesion Policy in the 2014-2020 period, in Poland
Ocena wpływu inwestycji z zakresu transportu realizowanych w ramach polityki spójności w perspektywie 2014-2020
OP Development of Eastern Poland 2014-2020; OP Infrastructure and Environment 2014-2020; All ROPs 2014-2020; Interreg V-A OPs 2014-2020
dr hab. Michał Wolański, Dominik Makurat, Paulina Kozłowska, Ernest Czermański, Karolina Orcholska, Beata Paczek, Wiktor Kijania, Marcin Pinkosz, Julia Widłak, Mirosław Czerliński, Bartosz Jarecki, Wojciech Szłapacki, Patryk Pochoda, Karol Trammer, Denis Wawreczko
Wolański sp. z o.o., www.michalwolanski.pl
The evaluation focuses on the assessment of effectiveness and socio-economic impact of transport projects under Cohesion Policy 2014–2020, in Poland. It also evaluates implementation processes and provides recommendations for the future.
To assess the contribution of Cohesion Policy measures to addressing transport bottlenecks, the impact on mobility and socio-economic outcomes and provide recommendations for future transport investments.
Desk research.Analysis of statistical and monitoring data.12 interviews with representatives of MA and IB, beneficiaries and central authorities.Survey of 31,590 inhabitants and 528 beneficiaries.10 case studies.Counterfactual method (SPSM).TBE approach.Expert panel.
5,700 km of roads and 3,850 km of railways were modernised.Travel times improved by 20% for one-third of intercity road routes, but rail performance remains inadequate​​.There is limited evidence of significant GDP growth or socio-economic transformation, because of infrastructure saturation​​.Road accidents decreased by 6%-35% annually from 2015 to 2021, with key contributions from bypasses and expressways​​.
Shift priorities toward sustainable transport and public mobility.Enhance synergy between transport programmes and spatial planning.Invest in rail network capacity and intermodal solutions​​.
Evaluation of measures on continuing education under the Administrative reform OP, 2007-2013 and the Public Sector Reform OP, 2014-2020, in Greece
Έκθεση Αξιολόγησης του εκπαιδευτικού έργου του Ε.Κ.Δ.Δ.Α. (Συνεχιζόμενη Εκπαίδευση)
OP Administrative reform 2007-2013; OP Public Sector Reform 2014-2020
Spyrou Merkouri 38, 116 34 Athens, e-mail: info@cmtprooptiki.gr, https://cmtprooptiki.gr/
This evaluation analyses measures on continuing education and training of public officials in Greece, implemented in 2007-2013 and refinanced under the Public Sector Reform OP in 2014-2020.
To assess the level of implementation of the measures, as well as their effectiveness, efficiency, and relevance.
Analysis of documentation.Analysis of monitoring data.Surveys of stakeholders, trainers and participants.
Training courses are generally relevant for participants when there is a preliminary identification of needs.The use of e-procedures for selecting trainees is considered positive by both participants and trainers, as they are standardised and reduce the administrative burden.Among the problems encountered by the IN.EP (Training institute), there are: shortages of qualified trainers, difficulties in providing classes through e-learning tools and difficult collaboration with some international partners.Trainees are generally very satisfied with the quality of training attended and the results achieved. The average cost of training is EUR 188,26 per participant.
Support trainers in the use of e-learning tools.Mainstream awareness of distance learning programmes in order to attract a larger number of participants and design training classes based on identification of needs.Ensure the quality of ICT infrastructure is maintained.Formulate a Strategic Plan for the development of IN.EP. transnational networks.
Evaluation of the scheme for sustainable employability for regions and sectors under the ESF OP, 2014-2020, in the Netherlands
Evaluatie esf-regeling duurzame inzetbaarheid regio’s en sectoren
Arie Gelderblom, Yvonne Prince, Frank Huurdeman (SEOR), Bert-Jan Buiskool (Ockham|IPS)
SEOR, www.seor.nl, gelderblom@seor.eur.nl; Ockham|IPS, www.ockham-ips.nl
The evaluation examines the scheme for Sustainable Employability for Regions and Sectors, which aims to keep workers motivated, healthy and productive up to retirement age. This is the third phase of the evaluation of the scheme, focusing on its effects and summarising the findings of all three phases.
To assess the extent to which the expected impact of the Sustainable Employability scheme to help workers keep motivated, healthy and productive up to retirement age has been achieved.
Interviews with 42 participating companies.
62% of companies claim that the expected results have been achieved or exceeded.The majority of companies state that projects have helped to increase knowledge and understanding of sustainable employability both in the company and among employees.According to 81% of companies interviewed, after the completion of projects, there are lasting effects, in the form of continuing activities (64%), changes in human resource policies (33%) and follow-up activities (10%).Some companies also noted an increase in labour mobility and reduced sick leave.In more than two-thirds of the participating companies interviewed, COVID-19 has had an impact on the development of sustainable employability policies because of the safety measures put in place.According to respondents, without ESF support, the project and pilot would not have been carried out or would have been on a smaller scale.Cooperation between the different parties involved is considered to be an important factor of success, while the main bottlenecks include the reluctance of employees to engage, the administrative burden and the difficulty of focusing on sustainable employability in the long-term.
Because of the lack of a database with contact details of participating companies, it was not possible to conduct a survey of all participants. Respondents often recognise that it is difficult to distinguish the impact of the project from other factors. The sample of companies interviewed is not fully representative, as companies with a positive perception of the project are more likely to participate. In addition, the companies interviewed were suggested by project leaders. Effects on sick leave rates and labour mobility are difficult to measure within the duration of the project.
Ensure a balanced reach of applicants across sectors and regions through communication activities.Give more importance to the strength of proposed partnerships in the evaluation criteria.Ensure dissemination of project results.Encourage exchange of experience between projects, companies and employees.Consider reaching less-known companies or those less engaged in sustainable employability.Ensure awareness of the importance of sustainable employability among managers.Involve employees in the formulation and further implementation of the project.Ensure that interventions are embedded in existing sustainable employability policies or are complementary to them.
Evaluation of the communication system for the Śląskie Regional OP
Ewaluacja systemu komunikacji funduszy europejskich oraz zrealizowanych działań informacyjnych w ramach RPO WSL 2014-2020
Cezary Przybył, Paweł Huras, Wojciech Pieniążek, Sebastian Pałka, Aleksandra Kamińska, Michał Marciniak, Adam Grajek, Magdalena Głomska, Magda Matysiak, Klaudyna Nowińska
Evaluation of the Communication System and Information Activities undertaken as part of the Śląskieregional OP 2014-2020.
To evaluate the communication system of the OP and its functioning between 2014 and 2017.
Interviews with MA representatives, Intermediate Bodies and other stakeholders.Online survey of stakeholders.Questionnaire to applicants and potential and actual beneficiaries.Group interviews with beneficiaries, projects participants and MA staff.Linguistic analysis of information and promotional material.
Beneficiaries, potential applicants and the general public require more detailed information on OP regulations, calls for proposals and examples of the tangible effects of the OP.The language used on the OP websites is too complicated.The current indicators show that most of the Communication Strategy’s objectives have been achieved.There is sufficient coordination between the MA and the Intermediate Bodies.
There is a need to:Continue profiling potential beneficiaries.Improve indications on how and where to find OP information.Carry out an information campaign targeting entrepreneurs.Further simplify the language used in promotional activities.
Evaluation of OP contribution to the Europe 2020 strategy under the National OP Legality, part 2
Rapporto di Valutazione sul contributo del Programma alla Strategia dell’Unione – Parte 2 Servizio di valutazione del PON Legalità 2014-2020
IRS - https://www.irsonline.it/ - irsroma@irsonline.it; https://www.consorziomipa.it/; https://www.ptsclas.com/contatti/
The report is the second part of the evaluation on the contribution of the 'National Programme Legality 2014-2020' to the Europe 2020 Strategy. Specifically, it discusses the key findings from the evaluation of projects aimed at enhancing the administrative capacity of the Public Administration, which were funded by the ERDF and ESF and implemented through Axes 1 and 5 of the Programme. Additionally, it examines the improvement of security standards of production systems, which were financed by the ERDF implemented through axis 2 of the Programme.
To draw from the previous report’s findings on Axis 1 interventions of the 'National Programme Legality 2014-2020,' targeting administrative strengthening against organised crime and corruption, to assess short-term effects, comparing them with other Axis 1 projects to improve analytical capacity, as well as some from Axis 5, on improving competences, and medium to long-term outcomes for projects operating over a longer period.
Desk analysis. Elaborations of data from CEN (Electronic data Centre of the Police); MIPGWEB (acts of Judicial Police), INPS (National Presidency); ADM (Agency of Customs and Monopoly); the Ministry of Interior. 2 surveys of beneficiaries of Axes 1 and 5 (7 and 6 responses respectively). 9 interviews with beneficiaries of Axes 1 and 5. 5 interviews with selected stakeholders of Axes 1 and 5. Survey of beneficiaries of Axis 2 (12 respondents). 4 focus group with beneficiaries of Axis 2. 9 interviews to beneficiaries and stakeholders of Axis 2.
INPS has developed applications used by central offices (20 operators) and territorial offices (1,748 users); the new ADM systems involve approximately 1,000 internal users; in the ARGO PanOptes system, which will be used by the Police Headquarters, over 1,900 cameras and about 1,400 license plate readers have been included. Improvement of information systems (Axis 1) has contributed to strengthening analytical capacity through increased interoperability, efficiency of controls and investigations, and simplification of procedures. The interventions to improve skills (Axis 5) have sometimes favoured the involvement of various entities, but it is still not possible to detect evidence of increased efficiency. New video surveillance systems improve the strengthening of territorial control, especially for operational systems working for over three months. 7 field operators indicated that they experienced changes in their perception of security following the installation of the systems.
In order to properly estimate the effects of the administrative strengthening strategy put in place, it will be necessary to wait for the conclusion of all interventions. By December 2022, only two of the 9 projects financed under Axis 1 and 15 of the 51 projects under Axis 5 were completed. No project was concluded, on improving security standards, though 7 of the 25 funded were at a very advanced stage. The contribution of the projects to actual security is not measurable.
Complement the strengthening of the interoperability between the different databases by analysing organisation. Define a set of sustainable project indicators, equipped with signalling capabilities and coordinated with other performance monitoring tools. Implement clear rules and activate task forces for beneficiaries to mitigate the complexity of project management procedures. Improve hardware and network infrastructure to handle Big Data or to manage video surveillance networks and license plate reading devices centrally. Encourage the Ministry of the Interior to play a leading role in fostering collaboration between law enforcement and other public entities to align strategies and activities.
Interim evaluation of the ITI Danube Delta supported under the Regional South-East ERDF Programme, 2021-2027, in Romania
Evaluarea intermediara timpurie a Investitiilor Teritoriale Integrate Delta Dunarii, Programul Regional Sud-Est
Liliana Lucaciu, Eugen Perianu, Monica Roman, Alexandra Sălcudean, Corina Roșu, Andreea Savin, Diana Manoleli, Adina Dragoman, Alina David, Irina Stăvilă, Ana Miruna Toma, Roberta Vancea, Laura Voinea, Aurel Rizescu, Elena Bacale
Civitta, https://civitta.com/get-in-touch/; Lideea Development Actions srl, office@lideea.eu
The evaluation examines the interventions of the Danube Delta Integrated Territorial Investment (ITI DD) area, with a focus on their management and implementation, from 21 October 2022 until 31 December 2024.
To contribute to the improvement of the implementation process and performance of the ITI DD by identifying implementation problems and proposing practical solutions for them and improving the quality and impact of the programme.
Literature review.Document research.Analysis of monitoring data and statistical office data.Analysis of administrative data.Interviews with MA and beneficiaries.Survey to beneficiaries (38% response rate, 41 responses).2 online workshops with a total of 12 people and 5 beneficiaries.Online focus group with 13 representatives of the MA.Expert panel.Case studies at project level.
The objectives and measures implemented in the Danube Delta ITI area continue to be relevant to the current socio-economic situation of the region, as well as to the needs identified in the Strategy for Sustainable Development of the Danube Delta.The directions set by the Strategy for the Danube Delta remain valid and a priority.The programme requires continuity and consolidation to reduce structural gaps and support the sustainable development of the Danube Delta area.The current progress of implementation is moderate (24% of funds have been allocated). The implementation is affected by insufficient financial, human and technical resources despite institutional progress and support provided by a specialised association.
Short timeframe of the evaluation (three months). Output analysis based solely on the assumed targets of the contracted projects.
Support the capacity of beneficiaries, through frequent training sessions and tailor-made technical support.Optimise the selection process by increasing the number of evaluators and outsourcing evaluation.Clarify technical requirements and set realistic deadlines before contracts are signed.Accelerate the launch of calls for projects, in particular for complex investments.Introduce strict contractual clauses to ensure compliance with the implementation schedules, including deadlines for launching public procurement and penalties in case of undue delays.
Evaluation of the complementary between the Research, Development and Education OP and other OPs, 2014-2020, in the Czech Republic
Zpráva z hodnocení koordinace mechanismů a procesů vedoucích k plánování a realizaci mechanismu synergických a komplementárních vazeb mezi OP VVV a jinými programy
The Report examines the complementarities and synergies between the Research, Development and Education OP and other OP as part of a broader evaluation of the OP.
To assess the coherence and complementarity of interventions financed by the OP with other OPs.
Desk research.Survey of applicants and beneficiaries, 40% response rate.Focus groups (3 with applicants and beneficiaries; 1 with the MA).Interviews.A case study/
The OP’s management procedures are consistent with the Unified Methodological Framework set up in the Czech Republic for 2014-2020 to ensure complementarities and coordination between OPs.Memoranda for cooperation have been agreed with the Enterprise and Innovation for Competitiveness OP and the Integrated Regional OP.Means have been established to coordinate the preparation of calls, though there is limited possibility to coordinate the announcement of coordinated calls.The option of involving external evaluators in the project selection process has not been used.Up to end-2016, in 60% of the OP’s calls, complementarity with other programmes was explicitly mentioned.
Analysis of synergies at project level was not possible.
Evaluation of benefits from microprojects funded under the Czech Republic-Poland Interreg OPs, 2007-2013 and 2014-2020
Evaluace přínosu Fondu mikroprojektů programu Interreg V–A Česká republika–Polsko
OP Česká republika-Polsko 2007-2013; Interreg V-A-Czech Republic-Poland 2014-2020
Evaluuj.czProkopa Velikého 2395, Brandýs nad Labem-Stará Boleslav, www.evaluuj.cz, email: info@evaluuj.cz
The evaluation examines the effects of microprojects funded under the Interreg OPs in the two periods.
To assess the context in which projects are implemented under the Microprojects Fund and their enabling factors.
Interviews.Case studies.Survey of beneficiaries (implementers of microprojects).Programme documentation.
Micro-projects are mainly concerned with promoting tourism, citizens’ events and cooperation in education, and seem to be effective in pursuing the Fund’s objectives and meeting beneficiary expectations.Main success factors are the quality of project applications, timing and the promotion of the project, and the active participation and financial stability of project partners.Investment and community projects are complementary, the former being focussed on long-term development, the latter on awareness-raising. The administrative complexity of microprojects is considered one of the main barriers to applying for support.Most of projects funded in 2007-2013 seem to have been sustainable because of the financial stability of public bodies, experienced NGOs and the small geographical location involved.
Consider concentrating funding in innovative projects and diversifying the Fund’s objectives to take account of the variety of regional needs.Clarify the expectations of stakeholders to reduce the deadweight of projects funded.Simplify administrative procedures, especially for smaller projects.Detail evaluation criteria (in particular sustainability of projects) and improve the monitoring system.
​​Evaluation of the Smart Specialisation Strategy, under the Research and Innovation ERDF/ESF OP, 2014-2020, in Italy​
PON R&I Valutazione trasversale PROGRAMMA OPERATIVO NAZIONALE FSE-FESR RICERCA E INNOVAZIONE E STRATEGIA DI SPECIALIZZAZIONE INTELLIGENTE (S3)
The evaluation analyses the contribution of the OP to the implementation of the national and regional Strategies for Smart Specialisation. Data used go up to December 2022.
To assess how the different measures of the OP (targeting only transition and less developed regions) contributed to building and consolidating regional innovation ecosystems, as well as to fostering synergies with EU programmes such as Horizon 2020 and COSME.
Document analysis.Analysis of monitoring data (primarily financial data; output indicators are provided for just one measure).Analysis of statistical data.Online interviews with MAs and key stakeholders.Survey of MAs of regional OPs, regional departments responsible for competitiveness and innovation, and regional S3 experts.Survey of university grant (StudioSì) beneficiaries (90% response rate).
Support for human capital, through funding for innovative industrial PhDs and researcher mobility, effectively promoted collaboration between universities, businesses, and research institutions. The measure contributed to developing advanced skills aligned with S3 priorities, improving training quality and employment outcomes. The alignment with European measures, such as the Marie Skłodowska-Curie Actions, was strong.The measure supporting tertiary education, implemented through the “StudioSì” financial instrument (zero-interest loans, managed by the EIB), helped to expand access to university education in strategic areas.Emergency measures introduced during the COVID-19 crisis, such as grants and non-repayable contributions, successfully ensured educational continuity and supported students in vulnerable situations.Investments in research infrastructures strengthened the scientific and technological capacity in the target regions, contributing to the consolidation of national and international networks. The measures were consistent with the European Strategy Forum on Research Infrastructures (ESFRI) roadmap.Funding for collaborative projects in technological clusters and enabling technologies resulted in a substantial number of public–private partnerships (over 2,000), with positive impacts on skilled employment and technology transfer, particularly in aerospace, health, agrifood, and advanced manufacturing. Less positive results were observed for blue growth, green chemistry, design and Made in Italy, and sustainable mobility.
For evaluating the measure “StudioSì” financial instrument, detailed data were not accessible from the EIB.
Improve coordination between national and regional levels through shared governance.Promote public-private collaboration and exchange of best practices across regions.Strengthen the alignment with EU programmes and support the internationalisation of research and innovation.
Evaluation of the Greece-Cyprus Interreg OP, 2014-2020
Επικαιροποίηση της πρώτης Αξιολόγησης της πορείας εφαρμογής και της Επικοινωνίας και Αξιολόγηση Επιπτώσεων του Προγράμματος Συνεργασίας «INTERREG V-A ΕΛΛΑΔΑ- ΚΥΠΡΟΣ 2014-2020» του Στόχου Ευρωπαϊκή Εδαφική Συνεργασίας
The evaluation examines the implementation and impact of the OP, up to end-2020.
To update the first evaluation of the OP, to assess the impact of the interventions, and to evaluate the communication strategy up to end-2020.
Analysis of relevant documentation.Analysis of monitoring and official statistical data.Theory of Change building and testing.Survey of project partners.Analysis of traffic data and targeted research on users of the Municipality of Chania digital application.Comparative analysis of treated and non-treated organisations in respect of energy saving in public buildings.Survey of those responsible for communication of projects (response rate: 38%).
The OP will reach all the output indicator targets except one. Result indicators show good progress, though the design of some indicators was not appropriate.Unit costs of the interventions implemented are lower than expected for most indicators. The ratio between the % of target achievements and % of payments is 1.35.The ICT applications developed by the Smart Cities project were used by 44% of the population, according to the survey.Projects like “GoBrand” could increase exports of the cross-border zone from 0.6% to 1.8% (based on data collected from surveys).Projects could contribute to a reduction in energy consumption by 13%. In public buildings and in pollutants by at least 7%.Projects could lead to a long-term increase in tourist accommodation by up to 14% and to a reduction of 51% in the waste disposed of by landfill.
For 4 out of 9 indicators the analysis of results cannot be considered reliable because of the early stage of projects and conceptual problems of linking interventions to the indicator.
Put more emphasis on projects with a shorter implementation horizon.Redesign some result indicators in terms of baseline and target values.Start communication activities earlier and use new technologies more.Continue to support entrepreneurship, the blue economy, tourism, digital technologies, the circular economy, and water resource management, and consider abandoning support for maritime transport.
Valutazione degli interventi riconducibili alla linea di programmazione 2014-2020 “Investimenti Territoriali Integrati”
ROP Umbria ESF 2014-2020; ROP Umbria ERDF 2014-2020
Lattanzio, https://www.lattanziokibs.com/en.html - IRIS, https://www.irisricerche.it/ - ISRI, https://isri.rm.it/category/commessa/studi-e-ricerche/
The evaluation examines Integrated Territorial Investment (ITI) in the Trasimeno area, funded under the ESF and ERDF OPs up to November 2021.
To assess the governance, implementation and results of measures up to November 2021.
Analysis of official statistics (ISTAT, INFOCAMERE, MIBACT).Analysis of monitoring data.In-depth interviews with those responsible for the measures interventions and other major stakeholders.
As at November 2021, 76% of the budget had been allocated to selected projects but only 26% spent.Delays in implementation are due to limited local administrative capacity and to the delayed adoption of the control and management system.There is a mix of bottom-up and top-down approaches, involving regional and local authorities in the design of the ITI.There is no specific monitoring system for the ITI.The coordination committee for the ITI is not effective.Technical assistance played a central role in implementation.
Start the co-design of ITI as early as possible for 2021-2027.Define a simplified management and control system immediately after the new governance model for the Trasimeno ITI is approved.Expand the range of beneficiaries.Improve multilevel governance, monitoring and coordination. Provide continuing training for beneficiaries.
Ongoing evaluation of the criteria and system of selection of projects in the Zachodniopomorskie OP
Ewaluacja bieżąca kryteriów i systemu wyboru projektów RPO WZ 2014-2020
Pracownia Badań i Doradztwa “Re-Source” Korczyński Sarapata sp.j.
ul. Spławie 53; 61-312 Poznań, Tel: 61 622 92 06-07, biuro@re-source.pl
This evaluation assesses the project selection criteria and system in the Zachodniopomorskie Voivodeship OP 2014-2020.
To assess the OP project selection criteria and system for the OP.
Desk research;benchmarking;CATI/CAWI;interviews;panel of experts.
The OP documentation is generally clear with regard to procedures and selection processes; the capacity of the administrative institutions involved in implementation of the OP is sufficient;the application assessment period is variable, the shortest assessment lasted one day, the longest 708 days;the method for recruiting experts is satisfactory.
The response rate to the questionnaires the co-financing application survey was 56%.
There is a need to:ensure full access to project selection documents;unify calls related to the same area of interventions;minimise issues that limit the website availability and so access to OP documentation.
Evaluation of the effectiveness, efficiency and implementation of the Sterea Ellada OP, 2014-2020 – 3rd deliverable
Σύμβουλος Αξιολόγησης αποτελεσματικότητας, αποδοτικότητας και εφαρμογής του Ε.Π. Στερεά Ελλάδα 2014-2020 (έτη αναφοράς 2014-2016) - 3ο παραδοτέο
The evaluation examines the implementation of OP 2014-2020, up to 24/05/2018.
To assess the effectiveness and efficiency of the OP, and the achievement of targets up to May 2018 as well as the validity of the intervention logic and to make recommendations and prepare the methodology for the future impact evaluation.
Desk research.Analysis of monitoring and statistical data.
The OP can be judged to be relatively effective in terms of specialisation, calls for proposals and projects selected but less so as regards contracts and expenditure.The degree of achievement of output targets is very low up to May 2018.The very limited number of projects completed makes it difficult to assess the overall efficiency of the OP.
Speed up the implementation of the OP.Adjust target values of indicators and re-allocate resources between PAs.Revise the OP in response to external changes (ability of businesses to access loans) and programme failures.
Evaluation of a long-term result indicator on employment under the Podlaskie OP, 2014-2020
Ocena wpływu wsparcia Europejskiego Funduszu Społecznego na poprawę sytuacji pracowników i przedsiębiorstw w województwie Podlaskim
Krzysztof Gutta, Katarzyna Gutta, Beata Belica
EU-CONSULT sp. z o.o., ul. Toruńska 18 C, lokal D, 80-747 Gdańsk
The evaluation examines how support from the OP has contributed to improving the situation of project participants in the labour market.
To estimate the value of the long-term result indicator for employment and assess the effectiveness and usefulness of the support provided.
Desk research.Analysis of monitoring and statistical data.Survey of 4,635 final recipients.Expert panel.
Future projects need more flexibility, both for the authorities and beneficiaries, to face unforeseen circumstances.Measures have not led to an increase in the employment rate commensurate to the expenditure financed.The most common reason for participating in a project is to acquire skills and knowledge, and less to find a job.There is a problem of outflow of people, and so workers, from the region.Support has been also targeted at women who are inactive because of childcare.
Promote projects using hybrid and remote work and the use of IT for education purposes.Put greater emphasis on acquiring new or updated competences, skills and knowledge.Provide more support for the employment of people aged 50 and more, and women who are inactive.Go beyond the indicator when assessing the effectiveness of projects.
Process evaluation of the Parents, Childcare and Employment ESF project, 2014-2020
Process Evaluation of the Parents, Childcare and Employment Project
OP West Wales and the Valleys ESF 2014-2020; OP East Wales ESF 2014-2020
The evaluation assesses the procedure and the implementation of the Parents, Childcare and Employment (PaCE) project which covers the entire region of Wales and is funded by the two ESF OPs East Wales and West Wales and the Valleys for the 2014-2020 period. The project aims to assist 7,884 economically inactive and unemployed parents, lone parents and parents in a workless household or assisting a second earner in a working household (between 16-24 and 25 and over) and helping 1,577 parents into sustainable employment.
The study is aimed at assessing the design and implementation of PaCE, as well as its effectiveness, i.e. how it is helping participants to overcome barriers to employment, to acquire job-relevant skills and to take up job-relevant opportunities.
The main methods used are:Desk research reviewing the strategic and policy context in which the PaCE operates.Interviews with stakeholders involved in the design and implementation of PaCE.Telephone survey of 60 participants in the project.Theory of change analysis.
Project design and development:The PaCE replicates the successful elements of earlier initiatives and modifies the way the policy operates according to the lessons learned.Shift of some tasks from local authorities to PaCE Advisers.Implementation:project launched in two phases: a pilot one in Pembrokeshire, Carmarthenshire and Ceredigion and a second one where it was rolled out in all region.The roll-out of key elements of PaCE took longer than expected affecting the initial delivery of outcomes (number of participants engaged and jobs entries achieved is still behind the target).Initial lack of funded training.Misunderstanding evident among PaCE Advisors on how to apply the childcare support.Service integration:Most PaCe Advisors have successfully integrated into Jobcentres, but their integration has been more challenging in some family-based settings.The flexible, person-led approach is welcomed by all stakeholders and participants.Delivery of outcome and performance:Given the particular socio-economic conditions of the target group, the employment outcomes for 20% of participants (the PaCE target) is challenging (particularly in rural areas which suffer from poor infrastructure).Monitoring:Administrative burdens on service providers were identified as well as some confusion among PaCE Advisors on what data they should collect and how.
The recommendations identified are:Produce a formal agreement such as a memorandum of understanding to set out roles and responsibilities of key partners (in particular local authorities).For PaCE Advisors: formulate a project-specific induction process; develop a frequently asked questions sheet to be used as a working document; introduce a buddying/mentoring system to share learning among them.To improve the integration of services: undertake close monitoring of referral routes; explore additional ways of increasing the diversity of routes for the PaCE project. Incorporate examples of eligible/ineligible evidence in the internal guidance/FaQ documentation.Introduce robust monitoring of participants and employment outcomes and commission a summative evaluation in order to use its findings to improve the project if necessary.Review project profiling six months after the project is fully operational to identify whether a re-profiling of targets is necessary.
Cohesion Policy contribution to social infrastructure development in 2007-2013
Wpływ polityki spójności 2007-2013 na rozwój i funkcjoNowanie infrastruktury społecznej
Jan Szczucki, Maciej Gajewski, Robert Kubajek, Ewa Kusideł, Małgorzata Zub, Justyna Witkowska, Katarzyna Górniak, Bogdan Pietrzak
Policy & Action Group Uniconsult Sp. z o.o., ul. Kierbedzia 4 , 00-728 Warszawa, Tel: 022 256 39 00, e-mail: biuro@pag-uniconsult.pl
The evaluation assesses the effects of ERDF programmes in the 2007-2013 period (16 regional OPs, the Infrastructure and Environment OP and the Eastern Poland OP) on social infrastructure of various kinds in Poland at the national, sub-regional and voivodeships level. The report provides evidence of the efficiency and effectiveness of interventions, their sustainability over time, their complementarity with other forms of support and investigates the viability of the supported interventions.
To assess the impact of social infrastructure interventions supported by ERDF OPs in Poland in the 2007-2013.
The evaluation uses qualitative and quantitative methods:desk research of OP documents and reports;econometric and statistical analysis;in-depth interviews with representatives of the public administration, beneficiaries, staff of the infrastructure, NGOs;CATI survey of 900 inhabitants of three regions;CAWI survey of 1,374 project’ beneficiaries;10 case studies and 50 mini case studies;panel of experts.The data used are: from interviews, surveys and an expert panel and secondary data from the monitoring system and official databases (portals: stat.gov.pl, strategist.stat.gov.pl mojapolis.pl, diagnoza.com).
Overall projects supporting the development of social infrastructure increased the availability of facilities and the quality of services and their diversification.5,695 projects were co-funded by EUR 10.8 billion of financing (31% going to sport and tourism, 27% to education, 22% to healthcare, 16% to culture and 4% to social assistance).4,714 pieces of infrastructure were built or renovated.322 ambulances were purchased and 314 new university laboratories for teaching and R&D activities were equipped.As a result more than 10,000 specialist health examinations were carried out, 249 new tourist information centres were established, and 11,000 km of hiking trails and cycle paths were constructed or modernised.There was a high level of effectiveness of projects relating to health, education and culture. In the case of tourism, sport, recreation and social assistance, there was no statistical evidence to link the improvement in infrastructure to the intervention.The investment supported generated 14,300 jobs, including 208 in research.
Only partial statistics and project monitoring data were available for 2015.
There is a need to:carefully assess the applicant’s capacity to finance the operations of a facility and the availability of similar services;involve businesses in the assessment of tourist and cultural applications;promote partnership projects through an appropriate system of evaluation criteria and identify example of good practice in the management of partnership projects to minimise costs;carry out information training for potential applicants on VAT eligibility and state aid rules.
Mid-term Evaluation of the Castilla-La Mancha OP, 2014-2020
Informe de evaluación del Programa Operativo FEDER de Castilla-La Mancha 2014-2020. Período 2014-2016
Red2Red Consultores S.L., Calle María Pedraza, 30 Planta 3ª, 28039 Madrid. España, TLF 915 501 170, red2redconsultores@red2red.net , www.red2red.net
The evaluation is concerned with assessing the pertinence, coherence of the logic of intervention as well as the progress of the OP Castilla-La Mancha 2014-2020.
The objective of the evaluation is a review of the progress made so far in implementing the OP and the results achieved. The objective is also to examine the appropriateness of the overall OP logical framework.
The evaluation adopted qualitative and quantitative methods:Desk research;analysis of relevant policy documents;analysis of statistical data;on-line survey.The data used are: primary data from on-line survey and secondary data, mainly financial data on financial progress and the OP implementation. Relevant document (e.g. OP document, ex-ante evaluation, the National Evaluation Strategic Plan, etc.) are used as well.
The main findings from the evaluation can be summarised as follows:The OP addresses regional needs and it has a high level of internal and external coherency.Various elements caused late start in implementing the OP: late approval of OP, overlap between previous and current period, lack of administrative capacity and administrative overload, difficulties in selecting projects.The late start of the OP affected the achievement; most targets set in respect of result and outputs were not achieved.The lack of data limited the possibility of analysis.The low rate of implementation calls for a reprogramming of the OP.
The evaluation highlights a lack of data especially in respect of outcome indicators.
The main policy implications are:Review the appropriateness of the OP structure and assess whether the priorities and actions are still in line with the needs of beneficiaries.Strengthen the OP implementation and make sure that financial and operational progress is made.Review the adequacy of the indicators of the Performance Framework and of the milestones.Address the ex ante conditions which are not yet met.
Evaluation of the use of financial instruments under the Śląskie OP, 2014-2020
Ewaluacja ex-ante zastosowania instrumentów finansowych w programie FE SL 2021-2027 oraz ocena tych instrumentów w ramach RPO WSL 2014-2020
The evaluation examines the implementation of financial instruments (FIs) under the OP.
To assess the effectiveness and rationale for implementation of FIs under the OP and the justification for their use in 2021-2027.
Monitoring and statistical data analysis.Documentation analysis.Interviews with 10 implementing institutions, 10 financial institutions, 15 housing cooperatives.Survey of 291 beneficiaries.Survey of 384 non-beneficiaries.Focus group.
According to 61% of beneficiaries, FI support contributed to improving financial results, to 55% to improving product quality and to 52% to strengthening price competitiveness.There are barriers to the implementation of FIs as regards energy efficiency, including the lack of legal clarity and stability.The main factors affecting the effectiveness of the measures are COVID-19, the war in Ukraine, supply chain breaks, rising construction costs and increased interest rates.
Maintain the three-tier structure for the implementation of FIs (MA, fund of funds, financial intermediary).Use FIs to support investment and the internationalisation of SMEs.Implement non-repayable or mixed instruments to support SME R&D.
Evaluation of Customer Engagement and the New Business Start Up Support projects 2007-2013
Final Evaluation of Customer Engagement and New Business Start Up
Huw Bryer (Old Bell 3 Ltd); Professor Max Munday (Cardiff University)
Old Bell 3 Ltd., IFF Research and the Welsh Economy Research Unit (WERU) of Cardiff University
The evaluation is the final report on the Customer Engagement (CE) and New Business Start-up Support (NBSS) projects, co-financed by the ERDF for 2007-2013.
To evaluate the impact of the CE and NBSS projects, particularly against the performance indicators and targets outlined in the project business plans.
The evaluation use both qualitative and quantitative methods:Desk research.Analysis of background literature, policy documents, monitoring data.CATI survey of enterprises (600) and individuals (100) supported.Visits to enterprises supported (25).Semi-structured interviews with key stakeholders (33) (e.g. Welsh Government officials, contracted delivery providers, youth entrepreneurs).Analysis of secondary data from the monitoring system.The data used are primary data from interviews, visits and survey and secondary data from the monitoring system and the ERDF Business Survey 2015.
The NBSS project exceeded all its targets except for the number of individuals assisted. 11,235 new enterprises were created (17% of the total recorded business births in Wales between 2008 and 2014).The CE project missed all its targets but supported 5,446 enterprises (6% of active enterprises in Wales in 2014).Enterprises supported were mainly young (less than 5 years old), micro businesses (0-4 employees) serving local markets and with a small turnover (less than £50,000).Of the enterprises surveyed, 14% had created jobs (485 new jobs), 8% had safeguarded jobs (205 jobs), 51% had increased their turnover, 46% had increased their profitability and 53% had introduced new products of services as a result of the support. The evaluation found evidence of a degree of deadweight in relation to the support delivered (36-47% of the gross new jobs created would have occurred even without the support). It also found evidence of small displacement effects (1 out of 10 new jobs created) and of a fairly modest multiplier effects (0.4 jobs).Given deadweight, displacement and multiplier effects, the evaluation concluded that for every 100 new jobs created an estimated 75 were additional and could be attributed to the support provided.
Segmented support should be provided. Information should be available to all businesses through the Business Wales website, whereas tailored support should be provided to more promising businesses.There should be a focus on role models and youth ambassadors talent spotting young people with the aim of directing them into business support services. Advisors should be specifically appointed to support young people setting up their enterprises.Improvements are needed in the accuracy, robustness and inter-connectedness of the on-going monitoring information, in order to have regular feedback on the progress of projects.Data monitoring procedures should record company registration numbers for enterprises supported and VAT registration numbers, so as to enable more robust counterfactual impact evaluations.
Assessment of the impact of the ESF on the social economy under the Kujawsko-Pomorskie OP 2014-2020
Ocena wpływu wsparcia EFS w ramach RPO WK-P 2014-2020 na wzmocnienie sektora ekonomii społecznej
DYSPERSJA - Anna Chrościcka, ul. Szymanowskiego 4 lok. 65, 03-477 Warszawa, E-mail: biuro@dyspersja.com
The evaluation examines the impact of the Kujawsko-Pomorskie OP on the development of social enterprises and the sustainability of jobs created in them. It also examines the social economy support system, including the Regional Centre for Social Policy (ROPS) and Social Economy Support Centre (OWES) activities.
To assess the impact of the OP on establishing social enterprises and developing their potential and the relevance, effectiveness and usefulness of the support provided, the degree of implementation of the long-term ESF indicator of the number of jobs created in social enterprises lasting at least 30 months and their quality and sustainability.
Desk analysis.CAWI survey of institutions, social enterprises (59) and local self-government units (38).CATI survey of 500 project participants.16 interviews with institutions.Benchmarking.Case studies.Workshop.
Jobs created in social enterprises have proved to be sustainable.72% of the job created lasted for at least 30 months.The survey results show that the support provided is effective.The positive effects on people who received help is partly because that was not focused on the most excluded.Social enterprises do not often have sufficient potential to reintegrate those who are socially excluded and there is insufficient cooperation with specialist reintegration bodies.While social enterprises obtain significant revenue from market activities, they need support to acquire customers and to increase their scale of operations.The accreditation system does not guarantee that the services provided by the Social Economy Support Centres (OWES) are of high quality.The current list of tasks for the Regional Centre for Social Policy (ROPS) is too wide for all of them to be implemented successfully and the Centre needs to be better informed about OWES activities and to have a greater impact on their performance.
The evaluation was carried out during the COVID pandemic which limited the field work.
Increase the regular exchange of information between OWES and ROPS and the impact of ROPS on OWES activities.Assign levels of priority to ROPS tasks for supporting the development of the social economy.Expand the possibility of supporting the socio-economic reintegration of those at risk of social exclusion employed in social enterprises.Extend the possibility of social enterprises using various forms of service provision, such as vouchers, in the next programming period.Keep the catalogue of social enterprises on the ROPS website up to date.
Evaluation of the objectives and results of the Melilla ERDF OP, 2014-2020
Evaluación de los objetivos y resultados de la estrategia del PO FEDER Melilla 2014‐2020 para el IAE ampliado 2019 y de cumplimiento del marco de rendimiento - INFORME
The evaluation is carried out for the 2019 Annual Implementation Report of the Melilla ERDF OP, 2014-2020 and relates to progress in implementing the OP over the period 2014-2018.
To assess the progress in implementing the OP and its effectiveness in relation to the achievement of the performance framework targets and result indicators.
Desk analysis of programming documents and other relevant national and regional documentation.Analysis of monitoring data.Qualitative analysis of primary data from managing bodies.
The financial execution of the OP was delayed by problems in the start-up.The OP has positively contributed to the achievement of the objectives of the Regional and Cohesion Policy of Melilla.With the exception PAs 2 and 6, the performance framework has been fulfilled. Non-compliance is due to delays in hiring in relation to PA2 and in the completion of the desalination plant financed by PA6.Documents, measures, networks and working groups have been established up to ensure that the MA and IBs have sufficient administrative capacity. Appropriate measures have been taken to integrate the horizontal principles into the execution of the OP.
Statistical data are limited and lack reliability because of the small sample size. In addition, for some indicators there are no data available.
Transfer the Performance Reserve of PAs 2 and 6 which have not reached the milestones to IP 4.e and to a new priority, IP 7.c.Include a new IP and a new specific objective, IP 7.c and SO 7.3.1, and so introduce a new beneficiary, the Port Authority, which has already demonstrated in previous periods a capacity to absorb finance.Review the methodology for calculating progress in relation to the outcome indicators.Update the values of targets that have been affected by reallocation of the Performance Reserve.
Evaluation of the 2014-2020 País Vasco ERDF OP: 2017 annual report
Evaluación de los objetivos y resultados del Programa Operativo FEDER de País Vasco 2014-2020 para el informe anual de 2017
IKEI - Parque Empresarial Zuatzu, Edificio Urumea, Planta 1ª, 20018 Donostia-San Sebastián, Tel. +34 943 42 66 10, ikei@ikei.es, www.ikei.es
The report focuses on the implementation of the 2014-2020 País Vasco ERDF OP and provides updated information on the efficiency and effectiveness of the OP by measuring the progress of relevant outcome and result indicators. The report also investigates the coherence of the OP intervention logic with the objectives of the programme and the integration of the horizontal principles.
To perform an intermediate evaluation (2017 annual report) of the efficiency, effectiveness and of other implementation activities of the País Vasco ERDF OP.
Analysis of OP’s monitoring data and databases from EUROSTAT and regional/national statistics.
Out of the 34 outcome indicators, 13 are found to achieve a high degree of efficiency (for 6 indicators no data are available).13 result indicators obtained a high degree of efficiency and the other 4 a medium degree.The OP is strongly aligned with the Europe 2020 strategy but the contribution of the OP in terms of results indicators is quite limited (as compared with the whole Basque economy).The achievements of targets within the performance framework is high for TO1, TO4 (although not as regard the renewable energy aspects) and TO5, medium form TO3 and low for TO2 and TO6.The rate of the OP financial efficiency is high (reaching 90% of the planned expenditures for 2016).OP contribution to climate change objectives is potentially relevant but efficiency of priority axes covering TO4 and TO6 is low.The review of the logical framework indicates that the specific objectives of the OP are consistent with the regional needs and challenges.
The early stage of implementation of the OP does not allow the use of certified expenditures and certified data from supported operations.
It is recommended to pay attention to the indicators showing a low degree of efficiency or for which no data are available (especially under TO2, TO3 and TO4) and to indicators that have already reached the 2023 target.
Evaluation of the calculation of long-term result indicators for the ESF and the YEI under the Knowledge, Education, Growth OP, 2014-2020, in Poland
Badanie ewaluacyjne dotyczące wyliczenia wartości wskaźników rezultatu długoterminowego w ramach Europejskiego Funduszu Społecznego oraz Inicjatywy na rzecz zatrudnienia osób młodych. Pierwsza analiza wskaźnikowa (2020)
Adam Grajek, Aleksandra Kamińska, Paulina Kowalczyk, Paweł Penszko, Cezary Przybył, Małgorzata Zub
The study examines the effect of the support provided under the OP for the YEI in terms of improving the situation of young people (aged 15-29) on the labour market.
To estimate the value of the 7 long-term result indicators for the YEI under the OP.
Desk analysis of monitoring data, data gathered in previous research and official statistics.CAWI survey, supplemented by CATI, of 504 participants.
The values of two YEI indicators (number of people participating in lifelong learning and number of self-employed 6 months after leaving the programme) are low and the indicator on increasing in the share of full-time employees is far below the target.The most difficult target group to reach are the inactive.Support has not eliminated fundamental barriers faced by those with disabilities (as shown by most indicators).74% of men and 67% of women participating found employment, the most important obstacle for women being taking care of dependents.96% of self-employed respondents stated that they would continue in business after support came to an end, but this is mainly because of their obligation to maintain economic activity as recipients of ESF support.
Make the control and accounting regulations more flexible because of the coronavirus pandemic.
Evaluation of EU support for ICT activities 2007-2013
ES fondu ieguldījumu izvērtēšana informācijas un komunikāciju tehnoloģiju pasākumu atbalstam 2007.–2013. gada plānošanas periodā un šo ieguldījumu ietekmes noteikšana
OP Infrastructure and Services 2007-2013
SIA “PricewaterhouseCoopers”, baiba.apine@pwc.com
The evaluation examines the projects co-financed under the “Development of Information System and Electronic Services” measure of the Infrastructure and Services OP 2007-2013 in Latvia.
To assess the effectiveness and impact of ICT projects co-financed under the OP to improve public services.
Desk research of OP documentation and previous evaluations;analysis of monitoring data;in-depth interviews with main stakeholders;survey of main stakeholders;3 focus groups (of construction authorities, transport and logistics and universities).Data come from programming and legislative documents, the monitoring system, national and international statistics (Central Statistics Bureau, Eurostat, OECD, World Economic Forum, Digital Society Indicators).
Overall, EUR 93 million was invested in ICT over the period.Investment in culture (EUR 14.7 million) resulted in the creation of 40 new e-services.Investment in education (EUR 9.5 million) extended the functionality of the information system, developed the exchange of data and improved data retrieval.Investment in health (EUR 10.6 million) resulted in the creation of 40 new e-services to increase the efficiency of healthcare.Investment in social welfare (EUR 4.3 million) resulted in 17 new e-services to further "Balanced Territorial Development" and "Equal opportunities".Investment in public administration (EUR 28.8 million) resulted in 41 e-services as well as new infrastructure and the development of information systems.The overall social impact of ICT is estimated to be 5.1 (out of 7) due to the wide internet availability, high indicators of ICT usage and the access to basic services.
There is a need to:ensure the involvement of end-users in the design and development of systems;use open standards and open source technologies;ensure inter-operability with other Latvian and European systems; include the results of scientific and industrial research in systems.
Assessment of the impact of the Warminsko-Mazurskie OP, 2007-2013 on the development and improvement of network connections
Ocena wpływu Regionalnego Programu Operacyjnego Warmia i Mazury 2007-2013 na rozwój i modernizację połączeń sieciowych województwa warmińsko-mazurskiego
ROP Warmińsko-Mazurskie Voivodeship 2007-2013
The evaluation examines the effect of projects financed by the Warminsko-Mazurskie OP 2007-2013 in improving networks in the region.
To assess the impact of projects financed on networks in the region.
Desk analysis of publications, reports, newsletters, statistical data, information available on websites, etc.CATI of people affected, businesses, beneficiaries, and non-beneficiaries (1031 respondents).Internet panel with people in the region and a sample from other parts of Poland (1543 respondents).34 IDIs with the MA, directors and representatives of departments, project managers, social organisations and the press.PAPIs with users of completed projects and people living nearby (511 respondents).Statistical analyses.Mini-FGIs – group interviews on preliminary versions of recommendations in seven groups of 4-5 experts.Counterfactual analyses.16 separate case studies.
Priorities for transport, environmental protection infrastructure and IT infrastructure were implemented satisfactorily.The majority of beneficiaries reported that projects had improved their profitability and the effects had been sustained. Over 65% of projects would not have been implemented without EU support and a third would have been only partly implemented.Even after the investment, there are areas where improvements are still required and specific problems of the region were not properly addressed.There is a discrepancy between the investment carried out and the barriers to the development of the region. While the focus was on the development of road infrastructure and transport availability, people are more interested in combating unemployment and low salaries.Businesses identified bureaucratic problems, lack of funds for development, lack of qualified personnel, and low regional internal demand. The main infrastructure problem reported was commuting by train, bus, and plane.The main deficiency of the programme was neglecting investment in city transport and rural area buses and focusing financing on the rail link to the regional airport. Deficiencies were also identified in water supply and sewerage infrastructure.The project selection system and promotion and information activities were assessed as very good.
Consider redirecting investment in the region to economic growth and combating social exclusion.Introduce the possibility to finance projects to increase the energy efficiency of buildings.Create the conditions for the development of broadband networks.
Thematic evaluation of action 2.3 under the Friuli Venezia Giulia OP, 2014-2020
VI Rapporto tematico di valutazione – L’Azione 2.3 del POR FESR
ROP Friuli Venezia Giulia ERDF 2014-2020
The evaluation examines the action aimed at supporting SMEs through investment in infrastructure (2.3.a) and in innovative ICT solutions (2.3.b).
To assess the results and effects of action 2.3 to support SME investment in infrastructure and innovative ICT solutions.
Desk review. Analysis of monitoring data. Survey of enterprises supported between December 2022 and February 2023 (25% response rate out of 999 surveyed). Counterfactual analysis based on matching and conditional difference-in-difference, using administrative data from AIDA.
Up to December 2022, 1,217 projects were supported in 999 enterprises through a total of EUR 250 million. Over two-thirds of resources were allocated to investment in infrastructure. Support mainly went to investment in process innovation (84% of enterprises surveyed), organisational innovations (63%) and new services and products. Half of enterprises surveyed introduced or improved advanced manufacturing software and to a lesser extent cloud computing and industrial robotics. 83% of enterprises surveyed would have undertaken the investment even without support though on a smaller scale or later. The counterfactual analysis shows supported firms, though only the small ones, increased employment and revenue more than non-supported ones. COVID adversely affected sales, client and supplier relations and investment in growth and business development for 60% of firms surveyed. 51% of firms surveyed were adversely affected by the Russia-Ukraine war because of the increased cost of energy and materials and reduced sales.
Limited availability of information on supported and non-supported enterprises.
Continue the support in the 2021-2027 period. Focus selection criteria more on innovation, technological level of investments and revise eligible minimum costs to avoid very small projects.
Assessment of the effect of measures for risk prevention under the Zachodniopomorskie OP, 2007-2013
Ocena efektów wsparcia Działania 4.5.2 Zapobieganie Zagrożeniom realizowanego w ramach IV Osi Priorytetowej Infrastruktura Ochrony Środowiska Regionalnego Programu Operacyjnego Województwa Zachodniopomorskiego na lata 2007 - 2013
ROP Zachodniopomorskie Voivodeship 2007-2013
00-807 Warszawa, Al. Jerozolimskie 96 lub 44-206 Rybnik, ul. Kowalczyka 17
The evaluation assesses the effects of the support provided under sub-measure 4.5.2 “Risk Prevention”, under the Zachodniopomorskie Voivodeship regional OP 2007-2013.
To identify the estimated value of losses which would have occurred in the absence of support and to make recommendations for 2014-2020.
CATI survey of beneficiaries and applicants;in-depth interviews with beneficiaries;interviews with applicants;interviews with experts;analysis of programming documents;analysis of calls for proposals;case studies.
The interventions contributed to the progress made at regional level on protection against natural disasters;the projects supported led to the renewal of basic equipment, especially of fire brigades;the time needed to begin a rescue operation was significantly reduced;outcome indicators do not enable the impact of projects on the region to be assessed;the average loss per risk occurrence was larger in non-supported municipalities;from 2012, municipalities supported experienced a systematic reduction in losses from natural hazards;the interventions increased moral among Firefighters, the number of volunteers and the perceived security of inhabitants.
There is a need to:diversify the co-financing levels of projects;provide assistance to interested applicants;refine the indicators used by taking account of the local effects of interventions;assign additional points in project assessments to those that entail training of rescue units.
Ex-ante study of financial instruments for the Toscana ERDF Programme, 2021-2027
Valutazione ex ante degli strumenti finanziari del POR FESR 2021-2027
The evaluation examines the possibility of using financial instruments in the 2021-2027 programme.
To assess market failures, the financial need, the legislative framework, the added-value and leverage effect of the use of financial instruments in 2021-2027.
Official statistics.Analysis of programme documents.
Over the last few years, self-financing of companies has increased, while credit from banks has decreased and venture capital and equity have been used very little.The multiplier effect (ratio between total financial resources potentially available for final recipients and EU funding) is estimated at 8.Loans and guarantees seem to be appropriate to support productive and R&D investments, equity for innovative start-up businesses.
Evaluation of the effectiveness of the Dytiki Ellada OP, 2014-2020
Αξιολογήσεις Αποδοτικότητας / Αποτελεσματικότητας για το Ε.Π. «Δυτική Ελλάδα 2014-2020» - 4ο Παραδοτέο: Τελική Προσαρμογή
The evaluation examines the implementation of the measures under all priority axes and the Smart Specialization Strategy (S3) of the OP. Data used go up to March 2020.
To examine the implementation progress for each thematic objective, to assess the contribution to regional, national and EU strategies, to examine the outcomes of the S3 and to test the logic of intervention of the programme and propose revisions.
Document analysis.Analysis of monitoring and statistical data.Analysis of the logic of intervention.Review of the SWOT analysis of the Smart Specialization Strategy.
The issued calls represent 149% of the programme’s total allocation. Contracted projects cover 47% of the allocation, while actual payments amount to 25% of it.Education/Health/Welfare infrastructure interventions are the more advanced measures while those on transport shows the lowest level of implementation.The objectives and indicators of the OP are explicitly mapped to EU2020 thematic goals and several projects aligned with EUSAIR pillars (e.g. wastewater, ports/TEN-T, Natura 2000).Funding needs are lower than expected because many interventions are contracted well below budget (e.g. construction discounts often >50% of the initial price).The OP’s intervention logic remains coherent and valid; no structural redesign is needed.
Output indicators in the monitoring system are recorded only when a project’s physical object is fully completed and are updated annually (early January), so reported values can lag behind actual progress. The “calls issued” figures also include re-issued calls and cases with no proposals, which can overstate activation. The SWOT update excludes COVID-19 impact (ongoing at that time).
Keep the OP strategy but take account of COVID-19 effects and shift funding to high-impact measures (business support, workers/unemployed, social inclusion, health equipment and materials).Refine the S3 by adding three priorities (e-mobility with renewable energy storage, circular economy, and Industry 4.0) and emphasise the link between research and productivity.Reinforce interregional partnerships—especially via EUSAIR/INTERREG—with neighbouring and Adriatic–Ionian regions to leverage complementarities (e.g., tourism).Speed up transport and energy infrastructure measures.Revise the indicators’ system, avoiding double counting, and ensure that projects reflect revised indicators.
Evaluation of SME Competitiveness Measures of the 2014-2020 OP
2014–2020 M. Europos Sąjungos fondų investicijų veiksmų programos 3 prioriteto "smulkiojo ir vidutinio verslo konkurencingumo skatinimas" poveikio vertinimo paslaugos
OP for EU Structural Funds Investments 2014-2020
Vilnius; Tel: +370 52 730101; Email: contact[eta]visionary.lt
This evaluation examines 20 measures funded by the “Enhancing the competitiveness of SMEs” OP to promote entrepreneurship (IP 3.1), to develop new business models for SMEs, especially as regards internationalisation (IP 3.2) and to support the capacity of SMEs to grow in regional, national and international markets and to innovate (IP 3.4).
To assess the contribution of the 20 measures to Europe 2020 in terms of their effectiveness, efficiency and impact in order to provide insights and recommendations for 2021-27.
Literature review, desk analysis of documentation;analysis of monitoring data and of secondary data from external statistical sources;interviews with national representatives responsible for the implementation of support measures and with other external experts;focus group;on-line surveys of beneficiaries, of non-financed companies and of companies that did not participate in the Programme (764 treated companies and 666 not treated companies);counterfactual impact analysis;case studies in order to carry out a detailed cost-benefit analysis of specific support measures;desk analysis of international examples of “good-practice”.
Measures under IP 3.1 "promoting entrepreneurship" will probably have the greatest impact on the competitiveness of SMEs and will directly contribute to Europe 2020. The counterfactual impact assessment shows a positive impact on business productivity. Most beneficiaries surveyed intend to continue investing their own resources in innovation and industrial development when support comes to an end.Measures to support the internationalisation of SMEs (IP 3.2) will not have significant short-term effects on exports, since they have involved only small one-off activities.Measures to promote new businesses are likely to have medium-to-large effects on competitiveness and employment, though they are not yet appreciable because not enough time has elapsed. Financial instruments (FIs), in particular venture capital funds, will play a key role in supporting competitiveness. The large amount of funding allocated should have a lasting impact on the growth and profitability of new companies.The impact of measures to support eco-innovation will only be moderate. Participation of Lithuanian SMEs in measures was low and part of planned funding risks not being allocated. Very few companies funded report intending to continue investing in eco-innovation in the future.Consultancy initiatives are low-value activities and are more expensive to manage than other grants of supporting the competitiveness of SMEs or FIs. Advisory measures will have a small impact on business performance as they are one-off and of low quality.
Several limitations and the means of overcoming them are indicated in the counterfactual analysis (short time from the end of interventions, low reliability of data from survey).
There is need to:strengthen investment supporting the competitiveness of SMEs, combine grants, non-financial measures (mentoring) and risk capital to stimulate the creation of new high-tech businesses; support the development of a business angel community in Lithuania by specific incentives, implement specific initiatives to create and maintain a business-friendly environment (e.g. tax breaks for start-ups, improving the availability of high-tech skills);integrate measures to support the internationalisation of SMEs into long-term and multi-stage business development programmes; emphasise the financing of fast-growing start-ups, create transnational industrial partnerships; attract new high-tech companies from abroad; implement the Industry 4.0 paradigm and technologies;review less effective support measures such as eco-innovation and advisory measures; finance eco-innovation by other types of incentives (e.g. tax breaks); communicate and inform companies on the opportunities and benefits expected from this type of investment; limit advisory measures to highly specialised services (e.g. legal advice on intellectual property);increase the share of FIs in support of SME competitiveness;promote measures combining different forms and sources of co-financing (e.g. grants and non-repayable loans) and integrated measures for business development that finance several types of activity in a single project (consulting, training, industrial development);reduce the administrative burden on the applicant, particularly in the early project selection stages.
Evaluation of vocational measures in Nordrhein-Westfalen
Das Handlungsfeld "Berufs- und StudieNorientierung" im Landesvorhaben "Kein Abschluss ohne Anschluss Übergang Schule-Beruf in NRW" - Ergebnisse der Evaluation - Forschungsbericht
Stöbe-Blossey, Sybille / Brussig, Martin / Kirsch, Johannes / Ratermann, Monique / Boockmann, Bernhard / Nielen, Sebastian / Puhe, Henry / Kleinemeier, Rita, Forschungsteam IAQ, IAW und SOKO. Düsseldorf: Ministerium für Arbeit, Integration und Soziales de
IAQ, Universität Dusiburg- Essen, Institut für Arbeit und Qualifikation, Gebäude LE 47048 Duisburg, Tel: +49 203 379 - 1806
Impact evaluation of ERDF OPs 2007-2013 'Convergence' and 'Competitiveness and employment'
Evaluation des impacts des programmes opérationnels FEDER 2007-2013 “Convergence” et “Compétitivité régionale et emploi”
OP Hainaut ERDF 2007-2013 Objective Convergence; OP Wallonie (hors Hainaut ) ERDF 2007-2013 Objective Competitiveness and Employment
The evaluation concerns the ERDF 'Convergence' (Hainaut) OP and the 'Regional competitiveness and employment' OP in Belgium, particularly, Axis 1: ‘Enterprise creation and employment’; Axis 2: ‘Development of Human Capital, knowledge, know-how and research’ and Axis 3: ‘Balanced and sustainable territorial development’.
The evaluation assesses the effectiveness, efficiency and impacts of the two OPs, especially on employment. The evaluation also assesses the impact on enterprise competitiveness, the overall level of innovation and the attractiveness of the region.
The evaluation assesses the effect of the OPs using a theory based approach and various qualitative and quantitative methods:in-depth interviews with different stakeholders (about 30 interviews with implementing agencies; around 10 with beneficiaries reporting the largest number of job created);analysis of statistical data (monitoring data and data recorded in EUROGES on the jobs created);on-line survey to different types of beneficiary;case studies (one for each of the 10 measures implemented by the two OPs);analysis of relevant policy documents and studies;focus groups with implementing authorities and beneficiaries.The data used are: from interviews, an on-line survey and focus groups; secondary data from the monitoring system, OPs documentation and implementation reports, studies, and statistical databases, such as EUROGES and Eurostat.
OP effectiveness: Axis 1: almost 2,881 enterprises supported in their investment and nearly 3,817 enterprises receiving support services;Axis 2: 27 research centres and universities supported, nearly 800 companies receiving R&D support;Axis 3: the expected results were not achieved, land purchase appears to be the major obstacle for infrastructure projects.OP efficiency: The cost per job created amounted to EUR 63,000 under OP ‘Regional Competitiveness and Employment” and EUR 53,000 under the OP ‘Convergence”.OP impact: Nearly 10,000 direct jobs created over the programming period.The OPs supported over 380 companies a year. 53% of enterprises surveyed reported that their turnover increased over the period, but only 14% attributed this to ERDF support.The impact of ERDF support on innovation was small, in line with the limited support going to RDTI in Wallonie in relation to total expenditure.The OPs helped to increase the visibility and reputation of research centres and universities.Added value: Only 14% of the beneficiaries surveyed reported that the project would not have been implemented without ERDF support.
The study indicated that data on jobs created should be interpreted cautiously since most of the direct jobs identified were not directly identified by the monitoring system.
Improve the targeting of business investment interventions using financial instruments as well as grants and adapting aid to the size of companies and their circumstances so as to support companies with strong potential for growth and employment creation.Strengthen the impact of interventions on innovation by adopting a demand pull approach.Integrate sanitation and infrastructure projects better into urban strategies.Adapt the indicator system better to the desired effects of programmes.
Evaluation of measures for tourism attractiveness supported under the West Wales and the Valleys ERDF OP, 2014-2020. Final report
Evaluation of Tourism Attractor Destinations: final report
Brett Duggan, David Howells, Peter Cole, Kara Stedman
The evaluation examines the Tourism Attractor Destinations (TAD) measure under the OP. The development and delivery of TAD projects spanned the period of 2018-2023.
To assess the implementation and outcomes of the measure as well as to draw lessons learned to inform future strategic priorities to support the visitor economy.
Desk research (ex. policy documents and strategic plans for the tourism sector).Analysis of monitoring data and statistical office data.Analysis of administrative data.Scoping interviews with 8 “Visit Wales” and Welsh Government representatives.Interim evaluation fieldwork with 13 projects.Interviews with Welsh Government representatives and strategic stakeholders.Interviews with representatives of TAD projects.Visit surveys from 4 projects.
The TAD measure was strategically well aligned with Welsh Government policies to support tourism and regeneration.Significant investment in a relatively small number of destinations has succeeded in enhancing the tourism offer, attracting new visitors and serving as a catalyst for other tourism-related and regeneration activity.The scale of investment in a small number of sites distinguished the operation from other investment funds such as the Tourism Investment Support Scheme, Wales Tourism Investment Fund and Micro Small Business Fund.Generating sufficient revenue streams to cover operational and management costs continues to be a challenge for several projects, particularly those operated by public agencies.The TAD measure has supported and served as a catalyst for the regeneration of local areas. It contributed to well-being by providing access to physical recreation opportunities for residents in local communities and visitors.TAD projects enabled new local partnerships to be formed which supported project delivery. The network of joint beneficiaries also benefitted from opportunities to discuss project delivery and learn from each other’s experiences.
Only 4 TAD sites had survey responses, and the total number of completed answers was relatively low (371).
Ensure that projects carry out self-evaluation activity, including the collection of visitor data.Undertake longer term evaluations of sites in order to understand their impact on visitor perceptions over time and their contribution to enhancing the local and regional tourism landscapeExplore further opportunities to facilitate collaborative marketing and promotion between TAD projects.Consider how TAD projects can be further supported through strategic programmes to continue their growth.
Evaluation of the regional Research and Innovation Strategy for Smart Specialisation (RIS3) under the Aragón ERDF and ESF OPs, 2014-2020
OP Aragón ESF 2014-2020; OP Aragón ERDF 2014-2020
Raquel Ornat Clemente, Enrique Navarro Rodriguez
Dirección General de Investigación e Innovación del Departamento de Ciencia, Universidad y Sociedad del Conocimiento, https://www.aragon.es/organismos/departamento-de-empleo-ciencia-y-universidades/direccion-general-de-ciencia-e-investigacion
The evaluation examines the implementation of measures for efficiency and sustainability of resources (energy and water), competitiveness (logistics and transport materials), well-being and quality of life (agri-food industry, tourism, and health) funded by the RIS3 under the OP. Data used go up to September 2020.
To assess the achievement of the objectives of the RIS3 and assess the effectiveness of the measures to promote research and innovation. To analyse the role of RIS3 in transforming the regional context and guiding actions towards smart specialisation in key priority sectors.
Analysis of statistical data (Eurostat, INE, IAE). Analysis of Monitoring data (Monitoring system and public departments). Analysis of Administrative data from INAEM, IACS (financial and employment registers). 27 questionnaires with key actors of the R&D system. 4 focus groups, involving representatives from public institutions, businesses, academic entities and research organisations. Analysis of secondary data (reports from parallel programmes, documents on PAID-RIS3, InnovAragón).
Stakeholders demonstrated great motivation and participation in the research and innovation system, but lack of appropriate institutional coordination limited the overall effectiveness of the strategy. The strategy placed excessive emphasis on basic research and training and neglected applied development and internationalisation. Although some measures led to positive results, the almost exclusive reliance on grants and agreements hindered the implementation of more innovative tools such as Public Procurement of Innovation (CPI). The number of innovative companies in the region decreased significantly from 867 in 2011 to 498 in 2018, and the innovation intensity is below the national average. The strategy created opportunities for collaboration between stakeholders, but some objectives were overambitious relative to the region’s capacities and context.
Indicators are often too generic and not well-aligned with specific measures. Quality of available data is limited, with some data being incomplete or poorly structured. Difficulties in involving stakeholders in the evaluation, which limited the comprehensiveness of the data collected.
Develop a robust governance structure to ensure effective management and the engagement of all stakeholders. Expand funding instruments and integrate innovative tools to avoid exclusive reliance on European funds. Encourage transnational and cross-sectoral projects to strengthen competitiveness.
Evaluation of the criteria and system for project selection in the Kujawsko-Pomorskie OP
Ewaluacja bieżąca kryteriów i systemu wyboru projektów RPO WK-P na lata 2014-2020
Pracownia Badań i Doradztwa “Re-Source” Korczyński Sarapata sp.j. ul. Spławie 53 61-312 Poznań
The evaluation assesses the project selection criteria and surrounding procedures for the Kujawsko-Pomorskie Voivodship regional OP for 2014-2020.
The main purpose of the evaluation is to assess the criteria and procedures for project selection to ensure that funding is used in the most effective way. 57 measures and 105 schemes are covered.
desk research;benchmarking;project selection criteria validation;CAWI/CATI interviews;interview with Managing Authority;two panels of experts.
The current planning of projects is assessed positively;transparency of the system is ensured by of regulations providing information on how the system works;calls for tender and rules for tendering are assessed positively;criteria for selecting projects were considered objective, verifiable and precise.
Spread calls for proposals over the year rather than concentrate them in a specific period;provide information on calls for proposals for the coming years;put more emphasis on the dissemination of information;ensure that the principle of impartiality is respected;if there are negotiated procedure, make sure that they do not hinder the timely implementation of projects;make clear the kind of information which applicants need to provide;ensure that horizontal principles are included in the selection criteria.
Meta-analysis of evaluations assessing ESF support in Poland (2018 edition)
Metaanaliza wyników badań ewaluacyjnych dotyczących oceny wsparcia z EFS – RAPORT CZĄSTKOWY 2018
Zofia Bieńkowska, Adam Grajek, Paweł Huras, Aleksandra Kamińska, Sebastian Pałka, Paweł Penszko, Wojciech Pieniążek, Cezary Przybył, Agnieszka Śnieżek, Małgorzata Zub
Evalu Sp. z o.o. , ul. Dzika 19/23 lok. 55, 00-172 Warszawa
The evaluation summarises the results of the 38 evaluations of ESF support in Poland in 2017.
To summarise the results of the evaluations of ESF support in 2017.
Meta-analysis of studies.Group interviews.Analysis of databases.
Support for day care homes and assistance to young people to enter the labour market, has delivered the desired results.53% of ESF supported enterprises continued to operate after funding ended.The tendency of officials to recruit participants with favourable labour market prospects led to a decline in effectiveness of interventions.Support provided by relatives and co-workers is identified as a key factor which significantly affects their chances of finding a job.
There is a need to:Facilitate the use of ESF support and knowledge about it.Ensure the competence of officials to deal with integration and activation.Analyse and create new measures at institutional level.
Evaluation of support for female victims of violence under regional OPs and the Public Sector Reform OP, 2014-2020 - Final report
Τελική Έκθεση Αξιολόγησης στην Μελέτη αξιολόγησης της λειτουργίας των δομών υποστήριξης γυναικών θυμάτων βίας στα ΠΕΠ και στο ΕΠ ΜΔΤ
OP Public Sector Reform 2014-2020; All ROPs 2014-2020
The evaluation examines support for women victims of violence financed by the regional OPs and the Public Sector Reform OP, 2014-2020, covering the period 2015- 2018. The regional OPs provide support services in 61 regional centres (under TO9), while horizontal operations are financed at national level to ensure a common approach (TO11).
To assess implementation and identify any bottlenecks and shortcoming, the results of interventions and the impact.
12 interviews with those responsible for implementing the OPs and managing the database;a focus group with 20 people involved and not involved in the support;survey (questionnaire) of those providing support (244 responses out of around 590 sent);survey (questionnaire) of individuals supported (228 interviewed);analysis of indicators.
The interventions are consistent with the specific objectives of the OPs and with the national Social Integration Strategy.The network of centres has expanded the target group by providing services to a wide range of beneficiaries. Certain population groups (e.g. Roma, disabled) have not been covered sufficiently.Beneficiaries report being satisfied with the services received except legal support.Shortcomings have been identified in the psychological supervision provided.The performance indicator targets suggest satisfactory progress.Communication activities led to an increase in the number of women visiting centres, but they are focused too much on domestic violence and sexual harassment.There are shortcomings in the professional staff providing child support.Beneficiaries are particularly interested in job counselling, but this does not seem to affect the way they approach work.Support was useful in helping beneficiaries access the labour market.
It was difficult for the survey to reach either because they were supported a long time before or for problems of data confidentially.
Provide additional support on work-life balance and on legal rights;expand beneficiaries so that the network becomes a fully inclusive, covering employment, education and other issues as well as violence;provide training to network personnel;improve the SOS Helpline;upgrade the database;strengthen dissemination activities;provide psychological supervision to staff.
Evaluation of the implementation of the Central Macedonia OP (ERDF+ESF) 2014-2020
Αξιολόγηση της πορείας εφαρμογής του Επιχειρησιακού Προγράμματος Περιφέρειας Κεντρικής Μακεδονίας 2014-2020
Averof 30, 10433, Athens, Greece; tel: +302108214407; email: speed@speed.gr
This is the first evaluation of the implementation of the Central Macedonia OP 2014-2020, aimed at suggesting adjustments to improve its efficiency and effectiveness.
To assess the effectiveness and efficiency of the implementation of the OP up to end-May 2018, to review the timeliness of the strategy and internal coherence of the intervention logic, to assess the feasibility of the milestones set in the performance framework and to prepare the basis for future impact evaluations and assess the availability of necessary data.
The analysis is based primarily on monitoring data, supplemented by information from documentation and interviews with MA executives. Statistical and qualitative data are used mainly to identify delay factors and data from 2007-2013 are used to assess certain aspects of different types of interventions such as average cost and average completion time.
Financial progress differs between Axes. PA4, PA6 and activities involving healthcare and minority groups (Roma and migrants) are delayed because of lack of organisation and staffing of beneficiaries.The outcomes of the OP seem likely to surpass the targets set in most sectors by 2023, except those relating to support of business due to delays at central level.The OP remains timely, coherent and consistent. It is compatible with the goals of recent national and regional strategies. Changes in needs and the persistence of certain socio-economic factors dictate the need for marginal shifts of resources from 'sustainable development' to 'inclusive growth, largely for reasons of efficiency.
The evaluation suggests:removing two Specific Goals (6F1 and 9C1);abolishing or changing 3 result indicators and modifying the method of measuring another;revising targets for 6 Specific Objectives;changing the way the economic indicator F100 is calculated.And to increase the rate of implementation:activate State aid measures, as these represent 15% of the OP budget and contain projects which can be implemented quickly;accelerate projects and stabilise payment flows in respect of Specific Objectives 5a1, 6b1 and 10a;accelerate the rate of implementation of Sustainable Urban Development plans, both because they account for 10% of the OP budget and because they will produce leverage effects in different IPs, as well as increasing the visibility of the OP.
Evaluation of communication activities of OPs
A 2014-2020 közötti operatív programok végrehajtásával kapcsolatos kommunikációs tevékenység értékelése
Field Consulting Services Zrt. & Collectivo-Intelligence Kft.
OP for the implementation of YEI in mainland France and outermost regions – YEI long-term result indicators, 2017 survey
Note d’analyse des indicateurs de résultat à longterme – Résultats de l’enquête à 6 mois 2017PO national IEJ
OP for the implementation of YEI in mainland France and outermost regions
This is the 2017 annual report on the OP for the implementation of YEI in France 2014-2020.
To assess the effects of the OP and its progress toward achieving its targets.
The evaluation is based on a survey of a representative sample of beneficiaries (total sample size=10,000; response rate=31%).
Over 28,000 participants were employed, six months after the end of their participation (12,000 in the 2016 survey; the new figure is 137% of the 2023 target).480 participants were self-employed 6 months after the end of their participation (170 in 2016; the new figure is 289% of the 2023 target).Over 11,000 participants were following additional training, an apprenticeship or a traineeship 6 months after the end of their participation (3,300 in 2016 survey; the new figure is 206% of the 2023 target).60% of the participants received at least the offer of a job, traineeship, apprenticeship or training proposal according to data from project managers (22% according to the survey).The employment rate of participants is high (40% after participation, 49% after 6 months, 50% at the time of the survey) and tends not to diminish over time. The figures are higher than in the 2016 survey.58% of respondents consider their job to correspond with their level of qualifications or to be higher.Over 67% of respondents had a net salary of over EUR 1,000 a month (10 percentage points difference between men and women).72% of respondents had a full-time job (10 percentage points difference between men and women).Education level is the most important influence on the chances of finding employment, the least well educated being more likely to follow a training or study pathway after participation.
The figures presented in the report are restricted to participants who left the programme between 1 July 2015 and 30 June 2016 (57,328).
Analysis of investment needs and forms of support in Kujawy i Pomorze in 2021-2027
Analiza potrzeb inwestycyjnych i form wsparcia w województwie kujawsko-pomorskim
Policy & Action Group Uniconsult; Imapp Consulting
Policy & Action Group Uniconsult Sp. z o. o., ul. Kierbedzia 4, 00-728 Warszawa, www.pag-uniconsult.pl; Imapp Consulting Sp. z o.o., ul. Puławska 145, 02-715 Warszawa, www. imapp.consulting.eu
The study identifies investment needs and possibilities of using financial instruments (FIs) in the 2021-2027 OP.
To assess the possibilities of using FIs in 2021-2027 and investment needs and the financing gaps in the main areas of regional development.
Desk research.Analysis of monitoring data and official statistics.Interviews.CATI survey of 1022 enterprises.CAWI survey of 71 local government units.CAWI survey of 225 final recipients.
The financing gap for support of SMEs is estimated at PLN 250 million (around EUR 55 mn).The financing gap for local government investment is estimated at PLN 225 million a year (around EUR 50 mn).
Do not use FIs for R&D projects.Establish financial and mixed instrument as forms of support for SMEs.Use FIs for energy efficiency project and for those generating savings in energy consumption or revenue from energy sales.
Evaluation of measures for competitiveness under the Emilia Romagna OP, 2014-2020
Asse 3 Competitività e attrattività nel sistema produttivo - Azioni 3.3.2 e 3.3.4
https://www.irsonline.it/en, https://www.nomisma.it/
The evaluation examines measures to stimulate business competitiveness, especially in respect of culture and tourism, handcrafts, and retailing, in the light of the effects of Covid, up to February 2022.
To investigate the characteristics of the projects, the results achieved up to February 2022 and to make recommendations for 2021-2207.
Analysis of monitoring data.Survey of beneficiaries (66% response rate).
73% of beneficiaries report having achieved the objectives set. For those that have not, the main reasons are the pandemic and insufficient financial resources.The OP output target for new products introduced to the market has already been achieved, while 18% of beneficiaries report having increased employment as a result of projects.Almost 90% of beneficiaries would have abandoned projects or reduced their size without support. 78% of beneficiaries plan to invest in the green economy, especially in energy efficiency, and digital solutions, though only 22% of them know about Digital Innovation Hubs, which represent a link between companies, research and the financial system.
Simplify administrative procedures and requirements for beneficiaries.Increase financial support and ensure availability of competences for small firms to facilitate their investment plans.
Evaluation of the use of ESI Funds in respect of energy under the Large Infrastructure OP, 2014-2020, in Romania
Evaluarea utilizarii Fondurilor ESI în sectorul energetic
World Bank in Romania, https://www.worldbank.org/en/country/romania
The evaluation examines the use of ESI Funds in respect of energy from April 2021 to March 2022, assessing 85 projects.
To assess the relevance, effectiveness, efficiency, coherence, and sustainability of 85 projects carried out between April 2021 and March 2022.
Documentation analysis.Analysis of data from the National Institute of Statistics and the National Energy Regulatory Authority.Analysis of monitoring data.Individual interviews with experts and beneficiaries, and group interviews with the MA and grant recipients.Analysis of ex-ante CBAs prepared by beneficiaries.9 case studies.
The level of implementation of energy projects is low, and they may not produce expected results by end-2023; because of inflation, more funding will be spent on less physical progress and the war in Ukraine may lead to delays in the import of supplies.The Transgaz project under SO 8.2 (gas network extension) has been completed.All SOs under PAs 6 and 7 and SO 8.2 are unlikely to be completed by 2023.The main obstacles to project implementation are insufficient capacity to assess projects, non-compliance with EU State aid rules, public procurement and supervision of large works procedures, and an increase in supply costs.Despite the negative internal rates of return, projects are justified by non-financial benefits, i.e. energy savings, reduction in energy losses, and increased renewables.Overall, progress in pursuing energy objectives can be directly attributed to the EU interventions, except for PA7, where a few other resources have been invested in the district heating system.Beneficiaries would not have implemented projects without EU support (except for SOs 6.2 and 6.4), or the payback period would have been longer.
The impact (including potential impact) of interventions could not be assessed at this stage. Monitoring data have some limitations. It was difficult to reach some key people in the MA. Some result indicators are not suitable because of lack of data and do not capture the expected change. For SOs 6.3 and 10.2, it is difficult to assess the progress likely to be achieved by the end of the period and which projects should be shifted to the next period. The cost-effectiveness of projects cannot be properly assessed at this stage because of uncertainty over future benefits.
Provide additional support to public authorities to prepare for public procurement.Revise energy policy priorities in the light of the latest EU policy developments in preparation for 2021-2027.Complete as many projects as possible under the current OP and continue the others under the 2021-2027 OP.Strengthen strategic planning and evidence-based policy making in the energy area.Ensure capacity building of staff.Ensure knowledge sharing between current and potential beneficiaries.Reassess cost-effectiveness of projects after the energy and economic crisis.In 2021-2027, coordinate energy efficiency in buildings with measures such as the renovation of district heating systems.
Evaluation of the Programme for Early Diagnosis of Chronic Kidney Disease under the Łódzkie OP, 2014-2020
Ewaluacja „Programu wczesnego wykrywania przewlekłej choroby nerek dla mieszkańców województwa łódzkiego” w ramach RPO WŁ 2014 - 2020
dr I. Franckiewicz-Olczak, dr A. Podkońska, K. Szczepaniak, M. Białuska, E. Pach, prof. dr hab. n. med. M.Nowicki
Biuro Badań Społecznych Question Mark Sp.z o.o.; Al.. 1 maja 9 lok. 4, 90-717 Łódź, www.questionmark.pl
Evaluation of the implementation of the Programme for early detection of chronic kidney disease in Łódzkie.
Assessment of the effectiveness, efficiency, usefulness and durability of the Programme.
Desk research.Survey of 333 Programme participants.24 interviews with patients.16 interviews with institutions involved in the implementation of the Programme.Interview with experts.
Increased knowledge and awareness of a healthy lifestyle is the most important and durable effect of the programme; the least durable effect is regular reporting of diagnostic tests.The programme has contributed little to the early detection of kidney disease in the region because of lack of interest in participating, the need to fill out extensive forms and the age limit of beneficiaries.The programme has provided services unavailable to people with diagnosed kidney disease, in the form of nephroprotection education and dietary consultations.
Remove age limits for participation in the programme.Supplement the support for people who have not been diagnosed with kidney disease with education activities and the possibility of consulting a nutritionist.Introduce the possibility of organising education workshops partly offline and partly online.
Analysis of reasons for reduction in III PA of OP KED competitions
Analiza przyczyn obniżenia kontraktacji w konkursach III OP PO WER
Internal evaluation of National Center for Research and Development
Final report. Part A: Evaluation of the OP. Part B: Implementation report
Endbericht. Teil A: Evaluierung des Operationellen Programms. Teil B: Durchführungsevaluierung
Mid-term evaluation of the ‘Administrative capacity’ PA of the Cohesion Policy Funding OP, 2014-2020, in Estonia
Prioriteetse suuna „Haldusvõimekus“ vahehindamine
Tauno Õunapuu, Merle Tambur, Katre Eljas-Taal, Merit Tatar, Tarmo Kalvet
LevelLab OÜ, Technopolis Group Eesti OÜ. Balti Uuringute Instituut
This is the mid-term evaluation of the ‘Administrative capacity’ PA of the Cohesion Policy Funding OP, which examines activities in three areas: development capacity of government, management capacity and quality of policy-making.
To assess the implementation and effectiveness of activities supported under the ‘Administrative capacity’ PA.
Documentation analysis;analysis of data from the Structural Support Operational System and Statistics Estonia;4 focus group interviews and 13 personal interviews with experts.
The quality of government decisions, exchange of information and transparency have improved.Local government activities supported were linked to administrative reform and created preconditions for improving development capacity, quality of policy-making and provision of public services. But development capacity in local governments remains uneven.Measures have helped to increase management capacity of government.The training system is decentralised and fragmented.Establishment of task forces and expert groups helped the cross-ministerial network of cooperation, but cooperation between authorities is often based on informal networks.The quality of impact assessments has improved, with more systematic involvement of stakeholders. But there is a persistent gap between evaluations findings and decision-making.Involvement of stakeholders and social partners in policy-making has failed to increase.There is no long-term action plan for the development of administrative capacity.
The evaluation model used is context-dependent, as success factors may change over time.
Monitor the administrative capacity of local governments regularly.Use best practice to strengthen cooperation between government organisations and reduce duplication.Establish a public sector competence centre for training and create a system for knowledge sharing.Initiate impact assessments before political decisions are taken.Develop a broader horizontal administrative capacity development plan for both local authorities and the state, as part of the Estonia 2035 strategy.Develop alternative means of financing to ensure sustainable development of administrative capacity.Establish formal cooperation networks.
Mid-term evaluation of supports for research from the National Centre for Research and Development – Smart Growth OP 2014-2020
Ewaluacja Programu Pomocowego NCBR - raport mid-term
Fundacja Idea Rozwoju - Policy & Action Group Uniconsult
Fundacja Idea Rozwoju; tel: +48506067751; IMAPP sp. z o.o.; Warszawa; tel: +48223905304; imapp@imapp.pl - Policy & Action Group Uniconsult Sp. z o.o.; Warszawa; Centrum Ewaluacji i Analiz Polityk Publicznych; Kraków
The evaluation assesses the support for research projects funded under the Smart Growth OP, focusing on three strategic interventions, TECHMATSTRATEG, BIOSTRATEG and STRATEGMED, implemented by the National Centre for Research and Development. This is the third of 5 planned evaluation reports.
To verify the intervention logic of the programme, assess achievements so far and identify the changes in behaviour of beneficiaries due to the support received.
Desk analysis.Analysis of monitoring data.Analysis of data on the socio-economic context.Surveys.Individual interviews and a focus group.
The beneficiaries selected differ from other enterprises in Poland by being larger and disproportionately in manufacturing. There is no risk however, of this having an adverse effect on market competition.As of May 2018, 6,858 enterprises had applied for funding (6,307 from the OP and 551 from the three national programmes).44% of the resources for axis I were allocated but only 27% of those for axis IV. The OP shows a delay of half a year as compared with the previous programming period. Payments amount to 22% of the total funds, so that projects are at an early stage of implementation.Micro enterprises represent 31% of the companies supported by the OP, small ones 24%, medium 22% and large 24%. Most companies have previous experience in R&D, implying there is only a small risk of failure in achieving objectives, but also a potential larger deadweight.70% of the beneficiaries had conducted R&D activities in the three years before the programme.In the short-term, support has increased enterprise spending on R&D, as expected.The projects increased cooperation of enterprises with research centres.So far, it is not possible to estimate the economic impact of the R&D projects supported on the economic situation of enterprises.
Only a small number of projects have been completed, making the empirical basis for the analysis of effects extremely limited.
There is a need to:increase funds for enterprises with little R&D experience;identify in advance risk factors for project failure;identify a means of supporting particularly innovative ideas;increase the attractiveness of financial instruments for final beneficiaries as an alternative to funding from the OP.
Evaluation of the implementation of PA3 of the Metropolitan Cities OP, 2014-2020, in Italy - year 2021
Report finale Indagine n. 1 – Rilevazione degli indicatori di risultato e analisi degli indicatori di output e di risultato dell’Asse 3 del PON Metro e supporto alla redazione della Relazione Annuale di Attuazione (RAA) – 25 luglio 2022
IRS, https://www.irsonline.it/ - PTSCLAS, http://www.ptsclas.com/en/home-en/
This is the final report of the quantification of result indicators for PA3 (TO9). The analysis extends up to 31 December 2021.
To assess the level of achievement of result indicators up to end-2021, 12 months after the end of projects.
Analysis of monitoring data.Interviews with representatives of IBs.
Action 3.1.1 (on housing poverty) did not reach the target of housing: 90% of the disadvantaged (48%) and of 85% for those living in families with no-one employed (55%).Action 3.2.1 (on marginalised communities, e.g. Roma) fell well short of the target of 85% of people finding a house after a year (63%) and of 60% for people finding a job or gaining a qualification after 1 year (22%).Action 3.2.2 (on the homeless): registered progress, but the target of 85% of the homeless finding a home after 1 year was not reached (50%).Action 3.3.1 (on social inclusion of the disadvantaged) exceeded the target of 20% for the activation of social inclusion networks in run-down urban areas (100%), and of 20% for those disadvantaged overcoming social exclusion (70%).
Result indicators for action 3.4.1 on innovative social services for multi-problem families were not quantified because they were outside the scope of the analysis. For some cities, follow-up data on the condition of beneficiaries were not available, because of the high turnover of social workers.
Train public officials in IBs as well as social workers, in monitoring and evaluation, to avoid lack of data.Focus activities on woman better to avoid a gender gap.
Summary of evaluations of the Brandenburg ERDF OP, 2014-2020
Begleitende Evaluierung des Operationellen Programms des Landes Brandenburg für den Europäischen Fonds für regionale Entwicklung 2014 bis 2020. Endbericht.
Michael Ridder, Kathleen Toepel, Gerhard Untiedt
MR Gesellschaft für Regionalberatung mbH, m.ridder@mr-regionalberatung.de; GEFRA Gesellschaft für Finanz- und Regionalanalysen; IfS Institut für Stadtforschung und Strukturpolitik GmbH
The evaluation summarises the results of previous evaluations of the OP, in different areas such as innovation and competitiveness, reduction of CO2 emissions, integrated development, and COVID-19 response.
To assess whether the measures contributed to the Europe 2020 objectives as well as to the thematic and specific objectives set in the OP, including the cross-cutting ones.
Analysis of official statistics.Analysis of monitoring data.Interviews with beneficiaries and experts.Analysis of previous evaluations of the OP.
Overall, the measures implemented made a valuable contribution to the Europe 2020 objectives, as well as to the thematic and cross-cutting objectives of the OP.Investments in university infrastructure have created the necessary conditions for research and development activities. Without the support, most of the innovation projects would not have been implemented, though this is not a sufficient condition for a successful technology transfer from science to industry.The measures have improved company competitiveness, but the gaps in productivity and income per capita with the average level in Germany remain.Measures concerning CO2 emissions focused primarily on energy efficiency and led to an annual reduction of 157,000 tonnes of CO2, though previous evaluations report that a different allocation of resources might have been more effective.Measures for integrated development were mainly implemented through individual projects and had a limited impact on the overall socio-economic development of the region.Regarding crisis management, commercial and infrastructural investments contributed to improving the competitiveness of tourism locations as planned.
On innovation, make further efforts to stimulate companies that have not yet been active in innovation.On competitiveness, support start-up activities, the ability to innovate and the internationalisation of SMEs.On project selection, introduce “hard” criteria, such as the expected reduction of CO2 emissions, to maximise benefits for the environment.
Impact of science, research and innovation activities in the 2007-2013 period
2007 – 2013. gada ES fondu plānošanas perioda darbības programmu “Cilvēkresursi un nodarbinātība” un “Uzņēmējdarbība un inovācijas” ieguldījumu zinātnes, pētniecības un inovāciju atbalstam izvērtējums”
OP Human resources and employment 2007-2013; OP Entrepreneurship and Innovation 2007-2013
The evaluation assesses the impact of R&I activities carried out during the 2007-2013 period on a selection of 21 research institutes.
To assess the impact of science and R&I related activities on the operation and sustainability of 21 research institutes selected for study.
Desk and literature analysis;interviews with agencies, ministries, policy makers and beneficiaries;survey of scientific institutes;statistical data analysis;case studies.
Most key indicators of the science and R&I system have improved significantly.The volume of publications has substantially increased and research cooperation with industry has improved in terms of both volume and quality.The mobility of researchers has increased and there was generally a more active participation in international projects and conferences.The research community currently lacks new young entrants, which are also needed to counter the effects of aging personnel.Despite the improvement of many key indicators, the performance at institutional level is quite mixed.
Some limits in the availability of data are reported.
There is a need to improve job opportunities for doctoral students and young scientists in science and industry.The attraction of research should be promoted to encourage international cooperation and cooperation with industry.The organisation of doctoral support programmes needs to be improved.The reliability of monitoring data needs to be assessed on a regular basis.
Evaluation of support for social work for young people under the Mecklenburg-Vorpommern ESF OP, 2014-2020
Bewertung der Förderung der Jugend- und Schulsozialarbeit
The evaluation examines the effectiveness and impact of measures on social inclusion and school inclusion under the OP.
To assess the impact of measures supporting social work for disadvantaged young people and social inclusion in schools in Mecklenburg-Vorpommern.
Monitoring data.Official statistics.Interviews with support providers and beneficiaries.Surveys of organisations involved.Case studies.
Social work helped to develop the personality of young people, career orientation, and cooperation with school management and other partners, which in turn helped to improve the school climate, so reducing absenteeism and drop-outs.The main ways that helped produce the observed results are the provision of demand-based support, integration of social workers into school operations, needs-oriented network building and training of teachers.Results are affected by external factors, such as spatial distances, the demographic composition of pupils, and the socioeconomic context.
It is difficult to isolate the effects of interventions from those of other factors, so findings should be considered as indicating the contribution of interventions to the results observed.
Increase cooperation between youth social work and school social work and between networks.Ensure systematic and continuing training of professionals in youth and school social work.Ensure transparency on ESF and national and local regulations.
Evaluation of PA1 “Welcoming new populations” of the Interregional Massif Central ERDF OP, 2007-2013
Evaluation des programmes Massif central sur la thématique « accueil »
Interregional OP Massif Central ERDF 2007-2013
Pierre Chouipe Consultant, pierre@chouipe-consultant.com
This evaluation examines from a qualitative angle whether the “Welcoming new populations” project of the Interregional Massif Central ERDF OP successfully encouraged the structuring of integrated local reception policies.
To examine the impact of the “Welcoming new populations” projects on the development of an integrated local reception policy.
Analysis of monitoring data.Document analysis.43 interviews with decision-makers, operators as well as territories not involved in the project.Online survey of project holders (50% response rate, n=47).8 territorial case studies (interviews with elected officials, partners and final beneficiaries).4 meetings in different territories to present the conclusions and suggestions.
Many projects encountered difficulties during their implementation, as the interpretation of the meaning of “welcoming” is different among stakeholders.The main value of the project is the collaboration between actors and the exchange of experience and good practice. However, the dissemination of experience is limited to development workers and does not involve elected representatives and thematic partners.Furthermore, the transfer of experience to regions not involved in the project is rare.The existence of a regional reception policy is not a prerequisite for the implementation of a local reception policy and is not sufficient, alone, to generate one. However, it can help ensure the long-term viability of these schemes after the funding.
Adjust the concepts and methods of the project to generate interest in all the partners and all the regions (from the level of the village to the Region, in all 6 Regions).Review the organization of the policy’s governance and partnership, both vertical and horizontal, political and technical (thematic partners), to capitalize experiences and organize the financing of investments accompanying structuring actions.
Thematic evaluation of TO10 actions funded under 3 OPs (Research Development and Education OP, Prague OP, and Integrated Regional OP), 2014-2020, in the Czech Republic
Výsledková tematická evaluace Dohody o partnerství pro období 2014-2020: TC10 Investice do vzdělávání, odborného vzdělávání, včetně odborné přípravy, pro získání dovedností a do celoživotního učení. Závěrečná zpráva.
OP Research, Development and Education 2014-2020; OP Prague - Growth Pole 2014-2020
Ernst & Young, s.r.o., Naviga Advisory and Evaluation, Czech Technical University in Prague
https://www.ey.com/en_cz; https://www.navigae.cz; https://www.cvut.cz
The report assesses the results achieved by projects funded under TO10 in 3 OPs. Data used go up to end-2022.
To examine the effectiveness and sustainability of the supported projects up to end 2022 and to monitor the effects of cohesion policy on socio-economic indicators.
- Analysis of monitoring and statistical data.- Analysis of administrative data.- Interviews.- Case-studies.- Text mining.- Factor analysis.- Theory of change.- Regression analysis (OLS linear model).- Counterfactual analysis (CR-SEQDD).- Spatial analysis.- Workshops.
- Support from ESI Funds is an important factor in developing the quality of education facilities.- Support helped to introduce quality assessment procedures in universities.- Between 2015 and 2022, ESI Funds accounted for 15-25% of investment in infrastructure and equipment in regional education (and up to 30% for tertiary education).- ESI Funds had a statistically significant positive effect on the share of children aged 3-4 enrolled in kindergarten.- However, ESI Funds are not a determining factor explaining the increase in the share of children with disabilities educated in regular classes.- Similarly, EU support had no immediate effect on improving children’s competences, the main determining factor being their socio-economic and cultural background. The effects of support will only become apparent in the long term.- Means have been developed to reflect labour market needs in the education system, but in practice, changes remain limited (no increased interest in studying technical and natural sciences and no partnership established with employers).- ESI Funds support the development of new means but their transfer to the standard education system remains a challenge.
Very short time series (covering only two academic years) for the cross-regional sequential difference-in-difference analysis of the causal relationship between the support for the development of education infrastructure and the development of professional and transferable competences.
- Provide more support for building childcare infrastructure for children under 3.- Strengthen the systematisation of results and their introduction into the education system.- Involve all schools (including the less active ones) in the dissemination of results and innovations.- Develop partnerships with employers to adapt education in secondary schools effectively to labour market needs.- Maintain support for the internationalisation of universities.
Evaluation of measures for general, vocational and lifelong education under the Opolskie OP, 2014-2020
Ocena wsparcia w zakresie edukacji w ramach Regionalnego Programu Województwa Opolskiego 2014-2020 oraz analiza aspiracji edukacyjno-zawodowych uczniów szkół ponadpodstawowych i osób dorosłych
A.Śnieżek, P. Penszko, W. Pieniążek, M. Cichowicz-Major, J. Piekutowski, S. Pałka, C. Przybył, A. Grajek, B. Tomczak, M. Marciniak
The evaluation examines the implementation of the measures under the OP, in particular in relation to the consequences of COVID-19 and the social and professional aspirations of secondary school students and adults.
To assess the usefulness and effectiveness of the support for education, in terms of meeting the needs of recipients and current education challenges and the impact on improving the situation of project participants.
Analysis of monitoring data and official statistics.11 focus group interviewsSurvey of 1,418 people.3 in-depth interviews.1 expert panel.
Almost all of the high school students surveyed plan to go to university, more than half of them outside the region.Schools are not sufficiently prepared to admit students from Ukraine.The effectiveness of Sub-measure 9.1.5 “Scholarship aid programmes” is high, as scholarships have helped reduce the difficulties caused by the transition to distance learning.Resources allocated by the MA are not sufficient to meet the needs of the schools, especially for equipment.Support for continuing vocational education is reported to be highly usefulness and innovative and is suitably tailored to local needs.
Include Opolskie colleges and universities in education projects in 2021-2027, to prevent education outward migration.Support the mental and physical health of students after the period of remote learning.Support systematic work with Ukrainian students to cooperate with NGOs and to integrate Ukrainian children and young people into the local environment.Differentiate the value of scholarships on the basis of the stage of education and student achievements.Continue support for infrastructure, especially as regards vocational training.
Evaluation of the indicators system in the Veneto ESF OP, 2014-2020
The evaluation examines the significance, relevance and measurability of output and result indicators adopted by the Veneto ESF OP.
To assess the significance and relevance of the indicators used by the OP, to highlight any information deficiencies that should be filled by indicators and make suggestions for improving the indicator system.
Desk analysis of EU and regional documents, reports and guidelines.Analysis of monitoring data.
The common output indicators are in general relevant in respect of the OP strategy, but there are exceptions.Under IP 8vii “Modernisation of labour market institutions”, most of the selected performance indicators, have a low level of consistency with the logic of the measures implemented.In the case of IP 8.vii, it will be difficult to reach targets.The target for IP 8.ii is overestimated.The result indicators CR05 and CR09 are consistent with the objective of IP 9v but do not enable the results of measures for strengthening entrepreneurship and the social economy to be fully measured.
Integrate the common output indicator (CO22), with two other indicators that refer to the number of public authorities and services supported.Remove the specific output indicator PO05 which is difficult to assess.Revise the methods for calculating the result indicators CR05 and CR09.
Evaluation of the benefits of international mobility projects financed under the Employment OP, 2014-2020, in the Czech Republic
Kvalitativní šetření přínosů projektů Mezinárodní mobility
Geologická 2, 152 00 Praha 5, Czech Republic
The study focuses on four international mobility projects (out of a total of 14) targeted at young disadvantaged and which are funded under the OP.Data used cover the period February 2022 – March 2023.
To assess the benefits of projects aimed at increasing international mobility and social inclusion of disadvantaged young people aged 18-30 (disabled, LTU, inactive, low skilled).
25 interviews with participants. 12 interviews with project managers. Focus groups.
There are various objective (health situation, education level) and subjective (motivation, self-confidence, distance from family) obstacles to participation in the projects. Recruitment of participants was negatively affected by the existence of competing projects of labour offices for the same target group. The importance of quality and experienced mentors was key for project success. After returning from their internship abroad, the main issue was a lack of motivation. The main outcome of the projects is an increase in self-confidence and autonomy of participants.
The results of the study are purely based on the subjective assessment of participants and project managers.
Release the eligibility conditions (and in particular the duration of registration at the labour office). Ensure the presence of competent coaches and mentors with psychological or psychotherapeutic experience.
Second evaluation report of Specific Objective A1.1 under the Baden-Württemberg ESF OP, 2014-2020
Zweiter Evaluationsbericht zum spezifischen Ziel A.1.1: Teilnehmer*innenperspektive und Wirkungsanalyse
ISG Institut für Sozialforschung und Gesellschaftspolitik GmbH, Weinsbergstr. 190, 50825 Köln, scheller@isg-institut.de
This second evaluation report on SO A1.1 of the OP concerns the effects of measures to integrate unemployed, especially long-term unemployed and other vulnerable people.
To examine participants’ judgements on procedural issues relating to measures for integrating the unemployed and vulnerable groups into work and to estimate their effects.
Analysis of monitoring data.Survey of participants between 2015 and 2017 (290 out of 1700 unemployed, and 55 out of 289 employed).Counterfactual analysis using administrative data from the Federal Employment Agency (Integrated employment biographies).
By the end of 2019 5,312 people had received support, 3,836 of them unemployed (80% of the target) and 70% women, reflecting projects being targeted at single parents.43% obtained a job after participation, 48% for men and 41% for women.According to the survey, Job Centres played a significant role in supporting people to participate in projects; cooperation with companies was problematic and respondents consider their income insufficient even after participation, though report that their social life and job prospects had improved.After 15 months, the employment rate of people supported is 10 pp higher than for the control group, while the probability of remaining on unemployment benefit (SGB II) is lower. Effects are also positive for a job lasting 6 consecutive months.The impact on the employment rate is positive for both men and women, but higher for men.
The sample of people interviewed is not fully representative of the characteristics of participants, though differences are not large.
Differentiate cooperation with companies by target group.Monitor and support people for longer, beyond the first transition into work.
Evaluation report on the implementation of the Liguria ESF OP, 2014-2020
II report dell’analisi valutativa dei risultati a metà periodo rispetto a ciascuna PdI ed OS
Silvio Casucci; Eugenio Corazza; Aldo Curreli; Debora Barucca; Enrico D’Angelillo; Gianluca Vari
The evaluation examines the implementation of the OP at the end of 2019.
To assess the strategy, the implementation of the OP and the results achieved up to the end of 2019.
Desk analysis of relevant documents.Statistical data.Monitoring data.
The socio-economic context has not changed dramatically from that when the Programme was formulated, so the strategy remains valid, but COVID-19 could necessitate future adjustments.Around 60% of the budget had been allocated and around a third of funding had been paid at the end of 2019.There is mostly good progress toward the achievement of output indicators, though there are some delays as regards measures for young people.Under the SO of increasing female employment, almost no woman receiving vouchers found a job after 6 months.Under the SO of promoting the integration of the unemployed, 33% of participants were employed after 6 months, though under the SO for young people, the employment rate was 67% after 6 months; but only 18% of vulnerable participants found a job after 6 months.
No limitations are indicated in the evaluation.
Consider introducing innovative measures.Simplify procedures to improve he implementation.
Impact evaluation of measures for the prevention of natural risks under the Corse OP, 2014-2020
Evaluations d’impact du programme opérationnel FEDER-FSE Corse 2014-2020 Evaluation relative au volet Risques naturels (Pi 05b)
http://www.teriteo.fr; contact@teriteo.fr
The report analyses the impact of measures to prevent natural risks financed under IP 5.2 of the OP up to 24 March 2021.
To assess the level of implementation of projects and the main effects up to March 2021 and to make recommendations for 2021-2027.
Documentation analysis.Interviews with beneficiaries and other stakeholders.3 case studies.Analysis of monitoring and cartographic data.
8 projects are financed under IP 5.2, with 35% of total funds committed (EUR 4.2 million) and 40% of certified expenditure (EUR 7.9 million), but indicators are only 26% of 2023 targets.5 projects are managed by the municipality of Ajaccio, 2 by the Corsican Environment Office and one by the Marine Hydrographic and Oceanographic Service.Projects financed contribute to preventing natural risks, but impact is very limited as only 8 measures were being financed at the time of the evaluation (up to March 2021). Protection against floods is still low and the Programme d’Action de Prévention des Inondation (PAPI) has been adopted only in Ajaccio.Projects have helped to increase knowledge of people about the risks relating to asbestos, floods and coastal erosion.Transfer of competence in managing of aquatic environments and flood prevention (GEMAPI) from municipalities to inter-municipalities has caused delays in implementing projects.
Support the adoption of the PAPI in all areas at risk of flooding.Identify the issues causing delays in implementing projects.
Evaluation of support for unemployed over 50 under the Employment OP, 2014-2020, in the Czech Republic
Vyhodnocení podpory nezaměstnaných osob 50+ podpořených v soutěžních a regionálních individuálních projektech
Simona Lipková, Kristýna Janhubová, Martin Dytrych
Ministry of Labour and Social Affairs, evaluace@mpsv.cz
The evaluation examines competitive projects to help those over 50 to return to the labour market and regional projects to solve regional labour market issues.
To assess the socio-demographic characteristics of participants in competitive projects to help those over 50 to return to the labour market and regional projects to tackle regional labour market issues, the scope of support, the characteristics of those supported, and their economic activity.
More women than men participated in projects, the average age of participants being 55.More people with disabilities participated in regional projects (42%) than in competitive projects (8%).Competitive projects more often provide a combination of different types of support than regional ones.Under the competitive projects, the share of those employed rose from 3% at the beginning of support to 44% 18 months after.Under the regional projects, the share of those employed increased from 1% at the start to 61% 18 months after.
The analysis was based on selected projects and so the data used does not include all people supported. Some projects are still being implemented, so their results cannot be considered as final. The data on competitive projects is included in the system directly by the beneficiary, which may cause some administrative errors. The analysis is descriptive only and conclusions on causal relationships cannot be drawn.
Evaluation of the results of measures on SMEs competitiveness under the Enterprise & Innovation for Competitiveness OP, 2014-2020, in the Czech Republic
Výsledková tematická evaluace Dohody o partnerství pro období 2014-2020: TC3: Zvýšení konkurenceschopnosti malých a středních podniků. Závěrečná zpráva.
Ernst & Young, Naviga Advisory and Evaluation, Czech Technical University in Prague
This evaluation measures the effectiveness, impact and sustainability of support for the competitiveness of SMEs, as well as its contribution to socio-economic indicators. Monitoring data are up to April 2023.
To assess the results and impact of measures to improve the competitiveness of SMEs, and the critical factors encountered. To suggest ways to tackle sustainability-related risks of the projects and to assess the effects of the measures on national socio-economic indicators.
Counterfactual analysis (CR-SEQQDD). Linear regression. Territorial analysis using GIS tools Academic literature review. Theory of Change testing. 3-member expert panel. Analysis of official statistics and monitoring data. Interviews and surveys.
The measures helped increase the availability of business infrastructure for SMEs and led to increased use of financial instruments. SMEs improved their way of presenting activities at trade fairs and introducing products into new markets. The projects did not have a clear effect on strengthening knowledge-intensive entrepreneurship nor on improving the provision of advisory services or on the image of business in society.
Part of the budget for these measures was also used for other objectives. Several project indicators were set by SMEs, which might have led to inflated estimates of effectiveness. Given a limited data availability, growth of sales was often taken as a proxy for assessing impact.
Strengthen the collection of data on the economic indicators of SMEs for more precise targeting of ESI Funds programmes. Reflect the current offer of consultancy services on the market, and absorption capacity and demand on the side of potential users in the programme settings. Re-establish the ratio of non-repayable support in favour of financial instruments for projects that have potential. Target at least part of the measures on knowledge-intensive SMEs, including new ones, and adapt the calls accordingly.
Evaluation of the new childcare service (‘Mikrojesle’) financed by the Employment ESF OP, 2014-2020, in the Czech Republic
Závěrečná evaluační zpráva z pilotního ověření nové služby péče o nejmenší děti
Markéta Horpeniaková, Eva Veselková; Evaluation unit, Ministry of Labour and Social Affairs
The evaluation examines micro-crèches (‘mikrojesle’) providing care for young children (from 6 months to 3) in small groups as a means of improving the availability of affordable and high-quality childcare and of facilitating reconciliation of work with family life.
To assess the success of micro-crèches (mikrojesle) in making it easier for parents to reconcile employment with caring for children under the Employment OP.
Monitoring data.Survey of 72 beneficiaries (response rate 97%).Survey of 508 parents (response rate 57%).In-depth interviews of beneficiaries.
57% of parents chose mikrojesle because they provide better quality care than other larger facilities, in being more suitable for young children.80% of beneficiaries interviewed plan to continue operating the mikrojesle after the completion of the project.Four months after placing their child in mikrojesle, the number of parents in employment increased by over 20% and the share working less than 25 hours a week declined by 14 percentage points.Almost all mothers report that in the absence of mikrojesle, they would have to care for their children themselves.Almost half of parents thought there should be a guaranteed place for children from two years old.The average cost per child is higher for mikrojesle (EUR 630) than for similar facilities with a capacity of up to 6 children (EUR 510).
To increase the number of mikrojesle (or at least maintain the existing ones).To continue promoting individual care for young children.To set up sustainable funding for mikrojesle for the next period.
Evaluation of the implementation of Specific Objective 8.6.1. under the EU Structural Funds Investment OP, 2014-2020
2014-2020 metų Europos Sąjungos fondų investicijų veiksmų programos 8.6.1. uždavinio „Pagerinti vietines įsidarbinimo galimybes ir didinti bendruomenių socialinę integraciją, išnaudojant vietos bendruomenių, verslo ir vietos valdžios ryšius“ įgyvendinimo
Didžioji g. 25-6, LT- 01128, Vilnius Vilnius, Tel: 852154075, www.bgiconsulting.ltinfo@bgiconsulting.lt, www.bgiconsulting.lt
The evaluation examines the implementation of the Specific Objective to “Improve local employability and enhance social integration of communities, by making use of relationship among local communities, businesses and local governments” under the EU Structural Funds Investment OP, 2014-2020.
To examine the implementation of the measures for local communities, assess the institutional capacity of Local Action Groups (LAGs), identify the areas to be improved and make recommendations on the implementation of local development strategies.
Analysis of indicators;analysis of public statistics;surveys of local institutions;interviews with representatives from national institutions, LAGs and project promoters;desk analysis of documentation;field research and case studies.
23 local development strategies of 39 approved have been initiated;urban LAGs, responsible for implementing the local development strategies, are performing their tasks satisfactorily;the organisation of LAG is made difficult by insufficient funding to ensure activities are undertaken well;two bodies are involved in the selection and implementation of local development projects (LAGs and the ESF Agency), which increases the administrative burden on applicants and project promoters;some local development projects are focused on services already available in urban areas, which limits the opportunity to provide new services;local development projects help to involve various groups in voluntary activities;since implementation is still low, it is not possible to assess the contribution of local development projects to the achievement of local development strategies.
There is a need to:facilitate the implementation of existing local development projects, by providing detailed description of the selection procedures, adopting clearer selection criteria and reducing the administrative burden on applicants;in the longer-term, to involve more members of target groups in local development strategies, to prepare a common protocol on the future prospects of LAGs, to reduce their number and to identify larger territorial units to implement the CLLD approach.
Evaluation of measure 3.2 "Labour market services to ensure better opportunities for participation in employment" under the OP for Cohesion Policy Funding, 2014-2020, in Estonia
Meetme 3.2 „Tööturuteenused tagamaks paremaid võimalusi hõives osalemiseks“ tulemuspõhise vooru hindamine
Tauno Õunapuu, Merle Raun, Delis Lauringson, Laura Argus
The evaluation examines the impact of measures, financed under the OP from 2020 to 2022, targeting 55-year-old NEETs (not in employment, education or training) to increase their employability.
To examine the effectiveness, positive aspects and potential bottlenecks of measures aiming to increase the employability of 55-year-old NEETs.
Introductory interviews with the representatives of the Ministry of Social Affairs (SOM) and the representative of the State Shared Service (RTK).Analysis of monitoring data and official statistics.Analysis of project documents.In-depth interviews with 18 beneficiaries.In-depth interviews with representatives of SOM.Focus groups and validation seminar with representatives of RTK.
The current regulation is too complex and open to multiple interpretations which generates uncertainty and confusion.The results-based funding model makes it possible to achieve high-quality results by giving beneficiaries greater freedom to allocate costs in a targeted and flexible manner.However, beneficiaries were not able to check whether project participants met the requirements for accessing and whether participants were employed after the end of the project.Some projects took place in areas where these measures had rarely been organised, but not all regions with higher-than-average unemployment were covered.446 people participated in the activities. 86% of them completed the planned activities.The activities had a positive impact on the participants’ self-development and self-confidence, communication and digital skills, and on the knowledge of the law, though they do not seem to be sufficient to overcome the main barriers to employment (lack of jobs, discrimination of older people, etc.).Training and counselling on psychological issues, creating new acquaintances were considered valuable, but online trainings generated low interest in participants.
Simplify the structure and wording of the regulation and funding procedure.Increase communication regarding the entire process in the application and early stages of the project.Continue with merit-based application rounds.Use a simpler funding scheme (initial flat-rate payment of 40%, 30% during implementation and 30% upon completion).
Ex-post evaluation of the measure supporting income under the Puglia OP, 2014-2020
Rapporto di Valutazione ex post 2014-2020 "Inclusione sociale attiva"
Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Puglia
The evaluation examines the implementation and impact of the “Dignity income” measure in Puglia, which aims to combat poverty and promote social and labour inclusion for vulnerable people. Data used go up to 2023.
To determine how effectively the measure has reduced poverty and social exclusion in Puglia, whether the instrument facilitated the integration of beneficiaries into the labour market, and improved their social inclusion through training and community work.
Document analysis.Analysis of statistical and monitoring data.Survey of 45 territorial units (ATS) responsible for implementation (71% response rate).Survey of 17,150 beneficiaries for the years 2018, 2019, 2020 (519 responses-3% response rate).14 in-depth interviews with representatives of territorial units.6 focus groups with public and private stakeholders.
The measure significantly improved the psychosocial well-being of beneficiaries, with many reporting reduced anxiety related to financial insecurity and an overall improvement in quality of life.Women benefited more than men, as 73% of female beneficiaries reported improvements in well-being, compared to 61% of male beneficiaries.The outcomes of the measure in terms of labour market integration are limited: only 7% of beneficiaries who participated in training or internships found stable jobs; the training programmes offered were often misaligned with beneficiaries' skills and the needs of the local labour market.Impact on poverty reduction among children and school dropout rates is modest, particularly in poorer areas like Taranto and Foggia.Significant local disparities emerged, with urban areas showing a higher number of beneficiaries and better access compared to rural services.
Data gaps due to missing responses from some territorial units partially affected the information collected through the surveys.
Enhance training quality and relevance by making training programmes more targeted and aligned with the specific needs of the local labour market.Introduce incentives for businesses to hire beneficiaries after internships or training programmes.Strengthen the psychosocial and healthcare support for beneficiaries, especially those facing mental health problems.Reevaluate the access criteria based on the ISEE indicator, which may not always reflect the true economic vulnerability of families (e.g., those with significant assets but low liquidity).Enhance ongoing monitoring of beneficiaries even after the end of the support period.
Results of environmental projects implementation in the 5 priority of the OP I and E
Efekty realizacji projektów przyrodniczych w V priorytecie POIiŚ
EU-CONSULT Sp. z o.o., ul. Wały Piastowskie 1, 80-855, Gdańsk, www.euconsult.org
Evaluation of the support for vocational education and lifelong learning under the Podkarpackie OP, 2014-2020
Ocena wpływu wsparcia w zakresie kształcenia zawodowego i ustawicznego
Eu-Consult Sp. z o.o., ul. Toruńska 18C, lokal D; 80-747 Gdańsk; www.eu-consult.pl
The evaluation examines the effects of support for vocational education and lifelong learning under the OP.
To assess the impact of support for VET and lifelong learning on the quality of education and competences in the region (PA 9), the regional labour market (PA 7) and spatial and social cohesion (PA 6).
Desk research.6 interviews.Surveys with 201 beneficiaries, 1415 project participants, and 204 potential beneficiaries.16 case studies.Workshop with MA and key stakeholders.Delphi panel.
Co-financing for equipping workshops in schools and vocational training institutions had the largest effect on improving vocational training conditions.Projects on continuing education helped to improve employee and potential employee qualifications and competences, and to meet employer needs for qualified, competent personnel.
Implement projects to increase the quality of teaching staff in vocational education.Implement projects for improving competences in foreign languages and digital skills, particularly among people over 50.Implement vocational courses, including support for occupations with shortages.Support continuing education in the future programming period, especially in relation to the qualifications most needed on the labour market.
Impact evaluation of investment in education and vocational training infrastructure (KAI 3.4) in 2007-2013
Evaluarea de impact a intervenţiilor POR 2007-2013DMI 3.4. Reabilitarea modernizarea dezvoltare si echiparea infrastructurii educationale preuniversitare, universitaer si a infrastructurii pentru formarea profesionala continua
Liliana Lucaciu, Monica Roman, Stefana Buhaescu-Ciuca, Alina Bosoi, Serban Totescu, Elena Botezatu
Civitta, Archidata, NTSN Connect, Structural Consulting Group
This is an ex-post evaluation of the impact of the interventions financed by the Regional OP 2007-2013 to improve the quality of education infrastructure.
To assess the net effects of the interventions and to identify the factors responsible for them as well as those which produced the best results.
desk analysis of relevant documents;interviews with MA, Ministry of National Education, the Romanian agency for quality assurance in pre-university education (ARACIP), directors and experts of RDAs;analysis of monitoring data;7 field surveys;focus groups in each region and nationally with stakeholders;counterfactual analysis;case studies of different research areas;
Over 220,000 students benefited from the investment in education infrastructure through the more comfortable and stimulating learning environment created.Only one project was funded on training infrastructure as against 35 planned.17% more buildings were totally or partially rehabilitated in schools financed than in those not.The energy efficiency of the buildings concerned was increased.It was difficult to achieve complementarity of the OP investment in educational infrastructure with projects funded by the Human resources OP.Investment in university campuses improved access to education, offering students quality housing at affordable prices.
The quality of some data was limited and there were inconsistencies between the different databases.
continue to invest in education infrastructure with a more strategic approach to prioritise investment at the national level;support interventions which most affect academic performance;develop guides for creating a modern learning environment;increase the number of accommodation places;support schools in rural areas with facilities for a modern learning environment;promote examples of good practice;develop a collaboration protocol between the MA and the Ministry of Education for preparing data necessary for impact assessment and avoid collecting data directly from beneficiaries.
Evaluation of measures for social inclusion under four OPs of the Partnership Agreement, 2014-2020, in the Czech Republic. Final report
Výsledková tematická evaluace Dohody o partnerství pro období 2014-2020: TC9: Podpora sociálního začleňování a boj proti chudobě a diskriminaci. Závěrečná zpráva.
OP Research, Development and Education 2014-2020; OP Employment 2014-2020; OP Prague - Growth Pole 2014-2020; Integrated Regional OP 2014-2020
The evaluation examines measures under TO 9 - Promoting social inclusion and the fight against poverty and discrimination - under 4 OPs. Data used go up to 31 December 2022.
To assess the main results and impact of measures on social inclusion and fight against poverty and discrimination. To assess whether and to what extent they have contributed to the socio-economic trends of the country.
- Analysis of relevant documents.- Analysis of monitoring and statistical data.- 17 Interviews with beneficiaries and external experts.- Linear regression.- Counterfactual impact evaluation (CR-SEQDD approach).- Experts’ assessment.
- Sequential analysis shows that there is a statistically significant relationship between the decreasing unemployment rate of vulnerable target groups and the funding from ESI Funds.- The Social Exclusion Index for the Czech Republic has fallen, both at regional and national level.- In the most deprived areas, social inclusion activities have been better coordinated and their relevance with respect to the needs of target groups has increased.- The absence of a Social Business Act because of a lack of legal framework is a problem for the development of social enterprises.- ESI Funds helped develop specialised healthcare services and make them available to vulnerable people.
Data from the monitoring system on social entrepreneurship or social housing are missing and impact cannot be assessed.
- Ensure that measures address the most vulnerable target groups.- Strengthen the management of funds.- Set a legal framework for social housing and social entrepreneurship.
Evaluation of the 'Internationalisation of Vocational Training' measure supported by the Baden-Württemberg ESF OP, 2014-2020
Evaluationsbericht: Förderlinie "Internationalsierung der beruflichen Ausbildung"
Georg Kalvelage and Friedrich Scheller - ISG
ISG Institut für Sozialforschung und Gesellschaftspolitik GmbH, E-Mail: kalvelage@isg-institut.de
The evaluation examines the measure, which finances traineeships abroad and advisory services for dual trainees.
To assess the results of traineeships abroad and advisory services for dual-system trainees financed by the measure.
Literature and project documentation analysis.Monitoring data.Survey of participants (51% response rate, out of a sample of 589).7 Interviews with training companies.
Support in the application and placement of internship abroad is positively assessed by all stakeholders.Measures allow trainees from smaller companies and less internationally oriented areas to work abroad.Participants have a good initial level of training, are not disadvantaged and are self-motivated. The overrepresentation of women is a positive aspect as it compensates for the gender disadvantage in vocational training.Participants report improved self-confidence and reduced language barriers.Companies supported the measure to provide additional opportunities for trainees and to increase the quality of the training provided.Overall, the measure has helped to strengthen the competences of trainees, improve their qualifications and increase the quality of vocational training.
Involve regional business associations and chambers of commerce more.Pay more attention to small companies.Increase complementarity between the ESF and other sources of funding.
Evaluation of the Sachsen ESF OP, 2014-2020 - Final Report 2018
Endbericht 2018 - (Wirkungs)evaluierung 2018 und Fachspezifische Evaluierungsfragen
Moysies & Partner, IT- und Managementberatung mbB, Mittelweg 56a, 20149 Hamburg , tel.: +49 (040) 211 07 61-13, e-Mail: andreas.stahn@moysies.de
The evaluation presents the results of third round of an on-going evaluation of the OP.
To assess the effectiveness and efficiency of interventions financed by the OP and the impact of 15 labour market measures.
Analysis of monitoring data and statistical data (e.g. JobPerspektive Sachsen).Interviews with project managers and stakeholders.Surveys of beneficiaries on several measures, i.e. Start-up consultancy (300 former participants, response rate 61%), Grants for technology-driven start-up (300 participants), InnoExperts (300 participants), Junior research groups, Research groups, Integrated training (300 supported SMEs).Evidence from previous evaluations.Case studies (two on ‘projects for production schools’ and two on ‘Employment opportunities for disadvantaged young people’).
The OP implementation is proceeding positively and targets set are expected to be met or exceeded.Horizontal principles are satisfactorily supported by the OP.On IP8.iii, the quality of consultancy received is assessed positively. All technology-driven start-ups surveyed opened a business within six months after receiving the grant, support mainly being used to develop products or services.On IP8.iii, the number of companies participating in the InnoExpert measure (grants to employ people on technology-based projects or innovation management) is as expected, though the involvement of those of 55 and over is lower than expected. 75% of those recruited were still employed in the companies surveyed after support ended.On IP9.i, the measures had a limited effect in helping people find employment mainly because of a lack of work experience of participants and an insufficient timeframe to reduce placement barriers.On IP9.i, projects ‘for production schools’ have helped to stabilise participants.On IP9.i, the multiple problems of many disadvantaged young people make it difficult to get them into training or employment.On IP10.iv, companies participating in the ‘Integrated training’ measure were mostly very satisfied with it, the training being carried out in most cases in collaboration with education providers.
Improve the selection and monitoring of start-up projects.Merge the measure ‘Improve employability of long-term unemployed’ with ‘Individual support to enter the labour market’ and redesign it to increase the effectiveness of projects.Improve projects ‘for production schools’ by expanding the types of project supported to include e.g. socio-education support and by offering individual support based on needs.Provide a longer chain of support for disadvantaged young people.Reconsider the funding periods of the measure ‘Junior research Groups’ to be more in line with doctoral studies.
Evaluation of the communication strategy of EPANEK 2014-2020 and the degree of satisfaction of its potential beneficiaries and beneficiaries
Αξιολόγηση της στρατηγικής επικοινωνίας του ΕΠΑνΕK (ΕΣΠΑ) 2014-2020 και του βαθμού ικανοποίησης των δυνητικών δικαιούχων και ωφελούμενων του
OP Competitiveness, entrepreneurship and innovation 2014-2020
Prorata Public Opinion Surveys and Communications Applications s.a.
Palaiologou Benizelou 7, Athens, 10556, tel: +30 211 103 1900https://prorata.gr/
Ex-ante study of financial instruments under the Provence-Alpes-Côte d'Azur and Massif des Alpes programme, 2021-2027
Evaluation ex-ante à la mise en place d’instruments financiers dans le cadre du Programme FEDER-FSE+-FTJ 2021-2027 de la Région Provence-Alpes-Côte d’Azur et Massif des Alpes
The report examines the need and potential use of financial instruments (FIs) under the programme. Data go up to Spring 2020.
To define a strategy for using FIs under the future programme.
Analysis of regional strategic and operational documents.Analysis of national statistics.20 interviews.3 thematic workshops with relevant stakeholder.Interviews with regional financial intermediaries (RSI, Bpifrance, France Active...).
A need for FIs has been identified for business development, innovation and renewable resources, but not for urban development, energy efficiency and vocational training measures.FIs for SMEs development are in place (RSI, South Guarantee) and appreciated but there are shortages for SME recovery from the crisis, takeovers and mergers, particularly in the low-innovative sectors.Companies with financial leverage (equity debt) have steadily declined since 2014, and their number is only a fifth of the national average.Net business creation has risen since 2014, because of an increasing number of start-ups and a decline in closures.Current developments in energy production do not make it possible to achieve the ambitious regional targets for 2030. The development of some strategic renewable sources is stagnating because of difficulties of social acceptability.
Use FIs more for financing company investment and energy transition.Mobilise the necessary resources for facilitating and monitoring the use of FIs.Follow three principles in FI strategy: alignment of FIs with strategic priorities; complementarity of regional investment with existing means of financing tools at national level (non-duplication); and ensuring the projects funded generate evident effects.Focus ESI funds on a limited number of measures with critical mass and high added value.
On the track of impact – evaluation of the Sustainable Growth and Jobs OP
Vaikutusten jäljillä. Kestävää kasvua ja työtä rakennerahasto-ohjelman vaikuttavuuden arviointi
MDI: Tommi Ranta, Elina Auri, Benjamin Heikkinen, Minna Mayer, Juho Nyman - Kuntoutussäätiö: Jaakko Harkko and Sari Pitkänen - Spatia: Timo Hirvonen, Arja Jolkkonen, Petri Kahila, Arja Kurvinen, Teemu Makkone - TK-Eval: Keimo Sillanpää, Tommi Ålander
This is an impact evaluation of the Sustainable Growth and Jobs OP 2014-2020 financed by both the ERDF and the ESF, supporting projects in 5 areas: SME competitiveness; smart specialisation; employment; education and training and social inclusion.
To provide an overview of the effectiveness and impact of the OP.
Counterfactual analysis;desk analysis;surveys of ESF beneficiaries, businesses and representatives of chambers of commerce;analysis of monitoring data;interviews and focus groups;case studies.
The number of new companies created by the support is modest compared to expectations.The growth of funded companies was good, but counterfactual analysis does not provided evidence of the effectiveness of support.Only 25% of participants are satisfied with the reduction of carbon emissions and efficient use of energy.New networks between research centres and businesses, important for territorial development were created.The target of at least 30% of beneficiaries being young seems difficult to achieve.38% of recipients were in employment 6 months after the completion of programmes.The education projects funded have helped participants acquire new skills (particularly digital ones).Interventions helped to reduce early school leaving.Social inclusion projects have increased job opportunities for disadvantaged groups.
ESF: The results may be uncertain because of lack of data for some beneficiaries.ERDF: The size of control group was not large enough to compare with the treated group.
There is a need to:increase the amount of investment going on R&D;make projects more consistent with the needs of SMEs;focus social inclusion measures on people of middle age as well as on younger and older groups;repeat counterfactual analysis in the future.
Strategic evaluation of information and promotion activities implemented within OP HC
Ewaluacja strategiczna działań informacyjNo-promocyjnych realizowanych w ramach PO KL
Agata Rauzer, Andrzej Gołoś, Anna Domaradzka-Widła, Bartosz Ledzion (red.), Katarzyna Seferyńska, Paweł Śliwowski
Evaluation of the effects of Cohesion Policy in Poland, in 2014-2020, on the development of cities and their functional areas
Wpływ polityki spójności 2014-2020 na rozwój miast i ich obszarów funkcjonalnych
All OPs 2014-2020 except OP Technical Assistance
Przemysław Bruhn, Michał Cichocki, Wojciech Dawid, Natalia Dziarmakowska, Violetta Drabik-Franiewska, Cezary Gołębiowski, Anna Górecka Ojdana, Aleksandra Jadach-Sepioło, Iwona Kania, Jolanta Kluba, Aleksandra Kowalska, Katarzyna Lipczyńska, Michał Niebylski, Adam Rybkowski, Paulina Sęk, Katarzyna Spadło, Weronika Zych, Edyta Tomczyk
The evaluation assesses how Cohesion Policy 2014-2020 interventions have shaped the socio-economic growth, environmental resilience and quality of the services of Polish cities and their surrounding functional urban areas (FUAs).
To assess the impact of the EU Cohesion Policy 2014–2020 on the development of Polish cities and their functional areas.
Desk analysis of relevant documents.Analysis of monitoring data and official statistics.Survey of 562 municipal offices.40 interviews with MA and IBs.Interview with officials of the Urban Policy Department of the Ministry of Development Funds and Regional Policy.7 focus-groups with local governments and NGO representatives.22 case studies of individual cities or functional urban areas.7 expert panels.Expert workshop.1 implementation workshop.
Around 80,000 projects were implemented in cities and/or FUAs. Most of them were in the area of economic development and labour market (31%) and urban mobility (29%).67% of programme indicators met their targets, 13% were exceeded while 20% fell short of them.Projects covering whole functional areas were more cost-efficient than those implemented in single cites, especially in transport, energy and water-sewage infrastructure.86% of municipalities expect benefits to persist and no policy area shows a risk of non-durability of the effects.Interventions under Integrated Territorial Instrument improved social, economic and spatial cohesion by animating inter-municipal cooperation and prioritising mobility, environment and social services.
Missing regional statistics: for the R&D expenditure indicator the Central Statistical Office had no values for 2023 for Lubuskie and Warmińsko-Mazurskie, so those two regions could not be included in the territorial breakdown of results.
Continue and strengthen ITI mechanisms to anchor cooperation at the municipal level.Make greening and universal design mandatory in future urban projects to replace “over-concreting” and improve accessibility.Align different funding streams in each city to avoid fragmentation and maximise synergies.Extend initiatives that build local-authority skills and promote knowledge exchange.Simplify rigid technical criteria and secure follow-up funding for soft/social projects.Embed anti-sprawl measures and stronger spatial planning conditions in new urban investments.
Evaluation of the progress of performance framework indicators under all OPs in Romania – 2014-2020, third report
Evaluarea progresului în îndeplinirea indicatorilor din cadrul de performanță – Tema E. Al treilea raport.
All OPs 2014-2020 except OP Technical Assistance 2014-2020
Laura Trofin, Graham Hay, Irina Lonean, Vasile Alecsandru Strat, Adelina Alexe, Dan Dutu, Jack William Spencer, Robert Petraru, Hakub Zagdanski
QURES Quality Research and Support SRL, Dem. Valea Danului 42/8, Sector 6, București, email: contact@qures.roCambridge Econometrics, Covent Garden, CB1 2HT, Cambridge, e-mail: info@camecon.com
The evaluation examines progress in achieving the Performance Framework targets of the 2014-2020 OPs up to June 2021.
To estimate the probability of OPs’ Performance Framework indicator targets for 2023 being met on the basis of data up to June 2021.
Data from the Unique Information Management System for Structural Instruments (SMIS) and Electronic Monitoring System (eMS).Interviews and focus groups with Managing Authorities.
Despite delays due to COVID-19 most of the PAs of the Regional OP will meet or exceed the 2023 targets.A large number of PAs of the Large Infrastructure OP are at risk of targets not being met.There are no major risks of the targets of the Competitiveness OP not being met, except for some financial indicators.For the Human Capital OP, only PA 3 (TO8) is not at risk of targets not being met and progress of other PAs is slow.For the Administrative Capacity OP (TO11), only the financial indicator targets under PA 2 are at risk of not being met.For the Interreg Romania-Bulgaria OP, targets are likely to be met, despite COVID-19 and delays in implementation and reimbursement procedures.For the Interreg Romania-Hungary OP, financial indicator targets are at risk of not being met.
The forecasting exercise produces estimates only for contracted projects. Estimates cannot be made for all indicators, since some of them have only been implemented recently and few data are so far available.
Accelerate contracting procedures for OPs with targets at risk of not being met.Produce clear estimates of which projects can be implemented by 2023.Provide adequate support to beneficiaries for OPs experiencing reimbursement issues.
Evaluation of the Andalucía ESF OP 2014-2020 for 2019
Informe de Evaluación del Programa Operativo FSE de Andalucía 2014‐2020. Annualidad 2019
Dirección General de Fondos Europeos, Junta de Andalucía
This interim evaluation examines the Andalucía ESF OP 2014-2020, covering employment (TO8), social inclusion (T09) and education (TO10).
To assess the OP strategy, implementation, its contribution to Europe 2020 and to horizontal principles.
Desk analysis of relevant documents and official statistics (EUROSTAT, National Statistics Institute, Institute of statistics and cartography of Andalusia)Analysis of monitoring dataOnline survey of management centres.
The OP is still relevant given regional needs.At the end of 2018, 51% of funding was allocated.7 out of 17 result indicators show positive progress towards targets, but the remaining indicators are below 65% of targets.29% of participants in the "Employe@30+programme" were employed after 6 months.Support led to 3011 becoming self-employed (30% long-term unemployed and 27% unemployed under 30).The OP shows solid progress on gender equality and non-discrimination.
Many measures are planned to be implemented between 2019 and 2023 which could not be covered.
Evaluate the measures launched in the futureConsider reviewing indicators where targets are at risk of not being reached by 2023Reallocate the Performance Reserve to support for sustainability and quality in employment.
Final Evaluation of High-Performance Computing Wales 2007-2013
Final Evaluation of High performance Computing Wales
The evaluation assesses the High Performance Computing (HPC) Wales project 2007-2013 (a collaboration between 6 Welsh universities aimed at providing access to HPC facilities and training to businesses, researchers and students).
The objective of the evaluation is to assess the effectiveness and impact of the HPC Wales project and to provide recommendations for improvement.
The evaluation used qualitative and quantitative methods:Desk research.Analysis of previous project reports and documents.Analysis of monitoring system data.Interviews with key stakeholders and employees.Surveys of beneficiaries (businesses, students, researchers and academics).
The main findings from the evaluation can be summarised as follows:Special Purpose Vehicle (in the form of a company limited by guarantee) proved to be an appropriate system to facilitate the purchase of HPC infrastructure, create networks and mitigate risk and promote trade.Progress was initially constrained by: 1) a protracted procurement process; 2) establishment of an infrastructure system; 3) operational staff recruitment.In-house expertise in application (performance tuning, scalability, etc.) and domain-specific support in helping users to exploit the full potential of HPC is critical for the second phase of the project.Despite the initial delay, the project fully achieved 8 of the initial targets, while 5 were 80% achieved and two 60%.The project was successful in securing funding from Horizon 2020 and Innovate UK/TSB (UK’s innovation agency).HPC Wales had a measurable impact on businesses supported and there is ongoing demand for HPC Wales services.Among the services provided, training services can lead to future projects, tenders and long-term benefits.It remains unclear whether the current ownership model or the outsourced rental model is the most cost effective one.HPC Wales has provided a strong foundation on which to build a long-term sustainable model (1/3 core grant, 1/3 commercial/project income and 1/3 from competitively sourced grant income or partner institutions).
Project follow-up should be based on the ongoing provision of services to businesses and training activities (the latter to be considered under a separate bid). HPC Wales stakeholders should ensure that adequate transition arrangements are put in place to secure these assets and future operation of HPC services.HPC activities within the Stakeholder institutions are aligned alongside the successor HPC Wales II project. The associated facility should remain in-house and support infrastructure should be strengthened providing the interface between the facility and research staff.The HPC Wales brand should be exploited and the stakeholders institutions should capitalise on their successful Horizon 2020 bid to create a Network of HPC Competence Centres (SESAME Net).Since recognised value in terms of expertise, products, services and brand was created, consideration should be given to how these assets should be retained and developed for Wales.
Evaluation of the design and implementation of the Employment OP, 2014-2020
Bericht der Begleitenden Evaluierung zu Konzeption und Umsetzung
Hedwig Lutz (WIFO), Isabella Juen, Maria Köpping, Andrea Leitner, Mario Steiner (IHS), Ferdinand Lechner, Helga Reichert, Claudia Sorger (L&R Sozialforschung)
WIFO - Österreichisches Institut für Wirtschaftsforschung - Arselnal, Objekt 20, 1030 Wien; Tel: +43 1 798 26 01 - 0; hedwig.lutz@wifo.ac.at
The evaluation examines the implementation of the four PAs the ESF OP in Austria.
To assess the implementation of the 4 PAs of the OP: employment (PA1), poverty (PA2), reducing early school leaving in vocational education and promoting lifelong learning (PA3), and measures in Burgenland, a transition region (PA4).
Desk research.Monitoring data.Interviews with stakeholders.Online surveys.
78% of funding is planned to be allocated to PAs 2 and 3.About 140,000 people (mainly unemployed or inactive) have participated in the OP up to end-2018.57% of participants are men (50% target for women), 60% are <25, 78% with ISCED 0-2, 63% are minorities, migrants or from outside the EU.Only 43% of the Performance framework financial milestone for PA1 was reached by end-2018, 85-107% for the other PAs. Output indicator milestones were mostly exceeded for all PAs.30% of participants found a job 6 months after exit and for 46% of those already employed, the labour market situation improved.Small beneficiaries can be discouraged from ESF projects because of the administrative burden and pre-financing requirement.There is a lack of a body responsible for disability mainstreaming and accessibility.The administrative burden from managing ESF projects and data monitoring is a key challenge across all PAs.
Monitoring data are based on information only up to December 2018.
Improve communication with beneficiaries to improve implementation.Promote a good staff planning for administering projects.­
Mid-term evaluation of the progress of the Border, Midland & Western ERDF OP, 2014‐2020, in Ireland
Mid Term Evaluation of the Border Midland & Western Regional Operational Programme 2014‐20
Border, Midland and Western Regional OP 2014-2020
Fitzpatrick Associates Economic Consultants - RSM Consulting UK Ltd
https://fitzpatrick-associates.com/ - https://www.rsm.global/ireland/
The evaluation examines progress in implementing the OP up to end-2017.
To assess the achievement of targets of the OP up to end-2017, its adequacy in relation to territorial needs and the functioning of management arrangements structures.
Monitoring data.Official statistics.Interviews with stakeholders, including the MA.
PA1 (R&D), 3 (SME competitiveness) and 4 (energy efficiency) are progressing well, while PA 2 (ICT) and 6 (environmental protection) are delayed. However, final targets are expected to be met by all PAs.Some schemes did not have adequate means of implementing and monitoring horizontal principles.Relations between the MA and other bodies involved rely more on informal links than on a formal management structure and this may affect the reporting of expenditure.Delayed introduction of the eCohesion system led to delays in the reporting of expenditure.
The actual impact of the OP could not be assessed, because of the early stage of implementation and the limited size of the OP in relation to measures financed by other sources.
Speed up the commitment of expenditure to meet intermediate targets and revise final targets of the OP when already met.Closely monitor the implementation of measures to ensure the achievement of final targets.Revise the means of calculating some of the indicators.Implement strategies to improve integration of horizontal principles in the measures.
Evaluation of PA1 “Strengthening applied research, development and innovation” of the Brandenburg ERDF OP, 2014-2020
Endbericht. Prioritätsachse 1 - Stärkung von angewandter Forschung, Entwicklung und Innovation
IfS Institut für Stadtforschung und Strukturpolitik GmbH - toepel@ifsberlin.de; MR Gesellschaft für Regionalberatung mbH, GEFRA – Gesellschaft für Finanz- und Regionalanalysen
This report takes stock of the 5 measures aimed to strengthen applied research, development and innovation, financed under the OP.
To investigate how and under what circumstances the 5 measures on research and innovation work. To examine implementation and progress of the measures and to assess impact.
Analysis of monitoring data and official statistics.Literature analysis.Expert interviews.Telephone interviews with beneficiaries.2 online surveys of StaF beneficiaries (2018, 2022) and unsuccessful applicants (2018).Innovation biographies to assess ProFIT and StaF (2019 – 2022).Analysis of a separate survey on Brandenburg Innovation Support (2021), which included ProFIT, Big Digital and WTT/Cluster.
The investments in higher education infrastructure (SO1) enabled the production a wide range of scientific publications, were used for doctoral studies and teaching, and prompted several collaborative projects with companies and other research partners.Measures to improve R&D and innovation capacities of companies (SO2) reached a wide range of industries and the number of companies cooperating with research institutions exceeded the output targets.Deadweight of the measures is low, many SMEs would not have carried out R&D projects without EU funding.Significant R&D efforts were still needed to bring a new product to market or to implement a digitisation project after the end of the funding.The support for individual and collaborative projects of cluster-related research in the pre-competitive field of non-university research (SO3) led to the creation of new projects, the development of contacts or collaborations with companies and generated follow-up activities after the end of the projects.Yet even after the project, further research works, tests and contact with companies were needed to be ready for application.Support for cluster management and transfer points (SO4) helped draw the company attention to the potential of research and support contacts, but increased visibility and awareness is needed to be fully effective.
SO1: Strengthen research focus on higher levels of technological maturity and on the needs of SMEs.SO2: Continue funding R&D projects and focus on how to increase the demand of SMEs.SO3: Consider funding research projects that can reach technology maturity within the duration of the project or ensure follow-up measures after the funding.SO4: Ensure knowledge transfer by moving funding through universities rather than separate structures with limited human resources, to overcome turnover in clusters.
Mid-term evaluation of the Competitiveness and Internationalisation OP, 2014-2020, in Portugal
Avaliação Intercalar do Programa Operacional Competitividade e Internacionalização
OP Competitiveness and Internationalisation 2014-2020
EY Parthenon: Coordenation: Sandra Primitivo, Rui Faustino. Team: Filipa Lopes, Hermano Rodrigues, Manuel Reis, Margarida Lourenço, Pedro Casimiro, Pedro Oliveira, Raquel Morgado, Rodolfo Junqueira, Rui Ferreira.
The report is a mid-term evaluation aimed at assessing the impact of the Portuguese Competitiveness and Internationalisation OP (COMPETE 2020) 2014-2020, financed by the ERDF, the ESF and the CF, from 2014 until the end of 2021. COMPETE 2020 sets out to strengthen business specialisation, SME skills and strategies, and context conditions for business activity.
To determine the effectiveness, efficiency, impact, relevance, coherence, and EU added value of the measures implemented under COMPETE 2020 on the competitiveness and internationalisation of the Portuguese economy.
Desk research on legislative and programme-related documents.Quantitative analysis of statistical and monitoring data.Theory of Change.13 interviews (with the OP’s managers and with public officials of relevant departments).4 case studies.5 focus groups.3 surveys: 1 targeting business beneficiaries with CRII and REACT-EU operations (29% response rate), 1 targeting other business beneficiaries (44% response rate), 1 targeting non-business beneficiaries (29% response rate).A counterfactual analysis.
The OP performance is generally positive. Most indicators have been met or will be met, the main exceptions being R&D projects.Support for SME competitiveness has mobilised the necessary demand, the achievement rate of employment sustainability projects is low, results for vocational training are mixed, while targets for transport projects are expected to be met.Public Administration targets for IT strengthening should be met, while worker vocational training measures could be improved.REACT-EU targets for business activity and maintaining jobs were met.The resources mobilised for the OP have proved adequate, although their use has not always been efficient, due to administrative and bureaucratic procedures, lengthy selection processes, and the shortage of qualified support staff in beneficiaries.OP support targeted higher performing companies, leading to improvements in competitiveness, internationalisation, and labour productivity.No changes were observed with respect to specialisation, and investment in R&D remained below EU targets.The OP has shown coherence with other policy instruments, has proved to be relevant to the macroeconomic context and has shown to be adaptable to developments in this.The OP has succeeded in leveraging both private and public sector funding for investment.
Strengthen incentives for firms to invest in R&D and innovation.Enhance the attractiveness of vocational training.Streamline the application submission procedure and the project selection process.Simplify administrative procedures, by introducing more up to date and effective technological tools, improving the qualifications of public officials, and strengthening data-sharing within the public administration.Maintain and refine measures to support the digital transformation and capacity building in public administration.Re-design measures for training public administration staff.
Evaluation of the Modernisation of Vocational Education (MOV) project under the Research, Development and Education OP, 2014-2020, in the Czech Republic - 1st interim report
Evaluace individuálních systémových projektů podpořených z PO 3 OP VVV-II Část II: Evaluační okruh B – Evaluace projektu MOV, 1. PRŮBĚŽNÁ ZPRÁVA
This is the first interim report on the evaluation of the Modernisation of Vocational education (MOV) project, financed by the Research, Development and Education OP 2014-2020.
To assess the management and implementation of the MOV project.
8 Individual interviews with those responsible for implementation and with employers.Group interview with project managers.Case studies (8 schools).Survey of educators involved in the project (88 out of 464).Questionnaire to education institutes (5).Panel of experts.
The implementation of the project is in line with plans. The excessive administrative burden is considered to be the only obstacle.According to a survey, only just over a third of educators use the National Register of Qualifications - NRQ on a regular basis and find it beneficial. Most of the others are unaware that it exists.45% of educators regularly attend expert panels of the MOV project and find benefits from doing so.84% of educators reported improving their knowledge as a result of the MOV project.For education institutes, the MOV project could improve the application of NRQ.Schools expect to benefit from the project by being able to network with other schools and share examples of best practice.
It is too early to assess the effects of the MOV project.
Consider promoting project benefits more to schools and employers.Create more awareness of the advantages of the NRQ.
Innovation Barometer. Evaluation of the Smart Growth OP in Poland, 2014-2020
Barometr Innowacyjności. Program Operacyjny Inteligentny Rozwój
Stanisław Bienias, Tomasz Gapski, Paulina Skórska, Rafał Trzciński (IDEA)
MCM Institute Poland, www.mcminstitute.pl; Realizacja, www.realizacja.com.pl; Exacto, www.exacto.pl; IDEA Instytut, www.ideaorg.eu
The evaluation examines 10 measures supporting innovation in SMEs financed under the OP: innovative services for SMEs, innovation vouchers, internationalisation, market research, etc. Data are updated at 2021.
To assess the results and the effectiveness of support for 10 measures (innovative services for SMEs, innovation vouchers, internationalisation, market research, etc.) up to 2021.
Analysis of monitoring and statistical data.Online survey of beneficiary companies in two rounds: (3 months after signing the agreement (39% of 2,376 companies covered) and 12 months after signing the payment request (34% of 1,092 companies covered).
The measures contributed only to a limited extent to increasing R&D activity, participation in industry clusters, innovation potential and employment in the medium term, according to survey respondents.Beneficiaries made little use of measures to protect intellectual property rights of innovation.The impact of the support on the competitiveness of companies has been very limited, especially because of the economic slowdown caused by the pandemic.The internationalisation of the companies supported, especially micro and new firms, increased only slightly after the projects.
Consider the modular nature of support, the inclusion of research units, the relatively low entry threshold for enterprises, when designing future support for R&D in SMEs.Orient support towards innovative projects and the potential of companies to increase their ties to centres supporting innovation.Ensure implementation of both promotional and system solutions (including design innovations).Focus more on changing the employment structure rather than employment growth.
Intermediate Evaluation of the Technical Assistance OP, 2014-2020
Evaluación Intermedia del Programa Operativo de Asistencia Técnica
The evaluation summarises the results of the mid-term evaluation carried out in 2017 of the operations undertaken up to the 31 December 2016 under the Technical Assistance OP in Spain.
To assess the state of implementation of the OP and the quality of management.
Quantitative-based questionnaire;analysis of indicators;workshops.
The OP’s strategy of intervention remains valid and relevant.The rate of implementation of the programme under each Objective has been satisfactory.There are difficulties in applying the simplified cost option and in complying with the indicator system.The current management of the OP is severely understaffed.
There is a need to:Improve and continue the training of Intermediate Bodies;increase the dissemination of ESF evaluations;establish an obligation to carry out periodic evaluations.
Impact of measures for sustainable development of business support infrastructure and remediation of polluted industrial sites in 2007-2013
Evaluarea de impact a DMI 4.1 „Dezvoltarea durabilă a structurilor de sprijinire a afacerilor cu importanţă regională şi locală” şi a DMI 4.2 „Reabilitarea siturilor industriale poluate şi neutilizate şi pregătirea pentru Noi activităţi”
Constanţa MIHĂESCU , Andrea GRAMILLANo, François LEVARLET, Pietro CELOTTI, Francesco SALVAGNINI, Massimo BRESSAN
SC ACZ Consulting SRL - str. Ștefan Velovan, bl. 23A, Craiova, jud. Dolj, Tel: 0351 44 20 44 - T33 SRL via XXV Aprile, nr. 28/b, Ancona, Italia, Tel: +39 0718 98093 , Iris SRL via G. Verdi, nr. 40, Prato, Italia, Tel: +39.0574.607522
The evaluation assesses the effects of measures included in 4.1 "sustainable development of business support infrastructure of regional and local importance" and 4.2 "remediation of polluted and unused industrial sites and preparation for new activities" of the Romanian Regional OP 2007-2013 and indicates possible future monitoring and evaluation activities.
Measuring the effects of the measures and identifying factors and conditions affecting their successful implementation.
Literature review, semi-structured interviews, questionnaire, ex post cost-benefit analysis, case studies, administrative data, panel of experts and a benchmarking analysis.
Interventions under measure 4.1 (TO3):The measures performed well in terms of the business structures supported and jobs created (2,105 jobs created, 68% of the target).The two main factors lengthening the selection process were the financial crisis and the definition of property rights.Business centres provided infrastructure for offices, meetings and conference; fewer measures provided services for innovation and R&D.The OP provided crucial funding at a time of investment crisis;Interventions under measure 4.2 (TO6):The measure addressed the need for remediation that could not be met by local authorities without EU funds;Public procurement rules, legislation on site and property insurance and a lack of companies specialised in soil decontamination hindered the implementation of projects.
The lack of completed projects under measure 4.2 limited the evaluation.
Financial instruments might be considered for the 2014-2020 programming period.The private sector needs to be more involved to accelerate the implementation process and the sustainability of business centres.The development of business structures for innovation and technology and knowledge transfer should be fostered.
Evaluation of the e-learning tool TRIO, financed by the Toscana ESF OP, 2014-2020
La Formazione a distanza tramite il portale TRIO della Regione Toscana
Lucia Fiorillo, Andrea Naldini, Marco Pompili - Ismeri Europa
The evaluation examines the results of the placement survey in respect of the TRIO e-learning tool financed by the OP.
To assess whether motivations and expectations of participants in the learning activities have been met, the effectiveness, strengths and weaknesses of the TRIO tool and the employment situation of participants after the training.
Official statistics.Monitoring data.Survey of final recipients (response rate 23%).Survey of organisations providing e-learning courses (response rate 54%).
Most of participants are workers and enrolled in the training because of a request from their employers. Most of participants are satisfied with the training received, especially those who enrolled spontaneously in the courses (often unemployed), and people over 45.Positive aspects of TRIO are the ease of following a course from home, flexibility of training and the wide choice of courses available, while negative elements are the few opportunities for interrelationship and limited update of training content.Overall the employment rate of unemployed participants 6 months after is 31%, but 41% for those with tertiary education. Covid-19 dramatically increased registrations for TRIO training, especially because of the more time available to attend courses.
Annual evaluation 2019 of the Extremadura ERDF OP 2014-2020
Evaluación de los objetivos/resultados del PO FEDER 2014-2020 de Extremadura para el informe anual de 2018 a remitir en 2019 y de cumplimiento del marco de rendimiento. INFORME FINAL
The evaluation examines the implementation and results of the Extremadura ERDF OP 2014-2020 over the period 2014-2018, covering all TOs.
To assess the implementation of the OP, its relevance in relation to needs, the achievements as measured by financial and result indicators, the impact in terms of the contribution of the OP to achieving the specific objectives, the contribution to Europe 2020 and the application of the horizontal principles.
Desk analysis of programming documents, monitoring and statistical official data;interviews with and online questionnaires to 17 implementing and 15 managing bodies.
The OP strategy remains valid, as the socio-economic context has not changed much.Monitoring and evaluation have been intensified, but management operations are difficult to monitor and corrective measures cannot be easily implemented where necessary.There have been delays in implementing the OP.The result indicators show good progress towards targets, but largely because of favourable socioeconomic developments.Most PAs have met performance framework milestones, but there were delays in verifying expenditure under PAs 1 and 9.Education and healthcare infrastructure needed has been built and digital public services developed.Horizontal principles have been suitably applied.
It was not possible to carry out an analysis of efficiency, because unit costs could not be calculated from the data available.
Continue developing training adapted to management needs;provide qualified technical staff to the management bodies to reduce workload;consider transferring funding from IP2.1, on which progress is slow, to IP2.3, which is achieving significant results in terms of digital public services in education and healthcare;propose a new result indicator for SO4.5.1 to monitor “Emissions of greenhouse gases in transport”;provide specific training on equality of opportunity between women and men;develop result indicators in 2021-2027 directly related to co-financed activities.
Evaluation of the material progress and the results of the Wielkopolskie OP, 2014-2020
Ewaluacja postępu rzeczowego i rezultatów Wielkopolskiego Regionalnego Programu Operacyjnego na lata 2014-2020
Maciej Piotrowski, Michał Thlon, Jakub Grabowski, Magdalena Marciniak-Piotrowska, Artur Kowalczyk, Katarzyna Grudzień
The evaluation assesses the degree of attainment of the OP indicator targets, with particular focus on the Performance Framework indicators.
To assess the progress of the OP’s implementation, the main factors affecting the effectiveness of measures and the impact on the main macroeconomic indicators.
Monitoring and statistical data analysis.Documentation analysis.Macro modelling.18 interviews with implementing bodies.CAWI/CATI survey of 1347 beneficiaries.2 focus groups.Expert panel.Workshop.
There is no risk of missing the performance framework targets.The most important factors for the projects implemented are the COVID-19 pandemic and the increase in the prices of goods and services.According to beneficiaries, the sustainability of projects is high (over 68% of respondents).The evaluation results confirm the positive impact of the intervention on GDP over the period analysed, despite anti-growth factors such as the COVID-19 pandemic and the war in Ukraine.
Carry out an in-depth study of the reasons for the termination of the agreement by beneficiaries.Implement an IT system to provide a comprehensive service for beneficiaries.Give priority to calls for proposals for R&D projects from enterprises and research centres.Support firms and research centres according to the socioeconomic needs of the region.
Assessment of the factors obstracting employment within subsidized employment in the projects of ROP Śląskie 2014-2020
Ocena czynników utrudniających zatrudnianie pracowników w ramach zatrudnienia subsydiowanego w ramach projektów RPO WSL 2014-2020
DYSPERSJA - Anna Chrościcka, ul. Szymanowskiego 4 lok. 65, 03-477 Warszawa, E-mail: biuro@dyspersja.com, Telefon: (22) 670-10-93, Faks: (22) 670-10-93
Interim report on the effects of ESI funds on regional development, 2014-2020, in the Zlínský region in Czechia
Výsledková evaluace přínosů Evropských fondů na regionální úrovni: Úkol 5 a 6 - Zlínský kraj
The evaluation is aimed at assessing the effects of six OPs (Enterprise and Innovation for Competitiveness OP, Environment OP, Integrated Regional OP, Employment OP, Transport OP, Research Development and Education OP) on regional development in the region up to June 2022.
To assess the outputs, results, relevance, and benefits of the measures in the region financed under 6 OPs up to June 2022, as well as the effects and added-value of the integrated urban and community-led local development (CLLD) measures.
Desk research.Analysis of statistical and monitoring data.Interviews.One case study.Surveys.
Projects supported have strengthened the competitiveness of enterprises in medium-sized municipalities and of start-ups.Projects have provided significant support to the cultural sector. Several revitalisation projects were initiated and Luhačovice is now preparing for inclusion in the UNESCO list. Overall, the number of visitors in the region has increased, but tourist promotion of the region needs to be further developed.277 social housing units were created in medium-sized municipalities and the availability and accessibility of services was increased.The needs of the region as regards transport have not been adequately addressed and there has been no major improvement.Thanks to the LAG, the administrative capacity of municipalities in the Rožnovsko micro-region has been strengthened and projects can start inter-municipal cooperation.In Rožnov pod Radhoštěm, projects had a positive impact on business competitiveness, the quality of teaching, and in connecting education programmes with the labour market. Projects in the sub-region would not be possible without EU funding (67% of municipalities report not being able to implement projects without this).ITIs have supported projects on innovation, competitiveness, education, social services and environmental improvement.CLLD was an important source of investment funds for smaller municipalities and small businesses.Besides providing financing for smaller development projects in municipalities, LAGs were the drivers of activities and helped to increase the absorption capacity of the region.Integrated Regional OP investment projects are unavailable to small non-profit organisations, as they are unable to receive loans to pre-finance projects.
Lack of detail of available socio-economic data prevented the use of counterfactual analysis.
Estimation of the long-term result indicator "Number of persons in a better situation on the labour market six months after leaving the programme", under the Podkarpackie OP, 2014-2020
Ewaluacja na potrzeby oszacowania wartości wskaźnika rezultatu długoterminowego Liczba osób znajdujących się w lepszej sytuacji na rynku pracy sześć miesięcy po opuszczeniu programu - woj. podkarpackie
The evaluation assesses the impact of support on the professional situation of participants in ESF projects. The evaluations focused on people employed at the time of participation in projects implemented in 2021-2023.
To estimate the long-term indicator "Number of people in a better situation on the labour market six months after leaving the programme" of participants in ESF projects and identify the most effective forms of support.
Desk research (programme documents, legal acts).Analysis of monitoring data.CAWI/CATI survey of 4,085 project participants.
The gross value of the long-term indicator related to the number of people in a better situation in the labour market six months after leaving the programme was 8.81%. This value was slightly higher than in the previous measurement (7.92%).The net value of the indicator, which is based on participants’ opinions, was equal to 3.84%.The improvement was noted primarily among young people under 25 years (24.42%) and those with junior high school (24.24%) and primary education (10.71%).The most effective forms of support were subsidies to wages or costs of running a business, reimbursement of the costs of care for dependent persons and grants to start a business.
Difficulties of reaching respondents in the survey.
Impact of the Podlaskie Voivodeship OP for 2007-2013 on the regional economy using the HERMIN model
Ocena oddziaływania Regionalnego Programu Operacyjnego Województwa Podlaskiego na lata 2007-2013 na gospodarkę regionu przy wykorzystaniu aplikacji HERMIN-REG PODLASKIE
Urząd Marszałkowski Województwa Podlaskiego, Departament Rozwoju Regionalnego, Regionalne Obserwatorium Terytorialne, ul. Poleska 89, 15-874 Białystok
Simulations using the HERMIN model are carried out on the impact of the Podlaskie Voivodeship OP for 2007-2013 on the growth rate, the situation of enterprises, gross value added (GVA), employment and standard of living.
To assess the impact of the OP on the region and the effectiveness of the programme.
Simulations were made using the HERMIN model (based on monitoring data).
The value of transfers implemented under the OP in the reference period 2007-2016 amounted to almost PLN 4 billion (around EUR 932 million).Targets were achieved for most indicators.Taking account of all EU funding since 2004, the OP is estimated to have increased GDP by almost 2.5% and GDP per capita of Podlaskie by 1-1.4 percentage points a year.The OP reduced the development gap with other EU regions only partly.The OP increased investment in enterprises.It raised gross household disposable income by PLN 4.7 billion (around EUR 1,095 million) between 2007 and 2023.Over 20% of the increase in the regional employment rate of those aged 20-64 between 2004 and 2015 is attributable to the programme.
Evaluation of measures for Roma communities under the Human Resources OP, 2014-2020, in Slovakia
Hodnotenie pokroku k naplneniu cieľov PO6 Technická vybavenosť v obciach s prítomnosťou MRK v rámci OP ĽZ
Octigon, a. s., Projektové služby, s. r. o., RADELA s. r. o.
info@octigon.sk; office@pseu.sk; radela@radela.sk
The evaluation examines the progress towards the objectives of the PA 6: “Technical equipment in municipalities with marginalised Roma communities (MRC)”, up to end-2019.
To assess efficiency, effectiveness, and relevance of the interventions under PA6, “Technical equipment in municipalities with marginalised Roma communities (MRC)”, up to end-2019.
Analysis of monitoring data from the IIB and administrative data (publicly available database, register of social enterprises).Interviews with the institutions involved in project implementation.Interviews with a sample of eligible applicants and beneficiaries of non-repayable financial contributions (NFC).
The financial progress of PA6 is very slow and financial instruments had not yet been activated at the end of 2019.Achievement of target values for several output indicators is at risk, since many indicators are below 30% of the target.PA6 is potentially very important to achieving Europe 2020 objectives for poverty and social inclusion.Main obstacles to implementation are: the co-financing rate of 5% (especially for the non-profit sector), delays or cancellation of calls, and the large number of controls.Measures seem to be efficient in terms of average costs per unit of output, though this is a provisional assessment.Allocation of funds to PA6 was in line with the identified needs of MRCs.
Data on indicators were limited, as verification of targets is planned in 2021 and 2023.
Increase publicity for eligible applicants to speed up implementation of PA6, including a common campaign with PA5, and monitor the implementation of projects.Continue to reduce the administrative burden.Provide practical assistance to applicants.For the next period, identify a single institution for coordination of support for MRCs; map the needs of municipalities with MRCs; raise awareness of financial instruments among all stakeholders and exchange good practice with other EU countries in this area.
Evaluation of the implementation of projects of strategic importance for socio-economic development under the Małopolskie OP, 2014-2020
Ewaluacja realizacji projektów o strategicznym znaczeniu dla społeczno-gospodarczego rozwoju regionu w ramach Regionalnego Programu Operacyjnego Województwa Małopolskiego na lata 2014-2020
The study examines the effects of key strategic projects implemented under the OP on the social and economic development of Małopolska.
To assess the OP procedures for selecting projects of strategic importance for the region and their effects on the pursuit of specific public policy goals.
Desk research of the programming documents, and evaluation reports.Analysis of monitoring data and official statistics.59 interviews with beneficiaries, institutions involved in implementation and local government.Benchmarking (against Spain, Croatia and Greece).11 case studies.
The non-competitive projects undertaken are highly strategic and justify the legitimacy of a non-competitive selection procedure.The non-competitive ESF projects have helped to pursue labour market, social economy and lifelong learning goals.There is a risk of both ERDF and ESF projects being delayed by COVID-19 constraints.Key strategic projects respond to socio-economic challenges in a comprehensive way.
Consider implementing strategic projects on telemedicine or targeted social services, given demographic trends.Include environmental projects among those planned for implementation in a non-competitive way.Take action to increase the potential for investment on the basis of public-private partnerships.Initiate a discussion with the education community to review the experience of the effects on education of the COVID pandemic develop a long-term action plan taking account of the risks and opportunities of distance learning.
Ex-post evaluation of the Kujawsko-Pomorskie OP, 2007-2013
Ewaluacja ex post Regionalnego Programu Operacyjnego Województwa Kujawsko-Pomorskiego na lata 2007-2013
ROP Kujawsko-Pomorskie Voivodeship 2007-2013
Pracownia Badań i Doradztwa “Re-Source” Korczyński Sarapata sp.j. ul. Spławie 53; 61-312 Poznań
The ex-post evaluation relates to interventions financed under the regional Operational Programme for Kujawsko-Pomorskie in 2007-2013, which were aimed at increasing the competitiveness of the region and its social and economic cohesion.
The objective is to assess the effects of the support provided by the regional Operational Programme in Kujawsko-Pomorskie, mainly in terms of achievements in relation to the specific objectives.
The following methods were applied:desk research;focus group interviews with beneficiaries;CAWI surveys of beneficiaries;CATI surveys of adult inhabitants of the region;difference-in-difference method;in-depth telephone interviews with partners, final beneficiaries and operators;individual in-depth interviews with employees of the MA;expert interviews with scientists in the regional universities and representatives of "third sector" organisations;case studies.
The results of the study are as follows:implementation: The programme was properly planned and its implementation is considered positively. However, the long time taken to assess applications is seen as a major bottleneck. The division of tasks and coordination between the units was clear. The relatively small administrative burden on beneficiaries and the information/promotion activities are also assessed positively;management: The monitoring system is considered average, the main issues being the lack of reporting data, imperfections in the National Information System and the absence of a local monitoring system. The control system is assessed as relatively flexible;the objectives of the programme were largely achieved, the results in terms of job creation being in particular significantly better than expected. The objectives of all priority axes were broadly achieved except as regards ICT infrastructure;for almost all context indicators, the situation improved (in line with the nation-wide trend), especially social developments;the funds spent relative to the results achieved indicates that they provided tangible results.
managing Authority: bureaucracy, changing legislation and unclear guidelines, delays in signing agreements, late payments;applicants: ineligible expenditure, problems with contractors, real costs exceeding planned costs, insufficient human resources, difficulties in obtaining permits, problems in public procurement procedures;changes of prices/currency exchange rates.These problems are not specific to the programme but were also reported in other regions.
Second Evaluation Report on the Youth Guarantee and the Youth Employment OP, 2014-2020
Secondo rapporto di valutazione della Garanzia Giovani e del Programma Operativo nazionale Iniziativa Occupazione Giovani
Giuseppe Di Battista, Roberto Landi, Cristina Lion, Vanessa Lupo, Federico Orfei, Katia Santomieri, Simona Pizzuti, Veronica Sciatta, Anna Maria Senatore, Paola Stocco, Enrico Toti
ANAPAL – Agenzia nazionale per le politiche attive del lavoro, Via Fornovo, 8 - 00192 Rome (IT), www.anpal.gov.it
This is the second evaluation report of the Youth Guarantee and Youth Employment OP 2014-2020 co-financed by the ESF and YEI in Italy.
To assess the implementation, effectiveness and impact of the OP with a particular focus on the quality of job offers.
Desk analysis;analysis of monitoring and financial data;CATI and CAWI surveys of participants (12,000) and non-participants (8,000) carried out between May and September 2017;participant satisfaction survey of 28,000 individuals (7.5% response rate);multivariate analysis;counterfactual analysis based on covariate matching.Data used are from the monitoring system (SIGMA, regional data), the ANPAL unitary information system, INPS (Italian National Social Security) and surveys.
The PES need to be further strengthened after the reform in 2015 in terms of staff and organisation (the satisfaction survey showed overall satisfaction with the professionalism of the PES staff, but less with of the organisation and waiting time).583,619 young people (56% of those applying) participated in an active labour market measure, 56% aged 19-24 and on average more educated and active than non-participants.Up until September 2018, 720,032 individual measures had been implemented, 58% of them traineeships, 23% recruitment subsidies and 13% training courses.75% of participants had at least some work experience during the programme and half of them were in employment at the end of September 2018.57% of those with tertiary education were placed in jobs as against 43% of those with lower secondary education. The figures were also higher for those aged 15-18 than those older and those less disadvantaged on the labour market as well as those living in the north.23.5% of offers to participate in active measures were refused (higher for the start-up support) and 10% of measures were not completed.Participation in active measures increases the likelihood of finding a stable job as compared with non-participating (8 percentage points higher), but the difference disappears in the long-run.In southern regions, the positive effects on employment are evident sooner than in norther regions, mainly because of the use of recruitment subsidies.The proportion of those participating in traineeships who were in employment 18 months after completion was 9.5 p.p. higher than for non-participants, but 9 p.p. lower for those participating in education.
Evaluation of the APIv projects supported by the Research, Development and Education ESF OP, 2014-2020, in the Czech Republic – Fourth interim report
Evaluace individuálních systémových projektů podpořených z PO 3 OP VVV-II Část II: Evaluační okruh A – Evaluace projektů APIV 4.průběžná zpráva
The evaluation examines the implementation of two projects, “Inclusive Education Action Plan – APIv A” and “Promoting joint education in pedagogical practice – APIv B”. This is the fourth interim report and covers the period January-December 2021.
To assess the implementation of measures APIv A (“Inclusive Education Action Plan”) and APIv B (“Promoting joint education in pedagogical practice”) in 2021.
Desk review.Evidence from previous evaluations on the same projects.Interviews with project managers, project administrator, internal evaluators and coordinators of different activities.Online survey of 33 schools involved.Case studies: interviews with 5 supported schools and 5 non-supported.Expert panel.
Due to the Covid pandemic, some measures of APIv A and B have been extended to 2022.In both projects, many activities were shifted or modified to online forms of education support due to the pandemic.The administrative burden remains a common barrier for both projects creating unreasonable demand on capacity.Although beneficiaries are largely satisfied with online activities, mentoring and internship are considered less suitable for online provision.The main risk as regards implementation for both projects remains the pandemic and the restrictive measures put in place. This may make necessary to transfer many activities to the online provision.Other risks are: the broad scope of APIv A where the implementation of each key activity involves a large number of sub-tasks, and difficulty of establishing successful cooperation with the media to present joint learning (APIv B).
Less willingness or more limited possibilities of schools to be involved in the field investigation and case studies due to the pandemic restrictions.
Do not cumulate too many activities within the same project.Plan follow-up activities properly.Make more systematic use of output from core activities.
Evaluation of the implementation of horizontal principles under the Growth and Employment Austria OP, 2014-2020
Endbericht. Leistungspaket 6 – Querschnittsthemen
OP Investments in Growth and Employment Austria ERDF 2014-2020
Ursula Mollay, Arndt Münch, Ulrike Stroissnig
ÖIR GmbH, mollay@oir.at, www.oir.at; convelop – cooperative knowledge design gmbh, www.convelop.at; KMU Forschung Austria, www.kmuforschung.ac.at; ÖAR GmbH, www.oear.at; ÖGUT GmbH, www.oegut.at; Spatial Foresight GmbH, www.spatialforesight.eu
The evaluation examines the sustainable development and equal opportunities and non-discrimination principles under the OP.
To assess the extent to which horizontal principles are taken into account in implementing projects, and the usefulness of the questionnaire to project promoters before project approval, which aims to increase awareness of horizontal principles.
Previous thematic evaluations.Questionnaires to project promoters before the approval of projects.Survey of funding bodies on specific measures aimed at taking account of the equal opportunities and non-discrimination principle.
While the sustainable development principle is strongly anchored in project design and implementation, the equal opportunities and non-discrimination principle is considered only to a limited extent.64% of projects contain elements relating to sustainability and 10% elements relating to equal opportunities and non-discrimination.40% of project promoters have sustainability certifications, and only 27% certification as regards equal opportunities and non-discrimination.The equal opportunities and non-discrimination principle is considered more at company level than at project level.Both horizontal principles are taken into account in urban and territorial measures in particular, the sustainability principle in measures to reduce CO2 emissions.Funding bodies show limited ambition and activity in relation to equal opportunities and non-discrimination.The questionnaire to project promoters is a useful evaluation and awareness tool.
Aim to have the questionnaire completed by all projects, except for CLLD, to ensure that a detailed assessment can be carried out.Eliminate irrelevant questions and carry out a plausibility check to increase the quality of the questionnaire.Ensure that IBs provide advice to project promoters on horizontal issues in order to raise their awareness of them.Take more account of horizontal principles during the tendering phase.
Socio-economic analysis in preparation for the Bayern ESF+ Programme, 2021-2027
Sozio-ökonomische Analyse ESF Bayern 2021-2027
Dr. Jenny Bennett, Stefan Feldens, Maik Oliver Mielenz, Christian Muscheid
Institut für Sozialforschung und Gesellschaftspolitik GmbH, Weinsbergstraße 190, 50825 Köln, E-Mail: Bennett@isg-institut.de, Tel.: +49 221-130 655-56
The study analyses the socio-economic situation in the region to prepare for the 2021-2027 Programme.
To assess the needs of potential beneficiaries and of the region, and to identify possible modifications to the present OP.
Between 2014 and 2018, the population grew by 3%, but the increase was mostly due to inward migration. After 2033, population is expected to fall.GDP growth and labour productivity show an upward trend. The competitiveness of the economy has been maintained and the region is one of the most innovative in Europe.Between 2014 and 2018, the employment rate increased, but the gender gap is still significant. In 2018, the at-risk-of-poverty rate was slightly higher than in 2014.26% of employees are at risk of being replaced because of automation and digital technologies. Investment in relevant technological applications and in training for workers is needed.The proportion of young people without secondary education increased by 1.2 pp. between 2014 and 2018 to 5.5%.
Annual evaluation report 2018 on the Toscana ESF OP, 2014-2020
The evaluation is the 2018 annual report on the implementation of the Toscana ESF OP 2014-2020.
To assess the relevance, coherence and progress towards the achievements of the targets set.
desk review;analysis of monitoring data and statistical data (Istat, Eurostat, INPS).
The OP implementation is overall satisfactory with progress on PA C (on training and education) most advanced, while PA B (on social inclusion) has allocated only half of funding. The OP is well managed and organised.The OP reached a range of target groups, particularly the inactive, unemployed and NEETs.6 months after completing measures, between 50% and 90% of participants were employed, but only 27% of the most disadvantaged groups.The OP objectives are still relevant and coherent with other regional and national policies.
Data on expenditure and participants of national OPs broken down by region, monitoring data of ‘interprofessional funds’ on continuing learning and data on beneficiaries of other regional policies are limited.
focus more on men unemployed with most difficulty of reintegrating into the labour market;maintain support for more disadvantaged groups and areas.
Evaluation of the efficiency and effectiveness of the Ionia Nisia OP, 2014-2020
Έκθεση Αξιολόγησης της αποδοτικότητας και αποτελεσματικότητας του ΕΠ Ιονίων Νήσων 2014-2020 - 2ο παραδοτέο
info@eeogroup.gr, https://www.eeogroup.gr
This evaluation examines progress in the implementation of the OP up to February 2021.
To assess the efficiency and effectiveness of the OP to inform the revision of the OP in 2021 and progress towards targets and their appropriateness after the pandemic.
Analysis of monitoring data.Analysis of statistical data from the National Documentation Centre, ELSTAT and Eurostat.Analysis of relevant documentation.
Overall implementation of the OP as reflected in output indicators is satisfactory but it is expected to be affected more by the pandemic, and financial resources are considered to be too limited for both the ERDF and ESF.45% of the measures cost less than planned, 40% are in line with the plans, and 15% cost more.Monitoring data on output indicators are not regularly updated.PA 4 and 5 are progressing well but PAs 1, 2, 3 are not in line with Performance Framework targets for 2023.The intervention logic of the OP remains relevant.The OP is making a significant contribution to the objectives of the EU Strategy for the Adriatic and Ionian Region, as indicated by 85 projects.There were delays in implementation because of the late selection of an IB for State Aid operations and changes in the public procurement framework.
Include additional output indicators in the OP (mainly those linked to Covid-19), and revise indicators in line with the updated intervention logic.Increase resources for PA 1.Speed up procurement and payments for integrated projects.
Mid-term evaluation of the OP for Cohesion Policy Funding, 2014–2020
Euroopa Liidu Ühtekuuluvuspoliitika fondide rakenduskava 2014-2020 vahehindamine
Katre Eljas-Taal, Merit Tatar, Maarja Käger, Marek Tiits, Tarmo Kalvet, Merle Tambur, Aija Kosk, Keiu Friedenthal, Karmen Kert, Maarja Toots, Ülo Kask, Martin Kikas, Garri Raagmaa, Märt Masso, Liina Osila, Indrek Kadarik, Laura Aaben, Hanna-Stella Haar
Technopolis Group Eesti, Ernst & Young Baltic AS, Balti Uuringute Instituut, Praxis Centre for Policy Studies
The mid-term evaluation concerns the OP for Cohesion Policy Funds 2014–2020 which has a budget of EUR 4.6 billion and covers all TOs.
To assess the relevance, efficiency, impact, achievement of strategic objectives and sustainability of the OP.
Desk analysis, literature review;Online surveys of partners and implementing bodies (74 respondents out of 356 invitations);3 Workshops with sectoral experts;7 Focus groups with partners, IBs and implementing agencies;25 Interviews with those involved in implementation.
The strategy is still consistent with needs and there is no need for major changes.At the end of 2018, 32% of funding was spent and, in most PAs, financial milestones were achieved. In the 5 PAs with the worst financial performance, the main reasons were late start of measures, lower than expected demand, administrative reforms and limited capacity of local government.Result indicators show improvement in access to the labour market (PA3), improvement in the administrative capacity of the public sector (PA12), development of ICT services infrastructure (PA11), development of energy efficiency (PA6) and mitigation of environmental risks (PA8).ESI Funds is helping to develop an economy with greater knowledge intensity and lower resource intensity, though the impact of the corresponding investment has not yet materialised. The OP is in line with the Estonia 2020 strategy, Europe 2020 and the Baltic Sea Strategy. The OP is also coherent with the Regional Development Strategy and the country-specific recommendations.Balanced regional development remains a major challenge which is not evenly addressed across PAs; metropolitan areas have received more funding than peripheral areas.
There is need to:provide additional funds from the performance reserve to activities where there is demand for additional funds and capacity for absorption; which are continued in the next programming to ensure sustainability and which create added-value such as R&D and innovation, education and regional development;develop result indicators more directly linked to the objectives of interventions;improve consultation with partners and their engagement;continue with inter-agency cooperation and central coordination for capacity building;give greater attention to increasing the administrative capacity of local authorities and include these in the definition of local development policies;implement a joint programme-based investment measure for major regional urban centres, in place of the current application-based measures.
Evaluation of the results of the integrated territorial programmes 2014-2020
Az Integrált Területi Programok eredményeinek értékelése 2014-2020
OP Territorial and settlement development 2014-2020; OP Competitive Central-Hungary 2014-2020
Péter Kolossa, Zsuzsanna Földi, Sona Avenasian, Marcell Borbély, Viktor Molnár, Eszter Oblath, Katalin Peredi, Tamás Saád, Péter Stipkovits, Alexandra Szabó
Field Consulting Services Zrt., http://fieldconsulting.hu/, iroda@fieldconsulting.hu
The evaluation explores how ITP tools and their implementation have contributed to the achievement of the development objectives of the Counties and Municipalities programmes which are eligible under TOP (Territorial and Settlement Development OP) and VEKOP (Competitive Central Hungary OP) funding.
​​To assess how the use of resources under ITPs (43 ITPs in total) with a spatially diverse logic has resulted in development patterns at national level and how the absorption of resources, irrespective of administrative boundaries and ITP areas, affected regions and groups of municipalities with similar characteristics (lagging districts, disadvantaged small rural areas).​
​​Desk analysis. Logical analysis of Strategic Cycle of interpretation of ITP implementation. Descriptive analysis. Statistical analysis with geospatial tools for thematic and territorial mapping. Correlation analysis of development indicators (4 groups of indicators: labour market, demography, business, tax statistics). Interviews with experts of the institutional system (4 counties) and other experts. Survey of territorial actors with ITP, covering 18 counties and 22 MJV (Cities with county rights).
​​ITPs were not effectively used as strategic, "living" documents to support balanced territorial development: while planning and implementation structures were established, continuous monitoring and phased evaluation were rarely carried out. There is no significant correlation between aid amounts and the development indicators considered. Counties and municipalities use ITP instruments with varying degrees of intensity, often allocating resources without considering territorial disparities. Municipalities prioritise local development over regional cooperation, leading to a lack of collaborative efforts in initiating regional projects. While lagging districts receive TOP grants at higher rates, grant amounts vary significantly because of inconsistent fund allocation by counties. ​
Although the TOP and VEKOP projects were mostly completed, the timing is not ideal, as some recommendations from the evaluation may be too late to affect the ITP planning for 2021-2027.
​​Empower counties with resources and authority to monitor and manage ITPs, including implementing corrections and conducting programme-level management. Ensure territorial actors consider previous funding use, including TOP and other domestic sources, when setting objectives in current spatial development documents. Ensure necessary framework conditions are in place to put the ITP cycle into practice (so that regional actors have flexible access to all databases, with a financially sustainable toolkit, to review the dissemination activities of the TOP Plus MA). Improve management capacity across all organisational groups involved in the ITP budget. Intensify the use of thematically and spatially integrated interventions to adopt a more balanced approach and to maximise the spatial development impact of ITPs.​
Ex-ante study on the use of simplified cost options under the ESF+ National Programme, 2021-2027, in the Netherlands
Eindrapport: Ex ante uitvoeringsonderzoek naar nieuwe SCO-methodiek voor subsidiëren casemanagement
Bram Berkhout, Leonore Das, Irene van Eldik, Anne van Schothorst
Berenschot Groep B.V., www.berenschot.nl, b.berkhout@berenschot.nl, l.das@berenschot.nl, i.vaneldik@berenschot.nl, a.vanschothorst@berenschot.nl
The study examines the feasibility of applying a new simplified cost option (SCO) methodology for subsidising the administrative activities of projects.
To gain insights from the pilot cases conducted in three labour market regions (Amsterdam, Tiel, and Leeuwarden) and from the 32 regions not yet applying the new SCO methodology, in order to assess the feasibility of nationwide implementation and identify obstacles.
Desk analysis of programme and regulatory documents. Focus groups with the employees involved in SCO development and control from of the implementation agency (Uitvoering Van Beleid - UVB). Online survey with municipalities of the 32 non-pilot labour market regions (59% response rate). Focus groups with representatives from the 32 non-pilot labour market regions.
Respondents from the pilot labour market regions reported a significant reduction in administrative workload, as fewer documents are required with the new SCO methodology. However, the SCO methodology generated greater administrative effort for participants and unclear definitions of eligible activities led to inconsistencies in applications. The new methodology allowed greater flexibility in planning contact moments, making it easier to adapt to existing schedules and participant needs. Specifically, Tiel municipality reported time savings; Leeuwarden implemented digital tools but faced some complexity; Amsterdam struggled with data systems and high caseloads. According to respondents from the pilot test, the EUR 4.20 daily rate for administrative activities is insufficient. Differences in technical systems and digital capacities across municipalities limited the effective implementation of the SCO. Non-pilot regions recognise similar benefits to pilot ones and consider the rate insufficient too, raising concerns about feasibility, workload, and financial sustainability of the SCO methodology.
Ensure that municipalities and implementing organisations adapt and test their ICT systems for the SCO methodology, with supplier support. Establish rules centrally and maintain them in a regularly updated knowledge base. Provide targeted technical and administrative support to municipalities with limited capacity. Review the EUR 4.20 daily rate and differentiate compensation by type of activity. Strengthen implementation through consistent communication, training, and monitoring.
Evaluation of the Building Better Opportunities initiative, 2014-2020 – Annual Report 2019
Building Better Opportunities Evaluation. Annual Report 2019
This is the third evaluation report on the “Building Better Opportunities” (BBO) initiative, financed by the ESF which supports local projects for social inclusion and local job for harder-to-reach groups. The projects are implemented by the Voluntary, Community and Social Enterprise (VCSE).
To identify what works well and why and to assess the results of BBO.
Desk analysis of documents, previous evaluation reports and monitoring data;16 project visits;survey of participants (186 for the third report; 500 for the final sample);focus group with funding officers (Big Lottery fund - BLF);2 network events and online networking for the projects to interact.
By the end of 2018, BBO projects had involved 50,579 participants, 67% of the target planned for 2023. Targets were exceeded for participants with disabilities (147%) and those over 50 (103%), but not for inactive participants, including NEET (51%).The main success factors are: the presence of partnerships implementing the projects, the strong relations with external referral partners and the targeted use of social media and campaigns.Over time, administrative requirements have become better defined and grant holders and partners have improved their capacity to deal with procedures.Projects focusing on specific target groups have been able to develop tailored support. For example, for groups with language needs, an informal approach, based on conversation clubs, helped to increase attendance; while volunteering was particularly effective for the older age groups.5,330 participants, or 11% of the total (44% of the target) found a job.91% of respondents to the survey reported that BBO support met their needs and referred to the tailored nature of the support, the role played by mentors and the non-judgemental approach as being positive elements.84% of respondents reported their self-confidence and well-being had improved.
Evaluation of the use of financial instruments under the regional OPs in Normandie, 2014-2020, with a view to 2021-2027
Evaluation ex ante relative aux instruments financiers FEDER, FSE+, FEADER et FEAMPA
ROP Haute-Normandie 2014-2020; ROP Basse-Normandie 2014-2020
www.ey.com/fr; pierre-aymeric.dewez@fr.ey.com
The evaluation assesses support through financial instruments under the regional OPs to make proposals for 2021-2027.
To assess the implementation of financial instruments in 2014-2020 and their current relevance to define an investment strategy for 2021-2027.
Monitoring data.Official statistics.Survey of beneficiaries (46% response rate) and non-beneficiaries (33%).Interviews with MA and stakeholders.Workshop.
Several financial instruments were deployed in 2014-2020 covering different stages of company development, though their scope was limited for some stages and for some enterprises targeted.According to beneficiaries, the use of financial instruments led to increased financial strength, innovation, job creation and business growth.However, the financial structure of companies often remained fragile and access to credit difficult.
It was not possible to carry out an impact analysis because of the limited data available.
Strengthen unsecured loans for small companies and for small projects.Develop financial instruments for pre-seed and seed phases of firm development.Introduce specific instruments, including unsecured loans, for companies in the social economy.
Research and Innovation Strategy for Smart Specialisation in Tuscany
Strategia di Ricerca e Innovazione per la Smart Specialisation in Toscana
ROP Toscana ESF 2014-2020; ROP Toscana ERDF 2014-2020
The evaluation is concerned with the Smart Specialisation Strategy (RIS3) implemented in Toscana in the 2014-2020 period and co-financed by the ERDF and ESF.
To assess progress in implementing the strategy.
Desk research of OP documentation (i.e. ERDF and ESF Annual Implementation Reports);analysis of monitoring data.
RIS3 identifies three technological priorities: ICT and photonics, smart industries and chemicals and nanotechnology. It also aims to support R&D, innovation and framework interventions.The overall contribution to the Strategy from the ERDF is EUR 446.6 million, mainly invested in making the most of and preserving the cultural heritage and enterprise support for technological progress.Overall ESF co-financing is EUR 58.2 million, mainly invested in vouchers for post-graduate courses, traineeships and training workshops and in technical and vocational education and training.As a result of interventions, almost 400 businesses have been supported to introduce new products (252% of the target).Over EUR 9.2 million of private investment has been attracted (477% of the target).
Data were available only up to 31 December 2016.
Evaluation of the WEBNET project under the Employability and Inclusion ESF OP, 2014-2020, in Ireland
The evaluation examines the WEBNET programme (a combination of mentoring, training and networking) which is designed to support women in Tallaght and the surrounding area to become entrepreneurs.
To assess progress made by the WEBNET programme (providing mentoring, training and networking) in relation to its objectives.
Monitoring data.Informal feedback from participants (via Facebook, WhatsApp, or emails).Survey of 46 participants.
From April 2017 to February 2020, 168 WEBNET sessions/classes were provided.A total of 118 women participated (target 90), half of whom were disadvantaged.70% of the participants were in employment on completion of the programme (83, target 60); 69% were still so 6 months after exit. In addition, 82 women gained a qualification upon leaving.There was a largely positive feedback from participants.
Ongoing evaluation of the implementation of the Research, Development and Education OP, 2014-2020, in the Czech Republic - Progress Report 03
Průběžná evaluace implementace Operačního programu Výzkum, vývoj a vzdělávání, průběžná zpráva 03
This is the third progress report on the OP which examines the performance up to June 2017.
To assess the extent to which objectives have been met, to identify factors affecting implementation and to provide feedback and recommendations to the MA.
Analysis of monitoring data.Surveys (CAWI) of PA3 projects.Two case studies for PA3 projects.Two focus groups for PA1 and 2 projects.
PA 1 (Research): around half of output indicators could be at risk of non-fulfilment. There were delays in payments for projects from call 02_16_013, because of legal issues.PA 2 (Universities and human resources): progress of output indicators is overall satisfactory, but the risk of not achieving 2018 milestones is high.PA 3 (Equal access to preschool, primary, and secondary education): there is no risk of non-fulfilment of 2018 milestones, but there could be a lack of qualified personnel.Cross-cutting issues: the obligation to provide information on salaries of employees in projects may violate personal data protection legislation.
Speed up the timing of payments.Verify compliance of the monitoring system’s requirements with personal data protection legislation.Simplify the administration of projects.
Evaluation of measures supporting the transition to a low-carbon economy, under the Partnership Agreement, 2014-2020, in the Czech Republic. Final report
Výsledková tematická evaluace Dohody o partnerství pro období 2014-2020: TC4 Podpora přechodu na nízkouhlíkové hospodářství. Závěrečná zpráva.
OP Environment 2014-2020; OP Prague - Growth Pole 2014-2020; OP Enterprise and Innovation for Competitiveness 2014-2020; Integrated Regional OP 2014-2020
The evaluation examines the measures promoting a low-carbon economy under the Partnership Agreement up to 30 April 2023.
To assess the effectiveness, impact and sustainability of the measures promoting a low carbon economy, especially through the promotion of renewable energy sources (RES), up to April 2023.
Analysis of relevant documents.Cross-Regional Sequential Difference in Difference Analysis (CR-SEQDD).Analysis of monitoring and statistical data.Case studies.Expert assessment.ToC elaboration and testing.
Support for RES did not have the expected impact, as less than 1% of TO4 resources were used for this and interest of applicants in renewable energy projects was low.Support contributed to improve energy efficiency in the housing sector, though with regional variations, as well as the energy performance of public buildings.The reduction in final energy consumption of beneficiary institutions is below expectations.Support from ESI Funds increases the demand for some energy measures that could overload energy networks and represent a threat to the functioning of the national electricity system.There is limited coordination between OPs, as measures often overlap instead of being complementary.
The evaluation of project results is, in some cases, limited by a lack of data and some impact indicators are not fully appropriate. The analysis of efficiency is slightly distorted by the heterogeneity of project size in terms of resources allocated.
Increase cooperation between MAs and relevant stakeholders to increase the coherence of the interventions financed.Prioritise complex energy saving projects with the potential to achieve a greater overall effect and maximize energy savings.Focus support on places with a higher concentration of older housing.
Impact assessment of measures from all the regional ERDF OPs, 2007-2013, in England, on the performance of SMEs
ERDF 2007-2013 Analytical Programme Workstream One. Economic Impact Counterfactual Assessment
The report aims at assessing the economic impacts of 10 regional ERDF OPs on SMEs’ economic performance. Analysed data go up to the beginning of 2013.
To assess the net economic impact of ERDF interventions on SMEs and to provide recommendations about how to best approach the ex-post evaluation of 2007-13 programmes.
Desk research of previous studies and evaluations.Counterfactual impact assessment using DiD, based on Inter Departmental Business Register (IDBR) data.
There is statistical evidence that ERDF support has had an overall positive impact on the performance of the SMEs supported. The size of impact increases with the intensity of funding.The impact is greater for companies in the manufacturing sector than those in the service sector.The impact is more evident on turnover than employment overall. This might indicate a larger effect on productivity or simply reflect the lag between the effects on turnover and employment growth.Schemes focused on strengthening the R&D base have less of an impact than other types of schemes, but the effects of this kind of support normally take longer to materialise.Beneficiaries receiving support prior to 2011 had larger turnover growth over 2010-12 than non-beneficiaries. This seems to be associated with higher productivity growth than employment growth.
Sample attrition and the timing of the impact assessment had the effect of substantially reducing the sample size for the econometric analysis, which limits sample representativeness. Data from the IDBR only cover some measures of business performance, so not all the effects of support could be identified.
Reflect in advance on the timing and methods to be used to evaluate impact of the 2014-2020 OPs.Consider supplementary approaches to measuring the impact of ERDF investments having no direct beneficiary SME.Reflect the data requirements of the selected method on the monitoring arrangements for the programme.
Progress in implementing the Abruzzo ERDF OP 2014-2020
Rapporto sui progressi del POR FESR Abruzzo 2014-2020
IRS – Istituto per la ricerca sociale; Nomisma – Societa’ di studi economici
IRS, www.irsonline.it; Nomisma, www.nomisma.it
The evaluation assesses progress in implementing the Abruzzo ERDF OP, aimed at boosting economic growth, and increasing competitiveness, innovation, R&D and preventing hydrogeological and seismic risk.
To assess the coherence, effectiveness and relevance of the interventions and their contribution to Europe 2020.
Desk research of OP documentation;analysis of monitoring data;analysis of secondary data (context indicators).
The interventions continue to be relevant. A new axis has been introduced to support the economic recovery of the areas affected by the earthquake of 2016 and 2017.PA I ‘RTD and innovation’ – certified expenditure represents 15% of total funding and 61% of the 2018 milestone in the performance framework.PA II ‘Diffusion of digital services’ – certified expenditure represents 1.2% of funding and 6% of the 2018 milestone.PA III ‘Competitiveness of the productive sector’ – certified expenditure represents 13% of funding and 77% of the 2018 milestone.PA IV ‘Promotion of a low carbon economy’ – No expenditure has been certified.PA V ‘Reduction of hydrogeological risk’ – certified expenditure represents 0.3% of funding and 1.7% of the 2018 milestone.PA VI ‘Protection and valorisation of the natural and cultural heritage’ – certified expenditure represents 3% of funding and 12% of the 2018 milestone.PA VII ‘Urban sustainable development’ – no expenditure has been certified.PA IX ‘Prevention of hydrogeological and seismic risk and support for the economic recovery of the areas affected by the earthquake of 2016 and 2017’, no expenditure has been certified.
The effectiveness of the implementation of the OP Smart growth (Module I, stage II) in Poland
Ocena skuteczności wdrażania PO IR przez NCBR, sprawności obsługi projektów oraz identyfikacji dobrych praktyk w działaniu 1.1 POIR (Moduł I, Etap II)
The evaluation assesses the implementation of the OP Smart Growth by the National Centre for Research and Development (NCRD), focusing on the project selection process and the monitoring system.
To assess the project selection process of the OP Smart Growth and the related monitoring system.
Desk analysis;focus Groups;in depth Interviews;analysis of monitoring data.
Applicants have a positive view of NCRD procedures;the OP website includes all the information needed by applicants, but its structure limits its actual availability;the selection system works well and the experts participating are well prepared.the 3 (in some cases 4) project milestones required for monitoring purposes lead to excessive detail in the work plan, hindering the management of the R&D process.
There is a need to:reduce the reporting frequency in measure 1.1.;increase the number of project supervisors;dedicate more resources and attention to beneficiaries at risk of financial loss.
Evaluation of progress of YEI indicators under the ESF Human Resources OP, 2014-2020, in Croatia
Evaluacija napretka mjerenog pokazateljima dugoročnijih rezultata sa zadanim ciljnim vrijednostima za IZM za 2020. godinu - Završno izvješće
OP Efficient Human Resources ESF 2014-2020
The evaluation examines the common result indicators on the employment status of participants 6 months after the end of the interventions up to 2020.
To assess the progress of YEI indicators on the employment status of participants 6 months after exiting measures up to end-2020 .
Analysis of monitoring data.CATI and CAWI survey of participants between July 2019 and June 2020 (338 respondents out of 1281, 5% margin of error) and between 2016 and June 2019 (586 respondents out 1675, 4% margin of error).
Of the 338 participants between July 2019 and June 2020, 61 were in education and 241 were employed 6 months after leaving.Of the 586 participants from 2016 to June 2019, 108 were in education and 425 were employed 6 months after leaving.
The margin of error of the survey was over 2%, which creates uncertainty over the results for subgroups and their generalisation.
Ensure completeness and accuracy of contact details of participants.
Evaluation of the effects and impact of the Prague Competitiveness (ERDF) OP, 2007-2013
Evaluace efektů a dopadů dosažených realizací OP PK
Prosecká 412/74, Praha 9, + 420 222 365 459, info@regiopartner.cz
This is the final report of the Evaluation of the effects of the Prague – Competitiveness OP.
To assess the effects and achievements of the Prague – Competitiveness OP in the different Areas of Intervention (AIs).
Desk analysis of programming and project documents;Analysis of monitoring data;Survey of applicants and beneficiaries;Interviews with beneficiaries;Case studies (targeting 14 projects and all areas of intervention);Theory of change analysis.
The impact of interventions relating to knowledge transfer was largely positive.Projects aimed at the promotion of a favourable business environment had minimal effects. Overall, the competitiveness of SMEs was improved, though with few spill-over effects on non-supported enterprises.Projects involving the revitalisation of waterways had a significant impact on flood-risk management.Energy saving and renewable energy projects contributed to improving the urban environment and its sustainability.Barrier-free access to the metro and development of the cycling network were successfully implemented, improving efficiency and access to public transport.E-government and support for public broadband improved the overall administration efficiency of the city.
At programming level, there is a need to differentiate between "key" activities and "supportive" activities.There should be greater flexibility in approving changes to ICT projects.When assessing or managing projects, account needs to be taken of their financial scale and the practical difficulties of carrying them out.
Evaluation of the environmental, social and economic effects of investments supported under PAs IV and V of the Opolskie OP, 2014-2020
Ocena efektów ekologicznych, społecznych i gospodarczych inwestycji wspartych w ramach IV i V Osi Priorytetowej RPO WO 2014-2020
Dr. Jakub Grabowski, Malwina Pietrzyk, Małgorzata Zub, Aneta Gierowska, Łukasz Groblewski
The report assesses the impact of investments supported under Priority Axes IV and V of the OP.
To assess the impact of measures aimed at strengthening the environment and cultural heritage under the OP.
Analysis of monitoring and statistical data.ToC testing.Survey of 253 beneficiaries.3 interviews with experts and the beneficiary.5 focus groups with beneficiaries.5 case studies.Expert panel.
Investments in environmental protection, waste management, and cultural heritage significantly contributed to regional development. Most of the projects are achieved or are on track to achieve their intended outcomes.Projects that comprehensively addressed problems were the most effective.Limited financial resources have led to underinvestment in certain areas, including cultural heritage.The main challenges of the projects included their durability and adaptation to external factors like the COVID-19 and geopolitical changes.There is a need for increased focus on climate change adaptation and sustainable resource management.
Prioritise comprehensive and integrated approaches to address complex regional challenges.Enhance support for cultural heritage projects to meet the need for funding in this sector.Make the measures adaptable to changes of the context, including pandemics and geopolitical shifts.Put greater emphasis on sustainability and climate change adaptation in regional development strategies.Ensure continuous monitoring and evaluation to ensure the effectiveness and efficiency of the measures.Develop new funding mechanisms to support areas with underinvestment and ensure the durability of project results.
Survey in 2016 of the satisfaction of Monitoring Committee members with the work of Committees in 2007-2013
A monitoring bizottsági tagok elégedettségének felmérése, 2016
The evaluation examines the satisfaction of Monitoring Committes (MCs) members of 2007-2013 OPs in Hungary.
To assess the satisfaction level of Monitoring Committee members in relation to its effect on OPs, to help MAs to strengthen the partnership process for 2014-2020 and establish the baselines for the relevant of the Public Administration and Civil Service Development OP.
Overall, members of MCs are satisfied with their impact on OPs.Members with voting rights are more satisfied than members of civil society organisations.Dissatisfaction is expressed mainly with the time available for discussion and the prior availability of material.Civil society organisations feel little involved sometimes, seeing MCs mainly as part of a dialogue between MAs and the Commission.Informal contact time has a high value.
There are some limitations in sampling procedures.
Raise the professionalism of members of MCs and increase informal contact opportunities.
Contribution to innovation and competitiveness support under the Sachsen-Anhalt ERDF and ESF OPs, 2014-2020 – Thematic report on scientific institutions
Beitrag des EFRE und des ESF zu Innovation und Wettbewerbsfähigkeit - Teilbericht "Wissenschaft"
OP Sachsen-Anhalt ESF 2014-2020; OP Sachsen-Anhalt ERDF 2014-2020
Marco Baldauf, Dr. Thorsten Lübbers, Miguel Riviere, Johanna Wittenberg
Ramboll, Chilehaus C - Burchardstraße 13, 20095 Hamburg, de.ramboll.com
The evaluation examines 4 measures under the ERDF and the ESF Sachsen -Anhalt OPs: Development of application-oriented public R&D infrastructure (IP 1.1); Application-oriented R&D activities in universities and research institutes and incentives for top-level research (IP 1.1); Development of the economic innovation infrastructure (IP 1.2); Internationalisation of higher education institutes (IP 10.ii).
To assess the contribution of the measures to support innovation and competitiveness, the relevance of the strategy and the appropriateness of its implementation.
Desk analysis of programming documents, monitoring and statistical data.Workshops with experts to identify the theory-of-change.8 interviews with experts and managing bodies.15 case studies.
Given the socio-economic situation in Sachsen-Anhalt (low public and private expenditure in R&D, high demand for skilled workers), the four measures are strategically relevant.The output indicator targets are expected to be met or exceeded in almost all cases by 2023 and there are few problems of implementation.The cost of submitting applications is judged by a majority to be significant, but in line with those of other funding programmes.The requirements for funding from the Investment Bank are often viewed critically (unclear communication, change of requirements in the course of funding, long processing periods).Several projects have already yielded scientific results in terms of high-profile publications and projects funded under the "Internationalisation at Universities" are already contributing to scientific education and training.The projects supported increase the visibility of scientific institutes funded and will enable them to raise significant amounts of third-party funding, both public and private.The projects supported provide significant impetus for the expansion of the knowledge base and the further development of innovative products and services for companies cooperating with the institutes concerned.The effects on the regional economy are limited by many of the companies concerned being based in another Land.
Continue the measures in the next programming period.Examine the possibility of obtaining ERDF financing for the “Internationalisation in higher education” measure, given its strong technical and scientific focus and its contribution to application-oriented R&D.Emphasise that regional companies should be actively involved in the design and implementation of projects.The new infrastructure supported and the results of R&D financed measures should be communicated more to the regional economy.Optimise administrative procedures (e.g. grouping all necessary forms and supporting documents into one application document).
The effects of training in 2015 under the Piemonte ESF OP, 2014-2020
Gli effetti dei corsi erogati in Piemonte nel 2015
Luisa Donato, Maria Cristina Migliore, Carla Nanni, Samuele Poy, Igor Benati, Elena Ragazzi, Lisa Sella
This evaluation assesses the impact of training courses carried out in Piemonte in 2015 and co-financed by the ESF, 2014-2020.
To assess the impact of training in 2015 on the employment status of beneficiaries of ESF support.
Desk analysis;statistical analysis;counterfactual analysis (Propensity score matching).
31% of beneficiaries (9,426) were employed one year after the completion of training. For training courses to integrate people into employment (5,045 beneficiaries), 35-38% were in work one year after.The probability of the supported group being in work was 12- 20 percentage points higher than the control group.The largest positive effects were for training courses for graduate and those in the health sector, but a positive impact was found for all courses.The impact was largest for: young people, those unemployed for under 12 months and those on long training courses. No differences were found between men and women.
Evaluation of measures for participation of women in employment under the Sachsen-Anhalt ESF OP, 2014-2020
Bewertung des SZ 3 - Evaluation Frauenspezifischer Projekte
Friederike Scholz, Nina Schwarz, Johanna Washington - Ramboll
The evaluation examines measures to support the employment of women.
To assess the ESF measures to support the employment of women in the region.
Official statistics.Monitoring data.Interviews with experts.Surveys.Case studies.
The interventions have achieved positive results, especially in terms of career planning for women in technical and scientific professions, helping to increase equality between men and women.The indicator system is not sufficient to give a complete picture of the improvements in the working situation of women.The projects have encouraged participants to take further steps in education and achieve higher qualifications.Some of the measures have reached more target recipients than planned, including beyond the region’s borders.
Only phone interviews could be conducted because of COVID-19. For some aspects data are partial and not useful to answer all evaluation questions.
Extend the target group of recipients to younger people to strengthen the vocational orientation of measures.Exploit synergies between the various measures to increase their final effects.Ensure the accurate and timely collection of data by beneficiaries to improve the monitoring and evaluation of measures.Introduce monitoring indicators better able to describe the measures implemented.
Impact evaluation of the YEI in Portugal – preliminary analysis
Avaliação do Impacto da Iniciativa Emprego Jovem (IEJ) - Relatório Intermédio Revisto
OP Social Inclusion and Employment 2014-2020
IESE (Instituto de Estudos Sociais e Económicos)
This follows up the process evaluation of January 2018 and covers the period Sept 2013-July 2019. It is a preliminary to a proper impact evaluation planned for April 2020. YEI in Portugal is implemented under the Social Inclusion and Employment OP and provides support for traineeships (80% of the YEI budget), recruitment, education and training and entrepreneurship.
To assess the impact of the YEI programme.
desk research;data analysis (monitoring data, statistical data, administrative registers);interviews;workshop;survey of former YEI participants.
Overall, implementation in financial term is progressing reasonably.There are difficulties in reaching inactive NEETs and implementing the qualification/education measures.The participant profile broadly corresponds to NEET characteristics (mainly women, older with high education) but LTU and other disadvantaged groups are under-represented.70% of unemployed YEI participants completed the programme, 45% received an offer of employment, continued education or apprenticeship/traineeship, and 40% continued their studies or were working immediately after exit. Half of participants were in employment after exit; but only 19% for those in employment incentives programmes.After 6 months, 71% of participants were employed (2023 target of 40%), but only 0.7% were self-employed (target 10%).Without the ESF, the national funding allocated to YEI interventions would have been lower. YEI provides tailor-made support for target groups who would not otherwise be supported. The implementation of the Youth Guarantee/YEI helped to speed up and expand PES activities (e.g. creation of a specific position in Job centres to assist companies and universities).
Only preliminary results are presented in the report. Due to data issues, most findings only relate to ‘Recruitment incentives’ and ‘Traineeships’.
No policy implications drawn – postponed until final report.
Evaluation of the "Strengthening the Social Labour Market" measure supported by the Hamburg ESF OP, 2014-2020
Wirkungsanalyse zum Landesprogramm "Den sozialen Arbeitsmarkt stärken"
Thorben Rohwer-Kahlmann, Marcus Neureiter, Simon Böhmer
Ramboll Management Consulting GmbH, www.ramboll.de/management-consulting
The evaluation examines the results of 8 projects financed under the measure, which is aimed at improving social participation and employability of those far from the labour market. Data are up to end-2020 or mid-2021. This is a first analysis of a broader impact evaluation.
To analyse the types of target groups reached, the implementation of the 8 projects examined, aimed at improving the social participation and employability of those far from the labour market and the effects on employment.
Analysis of monitoring data.Analysis of data on participants collected by project promoters.Data from the Federal Employment Agency and Job Centre Hamburg.Survey of participants (166 respondents, 50% response rate).Five interviews with official institutions and an association.Survey of relevant urban district institutions (21 respondents, 64% response rate).
Cooperation between project promoters and job centres was described as result-oriented and satisfactory by institutions and associations involved.In 2020, a total of 339 people took part in the employment activities of the 8 projects, in line with the planned target.The participants were largely satisfied with support from project promoters and the work experience provided, which had a large degree of flexibility both in terms of working time and content.The work experience is reported to have helped improve social and professional skills (75% of respondents), motivation for work (85%) and chances to access the labour market (50%).For over 70% of participants, employment has also had a positive impact on their living situation and social participation.According to the survey, projects have benefited urban areas by strengthening local cultural and support services, improving their attractiveness and strengthening social participation and cohesion.
The early stage of implementation of projects limits assessment of the effects (projects started in 2019 with a possible duration of 5 years). There are also no monitoring data for two projects.
Final evaluation of the "Fit for Work - Opportunity Training" measure, financed under the Bayern ESF OP, 2014-2020
Abschlussbericht. Evaluation von "Fit for Work - Chance Ausbildung"
ISG - Institut für Sozialforschung und Gesellschaftspolitik GmbH, https://www.isg-institut.de
The evaluation assesses the impact of the “Fit for Work” vocational training programme targeting disadvantaged young people.
To assess the effectiveness of the measure, the characteristics of participants, the cost borne by beneficiary companies, and the contribution of the measure to establishing and stabilising the training relationship, as well as its medium and long-term effects.
Desk research on programme documents and scientific literature.Quantitative analysis of statistical and monitoring data.Theory-based approach.Interviews (2 with Land officials involved in coordinating and implementing the measure, 20 with beneficiary companies).Online survey carried out in 3 waves (2019, 2020, 2021) targeting beneficiary companies (29% response rate).Mixed (online/post) survey carried out in 5 waves (2019 to 2023) targeting training participants (14% response rate).
93% of the available budget has been spent.Companies supported amounted to 102% of the target.Women were under-represented among participants (1 in 4) and half of all participants had a migrant background.93% of companies supported are SMEs and 83% of companies received grants only once, because smaller firms have fewer trainees.The measure reached a large proportion of companies in search of trainees that have limited resources to devote to training.The significance of the grant is inversely proportional to the size of the company supported.The amount of grant is appropriate: it has a limited stabilising effect, but more funding would not achieve greater stabilisation.Those completing the training exceeded the target, and the subsequent transition to the general labour market was successful for most trainees.The grant has had positive effects for both companies with a need for trainees and potentially disadvantaged apprentices who risk remaining excluded from training.
There is a difference between the sample surveyed and the population due to selection bias, which is compensated by using a logistic regression and by examining ESF monitoring data for all participants.
Only target support at SMEs.The categories of eligible disadvantaged could be expanded.Consider the possibility of funding part-time training.The funding period could be shortened and the monthly grant increased.Reinforce communication campaigns targeting potential beneficiary firms.Modernise the website and optimise the search engine.
Evaluation of the effects of Territorial Cooperation and European Neighbourhood programmes, 2014-2020, on regional development in Poland
Wpływ Programów Współpracy Terytorialnej Polityki Spójności i Polityki Sąsiedztwa na rozwój Polskich Regionów w perspektywie 2014-2020
Poland-Russia ENI CBC Programme 2014-2020; ENI CBC Poland – Belarus – Ukraine Programme; Interreg V-A-Poland-Slovakia 2014-2020; Interreg V-A-Poland-Denmark-Germany-Lithuania-Sweden (South Baltic) 2014-2020; Interreg V-A-Poland-Germany/Saxony 2014-2020
Beata Belica, Przemysław Bruhn, Cezary Gołębiowski, Anna Górecka-Ojdana, Paulina Kowalczyk, Katarzyna Lipczyńska, Michał Niebylski, Adam Rybkowski, Weronika Zych
The evaluation assesses the implementation and effects of twelve European Territorial Cooperation programmes in Poland.
To assess the impact of the programmes on the integration of border areas and on the improvement of institutional capacity to undertake activities boosting economic and social cohesion between Member States.
Desk research (legal acts, programme documents, previous evaluations).Analysis of monitoring and statistical data.32 interviews with representatives of institutions implementing programmes.Survey of 1,040 Polish project partners.28 case studies.Analysis of cooperation intensity using the isoline method.Network analysis.12 dyads/triads interviews with representatives of the evaluated programmes.Delphi method with 3 experts.Structuring workshop.
The sustainability of project results can be preserved long after completion, especially for infrastructure projects, although further funding may be required.The support was mainly used by local government units, their unions and NGOs, with a significant proportion of partners from cities with county rights.Small/micro-projects under the Small Projects Fund have contributed to increasing cooperation, cultural integration and the tourist attractiveness of border areas.The simplification of the accounting procedure for small/micro-projects and the use of advance payments are particularly beneficial for entities with limited financial capacities, such as small municipalities and NGOs. However, these aspects are not sufficiently publicised.Crises such as the COVID-19 pandemic, the war in Ukraine and rising inflation have significantly impacted project implementation.
It was not possible to estimate the impact of the programmes on the territorial cohesion of the regions, as they were influenced by a number of Cohesion Policy instruments, often of a much larger dimension than the intervention evaluated.
Maintain support for cross-border cooperation programmes.Promote activities within the two border counties, particularly in rural or urban-rural municipalities, by providing additional bonus points for such projects.Continue to support small/micro projects in the 2027+ period, while maintaining their scope.Continue the possibility to grant advances to beneficiaries of small projects in the 2027+ perspective.Make information on advance payments and other financial support more visible for applicants.Develop a set of procedures to facilitate the implementation of projects by partners in case of unexpected events.
Impact of the measures implemented in respect of culture under the Infrastructure and Environment OP, 2014-2020, in Poland
Wpływ działań podejmowanych w ramach VIII osi priorytetowej PO IiŚ 2014-2020 na osiągnięcie celu szczegółowego jakim jest lepsza dostępność infrastruktury kultury i dziedzictwa kulturowego oraz wzrost kompetencji kulturowych społeczeństwa jako ważnych elementów konkurencyjności gospodarki
J. Szczucki, M. Gajewski, H. Kalinowski, R. Kubajek, E. Kusideł, A. Matejczuk-Rosa, M. Ośka, B. Pietrzak, A. Rzońca, J. Witkowska
Policy & Action Group Uniconsult, www.pag-uniconsult.pl;Imapp Consulting, www.imapp.consulting
The evaluation examines cultural infrastructure projects implemented under the OP.
To assess the impact of support on the availability of cultural infrastructure and cultural heritage and the cultural competences of society.
Desk analysis of relevant documents.Analysis of monitoring data and official statistics.Econometric analysis.CAWI/CATI survey of 93 beneficiaries and 26 local government units.Interviews with implementing bodies and beneficiaries.Case studies.Workshops (design thinking).
The programme supported 404 cultural and cultural heritage sites, which constitute 104% of the target value.Beneficiary sites increased visits by 4.3 million annually.One in three projects resulted in an increase in social activity of residents.The impact of completed projects on the financial situation of cultural institutions is uncertain and depends on the situation of a given institution and the scope of the project implemented.
Place more emphasis on supporting activities aimed at improving the skills of beneficiary staff.Ensure that projects are prepared and implemented with greater public participation.Conduct an analysis of the usefulness of "pro-accessibility" solutions several years after the projects.
Final report. Evaluation of the Fachkräfteprogramm
Abschlussbericht. Evaluierung des Fachkräfteprogrammsdes Landes Nordrhein-Westfalen.
Evaluation of the effects of measures for the employability of vulnerable people and the social economy under the Trento ESF OP, 2014-2020
Valutazione degli effetti in termini di miglioramento dell'inclusione sociale e della partecipazione al mercato del lavoro degli interventi a favore delle persone maggiormente vulnerabili
The evaluation examines the effects of measures carried out from 2015 to 2020 to increase employability of the most vulnerable and to strengthen the social economy under the OP.
To assess the main features of projects, beneficiaries and final recipients and the effects of the measures implemented from 2015 to 2020.
Monitoring data.Official statistics.Administrative data (hiring registers - “Comunicazioni obbligatorie”).Survey of 23 beneficiaries (87% response rate).3 Case studies.
749 vulnerable people were reached by measures to increase their employability between 2015-2020, most of them men, unemployed and with low education.13% of recipients were in a job placement pathway six months after entering a measure and 26% employed (33% 1 year after).The main results are increased self-confidence and motivation of recipients rather than increased skills.Measures for the social economy seem to have been less effective and participation has been limited.Factors that improved the effectiveness of projects are cooperation with partner networks, individualised support activities, traineeship in a third body, and the issue of competence certification, though this last has been used to a limited extent.
Survey results need to be read with caution, as responses may be affected by self-selection bias.
Include a specific position charged with supporting activities to complement training.Ensure more flexibility for beneficiaries to manage training plans and final recipients, including with the help of a system for monitoring the soft skills acquired by recipients.Strengthen collaboration with local actors in the planning phase.Reward scouting activities of companies hosting trainees.Integrate measures for employability and the social economy more.
Analysis of EU investments in digital connectivity in Lithuania to prepare for 2021-2027
Išankstinis ES fondų investicijų į skaitmeninio junglumo gerinimo veiklą 2021–2027 m. finansavimo laikotarpiu vertinimas
JSC Smart Continent LT, T. Kosciuškos g. 24-104, LT-01100 Vilnius, Lithuania, www.smartcontinent.com, lt@smartcontinent.com
This is an analysis of EU investments in gigabit connectivity, which will feed into the National and Regional Broadband Plan.
To assess the needs for EU investments to meet the EU gigabit connectivity targets; to identify measures to meet the demand for very high capacity networks; to define the possibilities and means of providing technical and expert assistance to strengthen local capacity; and to make recommendations for the development of the National and Regional Broadband Plan.
Analysis of secondary data.Legal analysis.Analysis of statistical and administrative data.Analysis of intervention logic.10 interviews.CAWI surveys of operators.Focus group.
The ERDF-supported RAIN3 project is expected to lead to a broadband network covering 9150 km2 of ‘white areas’ and connecting 254,000 households.Other projects supported by the Structural Funds OP 2014-2020 have helped to upgrade public Internet access points, mainly in rural libraries, and the development of ICT skills of library staff and people generally.The Digital Agenda targets for 2020 – providing Internet access of at least 30 Mbps to all households and of 100 Mbps to 50% of households – will not be met.In 2014-2019, general broadband penetration in Lithuania increased by 10 p.p. and that of at least 100 Mbps by 26 p.p., while the gap in coverage between urban and rural areas was reduced.EUR 156 million investment is needed to cover all households and TEN-T roads with 100 Mbps connection.By 2022, private investment in broadband infrastructure is estimated to be up to EUR 7.2 million.
Use EU and national investments to develop broadband in ‘white areas’.Initiate a review of the Regulations for the installation, marking, maintenance and use of electronic communications infrastructure.Prepare a detailed investment plan for covering ‘white areas’.Simplify the requirements for the construction of communication towers and for the use of land to install broadband infrastructure.Inform the general public about 5G technology so as to dispel existing myths.Continue providing Internet connections in public institutions and promote the use of the Internet.Harmonise the data collection methods for state interventions with that used by the Communications Regulatory Authority.
Evaluation of proposed allocation of the performance reserve of the Molise ERDF ESF OP, 2014-2020
Valutazione della proposta di attribuzione della riserva premiale
Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Molise
The evaluation examines the proposal for allocating the performance framework reserve of the Molise ERDF ESF OP, 2014-2020.
To assess the proposal by the MA of how to allocate the performance framework reserve.
The proposed reprogramming increases thematic concentration in line with EU regulations.The decision to allocate the ERDF reserve to PA2 (ICT) seems justifiable, because of its good governance and persisting regional needs.The decision to allocate the ESF reserve to PA6 (Education) seems justified by its absorption capacity.
Concentrate ICT interventions on regional competitive advantage, especially with a view to 2021-2027 programming;Plan interventions in favour of business creation for the long term unemployed.
Ex-post evaluation of the support to transport under the Infrastructure and Environment OP, 2007-2013
Badanie ex-post wsparcia sektora transportu w Programie Infrastruktura i Środowisko 2007-2013
Joanna Obarymska-Dzierzgwa, Edyta Boratyńska-Karpiej, Aleksander Wołowiec
Centrum Unijnych Projektów Transportowych, Plac Europejski 2, 00-844 Warszawa
This evaluation assesses the long-term impact of 2007-2015 transport projects carried out under the Infrastructure and Environment (IE) OP. The effectiveness, efficiency, consistency and complementarity of transport projects are assessed, taking account of the specific objectives of the OP.
To assess implementation of the infrastructure interventions undertaken under the IE OP.To assess the impact of the projects carried out on the objectives set out in the National Strategic Reference Framework.
Analysis of project documentation (applications for funding, feasibility studies, etc.).Web research (additional information on investments).CAWI, interviews with: General Directorate of National Roads (GDDKiA), Fire Departments, Police, road and rail associations, businesses, local authorities, Provincial and Marshall's offices, airport managers, civil aviation authorities, the Rail transport Office, the Polish air navigation Agency, and those responsible for managing special economic zones and industrial parks.Focus groups.Case studies.Spatial analysis using GIS software.Computer simulations.Analysis of travel-time and traffic flows.Monitoring data.Eurostat data.
Much of road network has been developed, particularly in relation to major routes (NSDC 2030 and TEN-T).The number of road accidents has fallen, though the costs that they give rise to have increased.The infrastructure of Polish seaports is still poor and does not meet most of the maritime transport requirements.The provision of air transport services has led to increased economic activities.
There is a need to place more emphasis on reducing bottlenecks, both in certain agglomerations (e.g. Warsaw) and between them.The conditions and opportunities need to be created for obtaining external funds to ensure further development of road safety measures.Support for investment in maritime transport and seaports needs to continue.There is a need to optimise the implementation of rail investment and to ensure a stable medium and long-term financial framework for the maintenance of rail infrastructure.
Evaluation of the implementation of the Baden-Württemberg ESF OP, 2014-2020
Evaluationsbericht zur regionalen Umsetzung des ESF in Baden-Württemberg
Christian Loschelder, Uta Micic, Kai Sattler, Dr. Friedrich Scheller
ISG Institut für Sozialforschung und Gesellschaftspolitik GmbH. Weinsbergstr. 190, 50825 Köln. Contact person:. Gerald Engasser (Head) Tel: 0711 123-3614, E-mail: ESF@sm.bwl.de
The evaluation assesses the implementation of the Baden-Württemberg ESF OP 2014- 2020.
To assess the implementation of the OP and the progress towards achieving the targets set, focussed on measure B1.1 (to improve employability of groups particularly at risk of poverty and exclusion) and measure C1.1 (to prevent school drop-out/improve school ability of young people).
Desk analysis;interviews with experts, representatives of the Managing Authority and regional working groups (Arbeitskreisen-AK);two case studies;analysis of the working group strategies and applications for support;analysis of monitoring data;survey of regional actors.
General findings:Up to 2017, 525 projects were launched in the region, 327 under measure B1.1 and 198 under measure C1.1. Projects were managed by regional working groups, each of them dealing on average with 8 projects for measure B1.1 and 5 for measure C1.1.90% of the projects included horizontal principles (equal opportunity and non-discrimination) in their objectives.Administrative requirements are seen as too burdensome.The promotion of knowledge transfer and the use of new methods/instruments to reach specific target groups are seen as innovative elements (in particular for B1.1).The regional distribution and composition of working groups enabled relevant local cooperation.Measure B1.1:14,326 people participated in projects of which 68% were long-term unemployed and the others disadvantaged.Participants are almost equally divided between men and women;Participants went through job centres to access projects.Two thirds of participants received collective support and the rest individual coaching.After participation, 42% of the long-terms unemployed (and 49% of those in particularly disadvantaged) were in employment, in vocational training or had obtained qualifications.Measure C1.1:22,289 people participated, over 60% of them men.88% were under 25, mostly young people who prefer to work, interested in changing school and refugee minors.Projects were often implemented in schools.
It is mentioned that results of surveys of beneficiaries of the two measures are not included and will be analysed in a separate report.
The measures supported proved to be successful in terms of objectives pursed, organisation and services provided and should be continued in the future. Simplification of administrative requirements might also be useful.
Evaluation of the communication strategy of the OP Human Capital 2014-2020
Avaliação da Estratégia de Comunicação do POCH
Intermediate evaluation of the Research and Innovation Strategy for Smart Specialisation (RIS3) of the Castilla y León ERDF Programme, 2021-2027
Informe de evaluación intermedia de la RIS3 de Castilla y León 2021-2027
The mid-term evaluation of the RIS3, 2021-2027, focuses on three aspects of the RIS3: design, implementation and results.
To assess the coherence of RIS3 objectives with national programmes; the execution, governance, and sustainability of the measures; and the progress of indicators.
Document analysis. Analysis of monitoring data of RIS3 specific indicators. 60 in-depth interviews with key actors of the R&I local ecosystem. Three targeted questionnaires of companies, researchers and civil society members. Eight working groups to review the results of the ex-post evaluation of RIS3 2014-2020 and the mid-term evaluation of RIS3 2021-2027, involving the RIS3 Technical Management Group, the RIS3 Coordination Committee, the people responsible for the Digital Agenda, and Emblematic Initiatives in Aerospace, Cybersecurity, Health, and Bioeconomy.
The 2021-2027 RIS3 is coherent with national and international R&I frameworks. The RIS3 context indicators show mixed progress: R&D investment initially grew but may fall short of 2024 targets. While innovation rates and tech sector employment remain promising, high-tech export and internet coverage targets appear overly ambitious. Improvements in digital competencies and engagement with public administrations are on track. Impact indicators show positive trends in scientific publication rates, digital competencies and innovation performance. The strategy currently focuses on too many priorities to enable the effective monitoring of selected areas of strategic importance. Nearly all results indicators improved in 2022 compared to 2021, and this suggests greater effectiveness of the projects.
Simplify the RIS3 strategy’s objectives and areas of action and make them easier to monitor. Advance transformative roadmaps by refining Emblematic Initiatives measuring and establishing them as central to priority development within an open entrepreneurial discovery process. Update the regional specialisation priorities by incorporating new concepts and areas. Adjust 2027 context indicator targets for greater precision, realign financial planning based on initial execution and growth estimates, and update the RIS3 Management Guide's analysis to include new actors.
Evaluation of Communities for Work 2014-2020
Evaluation of Commuities for Work - Stage 2: Process and Outputs Evaluation Report
This evaluation assesses the Communities for Work (CfW) programme, jointly funded by the Welsh Department for Work and Pensions (DWP) and the ESF.
To assess how the programme has been set up and operates and its effectiveness.
Desk analysis of programming documents and previous evaluations.Interviews with policy makers; stakeholders; training providers and participants.Development of a theory of change.Online survey of delivery staff.Case studies of single participants.
The number of participants as at February 2017 was in line with projections but falls short in relation to the programme’s 5 year life-span.The number of participants progressing into employment was in line with the planned objective, with the exception of young NEETs.Addressing the barriers to NEETs could result in huge economic and social returns.It is uncertain how local context will affect delivery and relationships between CfW teams and partner teams.The rationale for targeting particular groups is well established and justified by the fact that not all groups require the same support.
The arrangements for working with local Job Centres need to be strengthened.An internet presence for CfW needs to be developed.Adviser caseloads need to be reviewed to ensure that clients are progressing.Plans for engagement with other partners need to be drawn up.
Ex-post evaluation of Cohesion Policy in 2007-2013 in Pomorskie
Ewaluacja ex post efektów polityki spójNości 2007-2013 z perspektywy regionalnej na przykładzie województwa pomorskiego
Evalu Sp. z o.o. ul. Dzika 19/23 lok. 55 00-172 Warszawa, WYG PSDB Sp. z o.o. ul. Bitwy Warszawskiej 1920 r.7 02-366 Warszawa
The evaluation focuses on the effects of Cohesion policy 2007-2013 in the Pomorskie Voivodeship (PL). Projects covered were funded through the Regional Operational Programme, the OP infrastructure and environment, the OP innovative economy and the OP Human Capital. The research focuses on the impacts, the effectiveness and the sustainability of the ERDF- and ESF-funded projects on regional economic competitiveness, living conditions (social infrastructure, natural environment and resource efficiency) and transport accessibility.
The evaluation is aimed at measuring the impact, effectiveness, efficiency, sustainability and scale of Cohesion policy programmes on:competitiveness of enterprises;employment and human capital development;social infrastructure;the natural environment and resource efficiency;transport accessibility.
The report is based on:analysis of monitoring data;CAWI/CATI and personal interviews;case studies;econometric models; counterfactual impact evaluation (Generalised Propensity Score Matching);desk research.
Economic competitiveness:there was an overall positive impact of Cohesion policy interventions on enterprises competitiveness and employment; cluster initiatives were supported but proved to be mostly unsustainable;the impact on business innovation is not clearly positive. Expenditure on industrial innovation (mostly in large and medium-sized enterprises) increased but the proportion of innovative companies declined;interventions contributed to the decline of the regional unemployment rate;there was a positive effect on the competences and skills of workers.Living conditions:access to social infrastructure and services (Culture, Education, Health, urban regeneration, sport and leisure, ICT) in weak areas was significantly improved;initiatives in healthcare and for urban regeneration had positive impact on employment;e-services are still not sufficiently developed;cohesion policy interventions were crucial for water and wastewater management;there was a positive impact on flood protection, protection of natural assists and waste management;projects supporting energy production from renewables did not fully meet the targets. Transport:road and rail transports was improved but need to be better inter-linked;investment in urban public transport significantly increased the number of passengers;ports need further investments.
Initiatives for business support lack common indicators.National data from PUP do not distinguish those benefiting from ESF measures.Complete data on enterprise own spending on developing human capital are missing.Indicators on projects relating to waste collection and improving energy efficiency are not fully available.
Economic competitiveness:strengthen support for innovation in small and micro-sized enterprises and R&D start-ups; reward selection criteria for projects on R&D and involving cooperation with research units;implement projects in areas of low economic activity;foster intervention on economically inactive people;assess the quality of training interventions in enterprises;implement job rotation (especially in small enterprises); intensify efforts to make the social economy sector non-reliant on public support.Living conditions:create common, interdisciplinary cultural products; adapt education to the needs of the labour market;promote social action in urban regeneration projects;coordinate e-Government activities better;avoid duplication of support for renewable energy.Transport:foster complementarity between projects and the links between different modes of public transport.
Second Interim Evaluation of the Employment, training and education ESF OP, 2014-2020
2-A Evaluación Intermedia del Programa Operativo de Empleo, Formación y Educación, anualidad de 2019
OP Employment, training and education ESF 2014-2020
This is the second on-going evaluation of the Employment, Training and Education OP, covering all of Spain and providing support to employment (TO8), social inclusion (TO9) and education (TO10).
To assess the implementation of the OP at the end of 2018, the results achieved and the first impact of the OP and to suggest possible improvements.
Documentation analysis13 interviews with MA and IBsOnline survey of beneficiaries and managing agencies (186 responses)2 case studiesAnalysis of monitoring dataSurvey of final recipient satisfaction (1068 out of 614,476).
The OP strategy is consistent with needs in respect of employment and educationAt the end of 2018, 79% of funding was allocatedAdministrative procedures absorb an excessive amount of effort and it is difficult to reach the expected number of people in less developed regionsFor employment indicators, women show better performance, while for education, it is the reverseThe employment status of participants is worse after 6 months than immediately after completion, implying a lack of sustainability of outcomesGender and the region of residence are not correlated with an improvement in the employment situation after 6 months. Increases in age are associated with declining probability of the employment status being improved. The long-term unemployed show better results, but ethnic minorities and those with disabilities are less likely to show an improvement in their employment situation.
Lack of detailed financial information limited the efficiency analysis, telephone contacts of organizations were not always available and it was not possible to obtain the data needed to carry out a counterfactual analysis.
Carry out a follow-up for PA 2Monitor the activities of IBsMaintain a similar OP structure in the futureEstablish a "roadmap " to improve the management of the OPDevelop employment and training measure that focus on sustainability over timeEncourage operations aimed more directly at older peopleContinue the fight against early school leaving as a transversal aimIntegrate ESF measures for people with disabilities with other measures in the countryCollect data for a counterfactual analysis.
Evaluation of the implementation of horizontal policies under the Świętokrzyskie OP, 2014-2020
Ewaluacja wdrażania polityk horyzontalnych w ramach RPOWŚ 2014-2020
Maciej Maj, Jolanta Maj, Michał Niebylski, Oliwia Mazan, Joanna Jaworska, Weronika Walo, Patrycja Klimek
Openfield Sp. z o.o., 45-057 Opole, ul. Ozimska 4/7, e-email: office@openfield.pl, www.openfield.pl
The evaluation examines the measures taken to incorporate the horizontal principles into the Świętokrzyskie OP, 2014-2020.
To assess the implementation of horizontal principles under the Świętokrzyskie OP and their effects.
Desk analysis.CAWI/CATI survey of 327 beneficiaries and 383 participants in projects.30 interviews with institutions, experts, beneficiaries and participants.Case studies.Workshop with the Marshal’s Office to discuss recommendations.
2,9% of funding is allocated to IPs which have an impact on the principle of sustainable development.37% of funding is allocated to IPs which have impact on social cohesion and reducing inequalities principle.The project application and selection system ensures the compliance with the horizontal principles. Projects demonstrating a negative impact on horizontal principles are excluded from the process.The implementation of horizontal principles is monitored by indicators to track progress.
Increase the potential of members of the Monitoring Committee by trainings and improve communication.Focus the training of MA personnel, beneficiaries and recipients of support more on horizontal policies.Ensure that the selection criteria reflect to greater extent sustainable development issues.
Evaluation of Action 4.2 "Support to families for the payment of university fees" and of Action 4.3 "Scholarships for deserving students and those in financial difficulty" supported by REACT-EU, under the Research and Innovation OP, 2014-2020, in Italy
Rapporto di valutazione PON Ricerca e Innovazione 2014-2020, all prova del diritto allo studio
Valeria Aniello, Maria Coronato, Tommaso Pizzella
The evaluation assesses the implementation and outcomes of two measures (4.2 and 4.3), which aim to promote the right to education for economically disadvantaged university students. These measures are financed through REACT-EU, under the OP.
To assess the relevance, coherence, efficiency, effectiveness and EU added value of the REACT-EU measures aimed at disadvantaged university students.
Literature review and desk analysis of both measures. Theory-based approach. Analysis of statistical office data from the Ministry of Education, from Anvur (National Agency for the Evaluation of the University and Research System) and the Right to Study Observatory. Analysis of monitoring data. 17 interviews, concerning the measure 4.3, with EDISU (Regional Bodies for the Right to Study) implementing bodies (10 interviews), with beneficiaries of scholarships (5), and with monitoring officers (2). 7 focus groups with EDISU representatives from each Region where the measure 4.3 was implemented. 3 focus group with beneficiaries of the scholarships of the measure 4.3
Measure 4.3 was targeted at Regions that had not yet achieved full coverage of eligible scholarship recipients, as at 2021 (Abruzzo, Molise, Calabria, Sicilia). Beneficiaries were selected through objective merit and income-based criteria. Despite delays, the measures expanded access to education and increased scholarship amounts, enabling over 12,000 students to benefit from additional REACT-EU resources. Key success factors included strong coordination between regional and central administrations and an efficient existing governance system. According to interviews, REACT-EU resources were additional and essential to expanding beneficiary coverage and ensuring scholarship amounts. A key concern is the lack of clarity about future funding, despite EDISU's strong coordination efforts.
Ensure clarity on whether support can be extended under the next programming period. Establish transitional mechanisms to preserve institutional practices, if the initiative terminates.
Mid-term evaluation of the ESF, 2014-2020
Midvejsevaluering af den danske socialfond 2014-2020
OP Education and Entrepreneurship 2014-2020
The interim evaluation focuses on four areas of interest (three ESF priorities) in the ESF 2014-2020 in Denmark.
To assess the contribution of projects to meeting the targets set for each priority, including the specific objective.
Desk and data analysis.Quantitative analysis.Interviews with interested parties.
In all priority axes, the indicators identified do not express the extent to which projects provide improvements for participants.Project managers have encountered difficulties in making the small changes needed to the projects.64% of the funding for priority 1: Entrepreneurship and job creation is committed, 49% for priority 3: Social inclusion and 48% for priority 4: Education. The first priority is likely to be fully implemented during the programme period, while priorities 3 and 4 require greater effort to attract the necessary applications to do so.Priority 1.1a (entrepreneurial consulting): there are difficulties in achieving the objectives in many projects and some overlap with projects under Priority 2 of the Regional Fund "Other growing companies".Priority 1.1b (teaching entrepreneurship): the planned target is probably too optimistic and little importance is given to such training in the school system.Priority 1.2 (skills development): the activities are well adapted to the needs of companies.Priorities 3a (Youth education for young people with special challenges) and 3b (Inclusion through social economy enterprises): they require a long time to produce an impact. There seem to be difficulties in managing these projects, due to continuing reforms in education and employment system.Priority 4.1 (better professional training): measures are relevant, but have little influence on the choice of education of the target group.Priority 4.2 (growth of competences): collaboration between businesses, municipal work centres and education institutions is an important prerequisite for improving the work force, but it has been difficult to establish in many cases.
There is a need to:introduce project performance indicators;give greater autonomy and responsibility to project managers;create networking that motivates participants;develop an entrepreneurial culture in the education system;avoid, under priority 1.1a, overlap with priority 2 of the Regional Fund "Other growing companies";define better the target group under priorities 3a and 3b; make more visible the added-value of projects under priority 3a;extend the target group under priority 4.
Evaluation of the Modernisation of Vocational Education (MOV) project under the Research, Development and Education OP, 2014-2020, in the Czech Republic - Final Report
Evaluace individuálních systémových projektů podpořených z PO 3 OP VVV-II Část II: Evaluační okruh B – Evaluace projektu MOV, Závěrečná zpráva
Křižíkova 2987/70b, 612 00 Brno, https://navigae.cz/kontakty/
The evaluation examines the project, which aims to increase the quality of vocational training and to promote employability of graduates. This final report concerns the assessment carried out in 2021-2022, and is a follow-up of three previous interim reports.
To assess the project’s implementation, the extent of achievement of objectives, the effectiveness of the exchange of experience among schools and stakeholders, the awareness of the National Register of Qualifications (NRQ) by potential users, the cooperation with other projects and the unintended effects.
Analysis of documentation and monitoring data.Survey of teachers involved in the project (92 responses out of 398) and representatives of all secondary vocational schools in the country (256 out of 863).
The project objectives have been achieved and the indicator targets met or exceeded.According to the survey, around 81% of respondents apply the knowledge obtained in practice, mostly in teaching, adapting teaching texts, and workbooks.Almost 50% of respondents share the knowledge obtained with their colleagues and employers.Over 50% of respondents use the NRQ and find it useful or quite useful, 33% do not use it.According to the teachers involved in the project, the main benefits of the project were in the proposal for planning, organisation and evaluation of the implementation of practical teaching, optimised School Education Programmes using the NRQ, and examples of good practice.The unintended effects of the project are increased knowledge about the NRQ, greater awareness of options for using project outputs, and new information about the work of schools.
It is difficult to carry out an in-depth efficiency analysis, because of problems of finding relevant comparative data and of assigning a specific amount of funding spent to different project outputs.
Consider changes which would reduce duplication in the internal and external evaluations of the project.Review the requirements for the printed versions of the reports.
Measuring indicators on satisfaction under the Technical Assistance OP, 2014-2020, in the Czech Republic - year 2018
Zjišťování míry naplnění indikátorů spokojenosti za rok 2018
Lenka Jansová, Lenka Jantačová (Ministry of Regional Development)
The study examines the level of achievement of indicators on satisfaction under the OP.
To assess the level of satisfaction of users with the monitoring system; and employees with personnel policy and the education system and with working conditions.
Survey of four target groups in December 2018; 5469 online questionnaires sent with a response rate of 41%.
At the end of 2018, the indicator on satisfaction with the information system was at 60%, 20 pp. less than the target for 2023The indicator relating to personnel policy and education system was at 64%, 8 pp less than the targetThe indicator on working conditions was at 67%, 3 pp less than the targetOverall, the level of satisfaction increased over the period.
There are some problems with data collection, such as changes in definitions, making it necessary to reformulate the individual items in the questionnaire.
Implement recommendations from previous yearsEstablish a helpdesk for the monitoring systemProvide a manual or handbook on the monitoring system.
Evaluation of the Performance Framework of the Toscana ESF OP, 2014-2020
Rapporto di valutazione sul quadro della Performance
The evaluation assesses the implementation of the Toscana ESF OP 2014-2020, which is aimed at supporting employment, education and social inclusions and at strengthening the regional administration, and the application of the Performance Framework (PF).
To assess progress towards achieving the PF targets and the management tools applied by the MA to improve the OP’s implementation.
Desk analysis of OP documentation and the EU regional policies;analysis of monitoring data;participation in Monitoring Committee meetings;in-depth interviews with the MA and officials involved in the OP’s implementation.
The PF is composed of 4 financial and 9 output indicators, the targets for which have been achieved giving access to the performance reserve.All the PAs have achieved their financial indicators.The output indicators have in some cases exceeded their targets by several times.The achievement of the PF derives from several structural and other measures implemented, including:detailed planning of activities;an effective system of incentives;introduction of procedural improvements;major investment in information systems;the use of over allocation and the region's own resource at the beginning of the period;the timely adoption of measures to accelerate spending (such as automatic selection procedures).
There is a need to:maintain focus on the implementation of PA B “Social Inclusion”, where new participants have been activated recently;reflect on procedural simplifications and the use of more advanced information systems, in view of the changes in the 2021-2027 regulations;continue to ensure that beneficiaries operate in a timely and effective way.
Evaluation of the contribution of the 2007-2013 Urban Initiative to social inclusion and equal opportunities in Spain. Final report
Evaluación de la contribución de la Iniciativa Urbana 2007-2013 a la inclusión social y a la igualdad de oportunidades Informe final
Dolores García Martos and Bárbara López de Quintana Palacios (Instituto de Estudios Fiscales-IEF)
The evaluation examines the contribution of the Urban Initiative 2007-2013 funded by the ERDF to the improvement of social inclusion in Spain.
To assess the mechanisms through which projects funded by the Urban Initiative helped to improve social inclusion and equal opportunities in deprived urban areas in Spain.
Interviews with beneficiaries.National statistics.Case studies.
The projects mainly focussed on providing infrastructure equipment and public services and less on fostering cooperation networks and economic activity.Some non-residents of the affected urban areas expressed discontent because they were denied access to interventions.Although projects improved opportunities in deprived areas, this did not lead to any significant reduction in unemployment or job insecurity, because of the economic crisis in the 2007-2013 period.There was a problem of sustainability for projects involving services in education, job integration and network development, since they take longer to be effective and require commitment from local authorities.
Results of the evaluation cannot be generalised, since only specific projects were analysed and for most of them no data were available from monitoring systems.
Define an integrated model of governance and encourage coordination between different public authorities and stakeholders.Define clearly the expected results of measures.Ensure continuity of interventions and an adequate monitoring system.Improve communication of measures to people.
Evaluation of implementation of the Research, Development and Education OP, 2014-2020, in the Czech Republic – 6th progress report
Průběžná evaluace implementace Operačního programu Výzkum, vývoj a vzdělávání, průběžná zpráva 06
This is the sixth implementation report of the OP.
To assess the progress of the OP to end-December 2018, to identify the factors that affect the OP’s implementation and to formulate recommendations for overcoming these.
Analysis of monitoring data and other sources provided by the Ministry of Education, Youth and Sports.Interviews with stakeholders, project implementers and target groups.Focus groups with project implementers and target groups.6 surveys with target groups.4 case studies.
Physical progress is mostly in line with plans, whereas financial progress is less.Out of 156 indicator targets, 38 are at risk of not being met, 80 of being exceeded.At end-December 2018, 77% of funds had been committed and 39% spent.The support for international mobility of researchers is assessed positively.Contacts established with foreign partners are important for international mobility projects.
It is not possible to measure the response rates to the surveys, since the respondents (target groups) were reached via beneficiaries of calls and the evaluator does not know the exact number of respondents contacted. The impact of organisational changes could not be assessed as little time had passed since their implementation.
Revise the OP with the aim of increasing selected indicator targets.
Evaluation of the France-Germany-Switzerland (Rhin supérieur/Oberrhein) Interreg OP,2014-2020 - Specific Objectives 1, 2 and 3 of PA ‘Smart Growth’
Evaluation d’impact du programme INTERREG Rhin Supérieur (2014-2020) pour la Région Grand-Est - Lot n°1 : Axe A - Croissance intelligente
Interreg V-A-France-Germany-Switzerland (Rhin supérieur/Oberrhein) 2014-2020
ADIT - Euro Institut - Strasbourg Conseil
https://www.adit.fr/fr/business/deploiement-des-affaires/entreprises-territoires/ - https://www.euroinstitut.org/fr/ - http://www.strasbourg-conseil.com
The evaluation examines three specific objectives of the ‘Smart Growth’ PA which are strengthening cross-border cooperation between higher education institutes, universities and research centres (SO1), strengthening the link between the research sector and enterprises (SO2) and supporting the development of innovative processes and products (SO3).
To assess the level of achievement of targets set and the impact of the measures supported of 15 projects supported under Smart Growth PA.
Desk and literature review.Analysis of monitoring data.Interviews with 20 project leaders and 40 R&I implementation managers.4 case studies.
The achievement of targets set vary across projects under SO1 (strengthening cross-border cooperation between higher education institutes, universities and research centres). Targets are close to being achieved under SO3 (supporting the development of innovative processes and products) while they are far from being achieved under SO2 (strengthening the link between the research sector and enterprises).Under SO1, administrative complexity of cross-border use of research infrastructure, tariff conditions for SMEs in neighbouring countries in technology transfer projects, and difficulties of creating real cross-border clusters have limited the increase of researchers in cross-border cooperation.In the 15 projects supported, there are 19 SMEs involved out of 277 partners.The participation of SMEs in projects is hindered by long and time-consuming procedures, State-aid rules, and intellectual property legislation.Interventions have helped to strengthen cross-border links between research institutes and enterprises but not the implementation of real cross-border clusters.Many companies have relied on existing networks and prior personal knowledge to undertake projects.Under SO3, projects have developed many applications and innovations.
For the next programming period , ensure that major scientific and technological challenges are well covered by the programme by developing pilot projects, e.g. on energy, industry 4.0, health, the bio-economy, social innovation, mobility, and climate change and remain flexible as regards future areas of production, such as those relating to bio-production.Involve more enterprises, e.g. through financial incentives.Simplify the programme and consider making co-financing larger and lengthen the duration of R&I projects.Align Interreg more with regional R&I programmes and with French and German S3 strategies.
Evaluation of social inclusion measures financed under the Human Capital OP, 2014-2020, in Romania
Evaluarea interventiilor in domeniul incluziunii sociale finantate prin PO Capital Uman 2014-2020
Liliana Lucaciu - Team Leader, Monica Roman, Andreea Savin, Ana Bulai – key experts, Dana Vişoreanu, Alina Ionescu, Corina Rosu, Diana Manoleli, Claudia Petrescu, Alexandra Sălcudean, Andra Jurj, Roxana Ursu, Cristina Stefan
Archidata SRL, Civitta Strategy and Consulting SA, LIDEEA Development Actions SRL, GOLDBACH Primes SRL. info@archidata.it; info.ro@civitta.com
This evaluation examines impact of the measures financed under PA4 ‘Social inclusion and fighting poverty’ and PA5 ‘Community led local development’ of the OP. Monitoring data used go up to 30 April 2023.
To assess impact of the measures for social inclusion and combating poverty and to support decision making of the 2021-2027 programmes.
Desk analysis of relevant documents.Analysis of monitoring and statistical data.Interviews with stakeholders.Workshops, surveys and focus groups.Case studies.Counterfactual impact evaluation.Expert panel.Theory of Change elaboration and testing.
More than 550.000 vulnerable persons and over 411 marginalised communities benefitted from support. Over 600 social services were financed and more than 100.000 medical and social staff trained.Despite the measures contributed to the improvement of the situation of the groups supported, needs remain significant.The uneven distribution of social services remains a critical problem for the most vulnerable.The programme had a good internal coherence. However, ensuring complementarity of soft and hard measures (e.g. developing services and infrastructure) remained challenging in practice.Sustainability of support is a critical aspect, because of a lack of financial resources and qualified personnel, especially in disadvantaged areas.
Programme-level data have limited accuracy because of differences between monitoring systems. There is a lack of data at national level on some vulnerable groups (homeless, drugs dependents, prisoners). Stakeholders were not fully available to provide data and participate in interviews and surveys.
Ensure that the new programme focuses on the most vulnerable groups and communities, piloting new methods and approaches to this end.Ensure an adequate provision of medical and social services to the most vulnerable.Improve data collection, also covering all vulnerable categories and areas.Continue administrative simplification.
Socio-economic analysis in preparation for the Berlin ESF+ Programme, 2021-2027
3. Bewertungsstudie: Sozioökonomische Analyse zur Ableitung strategischer Eckpunkte für die Förderperiode nach 2020
Marco Puxi, Ulrike Kaden, Almuth Richter
ISG - Institut für Sozialforschung und Gesellschaftspolitik, www.isg-institut.de; puxi@isg-institut.de; kaden@isg-institut.de; IfS – Institut für Stadtforschung und Strukturpolitik, www.ifsberlin.de
The study examines the socio-economic situation in Berlin on the basis of developments over the 2014-2019 period and identifies the key challenges in preparation for the 2021-2027 period.
To analyse the socio-economic framework and to identify the key challenges for ESF funding after 2020, taking account of the effects of Covid-19.
Official statistics.Previous studies on education and labour market.
In 2014-2019, economic developments in Berlin were favourable, with the GDP per capita in 2019 exceeding the country average.R&D expenditure in Berlin is higher than the German average but business R&D is lower.Even though the number of employed increased between 2014 and 2019, the overall and female employment rates remain below the country average.There is a shortage of skilled workers in education and training, public administration, and social security. It is estimated that the shortage of skilled workers will increase up to 2029.Men less often obtain a higher education qualification, more often leave school without a qualification (in particular those whose native language is not German), and are more often left without training opportunities.The risk of poverty has continued to increase since 2019, particularly among the low-skilled, those with a migration background, single parents and older people.Because of Covid-19, in the first half of 2020, there was a significant decline in economic activity, and between March and October 2020, unemployment increased by almost a third.
Although the analysis covers the period 2014-2019, because of data availability issues, in some parts, the period examined is shorter. The medium- and long-term impact of Covid-19 on economic activities cannot be assessed yet.
Evaluation of policies for employment and social inclusion financed by the Veneto ESF OP, 2014-2020
La valutazione delle politiche di contenimento della disoccupazione e di prevenzione dell’esclusione sociale
The evaluation examines several types of ESF measures to support employment and social inclusion in Veneto. The main focus is on the “voucher for employment“ - (Assegno per il lavoro) scheme - which is targeted at supporting the unemployed of 30 and over through counselling, training and other activities.
To assess the relevance and implementation of measures concerned, the effectiveness of the “voucher for employment” scheme and to make recommendations.
Desk analysis of documents.Analysis of monitoring data.Survey of participants (2,541 individuals).Interviews with PES, training and labour service providers and social partners.Benchmarking against Lombardia, Piemonte and Emilia Romagna.
Participants in the work experience scheme have a higher level of education than those in the voucher scheme and the share of women is larger.90% of participants report not having difficulties accessing the ESF measures.22% of respondents found a job after participation, 19% after six months. Two third of jobs being temporary.42% of respondents report the job found being consistent with the contents of the measure.Half of respondents receiving training (60% of the total) report that it was not very useful.The voucher for employment scheme is not appropriate for those most distant from the labour market.
Continue the work experience and voucher for employment schemes.Introduce traineeships under the voucher for employment scheme.Improve the profiling system of the voucher scheme by combining qualitative and quantitative criteria.Improve the consistency between the contents of training and enterprise requirements.
Evaluation of the effects of PA II (e-government) under the Digital Poland OP, 2014-2020
Ewaluacja efektów realizacji II Osi priorytetowej (E-administracja…) Programu Operacyjnego Polska Cyfrowa na lata 2014-2020
The evaluation examines the effects of the OP on the development of e-government at the central administrative level.
To assess the effects of the support provided under PA2 of the Digital Poland OP in relation to the objectives.
Monitoring and statistical data analysis.CAWI/CATI survey of 81 beneficiaries.14 interviews with beneficiaries.4 interviews with implementing bodies.2 expert panels.
All strategic result indicator targets have been achieved, despite Covid-19, high prices and changes in the supplier market adversely affecting the implementation of projects.Digital projects implemented have helped to combat and counteract the negative effects of Covid-19.There are still significant unmet needs as regards digitalisation of public administration.
Continue to develop e-services in areas with greatest needs, e.g. in healthcare.Provide applicants and beneficiaries with advise and training throughout the life cycle of projects.Expand the number of information channels on organised training for beneficiaries.
​​Evaluation of the National Strategy for the inclusion of Roma, Sinti and Camminanti, financed under several national and regional OPs, 2014-2020, in Italy​
Rapporto sul cambiamento prodotto dalla Strategia Italiana RSC 2012-2020
NOP Social Inclusion 2014-2020; ROP Lazio ESF 2014-2020; NOP Metropolitan Cities 2014-2020
​​Marco Accorinti, Monica Andriolo, Loredana Cerbara, Enrico Mascilli Migliorini, Maria Alessandra Molè, Antonio Sanguinetti, Sandro Turcio, Mattia Vitiello​
​Istituto di Ricerche sulla Popolazione e le Politiche Sociali (IRPPS), https://www.irpps.cnr.it
The evaluation assesses the effects of projects targeting Roma, Sinti and Camminanti (RSC) communities, which aimed at improving housing, education, employment, healthcare and addressing anti-Roma discrimination. Data used go up to 2020.
To assess the effectiveness of interventions promoting social inclusion and combating discrimination towards marginalised communities. To assess whether the projects implemented under the strategy have achieved their objectives and to provide insights for the future.
Desk research on RSC Strategy and implemented projects, including academic literature, web material and press reviews.Realist evaluation approach (TBE).Analysis of monitoring and statistical data.27 case studies of projects implemented at different territorial levels relating to housing (1), health (1), education (8), work (5), governance (4) and anti-Gypsyism (8).Over 28 semi-structured interviews with representatives of the National Office Against Racial Discrimination (UNAR), public administrations at different levels, other stakeholders (i.e. local associations, research institutes and universities), and representatives of the RSC community.
Projects in the education sector led to increased school enrolment and participation of children from Roma, Sinti, and Camminanti communities.Housing projects helped create a statistical framework on the living conditions of marginalised communities, but housing deprivation remains a big challenge.While some projects successfully integrated participants into the workforce and helped secure long-term jobs, challenges remained in providing sustainable employment opportunities, particularly in more marginalised areas.Despite differences across regions, an improved access to healthcare for Roma, Sinti, and Camminanti people emerged, particularly due to training programmes for health professionals.Many projects led to strengthened partnerships between local, regional and national stakeholders, but coordination is problematic.Despite awareness campaigns and cultural initiatives, prejudice and discrimination remained prevalent in some regions, highlighting the need for more long-term, systemic efforts.
Impossibility to collect and process personal data relating to racial or ethnic origin, due to EU Regulation 2016/679 (Article 9). Some projects were too limited in time and data tracking was partial in certain cases.
Ensure long-term sustainability and better coordination of projects.Increase participation by involving RSC communities, especially women and the youth, in project design and implementation.Strengthen anti-discrimination initiatives and integrate cultural promotion to improve societal recognition of RSC communities.Adapt intervention to local contexts and link between them housing, education, health and employment initiatives.
Evaluation of measures in the environment, culture and transport sectors under the Kriti OP, 2014-2020
Τελική Έκθεση Αξιολόγησης για το Έργο: Αξιολόγηση των επιπτώσεων των υλοποιημένων δράσεων και πράξεων των τομέων περιβάλλοντος, πολιτισμού και μεταφορών, οι οποίες εμπίπτουν στους Ειδικούς Στόχους και τις Επενδυτικές Προτεραιότητες ΕΤΠΑ του ΕΠ «Κρήτη» 2014-2020
52 Gkanogianni Street, Zografou, 15773, tel: 210 7752120, e-mail: geopar1983@gmail.com
The evaluation examines implemented interventions and projects in the environment, culture and transport sectors, under the OP. Data used goes up to November 2023.
To assess the achievements and impact of projects implemented in the environment/culture (140 projects) and transport (10 projects) sectors.
Analysis of monitoring data. Preliminary impact assessment of all projects: identification of the possible impact in terms of direct/indirect effects, significance, and duration (based on secondary data provided by the MA) and analysis of the actual impact only for completed projects. Analysis of the results of 62 selected projects (application of the Proportionate Stratified Sampling method). Interviews and survey (response rate of 96.8%, 60 out of 62 requests) addressed to the main participants/people involved in the implementation of the projects.
Completed projects have achieved their outputs and results to a very large extent. Effects of the interventions in the medium and long term are not expected to differ significantly from short term ones and they are likely to be sustainable after the end of the programme. The implementation of both completed and ongoing projects do not present significant problems, apart from small issues due to context factors (e.g., Covid-19, energy crisis).
Some projects were not completed at the time of the evaluation or the time elapsed since their completion was too short to record the results.
Ensure that the selected result indicators are fully compatible with the specific characteristics of each measure. Establish and maintain a monitoring system for result indicators to ensure alignment with the measures’ goals and track progress effectively.
Evaluation of 8 ERDF programmes 2014-2020
Utvärdering av åtta program Regionala utvecklingsfonden. Tematiskt mål 3. Delrapport 2 om målet: Öka konkurrenskraften hos små och medelstora företag
This is the second report on the evaluation of TO 3 in Sweden, which supports the competitiveness of SMEs.
To give an overview of the results from the evaluation's second phase.
Documentation analysis;qualitative analysis;project manager interviews;case studies.
In general, the projects are in line with the challenges identified and specific objectives.Almost two-thirds of the projects examined have good project logic. In some cases, the project logic has been reworked or clarified during the project’s implementation.Less than half of projects examined will make a substantial contribution to the achievement of programmes objectives.Most of the regional programmes are not set up in a way that promotes efficiency.The performance indicators often measure changes in output or turnover rather than competitiveness.In two framework programmes examined in Upper Norrland, there is a risk that support will be a means to redistribute resources more than a way of strengthening competitiveness.
It is too early to make a quantitative evaluation of the long-term effects in relation to a control group.
Projects should target SMEs with a high potential to increase their competitiveness and expand into new markets.Projects need to take into account geographical, industry-specific and historical context in order to be designed in a way that effectively addresses barriers to increased competitiveness.Projects aimed at creating direct effects in SMEs should be evaluated quantitatively and in relation to a control group. This evaluation should begin no earlier than 2022.
Evaluation of implementation and effects of the Greece-Republic of North Macedonia Interreg OP, 2014-2020
Ενημέρωση της 1ης αξιολόγησης υλοποίησης και επιπτώσεων του προγράμματος συνεργασίας «INTERREG IPA CBC Πρόγραμμα Ελλάδα - Δημοκρατία της Βόρειας Μακεδονίας 2014-2020»
IPA CBC Greece-The former Yugoslav Republic of Macedonia 2014-2020
The evaluation examines implementation and effects of the OP up to end-June 2021.
To assess the effectiveness, efficiency, relevance and the effects of the OP up to end-June 2021, as well as its communication strategy.
Official statistics.Monitoring data.Interviews with MA and members of the Joint Secretariat.Survey of beneficiaries, potential applicants and other stakeholders.Review of OP documentation and deliverables.
The OP is on track to meet the targets set for 2023, despite initial delays and challenges posed by Covid, for which sufficient response measures have been taken.Delays were mainly due to differences between IPA and national funding regulations, and the very large number of beneficiaries.Communication activities make use of a wide variety of measures and are effective according to stakeholders.Projects are contributing to pursuit of the objectives of the OP, as most of result indicators have already achieved their targets or are on track to do so.The implementation of programmes has benefited from past experience and from training to perform tasks more effectively.
Findings on impact are based on assumptions as most of projects were not completed at the time of the evaluation.
Increase focus on dissemination of results of the OP.Ensure systematic data collection and set a framework for determining the indicator values in the monitoring system.Provide practical-based training for beneficiaries.
Evaluation of the communication strategy
Chilehaus C – Burchardstraße 13, 20095 Hamburg
Παροχή Υπηρεσιών για την Εκτίμηση Των Επιτευγμάτωντων Παρεμβάσεων της Κοινής Υποστήριξης τουΕυρωπαϊκού Κοινωνικού Ταμείου και της ΠρωτοβουλίαςΑπασχόλησης των Νέων συμπεριλαμβανομένης και τηςΕφαρμογής της «Εγγύησης για τη Νεολαία» όσον αφορά τη Βιώσιμη
OP Employment, Human Capital and Social Cohesion 2014-2020
YEI is implemented in the Employment, Human Resources and Social Cohesion OP under a specific Priority Axis and by end-2018, 6 projects had been approved for a total budget of EUR 40 million, involving the provision of work experience for young unemployed graduates, incentives for hiring the unemployed under 25 and NEETs aged 25-29, and providing opportunities for Law graduate NEETs under 30 years to acquire their licence and training opportunities for architecture and civil engineering graduate NEETs in the same age group under years of age and guidance for NEETs generally.
To assess whether the objectives have been reached and in which cases the impact on youth employment was largest, to identify how the measures currently in place can be usefully modified before the end of the 2014-2020 programming period and to assess the quality of job offers, the achievements of the measures supported, the sustainability of the results and the contribution of YEI to the implementation of the YG.
Literature review of OP documents, the National action plan for youth, etc.Interviews with authorities involved in the design and implementation of YEI.Survey of participants 6 months after exit.Peer reviews.Data: monitoring system, National Statistical Institute, Eurostat, ILO, and PES.
YEI implementation is satisfactory. Targets for output indicators should easily be reached for each measure.Apart from larger recruitment incentives for green and blue jobs, the innovative aspect of measures is the targeting of NEETs.There are other measures financed by ESF which are aimed at young unemployed and are complementary to YEI. The fact that the same authority manages YEI and these programmes is crucial for ensuring coherence between them.Unit costs vary from EUR 300 (guidance for NEETs) to EUR 10,000 (providing training opportunities for architecture and civil engineering graduate NEETs).6 months after exit, over half of participants in work experience projects were in employment, 5% were self-employed and the same proportion were in education or training.39-47% had a full-time permanent employment contract.A significant number of participants remained in the enterprise in which they were placed to gain work experience.Over 90% of participants were satisfied with the measure and thought that it improved their personal and professional skills.The main reasons for drop-outs are low pay, poor working conditions and jobs not meeting requirements.A unit cost of EUR 4,681 is similar to that of a similar work experience measure.
Of the six measures supported, detailed results are only available for one (‘Acquisition of work experience by young unemployed graduates’).
There is a need to:adapt examples of good practice identified in other countries (i.e. to improve networking among stakeholders, to organise a regular forum for exchange of views, to cooperate with schools to prevent early school leaving, to set up teams of young unemployed graduates to help identify NEETs, to establish a one stop shop in each province to provide guidance for young people);examine the possibility of partnerships with private employment agencies;continuously update the mapping of data on NEETs;introduce specific conditions to limit redundancies in enterprises when participants are working in them.
First Impact Evaluation of the Mecklenburg-Vorpommern-Brandenburg-Poland Interreg OP, 2014-2020
Erste Impact-Evaluierung im Kooperationsprogramm Interreg V A Mecklenburg-Vorpommern / Brandenburg / Polen im Rahmen des Ziels "Europäische territoriale Zusammenarbeit" des Europäischen Fonds für Regionale Entwicklung (EFRE) Endbericht
Interreg V-A-Germany/Mecklenburg-Vorpommern-Brandenburg-Poland 2014-2020
INFRASTRUKTUR & UMWELT: Professor Böhm und Partner, Julius-Reiber-Straße 17, D-64293 Darmstadt, mail@iu-info.de
The evaluation examines the Mecklenburg-Vorpommern-Brandenburg- Poland Interreg OP, which contains four PAs, PA 1 “Conservation and Protection of the Environment and Promotion of Resource-Efficiency”, PA 2 “Promotion of Sustainability of Mobility and Removal of Bottlenecks in Important Network Infrastructures”, PA 3 “Investment in Education and Vocational Training for Competences and Life-Long Learning” and PA 4 “Improvement of the Institutional Capacities of Public Authorities and of Efficient Public Administration”.
To assess the effectiveness and impact of the OP by examining expected results of projects and the outcome indicators, to identify aspects of implementation that do not work effectively and efficiently and should be modified and measures that increase the effects of projects.
desk analysis of project results, partnerships and administrative procedures;online survey of all lead project partners (187, response rate, 35%);5 case studies;9 in-depth interviews with OP bodies;network analysis using the online “Graph Commons” tool to assess project partnerships.
Targets for finance and output indicators should be achieved, except for output indicators 3.1 “Total length of road connections” and 1.3. “Length of tourist pathway and road infrastructure”.The output indicator “Number of activities” is considered to be ambiguous.The quality of the projects is only satisfactory and there is potential to increase the added-value of the OP by eliminating obstacles to cross-border cooperation.While the OP management is relatively efficient, there are a number of problems to resolve, including conflicts in the Monitoring committee because of insufficient consultation, eligibility rules seem to be complex and the electronic monitoring system inaccessible.
Examine the contribution of projects to cross-border cooperation more closely.Introduce some form of exchange prior to deciding on projects in the Monitoring committee.Simplify and unify eligibility rules.Reduce the complexity of the monitoring system and increase its user-friendliness.
Evaluation of Action 11 'Practice classes' of the Bayern ESF OP 2014-2020
Evaluaionsbericht zur Förderaktion 11: Praxisklassen Bayern 2014-2020
Institut für Sozialforschung und Gesellschaftsppolitik GmbH - Gorgasring 2, 13599 Berlin, www.isg-institut.de
The evaluation examines the “Practical classes” scheme providing practical teaching in secondary schools and socio-education support to reduce the number of drop-outs, financed by the Bayern ESF OP. The evaluation is one of a series on different ESF measures, summarised in DEE53.
To assess the results and effects of the scheme.
Analysis of monitoring and administrative data on employment.Online survey of school authorities (78% response rate out of 56).Telephone survey of participants six month after the completion of the praxis class (64% response rate out of 300 contacted).Interviews with two representatives of the Bayern Ministry for Science and Arts and with three administrative districts.2 case studies of praxis classes.
Practical classes were already funded in the previous period and are a well-established part of the school system.Almost half of the schools rated the practical classes as a very good tool for achieving the secondary school leaving certificate.The practical classes are rated as very good by over 80% of participants.All respondents report an improvement in school performance.80% of participants assume that the practical class will improve their chances on the job market.Only 11% of participants dropped out of the practical classes.Only 42% have entered vocational training after the practical class (target 55%).It is unclear how far the placement of participants crowds out “normal” school leavers.
Causality between the practical class and results has not yet been seriously analysed.
Consider providing follow-up support after leaving the practical class (based on personal relationships established).Consider involving the Chamber of Crafts as an advisor as they provide other support for those interested in an apprenticeship.
Evaluation of measures for developing an early childhood service system, financed under three OPs, 2014-2020, in Hungary
A koragyermekkori szolgáltató rendszer fejlesztése és hátránykompenzáló szerepének értékelése
OP Human Resources Development 2014-2020; OP Territorial and settlement development 2014-2020; OP Competitive Central-Hungary 2014-2020
Zsombor Farkas, András Gábos, Ágnes Hárs, Orsolya Kisgyörgy, Katalin Nagy
The evaluation aims to assess 20 calls for proposals to improve the childcare services system and to support parents’ employability, financed under three OPs: the Human Resources Development OP (HRD OP), the Competitive Central Hungary OP (CCH OP) and the Territorial and Settlement Development OP (T OP). Data used go up to October 2023.
To assess the policy context, achievement of targets, impact, limitations and sustainability of the measures supporting childcare services implemented under the selected OPs.
Desk research. Analysis of statistical, monitoring, and administrative data. Structured interviews with stakeholders and beneficiaries. Survey of beneficiaries.
77% of the 115 projects financed under the HRD OP created new childcare services. HRD OP and CCH OP measures improved the childcare system by creating a nationwide methodological network and developing a system of early childhood intervention. As of September 2023, measures under the T OP and CCH OP created 6500 new crèches and 2000 nursery places for 0-3 years old children. There were some overlaps between the measures funded by the various OPs, especially in terms of target groups reached. Staff’s lack of training, COVID-19 and the presence of refugees from the war in Ukraine have considerably reduced the overall services’ quality and sustainability over time. Sustainability of childcare services is needed to foster women’s employment rate.
Some data from administrative documents are missing (Children’s House). The surveys on measures under the Territorial and Settlement Development OP had a relatively low response rate (13% and 20.7%).
Improve early childhood measures both quantitatively and qualitatively, and avoid overlapping of similar services. Ensure cross-sectoral and inter-stakeholder cooperation as well as high quality training for staff.
Evaluation of the social centres financed by the Epirus ESF OP 2014-2020
Αξιολόγηση των οινωνικών Δομών που χρηματοδοτούνται από το ΕΠ Ήπειρος 2014 -2020
Mavrokordatou 1-3, Athens 106 78, e-mail: info@eeogroup.gr, www.eeogroup.gr
The evaluation examines the progress of measures under IP 9.ii, 9.iii, 9.iv of the OP, involving community centres, the provision of essential services, day-care centres for the elderly (KIFI) and those with disabilities (OHRC) and arrangements for the homeless.
To assess progress in implementing projects to provide community centres, day-care centres for the elderly and people with disabilities and arrangements for the homeless and to make policy recommendations.
Analysis of OP monitoring data.Survey of 37 coordinators and managers of social services.Survey of a sample of individuals receiving services from Community Centres (378 out of 11910), social grocery stores (313 out of 1610), social pharmacies (222 out of 617), and those responsible for day-care centres for the elderly and disabled.4 focus groups with coordinators of Community centres, executive officers of the OP, Homeless shelters and Social Welfare Directorates.
The services provided contributed to achieving the objectives of national and regional strategies for social inclusion and combating poverty.The absorption rate averages 87%.Almost all output indicator targets had been achieved or exceeded up to end-2020.Only one Community centre had extended its services under IP 9.ii and the monitoring system for IP 9.iii and IP 9.iv measures had not been activated.Community centres did not make as many referrals to employment services as planned because of higher demand for welfare benefits. The Integrated information system has defects and data on individuals are registered in different ways. This, combined with the administrative burden, limited the effectiveness of measures.Cooperation between social services and other public and private bodies, suitable staffing levels and the maturity of services are important for projects being effective.The unit costs of most of the measures have been lower than estimated because of the scale of participation.The added-value of the services provided was the creation of the basis for the National Social Policy, the strengthening of the social network for combating poverty and the opportunity given to those with disabilities to integrate into social life.
The efficiency analysis could not be carried out for all the measures because unit costs were not always available.
Organise training for the social service staff on the interconnection between social and labour market services.Upgrade the unified information system, configure it to export statistics and create a digital forum for all the community services.Sign cooperation agreements between community services and local bodies with a clear division of responsibilities.Disseminate the experience gained for the provision of basic services in other municipalities.
Evaluation of the implementation and first effects of the Measure 3.5 ‘Accessible university’ and ‘Accessibility knowledge centre’ of the Knowledge Education Growth OP, 2014-2020, in Poland
Ocena realizacji i pierwszych efektów konkursów PO WER Działania 3.5 „Uczelnia dostępna” oraz „Centrum wiedzy o dostępności”
Andrzej Dziurdzik, Joanna Kwinta, Tomasz Klimczak, Natalia Krygowska Nowak, Adam Miller, Małgorzata Zub
EGO S.C., www.evaluation.pl;LB&E Sp. z o.o., www.lbie.eu
The evaluation investigates the relevance and usefulness of the measures to ensure equal opportunities for people with special needs of participating in the higher education process.
To assess implementation and first results of the 'Accessible university' and 'Accessibility knowledge centre' projects, aimed at ensuring participation in higher education process for people with special needs.
Analysis of monitoring data and official statistics.5 interviews with representatives of institutions involved in project implementation.Survey of 196 beneficiaries (project managers) of the ‘Accessible University’ calls (75% response rate) and survey of 622 university employees involved in training activities (12% response rate).8 interviews with beneficiaries of the ‘Accessible University’ and 4 interviews with beneficiaries of the ‘Accessibility knowledge centre’.2 interviews with representatives of medical universities that did not apply for funding and 8 interviews with representatives of organisations working for people with disabilities.24 individual interviews with students with special needs.3 interviews with authors of the study ‘Models of support for universities to increase their accessibility for people with disabilities’.2 expert panels.
There were criticalities in relation to the administrative burden and project management, particularly for larger-scale projects.Cooperation with partners and stakeholders for the implementation of projects was positive, including with people with disabilities.More than half of the universities have completed organisational and procedural tasks. Progress is evident in relation to accessibility of assistance services and online accessibility, but limited in adapting the infrastructure.
Reduce administrative burden and simplify project management, especially for large-scale projects.Ensure all types of disabilities, including mental disorders and autism, are adequately addressed in future projects.Strengthen partnerships with disability communities and other stakeholders for effective project implementation.Prioritise infrastructural improvements to enhance physical accessibility to campuses.Enhance training programmes for university staff and students to improve disability awareness and inclusion.Provide regular assessment of project outcomes and flexibility to adapt strategies based on emerging needs.
Evaluation of the ESIF-Communication Strategy of the Sachesn-Anhalt ESF and ERDF OPs, 2014-2020
Bewertung der ESIF-Kommunikationsstrategie
Ramboll Management Consulting GmbH, Chilehaus C - Buchardstr. 13, 20095 Hamburg, +49 40 30 20 20-137, marcus.neureiter@ramboll.com
Evaluation of measures for continuing vocational training under the Brandenburg ESF OP, 2014-2020
Endbericht. Bewertung der Förderung der beruflichen Weiterbildung im Land Brandenburg mit dem Europäischen Sozialfonds (ESF)
SÖSTRA GmbH - Institut für sozialökonomische Strukturanalysen
The evaluation aims to assess the contribution of the continuing education measures funded under PA C "Investing in education, training, and vocational training for skills and lifelong learning" (IP 10.iii) to ecological sustainability and enhancing employment-related skills.
To assess ESF-funded continuing training measures for identifying ways to enhance ecological sustainability in company training and the overall effectiveness of ESF funding.
Desk research. Analysis of output and results indicators. 3 focus groups with programme managers. 12 focus groups with stakeholders (chambers, training alliances, educational service providers) and company professionals involved in continuing training (16 people in total).
Of 17,178 participants who completed vocational qualification initiatives, 15,027 obtained a qualification, surpassing the planned target. But older individuals participated to a limited extent and were only 55% of the target. 57% of participants were men. Women showed more interest in individual access programmes than in company-funded ones. According to focus groups, the demand for training arises from the need for specialised skills and for improved sustainable production processes. Large companies, particularly in energy supply, drive ecological transformation, while SMEs lack these capabilities. In Brandenburg, continuing education providers vary in their preparedness for future challenges: those closely collaborating with companies are well-equipped for changes, those reliant on commissions and job centres less so.
Maintain individual access to continuing vocational training while preserving the "Brandenburg Settlement Package". Encourage education providers to develop adaptable curricula in response to ecological or digital transformation. Support continuing training alliances in Brandenburg, focusing on SME workforce development, involving industry-specific companies, with priority given to those promoting ecological transformation. Focus on raising awareness and attracting low-skilled workers to vocational training, emphasising individual and institutional engagement. Explore the promotion of model projects to improve vocational education formats for specific groups, taking account of factors like duration, methodology, and integration into working processes.
Evaluation of the contribution of ESI Funds, 2014-2020, to regional development in Moravskoslezský, in the Czech Republic
Výsledková evaluace přínosů Evropských fondů na regionální úrovni: Úkol 5 a 6 - Moravskoslezský kraj
The evaluation examines results and benefits of the EU funds at the level of the Moravskoslezský region. Data used go up to 30/06/2022.
To evaluate outputs and results of the interventions funded under EU funds.
Analysis of official statistical data. Analysis of monitoring data (MS2014+). Case study (ORP Havířov). Interviews with selected stakeholders. Survey of municipalities and citizens. Counterfactual analysis (DiD, PSM, regression analysis).
CO2 emissions decreased by 4% in the 2014-2020 period mainly due to the projects that aimed at improving the energy performance of buildings. EU-funded projects have led to an increase in the quality of education and level of inclusion, as 3,409 Roma pupils have been included in education. In the 2014-2020 period, the unemployment rate in the region decreased by 4.1%. The projects supported 5,387 unemployed people, including the long-term unemployed. There is a strong positive relationship between the support from EU funds and the increase in the number of researchers and GDP per capita. In total, 68 km of roads were newly built or reconstructed, which improved transport accessibility in some parts of the region. The correlation between EU funds and transport improvement is significant. No significant quantitative impact of EU support is found in relation to social exclusion, ecological stability and employment. Tourism and internet coverage were not significantly supported by EU funds.
For some TOs the quantification of the benefits of EU funds is not possible because of a lack of granular data or absence of projects implemented.
Continue to use the “RESTART” strategy to solve key problems and increase absorption capacity. Reduce the administrative complexity for the preparation of the applications, preparation and implementation of public contracts, payment claims, sustainability and promotion of projects. Increase the interdependence of methodological and systemic projects in the field of education and their overlap into practice.
Evaluation of the Empowerment Plan and the Coaching Plan for PES staff funded under PA4 of the Lombardia ESF OP, 2014-2020
Rapporto di valutazione tematico. I piani di empowerment and coaching per il personale dei centri per l'impiego e delIe AFOL in Lombardia
https://www.irsonline.it/ - irsmi@irsonline.it; https://www.cogea.it/ - consulenza@cogea.it
The report examines the implementation of the plans for PES staff financed by the OP, which aim to improve the efficiency and effectiveness of employment centres, covering the period 2018-2022.
To assess the achievements of the two plans for improving the efficiency and effectiveness of Employment Centres over the period 2018-2022.
- Desk research.- CAWI survey of participants.- Two focus groups with main stakeholders.
- There were 268 participants in the Empowerment plan and 735 in the Coaching plan, who were typically women, under 45, and with high education.- Both plans increased the quality of work and had a positive effect on the organisation of employment centres and the professional skills of staff.- Improvements are still needed in public-private cooperation, career guidance, job placement, support for firm creation and management diversity.
- Strengthen the managerial role of coordinators.- Improve networking at provincial level and in each employment centre.- Monitor the accompaniment of new recruits, improve the applicability of learning at work, and facilitate transfer of learning.
Annual evaluation report of the Veneto ESF OP, 2014-2020, 2018
This is the 2018 annual report on the implementation of the Veneto ESF OP.
To assess the progress of the OP up to end 2018, and the relevance of operations in relation to the development of regional needs and the socio-economic situation.
Monitoring data.National statistics.Evidence from other evaluations.
In 2018 there was a marked acceleration in the implementation of the OP and all PAs have achieved performance framework targets for 2018.Most recipients of support are men (56%), people with lower secondary school education, employed or inactive.Despite improvements in the economic situation, measures are still relevant to regional needs and are generally coherent with other measures at regional and European level.Continuing training is mainly undertaken by micro-enterprises.Beneficiaries are mostly satisfied with the continuing training, but they would prefer it being more practical.Continuing training measures had a positive impact in reducing the unemployment rate but had no effects on value-added.
The analysis of impact, that comes from a previous evaluation, is only an indicator of how the OP is progressing. A more accurate analysis will be carried out at the end of the period.
Improve communication of the OP through the internet, especially to reach specific beneficiary targets.Keep a large financial allocation for PA1 (employment), to face a possible future economic crisis.Improve social responsibility awareness of companies, especially through the internet and social media.
Evaluation of the Small Projects Fund of the Poland-Germany/Sachsen Interreg OP 2014-2020
Ewaluacja wdrażania i wpływu Funduszy Małych Projektów na realizację celu IV osi priorytetowej Programu Współpracy INTERREG Polska-Saksonia 2014-2020
Interreg V-A-Poland-Germany/Saxony 2014-2020
EU-CONSULT Sp. z o.o.; Kreatus Sp. z o.o.
EU-CONSULT Sp. z o.o., www.eu-consult.pl; Kreatus” Sp. z o.o., www.kreatus.eu
The evaluation examines the impact of ‘small projects’ funded under PA4 of the Poland-Germany/Sachsen OP on cross-border cooperation and on the socio-economic development of border areas.
To assess the impact of Small Projects on socio-economic changes in the Poland-Sachsen border areas.
desk research;in-depth interviews with controllers, representatives of Euroregions and project partners;case studies;benchmarking;mystery client, mystery calling, mystery web;interviews with applicants;social network analysis.
Small projects address the main problems and challenges of border areas and help to achieve programme objectives.New partnerships account for 20% of all project partnerships.The most important barriers relate to organisational and legal-administrative issues.Micro-projects support the strengthening of local communities and Poland-Sachsen cooperation.There is only a small degree of complementarity with other activities of the OP and with other OPs.
Intensify training and information activities on aspects that were difficult for applicants (e.g. the estimated number of participants).Consider a mandatory number of hours of a neighbour's language learning with elements of intercultural training in the next period.Increase awareness of beneficiaries of EU horizontal principles.
Evaluation of measures for the valorisation of artistic, cultural and natural resources under the Emilia-Romagna ERDF OP, 2014-2020
Rapporto tematico: Asse 5 valorizzazione delle risorse artistiche, culturali e ambientali
https://www.irsonline.it/, irsbo@irsonline.it; https://www.nomisma.it/, info@nomisma.it
The evaluation examines the implementation of measures aimed at developing the cultural and natural resources of the region under the OP.
To investigate the characteristics of the projects and beneficiaries, the benefits of upgraded cultural heritage for the development of the local territory and to provide suggestions for future projects.
Analysis of monitoring data. Survey of beneficiaries (60% response rate).
Projects are mainly on hiking and cycling pathways, redevelopment of public buildings with a view to their cultural and artistic use. Most of projects are part of a wider strategy of beneficiaries or of the local territory. Delays affected most of the projects and were mainly caused by the increase of prices of raw materials and difficulties in their acquisition, by Covid and by a lack of human resources of local authorities. According to beneficiaries, projects improved access to cultural and natural heritage, and led to increased tourist flows, but the monitoring system is only partly able to capture this result. Projects helped increase employment for less than 29% of beneficiaries surveyed.
Preparatory study on the RIS3 strategy in Guadeloupe for the ERDF and ESF+ Programme, 2021-2027
Evaluation de la stratégie de spécialisation intelligente (S3) de la Guadeloupe
The report examines the Guadeloupe’s Smart Specialisation Strategy (RIS3) for 2021-2027 around 8 priorities: circular economy, preservation of the environment, agriculture, energy, mobility, health, tourism and sport.
To review Guadeloupe’s RIS3 for the 2014-2020 period in order to prepare the 2021-2027 strategy.
Literature review.Analysis of monitoring data.Analysis of official statistics.20 in-depth interviews with the main stakeholders of the Smart Specialisation Strategy.Final workshop with 30 participants.
The RIS3 of the 2014-2020 period has enabled major research projects to be carried out, thanks to the cooperation between the various research actors (public bodies, laboratories, research institutes, companies etc.).The Regional Innovation Network was the basis for the implementation of the strategy, as it brought together all the actors of the research and innovation ecosystem.The structure of the 2021-2027 strategy includes 8 priorities: circular economy and waste management, preservation of the environment and biodiversity, agriculture, renewable energies and energy efficiency, sustainable mobility, demographic change and health, sustainable tourism, sport.The eight priorities are all designed to promote three types of growth: green growth, blue growth and inclusive growth. In addition, the digital transition is identified as a cross-cutting axis in addition to these 8 priorities.
Due to COVID-19, interviews could not be conducted physically and had to be conducted online.
Ensure that the Guadeloupe Region will drive the strategic governance of the strategy.Assign the lead of the operational governance of the RIS3 to the Research and Innovation Network and the future Guadeloupe Technopole.
Evaluation of business training for vulnerable refugees under the Education and Entrepreneurship OP, 2014-2020
www.cowi.com - E-mail: cowi@cowi.com - Tel: +45 56 40 00 00
The evaluation examines business training measures implemented in Denmark under the OP to support social inclusion of refugees.
To assess whether the business training measures at the end of 2020 met the targets set and the evidence on the results.
Monitoring data.Interviews with beneficiaries and implementers.
The holistic approach of the project strengthened the competences and integration opportunities of participants.The ‘Place and Train’ programme supported traineeships, mentoring and coaching of participants with the involvement of the Danish Refugee Council and the partnership established was effective.100% of the activity targets were met at the end of the projects and 92% of the output targets.Covid-19 has affected some of the activities and there has sometimes been a need to keep up with developments in legislation.
Support the involvement of civil society organisations and municipalities to insure effectiveness.Use evidence drawn from the project to strengthen the methodology for planning future measures.Improve the match between beneficiaries and companies.
Evaluation of ICT measures under the Enterprise and Innovation for Competitiveness OP and the Integrated Regional OP, 2014-2020, in the Czech Republic. Final report
Výsledková tematická evaluace Dohody o partnerství pro období 2014-2020: TC2 Zlepšení přístupu, využití a kvality informačních a komunikačních technologií. Závěrečná zpráva.
OP Enterprise and Innovation for Competitiveness 2014-2020; Integrated Regional OP 2014-2020
This evaluation examines the impact of measures to improve ICT access, usage and quality in the Czech Republic, with emphasis on the public administration and businesses.
To assess the results and impact of the measures and their contribution to socioeconomic trends (including relative to other EU members).
​​Cross-Regional Sequential Difference-in-difference (CR-SEQDD). Analysis of monitoring and statistical data (e.g. Czech Statistical Office, Czech Institute of Health Statistics).Literature review.Linear regression. Theory of change testing. Expert verification (two additional experts beside the evaluators). 13 Interviews.6 Case studies.Text mining.
High-speed internet access (and to a lesser extent usage) increased significantly in the country, but the impact of the measures was negligible.The digitalisation of businesses was strengthened as a direct result of the measures through the acquisition of 687 information systems and the construction and modernisation of 74 data centres.The digitalisation of public administration was also strengthened, with a direct effect of the measures on 118 municipalities. However, digitalisation in both public and private sectors remains below EU average despite these improvements.The measures have saved 3.5% of the state budget and 20.1% of regional and municipal budgets.
The relatively recent provision of support for high-speed ICT coverage has limited the room for impact analysis. The generic words in some of the titles of the calls provided an unrealistic picture of some types of support in the text mining analysis. The usability of regional data was sometimes limited.
Continue to support high-speed internet expansion but coordinate better with local authorities and other stakeholders.Continue to support the digitalisation of businesses but improve the targeting of highly specialised ICT-related job posts in businesses.Continue to support the digitalisation of public administration but focus more on centralised services, better interoperability between institutions, and increasing citizen awareness of digitalised services.
Impact evaluation of measures on healthcare under the EU Fund investments OP, 2014-2020, in Lithuania
2014–2020 m. ES fondų investicijų sveikatos srityje poveikio vertinimas
http://www.bgiconsulting.lt/en/, info@bgiconsulting.lt
The evaluation examines 34 measures addressing healthcare services. Data used go up to 01/09/2022.
To assess the main results and impact of the 34 measures financed by the OP to improve healthcare services.
- Analysis of statistical and monitoring data.- Survey of 144 project promoters.- Semi-structured interviews with 34 beneficiaries.- Focus group discussion with 7 representatives of NGOs.- Cost-effectiveness analysis.- ToC building and testing.
- The mortality rate of the population aged 0-64 due to Circulatory system diseases has been declining in target territories, from 2014 to 2021. EU support is estimated to have had positive, though limited, effects on this.- The mortality rate from cerebrovascular diseases in target territories has been decreasing from 2014 to 2021. EU funding primarily contributed to network-based investments in medical equipment of stroke treatment centres and care hospitals.- The mortality rate from malignant neoplasms has been decreasing from 2014 to 2021. EU Funds’ investments have a direct impact on this decrease, though impact is expected to be larger when projects will be completed.- The mortality rate due to external causes has been declining from 2014 to 2021, but impact of EU funding is not yet evident because of implementation delays.- The analysis of project efficiency shows that in most cases similar investments led to similar results.- Effectiveness of measures has been sometimes hindered by lack of cooperation between health authorities at national and regional level.
Not all planned investments have been completed.
- Strengthen cooperation with NGOs, communities and businesses to expand the range, supply and quality of healthcare services.- Support collaboration between general practitioners and public health bodies to increase health literacy and promote healthy lifestyles among different population groups.
Ex-post evaluation of the Kujawsko-Pomorskie OP, 2014–2020
Ewaluacja ex-post Regionalnego Programu Operacyjnego Województwa Kujawsko-Pomorskiego na lata 2014-2020
The evaluation examines the extent to which the OP achieved its objectives.
To evaluate the effectiveness of the OP in achieving its strategic objectives, particularly in relation to the promotion of economic development, digital transformation and environmental sustainability​.
Desk research.Analysis of monitoring and statistical data.17 interviews with representatives of institutions.Survey of 361 beneficiaries.3 group interviews with stakeholders.
Support for SMEs improved regional competitiveness, though collaboration between these and research centres remained limited.A significant reduction of CO₂ emissions and improvements in waste recycling were observed, which were in line with sustainability goals.Labour market activation programmes were highly successful, particularly for the long-term unemployed and individuals over 50.The limited adaptation ability and administrative complexity of financial instruments reduced their effectiveness.Limited coordination among institutions hindered the full realisation of Programme synergies.Urban areas benefited more from interventions than rural ones, and this tended to increase regional inequalities.
The evaluation was made more difficult by the fact that some respondents had changed their responsibilities or workplace, and that in some cases respondents were unable to recall details on the implementation of the OP.
Strengthen partnerships between businesses and research institutions to foster innovation.Increase financial incentives for boosting the use of energy efficiency solutions and renewable energy.Tailor workforce activation policies to better support marginalised groups.Simplify procedures and enhance flexibility in financial support mechanisms.Implement targeted support for rural areas to reduce development disparities.Improve collaboration mechanisms across various levels of governance and policy fields.
Market study to assess the relevance of a CO2-Loan-Fund in Sachsen-Anhalt for the 2021-2027 period
Marktstudie zur Relevanzprüfung eines CO2-Darlehensfonds für das Land Sachsen-Anhalt
The evaluation explores the possibility of using a loan-fund for measures on energy efficiency in Sachsen-Anhalt in 2021-2027.
To assess the relevance, coherence and reception of a possible loan-fund for reducing CO2 emissions and improve energy efficiency in Sachsen-Anhalt.
Official statistics.Expert panels.Interviews with beneficiaries.
A CO2-loan-fund could be used for financially weak municipalities, which view the use of a repayable and amortisation grants favourably.Non-profit organisations are only partly attracted by repayable subsidies, as they would compete with low-interest loans from banks.Strong companies show no interest in a CO2-loan-fund.
There are some limitations in the monitoring data.
Define specific financial instruments for municipalities, non-profit organisations and companies, avoiding overlap with existing loan schemes.Maintain the current subsidy schemes for CO2 reduction at the same time.
Evaluation of the effects of Cohesion Policy on the main socioeconomic indicators, in 2014-2020, in the Czech Republic
Efekt politiky soudržnosti v České republice z pohledu celonárodních socioekonomických ukazatelů (součást velké evaluace: Výsledková tematická evaluace Dohody o partnerství pro období 2014–2020). Závěrečná zpráva.
All OPs 2014-2020 except the Technical Assistance OP
Ernst and Young, romana.smetankova@cz.ey.com
The evaluation examines the effects of support from Cohesion Policy Funds in 2014-2020, on the socioeconomic development of the Czech Republic.
To assess the impact of ESI Funds on the development of Czech Republic, in relation to changes in the main socioeconomic indicators.
Analysis of monitoring data and official statistics. Regression analysis used for panel data analysis. Surveys and interviews. CIE (CR-SEQDD method).
Despite significant investments in highway, road and rail infrastructure, including from ESIF, the Road Network Quality Index remains one of the lowest in Europe. Between 2015 and 2022, the regional innovation index of Czech republic has improved significantly. However, ESI Funds support seems one of the components of the overall R & D and education ecosystem, but not as a decisive factor and impact cannot be proven statistically. ESI Funds seem to have contributed to improve the “people at risk of social exclusion” index, also through the creation of social enterprises, increased social housing and access to modernised health care services. Support from Cohesion policy plays a significant role for the areas of energy efficiency and reduction of CO2 emissions. ESI Funds helped decrease the mortality rate of SMEs, but there is no effect on creation of new companies, especially innovative ones.
Impact of ESI Funds on socioeconomic indicators is very difficult to demonstrate quantitatively, because of: their small share on total public expenditure as well as the broad focus of ESIF investments, the little time elapsed since the end of projects, not fully suitable indicators to cover the areas supported, and the potential presence of reverse causality between the variables examined.
Evaluation of interventions under SO 1.3 and SO 2.3 of the Administrative Capacity OP, 2014-2020, in Romania
Servicii de evaluare a principalelor interventii in cadrul axelor prioritare ale POCA - LOT 3 - Asistenta financiara acordata in cadrul OS 1.3 si OS 2.3. Raport de evaluare 2020
Archidata, Civitta, NTSN Conect, Structural Consulting Group
Archidata, https://www.archidata.it/; Civitta, https://civitta.ro/; NTSN Conect, http://www.ntsn.ro/contact_ro.htm
The evaluation examines the measures under SO 1.3 “Development and implementation of standard systems and modern and efficient management tools for judicial institutions” and SO 2.3 “Ensuring greater transparency and integrity in the judiciary system to improve access to, and quality of, the services provided“.
To assess the effectiveness, efficiency, impact and sustainability of interventions in respect of management tools for the judicial system and improving access to it, and the extent to which needs are being met.
Documentation analysis and literature review.Analysis of monitoring data.19 interviews with European fund managers.Interview with the MA and two with rejected applicants.Surveys of final beneficiaries and legal professionals.5 case studies on the projects implemented.Focus group.Panel of experts.
As of November 2020, 4 out of 33 projects had been completed.Targets were not achieved for most output and result indicators.Projects under SO 1.3 (for improving the judicial system) have helped to reduce the duration of criminal investigations.Projects under SO 2.3 (on improving access to the judicial system) have helped to improve the quality of judicial services and access to justice (output indicator targets were achieved or exceeded).The greatest impact is expected from SO 1.3 projects for strengthening institutional capacity, supporting institutional procedures and streamlining management.Projects under both SOs will make an important contribution to the computerisation of the judiciary.Unexpected effects include strengthening communication and cooperation between project partners, increasing capacity to access grants and improving legal knowledge.The procurement process and the COVID-19 pandemic have led to delays in implementation.Communication activities concerning both potential beneficiaries and the general public are well defined.
Because of the limited progress in project implementation, only the expected impact is assessed.
Set indicator targets on a realistic basis to ensure that they are achieved.Limit the changes in project officers in the MA by providing the necessary staff to monitor projects and perform other tasks that are not directly their responsibility.Carry out an in-depth analysis of the needs of potential beneficiaries and target groups.Given the COVID-19 pandemic, extend all projects.Form a group of public procurement experts to support applicants encountering problems in the procurement process.Carry out an analysis at project level of reimbursements made, savings and financial reallocations.In the future, organise non-competitive calls for all strategic organisations in the judiciary.In future projects, organise inter-professional training sessions.
Evaluation of the England ERDF OP, 2014-20 – Phase 2
National Evaluation of English ERDF Programme 2014-20. Phase 2 Report:Interim Impact Evaluation
The evaluation examines the results and first effects of the OP, with a specific focus on interventions for SMEs under TOs 1, 2, 3 and 4. Data go up to the end of 2019 and partly to October 2020.
To assess the validity of strategy, the implementation and delivery of the projects, progress in achieving targets and the first effects of the measures supporting the SMEs up to end-2019.
Analysis of official statistical data and monitoring data.Interviews with those responsible for programme management.Desk analysis of documents on the results of 80 projects under TOs 1, 3 and 4.Survey of 4,200 enterprises receiving support for business start-up, SME competitiveness, RTDI and resource efficiency.Counterfactual analysis using firm-level data (from ONS Business Structure Database (BSD)). CIE on firms supported in 2016-2018, using a mix of PSM and DID. Spatial and time-differencing methods for assessing territorial effects.
The OP has made a satisfactory progress towards the targets. Implementation of low carbon economy measures was difficult because of the large budget for this and limited local capacity to manage complex projects.Simplification of delivery procedures for business support produced benefits as well as an understanding of business needs.Businesses supported report high added-value from the funding received, an increase in employment, especially for SMEs accessing grant and repayable finance, and to a lesser extent for start-ups.According to the counterfactual analysis, support for businesses is responsible for 16-38% (depending on the year) of the 34,000 gross jobs created.Enterprises supported also increased turnover more than the control group. At territorial level, projects implementing place-based measures (e.g. incubators, broadband infrastructure) are estimated to have created up to 10,500 jobs.
The fact that only 80 projects were analysed limited the number of PAs covered and did not permit robust evidence to be identified at IP level. BSD data available only enabled businesses supported in 2016-2018 to be considered.
Monitor the emerging local impact of Covid-19 and the potential need for further action.Monitor delivery organisations which may face challenges due to Covid-19.Monitor potential loss of projects, especially in PAs 1 and 4.Improve integration between the national programme and local projects as regards the low carbon economy (avoid duplication).Improve the quality of project documentation.
Evaluation of the FIF and ZEMIGRA measures supported by the Sachsen-Anhalt ESF OP, 2014-2020
Evaluationsstudie zu den Förderprogrammen FIF und ZEMIGRA
Ramboll Management Consulting GmbH, Chilehaus C - Burchardtstraße 13, 20095 Hamburg, www.de.ramboll.com, Tel.: +49 40 30 20 20-137
The evaluation examines the FiF and ZEMIGRA measures, the first (‘Skilled Workers in Focus’), a central advisory and network scheme to maintain and attract skilled workers; the second (‘Centre for Migration and Labour Market’) providing support to companies, volunteers and regional organisation to integrate migrants into the labour market through advice, training, networking activities and an exchange platform.
To assess the achievements of the measures supported to maintain and attract skilled workers (FIF) and to provide advice, training and other support to ease the integration of migrants into the labour market (ZEMIGRA), the socio-economic conditions and ways of improving the two measures in the next programming period.
Desk review30 interviews with beneficiaries and 20 with stakeholdersAnalysis of monitoring and statistical dataWorkshop with stakeholders.
Target groups reached are quite diverse: those re-entering the labour market, returning from abroad or with a migration or refugee background, SMEs and the care sector for FiF and mainly companies for ZEMIGRA.The main challenges are to reach low-skilled workers, small businesses and those with little contact with the support system.Measures are not adequately known in rural areas and by local support organisations.Target indicators were reached and exceeded for FiF but not completely so for ZEMIGRA, partly because this is a new measure introduced in 2017 while FiF has been in place since 2007.Both measures are complementary with similar forms of support at the regional level.FiF has an important role in raising awareness, activating and motivating beneficiaries; ZEMIGRA in establishing networking and cooperation besides raising awareness.Positive factors for both measures are their accessibility, easy communication channels and good cooperation between those involved; less positive factors are high stuff turnover, technical barriers and low awareness of them.The need of skilled workers remains relevant for 2021-2027.
Continue support of the two measures in the next programming period.Ensure coordination between different policy measures in this area avoiding overlap.Continuing training for skilled workers, including those from abroad.
Second evaluation of the Technical Assistance OP, 2014-2020
Segunda evaluación Programa Operativo de Asistencia Técnica 2014-2020
This is the second evaluation of the national Technical Assistance OP, with a budget of EUR 35 million and providing support to ESF and YEI OPs (training of managers, evaluations and studies, etc.).
To assess the implementation, efficiency, effectiveness and impact of the OP.
Desk analysis;analysis of monitoring data and indicators;survey of direct beneficiaries of the OP (33 responses out of 52);two focus groups and interviews.
The OP objectives are in line with the problem identified. The main weakness concerns the system of indicators.OP progress in terms of physical indicators is satisfactory, though funding is highly concentrated to two beneficiaries.Operations are helping the management of the Funds through capacity building and external contracts.Beneficiaries regard the effectiveness of the evaluations funded by the OP to be low and communication activities are also not very effective.Training and the software application FSE14-20 are the most effective measures for both multi-regional and regional intermediate bodies.There are a lack of indicators and concrete measures on gender equality.The OP has had a positive impact on the functioning of the ESF in Spain, highlighted by the increased exchange of experience and good practice, the increased number of positive audit reports and the reduction in the financial error rate.There are changes in final beneficiaries that would not have occurred without financial support. The IBs recognise that there have been improvements in the management of the ESF in Spain, but are uncertain whether these are a consequence of the OP.
The data available are insufficient to carry out efficiency, effectiveness and impact analysis.
Identify needs better;strengthen participation mechanisms;complement the current system of indicators;focus on introducing innovative elements into the OP;review the communication strategy and targets;refocus operations on evaluation;strengthen operations to reduce the administrative burden;improve the IT tool.
Impact evaluation of the Italia-Austria Interreg OP, 2014-2020 - Year 2022
The evaluation examines the contribution of the measures supported to the objectives of the OP.
To assess the impact of the measures financed by the OP.
Analysis of output and result indicators.ToC testing.Analysis of documents.Survey of lead partners (55 completed answers) and CLLD stakeholders.8 interviews with CLLD lead partners.12 case studies.
All the measures analysed for each SO have achieved result and output targets and seem to be effective.Measures under SO 1 “Enhancement of the R&D capability of the regional economies” have helped to involve researchers in the area, to spread know-how, to generate investment in machinery and infrastructure, and to develop new competences.Measures under SO 2 “Promotion of R&D investment through cross-border cooperation” have helped to create or maintain jobs for researchers, establish R&D networks, promote innovation and competences.Projects under SO 3 have supported the enhancement of innovation in companies.Measures under SO 4 “Valorisation of the natural heritage to improve tourism” have helped to do this and to increase tourist numbers.Measures under SO 5 “Enhancement of cross-border institutional cooperation“ have contributed to risk prevention, civil protection, sustainable mobility and healthcare.CLLD measures under SO6 have helped to improve cooperation between stakeholders through greater awareness and spread of good practiceThe OP has helped to reduce linguistic, administrative, and legal barriers in a limited way.According to survey respondents, the major barriers to the operation of the OP are reporting procedures and a lack of information about calls for proposals.
There was only a limited number of responses to the survey on measures under SO 3.
Update the underestimated output/result targets.Define more effective measures to improve R&D.Include the possibility of indicating the thematic area to which the project contributes in application documents.Identify the thematic areas in which to improve institutional cooperation in the next programming period.Reduce the administrative burden.
Evaluation of the communication strategy of the OP Schleswig-Holstein ESF 2014-2020
Durchführungsevaluierung: Bewertung der Umsetzung und Ergebnisse der Kommunikationsstrategie, der Maßnahmen zur Berücksichtigung der übergreifenden Fördergrundsätze gemäß Art. 7 und 8 VO (EU) 1303/2013 sowie der Rolle der Partner
Moysies & Partner - OT- und Managementberatung, Mittelweg 56a, 20149 Hamburg
Mid-term evaluation of the INNOship programme financed by the Smart Growth OP, 2014-2020, in Poland
Katarzyna Gano, Monika Kordowska, Katarzyna Krok
The evaluation examines the innovation support programme for the shipbuilding sector, the barriers to the implementation, the appropriateness of the measures and their effectiveness. The study covered the implementation period up to December 2020.
To assess the validity of the programme's assumptions and the effects of the programme in terms of added value for the shipbuilding sector.
Analysis of monitoring data and official statistics. Survey of beneficiaries and unsuccessful applicants (no information on technique and sample). 4 individual in-depth interviews. Expert panel. Analysis carried out by 2 independent experts.
The needs justifying the funding of the programme are still valid and the programme is a unique source of R&D financing for the shipbuilding sector in Poland.The solutions developed under the INNOship programme respond to the needs of the specific entrepreneurs but do not create added value for the sector. In addition, they are not sufficiently innovative.The lack of formalised cooperation within the projects has a negative impact on the scalability and synergies of the solutions developed.Long and complex procedures make it difficult to apply for support.
The early stage of implementation limits the ability to fully assess the effectiveness and durability of the effects.
Establish priority areas of support, including environmental issues and low emission standards.Introduce solutions for the implementation of projects in partnership.Intensify information and education activities.Introduce solutions to limit the risk of foreign entities acquiring ownership of project results.
Evaluation of the contribution to the Europe 2020 Strategy of the Veneto ESF OP, 2014-2020. Second report
Valutazione del contributo del Programma alla Strategia dell’Unione per il servizio di Valutazione operativa e strategica del POR FSE 2014-2020 - secondo report
IRS, https://www.irsonline.it/ - PTSCLAS, https://www.ptsclas.com/ - GRETA, https://www.greta.it/index.php/it/
The evaluation examines the contribution of the OP to the achievement of the Europe 2020 objectives up to 31 December 2021.
To assess the contribution of the OP to the pursuit of the Europe 2020 objectives up to end-2021.
Documentation analysis.Analysis of statistical data.Analysis of monitoring data.
PA 1 has contributed to the employment objective and PA2 to the social inclusion one.Interventions financing High Technical Institutes (ITS) are those contributing most to increasing tertiary education of those aged 30-34.Support for scholarships, applied research and training contribute to the R&D objective.As of end-2021, the level of achievement of targets was satisfactory.
For some output indicators monitoring data only go up to 2018 or 2020.
Socio-economic and SWOT analysis in preparation for the Baden-Württemberg ESF+ Programme, 2021-2027
Sozioökonomische Analyse und SWOT-Analyse für Baden-Württemberg im Rahmen der Erstellung des OP des ESF in Baden-Württemberg 2021-2027
Stefan Feldens and Dr. Friedrich Scheller
ISG Institut für Sozialforschung und Gesellschaftspolitik GmbH, Weinsbergstr. 190, 50825 D-Köln, E-Mail: feldens@isg-institut.de, Tel.: + 49 221-130 655-54
This is a socio-economic study and SWOT analysis of the region to prepare for the 2021-2027 ESF+ Programme.
To identify ESF+ financing priorities in the region for the 2021-2027 programming period.
Population is ageing implying a need to adapt infrastructure to future needs.The region’s international competitiveness has been maintained, especially as regards manufacturing.Employment rates have increased slightly, although gender differences still persist in terms of earnings and occupational mobility.28% of workers in 2016 are estimated to be at risk of losing their jobs because of automation and digital technologies, much more than in 2014.Unemployment declined between 2014 and 2018, but the at-risk-of-poverty rate in 2017 was 12%.The proportion of early school leavers increased by almost 2pp between 2014 and 2017 to 6.6 %.
Evaluation of High Technology Network in Emilia Romagna, 2014-2020
Rapporto Tematico: Rete dell’Alta tecnologia
www.irsonline.it - Via Castiglione 4, 40124 Bologna - irsbo@irsonline.it; nomisma.it - Strada Maggiore, 44 – 40125 Bologna - info@nomisma.it
This evaluation examines business research and innovation under the Emilia-Romagna ERDF OP, 2014-2020 (IP 1.2), focusing on 1) Collaborative research and development projects for businesses (Actions 1.1.1 and 1.1.4); and 2) Strategic industrial research projects aimed at the priority areas of the regional Smart Specialization Strategy (Action 1.2.2).
To assess the results of the research projects and their effects on enterprises and the regional innovation system.
Desk analysis of programming and implementation documents.Statistical analysis of monitoring data.For Action 1.2.2, on-line survey of beneficiaries – research laboratories of the Regional High Technology Network (response rate 86%).For Action 1.2.2, on-line survey of potential recipients – companies who participated in the research projects (response rate 41%).
54% of the 136 collaborative R&D projects financed up to mid-2019 were in the S3 area of "mechatronics" and 46% on “sustainable development”.Almost all the beneficiaries are innovative companies that already had an in-house R&D unit. 20% had no previous collaboration with universities and/or research laboratories before.248 people have been hired in companies to carry out the research projects; only 13% of companies experienced critical problems identifying and recruiting specialised personnel.241 cases of collaboration have been activated between the companies supported and research laboratories, 84% in the region. Only 11% experienced difficulties in finding suitable project partners.24 patents have been filed at the conclusion of projects (target is 50).Over 77% of the companies supported did not participate financially in the project.The relationships between companies and research centres has been strengthened. In most cases, collaboration led to further joint research initiatives.Almost all projects have ended with the creation of industrial prototypes. The researchers supported in most cases obtained an additional grant from the research centre or were hired by companies after projects came to an end.The main difficulties concern the administrative burden on submission and management of projects and the transfer of research results to companies.
Further simplify the administrative procedures for access to regional funding under Action 1.2.2 to facilitate participation of SMEs.Provide more technical assistance on submission of applications and implementation of projects.Involve local businesses more in defining the contents and implementation procedures of regional calls.Focus more on the results actually achieved by projects than on the administrative requirements.Extend funding to encourage the greater and more active involvement of SMEs in the implementation of projects and in the industrial exploitation of research output.Increase the maximum duration of research projects and size of partnerships.
Mid-term evaluation report of the Portugal Human Capital ESF OP, 2014-2020
Avaliação Intercalar do Programa Operacional Capital Humano
The report presents the results of the mid-term evaluation of the OP. It is based on a theory of change and it combines the findings of recent thematic evaluations with those of a specific survey carried out in this evaluation. Monitoring data go up to December 2021.
To assess the impact of the OP as well as its efficiency, effectiveness, relevance, coherence and EU added-value, and to make recommendations for the implementation of the 2021-2027 ESF+ OP ‘Demography, Qualification and Inclusion’.
Theory of change as part of theory-based approach.Desk research.Interviews with IB and beneficiary bodies.Survey of participants and beneficiary bodies.3 focus groups.1 case study on Technical assistance.
The OP reached over 900,000 participants (including 300,000 young people supported in professional courses and 74,000 teachers).By end-2021, the value of both output and result indicators were in line with the 2023 targets.12 months after exit, 45% of training participants were in employment.The average cost per participant ranges from EUR 410 (under PA4 ‘Quality and innovation of the education and training system’) to EUR 5,452 (under PA1 ‘Promotion of young people’s education’).The Human Capital OP is very complementary to the regional OPs, but less so to the ERDF programme.Without the ESF, the intensity and quality of vocational education would be significantly lower, 93% of respondents reporting that they would not have gone on to higher education otherwise.The ESF has introduced innovations in the education and training system such as the adoption of new teaching practices.
Ensure more balance between the resources allocated to training for young people and those to training for adults.Allocate progressively ESF+ to the most innovative aspects of education and training policies.Increase articulation between ESF+ and ERDF support for innovation and competitiveness.Involve IBs and beneficiaries more actively in the communication strategy of ESF+ programmes.Increase the use of the simplified cost option.
Ex-post evaluation of the Smart Specialisation Strategy under the Puglia OP, 2014-2020, in Italy
Rapporto di Valutazione ex post 2014-2020 “Strategia di specializzazione intelligente”
The evaluation aims to assess the progress and results of the Smart Specialisation Strategy (SmartPuglia 2020) for the 2014-2020 period, focusing on its effectiveness in fostering technological innovation and regional competitiveness. Data used go up to 2019.
To assess the effectiveness of S3 measures, the achievement of specific objectives (e.g., promotion of innovative markets and digitalisation), their alignment with the strategy, the results and impact on the regional development, and to identify strengths, weaknesses, and areas for improvement.
Document analysis.Analysis of monitoring data.Analysis of statistical data (ISTAT, SVIMEZ, EUROSTAT, IMF).Online survey of 185 companies actively contributing to the regional S3.
Measures to promote technological innovation in processes and products helped improve productivity and reduced costs.Employment has grown, particularly among young graduates in technical and scientific fields.Technological upgrades allowed companies to expand products and services in the national market, but internationalisation was limited.Many of the companies declare they would have invested even without support from the ERDF measures, but this accelerated technological innovation and the acquisition of new knowledge.There is a geographic concentration of beneficiary companies, especially medium and large ones, and they show a higher concentration of researchers with advanced qualifications such as PhDs.
Policy implications are not set out in the report.
Evaluation of the ‘Impulses of labour market policy’ programme under the Hessen ESF OP, 2014-2020
Evaluationsbericht über das ESF-Programm „Impulse der Arbeitsmarktpolitik“
ISG Institut für Sozialforschung und Gesellschaftspolitik GmbH (ISG), www.isg-institut.de, Bennett@isg-institut.de
The evaluation examines the programme for improving the employability and integration into the labour market of disadvantaged groups. This final report builds on the interim report of 2019.
To assess the funding priorities, the target groups reached and the social needs addressed by the programme to increase the employability and labour market integration of disadvantaged groups, the implementation issues, results, effects and sustainability of funding.
Analysis of monitoring and project data.Interviews with 11 project promoters of completed projects.Survey of beneficiaries of completed projects, carried out in 2019 and 2022.
As of January 2022, 89 model projects were approved, 67 of them with different social innovation ideas. Almost all ESF financing has been absorbed.59 of 77 projects involving participants were completed by January 2022, reaching 6,653 participants, 75% of them women and 85% unemployed or inactive before starting the project.35% of those previously not working were employed after leaving the project.19% of all participants took part in further education or completed education or vocational training, 17% dropped out.Project costs per participant vary from EUR 651 to EUR 78,079, depending on the project.Project promoters claim that they achieved quite good rates of placement in work and training.Success factors were group dynamics, networking and personal support, while self-learning, handling digital technology and language barriers were considered the main obstacles for participants.Factors which promote project continuity are digital learning fostered during Covid-19 and support from the Land government, while inhibiting factors are lack of financial resources and changing social needs.
As of January 2022, there was no reliable information on the ‘Successful model projects’ result indicator. Not all project promoters were able to provide an assessment of the continuation of their project at the time of the interview.
Include a wider range of topics and take account of the shortage of skilled workers in selected areas in future calls for funding.Aim to attract new project promoters to generate new innovations.Assess the possibility of implementing specific measures for the unemployed or inactive (especially women) in rural areas.Involve those aged 55 and over more.Reconsider the ‘Successful model projects’ result indicator and its definition as a measure of success.Support project promoters in attracting alternative sources of finance and in public relations.
Social Inclusion and Community Activation Programme (SICAP)
Social Inclusion and Community Activation Programme - 2015 and of year report
The evaluation covers the Social Inclusion and Community Activation Programme (SICAP) which is aimed at reducing poverty and promoting social inclusion and equity through local, regional and national engagement and collaboration.
The main objective is to describe the main features of the programme and to present evidence on its performance. It provides details of programme participants, the type of support provided and the outputs achieved.
The evaluation uses qualitative methods, only:Desk research.Analysis of monitoring system data.Case study.The data used come exclusively from the programme Integrated Reporting and Information System (IRIS) and the commentary included in the End of Year Progress Reports prepared by Programme Implementers that also submitted the case studies.
As regards Local Community Groups (LCGs) (Goal 1):2,322 LCGs assisted in respect of training which in turn helped over 125,000 people.867 LCGs supported to participate in local, regional or national decision-making.213 LCGs assisted to leverage additional funding of EUR 3.2 million.143 LCGs assisted in developing social enterprises.As regards Educational support to individuals (Goal 2):16,705 people received education support.13,434 people participated in Life Long Learning courses.As regards employment support to individuals (Goal 3):23,546 people received support.1,337 people found jobs.4,687 people went into self-employment.As regards the Youth Employment Initiative:3,958 NEETs were supported.
The key policy implication mainly concerns how to operationalise programme requirements, including:Communication of changes in programme guidelines, registration forms and IRIS should be more effective and training better-timed.Barriers to participation need to be reduced.Engagement with specific target groups such as new communities, Roma, lone parents and young people needs to increase.Data recording needs to be improved.The coverage of the programme needs to be extended people experiencing hardship due to low income levels or in a precarious employment situation (e.g. self-employed, farmers).The requirement that 50% of those supported by SICAP should live in disadvantaged areas needs to be modified.
On-going evaluation of investment in growth and employment supported by the Hessen ERDF OP in the period 2019-2023
Begleitende Evaluierung des operationellen Programms für die Förderung von Investitionen in Wachstum und Beschäftigung in Hessen aus Mitteln des Europäischen Fonds für regionale Entwicklung (EFRE) im Zeitraum 2019 bis 2023 und Ad-hoc-Bewertungen zur Vorbereitung von Anträgen zur Änderung des IWB-EFRE-Programms Hessen
GEFRA - Joanneum Research Forschungsgesellschaft mbH - Kovalis, Meyer Stefan
info@gefra-muenster.de; prm@joanneum.at; meyer@kovalis.de
The evaluation examines SME business investment (ML 2.2.2); energy efficiency in SMEs, use renewable energies and establish recyclable material cycles (ML 3.1.1); applied energy research, pilot and demonstration systems, market penetration (ML 3.2.1); advice measures (ML 3.2.2); and municipal investment for the conversion of industrial, transport and military brownfield sites (ML 4.1.2).
To assess the implementation and effects (where possible) of SME business investment (ML 2.2.2); energy efficiency in SMEs, use of renewables and establishment of recyclable material cycles (ML 3.1.1); applied energy research, pilot and demonstration systems, market penetration (ML 3.2.1); advice measures (ML 3.2.2); and municipal investment in the conversion of brownfield sites (ML 4.1.2).
Desk reviewAnalysis of monitoring and statistical dataSurvey for ML 3.2.1 (73% of 26 project managers)Case studies for ML 3.1.1, 3.2.1 and 4.1.2Regression analysis for ML 2.2.2 using data from the Federal Office for Economic Affairs and Export Control (BAFA)Evidence from previous evaluationsInterview with experts and beneficiaries for ML 4.1.2.
As of August 2020, 60 supported investment in SMEs in the weaker parts of the region, with total expenditure of EUR 127.27 million, EUR 22.26 million of which was from the ERDF and the rest from private sources.Most targets have been achieved.435 new jobs were created (30% taken up by women) and 1,503 secured (23% filled by women).Investment per job created by ML 2.2.2 projects was EUR 278,989.Without investment, many ML 2.2.2 projects would not have been undertaken or would have been on a smaller scale.As of November 2020, 43 ML 3.1.1 projects had been approved as part of the PIUS-Invest programme, with average expenditure of EUR 915,000, 25% covered by the ERDF.Projects are expected to save over 15,700 metric tons of CO2 equivalent per year, at an average cost per ton of EUR 2,450.At least one organisational innovation is implemented by each project which is expected to improve the competitiveness of SMEs.Procedures were considered burdensome because of the administrative workload and documentation to be provided.As of October 2020, 19 ML 3.2.1 projects were funded as part of the FPG 960 programme.Expenditure was low mainly because of the introduction of new funding procedures.The administrative workload was considered excessive.As of October 2020, EUR 5.5 million of ERDF resources had been used to fund the establishment of the Hessen Regional Energy Agency.6 ML 4.1.2 projects were supported with EUR 2.5 million from the ERDF.Support helped revitalise old military and industrial buildings and clean up contaminated sites for commercial use. The funding also contributed directly to company relocation into urban area, removal of potential hazards from land and improvement of the attractiveness of cities.There were strong synergies with other urban development projects.Demand for funding was lower than expected.
Continue the support for ML 2.2.2 and ML 3.2.1 in the next programming period.Give a better definition of the regional aid landscape.Change the strategy to national SME support for innovative investment projects.Improve administrative procedures for ML 3.1.1 and ML 3.2.1.Assess the need for ML 4.2.1 support.
Evaluation of the implementation of PA2 “Knowledge and competences” of the Wallonie-Bruxelles ESF OP, 2014-2020
Évaluation de la mise en oeuvre de l’axe 2 « Connaissances et compétences »
https://www.bdo.be/nl-be/, https://sonecom.be/, https://www.ulb.be/
The evaluation examines the implementation and effects of measures under the PA.
To assess the relevance, coherence, effectiveness and efficiency of the measures under the PA up to 2021.
Literature review.Analysis of monitoring data and official statistics.Survey of final recipients.Interviews with training bodies.
Measures for increasing competences are relevant for territorial needs, though some of them should be identified more precisely.The distinction between measures targeting people and systemic interventions appears artificial and not always coherent.Most of the output indicator targets have been reached, though this is less so for result indicators.51% participants in training in 2018 had only short term contracts or were inactive 12 months after training, against 33% for participants in 2015.Result indicators are not able to capture long-term effects of the measures nor is the survey of final recipients, as it is too limited.Follow-up support for participants in training is very weak and seems focussed only on collecting a few data or encouraging potential participants to take up training opportunities.
Additional data on the employability and adaptability of participants in the measures are needed to assess efficiency and effectiveness in more depth.
Define needs more precisely and make use of unambiguous concepts.Encourage groupings of training bodies whose projects have the same objectives to create more synergies.Introduce new monitoring indicators able to better capture the effectiveness of measures.
Evaluation of the communication strategy of the Kriti OP, 2014-2020
Μελέτη Αξιολόγησης της Στρατηγικής Επικοινωνίας του ΕΠ Κρήτη 2014-2020: Με το Βλέμμα στην Περίοδο 2021-2027
Hellenic Mediterranean University, Department of Business Administration and Tourism
Estavromenos, 71004, Heraklion, Tel: 2810 - 379677, https://bat.hmu.gr/en/home/
Evaluation of the Interreg V-A-Germany/Mecklenburg-Vorpommern-Brandenburg-Poland OP, 2014-2020
Update der Impact-Evaluierung im Rahmen des Ziels „Europäische territoriale Zusammenarbeit“
INFRASTRUKTUR & UMWELT Professor Böhm und Partner
The evaluation examines the achievement of OP objectives. Data are as of December 2022.
Analysis of monitoring data. Interviews with the MA.
Overall, 62 projects were approved of which half under PA IV (Cross-border cooperation), a third under PA I (Nature and Culture) and the others under PA II (Transport and Mobility) and III (Education). Two thirds of them were completed by end-2022. Targets are achieved or in the course of being achieved. There were issues with data collection of the monitoring system because of a lack of a clear definition of indicators and methodological guidelines.
Clearly define indicators in the OP as well as the method to record them. Analyse early and systematically the risks of not achieving objectives and take appropriate action timely.
Evaluation of the effect of the Podlaskie OP 2014-2020 on the quality of vocational education in the region
Ocena wpływu wsparcia RPOWP 2014-2020 na jakość kształcenia zawodowego w województwie podlaskim
Joanna Jaworska, Oliwia Mazan, Maciej Maj
The evaluation examines the measures on vocational education under the OP, which are intended to improve competences and infrastructure of vocational schools.
To assess the effects of support for vocational education quality, its effectiveness, efficiency and usefulness.
Analysis of monitoring and official statistical data.Documentation analysis.10 individual in-depth interviews.Survey of 1,266 participants.1 policy implications panel.CIE (DiD).
Support has led to improvement of competencies and qualifications of students, which increase their chances in the labour market.A success factor is the universality and complexity of the support received by students.Support for teacher training is the most effective measure for teachers/instructors of practical vocational training.Cooperation between schools and employers is still critical, as there is inadequate communication of benefits to businesses.
Keep favouring a combined use of various forms of support for participants.Include measures to improve infrastructure along with those on competences.Ensure cooperation between vocational education institutes, universities and employers.
Evaluation of “Inter-company apprenticeship training” under PA1 (Sustainable and high-quality employment and support for labour mobility) of the Bayern ESF OP, 2014-2020
Endbericht. Evaluation der Förderaktion 8: Überbetriebliche Lehrlingsunterweisung
ISG - Institut für Sozialforschung und Gesellschaftspolitik GmbH;https://www.isg-institut.de
The evaluation examines to what extent participants of apprenticeship courses (ÜLU) improve their vocational skills and knowledge and find employment following the courses’ completion. It covers the implementation period between 2014 and 2020.
To assess whether the apprenticeship courses improve the vocational skills and knowledge of apprentices and compensates for deficiencies due to an increasing specialisation of training establishments or a lack of adaptation to current technical developments in companies.
Literature review pertaining to ÜLU.Analysis of statistical data (Bavarian Crafts Day (BHT), Federal Institute for Vocational Education and Training (BIBB), etc.).Analysis of internal administrative documents (Chamber of Crafts)Analysis of monitoring data.Online surveys of participants who provided their emails (approx. one third of participants) with an average response rate of 15% (four survey waves in total).
85.2% of participants have successfully completed their training, exceeding the objective of 67%.The quality of training is positively assessed by the majority of respondents.The proportion of participants who have become acquainted with new artisanal processes, new technical equipment, new fields of activity or environmentally friendly procedures through the courses remains below expectations.Participants were more critical about the courses’ capacity to compensate for deficiencies of the training system.The transfer rate into job for those who successfully completed training was 52%, which is lower than the average for all training occupations in Bavaria (75%).
Analysis of the inclusion of people with disabilities in the Employment and Social Inclusion OP and OP for the implementation of YEI in mainland France and outermost regions
Note d’analyse de la prise en compte des personnes en situation de handicap dans les programmes opérationnels nationaux (POn) FSE et IEJ
OP for the implementation of YEI in mainland France and outermost regions; NOP Employment and Social Inclusion ESF 2014-2020
This is an evaluation of the inclusion of those with disabilities in the national ESF and YEI OPs 2014-2020 in France.
To assess the effects of the OPs and their progress toward achieving their targets.
Analysis of data from the monitoring system (MaDemarcheFSE);survey of a representative sample (4,100) of beneficiaries (with a response rate of 15%).For EU comparisons, data are taken from the Commission's Open Data portal.
For the national ESF OP:about 93,000 people with disabilities participated in one of the activities supported by the OP (6.4% of total participants);at the end of participation, 27% were in employment (as compared with 32% of all participants) and 7% were in training (the same as for all);6 months after participation, 47% of those with disabilities were in employment (as compared with 56% of all participants). For the national YEI OP:6,300 people with disabilities participated in one of the activities supported by the OP (2.1% of all participants);at the end of participation, 25% were in employment (as compared with 34% of all participants) and 15% were in training (as compared with 12% of all participants).
The figures in the report go up only to 4 January 2018.
In order to increase the participation of the disabled in ESF programmes, there is a need to:conduct information and communication campaigns as well as targeted training so that more associations for the disabled can obtain ESF support;launch calls for projects specifically targeting the inclusion of people with disabilities;to reduce the administrative burden on organisations already supported by the OPs and to introduce simplified payment and audit schemes, as well as to introduce simplified procedures for first-time applicants and smaller organisations.
Assessment of the availability of support for NGOs under the Kujawsko-Pomorskie OP, 2014-2020
Ocena dostępności RPO WK-P na lata 2014-2020 dla organizacji pozarządowych
Evaluation Unit in Marshal's Office of the Kuyavian-Pomeranian Voivodeship in Toruń, Department of Regional Development
Plac Teatralny 2, 87-100 Toruń, , Tel. 56 62 18 653, www.mojregion.eu
This evaluation examines the possibilities and limitations relating to NGO applications for financial resources under the Kujawsko-Pomorskie regional OP for 2014-2020.
To identify the possibilities and limitations relating to NGO applications for financial resources.
Desk research (programming documents, applications).CAWI/CATI interviews with NGOs.
Most applicants have limited experience of applying for EU funds.Applicants’ poor knowledge of the application process led to the rejection of most applications.The main barriers are related to the detail and complexity of documents, as well as the amount of information included and requested in the application and by financial rules.There is an overall scarcity of knowledge about the OP.
There is a need to:Simplify the requirements for small scale projects.Avoid issuing multiple calls in the same period.Improve the OP website in terms of information provided.
Evaluation Report 2017 on the Employment, human capital and social cohesion OP, 2014-2020
Έκθεση Αξιολόγησης 2017/Ε.Π.Απασχόληση, Ανθρώπινο Κεφάλαιοκαι Κοινωνική Συνοχή
Marathonodromou 83, Marousi tel:+30 210 672 5966, remaco@remaco.gr, G. Papandreou 5,P.O.Box 71306, Heraklion Crete, tel: +30 2810 361242 , info@etam.gr
This is the 2017 evaluation report on the Employment, Human Capital and Social Cohesion OP in Cyprus assessing progress in implementation as well as effectiveness and efficiency.
To assess progress in implementation and the effectiveness and efficiency of the OP.
Desk Analysis;interviews;analysis of indicators;analysis of project documentation.
70% of funding has been allocated;the efficiency of the OP is not particularly high;the absorption rate is rather low;there is a significant delay in integrating data from the measures supported.
The number of projects undertaken is still small, so relatively little data are as yet available.
There is a need to accelerate staff recruitment and the external outsourcing of staff, as well as to improve procedures.
Evaluation of PA C ‘Investment in skills and lifelong learning in education, training and vocational training’ under the Berlin ESF OP, 2014-2020
Ergebnisse und Wirkungen in der Prioritätsachse C
Dr. Ulrike Kaden, Marco Puxi und Dr. Katja Seidel (ISG), IfS Berlin
ISG Institut für Sozialforschung und Gesellschaftspolitik GmbH, www.isg-institut.de - IfS Institut für Stadtforschung und Strukturpolitik GmbH, www.ifsberlin.de
The evaluation examines PA C of the OP aimed at supporting young people with multiple disadvantaged, in particular those with migrant background, to obtain a diploma or participate in training under SO C1 (measures for those aged 15 to 19) and SO C2 (measures for those aged 20 to 23).
To assess the impact of the measures supported for disadvantaged young people under SO C1 (for those aged 15-19) and SO C2 (for those aged 20-23).
Desk review. Analysis of monitoring data. Interviews with service providers and experts. Surveys of beneficiaries. Case studies of measures. Focus groups. Counterfactual analysis (PSM).
By end-2019, 783 projects had been approved under PA C1 and C2 and EUR 154 million invested. 60% of participants had a migrant background and 18% a refugee background, 58% were male and over half had only basic education. Over half were in education, 30% unemployed and 8% inactive. Only a few participants aged 15-19 under SO C1 succeeded in integrating into regular education and many, especially those with a migrant background, had a low level of skills even after leaving because of dropping out and remained inactive or unemployed. Under SO C2, support increased the likelihood of employment, although it remains hard for participants to move out of unemployment. Both SO C1 and C2 measures contributed to improving professional qualifications. COVID had a negative impact on vulnerable target groups. The temporary termination of face-to-face teaching and the more limited opportunity of gaining professional experience made the transition from school harder for young people with special needs, while employment opportunities for the low-skilled were reduced. Positive side effects are that promoters gained knowledge, communication between service providers was improved and workload in schools were reduced.
COVID affected the collection of data through surveys and interviews in 2020.
Focus better on sub-target groups. Clarify the importance of integrated language support. Design education and support chains better. Strengthen cooperation of sponsors and partners to better address the multiple problems of beneficiaries. Organise the transition into training or education better. Implement cross-cutting objectives better.
Annual evaluation report of the Marche ESF OP, 2014-2020 – 2019
The evaluation examines the level of implementation of the Marche ESF OP in 2019.
To assess the relevance, implementation of measures implemented, the achievement of physical and financial targets and coverage of target groups.
Desk analysis.Analysis of monitoring and statistical data.Interviews with relevant stakeholders.Survey of the population in the region on their awareness of the ESF (1,000 surveyed).Survey of participants of measure 8.v to assess their job market situation 6 months after support (320 interviewed, 42% response rate).Evidence from other evaluations.
The OP strategy remains relevant and coherent with other regional polices.The pace of implementation is slower than in the previous period, mainly because of a late start and organisational/management changes after the reform which modified the role of provinces in Italy (the COVID emergency might cause even further delays).No IPs reached over 50% of targets and result indicators remain low.Unemployed represent 75% of the OP target groups and 10% of them have been reached by measures.Over 3,064 people have participated in ESF measures (1,621 in training and 1,443 in grants and traineeships), 45% being employed 12 months after exiting support, with higher rates for training under IPs 10.iv and 8.i and lower rates for sectoral training (IP 8.i) and training for young people (IP 8.ii).The difference in employment rates between those supported and those not supported is not statistically significant. It is particularly negative for the training of young people, mainly because of those supported being more disadvantaged.Those participating in traineeship had an employment rate 5 percentage points higher than those not supported after exiting support.Horizontal principles are adequately taken into account.ITI, in the form of innovative urban infrastructure and the development of human capital, has been undertaken in Ancona, Ascoli Piceno, Fermo, Macerata, Pesaro-Urbino and Fano.
Evaluation of the impact of measures supporting social infrastructure and the social economy, as well as measures on transport, environment, and culture under the Notio Aigaio OP, 2014-2020
Αξιολόγηση της επίπτωσης των δράσεων υποστήριξης κοινωνικών υποδομών, χρηματοδοτικής ενίσχυσης των Κοιν.Σ.Επ. καθώς και των δράσεων μεταφορών, περιβάλλοντος και πολιτισμού
The evaluation examines the implementation and impact of measures promoting the social infrastructure and social enterprises, transport, environment and culture under the OP.
To assess the achievement of targets, the effectiveness, efficiency, and impact of the measures financed under the OP on social infrastructure, the social economy, transport, environment and culture.
Literature review.Analysis of monitoring and statistical data.Interviews with Managing Authorities and beneficiaries.Survey of beneficiaries.Thematic focus groups.
Beneficiaries are satisfied with the results of the measures (achievement of targets) but less so in relation to impact (e.g., lifting insular isolation, reduction of disparities, making the region a leading tourism destination).Educational infrastructure was mainly implemented on islands with large population. This has sometimes led to young families moving from little islands to more advantaged areas.Projects on welfare infrastructure relate to a wide range of services (especially for vulnerable people) and had a particularly positive impact and a significant multiplier effect, as well as projects promoting environment and energy efficiency.Pilot projects on social cooperative enterprises faced several criticalities, such as limited administrative burden and limited administrative capacities of beneficiaries. The sustainability of the projects is also uncertain.Transport infrastructure projects had a particularly positive impact, both for the population of the region and for tourism. However, the needs of smaller islands, lacking airports and large ports, remain significant.In relation to measures in the area of culture, economic and social benefits per financial unit spent are reported to be larger in small communities and isolated islands, than in large ones.
Continue upgrading and expanding health and educational infrastructure and improve access for inhabitants of small islands.Adopt simplified cost options and reduce bureaucratic requirements of project management, monitoring and audit for projects on social cooperative enterprises.Continue investing on transport projects, not on the basis of a strict cost-benefit analysis and focus on road safety and the needs of smaller islands.Integrate cultural projects into networks and ensure ownership of projects by local actors.
Evaluation of the implementation of the Technical Assistance OP, 2014-2020, in Romania – report for 2021
Al treilea raport de evaluare a Programului Operațional Asistenta Tehnica 2014-2020, Romania
Michele Giove, Laura Trofin, Sofia Stoican, Vasilica Ciuca, Eva Militaru, Speranta Pirciog, Madalina Popescu, Liliana Lucaciu, Valentin Alexe, Alexandra Maria Baciu, Andra Corina Jurj, Maria Banu
ERNST&YOUNG SRL, https://www.ey.com/ro_ro; INCSMPS, https://new.incsmps.ro/en/homepage/; QURES, https://www.qures.ro/
The evaluation examines the contribution of the OP to strengthening the capacity of beneficiaries, ESI Fund transparency, and GDP and employment, up to end-2021.
To assess the achievement of targets, intended and unintended effects, their sustainability, and good practice up to end-2021.
Monitoring and statistical data.Interviews.Survey of project managers, beneficiaries, participants in training.Case studies.2 Focus groups.Input-output econometric model.
Progress of the OP accelerated in 2021, partly because of the increased capacity of beneficiaries, in terms of quality of projects proposed and project management skills.Transparency of ESI Fund management has increased, in terms of involvement of social partners, though the communication strategy is not sufficiently clear.Unintended effects are improved collaboration between beneficiaries, exchange of ideas, increased participation of social partners.Training for beneficiaries and methods for selecting social partners are likely to have sustainable effects, while this is less the case for support to cover labour costs.Institutional instability, complex procurement procedures and administrative bottlenecks have adversely affected the results achieved, while a good understanding of the needs of social partners has been a positive factor.The OP’s direct, indirect and induced effects are estimated at 0.03% of GDP, the main effect being more in terms of strengthening the capacity of beneficiaries.The direct contribution to employment is estimated at 900 jobs created and the overall contribution at 2,119 FTE jobs.
Limited progress of the OP and the small number of responses to some survey questions created difficulties, but data have been triangulated to overcome this problems. The sustainability analysis is still preliminary, given the progress of the OP.
Analyse the needs and potential beneficiaries of the OP in the private/nongovernment sector.Support capacity building in project and resource management in beneficiary organisations.Continue information activities and dissemination of project results.
Evaluation of the communication activities to support potential beneficiaries within INTERREG V A Romania-Hungary Programme
Realizarea evaluării activității de comunicare în sprijinul potențialilor beneficiari din cadrul Programului Interreg V-A România-Ungaria
Andrea Floria, Cristina Cojoaca, Irina Ciocirlan
SC ACZ Consulting SRL - str. Ștefan Velovan, bl. 23A, ap. 12. Craiova, jud. Dolj, Tel: 0351 44 20 44 - T33 SRL via XXV Aprile, nr. 28/b, Ancona, Italia, Tel: +39 0718 98093 - Iris SRL via G. Verdi, nr. 40, Prato, Italia, Tel: +39.0574.607522
Evaluation of REACT-EU measures financed under the Bremen ERDF OP, 2014-2020
Carla Harnischfeger, Luise Schrader, Dr. Lukas Vonnahme
Ramboll, carla.harnischfeger@ramboll.com
The evaluation analyses the implementation and results of measures under the TO "Supporting crisis management in the context of the Covid-19 pandemic and its social consequences and preparing a recovery for a green, digital and stable economy” of the OP. The data collected go up to November 2023.
The aim of the evaluation is to analyse the effectiveness, efficiency and impact of measures funded by REACT-EU. These concerned: LuRaFo – Aerospace Research Programme; AUF – Financing of applied environmental research projects; PFAU – Programme for the Promotion of Applied Environmental Techniques; expansion and equipment of research and development infrastructure and transfer centres; digitalisation of public spaces.
Analysis of the intervention logic.Analysis of monitoring data.Interview with the MA on the analysed measures and on the possible influence of external factors.
It is likely that the expected results of the Lu.Ra.Fo measure will be achieved, although there is a risk that not all projects will be completed on time and that fewer private investments than expected will be implemented.It can be assumed that the PFAU measure will have a positive impact on innovation activities and contribute to strengthening innovation and competitiveness of companies in the field of environmental technologies.AUF measures show low financial progress but a positive contribution to the expected impact is considered as likely.It is plausible that measures on the expansion and equipping of R&D infrastructure will contribute to the expected results in the long run.
Since only 3 of the 59 projects approved have been completed at the time of the evaluation, the impact assessment is not definitive.
Evaluation of the indicator system under the Research and Innovation OP, 2014-2020, in Italy
PONR&I Validità del sistema di indicatori del PON e della quantificazione dei relativi target
Nicola Drago, Giacomo Grazini, Marco Marchese
The report aims to assess the indicators system of the OP. Data used go up to December 2022.
To assess the coherence, representativeness and the level of achievement of the programme’s indicators and the validity of targets, to improve design and implementation in the 2021–2027 programming period.
Document analysisAnalysis of monitoring data and statistical data (from ISTAT and MUR Statistical Office).SMART analysis of indicators (Specific, Measurable, Achievable, Relevant, and Timely).Effectiveness analysis of indicators using MEANS (Means for Evaluating Actions of a Structural Nature) methodology.Focus groups with the Technical Assistance staff, the Monitoring Office, and working groups with all the teams responsible for the programme evaluations.
The indicator system of the programme is overall coherent, balanced and relevant to programme objectives.23 out of 38 indicators had already reached or were on track to reach their targets.COVID-19 reprogramming indicators showed misalignment due to the exceptional context and needed further reassessment.Some methodological issues emerged (e.g. over/underachievement, narrow scope of certain indicators).The monitoring system is reliable but too complex and slow in reporting updated information.
Shift the focus of indicators from outputs to outcomes, to capture the social and economic impact of the measures.Refine some of the indicators and adopt shared standards across regional, national, and EU levels.Plan evaluations from the start, including impact assessments.Provide training to beneficiaries on indicator reporting and develop a unified IT system to integrate data.
Annual evaluation of the implementation of the Lombardia ERDF OP, 2014-2020 - year 2021
Rapporto di valutazione 2022 - per l'annualità 2021
Fondazione Giacomo Brodolini - MBS - IZI
https://www.fondazionebrodolini.it/ - info@fondazionebrodolini.eu; https://mbs.it/ - info@mbs.it; https://www.izi.it/ - mail@izi.it
The evaluation examines the OP’s implementation in 2021.
To assess progress towards the achievement of targets set up to end-2021.
Desk analysis.Analysis of monitoring and administrative data.Interviews of those responsible for the management of the different PAs.Interviews with beneficiaries in particular of PAs 5 and 6.CAWI surveys of beneficiaries of some measures under PAs 3 and 6.Collection of materials for case studies under PAs 1 and 4.
By end-2021, 90% of resources were committed and 71% spent. Calls focused on specific areas and the ability to reply quickly to the COVID emergency, e.g. by simplifying procedures and extending project implementation.Two new projects under PA1 (TO1) were launched in 2021 for enterprises: ‘Bando brevettti 2021’ to acquire services for technological, strategic, commercial innovation and ‘Bando Tech Fast Lombardia’ to support projects for technological and digital innovation. Other measures supported are: ‘Smart Fashion and Design’ in manufacturing and the creative and culture sector; ‘Fashiontech’; ‘Bando Covid’ that supported health sector projects, involving 36 public and 14 private research centres.On PA2 to extend ultra-wideband across the region, no major progress was made in 2021.On PA3 (SME support), two new measures were launched (‘Bando Ripresa’ and ‘Internatinalisation of SMEs’) to support sectors most badly hit by the Covid pandemic and administrative procedures where improved and simplified.On PA4 (TO4), measures included the construction of new cycle paths improving the energy efficiency of public building using innovative means, though difficulties arose because of a lack of experience of public authorities in managing this kind of project.On PA5 (urban development), funding not used was allocated to other PAs in 2020. Implementation issues show a need to strengthen administrative capacity.On PA6 (support for four strategies in inner-city areas), implementation is slow mainly because of the complex governance system put in place.
Continue the support of networking among partners in 2021-2027.Use a mix of grants and financial instruments in the next period.Reinforce support for patenting.Focus on strategic issues such as healthcare, sustainability, public-private networks of research, the innovative capacity of the regional economy and development of innovative ecosystems.Improve the model for inner city areas.
Evaluation Report of the OP Canarias ERDF 2014-2020
Informe de Evaluación del Programa Operativo FEDER de Canarias 2014-2020. Anualidad 2017
Evaluation of the State Aid Scheme for intermodal transport projects under the Infrastructure and Environment OP, 2014-2020, in Poland
Ocena Programu pomocy publicznej na realizację projektów w zakresie transportu intermodalnego
dr M. Wolański, P. Kozłowska, Ł. Widła-Domaradzki, J. Kaczorowski, J. Widłak
Wolański, www.michalwolanski.pl; EGO – Evaluation for Government Organizations, www.evaluation.pl
The evaluation examines the implementation of the State Aid Programme for intermodal transport.
Analysis of the potential positive and negative effects of the programme and the presumed impact on intermodal transport and beneficiaries.
Analysis of existing data (graphical and spatial analysis of statistical data, document review).Interviews with project designers, experts assessing projects and successful and unsuccessful applicants (46).Evaluation workshops.9 case studies.CIE using PSM.Panel of experts.
The support was very attractive, especially for rolling stock projects, but there is a need to implement complex projects over the entire logistic chain.The long duration of calls and their limited number have been problems.Accumulation of investments made it difficult to implement investment projects at certain stages.Effects of the intervention have been limited by the pandemic.There has been a lack of complementary measures from other institutions, institutional weaknesses of other bodies and a lack of recognition of the importance of intermodal transport in strategic documents.
Limited willingness to cooperate by non-beneficiary bodies that failed to obtain financial support.
Continue the support for intermodal transport in its present form for infrastructure projects and those involving purchase of terminal equipment.Increase frequency of calls and training for beneficiaries.Adjust selection criteria to the objectives and specific features of the scheme.Give preference for projects with an infrastructure component and those in attractive locations.Create a strategic document and an institutional unit devoted to intermodal transport.
Final evaluation of the implementation, effectiveness, efficiency and impact of ESF and YEI support (including the Youth Guarantee)
Noslēguma izvērtējums „Eiropas Sociālā fonda atbalsta un īpašā piešķīruma Jauniešu nodarbinātības iniciatīvas, tostarp garantijas jauniešiem shēmas īstenošanai, investīciju lietderība, efektivitāte un ietekme”
The evaluation assesses the impact on youth employment and wages 18 months after exit of the three strands of YG in Latvia:Active LMP measures implemented by the PES (SEA) to facilitate transition to employment (YEI Specific objective 7.2.1.1).Initial vocational training programmes implemented by the State Education and Development Agency (VIAA) (YEI SO 7.2.1.2).“Know and Do!” project to identify and support inactive young NEETs (ESF SO 8.3.3).
To assess the effectiveness, efficiency and impact of these programmes on the labour market situation of young people during the period 2014-2018.
Analysis of documentation.Case studies.In-depth expert interviews.Focus group discussion.E-survey of participants.Analysis of statistical data.Econometric analysis.Counterfactual evaluation (propensity score matching).Data from the monitoring system, PES, Ministry of Education and Science and the State Revenue Service databases and the Central Statistical Bureau.
Individual work with young people at high risk of unemployment is a prerequisite for a successful outcome.Between 2014 and 2017, over 100,000 young people were supported by YG in their job search. Long-term support was provided to over 28,000 YEI participants, including over 19,000 registered unemployed (including almost 4,000 long-term unemployed), and 800 inactive and 2,400 employed. The project “Know and Do!” involved 1,550 inactive NEETs.Because men seldom register as unemployed, they are considered missing NEETs (65% of participants in “Know and do!” are women).Participants reported positive feedback on the measures supported (technical skills, level of motivation, social skills), though less so among participants with basic or general secondary education.Immediately after exit, 48% for participants in First job experience were in employment as were 38% of those in subsidised jobs, 25% of those in the ‘Development of skills required for work in the non-governmental sector’ programme, 13% of those in continuing vocational education and in Support for self-employment measures and 10% of those in Non-formal education and Training of drivers.Around 10% of participants were employed in a full-time job.The proportion of participants in employment after 6, 12 and 18 months was generally larger than in the control group for most of the measures (except Non-formal education after 18 months and Support for self-employment after 12 and 18 months).None of the activities supported is estimated to have a statistically positive impact on wage, i.e. relative to the control group, except for disabled people participating to initial vocational education programmes.Cost effectiveness measured by average wages was lower than for control group for all measures except First job experience (after 6 and 12 months).‘Developing skills for work in the non-governmental sector’ is estimated to be the most cost effective measure in terms of participants in employment and number of days worked.
The following external circumstances might affect the results:The recovery of the economy after 2012.Immigration abroad after participation (issue of missing data).Demographic trends (number of young people is declining).Data not available for all participants due to the timing of the evaluation.The very low response rate to e-survey of participants (only 478 out of 30,000 – i.e. less than 2%).
There is a need to:perform an audit of data and information systems;provide more individualised support for low educated young people in “Know and Do!”;further promote the participation of young disabled in initial vocational education and job subsidy programmes;devote additional effort to identifying the missing NEETs;review the range of support activities offered to young people with basic or low levels of education;strengthen cooperation between career advisers and employers.
Evaluation of the interventions to facilitate transition from education into work under the Mecklenburg-Vorpommern ESF OP, 2014-2020
Bewertung der Förderung zum spezifischen Ziel C.2.1 "Verbesserung der Übergänge von der Schule in die Berufsausbildung"
ISG Institut für Sozialforschung und Gesellschaftspolitik GmbH, Otternbuchstra. 35, 13599 Berlin, Roth@isg-institut.de, Tel.: +49 30 - 650 43 90
The evaluation examines support for the promotion of extra-curricular vocational guidance financed by IP 10.iv of the Mecklenburg-Vorpommern ESF OP, 2014-2020.
To assess the implementation of extra-curricular vocational guidance measures and how and under what circumstances they help to smooth the transition from school into work.
Analysis of monitoring data.Analysis of administrative data of the Federal Employment Agency.Case studies (documentation analysis and interviews with stakeholders).Focus group with participants (young school students).
The implementation of extra-curricular activities is successful when local actors are highly involved and help to identify suitable companies for work placement (module B), company visits (module C) and vocational guidance camp (Module D).Implementation requires strong links between the curricular activities of the school and the extra-curricular courses to be successful.The effects of extra-curricular vocational guidance on participants’ perception of their improvement differ between the modules, in line with the intervention logic: module A: career choice courses work better when supplemented by curricular activities in schoolmodule C: company visits help students discover new occupations and/or to know better whether occupations fit their interestsmodule D: external experts helped students develop relevant soft skillsmodule E: vocational guidance camps helped students to be able to work in teams.
It is difficult to isolate the effect of ESF measures from those of similar measures developed by schools through their curricular activities or by the Federal Government, employment agencies and trade associations.
Increase awareness of high school teachers of career guidance.Encourage integration between extracurricular and curricular activities.Foster a pluralist approach to vocational training by involving local actors to identify sector-specific training needs.Increase integration between module B (work placement) and module E (Vocational Guidance Camp).
Mid-term evaluation of the West Romania ERDF Programme, 2021 – 2027
Programul Regional Vest 2021 – 2027 Raport de analiză intermediară (mid term review – MTR PR Vest)
The mid-term evaluation examines the progress of the implementation of the programme. It covers the implementation period between 2022 and 2024.
To assess the implementation of the resources allocated by the programme and its contribution to the achievement of national objectives in the field of energy, climate and social rights. To provide recommendations for the future of the programme.
Analysis of monitoring data (Integrated Management System).Document analysis (Country report, Country-Specific Recommendations).
61 calls for projects were launched, covering 73% of the non-reimbursable allocation of the programme.208 contracts were signed covering 44% of the non-reimbursable financial resources of the programme.Various actions have been implemented to strengthen the administrative capacity of the stakeholders involved in the implementation of the projects.Specific actions have been launched to implement the New European Bauhaus initiative.Some progress has been achieved to map the provision of higher education in relation to labour market needs.
Launch calls for the remaining resources allocated, including those supporting the STEP Platform and social housing.Strengthen the administrative capacity of local authorities in addressing key investment needs and complex projects.Develop digitalisation and simplification processes.
Measurement of the long-term OP indicator “Number of people in a better situation on the labour market six months after leaving the programme” under the Knowledge Education Growth OP, 2014-2020, in Poland
Drugi raport z II modułu badania: Wyliczenie wartości wskaźnika rezultatu długoterminowego” – „Liczba osób znajdujących się w lepszej sytuacji na rynku pracy sześć miesięcy po opuszczeniu programu”
Stanisław Bienias, Krzysztof Kasparek, Maciej Koniewski, Piotr Stronkowski
The study examines the long-term result indicator under all the IPs of the OP. The indicator refers only to people employed before participation and who completed measures before the end of 2020.
To calculate the result indicator at IP and OP level for those completing participation in measures before the end of 2020.
Analysis of monitoring data.Survey of participants through CAWI (1,100 interviews) and CATI (3,208 interviews). Response rate of 4%. Sampling was applied to IP 10iii and 11.i (margin of error of under 5%).6 interviews with OP managers and project promoters.
13% of participants employed improved their situation 6 months after by moving into a position requiring higher skills or responsibility (8%) and shifting from a fixed-term to an open-ended employment contract (3%).Improvements are more frequent for people under 25 and the lower educated, and for measures targeted at healthcare workers (IP 10ii), the social economy (IP 9v), and university students (IP 10ii).According to participants, participation in the projects contributed to these improvements to a modest extent.The indicator seems to be related more to the natural development of the labour market than to the support measures.The indicator is lower than planned (23%), probably because of a mismatch between the target and the types of measure, which are aimed at those with a relatively good employment situation, rather than because of the poor design and implementation of the measures.
Limit the use of the indicator only to measures aimed at improving the labour market situation of those in employment.
Estimation of selected performance and result indicators of priority 7 of the Regional OP for Pomorskie 2007-2013 - Healthcare and rescue system
Oszacowanie wybranych wskaźników rezultatu i oddziaływania Osi Priorytetowej 7. Ochrona zdrowia i system ratownictwa Regionalnego Programu Operacyjnego dla Województwa Pomorskiego 2007-2013
Autorzy raportu: Marta Cichowicz-Major, Agnieszka Rudolf, Cezary Przybył
EVALU Sp. z o.o. (dawniej Agrotec Polska Sp. z o.o.)ul. Dzika 19/23 lokal nr 55 )00-172 Warszawa
The evaluation relates to interventions financed under priority 7 – Healthcare and rescue system – of the ESF Regional Operational Programme for the Pomorskie region 2007-2013.
The evaluation is aimed at estimating two indicators:the result indicator for priority 7 of the Regional Operational Programme for Pomorskie 2007-2013 (OP 7): Area of the Pomorski voivodeship covered by the life-path (% of the voivodeship);the performance indicator fatality rate of victims in emergencies covered by the life-path (%).
The evaluation adopts a mix of methods:Desk analysis of relevant documents and data;CAWI and CATI interviews with recipients of support.
The main findings are:the result indicator Area of the Pomorskie covered by the life-path (%) increased from 13% to 33% in 2015. The index value increased steadily over the period as a result of the availability of new ambulances so reducing the “maximum time to reach people in emergencies”;the impact indicator “The fatality rate of victims in emergencies” covered by the life-path was reduced from 3.3% in 2011 to 2.7% in 2015.
The evaluation highlights that data received from the 9 Medical Rescue Teams were unusable because of their non-comparability.
There is one main policy recommendation: to continue support to hospital emergency wards in the region.
Evaluation of vocational education and training in the Lombardia ESF OP 2014-2020
Approfondimento tematico: la filiera professionalizzante Lombardia
RTI Istituto per la Ricerca Sociale, COGEA – Consulenti per la Gestione Aziendale
The evaluation assesses the 11 most important VET areas under the ESF Lombardia OP 2014-2020; agri-food; furniture and design; chemistry and biotechnology; construction; energy; ICT, audio-visual and multimedia; mechanical engineering; fashion; business services; transport and logistics; and tourism.
To assess whether or not the regional training system is able to provide programmes which meet the needs of the market and whether or not it is able to identify and support the research and innovation skills required by the industry.
Desk analysis of the training provided in 2017-18 by the regional VET courses, the Higher Technical Education and training paths and the Higher technical Institute courses.In-depth interviews with training institutions and other bodies involved in the regional vocational sectors.
Overall, the regional system provides wide and coherent training, with 436 VET courses, 128 Higher Technical Education and training courses and 221 higher technical Institute courses for 3,000 students in the second and 5,000 in the third.Between 2015 and 2017, the region allocated EUR 614 million (EUR 126 million from Axis III of the ESF OP) to VET courses and EUR 34.5 million (EUR 27 million from the ESF OP) to Higher Technical Education and training and Higher technical Institute courses.For 9 of the 11 sectors analysed, the training system provides courses up to the level of non-academic tertiary education.For some of the sectors, training is either incomplete (for “chemistry and biotechnology” there are no IV year courses following the three-year qualification) or is not provided because of lack of interest (construction).
Although the planning of VET courses takes account of the overall supply chain, information and guidance is limited and should be increased.The provision of VET courses should be flexible enough to respond to the changing needs of the market and ensure job opportunities for participants.
Evaluation report of the La Rioja ERDF OP 2014-2020
Informe de evaluación para la anualidad 2016 del programa Operativo FEDER de La Rioja 2014-2020
José María de Escuza Kalea, 1-3, 48013 Bilbo, Bizkaia, Tel: 944 05 00 43
The report provides an evaluation of the implementation of the 2014-2020 La Rioja ERDF OP to be presented to the Monitoring Committee in 2017 and covers the period from 2014 until 2016.The evaluation focuses on the review of the intervention logic and on the implementation aspects of the OP through analyses of the measures adopted for the management, monitoring, evaluation and control of the programme, and through the assessment of the efficiency and effectiveness of the OP.
The objectives of the evaluation are to assess the coherence of the intervention logic, the appropriateness of the management structures and the effectiveness and efficiency of the OP implementation.
Analysis of programming documents (OP, ex-ante evaluation, communication strategy of the OP etc.), of secondary data (OP’s monitoring data and national and regional official statistics) and interviews with the key actors of the OP.
The logic of the intervention of the OP continues to be valid because the socio-economic context has not changed much.The main difficulties identified resulted from the delayed approval of rules on eligible expenditure for ERDF OPs and from changes in procedures by the Management Authority.The financial implementation of the OP is low and there is a risk of non-compliance with the milestones set out in the performance framework, especially with regard to TOs 1 and 2.Many of the result indicators suggest satisfactory progress but it can be assumed that the main effect did not come from the OP since it was only at an early stage of implementation.Horizontal principles are addressed by the OP but stronger effort should be made in respect to equal opportunities.
Only a small number of activities have been completed; this has affected the effectiveness indicators. Furthermore, certified expenditures were not yet available.
The evaluation recommends to:Develop a detailed schedule of activities for agencies’ managers/beneficiaries;strengthen the financial implementation of T01 and TO2 and support R&D projects in line with the RIS3;enhance the principle of equal opportunities throughout the OP.
Final evaluation of the implementation of the Innovation and Sustainable Growth in Businesses OP, 2014-2020, in Denmark
Slutevaluering af den danske Regionalfond 2014-2020
OP Innovation and Sustainable Growth ERDF 2014-2020
The evaluation examines the implementation and achievements of the OP. Data are updated at August 2023.
To improve the quality of design and implementation of future measures and to assess the effectiveness and efficiency of the OP.
- Interviews with project operators and case handlers.- Analysis of relevant documentation.- Analysis of monitoring data.- Analysis of administrative data from the Danish Business Authority.- Progress evaluation report of projects financed by the OP (163 report) analysed with AI.
- Most of the specific objectives and targets have been met for all priorities.- The programme has directly contributed to meet several EU2020 targets, such as R&D expenditure, employment rate, reduction of CO2 emissions.- There is a wide difference in the cost-effectiveness of measures. This is low for the "cooperation for innovation" priority but very high in relation to energy and resource efficiency.- Measures for "cooperation for innovation," are coherent with other national measures (Innovation Fund and Development and Demonstration Programmes); however, measures for "growing enterprises" and '"resource efficiency," show overlaps with local, regional and national measures.
- Strengthen synergies between initiatives on collaborations on innovation and focus on scale-ups and growth-oriented skills development.- Involve knowledge institutions and business houses as project partners in the future.- Include more nuanced indicators of business growth to ensure a more real reflection of the different ways in which the effects of projects can materialise.- Ensure greater transparency and operationalisation of green business plans.
Employment and Social Inclusion OP, 2014-2020 – analysis of long-term outcome indicators, results from the 2017 survey
Note d’analyse des indicateurs de résultat à long terme (6 mois) du PON FSE- Enquête 2017
This is the 2017 annual report on the ESF OP in France 2014-2020.
To assess the effect of the OP and its progress toward achieving its targets.
The evaluation is based on a survey of a representative sample of beneficiaries (90,000 out of 952,567 with a response rate of 27%).
6 months after completing programmes, 46% of the participants who were unemployed or inactive were in employment and 30% of those in employment have improved their situation on the job market.Participants in IP 8.1 activities (access to employment for jobseekers and the inactive, particularly the long-term unemployed) have the highest employment rate (58%), while those in IP 9.1 activities (active inclusion, including with a view to promoting equal opportunities, active participation and better employability) have the lowest rate (38%).The change in the employment rate between immediately finishing activities, 6 months afterwards and at the time of the survey shows the biggest increase among participants in IP 8.5 activities (adaptation to change of workers, companies and entrepreneurs) and the smallest increase among those in IP 8.3 activities (self-employment, entrepreneurs and start-ups).Of the 26% of unemployed or inactive participants employed at exit, 69% were still employed 6 months later and 56% at the time of the survey; among the 51% of the unemployed or inactive participants, who were looking for work at exit, 42% were in employment 6 months later and 33% at the time of the survey.
The figures presented in the report are restricted to participants who left programmes between 1 January 2015 and 31 March 2017.
Evaluation of measures for victims of trafficking supported by the Piemonte ESF OP, 2014-2020
Renato Cogno, Luigi Nava, Gianfranco Pomatto, Martina Sabbadini, Antonio Soggia
The evaluation examines the measures adopted under the OP to support the social and employment inclusion of victims of human trafficking.
To assess the level of implementation of the measures taken to support victims of trafficking.
Analysis of monitoring data.15 interviews with bodies implementing the projects.Workshop with stakeholders involved.2 focus groups with those working on the Anello forte and POR Tratta projects.
Between December 2017 and November 2019, 1,035 people (87% of them women, 83% from Nigeria) had contact with the Anello Forte network established by the Region in collaboration with public and private bodies to identify victims of trafficking, provide shelter and begin processes to integrate them into education and employment. 300 people received individualised support of different kinds. b) 97 people participated in active inclusion measures (language teaching, career guidance, psychological, social and cultural support). Of these, 27 participated in traineeships and 8 obtained a job. c) The main weaknesses were the severe difficulties of the target group, their low employability and problems of accessing active labour market measures, issues with burdensome procedures, and managing relationships with enterprises.
Strengthen the teaching of Italian.Use cultural mediators if participants do not speak Italian.Develop innovative means of providing childcare for women receiving support.Focus vocational education on the more promising sectors.
Evaluation of the relevance and effectiveness of the preventive programmes under the Knowledge Education Growth OP, 2014-2020, in Poland
Ocena trafności i skuteczności programów profilaktycznych realizowanych w ramach Programu Wiedza Edukacja Rozwój
The evaluation examines the quality of health policy programmes supported by the ESF, methods used to implement them and their effects.
To assess the relevance of support for health needs, the effectiveness of prevention programmes and the efficiency of their implementation, with a focus on three selected programmes.
Desk research.Surveys of beneficiaries (68), patients (1050) and medical bodies (550).Interviews with project coordinators (24) and central government (6).4 group interviews.Design thinking workshop.
Programmes are tailored to needs and have increased the availability of preventive services.There might be a risk of the budget being unused because of the small number of patients as a result of insufficient promotion and the COVID pandemic.Some patients are reluctant to participate in programmes because they are afraid of the diagnosis, discouraged by procedures and do not have the time (especially if they are working).Another problem is the low motivation and commitment of primary healthcare workers, and attracting patients strongly relies on them.It is too early to assess effects but there seems to be greater detection of diseases and improved competence of medical personnel.The allocation of funds and their division into direct and indirect costs have caused problems.
Carry out further preventive activities but with intensified promotion.Engage companies and employers' organisations in promoting a healthy lifestyle.Introduce modern means of designing preventive measures (such as an app reminding people about medical examinations).Extend the functionality of the Individual Patient Account (IKP) by adding an education and training module.
Evaluation of technical assistance measures financed under the Interreg V-A-France-Germany-Switzerland OP, 2014-2020
Evaluation de l'assistance technique du programme INTERREG V Rhin supérieur
Institut de la gouvernance territoriale (IGT) ; SynappCity
https://igt-itg.eu ; https://synappcity.com
The report focuses on the Technical Assistance Axis of the OP. Technical assistance has been implemented through a nine-year support project. Data used go up to 14 June 2018.
To analyse the results of the technical assistance project in terms of management and implementation of the OP.
Document analysis.Monitoring data processing.11 interviews with stakeholders.Workshops.Case studies.
The physical and financial progress of the measures is in line with the expected targets. The measures respond to the needs of beneficiaries, but communication and dissemination activities might be strengthened.Compared to the previous programming period, new tools and services have been introduced which have significantly improved the support to beneficiaries during project preparation and implementation.The organisation of human resources by the Managing Authority and the Joint Secretariat is effective and fully responds to the objectives of good management and accompaniment of beneficiaries.Initial issues, such as the definition of national co-financing and delays in staff hiring were overcome at the time of the evaluation.
The limited resources allocated to this evaluation did not allow an in-depth analysis of some aspects requiring the involvement of final beneficiaries of the OP.
Strengthen communication and dissemination activities of achievements and opportunities offered by the OP, also through the involvement of local stakeholders.Strengthen the administrative capacity of Managing Authorities/Joint Secretariat and Programme partners, in particular on the French side.
Analysis of Vocational training courses funded by the Toscana ESF OP, 2014-2020
La formazione professionale nella programmazione 2014-2020
The evaluation examines training courses funded by the OP to help people to enter the labour market.
To analyse the implementation of different types of training activities (courses and vouchers).
Descriptive analysis of monitoring data.
At the beginning of 2019, over 8,000 people had participated and over half of them were still involved in courses.Most of participants are under 30 with a secondary school qualification.On average, the cost per participant amounts to around EUR 5,000 (except for vocational courses at around EUR 10,000 per person).After 6 months of exiting courses, 39% of participants in ITS (Istruzione tecnica superiore ) courses were in employment and between 13% and 20% for those in other types of courses. Overall employment rates are higher for training courses than vouchers.
Impact evaluation of the Estonia-Latvia Interreg OP, 2014-2020
Programme impact evaluation of Interreg V-A-Estonia-Latvia Programme 2014-2020
Merit Tatar, Mart Veliste, Richard Henahan, Andra Somelar
Institute of Baltic Studies, www.ibs.ee/en/
The evaluation examines the Estonia-Latvia OP and whether it has created positive changes in and outside the programme area.
To assess the effectiveness, efficiency, impact, relevance, added value and sustainability of projects supported, why some were successful and others not, and the internal and external constraints in achieving impact.
Desk research of programme documentation, implementation and final reports, previous surveys and other documents.Analysis of statistical and administrative data.Interviews with OP authorities (13) and project partners (35).Online survey of project partners supported (60% response rate).
Up to 2019, most output indicator targets had been met or exceeded.Projects under SO 1.1 (TO3) contributed to cooperation and knowledge sharing between project partners and facilitated business connections.Effects of support under SO 1.2 (TO3) are observed in jointly-developed products and services and new jobs.Under SO 2.1 (TO6), the target for improved natural and cultural heritage sites which are more attractive for visitors was exceeded.SO 2.1 projects have also led to improved marketing and communication skills and the development and improvement of cross-border networks.Support under SO 2.2 (TO6) has increased awareness of energy saving and of waste sorting and re-use.The large-scale investment under SO 3.1 (TO7) led to the network of ports becoming functional, international quality standards being met, and the safety of sailors increasing.According to survey respondents, SO 4.1 (TO8) projects improved conditions for labour mobility and cooperation over traineeships.According to respondents, almost 90% of completed projects have achieved their objectives and 98% of ongoing ones have either done so or are on track to doing so.Without programme funding, respondents report that projects could not have been undertaken or would have been on a smaller scale.The administrative burden and financial reporting requirements are considered the main problems.The main success factors are effective partnerships, experienced leadership and good knowledge of the target group.
The relevance and impact of the programme in a wider strategic context were not assessed and nor were the programme result indicators, since another evaluation is expected to do this, together with assessing the impact of projects. In addition, at the time of the evaluation, some projects were still ongoing, and only 19 of 33 finished projects could be assessed. Survey and interview responses may be biased to some extent.
Strengthen the links between indicators and the impact of projects.Clarify the methodology for defining and measuring result indicators.Consider providing larger funding to more focused projects.Simplify reporting procedures and reduce bureaucracy.
Evaluation of measures for the footwear industrial district under the Puglia OPs, 2007-2013 and 2014-2020
Rapporto di valutazione locale: La ripresa del distretto calzaturiero di Casarano in Sud Salento
OP Puglia ESF 2007-2013; ROP Puglia ERDF 2007-2013; ROP Puglia ERDF ESF 2014-2020
Gianfranco Viesti, Veronica Notaro, Benedetta Parenti
Università di Bari, Centro Rossi Doria Roma Tre, Università di Napoli l’Orientale; gianfranco.viesti@uniba.it, benedetta.parenti@unior.it
The evaluation aims to assess the measures for the textile-clothing-footwear (TAC) industrial district, under the regional OPs 2007-13 and 2014-20. Data used go up to July 2022.
The research aims to understand how the measures for footwear industrial district have supported its productive recovery and competitiveness.
Analysis of official statistical data. Analysis of Monitoring data. Analysis of Administrative data. Surveys of 322 respondents from a single company of the district. 28 semi-structured interviews (including 9 in-person) with local entrepreneurs and key informants. 19 qualitative interviews with workers from three local companies. Panel of experts. Analysis of secondary data.
ERDF funds have supported investments in advanced technologies and infrastructure, improving productivity and enabling the district to specialise in a high-value segment, in line with the Smart Specialisation Strategy. ESF measures have supported projects aimed at developing workforce skills and territorial tools such as PIAs, tailored to local needs, and have facilitated the adaptation of the workforce. The measures implemented have improved labour standards, with incentives towards regularisation and safety that helped reduce informal economy. ERDF and ESF have strengthened the network of suppliers, aligning with the S3, which promotes competitive clusters.
A restricted sample size, with participants largely from one company of the district, limits the generalisability of findings across the district. Additionally, detecting informal labour, particularly home-based work often carried out by women, proved to be challenging.
icy implications (which seem to be reliable) Encourage ongoing training to develop advanced skills aligned with technological advancements and luxury market demands. Enhance labour regularisation by increasing oversight to reduce informal work, especially in home-based roles, and promote safety standards. Promote innovation and digitalisation by boosting incentives for firms investing in advanced technologies and digital processes to enhance global competitiveness. Foster collaboration among district companies to create an integrated ecosystem, supporting shared skills and joint projects within the district.
Evaluation of support for long-term unemployed and family needs communities in the Brandenburg OP, 2014-2020
Endbericht: Evaluierung der Förderung der Integrationsbegleitung für Langzeitarbeitslose und Familienbedarfsgemeinschaften
Institut SÖSTRA - Institute for Socio-Economic Structural Analysis
SÖSTRA, Torstraße 178, 10115 Berlin, www.soestra.de
The evaluation assesses the implementation of the Brandenburg ESF OP 2014-2020, focusing on support for integrating the long-term unemployed into employment, along with low-skilled workers, single parents, those with disabilities, migrants and the elderly.
To assess progress towards achieving the targets and the way funding is managed.
Desk research of the OP and project documentation;literature review of comparable funding programmes for long-term unemployed;analysis of monitoring data;panel of experts;5 case studies of selected projects in support of integration;two semi-standardised surveys, one of implementing bodies (62 responses, 90% response rate) and the other of final beneficiaries (744 responses, 21% response rate);workshop with the programme manager and representatives of the 5 case study projects.
By 30 September 2017, 3,600 people had participated in the training (64% of them women) and 2,414 had already exited from the programme.Of the latter, 32% were in employment, 30% in subsidised employment and 12% in education.Of those in employment, 72% were in jobs with basic requirements and 28% in jobs requiring a professional qualification. 52% of those in employment worked part-time (79% of them women).6 months after the end of the programme: 34% were in employment, 11% in further education or vocational training, 5% inactive and 5% unemployed.
Data on the employment situation 6 months after the end of the programme only cover 52% of those exiting the programme by end-September 2017.
With regard to the education indicator, consideration should be given to including school completion as a successful outcome. Other positive referral outcomes, such as placement in the Federal Volunteer Service or volunteering generally, could also be included.With regard to the horizontal objectives, in line with the target set out, 50% of women also be retained in jobs.A potential future counterfactual impact assessment should be based on the random or quasi-random provision of support (e.g. possibly by varying the time at which support is offered to different groups of participants).
Evaluation of the impact of the Warmińsko-Mazurskie ERDF OP, 2014-2020 on increasing transport accessibility and territorial cohesion
Ocena wpływu RPO WiM 2014-2020 na zwiększenie dostępności i spójności transportowej województwa warmińsko-mazurskiego
EU-CONSULT, Infra Centrum Doradztwa, Polska Agencja Ewaluacji Sektora Publicznego
The study focused on the impact of the transport interventions implemented under the OP.
The main objective of the study was to assess the current and potential impact of transport infrastructure support on improving the accessibility and transport coherence of the region.
Documental analysis.Analysis of statistical and monitoring data.Survey of beneficiaries, of implementing institutions and key stakeholders.Survey of local residents (n=135).Interviews with implementing institutions and key stakeholders.Case studies.Cost benefit analysis.Benchmarking.Expert panel.Counterfactual analysis.
Implementation of the OP has had a greater impact on improving internal accessibility than external accessibility of the region.The investments strengthened the coherence of the transport network, though they were largely focussed on increasing the capacity of existing roads.The measures had additional positive effects, such as on the environment (reduction of CO2 emissions) or on socio-economic aspects (e.g. change in the transportation behaviour of residents).Road projects have significantly increased the level of safety of region's roads.
Focus more the measures on the promotion of micro mobility, eco-mobility, e-mobility and public transport.Complement investment projects with activities aimed at educating the public about ecological forms of transportation.Complement transportation interventions with Park&Ride infrastructure.
Evaluation of the ‘SAFI – Salzburg Women’s Initiative’ project under the Employment Austria ESF OP, 2014-2020
Evaluierung der ESF-Umsetzung für Salzburg. Projektbericht "SAFI - Salzburger Fraueninitiative"
Institut für Berufs- und Erwachsenenbildungsforschung (IBE), www.ibe.co.at
The evaluation examines the initiative, targeted at women with low-income living in the province of Salzburg, to provide support, help their personal development and prepare them for the labour market.
To assess the implementation, effectiveness and effects of the initiative to support women with low income, the success factors, the potential for improvement and the contribution to cross-cutting issues.
Documentation analysis.Analysis of data in the ZWIMOS database and the SAFI participants database.Interview with SAFI project managers.Focus group with 4 people from the initiative.Data collected by the International Research Centre for Social and Ethical Issues (interview with project managers, focus group with employees, research report).
70 participants had completed the project up to the beginning of 2020 and 7 had dropped out.77% of SAFI participants were employed or in training at the time of leaving the project, exceeding the target.The project helps to stabilise the living conditions of participants, strengthen their self-esteem and competences, and provide support relevant for the labour market, including career advice and coaching on applications.Success factors include voluntary participation, low requirements for accessing the initiative, follow-up of women when they stop reporting, and individualised support.
The number of participants is small. The impact of SAFI could not be assessed because of data protection requirements and so restricted access to relevant data. The results for the labour market status were based on the data registered by project managers; the sustainability of project results could not be assessed.
Regionalise the initiative, extending it to other locations.Provide joint training courses to employees and adopt a flexible working time model.Reach more women in precarious jobs.Provide more possibilities to unemployed women with health issues.Provide free German courses to participants.
Mid-term evaluation of material progress of the Smart Growth OP, 2014-2020, in Poland
Ewaluacja mid-term postępu rzeczowego Programu Operacyjnego Inteligentny Rozwój 2014-2020
Karolina Dobrowolska, Tomasz Klimczak, Bartosz Ledzion, Adam Miller, Adam Płoszaj, Zuzanna Popis, Korneliusz Pylak, Elżbieta Wojnicka-Sycz
Openfield Sp. z o.o., ul. Ozimska 4/7, 45-057 Opole, www.openfield.pl/ - IPC Instytut Badawczy Sp. z o.o., ul. A. Ostrowskiego 9, 53-238 Wrocław, www.instytut-ipc.pl
The evaluation examines the implementation of the Smart Growth OP 2014-2020, which has four Priority Axes: I. Increase R&D activities of enterprises; II. Increase capacity of enterprises to implement RDI activities; III. Increase capacity of enterprises to internationalise and increase the use of venture capital to finance innovations; IV. Improve scientific-research capacity.
To assess the progress of the OP at the end of 2018, including the implementation of the Performance Framework for the mid-term review, and the contribution to Europe 2020.
Desk research on OP documentation and strategy papers;34 interviews with representatives of the MA, IBs and beneficiaries of non-research projects;panel of experts, including representatives of the MA, IBs and external experts;analysis of monitoring data.
PA I:EUR 3.85 billion allocated, of which almost 64% contracted.Intermediate output targets have been achieved and final objectives are not at risk.The risk of not achieving target values of some indicators is due to the limited extent of implementation and termination of the agreement with one of the beneficiaries.PA II:Over EUR 1 billion allocated, of which 73% contracted.Intermediate output targets have been exceeded and final objectives are not at risk, except for the financial indicator in more developed regions.There are delays in some outcome indicators (e.g. ‘growth in employment levels in supported enterprises’ and ‘number of entities implementing projects in the area of industrial property’) particularly in less developed regions, because of lower than estimated number of enterprises supported and low activity of enterprises in industrial property protection. PA III:Over EUR 2.35 billion allocated, of which 71% contracted.Intermediate output targets have been achieved and final objectives are not at risk.There are difficulties in the implementation of activities under IP 3.1 (activities aimed at increasing capital financing of innovation) because the Loan Fund for innovation has been removed and due to the availability of other investment instruments.PA IV:Over EUR 1.22 billion allocated, of which 74% contracted.One of the financial indicators for less-developed regions has not been achieved compromising the allocation from the performance reserve, which is planned to be transferred to PA II, but this does not jeopardise the achievement of the objectives of the PA as a whole.
Data only go up to 31 December 2018. Expected results of the indicator values for 2023 were estimated taking into account completed, on-going and calls scheduled for 2019.
There is a need:under PA I, to provide funding in the next period to support the implementation of research results co-financed in the current period;under PA IV, to increase allocation to finance all positively assessed projects and in case of non-usage of all the allocation, to reformulate the 2020 call (e.g. reducing the minimum value of eligible costs);to reallocate funds between financial instruments and to transfer funds from financial instruments to grants.
Mid-term evaluation of progress in implementing the Lubelskie OP 2014-2020
Ewaluacja mid-term postęp rzeczowego i systemu realizacji RPO WL 2014-2020 dla potrzeb przeglądu śródokresowego, w tym realizacji zapisów ram i rezerwy wykonania
Michał Marciniak, Monika Sochaczewska, Marta Cichowicz-Major, Adam Rybkowski, Paweł Hess, Iwona Kania, Sebastian Pałka, Paweł Penszko, Małgorzata Zub, Marta Mackiewicz, Janusz Kornecki, Ewa Dzielnicka, Katarzyna Lipczyńska, Paulina Gawryś, Jerzy Radziwiłł
ECORYS Polska Sp. z o.o., ul. Solec 38 lok. 105, 00-394 Warszawa, e-mail: ecorys@ecorys.pl, EVALU Sp. z o.o., ul. Dzika 19/23 lok. 55, 00-172 Warszawa, e-mail: evalu@evalu.pl
The report is a mid-term evaluation of the Lubelskie OP 2014-2020.
To assess progress in pursuing the specific OP objectives with particular attention to the achievement of the performance Framework objectives and the contribution to Europe 2020 .
desk review;analysis of statistical and monitoring data;11 interviews with institutions involved in the implementation of the OP;survey of beneficiaries on the preparation and implementation of projects;statistical and econometric analyses;panel of experts.
There were no problems in achieving the Performance Framework.The assumptions made at the programming stage (relating to specific issues identified in particular areas, level of unit costs in different types of project etc.) have been confirmed except for minor exceptions.Despite external factors identified in the case of some of the priority axes, no major obstacles in implementation are evident.
end the implementation of selected measures under IP 4c and 4e;change target values of selected indicators (e.g. IP 2c, 3c, 4a, 4b, 6a-d, 7b, 7d, 8iii, 10i, 10iii, 10iv, 9a, 9b, 10a);reallocate funding for some measures (e.g. 1b, 7b, 10iv, and 10.3); mobilise the performance reserve in PA3;verify the methodology for estimating target values for indicators in IP 5b;update the socio-economic analysis for the labour market because of significant changes in recent years;end projects under the Integrated Territorial Investments (ITI) of the Lublin Functional Area or reallocate unused funds for the same activities, except for Territorial Policy;provide more funds for ITI projects on transport and adaptation to climate change and specifically to low-carbon economy projects (post 2020).
Mid-term evaluation of the West-Nederland OP, 2014-2020: low-carbon economy
Tussentijdse evaluatie Kansen voor West II Prioritaire as Koolstofarme economie
Hans van der Zwan, Babette Beertema, William van den Bungelaar
European and Regional Affairs Consultants (ERAC), www.erac.nl
This mid-term evaluation examines the implementation of PA2 ‘Supporting the shift towards a low-carbon economy in all sectors’ of the West Nederland OP.
To identify the learning effects that can be used in shape the OP for 2021-2027.
documentation analysis;16 semi-structured interviews.
Out of 23 applications received since 2015, 12 projects were approved.Indicators for sustainable energy generation are expected to exceed their targets, while targets for reducing energy use are unlikely to be achieved.Interest in the low-carbon economy is lagging behind, because of the difference in expectations between project applicants and assessors.The OP text is complex and opaque.The lack of a “value for money” method makes it difficult to assess low-carbon projects.The expert committee has only one expert on energy.The roles of those involved in the management of the programme are not entirely clear.Although the number of projects is small, they produce the right output.
Due to the early stage of project implementation, the results are limited or non-existent, which does not enable a theory-based analysis to be applied as envisaged.
strengthen the way the programme is communicated;cooperate with the other subsidy providers;clarify the guidelines and extend the time for applications;optimise the “CO2 value for money” method;involve more specialists in different areas as experts;organise discussions between the expert committee and support centres to exchange information and clarify roles;conduct market analysis of the current subsidy landscape;clarify the wording of the OP;consider a more qualitative project assessment system.
Assessment of the effect of support for people in the most difficult situation on the labour market, 6 months after completing participation - Mazowieckie OP 2014-2020
Ocena wpływu wsparcia kierowanego do osób w najtrudniejszej sytuacji na rynku pracy w województwie mazowieckim na ich sytuację po zakończeniu udziału w projekcie
ul. Toruńska 18c lokal D, 80-747 Gdańsk, www.eu-consult.pl
The evaluation assesses the effect of the support provided under the ROP Mazowieckie Voivodeship 2014-2020 on people in the most difficult situation on the labour market, six months after completing participation in the Programme. The assessment focus on the IPs 8.iv, 9.i, 9.iv, 9.v, 10.i, 10.iii and 10.iv.
To assess the impact of the Programme to improve the employment situation of participants 6 months after leaving the interventions.To identify the forms of support with the greatest impact on changing the participants’ labour market situation.
Desk research;on-line survey of project beneficiaries which have left the programme since at least 6 months (stratifies sample by IP, gender and sub-action; total sample of 2,262 individuals out of 13,336);survey of entrepreneurs and other types of employers employing project participants 6 months after the end of the project;individual interview;case study;panel of experts;benchmarking;SWOT analysis.
The gross value of the indicator "Number of people in a better situation on the labour market six months after leaving the interventions” is about 13%. The highest value of the indicator refers to interventions under IPs 8.iv (25%) and 10.iv (19.8%); the lowest value refers to interventions of IP 9.iv (0%).Improvements in the labour market situation refers to: transition from a temporary to a stable employment (4.3% of participants); transition from a part-time to a full-time employment (0.8%); increase in salary (26%).Deterioration in the labour market situation refers to: transition from a stable to a temporary employment (1.1% of participants); transition from a full-time to a part-time employment (0.1%); decrease in salary (26.3%).No change has been observed in almost all of the IPs. The overall indicator is 44.5%. The highest value of the indicator refers to interventions under IP 10.i (56.7%); the lowest value refers to interventions of IP 8.iv (25%). The form of support with the greatest impact on changing the participant status was training and courses.One year after leaving support, about 58% of the participants affirm that the effects of support in the form of a change in their professional situation persist. About 21% of them indicate that their professional situation improved and only 2% that their situation deteriorated. Improvement consisted mostly in: increase in salary, job change with better employment prospect for professional development, job change which is more suitable to participant, job change with higher responsibility, continuing education, job change to a more stable position.The main issue in programme implementation related to the recruitment of participants, bureaucratic barriers and problems related to the settlement of the project (mainly resulting from the need to apply EU procedures).
Data are up to 30 June 2018. For those measures where the number of observations were limited, the whole population was considered. However, for some sub groups (e.g. 9.v) the whole population could not be reached.
There is a need to:increase support addressed to the group of people below 30 years old that, due to its specific characteristics (young people, often well-educated, being young parents) require specific forms of support;enhance support delivered through e-learning;reward projects which integrate several forms of support (e.g. professional counselling combined with training) at the stage of project selection.
Evaluation of measures concerning the education of medical professionals under the European Funds for Social Development programme, 2021-2027, in Poland
Ocena działań FERS dotyczących kształcenia pracowników sektora ochrony zdrowia
The evaluation examines early implementation of measures dedicated to postgraduate education for doctors, nurses and midwives, and skills for non-medical staff.
To assess the relevance and effectiveness of the measures for training of medical and non-medical staff, including an assessment of implementation conditions enabling effective delivery.
Desk research.Analysis of statistical and monitoring data.Survey of 17 beneficiaries and 895 participants.Focus group with representatives of institutions.19 interviews with beneficiaries.Expert panel.11 case studies.
The measures are assessed as well-targeted to core human resources challenges in healthcare and aligned with ESF+ strategic objectives.Beneficiaries indicate high demand and 93% of participants believe the support is strongly tailored to professional needs.Effectiveness of learning is linked to adapting formats to recipient characteristics. Mixed/hybrid models are reported as effective and positively rated, with differences by age and role.The quality of the implementation system is assessed very positively, while identified risks include uneven quality of assessors’ judgements and a lack of uniform guidance on sustainability of results.
The report explicitly notes the early stage of implementation, which constrains outcome measurement.
Streamline administrative/documentation processes for participants and providers, prioritising digital document flows where feasible.Build “hybrid-by-design” training standards as a default option.Introduce minimum functional requirements for e-learning components, to avoid technical problems.Use early-warning monitoring for delivery risks to ensure target achievement under high demand conditions.
Mid-term Evaluation of the Andalucía ESF OP 2014-2020
Informe de evaluación del Programa Operativo FSE Comunidad Autónoma de Andalucía 2014-2020
Dirección General de Fondos Europeos, Junta de Andalucía; Saet Consultores
This is the mid-term evaluation of the ESF OP for Andalucía 2014-2020.
To assess the financial and operational progress of the OP and to examine whether the logical framework is still appropriate.
The evaluation adopts a mix of qualitative and quantitative methods:Desk research;analysis of relevant policy documents;analysis of monitoring data;two on-line surveys of implementing bodies;analysis of official statistics from Eurostat and national sources.
The intervention strategy proposed in the OP continues to be in line with regional needs.There are delays in physical and financial implementation, mainly because of delays in completing the 2007-2013 OP and the difficulty of budgeting during 2014 and 2015 due to financial constraints.The main obstacles to OP implementation are difficulties in the interpretation of Regulations, in interpreting the eligibility of expenditure, with internal communication; in hiring qualified personnel and in the administrative processing of financing requests.Delay means it is impossible to assess result indicators.
Data go up to 31 December 2016. The low level of physical and financial implementation limited the possibility of assessing result indicators, so the assessment is based only on expenditure incurred.
ESF co-financing should be focused on providing employment guidance to those of 30 and over rather than to younger age groups who are already covered from other sources.The actual absorption capacity of PA2 activities, directed at women, the disabled, migrants and other disadvantaged groups, should be assessed and resources possibly allocated to other activities to support the employability of the same groups.The definition of the simplified cost module and performance indicators should be accelerated in respect of PA3 activities (particularly school guidance and training courses for languages teachers) so as to increase the rate of implementation.
Evaluation of Business Programme measures funded by the “Education and Skills” ESF+ Programme and by the “Smart Green and Digital Transition of Enterprises” ERDF Programme, 2021-2027, in Denmark
Pluss: https://pluss.dk/, post@pluss.dk; IRIS Group: https://irisgroup.dk/, irisgroup@irisgroup.dk
The evaluation examines the implementation and preliminary effects of the “Enterprise programme II”, which is structured around 6 different themes (circular economy, digitalisation, export markets, green skills, recruitment, strategy and management). It covers the implementation period between 2022 and 2024.
To assess the implementation of the measures for businesses and identify the main features of the beneficiaries. To assess the short-term effects of the measures. To provide recommendations to adjust and improve the programme.
Document analysis.Analysis of monitoring data (applications and rejections, main features of the beneficiaries…).Online survey of companies (876 respondents).Interviews with operators.Focus groups with business developers and consultants.92 interviews with beneficiaries (SMEs).36 interviews with non-users.10 interviews with partners (clusters, industry organisations, researchers).
Most of the beneficiaries are established SMEs from the transport or wholesale sector with less than 50 employees, who have already received an external support or guidance.The majority of beneficiaries acquired knowledge of the programme through their business centres.80% of beneficiaries underline that the programme has already improved (37%) or will improve (43%) their productivity.2/3 of the beneficiaries consider that the programme has improved or will improve their sustainability.The majority of the SMEs consider that the programme was crucial to implement new development initiatives.A significant proportion of beneficiaries consider that the administrative requirements of the programme were too high and the application process was cumbersome, especially for less experienced beneficiaries.
Very low response rate to a follow-up survey of SMEs, which could not be used.
Simplify the application process and improve the user-friendliness of the application system.Clarify the monitoring requirements and reduce the number of documents to send.Increase the focus on inexperienced SMEs.
Evaluation of the implementation system for the Pomorskie OP, 2014-2020
Ocena systemu realizacji Regionalnego Programu Operacyjnego Województwa Pomorskiego na lata 2014-2020
Pracownia Badań i Doradztwa “Re-Source” Korczyński Sarapata sp.j., ul. Spławie 53; 61-312 Poznań, Tel: 61 622 92 06-07, e-mail: biuro@re-source.pl
The evaluation examines the implementation system of the Pomorskie OP, as regards the institutional structure, organisation and management system in terms of effectiveness, efficiency, usefulness and relevance.
To assess the effectiveness and efficiency of the OP’s institutional system, specific mechanisms and instruments and to make recommendations on implementation.
Desk research;benchmarking (on basis of regional and national OPs);in-depth interviews with representatives of the MA and beneficiaries;in-depth telephone interviews with FUA representatives and project managers;online surveys;case studies of mechanisms implemented;expert panel.
The structure and division of tasks between institutions responsible for implementation was assessed positively.There are shortages of personnel in the institutional system.National and Community regulations have a strong impact on the regional level of implementation, mainly because of long waiting time for the necessary regulations to be introduced and the imprecise scope of these.Procedures concerning the various stages of implementation were assessed as being clear and legible.Mechanisms used to ensure effective connection of the OP with the Pomorskie Development Strategy 2020 were assessed as being effective.
There is a need to:standardise the form and scope of obligations of the different institutions, limiting the risk of delay and difficulties resulting from long decision making processes;enable the MA to prepare additional information material in relation to "horizontal" guidelines;simplify the process of project assessment and selection;limit the amount of documentation expected from beneficiaries;increase coordination between the OP and the Regional Development Strategy;improve coordination at regional and national level.
Evaluation of measures under PA1 "Promoting Innovation" of the Niedersachsen ERDF/ESF OP, 2014-2020
Evaluationsbericht zur Prioritätsachse 1 „Förderung der Innovation"
This is the final evaluation report on three Specific Objectives (SOs) to promote innovation under PA1 of the OP, up to 30 June 2022.
To assess the level of implementation, the main results and impact of the measures financed under PA1 up to end-June 2022.
Analysis of monitoring data.Documentation analysis.Online surveys of beneficiaries (response rate 52% for SO1, 25% for SO2, and 43% for SO3).Testing of the intervention logic of each SO.
There is a good level of achievement of output and financial targets.Institutes supported can participate in international scientific competition and receive third-party funding.Competitiveness of the companies supported has improved through product and process innovation leading to increased turnover or cost savings.Projects supported have contributed to the scientific development of research institutes.Causal mechanisms leading to the expected impact are confirmed by beneficiaries.
Assessing impact on employment was not possible because of lack of data.
Mid-term evaluation of the design and structure of the JTF Programme, 2021-2027, in the Netherlands
Mid-term evaluatie - JTF-programma Nederland: Procesevaluatie opzet en inrichting
Tessel Blom, Marenne Massop, Femke van Wijk, Ben Bresser, Vincent Mosmuller, Jeroen van der Teems; Gert-Jan Lindeboom, Bert-Jan Buiskool
Dialogic innovation & interaction, blom@dialogic.nl; Ockham IPS, b.buiskool@ockham-ips.nl, g.lindeboom@ockham-ips.nl
This evaluation examines the implementation and execution of the Just Transition Fund, which aims to support the transition to a low-carbon economy. Implementation of the measures commenced in 2023. The cut-off date of the monitoring information used during the evaluation was end-March 2024.
To examine the implementation of the projects and provide lessons for the remaining programming period.
Document analysis (relevant policy documents, reports and other publications, public information published by the implementing bodies and reports from the monitoring committee).Analysis of monitoring data.Analysis of administrative data.46 interviews with programme stakeholders.
The programme has an acceptable form of operational cooperation, although regions experience too little autonomy to organise tasks efficiently, and the central government has little resources to adjust the content of decentralised programmes.Engaging MAs of other regional ERDF programmes in the implementation of this programme has fostered synergies in both its planning and execution.The intensive support for applicants in the project preparation and the early involvement of external parties in project development appeared to be an important factor in securing sufficient high-quality project applications.All project applications adhered to the EU horizontal principles, although some guidance was required to understand the “do no significant harm” principle.Although the EU spending requirements have been significantly reduced, it will remain a challenge to complete the financial reporting of a sufficient number of projects by the end of 2026.
Delegate formal implementation tasks to regional bodies in future programmes so that they can respond better to changing circumstances.Include more guidance to ensure consistent application of the horizontal principles.Further define the definition of “social objectives” in the context of the administration agreements, including budget distribution and possible sanctions.
Evaluation of family harmonisation and professional life measures in Greece. Interim report on results
Παραδοτέο Γ: «Ενδιάμεσο παραδοτέο της αξιολόγησης των αποτελεσμάτων της δράσης ως προς τον ωφελούμενο πληθυσμό» στο πλαίσιο του έργου με τίτλο: «Αξιολόγηση των δράσεων εναρμόνισης οικογενειακής και επαγγελματικής ζωής στο ΕΠ ΑΝΑΔ ΕΔΒΜ και στα ΠΕΠ»
This is an interim report on the results of the "Evaluation of family harmonisation and professional life measures in the Human Resources Development, Education and Lifelong Learning OP and the regional OPs" in Greece.
To summarise the results from the measures in quantitative terms.
Data analysis;data collection;focus groups;semi-structured questionnaires.
In 2016-2017, the number of participants increased (to 75,070) compared with the two previous periods (2014-2015; 2015-2016).The vast majority of participants were women.The majority of participants came from Central Macedonia and Attica.The employment status of participants varied significantly in the three periods. In 2016-2017, 50% of participants were unemployed and 47% employed.The satisfaction of mothers with the childcare measures increased significantly from 2014-2015 to 2016-2017.86-90% of participants were still in employment after completing the measure. One in 4 respondents reported that they would have stopped working without the support of the ESF.70% of unemployed and inactive participants reported that the ESF helped them to find a job.98% of beneficiaries report that they were satisfied with the way measures were implemented.
There is a need to:increase the funding for the measure;review the selection criteria in order to attract more women;simplify submission procedures;increase the number of social care places.
Analysis of the structure of services, service providers and clients registered in the BUR Database of development services under the ESF OP, 2014-2020, in Poland
Ewaluacja „Zpływ bazy usług rozwojowych na jakość i rynek usług rozwojowych oraz ocena podmiotowych systemów finansowania – ETAP II”
Danae Sp. z o.o., ul. Bora-Komorowskiego 19B, 80-377 Gdańsk
The study examines the functioning of the ESF support model at national and regional level for entrepreneurs and employees.
To assess the functioning of the database of development services (BUR) at the national level, its quality and its effect on the structure of services and differences in regional financing systems in terms of effectiveness, efficiency and utility.
Desk research.CATI/CAWI surveys of employees, clients, service providers and operators.122 interviews with MA, operators, service providers and enterprises.Focus groups.
4,619 entities providing development services are registered in the database of development services (BUR).Since the beginning of the database, 335,352 development services have been published, 20% of which were actually provided.161,825 people have benefited from the services.The development services are considered to be tailored to the needs of companies.As result of the services, employees have increased their knowledge and competences, their self-confidence, employability, motivation and involvement in their work.Only 25% of respondents reported that they would use the development services if they were not subsidised.
The differences in the types of service in the database and in the classifications used in national statistics make it difficult to assess how services should be adjusted to the needs of the economy.
Continue the functioning of the BUR and PSF (regional) databases in 2021-2027.Strengthen the distribution of specific types of service by raising the threshold for their financing.Use cost-benefit analysis to determine the best model for the next period.Unify the system at least for key aspects (collection of cost data, documentation, principles of control over the development services implemented).
Evaluation of tourism measures under the Thüringen ERDF OP, 2014-2020
Evaluierung der Maßnahmen zur Tourismusförderung
dwif-Consulting GmbH, Sonnenstr. 27, 80331 München; Tel: +49 (0)89 - 237 028 9-0; E-mail: info@dwif.de, m.seibold@dwif.de, m.kantsperger@dwif.de
The evaluation examines measures to promote tourism in Thüringen. The analysis is based on 59 projects approved between January 2014 and December 2020.
To assess effectiveness, efficiency and impact of the measures to support tourism, on the basis of 59 projects approved between January 2014 and December 2020, and to make recommendations for improvement.
Documentation analysis.Analysis of statistical data.Survey of beneficiaries (24 responses, 63% response rate).3 discussions with Land authorities and 18 interviews of beneficiaries.5 case studies.Workshops to discuss the evaluation design and preliminary results.
ERDF support for tourism measures amounted to EUR 41.2 million, out of EUR 55.2 million in total.At the time of the evaluation, 26 of 59 projects had been completed, mainly in marketing.Because of Covid-19, the result indicator targets (on overnight stays) are unlikely to be achieved, but the overall impact of the pandemic on the implementation of projects is limited.The measures enabled public infrastructure to be improved and upgraded.Marketing measures have promoted the image and awareness of Thüringen as a cultural destination.According to respondents, without ERDF support, the majority of projects would not have been implemented or would have been smaller in scale and without an international focus.New target groups have been reached.The majority of beneficiaries were satisfied with the programme.Some selection criteria are considered not suitable for tourism projects or difficult to operationalise.
The wide spread of public infrastructure measures does not allow the effects at regional level to be assessed. It is not possible to assess how the use of funds was distributed across national or international target markets because marketing activities were not recorded in detail for all projects. Most of the structural measures had not been completed at the time of the evaluation, so their effects on overnight stays could not be assessed. The effects of advertising measures on overnight stays are unclear because of a lack of data to establish such a link.
Continue supporting marketing activities.Maintain a two-step application procedure.Ensure that decisions on the eligibility for funding are transparent and comprehensible.Intensify support for grant recipients, including through training and advice on public procurement, and ensure an exchange of experience.Simplify procurement procedures.Define result indicators in a way that they relate more to the content of the measures.Reduce the administrative burden.
Evaluation of PA 2 “Use of information and communication technologies” under the Competitiveness and Cohesion OP, 2014-2020, in Croatia
Vrednovanje učinka Operativnog programa Konkurentnost i kohezija 2014. – 2020. Prioritetna os 2: Korištenje informacijske i komunikacijske tehnologije (Grupa 2)
info@equinox.hu ; ivana.katuric@urbanex.hr
The evaluation examines the progress in implementing measures relating to ICT financed under the OP (PA 2) and their effects. The measures concern the development of broadband and e-services. Monitoring data used go up to end-2021.
To assess the relevance, effectiveness, efficiency and impact of the ICT measures implemented and the achievement of short- and long-term objectives. To assess the sustainability of the effects observed.
Analysis of monitoring data and official statistics. Theory of change testing. Analysis of strategic documents, calls. Interviews with MA and other national institutions (e.g. Agency for Network Activities, Office for the development of the digital society). Survey of beneficiaries of e-services measure (3 replies out of 20). Case study.
The number of e-services available (30) increased by 75% between 2019 and 2021). The corresponding result indicators on the use of e-services by individuals also increased, but this cannot be reliably attributed to the measures implemented. Selection criteria of calls for broadband development did not encourage beneficiaries to cover disadvantaged areas. Too strict criteria also resulted in implementation delays. Beneficiary local authorities lack the expertise and financial resources to implement projects effectively, and national formal procedures are too complex and lengthy. Ex-post controls of public procurements procedures and expenditure claimed are extremely time-consuming, because of both the limited expertise of beneficiaries and the inadequateness of the control system.
Most projects were still ongoing at the time of the evaluation, so the effects of the measures observed are preliminary.
Improve the capacity of MAs and beneficiaries to manage EU funded projects. Ensure that selection criteria encourage applicants to provide services in disadvantaged areas and improve the monitoring system. Develop the capacity of beneficiaries through training and simplify administrative procedures. Monitor project budgets constantly to ensure their financial sustainability and keep a funding aside to tackle unexpected circumstances.
Operational Evaluation of the Extremadura OP reprogramming
Evaluación por reprogramación del programa Operativp FEDER de Extremadura 2014-2020
Calle de la Mancha, 1, 28823 Madrid, Teléfono: 918 83 80 08
The evaluation assesses the financial reprogramming and implementation of the Extremadura OP for 2014-2020.
To examine the rate of financial implementation of the OP and to assess whether the reprogramming is coherent with the initial OP strategy.
A combination of qualitative and quantitative methods based on documentation and secondary, mainly financial, data from the Fondos 2020 accounting system and intermediate bodies.
The main purpose of reprogramming was to include an increase in ERDF support of EUR 40.3 million and national co-financing of EUR 10.1 million.In the new programme, the funding for investment supporting the shift towards a low-carbon economy (TO4), promoting social inclusion, combating poverty and any discrimination (TO9) and Education (TO10) has been increased.The new programme includes a new investment priority (IP 4.5 ‘Promoting low-carbon strategies for all types of territories’).Changes were made to output and outcome indicators and the target values.These modifications did not change the strategy or the intervention logic of the OP.
As at 31 December 2016, no certified expenditure had been reported, so the analysis of financial implementation is based on the incurred spending.
No policy implications per se. The reprogramming does not involve any significant change in the OP strategy.
Evaluation of the implementation of the partnership principle in 2014-2020 OPs
Ewaluacja realizacji zasady partnerstwa w ramach perspektywy 2014-2020
All OPs 2014-2020; Interreg V-A-Poland-Slovakia 2014-2020, Interreg V-A-Poland-Germany/Saxony 2014-2020
EVALU Sp. z o.o., ul. Dzika 19/23 lok. 55, 00-172 Warszawa, http://evalu.pl/
Evaluation of the implementation of the partnership principle in the 2014-2020 period.
To assess the implementation of the partnership principle and to make recommendations for the current and next programming period.
Desk research (of legal, programming and operational documents, reporting data, results of evaluation studies and other analyses).Survey of MA and of implementing bodies.166 interviews.4 case studies.Expert panel.
External partners feel little involved because of lack of control over the spending of public money and limited possibilities of influencing the effectiveness of implementation.Some members of the Monitoring Committee believe their role is only limited to legitimising MA activities.There is no forum for the exchange of experience in respect of horizontal policy.The role of different types of body and partner is described in different documents in different ways.The documentation of the activities of working groups and external communication is not adequate.The partnership principle is being implemented only to a minimal extent in programming for 2021-2027 because of COVID-19 and delays in the current programme.
Promote measures that strengthen partnership at the local or sub-regional level.Reduce the role of the MC chairman and create an MC ‘executive body’ with equal participation of all categories of partner.Consult MC members on call for proposal rules.Introduce the possibility of immediate financing of necessary training for MC members, finance the costs of partners participating in MC activities from outside the administration and the coordination and networking activities of civil society bodies.Harmonise the Guidelines on MCs and on the implementation of the partnership principle with the regulations.Recruit social partners through industry organisations that then select members according to the main thematic areas supported by the programme.Define reporting requirements for working groups similar to those for the MC. Strengthen the role of the MC Secretariat.Make greater use of online or mixed public hearings in the consultation procedure for 2021-2027.
Analysis of social innovation projects financed by the Employment OP, 2014-2020, in the Czech Republic
Kvalitativní šetření inovačních projektů PO 3 OP Z - 2. vlna: Etapa 1
MindBridge Consulting a.s: Ing. Tomáš Kravka, Ing. Ladislav Klika, Ing. Šárka Vidlařová
www.mindbridge.cz/en/, info@mindbridge.cz
The evaluation examines nine projects on social innovation financed by the OP. The analysis was carried out between the end of July and the beginning of September 2020.
To identify factors that affected the success of 9 projects financed by the OP, the importance of cooperation in implementation and their potential sustainability.
Interviews with 13 implementers, 9 project assessors and 38 other stakeholders.Desk analysis of project documentation.
The main factors affecting results are careful preparation of the initial phases, a highly professional team and the involvement of experts, as well as an ability to reach vulnerable groups and cooperation with local authorities. For all projects the key indicator targets had not been met as at mid-2020, but for almost all projects there were no significant changes from the initial plans.In many projects, the role played by social services was fundamental.For most of the projects, sustainability is uncertain and conditional on future financing, as well as on political support.
Those interviewed were indicated by project developers themselves, so self-selection bias can be a factor. Some interviews were carried out up to 3 years after the start of the project and respondents could not remember all the details. The heterogeneity of stakeholders made it difficult to use a uniform approach to process their responses.
Distinguish between issues of sustainability and replication of existing projects.
Evaluation of the implementation of Just Transition measures under the European Funds for Śląskie Programme, 2021-2027
Ewaluacja dotycząca postępu rzeczowego i finansowego FE SL 2021-2027 w zakresie funduszu FST na potrzeby przeglądu śródokresowego
The evaluation examines the implementation of the Just Transition Fund (JTF) in Śląskie.
To assess the progress in implementing the JTF and to provide recommendations for possible modifications to the Programme.
Desk research. Analysis of monitoring and statistical data. 14 interviews with representatives of MA and IBs. 17 interviews with project beneficiaries. 3 interviews with experts in regional transformation topics. 3 interviews with social and economic partners. 3 interviews with civil society representatives. 2 interviews with representatives of the Małopolska region. Benchmarking of 4 regions. Focus group with MA and IBs. Survey of 71 local government units.
JTF interventions are generally well-aligned with the region’s needs, although delays in the implementation of the programme create time pressures. Outplacement and retraining measures for miners are limited by employment guarantees under the Social Agreement, but they remain crucial for employees of mining-related sectors. Demarcation between JTF, ERDF and ESF+ has mostly prevented overlaps, although in certain areas competition between funds still occurs. About 65% of the total JTF allocation was committed by mid-2024, which puts pressure on the full absorption, given the 2026 deadlines. Several outcome targets are at risk of not being fully met due to cost escalation associated with high inflation and the late start of the programme.
Continue JTF support for the region in the next programming period, recognising that the scale and duration of the transition exceed current resources. Strengthen preferences for enterprises in mining-related industries to encourage diversification and redeployment of the workforce. Explore flagship or “priority” projects in areas such as post-mining land reclamation and innovative industries, boosting new growth drivers for the regional economy.
Evaluation of the material progress and results of the Podkarpackie OP, 2014-2020, in terms of socio-economic development of the region and achievement of the objectives of the Europe 2020 Strategy
Ewaluacja podsumowująca postęp rzeczowy i rezultaty RPO WP 2014-2020, ich wpływ na rozwój społeczno-gospodarczy województwa oraz na osiąganie celów Strategii Europa 2020
EVALU Sp. z o.o., ECORYS Polska Sp. z o.o.
The evaluation investigates the impact of the Programme on the socio-economic development of the region.
To assess the progress of the implementation of the Programme and its impact on the socio-economic development of the region.
Desk analysis of relevant documents.Analysis of monitoring data and official statistics.11 interviews with implementing institutions.Expert panel.
The large majority of objectives and indicators have been met.GDP per capita in the region is estimated to be 0.2-3.9 pp higher (depending on the year) than it would have been without EU funding.In 2021, the employment rate for 20–64-year-old people reached 63.1%, higher than the 60.5% estimated in the case of absence of EU funding.The main factors affecting the effectiveness of the programme were COVID-19 and the war in Ukraine, since the region borders Ukraine directly.
Carry out an evaluation of the sustainability of support for business start-up for young people.Increase support for the development of regional road infrastructure (particularly in border areas) to improve the business environment, in response to the war in Ukraine.
Final Evaluation of the communication strategy of the OP Employment, Training, and Education 2014-2020
Evaluación Final de la Estrategia de Comunicación del Programa Operativo de Empleo, Formación y Educación 2014-2020
Evaluation of the effects of social service infrastructure projects under the Łódzkie OP, 2014-2020
Wpływ inwestycji w infrastrukturę w ramach osi priorytetowej VII RPO WŁ 2014-2020 na dostępność i jakość usług społecznych
Krzysztof Gutta, Tomasz Gutta, Katarzyna Gutta, Michał Dominik, dr hab. Joanna Hołub-Iwan
The evaluation examines how support for investment in infrastructure has contributed to the availability and quality of social services in the region.
To assess the impact of investment in social infrastructure on the availability and quality of social services.
Documentation analysis.Analysis of monitoring data and official statistics.Survey of 1,136 inhabitants of the region.Survey of 144 beneficiaries.Survey of 172 self-government units.13 interviews with representatives of implementing institutions.Benchmarking of 3 regions.
Projects seem to have helped to reduce intra-regional disparities in access to services, but preschool infrastructure is not sufficient in rural areas. There is much need for the development of health and social services in a deinstitutionalised form.After the pandemic, the use of ICT by schools and students has increased.There is a need to develop e-health services, partly to reduce shortages of medical staff.
Continue support focussing on rural areas and publicise project funding opportunities better.Encourage NGOs to establish aid and support facilities.Continue ICT projects at all stages of education, including for distance learning training.Continue support for ICT.Promote greater cooperation between vocational schools and employers.
OP Human Resources and Employment 2007-2013
Evaluation report on literacy and basic education under the Baden-Württemberg ESF OP, 2014-2020
Evaluationsbericht: ESF-Programm Alphabetisierung und Grundbildung im Rahmen der Evaluierung des Operationellen Programms des Europäischen Sozialfonds in Baden-Württemberg 2014–2020
Georg Kalvelage, ISG Institut für Sozialforschung und Gesellschaftspolitik GmbH
kalvelage@isg-institut.de – ISG, Weinsbergerstraße 190, 50825 Köln 0221-130 655-30
The evaluation assesses the measures under specific objective (A5.1) “strengthening of the competitiveness of the working population and medium-sized enterprises” supported by the Baden-Württemberg ESF OP, 2014-2020, which involve literacy and basic education courses for adults.
To assess the funding, measures, output and effects of the measures concerned.
analysis of all project applications and case reports submitted by beneficiaries;expert interviews with the Ministerium für Kultus, Jugend und Sport and the specialist department for basic education and literacy sponsored by the programme; analysis of monitoring data;survey of participants (by phone);survey of beneficiaries;case studies of two beneficiaries.
There is a divergence between the initial plan and actual implementation as regards the target group, mainly because the majority of participants are migrants and are not representative of illiterate people in Germany.The attraction of participants is a major challenge.Flexibility in deciding the content and organisational aspects of courses enabled individualised support to be put in place. The possibility of participating in a maximum of two courses with 80 teaching units per semester is too restrictive.The courses are primary aimed at creating self-confidence and teaching basic learning competences, and less at achieving a higher knowledge of the language given the duration of courses.Participation in courses helps participants to have a more stable position in the labour market.
finance longer-term courses on literacy and basic education;target the needs of participants better (e.g. focus on the ability to express themselves orally rather than writing skills);involve additional staff to attract participants, e.g. by involving the specialist department for basic education and literacy that is sponsored by the programme;advertising the courses organised through employers better.
Evaluation of support for SME competitiveness in the Regional OP, 2014-2020
Evaluarea intervențiilor POR 2014-2020Competitivitatea întreprinderilor mici şi mijlocii
Virgilio Buscemi - Lattanzio Advisory Spa
The evaluation examines support for SMEs and business incubators under PA 2 of the Regional OP, 2014-2020, aimed at consolidating their market position, improving the survival rate of microenterprises and increasing SME competitiveness.
To assess the net contribution of the OP to achieving the specific objectives of PA2 and SMEs competitiveness and how the effects are produced, how far they are sustainable and what other effects are produced.
literature review; semi-structured interviews with OP management at central and regional level;survey of enterprises supported;3 focus groups with stakeholders;a panel of experts;4 case studies.
The survival rate of microenterprises 3 years after establishment increased from 63% in 2011 to 69% in 2016. The 2023 target of 73% is likely to be reached. SME labour productivity has also increased since 2011 and the target should also be met because of new technology and more efficient equipment.According to the stakeholders, the OP has significantly contributed to the consolidation of beneficiaries’ position on the market and their survival, product diversification and quality improvement.The conclusion from the case studies is that the contribution is proportional to the investment.The OP has simplified submission, evaluation, contracting, monitoring and control, but administrative obstacles deterring SMEs need to be alleviated.
Time constraints because of delays in preparing data collection methods and the limited availability of stakeholders during winter holidays.Limited availability of beneficiaries to participate in the survey and focus groups.The monitoring database was incomplete and contained errors.A counterfactual analysis was not possible because projects were completed too recently to produce effects.
assess the scope for using integrated financial mechanisms, allowing SMEs to apply individually or in partnership for overall funding covering different needs;extend the financing of other types of business support infrastructure to innovative clusters or encourage sectoral/specialised incubators in competitiveness areas;introduce new public policies to support SME innovation and internationalisation, which reflect real development needs and are coherent at the national level;encourage beneficiaries to apply again through "cascade" financing.involve stakeholders, e.g. SME associations, more actively throughout the 2021-2027 period;create a network of SME information and support offices;improve and simplify project selection and implementation procedures.
Evaluation of the support provided in the areas of social inclusion and poverty under the Łódzkie OP, 2014-2020
Identyfikacja oraz ocena efektów wsparcia udzielanego w ramach OP IX RPO WŁ 2014-2020 w obszarze włączenia społecznego i walki z ubóstwem
Paulina Kowalczyk, Anna Górecka-Ojdana, dr Jolanta Kluba, Paulina Sęk, Weronika Zych
The evaluation examines the effectiveness of the following measures: Measure IX.1 - Active integration of people at risk of poverty or social exclusion, Measure IX.2 - Services for people at risk of poverty or social exclusion, Measure IX.3 - Development of the social economy.
To assess the impact of the measures supporting social inclusion and tackling poverty on the reduction of social exclusion and poverty in Łódzkie.
Desk research (programme and strategic documents).Analysis of monitoring and statistical office data.19 in-depth telephone interviews with programme institutions, beneficiaries and experts.CAWI/CATI survey of 187 beneficiaries.CATI survey of 1,105 final recipients of support.1 focus group.4 case studies.Cartographic method.
The interventions in the area of social inclusion proved to be effective overall, in terms of achievement of basic targets.The functioning of social economy entities and social enterprises on the regional and local market is still not satisfying, as there is a misconception that the services they provide are of low quality.Employment stability in these entities is improving as a result of the intervention, but a significant proportion of employees is still employed only for the duration of the project. This is mainly due to the financial problems of these entities.In the 2021-2027 period, problems in the area of social inclusion remain relevant, but due to the improvement in the economic situation, the basic characteristics of the recipients of support have changed. The most vulnerable groups on the labour market include economically inactive people of post-working age, older people, and those with a low level of education.The interventions did not adequately address the needs of people leaving foster care, who are faced with the challenge of becoming independent.
Extend the scope of intervention to include measures promoting cooperation between social enterprises and local government units in the area of social services, especially care services.Extend the scope of support to include a component known as an “investment subsidy,” i.e. financial resources that would allow social enterprises in a specific industry to develop their activities and increase their economic efficiency.In 2021 -2027, focus support for social inclusion on people in particularly difficult situations, i.e. economically inactive people aged 50+, those with a low level of education or with health issues, etc.
Final evaluation of the work experience measure for young people, “Torno subito”, under the Lazio ESF OP, 2014-2020
Rapporto di valutazione finale - Torno Subito
The evaluation examines the measure in 2015 and 2016. The measure includes two phases: work experience in Italy and abroad followed by work experience in Lazio.
To assess the implementation and management of the measure in 2015 and 2016, the characteristics of participants, the gross and net results on the employment and other qualitative aspects.
Desk analysis of documents and monitoring data.Analysis of administrative data (at the beginning and the end of work experience).Counterfactual analysis (RDD method).Interviews with main stakeholders.
Both 2015 and 2016 calls show positive results in terms of the employment rate of participants, which was around 30% 12-18 months after for both.According to the counterfactual analysis, people supported under the 2015 call have an employment rate 6 pp higher than the control group 12 months after and 3 pp 18 months after the projects, while for the 2016 call, it was 10-11 pp higher.Impact is greater for men and for those with a secondary school diploma.7-18% of participants did not finish programmes, mainly because they accepted job offers before doing so.
In the counterfactual analysis for the 2015 call, the two groups compared have some differences which could affect the results. In general, the administrative data used could underestimate actual effects, as they do not cover self-employment.
Strengthen the appeal of the measure, e.g. through recruitment hiring incentives (not necessarily financial ones).Involve more local organisations potentially interested in hosting a participant for phase 2 of the project.Assess in advance the actual interest of hosting organisations in hiring participants after the completion of the experience.Provide follow-up measures for participants not hired.
Study of the guarantee fund for SMEs in response to the COVID pandemic
Evaluation of the impact of the Warminsko-Mazurskie OP, 2014-2020, on the improvement of the natural environment and the rational use of resources
Ocena wpływu RPO WiM 2014-2020 na poprawę środowiska przyrodniczego i racjonalne wykorzystanie zasobów województwa warmińsko-mazurskiego
The evaluation investigates the impact of the OP on the environment and the use of natural resources.
To assess the existing and potential impact of the support for the preservation and protection of the natural environment, resource efficiency and adaptation to climate change.
Analysis of monitoring and statistical data.23 interviews with the entities responsible for implementing different priorities.Survey of 135 self-governments.Survey of 60 beneficiaries.Survey of 540 inhabitants of the region.Focus group interview with experts and representatives of institutions.Counterfactual analysis.Discussion panel.Benchmarking.Case studies.Interviews with 10 companies from the tourism and the agro-food industries (Dyad method).
The OP helped increase waste recycling capacity, build waste management plants, and improve selective waste collection.The measures had a significant impact on water and sewage infrastructure. Many households benefited from improved wastewater treatment and there was a significant increase in the use of sewage systems in the areas supported.Projects protected thousands of hectares of natural habitats and improved biodiversity, and the total area of nature reserves exceeded the plans.The OP helped improve the water conditions of an important regional lake, because of the decreased industrial and municipal sewage volumes discharged into the waters.The OP enabled the purchase of fire engines and the implementation of a threat monitoring and early warning system, which increases the protection against forest fire and floods.
Continue supporting environmental protection and focus on EU directives and Poland's accession obligations.Modernise household septic tanks and drainage stations in rural areas, as building sewage networks is difficult in those areas.Improve the management of wastewater in non-urbanised areas.Provide comprehensive support for asbestos waste removal, including costs related to transport and storage.Focus future investments on adapting to climate change and protecting freshwater resources.Promote sustainable development initiatives and raise the general public awareness on environmental protection.
Evaluation of the efficiency and effectiveness of the Thessalia OP, 2014-2020
1η Αξιολόγηση Αποδοτικότητας & Αποτελεσματικότητας του Περιφερειακού Επιχειρησιακού Προγράμματος Θεσσαλίας 2014- 2020
The evaluation examines progress of the OP, up to end-2017.
To assess progress, effectiveness and efficiency of the OP.
Desk analysis of OP documentation.Analysis of monitoring data.
The intervention logic is relevant with respect to the priorities set by regional strategies.All PAs show a low level of commitments and expenditure, because of implementation delays, which mean limited achievement of the Performance framework milestones.Measures under PA2 and PA4 are the most effective.Efficiency, in terms of costs per unit of output, is lower than expected.
The set of result indicators need to be updated to better reflect the OP strategy.
Evaluation of impact of Structural Funds on transport systems in 2007-2013
2007-2013 m. laikotarpio ES struktūrinių fondų poveikio transportui vertinimas
OP Promoting Cohesion 2007-2013; OP Economic Growth 2007-2013
This is an ex-post evaluation of the measures to improve the Lithuanian transport system co-financed by the Structural Funds 2007-2013. The focus is on the projects funded by the National Programme for the Promotion of Cohesion (3rd Priority "Environment and Sustainable Development") and the National Programme for Economic Growth (Priorities 4 "Essential Economic Infrastructure" and 5 "Trans-European Transport Network Expansion").
To assess the coherence between the interventions and the objectives set by the National Strategy in the light of the main problems of the Lithuanian transport network;to evaluate the impact of 2007‒2013 EU funding on the different modes of transport;to assess the efficiency and effectiveness of the projects implemented;to evaluate the broader effects on society and the economy.
Desk analysis of the main strategy and programming documents;statistical analysis, causal analysis, expert assessment;interviews with the main institutions involved in the management and implementation of measures;online survey of project beneficiaries;econometric analysis.
The evaluation highlights:The appropriateness of the funding measures given the objectives of the National Strategy and the problems of the transport network in Lithuania;the positive effects on the environment, transport safety and socioeconomic cohesion of the projects for all modes of transport;the adequacy of the funding schemes, procedures and management systems for all the measures analysed;the positive long-term effects in terms of increased social cohesion, increased labour mobility and more favourable conditions for business development.the estimate that each 1 Euro added EUR 1.86 to GDP by 2013 and will add EUR 2.5 by 2020.
The evaluation recognizes that the effects of transport interventions usually take 4-5 years to materialise fully, i.e. long after the evaluation was completed.
Inter-institutional cooperation should be improved;the Transport Investment Directorate should be directly involved in planning and setting priorities;a more detailed analysis of the current state of inland waterways should be undertaken in order to define an Action Plan for the further development of water transport;the long-term planning capacity of the institutions involved in the management and implementation of funding measures and projects should be improved.
Meta-analysis - Summary of main findings from evaluations in 2007-2013 and 2014-2020
Jana Drlíková, M.A., Mgr. Tomáš Novotný, RNDr. Jana Bystřická
Evaluation Unit of Ministry for Regional Development
The report summarises the main findings from evaluations on the ERDF, ESF and CF carried out on the present and previous period in the Czech Republic.
To synthesise the findings of the evaluations.
The EU funds have contributed to the growth of GDP in the Czech Republic and regions and the improvement in the quality of life.The largest short-term impact is from investment in transport infrastructure, the largest long-term one from investment in human resources.EU funds have helped to increase RTDI and enable research organizations to participate in international projects.They have also enabled energy intensity and negative environmental effects to be reduced.They have led to R&D moving towards excellence and improved the quality of research teams, as well as of regional education.The innovation performance of enterprises is increasing as is collaboration with research centres.Faster preparation of construction projects is hindered by legislative barriers, while digitalisation of applications of environmental projects has eased the administrative process.Cooperation with partners has improved and the management of EU funds is less affected by irregularities and corruption than other areas of public administration.
Give more stability and continuity to the implementation of EU funds.Simplify and clarify methods and procedures and focus on the transfer of good practice.Create an enabling and supportive environment for a greater role for financial instruments and simplify and consolidate them.Target the calls as regards R&D and education better.
Evaluation of the implementation of PA 3 "Reduction of CO2 Emissions" under the Niedersachsen ERDF/ESF OP, 2014-2020 - phase 1
Evaluationsbericht zur Prioritätsachse 3 Reduzierung der CO2-Emissionen Bericht zu Phase 1 – Durchführungsanalyse und Wirkungsmodelle
This first evaluation examines the implementation of the measures to reduce CO2 emissions over the period up to the end of 2019.
To assess the implementation procedures and the reasons for slow progress and other issues and to define a preliminary theory of change for a subsequent impact evaluation.
Desk analysis.Analysis of monitoring data.Interviews with all responsible specialist units and representatives of the relevant departments.
Up to December 2019, 253 operations were approved, 62% of the budget, though payments amount only to 14%, and progress in implementation is slow.The main obstacles to implementation are: the limited capacity of new beneficiaries; delays in applications for funding, selection procedures, preparation of guidelines and adaptation of procedures; and bottlenecks in external auditing.Lack of information on the implementation of financing instruments prevents the MA from carrying out realistic financial planning to avoid the risk of losing funding.
The evaluation includes an overview of possible implementation problems but not a more detailed analysis of implementation procedures.
Define different strategies in the financial programming for the different categories of region.Define a suitable combination of established instruments before introducing new ones.Experiment with pilot projects in funding areas for which there is no experience in order to learn.Prepare the basis for producing guidelines in time.In the case of new funding instruments and programmes with few likely projects, the application process should be started with the lowest possible threshold.
Evaluation of the training system for employees under the Technical assistance ERDF OP, 2007-2013
Evaluace systému vzdělávání zaměstnanců realizujících NSRR v programovém období 2007-2013
Institut evaluací a sociálních analýz (INESAN)
Institut evaluací a sociálních analýz (INESAN), HeřmaNova 22, Praha 7, tel:00420 220 190 597, info@inesan.eu, www.inesan.eu
The evaluation examines the “System of Education” training programme under the Technical Assistance OP 2007-2013 which is aimed at staff of Managing Authorities and implementing bodies.
The evaluation is aimed at assessing the content and outcomes of the training courses organised as part of the System of Education between 2009–2013 and making recommendations for the 2014-2020 period.
The evaluation uses both qualitative and quantitative methods:Desk research;analysis of OP documentation;CAWI survey of participants in training programmes (717 responses) and non-participants (78 responses);in-depth interviews with participants (8), coordinators and supervisors (25);The data used are: primary data from interviews and a survey and secondary data from OP documentations.
Content and outcome of the courses: 8,982 people attended the 622 courses;around 80% of participants viewed the courses as being useful and applicable to their work.Performance of System of Education: The courses increased the knowledge and skills of participants and were in line with their needs. 31% of participants reported an increase in their efficiency and 16% a change in their pattern of working;but the effect on organisation performance was reported to be low.Future educational need: An emphasis should be put on vocational training (especially on public procurement);the soft-skills courses should continue to be provided (on assertiveness, presentation and communication skills and time and stress management).Evidence for new system set-up: The registration system is reported to be lengthy and administratively demanding by participants and coordinators;more lecturers from public administration staff should be involved.
Content and course structure: multiple teaching methods should be used as well as modern methods (e.g. e-learning);system of education: what the courses offer needs to be clearer and the target group better defined (e.g. in terms of the level of expertise);evaluation system: training on data needs to be expanded and indicators of the performance of courses need to be better defined and monitored;organisation of the system: the system of registration needs to be simplified.
Evaluation of training and development services
Ocena zapotrzebowania na realizację usług rozwojowych w odniesieniu do potrzeb przedsiębiorców z województwa lubelskiego
Ul. Bitwy Warszawskiej 1920 r.7, 02-366 Warszawa. Tel. +48 22 492 71 04, www.wygpsdb.pl
The evaluation is concerned with the human development projects co-financed by the Lubelskie Regional Operational Programme 2007-2013 (LROP 2007-2013).
To assess the experience of employers as regards employee training.To identify the views of business on support being provided according to the characteristics of the enterprise (e.g. size and specialisation).To identify the sectors targeted for support.
The evaluation adopted qualitative and quantitative methods:Desk research.Analysis of relevant documents.In-depth interviews (17) with representatives of main institutions, employer organisations, training and consulting services.Focus groups (3) with representatives of enterprises.CATI survey (380) of entrepreneurs of the Lublin province.CAWI survey (320) of entrepreneurs operating in smart specialisation area.The data used are: primary data from interviews, surveys and focus groups; secondary data from official documents (including the partnership agreement, OP priorities and progress reports).
On the past experience of firms:In the two years before the survey, 48% of SMEs (32% in the smart specialisation area) took no development initiative. Medium-sized enterprises are those most willing to invest in human capital, especially in middle and lower level employees.Funds allocated were on average less than EUR 1,200 per enterprise. SMEs in the smart specialisation area invested more.On employers’ support of employee competencies:63% of the enterprises had undertaken activities to improve skills or planned to do so, but only 22% were certain to carry them out.On average, only three people were planned to be covered by training activities.The ‘bio-economy’, ‘medicine and health’ and ‘informatics and automation’ were the smart specialisation sectors most covered.
Emphasis should be placed on making employers aware of the benefits of appropriately targeted development of skills and qualifications.Awareness should be raised among employers of personal funding schemes to manage competencies and qualifications of employees.Additional criteria should be identified to enable the extent to which smart specialisation is the principal area of business for an SME to be verified.
Evaluation of the implementation of the principle of sustainable development under the Infrastructure and Environment OP 2007-2013 and 2014-2020, as well as the European Funds for Infrastructure, Climate, Environment programme, 2021-2027, in Poland
Ewaluacja sposobu realizacji zasady zrównoważonego rozwoju: badanie efektów programów POIiŚ i FEnIKS
OP Infrastructure and Environment 2007-2013; OP Infrastructure and Environment 2014-2020; Infrastructure, Climate, Environment PL CF/ERDF 2021-2027
Michał Wolański, Dominik Makurat, Paulina Kozłowska, Beata Paczek, Marcin Pinkosz, Julia Widłak, Michał Babicki, Mirosław Czerliński, Wojciech Szłapacki, Artur Dziubiński
The evaluation examines the effects of the implementation of the principle of sustainable development in projects under the OPs Infrastructure and Environment 2007-2013, 2014-2020 and the European Funds for Infrastructure, Climate, Environment 2021-2027 projects.
To assess how the programmes’ interventions contribute to the implementation of sustainable development goals.
Analysis of monitoring data and statistical office data.Econometric analysis (CIE based on the different degrees of policy exposure).Theory-based approach.Interviews with implementing institutions.18 case studies.4 expert panels.
Between 2007 and 2023, the share of renewable energy in electricity production in Poland increased more than twelvefold.Financing infrastructure projects in energy, transport and the environment through European funds is a key driver of sustainable development in Poland.The intervention also had visible economic effects: an additional one million euros of European funding increased industrial output per capita by an average of EUR 278.The counterfactual analysis found that the most successful projects combined various aspects of sustainable development, responding comprehensively to local needs. Synergy effects are particularly evident in projects that simultaneously address infrastructure, social and environmental issues. The main weakness remains in the areas of education and public awareness of sustainable development, along with the rising investment costs. Limited pro-environmental attitudes among residents mean that the positive effects of infrastructure projects are not fully used, which limits their sustainability and impact.
Increase the flexibility of financing rules and integrate hard and soft measures.Strengthen the flow of knowledge and communication in local governments in the field of sustainable development.Simplify the formal requirements imposed on beneficiaries in relation to sustainable development principles.
Evaluation of the coordination between the Molise ERDF/ESF OP, 2014-2020, and other funds
Servizio di valutazione del modello di sinergia/complementarietà implementato e correlato riesame della demarcazione tra fondi enunciata con riferimento al FEASR
The evaluation examines coordination between the ERDF and ESF and other European funds, in Molise.
To check whether the demarcation principle between funds was effective and the extent to which it was implemented, and to identify possible improvements for the next programming periods.
Analysis of relevant documents.Analysis of monitoring data.Survey of relevant stakeholders (18 responses out of 104).Desk comparative analysis.
In policy areas where more than one fund operates, the demarcation principle has worked satisfactorily, though cooperation between organisations was not fully effective.Overall, the EAFRD does not seem fully coordinated with the other two funds, mainly because of the different rules and implementation times of the measures, and limited administrative capacity.Respondents are reasonably satisfied with the coordination between EU funds, though they believe that synergy with the EAFRD is more spontaneous than planned.Other Italian regions (such as Emilia Romagna or Friuli Venezia Giulia) have established well-defined bodies, procedures and means for managing coordination.
Involve territorial actors more and define clearly tasks and roles.Simplify the demarcation rules.Introduce successful elements from other experience.Strengthen administrative capacity.Develop the ability of the inter-fund technical table to activate "technical groups".Improve the synergy between Funds, in the next programming period, through a shared construction of programmes, a clearer identification of demarcation principles and the involvement of the operational level of the administration.
Evaluation of ‘Basic education with work experience’ programme in Nordrhein-Westfalen in 2014-2020
Wissenschaftliche Begleitung/Evaluation der ESF-geförderten Maßnahmen der lebens- und erwerbsweltbe-zogenen Weiterbildung in Einrichtungen der Weiterbildung, Fördergegenstand: „Grundbildung mit Erwerbswelterfahrung“
Büro für Berufliche Bildungsplanung, R.Klein und Partner GbR
This evaluation assesses the "Basic education with experience in gainful employment" ESF measures implemented in 2015 and 2016 in Nordrhein-Westfalen.
To assess the effectiveness and impact of ESF-funded basic education programmes and the consequences for those in the target group who did not participate in the measures as well as to make recommendations for improvement.
Interviews;focus Groups;survey;qualitative control group survey;case studies.
The beneficiaries of the ESF courses are largely heterogeneous, and not all the desired target groups were reached by the interventions;social and cultural orientation proved to be crucial for implementation of the courses which targeted people with a migrant or refugee background;cooperation between educational institutions and municipal job centres, the Employment Agency, the Social welfare office and other regional structures has been effective in improving the access of participants;sustainable work and vocational guidance enabled participants to develop their own career aspirations;traineeships supported participants in their career choices and enabled learning in the workplace;longer working experiences proved to be more effective than short work placements;the best results were achieved when strategies where developed with the participants.
To take greater account of the financial situation and the availability of professional staff in future programmes;to ensure adequate flexibility of the programme;to ensure that the acquisition of experience and the skills needed to face life situations will be included in future ESF courses.
Evaluation of systemic projects on education funded by the Research, Development and Education OP, 2014-2020 in the Czech Republic – 3rd Interim Report
„Evaluace systémových a koncepčních projektů z výzev PO 3 OP VVV“ - Evaluační okruh A 3. Průběžná zpráva
Evaluation Advisory CE, Naviga 4, https://evaluace.eu/ - Naviga 4, www.naviga4.cz
This is the third interim evaluation of Systemic projects on education financed under PAs of the OP which are aimed at promoting equal access to good quality education in kindergartens and primary and secondary schools.
To assess the awareness that target groups have of Klima Action, strategic planning in schools and education projects as well as methodology used in internal project assessment.
Desk analysis of Local and Regional Action Plans and monitoring reports.Survey of school management, teachers, founders (2510 responses) and projects managers.Interviews with project managers.
Around half of the teachers and 60% of management staff in kindergartens and elementary schools are aware of KLIMA action.86% and 93% of representatives of kindergartens and elementary schools report that objectives have been met totally or partly, though the large number of children in classrooms is an obstacle to an individualised approach.Preparation of the Activity Plan (PA) and the School Action Plans (SAP) is seen mainly as a formal obligation to obtain grants.There has been an increase in awareness of the implementation of the SRP project – “Strategic management and planning in schools and territories”: over the past two years though 50% of kindergarten and elementary school teachers are unaware of it.Self-assessment for project teams is perceived positively, but it should be linked more to specific project milestones.
No substantial limitations are indicated.
Provide more information on outputs and feedback on self-assessment reports.Avoid duplication of information in the self-assessment of projects and in the PRINCE2 management system.Focus on raising awareness among school founders.
Centre OP, 2014-2020 - 6-month follow-up survey of participants in ESF/YEI vocational training 2018
Enquête de suivi à 6 mois des participants aux formations profesionnelles cofinancées par le FSE et l'IEJ
The evaluation assesses the overall situation of participants in vocational training financed by the YEI and ESF in France 6months after completion.
To assess the situation of participants 6 months after completing training and the quality of jobs, where applicable and to produce indicators.
In the post-training period, the number of participants becoming self-employed is relatively small.After participation, women are overall less likely to be employed than men.Following completion of training, women are more likely to participate in additional training.In 2017, 17% of trainees were attending another training programme or were employed under an apprenticeship or a traineeship 6 months after completing their training.
Evaluation of the 2014-2020 communication strategy
Ολοκληρωμένο Προσχέδιο Μελέτης Πρώτης Αξιολόγησης της Στρατηγικής Επικοινωνίας 2014-2020
OP Employment, Human Capital and Social Cohesion 2014-2020; OP Competitiveness and sustainable development 2014-2020
Aristotelous 11-15, P.O. Box 10432, Athens, tel:+30 210 5201460, lkn@lkn.gr
The evaluation focuses on the 2014-2020 joint communication strategy of the “sustainable development and competitiveness” and “employment, human capital and social cohesion” OPs. It describes the communication measures and provides an overview of the initial implementation stage of the strategy by measuring the performance of result and output indicators. Results from two surveys of the general public and enterprises are also described and recommendations for improving the strategy are made.
To undertake an evaluation of the implementation and effectiveness of the 2014-2020 joint communication plan.
Analysis of OP monitoring data and strategic documents; structured questionnaires through telephone interviews (CATI); analysis of previous evaluations.
The 2014-2020 communication strategy adopts a comprehensive set of information and communication tools, suitable for examining the three target groups (beneficiaries, potential beneficiaries and the general public), and incorporates all of the mandatory provisions of the regulations.Overall the strategy’s output indicators show very good performance especially as regards electronic information aspects, the targets for which have already been exceeded. Internet-based communication emerges as the principal communication method.The implementation of the strategy is highly efficient with lower than expected costs.The surveys show that EU funds are mostly focused on environmental protection, followed by improving education and lifelong learning and supporting research and innovation.Four of the six result indicators have already achieved their targets.
The report recommends updating the system of monitoring indicators as regards electronic information media and reviewing the budget for the communications strategy for the remaining years.
Evaluation of PA 15 “SME Initiative” of the Regional OP, 2014-2020, in Romania
Evaluarea POR 2014-2020. Axa Prioritara 15: Initiativa pentru IMM
Giovanni Familliari, Michele Alessandrini
SC ACZ Consulting srl, office@aczconsulting.ro; t33 srl, info@t33.it
The evaluation examines the impact of the uncapped guarantee instrument, “SME Initiative”, which aimed at facilitating access to finance for SMEs. It covers the implementation period between 2014 and 2020.
To examine the impact of measures to facilitate access to finance of SMEs on productivity, innovation and the capacity to grow in national and international markets.
8 interviews with: financial intermediaries; the European Investment Fund (EIF); the Competition Council; the Ministry of Research, Innovation and Digitalisation; the Ministry of Finance; and the Head of the MA of the OP.Analysis of monitoring data and official statistics.Analysis of administrative data (database provided by the Business List and EIF database).Survey of final beneficiaries (43 valid responses, response rate not indicated, 15% margin of error).Counterfactual analysis.Cost-effectiveness analysis.1 focus group with six representatives of the Regional Development Agency (RDA), evaluation experts and an expert panel.
The measure providing guarantee for SMEs helped improve access to finance, especially for micro-enterprises. The effects on turnover are sustainable and contributes in the long term to increasing labour productivity.Beneficiary needs are generally covered by a combination of different financial instruments; this measure appears to be more successful than grants in increasing turnover and productivity.The use of financial instruments allows publicly funded programmes to reach companies and sectors that could contribute to increase competitiveness in the country or region.Compared to other guarantee initiatives in Romania and other normal standard transactions, the SME Initiative is more manageable for financial intermediaries in terms of monitoring and reporting activities.Financial intermediaries are more willing to participate in EU schemes – because of the SME initiative - but they need capacity building in order to participate in the design and implementation of future financial instruments.
The opinion survey of final beneficiaries collected fewer responses than expected, despite numerous attempts to contact businesses. The duration of the evaluation was reduced from 11 months to 3 months, which limited the breadth of the counterfactual approach and the ability to contact businesses.
Consider the possibility of combining investment support or working capital with support for human resources (notably for financial intermediaries) in the future design of financial instruments.Improve lending conditions, shorten time to lend and include more information and communication activities for final beneficiaries.
Survey of primary school support under the Research, Development and Education OP, 2014-2020
Vyhodnocení dotazníku k výsledkové evaluaci OP VVV na základních školách
Evaluation unit, Ministry of Education, Youth and Sports
The report presents the results of a survey of headmasters and teachers to evaluate the OP interventions – Templates for kindergartens and primary schools I.
To report the results of a survey of nursery and primary school teachers on the templates financed by the OP.
Survey of headmasters and teachers (33% response rate).
According to the headmasters surveyed, tutoring is the main benefit of the templates, as these improved children’s results.Templates made the teaching more systematic and individualised and helped communication with parents.Reading clubs increased children’s interest in literature and reading, especially those from socially disadvantaged backgrounds.Activities on mathematical literacy increased children’s interest in the subject.The courses helped schools to know more about working with pupils with special education needs.Support also helped cooperation between schools.The majority of teachers are generally in favour of including all types of children in regular classes.A positive school climate, guidance on mutual tolerance and fewer children in the classroom are reported to be the most effective means of reducing early school leaving among socially disadvantaged children.
Evaluation of process settings in the Technical assistance ERDF OP, 2014-2020
This study assesses a number of aspects relating to the procedures followed in the Technical Assistance OP for the Czech Republic in 2014-2020.
To assess processes and potential issues in respect of the OP and cooperation between the Managing Authority and the Department of Project Management.
Desk Research.CAWI Survey.Group Interviews.
Overall, the implementation system is considered efficient and clear by the staff of the OPTA MA. A few beneficiaries, not directly involved in the implementation, stated that the system is burdensome particularly as regards the preparation of projects application and payments.Beneficiaries and staff of the OPTA MA consider some procedures included in the information system too complex.The average time from submission of the project application to its approval is 31 days. The average time for the approval of first-time payment requests was 63 days in 2016 and 48 days in 2017.The cooperation established between the MA and the Department of Project Management is proving useful.
There is a need to:Streamline procedures.Establish specific dates for the submission of reports.
Evaluation of the goals and objectives achieved in respect of the 2007-2013 OPs
Kiekybinis ir kokybinis 2007–2013 m. veiksmų programų pasiektų tikslų ir uždavinių vertinimas
OP Development of Human Resources 2007-2013; OP Promoting Cohesion 2007-2013; OP Economic Growth 2007-2013
Evaluation of the objectives achieved in the 2007-2013 OPs.
The evaluation summarises the objectives achieved by the four OPs co-financed by the ERDF, Cohesion Fund and ESF (the Human Resources Development OP, Economic Growth OP, Promotion of Cohesion OP and Technical Assistance OP) and the impact of the investment supported on the economy and quality of life over the 2007-2013 period.
The evaluation is a meta-evaluation aimed at presenting the aggregate findings from several different evaluations.
Impact of investment supported by the EU funds on the economy and quality of life:Each Euro invested generated an estimated EUR 1.38 of nominal GDP in return;labour productivity was 2% higher than it otherwise would have been, household consumption 3.2% higher and exports 1% higher;almost 266,000 jobs are estimated to have been created or preserved by projects funded by EU support (slightly less than half were preserved for over a year after projects were completed);the quality of life improved in many municipalities in 2014 compared to 2007, partly through the renovation of multi-apartment blocks, social housing and investment in the upgrading and development of health, education, social service, transport, tourist, water supply and wastewater management infrastructure.Goals achieved by the Human Resources Development OP:235,000 people participated in training over the period 2007-15;loans issued by the Entrepreneurship Promotion Fund to help start up small businesses created more than 3,800 new jobs;over 245,000 unemployed participated in active labour market measures;2,600 of the socially excluded participated in activities aimed at increasing social inclusion;22,000 employees participated in continuing training.Goals achieved by the Economic Growth OP:by the end of 2015, 592 jobs were created in research, 145 contracts were signed with SMEs and 228 laboratories were constructed or upgraded;the number of researchers in enterprises increased by 2,574 between 2007 and 2014;the number of applications filed with the European Patent Office per million population increased from 2.6 in 2009 to 16.6 in 2014;intervention projects created 215 new technology enterprises;EU investment accounted for 69% of the Lithuanian venture capital market. Loans prevented many enterprises with temporary liquidity difficulties from going bankrupt;10 industrial parks have a attracted 34 foreign investors;the share of public services accessible on the Internet increased from 37% to 95% between 2007 and 2013, the share of population with broadband access increased from 48% to 67% between 2009 and 2015;12% of district heating networks were upgraded; 138 km of gas pipelines were built;737 km of gravel roads were paved; 1,234 km of roads were built or reconstructed ;Goals achieved by the Promotion of Cohesion OP:368 multi-apartment blocks were modernised, improving living conditions for over 11,800 people;271 social housing units were built and 291 units were modernised;public infrastructure was upgraded in 241 residential areas in 52 of the 60 municipalities (mostly in Utena and Tauragė);EU Structural Funds accounted for 25% of all material investment in Lithuania between 2009 and 2013;to safeguard biodiversity and landscape: 22 visitor centres were built, 16 information systems were designed; 5 nature schools were opened and around 460 hectares of water were cleaned up;all old landfills were closed; access to main drainage was provided to around 153,000 people.
No implications for policy are included in the evaluation.
Evaluation of support for vocational guidance of students under the Baden-Württemberg ESF OP, 2014-2020
Evaluationsbericht zum Spezifischen Ziel C.4.1. (KOOBO)
ISG Institut für Sozialforschung und Gesellschaftspolitik GmbH, Weinsbergstr. 190, 50825 Köln, Email: info@isg-institut.de, www.isg-institut.de
The evaluation examines the measures providing vocational guidance to students in upper secondary schools over the period 2016-2020 under IP 10iv of the OP.
To assess the results of vocational guidance measures in the short and long term and their implementation over the period 2016-2020.
Desk analysis of literature and project documentation.Monitoring data.Interview with 3 Ministerial authorities.Survey of students before (673 respondents) and after (395 respondents) the measure.Survey of students in the last class of secondary schools 6 months after participation in the measure (158 respondents, 16% response rate).4 case studies (14 interviews with school management, teachers and project implementers).
The flexible approach to define the content of projects and the organisational aspects are positively assessed by students, teachers and school managers.A total of 19,797 students were reached by projects over the period, 58% of them male.77% of students participating obtained a qualification afterwards.The projects focused on how to approach the labour market and were significant in terms of individual career guidance.The older the participants, the more positive the changes made.Immediately after participation the share of students willing to continue their present education pathways declined and the share undertaking additional vocational training (the majority) increased.6 months after participation, 34% of students had applied for an apprenticeship, 30% had undertaken a traineeship, 30% had enrolled in a higher education programme, 22% had started training, 16% had applied for a job, 14% had started a job, and 9% had started tertiary education.
The limited number of students surveyed and the low response rate to the survey of students 6 months after participation prevent results being generalised.
Promote a greater integration of the measures in vocational education and training and more intensive involvement of education providers through a more active role of the Regional Directorate of the Federal Employment Agency.Reinforce traineeships, which proved to be an effective means of increasing career choice skills.Give priority to students in their final school year and to those with the least occupational skills.Focus more on sectors with a shortage of skilled workers.
Impact evaluation of Cohesion policy 2014-2020 on the development and functioning of social infrastructure in Poland
Wpływ polityki spójności 2014-2020 na rozwój i funkcjonowanie infrastruktury społecznej
OP Infrastructure and Environment 2014-2020; all regional OPs 2014-2020; Interreg V-A-Poland-Slovakia 2014-2020; Interreg-V-A-Poland-Denmark-Germany-Lithuania-Sweden (South Baltic) 2014-2020; Interreg V-A-Poland-Germany/Saxony 2014-2020
Jan Szczucki, Ewa Kusideł, Marzena Sochańska-Kawiecka, Justyna Witkowska, Maciej Gajewski, Robert Kubajek, Zuzanna Kołakowska-Seroczyńska, Edyta Makowska-Belta
Policy & Action Group Uniconsult, pag-uniconsult.pl, biuro@pag-uniconsult.pl; Badania Społeczne Marzena Sochańska-Kawiecka, badaniaspoleczne@autograf.pl
The evaluation examines the impact of the ERDF on the development and functioning of social infrastructure.
To assess the effects of support for investment in infrastructure in education and training, culture, healthcare, tourism, recreation and sports, social assistance and integration, and childcare and to make recommendations for 2021-2027.
Analysis of monitoring and statistical data.Documentation analysis.Econometric analysis.42 interviews with MA and beneficiaries.Surveys of 4089 beneficiaries and 900 residents of three Polish regions.10 case studies of projects.10 workshops with stakeholders and one on recommendations.
At the end of 2022, support for social infrastructure had involved almost 10,000 projects with a total value of PLN 43 billion, of which around 30% had gone to healthcare, partly because of combatting Covid.Projects have been complementary with 2007-13 ones, but ERDF support has been less complementary with that from the ESF.Projects are considered to be sustainable, with 81% of respondents reporting that the infrastructure will continue to be used in the same way in the future and 13% that new uses have been identified. But maintenance has become a major problem because if high inflation, especially of energy costs.Projects have contributed to improving access for people with disabilities.Econometric analysis was unable to confirm the positive impact of projects, perhaps because it was carried out too soon, but the largest improvements in accessibility were recorded in regions with the lowest initial levels.
28% of projects supported had not been completed at the time of the evaluation. The latest data available related to 2021.
Promote projects that include a coherent long-term management plan and those that with minor adaption could serve new purposes.Promote projects with soft measures combined with infrastructure.Involve stakeholders in project preparation.
Evaluation of measures implemented in Salzburg under the Employment Austria OP, 2014-2020
Evaluierung der ESF-Umsetzung für Salzburg
Thomas Lankmayer, Manuela Hiesmair, Karl Niederberger
The evaluation examines the results of measures for the employment integration of people disadvantaged, through innovative pilot projects and the provision of support services, over the period up to March-2021.
To assess the effects of the 6 projects for the integration into employment of disadvantaged groups up to March 2021 and their contribution to the promotion of equality between women and men, equal opportunities and access to the labour market of people with disabilities.
Administrative data.Monitoring data.Synthesis of the evaluations of the six projects.Literature review.Interviews and focus groups with projects’ staff.
52% of the 867 participants up to March 2021 were women, 87% of the total completed the activities regularly and the average duration of projects was 240 days.The ex post participation rate of beneficiaries in the labour market is higher than before the measures.Project helped to improve “soft outcomes”, such as health and attitudes to work, which favour the social integration of participants.Success factors are “identification” of participants with the objectives of measures and individualised support to take account of the heterogeneity of the target group.Childcare services helped to eliminate barriers to women's participation in social inclusion projects.
Data from ZWIMOS (a database of the Ministry of Social Affairs for ESF projects) and AMS (Labour market service) are not always comparable. Moreover, the analysis of soft outcomes is only partial.
Improve the analysis of soft outcomes of projects.Provide more targeted activities for groups with few opportunities to be re-employed.Consider implementing projects outside the Salzburg urban area.Strengthen the provision of childcare services.Develop a uniform data collection system.
Evaluation Report 2019 on the Lombardia ERDF OP, 2014-2020
Fondazione Giacomo Brodolini, MBS Consulting, IZI – Metodi, analisi e valutazioni economiche
Fondazione Giacomo Brodolini, MBS Consulting, IZI – Metodi, analisi e valutazioni economiche, www.fondazionebrodolini.it
The evaluation examines the implementation of the Lombardia ERDF OP 2014-2020.
To assess the level of implementation of the OP and the progress towards the achievement of targets set.
Documentation analysis;analysis of monitoring data and regional statistical data;interviews with the MA and main stakeholders;case studies;focus group .
By end 2018, the OP implementation was generally satisfactory. Some delays are evident for Priority Axis (PA) 2 (on high-speed broadband), mainly due to the complexity of involving a large number of local bodies and the nature of infrastructure investment and PA5, on urban regeneration due to the nature of interventions and unexpected issues in implementation. PA1 measures are coherent with the Smart Specialisation strategy.There is strong cooperation between SMEs, large enterprises and research centres.The involvement of SMEs with a less favourable attitude to innovation is positive as they probably would not have invested without support.PA4 measures (shift towards a low carbon economy), mainly focused on energy and sustainable mobility, are in line with the regional strategy of sustainable development.The MA has worked towards resolving implementation issues highlighted in previous reports.Calls are assessed as being innovative (e.g. use of financial instruments and public-private partnership) and of high quality (in terms of selection criteria and clarity).
Introduce procedures and strategies aimed at simplifying implementation;maintain an high level of internal and external communication;support those beneficiaries (mainly public bodies) which participate in public procurement;maintain dialogue with stakeholders.
Evaluation of support for business creation under the Friuli Venezia Giulia ERDF and ESF OP, 2014-2020
V Rapporto tematico di valutazione – Creazione e reti di imprese
ROP Friuli Venezia Giulia ESF 2014-2020; ROP Friuli Venezia Giulia ERDF 2014-2020
Luca Rossi; Andrea Ciffolilli; Marco Pompili; Carlo Miccadei; Vincenzo Angrisani
The evaluation examines the results obtained by measures for business creation under TO1, TO3 and TO8 of the OP.
To assess the main features of projects financed and characteristics of beneficiaries, the added value of EU funding, the performance of companies supported and the effects of Covid-19 on them.
Official statistics.Monitoring data.Administrative data (business registers).Survey of beneficiaries and non-beneficiaries of measures under TO1 (response rate 15%) and of beneficiaries of measures under TO3 (45%).Interviews with implementing bodies.Survey of participants of measures under TO8 (around 34% response rate).
IP 1.2:Most of the projects financed by TO1 are in advanced manufacturing systems, advanced materials and nanoelectronics.Lack of selectivity has led to non-innovative projects being financed and deadweight being high.Preliminary results show that the turnover of companies supported remained unchanged between 2018 and 2019 while employment increased by 20 pp; Covid-19 has led to a contraction in sales and investment.IP 3.1:Newly-created companies are mainly in retailing and tourism; the presence of women and young people is significant and deadweight seems low.Most of the companies supported have grown, but those in cultural and creative activities declined because of Covid-19.TO8:Participants in measures promoting entrepreneurship are satisfied with the support received, in terms of improving their enterprise culture, knowledge and technical competences and deadweight seems low.87% of newly created enterprises were still active after one year and, on average, performed well in terms of employment and turnover growth, though Covid-19 led to a reduction in investment and turnover in over 40% of them.
The effects observed of support are preliminary, as many projects only started in 2019.
Adopt a simpler application process and more effective selection procedures and reduce the administrative burden on beneficiaries.Ensure better integration between different ESF measures and between these and ERDF support.Include career guidance among ESF measures to better match participants with the training provided.
Evaluation of the effects of measures under PA6 "Environmental protection and sustainability of resources" of the Competitiveness and Cohesion OP, 2014-2020, in Croatia
Vrednovanje učinka Operativnog programa Konkurentnost i kohezija 2014.-202. Prioritetna os 6: Zaštita okoliša i održivost resursa (Grupa5)
Equinox consulting kft, info@equinox.hu;Urbanex, ivana.katuric@urbanex.hr
The evaluation examines PA6 ‘Environmental protection and sustainability of resources’ of the OP. Data are updated at 31/12/2021.
To assess the achievement of short-term and long-term objectives as well as the impact of measures to improve environmental protection and sustainable use of resources.
- Design and testing of the Theory of Change (ToC).- Analysis of relevant strategic document.- Interviews.- Analysis of monitoring and statistical data.- Case study.
- Overall, there is a good level of achievement of output and result indicators.- Cooperation and communication between stakeholders and managing and control bodies contributed to the effectiveness of the measures. On the contrary, insufficient human competences were one of the major obstacles for effective implementation. - Specific Objectives (SOs) 6c1 and 6c2: 15 cultural heritage development programmes were supported, 116 people were newly employed in the touristic sector, 126 infrastructures were supported in national and nature parks, and the number of visitors expected grew by 1.2 million people.- SOs 6i1 and 6e2: more than 5,200 m2 of brownfields and 32.39 hectares of land were rehabilitated, 22 landfills were closed and remediated, and the solid waste recycling capacity increased by 157,978 tonnes/year.- SOs 6ii1 and 6ii2: more than 242,000 additional people were served by improved water supply systems, and 63,000 by improved wastewater treatment systems.- SOs 6iii2: 5 plans for Invasive Alien Species management system were established, and 984 hectares of habitats were supported.
Three of the ten specific objectives do not have any completed projects by the cut-off date of the evaluation.
- Improve the description of indicators and set up a consistent way to collect and maintain data to facilitate the assessment of measures’ effects.- Reduce the duration of public procurement procedures.- Design the management and control system in a flexible way to make it adaptable to changes to projects’ budget and timing.
Evaluation of the implementation of the Climate Action and Sustainability Programme, 2021-2027, in Portugal
Operacionalização do Programa Sustentável 2030
Thematic Programme for Climate Action and Sustainability PL CF 2021-2027
Cristina Marta-Pedroso, Filipa Seiceira, Lia Laporta, Marco Henriques, Ramiro Neves, Tatiana Alves, Tiago Domingos
ATTHIS Consulting – Ambiente e Economia dos Recursos Naturais, https://atthisconsulting.pt/; PPLL Cosult, www.ppllconsult.pt
The report examines the implementation of the national programme for climate action and sustainability. Data used go up to January 2025.
To assess the management setup and intermediate bodies, the effectiveness of governance, progress towards targets, and performance of communication and visibility activities. To this end, the programme's functioning is examined across eight core dimensions, including administrative procedures and IT systems, the ability to mobilise quality demand, the speed and quality of project selection, and the adequacy of indicators and data systems.
Document analysis of the programme.Analysis of statistical and monitoring data.A first online survey of 17 beneficiaries covering 46 operations (excluding technical assistance).A second online survey of MA officers, with the aim to collect information on the application process, beneficiary capacity-building, support documentation, governance model, information system, and communication.Eight semi-structured individual or group interviews with representatives from the information system unit, operational units, managing bodies etc.Three focus groups, with 31 participants overall: one with the Climate Action and Sustainability Network (RACS, 6 participants), one with the Monitoring Committee (18 participants), and one thematic group on communication (7 participants).Case study on the two Intermediate Bodies (OI) of the programme - the Regional Directorate for Planning and Structural Funds (RAA) and the Regional Development Institute (RAM) - to assess how the delegated management model works in practice.Case studies covering six projects selected by the MA.A workshop linked to the case studies on projects, including beneficiaries and other stakeholders, to validate results and exchange learning lessons.
The programme design is broadly coherent, but implementation is lagging and there are risks of not achieving targets.By November 2024, 46 calls were opened (67% of allocation), while commitment was 28% and execution 1.7%.Main bottlenecks are authorisation steps, Court of Auditors clearance, procurement delays, and weak project maturity/planning at submission stage.The indicator framework is considered relevant, but data collection and validation are not yet fully consolidated in practice.Delegation to Intermediate Bodies is a strength for territorial proximity and implementation support, but staffing constraints and supervisory dependencies reduce speed of execution.Visibility of the programme has improved, but targets are not met, especially when it comes to engagement on digital platforms.
The indicators system is not fully consolidated, so data on projects might not be on time. Only three out of six authorities from the regional infrastructure offices participated in the workshop due to scheduling conflicts.
Simplify authorisation, procurement, and project maturity processes to eliminate bottlenecks and speed up execution.Improve the indicator framework and data validation systems for better performance tracking and timely adjustments.Increase staffing and empower intermediate bodies to reduce bottlenecks and accelerate decision-making.Improve digital engagement and communication targeting to raise the programme's visibility and reach.
Interim evaluation of the initial phase of ESF Specific Objective ‘Promoting sustainable employability of workers’
Tussenevaluatie Eerste Openstelling ESF-regeling Duurzame Inzetbaarheid
Bureau Bartels B.V., P.O. Box 318, 3800 AH Amersfoort, Tel: +31 33 4792020, email: info@brtls.nl, internet: www.bureaubartels.nl
El proceso RIS3 (Estrategias de investigación e innovacion para una especialización inteligente) en España
Dirección General de Fondos Europeos SG de Programación y Evaluación
This study examines the preparation and implementation of RIS3 in the Spanish regions, assessing the definition of each RIS3, its implementation status and its monitoring and evaluation.
To assess the implementation status of the various strategies and their monitoring and evaluation.
Questionnaire to the units responsible for RIS3;analysis of indicators.
Delays in RIS3 implementation have been caused by the increasing complexity of managing the ERDF, especially appointing Intermediate Bodies, launching new procedures, and defining eligibility criteria.Participatory governance for the development of RIS3 has led to new instruments and actions being proposed, which are limited by the existing regulatory framework, administrative procedures and the inertia of current systems.Fiscal consolidation resulted in fewer human resources at national and regional level, reducing the effectiveness in developing, implementing and monitoring the strategies.There is lack of a single approach to monitoring and evaluation of the RIS3s.
There is a need to:define monitoring indicators and plan evaluations of the RIS3;link the performance indicators to the activities financed in the RIS3;harmonise the indicators in the different RIS3, without losing their specific features.
Evaluation of measures to extend-hours of early childhood education under the Piemonte ESF OP, 2014-2020 - School year 2021/2022
Il sostegno al prolungamento dell’orario nei servizi educativi per la prima infanzia. Analisi di attuazione nell’anno educativo 2021/22
IRES Piemonte - https://www.ires.piemonte.it/ - ires@pec.irespiemonte.it
This evaluation analyses implementation of measures to extend hours in public early childhood education services (SPO) financed under the OP, 2014-2020. Data refer to school year 2021-2022.
To assess the implementation and management of the measure in the 2021-22 school year and highlight potential bottlenecks.
Documentation analysis.In-depth interviews with 13 municipalities and 20 professionals.Analysis of statistical and monitoring data.
26% of municipalities in the region received support, 50 out of 194.The highest participation rates were in the largest municipalities (over 20,000 inhabitants), while smaller ones (under 5,000 inhabitants) had the highest withdrawal rates.About 700 children benefitted from the service, 27% of those enrolled in the facilities of the municipalities supported.According to interviewees, critical issues are the length of selection procedures, lack of staff and inadequate monitoring in daycare centres.
Information on parents and their labour market history to estimate the impact of the measure on their employability is not available. The number of children benefitting from measures might be underestimated, since it comes from the daycare centre registers that are not adequately kept up to date.
Identify the list of beneficiaries in advance (by Spring) to make it possible to define the resources allocated in the budget for the following year.Simplify procedures for daycare centres.Support the capacity of (small) municipalities to plan and simplify administrative procedures.
Impact evaluation of specific objective 6 'Reducing the environmental impact in the context of the development of economy and town planning' under the Interreg Rhin Supérieur 2014-2020
Rapport final de l'évaluation d'impact de l'objectif spécifique 6 : "Réduire l’impact environnemental dans le cadre du développement de l’économie et de l’urbanisme dans le Rhin supérieur"
Reeleaf, https://www.reeleaf.nl/en - Spatial Foresight, https://www.spatialforesight.eu/home.html
Evaluation of interventions for gender equality under IP 8.iv of the Employment ESF OP, 2014-2020, in Austria
Prospect Unternehmensberatung - JOANNEUM RESEARCH Forschungsgesellschaft
Prospect Unternehmensberatung GmbH, Siebensterngasse 21/4, 1070 Wien, Austria, www.prospectgmbh.at - JOANNEUM RESEARCH Forschungsgesellschaft GmbH, Sensengasse 1, 1090 Wien, Austria, www.joanneum.at
The evaluation examines the implementation of 7 pilot projects on advisory services on gender-oriented approaches in enterprises, financed under the OP, including the results of 7 evaluations on each. The projects were carried out from mid-2017 to mid-2019; primary data used are updated at November 2018.
To assess the preliminary results from 7 pilot projects to provide advisory services to firms on gender equality and the implementation of the projects over the period from mid-2017 to mid-2019 and to make recommendations for the second round of projects.
Monitoring data.Survey of companies which used (32) and did not use the advisory services (230).Survey of stakeholders involved in project implementation (88).2 focus groups with project promoters.Interviews with experts (11).Workshop with Austrian, German and Swiss experts (10).Case studies on selected companies (20) through interviews with project promoters, managers and employees (123).Analysis of project documentation and communication material.
The pilot projects were successful in reaching companies of different size, employment structure, economic sector, and different previous experiences of gender-equality approaches.Companies were interested in participating if the advisory services addressed their current personnel needs (e.g. skilled worker shortages, staff turnover). The wish to promote gender equality played a minor role.Analysing concrete needs and challenges of companies is an important step towards defining appropriate gender-friendly measures.Success factors are company support in defining objectives and measures and the involvement of employees.Structural changes in the corporate culture are essential for the sustainability of the measures implemented.Companies supported were satisfied with the service, especially the content and the flexibility of the timing of the implementation.
The significance and availability of data are limited and uneven in the projects, because they were still being implemented.
Promote advisory services tailored to the characteristics, experience and needs of companies.Define uniform quality standards for the services.Raise awareness among enterprises of the benefits of promoting gender equality.Adopt two approaches to implementation: in the next round of projects, one for companies with skill-shortages, the other for those with career advancement and equal pays issues.
Assessment of the impact of support for education in the Lubelskie OP, 2014-2020
Ocena wpływu wsparcia oferowanego w ramach 12 Osi Priorytetowej RPO WL 2014-2020
AD REM Elżbieta Stosio-Sielach; Piotr Fuchs Smart Research
AD REM Pracownia badawcza Elżbieta Stosio-Sielach, ul. Macieja Palacza 120e/3, 60-278 Poznań; Piotr Fuchs Smart Research, ul. Wojskowa 5/73, 60-792 Poznań
The evaluation examines the effects of support for preschool, general, vocational and adult education in the region.
To evaluate the impact of support for preschool, general, vocational and adult education in the region.
Desk analysis.Survey (CAWI/CATI) of project participants and providers.Interviews with MA, Implementing bodies, parents of children attending pre-school facilities, adults participating in learning,Focus group interviews with school directors, project participants.
Access to pre-school education has been increased, particularly in rural areas: 4% of places in preschool education, 10% in rural areas, were created by the OP.Support of pre-school education is complex, assessed positively by stakeholders and parents, has benefited children’s development and is sustainable.The counterfactual analysis shows no impact of interventions on student results, though it might be too soon to assess and there are reliability issues in the exams analysed.Most of the activities did not continue after the end of projects.Interventions improved access to adult learning.VET students reported that participation in projects improved their situation on the labour market.
Part of the interviews had to be conducted online because of COVID-19.
Strengthen psychological and educational support for those in preschool and higher levels of education.Continue support for vocational education and increase career counselling for students.Continue support for schools, including investing in equipment, for people with disabilities also.Increase the competences of preschool teachers to work with disabled children.
Evaluation of the Asturias ESF OP - 2019
Evaluación del PO-FSE 2014/2020 del Principado de Asturias para el informe anual a presentar en 2019
Javier Mato Díaz (Coordinator), Angel Alonso Domínguez, Israel Escudero Castillo, Rosario González Arias
The evaluation examines the Principality of Asturias OP and is the annual evaluation report for 2019.
To assess progress in implementing the OP and its contribution to the horizontal aspects, the impact of the “Integrated Activation Pathways” (IP 9.i) for vulnerable people and the effects of PMAR scheme (IP 10.i) for preventing early school leaving.
Analysis of indicatorsInterviewsFocus groupsSurvey of participants (500)Counterfactual impact evaluation.
46% of funding was allocated, 30% for Axis 1B (TO8), which does not comply with the performance framework, mainly due to delays in implementing the measures foreseen.Under IP 9.i (SO 9.1.1), certified unit cost was 61% higher than forecast.In general, all measures incorporate the gender perspective and the principle of equal opportunities.In the case of Integrated Activation Pathways (IP 9.i), the risk of suffering from a mental disorder decreased after participating in the programme, for men by 57%, for women by 32%.For PMAR (TO 10.i) to prevent early school leaving of those with learning difficulties, the counterfactual analysis carried out showed a general positive effect of the scheme on participants in terms of easier access to secondary school, with no significant gender differences.
It was impossible to carry out an ex-post survey of non-participants in the Integrated Activation Pathways.
Reallocate financial resources from PA1B (TO8) to 2B (TO9)Improve the effectiveness and efficiency of interventionsIncrease awareness of gender inequalitiesIncrease the number of beneficiaries of Integrated Activation PathwaysStrengthen the PMAR interventions and provide training to teachers to avoid frequent turnover.
Second Interim Evaluation of the Friuli Venezia Giulia OPs, 2014-2020
Servizio di valutazione unitaria dei programme co-finanziati con Fondi dell’Unione Europea nel periodo di programmazione 2014-2020. Rapporto esteso 2019
Marco Pompili; Matteo Martella; Carlo Miccadei; Vincenzo Angrisani; Luca Rossi
Ismeri Europa s.r.l., Via G. G. Belli 39, 00193 – Rome, Italy; www.ismerieuropa.com
The evaluation examines the progress of the OPs and their contribution to the Europe 2020 objectives and analyses their more specific objectives – for the ERDF OP, the duration of the reporting phase of some measures, the first results of measure 2.3.a to support technological investment in SMEs and 2.3.b to support ICT investments in them and the results of the Communication Strategy. For the ESF OP, the focus is on the regional system of Higher Technical Education.
To assess the progress of the OPs and their contribution to the Europe 2020 objectives and, for the ERDF OP, the effect of investment on business strategies, firm performance and innovation capacity and the effectiveness of the communication strategy, and, for the ESF OP, the regional system of Higher Technical Education.
Documentation analysis;monitoring data;surveys of funded and non-funded enterprises and people;interviews with MAs and CCIAA and with 6 firms for ERDF;focus groups for ERDF;statistical data;theory-based approach for ESF.
Progress of ERDF and ESF OP is in line with objectives set and targets for 2023 will be met. Performance targets have been achieved or thresholds have been met.The ERDF OP is helping to meet the Europe 2020 strategy target for R&D.ERDF investment has contributed to achieving the Europe 2020 goals for energy and will potentially lead to a reduction in primary energy consumption of 23% and in CO2 emissions of 22%.The ESF OP contributed to innovations in regional employment policies and with the ERDF OP to human capital development.ERDF investment seems to have stimulated a more active attitude of businesses towards innovation.All output indicators show that communication strategy (ERDF) is performing well.Higher Technical Education targets have been achieved.
The call for ERDF projects did not clearly specify the types of investment to be made, so the 2.3.a measure was also used for traditional investment and by firms without a clear strategy for technological upgrading.
Monitor carefully ERDF projects that are strategic for jobs to be created;make methods used in the planning and verification process, e.g. the timetable for calls, clearer;continue measure 2.3 in the future, as it is consistent with regional industrial policy objectives;consider integrating ERDF measures with ESF strategies;simplify the regulatory and technical language used in tenders;continue investing in initiatives involving young people;use methods of dissemination to the general public, such as the social media, more widely;create Higher Technical Education governance at the regional level;accompany the Higher Technical Education system with training;integrate policies supported by the ERDF OP for research and sustainable economy into the ESF OP.
Analysis of the main territorial challenges, needs and transnational cooperation potential under the Central Europe Interreg Programme, 2021-2027
Analyse der territorialen Herausforderungen, Bedarfe und Potenziale für transnationale Kooperation in Zentraleuropa
Julia Grübler, Doris Hanzl, Helmut Hiess, Stefan Jestl, David Pichler, Joris Schröder, Roman Römisch
The Vienna Institute for International Economic Studies, Rahlgasse 3, 1060 Vienna, Austria; tel.: (+43-1) 533 66 10-86; wiiw.ac.at; gruebler@wiiw.ac.at
The study examines the main territorial challenges and needs in preparation for the programme in the 2021-2027 period.
To identify the main joint challenges and needs in the programme area and strategic areas for transnational cooperation, and to make recommendations for the programme in 2021-2027.
Documentation analysis.Analysis of statistical data.Survey of stakeholders (321 responses).Workshop with the members of the programme’s national committees.40 expert interviews.
The main differences in the Central Europe area are the West-East divide between countries and the urban-rural disparities within countries.Except in Austria and Germany, R&D activity in the programme area is low and mainly concentrated in urban areas.The environmental situation is improving, although many countries are still below EU standards.There are bottlenecks in the main cross-border connections and transport efficiency is low.A high concentration of economic, social and territorial problems, especially in rural areas, is leading to outward migration.A number of regions and cities have limited capacity to participate in territorial programmes.Coordination and cooperation with other territorial programmes and macro-regional strategies is complex.According to interview respondents, the main added-value of trans-national cooperation is the exchange and transfer of knowledge, cultural exchange and capacity building.
The programme should focus on three policy objectives: ‘a smarter Europe’, ‘a greener, low-carbon and resilient Europe’, and ‘a Europe closer to its citizens’, supporting 10 specific objectives which are largely consistent with these policy objectives.
Ex post evaluation of the Basilicata ESF OP, 2007-2013
Regione Basilicata, Dipartimento Programmazione e Finanze
The ex post evaluation report for the 2007-2013 programming period of the ESF Regional OP of the Basilicata Region.
To assess the impact of the OP and the effectiveness and efficiency of the interventions implemented, as well as their consistency with the OP objectives and their sustainability.
The evaluation uses both quantitative and qualitative methods:ad hoc questionnaires for each intervention analysed;focus groups;face-to-face interviews with relevant stakeholders;desk analysis of relevant documents and statistical data.
The expenditure on employability and social inclusion was larger than planned, as the OP was directed towards tackling the adverse effects of the economic downturn.The incentives for training and employment of (1,800) disadvantaged people had a positive effect on them in terms of their employment over the medium term.The programme to tackle poverty, supported around 2,750 families and kept the overall number in poverty lower than the Southern Italian average.The interventions for tackling social exclusion of disadvantaged people seem not to have had a strong effect.The interventions to promote training and higher education paths for the young had limited effects in preventing them from leaving the region.
The result indicators could not all be defined and the samples analysed were relatively small in many cases.
Most of the interventions financed in the 2007-2013 period can be continued in the 2014-2020 period.
The Impact of Cohesion Policy in Visegrad Countries
Efekty Polityki Spójności w Polsce, krajach Grupy Wyszehradzkiej i państwach partnerskich
ul. Bukowińska 22B, 02-703 Warszawa, imapp@imapp.pl,
This report is a summary of the study “The impact of cohesion policy 2007-2013 in Poland, countries of the Visegrad Group and partner countries.” It includes the cumulative results for all countries.
To set out the effects of the 2007-2013 Operational Programmes on the Visegrad countries.
The report is a synthesis of a previous study, In the latter the following methods were used:a simulation model of the impact of cohesion policy in the economies concerned, supplemented by secondary data – in particular input-output tables and data on international trade in goods and services by sector;desk analysis of additional data from Eurostat, OECD and European Commission and data on public procurement;case studies;CAWI survey.
Innovation and competitiveness:In Romania, support for micro-enterprises resulted in higher employment rates than in the control group;in Slovakia, beneficiary SMEs implemented new production methods and products, improving efficiency;in Croatia, knowledge and technology transfer between industry and academia was improved and new joint research projects were implemented.Human capital and labour market:In Hungary, pro-employment projects provided support to unemployed of 50 years and over;in Bulgaria, interventions reduced illiteracy in adults and increased the availability of schools for children with special educational needs;in Slovakia, interventions provided support to vocational education for young people and helped them to find employment.Transport:Investment improved the quality of the rail network, though in all countries it is still below the EU-15 average;in Czech Republic, the quality of airports was improved significantly;in Slovenia, ports were improved to the average level in the EU-15.Environment and Energy:In the Czech Republic, modernisation of heating systems in the public sector has stimulated the development of an energy-saving construction market;in Slovenia, photovoltaic infrastructure was installed;in Romania, interventions helped to increase the share of population covered by high quality drinking water from 8% to 32%.
Evaluation of measures for urban regeneration in small and medium municipalities under the Toscana ERDF OP, 2014-2020
Progetti di riqualificazione urbana Inclusiva e Partecipata (PRUIP): la valutazione del paesaggio urbano in comuni medio-piccoli della Toscana
Patrizia Lattarulo (IRPET), Silvia Ferrini, Gaetano Grilli, Sara Peppoloni, Marco Manetta (Università di Siena).
http://www.irpet.it/, patrizia.lattarulo@irpet.it; https://docenti.unisi.it/it/ferrini, silvia.ferrini@unisi.it
The evaluation examines the effects of urban regeneration projects financed under the OP in three cities: Arezzo, Lucca and Pistoia.
To assess the benefits of urban regeneration projects in Arezzo, Lucca and Pistoia, from the point of view of citizens.
Survey of citizens (final recipients of the measures – 60% response rate). Regression analysis on the survey data. Official statistics.
Building regeneration, especially combined with a multi-use aim, is generally preferred to redevelopment of squares. Green areas and cycle lanes are project features very often requested. It is estimated that citizens are willing to pay a maximum of EUR 110 per year to ensure that regeneration projects are carried out. According to the citizens surveyed, the situation in terms of green areas, cycle lanes, safety in public areas and integration in the neighbourhood has improved in 2022 as compared to 2017.
People over 60 are underrepresented in the sample used.
Mid-term Evaluation of the Principado de Asturias OP, 2014-2020
Evaluación del PO-FSE 2014/2020 del Principado de Asturias para el informe anual a presentar en 2017
Fco. Javier Mato Díaz (Coordinator), Israel Escudero Castillo, Rosario González Arias
This is the mid-term evaluation of the ESF OP for Principado de Asturias 2014-2020.
To assess the OP’s implementation, effectiveness, efficiency and impact.
The evaluation adopts a mix of qualitative and quantitative methods:Desk research;analysis of relevant policy documents;analysis of monitoring data;counterfactual impact analysis using propensity score matching;in-depth interviews of 18 beneficiaries;data from Public Employment Services and other official sources.
The OP’s implementation shows a significant delay, total investment is 16% of that planned for the period (as compared with a target of 38%).The delay in implementation is principally due to the late start of the Programme and the fact that several measures initially included in the OP were carried out outside the Programme.The effectiveness and efficiency of the OP are acceptable. Three outcome indicators already exceed their targets and four are close to achieving them. Unit cost are mostly below the expected values.The counterfactual impact evaluation revealed no statistically significant effect of the training on the probability of finding a job.There is a certain level of satisfaction among participants due to the perceived economic and psychological consequences of being in work, but the intervention does not seem to have improved their level of employability.
There is a need to:Improve and standardise the procedures for collecting information on beneficiaries;consider the possibility of reprogramming the OP to redirect funds towards PAs where the level of execution is highest;improve the tendering process in order to avoid concentration of calls for tender in specific periods of the year;consider strengthening guidance activities to improve matching between training activities and the specific characteristics of beneficiaries;promote, through incentives or other affirmative action, women's access to the training opportunities offered.
Evaluation of PA2 "Competitiveness of SMEs" of the Niedersachsen ERDF-ESF OP, 2014-2020
Evaluationsbericht zur Prioritätsachse 2 Wettbewerbsfähigkeit der KMU
Stefan Meyer (Kovalis), Lena Heitzenröder (ISG), Julia Schade (ISG), Marco Puxi (ISG), Darius Bosselmann (Kovalis)
Kovalis, Dr. Stefan Meyer, meyer@kovalis.de, www.kovalis.de - ISG Institut für Sozialforschung und Gesellschaftspolitik GmbH, info@isg-institut.de
The evaluation examines PA2 of the OP, focusing on business-related infrastructure in the first part and horizontal principles in the second.
To assess the implementation of PA2, focusing on business-related infrastructure and horizontal principles.
Desk reviewAnalysis of monitoring dataInterviews with 21 experts in technical departments, the NBank, funded and non-funded municipalities and executing agencies.Online survey of units responsible for 11 measures on the effects of Covid.Interviews with 13 experts from the MA and NBank on horizontal principles.
Implementation of business-related infrastructure was slow with delays in project approval and few projects completed in time mainly because of burdensome planning and coordination procedures in municipalities; scarce financial resources; lack of personnel capacity in municipalities; and price increases in construction.Horizontal principles have often not been taken into account in defining funding measures and in the guidelines produced by the MA.
Support municipalities in project planning and management e.g. via a central agency or central service providers.Optimise procedure e.g. through a two-stage process, increase funding rates for early applications and reduce the involvement of third parties in the approval process.Establish more intensive communication between those involved.Introduce a post-approval or price escalation conditions to reduce the risk posed by cost increases, particularly for financially weaker municipalities.Support better cooperation between municipalities and project sponsors.Give greater weight to horizontal principles by emphasising their relevance in guidelines, specifying expectations and objectives for each funding measure, improving monitoring and increasing the professional competence of stakeholders.
Evaluation of the cooperation of local action groups under the Employment OP, 2014-2020, in the Czech Republic
Vyhodnocení přínosu spolupráce aktérů místních akčních skupin
The evaluation examines the cooperation of local action groups (LAGs) under the OP.
To evaluate the benefits of cooperation between local action groups (LAGs), and to map examples of good practice up to March 2022.
Focus group with 7 representatives of municipalities and mayors.Focus group with 6 representatives of NGOs and businesses.Focus group with 8 LAG managers.Case studies: Šumperský countryside, Brdy-Vltava, and Orlicko LAGs.
LAGs are considered a useful means for establishing long-term collaboration between local actors. Knowledge of the local network and the proactive approach of LAG personnel are conditions for a successful and long-lasting partnership.NGOs are readily involved in LAGs, while businesses are likely to be involved only if there are business development opportunities.The most relevant contribution of LAGs is the promotion of social activities in municipalities.Main motivations for cooperation are the possibility of raising funding for territorial development projects, developing connections between local actors and improving the quality of local services.
Encourage greater involvement of businesses in LAGs.Ensure better coordination between LAG activities and regions, and synergies between investment and non-investment projects to ensure sustainability of innovative projects.Provide follow-up support to social enterprises participating in projects and for project accounting in small municipalities.
Evaluation of the effects of social investment on improving accessibility and quality of social services in the Mazowieckie Voivodeship in 2014-2020
Wpływ RPO WM 2014-2020 na poprawę dostępności i jakości usług społecznych w województwie mazowieckim
Krzysztof Gutta, Beata Belica, Michał Dominik, Tomasz Gutta, Katarzyna Gutta
The evaluation examines how social investments have contributed to improving the quality and accessibility of social services in the region.
To determine whether the measures have improved the quality and accessibility of social services and active integration and to assess the efficiency and effectiveness of the institutional system.
Desk research.Analysis of monitoring data and official statistics.Survey of 505 beneficiaries and 1154 project participants.419 interviews with residents of the region.20 interviews with representatives of institutions.8 interviews with beneficiaries.8 case studies.Benchmarking with 3 programmes.Delphi panel.SWOT analysis.Social Listening analysis.2 expert panels.Evaluation workshop.
There are significant disparities in access to social services and active inclusion opportunities across the region.Older people and those requiring daily assistance, including people with disabilities, are identified as a growing group in need of special support.The demand for mental health support has increased.Limited activity of bodies in applying for EU funds contributes to the significant differences in access to social services between areas.There is a lack of a long-term cooperation culture.Cooperation with project supervisors was found to be crucial in ensuring the quality of project implementation.
Prioritise addressing the existing "white spots" with limited access to social services, which require targeted investments to improve availability and promote active inclusion.Develop specialised support services tailored to the needs of older people and those in need of assistance.Allocate resources to meet the increasing demand for mental health support.Make efforts to enhance the capacity of bodies to apply for EU funds and foster a culture of long-term cooperation among beneficiaries and other relevant bodies.Establish support mechanisms similar to project supervisors to improve the implementation of projects.
Evaluation of the contribution of the Śląskie OP 2014-2020 to the achievement of objectives under PA8 'Regional personnel of the knowledge-based economy'
Ewaluacja sposobu, w jaki wsparcie w ramach Regionalnego Programu Województwa Śląskiego na lata 2014-2020 przyczyniło się do osiągnięcia celów w ramach VIII osi priorytetowej Regionalne kadry gospodarki opartej na wiedzy
Agnieszka Śnieżek, Michał Marciniak, Marta Cichowicz-Major, Magdalena Głomska, Aleksandra Kamińska, Paweł Penszko, Małgorzata Zub, Sebastian Pałka
EVALU Sp. z o.o., ul. Dzika 19/23 lok. 55, 00-172 Warszawa
The study examines the impact of the interventions on the knowledge-based economy under the OP.
To assess the effects of support for the knowledge-based economy and the achievement of targets and to make recommendations for labour market support for 2021-2027.
Desk research.Survey of 142 beneficiaries and 1531 project participants.Interviews with 8 implementers, 4 thematic experts and 4 operators of measure 8.2.Counterfactual methods.5 case studies.Benchmarking analysis in 7 regions.Group interview with MA and IB.
There is risk to the long-term sustainability of the effects of projects creating childcare places for up to 3-year olds in private institutions.There are creaming effects in supporting participants in pro-employment activities, most of whom have higher education and high employment potential living in large cities.There has been insufficient growth of childcare places for up to 3-year olds in rural and less urbanised areas.Territorialisation of support for enterprises through development services has not brought any added-value.Projects are well linked to smart specialisation of the region and to industries with high employment potential.
The methodological challenge was to create indicators that would enable whether there has been any increase in the level of competence and qualifications of employees supported to be determined.
Reserve part of 2021-2027 funding for maintenance of childcare places for up to 3-year olds.Apply a selection criterion promoting projects from rural/less urbanised areas.Maintain support for the development of day care centres and municipal and inter-municipal networks.Replace mechanisms of territorialisation of support for enterprises with bonuses for support in strategic intervention areas.Maintain health programmes for cancer prevention.
Evaluation of relevance and effectiveness of multi-funding and support for regions under the of Research, Development and Education OP 2014-2020 in the Czech Republic
Zpráva z hodnocení relevance a efektivnosti multifondovosti OP VVV a podpory kategorie regionů (více rozvinuté regiony, méně rozvinuté regiony) v rámci multifondového OP VVV - Průběžná evaluace
The report is a part of an on-going evaluation project to assess the implementation of the Research, Development and Education OP 2014-2020. It focuses on multi-financing by the ERDF and ESF and support to regions under PA1 and PA2.
To assess the relevance and effectiveness of the multi-funding approach and support to both more and less developed regions, with a focus on complementarities and additionalities of the calls.
Desk research of documentation and monitoring data.Interviews with stakeholders (11 individual and 1 group).Survey of beneficiaries.Internal expert panel.2 focus groups.2 case studies.Comparative analysis.Process analysis.
The multi-funding approach targeting both more and less developed regions is relevant in the current and future programming periods.Linking ERDF and ESF interventions at the general and project level is regarded as the main benefit of additionality.The administrative burden, project coordination issues and time differences between the calls under PAs 1 and 2 are considered to have limiting effects on both complementarity and additionality.
Include a uniform co-financing rate for all regions in the next programming period.Maintain the multi-funding approach in the next programming period.
Impact evaluation of the Central Europe Interreg OP, 2014-2020 – Phase 1
Impact Evaluation Report of the Interreg CENTRAL EUROPE Programme – Phase 1
Civitta - WIIW – the Vienna Institute for International Economic Studies
https://civitta.com/; https://wiiw.ac.at/
The evaluation examines the impact of the Central Europe Interreg OP, focusing on Calls 1 and 2, up to mid-December 2021.
To assess the achievement of objectives and effects of the OP, including the mechanisms that led to them.
Documentation analysis and literature review.Analysis of monitoring data.Interviews with programme stakeholders, project beneficiaries, end-users, and thematic experts.Surveys of programme implementers, project beneficiaries, and end-users.Cost-effectiveness analysis.Case studies.6 focus groups.Theory-based evaluation.
Out of 138 projects, 85 had been completed by mid-December 2021.The indicator targets had either already been achieved or will be by 2023.1,761 institutes adopted new or improved strategies and 3,369 applied new or improved tools and services, 62,452 people were trained, and 1,276 new full-time jobs were created.The OP has achieved its objectives in improving the capacity of public and private bodies, supporting cooperation beyond borders in Central Europe, and bringing EU-level priorities closer to local communities.Pilot measures enable new approaches in addressing challenges to be developed and tested, and are considered to be cost-effective and unlikely to have been implemented from other funding sources.The projects contributed to changing practices in individuals and organisations.According to beneficiaries surveyed, most project outputs are expected to last for over three years after projects ending, but their sustainability depends on additional funding, synergies with other initiatives and cooperation between project partners.
Assessment of output effectiveness made by the experts may be biased, and the experts might have had a different understanding of the scale for scoring used. The results of the cost-effectiveness analysis should be interpreted with caution, since the way the index is constructed is difficult to back up empirically, and it was sometimes difficult to clearly allocate costs to individual outputs.
Ensure a more systematic approach both to identifying and enabling synergies with national policies and EU programmes.Involve relevant stakeholders to ensure project results are used to inform policy decisions at local, regional, national and EU levels.Ensure sufficient technical capacity during the project selection process in order to finance high quality and innovative projects.Continue supporting pilot actions.Increase support for beneficiaries from less active regions.Promote the transfer of knowledge and the outputs and results achieved.Have a more systematic approach to collecting and disseminating good practice.Focus on creating functional links between the different territories and distributing benefits more fairly between them.Consider setting more ambitious indicator targets in the future.
Evaluation of communication, effectiveness and stakeholder involvement of the Alpine Space Interreg OP 2014–2020
Evaluation of programme communication, effectiveness and stakeholder involvement of the Interreg Alpine Space 2014–2020 Programme
The evaluation assesses the implementation and the communication strategy of the Alpine Space OP 2014-2020, as well as the involvement of stakeholders.
To assess the implementation of the OP, the communication strategy and the involvement of stakeholders.
Desk analysis of documentation;analysis of monitoring data and project websites;9 interviews with programme bodies;surveys of project leaders and observers (response rate: 97% and 16%, respectively);consultation with the EUSALP (EU Strategy for the Alpine Region) presidency and action group leaders.
At the end of January 2018, 33 projects had been approved (56% of the total estimated);for each of the specific objectives, targets and the milestones of the Performance framework are likely to be attained;the horizontal principles are integrated well into the management arrangements and reflected in the programming documents;the OP mainly contributes to three priorities of the Europe 2020 strategy: 55% of the projects contribute to achieving the Climate and Energy goal, 24% the R&D objective and12% the reduction in poverty, while they indirectly contribute to increasing employment;90% of the stakeholders consulted report that coordination between the OP and EUSALP is effective;For 29 of the 33 projects (88%), synergies are reported with other funds;the involvement of relevant target groups is generally satisfactory;“observers” have proved to be useful.
There is potential bias as regards the responses to the survey of project leaders since half of them are Italian.
There is a need to:review an output indicator and change the target value;continue supporting projects which raise awareness of the different possible roles and contributions of observers to projects;support projects on social innovation;increase interactions between the EUSALP and ESI Fund MAs;increase private sector involvement.
The impact of the Cohesion policy on selected macroeconomic indicators at the national and regional level using the EU Impact model in the period 2004-2018
Ocena wpływu realizacji polityki spójności na kształtowanie się wybranych wskaźników makroekonomicznych na poziomie krajowym i regionalnym za pomocą modelu EUImpactMOD
Henryk Kalinowski, Andrzej Regulski, Marek Antosiewicz, Jan Witajewski ; imapp consulting sp. z o.o, Fundacja Naukowa Instytut Badań Strukturalnych
Imapp consulting sp. z o.o, hello@imapp.consulting, www.imapp.consulting; Fundacja Naukowa Instytut Badań Strukturalnych, ibs@ibs.org.pl, www.ibs.org.pl
The evaluation uses the EU Impact model (a dynamic general equilibrium model) to estimate the impact of Cohesion policy in 2004-2018 at national and regional level.
To estimate the impact of Cohesion policy investment in business, infrastructure and human capital on GDP growth, employment, the standard of living, public finances and exports and on real convergence.
Macro-economic modelling (EU Impact model).
GDP in 2018 is estimated to have been 4.9% higher than it would have been without Cohesion policy.This has accelerated real convergence in GDP per head between Poland and the EU average.The employment rate of those aged 20-64 rose to 72.2%, 1.9 pp of which as a result of Cohesion policy investmentEU funding has helped slow down the pace of divergences between the richest and the poorest regions in the country.It has been the main driver of economic growth in the least developed regions, i.e. the Eastern Poland voivodeships and the Zachodniopomorskie region.
Letter to the EC on the first evaluation of the effectiveness, efficiency and impact of joint support from the ESF and the specific allocation for YEI for the implementation of the Youth Guarantee (YG)
The evaluation covers two national projects co-financed by the YEI and which are part of the ESF OP ‘Human resources and employment’:Active labour market policy measures for the young registered unemployed.Implementation of 2nd chance vocational education programmes.The report is mainly based on the results of a study conducted by the OECD in 2015 (Investing in Youth: Latvia, OECD Publishing, Paris). (Though it is indicated that the SEA signed an agreement with the OECD on the research involved in the study, it is not clear what the status of the study is and the extent to which it is an evaluation as such).The YG has been implemented since January 2014 in three stages (only the first two are supported by the YEI):Reinforcing the availability of active measures.Developing a second chance training programme of 1-1.5 years with the support of VET schools for those with very low skills.Promoting outreach activities to re-engage youth and the training of youth specialists in municipalities.The 1st stage is based on various existing programmes (and involves for instance allowances) while the 2nd sets out a strategy to create the second-chance programme based on a network of VET schools.
The objective of the (letter of) evaluation is to set out an assessment of the effectiveness, efficiency and impact of the YEI as required by the ESF Regulation.The OECD report provides a detailed assessment of the youth labour market and education system and a statistical analysis of the situation of disadvantaged young people in Latvia.The next evaluation (due in December 2018) will analyse in depth the longer-term labour market outcomes of the YG participants.
Due to the late implementation of the YEI, only data on participation in the various measures supported are presented.Data are based on surveys as well as administrative registers.
Due to the late implementation of the YEI and the small number of young people who completed the programmes, it is too early to assess the effectiveness, efficiency and impact of the YEI support.Findings are:The high number of caseloads among the SEA staff may explain the relatively slow start for some of the measures.Preliminary results for the 1st stage of the YG show that it helped to maintain the share of newly registered unemployed young people participating in an active programme (in particular in training programmes). Participation of young disadvantaged in measures providing hiring subsidies to employers remains low.
(Note: Lack of data is not indicated as a limitation as such)Caseloads of PES counsellors are large limiting their capacity to follow up on the most disadvantaged youth.Labour costs at the minimum wage are significant compared to the costs of workers on the average wage which may explain why participation of low-skilled youth is low in measures providing hiring subsidies.Links between the VET institutions and employers are weak, lowering the chances of participants in second-chance programmes of finding a job.Implementing an effective YG when unemployment is high is challenging as activation used to be weak in Latvia.A significant share of NEETs (about 30%) may be difficult to reach.
Improve the sharing of information on at-risk youth between schools and PES to further reduce early school-leaving and non-completion of VET and other training programmes.Develop a quality apprenticeship system, with tight links between school and employers while involving social partners in programme design.Further improve/reinforce career counselling in schools to motivate students at risk of dropping out to enrol in VET.Reduce caseloads per PES counsellor for the most disadvantaged youth by improving coordination between services and ideally by hiring additional counsellors.
Strategic study of the context and measures on digital competences in Veneto under the ESF+ Programme, 2021-2027
"Approfondimento tematico “Rafforzare e diffondere le competenze digitali”"
IRS, https://www.irsonline.it/en - PTSCLAS, https://www.ptsclas.com/en/home-en/ - GRETA, https://www.greta.it/old/en/index.html
This study examines support for digital competences under the ESF OP up to end-2020, and the context for digitalisation of public services and developing digital competences among the population for them to better access public services.
To assess the needs of the population and public authorities for digital competences, and to identify how the future ESF+ can support the mainstreaming of digitalisation.
Benchmarking (Lombardia and Emilia-Romagna).In-depth interviews with stakeholders.Focus groups with stakeholders.Mystery user approach to assess the effectiveness of websites.Analysis of administrative data.Analysis of statistical and monitoring data.
Digital competences are key to regional competitiveness and fundamental to help SMEs most affected by Covid.The present level of digitalisation of training in Veneto is modest, though for employment and counselling services it is higher.Digitalisation of public services can help reduce the administrative burden.The digital divide should be tackled as it mostly affects women and older workers willing to re-enter the labour market.Teachers and educators should be supported in increasing their use of ICT equipment in schools.Financial support should be provided to those willing to create innovative start-ups, as they can improve regional competitiveness.
Improve the functionality of the regional website by highlighting the connection between training, counselling and employment services.Define a preliminary screening of student competences when enrolling.Reorganise the regional websites on training and employment services on the basis of good practice such as the ITS Veneto Academy and #orientati.Keep information on the regional websites always updated and provide adequate assistance to users.
Evaluation of the effects on employment of vocational training funded in 2017 under the Piemonte ESF OP, 2014-2020
L’effetto occupazionale della formazione professionale in Piemonte
Samuele Poy, Chiara Subrizi, Maria Cristina Migliore
The evaluation examines the effects on employment of vocational training measures funded by the ESF OP in 2017.
To assess the effects of vocational training measures in 2017 on the employment rate of participants, the most effective kind of training and the cost-effectiveness of the measures.
Monitoring data.Administrative data for PSM counterfactual analysis.
12 months after the interventions, the employment rate of participants was 6.6 pp. higher than for the control group (8.9 pp after 18 months).The higher employment rates come from temporary contracts and there are no visible effects on permanent contracts.The effect is barely positive for young people and only slightly so for the most long-term unemployed.Cost per job placement of training is estimated at EUR 62,000.
Final evaluation of the initial phase of the ESF Specific Objective ‘Promoting sustainable employability of workers – companies/institutions’, 2014-2020
Eindevaluatie eerste openstelling ESF-regeling Duurzame Inzetbaarheid – bedrijven/instellingen
1. Bureau Bartels B.V., P.O. Box 318, 3800 AH Amersfoort, Tel: +31 33 4792020, email: info@brtls.nl, internet: www.bureaubartels.nl
The evaluation is a follow-up of a study carried out in 2015 assessing the effects of projects promoting sustainable employment under the ESF 2014-2020 OP.
The evaluation aims to assess the efficiency, effectiveness and first effects of subsidies granted for the implementation of projects promoting sustainable employment in the workplace (projects for healthy and safe work; projects for reducing short-term absence because of illness, etc.). Several evaluation questions are addressed:What are the contents and scope of the interventions (addressed in the first interim report of 2015)?How is the scheme managed and implemented? (addressed in the first interim report of 2015)How effective are the projects?What lessons can be drawn for the future?
The evaluation uses a mix of qualitative methods:Interviews to 4 stakeholders.Desk research.Telephone interviews with participants in the scheme (a stratified sample of 350 in total out of 2,091 projects financed, of which 218 interviewed for the first interim evaluation in 2015, and 132 new ones interviewed for the 2016 report), namely employers, Work Council (WC) members and employees’ representatives.Data sources are: interviews with stakeholders and participants in the scheme; monitoring data on the implementation of the projects from the SZW Agency (an Inspectorate operating within the Ministry of Labour).
The main findings are:Efficiency: the analysis found that the provision of the scheme was efficient, as it supported the implementation of projects which for nine out of ten employers, would otherwise not have been implemented or would have been carried out on a smaller extent. Only 9% of the employers interviewed stated that the project would have been undertaken without a subsidy.Effectiveness: two main issues emerged. On the one hand, awareness was raised among both employers and employees on the need for further investment to support sustainable employment. On the other hand, more than half of those interviewed stated that substantial changes relating to the organisation of work were made; in addition, measures were implemented to support employees, such as the provision of coaching, training, education courses (two-thirds of the interviewed) and changes in the way projects were undertaken (three-quarters of the cases).Impact: the overall effects were reflected in increased dialogue between employers and employees.
The evaluation highlights the fact that the number of respondents to different questions was uneven.
The main policy implications are:To avoid projects that are selected for financing not being implemented, a contact point should immediately be established and changes in the implementation plan should be consistent with the aim of making employment more sustainable.To avoid rejection of applications due to documents or declarations not being complete, a checklist could be provided for applicants.Enterprise applicants should ensure they receive preliminary support from internal staff in order to ease implementation of projects.
Evaluation of the use of simplified cost options in Portugal in 2007-2013 and 2014-2020
Avaliação da Aplicação de Custos Simplificados
All OPs 2014-2020 except technical Assistance; OP Human Potential and Social Cohesion Madeira 2007-2013; OP Human Potential 2007-2013; OP Açores Objective convergence 2007-2013
The evaluation examines the application of simplified cost options (SCO) in Portugal in 2007-2013 and 2014-2020 OPs.
To assess the coherence and relevance of the measures adopted to simplify cost reporting and their efficiency and effectiveness.
Monitoring data.Interviews with MAs.Survey of MA and IBs (response rate 29%) and beneficiaries (11%).5 case studies.Focus groups.Benchmark analysis.
There are no inconsistencies between SCO rules and national legislation.Knowledge of SCO application procedures is still limited among MAs, IBs and beneficiaries.Control and audit guiding documents have been amended to incorporate specific guidelines on SCO, but the monitoring systems have not been systematically adjusted to incorporate the specificities and results of interventions using SCO.SCO methodologies have helped to reduce the administrative burden for the MA and beneficiaries and to speed up the project selection process, but no reduction in project management costs is evident.The adoption of SCO led to an increased focus on outputs and results in the implementation of measures.
Some specific aspects (e.g. reduction in error rates) could not be investigated because of the limited data available on measures which had adopted SCO shortly before.
Include SCO methodology in the programming documents for 2021-2027 and provide guidance and training for MAs and IB staff and beneficiaries.Promote the development of SCO methodology, involving beneficiaries and stakeholders.Adjust monitoring systems to incorporate information on SCO and facilitate audit procedures focussed on results of operations.Strengthen synergies between MAs and IBs to facilitate the adoption of uniform procedures on SCO.
Satisfaction of Monitoring committee members in 2014-2020
A monitoring bizottsági tagok elégedettségének felmérése, 2019
ZRI Závecz Research Kft; www.zavechresearch.hu
The evaluation examines the satisfaction of the members of the monitoring committees (MCs) of 2014-2020 OPs in Hungary in 2019.
To assess the satisfaction of MC members with their work, organisation, the number of meetings and the flow of information between themselves.
A survey of 153 members of monitoring committees.
A large share of respondents are satisfied with the organisation of meetings, with some critical only of the venue. A large majority of members are also satisfied with the technical content of meetings.Satisfaction is also high in relation to the informative nature of meetings, but a significant proportion report a need for more information about the results (26%) and impact of programme (33%). Members representing EU and civil society organisations are slightly less satisfied than average.Half of the members use the GIS software interface in their work, though only 17% of them are fully satisfied with it and 16% are dissatisfied.
The number of replies from the Human Resources, Economic Development and Innovation and Environmental and Energy Efficiency OPs Monitoring Committee members was less than 10.
Provide more content in the committees on good practice and results of OPs.Organise more informal events to exchange information.
Evaluation of RIS3 under the Veneto ERDF OP, 2014-2020
Sistema di Monitoraggio e Valutazione della RIS3
Lattanzio KIBS, https://www.lattanziokibs.com/en.html - IRIS, https://www.irisricerche.it/
The evaluation examines the implementation of RIS3 in the region. Statistical data used go up to 2021, monitoring data to January 2022.
To assess strengths and weaknesses of the RIS3 monitoring system up to end-2021/beginning 2022 and make suggestions for 2021-2027.
Documentation analysis (programme documents on RIS3, tenders).Analysis of monitoring data.Analysis of statistical data (ISTAT-EUROSTAT, ASIA-Local Units).Interviews with policy managers.
R&D expenditure has been increasing since 2017, including in relation to GDP.Designing RIS3 has been a long process and the governance of measures has been revised to make management more effective.The monitoring of implementation has not been systematic.The new RIS3 system for monitoring and evaluation for 2021-2027 will be based on a ‘circular matrix monitoring system’ providing relevant information on an ongoing basis.
Carry out an effective mapping and selection process for the initiatives that contribute to RIS3 in Veneto.Define the technological classification of projects at the beginning of the monitoring process to allow the competitive positioning of Veneto and its research potential to be assessed.
Evaluation of the fulfilment and revision of priorities of the Czech Republic-Poland Interreg OP, 2014-2020
Evaluace plnění priorit a specifických cílů a ověření změn rozvojových priorit programu Interreg V-A Česká republika –Polsko: Ověření změn rozvojových potřeb programu (2016 -2017)
https://home.kpmg/cz/en/home.html - https://evaluace.eu/en/about-us/
The evaluation examines the extent to which priorities and objectives under the Interreg OP, 2014-2020 have been achieved and the suitability of revisions to priorities.
To assess the capacity of the OP to respond to territorial needs and identify opportunities for development.
Interviews with MA and target groups.Surveys.Official statistics.Programme documentation.
The main problems of the region are outward migration of high-skilled people, long-term unemployment, misalignment between training and labour market needs, low development of services.The main opportunities for development are tourism, technical education and cross-border cooperation. The OP addresses the main regional problems and opportunities, especially as regards tourism, transport and education.The OP has synergies with other ESI Fund programmes in the Czech Republic.
Develop the tourist sector, using a destination management and marketing approach (DMO).Intensify cooperation between schools and companies, and make technical education more attractive.Increase employment in the tertiary sector.
Mid-term evaluation of PA 2 "Social Inclusion" under the Efficient Human Resources ESF OP, 2014-2020, in Croatia
Vrednovanje djelotvornosti, učinkovitosti i učinka OP Učinkoviti ljudski potencijali 2014. – 2020. – Grupa 2: Vrednovanje Prioritetne osi 2 „Socijalno uključivanje” (Srednjoročno vrednovanje)
Piotr Stronkowski, Tomaž Deželan, Lidija Japec, Branka Hodak, Mona Manojlović
WYG savjetovanje d.o.o., Adresa Ulica grada Vukovara 269 G, Zagreb, https://wyg-c.eu/
The evaluation examines PA2 up until 31 December 2019.
To assess the relevance, efficiency, effectiveness and impact of the measures supported up to end-2019.
Desk analysisAnalysis of monitoring and statistical dataIn-depth interviews with stakeholdersCAWI of project implementersCAWI/CATI of beneficiariesExpert workshop.
As of end-2019, a total of 91% of funding was allocated and 52% paid out.Activities planned under PA2 are in line with the needs set out in the national strategic documents, i.e. combating poverty and social inclusion, protecting the most vulnerable groups with a particular focus on less developed areas and increasing the availability of primary healthcare and social services in the community.Measures supported are complementary with similar national ones.Positive efforts have been made to target less developed areas, e.g. the five Slavonian counties.There is a need for further training of service providers.Support was focused on the unemployed and to a minor extent on groups facing more difficulties in the labour market, such as the inactive and people with disabilities.New skills and qualifications were obtained by participants in projects under IP 9.i.Participants reported a strengthening social ties, increasing self-confidence and more motivation to work.Projects have helped to provide better healthcare though the quality of services does not seem to have improved.There were delays in signing contracts and approving expenses, and the COVID19 pandemic was a further challenge.
Simplify the procedures and harmonise the practices of IBs.Simplify complicated structures of IBs and strengthen coordination and compliance with deadlines.Use online tools and reduce paperwork, use simple cost options and sample to check cost eligibility rather than checking each individual cost.Focus more on achieving results for final beneficiaries, e.g. by visiting projects on the field, strengthening coordination between authorities and aligning interventions.Consult with competent authorities to define operations and indicators in line with local needs.Focus on timeliness, suitability to individual needs, results and ensure continuity in service provision in order to improve the quality of social services provided.Focus more on the most vulnerable groups.Improve the quality of the monitoring system.
Evaluation of TO1 ‘Research, Development and Innovation’ under the regional OPs and the Multiregional OP for Spain, 2014-2020
Seguimiento y análisis sectorial del Objetivo Temático: Potenciar la I+D+i
Multiregional OP for Spain ERDF 2014-2020; all ERDF ROPs except OP Ciudad de Ceuta ERDF 2014-2020 and OP Melilla ERDF 2014-2020
Dirección General de Fondos Europeos — Subdirección General de Programación y Evaluación
The evaluation examines the progress of the implementation of the measures supporting research, development and innovation in Spain. It covers the implementation period between 2014 and April 2025.
To provide an overview of the implementation of the measures supporting research, development and innovation. To assess the progress of the main indicators.
Analysis of statistical data (Eurostat, Spanish Statistical Office).Analysis of monitoring data.Analysis of administrative data (Spanish Patent and Trademark Office).
EUR 5,474 million have been invested in research, development and innovation activities through the ERDF and the REACT-EU initiative.The implementation rate has been higher in the more developed regions which can rely on more developed R&D capacity and a greater ease of absorption.The ERDF and REACT-EU have mainly supported investments in public infrastructure and R&I projects (43%), companies and private research and innovation centres (37%) and cooperation projects between companies and research centres (13%).The R&D expenditure on GDP reached 1.49% in 2023 and is still far from the EU average (2.24%).The share of the R&D expenditure financed by the private sector in 2023 (48%) is still far below the target that was initially set for 2020 (60%).
Evaluation of Priority Axis 10 'Improving Education Infrastructure' of the Regional OP, 2014-2020, in Romania
Evaluarea POR 2014-2020. Axa Prioritara 10: Imbunatatirea infrastructurii educationale
Liliana Olivia Lucaciu, Monica Mihaela Roman
CIVITTA Strategy & Consulting SA, www.civitta.com; Lideea Development Actions srl, www.lideea.eu
The evaluation examines the measures to improve the education infrastructure under PA10 of the OP. Data used go up to mid-2023.
To assess the effectiveness, efficiency, sustainability, and impact of the measures to strengthen the education infrastructure.
Desk analysis: EU and national frameworks, ROP documents, annual implementation reports (AIRs), and previous evaluations. Analysis of monitoring data and official statistics. 12 interviews with MA, IBs, beneficiaries and contractors. Survey of 432 beneficiaries (124 responses, 28.7% response rate). 8 case studies. Focus group with 20 participants (MA and IBs). 2 workshops with beneficiaries and 3 with project evaluators (38 participants). Delphi Survey of 10 experts to refine findings and conclusions. Expert panel: 4 experts participated in two-stage discussions on impact and sustainability. Analysis of administrative data extracted from Single Management Information System (SMIS). GIS Maps and Visualisations. Theory of Change (ToC). Counterfactual reasoning (e.g., comparing observed outcomes to planned objectives and assessing the likely effects in the absence of interventions) without a real use of the counterfactual method because of the stage of projects.
The measures have led to the rehabilitation or construction of numerous educational facilities across Romania, and increased school capacity and accessibility. Preliminary data indicate higher enrolment rates, particularly in early childhood education and in rural areas, and a decrease in dropout rates at primary and secondary education levels in the areas supported. Projects focusing on integrating children from marginalised communities have been able to promote equity in education. The number of available places in educational institutions has increased, particularly in urban areas where demand is higher. The projects have generated temporary employment during construction phases and stimulated local economic activity. Improved school infrastructure has contributed to increasing property values and community satisfaction in regions receiving support.
The evaluation was carried out shortly after the implementation of the projects, so the long-term impact assessment is limited.
Improve data collection and analysis. Strengthen collaboration between MAs, regional bodies and beneficiaries to streamline processes and reduce inefficiencies. Prioritise additional funding and tailored interventions for vulnerable groups (Roma communities and children with special needs), to ensure equitable access to education. Implement region-specific strategies to account for local issues and differences in project effectiveness. Develop clear guidelines and support mechanisms to ensure the sustainability of redeveloped educational facilities.
Evaluation of the support for rail and road accessibility under the Kujawsko-Pomorskie ERDF OP, 2014-2020
Ewaluacja wpływu interwencji w ramach Regionalnego Programu Województwa Kujawsko - Pomorskiego 2014-2020 na dostępność komunikacyjną kolejową i drogową regionu
dr Michał Wolański, Beata Paczek, Wiktor Kijania, Karolina Orcholska, Michał Babicki, Dominik Makurat, Paulina Kozłowska, Julia Widłak, Marcin Pinkosz, Jakub Kaczorowski, Mirosław Czerliński
The evaluation examines the progress and results of measures for improving transport accessibility under the OP.
To assess the impact of projects on increasing the rail and road accessibility of the region.
Analysis of monitoring and official statistics.12 interviews with institutions.Survey of 43 beneficiaries, and 568 inhabitants.2 case studies.Evaluation workshop.
Despite some changes, part of the region is still characterised by limited access to public transport (especially rural areas).Road investments have enhanced traffic safety and improved conditions for pedestrians and cyclists, though statistics do not fully reflect this.The "design and build" formula extended the implementation time of investments and, in many cases, even made them impossible.Some bicycle paths are built in a fragmentary manner and are neither an alternative to individual transport nor a complement to public transport.
Start activities relating to the establishment of district-municipal associations as organisers of public collective transport.Develop bus transport and carry out additional analysis of the possibility of implementing on-demand transport.Further focus on measures improving the safety of pedestrians and bicycle traffic.Reduce the possibility to use the "design and build" formula to reduce delays.Continued expansion of the bicycle path network treated as an element of the transport system, and not only as tourist attractions.
Ex-post evaluation of the 2007-13 period in respect of energy
Ex-post evaluace programového období 2007-13 v oblasti nakládání s odpady a v oblasti snižování energetické náročnosti, oblast energetických úspor
OP Enterprises and Innovation 2007-2013; OP Environment 2007-2013
SPF Group s.r.o. Bozděchova 99/6 400 01 Ústí nad Labem and Evaluation Advisory CE, s.r.o. Křižíkova 70b, 612 00 Brno
This is an evaluation of the results of the policy measures taken in the 2007-2013 period in respect of energy, which were implemented through the Environment and the Enterprise and Innovation OPs.
To assess the benefits, impact and results of the interventions to improve energy efficiency.
Analysis of documentation;quantitative and qualitative analysis;case studies;data from the monitoring system, other official sources and surveys.
Results of the interventions generally exceeded expectations, with larger energy savings than planned.The reduction in energy losses has led to more efficient management of energy sources in the public sector, which has also allowed the use of environmentally-friendly sources (e.g. use of renewable sources or natural gas rather than fossil fuels).The case studies highlight the fact that companies supported could save on annual energy costs and improve competitiveness.Savings in the cost of building construction had a positive impact on public budgets.
In order to evaluate the net effect of the programme, the initially-planned counterfactual method had to be modified because of problems of data collection.
There is a need to:continue implementing similar measures in 2014-2020;add more monitoring indicators in order to encompass detailed results of the energy saving projects supported;maintain a fully electronic system to manage requests for support.
Mid-term evaluation of the implementation of the Poland-Saxony INTERREG Cooperation Programme, 2021-2027
Ewaluacja śródokresowa wdrażania Programu Współpracy INTERREG Polska-Saksonia 2021-2027
Interreg VI-A Poland-Germany/Saxony 2021-2027
dr Beata Belica, Przemysław Bruhn, Cezary Gołębiowski, Anna Górecka-Ojdana, Roksana Kołodziejczyk, Weronika Zych
The evaluation examines the implementation system of the Poland-Saxony INTERREG Cooperation Programme, in 2021-2027.
To assess the progress, effectiveness and efficiency of the programme and the relevance of the intervention logic.
Analysis of monitoring data and statistical office data.6 interviews with implementing institutions.6 interviews with beneficiaries.CAWI/CATI survey of 72 beneficiaries.4 Workshops.Delphi method.5 case studies.
The implementation of the programme and achievement of its objectives were largely affected by external factors, particularly the energy crisis, high inflation and the impact of the war in Ukraine. This necessitates a review of the target values of the indicators.Although the project selection system allows for projects to be selected that are consistent with the programme logic, the lack of coordination between calls for proposals for different measures was a significant barrier.Interest in the programme is low, also because of the differences in regulations between Poland and Germany, overly complicated procedures, and insufficient resources for project implementation. This led to a small number of projects being submitted.
Update and adjust the target values for certain output and result indicators.Maintain continuous call for proposals in the current financial framework.Increase the scale of promotion of the programme and change the way information about the call schedule and facilities for beneficiaries is communicated.
Operational Evaluation of the País Vasco OP 2007-2013 reprogramming
Evaluación Operativa por Reprogramación del Programa Operativo de País Vasco
Ministerio de Hacienda y Administraciones Publicas
Dirección General de Fondos Comunitarios. Secretaría de Estado de Presupuestos y Gastos
The evaluation is concerned with assessing the financial reprogramming of the OP País Vasco 2007-2013.
The stated objective of the evaluation is to examine the legal justification for the OP País Vasco 2007-2013 reprogramming.
The evaluation assesses the financial reprogramming of the OP using qualitative methods. The evaluation first analyses the legal foundations of the financial reprogramming and secondly whether the proposed changes are in line with the Lisbon and the EU2020 Strategies. Changes in the OP targets are also assessed.The data used are secondary, mainly financial. Reference is also made to the legal basis of the reprogramming.
The main purpose of reprogramming was to include private expenditure in the calculation of eligible national co-financing for the ERDF.The reprogramming also included various adjustments in financial allocation (particularly, an increase to Axis 1 Knowledge Economy and Business Innovation and Development).Some OP targets were also modified.Reprogramming found a legal justification in the Article 33.1 of the 1083/2006 Regulation, which allowed reprogramming (i) “following significant socio-economic changes”; and (ii) “following implementation difficulties”.The proposed financial modifications do not entail a substantial change in the content of the programme and do not entail any effect on the horizontal issues (such as environment and gender). The proposed changes are in line with the Lisbon and the EU2020 Strategies.
No limitation are indicated in the evaluation.
The proposed reprogramming is justified by the significant socio-economic changes that had occurred since the OP was agreed. The proposed adjustments did not entail any significant change in the OP strategy but only a reallocation of funds.
3rd Evaluation of the implementation of the Ipeiros OP, 2014-2020
3η Αξιολόγηση Εφαρμογής του ΕΠ Περιφέρειας Ηπείρου 2014-2020
The evaluation examines the implementation of the OP and in particular the contribution to the "smart specialisation strategy" in the following sectors: agri-food and gastronomy; tourism and culture and creative economy; academic institutions, ICT and youth entrepreneurship; health and well-being. Data used go up to November 2023.
To establish the effectiveness of the measures implemented under the OP, and the coherence with the intervention logic.
Review of OP related documents. Analysis of monitoring data. Analysis of statistical data.
The start of the OP in 2015 was very satisfactory due to the pre-existence of mature projects and, until 2020, there has been a steady acceleration of expenditure with a high degree of resource absorption (currently 94.8%). PA5 (Human resource development, social inclusion) has already met its targets. The other PAs are likely to meet the targets for 2023. Factors that delayed or hindered implementation are changes in the regulatory framework, the public procurement reform, lack of maturity of projects, the long expropriation procedures (for transport infrastructure) and the time required for the implementation of Integrated Territorial Investments. Most PAs show an efficiency indicator slightly above the unity.
There are some mistakes and misalignments between the data recorded in the monitoring system and real values.
Remodulate the OP indicator system to ensure that it coherently reflects the expected results of the measures. Reallocate the resources between PAs to increase effectiveness.
Evaluation of systemic and conceptual projects under the Research, Development and Education OP, 2014-2020, in the Czech Republic – Evaluation area C – 2° interim report
Evaluace systémových a koncepčních projektů z výzev PO 3 OP VVV - Evaluační okruh C, 2. Průběžná zpráva
The evaluation examines the implementation of the “Inclusive and High-Quality Education in Territories with Socially Excluded Localities” project, funded by the Research, Development and Education OP in the Czech Republic.
To assess the satisfaction of recipients and beneficiaries with the implementation of the inclusive education project and the methodological approach for a future impact evaluation.
Compilation of methodological material for the impact evaluation of the project is in progress.Most of the beneficiaries are satisfied with the support received from the Agency for Social Inclusion, though more practical activities are called for.There is some confusion over activities undertaken by different local plans.Most teachers believe that inclusive measures are too demanding and have adverse effects on other pupils.
Assessment of the methodological approach to adopt for impact evaluation is limited because of recent changes in the policy and lack of updated documents.
Describe in more detail the methodological material available for impact evaluation of projects.Create a management and control system for evaluators operating at local level to improve cooperation and evaluation design.
Evaluation of the effectiveness and efficiency of support in the area of social inclusion under the Pomorskie OP, 2014-2020
Ocena skuteczności i efektywności wsparcia w obszarze włączenia społecznego, w Regionalnym Programie Operacyjnym Województwa Pomorskiego na lata 2014-2020
Anna Araminowicz, Sebastian Gralec, Natalia Krygowska-Nowak, Joanna Kwinta-Odrzywołek, Magda Sierżyńska, Katarzyna Stankiewicz, Jakub J. Wróblewski, Agnieszka Zaborowska
EPRD Biuro Polityki Gospodarczej i Rozwoju Regionalnego Sp. z o.o., www.eprd.pl; MABEA sp. z o.o., www.mabea.pl
The evaluation included a comprehensive assessment of support from Priority Axis 6 (social and professional activation, development of social services, support for social economy entities) and Priority Axis 14 (integration of migrants).
To assess the effects of projects addressed to people in poverty or at risk of poverty and social exclusion, as well as to assess the support for social economy entities and the development of social services in the region.
Desk research (programme and project documentation, studies and reports).Analysis of monitoring data and official statistics.CATI/CAWI survey of 3,214 project participants and beneficiaries.24 interviews with representatives of various institutions involved in the implementation of the programme.5 focus groups with beneficiaries and grant recipients.Recommendation workshop.
Targets of indicators on the take-up of employment and improvement of competencies by participants were difficult to achieve, because of the mismatch with the needs and capabilities of participants, i.e. people at risk of poverty or social exclusion.Sustainability of project results is problematic, e.g. in terms of maintaining places of social services, jobs and social economy entities.Integrated projects combining infrastructure aspects and provision of social services are the most successful ones.As the sustainability of social enterprises depends on the demand for their services, supporting the creation of new jobs in such entities in the current market conditions does not seem to be fully suitable.An important factor influencing the quality of support provided is the experience of entities providing social services.Support for the integration of migrants was positively assessed by both project participants and entities involved in the implementation of activities, despite the excessive formalities burdening participants.
Holiday period during survey implementation and difficulties in agreeing dates for individual interviews.
Adapt indicators to the specifics of target groups.Conduct training and informational activities to raise awareness of the importance of applying horizontal principles to sustainable development.Improve the quality of projects in terms of diagnosis of needs, analysis of risks and planning of activities.Place more emphasis on support for social services and reintegration of the most vulnerable people, rather than on supporting the creation of new jobs in the social economy.
Impact evaluation of the Rhône-Saône ERDF OP, 2014-2020 – Lot 2
Evaluation de l’impact des FESI pour le Programme Opérationnel Interrégional FEDER Rhône-Saône 2014-2020 – Lot n°2 POI
The evaluation examines the contribution of the OP to the expected changes in the region.
To assess the usefulness of the measures financed and their contribution to achieving the objectives set, whether targets are likely to be met, the success factors and obstacles, the share of the changes observed attributable to the ESIF and possible improvements in implementation, and to reflect on the significance of the findings for 2021-2027 OP.
Documentation analysis.Data analysis.Interviews.Case studies on 15 completed projects.4 focus groups.Benchmarking.
By July 2019, 58 projects were programmed under IP 5.2 for a total of EUR 3.4 million.The agricultural measures helped 5 Chambers of Agriculture to develop expertise on reducing vulnerability, to create partnerships, and to carry out vulnerability analysis which has so far benefitted 25 farms.The ARAVIR study made it possible to assess the costs of potential damage from floods across the Rhône basin, while the application of the REVITER approach led to the development of a comprehensive vulnerability analysis to define a territorial action plan.By September 2019, 26 projects had been programmed under IP 6.4 amounting to EUR 4.6 million.The target of restoring 400 ha of wetland was exceeded by 200 ha.ERDF funding enabled the network of wetland managers to be relaunched, and the strategy of the Rhône Plan on wetlands to be strengthened.The scientific projects on the River Rhône promoted dialogue and knowledge sharing between managers and scientists and would not have been implemented without ERDF support.By September 2019, 9 projects under IP 7.3 were programmed for EUR 8.4 million.The modernisation of the Arles slipway has helped to maintain the river traffic and related jobs.The modernisation of the canal from Rhône to Sète is delayed.The redesign of the InfoRhône site has enabled the needs of the sailing community to be met and the safety of river navigation to be strengthened.19 projects under IP 6.3 were programmed for a total of EUR 3.8 million.71 km out of 151 km of planned cycle routes had been completed.There is a high level of satisfaction with basic aspects of cycle routes, but dissatisfaction with repair, toilet and parking facilities.The use of the ViaRhôna cycle route increased by 9% in 2015-2017.
It is not possible to make robust estimates of the overall costs needed to reduce the vulnerability to flooding. The final impact in terms of increasing waterway freight traffic remains uncertain and is to be confirmed. The expected effects of marketing the ViaRhôna route cannot be assessed.
Support projects in 2021-2027 which reduce the vulnerability of economic activities to flooding through the establishment of a network of competent technical experts.Deploy a proactive outreach and engagement strategy for local decision-makers to encourage restoration projects in priority wetlands.Accelerate the transition to a greener mode of freight transport.Stimulate improvements in cycle routes through ambitious selection criteria for projects.
Evaluation of performance of the Estonia-Latvia Programme
Performance Evaluation of the Estonia-Latvia Programme
SAFEGE Baltija Ltd, Kaļķu iela 5-1, Riga 1050, Latvia; tel. +37167333090Comperio Ltd., Ogļu street 1, Riga 1048, Latvia; tel. +371 29115356; comperio@comperio.lv; www.comperio.lv
Evaluation Report 2016 on the Competitiveness and Sustainable Development OP
Έκθεση Αξιολόγησης 2016/Ε.Π. Ανταγωνιστικότητα και Αειφόρος Ανάπτυξη
OP Competitiveness and sustainable development 2014-2020
The evaluation assesses the implementation the Competitiveness and Sustainable Development OP 2014-2020.
The aim is to assess the implementation of the OP and the progress made towards achieving the physical and financial indicators.
Desk research.Analysis of monitoring data.
Because of the early stage of the OP’s implementation, only 25% of the funding available has been allocated.
Speed up the OP’s implementation through specific actions such as intensifying checks of project certification to provide information and support to beneficiaries; providing training to MA personnel on evaluation and monitoring.Implement Priority Axis (PA) 2 on promotion and use of informal technologies and communications, PA 4A on protection of the environment and efficient management of resources and PA 4B on natural and cultural resources, as well as issue calls for proposals for the other priority axes.carry out systematic checks to ensure the correct implementation of integrated strategies for sustainable urban development.
Assessment of learning environments in preschools and general and vocational schools in Bulgaria to prepare for 2021-2027
Оценка и препоръки по отношение на образователната среда (ОС) в предучилищните институции, училищата в България, включително училищата, предлагащи професионално образование и обучение (ПОО)
OP Science and Education for Smart Growth 2014-2020; Education BG ESF+ 2021-2027
Husein Abdul-Hamid, Dessislava Kuznetsova, Maria Ustinova, Syedah Aroob Iqbal, Kristina Nikolova, and Borislav Zlatkov
World Bank, Bulgaria Country Office, Advance Business Center, 2, Samara Street, 1715, Sofia, www.woldbank.org/bulgaria, info@worldbank.bg
This is a study of the quality of schools in Bulgaria ahead of 2021-2027.
To assess the learning environment in schools, educational needs and the socio-economic context.
Official statistics.Official documents and studies on the education system in Bulgaria.Survey of school heads.
Despite recent increases in the investment in school infrastructure, this is still fragmented between the various programmes and there is no uniform long-term goal.Education stakeholders have developed capacity in innovative education practices through investment in the learning environment which could be used as a model for implementing successful policies.There is a lack of targeted monitoring of the effects of investment in the learning environment on the quality of education and outcomes.Preschools receive proportionally less funding nationally than other types of school.
Define a comprehensive medium-term plan for the learning environment, establishing basic standards and categorising schools in order to prioritise investment.Ensure monitoring of results, defining quality indicators to track the results of investment in addition to the quantitative ones on outputs which already exist.Make preschools eligible for funding for infrastructure from national and EU programmes, to provide an appropriate learning environment from an early age.
Evaluation of Communication strategy of the ERDF Emilia Romagna OP, 2014-2020
Rapporto Tematico: Strategia di Comunicazione del POR FESR
Evaluation of support for the unemployed under the Veneto ESF OP, 2014-2020
Approfondimento tematico “Il sistema di presa in carico delle persone senza lavoro”
IRS, https://www.irsonline.it/ - PTSCLAS, https://www.ptsclas.com/en/home-en/
The evaluation examines the measures for jobseekers financed under the OP, up to September 2021.
To identify the positive and critical aspects of measures up to September 2021 , and areas for improvement.
Desk analysis of relevant regional policies.Analysis of procedures.Interviews with the main bodies involved in active labour policies.Interviews with relevant PES staff.Analysis of monitoring data.
Overall, the system is capable of responding to the needs of recipients, but there is a lack of policies specifically focussed on vulnerable groups (migrants and older people).Accredited labour market agencies have strategic importance in implementing ESF measures.Positive aspects of the system of active labour policies are a customised approach, its multidimensional nature and collaboration between public and private agencies.Up to September 2021, 77% of recipients of “Assegno per il Lavoro” (voucher for personalised assistance) obtained a new employment contract.The employment rate of participants in work experience programmes is 41% 12 months afterwards and 25% in the case of employment pathways for workers at risk of unemployment.Participants in “Integrated measures for territorial cohesion” (AICT) have an employment rate of 31% 12 months after, while the rate is lower for public utility jobs.Participants in the “Work with a social Impact” scheme have an employment rate of 61% 12 months after.As of June 2021, 60% of participants in Youth Guarantee measures obtained fixed-term or apprenticeship contracts.
Make interaction with businesses more systematic and continuous.Continue to develop means of analysing the local employment system and forecasting skill requirements.Improve career guidance for both young people and adults.Continue to develop the use of digital services.Make cooperation between public and private agencies a structural element.Ensure integration between training and other active labour market policies.Ensure complementarity with other regional, national and EU funding.
Ex-ante study of the use of financial instruments under SO 5.1 of the Śląskie Programme, 2021-2027
Ocena ex ante zastosowania instrumentów finansowych w ramach priorytetu inwestycyjnego 6e Regionalnego Programu Województwa Ślskiego 2014-2020
Jan Frankowski, Henryk Kalinowski, Magdalena Ośka, Andrzej Regulski
Imapp Consulting Sp. z o.o., www. imapp.consulting.eu
The study identifies the possibilities of using financial instruments in respect of energy efficiency in the 2021-2027 programme.
To assess the appropriateness of using financial instruments for the energy modernisation of single-family buildings.
Desk research.21 interviews with implementing bodies, financial intermediaries, experts and local government units.CAWI survey of 167 local government units.Benchmarking analysis.Focus group.
The study indicates the existence of market failure and a suboptimal level of investment as regards energy efficiency.Implementation of a financial instrument could reduce the scale of investment needs and bring benefits to the region.Use of this instrument in the 2014-2020 period is too risky because of the limited time, limited range of support and the existence of other similar instruments.
Plan a financial instrument for energy modernisation of single-family buildings in the next period.Strengthen the potential of regional financial institutions to implement financial instruments for energy investment.
Impact evaluation of support for the unemployed under the Emilia Romagna ESF OP, 2014-2020
Tema 2 - Efficacia delle politiche formative e delle politiche attive del lavoro per sostenere l’inserimento e il reinserimento nel mercato del lavoro contrastando la disoccupazione di lunga durata – Valutazione di impatto controfattuale
Iris, www.irisricerche.it – Performer, www.performer.it
The evaluation examines the impact of training and traineeships for the unemployed implemented under the OP between 2015 and 2018.
To assess the effects of training and traineeships on the employment of those who participated between 2015 and 2018.
Analysis of monitoring data.Counterfactual approach (PSM, using administrative data from PES and “Comunicazioni obbligatorie” – COB).
After the training (between 2018 and March 2020), participants are less likely to have worked than the control group (69% vs. 81%).The more disadvantaged groups in employment terms are women and older people and those participating in short courses.
Evaluation of conditions of investment success and optimisation of Sub-measure II.1.1, Investment areas, in the Łódzkie OP
Wsparcie terenów inwestycyjnych w województwie łódzkim - warunki powodzenia inwestycji i optymalizacja systemu wdrażania Poddziałania II.1.1 Tereny inwestycyjne RPO WŁ 2014-2020
Barbara Leszczyńska, Oliwia Samołyk, Jan Wałaszek
Danae, ul. Szarotki 10/16, 02-609 Warszawa, Realizacja, al. KEN 93, 02-777 Warszawa
The evaluation assesses the investments funded by the Łódzkie OP 2014-2020 under IP 3a - promoting entrepreneurship and more specifically, Sub-measure II.1.1, financing projects carried out by Territorial Self-Government units (TSUs).
To identify reasons for the small number of project submissions under Sub-measure II.1.1 and to assess the selection criteria used and the functioning of the selection system.
Desk research;CAWI/CATI with a sample of 146 local government units (74% of total approached);interviews with 6 representatives of institutions supporting the development of investment areas and 10 representatives of SMEs ;case study of activities to support the development of investment areas;panel of experts.
Many TSUs reported feeling discouraged to apply for a proposal because the selection criteria are too rigid.The selection process is excessively long, which is not in line with the needs of investors and enterprises, which usually look for areas ready for investment, while the logic of the Sub-measure requires that TSUs find investors able to wait several months.Other obstacles for TSUs include a lack of investment areas close to them, no interest of investors or enterprises in their locality, and a lack of finance for the construction of access roads to the investment areas.There is a risk that the total allocation of funding will not be completely used. However, the potential interest of TSUs is considered to be higher than currently shown and a change in the project selection system could attract more applicants. In fact, 18% of municipalities participating in the study considered submitting an application for financing and 31% would like to submit one in the future.
There is a need to:review some eligibility and selection criteria (e.g. those requiring a minimum of 2 investment projects for the local institutions which want to participate and the minimum size of the investment area to be 3 ha);permit SMEs to submit applications;inform TSUs of the possibility of obtaining funding for the construction of roads to access the investment areas from other support provided by the OP.
Evaluation of the impact of general and vocational education support under the Dolnośląskie OP, 2014-2020
Ocena wpływu interwencji RPO WD 2014-2020 w zakresie wsparcia kształcenia ogólnego i zawodowego
Krzysztof Gutta, Beata Belica, Katherine Gutta, Emilia Karewicz
The evaluation examines measures on ​​education implemented under the OP.
To assess the relevance, durability, usability and effectiveness of support for general and vocational education in relation to the needs of participants up to 2022.
Analysis of monitoring and official statistical data.13 Individual Interviews.6 Focus Group Interviews.10 Case studies.Survey of 1135 participants.1 expert panel.
50% to 85% of projects would not have been carried out without EU funding.73% of recipients of "soft" support report improved competences and professional skills to a large or very large extent. 83% reported that support was tailored to their individual needs.Almost all result and output targets have already been achieved.Beneficiaries consider the effects of support to be durable, for at least two years.
Continue support for infrastructure and soft projects in 2021-2027.Focus future support on enabling exchange of experience between teachers, with the participation of experts and, for vocational training, enterprises.Include the possibility of providing ICT-equipped facilities and training on digital competences for students.
Guadeloupe Conseil Régional OP – Evaluation of the YEI in 2015
Évaluation de l’impact du programme européen « Initiative pour l’emploi des jeunes » en 2015 - Guadeloupe
ROP Guadeloupe Conseil Régional 2014-2020
The evaluation assesses progress in implementing the YEI in Guadeloupe in 2015, complementing the national YEI evaluation 2015.
EUR 22 million of YEI and ESF has been allocated in Guadeloupe (EUR 10.6 million from the regional OP and EUR 8.6 million from the regional component of the national OP). This corresponds to around EUR 1,600 for each NEET identified in 2014 (14,000);overall, 93% the resources available have been allocated (the same for both OPs);by the end of 2015, around 600 people had started national OP activities;out of a 28% sample of participants supported by the national OP, 73% were employed or had started training, while 14% had withdrawn from the programme.
There is a need to:produce a detailed territorial analysis of NEET needs in the region, of the measures provided and their shortcomings;collect more detailed data through the monitoring system (e.g. the types of activity undertaken by each beneficiary) in light of the counterfactual assessment foreseen in 2018;monitor the actual use of resources, in particular on critical activities (local activities of the regional OP).
Evaluation of the 'Reduction of CO2 emissions in specific urban areas' under the Bremen ERDF OP, 2014-2020
Evaluierung der Förderung zur "Senkung der CO²-Emmissionen in bestimmten städtischen Gebieten"
Carla Harnischfeger, Christine Nettersheim
Ramboll Management Consulting, Jürgen-Töpfer-Straße 4822763 Hamburg, carla.harnischfeger@ramboll.com
The evaluation examines the results of measures for the reduction of CO2 emissions in urban areas under PA3 of the OP up to October 2022.
To assess the achievement of the objectives of measures for reducing CO2 as at October 2022 and to make suggestions for the future.
Documentation analysis.Analysis of monitoring data.Expert interviews (specialist units and IBs).Interviews with beneficiaries.
Both the financial and physical implementation of the measures are progressing well.Administrative requirements relating to the energy renovation of buildings are considered burdensome.Values of the output indicator “Estimated annual decrease in greenhouse gas emissions” are not aligned with targets which do not seem to be calculated correctly. In addition to reducing CO2 emissions, projects have also supported awareness raising and competence building.The two-step structure of funding has not proved effective.
In the future, ensure a transparent and documented calculation of indicator targets.Set up a uniform methodological basis for determining CO2 reduction for all applications.
Impact assessment of cohesion policy on selected macroeconomic indicators in Poland, 2007-2013 and 2014-2020
imapp consulting sp. z o.o, ul. Puławska 145, 02-715 Warszawa, www.imapp.consulting; Fundacja Naukowa Instytut Badań Strukturalnych, ul. Wiśniowa 40B, 02-516 Warszawa, www.ibs.org.pl
Study of the impact of cohesion policy funding on selected macroeconomic indicators, from 2004 to now and in the programming period 2021-2027.
To assess the impact of cohesion policy funding on the macroeconomic indicators at national and regional level.
In 2019, GDP is estimated to have been 5% higher than without EU funding, the impact of which has been increasing since 2017, along with progress in spending the 2014-2020 funds.EU funding has led to an estimated additional 445 thousand being employed, increasing the employment rate by 2.0 pp, and to an estimated 3.1 pp in 2023. The unemployment rate is estimated to have been reduced by 0.9 pp.Estimates are that per capita real income in 2020 was increased relative to the EU average by 2.8 pp, and the Gini coefficient reduced by 2.7 and the absolute and relative poverty rates by 1.8 pp and 2.8 pp, respectively.EU funds stimulated economic growth in the poorest regions in particular, especially in Eastern Poland regions and Opolskie.
First report on environmental assessment of the Valle d’Aosta ERDF OP, 2014-2020
Primo rapporto tematico di valutazione ambientale
This report is the environmental assessment of the Valle d'Aosta OP, 2014-20.
To assess the contribution of the OP to environmental issues, to update the analysis of the regional environmental context and to analyse the integration of environmental issues in the programming and implementation of the OP and the potential contribution of projects financed to the sustainable development goals.
Desk analysis of programming and implementation documents.Analysis of official statistics.
The updating of the analysis of the territorial environmental context confirms the relevance of the regional strategy.The principles of sustainable development have been integrated into PA 4, which includes measures for energy efficiency and sustainable smart mobility, and Specific Action 6.3.8, which finances initiatives for protecting and supporting natural and cultural resource and environmental objectives. The other OP measures contribute indirectly to sustainable development goals.The project selection procedures incorporate the principle of sustainable development through the use of specific evaluation and reward criteria.
Integrate environmental issues more into initiatives to support business competitiveness.Design and implement an environmental monitoring system to measure the impact of projects financed by the OP.
Evaluation of the training needs of judicial staff under the Knowledge Education Growth OP, 2014-2020
Analiza potrzeb szkoleniowych pracowników wymiaru sprawiedliwości – określenie potrzeb szkoleniowych pracowników wykonujących czynności administracyjne oraz pracowników jednostek merytorycznych w jednostkach organizacyjnych wymiaru sprawiedliwości (sądach i prokuraturach)
The study examines the training needs of staff performing administrative activities and of courts and prosecutor offices.
To collect data on training needs in respect of "soft" and "hard" skills required by judicial staff.
Desk research.Survey of 4,362 courts and prosecutor office staff.Survey of 307 clients of courts and prosecutor offices.10 interviews with court and prosecutors office staff.4 focus groups with court and prosecutor office staff.Expert panel.
There is no systematic training tailored to legal and procedural changes in place and there is a need to develop both hard and soft skills of judicial staff. The main barriers to accessing to training are the insufficient number of training courses, the distance from training places and the burden of day-to-day duties. In the case of commercial trainings, price is also a barrier.Managers and directors need assistance to prepare them to manage units and develop managerial skills.
Develop training programmes in substantive knowledge and soft skills, including for managerial staff.Develop the training available using modern communication methods, such as teleconferences/webinars.Implement training activities before introducing legislative changes to prepare staff for them.
Evaluation of the support provided to innovation centres in 2007-2013
Ocena efektów wsparcia ośrodków inNowacyjNości w ramach działania 5.3 Wspieranie ośrodków inNowacyjNości PO IG oraz ich potencjału w świadczeniu usług proinNowacyjnych
Jarosław Chojecki Paweł , Paweł Huras, Magda Matysiak, Adam Miller, Sebastian Pałka, Cezary Przyby,ł Agata Rauzer, Patrycja Rumińska
WYG PSDB Sp. z o.o. ul. Bitwy Warszawskiej 1920 r.7 02-366 Warszawa, Evalu Sp. z o.o. ul. Dzika 19/23 lok. 55 00-172 Warszawa
The study is aimed at assessing the effects of an ERDF co-financed measure called ‘Supporting innovation centres’ (known as Measure 5.3) which was part of the Innovation Economy OP during the 2007-2013 programming period.This measure was designed to tackle the issues of insufficient innovation centres and the associated services.It falls under priority 5 of the OP (Diffusion of innovation) aimed at providing enterprises with high quality services and infrastructure to strengthen their innovative potential and the competitive position of enterprises through cooperation.
The objective is to assess the effects of support to innovation centres under measure 5.3. The evaluation was conducted between September and December 2016.
The study used a range of qualitative and quantitative methods:Desk research.In-depth interviews with representatives of institutions involved in the implementation of the measure.Case studies of projects supported.CAWI/CATI surveys of customers of innovation centres.Telephone interviews with potential customers of innovation centres.
Support was provided to 12 technology parks located in 7 provinces.In 2015, there were 828 tenants and 1,760 external customers.Investment was mostly made in large provincial cities.All projects involved investment in infrastructure and facilities.Beneficiaries are mainly commercial companies, either long-established or newly-created ones.Projects mainly took the form of office spaces and laboratories.A wide range of services was provided.There is a strong focus on support to start-ups.Virtually all parks cooperate with scientific or research institutes.The measure indirectly contributed to the creation of additional jobs both (in the parks and outside) and the revitalisation of urban areas.According to 70% of enterprises interviewed cooperation with the park has led to increased growth and innovation, for over 50%, cooperation has resulted in initiating or intensifying company R&D, and for 30%, establishing or strengthening cooperation with science centres.Full assessment of the support will not be possible for two years.
The biggest challenge was the application of the Public Procurement Law and ensuring fair competition and transparency in spending.Changing legislation also led to difficulties in implementation (such as from changes in fire regulations). Significant delays in the implementation were also due to financial problems of major contractors (construction works).
To take into account the specific features of the activities of individual innovation centres.To establish an electronic database on the functioning of the parks.To determine periodically whether current or planned activities threaten the sustainability of the project.To create a system to control the sustainability of projects.To create a web platform to present what technology parks in the country can provide.
Evaluation of measures supporting start-ups funded by the Thüringen ESF OP 2014-2020
Evaluierung der Gründerrichtlinie Teil A: Existenzgründungsberatung und Gründerprämien im Rahmen der Förderung durch den Europäischen Sozialfonds (ESF) im Freistaat Thüringen in der Förderperiode 2014 bis 2020
Brumm Angela, Fritsch Michael, Schiemann Frank, Walter Gerd
Institute for Social Economic Structural Analysis (SÖSTRA GmbH), info@soestra.de
The evaluation examines the measures to support start-ups funded by the OP, through intensive consulting, start-up advice and training through start-up passports, promotion of networking, and start-ups premiums for innovative projects.
To assess the coherence, effectiveness and effects of the measures supported, including consulting, advice, networking and start-up premiums.
Desk reviewAnalysis of monitoring dataIn-depth interviews with expertsFour case studies of consulting and networking projectsThree rounds of surveys of participants to the following measure: start-up advice and qualifications through start-up passports, intensive consulting, start-up premiums (1,822 people in the first round in December 2018, 1,048 in the second in September/October 2019 and 122 in the third in October/November 2020).
Overall 9,500 people were supported through different measures by end-2020: the largest number through start-up advice and start-up passports (42%) and networking projects (52%). Of these, 47% were women and 12% had a migrant background. Almost a third of participants lived in the cities of Erfurt, Weimar and Jena.59 start-up premiums were granted.12% of participants were self-employed before the support and 56% six months after the end of support (71% for those participating in start-up passports).92% of those surveyed generated sales shortly after the start of the business and 78% made a profit, though these declined between the second and third survey because of coronavirus pandemic restrictions. Hospitality, arts, entertainment and recreation were the most affected sectors.35% of those surveyed had recruited staff (most on employment contracts) and 20% planned to do so but had delayed because of the pandemic.
Due to the coronavirus pandemic, a further round of survey – originally planned for spring 2020 – did not take place.
Continue the support in the next programming period.Simplify the funding conditions and make them more flexible.Consider to make the entry requirements for the start-up premium more flexible.Extend the duration of funding.Ensure more transparency in the way funding opportunities are advised.
Operational Evaluation of the Murcia OP reprogramming
Informe de Evaluación operativa por reprogramación del Programa Operativo FEDER de Murcia 2014-2020
Region of Murcia. Ministry of Finance and Public Administration. Directorate-General for Budgets and European Funds
The evaluation assesses the financial and technical reprogramming of the Murcia OP for 2014-2020.
To examine the financial and technical reprogramming and to assess whether it is coherent with the initial OP strategy.
A combination of qualitative and quantitative methods based on secondary data retrieved from the Fondos2020 accounting system and official documents (e.g. yearly implementation report, Mid-term evaluation, etc.).
The main purpose of reprogramming was to include an increase in ERDF support (EUR 37.1 million) coming from technical revisions.The reprogramming was concerned with increasing finance for Business investment in R&I (TO1), ITC (TO2), energy efficiency (TO4) and water treatment (TO6).Some output and outcome indicators as well as some of the targets were changed.The strategy remains in line with the initial OP objectives though there was a slight change in the intervention logic due to the introduction of energy efficiency measures.
Since data on certified expenditure were not available, the financial analysis is based on actual spending (which, at 31 December 2016, represent 11% of total funding).
Evaluation of the effects of transport measures under the Małopolskie OP, 2014-2020
Ewaluacja wpływu projektów zrealizowanych w ramach RPO WM na lata 2014-2020 na budowę efektywnego i nowoczesnego systemu transportowego w Małopolsce
The evaluation examines how support for investment in infrastructure has contributed to the modernisation of the transport system in the region.
To assess socio-economic effects of transport projects, the enabling and hindering implementation factors and the complementarity of support with other investment.
Analysis of monitoring data and official statistics.7 interviews with experts and institutions responsible for programming and implementation.30 interviews with beneficiaries.Spatial analysis.6 case studies.Benchmarking analysis of 3 regions.Evaluation workshop.
There are delays in project implementation, often because of increased costs. Fragmented and uncoordinated railway investment and incoherent cycle routes are bottlenecks.The external accessibility of the region seems to have improved, but the effects of projects on internal accessibility are limited to big cities.Projects intended to facilitate travel are often ineffective.
Train applicants and beneficiaries in project scheduling and risk planning.Apply bonus criteria to projects integrating modes of transport and promoting multimodal transport.Promote projects that are complementary to the current network.
Evaluation of the project selection system under the European Funds for Lubuskie Programme, 2021-2027
Ewaluacja kryteriów oraz systemu wyboru projektów w ramach Programu Fundusze Europejskie dla Lubuskiego 2021-2027
Polska Agencja Ewaluacji Sektora Publicznego S.A.
The subject of the evaluation is the project selection system adopted under the 2021-2027 programme.
To assess the consistency of the criteria and project selection system with the objectives of the programme.
Desk analysis of relevant documents.Analysis of monitoring data and official statistics.18 interviews with implementing institutions.CAWI/CATI survey of 66 beneficiaries.SWOT Analysis.2 focus group interviews.
The project selection criteria effectively support the identification of those projects that contribute most to the achievement of the programme objectives.The project selection system supports the implementation of the programme strategic objectives, such as improving inclusion, innovation and protecting the environment.The achievement of key social and economic objectives is pursued by prioritising comprehensive projects and those targeting specific groups, including people with disabilities.
Increase the level of competence of applicants and other user groups in relation to the IT system for applying.Strengthen support for less experienced beneficiaries to increase their chances of success in the application process.
Evaluation of the effects of regional health programmes implemented under the Podlaskie OP, 2014-2020
Ocena efektów wsparcia Regionalnych Programów Polityki Zdrowotnej wdrażanych w województwie podlaskim
The evaluation analyses the effects of the implementation of the Regional Health Policy Programme financed under the OP.
To assess the effectiveness, usefulness and sustainability of Regional Health Policy Programmes financed by the OP.
Desk analysis of relevant documents and monitoring data.CAWI/CATI survey of participants in the programme (n=1,203).7 interviews with beneficiaries.5 interviews with experts.Evaluation workshops.
The programmes have been instrumental in raising awareness of the importance of prevention and treatment for maintaining good health conditions.The programmes have improved access to health services in Podlaskie.The main barriers to implementing the programmes were difficulties in recruiting participants because of external factors (COVID-19, price increases, need to obtain a wide range of personal information).
Simplify the process of recruiting participants, particularly by limiting the amount of personal data collected.Shorten the implementation period to reduce the negative impact of external factors that cannot be predicted at the planning stage.Continue supporting health programmes in the next programming period.
Study on the monitoring system and the macroeconomic model used under the EU Funds’ Investments Programme, 2021-2027, in Lithuania
Ekonomikos ir inovacijų ministerijos kompetencijai priskirtų iš Europos Sąjungos fondų lėšų bendrai finansuojamų ekonomikos sektorių būklės pokyčių vertinimas (3 etapas)
BGI Consulting - http://www.bgiconsulting.lt/, jonas@bgiconsulting.lt
The study examines the monitoring system and the macroeconomic model adopted under the 2021-2027 Programme, with a view to including the digitisation sector into them.
To identify appropriate physical and financial indicators to plan measures under the programme, to update the macroeconomic model used, to include the digitalisation of the economy and to assess the development of digitalisation in Lithuania from 2000 up to 2022.
Analysis of the monitoring system. Literature review. Analysis of statistical data on digitalisation. Macroeconomic modelling (HERLIT model).
Most of the main indicators on the use of ICT in Lithuania have increased substantially over the past 20 years and the values achieved are often above the EU average. The “share of enterprises selling via computer networks” and the “value added of computer-related activities” are the indicators to be integrated into the monitoring system of the Programme and in the macroeconomic model used for assessing the impact of EU funds. Investments in energy and R&I are expected to have the largest impact on GDP and job creation.
Collect additional data on the digital transformation (e.g., data on non-use of digital technologies by the population). Ensure the use of the HERLIT model by other Ministries other than that for Economy. Include only large-scale investments in the macro modelling. Collaborate with the Innovation Agency, to ensure the selection of the digitalisation projects with the highest return on investment.
The impact of ESI Funds in 2014-2020 and the priorities for 2021-2027
2014-2020 m. ES fondų investicijų poveikio Lietuvos ūkiui ir plėtros prioritetų 2021-2027 vertinimas
ESTEP VilniusAdress: Olimpiečių g. 1A-26, LT-09200 VilniusPhone number: 85269 0120Fax: 85269 0124info@estep.lthttp://www.estep.lt
The report examines the future impact of the ESI funds for the 2014-2020 period and sets out proposals for investment for the 2021-2027 period. Stakeholders of the programme were involved in the evaluation process.The overall effect on the Lithuanian economy is measured in terms of value-added, jobs, wages, consumption and production, export prices and other macroeconomic and sectoral indicators.
To assess the impact of the 2014-2020 ESI funds on the main macroeconomic indicators and different sectors;to define development trends in Lithuania after 2020 and to identify long-term priorities and possible areas of investment.
Econometric modelling, relevant literature review, analysis of secondary data from EUROSTAT, national statistics and Ministry of Finance database, OP monitoring data, interviews and discussion with experts.
The development of the Lithuanian economy and convergence of GDP per head towards the EU average is strongly affected by the investment co-financed by the ESI funds.GDP growth will be an estimated 0.9 percentage points higher a year as a result of the ESI funds.Increased investment will have a positive effect on the labour productivity growth.The impact of the ESI Funds on the trade balance will be negative, because it will increase imports more than exports.Lithuania will effectively return over half of the ESI Funds received in 2014-2020 to the net-contributor countries in the form of increased imports.Implementation of the current OP will increase the economic disparities between leading and lagging Lithuanian regions.Analysis for the post-2020 period shows that, in order to be competitive, Lithuania must continue to invest in energy, transport and information infrastructure and in healthcare and basic education.
To stimulate convergence between regions, financial transfers must be linked to structural reforms;there is a need to identify the specialisation of each Lithuanian county;the effectiveness and efficiency of the Lithuanian innovation system (training and lifelong learning, technological innovation) needs to be improved;there is a need to improve the efficiency with which EU funds are managed.
Operational Evaluation of the OP Canarias reprogramming
Evaluación Operativa por Reprogramación del Programa Operativo de Canarias
Mid-term evaluation of Priority 2 under the Regional South-East ERDF Programme, 2021-2027, in Romania
Evaluarea intermediara timpurie a Prioritatii 2 a Programului Regional Sud-Est
Liliana Lucaciu, Monica Roman, Alina David, Irina Stăvilă, Ana Miruna Toma, Roberta Vancea, Laura Voinea, Aurel Rizescu, Elena Bacale
Civitta Strategy & Consulting SA, info.ro@civitta.com; LIDEEA Development Actions srl, office@lideea.eu
This mid-term evaluation examines the effectiveness of the implementation mechanisms of five strands of interventions carried out under Priority 2 of the Programme, which aimed at energy efficiency, climate change adaptation, and the protection of biodiversity. This report examines data up to December 31, 2024.​
To assess the extent to which the implementation mechanisms of the actions related to improving the energy efficiency of public and residential buildings, strengthening of public buildings vulnerable to seismic risk and to the development of green infrastructure in Natura 2000, contributed to achieving the Programme’s objectives, as well as their strengths and vulnerabilities.
Desk research. Interviews with the MA and beneficiaries. Reconstruction of the logic of intervention. Analysis of monitoring and statistical data. Online workshop with 12 representatives of consultancy firms involved in the preparation of the applications. Online workshop with 5 beneficiaries. Focus group with 13 representatives from the MA. Survey with beneficiaries from 48 target entities across various administrative levels and institutions. Panel involving 4 experts with experience in regional development and urban mobility. Case studies.
Progress under Priority 2 is slow and uneven, due to delays in launching the calls and the slow implementation of projects. SO2.1 has the highest contracted value (over EUR 83 million, covering 75% of the allocated resources), but declared expenditure is extremely low (EUR 387,000), due to delays in implementation of complex large infrastructure projects. SO2.4 has seen strong demand for seismic risk reduction measures (39 projects submitted), but low interest in forest curtains, for which only one project has been contracted so far. SO2.7 has delays and no signed contracts yet, due to project complexity. The effectiveness of project implementation varies by ATU (Administrative Territorial Unit) size, with smaller units facing capacity gaps and relying on uneven quality of external consultancy. Priority 2 supports Romania’s National Energy and Climate Plan (NECP) through energy efficiency investments, but its impact remains limited due to the early stage of implementation and low beneficiary awareness of environmental commitments.
The short timeframe of the evaluation (three months) constrained data collection and analysis. Moreover, since most projects were either phased from the previous period or still in early stages, the analysis was limited to contracted beneficiaries.
Accelerate project implementation by introducing contractual penalties for implementation delays. Reorient SO2.4 funding toward climate risk prevention, given low uptake for forest curtains and urgent flood protection needs. Improve monitoring of SO2.7, mobilising key stakeholders to advance Natura 2000 site preparation. Build administrative capacity, especially in small and medium urban areas, through targeted training and technical support. Speed up evaluation and project’s selection and contracting by adding resources during peak periods and by creating dedicated teams, including energy efficiency experts.
Evaluation of communication activites under the ESF OP, 2014-2020
Rapporto di valutazione della Strategia di comunicazione
Efficiency of the ESF and YEI OPs: first results - Employment and Social Inclusion OP, OP for the implementation of YEI in mainland France and outermost regions, 2014-2020
L’efficience des opérations des PON FSE et IEJ : analyses, 1ers résultats et éléments de méthode
The evaluation reports the first results of the ESF and YEI OPs in France as at the end of 2016.
To review the average cost per participant of operations co-financed by the ESF and the YEI.
Analysis of data from the monitoring system.
The costs per participant between 2014 and mid-2016 are higher than those assumed when determining the targets initially included in the OPs.The gap is particularly wide for activities to combat early school leaving.At this stage, only some figures relating to cost effectiveness have been examined, further analysis being delayed until more information is available.
The analysis focused mainly on cost per participant and future reports are intended to examine cost effectiveness.
There is a need to:consider adjusting the 2014-2020 objectives;combine quantitative analyses with more qualitative data to better examine some hypotheses.
Annual evaluation report of the Toscana ERDF OP, 2014-2020 - 2020
Rapporto di valutazione annuale connesso alla sorveglianza - anno 2020
The evaluation examines the implementation of the OP up to December 2019.
To assess the achievements and to identify strengths and weaknesses in implementing the OP to inform the preparation of the 2021-2027 programmes.
Monitoring data.Interviews with those responsible for implementation.Desk analysis of relevant documents relating to 2021-2027.
PA1 (R&D) and PA 3 (SMEs) show a good level of progress, apart from the PA3 result indicators. Positive factors are the timely launch of calls; use of different types of selection procedures and experience in managing financial instruments.In 2019 there was an acceleration in the implementation of urban projects under PA6, though progress was still below the OP average.Progress of PA 2 (broadband) and PA4 (low carbon) is slow, especially in respect of infrastructure investment.A relatively large proportion of beneficiaries withdrew projects, a phenomenon which needs monitoring given the potential adverse effects of Covid-19.The current strategy of the OP is consistent with the 2021-2027 period, with the partial exception of ecological transition, for which major changes are needed.
Improve the monitoring system in order to collect more detailed information and improve coding in the system.
Evaluation of the Integrated Territorial interventions under the Peloponissos OP, 2014-2020 – 3rd report
Αξιολόγηση των Ολοκληρωμένων Χωρικών Παρεμβάσεων του Ε.Π. Πελοποννήσου 2014-2020 - 3ο παραδοτέο
Solonos 45, Kolonaki, Athens, 10672, +30 210 3007240, info@e-ron.gr, https://e-ron.gr/
The evaluation examines territorial interventions in the Municipalities of Kalamata, Argos-Mycenae and Nafplion, and in the Arcadia region, up to July 2020.
To examine the extent to which the objectives of the territorial strategies were achieved up to July 2020 and to propose improvements in implementation.
Analysis of project and programme documents.Analysis of monitoring data.
The level of implementation of the four approved territorial strategies was only partly satisfactory up to July 2020.There is a need to promote and implement territorial measures in Kórinthos.The proposal to include the "Cultural Path for the Towers of Mani" in the Action Plan for the Mani area seems to be useful to promote its particular cultural, historical and architectural character .The Taygetos LEU scheme for the Mani area will effectively integrate the measures for the region’s mountain areas through strengthening the ecosystem.
The evaluation of the efficiency of the measures is not reliable because of the small number of projects contracted as well as the difficulty of comparing them with similar operations elsewhere.
Ensure broad synergy between the Peloponnese Region, the Ministry of Culture and local organisations with respect to the Morea Cultural Route.Include hard and soft measures on both the functionality of infrastructure and the design of the communication strategy in the Mani area.Ensure the participation of entrepreneurs in the Mani ITI, through creating a "cluster" on the basis of a Local Quality Pact.
Evaluation of the implementation of the Research, Development and Education OP, 2014-2020, in the Czech Republic – 7th progress report
Průběžná evaluace implementace Operačního programu Výzkum, vývoj a vzdělávání, průběžná zpráva 07
This is the seventh progress report on the implementation of the OP, reviewing progress up to end-June 2019.
To assess the pursuit of objectives of the OP, to identify the factors affecting its implementation and to formulate proposals for overcoming these.
Documentation analysis.Analysis of monitoring data.Focus groups with university staff.Individual and group interviews with the implementer of the Digital Education Strategy (DES) and projects.Survey of higher education institutes (response rate 9%).
Out of 156 result and output indicator targets, 85 are expected to be exceeded and 36 risk not being met.2018 milestones and the N+3 rule for 2019 have been met.84% of funds have been committed.University staff are generally satisfied with the training courses and the infrastructure developed, but they take part mainly in short-term international cooperation (e.g. conferences) rather than long-term projects.The OP helps to fulfil the DES objectives, mainly by developing digital literacy and by making it possible to acquire the equipment needed to integrate new technologies in the classroom and by providing support in using them.
Revise the indicator targets of all PAs.
Evaluation of support for the protection of the cultural heritage and development of cultural resources under the Infrastructure and Environment OP, 2014-2020, in Poland
Ewaluacja podsumowująca postęp rzeczowy i rezultaty VIII osi POIiŚ 2014-2020 "Ochrona dziedzictwa kulturowego i rozwój zasobów kultury"
Izabela Franckiewicz-Olczak, Emilia Pach, Aleksandra Podkońska, Wojciech Roszka
Biuro Badań Społecznych Question Mark sp. z o.o, questionmark.pl
The evaluation examines the level of implementation of the measures concerned under the OP and the achievement of targets.
To assess the effectiveness of implementation, to identify possible changes needed to targets and means of improving the effectiveness of measures.
Analysis of statistical data.Survey of 117 beneficiaries.12 interviews with beneficiaries and three with relevant institutes.5 case studies.Expert panel.
The achievement of final result indicator targets is uncertain, partly because of COVID and the economic situation in Poland.Establishing cooperation with local entities is an essential element in improving the effectiveness of beneficiary activities.The interest of potential beneficiaries in the measures was strengthened by nationwide promotional events organised by EU funded institutions.
Introduce intensified and systematic monitoring of OP indicators to collect up-to-date data on the state of implementation.Make sure that beneficiaries consider the needs and potential of the local community, especially of the third sector, when formulating their projects.Continue promotional activities at the national level to increase participation in the measures.
Demand and supply effects of Cohesion policy, 2007-2013
A fejlesztéspolitika keresleti és kínálati hatásai
Klára Major, Gábor Balás, Bálint Herczeg, Kata Bördős, Gábor Kiss
The evaluation assesses the effects of the Structural and Cohesion Fund programmes for the 2007-2013 period.
To assess the macroeconomic impact of the 2007-2013 programmes.
The evaluation is confined to the use of quantitative methods:analysis of secondary data;macroeconomic impact analysis through a multi-sectoral CGE model;sensitivity tests;The data used are: secondary data from the monitoring database and from the Central Statistical Office.
EU funds are estimated to have increased GDP by 0.7% in 2008 and 5.2% in 2015 and the employment rate by 0.7 of a percentage point in 2008 and 2.3 percentage points in 2015;the positive effects on GDP and employment come from both the demand and supply side of the economy;EU funds are estimated to have increased capital stock by 1% of GDP in 2008 and 2.5% in 2015, particularly in (public administration, health, education and social welfare;EU support is also estimated to have increased productivity by 0.7 of GDP in 2008 and 1.5% in 2015, but only in some sectors, such as ICT;EU support had a stronger impact on the employment of the low-skilled the total number in employment is estimated to have increased by 55-70 for every EUR 3.2 million of grant, of whom 40-45 were low-skilled;there was a displacement effect of funding which is reflected in an estimated decline of exports and an increase in imports.
The evaluation clearly points out the limitations of the CGE model:it does not include the leverage effect of the additional demand generated by support on investment;liquidity and cash flow are also not included, so that changes in financial conditions which may affect the result of the estimation are not taken into account;enterprise size and the structure of the enterprises in a sector are not included in the model;it does not take account of changes in capacity utilisation;it excludes regional differences.
The sectoral distribution of Cohesion policy support is an important factor. Activities receiving support may have indirect effects on other sectors which are competing for resources. This may unintentionally reduce overall performance. Both the level of the labour intensity of production and the openness of a sector have a strong effect on the way it responds to an expansion of demand.
Evaluation of support for regeneration of small towns in war-affected areas in Croatia in 2014-2020
Vrednovanje tijekom provedbe 5 intervencijskih planova i Programa integrirane fizičke, gospodarske i socijalne regeneracije malih gradova na ratom pogođenim područjima
OP Efficient Human Resources ESF 2014-2020; OP Competitiveness and Cohesion 2014-2020
The evaluation examines pilot measures for the physical, economic and social regeneration of small areas that have been affected by war in the 1990s.
To assess the relevance, effectiveness, efficiency, coherence and sustainability of the pilot projects with a view to improving the management capacity of the programme.
Monitoring data.Survey of implementers of interventions in municipalities.Interviews.Focus group.Review of programme documents.
There is reasonable coordination between the MA and small town authorities but significant pressure on some IBs and some projects could not be carried out.Implementation rules are not fully clear and there are difficulties for cities in respect of public procurement, lengthy approval of expenditure and disbursement of funds by the MA.There is little progress as regards output indicators, but targets are often too ambitious given that disadvantaged areas have little experience of EU-funded projects and limited skills.The design of the programme ensures the involvement of stakeholders but rules on gender equality are poorly developed.Road maps established are not sufficient to ensure long-term sustainability of projects, especially because of limited administrative capacity.
Strengthen administrative capacity through targeted training in respect of implementation at all levels, transfer of practical experience and knowledge, and short webinars.Extend the programme to new disadvantaged areas, based on the evaluation and the conclusions of the focus groups.Define roadmaps of projects (infrastructure and soft projects) with clearly defined procedures, responsibilities and guidelines for implementation.Make more use of IT in the monitoring system to improve programme management.
Evaluation of consequences of Covid-19 on measures for sustainable mobility under the Calabria ERDF-ESF OP, 2014-2020
Valutazione in itinere delle politiche e degli interventi di promozione della mobilità sostenibile del ciclo di programmazione 2014-2020: implicazioni dell’emergenza Covid19 e nuovi scenari per la mobilità sostenibile urbana
The evaluation examines the implementation of sustainable mobility measures in the context of Covid-19 up to end-2020.
To assess progress in implementation and achievement of results, and the main changes in programming in response to Covid-19.
Official statistics.Monitoring data.Interviews with representatives of local transport companies.
Covid-19 led to reduced length of travel, short-distance journeys accounting for 75% of the total.Of the two major rail projects, one is progressing slowly and the other has been postponed because of the shift of funding to Covid-related needs.Local public transport is not well integrated and use of ICT very limited.
Reconsider unprofitable local transport in the light of the new situation and emerging needs.Improve monitoring of major projects.
Evaluation of support for schools under the Knowledge Education Development OP, 2014-2020, in Poland
Ewaluacja wspomagania szkół. Badanie wartości wskaźnika rezultatu długoterminowego Programu Wiedza Edukacja Rozwój
Idea Instytut and Piotr Fuchs Smart Research
The report focuses on three types of projects developed to support schools: projects supporting school institutions, training schools for teachers, and training of school managers. The reference period is May 2021-May 2022.
To assess the implementation of measures supported under IP 10i up to May 2022, and in particular the long-term result indicator ‘Share of supported schools using a comprehensive model for supporting school work thanks to ESF support’.
- Desk research.- Analysis of monitoring data.- 58 individual interviews and 3 group interviews.- Census survey of schools benefiting from support.- CATI with employees of institutions (response rate of 33%).- CAWI of school directors.- Comparative analysis on measure 2.10 (Supporting school work).
- Some 9,000 schools were supported (mostly kindergartens and primary schools), 15% of schools in the country.- 42% of the participating schools used a comprehensive model for supporting school work after receiving ESF support, much lower than the 60% target, the gap being mainly due to COVID-19, insufficient demand from schools, and overestimation of the target).- ‘Projects supporting school’: training matched participant needs.95% of participants provided effective support to schools after their training (e.g. in helping to identify schools’ needs).- Most schools considered that without support, the changes implemented would have been introduced anyway, but to a lesser extent or later.- ‘Teacher training schools’ were considered highly relevant and helped to increase competence, but cooperation between these schools and higher education institutes was poor. Ensuring their sustainability remains a major challenge.- ‘Training of school managers’ was assessed positively by participants, but it was difficult to attract participants.- Assessing the effects of the support on students is difficult.
In the comparative analysis, the schools supported, and the control group do not share the same characteristics in terms of regional location, public/private nature, target groups, school management, and school size.
- Find sustainable funding for ‘Teacher training schools’ and continue supporting the three projects examined in the evaluation.- Develop means for measuring students’ key competences.- Create a repository on effective methods to improve learning outcomes.
Evaluation of the impact of Cohesion Policy funding, 2014-2020, on the education and training system in Poland
Wpływ polityki spójności 2014-2020 na system kształcenia i szkolenia
OP Knowledge Education Growth 2014-2020; OP Technical Assistance 2014-2020; all ROPs 2014-2020; all Interreg OPs 2014-2020
Piotr Fuchs Smart Research, IDEA Instytut Sp. z o.o., Pracownia Rozwoju Przemysław Kozak
Piotr Fuchs Smart Research, www.smartresearch.pl; IDEA Instytut, www.ideaorg.eu; Pracownia Rozwoju Przemysław Kozak
The evaluation aims to assess the effects of measures for education and training funded under Cohesion Policy OPs in Poland.
To assess the effectiveness of the measures to improve the quality of human capital and access to high quality education, and the synergies between the various forms of support. To provide insights and recommendations for future measures.
Analysis of monitoring and administrative data. Survey of 2,765 territorial self-government units (27% return rate). Survey of beneficiaries (3,095 questionnaires completed). 46 individual interviews with MAs, representatives of universities and key beneficiaries. 9 case studies of projects. Counterfactual analyses - PMS technique for matching. Econometric modelling (multiple and logistic regressions). 2 workshops and 1 expert panel. Contribution analysis on 5 different intervention logics for each stage of education.
Support from the OPs was adequately tailored to the needs of the target groups. 76,600 new pre-school education places were created through ESF support and access to pre-school education increased by 18,1 children per supported municipality. The increase in the share of children covered by pre-school education is 1.6 percentage points higher for supported municipalities than for non-supported. Approximately 1.6 million people benefited from support for key competence development in the area of general education. This resulted in improvements in mathematics performance but not in other subjects. Service training for 145,000 teachers proved effective in enhancing competences. Over 400,000 students in high schools improved their competences, though no difference is found in the area of professional achievements with non-supported students; the only difference is found for the time needed for finding a job, which is 2-3 months less for those supported. Measures for adult learning are logically well-structured, but assessing their impact remains challenging because of measurement divergences.
2 main methodological limitations for counterfactual analysis: cities had to be excluded from the analysis relating to pre-school places; it is not possible to establish a clear relationship between funding and educational outcomes for some subjects because of lack of data on the type of educational project.
Increase the quality of pre-school education. In the area of general education, apply more extensive/alternative forms of support for municipalities which have problems to obtain EU financing. Strengthen the area of vocational counselling, especially in the final years of primary school. Strengthen vocational education and comprehensive projects in higher education.
Evaluation of the impact of the Interreg V-A-Poland-Germany/Saxony OP, 2014-2020, in the cross-border area
Ewaluacja wpływu Programu Współpracy Transgranicznej INTERREG V A Polska-Saksonia 2014-2020 na obszar wsparcia pogranicza polsko-saksońskiego
The evaluation examines the effects of cross-border projects implemented under the INTERREG-V-A Poland-Saxony.
To assess the effectiveness of the intervention and the impact of the programme on the socio-economic life of residents of border regions.
Desk analysis of relevant documents.Analysis of monitoring data and official statistics.CAPI survey of 1,216 tourists.CATI survey of 400 local residents.CAWI/CATI survey on 400 project partners.12 interviews with implementing bodies.12 case studies.Delphi method.Qualitative comparative analysis.
The main indicators of the programme were achieved and, in some cases, exceeded.The measures facilitated the development of cooperation between border areas and the removal of barriers between communities.The greatest benefits were observed in the areas of interpersonal relationships, institutional cooperation, valorisation of the culture and national heritage, and education.The vast majority of people living in the area supported (83%) have observed positive changes as a result of the programme.Two-thirds of border regions inhabitants consider that the changes brought about by the OP have improved their access to places of natural and cultural value.
Further support children, young people and regional staff in the development of language skills.Increase the share of NGOs among programme applicants.Intensify activities aimed at informing the population living in the border regions affected by the OP.
Evaluation of the Regional OP 2014-2020 technology transfer interventions
LOT3 Evaluarea Interventiilor POR 2014-2020, Tema 1 Promovara transferului tehnologic
The evaluation examines support for technology transfer financed by the Regional OP 2014-2020 under PA1 with the aim of increasing innovation in companies. The DG Regio pilot initiative for Lagging Regions covers the North-East and North-West regions.Because of delays, it is limited to assessing the measures concerned and the relationship with RIS3 in broad terms.
To assess the contribution of the Regional OP to smart specialisation in the regions, in terms of changing the vision, the added-value and the sustainability of the institutional structure.
literature review;17 semi-structured, in-depth interviews with the MA and independent experts;survey of potential beneficiary companies and research organisations;5 focus groups with stakeholders;2 case studies.
The OP has made a visible contribution to the implementation of the smart specialisation concept in regions, involving all relevant stakeholders and businesses in the planning process.The OP gave technical support to the development of the strategic planning methodology in line with EC Guidelines for RIS3, but not with that adopted at national level for research and innovation planning. RIS3 therefore runs parallel to regional development planning but is not fully integrated.The added-value from the OP consists of increased knowledge acquired by companies and research organisations, a clearer vision of the links with education and the labour market and the need for cooperation between companies and research centres.The barriers to technology transfer include access to finance, the high costs of transfer, intellectual property issues and state aid rules.Implementation was delayed, no contracts have been signed and a large number of projects submitted have been rejected. All of this may deter participation of companies and research centres in smart specialisation measures in the regions.
Participation in regional focus groups was reduced by the disappointment caused by the long duration of the project selection, high rejection rate and unclear procedures.
Harmonise the procedures for devising the National Strategy for Smart Specialisation with those of the RIS3.Consolidate institutional capacity for promoting innovation in the regions, creating a network of Councils for Regional Innovation, and providing support for strengthening capacity of the MA, RDAs and potential recipients of funding.Simplify the project selection process, reducing the administrative burden, adapting funding conditions to potential recipients and adjusting the applicability of the state aid rules using the Horizon programme experience.Ensure more flexibility in applying the JRC methodology for RIS 3 implementation.
Mid-Term Evaluation Study of the Communication Strategy of Madeira 14-20
Estudo de Avaliação Intercalar da Estratégia de Comunicação do Madeira 14-20
Rui Godinho (coordination), Ana Simões, Gisela Ferreira and Américo Alves
Evaluation of the contribution of tourism to increasing SME competitiveness under the Nordrhein-Westfalen OP, 2014-2020
Spezifische Evaluierung 1.3. Beitrag der Tourismusförderung zur Steigerung der Wettbewerbsfähigkeit von KMU
Dr. Sebastian Elbe, Dr. Judith Elbe , Dr. Florian Langguth, Dr. Ute Middelmann - SPRINT
Dr. Sebastian Elbe, SPRINT – info@sprintconsult.de
The evaluation assesses interventions to promote tourism in the region between 2018 and 2020.
To assess the contribution of support for tourism to strengthening SME competitiveness between 2018 and 2020.
• Desk review.• Case studies in respect of support for innovation, ­infrastructure and networks.• Focus groups and workshop with regional stakeholders.• Surveys of beneficiaries and SMEs supported.• Analysis of monitoring and statistical data.
• 52 projects from the ‘Erlebnis.NRW’ measure and 38 from ‘Regional 2016’ on innovative services, integrated infrastructure, and network creation as well as studies on development to support tourism were funded.• The 2023 target for the ‘number of tourists’ was already achieved in 2019.• The support of networking and workshops had a positive impact on the development of innovative services and infrastructure.• The Master Plan and its continuation in the regional tourism strategy was useful for conceptualising projects and targeting specific groups.• Horizontal principles were not adequately taken into account.
The majority of projects were not completed at the time of the evaluation and effects on regional development are expected to materialise later.
• Continue the promotion of innovation in tourism and also include digitalisation-related measures.• Focus on the construction of permanent networks.• Maintain spatial flexibility, supporting projects at both regional and national level and involve local actors.• Continue to develop a tourism strategy in the future
Evaluation of the criteria and system for selecting of projects within PA I and VII of the OP Infrastructure and Environment in Poland
Ewaluacja kryteriów i systemu wyboru projektów w ramach I i VII osi priorytetowych Programu Operacyjnego Infrastruktura i Środowisko 2014-2020
IBC GROUP Central Europe Holding S. A., ul. Mokotowska 1, 00-640 Warszawa, tel.: +48 22 250 18 39, fax.: +48 22 201 21 40; Klub Jagielloński Sp. z o.o., Rynek Główny 34 (Pałac Spiski), 31-010 Kraków
The evaluation examines the procedure for selecting projects under PA I (Carbon emission Reduction) and PA VII (Improvement in Energy Security) of the OP Infrastructure and Environment 2014-2020 in Poland.
To assess the criteria and system of selecting projects under PA I and VII of the OP.
Desk research on OP documentation and EU regulations;communication analysis;11 in-depth interviews with representatives of implementing authorities and applicants;CAWI survey of applicants and beneficiaries (with a 21% response rate);Panel of experts.Data come from interviews and the survey as well as from programming and legislative documents.
Criteria for project selection under PA I and PA VII are regarded as being in line with the objectives and assessed positively as regards their intelligibility, objectivity, lack of ambiguity and accuracy;organisation of the calls is also assessed positively, along with the information system, including for project selection, and the appeals procedure;for PA VII, there is a high concentration of funds on a few beneficiaries, which is in line with the characteristics of the market in which they operate;co-operation between the applicants and the implementing authorities on non-competitive calls allowed the proposals to be improved so that they could fulfil the objectives.
There is a need to:clearly define the scope of the measures under PA I in order to avoid duplication;simplify the application procedure under PA I, by focusing on criteria describing effectiveness and readiness to implement;consider the possibility of carrying out partial audits to analyse selected efficiency measures;simplify and reduce the amount of information required in project documentation for applicants and provide them with training.
Evaluation of the project selection system and criteria in the Lubelskie regional OP
Ocena kryteriów i systemu wyboru projektów RPO WL 2014-2020
Spławie 53; 61-312 Poznań, Tel: 61 622 92 06-07, biuro@re-source.pl
This evaluation assesses the project selection and evaluation procedure, including the related criteria, used by the Lubelskie regional OP for 2014-2020.
To assess the procedure and criteria used to select projects, to assess the achievement of product and financial indicators and to indicate possible improvements in the selection process.
Desk research.Benchmarking.CAWI/CATI interviews.In depth expert interviews.Panel of experts.
The projects’ selection system is overall reasonably transparent, impartial and non-discriminatory.Issues are reported concerning the correctness and intelligibility of the formulation of the criteria.Specific problems relating to correct wording were identified under both the ERDF and the ESF general criteria.The two IT systems which support the selection procedure work properly and provide all the necessary functionality.
There is a need to:Improve the monitoring of the OP objectives so that possible delays in implementation are taken into account.Improve the management of MA staff.Reduce the amount of ERDF required documentation.
Evaluation of the 'School projects/support for the young' scheme under the Brandenburg ESF OP, 2014-2020
Evaluierung der Förderung des Programms "Projekte Schule/Jugendhilfe 2020" - Endbericht
SÖSTRA Institut für Sozialökonomische Strukturanalysen
SÖSTRA GmbH, Torstraße 178, 10115 Berlin, www.soestra.de
The evaluation examines the 'School projects/support for the young' scheme financed by the Brandenburg ESF OP, made up of two parts: Model A to limit school drop-outs and Model B to ensure that participants complete compulsory education and are prepared for ­the transition to vocational training or further education.
To assess the implementation of the 'School projects/support for the young’ scheme.
Literature review.Analysis of monitoring data.Interviews with the MA, project promoters and stakeholders.2 case studies of a project, one of Model A and one of Model B.Interviews with 6 youth welfare offices.
1,152 students were supported between 2015-2019, split equally between Model A and Model B.The gender division was the same in the two Models, two thirds of participants were male, the average age being 13.5 in Model A and 15.3 in Model B, and support lasting 7.4 months on average in the former and 10 months in the latter.Cooperation between those involved is good, in part because of the advisory board which eases the work of youth welfare offices and schools.Cross-cutting issues have been taken into account in the implementation of the scheme.
Support should be continued in the next programming period.The learning approach developed by schools and cooperation with social services should also continue.
Investment in Health infrastructure and services in 2007-2013
Evaluarea de impact a DMI 3.1 „Reabilitarea, modernizarea, dezvoltarea și echiparea infrastructurii serviciilor de sănătate" şi a DMI 3.3 "Îmbunătățirea dotării cu echipamente a bazelor operaţionale pentru intervenții în situații de urgență"
Miruna MAZURENCU MARINESCU, Andrea GRAMILLANo, François LEVARLET, Pietro CELOTTI, Francesco SALVAGNINI, Massimo BRESSAN
SC ACZ Consulting SRL - str. Ștefan Velovan, bl. 23A, Craiova, jud. Dolj, Tel: 0351 44 20 44 - T33 SRL via XXV Aprile, nr. 28/b, Ancona, Italia, Tel: +39 0718 98093 - Iris SRL via G. Verdi, nr. 40, Prato, Italia, Tel: +39.0574.607522
The evaluation assesses two measures co-financed by the Regional OP 2007-2013: measure 3.1 – ‘Rehabilitation, Modernisation, Development and Equipping of Health Service Infrastructure’ and measure 3.3 – ‘Improving the provision of equipment in centres for emergency treatment'.
To assess the net effects of the measures and the underlying mechanisms.
The evaluation uses qualitative and quantitative methods:Literature review (of health infrastructure and services for emergency situations);analysis of administrative data and of the intervention logic;econometric and statistical analysis;customer satisfaction survey;cost-benefit analysis ;semi-structured interviews with the MAs and Ministry of Health;case studies (3);expert panel;benchmarking analysis;focus groups.The data used are primary data from interviews, surveys, questionnaires and focus groups and secondary data from OP documentation the monitoring system and official statistics.
On measure 3.1 - ‘Rehabilitation, Modernisation, Development and Equipping of Health Service Infrastructure’:the number of people benefitting from health infrastructure was increased (average number of daily users rose from 5,700 in 2008 to 8,314 in 2014);the quality of services, equipment and infrastructure was improved and user satisfaction was increased;the competence of staff was increased (through training) and new people were recruited.On measures 3.3 - ‘Improving the provision of equipment in centres for emergency treatment’:the average treatment time in mobile units was educed (from 30-45 minutes to 23 in rural areas and from 20 minutes to 11 in urban ones);support was given to creating an integrated organisation and regional cooperation in respect of large-scale, long-term treatment.
Several limitations are listed, in particular:Measure 3.1:there were a relatively small number of observations available;the tax code enabled only two beneficiaries to be identified;outcome indicators were measured in different ways at project level.Measure 3.3:information was received only from 7 out of 14 projects completed;the data available are limited.
Resources should be allocated according to the need.Health services need to be integrated at the community level to improve territorial coverage and reduce pressure on main health centres.Funding recipients should be required to submit project information on infrastructure supported.The use of e-health services and preventive medicine should be promoted as well as out-patient visits to reduce hospital costs and the pressure on main health centres.
Evaluation of the social inclusion specific objectives of the Integrated Regional ERDF OP, 2014-2020, in the Czech Republic
Evaluace programu IROP, Prioritní osa 2: specifické cíle 2.1, 2.2, 2.4 se zaměřením na sociální inkluzi
Ernst & Young, s.r.o., https://www.ey.com/cz/
The evaluation examines the 7 types of social inclusion project receiving support for infrastructure or equipment: social and community services, social housing, social entrepreneurship, pre-school facilities, formal education, lifelong learning and non-formal education.
To assess the extent to which ERDF support for infrastructure and equipment contributes to social inclusion in respect of 7 types of project.
Desk research.Interviews with beneficiaries, target groups and other stakeholders.Phone interviews.Online surveys.Case studies.
Overall, projects supported respond to the needs of target groups and municipalities, and are coherent with the Employment and Research Development and Education OPs.Cooperation between beneficiaries and other relevant stakeholders (such as NGOs) has proved to be effective, along with involvement of experienced project managers.Administration of projects is demanding and often requires outside help.In 2019, values for specific indicators are well below 25% of the target except for 3 programmes.The OP has helped to increase the territorial coverage and quality of social services, to create social enterprises and employment, to increase the number of pre-school places and to improve lifelong learning and non-formal education infrastructure.There have been positive effects on the quality of life, social situation, employment competencies and financial situation of target groups.Community centres alone cannot solve the major problems of target groups; social entrepreneurship has not markedly reduced unemployment of target groups because of the small scale of support. Formal education has not significantly increased the inclusion of children from disadvantaged families, and social housing has not motivated residents to increase their income because it means them losing entitlement to it.
It was difficult to get data from target groups and to asses measures which had recently ended; the replies from survey respondents were incomplete or subjective; and for some projects, the sample size was not large enough for the findings to be generalised.
Speed up assessment of project applications.Revise the set of indicators to better capture progress made.Establish phases of social housing so that target groups remain motivated to improve their financial situation.Create more projects for children under 3.Ensure cooperation with other institutions on the collection, processing and transmission of data.Simplify the administration of projects.
Annual evaluation report of the ESF OP, 2014-2020, in Toscana – year 2021
The Annual evaluation report (AER) examines the physical and financial progress of the OP until 2021 and primarily focuses on the effects of the Covid-19 pandemic on the regional socio-economic context.
To assess the financial and physical progress of the OP and the target groups covered, as well as to examine the effects of the pandemic on key socio-economic variables.
Desk research. Analysis of monitoring and official statistical data. Interviews with the MA.
The pandemic caused a 2.4% decrease in the active population in 2020, and a 2% decrease of people employed. However, in 2021, the trend reversed for both indicators, which increased by 0.1% and 0.8% respectively. COVID-19 led to a prompt reprogramming of resources, which resulted in an increased share of beneficiaries of IP 9.1 and IP 9.2 (32.3% of total beneficiaries). The OP has high coverage rates of the target populations overall, but women tend to be underrepresented in measures promoting job placement. The implementation of the OP shows a good progress: 10 out of the 12 physical indicators have already reached the target value for 2023.
Ensure effective integration with other regional and national policies, including the National Recovery and Resilience Plan, to address emergent social issues comprehensively. Utilise ESF funding to introduce and expand innovative practices, such as those for gender policies or green and digital transitions. Maintain flexibility in programming to address emerging challenges promptly, and support this by continuous monitoring and strategic evaluations.
Evaluation of innovative activities under the Human Resources OP, 2014-2020, in Slovakia
Doterajšie skúsenosti s podporou inovatívnych aktivít a návrhy na systémovú podporu inovácií z ESF+
Octigon, info@octigon.sk - Projektové služby, office@pseu.sk - RADELA, radela@radela.sk
The evaluation examines the innovation activities implemented under the OP, and makes recommendations for the next programming period.
To assess the extent to which the OP supports innovative activities, their contribution to achieving the specific objectives, the innovations with the most potential to become systemic changes, the groups using the innovations the most, and the main implementation issues, and to suggest measures for improvement.
The OP strategy partially supports the implementation of projects focusing on innovation, including social innovation.The OP intervention strategy is up-to-date, and despite several OP revisions, the innovation area has not been affected by significant changes.The performance of SOs varies, but the measures with an innovative aspect contribute to the IPs, TOs and PAs.Innovative elements can be found in every PA.The definition of ‘innovation’ and its measurement has posed problems.
Because of the early stage of implementation, only partial results in achieving the objectives are available. Because of Covid-19, the expert panels had to be shortened and conducted with a smaller number of participants.
Assess whether the proposed innovations meet the required quality and evaluate their impact.Define innovation, including social innovation, in national legislation.Provide training across all departments involved in the setting up, implementation and control of innovative projects in order to ensure uniformity throughout.Create a website or a database where the results of innovative projects are presented.Develop and promote local and international partnerships.Continue promoting innovative approaches in 2021-2027, and build future SOs based on the experience of previous ones.Reassess the measurable indicators based on the achievement of project goals.
Ex ante study of the draft documents of the Pomorze Programme, 2021-2027 - 2nd Phase
Ocena ex ante projektów dokumentów strategicznych i programowych województwa pomorskiego dla perspektywy finansowej 2021+. Etap 2 – Ocena ex ante projektu Fundusze Europejskie dla Pomorza 2021-2027
LB&E Sp. z o.o.; Bartosz Ledzion EGO – Evaluation for Government Organisation S.C.; Karol Olejniczak EGO – Evaluation for Government Organizations S.C
This is a comprehensive ex ante study of the draft regional OP.
To assess the strategy of the OP, the relevance and feasibility of priorities and specific objectives, the suitability of indicators, the feasibility of the financial plan, and the relationship with other policies.
Desk research (strategic, programming and implementation documents, legislation).36 interviews with representatives of institutions.4 interviews with representatives of self-governments.Logical models.Mini-focus group.Workshop with MA.
The scope of support is mostly in line with the problems and challenges diagnosed.The selection of target groups is appropriate.The vast majority of indicators selected are suitable.Allocation of funding between the various areas is appropriate.
Plan a smaller allocation of funding for investment in fixed and intangible assets.Revise some of the indicators.Add to the programme the elements that are missing.Define a list of strategic projects.
Evaluation of PA2 ‘Promoting the information society’ of the EU Structural Funds Investments OP, 2014-2020, in Lithuania
2014–2020 m. Europos Sąjungos fondų investicijų veiksmų programos prioriteto „Informacinės visuomenės skatinimas“ poveikio vertinimas
https://civitta.com/ - info.lt@civitta.com
The evaluation examines the impact of PA2 ‘Promoting information society’ on the development of the information society, the sustainability and continuity of interventions.
To assess the impact of PA2 (TO2) on the development of the information society, by identifying how and to what extent EU-financed investment has contributed to the relevant strategic OP indicators, the benefits and the sustainability of products created.
Analysis of statistical and monitoring data.Analysis of legislation and previous studies and evaluations.10 interviews with EU funding administrators and project implementers.Survey of projects implementers (30 responses out of 37, on 33 projects out of 43).Focus group.Reconstruction of the intervention logic.Case studies.Cost-effectiveness analysis.
13 out of 18 indicator targets have been met or will be met at the end of projects. One indicator is unlikely to be achieved, while the other four are difficult to predict because they are affected by other factors.The main hindering factors are lengthy public procurement processes, a lack of competence of workers and of common interest in the public sector to improve public administration.In a number of cases, without the subsidy, projects would not have been implemented to the same extent or at the same pace.The measures helped to promote the public’s use of the Internet and the digitalisation of public management procedures and increased e-service maturity.The socioeconomic benefits are estimated at EUR 1.89 billion, which mostly come from the time and money saved by being able to use e-services instead of having to travel to a physical location.The financing of the measures was sufficient.Since the start of the project, the number of Information System (IS) users has risen by 400%.According to stakeholders, the largest impact is generated by projects that reach the largest target groups, such as the state cloud project, the development of e-health services and the State Tax Inspectorate's smart tax administration system.Some projects implemented in 2007-2013 were continued in 2014-2020.Even though Covid-19 brought challenges to the implementation of projects, these have largely been overcome.
It was not possible to assess the progress of two result indicators on the use of the information system because their monitoring was interrupted. It is difficult to assess the extent to which public e-services have been used, since there was no obligation to monitor this, and because the real impact of interventions will only be evident 3-5 years after. There is a lack of a standardised mechanism for monitoring the use of e-services. At the time of the evaluation, some projects were still being implemented.
Promote inter-institutional cooperation not only among specialists, but also at management level.Assess the possibility of the mandatory monitoring of public and administrative e-services, such as through the Public and Administrative Services Information System (PASIS).Assess the possibility of enabling the Open Data and Digital Transformation Competence Centre to develop a methodology for monitoring public and administrative e-services.Carry out an ex-ante evaluation for 2021-2027, focusing on the relevance of investment and the priorities and an ex-post evaluation focusing on the long-term impact of interventions.Before starting the implementation of the 2021-2027 OP, create a working group to address the challenges relating to monitoring indicators.Continue developing the learning material on the EU funds, organising physical and online training.In project funding applications, include an additional requirement that three years after the completion of the project, service/product promotion activities must be carried out.Continue the practice of the Central Project Management Agency to include an additional question in project funding applications on how the project promoters plan to ensure long-term project funding.
Ex ante analysis for the establishment of a zero rate rebound loan fund in the context of the health crisis
Analyse ex ante pour la mise en place d’un Fonds de Prêt Rebond à taux zéro dans un contexte de crise sanitaire
Thematic evaluations under the Liguria ERDF OP, 2014-2020
IRS - https://www.irsonline.it/ - irsroma@irsonline.it; PTSCLAS - https://www.ptsclas.com/
The evaluation examines the measures on the Smart Specialisation Strategy (S3), digital infrastructure and SME competitiveness, financed under the OP, up to June 2022.
To assess the OP contribution to S3, the digitalisation of processes and the strengthening of SME competitiveness, as well as the main results achieved up to June 2022.
Documentation analysis.Analysis of monitoring and statistical data.Interviews with stakeholders.2 surveys of SME beneficiaries.Contribution analysis.2 Focus groups with experts to validate the contribution analysis.
Measures under PA 1 contributed to the pursuit of 4 out of 7 strategic objectives of S3 and cover all three 3 areas of specialisation of the strategy.11 out of 16 measures on digitalisation of administrative procedures and dissemination of digital services under PA2 have been concluded.According to the interviews, projects under PA2 have helped increase the efficiency of administrative procedures, e-payments, and e-services for the public, tourists and enterprises, despite delays in training of public officials and processing documents.Projects under PA3 enabled 82% of beneficiary companies surveyed to consolidate and innovate production processes, 76% to strengthen their organisational structure, 74% to deal with changes linked to the Covid pandemic effectively, 61% to strengthen their position on the market and 35% to create new products.30% of enterprises initiated collaboration with other firms during projects.
The validity of the surveys is uncertain, as the sample might include a large share of highly proactive companies, and companies covered are only those with a Registered Electronic Mail address.
Support collaboration on innovation to increase internationalisation.Expand initiatives fostering spill-over effects on innovation.Include in the monitoring system indicators able to measure the achievement of S3 objectives.Train company personnel on how to take advantage of digital infrastructure projects.Plan comprehensive projects for enterprises to support sustainable development over the long-term.
Evaluation report of the Caribbean area Interreg OP, 2014-2020 – year 2021
Rapport évaluatif 2021 portant sur le programme INTERREG Caraïbes 2014-2020
The evaluation examines the results of the OP aimed at promoting development of trade between the territories in the region.
To assess the results and the effects of the measures implemented and to identify the limitations and examples of best practices and possible future improvements.
Desk analysis of monitoring data and relevant documentation.Interviews with stakeholders.Surveys of 23 beneficiaries and 16 partners.
Projects under PAs 1 and 2 (innovation and business competitiveness) are coherent with the overall strategy but have obtained few results as regards "innovation" despite what was expected.Measures under PAs 3 and 4 (preparing communities for natural risks) are relevant and more strategic than in the previous programming period.PA 5 and 6 projects (exchange of know-how on environmental management through promoting the natural heritage) can potentially achieve the expected results. Their main logic is to invest in tourism as an essential driver of the economy and employment.The only project supported under PA7 in respect of research and cooperation in healthcare was strongly affected by the covid-19 pandemic and its implementation was delayed.The two PA9 projects (on renewable energy) have low potential to contribute to the overall strategy.PA10 projects help to increase linguistic competences and cooperation mainly through intangible investments.In general, ERDF resources are judged to be essential for the projects to be implemented.The support provided by the MA was in some cases not timely or incisive enough and there were payment delays that affected the implementation of the projects.
No substantial limitations are mentioned, it is simply emphasised that some expected effects on business R&D objectives, reducing energy consumption of public buildings and cooperation on healthcare are difficult to observe and assess at this stage.
Strengthen the communication strategy of the OP.Support project managers better through interim meetings.Strengthen the links and interaction between the MAs, the Joint Secretariat and the Regional Contact Point (RCP).Set up FAQs for all RCPs, preparing them before programming and improving them as the OP is implemented.Improve the transmission of information and documentation to the RCPs.Increase the number of information meetings and/or training activities with non-EU partners.Develop the means for monitoring and evaluating projects.
Evaluation of the Germany-Netherlands Interreg OP, 2014-2020 – year 2021/2022
Impact-Evaluation 2021/2022 INTERREG Deutschland-Niederland
Marnie von Fritsch (Ramboll Management Consulting), Dr. Jos Mevissen (Regioplan Beleidsonderzoek), Hetty Visee (Regioplan Beleidsonderzoek), Lara Westphal (Ramboll Management Consulting)
Ramboll Management Consulting, marnie.vonfritsch@ramboll.com
The evaluation examines the OP up to end-2021.
To assess the contribution of support to the achievement of the objectives of the OP of strengthening RTDI and deepening cross-border integration and its impact.
Defining a simplified Theory of change for each of the Specific Objectives.Analysis of monitoring data and of project documents.Interviews with MA.Survey of lead partners of all projects.
All target values of indicators had already been achieved or significantly exceeded by end-2021.Projects have contributed to the OPs objectives by implementing innovative projects, raising awareness of cross-border cooperation and building cross-border networks.Projects have made a visible contribution in terms of new knowledge being built up and competences being expanded and joint action in policy-making processes being strengthened.Projects potentially have long-term effects, by increasing competitiveness and improving cross-border cooperation.Success factors are the high level of personal commitment, in-house staff continuity and experience and the knowledge of those involved in implementing the OP.
Consider the lessons of the current experience in setting the result indicator targets for the next programming period.Promote projects in which citizens are directly involved.Continue to expand active public relations.Support more project applicants and those and implementing projects.Maintain the high level of personal commitment.
Report on progress in implementing the Veneto ESF OP
Rapporto sui progressi del Programma per il servizio di Valutazione operativa e strategica del POR FSE 2014-2020
This evaluation assesses the progress made in the implementation of the ESF Veneto OP 2014/2020.
To assess the progress in implementing the Veneto OP up to offend-December 2017.
Overall evaluation of certified expenditure is positive.Overall evaluation of the outcome indicators is positive.The Axis 4 indicator ‘Public administration or public services in national, regional or local level projects’ is largely achieved.The targets for Axis 1 and 3 participants are almost achieved.
Evaluation of ERDF and ESF indicators under the Śląskie OP, 2014-2020
Ewaluacja uzyskanych wartości wskaźników rezultatu EFS i EFRR w Regionalnym Programie Województwa Śląskiego na lata 2014-2020
Jakub Grabowski, Malwina Pietrzyk, Ewelina Baryła-Zapała, Aneta Gierowska
ASM - Centrum Badań i Analiz Rynku, asmresearch.com.pl / neurodata.pl
The evaluation examines the extent of achievement of some targets of the OP.
To examine the values of four long-term and one result indicators of the OP.
Analysis of monitoring data.Survey of 7,528 people.
17% of participants were “in a better situation on the labour market six months after completing participation in the programme”, 19% of men and 15% of women.240 “jobs existed in social enterprises for at least 30 months”.1,150 “places in pre-school education were functioning for two years after obtaining funding from the ESF”.All 711 newly created childcare places were still operating two years after receiving ESF funding.There were 470 “active forms of activation of the population living in revitalised areas”.
There was a problem with reaching survey respondents because of incorrect phone numbers. In addition, some of the respondents had difficulties in remembering the project because of the long time that had elapsed.
Evaluation of the indicator system for the Valle d’Aosta ESF OP, 2014-2020
Valutazione del sistema di indicatori del PO FSE 2014-2020
The evaluation examines the system of indicators of the ESF OP and progress towards 2023 targets.
To assess the significance and relevance of the indicators and progress towards targets and to identify monitoring system deficiencies.
Monitoring data.OP documentation and EU guidelines on monitoring systems.Interviews with OP technical assistance and MAs of other regions.
OP output and result indicators are highly relevant given the strategy and specific objectives.There are no problems in completing data for output indicators.A few result indicators have measurability problems (especially that on children receiving assistance services), mainly because of their wrong link to a specific objective or the failure of the monitoring system to keep track of annual participants instead of total numbers. 2023 targets for performance framework output indicators have already been achieved or are on track, but financial indicators are far from targets.
Replace the indicator R6S with a more relevant and measurable indicator, or involve implementation bodies and beneficiaries in reconstructing data on recipients.Define procedures to simplify transmission of data on indicators, on the basis of best practice from other regions.Create a data warehouse integrating data on ESF recipients and administrative databases on employment, education and training.
Evaluation of measures for internationalisation of SMEs under the Toscana ERDF OP, 2014-2020
Analisi dei risultati delle politiche per l’internazionalizzazione commerciale
The evaluation examines the features of projects for the internationalisation of SMEs under the OP.
To assess the main characteristics of projects for the internationalisation of SMEs.
Monitoring data.Official statistics.Literature review.
The demand for highly specialised and complex services for internationalisation has increased in Toscana.Projects mainly finance participation in trade fairs, promotion activities and advanced services such as the Temporary Export Manager.Only 50 of the 995 projects funded are in sectors included in the regional Smart Specialisation Strategy.
Annual Evaluation report 2018 of the Campania ESF OP 2014-2020
Rapporto annuale di valutazione - POR Campania FSE 2014-2020 -2018
Regional office for the assessment and verification of public investments - Campania Region
The evaluation focuses on the state of implementation of the Campania ESF OP 2014-2020 at the end of 2018.
To assess the procedural, financial and physical progress of the OP and the five PAs (I employment; II social inclusion; III education; IV institutional capacity; V Technical assistance).
Desk analysis;Analysis of monitoring data.
The strategy of the OP is appropriate and does not require substantial change.At the end of 2018, 18% of total funding was spent and 63% allocated to selected projects. In relation to the financial targets of the performance framework, PAs I and IV achieved about 86% of the target, PA II 103% and PA III 83%The OP reached 365,004 people, 263,440 of them under PA III (education and training).As regards the output targets of the performance framework, PAs I, III and IV exceeded the milestone of 2018, while PA II achieved only 78% of the target (number of unemployed).The OP is consistent with the Europe 2020 objectives; but the regional distance from the national target on poverty and social exclusion is still very large and there has been no progress towards reaching the employment rate target in the past few years.
​​Evaluation of measures for the digitalisation of education under the European Funds for Social Development Programme, 2021-2027, in Poland​
Jolanta Kluba, Paulina Kowalczyk, Dorota Kretkiewicz, Weronika Zych
ECORYS Polska, www.ecorys.pl, jolanta.kluba@ecorys.com; Grupa Badawcza DSC, https://www.grupa-dsc.eu/
The evaluation focuses on assessing the implementation, coherence, complementarities, risks, and early results of the interventions supporting the digitalisation of education. This includes the development of system-level tools and digital competences, as well as teacher training.
To assess the progress, preliminary results, consistency, complementarity, and risks of interventions supporting the digitalisation of education, and to formulate recommendations for further action.
Analysis of monitoring and statistical office data.23 interviews with key stakeholders.Expert panel.Focus group interview with beneficiaries.Comparative review of 16 regional programmes and the National Recovery and Resilience Plan.
In general, the measures represent a solid foundation for strengthening digital skills and tools in schools.The main problems with the digitalisation of education are organisational delays, staff shortages, and excessive bureaucracy, which slow down the implementation of the measures.Although the measures are in line with the national strategic directions, cybersecurity, digital hygiene and the fight against disinformation remain insufficient.The measures complement other programmes, but cooperation between institutions is fragmented and lacks good coordination.The use of Individualised Education Programmes and e-materials is uneven, and low awareness of these tools among schools hinders their widespread use.Cooperation between programmes, such as the National Recovery and Resilience Plan exists, but to be truly effective, it needs to be better linked at the level of everyday school operations.According to stakeholders, the effectiveness of activities depends on clear communication, easy access to training and the practical usefulness of digital tools.
Early stage of implementation, difficulty in assessing risks.
Strengthen cooperation between institutions responsible for digitalising education to ensure consistent and coordinated action.Increase the focus on less developed areas, particularly cybersecurity, digital hygiene, and fight to disinformation, to address the actual needs of schools.Improve communication and promotion of digital tools, particularly IEP resources and e‑materials, to raise awareness and encourage their use in schools.Define better synergies between programmes at the operational level, to enable schools to combine different forms of support effectively.Monitor delays and implementation barriers systematically and take early corrective actions to reduce the risk of not meeting the 2029 targets.
First thematic report – Results of the ESF co-financed “Employment and Labour Policies Integrated Plan” in Friuli Venezia Giulia in 2014-2020
I Rapporto tematico di valutazione – I risultati di PIPOL
Naldini A., Pompili M., Ciffolilli A., Miccadei C., Pastore F., Rossi L. and Ciccarella F.
Ismeri Europa. S.r.l. Via G. G. Belli, 39 – 00193 Roma, Tel.: +39(06)32691201, E-mail: mail@ismerieuropa.com
The evaluation assesses the effects of the “Employment and Labour Policies Integrated Plan” (PIPOL) co-financed by the ESF in the region in 2014-2020.
To assess the impact of PIPOL on employability and access to employment of beneficiaries.
The evaluation uses qualitative and quantitative methods:Desk research (of OP documentation and related studies);counterfactual analysis (using propensity score matching);survey of the beneficiaries.The data used are mostly from the monitoring system (OPOC system and Netforma), programme data (Ergonet), data on compulsory communications and data from the survey.
Beneficiaries have a higher chance of being employed than the control group (net impact of PIPOL is estimated to be 5-5.5%).The impact of PIPOL is larger for those participating in traineeships and training. The former increases the probability of obtaining a long-term job by 3%.PIPOL is more effective for women than men (+6% vs. +5%), people under 30 than those above (+6% vs. +4%), low educated and those from abroad.Training leading to a vocational qualification has a positive impact on the probability of finding a job, (especially more structured training of over 200 hours) Language trainings is found to have a negative effect.Traineeships that provide work experience have more impact than vocational training, especially those in manufacturing.
The evaluation focuses on the first phase of PIPOL, from 2014 to April 2018, in particular on extra-curricular training and traineeships completed by the end of 2016.
There is a need to:improve services that encourage contact with the labour market, particularly for “weaker” beneficiaries;boost the dual system within education;improve on-going monitoring of employability;repeat the impact analysis in the future to clarify some elements which remain unclear (e.g. whether or not effects tend to disappear in the long-run).
On-going evaluation of the management and implementation system of the Świętokrzyskie OP, 2014-2020
Ewaluacja on-going systemu wdrażania i zarządzania Regionalnego Programu Operacyjnego Województwa Świętokrzyskiego na lata 2014-2020
ITTI Sp. z o.o. ul. Rubież 46 61-612Poznań
The on-going evaluation assesses the management and implementation system for the regional OP for Świętokrzyskie in the 2014-2020 period, which is funded by the ERDF and ESF.
The objective of the study is to assess the management and implementation system of the regional OP for Świętokrzyskie. Three criteria were assessed: relevance of the arrangements, usefulness of the procedures and efficiency of the process.
The research was based on:Individual in-depth interviews with the Managing Authority (MA) and Intermediate Bodies (IB).Field observations.CAWI interviews of all employees involved in the management, implementation and control of the OP.CATI interviews of beneficiaries and unsuccessful applicants.Evaluation workshop summarising the results of the study.Data analysis: SWOT analysis, qualitative analysis of the replies to the interviews, quantitative analysis.
The division of tasks and responsibilities between the MA, the IB and the certifying authority is appropriate.The current procedures and instructions governing the management, implementation, control and certification processes are also appropriate and ensure efficient implementation of the OP.In addition, the current resources are considered adequate.However, in terms of human capital, the lack of an incentive scheme and insufficient wages for the employees involved in the management and implementation of the OP was highlighted.
Limitations are not indicated in the evaluation report.
The evaluation report recommends:The introduction of bonuses/awards based on performance as well as a greater emphasis on the professional development of the employees involved in the management and implementation of the OP.Specific training tailored to the capabilities of the employees should be organised.Reduction of staff turnover is necessary to improve the effectiveness of their work.To increase communication within the institution, a common repository of knowledge including examples of good practices in addressing difficulties should be created.
Process evaluation of the National Framework under nine Programmes 2021-2027, in Czechia
Zpětná vazba k JNR 2021-2027 pro přípravu metodického prostředí fondů EU 2028+
All Programmes 2021-2027, Interreg VI-A Czechia-Poland 2021-2027
Naviga Advisory and Evaluation; Grant Thorton
https://www.navigae.cz/, https://www.grantthornton.cz/en/
The report is a process-oriented evaluation of the Czech “National Framework” (JNR), 2021–2027, under nine OPs. Data used go up to September 2025.
To assess how the framework’s rules work in practice for the key actors—those who develop, apply, and are subject to them (MAs, implementation bodies, applicants and beneficiaries). To provide suggestions on how to streamline processes and significantly reduce the administrative burden for the 2028+ period.
Desk research to analyse the methodology and identify bottlenecks in the programmes.Analysis of monitoring data.22 in-depth interviews with applicants and beneficiaries including representatives of institutions receiving support from different OPs, representatives of ‘subsidy agencies’, representatives of overarching bodies, and other relevant stakeholders.A second round of 30 interviews with MAs and administrators of selected implementation areas.A survey of 309 users of JNR methodologies and programme management documentation to collect information on the main problems and bottlenecks during implementation.A second survey of a wide range of applicants and beneficiaries (2,300 individuals reached; 1,303 responses were used in the analysis – 57% response rate).Focus groups with 14 representatives of the MA of different programmes.A workshop with applicants, beneficiaries and representatives of other bodies to discuss the results of the evaluation process.
Underexploited digitalisation and sharing of information between implementing institutions is the most frequently cited bottleneck (i.e., beneficiaries being asked for information already available).The JNR’s flexibility is assessed positively by MAs, but beneficiaries report that the same flexibility produces low harmonisation and divergent approaches across authorities, reducing predictability and transparency.Controls and audits are described as overlapping and not well sequenced, which leads to repetitive submission of the same documents.Management of the risk of double-funding is constrained by the absence of a single, centralised database of grants/subsidies and by limited end-to-end digital traceability beyond the monitoring system (MS2021+).
The report acknowledges that the evaluation is exploratory in this phase and information provided by the stakeholders sometime can be inaccurate. Also, the assessment of programme management documentation explicitly excludes operational programmes under the responsibility of the Ministry of the Interior of the Czech Republic.
Complete and harmonise the JNR “core” guidance in areas currently handled inconsistently (notably DNSH, state aid/public support, firm-in-difficulty checks).Stabilise methodologies: reduce ad hoc amendments, publish consolidated versions, and introduce advance notice (around two months) for major changes, supported by a central overview of legislative updates.Push digitalisation to cut excessive documentation requests: link better the EU funds monitoring system (MS2021+) and public registers, enable acts/communication via the system, and grant control bodies access to documents in MS2021+.Improve calls and project selection: more objective criteria for simpler projects, simplified applications, and better calibration of call parameters using absorption/interest analysis.Pursue targeted legal changes (incl. budgetary rules) to support results orientation and wider use of simplified methods, and to reduce EU–national inconsistencies affecting controls and procurement and horizontal principles.
Evaluation of the impact of measures under TO3 “SME competitiveness” of the Provence-Alpes-Côte d’Azur ERDF-ESF OP, 2014-2020
Evaluation d’impact de l’OT 3 « Renforcer la compétitivité des petites et moyennes entreprises » du Programme Opérationnel FEDER-FSE de la Région Provence- Alpes-Côte d’Azur 2014- 2020
ROP Provence Alpes Côte d'Azur 2014-2020
Léonor Rivoire, Elisabeth Zaparucha, Marie Dubus, Aurélien Fichet De Clairfontaine, Yvan Meyer, Francie Sadeski
Technopolis, https://www.technopolis-group.com
The evaluation examines the impact of the measures, focussing in particular on the use of financial instruments: “Region Sud Investissement” (RSI) and “Sud Garantie” (SG), a guarantee fund.
To assess the effects of the use of financial instruments (an investment fund (RSI) and a guarantee fund (SG)) on the performance of firms supported, in terms of turnover, value-added, exports and employment.
Analysis of monitoring data.Administrative data (from business registers).Interviews with beneficiaries, financial intermediaries and other stakeholders.ToC testing.20 case studies.Semi-counterfactual analysis.Qualitative comparative analysis.
The RSI fund responds to the needs of companies, which would not have been able to access funding from other sources.The fund has also helped mobilise additional resources from co-investors that perceived that it strengthened the regional economy.RSI beneficiaries generally report better performance in terms of turnover, employment and value-added than non-beneficiaries, but not exports.The SG fund seems to have been useful in improving development plans of beneficiaries, but it is uncertain whether it has been important in mobilising additional investment.
A semi-counterfactual analysis is carried out, as the number of variables is too limited to carry out a proper one. Beneficiaries interviewed are not the initial sample selected, as many of those included could not be reached.
Make the RSI fund permanent and shift the focus of the SG fund towards green investments.Strengthen monitoring of the funding allocated, especially as regards financial intermediaries, so as to facilitate impact evaluation of projects.Make the role of the RSI more visible to beneficiaries, as they normally have more contact with financial intermediaries.
Evaluation of PA2 “Adaptation to adverse climate impacts with a focus on flood protection” of the Quality of Environment OP, 2014-2020, in Slovakia
Hodnotenie dopadov intervencií OP KŽP v oblastiach podpory prioritnej osi 2 adaptácia na nepriaznivé dôsledky klímy so zameraním na ochranu pred povodňami
The evaluation examines the implementation and the effects of flood protection projects funded between 2014 and 2020, such as the construction of dams and rainwater harvesting. The evaluation aims to establish the extent of citizens protection and property protection against floods due to the projects funded.
To assess the effectiveness, efficiency and impact of measures on flood protection, which include the construction of dams and rainwater harvesting.
Analysis of administrative data (flood risk assessments).Analysis of project documentation and monitoring data (annual report of the implementation of the OP).Analysis of official statistics.
The measures substantially contributed to the safety of residents and property protection in high flood risk areas.The population benefiting from flood protection measures increased by approx. 1,000 inhabitants, primarily in the Prešov region. Additional projects still under implementation could protect around 4,000 inhabitants, mainly in the city of Banská Bystrica.The programme helped improve flood protection for properties valued at EUR 37 million. Additional ongoing projects could protect properties worth more than EUR 279 million.The most popular type of water retention measure was rainwater collection, which positively affect climate change mitigation and flood risk reduction.The total area with retention capacity in the built-up area is expected to increase by almost 480,000 m² by the end of 2023.
The relatively low number of completed projects focusing on flood prevention and water retention measures implies that the benefits achieved are still limited.
The evaluation does not include any policy implications.
Evaluation of interventions on RIS3 funded by the Campania ESF OP, 2014-2020
Strategia regionale di ricerca e innovazione per la specializzazione intelligente (RIS3)
The evaluation examines the measures on education and training under the OP, which contribute to the RIS3.
To assess the effectiveness of the interventions on education and training, their ability to reach final beneficiaries, their innovativeness and the critical aspects in their implementation.
Although the ESF measures on RIS3 have helped to strengthen the strategy, most of the 2018 milestones were not achieved.ESF measures are mainly focussed on developing competences in technology-intensive sectors, providing support for start-ups, especially in the green economy, and improving digitalisation in schools.Public and private expenditure on research increased significantly in Campania in the period 2016-2018, but remains lower than the national and EU average.There are positive results for indicators on collaboration between companies and research centres, patent applications, use of ICT and innovation of the productive system.
Results of projects to improve health security and the efficiency of the healthcare system under the Infrastructure and Environment OP,2007-2013
Podsumowanie efektów wdrażania projektów w priorytecie XII Programu Infrastruktura i Środowisko
Krzysztof Gutta, Dr Tomasz Holecki, Dr Martyna Wronka, Piotr Gozdek, Joanna Czerepko
EU-CONSULT Sp. z o.o., ul. Wały Piastowskie 1, 80-855 Gdańsk, www.eu-consult.pl
This evaluation assesses the extent to which the goals set under Priority Axis XII of the Infrastructure and Environment OP to improve health security and the efficiency of the healthcare system were achieved.
To assess whether the main Priority Axis goals were achieved and to estimate the expected long-term effects of projects implemented under Measure 12.1 “Development of the medical emergency system” and Measure 12.2 “Investment in supra- regional healthcare infrastructure”.
Desk research;CAWI survey of beneficiaries;case studies of selected projects;in-depth interviews;panel of health experts.
All the targets at national level were achieved;the projects implemented helped reduce the mortality rate of patients and to increase the number of specialised health services;overall, the projects were characterised by relatively high efficiency, sustainability and usability;the impact on the number of patients treated per year was limited;from a long-term perspective, the projects improved the comfort and safety of patients and increased the availability and quality of diagnostic and therapeutic services;the projects improved technical infrastructure and the quality of medical equipment in health facilities.
There is a need to:increase the number of context indicators;increase the medical facilities located in the provinces;reduce restrictions on the use of equipment purchased under the OP;implement additional projects relating to geriatrics and psychiatry.
Impact evaluation of interventions under PA III and IV of the Infrastructure and Environment OP 2014-2020 to improve road accessibility and traffic in cities
Wpływ działań podejmowanych w ramach III i IV priorytetu Programu Infrastruktura i Środowisko 2014-2020 na poprawę dostępności drogowej i obciążenie ruchem w miastach
Marta Kindrat, Jacek Malasek, Marcin Świtała, Jakub Zawieska, Agnieszka Łukasiewicz, Leszek Kornalewski, Tomasz Polichnowski
Instytut Badawczy Dróg i Mostów, 03-302 Warszawa, ul. Instytutowa 1, email: ibdim@ibdim.edu.pl, www.ibdim.edu.pl, MGR MARTA KINDRAT, POLITRAFFIC SP. Z O.O., 00-867 Warszawa, al. Jana Pawła II 27
The report summarises a pilot study of two road investment projects (Poznań and the Lublin Functional Area) co-financed under PAs III (“Development of the TEN-T road network and multimodal transport”) and IV (“Road infrastructure for cities”) of the Infrastructure and Environment OP, 2014-2020, in Poland.
To develop, test and select the most effective and useful methods to assess the impact of measures taken under PAs III and IV on improving road accessibility and traffic congestion in cities, identifying difficulties related to their implementation during the pilot study.
Road traffic modelling;4 PAPI and CAWI surveys of 400 car and service users, 400 residents without personal transport and 110 freight companies;analysis of vehicle traffic volumes;cost and benefit analysis;spatial analysis;desk analysis of data from the Central Statistical Office's Local Data Bank and the Police Accident and Collision Records System.
The investments in Poznań and Lublin have resulted in shortening travel time, increasing average speed of vehicles and improving road safety. In Poznań, the number of road accidents fell by 54%, in Lublin by 74%, fatalities by 50% and those injured by almost 78%.28% of enterprises in Poznań and 19% in Lublin reported that the road improvements had a positive effect on their financial results.The investments contributed significantly to the development of an integrated transport system, improving accessibility, reducing road noise and the concentration of pollutants in the air.It is estimated to take 5 years for investment costs to be covered by benefits in Poznań and 3 years in Lublin.The use of traffic models enables the effect of investment on travel times, the development of the transport network, traffic flows and improvement in accessibility to be assessed.Analysis of statistical data on road accidents and so on, along with surveys of road users, is considered sufficient to assess the impact of road investment on road safety.Area-based traffic analysis enables the net effect of road investment on travel costs to be assessed.Analysis of road traffic volumes from GPS and surveys enables the development local economic activity to be assessed.Data on transport emissions, pollution and road noise, surveys of users and spatial analysis enable improvements in the urban environment to be measured.
The limited availability of updated area traffic models, which currently exist only for some urban agglomerations.
The economic impact of road investment on travel costs should be assessed using area-based traffic models.Despite the cost, CAWI survey methods should be used to improve the quality of data obtained. Questionnaire surveys should be conducted 2-3 years after an investment project has been completed.Analysis of road traffic safety should be carried out using the police database on road accidents and collisions (SEWiK, which contains details of the type, causes, consequences and location of the accident).
Evaluation of the effectiveness and efficiency of support for employment and lifelong learning under the Pomorskie OP, 2014-2020
Ocena skuteczności i efektywności wsparcia w ramach Osi Priorytetowej 5 Zatrudnienie Regionalnego Programu Operacyjnego Województwa Pomorskiego na lata 2014-2020
The evaluation examines the support for the vocational and educational activities for the unemployed and those at risk of unemployment.
To assess the impact of support on the increase of employment, the popularisation of lifelong learning and the availability of childcare for children up to the age of 3.
Desk analysis of relevant documents.Analysis of monitoring data and official statistics.CAWI/CATI survey of 228 beneficiaries.CAWI/CATI survey of 2,289 final recipients.11 interviews with implementing bodies.5 case studies
95% of participants who received a grant started their own business; more than half (56%) of the new companies were still operating at the time of the evaluation (one to six years after the end of the project).Childcare places created under the OP in 2020-2022 accounted for approximately half of all childcare places created in the region.69% of participants in training projects reported increased knowledge or skills. 13% of participants found employment.
Include psychological support as part of labour market projects.Implement more comprehensive diagnostic activities in relation to target groups.Implement solutions that support the creation of effective project partnerships.Support the adaptability of employees through individualized counselling.
Annual evaluation of the Marche ESF OP, 2014-2020 – 2020
Marco Pompili, Isabella Giorgetti - Ismeri Europa
The evaluation examines progress in the implementation of the OP up to end-2020.
To assess the validity of the OP strategy, especially in respect of Covid-19, progress in achieving objectives, the impact of interventions for the long-term unemployed, the adequacy of the monitoring system, and the effects of specific measures on gender equality.
Official statistics.Administrative data.Monitoring data.Interviews with MA.Survey of beneficiaries using the monitoring system (258 people).Survey of beneficiaries (585) and non-beneficiaries (214) of childcare vouchers.Evidence from previous evaluations.
The strategy of the OP is valid to respond to the effects of Covid-19 and implementation shows a good progress.One month after participation in employment measures, employment rates are higher for hiring incentives and education pathways, especially for men and the unemployed with previous work experience.Job fellowships and traineeships seem to be the most effective measures for the long-term unemployed, work experience and training vouchers seeming to be less effective.The monitoring system functions well overall, though in some cases beneficiaries consider it to be too rigid and insufficiently clear.Vouchers for childcare have led to higher quality care being chosen in many cases and have helped balance work and family responsibilities, especially for low-income families.
Ensure completion of 2014-2020 expenditure and a prompt start to the 2021-27 OP, as well as better coordination with other national and EU funding sources.Target job fellowships on long-term unemployed and traineeships more widely, and involve the PES more in the implementation of active labour market policies.Improve technical aspects of the monitoring system.
Evaluation report on the Competencies for development ESF OP, 2007-2013
Rapporto di Valutazione Complessiva PON FSE 2007/2013
NOP Competencies for development ESF 2007-2013
Fulvio Pellegrini, Claudia Villante, Claudio Maria Vitali, Lorenzo Mari
Deloitte Consulting srl. Via Tortona 25, MilaNo, Italy
The evaluation assesses the effects of the ESF National operational programme “Competencies for development”. The objective of the programme is to improve the quality of education and to strengthen the management capabilities of stakeholders and their self-evaluation capacity. The evaluation makes use of theory-based approaches as well as interviews, focus groups and surveys in order to identify the changes resulting from the implementation of the OP, to assess the appropriateness of the activities supported and to provide recommendations for future policies.
To measure the effects of the OP and to identify the strengths and weaknesses of the intervention logic and the implementation of the programme.
Desk analysis of programme monitoring documents;literature review;theory-based Programme Evaluation;focus group, semi-structured interviews, CAWI surveys.
The OP has promoted a cultural change among education stakeholders (in terms of management, performance measurement, self-diagnosis and evaluation) especially at central level.At local level very uneven results have been achieved in terms of improving education practices and of increasing the evaluation of education systems.While training initiatives reached a large number of people, they often do not provide a clear vision on how the skills taught could be used in the long-term.Key competencies and skills are still insufficient.Non-attendance levels have not improved.ICT and foreign language knowledge have improved significantly and the results of integrated learning activities abroad are very promising.ICT and web-based teaching opportunities are lagging behind due to low knowledge levels among teachers and infrastructure deficiencies.
Lack of ex-ante indicators for some areas (e.g. as regards the quality of educational and self-evaluation systems); a lack of previous evaluations on the themes covered.
There is a need to:Increase homogeneity of management practices and individual competences in order to obtain minimum common levels of achievement.Increase innovation in management approaches and learning activities and award beneficiaries able to produce solid results and make efficient use of investment.Define clear and measurable quality standards at national level for the educational system.Develop exchange of experience between schools.Foster international learning opportunities and the creation of international networks.Increase key competences and clarify their relevance in ordinary daily activities.Encourage the development of ICT infrastructure and skills in schools.Define a national strategy against school drop outs.
Evaluation of In-Work Support, 2014-2020
Evaluation of In-Work Support – Theory of change
OP West Wales and the Valleys ESF 2014-2020
Institute for emplyment studies and Old Bell Consultancy
The report is aimed at developing a theory change for in-work support (IWS), funded from 2015. It is the product of the second phase of an evaluation process composed of five stages.
To develop a theory of change of in-work support, in order to test whether IWS works in the way it was originally designed to do.
Desk analysis of programming documents and previous evaluations and studies on similar interventions;interviews with policymakers from the Welsh Government and senior representatives of delivery organisations;a workshop with stakeholders;a cross cutting analysis of findings from the above.
The IWS should focus on the most vulnerable groups in terms of work-limiting health conditions;several activities (levers for change) are expected to work: work focused health intervention (to facilitate the return to work), early or rapid intervention, flexible personalised support, multiple referral routes (in order to generate a more diverse population of participants);the main expected outcome is the return of participants to the labour market through, for example, reduced pain, and improvements in self-confidence of those with psychological problems.
Long term outcomes could not be studied within the timescale of the evaluation.
Policy implications are not included since it is only a part of an evaluation which will be completed in August 2018.
Impact evaluation of PA 2 and operations targeted at older workers under the ESF OP, 2014-2020
Evaluation d’impact de l’axe 2 du Programme Opérationnel national FSE 2014-2020, élargie à toutes les opérations du PON FSE pour le public sénior
The study examines PA2 of the OP, adaptation of workers, enterprises and entrepreneurs to change and healthy ageing measures covered under other priorities.
To assess the implementation of PA2, on adaptation of workers and enterprises to change, and its impact and to draw lessons for 2021-2027.
Desk research.Online surveys of participants (response rates 8-10%).43 interviews with stakeholders.Workshop.
The intervention strategy of PA2 is considered relevant to labour market needs of the workers.Projects selected, largely for training, accounted for 85% of the budget at February 2020.Vocational training of employees (SO3) and of those made redundant (SO4) accounts for 97% of participants.The results do not meet gender equality and healthy ageing objectives.The cost per participant finding a job averaged just over EUR 3,000, higher than under the other priorities.The ESF has increased the number of participants receiving support.For older workers, the budget is insufficient and there are strategic, operational and cultural barriers to reaching the target group.
Continue supporting the same types of measure.Extend support beyond training.Promote innovative projects in particular in terms of gender equality and healthy ageing.Take better account of the territorial dimension in the allocation of the budget.
Evaluation of the Italy-Croatia Interreg OP, 2014-2020 - Year 2021
Interreg V A Italy Croatia CBC Programme 2014-2020 - Operational Evaluation 2021
Lattanzio, www.lattanziokibs.com - IRIS, www.irisricerche.it - ACZ consulting, www.aczconsulting.ro
The evaluation examines the implementation of the OP.
To assess the implementation of the OP up to 2021.
Desk analysis.Analysis of monitoring system.Interviews.
There were delays in the start-up phase of the OP mainly due to the MA and JS being set up late, with the top positions changing during the programming period and the operational staff being incomplete for long periods. This also caused management problems, such as the lack of comprehensive and consolidated procedures and documentation.The creation of a new IT system for the monitoring system caused some delays and is considered not to be user-friendly by beneficiaries.Result indicators are relevant and specific and linked to the objectives and are generally suitable for monitoring and evaluation purposes.The share of partner organisations is quite balanced between Italy and Croatia, but this is less so for the lead partners, 77% of them being based in Italian regions (mainly Veneto, Friuli Venezia Giulia and Emilia Romagna).The overall involvement of most relevant partners is considered satisfactory.A good level of vertical partnership through cooperation of central and local bodies has been activated.
Avoid further changes in the MA and JS staff.Improve the cross-border dimension especially at project level.Develop a set of common indicators able to capture the cross-border dimension.Identify more exact targets that can better reflect the implementation stage of the OP.Consider the possibility of creating a set of network indicators.
Mid-term evaluation of the Amazonia Interreg OP 2014-2020
Évaluation à mi parcours du Programme de Coopération INTERREG Amazonie 2014 2020
Service Coordination Interfonds Pôle Affaires Européennes Guyane, www.europe-guyane.fr
This is the mid-term evaluation of the Amazonia Interreg OP which supports the integrated development of the French department of Guyane with the neighbouring Republic of Guyana, Suriname and the Brazilian states of Amapá, Pará and Amazonas.
To assess the territorial cooperation strategy, the relevance and coherence of the OP and progress in implementing it.
Desk research;monitoring data;semi-structured interviews with stakeholders (8);survey of beneficiaries.
The OP remains fully relevant to the challenges faced in the region.At mid-2019, expenditure allocated amounted to 69% of the budget, but less than 9% had been paid because of implementation delays.Apart from PA3 (natural/cultural heritage and biodiversity at cross border level) and PA8 (SME competitiveness at transnational level), the milestones have been reached.The communication strategy is considered to be efficient. 90% of the beneficiaries are satisfied with the information received and its accessibility.According to the survey of beneficiaries, the OP has helped to increase the span and quality of their projects, and for half of respondents, to introduce new working methods.
The report reflects the situation as at June 2019. The monitoring system was late in being implemented.
Strengthen the links with the non-EU OP partners.Improve the efficiency of the joint Secretariat.Translate into English all information on the OP published on the ‘Europe en Guyane’ website.
Evaluation of regional investment aid in regional OPs 2014-2020 in Poland
Ewaluacja regionalnej pomocy inwestycyjnej w zakresie celu tematycznego 3 w obszarze wzmacniania konkurencyjności mikroprzedsiębiorstw oraz małych i średnich przedsiębiorstw w ramach regionalnych programów operacyjnych na lata 2014-2020
All ROPs 2014-2020; OP Development of Eastern Poland 2014-2020
EVALU Sp. z o.o. (lider konsorcjum), ul. Dzika 19/23 lok. 55, 00-172 Warszawa, e-mail: evalu@evalu.pl; Stowarzyszenie Na Rzecz Rozwoju Rynku Pracy S-T-O-S, ul. Jana Brożka 18/4501-451 Warszawa, e-mail: stos@stos.org.pl; WiseEuropa, Aleje Jerozolimskie 99
The study examines regional investment aid (RIA) granted under 16 regional OPs.
To provide knowledge and evidence on the effectiveness, efficiency, proportionality and adequacy of regional investment aid.
Desk research of the programming documents, public statistics, evaluation reports and monitoring data905 CAWI interviews with beneficiaries and non- beneficiaries48 IDI with institutions involved in the implementation48 TDI with experts15 case studies6 FGI with enterprises and socio-economic partners3 expert panelsCounterfactual analysis (Regression Discontinuity Design, Difference-In-Differences, Propensity score matching)Macroeconomic modelling
« Pure » incentive effects (projects not being undertaken without public funds) in the case of RIA occurs rather rarely.The scale of investment of RIA beneficiaries was 2.7 times larger than in the 2 years preceding the support, while private capital involved 1.7 times larger.Revenue, productivity, entering new markets, and increasing market share increased in companies supports compared to the year preceding the support.Each PLN 1,000 of co-financing generated an increase in beneficiary expenditure on innovative activities in 2015-2017 by an average of PLN 1,509. But the support was not fully effective in relation to expected innovativeness.Employment growth in RIA beneficiaries was 4 p.p. higher than among unsuccessful applicants.Counterfactual analysis indicates a strong positive effect on GDP, employment, private investment and exports.
Implement three kinds of support for SMEs:a "wide scale" measure with a large allocation for financing ambitious development projects of technological innovation;a grant-based measure targeted at enterprises generating R&D results with significant levels of new knowledge;a support measure targeted at micro-enterprises and recently established businesses.
Operational Evaluation of the Cantabria OP reprogramming
Informe de Evaluación operativa por reprogramación del Programa Operativo FEDER de Cantabria 2014-2020
GobierNo de Cantabria, C/ Peña Herbosa 29, 39003 Santander, Cantabria, TeléfoNo: 942 207 100 (Centralita)
The evaluation assesses the financial reprogramming and implementation of the Cantabria OP for 2014-2020.
A combination of qualitative and quantitative methods based on secondary, mainly financial, data from the Fondos2020 accounting system. Official statistics (e.g. Eurostat, Instituto Cántabro de Estadística, INE, etc.) and relevant documents (e.g. EU 2020 Strategy, Partnership Agreement, OP, etc.) are also used.
The reprogramming concerned additional support from the ERDF of EUR 3.6 million, which is entirely allocated to TO 2.The reprogramming consisted in transferring funding from TO 1 and 3 to TO 2, 4, 9, 10 and changing some investment priorities (IP) in TO2 and 4. It also consisted in fixing a number of additional specific objectives (SO). Some indicators have been modified and target values adjusted.Despite the changes, the reprogramming did not change the OP strategy or the intervention logic.
Since there were not data on certified expenditure available, the analysis of progress in financial implementation is based on actual spending.
The reprogramming entails an increase in total financial resources (+ EUR 3.6 million) and does not involve any significant change in the OP strategy.
Evaluation of interventions to improve the quality of drinking water and reduce the environmental impact of wastewater under the Environmental and Energy Efficiency OP, 2014-2020, in Hungary
Az ivóvízminőség javítását és a szennyvizek okozta környezetterhelés csökkentését célzó fejlesztések komplex értékelése
OP Environmental and Energy Efficiency 2014-2020
Aqua Engine Kft. (Orsolya Kornélia Nagy) - Collective-Intelligence Kft (Tamás Cserneczky)
http://aquaengine.hu/ - 2040 Budaörs, Szép utca 50-52. A. Ep.
The evaluation examines progress up to June 2020 of interventions to improve the quality of drinking water and reduce the environmental impact of wastewater under PA2 of the OP.
To assess the implementation and results of the measures implemented and their consistency with the Drinking Water Directive and make recommendations for improvement.
Desk analysis of previous evaluations/studies.Monitoring data.Statistical analysis of primary and secondary databases.Interviews with target organisations.Survey of 40 water service managers.Case studies.
The number of water supply areas not complying with the Drinking Water Directive was reduced to 14 out of 365, 4% of the total. Around 114,000 people are still affected by the problem of lead water pipes.45% of the drinking water supply network is older than it was planned to be (30 to 50 years) and the sewerage system is in poor condition, as reflected in a 41% increase between 2012 and 2017 in the number of faults reported.Investment in wastewater management led to a significant extension of the sewerage system, but there is still a risk of the European Commission initiating infringement procedures.The physical and financial progress of the interventions is slow, and there is a risk of losing funding.Some of the indicators are not adequate for monitoring the physical progress of projects since targets are only verifiable after measures have come to an end.The lack of support for beneficiaries for preparatory activities has adversely affected the technical content of some projects.COVID19 led to only sporadic shortages and slight delays in the supply of building materials.
Energy efficiency is a contractual requirement that is difficult to monitor in qualitative and quantitative terms.There is an anomaly in the way the "Additional population benefiting from a better water supply (CO18)" target is fixed.It is difficult to verify the actual values of ongoing projects before their conclusion by analysing the EUPR data.At the time of completing the report it was impossible to estimate the effects of the pandemic as the second wave was still ongoing.
Increase investment in improving the efficiency of water supply networks.Review regularly water safety plans and make the activities eligible for future calls for proposals.Continue developing and improving the wastewater disposal and treatment system to achieve the required performance and mitigate the sanctions from the ongoing infringement procedure.Assign a broader and more defined role to the operators in preparing and implementing projects to ensure they are carried out smoothly.Use result indicators (RCR 41-44 and CCR 09) linked to the new priorities for 2021-2027.
Study on gender budgeting under the Bund ESF OP, 2014-2020 – year 2020
Institut für Sozialforschung und Gesellschaftspolitik, Weinsbergstr. 190, 50825 Cologne, www.isg-institut.de, e-Mail: viedenz@isg-insititut.de
The report examines the degree of participation, the level of expenditure and the achievement of targets by gender. Data go up to end-September 2020.
To analyse the activities and operations implemented under the ESF OP from a gender perspective.
Women account for 44% of total expenditure (target 46%).On average, EUR 4,204 has been spent per participant, for women EUR 845 lower than for men.Overall, women make up 49% of participants.63% of projects are gender-oriented and only 0.4% gender-specific.
Data on financial resources are available only at project level, not at individual level. For on-going projects information on actual resources spent is not available. Other variables are missing: for 14,100 people there is no information on education level, for example.
Interim Report on the Partnership Agreement
Relatório Intercalar do Acordo de Parceria: 2017
The evaluation concerns the Partnership Agreement for Portugal 2014 -2020 (PT2020).
To assess progress towards the achievement of the PT2020 targets.
The evaluation uses both qualitative and quantitative methods:Desk research.Analysis of relevant documents and of secondary data from the monitoring system.
Main outcomes by area:R&D: By June 2017 about 800 projects were contracted (123% of the 2018 target).SMEs: approximately 700 projects contracted by June 2017 (106% of 2018 target).Climate change adaptation: 43 km of coastal strip approved (of which 5 km completed) (2020 target: 50 km).​​Environment and efficient use of resources: 45% of resources committed and EUR 831 million allocated. Additional 1 million people committed to be served by improvements in the wastewater system and 105 thousand by improvements in water supply. 19,380 square metres of urban area rehabilitated.Transport: 2.5 km of regional highway (Madeira) completed by end of 2016. EUR 90 million of eligible investment approved.Employment: almost 40,000 young people and 11,000 adults supported in finding a job by the end of 2016. In 2016, about 26,700 young people and 2,800 adults attended training courses. 78% of adults and 51% of unemployed young people attending the training succeeded in finding employment, up to 4 weeks after the training. (For the Youth Employment Initiative: 25% of unemployed young people obtained work after 4 weeks following the end of a programme and 78% 6 months later).Health and social infrastructure: 96 social and health facilities approved (92% and 24% of the target for 2018 and 2023, respectively), covering 2.7 million people (2023 target: 4.5 million people).Education: 21,000 young people received training at ISCED 2 level and 112,000 young people at ISCED 3; there were 76,000 scholarships to higher education at ISCED levels 5, 6 and 7 and 2,320 doctoral fellowships. 17,000 adults were supported in school and/or vocational certification courses.
Mid-term Evaluation of the ESF Islas Baleares OP 2014-2020
Informe de Evaluación del Programa Operativo FSE de las Islas Baleares 2014-2020. Período 2014-2016
This is the mid-term evaluation of the ESF Islas Baleares OP 2014-2020.
To assess the OP’s implementation and achievements and to examine whether the logical framework is still appropriate.
The evaluation adopts a mix of qualitative and quantitative methods:Desk research;analysis of relevant policy documents;analysis of monitoring system data;analysis of the socio-economic context;in-depth interviews.Data used come from interviews and the monitoring system as well as from Official statistics (e.g. Eurostat, Instituto Nacional de Estadística, Observatorio Español del Emprendimiento, IBESTAT, among others).
The OP continues to respond to regional needs.The rate of OP implementation is in line with expectations.The OP’s design and implementation respect the horizontal principles.
Data go up to 31 December 2016. The assessment is based only on data on activities completed. Data on certified expenditure are not available.
For the result indicator CR04 ‘Participants who obtain a job, including on their own account, after their participation’, the baseline value and the target should be reviewed and adapted to the state of implementation.The initial conservative targets for the employment of research staff should be reviewed.The finance for the inclusion of vulnerable groups should be increased.
Evaluation of the measures for the employment of women under the “Active labour market policies” OP, 2014-2020, in Italy
Adeguatezza e rilevanza delle azioni introdotte nei territori volte a sostenere l’occupazione femminile
NOP Active Employment Policies 2014-2020
https://www.lattanziokibs.com/, https://www.irisricerche.it/
The evaluation examines the implementation of measures for the employment of women under the OP, up to the end of 2022.
To assess the adequacy and relevance of the measures implemented to help women participate in the labour market.
Analysis of official statistics.Interviews with the MA.Previous studies (based on a survey).
Projects implemented through the OP mainly concerned guidance and innovative methods by regions and territorial services to improve the work-life balance of women and to promote organisational innovation in firms.Companies participating in the training on flexible work are satisfied with its relevance, content and the material received but less with the timing and the long duration of activities.
Make wide use of the results obtained by experimental projects implemented by Regions and territorial service providers. Make the documentation developed by projects on organisational innovation in firms available to all users, to facilitate alignment to national legislation.Strengthen relationship networks developed by the projects.Organise further activities with schools and young people.
Evaluation of the implementation of the Ipeiros OP, 2014-2020 – 2nd assessment
2η Αξιολόγηση Εφαρμογής του ΕΠ Περιφέρειας Ηπείρου 2014-2020
Artemidos 3, Marousi, 151 25, tel: 2106725966, www.remaco.gr
This is a third deliverable of the second assessment of the implementation of the OP.
To assess the implementation, effectiveness and efficiency of the OP, the degree of achievement of Performance Framework objectives, and the timeliness of the intervention logic up to May 2021.
Documentation analysis.Analysis of monitoring and statistical data.
99% of the OP budget has been committed and 45% absorbed as of May 2021.Out of 5378 projects, 139 had been implemented up to May 2021.The implementation of TO9 ERDF measures is the most effective and that of TO3 measures the most efficient. The implementation of TO2 and TO8 measures is least effective and efficient.Most of the output indicators are expected to be achieved.The Performance Framework targets are expected to be met or overachieved, with an exception of PA3 indicators and PA2 one.The implementation of interventions was affected by delays in fulfilling conditionalities, the new project assessment and approval process, legal difficulties in selecting the intermediary, the reform of the Institutional Framework for Public Procurement, the lack of maturity of projects and the time required for ITI interventions to mature.The intervention logic remains timely.
Some data were incorrectly included in the information system.
Revise the OP and adjust the Performance Framework indicators as well as co-financed public expenditure under the PAs on the basis of two scenarios available.
Assessment of the readiness of Polish enterprises to be involved in public procurement and public-private partnerships
Ocena gotowości polskich przedsiębiorstw do realizacji zadań publicznych
Instytut Badawczy ProPublicum ltd., 01-515 Warszawa, al. Wojska Polskiego 29/138, www.propublicum.pl
The evaluation examines training and advisory projects in respect of public procurement (PP) and public-private partnership (PPP) funded by the Knowledge Education Growth OP 2014-2020 in Poland. The projects were aimed at informing SMEs about PP and PPP and increasing their involvement.
To analyse the involvement of SMEs in PP and PPP and to assess the effectiveness of the projects carried out to make SMEs aware of PP and PPP and to increase their involvement.
Desk analysis;70 interviews with enterprises (50 supported and 20 not supported) in relation to PP projects;30 interviews with enterprises in relation to PPP projects;20 interviews with representative of PARP (Polish Agency for enterprise Development).
Public procurement projects:The main reasons for the low level of participation of SMEs in PP are a lack of finance; the high level of bureaucracy and poor quality of tender procedures. Most respondents perceived PP to involve low prices and consequently low profits.The participation in advisory and training projects was high.There were limitations in the implementation of training activities, e.g. enterprises from different sectors and with different level of experience of PP were mixed together.Beneficiaries provided high-quality trainers and advisors.No actual changes were observed in the level of participation of enterprises in PP.PPP projects:The main difficulties discouraging SMEs from entering into PPPs are the small number of them in Poland; the complicated procedures, a lack of finance and the risk associated with the long-term nature of the investment involved.Only 178 enterprirses received training on PPP (as of August 2018), 19% of the planned number.A limitation was the exclusion of large companies and municipal enterprises which are those most interested in PPP.The content of the training was positively assessed.Overall, the expected results in the form of more firms being interested in PPPs were achieved.
It is worthwhile maintaining the current formula for the distribution of training and consultancy support and maintaining the selection criteria.Training groups should include participants with similar characteristics in terms of sector and experience.Advisory and helpdesk points should be considered to support enterprises already involved in PP.Large enterprises and public companies should be admitted to the training on PPP.
Evaluation of the implementation system of the South Muntenia ERDF Programme, 2021-2027
Evaluarea sistemului de implementare a PR Sud-Muntenia
office@uniques.ro; andreea.nazarie@uniques.ro
The report examines the implementation system of the programme. The analysis focuses on the effectiveness of call launch, project selection and contracting procedures, as well as the administrative capacity of the Managing Authority and regional beneficiaries. Data used go up to May 2025.
To identify structural barriers and best practices to improve fund absorption and the relevance of interventions in relation to the territorial needs.
Analysis of the intervention logic and the Theory of Change of the programme.Document analysis of programmes, guidelines, and procedures.Analysis of statistical and monitoring data.SWOT analysis applied to the intervention logic and procedures.37 Semi-structured interviews with MA staff, Monitoring Committee members, beneficiaries, and consultants.2 Focus groups with MA and other stakeholders.4 Case studies and comparative analysis to identify best practices at a European level.Online surveys of beneficiaries/applicants (414 responses) and the MA (80 responses).Calculation of the Administrative Capacity Index, based on checklists.Use of the Kirkpatrick Model to assess improvements in administrative capacity.
The intervention logic of the programme is robust and aligned with territorial needs, although some targets are ambitious relative to initial spending capacity.The MA demonstrates high technical competence, but faces challenges related to excessive staff workloads and the need to optimise internal workflows to reduce the evaluation timeframe.The capacity of beneficiaries varies to a large extent. Larger public authorities are experienced, but SMEs and smaller municipalities remain heavily dependent on the MA technical assistance.Call launch and contracting procedures are transparent, but administrative bottlenecks persist, slowing the transition from application submission to contract signature.The use of digital tools and technical manuals has improved procedural clarity, but further simplification is required to reduce the bureaucratic burden on applicants.The programme is well-aligned with national strategies and European Green Deal objectives, and ensures strong complementarity with other funds such as the RRF.External factors like legislative instability and complex public procurement rules continue to hinder the overall pace of implementation.
The limitations include the reduced representativeness of the surveys due to voluntary and online participation, the bias in internal data caused by incomplete questionnaire responses from MA staff, and the absence of certain key stakeholders in interviews.
Simplify and standardise technical documentation and call procedures to reduce the bureaucratic burden, error rates, and implementation timeframes.Invest in human capital through staff stabilisation and the provision of continuous training to prevent excessive workloads and the loss of technical expertise.Provide tailored technical assistance and mentoring for less experienced beneficiaries, focusing on critical issues such as green procurement and financial management.Accelerate the digitalisation of the entire project lifecycle to ensure greater transparency, speed, and a reduction in paper-based interactions.Strengthen strategic coordination between the regional programme and other programmes, such as the RRF, to avoid duplication and maximise territorial impact.Align monitoring with the "Theory of Change", prioritising the assessment of actual socio-economic outcomes over mere administrative expenditure.
Evaluation of the ReAct III Programme (2015-2019) financed under the West Wales and The Valleys ESF OP, 2014-2020
Evaluation of the ReAct III Programme: 2015 to 2019
Endaf Griffiths, Tom Marshall, Paula Gallagher, Sam Grunhut, Duncan Melville, Paul Bivand and Conor Stevens
The evaluation examines the REACT III programme, which provides support for people who have been made redundant or are under notice of redundancy, up to end-2019.
To assess the performance of the programme, the way support is provided and the impact and to make recommendations.
Desk research and literature review.Interviews with stakeholders, including Welsh Government officials and those involved in ReAct.Survey of 1,155 participants and interviews with 50 participants.Survey and interviews with businesses supported.4 case studies.Benchmarking.Cost–benefit analysis.
The programme is demand-led with no specific target group set other than redundant workers. This could lead to underrepresentation of categories that are most in need of support.Training programmes are well designed but need to be reviewed on an ongoing basis to ensure their effectiveness.Overall, the programme response to redundancy is quick and effective thanks to well established procedures, but how far they work in all areas, especially rural ones, needs to be investigated.Six months after the training, 80% of participants had found a new job, but some of them reported that the training did not affect the job they obtained but it helped them acquire the skills needed in their desired job.The administrative procedures are still burdensome.
The limited number of responses in the survey of businesses limits the validity of the results. The lack of data needed to build a valid control group for the counterfactual analysis led to a benchmarking approach which is weak in any case because of data scarcity.
Improve the data collection system.Periodically review the training programmes to ensure they are in line with needs.Target of people with most needs.
Evaluation of measures for the modernisation of higher education institutions under the “Science and Education for Smart Growth" OP, 2014-2020, in Bulgaria
Процедурата „Модернизация на висшите училища” в България: Уроци за образователно програмиране: Междинна оценка
The evaluation examines the effects of support for the modernisation of higher education institutions under the OP, implemented from August 2020 to December 2023. Data used goes up to June 2024.
To assess effectiveness, efficiency, impact, relevance, coherence, EU added value, and sustainability of the measures for improving the quality of education through digitalisation, training, and international cooperation.
Analysis of monitoring data and statistical office data.Realist Evaluation Approach (TBE).Interviews with 16 stakeholders and focus groups with 7 beneficiaries, field visits to 3 universities.
The modernisation measures were in line with national and European priorities, but financial resources for them were limited.The lack of experts caused inconsistencies between policies and the projects implemented, which limited the overall effectiveness.The main objectives, such as the adoption of digital technologies and continuous training for academic staff, were partially achieved. However, there were delays for measures for international mobility of students, PhD candidates, and researchers, due to legal and financial issues, and the impact of pandemic on mobility.The sustainability of results is a major concern, with many joint programmes unlikely to continue in the long term, although EU funding was crucial for the large-scale implementation of these projects.
Insufficient data on the concrete outcomes and impact, such as students' learning improvements, made it difficult to conduct a comprehensive and accurate evaluation of the effectiveness.​
Streamline administrative and financial procedures to reduce bureaucracy and improve efficiency in the management of funding and projects.Design policies that ensure the sustainability of initiatives beyond EU funding and enable long-lasting impact.Create more flexible opportunities for international mobility for students and researchers.
First intermediate evaluation of the Valle d'Aosta ERDF OP, 2014-2020
Primo Rapporto di valutazione intermedio - FESR
The evaluation examines the implementation of the Valle d’Aosta ERDF OP and the progress made.
To assess progress in implementing the OP in relation to the context, the targets set, adequacy of management and control procedures and contribution to macro strategies.
Monitoring data.National statistics.Interviews with MA.Desk analysis.
Financial and physical progress is overall satisfactory, but PA3 (SME competitiveness) and PA4 (Energy efficiency) are lagging behind.Delays for PA3 are due mainly to lack of interest and limited investment by companies and for PA4 to difficulties in launching new complex projects.Main management problems are insufficient staff and their limited competence and complexity of the monitoring system.Sustainable development aspects have been largely taken into account in the implementation of the OP but equal opportunities less so.
A few evaluations questions could be answered only partly because some projects were still to start.
Check the expenditure capacity of PA3 (SME competitiveness), and if needed shift funding from PA3 to PA1 (R&D).Monitor the implementation of delayed projects and check the suitability of an indicator in PA4 (Energy efficiency).Monitor training needs of MA staff and beneficiaries to speed up implementation.
Evaluation of the Slovenia-Austria Interreg V-A OP
Evaluation of the Cooperation Programme INTERREG V-A Slovenia-Austria
The evaluation concerns the Slovenia-Austria Interreg Programme 2014-2020.
To assess the efficiency and effectiveness of the OP structure and procedures, including project application and selection procedure, as well as the progress of implementation and the communications strategy.
The evaluation use both qualitative and quantitative methods:desk research;analysis of relevant documents;on-line survey of lead partners;in-depth interviews with selected programme bodies (MA, Joint Secretariat, National/Regional authorities, First Level Control).The data used are: primary data from interviews and surveys and secondary data from official documents and programme monitoring data.
Overall, the programme structure is fairly effective with room for improvement as regards cooperation at the programme level (especially on IT support on the electronic Monitoring System – eMS).The programme is likely to progress relatively well in all PAs, with some exceptions.The geographical distribution of the resources is uneven. The concentration of project is greatest in Podravska, Osrednjeslovenska, Pomurska and Graz. Other regions (e.g. Zasavska) receive less resources.Progress towards achieving the programme and project level communication objectives is very good.
The evaluation highlighted that the projects are still in the early phase of implementation and no reimbursement from programme funds has yet been made.
The MA and the Joint Secretariat should organise Bilateral Working Groups more often in order to address issues identified in the implementation of the programme.The implementation manual and workshop should be complemented with lessons learned from the quality assessments of application and practical examples.Improvements should be introduced to the eMS monitoring system.The MA and the Joint Secretariat should consider the possibility of adjusting reporting periods and avoiding overlaps with other programmes. They should also alleviate problems causing delays in the certification of expenditure.The MA, the Joint Secretariat and National/Regional authorities should increase the generation of projects in areas not yet covered sufficiently by those so far approved.
First evaluation report on Smart Specialisation Strategy under the Marche ERDF OP
Lattanzio, www.lattanziokibs.com – IRIS, www.irisricerche.it – ISRI, www.isril.it – CLES, www.cleseconomia.com – MBS, www.mbsconsulting.com
The evaluation examines the regional Smart Specialisation Strategy (RIS3) supported under the Marche ERDF OP, 2014-2020, which focuses on four cross-sectoral areas of intervention: sustainable manufacturing, domotics (home automation), mechatronics, health and wellness.
To assess the progress in implementing the strategy and the results obtained so far.
Desk research on OP documentation and RIS3;analysis of monitoring data;interviews with the MA and implementing bodies;analysis of secondary data (ISTAT, Eurostat);survey of key stakeholders (44 responses).
RIS3 for 2014-2020 is allocated over EUR 500 million, of which EUR 180 million come from the ERDF .Of the ERDF, 80% has been programmed, 67% committed and 19% spent.By end-2018, over 1,000 projects had been financed, 31% of funding allocated to projects supporting sustainable manufacturing and 30% to ITC. Projects on domotics (12%), mechatronics (15%) and health and wellness (12%) have received a smaller share.
Data only go up to 31 December 2018. Most interventions are still being implemented so analysis of results is premature.
There is a need to:share with the different programmes financing RIS3 the information from the monitoring systems so to have an overall view of the state of implementation;allocate to those areas so far receiving less support some of the funding not yet programmed;concentrate funding in the next programming period on a smaller number of objectives to make it easier to monitor and evaluate.
Impact Measurement and Monitoring 2016 – the ESI Funds effort in Syddanmark, 2007-2013
Effektmåling og monitorering 2016 - strukturfondsindsatsen i Syddanmark
OP More and better jobs; OP Innovation and Knowledge
Danmarks Statistik (Statistics Denmark) et al
Between 2010 and 2012, 45 projects were co-financed by the EU Structural funds in various areas (design and creative industries, energy efficiency, offshore, health and welfare innovation, tourism) in Syddanmark (Southern Denmark region). The majority of beneficiaries were SMEs in construction and manufacturing.
The objective of the study is to assess the impact of the Structural and Investment Funds on the companies supported (on their employment and turnover) between 1-3 years after the start of the projects.
Data on employment and turnover from Statistics Denmark company registers were used.The impact was assessed by comparing the development of the participating companies with a control group of companies (with similar characteristics: location, size, sector, historical growth) that did not receive support.The focus was on the period 2012-2015.
Employment in the supported companies increased in all areas examined (except energy efficiency) in the period 2012-2015. In total, about 1,600 new full-time jobs are estimated to have been created.Turnover of companies in healthcare, social welfare innovation and Tourism increased.There is however no indication that the funded projects have had positive impact on productivity.
Only companies with less than 100 employees were covered in the evaluation in order to reduce the risk of large companies dominating the overall results.It is recognised that some other characteristics (such as corporate motivation or development needs) which were not taken into account in the composition of the control group may bias the results.
Evaluation of the criteria and project selection system in the Opolskie OP, 2014-2020
Ewaluacja bieżąca kryteriów i systemu wyboru projektów RPO WO 2014-2020
This is the final report of the evaluation of project selection system and criteria under Opolskie Voivodeship regional OP 2014-2020.
To assess the project selection system and the related project selection criteria.
CAWI/CATI survey (project evaluators; beneficiaries; potential beneficiaries).Focus group of project evaluators.Interviews with beneficiaries and potential beneficiaries.Panel of experts.
The overlapping of calls for proposals and related deadlines forced a number of potential beneficiaries to refrain from applying.The MA policy of launching calls for proposals in the second quarter of the year rather than in the third which coincides with the end of the financial year was appreciated by applicants.The duration of calls for proposal has been made more flexible according to the number of documents and attachments required in the application.Most of the beneficiaries surveyed consider the project selection system to be relatively objective and impartial.Only 32% of applicants consider the availability of information on the MA website to be sufficient.
There is a need to reduce the amount documents required by calls for proposal.Calls for proposal should be scheduled in a more targeted way.Additional staff should be employed when workloads are heavy.The number of expert project evaluators should be increased.
Impact evaluation of support for people in the most difficult situation on the labour market, under the Podlaskie OP, 2014-2020
Ocena wpływu wsparcia kierowanego do osób w najtrudniejszej sytuacji na rynku pracy w województwie podlaskim na ich sytuację po zakończeniu udziału w projekcie
The evaluation examines the effectiveness of support and its usefulness.
To assess the effectiveness and impact of the measures and external factors on final results, as well as the degree to which the measures meet the needs of recipients.
Desk research.Monitoring and statistical data.CAWI/CATI survey of 228 beneficiaries and 1,907 participants.5 interviews26 Case studies.Delphi panel.
The majority of targets were already achieved or exceeded in the first quarter of 2021.Support under the OP is in line with the needs of project participants.The labour market situation of participants improved particularly because of the acquisition of practical skills and formal qualifications.
There were minor problems with the implementation of field research in relation to survey participation.
Take account of the needs of people and families in crisis; those dependent on others, including those suffering from mental disorders, and people leaving a protected environment.Provide more support in the form of: psychological and psychiatric services and health and social care.
Ex-post evaluation of measures on education under the Puglia ESF-ERDF OP, 2007-2013. Parts 1 and 2
Valutazione ex-post relativa alla programmazione regionale 2007-2013 in materia di istruzione - Programma operativo Puglia FESR-FSE 2007-2013
OP Puglia ESF 2007-2013; ROP Puglia ERDF 2007-2013
IRS, PTSCLAS Spa, Università degli Studi Aldo Moro
IRS - www.irsonline.it; PTSCLAS; Università degli studi Aldo Moro - https://www.uniba.it/it
This ex-post evaluation examines the "Rights in School" (Diritti a scuola -DAS) project under the OP, focussing on early school leaving and the acquisition of key competences by students. Data go up to 2014.
To assess the level of implementation, the quality of management and the impact, effectiveness and efficiency of the "Rights in School" (Diritti a scuola -DAS) projects carried out, on the basis of data up to end-2014.
Desk analysis of documentation.Interviews and focus groups with stakeholders.Analysis of monitoring data and official statistical data (INVALSI microdata from Ministry of Education, Eurostat, Istat).CIE (PSM).Case studies of good practice.
A Control Room set up by the Regional School Office to monitor and support schools in the implementation of the projects improved the management of the measures.Positive synergies were created between teachers, families of students with special needs in education, and psychologists and cultural mediators.According to the counterfactual analysis, the average scores in Italian and Maths of schools participating in the DAS were significantly higher than similar non-participating schools, especially for lower grade classes.The share of students who received a score below 25% in Italian and Maths) was not statistically significantly smaller in schools supported than the control group.Teachers in precarious jobs do not ensure continuous education for fragile students.
Monitoring data for the DAS projects are not complete. The very long delay between the implementation of DAS and the ex-post evaluation made an in-depth investigation problematic. Analysis of impact was feasible only for those classes involved in the national test of competencies (INVALSI). Data were not available data to directly investigate student dropouts.
Ensure ongoing monitoring in each phase of the project.Make sure that specific professionals are available in schools to support the most vulnerable.Reduce the workload of beneficiaries by implementing two-year instead of yearly projects.Make the activities more stable over time and ensure continuity of internal teaching staff, external specialists and managers.Target the measures more towards primary and lower secondary schools.
Strategic evaluation of the relevance of the Employment OP, 2014-2020
HOPE GROUP, s.r.o. Palackého tř. 10, Brno, 612 00, info@euservis.cz
The evaluation is concerned with relevance in respect of OP Employment 2014–2020 in the Czech Republic.
To assess the relevance of the measures undertaken or planned under the OP in light of changed conditions as well as of the intervention logic.
Desk research of OP documentation;data from the monitoring system and Eurostat;interviews with 20 representatives of the MA and 27 other key stakeholders;survey of service providers, applicants and beneficiaries (671 addressed, 29% response rate).
Despite significant changes in the context, the overall intervention logic is still valid. However, some of the activities should be reoriented.TO8 - Significant changes in the Czech labour market have occurred since the OP began; the unemployment rate has fallen below 5% and the share of young people has declined considerably.TO9 - The number of those at risk of social exclusion has increased. There remains a need to increase the quality and accessibility of social services along with activities to develop new and innovative methods of social work.
Activities should be focused on the most vulnerable target groups (people of 55 and older, long-term unemployed, low-skilled, people with disabilities) and be tailored to their needs.Consideration should be given to launching new calls to support technical education and technologically advanced solutions.
Second Operational Evaluation of the OP Generalitat de Cataluña reprogramming
Evaluación Operativa por Reprogramación del Programa Operativo de Cataluña
OP Generalitat de Cataluña ERDF 2007-2013
Evaluation of the emergency measures funded by Cohesion Policy 2014-2020, in Hungary, in response to the economic and social impact of COVID-19
A magyar kohéziós politikának a COVID-járvány gazdasági és társadalmi hatásai kezelésére adott válaszainak értékelése – Értékelési jelentés
OP Human Resources Development 2014-2020; OP Economic Development and Innovation 2014-2020; OP Environmental and Energy Efficiency 2014-2020; OP Competitive Central-Hungary 2014-2020
Equinox Consulting Kft., HÉTFA Research Institute
The evaluation aims to assess how the Covid-19 pandemic has affected the implementation of four 2014-2020 OPs (Human Resource Development OP, Economic Development and Innovation OP, Environment and Energy Efficiency OP, Competitive Central-Hungary OP) and what instruments have been put in place to counter the effects of the pandemic.
To analyse the economic and social impact of the COVID-19 pandemic in Hungary and on the implementation of OPs, to describe the response of Cohesion Policy to the pandemic, to assess its effects, focussing on SMEs, and to suggest possible measures for the post-pandemic period.
Desk research on previous evaluations, programme documents, and scientific literature.Analysis of monitoring data and official statistics.Counterfactual analysis using PSM based on administrative data.Macroeconomic modelling.24 interviews with institutional stakeholders and beneficiaries.Online survey of SMEs (2,554 responses).
The COVID-19 pandemic had a severe shock in all but a few sectors; manufacturing, construction and tourism experienced significant declines, while distribution and ICT were less affected.The pandemic slowed down project implementation significantly, but projects were still implemented.COVID response measures were introduced in 4 Hungarian OPs, some responding to the health and social impact of the pandemic, others helping firms and workers.Schemes designed to reduce the negative labour market effects of the pandemic were particularly successful.Support for the smallest enterprises was provided through loans and other measures outside of Cohesion Policy.Emergency measures implied a significant increase in the administrative burden and workload in public authorities.Mixed experience was reported with respect to digitalisation of audit procedures.Only 8-12% of firms would have gone bankrupt without emergency support.Enterprises supported performed worse than non-supported ones, but survival rates are higher.It is estimated that support helped the Hungarian economy sustain stronger GDP growth, but the labour market impact was negligible.
Impact and effectiveness have been difficult to quantify because several measures were mostly aimed at covering additional costs rather than at achieving specific targets.
Provide support to the smallest SMEs, which were not reached by emergency measures.Reinforce the territorial focus of Cohesion Policy measures, as the pandemic has deepened territorial imbalances.Strengthen digital literacy and improve the national IT infrastructure.
Evaluation of the National Youthreach Programme, 2014-2020
Evaluation of the National Youthreach Programme
Smyth E., Banks J., O'Sullivan J., McCoy S., Redmond P. and McGuines S.
Economic and Social Research Institute, www.esri.ie
The evaluation examines the Youthreach programme, co-financed by the ESF and established in 1989 to tackle early school leaving by providing second-chance education for young people who leave school before Leaving Certificate level.
To assess the Youthreach programme, including referral pathways, learner profiles, governance and reporting structures, funding and resources, the curriculum, and learner experiences and outcomes.
Desk-based research and secondary analysis of administrative and survey data;national survey of senior managers at Education and Training Board level (response rate: 86%);national survey of Youthreach coordinators and Community Training Centre managers (response rate: 63%);case-studies in 10 Youthreach and Community Training Centres;in-depth interviews with Youthreach staff, coordinators and members of the board, learners and past participants;consultative workshops with representatives from government departments and agencies in education, training, employment and social inclusion.
Overall, the programme works well providing participants with a positive experience of teaching and learning and fostering personal and social skill development.The total number of beneficiaries on the programme fell by 7% between 2015 and 2017 (from 10,675 to 9,932).The average cost per participants amount to around EUR 10,000 (costs are somewhat higher in Community Training Centres than in Youthreach centres). The average cost for Youthreach provision is only exceeded by that for specialist training programmes and is higher than for the Vocational Training Opportunities Scheme.In 2017, administrative data show non-completion rates of 14%.Of those not completing the programme, 20% found an employment, 19% engaged in another course, 23% left for other valid reasons and 38% simply left the course.Of those successfully completing the programme, 45% went on to other education or training (including apprenticeships) and 43% found employment, while 12% were unemployed.
There is a need to:review the allocation of programme places at national and local level, taking account of the number of early school leavers and the availability of existing places;develop and install a tracking system, linked to the Post-primary pupil database, to follow young people as they leave school;introduce a soft skills measurement tool to assess learner outcomes.
Evaluation of the Wielkopolskie Voivodeship development strategy up to 2020
Ewaluacja Strategii rozwoju województwa wielkopolskiego do 2020 roku. Wielkopolska 2020. Raport końcowy
Wojciech Dziemianowicz, Jacek Szlachta, Anna Dąbrowska, Julita Łukomska, Krzysztof Piróg, Izabella Jurkiewicz, Weronika Marchewa, Martyna Olczak, Karolina Oleszkin, Jakub Tarnacki, Aleksandra Wawrzyszak
Geoprofit Wojciech DziemiaNowicz, ul. Wiązana 26 F; 04-680 Warszawa, geoprofit@geoprofit.eu
This evaluation of the Wielkopolskie Development Strategy is part of the regional authorities' preparation for updating the main strategy document in the face of on-going changes and future challenges, focusing in particular on the territorialisation of the measures involved and the implementation management system.
To assess the regional development strategy as a basis for planning for the 2021-2027 period, covering the scope of interventions, the effects and the management systems.
Desk analysis of 8 strategies, 30 programmes, 7 plans, 113 papers and documents.Survey of 16 voivodeship marshals.In-depth interviews with the Voivodeship Self-government staff (40), experts (13), representatives of the areas covered by the ITI strategies and territorial mandates (3).Case studies of other voivodeship self-governments (6).
The diffusive-absorptive development model adopted in the strategy meets the region’s situation and is flexible as regards external factors.The development policy should be based on its strengths – a thriving economy and the multifunctional metropolis of Poznań.The region has great potential for innovation and the development of renewable energy.In 2013-2017, PLN 5.7 billion was on the implementation of the strategy’s objectives, 45% allocated to communications (mainly transport) and 25% to culture and the cultural heritage, health and social policy.The region is successful at obtaining funds from national programmes and for research.62 out of 90 indicators achieved the desired direction of change.The regional authorities have implemented numerous non-financial initiatives that also contribute to the pursuit of objectives.Monitoring the strategy is still a challenge.The strategy’s implementation system is of high quality but should be further improved.
Limitations concern the feasibility of assessing the effects of the strategy.
Use the updating of the strategy to strengthen existing relationships to build new one.Implement a Territorial Forum and other forms of building intra-regional relations.Develop in-depth scenarios of the region’s response to various external stimuli in a transparent, inclusive and understandable way.Develop the assumptions for a “programme of institutional strengthening of the region".Maintain the current development model but consider changes in the vision.Develop the system of communication, implementation and monitoring of the strategy.Develop rules for effective monitoring of strategic projects.Increase the territorialisation coherence of the development policy and integrate planned activities financed from different European funds so as to intensify the impact of the policy.
Impact assessment of EU co-financed investment in transport in 2007-2013 and 2014-2020
Euroopa Liidu struktuurivahenditest teostatud transpordiinvesteeringute mõjude hindamine
OP Development of Economic Environment; OP Development of Living Environment; OP for Cohesion Policy Funding 2014-2020
Katrin Pihor, Kaupo Koppel, Heret Knjazeva, Kadri Kuusk, Kristjan Piirimäe, Valentina Batueva, Margus Nigol, Dago Antov, Harri Rõuk, Laura Altin, Erki Saluveer, Kaisa Vent, Siim Esko
Katrin Pihor, Poliitikauuringute Keskus Praxis, Tornimäe 5, III korrus, 10145 Tallinn, Tel: +372 640 8000, www.praxis.ee
The evaluation provides an assessment of the effectiveness, impact and sustainability of transport investment in Estonia during the 2007-2013 period and of the estimated impact and sustainability of 83 transport projects financed in the 2014-2020 programming period.
The main objective of the evaluation is to assess transport projects implemented during the period 2007-2013 and the estimated impact and sustainability of 2014-2020 projects.
Map layers analysis;desk analysis of project applications and final reports;expert interviews and telephone interviews with entrepreneurs;descriptive statistics from public databases (Statistics Estonia, Tallinn Airport, AS Saarte Liinid, Estonian Road Administration road registry);business data from commercial registers (turnover, profit, number of employees);police and Border Guard data on registered traffic accidents;traffic counts using video counters and border patrol mobility sensors;census data from the Estonian National Road Registry and Positium’s data on passive mobile phone positioning.
Following the crisis, road construction prices dropped and many projects were completed with lower costs than originally planned.Considerable improvement in the quality of rail passenger traffic was achieved, including improved access and shorter travelling times, and the number of rail passengers increased.The current port and airport infrastructure satisfy the needs of local residents and passengers.The recommendations from the environmental impact assessment were largely taken into account in the implementation of projects.The development of transport infrastructure should not be considered a driver of local business activity, rather the opposite seems to be the case.
It was not possible to evaluate site specific environmental impact.
The evaluation includes the following recommendations:To continue the reconstruction of the rail network and develop alternative modes of transport;to ensure a stable and regular shuttle service to regional airports;to include bicycles in long term transport planning;transport infrastructure should meet the increasing demand in areas of economic development;future transport planning should take account of traffic forecasts on the main arterial roads.
Impact assessment of PA1 of the national ESF OP in France, 2014-2020
Evaluation d’impact de l’axe 1 du POn FSE « Emploi inclusion » sur la période 2014/2020
This evaluation focuses on PA1 of the OP, on access to labour market, self-employment and prevention of school drop-outs.
To verify the functioning of PA1 on access to the labour market, self-employment and prevention of school drop-outs and to assess the effects of the ESF support.
Desk reviewAnalysis of monitoring data40 interviews with stakeholdersSurvey of entrepreneurs (1,700 respondents – response rate of 5%)Interviews of early leavers8 case studies
470,000 participants have been supported, 76% of them receiving support to access employment.43% were employed at the end of participation and 59% after 6 months. The average cost per participant was EUR 500.58% of the unemployed/inactive participating in self-employment measures found employment upon exit and 53% after 6 months. 37% of those employed before reported an improvement in their labour market situation. The cost per participant averaged EUR 800.80% of participants in measures to prevent school drop-outs entered a vocational programme upon exit and half of them are still in one. The average cost per participant was EUR 1,500.Participants largely match the groups targeted.Mobility support was provided on an transversal basis.The ESF (with its financial constraints and performance framework) is not considered to promote innovation and experimental approaches.The administrative burden is considered a serious barrier by many beneficiaries.
It is recognised that the ESF has effects that are not reflected in the result indicators reported, especially for those who are far from the labour market.
Establish a funding distribution strategy based on regional specificities.Ensure that the staff assigned to ESF management is appropriate.Consider all targeted groups (not only young people).Make the OP objectives and eligibility criteria clearer to encourage a more strategic approach from stakeholders.Call on the leaders of associative networks to support their members in mobilising the ESF.Devote part of funding to innovative measures by lifting financial and administrative barriers.Analyse in detail the efficiency of measures.
Mid-term evaluation of the Aragón ESF OP, 2014-2020
Informe de Evaluación de los Objetivos/Resultados del PO FSE ARAGÓN 2014-2010, Informe a remitir en 2017
Gobierno de Aragón - Sección de Coordinación de Ayudas Públicas y Evaluación de los Programas Europeos.
This is the mid-term evaluation of the ESF Aragón OP 2014-2020.
To assess the OP’s implementation and to examine whether the intervention logic is still appropriate.
The evaluation adopts a mix of qualitative and quantitative methods:Desk research;analysis of relevant policy documents;analysis of monitoring system data.
The overall logic of interventions remains unchanged as the OP continues to be in line with regional needs.The OP’s implementation is delayed due to: the late approval of both it and the eligibility rules, difficulties in interpreting the application of the simplified cost method and budget constraints which limited co-financing.The OP shows a high level of effectiveness for all three IPs.
Data go up to 31 December 2016. Data on certified expenses were not available.
The excellent results achieved need to be carefully assessed. Implementing authorities need to verify on a case-by-case basis the values of the result indicators and to keep documentary evidence that makes subsequent verification possible.
Evaluation of the Smart specialisation strategy in the Bolzano ERDF OP 2014-2020
Rapporto di valutazione della Smart Specialisation Strategy
IRS, www.irsonline.it - PTSCLAS, www.ptsclas.com
The evaluation examines the contribution of the Bolzano ERDF OP to the Smart Specialisation Strategy.
To assess the contribution of PA1 (R&D) of the Bolzano OP to the specialisation areas, specific objectives, and indicators of the RIS3 and collaboration between those involved in innovation.
Monitoring data;survey of beneficiaries.
All the RIS3 areas have been supported by PA1 projects, especially energy efficiency and ICT.ERDF projects have contributed to 3 of the 4 RIS3 specific objectives, but have affected only a few indicators.Most beneficiaries report that implementing RIS3 has not brought benefits, thought they lack full information.Projects mainly have a “technology push” approach and collaboration between firms and research centres is good.
The evaluation only defines starting points for reflection on RIS3.
Improve the focus of the RIS3 areas and communication activities.Include RIS3 indicators in calls for proposals and improve the spotting of innovation opportunities in companies.Introduce new means of collaboration between firms and research centres.
Evaluation of social inclusion measures under the Human resources OP, 2014-2020, in Slovakia
Hodnotenie inkluzívnych opatrení v rámci ŠC 1.1.1 PO1 Vzdelávanie OP ĽZ
Ján Helbich, Lucie Bučinová, Vladimír Sodomka, Miroslav Košík
Octigon, https://octigon.sk/en - Projektové služby, http://www.pseu.sk/01/index.php - RADELA, http://www.radela.sk/
The evaluation assesses the effects of social inclusion measures funded under the OP.
To assess the effectiveness, implementation aspects, sustainability and impact of the measures.
Official statistics.Monitoring data.Interviews with project managers and 9 schools involved in the project.Survey of beneficiaries and final recipients.
Projects completed or reasonably advanced are considered to be successful and have helped increase the professional capacity of schools and strengthened the individual approach to children with special needs.Projects have made little contribution to increasing the number of children in pre-primary education and reducing the presence of Roma children in special classes/schools, but have helped reduce early school leaving.Project management is satisfactory, though more cooperation between the bodies involved is needed as well as a uniform management system for the methodology for inclusive education.Sustainability of projects is at risk because of a lack of planning of inclusive measures in schools and the related funding.Flexibility in the composition of the working teams is particularly appreciated by stakeholders, while a lack of activities for parents and for teaching and professional staff is considered a major shortcoming.
The representativeness of the survey samples is not always clear.
Ensure stable funding and higher salaries for professional staff involved in inclusive education.Increase training for professional staff and cooperation with parents.Strengthen inclusive measures in pre-primary education for marginalised communities.
Interim Evaluation of the Trento ESF OP, 2014-2020
Valutazione dell’esecuzione del programma operativo del Fondo Sociale Europeo a titolarita’ della provincia autonoma di Trento
This evaluation assesses the progress made in implementing the Trento 2014-2020 ESF OP.
To verify the validity of the OP strategy and the current state of the OP’s implementation and to assess its achievements in terms of Europe 2020 objectives.
Desk analysis;Interviews with the MA;Theory-based analysis;Analysis of financial data and indicators
The analysis of context which underlies the OP is still relevant;the OP is helping to increase the employment rate and reduce early school leaving;implementation of axis 4 on capacity building is limited since, it has been given low priority.
The theory-based analysis was limited to the impact of the OP on Europe 2020 objectives.
There is a need to:improve the implementation of some actions;further support the structures involved in the OP’s implementation;fine-tune the financial allocations of the Social Inclusion Axis.
Evaluation of support for energy efficiency and reduction of carbon emissions under the Podkarpackie OP, 2014-2020
Ocena wpływu wsparcia RPO WP 2014-2020 na efektywność energetyczną i emisyjność
Ul. Przejazd 4 lok. 77, 02-654 Warszawa, fundeko.pl
The evaluation examines the contribution of support from the OP to changes in energy consumption and emissions.
To assess the current and potential impact of the support in terms of development of renewable energy sources, the energy modernisation of buildings and air quality.
Desk research.Monitoring and statistical data.Interviews with 7 representatives of institutions, 2 experts on energy policy and 3 experts on assessing applications.Survey of beneficiaries (response rate 61%).6 Case studies.Evaluation workshop.
The most effective projects are those which simultaneously reduce greenhouse gas and dust emissions.Projects in 2021-2027 will be more complex technically, covering aspects such as energy storage and low-carbon hydrogen.EU support at the regional level should complement programmes implemented at the national level.Destabilisation of the local electricity distribution network is an unplanned negative effect of support for renewable energy.
Finance technologies that affect the development of renewable energy sources and improve energy efficiency and air quality.Cooperate closely with energy experts when defining the scope and rules of support.Ensure complementarity between EU support and national programmes.Limit the use of technologies that destabilise energy supply.Maximise consumption of energy produced from renewables.
Evaluation of Action 6 "Networking Activities between Universities and Companies" under the Bayern ESF OP, 2014-2020
Endbebricht. Evaluation der Förderaktion 6 „Netzwerktätigkeiten zwischen Hochschulen & Unternehmen“
ISG Institut für Sozialforschung und Gesellschaftspolitik GmbH, Weinsbergstraße 190, 50825, Köln, https://www.isg-institut.de
The evaluation examines Action 6 “Netzwerktätigkeiten zwischen Hochschulen und Unternehmen” under the OP
To assess the implementation and impact of the measures up to March 2022.
Desk and literature review.Analysis of monitoring and statistical data.Evidence from the interim evaluation carried out in 2018 (based on two interviews with stakeholders involved in planning and implementation carried out in 2015, 24 interviews of project managers of 12 projects between 2016 and 2018, and two case studies in 2018).Five waves of surveys of companies supported between 2017 and 2022 (32% response rate out of 979 companies contacted).Survey of university staff responsible for in 2022 (67% response rate out of 33).
Financial and physical implementation of projects supported have proceeded without any difficulties.By end March 2022, output indicator targets were nearly all achieved.69 knowledge transfer projects were launched by 16 universities and 31 networks were established.1,542 companies were supported, a third of them involved in online university training, the rest participating in events and network meetings with higher education institutions.93% of companies were SMEs, 29% producing economic and/or technical services and 56% with previous university experience.Administrative procedures are assessed as being quite complex, particularly for institutes without experience, and ESF requirements as being restrictive.
Continue support in the next period.Reduce requirements for receiving ESF support, particularly for smaller higher education institutes.Consider whether it is worth eliminating the different allocation of funds between projects involving online universities and those involving ‘offline’ ones.Consider eliminating the regional requirement for universities to participate.Stimulate companies to participate, e.g. by carrying out participatory examination or issuing certificates.Strengthen cooperation.
Partial evaluation of the Interreg V B programme alpine space and north-western Europe
Teilevaluation der Interreg V B Programme Alpenraum und Nordwesteuropa - Schlussbericht
Myriam Steinemann, Christoph Petry, Tohmas von Stokar
INFRAS, Binzstrasse 23, 8045 Zürich, Tel.: +41442059595, E-Mail: zuerich@infras.ch
Impact evaluation of PA1 focusing on skills development under the Investments in growth and jobs ESF OP, 2014-2020, in Sweden
Effektutvärdering av Europeiska socialfondens programområde 1 2014–2020
NOP Investments in growth and jobs ESF 2014-2020
The evaluation examines the impact of skills development measures under Priority Axis 1 (PA1) of the OP. The analysis focuses on the effects on the labour market for participants four years after the intervention and takes into account the horizontal principles. Out of 285 projects supported under the PA1, 173 were analysed, concerning participants aged 23-60 at the start of the intervention, who began training between 2016 and 2020.
To assess the impact of interventions to promote skills development, and to identify the mechanisms which contributed to or prevented change.
Reconstruction of the intervention logic.Counterfactual impact evaluation (propensity score matching, difference-in-difference, difference-in-difference-in-difference) based on official statistical data and monitoring data.Analysis of 10 projects through a documentation study.Two workshops with four ESF coordinators from different regions.
Project participants are more stable on the labour market than the rest of the working population at the entry of the projects: they have a higher average annual income; 52% of participants have been employed in the same workplace for over five years (compared to 45% of the working population), and 48% of them have a higher education level (compared to 36% of population). 68% of project participants are women.One year after the intervention, those who took part in skills development activities for at least 4 hours, earned around EUR 457 per year more than those who took part in the measures for 2-3 hours. Two years after the measures, the effect drops to approx. EUR 274 per year, while in the third and fourth years the effects are not statistically significant.Similarly, participants receiving skills training for 41 hours or more show larger estimated effects on their annual income, as compared to those who were in training for 25-40 hours or 4-24 hours.Skills development measures reduced the relative risk of being unemployed by about 30%.When differentiated by gender, skills development interventions show positive effects for women’s annual income, while there is no measurable effect for men. No difference between men and women with respect to staying employed was observed.Positive effects on income and remaining employed were also observed for all age groups and for those born in Sweden, while for those born abroad, the effects are uncertain.One of the main challenges of the projects has been to recruit participants, and to get them to take part in skills development to the extent desired.
The projects analysed include those targeting employed individual participants or those active in non-profit sector, whereas interventions related to collaboration or organisational development, as well as those targeting long-term unemployed were not analysed. Due to data limitations, long-term effects could not be assessed on: the participants' potentially strengthened position on the labour market leading to a more sustainable working life, productivity, and compliance with labour­ market needs. Furthermore, due to the lack of data in official registers, the impact for people with disabilities could not be assessed.
Increase project intensity (training hours) to ensure quality results.Include a wider range of indicators and targets that can better capture the results generated by the ESF support.Focus the objectives of the measures more clearly towards productivity growth at the individual level.Develop new metrics or indicators of cost-effectiveness, more focused on the benefits achieved rather than on costs per individual.Increase the attractiveness of the ESF by simplifying the administration of the projects and providing support to project implementers, especially new ones, as well as by increasing the quality of the needs analysis, to ensure that funding goes to those most in need.
Evaluation of the environmental and sustainability targets of the Rheinland-Pfalz ERDF Programme, 2021-2027
Evaluierung der umwelt- und nachhaltigkeitsbezogenen Ziele des EFRE-Programm 2021–2027 des Landes Rhineland-Palatinate
Katrin Böhme, Valentin Heimes, Tabea Knöll, Dr. Klaus Sauerborn
TAURUS ECO Consulting, www.taurus-eco.de
The evaluation examines how sustainable development and climate protection are integrated into the implementation of the Rheinland-Pfalz ERDF programme for the period 2021–2027.
To provide information on the extent to which funding contributes to the cross-cutting principle of sustainable development and identify areas for improvement.
Analysis of monitoring data.Document analysis.Expert discussions with intermediate bodies.
The programme is making a significant contribution to the implementation of the European Green Deal, particularly with regard to ensuring a sustainable energy supply.The promotion of energy-efficient public building renovations has been particularly successful due to high demand and quick project approval. The approved projects have exceeded the planned targets by a factor of five, achieving a net floor area of 26,895 m². In addition, the funding generated other effects, such as the reduction of costs for public budgets, the increased quality and comfort and awareness raising.The audit procedures for compliance with the cross-cutting objective of sustainable development and for climate impact assessment in ERDF funding are functional overall, but can be improved in terms of methodology.Due to the low level of implementation of sustainability measures so far, it is difficult to make any statements regarding the possible achievement of the 2029 objectives.
Accelerate the outflow of funds for measures that have so far received little or no authorisation, particularly for funding efficiency improvements in companies, intelligent energy systems and start-up infrastructure.Adjust the audit procedure methodology to enhance its effectiveness and reduce the administrative burden.
Evaluation of the impact of projects on the development and use of e-services under PA 2 of the Małopolskie OP 2014-2020
Ewaluacja wpływu projektów wspartych w ramach 2 osi Regionalnego Programu Województwa Małopolskiego 2014-2020 na rozwój i wykorzystanie e-usług w województwie małopolskim
LB&E Sp. z o.o. - EGO – Evaluation for Government Organizations s.c.
LB&E Sp. z o.o, Ul. Książenicka 98, 05-807 Owczarnia, www.lbie.eu; EGO – Evaluation for Government Organizations s.c., ul. Podmokła 53 B, 04-819 Warszawa, e-mail: evaluation@evaluation.pl
The study assesses the impact of the measures to increase the access of citizens and entrepreneurs to public information and e-services.
To assess the impact of projects to improve the quality and availability of public e-services and internal and external procedures in public authorities and to make recommendations for 2021-2027.
Desk research of programming documents.30 interviews with implementing institutions, experts, local government units and beneficiaries.CAWI/CATI survey of 148 local government units, 660 local citizens, 396 enterprises and 1079 final recipients.Focus group with public administration, experts and enterprises.Evaluation workshop.
Support has enabled 60% of regional inhabitants to access new e-services provided by municipalities.96% of beneficiaries report that projects helped to increase computerisation and modernisation of ICT systems, 78% that they increased internal integration of ICT systems and 67% that they increased the level of IT security.Only 25% of inhabitants surveyed indicate high interest in using e-services and only 5% very high interest.93% of inhabitants surveyed report an interest in e-health services.
Continue support for the development of e-services in public bodies, in particular local government units, in 2021-2027.Focus on supporting e-services at the highest levels of maturity.Promote projects implemented by partnerships as they are more cost-effective.
Evaluation Report on the Communication Strategy of the OP Peloponissos - 2nd deliverable
1η Αξιολόγηση της Στρατηγικής Επικοινωνίας του ΕΠ “Πελοπόννησος”
Address: 17 Arionos Str., 81100 Mytilene, GreeceTel.: +30 22511-00340E-mail: info@eupolis.com.gr Website: www.eupolis.com.gr
Fondazione Giacomo Brodolini, www.fondazionebrodolini.it - IRS www.irsonline.it
Evaluation of the information system of the Lazio ESF OP, 2014-2020
Valutazione del sistema informativo e delle fonti informative
The evaluation assesses the adequacy, efficacy and effectiveness of SiGeM, the new management and monitoring system adopted by the Lazio ESF OP, 2014-20.
To assess the capacity of SiGeM to collect and validate the data required for quantifying the OP output and result indicators, to highlight strengths and weaknesses of the monitoring system and to suggest changes to overcome the main critical issues arising from the analysis.
Desk analysis: checking the SiGeM operation and analysis of the available documentation.
The SiGeM is generally suitable for collecting all the data and information required to measure the OP indicators and is a step forward compared to the monitoring system used in the previous period.The main strengths are a single register for beneficiaries, recipients and participants, the automatic validation of information and quicker and more accurate calculation of correlated indicators, the complete digitisation of data and documents for each procedure or project; personalised access based on the role and responsibility of users to ensure the quality and security of information loaded into the system; a data warehouse that allows standardised reports to be extracted and "free" data for customised calculations.However, there are still issues about the quality of data provided directly by beneficiaries and delays in providing and updating.
There is a need to:supervise data entry by beneficiaries, defining standard procedures and specific guidelines in order to reduce errors and ensure the consistency and quality of data;ensure timely data entry and updating in order to monitor the actual state of the OP;systematically update the existing technical documents and manuals used by both beneficiaries and the regional authorities, with specific focus on procedures for measuring the OP indicators.
Impact evaluation of PA2 “Road Infrastructure (TEN-T)” of the Integrated Infrastructure OP, 2014-2020, in Slovakia
Hodnotenie vplyvu projektov financovaných v rámci prioritnej osi 2 Cestná infraštruktúra (TEN-T) OP Integrovaná infraštruktúra
Marek Carbol, Roman Čampula, Ivo Dostál, Jiří Dufek, Zdeněk Dytrt, Blanka Hablovičová, Eva Havlíčková, Zdeněk Hejkal, Vítězslav Křivánek, Jan Machanec, Petra Marková, Jitka Ondráčková, Leoš Pelikán
Centrum dopravního výzkumu - https://www.cdv.cz/
The evaluation examines the effects of two newly created sections of the Highway Network in Northern Slovakia (D3) funded by the Cohesion Fund, particularly the Žilina Strážov – Žilina Brodno and Svrčinovec – Skalité.
To assess the results of the new sections in terms of traffic diversion, travel time, fuel use, change in the use of motorised vehicles (passenger and freight), emissions, noise, as well as to assess the consequences on the quality of life of surrounding inhabitants.
Counterfactual analysis, which compares the situation observed in 2022-2023 with a scenario without project based on a model (Sources of data: Transport model, Traffic data from National Motorway Company, socioeconomic data from the Slovak Statistical Office).Analysis of monitoring data.6 interviews with local stakeholders.Survey of 110 citizens.
The Žilina project was effective in diverting 6,000 vehicles per day from the city centre through the newly built tunnel. It also had a positive impact in terms of reducing noise and emissions.The Svrčinovec - Skalité section took more than 20 years to build, and local communities are critical of the lack of connections to villages. However, it had a positive local impact, mainly by diverting 1,700 lorries per day from local roads and reducing fuel consumption and emissions.There is a discrepancy between expectations that noise would be reduced and complaints of citizens about increased noise levels.Both projects are said to have reduced travel times for citizens, encouraged increased car use, and had a positive impact on international transit between Slovakia and Poland. Their full economic benefits depend on the completion of the D3 motorway to fully connect the region with southern Slovakia and Poland.
Evaluation of the strategy for fighting fraud and corruption in the ESI Funds in 2014-2020
Hodnocení Strategie pro boj s podvody a korupcí v rámci čerpání fondů SSR v období 2014-2020.
ASTRA – Asociace pro transparentnost, z. s., Sokolovská 260/143 180 00 Praha 8, dufkova@astracr.cz
The evaluation assesses the strategy against fraud and corruption in implementing the ESI Funds in 2014-2020.
To examine the effectiveness of the strategy against fraud and corruption in implementing Cohesion policy.
Qualitative analysis of relevant documents on the main national and international requirements for combating fraud and corruption.
The Strategy contains measures preventing conflicts of interest and procedures to verify this.Applicants for funding are screened to check any previous involvement in frauds.A database needs to be created for sharing experience in identifying and dealing with a risk of fraud.The Strategy has set up an effective support system for anonymous reporting of suspected fraud.The Strategy does not include a tailored risk management system or specific recommendations to avoid risk.More emphasis should be put on the risk of fraud in the implementation of projects and public procurement procedures.Personnel responsible for selecting projects should be adequately trained to avoid frauds.
The report does not assess the financial implications of managing risk.
The Strategy should ensure increased transparency through timely, open and easily accessible information on all the procedures carried out by public bodies managing the Funds.A specific platform could usefully be established to share experience of identifying the risk of fraud and examples of good practice in combating it.Specific codes of conduct should be established for managing funds and for protecting whistle-blowers.
Evaluation of the Interreg Italy-Croatia OP, 2014-2020 - Year 2022
https://www.lattanziokibs.com/ - info@lattanziokibs.com; https://www.irisricerche.it
The evaluation examines the financial and procedural progress of the OP up to December 2021.
To assess the level of implementation of the OP up to end-2021, its governance and the effectiveness of partnership.
Analysis of documentation.Analysis of monitoring data.Online survey of lead and project partners (202 responses).Interviews with a sample of lead partners of projects.Focus group with project managers.
Most of the indicators had achieved more than 50% of their target by end-2021.The number of partners involved in projects are well above the minimum requirements of the calls. 56% of the organisations involved are Italian and 44% Croatian.Croatian NUT2 regions have the highest number of partner organisations in the projects and so the largest administrative burden.The added value of projects is the exchange of good practice for addressing common problems.
The use of NUTS as geographical units does not allow areas to be compared directly because of variations in population density.
Consider limiting the number of single participants in projects. Introduce regulations or incentives to increase the involvement of partners with technical, scientific or operational specialisation. Inform potential stakeholders about upcoming calls.
Impact evaluation of the Austria-Bavaria Interreg OP 2014-2020
Wirkungsevaluation des Interreg V-A Programms Österreich-Bayern 2014-2020
Daniel Zwicker-Schwarm, Dr. Roland Scherer, Dr. Kristina Zumbusch
Institut für Systemisches Management und Public Governance IMP-HSG, Universität St. Gallen, Defourstraße 40a, CH - 9000 St. Gallen, Tel.: +41712242525
The evaluation examines the Interreg Austria-Bavaria OP 2014-2020, which has three priorities: 1) improving cross-border capacity R&D and innovation; 2) protecting the environment and promoting resource efficiency and 3) promoting co-operation on legal and administrative issues.
To assess the impact of the OP, to recommend short-term modifications and to draw lessons for the next programming period.
Desk research of OP documentation (implementation report, project reports);analysis of data from the monitoring system;focus group.
PA 1: under SO1 ‘Development of cross-border RDI capacity in higher education’, all programmed funding (EUR 7.9 million) was allocated; under SO2 ‘Increase business related R&D&I’, (EUR 11.4 million) 91% of planned funding was allocated. PA 2: under SO3 ‘Sustainable tourism development’ all programmed funding (EUR 10.7 million) was allocated; under SO4 and 5 ‘Improving biodiversity and protecting the environment’ (EUR 6 million) 85% of planned funding was allocated. PA3: under SO6 and 7 ‘Strengthen cross-border structures supporting regional governance and cooperation for greater social and economic integration’ (EUR 15.2 million) 66% of planned funding was allocated.The OP financed 63 large projects and 120 small projects, only a few of which are completed (15 large projects and 57 small ones).Under PA 1, output indicators are not yet fully achieved but the approved projects suggest that they will be by the end of the period. The result indicators (personnel in R&D and established networks) are already achieved, but there is a limited link between projects and result indicators.Under PA 2, three out of five output indicators are not yet achieved, but tourism projects are well under way but those of SO5 (green infrastructure) are delayed. All the result indicators are already achieved.Under PA 3, implementation of projects is delayed. None of the output and results indicators have been achieved yet and for some (e.g. long-term collaborative partnerships) their final achievement is at risk.
There is a need to:ensure coherence between projects and to seek synergies between stakeholders;prioritise resource efficiency projects in the next programming period;define results indicators so that they are as close as possible to the actual change aimed by the intervention.
Evaluation of the implementation of horizontal principles under the Podkarpackie OP, 2014-2020
Ewaluacja realizacji zasad horyzontalnych przyjętych w ramach RPO WP 2014-2020
The study examines the implementation of horizontal principles and the effects under the OP.
To assess the implementation of horizontal principles under the OP, the effectiveness of measures in relation to the needs of recipients.
Desk analysis.CAWI/CATI survey of 752 beneficiaries of support.37 interviews with institutions responsible for implementation of the OP, the Monitoring Committee, the Evaluation Steering Group, the Project Assessment Committee and civil society.Benchmarking.12 case studies.Evaluation workshop.
Measures adopted ensure the effective implementation of horizontal principles.55% of all projects are neutral in relation to equal opportunities for women and men.The implementation of the partnership principle is assessed positively.The level of awareness of the need to apply horizontal principles is high among both beneficiaries and managing bodies.Beneficiaries do not understand the principles of sustainable development and equal opportunities for women and men.The administrative burden in respect of implementing horizontal principles is small.The principles are treated as an operational standard rather than as an additional requirement.Implementation of the principles produces measurable economic, social and environmental effects and enables beneficiaries to be seen as socially responsible.
Continue training support of beneficiaries and applicants.Prepare a catalogue of the types of project in which the impact on particular horizontal principles is indisputable and does not require further justification.Create a code of good practice showing examples of how to implement the principles in practice.Create an ad hoc task group if necessary rather than appointing the Evaluation Steering Group to oversee respect for the principles.Place a greater emphasis on the knowledge of experts assessing applications of horizontal principles.
Evaluation of measures promoting inclusion of foreign children under the Prague - Growth Pole OP, 2014-2020 – 2nd phase
Evaluace aktivit se zaměřením na žáky s odlišným mateřským jazykem v rámci specifického cíle 4.2 Zvýšení kvality vzdělávání prostřednictvím posílení inkluze v multikulturní společnosti OP PPR v rámci výzvy č. 54: 2. etapa
Instrategy Consulting, s.r.o.; SCHOLA EMPIRICA, z.s.
www.instrategy.cz, info@instrategy.cz www.scholaempirica.org/en/, info@scholaempirica.org
The evaluation examines support for social inclusion of children with a foreign mother tongue under the OP, through bilingual assistants, tutoring, Czech language courses and teacher traineeships abroad. Field activities were carried out in 2020 and from November 2021 to February 2022.
To assess the main changes experienced by children, teachers and parents up to February 2022 as a result of the implementation of the measures, including bilingual assistants, tutoring, Czech language courses and teacher traineeships abroad.
Focus groups with education staff of schools and parents.Survey of parents of children (12 questionnaires collected).Quantitative analysis of the TALIS (Teaching and Learning International Survey) data targeting school staff in 2020 and 2021 (9 schools participating – 254 questionnaires sent in 2020 and 2021).Theory based approach and Outcome harvesting approach.Case studies on 9 projects.Standardised inventory (Strengths and Difficulties Questionnaire) survey of 190 children from 9 schools.
According to those interviewed, the learning outcomes of children, especially in the Czech language, improved over time, as well as their social behaviour.The measures have increased the inclusion of children, including through raising awareness among teachers of the culture of the families concerned.The promotion of foreign language courses (mostly English courses) for Czech pupils helped communication among children and improved classroom environment.According to teachers, traineeships abroad did not help increase their ability to deal with children with a different mother tongue, but they were useful for exchanging experience with colleagues.Financial support to employ bilingual assistants, extra resources for tutoring and Czech language courses is much appreciated by teachers.
Because of the very different types of measures implemented, the analysis cannot provide a real overview of the effectiveness of the programme and the sample covered by the TALIS survey is quite small: 92 responses in 2020 and 112 in 2021. Given the absence of a control group to compare the results of the surveys, and insufficient monitoring over time, caution is needed in drawing any conclusions.
Plan systemic and coordinated support to teachers and children with a different mother tongue in the Ministry of Education, Regions, and individual schools.Design methodological guidelines for school staff on how to work with children with a different mother tongue.Provide individual support to children with a different mother tongue because it helps them to become socially integrated.
Evaluation of YEI interventions, 2014-2020
Αξιολόγηση των Παρεμβάσεων της Πρωτοβουλίας για την Απασχόληση των Νέων
The evaluation is concerned with assessing YEI interventions being implemented within the OP Human Resources Development, Education and Lifelong Learning (2014GR05M9OP001). It is the first of two evaluations MS need to undertake (Article 19(6) of the ESF Regulation).
The evaluation has two objectives:Objective 1: to assess the quality of employment (incl. self-employment) and traineeship offers upon leaving a YEI intervention.Objective 2: to assess the quality of the design and of the implementation of the YEI.
Objective 1 above: Participants surveys based on representative samples. There are 3 different questionnaires (employment, traineeship and self-employment).Objective 2 above: Interviews with executives from the MA, beneficiaries, PES, Vocational training centres (KEK) and businesses.
Quality of employment and traineeship offers:Employment: 83% of offers were from companies that participants did a traineeship with under YEI. Main reasons for rejecting an offer were salary related.66% of participants were satisfied.Self-employment: 58% of participants work in a family business with (7%) or without (51%) a salary.63% of the participants were still self-employed, or family workers, at the time of the survey.47% decided to become self-employed because they could not find a job; 21% wanted to continue the family business; 28% chose self-employment for reasons connected to their studies.Traineeships: 74% of offers were from companies that participants did a traineeship with under YEI.84% were satisfied with the working conditions, 83% with the working hours, 54% with the social insurance coverage and 53% with the amount of the subsidy.Design and implementation of YEI:Design: Strong points: transfer of focus from the design bodies and training providers to the participants; the voucher system accelerates implementation procedures compared to previous practices; the selection system is completely transparent.Weak points: lack of reliable information; lack of qualitative criteria for training providers and programmes.Implementation: Strong points: effort to connect the needs of participants to the needs of the labour market; broad spectrum of the theoretical training covers both the needs of the labour market and participants’ skills; publicity of interventions.Weak points: theoretical training cannot always be linked to practical training; skill mismatches of participants and the specifications of planned actions and traineeships; difficulties in creating homogeneous (in terms of participants) training courses; limited participation of enterprises in the selection of participants; deficiencies regarding counselling.
Need for an integrated, coherent and transparent system for understanding labour market needs, at central, regional and local level.Apply qualitative criteria for training providers/and programmes and reduce their number. Connect training programmes to specific training places and participants with common backgrounds. Two types of training are recommended: (1) for new entrants to the labour market and (2) for specialised training. Connect actions to certification of qualifications.Increase the involvement of businesses in the selection of participants.More specialised counselling.Create a control, feedback and evaluation system focusing on improving quality and not on 'indicators' (would require greater investment in human resources).The electronic monitoring and the interconnection between the MA, beneficiaries and providers need be generalised.Proposal to reverse the process: from registration of jobs to the choice of participants, and from there to deciding on the training.Improvements regarding practical issues.
Evaluation of the ‘Custodi del territorio’ project for active ageing under the Umbria ESF OP, 2014-2020
Rapporto di valutazione del Progetto “Custodi del Territorio” Promozione e valorizzazione dell’invecchiamento attivo
Marta Scettri, Maria Caterina Capuano, Meri Ripalvella
Agenzia Umbria Ricerche (AUR), https://www.agenziaumbriaricerche.it/
The evaluation examines the ‘Custodi del territorio’ project for active ageing of those of 65 and over through local crafts, training of volunteers for supervision and guidance of museums, libraries and cultural heritage, and meetings and seminars for the dissemination of experience.
To assess the activities carried out under the project and their results, to outline policies and interventions on active ageing at regional, national and international levels and identify good examples.
Documentation analysis.Analysis of statistical data.Interviews with project organisers and lecturers.
In 2009-2018, the population in the project area fell by 3.2%, while population of 65 and over increased by 7.7%.From 2011, visits to the project area increased by over 50%, to over 200,000 in 2018.Around 100 people have participated in artisan workshops, run by those of 65 and over who are experts in the crafts concerned.At the end of the project, some participatory workshops continued in Il Quadrifoglio social cooperative centres and organisations that participated in the project.Three training courses (15 hours each) have been organised, with a total of 53 participants.Activities under the project were also open for those below 65, which allowed 20 young people to participate in various initiatives.
Set up a place in Perugia, where older people can meet and share their skills and knowledge with those interested.Establish PhD scholarships to incentivise research on ageing.Provide courses on the education of ageing for training agencies.
Interim evaluation of the implementation of the Efficient Public Administration OP, 2014-2020 in Slovakia
Pravidelné hodnotenie plnenia čiastkových cieľov na úrovni prioritných osí OP EVS (PO1, PO2, PO3, PO4)
The evaluation focuses on the implementation and results of the OP. Data used go up to September 2022.
To assess the level of implementation and results of the measures for increasing the efficiency of the public administration up to September 2022 and to make recommendations for the future.
Analysis of monitoring and statistical data.Analysis of relevant documents and studies.Focus group with stakeholders.
Some output indicators have a low target, which was already achieved by 2019.Result indicators were not defined for some OP objectives.Several activities planned to achieve OP milestones were not actually carried out.The intervention logic of the measures seems only partly aligned with the achievement of OP objectives.Financial targets for 2023 are unlikely to be met, as only 54% of the budget had been allocated by December 2021.
Speed up the implementation of projects, calls and grant applications, and improve compliance with payment deadlines.Mitigate administrative capacity risks by creating working conditions which limit staff turnover.
Evaluation of the results of measures in the area of education, including digitisation of education, under the Knowledge Education Growth OP, 2014-2020, in Poland
Metaewaluacja rezultatów działań PO WER w obszarze edukacji oraz rola e-materiałów w cyfryzacji oświaty
Piotr Fuchs, Przemysław Kozak, Zofia Rutkowska, Elżbieta Stosio-Sielach
Piotr Fuchs Smart Research, smartresearch.pl;Pracownia Rozwoju Przemysław Kozak, rozwoju.edu.pl
The evaluation consists of two parts. The first is a meta-evaluation of the results of the OP, while the second part assesses the extent to which schools and teachers use the educational e-materials acquired under the OP and made available on the Integrated Educational Platform (ZPE).
To assess effectiveness, usefulness, and sustainability of the results of the educational measures under IPs 10i, 10iii and 10iv of the OP. To assess the extent to which electronic educational materials are used.
Desk research of program documentation and literature.Analysis of monitoring and statistical office data.CATI/CAWI survey of 450 psychological-educational counselling centres.CAWI/CATI survey of 565 schools for vocational training.Interviews and focus groups with Ministry of National Education and representatives of project implementers in the 22 areas.5 expert panels.CAWI survey of 4,253 school employees and 594 training participants (concerning e-materials).27 interviews and 1 expert panel (concerning e-materials).
The measures on the educational system led to better preparation of counselling centres to provide services to students, including students with special education needs.In relation to projects on lifelong learning, the expected effect regarding better cooperation at the national level on qualifications was not confirmed. However, the projects helped reform the system of career counselling in schools and develop new Qualification Vocational Courses (KKZs) and Vocational Skills Courses (KUZs).Innovations implemented in the field of vocational education contributed to its better adjustment to the expectations of employers and the needs of the labour market. Nevertheless, vocational education projects did not lead to expected effects in relation to the use of model solutions in Practical Vocational Training (PNZ) and cooperation between higher education institutions and vocational education schools.In the case of the use of e-materials, most indicators reached a value close to the target.
Integrate the needs of pupils with special needs into the legal framework.Ensure that the mandatory functional analysis takes user experience into account when working on projects for designing and developing usable products.Introduce additional bonus criteria for employers applying for EU funds to encourage greater involvement into vocational training.
Evaluation of revitalisation measures financed under the Lubelskie OP, 2014-2020
Ewaluacja wpływu działań rewitalizacyjnych na sytuację obszarów zdegradowanych, w tym na ożywienie społeczne i gospodarcze oraz na jakość życia mieszkańców
The evaluation examines how revitalisation projects under the OP have improved the quality of life in degraded areas.
To assess the impact of revitalisation measures on the improvement of conditions in degraded areas, including the quality of life for residents and socio-economic regeneration.
Desk research (programme documents, national and EU guidelines, data on co-financed projects, local revitalisation programmes).Analysis of monitoring and statistical data.5 interviews with MA and IB, 5 interviews with LAGs.300 interviews with project participants.Survey of 66 beneficiaries and 44 social welfare offices5 case studies.Panel of experts.
Revitalisation projects focused mostly on infrastructure investments.Projects show significant effects in relation to social and employment issues.Projects were implemented in 97 municipalities in Lubelskie, and most of funds were used in Lublin.The measures contributed to the social and economic regeneration of the areas, helped improve residents' quality of life and create attractive tourist areas.
Assessing the cost-effectiveness was problematic because projects were not comparable in terms of scale or scope. Projects from late 2021 onwards have been affected by galloping inflation, which also made it difficult to assess their effectiveness.
Allow activities within the projects to be financed by sources other than the ESF.
Mid-term review of the financial and material progress of PA 2 of the Infrastructure and Environment OP, 2014-2020, in Poland
Ewaluacja śródokresowa dotycząca postępu rzeczowego II osi priorytetowej Programu Operacyjnego Infrastruktura i Środowisko 2014-2020 oraz wkładu tej osi w realizację Strategii Europa 2020
Radomir Dyjak, Małgorzata Kachniarz, Izabela Krok-Baściuk, Agnieszka Puzio, Alicja Weremiuk
FUNDEKO Korbel, Krok-Baściuk Sp. J., ul. Baśniowa 3, lok. 311, 02-349 Warszawa, biuro@fundeko.pl, +48 22 658 03 70 - Fundacja IDEA Rozwoju, Wierzbica 57B, 05-140 post Serock, www.ideaorg.eu
This a mid-term review of the implementation of PA 2 of the Infrastructure and Environment OP, 2014-2020, covering water management, seashore protection, environmental protection, education, information and monitoring, wastewater and waste management and improvement of the urban environment.
To assess the financial and material progress in relation to objectives, the contribution to Europe 2020 and national development priorities and absorption potential.
Desk research of programme and strategy documents, monitoring data, official statistics and relevant reports13 individual in-depth interviews with the MA22 case studies and telephone interviews with beneficiariesCAWI survey of beneficiaries, unsuccessful applicants and potential onesAnalysis of indicator systemEfficiency analysisAnalysis of the possibilities of assessing the expected net impact in 20232 expert panels
The main allocation and performance reserve are likely to be fully used and there is the potential to absorb additional funds of PLN 2.8-5.7 billion.The intermediate objectives and the final objectives for the framework performance indicators are likely to be achieved.The strategic result indicators are likely to be achieved, except for the share of municipal waste collected selectively and the share of Natura 2000 sites covered by management plans.PA 2 projects contribute to the pursuit of the priorities set out in EU and national strategic documents, covering protection, preservation and improvement of natural capital, effective use of resources and the health and well-being of citizens.
Impact assessment is only pre-analysed and the possibilities of an ex-post evaluation examined.
Use the remaining PLN 60-75 million for Action 2.4 (urban environment).Direct performance reserve resources first to covering contracts that exceed the Action 2.1 allocation (waste management), sign contracts for additional projects included in the list identified, and increase EU co-financing in projects implemented under Actions 2.1 and 2.3 (nature protection). Use any remaining funds to finance additional projects under Action 2.2 (wastewater management).Use any remaining funds for Action 2.3 (nature protection) and, secondly, for Action 2.2 (wastewater management).
Ex-ante study of potential interventions to be financed under the Material Deprivation Programme, 2021-2027, in Lithuania
Paramos materialiniam nepritekliui mažinti 2021–2027 m. programavimo laikotarpio galimų finansuotinų veiklų išankstinis vertinimas
Addressing Material Deprivation LT ESF+ 2021-2027
The study assesses the most effective methods of supplying food and hygiene products to final beneficiaries and implementing complementary activities in the 2021-2027 programming period to reduce material deprivation, aiming to eliminate the stigmatisation of the target group.
To analyse the experience of municipal administrations in the reduction of material deprivation, to assess the feasibility of transferring FEAD funds for food and essential goods to electronic cards in the 2021-2027 period, to assess the complementary activities carried out by partner organisations in the 2014-2020 period and to identify the most effective practices.
Analysis of legal acts and literature.Analysis of statistical and administrative data.Online survey of partner organisations (76 responses).25 semi-structured interviews with representatives of the Ministry of Social Security and Labour, the European Social Fund Agency (ESFA), municipal administrations and non-governmental partner organisations, as well as representatives of retailers.Exchange of good practice.
62% of surveyed municipalities have experience with the social card system.According to respondents, the most efficient way of organising the electronic card system would be to centralise the administrative processes, but to allow municipalities to decide which retailers to sign contracts with.The electronic card system would allow to significantly reduce administrative costs.The main risks of the electronic card system include limited access to supermarkets for beneficiaries with mobility issues, price differences between retailers, and potential abuse of the card by beneficiaries (e.g. by buying tobacco and alcohol).For FEAD, only funds transferred to cards for the purchase of food and other goods actually used by final beneficiaries can be considered eligible costs; therefore, when assessing the possibilities for simplification, financing not linked to costs could be considered.In 2014-2020, partner organisations carried out various types of complementary activities aimed at social integration: individual consultations, excursions, camps, training, health programmes, and psychological assistance. However, these activities rarely had a measurable goal and monitoring indicators.Complementary activities are mostly focused on children, but they lack continuity.
Appoint ESFA as the project implementer responsible for the indirect provision of FEAD support.Include municipal administrations in FEAD projects as partner organisations, ensuring their involvement in the selection processes of retailers.Organise the selection process by means of an open tender and provide flexibility for selecting more than one retailer in each municipality.Provide the opportunity for beneficiaries to choose and change the shopping centre cards themselves, by filling out an application at the social assistance centres.For beneficiaries living in remote areas and with mobility impairment, encourage the possibility to order food and other goods online or by phone or provide other solutions.Continue the implementation of complementary activities at the municipal level, ensure their continuity and set measurable indicators.Target more motivated groups, such as children and young people, invest in long-term individual consultations, and maintain constant contact with beneficiaries by phone, email, etc.
Evaluation of systemic interventions by the Human Resource OP 2007-2013 and the Human Resource, Education and Lifelong Learning OP 2014-2020 (ESF+YEI)
Μελέτη για την αξιολόγηση των συστημικών παρεμβάσεων του ΕΠ "Ανάπτυξη Ανθρώπινου Δυναμικού 2007-2013" και του ΕΠ "Ανάπτυξη Ανθρώπινου Δυναμικού Εκπαίδευση και Δια βίου Μάθηση 2014-2020"
OP Development of Human Resources 2007-2013; OP Human Resources Development, Education and Lifelong Learning 2014-2020
This is the ex-post evaluation of the Human Resource National Programme 2007-2013 and of the Human Resource, Education and Life-Long Learning National Programme 2014-2020.
To assess the appropriateness and relevance of the 2007-2013 national strategy and the objectives of the programme and the efficiency, impact and added value of the 2007-2013 OP; to assess the adequacy of the intervention logic for the 2014-2020 period.
Desk analysis of policy documents.Statistical analysis of official data (LABOUR SURVEY RESOURCES Nssg; Eurostat; ELSTAT).Surveys of stakeholders.
On 2007-2013:The OP was not implemented as planned and this caused delays and limited the effectiveness of the interventions.Interventions were not always consistent with actual needs.Intervention 1 “Development of a system to define the needs of the labour market” laid the basis for the 2014-2020 period.Intervention 6 “Development of an institutional, regulatory, administrative and financial framework with a view to promoting the social economy” facilitated the establishment of social businesses, but the ambitious design was not fully implemented.Intervention 8 “Review and expansion of the role and powers of the National Observatory for disabled” was not fully implemented, as more coordination is needed between the Ministry of Labour and the Ministry of Social Protection.On 2014-2020The implementation of the OP needs to be strengthened.Intervention 1 “Development of a system to define the needs of the labour market” continues to lack effectiveness.Intervention 4 “National mechanism for coordination, monitoring and evaluation of social inclusion policies and social cohesion” proved to be relevant for pursuing Europe 2020 objectives and to have significant links with regional and national programmes.
Some result indicators are missing and in some cases, there is a lack of data to analyse the adequacy of the 2014-2020 programme with respect to the socio-economic context.
There are no clear policy implications set out in the report.
Evaluation of interventions on energy efficiency under the Veneto ERDF OP, 2014-2020
Valutazione degli effetti dell’efficientamento energetico presso le imprese e gli edifici pubblici
V. Buscemi , M. Bressan, D. Quatrini, P. Paris, F. Costa, F. Bracci
Lattanzio, www.lattanziokibs.com - Iris, www.irisricerche.it
The evaluation examines two measures for improving energy efficiency in enterprises and public buildings up to mid-2020 under TO4 of the OP.
To assess the results of the measures to improve energy efficiency in enterprises and public buildings and to identify examples of good practice as for future measures.
Desk Analysis (including results of previous survey).Analysis of monitoring data and official statistics.
According to the “Odyssee-MURE” Horizon 2020 project, the most effective measures to improve energy efficiency in Italy are mandatory energy audits, market incentives, efficient lighting systems and co-generation installations.Despite several strengths, Veneto has inefficient public transport and low energy efficiency of buildings.A large number of applicants is reported for measures on energy efficiency in public buildings, while only some of the funding for enterprises was taken up.
Strengthen partnerships with the private sector.Incentivise industrial research activities on energy efficiency.Promote training of energy managers and energy auditors.Involve businesses in the planning phase.Promote greater synergy with other sources of finance.
The effect of the Database Service Development under the Knowledge Education Development OP, 2014-2020
Wpływ Bazy Usług Rozwojowych na jakość i dostępność usług rozwojowych świadczonych na rzecz przedsiębiorców i pracowników z uwzględnieniem oddziaływania Podmiotowego Systemu Finansowania
Danae Sp. z o.o., Ul. Szarotki 10/16, 02-609 Warszawa, Realizacja Sp. z o.o., Al. KEN 93, 02-777 Warszawa
The evaluation assesses the Database Service Development (BUR), a free, online database implemented under the Knowledge Education Development OP in Poland in 2014-2020.
To assess the impact of BUR on the quality and availability of development services provided to employers and employees.
CATI/CAWI interviews;individual interviews;statistical analysis;theory of intervention;swot analysis.
BUR has improved the availability of development services by providing standardised information on them;the development services have improved the knowledge and skills used by employees in their daily work;BUR is only partly recognised as a commercial tool among potential users and is mostly used to obtain funding;a number of barriers limit the use of BUR by 50-year old employees and the low educated;over half of those that benefitted from BUR services have tertiary education.
There is a need tofocus information and promotional activities on SMEs;conduct a nationwide BUR promotional campaign;improve BUR functionality by allowing comments to be added on the services provided;conduct systematic data analysis on the services included in BUR and the people who use them.
Evaluation of support for job creation in social enterprises funded under regional OPs in Poland - 1st. measurement (2021)
Ewaluacja wsparcia EFS na rzecz przedsiębiorstw społecznych z pomiarem wskaźnika rezultatu długoterminowego Liczba miejsc pracy istniejących co najmniej 30 miesięcy, utworzonych w przedsiębiorstwach społecznych – pomiar I (2021 rok)
All ROPs 2014-2020 except ROP Śląskie Voivodeship 2014-2020
Openfield, Instytut Badawczy IPC, EDBAD Pracownia doradczo-badawcza Maciej Mroczek
Openfield, openfield.pl; Instytut Badawczy IPC, www.instytut-ipc.pl; EDBAD Pracownia doradczo-badawcza, www.edbad.edu.pl
The evaluation presents the results of the first measurement of the result indicator on jobs created in social enterprises between 01/07/2015 and 31/12/2018.
To estimate the effectiveness, durability and utility of the ESF support for social enterprises between July 2015 and December 2018 on the basis of the “Number of workplaces in existence for at least 30 months created in social enterprises”, and to make recommendations for improvement.
Desk research.Survey of 743 people.45 interviews.
The long-term result indicator is expected to achieve the target set for 2023.The support is highly effective because the measure is comprehensive and does not only cover funding in the form of grants.Support covers development services for newly emerging social enterprises and business consulting.Jobs created as a result of the measure are durable.
Define a uniform, standardised method for calculating the indicator.Continue the measures, which are highly effective and have a high level of sustainability, in 2021-2027.Continue comprehensive support for job creation which is not limited to financing.
Evaluation of PA 2 “Improving SME competitiveness” of the Regional OP, 2014-2020, in Romania
Evaluarea POR 2014-2020. Axa Prioritara 2: Îmbunătăţirea competitivităţii întreprinderilor mici şi mijlocii
SC ACZ Consulting srl, t33 srl; office@aczconsulting.ro, info@t33.it
The evaluation examines support for micro-enterprises and investments in incubation facilities funded through PA2 of the OP. It covers the implementation period between 2014 and 2020.
To examine the impact of support for micro-enterprises and investments in incubation facilities to improve the competitiveness of SMEs.
Analysis of statistical data from Eurostat and the National Institute of Statistics.Analysis of monitoring data.4 interviews with the MA, the Competition Council, the Ministry of Research, Innovation and Digitalisation, and the Ministry of FinanceSurvey of beneficiaries (349 valid responses, 12% response rate, 4.9% margin of error and considered representative)Counterfactual analysis.CBA.2 case studies: 1 of an incubator and 1 on the risk capital/equity instrument. Both case studies involved interviews with two beneficiaries.1 Focus group with five representatives of the Regional Development Agency , evaluation experts and panel experts.
Support for micro-enterprises and SMEs was successfully implemented and generated more private investments than expected.The financial support allowed companies to finance new products and services, increase production capacity and make investments. It helped improve financial processes, create or consolidate organisational culture, introduce concepts relating to corporate governance, etc.A spillover effect in other territories is observed for close to half of the beneficiaries as they have customers and suppliers outside of their territory. Spillover effects are also expected throughout the supply chain.Progress with business incubators was more limited, as public authorities do not have the same expertise as private companies.The counterfactual analysis indicates that both measures have positive and statistically significant effects on job creation and fixed assets.The support did not address companies’ resilience to COVID-19 crisis.
Some statistical and result indicators were not up to date. No representative of the business sector was willing to participate in the interviews. Statistical analyses are used for exploratory purposes, rather than to provide a final indication on the relationship between the amount of funding and other variables.
Make publicly owned incubators open to private ownership, as private companies have more experience in supporting start-upsInclude sectoral approaches, particularly for incubators, to promote investments in specific sectors.
Mid-term evaluation of the implementation of action plans on health
Ocena realizacji planów działań w sektorze zdrowia - I raport cząstkowy
OP Infrastructure and Environment; OP Digital Poland; OP Knowledge Education Growth; All ROPs 2014-2020
dr Marcin Pierzchała, dr Piotr Karniej, Jarema Piekutowski
IBC GROUP Central Europe Holding S. A., ul. Mokotowska 1, 00-640 Warszawa, Tel.: +48 22 250 18 39, fax.: +48 22 201 21 40
The evaluation reviews 36 Action Plans implemented between 2015 and 2016 on various measures relating to healthcare co-financed under Cohesion policy (Infrastructure and Environment OP, Digital Poland OP, Knowledge Education Growth OP and regional OPs).
To assess Action Plans as a coordination tool (relevance, effectiveness, consistency, usefulness) as well as their impact on healthcare.
The evaluation uses qualitative and quantitative methods:desk research (strategy papers, ministerial guidelines, OP documents);in-depth interviews (23 interviews with representatives of the Ministry of Health, Ministry of Development, National Health Fund and health departments in different regions); CAWI and CATI surveys (105 answers);focus group.The data used are primary data from interviews, surveys and a focus group; and secondary data from the OP documents and national regulations)
Action plans are important tools for coordinating public interventions in healthcare;action plans are in line with the objectives of the National Strategic Framework Policy Paper for health protection 2014-20);there are problems in the implementation of some of the Action plan provisions (e.g. interventions in areas where staff is not adequate or improper definition of target groups);the monitoring system (94 indicators) is satisfactory, transparent, objective and measurable.
Indicators are needed to facilitate the monitoring of the implementation of the Policy papers tools;the representativeness of the Steering Committee needs to be increased to improve Action plan effectiveness;data gathering by Managing Authorities needs to be encouraged along with data aggregation for the Steering Committee and Ministry.
Evaluation of the contribution of ESI Funds, 2014-2020, to the National Ocean Strategy 2013-2020, in Portugal
Avaliação do Contributo do Portugal 2020 para a Estratégia Nacional para o Mar 2013-2020
All ROPs 2014-2020; OP Atlantic Area 2014-2020; All OPs 2014-2020, except OP Technical Assistance 2014-2020;
The evaluation examines the contribution of the European Structural and Investment Funds (ESI Funds) to the National Ocean Strategy 2013-2020. It covers the implementation period between 2013 and 2020 and includes 12 OPs and 2 Cooperation Programmes.
To assess the effectiveness, efficiency and added value of the measures implemented through ESI Funds on the National Ocean Strategy.
Analysis of monitoring data.Analysis of statistical data.Literature review.Two workshops to discuss the ToC, the draft conclusions and recommendations.15 interviews with key stakeholders (Directorate General for Maritime Policy, managing authorities…).6 case studies covering the main projects in terms of funding.7 thematic focus groups (e.g.: competitiveness of the fisheries sector; ports, transports and logistics).One survey with business project holders (260 respondents, response rate: 32%).One survey with non-business project holders (227 respondents, response rate: 44%).
In addition to the constraints inherent to COVID-19, the difficulty of attracting human resources and the time needed to fulfill public procurement procedures affected the level of projects implemented.Supported projects contribute to strengthening scientific and technological capacity in the Ocean economy and reinforce human capital through the funding of doctoral grants.Projects contribute to stimulating new areas of intervention with high economic potential, such as marine biotechnology.ESI Funds have an important additionality effect as 80% of the eligible investment would not have been carried out without EU support.The additionality effect was higher for projects implemented by public administrations and investments on the internationalisation of small companies or focusing on the fisheries sector.
Difficulty of collecting monitoring data. Absence of targets for most of the output and result indicators. Difficulty of covering a sample of respondents in the two surveys representing the diversity of supports associated with 33 IPs covered by the evaluation. Limited time elapsed to assess the results and impacts of the projects.
Promote complementarity between the diverse sources of funding of the National Ocean Strategy.Deepen the evaluation process of the next National Ocean Strategy and maintain a mid-term evaluation.Simplify the procedures to certificate expenditure and payments.Strengthen support for initiatives stimulating R&D and innovation in the Ocean Economy.Design an action plan to foster knowledge of the deep sea.Ensure continuity of support for the sustainable exploitation of marine mineral resources and the promotion of offshore renewable energy.
Effects of support for young people under OP Knowledge, Education and Growth 2014-2020 in Poland – V report
imapp consulting sp. z o.o, hello@imapp.consulting, www.imapp.consulting; Fundacja Naukowa Instytut Badań Strukturalnych, ibs@ibs.org.pl, www.ibs.org.pl; IQS Sp z. o.o., kontakt@grupaiqs.pl, www.grupaiqs.pl
The evaluation is the last of 5 reports to examine the implementation and results of measures implemented under PA 1 of the Knowledge, Education and Growth OP to support young people.
To assess the situation of NEETs aged 15-29 6 months after participating in OP-supported measures.
statistical analysis of data on characteristics of participants and forms of support using social insurance data;survey of participants - representative group of 1,673 people.
6 months after the programme 66% of participants were in employment; improvement on the labour market conditions helped to achieve this.29% returned to education (target 44%) because programmes were primarily oriented towards employability rather than education.Fewer participants than expected were in self-employment (11%, target 23%) because of a cut in start-up grants.24% of participants were neither in employment nor in education.45% of participants with disabilities found a job, much less than average (66%).
Indicators are calculated on the basis of administrative data from Social Insurance records which exclude those not paying contribution (e.g. students working on civil contracts, those employed in the grey economy) but it should not affect the results significantly.
improve the effectiveness of support for people with disabilities;implement a comprehensive and flexible support package for women to get back to work after maternity and childcare leave;adjust programmes better to specific needs of individuals;undertake research to better understand long term employment effects (and other effects) of labour market activation of young people.
Summary of the effects of projects implementation in Priority XIII Higher education infrastructure in OP Infrastructure and Environment 2007-2013
Podsumowanie efektów wdrażania projektów w ramach Priorytetu XIII Infrastruktura szkolnictwa wyższego Programu Operacyjnego Infrastruktura i Środowisko 2007-2013 (PO IiŚ)
Cezary Przybył – kierownik badania, Sebastian Pałka, Paweł Huras, Magda Matysiak, Wojciech Pieniążek, Jarosław Chojecki
Agrotec sp. z o.o. ul. Dzika 19/23 lokal 55, Tel: + 48 22 403-80-26, e-mail: agrotec@agrotec.pl
Evaluation of measures for the energy sector financed under the Large Infrastructure OP, 2014-2020, in Romania
Evaluarea proiectelor PO Infrastructură Mare 2014-2020 în sectorul energie
The evaluation examines energy projects implemented under the OP. Monitoring data used go up to the end of June 2023.
To assess the relevance, efficiency, effectiveness and impact of the support in the energy sector and to draw lessons learned for 2021-2027. To assess the sustainability of support and the factors affecting the results of each type of measure.
Desk analysis of relevant documents.Analysis of monitoring data and official statistics.Interviews with stakeholders.Case studies.Theory of Change.
There have been significant delays in the implementation because of delays in project selection, resulting in increased costs and risk of obsolescence of technologies.Many projects under IPs 4a, 4c, 4d and 7e will have to be phased out and postponed to the next programming period.In general, progress in implementing the measures is very low (20% absorption rate) and there is a low rate of achievement of output and outcome indicators.A national strategy concerning the energy sector was initially missing. It was later incorporated into the National Integrated Energy and Climate Change Plan and the NRRP.
Low accuracy of monitoring data (i.e., data on indicators) and significant delays in the implementation of the measures. Issues with data collection because of the limited contribution from key stakeholders and restrictions caused by Covid-19.
Make a coherent use of evidence-based policy and strategic planning to inform decision-making in the energy sector.Improve the capacity of the MA (to ensure timely assessment of projects) and beneficiaries (for project preparation and implementation).Carefully analyse the decision to phase out projects.
Evaluation of the implementation of the POIA OP
Interim evaluation of the implementation of the Investment in growth and jobs ESF OP, 2014-2020
Effektutvärdering av Socialfondsprogrammet 2014-2020. Delrapport: programstruktur och implementering
This is the interim report on the second evaluation of the Investment in growth and jobs ESF OP 2014-2020, focusing on its structure and implementation (the final report will focus on results).
To assesses the coherence and the relevance of the programming of the OP and the quality of implementation.
Desk analysis of relevant OP documentation and studies.Interviews with national and regional representatives of 8 organisations involved in the OP’s monitoring committee.Site visits to the ESF Council regional offices.
The OP and other relevant European, national, and regional strategies pursue common goals.The OP has prioritised efforts to support people particularly far from the labour market, who are most in need. However, because of extensive administrative requirements, some of the small municipalities and civil society organisations seem discouraged from applying for support. Interventions under PA1 (TO8) have a relatively large proportion of highly educated personnel in the public sector, while employees in the private sector are involved less. This suggests that the business sector does not consider the OP as relevant to their needs.Under PA2 (TO9) young people under 25 and the most disadvantaged have not been reached as expected.The OP displays shortcomings as regards the mobilisation of potential applicants, the unclear division of roles between regional and national authorities, and differences in the interpretation and application of regulations.There are large differences in implementation between regions.In practice, the horizontal principles have been reduced to gender equality.
The findings of the evaluation must be considered preliminary and they could be revised in the final report. Data availability does not allow all the evaluation questions to be answered.
Evaluation of measures to promote social innovation under the Brandenburg ESF OP, 2014-2020
Endbericht Evaluation der Richtlinie Förderung sozialer Innovationen
Institut für Sozialökonomische Strukturanalysen Berlin (SÖSTRA)
The evaluation examines the results and achievements of support for social innovations on two investment priorities: 1 - adapting workers, businesses and entrepreneurs to change; 2 - active inclusion, promoting equal opportunities and actively improving employability.
To assess the feasibility and effectiveness of 20 model projects promoting social innovation.
Document analysis of project concepts and factual reports of the projects.Analysis of monitoring data.Survey of all 20 project promoters.Online workshops with project managers of the model projects.Online expert discussions with representatives of specialist units from the Ministry for Economic Affairs, Labour and Energy.
Different methodological approaches were taken by the model projects. Those focusing on building networks with different professionals were the most effective in promoting social innovation, as the different perspectives involved favoured tackling the challenges.COVID had an ambivalent impact on model projects: some could not fully achieve their goals, whereas others exceeded theirs by deepening digital teaching and learning methods.Project promoters had difficulties disseminating new solutions beyond the circle of participants.
Continue focus on the testing of social innovations and the development of new solutions and model projects in the future ESF programming.Require the active involvement of cooperation partners from the outset of projects.Define the use and sustainability of projects after the funding, as well as how new solutions will be disseminated, from the beginning of the projects.
Operational Evaluation of the Navarra OP reprogramming
Informe de reprogramación por revisión técnica del Programa Operativo FEDER 2014-2020 de Navarra.
DIRECTORATE-GENERAL FOR COMMUNITY FUNDS. State Secretariat for Budgets and Expenditures
The evaluation assesses the financial reprogramming of the Navarra OP for 2014-2020.
To examine the OP reprogramming and assess whether it is coherent with the original OP strategy.
A combination of qualitative and quantitative methods based on secondary data retrieved from the national and regional accounting systems (Fondos2020 and SAP) and official documents (e.g. OP documents).
The main purpose of reprogramming was to include an increase in the ERDF support of EUR 1.13 million.The reprogramming consisted of an increase in financing for TO3, IP3.3 ‘Supporting the creation and the extension of advanced capacities for product and service development’ and a small increase in the 2023 target value of one output indicator.The reprogramming did not change the OP strategy or the intervention logic.
The reprogramming does not entail any significant change in the OP strategy.
Impact evaluation of measures improving natural and cultural attractiveness financed under the Pomorskie OP, 2014-2020
Ocena wpływu interwencji w zakresie atrakcyjności przyrodniczej i kulturowej w ramach Regionalnego Programu Województwa Pomorskiego na lata 2014-2020
Marta Cichowicz-Major, Wojciech Onyśków, Adam Grajek, Michał Marciniak, Aleksandra Kamińska, Beata Dec, Agnieszka Śnieżek, Wojciech Pieniążek, Cezary Przybył, Sebastian Pałka, Zofia Bizacka
The evaluation examines the impact of the measures in the area of cultural and natural heritage implemented under the OP.
To assess the support of the OP in using the cultural and natural heritage and increasing the tourist attractiveness of the region.
Documentation analysis.Analysis of monitoring and statistical data.24 interviews with implementing institutions and key stakeholders.CAWI/CATI survey of beneficiaries (n=49).Expert panel.7 case studies.Workshop on recommendations.
The scale of the projects implemented in the field of culture is insufficient in relation to regional needs.Natural heritage projects have a visible environmental, social and economic impact, particularly in the tourism sector.As a result of the measures, it is expected that the number of visits to the supported cultural and natural heritage sites and tourist attractions will increase by 1.27 million people per year.
Increase the promotion of newly created cultural products.Emphasise the importance of conducting business in support facilities as part of the selection criteria.Give preference to the participation of entities that carry out activities in support of active tourism as part of strategic projects.
Study on the potential application of simplified cost options in Latvia in 2021-2027
Vienkāršoto izmaksu piemērošanas iespējas 2021.-2027. gada ES fondu plānošanas periodā
Cohesion policy funding LV CF/ERDF/ESF+/JTF 2021-2027
KPMG Baltics and Oxford Research Baltics
https://kpmg.com/lv/en/home.html, esturca@kpmg.com, kkovalevskis@kpmg.com; https://oxfordresearch.lv/
The study examines the possibilities for expanding the use of SCO in 2021-2027. Additional attention is paid to the use of SCO for the costs of training and staff remuneration, proposing two horizontal methods for these expenditure categories. Only direct eligible costs are covered.
To assess the feasibility of using SCO in 2021-2027, particularly for training programmes.
Documentation analysis.Analysis of monitoring data and data from the State Revenue Service and Central Statistical Bureau.Analysis of data for the planned SOs, measures, performance indicators and costs for 2021-2027.Survey and interviews with Ministries.Analysis of the use of SCO in other EU Member States.
The application of SCO is mandatory for all costs in measures where project costs do not exceed EUR 200,000.In measures, where the usefulness and practical feasibility of SCO is limited, their application is optional and may be limited to the use of horizontal SCO methods.It is not appropriate to apply SCO in measures where the requirements of State aid rules for commercial activities must be met, or where the measures are implemented through financial instruments.The experience of other EU countries shows that SCO is most often applied in social inclusion projects and the provision of social services and infrastructure.
At the time of the study, 2014-2020 measures were still ongoing and the 2021-2027 OP was being drafted, which limited the data available. In addition, the collection of 2014-2020 project-level data was difficult because often project implementers had different interpretations of both the wording and the values to be indicated.
Develop a horizontal SCO methodology for training activities, consisting of an hourly rate and the cost of a lesson for one student.Develop a horizontal SCO methodology for the remuneration of project management staff, consisting of an hourly rate and a flat rate.Prevent situations where the horizontal SCO rates for individual measures differ significantly from the actual project costs, assess the possibilities of applying horizontal SCO and, if necessary, measure-specific SCO rates.
Analysis of the Cohesion Policy programming process across all the Programmes, 2021-2027, in Poland
Przebieg i uwarunkowania procesu programowania perspektywy finansowej 2021-2027
The report aims to analyse the programming process of Cohesion Policy for the 2021-2027 period in Poland, including the legal, strategic, and organisational factors that influenced it. Data used go up to end-2024.
To identify the challenges faced during the preparation of the programming period in Poland and provide recommendations for future improvements. Additionally, to compare the country’s programming process with other EU Member States, highlighting good practices and areas for improvement for the post-2027 period.
Document analysis of national and EU strategic and legal documents, reports, and other official sources.Individual and group interviews: conducted with 64 key stakeholders, including representatives of Polish institutions, the European Commission, and other EU Member States.CAWI surveys: online surveys of several MAs (37 respondents).
The programming process was affected by delays in EU regulations and the need to develop parallel documents, which increased time pressure and made it difficult to maintain consistency between various programmes.The European Commission played a dominant role in the negotiation process, and while informal dialogue was appreciated, it also increased administrative costs and generated uncertainty.The lack of a shared digital platform for document drafting caused difficulties in managing the various versions, while cooperation between Polish institutions was affected by delays in political and financial decisions.Good practices emerged, such as cooperation between central government and regional governors, which facilitated the negotiation of regional programmes, and the inclusion of the civil society through consultations and public dialogues.
Challenges related to personnel changes, limited access to complete or updated information, particularly regarding public consultations.
Adopt a more flexible, less detailed approach for the Partnership Agreement to improve adaptability and reduce the administrative burden.Establish a shared digital platform for document management to enhance coordination and reduce mistakes.Strengthen collaboration between central and regional institutions and ensure more structured and transparent informal dialogues with the European Commission for future programming processes.
Evaluation of family harmonisation and professional life measures in Greece - Interim report
Παραδοτέο Β: «Ενδιάμεσο παραδοτέο της αξιολόγησης του σχεδιασμού των προγραμμάτων με έμφαση στις διαδικασίες υλοποίησης και αξιολόγησης των αποτελεσμάτων της δράσης ως προς τους στόχους των επιμέρους Επιχειρησιακών Προγραμμάτων» στο πλαίσιο του έργου με τ
The report is part of a wider evaluation. It sets out interim findings on implementation procedures from the "Evaluation of family harmonisation and professional life measures in the Human Resources Development, Education and Lifelong Learning OP and the regional OPs". The measures concerned provide care to children, teenagers and people with disabilities.
To assess the design, implementation and results of harmonisation measures.
Desk research;analysis of documentation;data analysis.
In 2016-2017, several innovations were introduced on the criteria and selection procedures and the administrative rules and regulations, which were effective, though some changes in the monitoring system impose a burden on the organisations carrying out the measures.Over time, the coverage of the target population was increased, though there is still a significant proportion not covered.There are problems in the procedures for allocating childcare place to children.Overall, the continuous changes in the system for managing the measures were destabilising.
There is a need to:add resources to cover a larger proportion of the target population;improve the procedure for placing children;review the cost accounting system for childcare and care services.
Evaluation of Technical Assistance under the Partnership Agreement in the Czech Republic in 2014-2020
Evaluace nastavení technické pomoci v rámci Dohody o partnerství
Ernst & Young, s.r.o., https://www.ey.com/en_gl, Na Florenci 2116/15, 110 00 Praha 1 – Nové Město
The evaluation examines the implementation of technical assistance under the TA OP (“large TA”) and under the other OPs (“small TA”) in the Czech Republic.
To assess the implementation of technical assistance in 2014-2020 to make recommendations for organising it in 2021-2027 and to investigate the possibility of adopting Simplified Cost options.
Analysis of documentation.Analysis of monitoring and administrative data.Focus groups.Interviews with technical assistance staff.
Implementation of TA has sometimes been delayed because of overly specific activities and expenditure on small projects, which are hard to check.The current administrative framework for implementing TA is regarded as burdensome and simplification is needed.Delays might also be caused by duplication of controls over project expenditure.Simplified Cost Options would be suitable for TA, though they might pose some problems (e.g. in the case of measures funded by several OPs).The one-off payments option could be applied to TA, provided that reimbursement is clearly linked to outputs.
Centralise definition of horizontal topics affecting the implementation of both types of TA.Simplify management of TA projects, to lower administrative burden.Simplify controls over expenditures and avoid duplications (ESIF and non-ESIF).Use a flat rate calculated on the basis of personal expenditure which would cover several types of expenditure.
Evaluation of communication activities of National Coordination Authority in 2020 - post-test
Vyhodnocení komunikačních aktivit MMR-NOK 2020-posttest
Ex-ante study of market failure and sub-optimal investment in Mazowsze for the 2021-2027 Programme
Ewaluacja ex-ante zawodności rynku i nieoptymalnego poziomu inwestycji na terenie województwa mazowieckiego w kontekście wdrażania Regionalnego Programu Operacyjnego Województwa Mazowieckiego na lata 2021-2027 (RPO WM 2021-2027)
Ul. Toruńska 18C, lokal D, 80-747 Gdańsk, www.eu-consult.pl
The study identifies investment needs and possibilities of using financial instruments (FIs) in Mazowieckie in 2021-2027.
To assess the possibilities of using FIs in the 2021-2027 programme and to estimate the financing gap in the main areas of regional development.
Desk research.Analysis of monitoring data and official statistics.30 interviews with implementing bodies.CATI/CAWI survey of beneficiaries, non-beneficiaries and citizens.Expert panel.Benchmarking.5 case studies.SWOT analysis.PEST analysis.Delphi method.Qualitative Comparative Analysis.Evaluation workshop.
The estimated financing gap for investment is PLN 154.7 billion (around EUR 34 mn), the largest gaps being in respect of entrepreneurship/innovation, transport and environmental protection.In these areas there are mainly small and micro firms operating in the market for a short time, often planning to initiate innovative activities (start-ups).Less than 10% of unsuccessful applicants implemented their projects using funding from other sources.
Carry out a broad information campaign showing the advantages of FIs.Provide repayable or mixed support for entrepreneurship, energy and regeneration projects.Focus more on loans instead of guarantees and securities in 2021-2027.
Evaluation of the impact of EU investments on macroeconomic indicators and economic sectors in Lithuania in 2014-2020 and 2021-2027
ES investicijų poveikio Lietuvos makroekonominiams rodikliams ir atskiriems ūkio sektoriams vertinimas
OP for EU Structural Funds Investments 2014-2020; EU Funds’ Investments LT CF/ERDF/ESF+/JTF 2021-2027
BGI Consulting, Aukštaičių g. 7, LT-11341 Vilnius, Lithuania, www.bgiconsulting.lt, info@bgiconsulting.lt
The evaluation estimates impact of EU investments in 2014-2020 and 2021-2027 on macroeconomic indicators, public policy areas and economic sectors. The analysis covers both investments already implemented and those planned in 2014-2020 and in 2021-2027.
To assess the impact of EU investment on the Lithuanian economy, economic sectors and public policies in 2014-2020 and of that planned in 2021-2027.
Documentation, previous studies, evaluations and literature.Monitoring and statistical data.7 interviews.Data from the Employment Service and the Board of the State Social Insurance Fund (SODRA) under the Ministry of Social Security and Labour.Theory-based evaluation.Counterfactual impact assessment.Macroeconomic modelling (HERLIT).
2014-2020:It is estimated that in 2014-2035, EU investment will generate EUR 15,96 billion of additional GDP.In 2014-2023, annual gross fixed capital formation is estimated to be on average 10.8% higher, annual household consumption 3.2% higher, and annual FDI 4% higher than without EU funding. In 2024-2035, after the end of expenditure, the three will be higher by an estimated 0.5%, 0.3% and 1.6%, respectively. The impact on the net trade balance is estimated to be negative during the implementation of investments, but positive afterwards.In 2014-2023, the EU investments are estimated to create 28,000 new jobs a year, to reduce unemployment by an average of 1.9 pp a year and to increase wages by 4.6%. In the long term, the positive effects will remain but will be smaller.The EU investments contribute to convergence between the central and western region and the capital region.Participation in 5 projects supporting employment led to the number of months worked in the first year being from 8 to 33 pp higher and wages from 3 to 16 pp higher than for the control group, depending on the project. The positive effects remain visible in the third year but smaller. The largest impact is estimated for those over 54 and the long-term unemployed, the smallest for the unskilled and those with disabilities.In 2014-2035, the additional GDP generated because of investments is estimated to exceed the cost by 2.2 times, and in education, science and sport, 3.1 times.On average, in 2014-2035, each EUR 1 of EU investments is estimated to generate EUR 1.72.2021-2027:In 2021-2042, EU investment is estimated to generate additional EUR 11.02 billion of GDP.In 2021-2030, EU investment is estimated to increase the growth rate of gross fixed capital formation by 6.4% a year, household consumption by 2.1%, and FDI by 2.0% and by 0.3%, 0.1% and 0.9%, respectively, in 2031-2042.In 2021-2030, on average, EU investment is estimated to create 16,270 new jobs a year, to reduce unemployment by 1.1 pp a year and to increase wages by 2.7%.In 2021-2045, the additional GDP generated is estimated to exceed the costs of investment by 2.17 times and in education, science and sports by 3.45 times.
When assessing the impact of EU investments in different policy areas in 2021-2027, the intervention logic was considered only in general terms, and models based on the theory of change were not developed in each area. Because investments in healthcare have a longer pay-off period, the macro model cannot fully reflect their long-term impact. Since most public interventions have other goals than economic growth, it is not possible to fully assess their benefits on the basis of economic rates of return alone.
Plan EU investments by distributing them more evenly across years and by gradually integrating them into the economy.Where appropriate, plan investments based on cost-benefit analysis.Carry out an additional analysis of active labour market measures to assess the efficiency of projects implemented by the Employment Service.Perform an analysis to identify active labour market measures which have the highest impact on employment and which activities are the most efficient.Consider the possibility of introducing measures which would slow down the fast growth of the construction sector.Seek alternative funding sources to maintain the high quality of public infrastructure.
Summary report on result indicators and sustainability of the Development of Eastern Poland OP projects
Raport podsumowujący stopień realizacji wskaźników rezultatu i trwałość projektów PO RPW
Department of Analyzes and Strategies, Section for Monitoring and Evaluation of Innovative Projects, ul. Państwa 10, Warszawa
The evaluation examines the interventions supported under the Development of Eastern Poland OP (OP DEP), 2007-2013, which covers five regions (Warminsko-Mazurskie, Podlaskie, Lubelskie, Podkarpackie and Świętokrzyskie) and is aimed at increasing growth and standards of living via investment in transport and innovation infrastructure and tourist potential.
To assess the effects and sustainability of projects completed under the OP DEP based on result indicators.
The summary is based on a CAWI survey of beneficiaries and horizontal result indicators; specific project indicators; qualitative information on jobs created and information on the target groups of individual projects.
Up to the end of 2016, beneficiaries reported the creation of about 3,000 jobs (88% of the target), 25% of them going to women and454 full-time research positions (183% of the target), 32% of them for women.At the end of November 2016, targets were achieved for 652 out of 835 result indicators. The share achieved is significantly higher for projects that have already completed their monitoring obligations.For projects that have not yet completed the monitoring obligation, failure to achieve the target is mainly a result of the project being ongoing (the target likely to be achieved in the coming years or the survey being conducted a month earlier than previous rounds).
The indicator values do not cover the whole calendar year and are not calculated on the basis of the outsourced commissioned research.
Evaluation of the impact of projects on the natural and cultural heritage under the Małopolskie OP, 2014-2020
Ewaluacja wpływu projektów realizowanych w ramach 6. osi priorytetowej Regionalnego Programu Operacyjnego Województwa Małopolskiego na lata 2014-2020 "Dziedzictwo regionalne" na podniesienie atrakcyjności i konkurencyjności regionu
The evaluation examines the impact of projects funded under the PA on the cultural heritage and attractiveness and competitiveness of the region of the Małopolskie OP.
To assess the impact of projects on the protection, development and promotion of the natural and cultural heritage and the development of cultural assets.
Desk research of strategic and programming documents, draft regulations for 2021-2027+, evaluation and expert reports and other studies.Analysis of monitoring data and public statistics.Interviews with different departments of the Marshall’s Office.Interview with beneficiaries.Counterfactual study using the mixed-mode CAWI and CATI method.Case studies.Expert panel.Workshop.
Projects respond to identified needs. Most output and result indicators will be exceeded.Problems with applications arise only for projects on the recreational and tourist development of water reservoirs.The programme mainly creates jobs that have little impact on the development of skills in the leisure industries.The programme mainly contributes to the development of one-day tourism in areas supported.Disproportionately large funds are directed to areas where tourism potential is already high, not to those that are in a bad situation.
Continue support for the renovation of monuments, developing cultural centres, especially in rural areas and protection of the intangible heritage.Favour projects of a complex nature and those which encourage the development of cooperation networks relating to the local environment.Discontinue support for the organisation of cultural events.Continue activities protecting natural resources and further support ecological education centres.
Evaluation of the implementation of the England ERDF OP, 2014-2020
National Evaluation of the English ERDF 2014-20 Programme Phase 1 Report: Process Review
Hatch Regeneris, Belmana, BMG Research, Middlesex University
The evaluation examines the implementation of the OP, up to 2019.
To assess the key factors contributing to the effectiveness of management, delivery systems and processes of the OP.
Analysis of intervention logic. Interviews with 21 representatives of the Managing Authority. Fifteen interviews with representatives from various Government Departments, GPB (Growth Programme Board) national sub-committees and technical assistance partnership representatives. Analysis of monitoring data. Survey of members of ESIF sub-committees, which gathered 98 full responses. Survey of beneficiaries and non-beneficiaries (223 responses overall, 24% response rate - 84% of respondents are beneficiaries).
The programme established a clear national framework, allowing local strategies in 38 LEP (Local Enterprise Partnerships) areas to adapt to specific needs. The establishment of regional Growth Delivery Teams (GDT) and of Central Policy Team has ensured strategic alignment with national goals, enhanced collaboration and supported the government's levelling up priorities. A well-structured team of key experts has been crucial for integrating the national OP with local ESI Funds strategies, despite challenges due to programme complexity. The resources are appropriately spent, as just 0.3% of expenditure failed to meet the required EU standards. Despite some issues (complexities in design and management system), the OP has successfully met key targets, including on expenditure, suggesting that future targets will likely be met as well.
The analysis heavily relies on stakeholder perceptions and objective data on the OP management are limited.
Design programmes with greater flexibility concerning resource allocations, moving away from rigid thematic funding envelopes. Allow sufficient time when preparing local strategic and investment plans at the start of a growth programme. Simplify administrative requirements. Increase access to thematic expertise at the local level to enhance programme and project development. Allow more local discretion and autonomy on adjustments to changed local strategies and priorities.
Evaluation of the Romania Bulgaria Interreg OP, 2014-2020 - Part 2 Communication strategy
Servicii de evaluare privind implementarea si impactul programului Interreg V-A Romaina-Bulgaria. Raport de evaluare privind comunicarea.
SC ACZ Consulting SRL & VVA Group SRL, Ștefan Velovan Str., no. 23A, Craiova, Dolj Conty, Romania
Avaliação da Implementação, Eficácia e Eficiência da Iniciativa Emprego Jovem (IEJ)
CESOP – Universidade Católica Portuguesa
Ana Cláudia Valente (Coordenação, CESOP-UCP)Clara Correia (Quaternaire Portugal)Jorge Cerol (CESOP-UCP)Maria de Lurdes Cunha (Quaternaire Portugal)Mariana Rodrigues (Quaternaire Portugal)Paulo Feliciano (Quaternaire Portugal)Sónia Trindade (Quaterna
This is the final version of the first evaluation report on the YEI in Portugal, implemented under the Social Inclusion and Employment OP and providing four types of support: for traineeships, recruitment, education or training and entrepreneurship. It covers the period September 2013-March 2016 and was published in January 2018. It is concerned with the efficiency and effectiveness of implementation.
To assess whether YEI responds to the needs of the NEETs, is implemented according to the objectives, the conditions in place and the first results in terms of effectiveness and efficiency and to help prepare for the future impact assessment.
Desk research of documentation;analysis of data from the monitoring system, the National Statistics Institute, social security records, PES and Eurostat;in-depth interviews with relevant stakeholders; focus groups and interviews with YEI participants;survey of former YEI participants.
The NEET rate is relatively high among those aged 25-29, the low educated (though the rate among the highly educated aged 25-29 doubled between 2009 and 2015), people from abroad and with disabilities and those in Azores and Madeira.The most disadvantaged are under-represented because of difficulty of reaching them.YEI implementation is based on an extensive network of partners.Most NEETs are unemployed rather than inactive.Targets for output should be easily reached or even exceeded. Targets for result indicators (except for self-employment and participation in continuing education, training programmes, apprenticeships and traineeships 6 months after participation ended) are already exceeded.The completion rate is 86%. Within 4 weeks of completing participation, 73% received an offer and 71% pursued studies/training or acquired qualifications or a job. 6 months after, 70% were in employment and 6% in continuing education or training, apprenticeships or traineeships.44% of participants obtained open-ended employment contracts, 96% of them full-time and for 60% professional training was guaranteed by the employer. More than half of participants who accepted a job offer after 4 weeks reported that it was very suitable.The preliminary average unit cost per participant is EUR 1,602.
Data are available in the monitoring system for only traineeships and recruitment support. The samples used for analysis of disadvantaged groups were limited in size. There is a lack of representative information on implementation costs and it is difficult to obtain data on standard costs at EU level.
There is a need to:increase the focus on less qualified young people, inactive, LTU, women and other disadvantaged groups, as well as those living in rural areas and in Azores, Madeira, Algarve and Alentejo;accelerate implementation of education, training and entrepreneurship measures;strengthen the system for mobilising unregistered NEETs;clarify the articulation between the YEI and the YG (in terms of target groups and objectives).
Annual evaluation report of the Veneto ESF OP, 2014-2020, 2017
Rapporto annuale di Valutazione per il servizio di Valutazione operativa e strategica del POR FSE 2014-2020
The evaluation assesses the implementation of the Veneto ESF OP 2014-2020.
To assess the relevance and coherence of the programme, progress towards achieving the targets set and the contribution to the Europe 2020 Strategy.
Documentation analysis;analysis of monitoring data (financial and physical data);interviews with representatives of the MA and main stakeholders.
The pace of absorption of resources accelerated in 2017 compared to previous years for all PAs, except PA4 (on institutional capacity). 41% of the total funding was committed and 32% paid up at the end of 2017.Up to end-2017, 2,947 projects had been implemented. Of these, more than half were under PA1 on Employability.There were 85,010 participants in measures, 51% of them in PA1 projects.57% of participants were men, around a third under 25, just over 40% with lower secondary education, 45% employed, 31% inactive and 24% unemployed and 12% migrants and ethnic minorities.The horizontal principles of equal opportunities and non-discrimination and sustainable development have generally been applied.The targets set have been exceeded for PA4 and are in the process of being reached for P1 and PA3.The programme is in line with local needs and coherent with the regional Unitary Strategy 2014-2020.The OP system of management, auditing and control is well organised and is performing well. The monitoring system is also adequate and effective.
Employment and Social Inclusion, OP 2014-2020 – audit of the performance monitoring system and proposal for a performance framework management method
Audit du sytème de suivi de la performance et proposition d'une méthode de pilotage du cadre de performance
The evaluation assesses the monitoring system and the performance framework management method of the ESF OP (Employment and inclusion in Metropolis) 2014-2020 in France.
To assess the performance of the ESF OP 2014-2020 and the achievement of the Performance framework objectives and to review the performance monitoring system.
The performance monitoring system has weaknesses as regards information between the parties involved in the ESF which potentially affect the achievement of final objectives.While some objectives appear to be realistic and achievable, the relevance of the indicators remains uncertain because their calculation method is not known or the way the target population is identified is questionable. Intermediate targets ​​appear in general to be realistic.The monitoring system “Ma Demarche FSE" is incomplete and still under construction.
The quality of the information and documentation available and accessible in the monitoring system needs to be improved and all participants need to be covered.Operations need to be selected which are in line with the objectives and the participation of beneficiaries should be promoted.
Evaluation of support for care facilities for pre-school children in the Czech Republic under the Employment OP, 2014-2020
Evaluační zpráva výzev 35 a 36 OPZ - Evaluace dětských skupin
The evaluation examines the support for the provision of private care facilities for pre-school children.
To assess the results of support for care facilities for pre-school children, efficiency and the interest of target groups in them.
Monitoring data.Official statistics.Survey of beneficiaries.
The establishment of private childcare facilities has led to mothers being able to work more and increase their salaries; the employment rate of mothers rose from 37% to 74% after the intervention.34% of recipients reported that without support they would have provided care, reducing their working time.Care facilities are mainly used by parents with higher education and higher income levels.Privately-owned facilities are more expensive than public ones, though they managed to reduce fees because of funding received, especially in Prague.Parents with children under three are generally most interested in using the facilities.
The evaluation is based on a pre/post comparison because of the impossibility of establishing a control group.
Monitor access to childcare facilities in villages, where mothers bear childcare responsibility in most cases.Examine in more detail the reasons for parents’ preference for state-owned kindergartens over private ones.Monitor the costs of childcare that seem too high, detailing the services provided.Explore the potential interest of employers in creating company childcare facilities.
Study on the possibility of using financial instruments under the Pomorze Programme, 2021-2027
Przeprowadzenie badań terenowych i analiza wyników na potrzeby opracowania Strategii Inwestycyjnej dla instrumentów finansowych w programie Fundusze Europejskie dla Pomorza 2021-2027
M. Gajewski, H. Kalinowski, R. Kubajek, A. Matejczuk-Rosa, M. Ośka, J. Szczucki, J. Witkowska
Policy & Action Group Uniconsult, pag-uniconsult.pl; Imapp Consulting sp. z o.o., imapp.consulting.eu
The study assesses the demand for repayable financing in the areas supported by the 2021-2027 OP.
To provide suggestions for the preparation of the investment strategy, which is part of the ex-ante assessment of financial instruments.
Documentation analysis.Analysis of monitoring data.14 interviews with potential financial intermediaries.9 Focus groups with key FI stakeholders.CATI survey of 1,024 SMEs.CATI survey of 102 housing associations.CAWI survey of 206 local government units.CAWI survey of 47 universities and training institutions.
Between 18% and 39% of SMEs are potentially interested in using a loan for financing their investments.The interest in a loan is positively correlated with the size of the company (36% of micro enterprises vs 57% of medium-sized enterprises).For 80% of SMEs, the acceptable interest rate is 4%.The demand for loans of local government units is about PLN 10 million (over EUR 2 million) and covers the needs of about 90% of the units surveyed.
Lower the own contribution required for financial intermediaries.Launch a loan for own contribution for financial intermediaries.Implement a pilot project for the first Regional Investment Fund in the field of thermal modernization of public utility buildings.
Process evaluation of the implementation of a coordinated approach to socially-excluded localities under the Employment OP, 2014-2020, in the Czech Republic
Evaluation Advisory CE s.r.o., Palackého tř. 10, 612 00 Brno, https://evaluace.eu/
This is the final report summarising the results of the Coordinated Approach to Socially-Excluded Localities (KPSVL) under the OP.
To assess the results of the Coordinated Approach to Socially-Excluded Localities (KPSVL), its strengths and weaknesses and the main factors affecting financial absorption.
Analysis of monitoring data.Surveys of local consultants of the Agency for Social Inclusion, managers and applicants for support.Interviews with stakeholders.9 case studies.
Out of 341 applications, 230 were successful. The lowest success rate in applications for support was in respect of social entrepreneurship.The KPSVL covers a wide range of themes, which makes it easier to tackle a large number of issues.The complexity of the management of projects is regarded as an obstacle to implementation. The capacity of potential applicants to fill qualified positions is limited.Cooperation between partners is sometimes difficult because of a lack of commitment.The limited commitment of municipalities in implementing projects hindered their success.Beneficiaries report that one of the main benefits of KPSVL is the availability of experts to support the implementation of measures.Sustainability of interventions is affected by inadequate follow-up.
Provide ad hoc support, including training, to applicants to avoid errors in applications.Improve cooperation between partners in implementing projects to increase financial absorption.
Evaluation of the progress and impact of the Lisboa ERDF and ESF OP, 2014-2020
Avaliação Intercalar do Programa Operacional Regional de Lisboa
Sandra Primitivo, Catarina Pereira, Diogo Martins, Heitor Gomes, Inês Andrade, Luis Carvalho, Larissa Dowdney, Manuel Reis, Nuno Morim, Raquel Morgado, Rui Faustino, Pedro Casimiro
Ernst&Young Parthenon, https://www.ey.com/en_gl/strategy/about-ey-parthenon
The evaluation investigates the progress of the OP and its contribution to the expected results, up to end-2020.
To assess the relevance, coherence, effectiveness and efficiency of the OP up to end-2020, to identify its potential or actual impact and EU added-value.
Analysis of monitoring data and official statistics. Interviews with MA, IBs, banks. ToC testing. 2 surveys (of business promoters and non-business entities). 3 case studies. 5 focus groups. Contribution analysis. Literature review.
115% of the budget was committed at the end of 2020, but only 64% of the budget spent. PA3 (TO4) risks not achieving its objectives, but this has little effect on the overall performance of the OP. Selection criteria did not allow the best projects to be chosen, especially for TO3 and TO4, which have low approval rates and relatively high abandonment rates. The multilevel governance and partnerships with local entities has helped to improve leadership and social capital. The administrative burden, the limited predictability of the tenders, the weaknesses of the information system and limited resources hindered the achievement of results. Because of the limited financial resources of the OP, the impact does not seem to be significant, especially on R&D, SME competitiveness and healthcare, but the OP seems to have helped reduce unemployment. The added-value of ESI Funds is evident in terms of expanding and accelerating investment, especially in areas of market failure and after the pandemic.
Assessment of impact is preliminary because of the limited share of projects completed.
Strengthen lifelong learning and align it with regional needs. Increase synergies between ERDF funding for RTDI and other EU and national funding for it. Increase the use of financial instruments especially for business support and entrepreneurship. Strengthen the attractiveness of energy efficiency incentives and equal opportunity measures. Increase participatory policy planning and improve methods of calculating monitoring indicators.
Evaluation of training for enterprises under the Puglia ESF OP, 2014-2020
Regional Evaluation and Verification Unit for Public Investments, nvvip.puglia@regione.puglia.it
This is an on-going evaluation of support for training for enterprises under PA10 of the OP.
To describe the main characteristics of measures implemented to support training in enterprises and to identify the factors which affect their effectiveness.
Analysis of data on project applications.4 surveys of beneficiaries and those who waived support.
As opposed to large enterprises, SMEs face obstacles in participating in projects because of their lesser capacity to allocate resources to planning activities.Multi-company interventions have given the opportunity for businesses with lesser capacity to establish partnerships for training activities.According to the majority of respondents, training has improved work force skills.60% of single-company and 73% of multi-company respondents report that training helped them to retain their work force.77% of single-company and 92% of multi-company respondents reported that the organisation and management of the company had improved.Respondents find administrative procedures complicated and burdensome.Funding was waived because of a lack of suitable people in the company, a lack of time and difficulties in interacting with the training provider.
Perform a counterfactual analysis of training interventions.Obtain more information on project applications and strategic-operational priorities.Conduct case studies of applicants.
Evaluation of environmental measures implemented under the Large Infrastructure OP, 2014-2020, in Romania
Evaluarea interventiilor PO Infrastructura Mare 2014-2020 in domeniul mediului (2023)
The evaluation investigates the environmental measures financed under the OP. The evaluation was carried out towards the end of the implementation period and monitoring data are updated as of June 2023.
To assess the effectiveness, impact, including unintended effects and spill-over effects, and the sustainability of the measures relating to environmental infrastructure, environmental protection and risk management.
- Theory based approach.- Analysis of monitoring data and official statistical.- Counterfactual impact evaluation (only for waste management).- Survey of beneficiaries.- Interviews with key actors.- Focus group.- Case studies.- CBA.- Benchmarking.
- Waste management: despite the allocation of significant funds, only progress in waste collection has been made, but not in recycling.- Water management: the population connected to drinking water and wastewater networks has increased but much less expected.- Low capacity of waste and water management service providers has caused delays in the implementation of the projects.- Biodiversity: the measures have a high added value from a financial point of view, because of the lack of other funding.- Risk management: the measures make a positive contribution to protecting the population from floods, but there is a lack of human resources, strategic vision, and interinstitutional cooperation.- Historically polluted sites: the objective relating to the reduction of contaminated sites has been fully achieved.- For most measures the effects are only at the local level.
The timing of the evaluation is not fully appropriate because the measures take time to produce effects. Large-scale effects are difficult to identify and to distinguish from other factors affecting the results. Beneficiaries were not always available to provide information for evaluation purposes.
- Disseminate lessons learned and best practice to avoid future delays and bottlenecks.- Include in the CBA a sensitivity analysis to measure the impact of relevant factors on the financial and economic sustainability of the infrastructure built.- Strengthen partnerships at local level in relation to the protection of natural areas and biodiversity.- Ensure capacity building, training and inter-institutional cooperation for the mitigation of climate change-related risks.
Evaluation of the communication strategy of the Schleswig-Holstein ERDF OP, 2014-2020
Evaluierung der Kommunikationsstrategie des OP EFRE Schleswig-Holstein 2014-2020
Dr. Stefan Meyer - Gesellschaft für Finanz- und Regionalanalysen, Münster IfS Institut für Stadtforschung und Strukturpolitik GmbH, Berlin Kovalis (GEFRA) and Prof. Dr. Annekatrin Niebuhr - Christian-Albrechts-Universität zu Kiel
GEFRA: Ludgeristr. 56. 48143 Münster, Tel: +49-(0)251-2089-1992, E-Mail: info@gefra-muenster.de. - Dr. Stefan Meyer: Am Wall 174. 28195 Bremen, Tel: +49-(0) 0421-33048383, E-Mail: meyer@kovalis.de
The evaluation assesses the communication strategy of the Schleswig-Holstein ERDF OP 2014-2020.
To assess the communication strategy and the main sources of information as regards citizens and companies.
The evaluation mainly uses quantitative methods:Desk research of OP monitoring data.Telephone survey of 800 private households and 200 companies;Data come from the survey and the OP monitoring system. Comparison is made with a previous survey carried out in 2010.
About 90 information events have been organised reaching around 2,750 people.Since the beginning of 2016, the ERDF portal received 200- 300 visits a month, in line with the 2007-13 period but less than expected to achieve the annual target (5,500 views).30% of people in the region are aware of the ERDF programme and its aims (close to the target) and almost 80% of people and 70% of companies are interested or very interested in EU institutions and issues, while around 35% of both consider the support provided by the programme as very important.People and companies in the region, however, show little knowledge of the actual content of funding programmes (similar to 2010 survey).People and companies rely on different sources of information. For the former, it is mainly the media, for the latter business promotional events and the internet.
The evaluation does not set out any policy recommendations.
Evaluation of the contribution of ESI Funds to regional development (Tasks 5 and 6) in Středočeský in 2014-2020
Výsledková evaluace přínosů Evropských fondů na regionální úrovni: Úkol 5 a 6 - Středočeský kraj
Ernst & Young, s.r.o., Na Florenci 2116/15, 110 00 Praha 1 – Nové Město
This interim report is part of the project “Evaluation of the Benefits of EU Funds at Regional Level” (Tasks 5 and 6) covering all the ESI Funds with a specific focus on Středočeský (Central Bohemia) in 2014-2019. The analysis extends to end-2021.
To assess the results and impact of ESI Funds and the EU added-value of the ITI and CLLD instruments, in Středočesky (Central Bohemia), mainly over the period 2014-2019.
Analysis of monitoring data.Analysis of statistical and administrative data.Interviews with 22 stakeholders.Survey of citizens and municipalities (58 respondents).2 focus groups with stakeholders.Counterfactual analysis (DID and PSM combined).Case study of the Mladá Boleslav administrative district.
The Structural Funds supported intermodal transport, road reconstruction, the development of highways and alternative mobility, especially cycling. However, the impact on traffic is small and differences between more and less developed municipalities are still significant.The counterfactual analysis shows only a weak relationship between higher intensity of funding and impact of road reconstruction.EU funding supported the Central Bohemian Innovation Centre and led to 158 new researchers being employed. The counterfactual analysis found that municipalities receiving a higher intensity of support have better results, in terms of staff increases.Support for social inclusion involved social housing being renovated or built and social services focusing on the inclusion of 3,000 children in education. New services were also introduced for an ageing population.The impact of EU funding on reducing CO2 emissions is still uncertain and is not significant on waste management in municipalities with high intensity of support.Impact of funding on employment is also not significant, the counterfactual analysis showing only a slight decrease in unemployment rates in the more supported regions, which is more attributable to general socioeconomic trends.Culture and tourism were not adequately supported.Smaller municipalities benefited from CLLD projects, which supported renewal of pavements and improvement in infrastructure, especially schools, in almost 100 municipalities.ITI projects focused on public transport and road connections to reduce the isolation of rural areas most distant from Prague and Mladá Boleslav.
Social inclusion indicators have been quantified only since 2016. The contribution of ESI Funds to TO11 was not assessed, since no suitable socio-economic indicators were identified.
Evaluation of the YEI (”First-challenge 2015”), phase one
VredNotenje Pobude za zaposlovanje mladih (program »Prvi izziv 2015«) - znižanje brezposelNosti mladih, ki niso zaposleni in se ne izobražujejo ali usposabljajo v starosti od 15 do 29 let v kohezijski regiji vzhodna Slovenija: Prva faza
OP for the Implementation of the EU Cohesion Policy 2014-2020
Deloitte Slovenija, Dunajska cesta 165, 1000 Ljubljana, www2.deloitte.com/si
The evaluation is concerned with assessing the YEI element of the Operational Programme for the Implementation of EU Cohesion Policy in the Period 2014-2020 ESF ‘Employability, Inclusion and Learning’ 2014-20 (Programme ʿFirst Challengeʿ (Prvi izziv). It is the first of two evaluations MS need to undertake (Article 19(6) of the ESF Regulation).
The objective of the first phase of evaluation is an assessment of the relevance of the programme »Prvi izziv« (key issues and target group’s expectations), adequacy of the monitoring system (system, indicators), the possibility of carrying out a counterfactual evaluation and non-discrimination/equality mainstreaming.
The evaluation is based on qualitative methods (interviews, document analysis).
The first phase of the evaluation showed that the measure properly addresses key issues and target group expectations.The monitoring system established will provide possibilities for both quantitative and qualitative evaluation of the implementation of the programme in 2018.The evaluator proposes to establish parallel control groups in the second half of the year 2018 in order to carry out the counterfactual evaluation.Very limited “dead weight” effect of the measure is expected.The programme promotes non-discrimination since the invitation to participate does not discriminate within the target group.
Evaluation of the contribution of the Veneto ESF OP, 2014-2020, to the region’s economic development
Valutazione del contributo del Programma alla Strategia dell’Unione per il servizio di Valutazione operativa e strategica del POR FSE 2014-2020
The evaluation assesses the contribution of the Veneto ESF OP 2014-2020 to the region’s economic development, covering all PAs (on employability, social inclusion, education and training), except PA4 on institutional capacity.
To assess the extent to which the policies supported by the ESF had an impact on regional economic development and the labour market.
Macroeconomic model GREM-VE (GRETA Regional Econometric Model) to estimate the impact of interventions. Data are from ISTAT, the OP monitoring system and simulations of macroeconomic indicators from the IMF for the period 2018-2023.
Domestic demand is estimated to have increased by EUR 102 million between 2016 and 2017 because of ESF support.The income of beneficiaries was mostly supported by the measures on employability in 2015 and 2017 and by those on social inclusion in 2016.The impact on GDP is estimated at 0.05% in 2017 and 2018 and it tends to decrease from then on because of the easing of the effect on domestic demand. The main effect is on services and construction.The estimated impact on the employment rate is more on women than men, the employability measures being most effective for both. The positive effects are evident for all age groups but are less for those of 65 and over and 15-24.
Analysis of the market and forms of support for PA2 - ‘Business development and competitiveness of SMEs’ of the Competitiveness Programme, 2021-2027, in the Czech Republic
Analýza tržní situace a specifikace vhodných forem podpory pro účely přípravy priority 2 - Rozvoj podnikání a konkurenceschopnosti MSP operačního programu zaměřeného na konkurenceschopnost pro období 2021 - 2027
Competitiveness Programme CZ ERDF 2021-2027
ASOCIACE PRO EVROPSKÉ FONDY, Z.S., BUDĚJOVICKÁ 2056/96, 140 00 PRAHA 4 - KRČ, http://apef.cz/
The study identifies market failures to justify supporting the PA of the future Programme.
To assess the areas for support under PA2 of the 2021-2027 Programme.
Interviews with stakeholders.Survey of stakeholders and beneficiaries.Analysis of monitoring data and official statistics.Panel of experts.
The main market failures are: negative externalities from distorted competition in the market; asymmetrical information when purchasing goods; uneven transaction costs and insufficient coordination between market players.Specific market failures as regards PA2 are: long period before there are returns to investment; difficult access to credit for smaller companies and to industry 4.0 digitisation/automation for smaller companies; and predominance of bigger companies in foreign market fairs.The flexibility allowed by the EU for State aid might support smaller companies in consolidating their presence in the market.
Support the use of advanced technology by SMEs.Ensure support for smaller and riskier companies, including through financial instruments or State aid.Support the internationalisation of SMEs, especially smaller ones.
Ex-ante study of financial instruments under the Abruzzo ERDF programme, 2021-2027
La valutazione ex ante degli strumenti finanziari del PR FESR 2021-2027
The report examines the possibility of using Financial Instruments (loans and guarantees) targeting regional SMEs and innovative start-ups in 2021-2027. The financial instruments (FIs) included in the programme aim at supporting the local productive system and the transition towards a circular economy.
To provide suggestions to the MA on the management of FIs in 2021-2027, by analysing their characteristics, the leverage effect, and the lessons learnt from the 2014-2020 programming period.
Desk research of strategic and normative documents at regional, national and EU level, programme documents, and previous evaluations.Analysis of statistical and monitoring data.Interviews with relevant stakeholders (representatives of financial intermediaries and relevant business sectors).
The use of financial instruments during the 2014-2020 programming period has been particularly complex and required a lengthy implementation process.Access to credit is difficult for SMEs because of their size, the consequences of COVID-19 and rising management costs. The reduction of public guarantees may further reduce their capacity to access credit.The Guarantee Fund Abruzzo Crea, funded by the ERDF OP 2014-2020, had limited success, due to complex activation mechanisms and the limited interest in the measure by the banking system.For the 2021-2027 programme, loans and guarantees support investments of innovative enterprises and investments for the transition to a circular economy, and they seem suitable for responding to the regional needs.The estimated leverage effect of loans is 2.38, while the leverage effect of guarantees varies according to the fund’s management cost and the required gearing ratio.
The proposed suggestions are based on the economic framework at the time of the study, which may change during the implementation phase.
Combine grants and subsidised loans to strengthen the effectiveness of the measures for innovative enterprises and the circular economy.Consider conceding a further subsidy to reduce the interest rate on financial operations in general, to facilitate access to credit of SMEs.
Evaluation of measures for sustainable transport in Hungary, in 2014-2020
OP Economic Development and Innovation 2014-2020; OP Integrated Transport 2014-2020; OP Territorial and settlement development 2014-2020; OP Competitive Central-Hungary 2014-2020
FŐMTERV Zrt. – Collective Intelligence Kft. - Field Consulting Services Zrt.
www.fomterv.hu - https://www.fieldconsulting.hu/index.php/hu
The evaluation examines the achievements of the interventions on transport under several OPs in Hungary, in 2014-2020.
To assess the effects of interventions on transport, especially in terms of financial, environmental, social and institutional sustainability.
Monitoring data.Official statistics.Interviews with MA and beneficiaries.Surveys.Case studies.
The interventions on sustainable transport are progressing, though the achievement of some targets is at risk, partly because of measures put in place to fight COVID-19.The management structures are incomplete and there are several bottlenecks in the administrative system.For some projects, the most critical points have been omitted to ensure the achievement of targets.Sustainability aspects are not always taken into account by projects, especially in respect of future maintenance costs.The introduction of road tolls has not succeeded in diverting road traffic to other modes of transport. Interventions in Budapest have been the most effective, but in other parts of the country, especially in rural areas, measures have not resolved major issues.
An in-depth assessment of the administrative burdens and project lead times could not be carried out because of lack of data.
Strengthen the strategy-making and coordination of functions in the administrative system.Continuously monitor the administrative burden, costs and capacity. Introduce a project management flat-rate fee to make the accounting of wage costs easier.Take more account of sustainability aspects in project selection.Ensure a more even territorial distribution of funds and promote large-scale projects in rural regions.
Evaluation of PA 8 "Combating poverty through active inclusion" of the Niedersachsen ERDF and ESF OP, 2014-2020
Abschlussbericht zur Evaluation der Prioritätsachse 8 „Armutsbekämpfung durch aktive Eingliederung“
Sabine Wellmer, Hans Verbeek, Marco Puxi
ISG - Institut für Sozialforschung und Gesellschaftspolitik, www.isg-institut.de
The evaluation examines the effects of the active inclusion measures under the OP.
To assess the contribution to poverty reduction of three programmes for social inclusion and their sustainability: ’Qualification and work’ (QuA), youth workshops and pro-active centres (PACE), and labour market reintegration of prisoners and ex-prisoners.
Monitoring data.Official statistics.3 case studies.Focus group.CBA.Surveys of beneficiaries (response rate of 30%, 45% and 56% for the three measures).CIE (PSM), using administrative data.
The QuA programme had positive effects on the immediate and long-term integration into employment of participants.Youth workshops and PACE helped increase vocational education and employability of disadvantaged young people, especially through measures to reduce early school leaving.About a third of participants in measures for labour integration of prisoners and ex-prisoners found a job immediately after.The low level of qualification of participants in all the measures prevented them from obtaining well-paid jobs, so that it was possible for relatively few participants to find sustainable employment.The measures have improved the social inclusion of participants.
Not all target groups could be covered by the survey and the characteristics of respondents differ from the overall population of participants (women and migrants are overrepresented).
Consider a more comprehensive definition of poverty, including personal and social stability, and include people having precarious or low-paid jobs in the target group.Increase “PACE” project staff to expand preventive support and improve its quality.Ensure more detailed information is obtained on the employment status of participants afterwards to better measure the results.Adapt workshops to regional labour market needs and expand contacts with local companies to support the integration of participants.
Evaluation of the project selection system under the European Funds for Łódzkie Programme, 2021-2027
Ewaluacja bieżąca kryteriów i systemu wyboru projektów Fundusze Europejskie dla Łódzkiego 2027
Maciej Gajewski, Robert Kubajek, Wiktor Mrozowski, Tomasz Schimanek, Jan Szczucki, Justyna Witkowska
The subject of the evaluation is the project selection system of the programme.
To assess the criteria and project selection system of the programme for consistency with its objectives.
Analysis of monitoring data.Desk analysis of relevant documents.13 interviews with implementing institutions and 12 with experts.2 focus group interviews with unsuccessful applicants.3 case studies.CAWI/CATI survey of 587 beneficiaries.CAWI/CATI survey of 24 members of the Monitoring Committee.Workshop.
The set of criteria is coherent, rational and correctly linked to the programme's objectives and logic.The project selection system's key strengths are its objectivity, impartiality and transparent procedures.Although the overall assessment of the selection system is generally positive, the length of the procedures remains problematic, according to 58% of beneficiaries.
Prepare a manual presenting the criteria used for each call for proposals, together with interpretative guidelines.Organise training courses and information meetings for measures with the potential to attract a high number of applicants.Organise information meetings, training sessions and webinars on horizontal principles.
Evaluation of the Knowledge, Education, Growth OP, 2014-2020 - achievement of objectives for PA V (health)
Ewaluacja on-going realizacji celów V OP PO WER
Grupa BST Sp.zo.o., ul. Mieczyków 12, 40-748 Katowice, EU-Consult sp.zo.o., ul. Toruńska 18 C, lok. D, 80-747 Gdańsk
The evaluation assesses priority axis V (PA V) on healthcare services of the Knowledge Education Development OP 2014-2020 in Poland.
To assess the relevance, coherence, effectiveness and efficiency of the measures implemented under PA 5.
Documentation and data analysis;SWOT analysis;panel of experts;CATI surveys of beneficiaries and of a group of citizens who used healthcare services;IDI survey of representatives of the Managing Authority.
Investment priorities identified under PA 5 are relevant given the specific needs and coherent with the main national strategies in this area.The current level of implementation of projects is satisfactory. Some limitations were identified in the form of the constant need to review preventive programmes by the Agency for Health Technology Assessment and Tariff System, some issues in the tendering process as regards prevention measures and some delay in the implementation of measure 5.4 due to the characteristics of the target group.Previous experience in carrying out projects had a positive effect on implementation.The planning and functioning of the measures under PA 5 are satisfactory as is the institutional capacity of the Ministry of Health.Project selection procedures were also appropriate.Without ESF support, projects would either not have taken place or would have done so on a smaller scale.
There is a need to:introduce some changes in measure 5.4 to make it more attractive for doctors to participate (e.g. by increasing the number of education credits or online training and e-learning);change the rules applying under measure 5.1 to avoid the under-utilisation of funds which was due to the division into macro-regions;create a catalogue of prevention programmes to avoid the involvement of the Agency for Health Technology Assessment and Tariff System and make the process more effective;adapt the measure planned to changing needs;identify and disseminate examples of good practice.
Impact evaluation of the Bolzano ESF OP, 2014-2020
Secondo rapporto di valutazione di impatto del Programma Operativo del Fondo Sociale Europeo
Andrea Naldini, Eleonora Peruccacci, Camilla Pangallo, Lorenzo Zecchin
Ismeri Europa srl - www.ismerieuropa.com
The evaluation examines the results achieved by the three PAs of the OP, employment (PA1), social inclusion (PA2), and education and training (PA3) up to September 2022, and how design and management can be improved. The report complements the findings of previous counterfactual and thematic evaluations carried out in 2020-2021.
To assess the contribution of the OP to the social and economic development of the region, and to draw lessons for 2021-2027.
Desk research.Analysis of statistical data.Analysis of monitoring data.Analysis of administrative data.14 interviews.Survey to training institutions.Focus groups.
By September 2022, the OP involved some 73,000 participants, 95% under PA3 (Education and training).The OP generally reached the target groups, but support could have been strengthened for women and NEETs.The measures supported are complementary to similar ones financed from other sources.Distance learning helped to maintain activities during the pandemic (though only among young people not too far from the labour market).The ESF allowed transversal skills to be supported through social innovation projects.The experience of educators is crucial for increasing the effectiveness of policies to combat early school leaving.
Further strengthen partnership with stakeholders.Focus on psychological as well as education aspects of early school leaving.Continue to support social innovation projects.
Evaluation of research and innovation under the Aquitaine, Limousin and Poitou-Charentes OPs, 2014-2020 – phase 1
Evaluation des FESI 2014-2020 - Marché subséquent n°4 - Objectif thématique 1 - Recherche et Innovation - Phase 1
ROP Aquitaine 2014-2020; ROP Limousin 2014-2020; ROP Poitou Charentes 2014-2020
Yann Cadiou, Sébastien Guilbert, Karine Lanoix, Aurélien Fichet de Clairfontaine, Vanina Choe, Valentin Mangiavellano, Yvan Meyer
Technopolis, www.technopolis-group.com; Edater, www.edater.fr
The evaluation examines the measures for research and innovation (R&I) up to end-2020.
To assess the results and effects of measures up to end-2020, the efficiency of the OPs, the relevance of the objectives and measures implemented, and the programming dynamics in relation to the RIS3 areas, and to make recommendations for 2021-2027.
5 scoping interviews.12 interviews.Surveys of higher education, research and innovation bodies (36% response rate).Survey of enterprises (45% response rate).Physical and financial balance sheet data analysis.Before-after analysis.
By end-December 2020, over 80% of funding for TO1 had been allocated and 136 out of 428 projects had been implemented.The majority of public research bodies expect to achieve project objectives.The ERDF triggered the decision to launch the projects, has accelerated their implementation, and enables additional funding to be mobilised. Without EU support, most of the projects would not have been carried out.The administrative burden and Covid-19 were the main obstacles to implementing projects.The majority of public research bodies and intermediaries believe that the monitoring and closure of projects are complicated.According to public research bodies, the three main outcomes of the ERDF interventions are an increase in scientific publications, restructuring and the development and construction of areas devoted to R&I activities, and an increase in theses defended.The projects implemented will make it possible to develop innovation in enterprises and new product, and to strengthen the competitiveness of SMEs.According to before-after analysis, the average turnover of enterprises increased by 32% between the year before obtaining funding and two years after. Over the same period, recruitment increased by 18%, with an average of 5.3 jobs being created.
By end-2020, only 31% of projects had been completed. It is difficult to assess the results of the survey of public research bodies by OP, because there were only three responses from the Limousin OP and 13 from the Aquitaine OP. The small number of responses to the survey of enterprises also did not enable an OP-specific analysis to be carried out. The result indicator data for 2020 will only be available in 2022.
Improve the monitoring and guidance system.Reflect on the implementation of objectives to allow a better understanding of the impact of the ERDF on final beneficiaries.Consider supporting innovation projects in large companies through loans or a combination of loans and subsidies.Support activities for the implementation of innovation projects in SMEs.Simplify administrative procedures.
Ex-ante study on the possibility of using financial instruments under the Competitiveness Programme 2021-2027, in the Czech Republic
Předběžné posouzení finančních nástrojů Operačního programu Technologie a aplikace pro konkurenceschopnost 2021 - 2027: Závěrečná zpráva
www.pwc.com/cz/cs.html - cz_info@pwc.com
This is an ex-ante study on the use of financial instruments (FIs) in the programme, covering SOs 1.2, 2.1, 4.1, 5.1, 6.1. Data used range from September 2021 to 30 June 2022.
To carry out an ex-ante assessment of the use of FIs in the programme, consistent with the 2021-2027 regulatory prescriptions.
Analysis of relevant policy documents.Surveys of 1,000 beneficiaries of the programme and of National Development Bank FIs.Interviews with 20 stakeholders.
Commercial loan guarantees, either alone or in combination with subsidies, are planned for SO 1.2 measures on digital business, technology 4.0 and data centres and seem to be adequate (around EUR 39mn).Commercial loan guarantees, discounted loans and capital/quasi-trading investments are identified for SO 2.1 measures on expansion warranties, expansion credits and risk capital and seem to be adequate.Preferential loan/guarantees for a commercial loan in combination with a subsidy are envisaged for energy savings measures under SO 4.1 and this seems to be suitable.Commercial or preferential loans either alone or combined with subsidies, are envisaged for SO 5.1 measures for industrial water management. The EUR 5mn allocated seems too low given the needs of large enterprises.Guarantees for commercial loans in combination with a subsidy are indicated for SO 6.1: measures on the purchase of alternatively powered vehicles, and building recharging and filling stations in enterprises, which seem adequate .
Set up a means outside the ESI Funds for SO 5.1, e.g. under the Just Transition Mechanism, which allows the financing of the development and modernisation of water infrastructure in companies and the public sector.
Intermediate evaluation of the Social Inclusion and Social Economy OP, 2014-2020
Evaluación Intermedia 2019 del Programa Operativo de Inclusión Social y de la Economía Social
OP Social inclusion and social economy ESF 2014-2020
This the second interim evaluation of the Social Inclusion and Social Economy OP examines interventions financed under the 4 PAs, to promote 1) sustainable and good quality employment (IP 8.iii and 8.iv), 2) social inclusion and combating poverty and discrimination (9.i, 9.ii, 9.iii, 9.v), 3) social innovation (9.i) and 4) transnational cooperation for the exchange of good practice on social inclusion (9.i).
To assess the implementation, effectiveness, efficiency and impact of the OP.
Desk analysis of OP documentation and monitoring data and national social inclusion policiesInterviews with the MA and IB and beneficiariesQualitative analysis of data from an on-line survey of beneficiaries (160 respondents)Case studiesQuantitative analysis of data from a telephone survey of participants (1,000, out of 460,000)Logistic regression on OP monitoring data.
The degree of financial implementation is satisfactory: 86% of funding has been allocated and eligible expenditure amounts to 42% of fundingThe OP has reached more men than women, which is not in line with expectations, but more vulnerable people have been coveredThe main weaknesses are the design of operations which does not correspond to territorial specificities and inadequate coordination between MAs and other implementing bodiesThe logistic analysis indicates that vulnerability and living in more developed regions are linked to better chances of positive results being obtainedEqual opportunities for men and women was applied successfullyThe strategy is coherent with the Europe 2020 objective of reducing poverty and social exclusion, and with the European Pillars of Social RightsEU added value is achieved through measures being developed where there was no other support for social inclusion and social economy and an extension of the evaluation culture.
Data available does not allow a counterfactual impact evaluation and analysis of unit costs to be carried.
Boost the financial implementation of the OP in less developed and transition regions Encourage women to participate in the OPIncrease cooperation with third sector bodiesIntensifying coordination between managing bodies to better understand local needs.
Evaluation of the Veneto ESF OP, 2014-2020 - 2021 Report
The evaluation examines the implementation of the OP as at end-2020.
To assess the progress of the OP up to end-2020, the relevance of the strategy, the territorial response to the measures implemented and the functionality of the management and control system.
Desk analysis of OP documentation.Analysis of monitoring data.Analysis of statistical data and administrative data on employment.Questionnaire interviews with representatives of implementing bodies.
Over EUR 195 million was allocated in 2020 to measures in response to Covid-19, to support jobs and economic activities.Most of the output and result indicator targets for 2023 have already been reached or are close to being so.73% of projects are located in urban centres with high income levels and are not in areas with the most serious socio-economic needs. Reaching the target population in peripheral municipalities is particularly critical. The OP strategy is still relevant given socio-economic developments in 2020.The OP’s management and control system is functional and adequate.
Evaluation of SICAP pre-employment support under the Employability, Inclusion and Learning ESF OP, 2014-2020, in Ireland
Evaluation of SICAP pre-employment supports
Adele Whelan, Judith Delaney, Seamus Moguinness, Emer Smyth
The evaluation examines the effects of pre-employment interventions under SICAP managed by Local Development Companies (LDCs), especially for those more distant from the labour market.
To assess the employment impact of the interventions under SICAP managed by Local Development Companies (LDCs), the effects on soft skills and the effectiveness of the implementation approach.
Administrative data on the characteristics of participants and of interventions.Counterfactual analysis (PSM based on the Integrated Reporting and Information System database).Survey of 45 LDCs (91% response rate).5 case studies (interviews with LDCs staff, participants, employers, stakeholders).
The SICAP holistic approach was widely seen as relevant and suitable for those most distant from the labour market.The intense one-to-one support is considered a crucial element for developing self-confidence and motivation and for assisting participants to find and stay in employment and/or for identifying suitable education/training courses.LDCs play a key role in providing a connection to local services and stakeholders, especially in respect of Local Employment Services and/or Jobs Clubs.56% of LDCs reported a lack of local mental health/psychological services; 40% the absence of substance abuse and family support, the latter being a greater issue in rural areas.The counterfactual analysis found that SICAP had a positive impact on employment in the short-term, especially for those with very low levels of education, the employment rate of participants being 18 percentage points higher than for the control group.Positive employment results come more from one-to-one interventions rather than group activities.The measures have led to increased motivation and self-confidence in both participants and LDCs staff, the relationship between the two being a crucial factor of success.
Data from the Live register could not be accessed, so those not receiving SICAP support for employment but only for personal development and life-long learning are used as a control group. Outcome variables were available only for 3 to 6 months from the interventions.
Provide more intensive support for groups that are furthest from the labour market and face multiple barriers.Extend and increase funding for outreach activities.Subsidise travel for participants in rural areas.Strengthen links with employers either directly or through increased interaction with regional skills fora, job fairs and similar.Increase awareness of SICAP support among those who could potentially benefit.Improve the tracking of those accessing SICAP support by using official administrative datasets.
Employment and Social Inclusion, OP 2014-2020 - analytical note on the indicator system and the targets
Note d’analyse du système d’indicateurs du POn FSE 2014-2020 et de ses cibles
This report assesses the monitoring system of the ESF OP 2014-2020.
To assess the relevance of the indicators with respect to the OP strategy and their consistency with the Specific Objectives (SOs) identified and to estimate the achievements by 2023.
Desk analysis of policy documents;analysis of data from the Viziaprog database.
Investment Priority (IP) 8.i – SO1: the indicators are very consistent with the objectives.IP 8.vii – SOs 1 and 2: relevance and consistency are high for SO1; relevance is high for SO2, but consistency is less so as Job Centre consultants are not covered by the target.IP 8.iii – SOs 1 and 2: for SO1, coherence and relevance are good, but for SO2, there are no outcome indicators.IP 10.i – SO1: coherence and relevance are good.IP 8.v – SOs 1, 2, 3, 4 and 5: coherence is good, but the “Number of projects to anticipate changes” indicator is too generic. The indicator for SO 2 covers only part of the activities. Relevance and coherence for SO3 are good. For SO4 and SO5, consistency is not covered.IP 8.vi – SO 1: the indicator is relevant but consistency is not covered.IP 9.i – SOs 1, 2 and 3: indicators are consistent but relevance is only partly covered in respect of SO1 and SO3.2023 target indicators: only the targets for IP 8.v are easily achievable.
The performance framework was not analysed and participant data are estimated. It does not take account of the development of the OP strategy.
Thematic evaluations should be carried out for some SOs, such as SO1 under IP 8.vii.Labour market support, especially for SMEs, should be carefully monitored.
Support to teachers under the Human Capital OP in Wielkopolskie, 2007-2013
Ocena wsparcia skierowanego do nauczycieli w ramach Działania 9.4 PO KL
Małgorzata Zub (redakcja), Agnieszka Białek, Karolina Malinowska, Paweł Penszko, Magdalena Smak, dr Dominika Wlaczak,
Instytut Badań Strukturalnych ul. Rejtana 15 lok. 28, 02-516 Warszawa, Polska e-mail: ibs@ibs.org.pl, www.ibs.org.pl, Tel: + 48 22 629 33 82
This evaluation is aimed at assessing Measure 9.4 of the Human Capital OP, which targets highly qualified education staff in Wielkopolskie Province. In particular, the study focuses on the utility rating of the projects and complementarity of support.
The objectives of the evaluation are to assess the utility rating and the complementarity of the support provided by Measure 9.4 for highly qualified teachers in Wielkopolskie.
Desk research.Survey CAPI/CAWI.Interviews.Monitoring data.Educational information system (SIO) data.Teaching material.
Part of the projects implemented were aimed at education development as envisaged in national and regional strategic documents, especially with regard to the ability of teachers to use ICT.The most common theme in the projects implemented relates to improving the use of ICT in teaching, which has been rarely used before.The results are inconclusive on how much teachers are trained to help students to develop social skills.
The focus of improvement should be decided by the school in cooperation with teachers rather than being imposed from above.There is need to develop local and supra-local cooperation to improve the quality of education, while ensuring the measures taken are complementary.There is also a need to define the evaluation criterion as regards the comprehensiveness of support.
Cohesion Policy impact on cross border Polish regions in 2007-2013
Wpływ polityki spójNości na rozwój przygranicznej i ponadnarodowej współpracy terytorialnej polskich regionów w perspektywie 2007­2013
OP ETC Polska (Woj. Lubuskie) -Brandenburgia 2007-2013; OP ETC Polska-Słowacja 2007-2013; OP ETC Południowy Bałtyk (South Baltic) 2007-2013
Michał Bil, Jan Charkiewicz, Paweł Dembiński, Bartłomiej Iwańczak, Izabella Jurkiewicz, Agnieszka Laskowska, dr Julita Łukomska, Wojciech Pander, Jarema Piekutowski, Agata Rauzer, dr Katarzyna Szmigiel-Rawska, Katarzyna Wołczkiewicz
WYG PSDB Sp. z o. o. ul. Bitwy Warszawskiej 1920 r. 7, 02-366 Warszawa, www.wygpsdb.pl
This evaluation assesses the impact of Cohesion Policy in 2007-2013 on the development of cross-border, interregional and transnational territorial cooperation in Poland.
To assess the implementation of the ETC OPs and their impact on cross-border areas.
Data analysis;CATI interviews;CAWI interviews;statistical analysis (linear models, regressions).
The objectives of the ETC OPs were in part achieved;the projects implemented contributed to improving transport and touristic infrastructure in border areas;they also increased the internationalisation of Polish local government bodies, NGOs and universities;their contribution to human resource development was generally positive and in some cases targeted more people than planned (OP CZ-PL);the Polish national OPs which operated in the border regions were complementary to the ITC programmes.
There is a need to:maintain a significant role for local authorities in the implementation of projects;further improve links between institutions and research centres;further improve transport networks and ICT;promote the integration of local public and private organisations through local cooperation networks.
Evaluation of the SYPO project on the development of teachers’ competences under the Research, Development and Education OP, 2014-2020, in the Czech Republic
Evaluace systémového projektu „SYPO“ - Průběžná zpráva 1
The evaluation examines the progress of the SYPO project on the development of teachers’ competences.
To assess the implementation and results of the SYPO project, the main obstacles and the opinions of the stakeholders involved.
Monitoring data.Surveys.Interviews.Focus group.
The project is progressing well and most of the activities completed have met their objectives.Methodological cabinets have been set up and teachers are satisfied with the support received through them, though greater involvement is sought and more cooperation between national and regional levels.Support for schools’ management is also effective and beneficiaries appreciate the possibility of individual support and sharing of experience.The large number of ESF projects for schools risks creating duplication of activities, lack of experts for some projects and teachers being overwhelmed by extra-curricular-activities.
Adjust activities in the light of recent legislative developments.Increase involvement of teachers in the methodological cabinets and cooperation between national and regional levels.Improve integration between different projects and differentiate activities according to school levels.Investigate whether findings from surveys might contribute to improving the project design.
Evaluation of the Hans-Fritze-Haus-Centre in Hamburg
Thematische Evaluation des Hans-Fritze-Hauses - Zentrum zur sozialen Integration suchtgefährdeter Menschen
Ramboll - Marcus Neureiter, Henriette Reichwald
Ramboll, Chilehaus C, Burchardstraße 13, D - 20095 Hamburg, www.de.ramboll.com
The evaluation assesses the effectiveness of the “Hans-Fitze Haus” project supported by the Hamburg ESF OP 2014-2020 for the social inclusion of vulnerable people, particularly those affected by drug and alcohol addiction in Hamburg-Harburg.
To assess the effectiveness of the “Hans-Fitze Haus” project.
Analysis of documentation and monitoring data;interviews with project staff and 9 interviews with other stakeholders;two surveys of beneficiaries, of 'day meetings' (27) and counselling and placement services (17).
The target group is mainly made up of men with an average age of 44, a low education level (75% have lower secondary education or lower) and mainly unemployed (70%). 61% of them are addicted, mainly to alcohol, 20% homeless and 33% were born abroad. The target group was expanded to include migrants from Eastern Europe in 2017, creating additional treatment and care needs.All output targets have been fully achieved.Trust plays an essential role in the success of the action taken.Cooperation with doctors, lawyers, job centres and police is important, but some difficulties persist with therapeutic facilities (due to a lack of resources) and job centres (due to their lack of understanding of the needs of the target group and the excessive formality of procedures).The survey results indicate that the project contributed to the social inclusion of those supported, particularly through placement services which helped to improve individual situations, day meetings to create social connections and social counselling to deal with various problems.Positive effects are also evident in the improvement in public spaces in Hamburg-Harburg.
There is a need to:maintain the project in the future to ensure a long-term perspective for staff and beneficiaries, since continuity and building up a relationship of trust are key factors for success;improve the day job market, as it contributes to the social integration of people, through measures to attract potential employers and further cooperation with job centres.
Evaluation of systemic and conceptual projects under the Research, Development and Education OP, 2014-2020, in the Czech Republic - Evaluation area A – 2°interim report
Evaluace systémových a koncepčních projektů z výzev PO 3 OP VVV - Evaluační okruh A, 2. Průběžná zpráva
The evaluation examines stakeholders’ satisfaction with measures funded by the Research, Development and Education OP, 2014-2020 in Czech Republic.
To assess the satisfaction of those involved in different action plans funded by the OP at regional and local level.
Regional and local plan managers are mostly satisfied with support received for entrepreneurship, vocational training and development of partnership, though less so with lifelong learning, transfer of information and overall improvement in education.Satisfaction is higher at local level than at regional.‘Model project’ applicants are satisfied with support for project submission.Action plans helped create new sustainable territorial partnerships between local players.
Improve support provided by experts for regional plans and provide individual support at local level.Focus interventions on concrete actions, avoiding non-essential administrative and formal activities.
Evaluation of the Job Integration Year (JIY) measure financed by the Bayern ESF OP 2014-2020
Evaluationsbericht zur Förderaktion 12: Berufsintegrationsjahr (BIJ) Bayern 2014-2020
Institut für Sozialforschung und Gesellschaftspolitik GmbH, Otternbuchstraße 35, 13599 Berlin
The evaluation examines the Job Integration Year (JIY) measure under the Bayern ESF OP 2014-2020, which supports the transition into education and employment of young people without a training place and in need of language tuition (in the 2015-16 school year, it mainly involved migrants and asylum seekers).
To assess the implementation and progress of the measure towards the targets set.
Analysis of monitoring data and data from federal and regional statistics;4 case studies based on interviews with 30 of those involved in implementation;two rounds of phone surveys of participants: in 2017, including refugees and asylum seekers (55% of sample of 379), and in 2019;two rounds of online surveys of vocational centres: in 2016 (60% response rate out of 41) and in 2018 ( 94% out of 35) and with those responsible for project expenditure (in 2016, 60% response rate out of 57 and in 2018, 77% out of 30);interviews with counties.
4,655 young people have participated in the JIY measure, a quarter of them women and two thirds migrants.48% of participants took part in vocational training after support (target of 55%).In 2015/16, participants were refugees and asylum seekers mainly unaccompanied men, for whom the main type of support was language training.In 2016/17, 2017/18, 2018/19, participants were young people who lacked motivation and self-confidence and had social behaviour difficulties, two thirds of them men with an average age of 16.5.Refugees and asylum seekers showed more motivation than these, but their integration into labour market was more difficult.42% of refugees and asylum seekers surveyed completed vocational training 6 months after their support ending.
It was not possible to carry out a counterfactual evaluation because of a lack of data. It is difficult to generalise the results of the surveys of participants as the samples covered were significantly smaller than the population.
Continue to include traineeships as part of JIY support;take more account of the needs of companies and further expand their cooperation with vocational schools.
First evaluation of the communication strategy of the ESF Lazio OP 2014-2020
Primo Rapporto tematico sulla Strategia di comunicazione del POR FSE 2014-2020
Impact evaluation of the ERDF Electronic Public Administration OP 2007-2013
Az Elektronikus Közigazgatás Operatív Program hatásértékelése. Zárójelentés
OP Electronic Public Administration 2007-2013
The study is an impact assessment of the Electronic Public Administration OP funded by the ERDF in the 2007-2013 period.
The evaluation is aimed at assessing the relevance, effectiveness and efficiency of interventions, as well as the sustainability and synergies with other OPs. The study also identifies examples of best practice possibly to be implemented in the 2014-2020 period.
Desk analysis;data analysis;case study;interviews with the Managing Authority.
Lack of coordination with other OPs;delays in implementing the OP due to the slow beginning of the 2007-2013 programming period and the slowdown in 2010 which led to a partial revision of the OP in 2011;difficulty of assessing effectiveness as the OP operational logic does not follow the objectives set and indicators are not adequate (i.e. they are not specific, measurable, reliable and often not clear);lack of experience in project management of public officials which is mainly evident in the non-respect of deadlines and difficulties in organising tasks;lack of competence in the application of public procurement procedures;modernisation and development of e-governmental infrastructure and electronic administrative services, but their use has been less than expected because of the lack of proper dissemination and identification of needs through accurate assessments;the IT system has been only marginally improved;some resources were not used efficiently.
The lack of reliable and measurable data is pointed out.
The policy recommendations are to:define specific, relevant, measurable and reliable indicators;create a single policy management in order to ensure respect of deadlines and the exploitation of synergies with other OPs;create a risk assessment mechanism at the project level;avoid shortening the time of public procurement procedures;separate tasks and clearly identify specific units in order to improve the quality of management, especially in the case of large projects and consortia;undertake cost-benefit analysis of the project design;launch surveys and apply user tests to define the needs of target groups and increase awareness of the new e-services available.
Follow-up evaluation of ERDF support in 2014-2020 for transport in Spain – 2019
Evaluaciones de seguimiento y analisis sectorial por objetivo tematico (SADOT) Transporte. Año 2019
Multiregional OP for Spain ERDF 2014-20; 13 ERDF ROPs 2014-2020
Ministerio de Hacienda - Dirección General De Fondos Europeos SG de Programación y Evaluación
The evaluation examines the implementation of measures co-financed by the ERDF for transport investment in Spain.
To assess the role of the ERDF in developing the transport system in Spain.
Most of the funding for transport in 2014-2020 goes to Andalucía and the transition regions in general.Financial execution of TO7 (Transport) measures shows good progress in the more developed regions but it is delayed in the others.Financial execution of IP 4.5 (Sustainable mobility) is still very slow.Many of the result indicator targets for 2023 have already been achieved.The ERDF helped to increase the number of rail passengers, though the ratio of ERDF financing to total investment in transport in Spain has declined over the years.
It was difficult to carry out a comparative analysis, but a proper impact analysis will be carried out in the future.
Evaluation of the use of Territorial Instruments in the 2021-2027 programming period, in Poland
Ewaluacja stosowania Instrumentów Terytorialnych ZIT i IIT w ramach perspektywy 2021- 2027
All regional Programmes 2021-2027; European Funds for Infrastructure, Climate, Environment, 2021-2027; European Funds for Eastern Poland, 2021-2027
Michał Wolański, Beata Paczek, Dominik Makurat, Michał Babicki, Karolina Orcholska, Julia Widłak, Marcin Pinkosz, Mirosław Czerliński, Paulina Kozłowska, Artur Dziubiński, Wojciech Szłapacki, Michał Żelazowski
The evaluation examines the implementation of territorial instruments in Poland under the 16 regional Programmes and two national Programmes.
To conduct a comprehensive assessment of the system for implementing territorial instruments in Poland, including the adequacy of their programming at the EU and national level.
Desk research.Analysis of statistical and monitoring data.110 interviews with local partnerships, self-governments, MA, ministries and LAGs.Survey of 710 self-government respondents.3 thematic groups.16 regional case studies.Cross-country interviews + comparative desk research (Germany, Italy).2 expert panels.
Large territorial differentiation in how ITI (Integrated Territorial Investments) and ATT (Another Territorial Tools) are designed and implemented across regions, including number of partnerships, scale of allocation and approach to project selection.Widespread use of guaranteed allocations for ITI/ATT, which improves planning capacity, but reverses the strategy logic by basing strategies on available funds instead of diagnosed needs.Weak institutional anchoring of partnerships, with 66% operating under loose cooperation agreements enabling easy withdrawal, which limits long-term stability.Monitoring systems are fragmented and insufficient, relying mainly on project-level indicators, which do not reflect broader changes in functional areas.Mismatch between needs and available funding for revitalisation ATT, with regional funds covering only about 33% of identified needs.Delimitation of functional areas is inconsistent, leading in many cases to “fictional” functional areas shaped by political or personal factors rather than real spatial–functional linkages.
Standardise functional-areas at national-level, because current regional discretion produces inconsistent, non-functional partnership boundaries.Align ITI financing with national policies, as reliance solely on EU funds weakens coherence with domestic instruments and contradicts subsidiarity principles.Strengthen institutional bonds of partnerships, including more durable legal forms of agreements.Adjust monitoring systems, including harmonised rules and indicators capturing territorial changes, not only those at project level.Change the logic of the programming from “allocation-first” to “needs-first”.Improve coordination of EU and national funding for Revitalisation ATT, as current regional allocations are far too small to meet demand.
Thematic evaluation of the monitoring and management systems for the Valle d’Aosta ESF OP, 2014-2020
La valutazione dei sistemi di monitoraggio, gestione e governance del POR FSE
This is an intermediate evaluation examining the implementation mechanisms of the OP.
To assess the coherence of measures funded by the ESF OP with the regional context and target groups and the way they are implemented, with special focus on training.
Monitoring data.Official statistics.Programme documentation.
The strategy of the OP is still coherent with regional needs, in terms of both target groups reached and objectives pursued.Measures mainly concern traditional training programme though in few cases research centres are also involved.The regional provision of training depends on a limited number of bodies, which may affect its quality.
The analysis is only aimed at reflections and suggesting a future, more in-depth analysis.
Widen the range of training measures, including funding directed at final recipients, such as vouchers.
Evaluation of support for disadvantaged people in the labour market under the Lubuskie Voivodeship OP 2014-2020
Ocena wpływu wsparcia kierowanego do osób w najtrudniejszej sytuacji na rynku pracy w ramach Regionalnego Programu Operacyjnego – Lubuskie 2020
The evaluation assesses the results of support for those disadvantaged on the labour market under Priority Axes 6 (Regional labour market) and 7 (Social sustainability) of the Lubuskie regional OP 2014-2020.
To assess the extent to which PA 6 and 7 projects helped to improve the situation of beneficiaries and the efficiency and effectiveness of support.
Desk Research;CATI with beneficiaries (55 out of 76);CATI with participants (364 out of 8,298);CATI with employers (a sample of 350 employers);Individual interviews with Managing authority;Telephone interviews with participants (14 interviews);1 Focus Group with beneficiaries;Experts Panel.
The unemployment rate in the Lubuskie voivodeship is close to the average for Poland; the most problematic targets are the younger, aged 25-34, and the low educated.Overall, the ESF measures are consistent with the general regional policies, labour market challenges and national policies.Several types of operations are funded, including career counselling, traineeships, subsidised employment, work experience, support for self-employment, etc..At the end of March 2018, 70% of projects under TO8 and 63% under TO9 were fully implemented. The main problems in selecting participants were a lack of time for people to participate in ESF activities, their willingness and personal issues. No severe difficulties occurred in the implementation phase of the projects and a limited number of individuals dropped-out from the operations.ESF projects are considered effective in improving the employment situation by beneficiaries and employers.Start-up funds and traineeships are the most effective forms of support. By contrast, public works and intervention works, were on the least effective. Complicated procedures, bureaucracy and insufficient financial support, were the major obstacles to achieving the intended effects.Because of the ESF, over 50% of participants surveyed found a job, which they consider would not have happened without support. Overall, the participants were satisfied with the support received.
There is a need to:increase measures focused on practical skills;further support long-term unemployed people;put more emphasis on activation programmes aimed at young people and low educated;provide psychological support to people with no motivation.
Evaluation of measures to develop the digital economy under the Aquitaine, Poitou-Charentes and Limousin ERDF-ESF OPs, 2014-2020
Evaluation des fonds européens, structurels et d’investissement en Nouvelle-Aquitaine - Marché subséquent n°7 : « Développement de l’économie numérique, accroissement de l’offre de services et des usages »
kouassivi@teriteo.frhttps://www.teriteo.fr http://www.taranconsulting.co
The evaluation examines the measures supporting the development of digital infrastructure and the use of new technologies or services by companies and the population. It covers the implementation period between 2014 and 2020.
To assess whether the measures improving the accessibility, use and quality of ICT have achieved their targets. To assess the coherence, efficiency and impact of the measures and identify the main factors explaining the results.
Desk review.Analysis of monitoring data.15 interviews conducted with the persons in charge of the measures in the Regional Council.28 interviews with project holders.7 case studies.Phone survey of companies in Limousine area (13 respondents, 81% of the those contacted).Online survey of project holders in Limousine area (15 respondents, 43% of the those contacted).
In terms of digital infrastructure, the ERDF support is relatively limited - EUR 22.7m over a total budget of 1.247 billion euros dedicated to regional infrastructure.Significant differences were visible in terms of financial expenditure for supporting companies between the former regions Aquitaine and Limousin, as the first one had implemented a more ambitious strategy for the digitisation of companies.The measures helped companies to make better use of ICT tools, which led to an increase of their turnover during the period.ERDF supported a high number of projects aimed at increasing the use of digital services.The projects achieved the expected changes, especially in the sectors of digital education, e-health and data.The support helped to increase the level of ambition and accelerate the implementation of new ICT services.
Establish a new digital governance framework at the regional level.Mobilise EU funds to support the Third Places regional network.Include a “digital responsibility” criterion in the selection of projects in 2021-2027.Promote the resilience of Fiber to the Home (FTTH) networks.
Evaluation of the state of achievement of the objectives of Technical Assistance OP, 2007-2013 in Poland in 2015
Ocena stanu realizacji celów Programu Operacyjnego Pomoc Techniczna 2007-2013 w 2015 r.
WYG PSDB Sp. z o. o., ul. Bitwy Warszawskiej 1920 r. 7, 02-366 Warszawa, Tel: 48 22 492 71 04, Fax: 48 22 492 71 39, www.wygpsdb.pl - Stowarzyszenie na Rzecz Rozwoju Rynku Pracy „S-TO-S” – ul. Brożka 18/45, 01-451 Warszawa
The report examines the results for individual PAs of the Technical Assistance OP, 2007-2013, assessing for each measure, the state of achievement of output indicators and, where available, result indicators, as well as the amount of funding allocated.
To assess the state of achievement of the objectives of the OP at the end of 2015 in terms of the result and output indicators and the absorption of funding.
Desk research of OP documentation, financial data, data on output and result indicators and previous evaluation reports.CAWI online survey of beneficiaries of training and occupational development.CAWI online survey of users of the IT monitoring and control system.
The MA systematically assessed progress in implementation, reallocating funds and correcting overestimated or underestimated indicator targets.The implementation of the OP was smooth from the beginning, due to good preparation and the relevance of the support provided.Output indicators were achieved in most cases.Not all result indicators were achieved, though where not the difference between outcome and target was usually small.The budget was largely spent.
There were problems with responses to the CAWI survey.
Improve the assessment of the demand for employees by institutions eligible for support.Address the problem of reaching the target for people participating in various forms of occupational development.Improve the way in which employees are informed about the training available.Better adapt the measures provided to improve qualifications to the needs of participants.
Impact assessment of KAI 3.2 interventions (social service infrastructure) in the Regional OP, 2007-2013
Evaluarea de impact a intervenţiilor POR 2007-2013DMI 3.2: Reabilitarea/ modernizarea/ dezvoltarea infrastructurii serviciilor sociale
Andreea Savin, Coordinator, Monica Roman, Elena Botezatu Eugen Perianu - Civitta Strategy and Consulting SA, Archidata srl, NTSN Consulting srl, Structural Consulting Group srl.
The evaluation examines the key area of intervention (KAI) 3.2 of the Regional OP, 2007-2013, which covered social service infrastructure investment in the 8 development regions of Romania to improve their quality and capacity and their balanced development across the country to ensure equal access of people to them.
To assess the impact of KAI 3.2 interventions and their contribution to the quality of the social services provided. The two evaluation questions were: (1) what are the net effects of the interventions and what factors were responsible for the results? (2) Which interventions produced the best results?
documentation research;5 case studies;16 interviews with key stakeholders;2 surveys of the social service providers and final recipients;6 focus groups with beneficiaries;analysis of administrative programme data;theory of change;panel of experts;SWOT and PEST analysis;territorial analysis;counterfactual analysis (PSM and DiD).
Investment under KAI 3.2 improved the quality of social service infrastructure, ensuring minimum standard in provision.Investment in the modernization and rehabilitation of residential centres for the elderly improved their quality of life and health.Investment in buildings improved the accessibility of people with disabilities.Investments increased the number of FTE staff.There was a significant impact from projects where there was complementarity between soft and hard measures.
There were delays in arranging interviews and obtaining databases and project documents.The availability of social service providers and beneficiaries of interventions to participate in focus groups was limited.The target group was not homogeneous affecting the counterfactual analysis.There was only a short time available for the evaluation.
continue to finance investment in social infrastructure, based on a detailed analysis of regional needs, together with social policies for the groups concerned;provide financial support to SMEs in the form of both grants and financial instruments;prioritise projects better, in line with regional needs to increase the efficiency and impact of investment in social infrastructure;combine investment in infrastructure with soft measures;ensure the long-term sustainability of investments by using “sustainability plans” from the project design phase and by means of creating complementarities with other sources of funding.
Evaluation of the interventions to promote social dialogue financed by the ESF under the Human Resources Development OP, 2007-2013
2007–2013 m. Žmogiškųjų išteklių plėtros veiksmų programos priemonės „Socialinio dialogo skatinimas“ poveikio, efektyvumo ir sukurtų rezultatų vertinimas
OP Development of Human Resources 2007-2013
Olimpiečių g. 1A-26, LT-09200 Vilnius, Tel: 85269 0120, info@estep.lt, www.estep.lt
The evaluation assesses the interventions carried out under the “Promotion of Social Dialogue” project under the Human Resource Development OP 2007-2013.
To assess the efficiency, effectiveness and impact of the social dialogue measures implemented.
The evaluation uses qualitative and quantitative methods:Desk research on public policy and administrative documents;analysis of the intervention logic;12 interviews with public policy makers, companies, institutions and organizations involved in social dialogues;survey of participants in social dialogues (employers’ and workers’ representatives) (with 815 responses or 6%);case studies of four sectors (law enforcement, education, forestry industry and services). The data used are from interviews and surveys and secondary data from the monitoring system and relevant documents (e.g. collective agreements).
Overall, 32 interventions were carried out with EUR 4.2 million of co-funding.Around 21,500 people successfully completed training courses (239% of the target); 284 collective agreements were signed (47% of the target) and 44 tripartite and bilateral councils and commissions were set up (220% of the target).The intervention logic is in line with the needs of the target groups and suited to the Lithuanian context and its lack of social dialogue tradition.The cost of implementing the Social Dialogue measure (one training session for each civil servant) was EUR 15.5 per person, not significantly different from the training costs of other central authorities but higher than the cost of training organised by municipalities.The interview data revealed that insufficient attention was paid to bilateral (or tripartite) informal social dialogue and more focus on concluding collective agreements, which are considered less efficient in both legal and economic terms.
For public policy-making and implementing bodies, there is a need to become more involved in formulating content for promoting social dialogue. Projects in the current implementing period should not be limited to employers and employees or to typical training but should provide personalised support.A database on working conditions should be created to monitor job quality in enterprises.A set of contextual, output and impact indicators needs to be established to measure the implementation of collective agreements, occupational safety and health committees and other forms of social dialogue.
Evaluation of the Out-of-Work Service (OoWS) under the East Wales and West Wales and the Valleys ESF OPs, 2014-2020 – final report
Evaluation of the Out of Work Service: Final Report
Lucy Loveless, David Scott, Oli Taylor, IzabelaJamrozik, Emma Lovatt
This is a second evaluation of the service to provide employment support to 16–24 year-old NEETs and those aged 25 and over who are long-term unemployed or economically inactive and with substance misuse and/or mental health issues or both. Monitoring data go up to May 2022, interviews were carried out in May-August 2022.
To update the findings of the first evaluation, to assess the performance of OoWS and how Covid-19 has impacted the achievement of targets and Cross Cutting Themes indicators, to examine the effect of Covid-19 on the OoWS and the extent to which the recommendations from the first evaluation were followed up, and to identify best practice.
Analysis of monitoring data.Interviews with 9 Lot leads, 22 peer mentors, 5 employment specialists, 10 delivery partners, 4 employers, and 35 participants (service users).
By May 2022, 18,110 participants had been recruited to the OoWS, but 46% had left the programme early because of lack of Internet access or relevant equipment, diminished motivation to find a job, or unmet expectations concerning needs.Covid-19 had a significant impact on the involvement of new participants.There have been difficulties in recruiting and retaining peer mentors, especially during Covid-19, which led to their high turnover.Covid-19 had a significant adverse effect on the mental well-being of OoWS service users, which resulted in support focusing more on this than employment.Targets for gaining qualifications or work relevant certification were significantly exceeded.By May 2022, 10% of participants had completed a work experience placement or volunteering, and 10% had entered employment (target 15%). 2% of participants maintained employment 6 months after leaving, although there have been difficulties in recording these data.Covid-19 regulations do not appear to have had a significant effect on job search or entering employment, but the proportion of participants gaining a qualification or a work-relevant certificate during the pandemic is 4 pp lower than before Covid-19 (20%).
Participant response rates to emails were low and some participants were difficult to engage given their mental health or time commitments. Some contact information did not include participant names, which led to a less personal approach during initial contact. Only 5 out of 35 participants interviewed were aged 16-24. There were difficulties in contacting delivery partners and employers. It was not always possible to reach the targeted number of interviews with OoWS providers because of the limited number of people in some roles at the time of the fieldwork. The monitoring data do not include the date at which the outcomes were achieved, therefore, any time-based analysis is impossible and it is difficult to detect the impact of Covid-19 measures on these outcomes. There are difficulties in estimating the employment rate of participants 6 months after leaving the programme.
Ensure competitive salaries, employment contracts of a sufficient duration, and a clear progression and training plan to retain peer mentors.Formalise and systematise training and development of peer mentors, including continuing career development options.Define more clearly the role of peer mentors.Develop procedures and pathways to identify when a participant should move from well-being related support to employability support.Develop hybrid models of support.Prioritise short courses as an engagement mechanism and stepping stone to work.
Impact evaluation of investment in education and RDI under the Structural Funds OP, 2014–2020 in Lithuania
Švietimo ir mokslo srities prioritetų „Mokslinių tyrimų, eksperimentinės plėtros ir inovacijų skatinimas“ ir „Visuomenės švietimas ir žmogiškųjų išteklių potencialo didinimas“ poveikio vertinimas
GedimiNo av. 50, LT - 01110 Vilnius, Lithuania, Tel: +370 5 2620338, www.ppmi.lt, info@ppmi.lt, www.vpvi.lt/en
The evaluation examines support for RDI and education under the Structural Funds OP, the progress made and effects of interventions .
To assess the progress and impact of investment in education and RDI and to make recommendations for the 2021-2027 period.
Desk research.Analysis of monitoring and administrative data.Interviews with institutions responsible for implementation of EU Funds, project managers, beneficiaries and other stakeholders.Surveys of applicants, project managers and partners.Case studies.
The RDI system and infrastructure remain highly dependent on EU funding.Measures implemented so far made a limited contribution to the use of RDI infrastructure and attracting external users.EU investment under TO10 has improved the educational environment for 10% of children in early childhood and pre-primary education.Investment in early childhood education infrastructure has not addressed the urban-rural gap, most investment being concentrated in cities.Projects in schools helped to improve results and to induce changes in organisation and operation of schools.Support for the development of infrastructure in 17 higher education institutions affects around 40% of students in 2019-2020.Projects to increase access to higher education and to match it with the labour market needs are likely to benefit individuals, though their scope is relatively small.Sustainability of investment in higher education infrastructure is affected by a decreasing number of those of school-leaving age.Higher education institutes have become more internationalised, foreign students accounting for 8% of the total and 1,588 students (under half those planned) spent part of their studies abroad.Interventions helped to increase qualifications of researchers and their participation in international scientific activities (publications, conferences, joint project applications).Despite investment in vocational and adult education, the demand for this is falling.
The majority of projects are still ongoing, so, in the main the expected rather than actual effects are assessed.
Provide incentives for increasing commercialisation of R&D and ensure regular funding for this.Support joint research and business RDI activities.Ensure availability and accessibility of early childhoold education.Provide schools with laboratory equipment.Ensure effective coordination of university network reform.Take measures to minimise disparities in student performance and their impact on access to higher education of those from vulnerable groups.Aim to increase funding, reduce the administrative burden and ensure flexibility in developing science competences.Implement targeted initiatives in lifelong learning, and consider provision of scholarships based on results.
Evaluation of the "Pro Abschluss" (pro graduation) programme as part of the evaluation of the “Investment in growth and employment” Hessen ESF OP, 2014-2020
Evaluationsbericht über die ESF-geförderte Initiative „ProAbschluss“ im Rahmen der Evaluierung des Operationellen Programms im Ziel „Investitionen in Wachstum und Beschäftigung“ in Hessen aus Mitteln des Europäischen Sozialfonds (ESF) 2014 bis 2020 im Auf
Christian Loschelder und Stefan Feldens, Institut für Sozialforschung und Gesellschaftspolitik GmbH, Weinsbergerstraße 190, 50825 Köln
Loschelder@isg-institut.de; feldens@isg-institut.de
The evaluation examines the “Pro Abschluss”, initiative to support “training coaches and post-qualification advisory offices” and “qualification checks” under the Hessen OP. It is aimed at unskilled employees without vocational training and has the objective of establishing a state wide, regionalised advisory system composed of coaches and training points and of providing a qualification check of people who have not gone through the dual training system.
To assess the consultation phase of the initiative, the way that qualification needs and the eligibility of those seeking advice is assessed and the qualification checks.
analysis of monitoring data, official socioeconomic statistics, and research studies;an expert interview with the programme officer of the HMWEVL;three in-depth case studies;two surveys, one of participants, one of other consultants.
By November 2018 3,500 employees had received consultation but only 275 had received a qualification check, largely because this was not a formal requirement. Advisory personnel have difficulties in contacting companies and convincing them of the benefits of post-qualification services.There is no large supply of suitable further training organizations.Employees who participated in the initiative lacked adequate support from their employers as regards post-qualification opportunities.For employees who received a qualification check employment opportunities improved and they were able to apply the skills acquired. Those not checked, did not benefit much at all.The main result of the advisory service is that half of employees reported becoming aware of the importance of further training. It has not been possible to establish a functioning post-qualification market.According to stakeholders, the objective of the initiative is too narrow and fragmented.
Because of missing contact details and the limited knowledge of the programme only two interviews with companies were possible. The survey of participating employees had a response rate of 10% so it cannot claim to be representative.
Evaluation of coordination mechanisms in healthcare in respect of Cohesion policy in 2014-2020 in Poland
Ocena mechanizmów koordynacji interwencji publicznej w obszarze zdrowia stosowanych w ramach polityki spójności w Polsce
Aldona Frączkiewicz – Wronka, Iwona Kowalska – Bobko, Piotr Gozdek, Michał Dominik, Łukasz Szemiako, Krzysztof Gutta
„EU-CONSULT”, Toruńska 18c D, 80-747 Gdańsk
The evaluation assesses the mechanisms to coordinate healthcare interventions co-financed by ESI Funds over the 2014-2020 period in Poland.
To assess the adequacy and coherence of the coordination mechanisms in respect of health services and their effect on the Polish healthcare system.
Desk research;in depth-interviews with experts and representatives of institutions involved in health services;CAWI questionnaire to representatives of the Steering Committee (SC) on the coordination of interventions on healthcare;panel of experts;benchmarking analysis.
OPs and other strategic documents of Cohesion Policy are in line with national strategies in respect of healthcare;the majority of respondents consider the coordination mechanisms a way of strengthening the healthcare system;the division of roles between the Ministry of Health and the SC is appropriate and complementary;the functioning of internal communication in the SC is satisfactory, as is the participation of relevant stakeholder;the level of funding for ICT in healthcare is not adequate.
There is a need to:consider the possibility of implementing monitoring activities as part of SC tasks;support the planning of coordination mechanisms through the use of evaluations and Health Impact Assessments;increase the involvement of teams which work under the SC.
Evaluation of social responsibility in ERDF-supported companies in Aquitaine, 2014-2020
Diagnostic de responsabilité sociale de entreprises : bilan concernant les actions aidées par le FEDER
The evaluation assesses social responsibility in companies supported by the ERDF in Aquitaine in France in 2014-2020.
To assess the extent to which social responsibility strategies are in place and implemented in the region.
Desk research;analysis of monitoring data;survey of beneficiaries;panel of experts.
The analysis covered 31 projects involving companies 80% of which were public sector ones.The sectors most represented were those involved in the production and distribution of energy and government services.Strategies on social responsibility are generally well defined and articulated.Over half of beneficiaries have established the means to reinforce and monitor social responsibility activities.In some cases, strategies on social responsibility should be better connected with human resource policies.Practices involving consumer information and protection are in place.
There is a need to:support a faster implementation of social responsibility strategies;carry out annual reviews of social performance.
Final Evaluation of the Bund ESF OP 2007-2013
Zusammenfassung des Endberichts der Evaluation des Operationellen Programms des Bundes für den ESF in der Förderperiode 2007-2013
Rheinisch-Westfälisches Institut für Wirtschaftsforschung, ISG Institut für Sozialforschung und Gesellschaftspolitik
The study is the final evaluation of the Federal ESF OP 2007-2013 for Germany.
To assess the effectiveness, coherence and relevance of measures supported by the German ESF OP for the 2007-2013 period and to report the main findings of 7 impact evaluations on specific interventions (which mainly used counterfactual analysis).
Descriptive analysis of physical and financial indicators of all programmes supported by the ESF OP.Surveys of implementing bodies and participants on transnational measures.Case studies of Bremen, Leipzig, Detmold, Stuttgart and Uckermark on specific ESF projects.
Achievements have exceeded targets for the vast majority of programmes supported.Measures supported were coherent with other regional labour market policies.Overall the measures supported were targeted according to the specific regional needs: regions with higher unemployment focused mainly on employability and transition from school to work; stronger regions mainly on life-long learning and disadvantaged groups. A high female participation rate in relevant programmes was achieved.The counterfactual analysis found some measures had positive effects on employment; some had partly positive effects; some effects that were not statistically significant (such as support for those with a migrant background).The OP was successful in responding to different regional labour market challenges and promoting coordination between the regional, national and EU levels.The decentralised management and control model adopted was successful in ensuring the active participation of employment agencies, Job centres and municipal offices providing social services.
There is a need to:Improve and strengthen the role of partnership in planning and implementation phases.Maintain close cooperation between federal and regional levels.Ensure sustainability of interventions.
Evaluation of the support for vocational education. Estimation of the values of long-term result indicators under the Knowledge Education Growth OP, 2014-2020, in Poland
Ewaluacja wsparcia szkolnictwa zawodowego. Badanie wartości wskaźników rezultatu długoterminowego PO WER
The evaluation examines the impact of measures to increase the use of practical and flexible forms of vocational education, implemented under the OP.
To assess the effects of projects to improve the flexibility of the vocational education system implementation and provide recommendations for the future.
Desk research.CATI/CAWI survey of 1,040 beneficiaries and final recipients.23 interviews.4 focus groups.
The indicator concerning the use of solutions developed for practical training (PNZ) reached the value of 12.8%, far from the target of 45%.The indicator on the use of solutions developed through cooperation with universities achieved the value of 4.5%, the target being 10%.All the schools covered by the measures on the use of modernised core curriculum for vocational education (PPKZ) provide such type of education in cooperation with employers.In spite of the dissemination activities undertaken, the survey shows limited awareness of the model solutions developed by schools providing practical training and cooperation with universities.The biggest barriers hindering projects on practical training and cooperation with universities were Covid-19 and the nationwide teachers' strike, which destabilised the work of educational institutions.
In the future, place more emphasis on the promotion of the solutions developed, including in vocational education institutions.Consider the possibility of introducing additional financial support for schools, to cover the costs of transport of students to universities and improve cooperation between the latter and schools.Increase the use of tools allowing classes to be conducted remotely, e.g. in the form of webinars.
Evaluation of the implementation of the Transport OP, 2014-2020
Hodnocení vývoje OP Doprava 2014-2020 včetně ověření rozvojových potřeb a nastavení synergických a komplementárních vazeb
NDCON s.r.o., Zlatnická 10/1582110 00 Praha 1www.ndcon.czE-mail: ndcon@ndcon.cz
The evaluation assesses the states of implementation of the Transport OP 2014-2020 in the Czech Republic and the relevance of its objectives in current socio-economic conditions.
To assess the state of implementation of the OP and the relevance of its objectives given prevailing socio-economic conditions.
Desk Research;theory of change;panel of experts;in-depth interviews;CAWI.
There is no need to modify the intervention logic of the OP, though it does not fully support the most pressing priorities;four major projects have been submitted, three of them approved at national level;the rate of implementing SOs 1.1, 2.1 and 3.1 will be slower than planned because of delays in obtaining building authorisation;the rate of expenditure on SO 1.4 and 2.3 is significant but there is a risk of delay because of the slow Integrated Territorial Investment approval mechanism.
It is recommended to:undertake a preliminary monitoring of projects in order to discard those which could be subject to delay;prepare alternative projects for SOs 1.1, 2.1 and 3.1 in the event of absorption issues with current projects;give preference to projects on the backbone of the rail system, especially to major projects.
Evaluation of the potential for simplification of the Nordrhein-Westfalen ERDF OP, 2014-2020. Final report
Vertiefende Analyse im Rahmen der Evaluierung des OP EFRE NRW 2014-2020. Analyse zu den Vereinfachungspotenzialen in NRW. Endbericht.
Dr. Jan-Philipp Kramer (Prognos AG), Holger Bornemann (Prognos AG), Matthias Woiwode von Gilardi (DLR PT)
The evaluation examines the potential in the implementation phase of the OP. Data used go up to March 2019.
To identify measures to simplify procedures and improve project management by the MA and IBs up to March 2019.
Documentation and context analysis.Online survey of 400 beneficiaries.33 interviews with experts.3 workshops with stakeholders.Analysis of monitoring data.
Several reforms of governance have been implemented, including a reduction in the number of agencies from 100 to 9, the creation of an ERDF framework Directive, and the reintegration of a quality management function into the MA.Reforms have led to a reduction of almost half in the time-to-grant process, but the administrative burden and reporting costs are perceived as being high by beneficiaries, including as compared with other OPs.
Strengthen support to applicants.Ensure more transparency of audit procedures.Support a clear division of responsibilities in the management of project calls.
Evaluation of the ESF Traineeships programme in Wales, 2014-2020
Evaluation of the Traineeships Programme: Final Report 2015-2019
Egglestone C., Aylward N., Melville D., Bivand P., (Learning and Work Institute); Allies O. and Burgess A. (Wavehill social and economic research)
Learning and Work Institute, www.learningandwork.org.uk - Wavehill social and economic research, www.wavehill.com
The evaluation assesses the Traineeships programme co-financed by the West Wales and the Valleys (WWV) and East Wales ESF (EW) OPs 2014-2020, under Priority Axis 3 (‘Youth Employment and Attainment’) to reduce the proportion of 16-19-year olds in Wales classified as NEETs and to help them into employment or further education or training, as well as to increase their confidence and motivation. The programme has three strands: Entry-level, Level 1 and Bridge-to-Employment.
To assess the effectiveness, efficiency and impact of the Traineeships programme during the period 2015-2019.
Desk research of programme documentation (Welsh Government statistics and relevant literature);analysis of the ESF Participant Survey (911 respondents);in-depth interviews with 7 Welsh Government officials and 17 people involved in delivery of the programme;telephone survey of 48 employers and 105 trainees;counterfactual analysis (Coarsened exact matching approach);cost-benefit analysis.
By end-2018, 10,749 people had completed the traineeship in WWV (42% of the 2022 target) and 5,168 in EW (28% of the 2022 target).Half of the Traineeship participants gained a qualification upon leaving the course (52% in WWV and 47% in EW; 2018 target: 56%), 31% entered employment (32% in WWV and 30 in EW; 2018 target: 28%) and over 14% entered education or training (18% in WWV and 13% in EW; 2018 target: 20%).Findings from the ESF Participant Survey show that 12 months after leaving the programme, 52% of participants were employed (including self-employed and apprenticeships), 22% were in education or training, 17% were unemployed and looking for work and 5% were inactive.Social engagement and the development of soft and employment-related skills were the most common benefits reported by participants.Compared to those engaged in ‘Entry Level’ or ‘Level 1’, those participating in Traineeships were more likely to obtain an higher job entry (16 p.p. more), show a higher rate of sustained employment over three months (16 p.p.), higher earnings (EUR 715 one year after the training; EUR 2,015 two years after the training) and work more days (34 more days in employment one year after the training and 61 more days two years after the training).Over a 2-year time horizon, the CBA estimates of the net present value of the programme range from EUR 26.6 million to EUR 91 million; for a 3-year time horizon, from EUR 75.4 million to EUR 139.7 million and for a 5-year time horizon, from EUR 165.5 million to EUR 229.9 million.
The sample size of the telephone survey of employers and trainees was too small to allow for a robust statistical analysis of the results. For the survey of employers, it was not possible to target specific groups. For the counterfactual analysis the control group comprises those at ‘Entry Level’ or “Level 1’. The matching process could not take account of information from the Pupil Level Annual School Census, which might have reduced the potential bias of the results from possible unobservable differences. The CBA involved several assumptions.
There is a need to:design the future employability programme to ensure that different provisions are flexible, offer a coherent pathway for young people and are targeted at young people with complex needs;review the range of provisions that the Welsh Government make available to young people so that all the key stakeholders are well aware of the different support available and adopt a joined-up approach referring to young people;establish a robust evidence base for future evaluations and research by continuing to improve the Longitudinal Educational Outcomes Study by adding data from the Pupil Level Annual School Census, the Department for Work and Pensions data on benefit receipt and, possibly, data from Careers Wales.
Evaluation of environmental, social and economic effects of support under PA VII ‘Protection of cultural and natural heritage’ of the Lubelskie OP, 2014-2020
Ocena efektów ekologicznych, społecznych i gospodarczych wsparcia w ramach Osi VII Ochrona dziedzictwa kulturowego i naturalnego w Regionalnym Programie Województwa Lubelskiego 2014-2020
EVALU Sp. z o.o., ul. Dzika 19/23 lok. 55, 00-172 Warszawa, evalu@evalu.pl; ECORYS Polska Sp. z o.o., ul. Solec 38 lok. 105, 00-394 Warszawa, ecorys@ecorys.pl
The evaluation examines the impact of projects on the cultural and environmental heritage under the OP.
To assess the impact of measures to protect the cultural and natural heritage in terms of their ecological, social and economic effects and the way they have been implemented.
Desk research.Individual interviews (MA, IB, public institutions, experts).CAWI/CATI interviews with project providers.Counterfactual survey of beneficiaries of completed projects and non-beneficiaries. CAWI/CATI interviews with institutions managing national parks, landscape parks and natural reserves.Case studies.
IP 6.3 (Culture)Achievement of main output indicator targets is not at risk, but result indicator targets will not be achieved because of COVID-19.The effects of the measures are mainly social and in part ecological. Effects on the economy are only indirect and will be delayed.Support has gone mostly to smaller communities and investments were widely dispersed across the region.Success factors for achieving positive socio-economic effects are: location of the support, presence of other cultural or natural attractions in close vicinity, partner cooperation, coordination of tourist activities in the region.IP 6.4 (Environment)A significant part of the region has received support, the shortening of tourist paths has stopped and the length of cycle paths doubled.Social effects identified are better access to ecological education and an increase in the prestige of local communities. Economic effects are the development of services for tourists.The most effective measure has been the formulation of plans to protect three country parks.
Support local cultural and natural resources, including country parks, in the next programming period.Focus investment on cycle paths on supplementing existing ones in attractive areas.Develop digital means for tourist information.
Evaluation of the effects of support for enterprises and their innovativeness and internationalisation in Dolnośląskie in 2014-2020
Ewaluacja efektów wsparcia przedsiębiorstw (w tym MŚP) oraz ich innowacyjności i internacjonalizacji w województwie dolnośląskim w perspektywie 2014-2020
Maciej Gajewski, Jan Szczucki, Robert Kubajek, Bogdan Pietrzak, Justyna Witkowska
Policy & Action Group Uniconsult Sp. z o. o., 00-728 Warszawa, ul. Kierbedzia 4, www.pag-uniconsult.pl
The study examines the effectiveness of support for enterprises under the Dolnośląskie OP in respect of innovation, internationalisation, the low-carbon economy and the labour market.
To assess the relevance of the OP intervention logic in respect of the needs of beneficiaries, to identify factors adversely affecting the pursuit of goals and to make recommendations for support for SMEs.
Desk research of the programming documents.Analysis of monitoring data and official statistics.Theory-based evaluation.19 interviews with implementing authorities.16 interviews with beneficiaries.3 case studies.Surveys of 1556 beneficiaries; survey of 352 non-beneficiaries.2 focus groups.Expert panel.
Measures implemented had a positive impact on R&D activities in enterprises (especially SMEs).Support fostered the implementation of innovative solutions in SMEs.Activities to internationalise enterprises have been less effective (particularly, support for preparing strategies to enter new markets).Projects in relation to the labour market have been very effective.
Barriers resulting from the COVID-19 pandemic led to a need to undertake online interviews instead of face-to-face interviews.
Continue support for R&D and innovation projects, including in the form of innovation vouchers.Continue support relating to the labour market, but take account of significant changes taking place.Use external expertise, especially to evaluate the most complex and knowledge-intensive innovative projects and R&D ones.
Evaluation of systemic and conceptual projects under the Research, Development and Education OP, 2014-2020, in the Czech Republic – Evaluation area C – 1° interim report
Evaluace systémových a koncepčních projektů z výzev PO 3 OP VVV- Evaluační okruh C, 1. Průběžná zpráva
The evaluation examines recipients and beneficiary opinions on the “Inclusive and High-Quality Education in Territories with Socially Excluded Localities” project, funded by the Research, Development and Education OP in Czech Republic.
To assess the satisfaction of recipients and beneficiaries with the implementation of the inclusive education project.
CAWI surveys.In-depth interviews.Monitoring data.Reports, plans and studies.
Most of the beneficiaries are satisfied with the support received from the Agency for Social Inclusion, though some changes in local consultants were problematic.There is some confusion about activities under different local plans.Complementary projects involve an administrative burden and coordination issues.Self-evaluation by the implementing team is considered important, though it involves only a few people.
Provide better training of local consultants and share information among the various actors.
Evaluation of the implementation of selected measures of PA4 of the Smart Growth OP, 2014-2020, in Poland and EC programmes for research development – MODULE II
Ocena wpływu realizacji wybranych działań IV osi POIR oraz programów KE na rozwój jednostek naukowych, pobudzenie współpracy i komercjalizacji oraz rozwój kadr B+R, a także na umiędzynarodowienie nauki polskiej i możliwości budowania partnerstw międzynaro
LB&E sp. z o.o., 05-807 Owczarnia, Książenicka 98, e-mail: kontakt@lbie.eu., www.lbie.eu; EGO – Evaluation for Government Organizations s.c., Warszawa, e-mail: evaluation@evaluation.pl, www.evaluation.pl
The study examines the internal and external complementarity of R&D support for research units under PA4 of the Smart Growth OP.
To assess the cohesiveness and complementarity of PA4 measures and sub-measures of the Smart Growth OP, and the effectiveness of the non-competitive project selection process used.
Desk research of programming documents.Analysis of official statistics and monitoring data.9 interviews with research units and 6 with firms cooperating with them.
Support is generally internally and externally complementary, but some measures compete with each other.There is inadequate support for investment in fixed assets which address research centre needs.Opportunities for financing projects in Mazowieckie are limited because of its status as a developed region, though there is a need for the development of R&D there.There is a lack of support for the commercialisation of the results of research and cooperation with businesses.
Increase support for investment in low and medium-level cost research equipment.Discontinue the multiplication of measures supporting R&D projects implemented by research-industry consortia.Withdraw the separate allocation for R&D support to Mazowieckie or set it at a significantly higher level (closer to 30%).Launch human resource development support measures for R&D for cooperation with businesses.
Evaluation of the Regional OP 2014-2020 interventions “Development of health and social infrastructure by increasing the accessibility of health services, the efficiency of emergency hospital care and increasing the coverage of social services”
Evaluarea intervențiilor POR 2014 - 2020Dezvoltarea infrastructurii de sănătate și sociale prin creşterea accesibilităţii serviciilor de sănătate, eficienței îngrijirii spitalicești de urgență si creşterea gradului de acoperire cu servicii sociale
The evaluation focuses on PA8 of the Regional OP “Improving Educational Infrastructure” for investment in health and social infrastructure to develop the overall system, reduce inequalities and increase social inclusion.
To assess implementation, the contribution to expected results, the regional specific features of the intervention, sustainability and the intervention logic.
documentation research; 6 case studies;8 focus group with beneficiaries;50 interviews with selected beneficiaries and RDAs;panel of experts;analysis of Administrative programme data.
The unmet need for medical treatment has already been reduced below the 2023 target and the population served by improved health services is also well above target.Reducing response times to calls requires improvements in the availability of: ambulances, staff and roads.The new equipment acquired has increased the quality and reliability of services and reduced waiting times.Around 30% of hospitalisation cases are for conditions that could be as outpatient.The creation of social services for the elderly is far below the target, only 9% having access.Competitive calls for proposal can direct funding to those most experienced in project application and not to the most pressing cases.Funding has mainly gone to developing regions with social needs and a demand for social services, but disproportionately to urban areas within these.Integration of people with disabilities requires funding for awareness-raising and the provision of social services.
In the absence of progress and monitoring reports, the evaluation results were predominantly based on estimates from contractual commitments and information from interviews. The large number of evaluations that took place in parallel considerably limited the availability of certain groups of participants. The limited access to quantitative data constrained the analysis, though alternative data sources were identified to compensate.
finance integrated community centres to improve the access of vulnerable groups to services, create outpatient/diagnostic and treatment centre by transforming small and medium-sized healthcare units;modify outpatient services and update the fees;ensure the MA and the Ministry of Health plan strategies together;develop clear means of coordination at national level for standardising the assessment of infrastructure, equipment, and staffing needs;clarify the status and institutional role of community centres;increase the role of the line ministries in monitoring the provision of services.
Evaluation of objectives and results of the Castilla-La Mancha ERDF OP, 2014-2020
Informe de Evaluación 2019 de los objetivos/resultados del Programa Operativo FEDER 2014-2020 de Castilla-La Mancha
The evaluation examines the progress in implementing the Castilla-La Mancha ERDF OP, 2014-2020.
To assess the progress of the OP, the achievement of performance framework targets and its contribution to the Europe 2020 Strategy, horizontal aspects and climate change.
Monitoring data;desk analysis;in-depth interviews.
PA2 (ICT) and PA6 (Protection of the environment) have achieved performance framework targets, though PA1 (R&D), PA3 (SMEs competitiveness), PA4 (Low carbon economy), PA9 (Social inclusion), PA10 (Education), PA14 (Financial Instrument) have not, but PA9 and PA14 were only included in the OP in 2018.Result indicators show good progress, apart from two.The OP contributes to the Europe 2020 Strategy, though in most aspects progress is still slow.The OP contributes to the achievement of horizontal principles and the fight against climate change, especially through PA4 (Low carbon economy) and partly PA6 (Protection of the environment).
Result indicators can be influenced by factors other than the OP’s implementation.
Boost the execution of lagging PAs, especially PA4 (Low carbon economy) and revise allocation of funding and indicators.Provide training to Managing Bodies to foster progress of the OP.
Evaluation of the impact of social infrastructure measures under the Małopolskie OP, 2014-2020
Wpływ interwencji podejmowanych w ramach 12 osi priorytetowej Infrastruktura Społeczna RPO Województwa Małopolskiego na lata 2014-2020 na zwiększenie dostępności usług zdrowotnych w regionie
Ul. Toruńska 18C, lokal D; 80-747 Gdańsk; www.eu-consult.pl
The study assesses the impact of support to health services in Małopolskie.
To assess the effectiveness of healthcare measures, the relationship of outlays to the effects planned and achieved, the sustainability of the measures and the adequacy of support relative to regional needs.
Desk research.45 interviews.Surveys (506 respondents).3 focus groups.11 case studies.
20% of respondents report an improvement in the state of healthcare in the region compared to 5 years before, but 30% believe that the situation has deteriorated.Some time is needed for the effects of measures implemented to be perceived positively by residents.Measures allowed teamwork to take place, which has contributed to the development of coordinated care, though these activities are still implemented on too small a scale.
Put more emphasis on information and promotion activities in relation to projects implemented.Provide residents with access to high-quality psychological and psychiatric care and develop further services for the elderly and those in need of care.Create a competitive advantage for services provided by healthcare organisations operating in the region, in particular by strengthening the position of the University Hospital in Kraków as a leading R&D and teaching institution.
Evaluation of the business programme financed by REACT-EU under the Education and Entrepreneurship OP, and the Innovation and Sustainable Growth OP, 2014-2020, in Denmark
Evaluering af erhvervsfyrtårne finansieret af den danske REACT-EU indsats
OP Innovation and Sustainable Growth ERDF 2014-2020; OP Education and Entrepreneurship 2014-2020
The evaluation examines eight local business lighthouses created in 2021 as part of the EU’s recovery plan and financed through REACT-EU funds. Each regional business lighthouse focuses on a specific sector: green energy from offshore wind turbines; robotics; water technology. The analysis extends from 2021 to the beginning of 2022.
To assess the effects of the eight regional lighthouses on the creation of better conditions for long-term and sustainable business developments.
Visit of each business lighthouse.53 interviews with operators, project partners and target groups of each business lighthouse during the visit or by telephone.Questionnaire survey with supported companies for 6 of the 8 lighthouses (99 responses – response rate not indicated).
Cross-cutting synergies were limited between lighthouses and cooperation with universities was insufficiently geared towards research gains and universities’ funding models.Most of the SMEs participating in business lighthouses were already involved in existing networks; the lighthouses nevertheless succeeded in engaging new companies and actors.SMEs report a limited return on activities, but operators and project partners expect further benefits for SMEs in the future.Larger companies are more easily involved in lighthouse projects if their benefits are tied to new development trails, rather than to the company’s existing business.The administrative burden of participating in the project is a barrier for many companies, particularly smaller ones.The strongest collaborations were those with a long history and established relationships. Collaboration problems arose because of the numerous project partners involved, conflicting local interests or an excessive organizational structure.
Maintain the current business lighthouses structures and ensuring continuity of their objectives.Ensure that research institution receive benefits from business lighthouses that can be used in research and teaching.Set less requirements for project applications to increase flexibility of projects and ensure their relevance through the project period.Give companies a central role to ensure adequate levels of private co-financing.
Interim evaluation of Priority 7 “Technical assistance” under the South-East regional ERDF Programme, 2021-2027, in Romania
Evaluarea intermediara timpurie a Prioritatii 7 Asistenta tehnica a Programului Regional Sud-Est
Liliana Lucaciu, Adina Dragoman, Alina David
Civitta Strategy & Consulting, https://civitta.com/; Lideea Development Actions srl, https://lideea.eu/
The evaluation examines the technical assistance projects financed under the programme. Data used go up to December 2024.
To assess the level of implementation and the main results of technical assistance projects, such as training, help desk services, development of strategic documents, digitalisation of processes, and communication campaigns.
Literature review. Desk analysis. Analysis of monitoring data. Analysis of the logic of intervention. Analysis of statistical data collected from Eurostat and the National Statistical Institute (NSI). Interviews with beneficiaries and representatives of the MA. 2 online workshops with 12 participants (5 beneficiaries). Online focus group with representatives of the MA (13 participants). CAWI survey of 112 beneficiaries (41 responses). Panel of 4 experts. 1 case study.
Two major non-competitive projects have been contracted under Priority 7, which represents 4.9% of the total budget of the programme: one implemented in 2022–2023 and completed, and one launched at the end of 2024 and currently under implementation. The projects helped to strengthen the administrative capacity of both the MA and the beneficiaries, especially small and medium-size local authorities. Despite the overall alignment with beneficiary needs, several persistent challenges were identified: excessive administrative burden, repetitive document requests, limited functionality of the monitoring platform, unpredictable call schedules, and low administrative capacity at local level. Technical assistance measures were well aligned with broader national reforms, including the digitalisation of public administration, simplification efforts, and the objectives of the 2021–2027 Partnership Agreement.
The short time frame of the evaluation (December 2024 – March 2025) constrained data collection and analysis, requiring several processes to be carried out in parallel. Updated data as of 31 December 2024 became available only at the end of January 2025, forcing a revision of preliminary analyses under tight deadlines.
Simplify administrative procedures and provide targeted support to improve access to funding for small and less experienced beneficiaries. Prioritise investments in staff training and digital tools to enhance the efficiency and transparency of implementation. Ensure continuity and stability of human resources of the MA to support timely project delivery and effective programme oversight.
Ex-ante study of the Śląskie Programme, 2021-2027
Ewaluacja ex ante programu Fundusze Europejskie dla Śląskiego 2021-2027
EVALU Sp. z o.o.; ECORYS Polska Sp. z o.o.
The study examines the intervention logic adopted by the OP for 2021-2027.
To analyse how the proposed measures respond to development challenges and socio-economic needs of the region, whether the planned specific objectives are achievable and the extent to which the programme will contribute to the implementation of horizontal policies.
Desk research (EU and national regulations, strategic and programme documents, procedures, evaluations and analyses).Official statistics.Monitoring data.38 interviews.5 focus interviews.Survey (2,353 respondents).Statistical analysis.Workshop on recommendations.
Needs are appropriately identified and the intervention logic seems correct, but statistical data are not used as much as they could be.The demand for support through financial instruments for revitalisation projects could be limited.The selection of indicators of the OP is, in principle, optimal.The methodology for the estimation of indicators is appropriate, but the inflation indicator should also be taken into account.
Supplement the identification of needs with more statistical data.Provide an in-depth assessment of the justification of using loans as forms of financing for revitalisation measures.Adjust the indicators selected better to the specificity of measures.Take into account current inflation and the forecast for 2021-2027 when estimating unit cost.
Evaluation of the involvement of companies under the Botnia-Atlantica Interreg OP, 2014-2020
Företagen i Botnia-Atlanticaprogrammet. En utvärdering av det gränsöverskridande samarbetsprogrammet med särskilt näringslivsfokus
Interreg V-A-Sweden-Finland-Norway (Botnia-Atlantica) 2014-2020
Thomas Westerberg, Michaela Danielsson, Karin Ibenholt, Sari Rannanpää, Göran Hallin
Kontigo AB, Katarinavägen 19 SE – 116 45 Stockholm, www.kontigo.se, +46 (0)8 562 262 40
The evaluation examines the role and involvement of companies in the implementation of the OP.
To assess the extent to which the business community is involved in the OP, its effects on the implementation and results of projects as well as the possible obstacles to business participation.
Interviews with MA.Monitoring data.Survey of beneficiaries (40% response rate).Interviews with beneficiaries.
Companies participating in the OP are mainly those in the energy, water and manufacturing sectors, most of them knowledge-intensive SMEs but larger companies are also involved.Main reasons for participation by companies and results achieved are: acquisition and use of new practical knowledge and establishment of new networks/contacts.There is limited involvement of companies in project design and planning, so the interventions do not always match the needs of businesses and few investments are carried out as a result of the projects.A positive factor is the clear planning of activities, while long distances between partners and limited involvement in the design of projects are critical points.
The limited scope of the interventions and the time needed to produce results did not make it possible to analyse the effects of the measures through indicators, but only through surveys and interviews.
Carefully map challenges and opportunities for companies as regards OP projects to involve them at the right time and in the most suitable way.Set high demands for project added-value and requirements to ensure long-term sustainability of results.Strengthen participation of Norwegian companies.
Evaluation of the Interreg Greece-Cyprus OP, 2014-2020
Τελική Έκθεση και επικαιροποίηση της 2ης Αξιολόγησης συμπεριλαμβανομένης της Στρατηγικής Επικοινωνίας και την αξιολόγηση επιπτώσεων
The report aims to assess the results and impact of measures implemented under the Interreg OP. Data used go up to December 2022.
To monitor the programme's implementation, to assess its effectiveness and the relevance of the intervention logic, and to provide recommendations for its future revision.
Desk research. Analysis of monitoring data. Analysis of statistical office data. Analysis of previous evaluations. Survey of 257 direct beneficiaries (response rate: 56%).
Four calls for proposals were issued with a total budget representing 77% of the programme. 55 projects were included, with total expenditure exceeding the initial budget.Most indicators exceeded targets, such as those relating to studies, projects and energy consumption reduction. However, others, like air pollution reduction and recycling rates, did not meet expectations.Despite delays caused by the pandemic and the increased cost of materials, the programme remains on track, with remaining projects expected to be completed by the end of 2023.The programme met its objectives in areas such as ICT, energy efficiency, and cross-border cooperation. However, new needs arising from the pandemic will require adjustments in the fourth revision.Beneficiaries expressed high satisfaction with communication and management, but suggested improvements in relation to the dissemination of information and results.
The evaluation faced challenges in data collection, particularly for environmental indicators. The survey responses potentially underrepresented some stakeholders.
Adjust the programme’s objectives and intervention logic to address emerging socio-economic challenges, particularly those relating to COVID-19 and geopolitical shifts.Improve data collection systems.Enhance communication strategies to increase participation and ensure better tracking of programme outcomes.
Mid-term evaluation of the Piemonte ERDF OP, 2014-2020
IRES Piemonte – Istituto di Ricerche Economico Sociali del Piemonte
via Nizza 18, 10125 Torino, ires@pec.irespiemonte.it
The evaluation concerns the measures implemented under the ERDF Piemonte OP 2014-2020, in particular Axes I (Research, Technological Development and Innovation), III (Competitiveness of manufacturing systems) and IV (Sustainable energy and quality of life).
To assess the coherence between the initial effects and the OP objectives as well as the state of implementation of the planned measures.
Analysis of data from the monitoring system, information collected by the regional administrative body and Finpiemonte (a regional finance company supporting economic and social development, research and business competitiveness), and data from ASIA (Statistical Register of Active Enterprises) and AIDA (financial statements and administrative data for the main public and private companies).In-depth interviews with beneficiaries.
Overall, about EUR 588 million have been programmed (60% of the total), EUR 206 million committed (about 21% of the total) and EUR 60 million certified (around 6% of the total).A total of 821 companies received support, much of it going to large enterprises (representing only 8% of total beneficiaries but 86% of employees) in the area of Torino (61% of beneficiaries) and mostly manufacturing ones (63% of beneficiaries).Initial evidence suggests that measures implemented under Axis I are in line with the OP objectives, in that they support the maintenance of innovation networks and the local exchange of knowledge and services, strengthen the link between businesses and research centres and consolidate the growth of the most innovative and promising technological sectors. Overall, some 500 researchers were employed under this Axis.
Account should be taken of the fact that digitalisation and technological change will affect the composition of the workforce of the companies supported. Accordingly, the existing workforce will probably have to acquire new skills, which implies: collaboration with local training providers to establish training courses based on companies’ needs and a strengthening of company internal training capacity.
Evaluation of research and innovation measures under the Bremen ERDF OP, 2014-2020. Report on thematic cluster “Economy”
Evaluierung der Spezifischen Ziele 1 bis 4 des Operationellen Programms EFRE Bremen 2014-2020. Teilbericht Themencluster Wirtschaft.
Ulrike Biermann, Piet Hausberg, Kevin Lübke (Ramboll)
Ramboll Management Consulting GmbH, Jürgen-Töpfer-Straße 48, 22763 Hamburg, https://de.ramboll.com, piet.hausberg@ramboll.com
The evaluation examines the measures on "Research, development and innovation” (RDI) and the "Aerospace research programme” (LuRaFo). Quantitative data go up to 2020, qualitative up to 2022.
To assess the level of implementation, the results, relevance, effectiveness and coherence of RDI and the Aerospace research programme (LuRaFo) up to 2022.
Documentation analysis.Analysis of monitoring data and official statistics.Interviews with 7 experts.Online surveys of 102 firms (34% response rate) and 58 scientific institutes (50% response rate).ToC testing and contribution analysis.
Measures are relevant for the needs of beneficiaries, but their coherence with both national and EU policies is limited.Financial implementation is progressing well and targets for 2023 are likely to be met.The application and approval process for grants is problematic because of bureaucracy and limited digitalisation.Projects have had a positive impact on the generation of scientific knowledge, research activities, innovation and cooperation, which has led to new and improved products, processes and services.Research institutes supported have benefited from the results of the projects, in terms of increased sales and the creation of new jobs for those highly qualified.
Continue finance for RDI and LuRaFo measures in the next OP.Continue schemes with a very low grant threshold and short-term application, as they represent opportunities for SMEs.Use more flexible flat rates for eligible costs.Simplify, streamline and digitalise project application and approval processes.
Interim evaluation of the Öresund-Kattegat-Skagerrak Interreg OP, 2014-2020
Delrapport inom utvärderingen av Interreg Öresund-Kattegat-Skagerrak 2014–2020
Ylva Grauers Berggren, Thomas Westerberg, Jennie Persson
Oxford Research AB, Norrlandsgatan 11, 111 43 Stockholm, Sweden; www.oxfordresearch.se; office@oxfordresearch.se
The evaluation examines progress in implementing the OP.
To assess the contribution of projects to achieving the objectives of the OP and the UN Global Goals.
Monitoring data.Interviews with project managers.Survey of beneficiaries.
The main results achieved by the OP are increased common knowledge, new and strengthened networks and increased exchanges between local actors.Financial sustainability after EU funding is uncertain for several projects, while some of them have continued to live through new projects.Targets have been achieved to a large extent, except, in particular, for projects for the development of new technologies.
As few projects have been completed at the time of the evaluation, the results are only preliminary.
Evaluation of the impact of public transport measures on urban mobility in 2014-2020, in Poland
Ocena wpływu działań podejmowanych w ramach polityki spójności w zakresie transportu publicznego na mobilność miejską w perspektywie 2014-2020
All OPs 2014-2020 except OP Technical Assistance 2014-2020; OP Smart Growth 2014-2020; OP Digital Poland 2014-2020; OP Knowledge Education Growth
dr Michał Wolański, Mirosław Czerliński, Karolina Orcholska, Marcin Pinkosz, Paulina Kozłowska, Wiktor Kijania, Dominik Makurat, Jakub Kaczorowski, Beata Paczek
The evaluation examines the effects of measures on urban transport in achieving urban mobility objectives.
To assess the impact of transport projects on transport systems, the environment and cooperation between local authorities.
Analysis of monitoring data.11 interviews with relevant bodies.12 interviews with organisers and operators of public transport and city representatives.Statistical data analysis (including "city ranking").Counterfactual analysis (SPSM method), using administrative data.Regression analysis.3 case studies.Survey of 800 residents of 5 cities.Analysis of Big Data on traffic flows.Panel of experts.
The pandemic reduced interest in public transport.There is no evidence of any significant impact of Cohesion policy on travel habits. Road traffic in cities and functional areas has increased, along with travel time between urban centres.The measures have helped improve the safety of passengers, though road safety is still unsatisfactory.Investment in urban mobility has intensified the process of suburbanisation.Investment in urban and rail transport has been accompanied in Functional Urban Areas by a parallel expansion of roads of all categories, which severely limits its effects.Fragmentation of public transport in FUAs is being increased by the establishment of poviat and commune associations, which bypass the public transport organised by cities.
Assessment of effects is limited by projects still being underway. The effect of the pandemic, which is difficult to isolate, did not allow the impact of the measures on the number of passengers to be determined.
Develop a national public transport financing model and support the reconstruction and further development of public transport.Reduce EU funding for selected types of road investment in cities, and promote measures for walking and cycling.Develop a national SUMP as a strategic document to examine mobility issues in the country, covering all modes of transport.Make financing investment conditional on the development of SUMPs for FUAs or on the implementation of SUMP packages and key activities.Develop national regulations to unify the system of discounts and subsidies and to enable municipal and regional transport systems to be integrated.Establish or increase local cooperation.
Territorial strategies under Cohesion policy in Abruzzo, 2014-2020
Territori e strategie territoriali per la politica di coesione della Regione Abruzzo
ROP Abruzzo ESF 2014-2020; ROP Abruzzo ERDF 2014-2020
IRS, https://www.irsonline.it/en - Nomisma, https://www.nomisma.it/
This study examines different territorial definitions relevant for Cohesion policy and assesses the experience of territorial strategies in 2014-2020, up to end-2020.
To assess different territorial concepts and their implication for territorial strategies and to examine their results up to end-2020.
Desk analysis of documentation.Analysis of monitoring and statistical data.
Municipalities have a different capacity to respond to the opportunities offered by the ERDF.Provincial capitals and the city of Avezzano host more than 20 projects, while there are no projects in a third of municipalities.Municipalities can be classified into four groups in respect of the number of ERDF projects and their presence in territorial networks: lively ones (33 municipalities), lonely leaders (37 municipalities) with a large number of projects but involved in few networks, latent ones (136) with few projects but involved in many territorial networks, and awaiting ones (99) with a few projects and involved in few networks.
Evaluation of education and vocational training courses (IeFP) under the Toscana ESF OP, 2014-2020
I percorsi di istruzione e formazione professionale: strumenti per la programmazione dell’offerta formativa
The evaluation examines the courses (IeFP) for young people financed by the OP up to the beginning of 2020.
To assess the region’s needs in relation to the vocational training provided by IeFP and the employment results of those receiving training up to the beginning of 2020.
Monitoring data.Statistical data.Administrative data from the regional Employment Informative System (SIL) and the Student Register.
Over 2014-2020, most people qualified as catering workers (42%), electronic workers (11%), and hospitality workers (9%).The regional demand for catering and hospitality staff is high but unstable in terms of duration and type of contract; the need for electronic workers is limited and unstable.80% of people who qualify continue their studies. Of the others, 47% find employment within 12 months.
Administrative data do not contain information on self-employment.
Study of the effects of Covid on the shipping sector in regions covered by the Infrastructure and Networks OP, 2014-2020, in Italy
Gli effetti della pandemia Covid 19 sul contesto di attuazione del PON: il settore marittimo
NOP Infrastructure and Networks 2014-2020
https://www.ecorys.com/italy/, https://green.unibocconi.eu/
The study examines the effects of Covid on the economic context of the sector, at the global and EU level, and more specifically in a few Italian southern regions.
To assess the impact of Covid on the shipping sector and to identify possible scenarios and suggestions for the future development of the sector.
Analysis of official statistics.Literature analysis.Delphi method (surveys of sectoral experts, in two rounds).
The pandemic led to a reduction in global maritime trade of over 3% in 2020 compared to 2019, leading to lower employment and worsened job quality, partly because of the restrictions and controls introduced.Maritime traffic has fallen more in Europe than in other areas of the world. Ports in the south of Italy seem to have been more resilient than those in the north.Measures to adapt to Covid have focussed on prioritising activities for shipping essential goods, and for tackling the pandemic.According to experts surveyed, shortening of global value chains will continue in the future, and this might strengthen the role of Mediterranean ports. Connection with rail transportation is also likely to become increasingly important.
Strengthen network connections to avoid bottlenecks in the supply chain.Support the creation of integrated logistic areas and increase cooperation between ports, institutions and operators.Improve the communication strategy to better manage the perception of risk.
Evaluation of the physical progress and the results of the Infrastructure and Environment OP, 2014-2020, in Poland
Ewaluacja podsumowująca postęp rzeczowy i rezultaty Programu Operacyjnego Infrastruktura i Środowisko (POIiŚ) 2014-2020 z elementami przeglądu celów końcowych ram wykonania w zakresie osi POIiŚ 2014-2020 wdrażanych przez Centrum Unijnych Projektów Transportowych
The evaluation examines the physical progress and results of the OP up to 2022, as well as the performance framework.
To assess the impact of transport projects on Europe 2020 objectives and the socio-economic environment and the added-value for Poland and the EU, and to identify pandemic and war-related effects on implementation.
Analysis of monitoring and official statistical data.10 case studies.CIE using SPSM.Panel of experts.
The unstable geopolitical situation adversely affects project implementation.In most cases the strategic result indicator targets were overestimated.Transport Blue Books ensure consistency and comparability of cost-benefit analysis of projects.Transport investments have a significant impact on the development of the transport network, especially infrastructure of national importance.The horizontal indicator for project environmental effects does not reflect results of low-emission public transport measures in particular cities.
Impact assessment is limited as only some of the projects were completed at the time of the evaluation.
Ensure the ongoing monitoring of projects to try to extend the eligibility period, increasing the allocation available and the amount of eligible costs.Verify the targets for strategic result indicators in the programme documents.Carry out an impact assessment at least one year after the end of the period to evaluate projects that have already been completed.Recommend the use of the CBA methodology developed for all publicly-funded projects.Estimate CO2 emissions on the basis of data from EU-funded public transport projects.
Evaluation of the Centro ESF-ERDF OP, 2014-2020
Avaliação Intercalar do Programa Operacional Regional do Centro
CEDRU - Centro de Estudos e Desenvolvimento Regional e Urbanowww.cedru.comgeral@cedru.com
The evaluation examines the implementation and effects of the OP and covers the period between 2015 and 2021.
To assess the relevance, coherence, effectiveness, efficiency and EU-added value of the OP and to identify its potential or actual impact.
Analysis of monitoring data.Literature review.Analysis of socio-economic official statistics.Previous thematic evaluations.Interviews with 9 key stakeholders.4 thematic focus groups with 7 to 10 participants.4 thematic case studies on employment, sustainable urban mobility and urban rehabilitation.One survey with business project holders (633 respondents, response rate: 23%).One survey with other project holders (144 respondents, response rate: 62%).
COVID-19, a lack of predictability and regularity of call for tenders, and lengthy application and payment processes hindered the implementation of the projects.81% of non-business project holders and 57% of business project holders reported that they would not have carried out their projects without the support.The OP contributed to improving primary and secondary education school facilities and promoting educational success and equal access to education.In the area of vocational training, the support met the needs of beneficiaries with the exception of Continuing Education and Training courses because new and higher professional technical programmes (CTeSP) led to a shift in demand between the two.Measures supporting micro-entrepreneurship met the needs of beneficiaries.Support for hiring highly qualified human resources led to positive changes in the behaviour of companies.The integrated urban regeneration measures improved the attractiveness of regional urban centres.
The limited share of completed operations (15%) was an obstacle to assess the impact of the OP.
Intensify communication activities to reach a full absorption of the funds and improve the predictability of call for tenders targeting companies.Maintain the support for the recruitment of highly qualified human resources and for micro-entrepreneurship.Promote financial instruments and facilitate their use.Maintain the investment in health, education and social equipment.Adapt investments in urban development to take into account of the size of the urban centres.Strengthen the link between preserving natural heritage and developing tourism.
Preparatory study on transnational cooperation in North West Europe post-2020
Transnational cooperation in North West Europe post 2020
Interreg VI-B North West Europe 2021-2027
Sebastian Hans, Kai Böhme, Frank Holstein
Spatial Foresight, www.spatialforesight.eu
The report investigates the most relevant policy areas for the North West Europe Interreg OP in the 2021-2027 programming period.
To identify the main areas in each policy objective (PO) which are relevant for the 2021-2027 programming period.
Desk analysis.Analysis of monitoring and statistical data.
For the ‘Smarter Europe’ PO, areas to support are on industrial production, smart specialisation and digitalisation as well as the circular economy and 4th industrial revolution.For the ‘Greener, low-carbon Europe’ PO, interventions might focus on river systems, air and water pollution, combating the loss of biodiversity, renewable energy and climate change adaptation. Possible core cooperation covers the Moselle and the Rhine areas.For the ‘More connected Europe’ PO, the objective of cooperation concerns the TEN-T corridor between the North Sea and the Mediterranean and the parallel Rhine-Alpine transport route.For the ‘More social Europe’ PO, cooperation should pay attention to the different language regions. It may be worth considering how to strengthen cooperation between Ireland and continental Europe after Brexit.
Evaluation of the Out of Work Service scheme under the Wales ESF OPs 2014-2020 – Final Report
Priya Shah, Richard Lloyd and Stuart Harries
ICF Consulting Services, www.icf.com; Arad Research, post@arad.wales
The evaluation is a follow-up to the interim report on the Out of Work Service scheme, examining progress up to October 2019 and reporting the perceived impact.
To assess the progress, effectiveness, and perceived impact of the Out of Work Service scheme and the OP’s contribution to the cross-cutting themes.
Desk research;case studies;analysis of monitoring and ESF Leaver’s Survey data;interviews with stakeholders (76 in the first round and 55 in the second);workshop with programme stakeholders and Welsh Government representatives.
The scheme did not change, except for employment of new peer mentors and changes in their responsibilities, but the turnover of peer mentors remains high.The participation rate in the scheme up to October 2019 was 81% of the target.At exit, 83% of the target number of participants obtained a work-related certificate, 99% of the target were actively looking for work, only 28% of the target went into work experience or volunteering, 27% of the target entered training or education and only 9% of all participants found a job.Participants are generally satisfied with the scheme, reporting increased confidence, self-esteem and being more ready to work.The main perceived effects of the scheme for participants include progress towards working and for peer mentors, gaining skills, experience and confidence.The scheme also helped the pursuit of horizontal objectives, such as tackling social exclusion, promoting equal opportunities and furthering sustainable development.
At the time of the evaluation, the scheme was still ongoing, which limits the results. In addition, due to the broad coverage of the scheme, it is not possible to identify a control group for purposes of a counterfactual analysis. There were also difficulties in contacting project participants and employers.
Incentivise peer mentors to stay in order to prevent disruption of the service.Increase the number of peer mentors.Ensure a standardised approach to peer mentor training and support.Ensure employer engagement in the service and increase the opportunities for participants to gain paid work experience.Review in-work support.Re-define and extend the targets for the scheme.Ensure sufficient resources to collect participant data, especially 6 months after exit.Increase awareness of the scheme.
Evaluation of measures on cultural heritage and cultural and creative systems financed under several OPs, 2007-2013 and 2014-2020, in Southern Italy
Esiti degli investimenti della politica di coesione sul patrimonio culturale e i sistemi culturali e creativi nel Mezzogiorno
OP Campania ESF 2007-2013; ROP Calabria ESF 2007-2013; ROP Sicilia ESF 2007-2013; ROP Basilicata ESF 2007-2013; OP Puglia ESF 2007-2013; ROP Abruzzo ESF 2007-2013; ROP Sardegna ESF 2007-2013; ROP Calabria ERDF 2007 - 2013; ROP Campania ERDF 2007-2013; ROP
NUVAP – DPCoe, segreteria.politichecoesione@governo.it
The evaluation examines measures on cultural heritage and cultural systems under several OPs in the 2007-2013 and 2014-2020 programming periods. Data used goes up to end-June 2023.
To assess how the interventions for the protection and enhancement of cultural heritage affect: a) the management and use of cultural assets, b) the local economic development, and c) innovation and collaboration processes among businesses and cultural operators.
8 territorial case studies concerning 45 projects. Each case study includes 10 interviews approx., mostly with institutions. Analysis of official statistical data. Analysis of monitoring data (from “Opencoesione”). Literature review of previous evaluations. Analysis of programming documents.
Investments in the cultural and creative sector have led to 60% of the museums supported implementing virtual tours, 70% of cultural infrastructures improving physical accessibility, and 50% digitising their collections. The interventions have created approximately 10,000 new jobs and increased the number of creative enterprises by 15%. 500 new cultural businesses started because of the funding too. Additionally, the capacities of cultural institutions have been strengthened and more than 200 collaborations have been established between cultural institutions and local actors. Many projects neglected the long-term sustainability of cultural infrastructures, and some institutions did not develop the necessary capacities to effectively manage the cultural heritage. The expected growth in cultural demand did not materialise, given that measures aimed at developing demand were marginal. The lack of integration between museums and cultural sites limited the effectiveness of investments, and the long timelines of investments and bureaucratic complexity further slowed project implementation.
Foster collaboration across different sectors (public, private, non-profit) to enable sustainable cultural development. Develop local capacity building initiatives to activate synergies in less responsive territories and avoid top-down approaches. Explore and adopt innovative management models such as Public-Private Special Partnerships (PPSP) to move beyond traditional concessionary approaches to public spaces. Support social participation in cultural activities and cultural welfare.
Impact of the Human Capital ESF OP, 2007-2013 on improving qualifications of human resources through lifelong learning in Dolnośląskie
Wpływ działań PO KL w województwie dolNośląskim na podniesienie kompetencji regionalnych zasobów ludzkich, w kontekście upowszechniania i wdrażania idei kształcenia ustawicznego
Ul. Ozimska 4/7,www.openfield.pl, Email:office@openfield.pl, Tel: 77 555 91 88
The evaluation examines selected activities undertaken under the Human Capital OP 2007-2013 aimed at raising the skills of the regional work force through lifelong learning.
To assess: the usefulness, quality and effectiveness of selected measures undertaken under the OP, their impact on reducing disparities in the region among adults in undertaking formal and informal education, and the effect of regional education advisory services on the spread of ​​lifelong learning.
Desk research of programming documents, public statistics, evaluation reports and monitoring data;CATI interviews with projects participants, schools/education institutions and employers (n=614);CAWI interviews with beneficiaries (n=60);IDI with institutions involved in the implementation, beneficiaries, social partners (n=20);focus group interviews (7) with projects participants and schools/education institutions;5 case studies;experts panel;SWOT analysis.
A widespread belief in the value of lifelong learning (98% of respondents) does not translate into people expanding their knowledge and skills.For 75% of participants, the support received was complete and comprehensive.It was most difficult to recruit people with disabilities and those living in rural areas.Projects providing advice on education had the lowest cost per participant.Projects do not seem to have reduced differences in access to lifelong learning between urban and rural areas.Employers do not know about the potential availability of education for employees in the region.
Implement measures of support in rural areas with limited availability of lifelong learning.Provide support for particular groups of young people to enter the labour market, as well as more effectively disseminating the idea of ​​lifelong learning.Adapt the education provided to the needs of the regional labour market.Provide individualised support better adapted to the needs of target groups.Provide additional support for those in a difficult situation to help them overcome barriers to participating in the project.
Evaluation of the implementation risks of the Just Development Transition Programme financed under the JTF, 2021–2027, in Greece
Έκθεση ανάλυσης και εκτίμησης κινδύνων εφαρμογής του Προγράμματος ‘Δίκαιη Αναπτυξιακή Μετάβαση 2021-2027’ και σχέδιο δράσης για την επικαιροποίηση της στρατηγικής και την αναπροσαρμογή του μίγματος των δράσεών του
This evaluation examines the implementation of the programme between 2021 and 2024.
To compile all available documentation to map the progress of the implementation of the programme and propose adjustments to the programme’s indicator system.
Analysis of monitoring data.Document analysis (ex. National Energy and Climate Plan-NECP, the Programme’s Mid-Term review…).
The financial progress of the programme shows a relatively satisfactory mobilisation of resources (64%), but there is no progress in the activation and implementation of Integrated Small-Scale Interventions (SSM).The intervention logic is in line with the recommendations of the European Semester, the revised NECP, the European Pillar of Social Rights and subsequent documents. Therefore, strategic targeting should be maintained.New challenges and needs have emerged over the last 3 years, e.g. the need to strengthen the entrepreneurship sector.
Create a distinct STEP priority to support investments in strategic sectors and align the Programme with the new challenges.Reallocate resources based on the new challenges and needs of certain intervention areas.Adapt the programme’s system of indicators to better reflect the intervention logic and/or the redesign of individual measures.
Evaluation of RIS3 under the Basilicata ERDF OP 2014-2020
Valutazione In itinere della Strategia Regionale per l’Innovazione e la Specializzazione Intelligente 2014-2020
The evaluations examines the implementation of interventions under the Smart specialisation strategy (RIS3) supported under the OP.
To assess the implementation of interventions to support RISS3 in the region.
Desk analysis.Survey of 5 thematic clusters.Analysis of monitoring data.
As of January 2022, 70% of funding for RIS3 was allocated (out of EUR 239 million) and 42% had been paid.Five clusters were created in five thematic areas (aerospace, automotive, bio-economy, energy and the creative and cultural industry) involving 200 enterprises and 20 research institutes.The RIS3 strategy and the associated calls have allowed clusters to organise their activities well, present well-structured research and innovation projects and create networks.Relationships between enterprises and research institutes have improved but exchange of knowledge is limited.Aspects which have not developed adequately are the ability to patent. the creation of innovative start-up and intensive knowledge enterprises.
Speed up and simplify procedures.Diversify the types of support to innovation based on the size of enterprises.Support the creation of knowledge intensive enterprises through specific measures.Support the five clusters in the next programming period.
Evaluation of State aid under Measure 1.3 of the Smart Growth OP, 2014-2020, in Poland
Ewaluacja pomocy publicznej udzielanej za pośrednictwem NCBR w zakresie pomocy udzielonej w ramach działania 1.3 POIR
Michał Przybyłowski; Piotr Tamowicz; Anna Zamojska
Taylor Economics SP Z O O, ul. Trzy Lipy 3, 80-172 Gdańsk, office@tayloreconomics.com
The study examines state aid for R&D from financial instruments (equities and quasi-equities) and grants under the Smart Growth OP.
To assess the direct and indirect effects of the support provided to R&D.
Desk research of programming documents and evaluation reports.Analysis of monitoring data and official statistics.CAWI/CAPI survey of beneficiaries and final recipients.Interviews with experts, implementing bodies and beneficiaries.Econometric analysis.
The programme has led to the equity gap being limited in terms of the investment implemented.The risk finance programme helped to increase the number of VC funds operating significantly.The intervention did not cause a negative crowding out effect in the private capital market.The large number of first-time-management teams with limited investment experience has been major weakness.Increasing the effectiveness and efficiency of the programme requires reducing the distance between the organisational culture of the administration and that of Venture Capital funds.
Extend the investment period until 2023 in the case of VCs that have started investing but have been operating only for a short time.Terminate agreements with funds that have not invested and allocate the funding released to a new call.Systematically check whether there is an R&D component in the projects submitted to Investment Committees.Launch a programme for building investment readiness in research units.
Evaluation of interventions to reduce the digital divide and to develop broadband under PA6 of the Trento ERDF OP, 2014-2020
Rapporto di Valutazione degli esiti degli interventi per la riduzione dei divari digitali e la diffusione della connettività della banda larga, nell’ambito dell’Asse 6
T33 - www.t33.it - info@t33.it; IZI - https://www.izi.it/ - mail@izi.it
The report examines the reduction in the digital divide and the spread of broadband from measures under PA6. Monitoring data used go up to 19 November 2021.
To assess the main results of measures and the need of schools for digital infrastructure and applications, and to draw policy recommendations for the future.
Documentation analysis.Analysis of monitoring and statistical data.Interviews with representatives of the Province authorities.Five case studies of schools.
Projects have absorbed funding mainly as a result of reprogramming the PA to reduce the financing allocated.16 projects were concluded as at November 2021 and 16 schools have a broadband connection.Most of the procedural activities have been delayed because of bureaucracy, infrastructure obstacles, time constraints due to school activities and the pandemic.New education activities have been made available by broadband and traditional activities are being performed better.Carrying out administrative and managerial activities is easier because of broadband.
Measures have been delayed and many are yet to produce results.
Complete the installation of broadband in schools.Keep investing in hardware and software for the integrated digitalisation of academic activities.Facilitate the transition of administrative and academic activities to the cloud.Support schools by training staff.
Evaluation of the impact of EU funds on culture in 2007-2013
2007–2013 metų Europos Sąjungos struktūrinės paramos poveikio kultūrai vertinimas
This is the final report of the impact evaluation of EU support to culture in Lithuania.
To assess the impact of the EU 2007-2013 Structural Funds on the quality of cultural activities and accessibility to them.
Analysis of monitoring data;interviews;case studies on support to culture in other EU Member States;theory-based analysis;survey of project managers and other stakeholders;analysis of cost efficiency.
389 culture-related projects were funded by EU Structural Funds.Public expenditure on culture amounted to 0.6% of GDP, a third financed by the EU.The quality and diversity of cultural activities were considerably improved, in part because of EU support for investment in infrastructure and equipment.Indicators show that all cultural activities were perceived by people as being more accessible.EU support contributed to improving governance and administrative capacity, particularly in the Ministry of Culture.The improvement in the availability of cultural activities had a limited impact on social cohesion.
A number of limitations concerning the availability of monitoring and other data are reported and the reliability of some indicators is questioned.
There is a need to:Further improve the quality and quantity of statistical data on cultural activities and infrastructure;give priority to cultural infrastructure on which renovation has begun but is still to be completed;continue providing training seminars to project applicants;establish cultural activities before cultural infrastructure projects are approved.
The concept of added value applied to European financing and especially to the ESF - Methodological note
Le concept de valeur ajoutée appliqué aux financements européens et particulièrement au FSE Note d'analyse théorique
Amnyos Edater avec Marjorie Jouen et Eulalia Rubio, Institut Jacques Delors et Yusuf Kocoglu, Université de Toulon
Preparatory study for the Thüringen ERDF Programme, 2021-2027
Sozioökonomische Entwicklung und SWOT-Analyse in Vorbereitung des OP 2021-2027
Davis Adedayo Eisape, Carla Harnischfeger, Johanna Washington
Ramboll Management Consulting GmbH, www.ramboll.com
The evaluation examines the main challenges for 2021-2027 in the region.
To support the preparation of the 2021-2027 OP.
There is still a need to support capacity building in research and innovation, for example in respect of cooperation between SMEs and research centres and improving the competences of workers as regards the new digital technologies.In recent years, business competitiveness has strengthened, but enterprises still need to become more international.The region performs well in respect of energy consumption, efficiency and renewables, but improvements are needed in transport, especially on CO2 emissions, and in environmental protection and adaptation to climate change.The decentralised urban structure is under pressure because of inward migration and declining population.
Invest in R&D, new smart technologies and expanding innovation capacity.Promote business investment.Invest more in emission-free vehicles.Create or restore near-natural habitat.Identify risk prevention measures to improve flood protection.Implement urban development strategies and use redeveloped areas for green spaces, social housing, and economic and communal activities.
Evaluation of the contribution of PAs 3 "Territorial intelligence", 4 "Transition towards a low carbon Wallonie" and 5 "Integrated urban development" of the Wallonie ERDF OP 2014-2020
Évaluation de la contribution des axes 3 « Intelligence territoriale » , 4 « Transition vers une Wallonie bas carbone » et 5 « Développement urbain intégré » à la réalisation des objectifs fixés dans le PO FEDER 2014 2020 « Wallonie 2020.EU »
The evaluation assesses the implementation of the PAs.
To assess the effectiveness, relevance, coherence and efficiency of the PAs up to end-2020.
Desk review10 pilot interviewsAnalysis of monitoring dataSurvey of 379 beneficiaries (response rate, 43%)Interviews with stakeholders and regional representatives6 case studies.
PAs 3, 4 and 5 have supported 116 projects distributed among 51 direct beneficiaries (mostly public sector) for a total budget of EUR 631 m, EUR 248 m from the ERDF.At end- 2020, projects were still in the implementation phase and many intermediate targets set had not been achieved but they are expected to be by 2023.Interventions are relevant for the region.Internal coherence of PAs is satisfactory, though some measures under different PAs overlap, and measures are complementary to existing local and regional schemes, particularly for brownfield redevelopment, multi-modality and environmental and energy transition projects, but less so for tourism and roads.Barriers include regulatory constraints; lack of consultation with partners and users on road, multi-modal and environmental projects; difficulties in regulatory and administrative monitoring; and the Covid emergency.Specific difficulties include the timing of clean-up and development of brownfield sites; mobilising co-financing for tourism projects; lack of expertise and training of stakeholders in environmental projects; lack of consultation with project partners and users on roads i; heavy and lengthy procedures, lack of legal expertise for renewable energy infrastructure and energy efficiency projects; and lack of knowledge of SMEs of energy transition opportunities and their problems of accessing public financing.Cooperation and exchange of experience have been out in place to overcome difficulties.The ERDF amount remains insufficient to meet all needs of projects and the budget initially planned was reduced for all those interviewed.Projects are part of a long-term territorial development perspective which should ensure sustainability.Without ERDF support, projects could not have been carried out on the same scale and time-frame.
Strengthen the project portfolio approach.Target more fields of intervention.Split the timing of the clean-up and development of brownfield sites.Select more mature projects to enable faster implementation.Adopt a more flexible financial approach and ensure that the budget made available corresponds to the expected results.Select indicators that are easy for beneficiaries to understand.
Evaluation of measures to develop the regional labour market under the Śląskie OP, 2014-2020
Ewaluacja sposobu, w jaki wsparcie w ramach Regionalnego Programu Operacyjnego Województwa Śląskiego na lata 2014- 2020 przyczyniło się do osiągnięcia celów w ramach osi priorytetowej VII Regionalny rynek pracy
Head of the survey - Agnieszka Śnieżek, Michał Marciniak, Marta Cichowicz-Major, Adam Grajek, Magdalena Głomska, Paulina Kowalczyk, Aleksandra Kamińska, Cezary Przybył, Paweł Penszko, Beata Tomczak, Małgorzata Zub, Sebastian Pałka, Zofia Bieńkowska
The evaluation examines the results achieved by labour market measures under the Śląskie OP and the impact of support for start-ups.
To assess the effects of support under PA7 (regional labour market) of the Śląskie ESF OP and the possibility of achieving the planned level of implementation and to design labour market support for 2021-2027.
Desk research.Interviews with MA, IB and labour market experts.Interviews with beneficiaries and project participants.Survey of beneficiaries, project participants and those that established a business without grants.
The regional labour market is characterised by an industrial monoculture and significant differences between areas.There is a clear focus on easier-to-reach target groups and limited efforts to reach the inactive, low-skilled, older people and those with disabilities.There is a risk of not achieving the target for the number of workers and low skilled supported.Activities on the labour market implemented by local action groups have been successful, which is important for the future implementation of CLLD.41 thousand miners are expected to be laid off because of policy on climate change.COVID-19 has adversely affected many enterprises and has led to a deterioration of the situation on the labour market.It is estimated that without support 45% start-ups would not have been established.Support is estimated to have had a positive effect on the survival rate during first three years of operating but no effect on employment.
Provide training for young and older unemployed for occupations with an identified need.Support restructuration of traditional industries and the development of new sectors.Develop ways of retraining miners.Continue the outplacement activities and standard labour market policy for those threatened by lay-off as a result of Covid-19.
Evaluation of the “ReImpuls” project for employing people distant from the labour market in Salzburg, under the Employment Austria OP, 2014-2020
Evaluierung der ESF-Umsetzung für Salzburg. Projektbericht "ReImpuls"
The evaluation examines the effects up to the end of 2019 of the measure to help people distant from the labour market into employment in Salzburg.
To assess the effectiveness, efficiency and impact of the measure, the contribution to equality and non-discrimination and the possibility of projects continuing.
Monitoring data.Administrative data from PES.Interview with project management.Focus group with project staff.Project documentation.
The employment rate of participants two years after increased only slightly as compared to before, as did their average earnings.However, most participants were not in employment up to one year after.The project helped improve the personal development of participants and resolve health and social problems, so increasing their self-esteem.Main success factors are: the individual and holistic approach of the projects, a combination of individual and group activities, flexibility of objectives and activities, cooperation within the project team and the heterogeneity of participants, which stimulate mutual understanding and reduce prejudices.Critical aspects are: difficulties in addressing basic education needs, high travel costs which limit participation, lack of post-project opportunities.
The sample analysed after completing the measure was too small to capture the actual effects of the project. In addition, only the gross impact is investigated, because it is not possible to establish a causal link with the project.
Extend the project’s duration, including on the basis of participant needs.Optimise equipment and financial resources for extra-project activities.Broaden the definition of success for these kinds of project, taking account of factors other than employment, such as health and social conditions.Widen the target group, including, for example, people receiving rehabilitation benefits.
Impact evaluation of the Belgium-Netherlands Interreg OP, 2014-2020
Impactevaluatie Interreg V Vlaanderen-Nederland
Interreg V-A-Belgium-The Netherlands (Vlaanderen-Nederland) 2014-2020
Idea Consult; Jozef II-straat 40 B1; 1000 Brussel; +32 (0)2 300 85 02; Steven.knotter@ideaconsult.be; www.ideaconsult.be
The evaluation examines the impact of the approved and ongoing projects of the Belgium-Netherlands Interreg OP, 2014-2020.
To analyse so far as possible the current impact of the ongoing investments.
a theoretical model is used to provide a deeper understanding of how the interventions/projects create the expected impact;desk research of documentation;analysis of monitoring data;interviews with project promoters, focus group with beneficiaries, review by internal experts;7 case studies.
The programme strategy complements the focus on economic valorisation of (technological) research.The high density of high-quality research and knowledge institutions and associated facilities and resources in the border region allows complementary collaboration between them.The OP supports a relatively small number of projects, but each relatively large in financial terms, so that they can generate the critical mass and scope to make a difference in structural terms and contribute directly to the results sought.Several successful projects explicitly link technological feasibility with socio-economic feasibility, so that market potential is taken into account in their development.Public R&D expenditure has increased significantly and is already above the target for 2023.The share of SMEs introducing innovations has increased slightly relative to the baseline, but it is still considerably below the target.The substantial increase in raw material productivity indicates that the border region is moving towards a more resource-efficient economy, but the direct contribution of projects to energy and resource efficiency is limited.Nature management, environmental protection and an “inclusive” cross-border labour market have received less emphasis in this OP.
As the OP and almost all the projects are still in progress, it is not possible to assess final results and the overall impact.
Promote and facilitate the involvement of companies in projects.Promote and strengthen structural partnerships/networks, by supporting targeted cooperation projects, which contribute to the joint strategic ambitions and objectivesGive a stronger focus to socio-technical transition as regards societal challenges.
Evaluation of social centres financed by the Attiki ESF OP, 2014-2020
Αξιολόγηση Κοινωνικών Δομών που χρηματοδοτούνται από το ΠΕΠ Αττικής
Solonos 45, Kolonaki, 10672, Athens, Tel: 210 3007240, 22310 20000, https://e-ron.gr/, info@e-ron.gr
The evaluation examines the interventions to support social centres (Community Centres, Basic service centres, Day-care centres for the Elderly, Centres for Day-care for people with disabilities, Homeless centres) under TO9 of the OP up to December 2019.
To assess the relevance, coherence, achievements and efficiency of the social centres financed by the OP.
Desk analysis of relevant documents.Analysis of monitoring data.Surveys and focus groups with the coordinators of the Community Centres, the heads of the other centres and MA Attica executives, the Directorate of Social Welfare and the Regional Observatory of Social Inclusion.Analysis of data from the Regional Social Inclusion Observatory and Attica Social Network.
The social centres financed by the OP are coherent with the main national and regional strategies for social inclusion, such as the National Strategy for Social Inclusion and the National Roma Strategy.Projects are generally relevant to the needs of beneficiaries and have helped improve their lives, though most of the centres lack administrative staff and specialists to provide all the services required.The COVID-19 pandemic highlighted the presence of social centres, especially Community Centres, as these provided services to a larger number of beneficiaries than planned.The projects are generally progressing well, and output indicator targets have already been met or are likely to be met.On average, 65% of the funding available had been absorbed by end-2019. The unit cost per beneficiary is for all the centres lower than projected.
The Information System used to collect data on beneficiaries has several technical issues and delays, and does not allow data to be integrated with those of other centres.
Implementation Evaluation of the PEACE IV and Interreg Programmes 2014-2020
First Implementation Evaluation of the PEACE IV Programme and INTERREG VA Programme
Interreg V-A-United Kingdom-Ireland (Ireland-Northern Ireland-Scotland) 2014-2020; Ireland-United Kingdom (PEACE) 2014-2020
This is the first interim evaluation of the Northern Ireland, Ireland and Western Scotland Interreg OP and the EU Programme for Peace and Reconciliation (PEACE IV Programme), both managed by the Special EU Programming Body (SEUPB).
To assess the extent to which the two programmes have been implemented effectively and efficiently.
Desk analysis of the programming and implementation documents;statistical analysis of administrative and monitoring data;17 interviews with Key stakeholders;online survey of successful applicants (81% of all Lead Partners of the two programmes);online survey of unsuccessful applicants (49% of total);16 focus groups and interviews with project beneficiaries;10 interviews with unsuccessful applicants;4 case studies.
In May 2017, the Interreg OP was at a more advanced implementation stage with 74% of the budget allocated as against 26% for the PEACE Programme. The electronic monitoring system is central to the management of the two programmes. The delay in its operation delayed the formal approval of both.The pre-application support provided by SEUPB is effective as demonstrated by the low rate of ineligible proposals received. A 24-month plan describing the calls for applications and the related administrative procedures, the expected results and the resources allocated helped in this regard.The simplification measures introduced in the 2014-2020 period have led to significant improvements in the assessment and selection procedures for projects. But beneficiaries complained about the continuous requests for updated information on the project being assessed, the difficulties and delays in collecting all the documentation necessary for submission and problems of uploading of information onto the eMS.Despite the MA encouraging the use of Simplified Cost Options (SCOs), they have not been applied as much as anticipated.The MA has developed a set of comprehensive support and training activities for project beneficiaries to help implement the programmes.At this early stage of implementation, the approved partnerships appear to be working effectively.
Expedite the allocation of funds and project expenditure to ensure that financial targets are achieved. Improve the functionality of the monitoring systems until the eMS database is fully operational, integrate it with other tools (e.g. Programmes’ website) and provide training to applicants and beneficiaries.Improve the content and delivery of the information workshops, to promote networking between potential partners.Simplify the project evaluation and selection procedures by further reducing the administrative burden for applicants.Provide additional training and support during the pre-application and project implementation phases.
First intermediate evaluation of the Bolzano ERDF Programme, 2021-2027
1° Rapporto di implementazione (riesame intermedio) del Programma FESR 2021-2027 Provincia Autonoma di Bolzano-Alto Adige
The evaluation aims to assess the implementation progress of four priorities of the OP: smart technologies, green transition, sustainable mobility and technical assistance. Monitoring data used go up to December 2024, while the online survey was launched in February 2025.
To assess effectiveness, efficiency, coherence, relevance, EU added value and visibility of the OP. To provide evidence for the mid-term review of the programme and to confirm the allocation of the ERDF flexibility amount.
Document analysis.Analysis of monitoring and statistical data.10 interviews, of which 6 with MAs and implementing authorities and 4 with beneficiaries.Online survey of all the beneficiaries of the programme with 76 answers (60% response rate).
The implementation of the interventions is mostly on track: the financial progress is good and indicators are moving toward the targets.The strategy remains relevant and coherent with EU and national priorities. ERDF support generates clear EU added value, as many projects would not go ahead without it.Main weaknesses concern the administrative burden, some implementation bottlenecks, and IT/simplification measures that still need refining.There is no reason to change the original internal allocation of the €14.8m ERDF flexibility reserve for 2026–2027.
Fine-tune some output indicators and targets, and consider reallocating resources released from downsized project.Maintain regular, structured stakeholder dialogue and implement a thematic study on communication within the ongoing evaluation.Do not introduce broader policy changes at this stage.Keep the current flexibility reserve of the ERDF.
Impact evaluation of the Luxembourg ESF OP, 2014-2020
Evaluation d’impact du Programme Opérationnel du FSE 2014-2020. Evaluation de fin de programme
Michela Bia, Franz Clement, Anne-Sophie Genevois, Blandine Lejealle
Luxembourg Institute of Socio-Economic Research (LISER); https://www.liser.lu/
The evaluation examines the impact up to June 2020 of measures for labour integration of jobseekers (IP 8i) and young people (IP 8ii), for active social inclusion (IP 9i), and for the acquisition of new skills (IP 10iii). Data up to June 2020.
To assess the impact up to June 2020 of measures for the integration into employment of woman and people aged 45 and over under IP 8i and the results of measures under IP 8ii for young people, under 9i for active inclusion and 10iii for the acquisition of new skills.
Counterfactual analysis of projects under IP 8i - PSM based on administrative data from the unemployment register (ADEM) and the social security register (IGSS).Online focus groups with project promoters.Online surveys of project promoters under IP 8.ii (11 respondents out of 16), IP 9.i (10 out of 12), IP 10.iii (12 out of 13).
8i: for women, the employment impact is negative and statistically significant 12 months after being registered as unemployed (8 pp less than for the control group); positive 24 months after (+1.6 pp) and positive and statistically significant after 36 months (+9.6 pp). For jobseekers aged 45+, the impact is negative and statistically significant 12 months after being registered as unemployed (-10.8 pp); it is positive after 24 months (+2.7 pp) and positive and statistically significant after 36 months (+13.2 pp).8ii: projects encouraged participants to identify unknown skills and to build self-confidence. Teamwork was an effective method.9i: projects made participants more aware of their skills and strengthened their self-confidence, especially in the case of migrants.10iii: participants were satisfied with training, which improved their skills and self-confidence.
Evaluation of the Communication Strategy of the Łódzkie ROP 2014-2020
Ewaluacja Strategii Komunikacji Regionalnego Programu Operacyjnego Województwa Łódzkiego na lata 2014-2020 oraz działań przewidzianych w Rocznym planie działań informacyjnych i promocyjnych na rok 2016 w ramach Regionalnego Programu Operacyjnego Województ
Wojciech Pieniążek, Cezary Przybył, Paweł Huras, Jarosław Chojecki, Sebastian Pałka, Magda Matysiak
EVALU Sp. z o.o. (former Agrotec Polska Sp. z o.o.) ul. Dzika 19/23 lok. 55 00-172 Warszawa
The evaluation focuses on the implementation and usefulness of the Communication Strategy of Regional Operational Programme of the Łódzkie Region for the years 2014-2020 as well as the promotional activities for the annual information and publicity plan in 2016.
Evaluation of the relevance and usefulness of the communication strategy of the ROP 2014-2020 and the identification of means of communication and ways of contacting the potential beneficiaries and regional target groups.Assessing whether objectives and targets in the 2016 annual information and publicity plan have been reached.
Desk analysis of regional data.PAPI, CATI, CAWI surveys.Interviews and focus groups.Analysis of user experience Analysis of language and means of communication used.
Analysis of resources at the disposal of the MA for implementing a satisfactory communication policy did not show any significant weaknesses.The implementation of the annual information and publicity action plan faces a significant organisational barrier in that the tender cycle makes it practically impossible to launch promotional activities at the beginning of the year.There is a significant knowledge gap in some groups of potential beneficiaries in sectors included in the intelligent specialisation strategy (especially the medical sector and the energy industry).The most effective channel of communication is directly between beneficiaries and fund managers.The means of informing the general public about EU funds are TV, Internet, press and radio.Residents do not associate benefits they obtain with EU Funds.
The main recommendations made are to:Expand the training provided to employees.Reduce the time limit for the adoption of the Annual plan for information and publicity and approve it by 15th November each year.Plan and conduct an information and promotional campaign addressing target groups with limited knowledge of European Funds.Provide the beneficiaries with a project manager for the whole period of project implementation.Because of the needs of the disabled, websites should be adjusted to comply with all WCAG 2.0 standards.Place more emphasis on the benefits going to people from EU Funds in the promotional communication.
Evaluation of funding action 14 ‘All-day offer for German classes’ under the Bayern ESF OP, 2014-2020
Evaluation der Förderaktion 14 „Gebundenes Ganztagsangebot für Deutschklassen“
Institut für Sozialforschung und Gesellschaftspolitik GmbH, www.isg-institut.de
The evaluation examines the all-day provision of German classes, targeted at children and young migrants with non-German mother tongue that have recently entered the Bavarian school system.
To define the target groups reached by the measure and the difficulties they face in making the transition to the Bavarian school system, the design of the measure, and achievements.
Monitoring data (up to July 1, 2021).Two surveys, of schools and principals (2016, 2019 and 2021 waves, response rates of 57% to 77%).Case studies (from the 2017 report).
As of July 1, 2021, the ESF approved for action 14 amounted to 59% of planned and 43% had been spent.11,200 participants had been registered by July 2021, and by 2023, the target is expected to be exceeded.Those pupils supported are 12 on average, 56% of them boys, nearly all of them migrants.The main difficulties faced by children and young people are that many of them have never been to school, and they are not familiar with the Latin alphabet.More participants than expected transitioned to a standard class or successfully obtained a school leaving certificate.During Covid-19, the drop-out rate did not increase and technical equipment was improved, but learning progress was reduced.Over half of schools surveyed attach great importance to working with parents, though a lack of German or parents not being used to participating in their children’s education pose challenges.The measure directly contributes to equal opportunity and non-discrimination objectives.
Because of external factors, it is difficult to draw a causal relationships between the intervention and results observed. No counterfactual impact assessment has been carried out. The funds planned for the 2021/22 and 2022/23 school years are not yet included in the monitoring data used in the evaluation. Social acceptance of migrants cannot be assessed, because the children supported in the German classes make up only a small proportion of those with a migration background in Bayern.
Maintain the funding period of one year and examine the possibility of extending support after transition to regular classes.Consider integrating therapeutic support in the measure.Extend the interaction between students in German classes and in regular ones.Develop proposals for administrative simplification, taking account of the funding requirements.Take account of the cross-cutting objectives in the German classes to a greater extent.
Evaluation of the selection system of the projects of the Digital Poland OP for 2014-2020
Ewaluacja systemu wyboru projektów w ramach Programu Operacyjnego Polska Cyfrowa na lata 2014-2020
Fundacja Idea Rozwoju - IMAPP - PAG Uniconsult - Uniwersytet Jagielloński
Fundacja Idea Rozwoju, Wierzbica 57b, 05-140 Serock, e-mail: , Tel: +48 506-067-751 - IMAPP sp. z o.o., u. Rejtana 15/25, 02-516 warszawa, Tel: +48 22 390 53 04, imapp@imapp.pl - Policy & Action Group Uniconsult Sp. z o.o. (PAG Uniconsult), ul. Kiebedzia
The evaluation concerns the project selection system of the Digital Poland Operation Programme 2014-2020 (OP DP).
The objective is to assess the project’s selection procedures for six measures of the OP DP: Measure 1.1. - Eliminating territorial differences in access to high-speed broadband Internet; Measure 2.1 - High availability and quality of public e-services; Sub measure 2.3.1 - Digital availability of public sector information from administrative and scientific sources; Sub measure 2.3.2 - Digital availability of cultural heritage sources; Measure 3.1 - Training activities for development of digital competences; Measure 3.2 - Innovative solutions for promoting digital activity.
The evaluation uses both qualitative and quantitative methods:Desk research.Analysis of OP documents, regulations and research studies.Analysis of monitoring data.Focus groups with authors of the OP to clarify the intervention logic.In-depth interviews with representatives of the institutions involved in the implementation of the OP.Survey (CAWI) of beneficiaries and unsuccessful applicants.In-depth interviews with beneficiaries and unsuccessful applicants.Expert panel.The data used are: from interviews, focus groups (including expert panel) and a survey and secondary data from research studies and the monitoring system.
Measure 1.1: the project selection system was burdensome for both applicants and implementing organisations.Measure 2.1: criteria for project assessment in the first call was sufficient to select the best and most important projects.Sub measure 2.3.1: the project selection procedures were functional though the criteria were demanding and a high rate of proposals were rejected.Sub measure 2.3.2: the criteria led to the most valuable projects being selected, but they caused some difficulties for applicants.Measure 3.1: different shortcoming were reported by applicants: municipalities concluded agreements with multiple applicants (though formally not allowed); defective software was used to submit applications; there were frequent and radical changes to the rules during the call.Measure 3.2: selection criteria were considered consistent, complete and appropriate for the types of project required.
Measure 1.1: the assessment system needs to be further simplified, public communication to be improved, additions and amendments to applications might be considered if formal errors are detected, and measures to standardise the assessment of proposals should be introduced.Measure 2.1: the introduction of a second expert to handle assessment appeals and the strengthening of training activities for applicants should be considered.Sub measure 2.3.1: the selection criteria should be modified, training for applicants strengthened and staff competences improved.Sub measure 2.3.2: the information surrounding the recruitment of applicants should be strengthened.Measure 3.1: municipalities need to be prevented from concluding agreements with multiple applicants, the application software needs to be improved and the competition rules clarified.Measure 3.2: the effectiveness and efficiency of the project selection system after the completion of the first assessment procedure in 2017 needs to be verified.
Evaluation of measures supporting green-blue infrastructure under the OP Territorial and Settlement Development 2014-2020 and the Programme Territorial and Settlement Development Plus 2021-2027, in Hungary
Zöld-kék infrastruktúrát támogató intézkedések értékelése
OP Territorial and settlement development 2014-2020; Territorial and Settlement Development Plus HU ERDF/ESF+ 2021-2027
MultiContact Consulting Kft., AQUA Engine Kft.
MultiContact Consulting Kft., 1115 Budapest, Bartók Béla út 105-113, https://multicontact.eu/hu/; AQUA Engine Kft. 2040 Budaörs, Szép utca 50-52. A. ép.
The report aims at evaluating green and blue infrastructural developments implemented in the 2014-2020 programming period, and at analysing the framework for investment starting in the period 2021-2027.
To assess the effectiveness and efficiency of projects involving green and blue infrastructure implemented in 2014-2020, and to examine projects funded up to now under the 2021-2027 programme, to identify lessons learned and make proposals for the upcoming calls on green-blue infrastructure.
Desk research on strategic and programme documents.Data analysis of official national statistics.4 interviews (2 with MAs and 2 with members of the Hungarian Chamber of Engineers).2 focus groups (one with local authorities, the other with designers).2 online surveys (targeting designers, with a 23% response rate, and project beneficiaries, with no response rate reported).
In most cases, renovations do not include the replacement of utilities, which cost a lot of money.When faced with cost increases, designers often reduce the green element of projects.Blue infrastructure projects require the advice of the Regional Water Management Council.As regards calls, project indicators and evaluation criteria are not standardised, making assessing proposals difficult.Climate resilience assessments have often been carried out retrospectively, undermining their usefulness for the planning process.Cooperation between construction firms and local governments in project implementation is still limited.The professional competence of MAs is often inadequate to appropriately oversee the implementation of projects.The public seems to prefer grey infrastructure (streets, parking spaces, etc.) to green, influencing the choice of local authorities.
Make project management a regional responsibility, as local administrative boundaries rarely follow natural boundaries.Revise existing technical standards to create a more flexible regulatory environment supporting the use of new materials and technologies.Encourage local authorities to acquire property to a greater extent because of the space requirements of green infrastructure projects.Create municipal environmental databases and maintain and update them.Promote cooperation between professionals from different Chambers.
Evaluation of the horizontal objectives in the Bund ESF OP, 2014-2020, in Germany
The evaluation examines the horizontal principles in the OP.
To assess the implementation of the horizontal principles by the OP and the measures supported.
Desk reviewAnalysis of monitoring dataEvidence from previous evaluations.
Overall, horizontal principles are recognised by the OP and linked to its implementation but are not adequately monitored.They are implemented in single programmes by separate departments.Project managers and beneficiaries have not received adequate methodological and technical guidance on how to implement horizontal principles.48% of participants in programmes funded by the OP are women, though 39% of disadvantaged young people, 43% of long-term unemployed, 44% of those with a migrant background in particularly difficult situations and 31% of business start-ups.Overall equality between women and men was achieved through programmes focused on women, while principles of equal opportunities and non-discrimination were mainly focused on those with a migrant background, older people and those with disabilities.The share of funding to support measures for inactive women, work-life balance and to increase the proportion of women in the social economy is small, and there is no specific support to increase the number of women in vocational education, particularly those with a migrant background and at risk of early school leaving.
It is difficult to measure quantitatively the impact of the OP on equality between women and men.
Prepare guidelines on how to implement horizontal principles at the project level and ensure adequate monitoring and evaluation of horizontal principles.Ensure more active governance of measures which support work-life balance.Link measure for supporting female entrepreneurship to the gender equality objective. Focus more on older people and those with disabilities.Strengthen the implementation of the ‘equality between women and men’ principle in all programmes for young people, particularly vocational training.
Evaluation of the investment for growth and employment under the Hessen ERDF OP, 2014-2020. Final Report 2021
Begleitende Evaluierung des operationellen Programms für die Förderung von Investitionen in Wachstum und Beschäftigung. Endbericht 2021
Dr. Björn Alecke, Prof. Dr. Gerhard Untiedt
GEFRA, Gesellschaft für Finanz- und Regionalanalysen (Untiedt & Alecke GbR), info@gefra-muenster.de - JOANNEUM RESEARCH Forschungsgesellschaft mbH POLICIES, prm@joanneum.at - Kovalis – Dr. Stefan Meyer, meyer@kovalis.de
The evaluation covers eight Action Lines (ML) supporting R&D, environment protection and SMEs, financed under the OP up to June 2022.
To evaluate the results and impact of the measures up to June 2022 and effectiveness and coherence and to draw lessons for 2021-2027.
Documentation and literature analysis.Analysis of monitoring data.Interviews with stakeholders.Surveys.Case studies.ToC elaboration and testing.
Projects on enterprise R&D have helped increase the technological knowledge of companies and their ability to generate new ideas and have increased turnover by 10-15%.Support strengthened collaboration between institutes and transfer programmes at universities and research centres.Support for regional R&D clusters has made limited progress, but it has contributed to the development of new products and services.Support for the ESA BIC incubator in Darmstadt has helped new competitive and sustainable start-ups to emerge.Financial progress of measures for the promotion of R&D, dissemination of innovative energy and environmental technologies, and environmental protection is good and projects seem to be successful.The electrification of the bus fleet has led to a reduction of CO2 emissions of 1,029 toe and to improved air quality and urban development in Offenbach.
For one measure, it was not possible to assess results for sales growth, cost reduction or job creation. For the transfer measure, impact could not be assessed since several projects were still ongoing. The incubator project is too small to produce measurable and direct effects on the result indicator of "Number of start-ups in Hesse".
Reduce the administrative burden, simplify procedures and better identify the needs of beneficiaries.Continue corporate R&D projects contributing to CO2 reduction in 2021-2027.
Evaluation of the communication strategy of the Social Inclusion and Employment OP, 2014-2020, in Portugal
Avaliação da Estratégia de Comunicação do Programa Operacional Inclusão Social e Emprego
António Manuel Figueiredo, Filipa Barreira, Mariana Rodrigues, Pedro Quintela, Pedro Guilherme, Américo Alves, Artur Costa
Evaluation of the ‘Citizens’ labour’ pilot project in the Bund ESF OP, 2007-2013
Evaluation der Modellprojekte "Bürgerarbeit". Endbericht
IAW: Dr. Tobias Brändle, Lukas Fervers, Andrea Kirchmann, Regina Weber, Prof. Dr. Bernhard Boockmann - ISG: Dr. Helmut Apel, Dennis EgeNolf, Dr. Michael Fertig, Dr. Martin Rosemann, Hans Verbeek, Marian Weimann
IAW Institut für Angewandte Wirtschaftsforschung e.V. an der Universität Tübingen, Ob dem Himmelreich 1, 72074 Tübingen - ISG Institut für Sozialforschung und Gesellschaftpolitik GmbH, Weinsbergstraße 190, 50825 Köln/ Gorgasring 2, 13599 Berlin
The evaluation is concerned with the pilot project ‘Citizens’ labour’ financed by the Federal budget (SGB-II integration budget) and ESF funds 2007-2013. This is the final report.
The objective of the evaluation is to assess the efficiency and impact of the pilot project, which consists of an ‘activation phase’ (6 months), including consulting and teaching activities, and an ‘employment phase’ (36 months), consisting of publicly funded work.
The evaluation adopted qualitative and quantitative methods:revision of the Desk research (revision of previous project interim reports);12 regional case studies (expert interviews and standardized surveys);standardised surveys of different participating and non-participating stakeholders (job centres, employee and beneficiaries);quasi-experimental analysis: propensity score matching (PSM);robustness analysis (alternative matching algorithms, placebo tests, instrumental variability estimates and difference-of-difference methods).The data used are primary data collected through standardised surveys of the participant and non-participants. PSM analysis is based on a group of randomly sampled beneficiaries out of 69,000 participants, whereas the 4 non-beneficiary groups are created from a random sample of four different non-participating groups of 125,000 each. Secondary data are also used, mainly the regional and individual datasets of the Federal Agency for employment, as well as the monitoring data from the ESF management system.
The impact assessment of the activation phase found that:the PSM estimate shows that beneficiary have an up to 35% higher integration rate compared to non-participants (a relatively large impact compared to the literature). The effect is delayed and is found to be at a maximum 180 days after the beginning of the activation phase (later the effect weakens and is reduced to zero after two years after the start of the activation phase);the positive effect is larger when other measures are added such as integration subsidies;the effects are partly heterogeneous. While there was no statistical difference between men and women, people between 25 and 49 seem to have an higher probability of integration than those older;the effects are heterogeneous according to the types of strategy applied by the job centres. This shows that the design and implementation of the activation phase can have an influence on outcomes. In particular, the creation of special teams has led to greater positive effects on integration into employment, whereas increased use of agents during the activation phase is more likely to have short-term effects.The impact assessment of the employment phase found that:the PSM estimate shows that, within the first two years, the integration rate would have been twice as high without participation in the employment phase;no significant difference was found between men and women or by type of strategy implemented by the job centres (e.g. coaching strategies);the negative effect can be explained partly by a locked-in effect (reduction in work search intensity by those involved in the project) and, particularly, by the fact that for public and civic jobs the employers had a possibility of choosing the person they considered the most appropriate, which is not always those receiving support through the ‘activation phase’ who are, by definition, problematic and difficult to integrate;the longer the employment lasts, the more pronounced the lock-in effect is. It is therefore necessary to consider how much the participants in the employment phase have benefited from a longer period of employment as regards other aspects such as employability or social participation and whether this justifies the significantly lower integration opportunities.With the regard to the efficiency analysis:due to the negative impact of the employability phase, the pilot project is not economically worthwhile;efficiency is even less than estimated as the cost-effectiveness analysis did take account of any additional expenditure of the municipalities and the Länder in the activation and employment phase as insufficient information was provided;the cost-effectiveness analysis did not take account of the economic effects of employability and social participation (potentially positive effects on the physical and mental health of beneficiaries and social impact of the work performed). This is because these elements are difficult to identify, to measure and to quantify in money terms.
Job centres participating in the project were disproportionately from the Eastern Länder, i.e. from regions with a weak labour market.Regarding the employment phase, no sensitivity analysis was performed regarding the effects that a different contract duration would have produced.
The main policy implications are:For the activation phase: focus on a particular target group (namely those between 25 and 49);focus on the types of strategy which are more successful has a positive (if not so strong) impact on the integration of participants. If the goal is medium- or long-term integration into the labour market ,specific teams exclusively dealing with activation and placement of participants should be considered in each of the job centresFor the employment phase: Only those without any other chance of integration should be admissible to the public or civic jobs in question. Therefore, a change in the framework conditions of the employment phase or in the definition of the target group is needed;to reduce the lock-in effect, the conditions attached to public sector jobs should be reconsidered, such as lower remuneration and a shorter duration time (three years was considered relatively long).
Impact evaluation of the EU funds on innovation and R&D activities in 2007-2013
Ewaluacja ex-post wpływu Funduszy Europejskich w ramach perspektywy finansowej 2007-2013 na podstawowe wskaźniki innowacyjności i działalności badawczo-rozwojowej
All OPs 2007-2013 except OP Technical Assistance 2007-2013
Marek Antosiewicz, Piotr Bartkiewicz, Jan Frankowski, Henryk Kalinowski, Magdalena Ośka, Andrzej Regulski, Jan Witajewski-Baltvilks
imapp sp. z o.o., ul. Bukowińska 22B, 02-703 Warszawa, imapp@imapp.pl, www.imapp.pl, Fundacja Naukowa Instytut Badań Strukturalnych, ul. Wiśniowa 40B, 02-516 Warszawa, ibs@ibs.org.pl, www.ibs.org.pl
The evaluation examines the innovation and R&D measures implemented under Cohesion policy in Poland in 2007-2013 (Innovative Economy OP; Human Capital OP; Infrastructure and Environment OP; Development of Eastern Poland OP and 16 regional OPs).
To estimate the impact of the measures supported according to selected innovation and R&D indicators and to assess the mechanisms involved.
The evaluation uses both qualitative and quantitative methods:review of documentation and the intervention logic;CAWI survey of beneficiaries (2,864);regression analysis;macroeconomic modelling (stochastic general equilibrium model - EUImpactMOD).The data used are primary data from the CAWI survey and secondary data from the monitoring system, the IEOP ‘Barometer of Innovation’, the Community Innovation Survey, the national Information System and Eurostat.
Estimates from regression analysis are that the measures implemented had the effect of increasing slightly the share of innovative enterprises, the share of sales of new and significantly improved products and the share of sales of high- and medium-high tech products in industry. It also led to a marginal increase in the share of high tech products in total of exports in 2007-2015 and expenditure on R&D and a larger increase in the number of R&D employees (from 1.5 thousand to 9.6 thousand in FTE terms);estimates from the macroeconomic model are that innovation measures led to an increase in GDP of around 0.3% in 2014 and of around 1% in the longer-term, while investment in innovative activities is estimated to be 4.5 percentage points higher.
Authors clearly point out the assumptions which limit the regression analysis and of the results of the macroeconomic modelling exercise.
Indicators measuring the economic effects of innovations should be monitored (e.g. gross value added per employee, total factor productivity);measures should be introduced to support companies with limited experience in innovation and R&D activities so as to increase the number of innovative enterprises in Poland.
Monitoring and analysis of TO1 Research, Development and Innovation
Evaluaciones de seguimiento y analisis sectorial por objetivo tematico SADOT. Investigación, desarrollo einnovación (I+D+i)
This is the second report on monitoring and analysis of TO1 and covers all the Spanish ERDF OPs investing in RDI (1 national OP and 18 regional).
To examine the general situation of RDI in Spain in parallel with the implementation of OPs investing in TO1 and the relationship between ERDF expenditure and RDI developments.
Desk analysis of official statistics and monitoring data.
Spain is far from achieving Europe 2020 and national Strategy for Science, Technology and Innovation targets for the main RDI indicators.ERDF support of TO1 amounts to EUR 5,006 million, 25% of the total.6 OPs of the 19 OPs investing in TO1 did not meet the RDI performance framework objectives.The result indicator “Percentage of companies which made technological innovations” have very ambitious targets for 2023, which will not be met. The targets for the result indicator “Percentage of companies with technological innovations that cooperate with universities and research centres”, will be achieved only for some OPs. The indicator “Spanish participation in consortiums of international H2020 projects” shows a significant increase on the previous programming, with Spain in first place in leading projects.It is difficult to discern a direct relationship between the ERDF support and developments in RDI. The financial crisis and budgetary restrictions have had much more of an impact. Changes in RDI expenditure have not been much affected by ERDF support for RDI, which accounted for only 5% of expenditure in the period 2007-2017, though more in transition (11%) and less developed regions (17%) than in more developed ones (4%).
Contribution of ERDF and ESF to Innovation and Competitiveness under the Sachsen-Anhalt OP, 2014-2020 (integrated report)
Beitrag des EFRE und des ESF zu Innovation und Wettbewerbsfähigkeit - Übergreifender Bewertungbericht
Ramboll, Chilehaus C - Burchardstraße 13, 20095 Hamburg, Tel.: +49403020200, www.de.ramboll.com
The evaluation examines the ERDF and ESF-financed interventions for improving innovation and competitiveness, financed by the Sachsen-Anhalt OP.
To assess the relevance of interventions, their implementation and their results and to identify their contribution to both the Regional Innovation Strategy (RIS) and Europe 2020.
Analysis of monitoring data and official statistics.Analysis of relevant documents.Workshop.Case studies.Online surveys of companies and research centres.
IP 1.1: There are no particular difficulties in implementation and financial and output indicator targets should be reached in 2023.IP 1.2: The output indicator targets should be reached in 2023 and measures have mobilised private funding with little apparent deadweight.IP 3.4: Deadweight seems to be limited.IP 10.ii: Both financial and output indicator targets for 2023 should be reached. Those interviewed reported support has led to the development of skills and the possibility of taking part in international conferences.Overall, the interventions are consistent with the RIS3 objectives.
Mid-term evaluation of the Hamburg ERDF Programme, 2021-2027
Halbzeitüberprüfung des Programms „EFRE 2021-2027 Hamburg“
Dr. Jonathan Eberle, Holger Bornemann, Felix Ginzinger, Renke Janshen, Nikolaus Bajor, Vincent Vogelsang
This mid-term evaluation examines the implementation of the ERDF programme in Hamburg. The report examines the data up to the 31 December 2024.
Identify new challenges, which may lead to an amendment of the programme, and provide feedback for the final allocation of the flexibility amount for the years 2026 and 2027.
Socio-economic analysis of Hamburg using data from European, federal and regional statistical offices, the federal employment agency and other data sources.Analysis of administrative data.Analysis of monitoring data.Analysis of previous evaluations.Document review.
The programme is relevant to the socio-economic situation of Hamburg.The programme focuses to a large extent on decarbonisation and energy efficiency and is coherent with the integrated national energy and climate plan (NECP), as well as with the country-specific recommendations (CSRs) for Germany adopted in 2024.The majority of the planned measures seem to be able to achieve their objectives by the end of the programming period, based on their implementation thus far.For a minority of measures, implementation was not as expected, because of too high initial assumptions, limited demand or funding requirements not taken enough into account.
Reduce the target value of the output indicator “Number of enterprises supported by grants” for the measure “Sustainable Logistics”, as it was shown that demand exists in particular for larger projects and the initial target was overestimated.Reduce the target value of the output indicator “Number of enterprises benefitting from the newly created incubator capacity” for the measure “Transfermilieus”, as demand is limited.Reduce the target value for “Number of SMEs using newly created incubator capacity” as demand was overestimated during the planning phase, and these spaces can conversely be in demand by large companies.
Evaluation of the Development Funds under the Northern Ireland ERDF OP, 2014-2020
This evaluation assesses the Crescent Capital III Fund and Bank of Ireland Kernel Capital Fund, collectively referred to as the Development Funds.
To review the objectives of both Funds and the appropriateness of the underlying model, to compare the support provided with equivalent support available to businesses in the UK and to identify the factors affecting the performance of the Funds.
Logic model approach;desk analysis;consultation with Fund managers, firms and other stakeholders;benchmarking against comparable funds.
Both Funds employ a competitive tendering procedure and two Fund Managers and have an appropriate scale of funding;the Funds have performed effectively to date;they have helped to create between 17 and 55 net jobs and net turnover by between £2.3m and £6.3m;they have reduced net labour costs by an estimate of between £1.1m and £3.8m.
The evaluation is made at a relatively early stage, mainly covering 2014-2016;there were only a small number of beneficiaries to be assessed;the context (economic and political) has been changeable.
To link the Funds to the wider pipeline of prospective businesses in which to invest, such as those spin-off and scale-up firms;to publicise the Funds to prospective investors and situate them in the wider funding system.
Impact evaluation of measures to strengthen SMEs competitiveness, under the Corse ERDF-ESF OP, 2014-2020
Evaluations d’impact du programme opérationnel FEDER-FSE Corse 2014-2020 - Evaluation COMPETITIVITE (Pi03a, Pi03d)
The evaluation examines the measures supporting business creation, transfer and competitiveness. It covers the implementation period between 2014 and 2020.
To assess the effects of the support for SMEs competitiveness on beneficiaries. To assess the relevance, efficiency and added value of the measures and provide recommendations for the 2021-2027 programme.
Quantitative analysis of statistical and monitoring data.8 interviews with the MA and institutional partners.5 interviews with project holders.4 case studies based on 1 interview with the project holders.Online survey of beneficiaries of loans and bank guarantees (response rate: 37%).8 interviews with companies supported via a bank counter-guarantee.
The programming rate of the two IPs is very different: 100,3% for IP 3.4 (“Increasing companies’ size and exports”) and 37,9% for IP 3.1 (“Business creation and transfer”).In the area of business creation and transfer, the measures helped improve access to bank financing, especially for companies in need of equity capital, and support job creation with an expected increase of 1,090 jobs for all IP 3.1 projects.Measures on companies’ development (IP 3.4) were concentrated on the aeronautics sector (70% of the resources), a long-standing partner of ERDF measures in Corse. The projects helped structuring the sector and getting access to new markets.On the other hand, the support has been more limited on the tourism sector, with only two projects supported.
Evaluation of the Spain-Portugal Interreg OP, 2014-2020
Evaluación de la eficacia y eficiencia del Programa de Cooperación Interreg V-A España-Portugal - POCTEP 2014-2020, y actualización de los datos medioambientales
The evaluation examines the implementation of the OP up to September 2019.
To assess the efficiency of management procedures, the implementation of the OP and the achievement of output targets that are part of the environmental scorecard.
Analysis of monitoring data.Interviews with MA, Joint Secretariat and Coordination Units.Survey of beneficiaries (504 contacted, response rate: 64%).Survey of key stakeholders.
The budget allocation by PA is consistent with the strategy of cross-border areas.The majority of the beneficiaries are satisfied with the reporting system and assessment criteria, though horizontal principles are considered only to a limited extent.According to beneficiaries, participation in partnerships has strengthened the relations between them and generated new knowledge.Overall, there is progress towards all the targets of the environmental scorecard, except for air quality and climate.
There are limitations as regards some output indicators because of the lack of homogeneous, updated, and stable data.
Simplify the application process.Prepare a catalogue of good practice on horizontal principles.Improve the functionalities of the Coopera 2020 platform.Provide more technical and human resources to the Joint Secretariat and Coordination Units.Strengthen gender mainstreaming and equal treatment.Review the methodology for assessing the contribution of projects to sustainable development and combating climate change.Review the definition and methodology of Strategic Environmental Assessment (SEA) indicators.
Study on economic sustainability in the Czech Republic in preparation of the Competitiveness Programme, 2021-2027
Analýza tržní situace a specifikace vhodných forem podpory pro účely přípravy priority 4 OP TAK - Efektivnější nakládání se zdroji operačního programu zaměřeného na konkurenceschopnost pro období 2021 – 2027
The study examines the state of play as regards economic sustainability to help plan the future Competitiveness Programme, 2021-2027.
To assess the degree of interest of companies in the Czech Republic in the issue of the sustainable economy and to identify the best ways of including it in the new programme.
Interviews with stakeholders.Survey of companies.Focus groups with experts on sustainable economy.
Interest in sustainable economy projects is raised by the financial return on them, the need to meet regulatory obligations and the risk of non-availability of raw materials.Obstacles are mainly low economic attractiveness of projects, the high level of investment and difficulties in accessing credit, and the uncertain/inappropriate regulatory framework.The administrative burden, long selection process, lack of transparency and non-availability of information also discourage applications for subsidies.
Reduce the administrative burden on companies and stabilise the regulatory framework, to increase applications for support.Ensure subsidies go to projects with a long period of economic return and to SMEs with difficulty in accessing credit.Increase the use of financial instruments.
Evaluation of financial instruments under the Smart Growth OP in Poland
Ewaluacja systemu realizacji instrumentów finansowych w ramach POIR
Taylor Economics: Michał Przybyłowski, Piotr Tamowicz, Anna Zamojska - PAG Uniconsult: Maciej Gajewski, Robert Kubajek, Bogdan Pietrzak, Jan Szczucki, Justyna Witkowska
Taylor Economics Sp. z o.o., ul. Trzy Lipy 3, 80-172 Gdańsk, Tel. 698653612, www.tayloreconomics.com - Policy & Action Group Uniconsult Sp. z o. o., 00-728 Warszawa, ul. Kierbedzia 4, tel. 22 256 39 00, www.pag-uniconsult.pl
The evaluation assesses the system of financial instruments (FIs) used in the three priority axes (PA I-III) of the Smart Growth OP in Poland in 2014-2020.
To assess the level of implementation of FIs.
Analysis of documentation and data.Interviews with members of the MA, intermediary bodies, institutions implementing the FIs, the Polish Agency for Enterprise Development and financial intermediaries.Benchmark analysis of FIs in EU Member States.Focus group interviews.Case studies.Workshop with institutions implementing FIs.Panels of experts.
Capital instruments:There are 5 capital instruments and by end-2018, 14 recruitment procedures for financial intermediaries had been initiated and 213 proposals had been submitted.The selection criteria and procedure adopted by PFR Ventures (the Implementing Authority for the Open Innovation instrument (PA II)) are consistent with those applied in programmes abroad.The multi-entity consortium structure which implements the capital instruments is considered cumbersome and may lengthen the decision-making process.Assuming the current rate of investment, around 22% of funds (or over PLN 500 million, EUR 117 million) for FI may remain unspent.Due to the market conditions, some FI may address the same target groups.Overall, capital investments provided by Cohesion Policy are not in competition with those provided by the European Investment Fund, but there may be a risk the latter will attract managing teams.Loan guarantees:The Guarantee Fund is implemented as a project operated by Bank Gospodarstwa Krajowego (BGK), which has a leading role in the market.By end-2018, only 37 guarantees had been granted for a sum of PLN 68.3 million (EUR 16 million), with an average guarantee of PLN 1.8 million (EUR 420 thousand). There is a risk that resources may not be used given the total funding of around PLN 2.2 billion (EUR 516 million).The system of implementing guarantees is generally appropriate and justified.
The limited access to some data because of confidentiality reasons.
There is a need to:Capital investments:simplify the entire implementation structure to improve the decision-making process;PFR Ventures should commit more resources to recruitment (e.g. by increasing staff);consider simplifying the assessment rules;provide a legal and economic analysis which assess whether and to what extent the implementation of tax reliefs is possible to support the use of capital funds.Guarantee instruments:promote close cooperation of BGK with lending banks to simplify procedures and the required documentation;consider for the next programming period the possibility of combining loan guarantees with grants.strengthen promotional and information activities targeted at businesses.
Mid-term evaluation of the Zachodniopomorskie OP, 2014-2020
Ewaluacja mid-term dotycząca postępu rzeczowego RPO WZ 2014-2020 dla potrzeb przeglądu śródokresowego, w tym realizacji zapisów ram i rezerwy wykonania.
ECORYS Polska Sp. z o.o., ul. Solec 38 lok. 10, 500-394 Warszawa, e-mail: ecorys@ecorys.pl, EVALU Sp. z o.o., ul. Dzika 19/23 lok. 55, 00-172 Warszawa, e-mail: evalu@evalu.pl
The evaluation examines the extent to which the OP’s objectives have been achieved, how external conditions have changed, and so where modification of the assumptions adopted at the programming stage need to be made, and how effective the institutional support system is in achieving the objectives.
To assess changes in the socio-economic situation in individual aid areas, fulfilment of interim and final objectives as expressed by relevant output and result indicators, progress in implementing IPs, their absorption potential and the possibilities of spending funds still to be allocated.
Desk research of strategic and programme documents, monitoring data and public statistics, evaluation and expert reports and other studies;in-depth Interviews with the MA, IB and selected representatives of self-governments;CAWI and CATI surveys of beneficiaries and unsuccessful applicants;econometric and statistical analysis;PESTEL analysis;benchmarking;expert panels;workshop.
The OP is being implemented efficiently.64% of funding has been committed.All the interim objectives for 2018 were met.There is still considerable absorption potential in all PAs and IPs.In most cases, target values of the strategic result indicators are expected to be achieved and even exceeded.In a majority of cases, territorial policy interventions are more efficient and useful than those covering the entire region.Entrepreneurs are willing to use refundable financial instruments, but mostly, not for investment.
There is a need to:allow new types of projects in selected activities;build absorption potential in activities where there is low interest, by providing support for applicants, expanding the group of potential beneficiaries and through information campaigns;change some indicator targets;direct selected intervention to areas in the least favourable position or to specific groups of beneficiaries;increase promotion of financial instruments.
Final evaluation of communication activities
Rapporto di valutazione finale delle azioni sviluppate dal piano di comunicazione
Evaluation of support for adult education under the Kujawsko-Pomorskie OP, 2014-2020
Ocena wsparcia EFS w ramach RPO WK-P 2014-2020 w zakresie kształcenia osób dorosłych
The evaluation examines measures for adult education under PA 8 "Active on the labour market" and PA10 "Innovative education".
To assess the relevance, effectiveness and usefulness of measures for the education of adults.
Analysis of monitoring data and official statistics.11 in-depth Interviews.Survey of 859 people.2 Focus Group Interviews.1 expert panel.
Support for adult education has had a positive impact on migration from the region.Beneficiaries implement horizontal principles, especially by using appropriate selection criteria for projects.47% of participants not working before participation in projects were not working after. 65% of participants working before the projects had stable employment after.Beneficiaries report positive effects in terms of effective vocational activation, good relations with project participants, greater interest of people in training, higher self-esteem and readiness of participants to take up employment and use new skills.COVID-19 and difficulties in the functioning of offices led to significant delays in projects.
Increase promotional activities for employers and people over 50, to increase their participation in projects.Intensify the training available in industry-related jobs, especially blue-collar jobs.Do not exclude projects with online activities in project appraisal.
Evaluation of problems connected to the implementation of recommendations created during the process of Cohesion Policy evaluation
Badanie problemów związanych z wdrażaniem rekomendacji opracowanych w ramach ewaluacji działań Polityki Spójności
Tomasz Klimczak, Adam Miller, Zofia Polańska, Bartosz Ledzion
The evaluation focuses on issues arising from the implementation of recommendations developed within the cohesion policy evaluation system.
To identify the factors influencing the implementation of recommendations and develop measures to improve this process in the future.
Desk analysis of relevant documents.Analysis of monitoring data.12 case studies.42 interviews with implementing institutions.CAWI/CATI survey of 68 evaluation units.2 Focus groups with implementing institutions.Workshop.
The main problems concerning the implementation of recommendations are the fact that they are outdated, they cannot be implemented without the appropriate involvement of the relevant institutions, and they are of insufficient quality.The mechanisms used to evaluate the quality of bids are not always effective, often resulting in the tender being awarded to the bidder offering the lowest price.Involving the recipients of the recommendations at the planning and implementation stages is a key factor in ensuring that the recommendations are relevant to their information needs.
Adapt research processes to meet the specific, previously defined needs of the decision-making process.Introduce an obligation to consult with the recipients of recommendations before they are finalised.Adopt a project-based approach to implementing recommendations, considering objectives, timelines, and necessary resources.
Mid-term evaluation of the communication and information activities under the Technical assistance OP, 2014-2020
Hodnotenie komunikačných a informačných aktivít v programovom období 2014 – 2020 (prvá fáza) - Záverečná správa
This is the mid-term evaluation of the support provided under the Communication Strategy of the OP Technical Assistance 2014-2020, including the 2016 Annual Communication Plan (ACP).
To assess the purpose and effectiveness of the communication activities undertaken on the basis of the 2016 ACP in relation to the Communication Strategy’s objectives.
The evaluation uses qualitative and quantitative methods:Desk research of relevant documents.Semi-structured (in-depth) interviews.Comparative analysis to assess real vs. planned performance of information and communications.Survey to assess the level of ESIF awareness among the general public.Online survey of professionals.The data used are primary data from interviews and surveys and secondary data from relevant documents and the monitoring system.
All the specific objectives of the Communication Strategy are relevant to the Partnership Agreement needs.The objectives of the 2016 ACP cannot be evaluated univocally as they were defined broadly and insufficiently. However, on the basis of indicators (e.g. level of ESIF awareness, level of ESIF perception) the interim achievement of objectives was satisfactory.The results of the activities under the ACP cannot be attributed to a single factor (factors affecting the results are e.g. problems in public procurement and parliamentary elections).The Webpage and media campaign made a major contribution to meeting the communication objectives, whereas the activities implemented by the Information and publicity department of the Government were less important.Most types of communication activities implemented are relevant for achieving the Communication Strategy’s objectives, though improvements are needed.
The authors specified that the results of the surveys should be considered as demonstrating a statistical trend and not a causal link.
There is a need to:Detail more specifically communication strategy objectives and to consolidate measurable indicators, including the setting of target values.Base the ACP background analysis on analysis of the activities carried out in the previous period.Improve the implementation of the information and communication activities by, among other things: replacing the general promotion of EU funds by promoting their positive impact; indicating the impact of EU-funded support in a specific region instead generally; establishing a common management structure for information and communication and tailoring the media channels used to the budget and objectives.
Evaluation of the effects the OP for EU Structural Funds Investment, 2014-2020, in Lithuania
2014–2020 m. ES fondų investicijų veiksmų programos tikslų pasiekimo vertinimas
The evaluation examines the achievement of the objectives of the OP and the impact of EU funding on the smart, sustainable, and inclusive growth. The monitoring data used go up to 31.12.2022.
To assess the contribution of the OP to the achievement of the EU and national strategic objectives of smart, sustainable and inclusive growth, and the contribution to socioeconomic changes, including in the long term.
- Literature review and desk analysis.- Analysis of statistical and monitoring data.- Analysis of administrative data from the database of companies receiving subsidies during downtime (from the Ministry of Social Security and Labour).- Case studies.- Online survey.- Semi structured interviews.- Expert panels.- Comparative analysis.- Analysis of the intervention logic.- Contribution analysis.- Meta analysis and peer review.
- In general, the relevance, effectiveness, impact and sustainability of the OP is significant. Impact is found to be limited in few areas, such as technological transfer, internationalisation of SMEs and eco-innovation, reduction of energy consumption, water supply and wastewater management, and development of regional transport connections.- The best results are achieved through a cross-cutting approach to clearly identified and defined challenges (e.g. as in the case of the transition from residential to community based social protection services).- Territorial allocation of funding was not always appropriate to answer the needs identified in the most effective and efficient way, i.e. funding only considered the territorial level of the municipality.- Financial instruments are not fully efficient if they are complemented with grant awarded for the same purposes and accessible to the same target group.- The OP strengthened the capacities of civil servants and employees managing EU funds, as well as social and economic partners. However, this improvement does not seem to be sustainable.- The result indicators of the OP do not reflect the actual impact of the measures, as they provide information more on the outputs.
- Concentrate the EU funds allocated to address complex, but well-defined problems, plan complementarities and avoid duplication of budgetary funding.- Address the issues of financial instruments as identified in the evaluation.- Focus funding on systemic change and the development of new services.- Continue and strengthen the CLLD approach and involve NGOs.- Simplify administrative procedures, especially for private beneficiaries.
Evaluation of waste management measures funded by the Environmental and Energy Efficiency OP, 2014-2020, in Hungary
Hulladékgazdálkodással kapcsolatos fejlesztések értékelése
The evaluation examines the implementation of measures for waste management funded under the OP.
To assess the progress of measures on waste management in relation to final targets, the procedures used and possible shortcomings.
Monitoring data.Official statistics.Survey of the general public.Interviews with national and local authorities and stakeholders.
The OP indicators are not able to fully monitor results of interventions and the achievement of EU objectives.Interventions have led to a general increase in waste treatment and recycling capacity, efficiency and public awareness.The high level of decentralisation of responsibilities along with their unclear scope have caused delays in project implementation and inefficiencies.
As several projects were still ongoing at the time of the evaluation, forecasts of results have been used instead of actual ones.
Increase the involvement of waste sector stakeholders and the general public to ensure achievement of 2021-2027 objectives.Define a set of indicators more directly linked to EU objectives in 2021-2027.Allocate more funding in 2021-2027, to stakeholders other than those in the public sector, and involve financial institutions in the management of repayable funds.
Evaluation of the efficiency, effectiveness and impact of the Austria-Hungary Interreg OP, 2014-2020, final report
Evaluation of the efficiency, effectiveness and impact of the programme - final evaluation report
M&E Factory: Christine Hamza, Angelos Sanopoulos; Rosinak & Partner ZT GmbH: Elisabeth Stix MultiContact Consulting Kft: Zsuzsanna Uszta, Éva Magyar
M&E Factory monitoring and evaluation GMBH, Sibergasse 25/10, AT - 1190 Wien, www.me-factory.eu; MultiContact Consulting Kft., Bartók Béla út 105-113, 1115 Budapest, multicontact.eu/en
The evaluation examines the interventions financed by the Austria-Hungary Interreg OP, in terms of their effectiveness, impacts and efficiency.
To assess the contribution of projects to the result indicators and their impact, the relevance of the strategy and their average costs.
Analysis of documents.Analysis of monitoring data.Online survey of beneficiaries.Interviews with experts.Network analysis.
TO3: the measures do not seem in line with the needs of SMEs and the focus of many projects is non-specific.TO6: the projects on the natural and cultural heritage lack a strategic approach; the natural environment and ecosystem projects have been delayed, but potential results are valuable; projects on the management of water bodies are working satisfactorily.TO7: the improvement of small cross-border links in the northern part of the region is not directly related to the TEN-T and might lead to increases in traffic, by providing alternative routes for those who want to avoid main roads and tolls; too many different activities in respect of sustainable mobility could hamper the effective implementation of projects, which are not always able to respond to needs.TO11: projects contribute much to cross-border cooperation between public authorities; projects on intercultural capacity focus on reducing language barriers and on training topics such as robotics, animal welfare and energy, on which little attention is paid at national level. In many cases the projects have intangible results, such as creating a better atmosphere for cross-border cooperation.Most participants in projects are public authorities, followed by special interest groups and NGOs.Respondents report that the Interreg process is time-consuming and bureaucratic, but most are positive about being involved in an Interreg project.
None in particular, but in some places it is underlined that the small number of projects and their early stage do not enable impact to be assessed.
Focus more in the future on environment and institutional capacity.Do not intervene in the support of SMEs.Focus more on sustainable transport than on the roads.Intervene on direct accessibility in the border region rather than linking to the TEN-T.Simplify and reduce the administrative burden.Increase synergies between projects and Specific Objectives.
Evaluation of the effects of SME support under the Integrated Infrastructure OP 2014-2020
Posúdenie efektov finančnej a nefinančnej pomoci MSP poskytovanej prostredníctvom národných projektov implementovaných v gescii MH SR v rámci OP Integrovaná infraštruktúra
Institute of Forecasting - Centre of Social and Psychological Sciences - Slovak Academy of Sciences
Centre of Social and Psychological Sciences, CSPS SAS, Sancova 56, 811 05 Bratislava, Tel: +421-2-5249-5114, E-mail: progasis@savba.sk
The evaluation examines the support for SMEs with particular focus on the National Business Centres (NBC), providing traineeships, incubation and growth programme and vouchers for creative industries, for marketing, ICT and design activities.
To assess the effectiveness of SME support, in the form of National Business Centres and vouchers for creative industries for design, ICT and marketing support and to make recommendations for 2021-2027.
Desk reviewMeta-analysis of evaluation approaches of similar projects implemented in OECD countriesAnalysis of monitoring and statistical dataInterviews with regional managers of each projectInterviews with enterprises supportedOnline survey of enterprises supportedCase study
7,801 enterprises were supported between 2017-2020, of which 3,554 were in the Bratislava region and otherwise in regions most in need, i.e. Banská Bystrica, Košice, Trenčín and Žilina.Enterprises supported were mainly in wholesaling and retailing, repair of motor vehicle and financial and insurance activities.Positive aspects of NBC are an individual approach and quality and reputation of consultants.Negative aspects of NBC are their insufficient visibility, high administrative costs, insufficient cooperation with other organisations, limited incubator capacity, weak links to science and research and lack of infrastructure in the regions.The average value of support (EUR 400) is considered too low to have a real impact on companies.424 enterprises out of 1,587 applicants received creative vouchers, with a total value of EUR 2.1 million and an average of EUR 4,915 per voucher.Nearly half were concentrated in the Bratislava region and most went on advertising and marketing and ICT activities.Despite efforts to simplify and streamline procedures, issues remained in the form of administrative complexity, long duration of the overall process, insufficient capacity of staff and lack of an appropriate IT system.
The COVID pandemic limited the possibility of carrying out face-to-face interviews and it was difficult to identify enterprises (only legal persons were identified).
Provide NBC services to all target groups across the countryProvide complementary servicesReduce the administrative burdenRaise awareness of the support available among target groupsBroaden the focus of voucher supportExpand the target group for vouchersRaise awareness among enterprises of supportExpand regional coverageReview public procurement requirementsOptimise administrative capacity and digitalise procedures.
Mid-term evaluation of the progress of the Opolskie Voivodship OP 2014-2020
Ewaluacja mid-term dotycząca postępu rzeczowego RPO WO 2014-2020 dla potrzeb przeglądu śródokresowego, w tym realizacji zapisów ram i rezerwy wykonania
The evaluation covers the whole Opolskie Voivodship OP 2014-2020. It examines whether the performance reserve indicators have been achieved and the need for changes in the OP. It also considers the possible value of the indicators at the end of the OP and makes recommendations for the next programming period.
To assess progress in achieving indicators and the Europe 2020 objectives, to consider the justification for the programme indicators adopted and to assess the absorption potential of the different Priority Axes (PAs).
Desk research of the EU and national legislation, strategic and programming documents, relevant evaluation reports and other studies, public statistics and monitoring data;in-depth interviews with the OP MA, Ministry of Investment and Development, beneficiaries of the Border Areas PA and IPs;focus groups with the OP Monitoring Committee representatives, MA and IBs;CAWI with self-government representatives;statistical, econometric and macroeconomic analyses;expert panel;TBE – result analysis of measures for which there is a performance reserve and Programme modifications are planned. Intervention logic has been recreated.
By the end of December 2018 78% of the OP allocation was contracted (83% for ERDF, 35% for ESF) and 32% certified (61% for ERDF, 22% for ESF).Performance reserve intermediate goals were all achieved.The method used to assess the target for the indicators is valid and the indicators are coherent with the intervention.The intervention logic is sound.For some IPs, there is still the potential to absorb additional funding, e.g. 1b, 3c (sub-measure 2.1.1.), 4e, 5b (measure 4.2), 8iv, 9iv, 10i, 10iv and 10a.On the other hand, for some IPs, the current allocation will not be used, e.g. 1a, 3c (sub-measure 2.1.4), 3a, 4a, 5b (measure 4.1), 6a, 8iii, 8vi, 8v, 9i, 10iii, 9b, 2c.Of the 213 indicators of the OP, for 48 (22 ERDF and 28 ESF) the achievement will be less than 85% of the target.For 33 indicators (mainly ESF) additional research is required to assess the target.
84 recommendations are made, including 20 on the need to reduce the indicator targets as there is a risk of not achieving them and 21 on the need to increase the targets as they are a forecast to be exceeded.6 recommendations concern reallocation of funds, 3 to increase them so as to achieve the target.10 recommendations are to increase the efficiency of interventions and 4 others to carry out additional analysis.13 recommendations are on the future financial perspective to increase the effectiveness and efficiency of support and reduce the administrative burden on beneficiaries, most addressed to the OP MA, but some requiring coordination between PAs or even EC action.
Evaluation of Invest Northern Ireland's Loan and Equity schemes under the Northern Ireland ERDF OP, 2014-2020
Evaluation of Invest NI's Loan and Equity Solutions
The evaluation examines the INVEST Northern Ireland (NI) equity and debt funds, which provide repayable risk capital to start-ups and SMEs, up to March 2021.
To examine how the schemes operate and their impact and to suggest improvements.
Theory of Change.Analysis of official statistics and monitoring data.Analysis of administrative Business Structures Database (firm-level data).Survey of enterprises (129 respondents, 28% response rate).8 case studies of enterprises.Counterfactual impact analysis (matching method with firm-level employment and turnover as outcome variables using Business Structures Database).
Equity funds supported company needs for seed and development capital, but their rate of return is uncertain.The debt funds are well established as regards the start-up, development, and growth phase of businesses, but the pandemic made the situation challenging for some of them.There are good links between the funds and other forms of business and innovation support.Funds achieved 98% of their target for enterprises supported and 89% of that for the amount invested.Enterprises are satisfied with the funds, though equity recipients are not satisfied with the “quality of services”.According to the survey, cumulative GVA increased by £508.2m and gross employment by 5,000 jobs for businesses receiving finance between April 2020 and March 2021. In net terms, the increases are estimated at £198.7m and 2,200 jobs.According to the counterfactual analysis, in the short-term employment was increased by 308 additional units and in the long-term by 1,252 units (2 jobs per enterprise on average), while turnover was increased by 37% in the short-term and 94% in the long-term.Overall, GVA is estimated to have been increased by £1 of public costs and the net cost per net job created is estimated at £25,100.
Impact and value for money estimates based on the data collected by the survey were limited by the willingness of businesses to provide information.
Closely monitor the recovery in the investment rate.Continue using the two development funds to ensure appropriate follow-on investment.Consider what can be done to drive-up investment penetration outside of the major urban areas.Maximise the returns from funds to Invest NI for reinvestment in future needs.Retain the flexibility of the funds to respond to change in SME demand.Consider conducting more regular research into the NI loan and venture capital finance market.Consider a more comprehensive approach to monitoring and evaluating economic development benefits.
Effectiveness and impact of the Infrastructure and Services OP, 2007-2013, on urban development
ES fondu 2007. – 2013. gada plānošanas perioda ieguldījumu pilsētvidē lietderības un ietekmes izvērtējums
OP Infrastructure and Services 2007-2013; OP Growth and Employment 2014-2020
Ministry of Finance, Ministry of Environmental Protection and Regional Development of the Republic of Latvia
This ex-post evaluation examines measure 3.6 ‘Polycentric development’ of the Infrastructure and services OP, 2007-2013, which financed infrastructure projects for urban development in transport, cultural and tourism and education. It also examines some aspect of the implementation of 2014-2020 funds.
To assess the socio-economic effects of urban projects, their sustainability and reasons for success or failure.
Desk analysis of strategic documents;analysis of monitoring and statistical data (and use of a specific urban development index);questionnaire to local authorities (12 responses out of 30);in-depth interviews with municipalities;19 case studies.
The main outputs were 125 km of urban roads, 12 km of pedestrian or cycling paths, 36 km of water and wastewater pipelines, 92 education institutions and 43 other buildings. The selection the approval process was complex and responsibilities unclear, increasing its length.The set of indicators for 2014-2020 better reflect the objectives and the logic of investment than those for 2007-2013.EU funds have a vital role, accounting for 81% of total investment in urban development since local authorities do not have the finance to invest in strategic projects.Projects were consistent with local strategies.Projects helped to improve urban outdoor space and access to services.Projects contributed to socio-economic development, but it is not possible to say by how much.Local authorities are not able to identify correctly the monetary gains and losses from investment.
The availability of data affected the selection of indicators used to calculate the urban index; some municipalities did not provided data on economic returns.
Consider financing integrated territorial development programmes which have a wider scope than a single local or regional authority;facilitate spatial development planning in a coordinated manner;implement projects combining investment in infrastructure and in human capital development and address new themes, such as smart municipalities;establish a national funding programme for urban and rural development in addition to EU funds;consider using cost-benefit analysis as one of the key criteria for selection of projects.
Evaluation of the Service centre for mobility of workers under the Hamburg ESF OP, 2014-2020
Thematische Evaluation der Servicestelle Arbeitnehmerfreizügigkeit
Simon Bohemian Henriette, Reichswald Marcus Neureiter (Ramboll)
Chilehaus C – Burchardstraße 13, 20095 Hamburg, www.de.ramboll.com
This evaluation assesses the measures implemented by the Service centre in Hamburg to support the integration of migrants, funded by the Hamburg ESF OP 2014-2020.
To assess the Service centre activities in terms of target groups reached and results and to make suggestions for the future.
Desk research and data analysis.Group exploratory discussion.Interviews with experts.Case studies.Observation of service centre activities.
The desired target groups have been reached and the planned outputs achieved.Advice and support are well managed by Service centre.The centre is well known by EU immigrants thanks to cooperation with foreign communities in Hamburg and the linguistic and technical expertise of the consultants employed.The centre adopts both a reactive (advice, support and mediation) and preventive (e.g. information campaign) approach. Focus is on advice on employment law, where there is a risk of overlap with others providing similar advice.The centre affects the working conditions of migrants in Hamburg both indirectly, by interacting with employers, and directly, through support for workers.The centre has improved transnational cooperation between authorities in different EU Member States and the understanding of the authorities in Germany as well as in the countries of origin of migrants through information campaigns.The centre will still be important in the future, given that the number of migrants in Hamburg continues to increase.
The choice of approach was partly due to difficulties in reaching individuals for interview or survey.
There is a need to:minimise the risk of overlap with other service providers;coordinate with other bodies in Germany and focus preventative efforts in the countries of origin of migrants;apply the Service centre model in other ESF projects with high innovation potential.
Evaluation of results of measures financed under the France-Italy Interreg OP, 2014-2020
Evaluation d'impact du programme de coopération territoriale transfrontalière Interreg V A - France - Italie ALCOTRA 2014-2020
Interreg V-A-France-Italy (ALCOTRA) 2014-2020
The evaluation examines the results of the measures funded under the OP up to end-March 2020.
To assess the effects of the OP projects up to end-March 2020.
Theory-based approach.Online survey of 121 project leaders (55 responses).14 case studies (10 simple projects, 2 PITEMs, Thematic Integrated Plans, 2 PITERs, Integrated Territorial Plans).30 interviews with project leaders.Analysis of monitoring data.
At 31 March 2020, certified expenditure was around 23% of funding but some 2023 output targets had already been exceeded.Beneficiaries are mainly public authorities and companies supported make up only 1% of the total.The cross-cutting effects of the OP are increased dialogue between institutions on the two sides of the border, increased use of digitalisation, improved skills of workers and project partners.Many projects support the introduction of product, organisational and process innovations.PA 1 and 2 projects (resilience to climate change) contribute to better environmental management, while PA3 ones (territorial attractiveness) have supported structural investment, new areas of tourism, improvements in fragile habitats and species, and eco mobility.PITEM projects have a wider “scale” of impact than simple ones and PITER projects have more “intensity” of impact (their concentration on small areas has the potential to produce greater effects).
Some of the result indicators are not in line, in terms of definition, with the 2014-2020 regulatory framework.
Define more precisely the objectives of integrated projects.Focus PITEMs solely on areas requiring long-term governance strategies.Pay more attention to the territorial nature of PITERs.Improve the system of indicators.Provide support to beneficiaries in implementation and monitoring in the future OP.Ensure more effective coordination with other policies and programmes.
Evaluation of measures for inclusive education under PA 3 of the Research, Development and Education OP, 2014-2020, in the Czech Republic
Evaluace individuálních systémových projektů podpořených z PO 3 OP VVV-II Část II: Evaluační okruh A – Evaluace projektů APIV - Závěrečná zpráva
Křižíkova 298/70b, 612 00 Brno, Czech Republic
The evaluation investigates two systemic projects supported by PA 3 of the OP: “Implementation of the Inclusive Education Action Plan – methodological support” (APIV A project) and “Support for Inclusive Education in Educational Practice” (APIV B project). Data go up to November 2022.
To assess progress in the implementation, the quality of management and the main results of the projects.
Document analysis of the 4 previous interim reports.Analysis of monitoring data.16 individual interviews with stakeholders.2 group interviews with policy representatives.
The support from the projects was generally perceived by representatives of schools as beneficial, especially in terms of increased professional qualifications and competences.Transfer of good practice took place at several levels (e.g. training and internships) and platforms for teachers and members of school counselling centres were created to share experience.The direct effect of the support provided on resolving individual challenges in the area of inclusive education could not be easily assessed, but projects increased awareness about this in schools.Cooperation as a compulsory activity proved to be successful, especially in relation to target groups.The projects succeeded in achieving the expected results.
It was not possible to carry out a counterfactual analysis for assessing impact because of the difficulty of identifying a control group (almost all schools received support). The required survey on the level of parental support for inclusive education was problematic because parents are not able to assess this.
The scope of individual projects should be reduced.Plan ad hoc follow-up activities to ensure projects’ sustainability.Make use of outputs from the National Education Institute and Ministry of Education and Sport more systematically.
Evaluation of measures for special education under the Human Resources Development, Education and Lifelong Learning OP and ESF regional OPs, 2014-2020, in Greece
OP Human Resources Development, Educatin and Lifelong Learning 2014-2020; All ROPs 2014-2020
ICAP Group SA - El. Venizelou Aven. 2, Kallithea 176 76, Greece, +30 21 0720 0000 , https://www.icapcrif.com/
The evaluation examines support for special educational pathways and new equipment and the employment of specialised staff in schools, up to end-2020.
To assess progress towards targets and the main results of the measures for students with special needs up to end-2020.
Analysis of relevant policy documents and literature.Analysis of monitoring data and official statistics.Analysis of administrative data from the Directorate of Special Education.Survey of workers in special education and tutors of students with special needs.Interviews with representatives of the departments involved in the planning of measures.4 focus groups with those involved in special education design and implementation.
Coherence of the measures with the “National Strategy for Social Inclusion 2021” is low, but it is high with the “Strategic Policy Framework for the Reduction of Early School Leaving in Greece” and the “National Deinstitutionalisation Strategy”.Regional OPs are only partially relevant to the needs of students with special needs.All output indicators have been either exceeded or almost reached.The measures are highly efficient, since they have been implemented at lower unit costs than planned.
Projects implemented through the National OP and the Regional OPs have not been targeted in relation to the result indicator “Percentage of schools providing specialised education support to pupils with disabilities”.
Evaluation of Local Action Plans for Education financed under the Research, Development and Education OP, 2014-2020 in the Czech Republic
Hodnocení Místních akčních plánů rozvoje vzdělávání Zpracovalo oddělení Evaluací (435), Odbor řízení OP
The report reviews the Local Action Plans for the Development of Education (LAPs) financed by the Research, Development and Education OP under PA3. LAPs are intended to ensure equal access to education, to develop skills in children and to promote cooperation between families and schools. They cover early childhood, pre-school and primary education, targeting children up to 15 with special educational needs.
To summarise the findings of previous evaluations on the functioning of the LAPs in order to obtain useful information for the design of the new LAP-II.
Desk analysis of programming and implementation documents, evaluation reports, including internal ones on individual LAP projects.Analysis of the main results of the LAP Functional Verification Survey and of the meetings organised in each region by LAP beneficiaries.
LAPs respond to the needs of regional education systems.Stakeholders are inadequately informed about LAPs and their potential benefits and have a low interest in actively participating in LAP projects partly because of the absence of financial incentives. Some are excluded from projects because of their poor technical and administrative capacity, others, e.g. socially excluded parents are not interested in being involved.Applicants and beneficiaries consider the technical and information support provided by the Ministry to be inadequate. Guidelines produced by the National Institute for Higher Education were published late and have been continuously modified without adequate and timely notification.Payments to LAPs from the Ministry have been delayed, causing problems in particularly for municipalities and NGOs.Beneficiaries report LAP monitoring procedures to be inadequate and time-consuming.
Strengthen information activities on LAPs, particularly for less motivated groups.Strengthen the technical and information support for preparing and implementing LAP projects, make guidelines easier to use and communicate any changes promptly.Simplify the monitoring procedures for LAPs by reducing the administrative burden on beneficiaries.Publish the next call for proposals when LAP projects currently in progress come to an end.
Evaluation of the horizontal principle of equality between men and women under the Mecklenburg-Vorpommern ERDF OP, 2014-2020
Schlussbericht. Umsetzung des Querschnittsziele Gleichstellung von Frauen und Männern in der Förderperiode 2014 - 2020.
OP Mecklenburg-Vorpommern ERDF 2014-2020
Steffi Kühn, Landesfrauenrat MV e.V. (LFR)
Steffi Kühn, kuehn@landesfrauenrat-mv.de
This evaluation analyses the contribution of the OP to the principle of equality between men and women.
To assess how and to what extent measures under the OP have contributed to gender equality.
Analysis of documentation. Interviews with policy implementers. Evidence from previous interim reports. Qualitative survey of beneficiaries.
The gender equality principle was taken into consideration in the design of the OP and of the measures to be financed according to the guidelines set by the State Women's Council (Landesfrauenrat). Selection criteria of projects need to be framed up according to guidelines on gender equality. Applicants need ad hoc support in defining the contribution of projects to gender equality. Monitoring indicators selected are not fully capable of capturing the contribution of projects to gender equality.
In 2021-2027, define monitoring indicators capable of capturing the contribution of projects to gender equality. Define more precisely to what extent and how the ERDF can contribute to gender equality.
OP for the implementation of YEI in mainland France and outermost regions – First analysis of YEI long-term result indicators, 2016 survey
Note d’analyse des indicateurs de résultat à long terme – Résultats de l’enquête PO national IEJ de mars-avril 2016
This is the first annual report (2016) on the OP for the implementation of YEI in mainland France and the outermost regions 2014-2020.
To assess the effect of the OP and its progress towards achieving its targets.
The evaluation is based on a survey of a representative sample of beneficiaries (total sample size=7,000; response rate=37%).
Over 12,800 participants were employed six months after the end of their participation (43% of the 2023 target).170 participants were self-employed 6 months after the end of their participation (75% of the 2023 target).Around 3,300 participants were in additional training, an apprenticeship or a traineeship 6 months after the end of their participation (48% of the 2023 target).59% of participants received at least one offer of a job, traineeship, apprenticeship or training proposal according to project managers (20% according to the survey).The employment rate among participants is high and does not tend to diminish over time (46% after participation, 50% after 6 months, 52% at the moment of the survey).57% of respondents consider their job to correspond with their level of qualification or to be higher.Over 58% of respondents had a net salary of more than EUR 1,000 a month (11 percentage points difference between men and women).74% of respondents had a full-time job (14 percentage points difference between men and women).Education level is the most important influence on the chances of becoming employed (20 percentage points difference between the most highly educated participants and the lowest).
The figures presented in the report are restricted to participants who left the programme before 30 June 2014 (117) and between 1 July 2014 and 30 June 2015 (25,644).
Evaluation of measures under PA4 of the Competitiveness and Cohesion OP, 2014-2020, in Croatia
Višekriterijske analize za prioritetnu os 4 Operativnog programa konkurentnost i kohezija
Regionalna energetska agencija Sjeverozapadne Hrvatske, Andrije Žaje 10, 10 000 Zagreb, http://www.regea.org
This is the final report on measures supporting energy efficiency and the use of renewables under PA4 of the OP. The analysis goes up to January 2019.
To assess the tendering and selection procedures in respect of measures for energy efficiency and use of renewables, as well as the implementation of projects.
Documentation analysis.Online survey of 343 stakeholders and beneficiaries.Analysis of monitoring data and official statistics.
The main issues as regards the application and selection process are unclear rules, complex procedures and low transparency, lack of experience in participating to EU-funded calls, short deadlines for applying, and understaffing of IBs.The main issues as regards implementation are a lack of clarity of implementation rules, delayed payments, insufficient support and communication from public bodies, too short deadlines, and lack of knowledge on the use of financial instruments.Overall, projects implemented under IP 4.3 have been the most appreciated by beneficiaries and stakeholders, and should be allocated further funds in the future.
Projects financed under IP 4.2 are still ongoing and monitoring data are not fully up-to-date.
Announce future calls at least 6 months before their opening.Strengthen the internal capacity of IBs.Allocate further funds to projects under IP 4.3.
Evaluation of measures for the employability of people with disabilities under the Sardegna ESF OP, 2014-2020
ISRI - https://isri.rm.it/, isri@isri.rm.it; IRIS - https://www.irisricerche.it/
The evaluation examines the call « Includis », on measures for labour market inclusion of people with disabilities, financed under the OP. Most projects were carried out between 2018 and 2021.
To assess the effects of projects carried out between 2018 and 2021 for people with disabilities on their employability and living conditions, the creation of networks, the factors limiting the results, and the replicability of projects.
Analysis of monitoring data and official statistics.Survey of all beneficiaries.Survey of stakeholders, final recipients and companies hosting final recipients for traineeships.Focus group.Studies on social inclusion measures and programming documents.
At the time of the survey (mid-2022), around 10% of participants had a job, often in a company other than that of their traineeship.According to final recipients, projects mainly helped them to be more active on the labour market, to increase self-esteem and acquire new competences.Creation of new partnerships and networks has been very limited and less satisfactory than expected in terms of planning innovative joint projects.Factors limiting the achievement of objectives are insufficient involvement of key institutional actors, inadequate matching between hosting companies and recipients of support, and limited flexibility and customisation of project activities.Projects might be replicated in different contexts and for different categories of vulnerable people.
Improve the selection of hosting companies, define hiring incentives for them and increase collaboration with the PES.Make project activities more flexibile and able to adjust to participant needs.Improve joint project planning between local actors.
Evaluation of PA3 (on competitiveness of SMEs) of the Pays de la Loire ERDF/ESF OP, 2014-2020
Évaluation portant sur l’axe 3 du PO FEDER-FSE Pays de la Loire 2014-2020 : Améliorer la compétitivité des PME
Mathilde Jauzein, Christophe Picamilh, Léonor Rivoire, Technopolis Group
www.technopolis-group.com;mathilde.jauzein@technopolis-group.com, christophe.picamilh@tehnopolis-group.com
The evaluation examines PA3 on SME competitiveness of the Pays de la Loire OP.
To assess the effectiveness, efficiency, coherence and relevance of support for SME competitiveness.
Desk research.Interviews with various stakeholders.Interviews with direct beneficiaries (10) and final beneficiaries (38).Online survey of 246 young innovative SMEs receiving incubation support (response rate 37%).5 case studies.Workshop with a technical committee.
PA3 objectives are in line with needs in the region.The support provided is coherent with the other PAs of the OP and complements other regional policies promoting business development.The key success factor for meeting targets is the involvement of a number of beneficiaries experienced in managing EU funds.Support gives rise to significant EU added-value, since without it, the measures would not have been carried out as extensively or in the same way (it provides additional financing but it also affects the way the management process is organised).
The study covers data only up to 15 January 2019.
Update the analysis of regional needs before the preparation for the next programming period.Continue with the same structure of the programme.Facilitate the access of more fragile beneficiaries to the programme (e.g. by an advance payment at the start of the project).Adapt the monitoring system to better reflect achievements (e.g. by defining programme-specific indicators and by setting more realistic targets).
Annual Evaluation of the Campania ERDF OP, 2014-2020
The evaluation examines the implementation of the Campania ERDF OP as at the end of 2018.
To assess the level of progress of each PA in the OP and the achievements.
Analysis of monitoring and statistical data.
At 31 December 2018, only 24% of total funding had been spent and less for TOs 4, 9 and 11.For PA2 (TO2), there have been difficulties in implementing digital infrastructure projects.PA3 (TO3) failed to achieved 2018 financial and physical milestones, as did TO9.The urban development PA failed to achieve any of the milestones.
Evaluation of the achievements of health programmes under the Warmińsko-Mazurskie OP, 2014-2020
Ewaluacja Programów Polityki Zdrowotnej dla mieszkańców województwa warmińsko-mazurskiego realizowanych w ramach RPO WiM 2014-2020
Paweł Janukowicz, Marcin Buczek, Jakub Kośman, Tomasz Łukaszewski, Bogdan Załęski
Regionalne Obserwatorium Terytorialne, Departament Polityki Regionalnej, https://bip.warmia.mazury.pl/
The evaluation examines the implementation of regional health programmes under the OP, which cover infectious disease prevention, rehabilitation and vaccinations.
To measure success in reducing disease prevalence, improving public health awareness and enhancing access to health services. To assess the financial efficiency, sustainability and long-term impact of the measures and provide recommendations for future policies.
Desk research (including analysis of interviews with participants carried out as part of the measures).Analysis of monitoring data.Focus group with MA representatives.Benchmarking of 2 programmes from other regions.Expert analysis.
Significant improvements emerged in disease prevention, vaccination rates, and rehabilitation outcomes.The cost per participant in some programmes (e.g. rehabilitation) was higher than in comparable regions, raising efficiency concerns.Some health programmes were only implemented in specific areas of the region, which limited their impact at the regional level.Certain groups (e.g. men, rural residents) were underrepresented as participants in the measures, which raises concerns about the programme inclusivity.Pandemic-related restrictions disrupted the implementation of several planned health measures​​.
Improve cost efficiency by optimizing expenditure per participant and identifying cost-saving measures.Expand geographical coverage to ensure equitable health benefits across the region.Enhance recruitment strategies to engage underrepresented groups.Strengthen long-term impact tracking by introducing more comprehensive follow-up assessments.Increase programme flexibility to better adapt to crises like COVID-19.
Evaluation of the benefits of Housing First projects for settled persons under the Employment OP, 2014-2020, in the Czech Republic
Ověření přínosů projektů Housing First pro zabydlené osoby - kvalitativní šetření
Tomáš Kravka, Ladislav Klika, Šárka Vidlařová
MindBridge Consulting, www.mindbridge.cz
The evaluation examines the benefits provided by the programme, which aims to eliminate homelessness by providing housing to people in need without prior preparation. The interviews were conducted in November-December 2021.
To assess the benefits of the Housing First projects according to those supported, as reported by them in November-December 2021.
30 interviews with people housed through the Housing First programme.
The people housed are highly satisfied with the accommodation acquired.The main benefits from the housing are reported to be peace, privacy, mental stability, more space, and significantly better living conditions.Solving the housing problem does not seem to be a sufficient prerequisite for improving family relations, which are often complicated.For children, moving to the housing provided enabled them to improve their development, education and leisure conditions, as well as their psyche and relationships with parents and siblings.According to most respondents, the programme stabilised or improved their mental state.The proactive, kind, friendly and human approach of social workers was particularly appreciated by respondents, often replacing psychological help.Covid-19 is not considered to have affected respondent contact with social workers.
For the interviewees, it was difficult to understand the essence of some questions, to reflect on and to express a position on a given issue, to formulate relevant responses (because of limited expressive abilities), and to describe their perception of some concepts. In addition, because of speech defects or poor articulation, the evaluators had some difficulty in understanding the responses provided.
Measuring the impact of Midtjylland Growth Forum projects in 2007-2013
Monitorering og effektmåling af Vækstforum Midtjyllands indsats igangsat i 2010-2012
Vækstforum Midtjylland (Midtjylland region growth forum) launched 14 projects in the years 2010-2012 (targeting more than 4,000 currently active private companies) co-financed by the ERDF and ESF to develop and sustain businesses in 6 areas: Energy and the Environment, Food, Tourism, Innovation and Business Development, Entrepreneurship and Education and competence development. The evaluation assesses the impact of interventions on employment, revenue and productivity in the enterprises supported using counterfactual methods.
To assess the impact of the measures on enterprises supported in terms of employment, revenue and productivity.
The proportion of companies with positive employment growth was greater in the treated group than in the control group in all areas of intervention.The effects of interventions cannot be attributed with certainty to project implementation for energy and the environment and tourism since the results are not statistically significant.Overall participation in the projects supported led to the creation of 1,800 new full time jobs.Positive effects on revenues are evident only for Entrepreneurship and Education and the development of competences.According to counterfactual analysis measures supported did not have a positive impact on productivity growth.
Important features were not included in the definition of the control group (e.g. corporate motivation and commitment towards growth).
Evaluation of broadband projects under the Marche ERDF OP, 2014-2020
Gli interventi del POR FESR Marche 2014-2020 collegati alla diffusione della banda ultralarga e le possibili integrazioni con le missioni del PNRR
The evaluation examines the implementation of broadband projects supported by the ERDF and their complementarity with the Italian Resilience and Recovery Plan.
To assess the level of implementation of broadband projects.
Desk reviewInterviews with the MA, municipalities and private companies.
There have been some changes in the funding available from the regional OP for broadband investment during the programming period.236 municipalities have undertaken investment and 29% of them had completed and tested projects by end- 2021.Although the implementation of projects has gone smoothly, there were some delays mainly because of a lack of experience of municipalities, technical aspects concerning the areas where broadband should be installed, lack of specialised companies and increases in the cost of materials.Positive elements include the facilitation and coordination role played by the Marche region and the MeetPAd platform which sped up the decision-making process and allowed better coordination between municipalities.
Evaluation of the OP Liguria ERDF, 2014-2020. Third report
Terzo Rapporto di valutazione del POR FESR 2014-2020
The evaluation investigates the communication plan, the territorial approach and the application of horizontal and sustainability principles under the OP. Data used go up to December 2022.
To assess: 1) the effectiveness of the OP’s communication tools; 2) the characteristics and the results of the territorial measures implemented; 3) the ways in which the OP promoted the integration of horizontal principles, including equal opportunities and environmental sustainability.
Analysis of relevant documents.Analysis of statistical and monitoring data.Interviews with stakeholders.Online survey of beneficiaries.
Beneficiaries consider the regional website to be the most relevant information channel, though only a quarter of them visit it frequently. Other digital channels, such as the newsletter and social media, are not sufficiently known and used.Measures on urban development strategies concern the digital agenda, energy efficiency and protection from hydrogeological risk. They are reported to have achieved their objectives and promoted the organisational and managerial innovation of the MA.Measures preventing hydrogeological instability have significantly contributed to improving safety and sustainability of the regional territory.Measures supporting areas of industrial crisis and free trade zones contributed to relaunching business investments. According to beneficiaries, these would not have been made without ERDF support.Measures supporting audiovisual and cinematographic productions potentially contributed to tourism development. However, it is not possible to directly attribute the positive trend to ERDF support.Measures to support inner areas through the financial instrument “Garanzia Artigianato Liguria” mainly financed the digitisation of enterprises and less energy efficiency of public buildings.The principle of equal opportunities is only partially addressed through measure 3.7.1, aimed at supporting social enterprises.The OP paid particular attention to the environmental sustainability, both directly (measures for energy efficiency and land protection), and indirectly (introduction of specific rewarding criteria in calls).
Provide timely information to potential beneficiaries on financing opportunities before the actual publication of calls for proposals.Improve communication from beneficiaries, also through appropriate award criteria in the proposal selection phase.Pay more attention to the principle of equal opportunities and environmental sustainability in business support measures, also through specific measures, in addition to rewarding criteria.
Evaluation of interventions for family assistance under the Piemonte ESF OP, 2014-2020
L'attuazione delle politiche per l'assistenza familiare 2019
The evaluation examines the first year of implementation of family assistance projects from Nov-2018 to Oct-2019.
To analyse the first year of family assistance projects, Nov 2018-Oct 2019, selection criteria, communication activities to reach potential beneficiaries and the first output.
Analysis of relevant documentation (preparatory study, project proposals…).Interviews with regional officials and those involved in 4 projects.Survey of project leaders.
12 projects were funded over the period. The level of advancement is low and only one project has carried out activities to match labour supply with demand.Up to end of 2019 the main outputs were career guidance to 1,032 people, certification of skills for 1,163, the involvement of 177 family assistants, consulting to 117 families, four activities to match labour supply with demand and 10 training courses for 115 people.The communication strategies are split between targeting specific groups of people and addressing everyone.The two main critical aspects identified in the interviews were relation difficulties from many actors operating in the same area and the burden implied by implementation procedures.
Evaluation of the effects of measures for the low-carbon economy under the Dolnośląskie OP, 2014-2020
Ocena wpływu działań podejmowanych w ramach Osi priorytetowej 3 Gospodarka niskoemisyjna RPO WD 2014-2020 na ograniczenie emisji gazów cieplarnianych i pyłów
Krzysztof Gutta, Katarzyna Gutta, Tomasz Gutta, Joanna Hołub-Iwan
The evaluation examines the effects of the measures to develop a low-carbon economy, financed under the OP.
To examine the effects of measures to foster the low-carbon economy, in terms of greenhouse gases and dust emissions, and the share of energy generated from renewables and energy efficiency.​
Desk research.Analysis of monitoring and statistical data.Survey of 351 beneficiaries and 92 applicants.22 interviews with implementing bodies.5 focus group discussions.28 case studies.2 expert panels.
Ongoing support is needed for low-emission economy interventions beyond 2020 to address remaining gaps and meet regional needs.The application process was identified as a barrier for beneficiaries, with some struggling to navigate the requirements and documentation.The projects led to economic and environmental benefits, but rising costs and inflation posed challenges to budget adherence.In the areas of renewable energy and energy efficiency in public and residential buildings the needs have not been fully met.Projects results are considered to have long-term sustainability, with a majority of projects expected to deliver benefits well beyond their formal evaluation periods.Implemented projects were generally complementary to other regional and EU-funded initiatives.Over 84% of surveyed beneficiaries believed that achieving similar benefits with lower costs was not feasible​.
The relatively small number of experts and their limited interest in participating in the survey made it impossible to complete the sample as intended.
Streamline application procedures and provide clearer guidelines to reduce barriers for beneficiaries.Adapt funding models to account for rising costs and economic fluctuations, ensuring projects can meet their objectives without financial shortfalls.Prioritise funding for projects in renewable energy and energy efficiency to maximise environmental and social benefits.Provide training and resources to local governments and stakeholders to improve their ability to implement and sustain low-emission projects.Ensure that future interventions align closely with regional, national, and EU priorities, leveraging complementarities for greater impact.Establish mechanisms for tracking long-term project outcomes and incorporate findings into future planning cycles to increase benefits.
Use of ERDF funds for rebound loans in the Hauts-de-France Region as part of the OP Nord - Pas de Calais and Picardie
Mobilisation de crédits FEDER dans le cadre de la mise en place du Prêt Rebond en Région Hauts-de-France (Programmes Opérationnels Nord – Pas de Calais et Picardie)
ROP Picardie 2014-2020; ROP Nord-Pas de Calais 2014-2020
Evaluation of effectiveness, efficiency and impact of the Efficient Human Resources ESF OP, 2014-2020, in Croatia
Vrednovanje djelotvornosti, učinkovitosti i učinka provedbe OPEHR i intervencija ESF-a prema kriteriju regionalne i lokalne zastupljenosti uz evaluaciju horizontalnih načela
Antonios Mousios, Maja Horvat, Mladen Vojković, Nives Miošić-Lisjak, Jakša Puljiz, Branka Hodak, Iva Šeler Sarajlić
WYG savjetovanje d.o.o. Ulica grada Vukovara 269 G, Zagreb, https://wyg-c.eu/ - MAP Savjetovanja d.o.o., Medulićeva 24a, Zagreb, https://www.map-consulting.hr/
The report examines the implementation and the impact of the OP up to December 2019.
To assess the efficiency and relevance of the support provided under the ESF and YEI, the effects on employment and living conditions of citizens and the contribution to horizontal principles, and to make recommendations to improve future programming and implementation.
Desk analysis of relevant documents and literature.Analysis of statistical and monitoring data.Counterfactual analysis (Fuzzy and PSM matching).Over 50 interviews with MA, other involved in implementation and project participants.Survey of completed projects (177 out of 453).Survey of stakeholders.Case studies of participants.
The measures failed to adequately identify specific regional/local needs, especially in less developed areas.Several internal factors affected implementation: limited administrative capacity, responsibilities and roles not well defined, unclear procedures and preparation of guidelines not being timely.The achievement of OP objectives was affected by delays from government changes, changed socio-economic patterns and the Covid pandemic.Achievements in relation to expected outcomes (milestones) are satisfactory.Counterfactual analysis of local initiatives for employment (LEDI) shows net positive effects from support of disadvantaged groups and the self-employed, but negative effects for public works for harder to employ groups.Positive effects are larger in counties with above-average unemployment rates in 2015 –2020.The ITI approach was not implemented uniformly and coordination with the Competitiveness and Cohesion OP was limited.
Meetings, interviews, and surveys were conducted online because of COVID-19. Because several measures had not been completed, the evidence is not conclusive.
Further strengthen the MA’s administrative capacity.Pay more attention to regional and local needs and to regional differences.Provide additional technical assistance to cities in the preparation of calls for proposals and in the implementation of projects.Provide clear definitions and guidelines on the ‘integrated approach to territorial development’.Consider all available options to ensure sustainability of positive effects.
Evaluation of projects for inner areas under the Umbria ERDF OP, 2014-2020
Valutazione degli interventi riconducibili alla linea di programmazione 2014-2020 “Aree Interne”
https://www.lattanziokibs.com/ - info@lattanziokibs.com; https://www.irisricerche.it/; https://isri.rm.it/ - isri@isri.rm.it
The evaluation examines measures financed under the National Strategy for Inner Areas (NSIA) in Umbria. Data used go up 31 December 2020.
To assess the management of the Inner Areas Strategy in Umbria and the main results achieved.
Document analysis. Analysis of monitoring and statistical data. Interviews with stakeholders and policy implementers. Surveys of stakeholders and policy implementers.
The implementation of the Strategy in Umbria shows a good performance compared to other regions, despite some delays. A placed-based development policy and a multilevel governance are key factors ensuring effectiveness of the measures. The MAs has played a relevant role in the programming and implementation of projects, providing constant support and coordination. Pre-existing conditions such as a strong track record of partnerships between territorial stakeholders positively affected social innovation. The rigidity of some of Funds’ regulation (i.e. EAFRD) has hindered financial absorption and implementation of projects in rural inner areas.
Delays in the implementation of the NSIA in Umbria do not allow for a thorough assessment of the results.
Use available national resources to overcome the EAFRD regulation’s rigidity. Support participatory local development by implementing ITIs through the integration of different Funds.
Evaluation of the measures for improving Llandaf and Radyr Train Stations, under the East Wales ERDF OP, 2014-2020. Final Report
Evaluation of the Llandaf and Radyr Train Station Improvements:Final Report
Dr Robert Francis (Carreg Las) and Dr Ian Johnson (Loxley Consultancy)
Carreg Las - https://www.carreg-las.com/; Loxley Consultancy - http://www.loxleyconsultancy.com/index.html
This is the final evaluation report on the measures to support intermodal mobility. The analysis was carried out between 2021 and 2022, three years after the projects came to an end.
To assess the level of achievement of targets and satisfaction of users and to identify the main impact of the measures on the use of public transport, three years after the work was completed.
Documentation analysis.Analysis of official statistics.Analysis of monitoring data.Survey of 644 beneficiaries.Interviews with stakeholders.
Measures are relevant for the needs of the region and have strong coherence with other regional policies and programmes, such as the Metro Station Improvements Programme (MSIP).Prior to the implementation of the measures, the number of users of the two stations was increasing slowly. After 2018, despite Covid, mobility to and from urban centres connected to the stations has steadily increased (by 5%).The result and output indicator targets have been achieved. Project management was overall effective.Train station users are generally satisfied with the improvements, in terms of increased security, clear signage and better access to ticket machines.
Covid-19 prevented any reliable analysis for 2020 and 2021 on the use of public transport. In addition, only 96 responses to the survey were collected, which makes it not fully reliable for drawing robust conclusions.
Support the involvement of the general community in the planning of new projects.Ensure monitoring and evaluation of the measures to implement corrective action in a timely way, if needed.
Results of the population survey 2015 about the ESF-communication strategy
Ergebnisse der Bevölkerungsbefragung 2015 zur ESF-Kommunikationsstrategie
ISG - Institut für Sozialforschung und Gesellschaftspolitik
Mid-term evaluation - the France-Belgium-Netherlands-UK (Two seas) Interreg OP, 2014-2020
Programme performance evaluation – Step 2 within the framework of a robust Two Seas 2014-2020 Monitoring & Evaluation system - Report on the evaluation of the mid-term impact
Interreg V-A-France-Belgium-The Netherlands-United Kingdom (Les Deux Mers/Two seas/Twee Zeeën) 2014-2020
The evaluation examines the France-Belgium-Netherlands-UK Interreg OP, 2014-2020 which focuses on four priorities: supporting cross-border initiatives to strengthen research and innovation, a shift towards a low carbon economy, improving the capacity to deal with climate change and promoting a resource efficient economy. There are 7 specific objectives (SOs):SO1.1 Framework conditions for innovationSO1.2 Technological innovationSO1.3 Social innovationSO2.1 Low-carbon technologiesSO3.1 Adaptation to climate changeSO4.1 Efficient use of natural resources and materialsSO4.2 Circular economy.
To assess the contribution of the cross-border projects supported to the OP objectives, their sustainability and transferability and sectoral and territorial focus.
Monitoring data;Analysis of project annual progress reports and application forms (AFs);Result indicator surveys;12 case studies.
The OP is helping to change the cooperation area through its thematic concentration. Contribution to change is however limited for SO3.1, SO4.1 and SO4.2 due to the small number of approved projects.2023 output targets should be reached except for technological and social innovation SOs. For 4 SOs, some of the targets can be achieved only if additional projects are implemented.The OP brings added-value by addressing border-related development challenges, sharing knowledge and experience and cooperation to define more effective solutions.All SOs bring knowledge benefits, increase skills, raise awareness, and promote the exchange of practices.Despite being too early to assess sustainability, projects promote knowledge transfer and capacity building and directly involve the private sector and institutions for a wider ownership and durability of results.Open access and visibility of benefits for end-users are the main conditions for output transferability.All SOs cover the expected sectors and issues and the more vulnerable areas and address the main development needs of the cross-border regions.
A comprehensive evaluation will only be possible once projects are finalised.
Identify for 2021-2027 the most promising sectors, issues and areas where cross-border cooperation is expected to be valuable.Collect evidence on the performance of cross-border cooperation as compared with other policy tools for addressing development challenges.Capitalise on evaluation activities and project monitoring strategies to support beneficiaries in surveying target groups and measuring outputs and results.Highlight the main risks for sustainability during implementation and conduct an ad hoc survey one/two years after the end of projects.Collect information on expected transferability.Identify transferable examples of good practice and make them accessible online.Communicate the actual and expected benefits better along with the potential for future economic scale-up.
Evaluation of measures for SMEs competitiveness under PA2 of the Schleswig-Holstein ERDF OP, 2014-2020. Final report
Begleitende Evaluierung des Operationellen Programms EFRE Schleswig-Holstein 2014-2020 (OP EFRE) Evaluierung der Prioritätsachse 2
The evaluation examines PA2 measures on SME competitiveness up to August 2022.
To assess the measures for SMEs competitiveness financed under the OP, their results and effects.
Analysis of relevant documentation. Expert discussion with the key actors of the OP. Literature analysis. Analysis of monitoring and statistical data. Standardised survey of final beneficiaries for each measure analysed (33% response rate on average).
SO 3 “Seed and Startup Fund II”: 86% of startups have developed new products or services or significantly improved existing ones and 76% have improved processes. SO 4 ”Productive investments of SMEs in structurally weak areas and SME funds”: the large majority of respondents (more than 80%) report a significant impact on employment growth in their company. SO 5 “Internationalisation of SMEs”: Almost half of the companies surveyed have increased their exports through participation in trade fairs. In addition, 25% of them increased their domestic sales. Projects contribute to all the cross-cutting principles (sustainable development, equal opportunities and anti-discrimination, gender equality). The support is particularly valued in terms of corporate development and innovation services, and the procedures adopted are considered appropriate by beneficiaries.
Continue the measures contributing more to innovation (SSF II, Invitations promotion, equity funds).
Interim evaluation of the Technical Assistance OP, 2014-2020, in Slovakia
This is an ongoing evaluation of the Technical Assistance OP, 2014-2020, in Slovakia, monitoring the financial and physical progress made in implementation as at December 2017.
To assess progress in implementing the OP on the basis of financial and output indicators, the extent to which the Performance Framework milestones have been achieved and the management of the OP.
Desk analysis of relevant policy documents.Interviews with relevant stakeholders.Analysis of data from the monitoring system and the MA administrative database.
At the end of 2017, 20% of the PA1 budget had been spent and 13% of PA2, but there is no risk of non-compliance with the N+3 rule.Physical implementation indicators do not always seem properly designed and data for assessing progress are sometimes lacking. For only one of the 7 indicators there is a risk of not meeting the target.There are delays in awarding contracts, this on average taking 57 days. But only 4 of the applications for payment considered exceeded the 90 days within which payments are supposed to be made.
The achievement of some indicators could not be fully assessed because of the slow progress in implementing some measures.
Revise some result indicators in order to avoid non-compliance;put in place communication standards internal to the MA to avoid delays in implementing the OP.
Evaluation of Support for Business under the West Wales and the Valleys and East Wales ERDF OPs, 2014-2020
ERDF Support for Business Evaluation: Final Report Summary
Oliver Allies, Paula Gallagher, Michelle Hollier, Michael Pang, Benjamin Reinertz, Prabhat Vaze, and Paul Woodcraft
The evaluation investigates the effects of measures supporting enterprises under the OP.
To assess the outcomes of the measures, the effectiveness, and the support received by beneficiaries in relation to cross-cutting themes (CCTs). To identify potential differences in the impact of different measures and in different business situations.
Analysis of monitoring data and programming documents. Analysis of statistical data. Survey of 1,040 businesses (13.5% response rate). Counterfactual analysis (administrative data from the Business Structure database). Interviews with key stakeholders. 6 case studies.
Marketing and promotion and business planning were the main types of support received, but among more established businesses there was a greater emphasis on customer engagement. The support offered through collaboration with research institutes had a very limited effect on improving the commercialisation of products, especially for micro enterprises and particularly because of a lack of financial support and alignment with the Innovate UK Fund. Despite this, commercialisation of new products has led to increased sales according to 65% of beneficiaries. 643 additional business have been created because of measures supporting start-ups. The CIE shows that over 10,000 new jobs were created as a result of ERDF funded support. The impact is mainly evident in the first two years after receiving the support and the effect diminishes over time. Support for SMEs to access new markets led to a 78% increase in turnover growth over three years for the companies supported. However, turnover growth for all the measures is smaller than employment growth, so the average productivity gap between Wales and the UK remains significant.
Since the intensity of support varies between the different measures and details of this were not available, it was not possible to take account of this in the analysis. Some beneficiaries were accidentally excluded from the surveys, so the results might be biassed. The typical limitations of CIE are also mentioned.
Continue the approach adopted to integrate CCT in service provision. Ensure close alignment and access to the Innovate UK Fund. Ensure digital support for businesses, as this is central for competitiveness. Include sales generation as a performance indicator of business support, alongside employment growth.
Evaluation of "EXI - Gründungsgutscheine" financed by the Baden-Württemberg OP, 2014-2020
Evaluationsbericht zur Förderrichtlinie "EXI Gründungsgutscheine"
Dr. Friedrich Scheller and Stefan Feldens
ISG Institut für Sozialforschung und Gesellschaftspolitik GmbH. Weinsbergstr. 190, 50825 Köln. Contact Person: Gerald Engasser (Head), Tel: 0711 123-3614, E-mail: ESF@sm.bwl.de
The evaluations assesses the "EXI-Gründungsgutscheine” (start-up voucher) funded by the ESF Baden-Württemberg OP 2014-2020, offering consultancy to entrepreneurs (e.g. assessing business plans, suggesting financing possibilities).
To assess the effectiveness of the start-up voucher measure.
Methods used are: desk research; interviews with experts and some of those responsible for implementing the measure; two surveys of beneficiaries (the first with a 22% response rate for a sample of 1,056, the second with a 29% response rate for a sample of 2,076); two case studies of promoters; and analysis of monitoring data.
Up to June 2018, 13 projects were launched, involving around 6,900 people split nearly equally between men and women. Half were between 25 and 39 and a third between 40 and 54; a quarter had a migration background. Overall education levels were high (40% had a university degree or PhD and 32% had completed VET). Most were employed when they received the voucher.The consultancy was particularly useful for reviewing business plans, for calculating expected costs and revenues and for providing information on finance. Professional and communication skills of consultants were assessed positively.Two thirds of the total surveyed implemented the advice after receiving it.The main sectors in which start-ups were active were health and welfare, ICT, technical and professional services and hospitality.
Evaluation of the Youth Employment Initiative 2018 - final report
Evaluacija Inicijative za zapošljavanje mladih u sklopu Operativnog programa Učinkoviti ljudski potencijali 2014. – 2020. za 2018. godinuIzvješće u tijeku provedbe
This report presents the final results of the evaluation of youth integration measures in Croatia. They relate to two IPs under the Efficient Human Resources OP:8ii/YEI on NEETs aged 15-29 at risk of being LTU (financed by YEI).8ii/ESF on LTU NEETs aged 15-29 (financed by ESF).The measures cover workplace training, education, employment subsidies, public works and self-employment subsidies.
To assess the impact and effectiveness of measures helping young people into employment.
Analysis of documentation;analysis of monitoring data;in-depth interviews with stakeholders;online and telephone surveys of participants and employers;counterfactual analysis (Propensity score matching).
The measures supported were already in operation before but used to be financed by the national budget and IPA.Financial continuity was considered a key factor in implementation while the heavy administrative burden and complexity of procedures were the main challenges.Measures financed by YEI on education and self-employment subsidies are still to be implemented.Only half of the target of 28,818 YEI participants was achieved by end-2018 (though 95% for measures financed by ESF).37% of YEI participants and 20% of ESF ones were employed immediately after exit and 1% in both cases were in education. The target was reached for YEI but not for ESF.6 months after exit, 68% of YEI participants (50% for ESF ones) were employed and 4% in education (2% for ESF), more than the target in both cases. The employment rates were highest for Employment subsidies (81% for YEI and 84% for ESF).YEI employment subsidies had the largest estimated net effect on employment (27% higher than for the control group); workplace training is estimated to have had no effect and public works a negative effect.75% of both YEI and ESF participants were offered a job after leaving the programme.70% of YEI and ESF participants reported their job was decent and were satisfied with it. 42% of the jobs were permanent and 94% full-time. Only 50% reported that their salary was in line with their expectations. Vocational training was included in half the jobs.Stakeholders consider YEI has significantly helped young people to enter the labour market and that the funds were spent on those most in need of support.Monthly costs per participant vary from EUR 292 for employment incentives to EUR 673 for self-employment support.The average cost per participant employed after 6 months was EUR 244,000 for YEI vocational training and EUR 17,630 EUR for YEI employment subsidies (EUR 69,000 for ESF employment subsidies).
Only the current situation is assessed and the final impact of the programme can only be assessed after its completion. For measuring the outcome of participants after 6 months, data from the PES were used, but when missing or incomplete, data were complemented by results from the survey of participants. Data for the control group came from the PES database.Due to financial constraints, the statistical margin of error for the sample size has been set to 2-5% (whereas the European Guidelines recommend one not exceeding 2%), which will affect the reliability of the results obtained when the analysis is carried out.
Contact details should be compulsory to include in the monitoring data.Further steps are needed to identify young NEETs and, in particular, the inactive (currently not covered by the measures examined).Another evaluation should be carried out once all projects are completed (which should also examine the impact of YEI on overall youth employment and unemployment rates).Further analysis is required to assess dead-weight and creaming to check if the funds are concentrated on those most in need.The workplace training measure should be reduced in importance in the future given the poor CIE results for it.Alternative financing should be found for measures under the responsibility of the Ministry of Education if funding is not possible under YEI 8ii.
Evaluation of South Wales metro phase 2 supported by the West Wales and the Valleys and East Wales OPs, 2014-2020
This interim evaluation examines the South Wales metro phase 2.
To analyse the baseline data against which the ERDF-funded operations can be evaluated, to assess how the South Wales metro phase 2 has been funded and procured, to identify good practice, lessons learned, and to assess the cross-cutting issues.
Documentation analysis.Telephone survey of over 1,000 residents.Interviews with stakeholders.Analysis of public transport data (timetables, rolling stock types, etc.) and rail demand data (Office of Rail and Road station entries and exits and LENNON ticket sales data).Analysis of Census, the Business Register and Employment Survey, and the Annual Population Survey/Labour Market Survey datasets.
It is estimated that by 2040, 177,900 tonnes of CO2 emissions will be reduced as a result of the South Wales Metro Phase 2 (SWMP2).Out of £738 million of project budget, £158 million come from the ERDF.In March 2020, rail infrastructure was transferred from (UK) Network Rail to Transport of Wales, which simplified the funding arrangements.For bidders, the procurement process was intense, but it allowed contractors to propose their own solutions and commercial arrangements, and overall was positively assessed.The early engagement with prospective funders accelerated the process of securing funding.The requirements of cross-cutting themes are well understood across the SWMP2 operations, and two cross-cutting theme champions have been appointed, providing the advice, guidance and data collection required to facilitate the achievement of objectives.
The demand and revenue forecasts and business cases for SWMP2 are based on pre-Covid-19 data. Travel-to-work data are over 10 years old and should be interpreted with caution. Staff changes at the Welsh Government, Transport for Wales and partner organisations have made it difficult to obtain detailed knowledge of the metro project.
Make use of existing infrastructure and ensure that SWMP2 could be extended in future.Create a public library of all project documents.Develop a clearly defined monitoring and evaluation framework based on a logic map.Create more focused monitoring and evaluation plans to document cross-cutting activities.Monitor community benefits in a quantified way.
Evaluation of support for VET for migrants under the Baden-Württemberg ESF OP, 2014-2020
Evaluationsbericht zur Förderlinie "Mehr Menschen mit Migrationshintergund in berufliche Aus- und Weiterbildung"
ISG Institut für Sozialforschung und Gesellschaftspolitik GmbH, Weinsbergstr. 190, 50825 Köln, scholz@isg-institut.de
The evaluation examines ‘The way to success: Vocational training – more people with migration background in vocational education and training’ programme, aimed at awareness-raising and counselling of families and the young people concerned as regards VET opportunities.
To assess how, why and for whom the ‘Way to success’ programme on VET for migrants works or does not work, its implementation and the initial results.
Documentation analysis.Expert group interview.Analysis of monitoring data.Analysis of data on the staff employed.Survey of participants.Survey of professional consultants.Survey of project promoters.Two in-depth case studies.
The financial size of projects varies significantly, but funds seem to have been used well.Mutual trust and knowledge of the native language are important to reach the target group.There are difficulties in recruiting qualified professional counsellors because of contracts being temporary and pay being limited.Discussions with young people are mostly about careers and job applications.Those supported include people with low skills or without vocational training.The average consultation time is around 16 months and the drop-out rate is 7.5%.56% of those using counselling services are women.63% of participants up to 2020 have obtained a qualification.Around half of young participants completed a traineeship or training programme during or after consultation.Participants were generally very satisfied with the consultations.The funding has directly helped to ensure equal opportunities and a reduction in discrimination.
Since the data on beneficiaries were collected by means of a questionnaire, because of education and linguistic reasons, the data validity for individual characteristics is limited. In addition, monitoring data cover only those who received consultation of at least 8 hours.
Ensure more human and organisational resources for advertising the availability of the programme in the migrant community.Establish and maintain a support network.Organise workshops or other types of support activity for trainees and trainers in respect of entrepreneurship.Continue the funding for the programme.
Provence Alpes Côte d'Azur OP, 2014-2020 - Evaluation of the implementation of the ESF
Evaluation de mise en oeuvre du Fonds Social Européen 2014-2020
This is the mid-term evaluation of the ESF part of the ERDF –ESF OP for Région Provence Alpes Côte d’Azur 2014-2020.
To assess the implementation and the progress of the operations of the OP co-financed by the ESF.
The evaluation adopts a mix of qualitative and quantitative methods:Desk research;analysis of relevant policy documents;analysis of monitoring data;in-depth interview with stakeholders;survey of applicants and beneficiaries;focus groups with stakeholders.
The implementation of the OP is progressing satisfactorily, although some parts are delayed.Over 7,700 participants were involved in programme activities (training courses, guidance services or specific pathways to facilitate return to work). 59% were unemployed, 31% of these long-term unemployed.Around 2,900 participants (1,400 women) obtained a qualification at the end of their participation; 1,800 of them were employed six months afterwards (around 890 of them women).The main obstacles to OP implementation concern the complexity of administrative procedures (particularly indicators and reporting rules) and delays in verifying payment claims, as well as a reduced number of project proposals due to bad management of the programme in the previous programming period.
Data on outcome indicators go up to December 2016.
The communication strategy should be more targeted and should follow a thematic approach. Future calls for proposals should be based on a clear schedule.The reference guide for beneficiaries should be adjusted to better cover the specific features of ESF regulations. Training sessions and workshops should be provided, linked with the envisaged launch of calls for proposals.The indicator monitoring system should be improved by checking the quality and reliability of programme data, including the procedures for data aggregation.
Evaluation of the Ceuta ESF OP: objectives and results
Evaluación de los Objetivos/Resultados del Programa Operativo FSE de Ceuta 2014-2020
The evaluation assesses the implementation of the Ceuta ESF OP 2014-2020, which focuses on support of sustainable and quality employment and labour mobility (TO8) and promotion of social inclusion (TO9).
To assess the OP implementation, effectiveness, efficiency and impact.
Desk review;analysis of monitoring data and statistical data (Eurostat – INE, Instituto Nacional de Estadística);interviews with MA.
The OP’s implementation (financial, milestones, objectives set) deviates to some degree from what planned due to delays in implementation and difficulties in certifying expenditure, although the objectives are expected to be reached by the end of the programming period. There is a significant discrepancy between physical and financial indicators mainly because of delays in updating the financial situation due to external audits.Needs identified at the beginning of the period are still relevant and there are no changes in the target groups to be reached.The LB is implementing the OP properly through its experience and know-how of the ESF and knowledge of the needs of beneficiaries.1,111 people benefitted from ESF measures (53% of planned). Of these, 88% were unemployed, 44% of them long-term and 55% with lower secondary education or below. 9% were integrated into the labour market after the intervention.Horizontal principles have been taken into account.
Analysis on the impact was not carried out because long-term results have not yet materialised and data from official statistics (Eurostat and INE) are of limited reliability.
Review the target values of IP 9.i result indicator as they are too high;update the monitoring system with the current values of financial and physical indicators.
Mid-term evaluation of the ERDF, 2014-2020
Midtvejsevaluering af den danske regionalfond 2014-2020
The interim evaluation focuses on three ERDF priorities: Enhanced innovation in SMEs; More growth companies; More energy and resource efficient SMEs within the ERDF 2014-2020 in Denmark.
Desk and data analysis.Quantitative analysis.Interviews with project promoters and administrators.
The projects are generally more focused on long-term effects than in the previous period.The resources committed are 61% of the total in priority 1 (Enhanced innovation in SMEs), 59% in priority 2 (More growth companies) and 38% in priority 3 (More energy and Resource efficient SMEs). Priority 3 requires a greater effort to complete within the period.Research institutes consider that the funded innovation partnerships provide limited research value, as the institutes focus on the benefits of long-term research, while companies focus on short-term market needs.The minimum requirement of the OP for at least three SMEs for each innovation collaboration is difficult to meet (priority 1.1a). Administrative costs are considered by respondents to be too high relative to the grant that can be received under priority 1.1b.The selection criterion of 20% growth in the number of employees for three consecutive years adopted under priority 2 is difficult to achieve.Under priority 3 projects vary in terms of achieving objectives in respect of the reduction in energy use, greenhouse gas emissions and materials. But there is greater uncertainty about the achievement of objectives in relation to climate change.Most companies supported have incorporates a green business model, but relatively few have incorporated material and energy savings into their production and sales organisation.
There is a need to:reduce the number of common programme indicators and to introduce project performance indicators;give greater autonomy and responsibility to project operators;make partnerships more attractive for research institutes;reduce the number of the companies required for a partnership for innovation;increase the synergy between priorities;require applicants to include in their project proposals a validation from an external expert of the realism of the objectives;request more innovation and collaboration with large enterprises in projects for energy and resource saving.
Strategic evaluation of methodological methods in the Marche ERDF OP, 2014-2020
The evaluation examines the relevance of interventions in the Marche ERDF OP 2014-2020, which has 7 PAs and is mainly aimed at promoting smart, sustainable and inclusive growth by boosting innovation and R&D, though also at increasing SME competitiveness, improving broadband infrastructure and services, increasing energy efficiency, adapting to climate change and preserving the environment.
To assess the relevance of interventions and their coherence with the Europe 2020 priorities and to consider the methodological approach that will be applied to the strategic evaluation in the second part of 2022.
Desk research of programme documentation;analysis of secondary data (ISTAT, Eurostat, Banca d’Italia);benchmarking;SWOT analysis.
PA 1 - interventions planned to strengthen RTDI appear coherent with the specific features of the economy and with the overall objective of increasing the propensity of companies to carry out R&D, especially in knowledge-intensive sectors.PA 2 - Improving access to information technologies: the finance allocated appears coherent with the need for digital services for the communication between companies and public administration.PA 3: the significant share of funding allocated to SME competitiveness is justified due to the adverse effects of the economic crisis.PA 6 - Protecting the environment and promoting the efficient use of resources: the allocation of funding is reduced for some of the specific objectives (e.g. promoting regional tourism and the cultural heritage) which are considered important.
At June 2019, the implementation of the OP was still limited so it is not possible to assess the validity of the OP strategy in relation to the results achieved.
For the methodological approach to be applied to the 2022 strategic evaluation:the evaluation should examine the results obtained by the OP as regards the specific objectives of the individual PAs;a mapping of secondary data sources is necessary to define precisely the information needed;the size of the OP budget does not justify the use of macroeconomic models to assess impact.
Implementation of Priority Axis 9, Workforce competitiveness, of the EU Structural Funds Investment OP, 2014-2020
Europos Sąjungos fondų investicijų veiksmų programos 9 prioriteto konkretaus uždavinio „Padidinti dirbančių žmogiškųjų išteklių konkurencingumą, užtikrinant galimybes prisitaikyti prie ūkio poreikių“ įgyvendinimo pažangos vertinimas
Didžioji g. 25-6, LT-01128 Vilnius, Tel: 852154075, 852153969info@bgiconsulting.lt, www.bgiconsulting.lt
This evaluation assesses the 8 measures implemented under Priority Axis 9 of the OP for EU Structural Funds investment, in order to improve future interventions.
To assess the relevance, efficiency, effectiveness and impact of the measures implemented under Priority Axis 9 of the OP, with a view of improving future interventions.
Theory of change;analysis of literature;analysis of statistical data;counterfactual analysis of the “Human Resources INVEST LT+” measures.
The measures assessed overall met the companies’ needs.The implementation logic of the measures did not fully meet the needs of employees.The measures had a positive impact in the short-term on workforce competitiveness but in the long-term their impact is limited.Most of the training activities were coherent, of high quality, and helped to improve knowledge and skills.Participation in activities funded under measure “Human Resources INVEST LT+” had no effect on the income of participants.
At the time of carrying out the evaluation, progress in implementing the different measures was far from uniform.
Further increase the competitiveness of the workers involved;ensure that the planned measures are in accordance with the Sector Strategy Papers;involve institutions in the development of measures such as apprenticeships and training.
Summary of system projects implemented by NCBR under Measure 1.5 of the Innovative Economy OP 2007-2013
Podsumowanie projektów systemowych realizowanych przez NCBR w ramach Działania 1.5 POIG
Michał Przybyłowski, Piotr Tamowicz, Katarzyna Bachnik, Marta Mackiewicz, Agnieszka McCaleb, Łukasz Konopielko, Marcin Łaskawski, Wojciech Szapiel
ECORYS Polska Sp. z o.o. ul. Solec 38 lok. 105 00-394 Warszawa, TAYLOR ECONOMICS SP Z O O, ul. Trzy Lipy 3, 80-172 Gdańsk
The evaluation is concerned with 6 projects implemented under Measure 1.5 of the Innovative Economy 2007-2013 OP. This is the final report.
To assess the OP project selection system, including the logistic process, as well as the adequacy and efficiency of the support granted to particular projects.
The evaluation adopted mainly qualitative methods:desk research;analysis of relevant documents (e.g. competition documents, applications submitted);in-depth interviews with the MA and beneficiaries;expert panels (6);benchmarking analysis (including 9 selected European Programmes);The data used are: primary data from interviews and panels and secondary data from tender documents and other EU programmes.
Assessment of the project selection system (including logistics):there is a lack of commonly accepted optimum number of selection criteria;the support of two foreign experts widened the perspective;delegation of financial assessment to one of the experts proved to be effective;applying different weights to criteria or unequal scales of assessment complicates the assessment and weakens transparency.Assessment of the support granted to particular projects:BRIDGE ALFA focused on riskiest projects, transferring most of the risk to the public sector. Its continuation may adversely affect competition in the private market;DEMONSTRATOR+ helped reduce the barriers to technology transfer and to minimise the risk from commercialising R&D activities;INNOLOT had a beneficial impact on strengthening cooperation between business and science and improved the overall competitiveness of the aviation industry.
During the selection phase there is a need to assess whether a company will handle the cash flow throughout the duration of the project;to minimise the risk that effects of the project will be transferred abroad, there should be an obligation to have an international partner;seminars to improve planning and analytical skills of SMEs might make it more likely that successful projects are designed;the establishment of online networking platform might enable experts and organisations to look for suppliers and partners for their consortia and to cooperate with different stakeholders;publication of case studies might facilitate the dissemination of knowledge on programmes.
Demand assessment for repayable financial instruments in the area of KED OP higher education
Ocena zapotrzebowania na instrumenty zwrotne i finansowe w obszarze szkolnictwa wyższego PO WER
Adam Miller (kierwonik badania), Szymon Piotrowski, Patrycja Rumińska
Intelligent specialisations - water economy
Inteligentne specjalizacje - EkoNomia wody
Michał Bil, Jan Charkiewicz, Paweł Dembiński, dr hab. Wojciech DziemiaNowicz, Izabella Jurkiewicz, Agnieszka Laskowska, Magda Lewandowska, Sylwia Patron-Ravida, Klaudia Peszat
Geoprofit Wojciech DziemiaNowicz, ul. Wiązana 26 F; 04-680 Warszawa, geoprofit@geoprofit.eu - ECORYS Polska Sp. z o.o. ul. Solec 38 lok. 105, 00-394 Warszawa, email: ecorys@ecorys.pl, biuro.rzeszow@ecorys.pl
Impact assessment of R&D support under the Franche-Comté Massif de Jura and Bourgogne OPs, 2014-2020
Évaluation d’impact portant sur le volet recherche et innovation des Programmes Opérationnels 2014-2020 FEDER-FSE Franche-Comté Massif du Jura et FEDER-FSE Bourgogne
The evaluation examines R&D support under the Franche-Comté Massif de Jura and Bourgogne OPs, 2014-2020.
To assess the impact, relevance and coherence of the OP strategies in respect of R&D, and their implementation aspects in order to identify lessons for the 2021-2027 period.
Monitoring and context data analysis;review of literature;20 interviews with institutional actors;40 telephone interviews with projects leaders;6 case studies;benchmarking on the basis of a comparative analysis with 3 other EU regions.
The strategies of the two OPs are coherent with EU, national and regional innovation strategies. Support is strongly focused on a limited number of areas of specialisation (micro-technical, sustainable food).The Franche-Comté Massif du Jura OP is concentrated on a limited number of projects closely linked to the regional innovation strategy. The Bourgogne OP is focused on small operations, supported by regional co-financing which has resulted in management bottlenecks and was more affected by the overhaul of regional policies after the merger of the two regions.Measures under TO1: in the Franche-Comté Massif du Jura OP have performed well, while they have shown little progress in Bourgogne. This is because the merger of the two regions in 2015 penalised Bourgogne more, the focus of the OP being on platform projects and incubators, for which there was a widespread sense of legal uncertainty and a structurally unfavourable environment with a dispersed research ecosystem with little capacity to attract beneficiaries with strong project management potential.The funding allocated by the Bourgogne OP does not have sufficient critical mass to generate a measurable leverage effect to counteract the unfavourable regional innovation situation.The first results suggest the ERDF has helped to consolidate research activities; encouraged the filing of patents; led to economic spin-offs, though many of the projects do not have direct commercial application; but produced disappointing results as regards the creation of innovative enterprises, due in part to high entry costs, particularly in industry.
Given the early stage of most of the projects, the effectiveness of the OPs can only be discerned from short-term results or the potential impact.
Maximise the leverage effect of funding.Take account of feedbacks when defining the future performance framework.Carry out studies on the benefits and risks of targeted financial instruments for innovation.Continue thinking about securing collaborative platforms.Formulate appropriate calls for projects.Adequately communicate the calls for projects.Strengthen support for project leaders.
Evaluation of the implementation of the Regional Health Programme, “Early detection and prevention of diabetes among residents”, under the Mazowieckie OP, 2014-2020
Ocena realizacji Regionalnego Programu Zdrowotnego Samorządu Województwa Mazowieckiego „Program wczesnego wykrywania i profilaktyki cukrzycy wśród mieszkańców województwa mazowieckiego”
Krzysztof Gutta, Tomasz Gutta, dr. hab. Joanna Hołub-Iwan, Katarzyna Gutta, dr. Magdalena Jażdżewska-Gutta, prof. dr. hab. Aldona Frączkiewicz-Wronka, prof. Krzysztof Strojek
The evaluation examines the implementation and results of the Mazowieckie Regional Health Programme for early detection and prevention of diabetes.
To assess the implementation of the programme, the achievement of the main and specific objectives, the effectiveness and efficiency of the measures. To identify problems/barriers, service quality, stakeholder views and unintended effects.
Desk research.Statistical and monitoring data.Interviews with representatives of MA and IB.Interviews with service recipients and medical staff.Interviews with beneficiaries.Survey of participants in programme.Survey of medical staff.Delphi panel.Case studies on programme participants and projects.Benchmarking of other regional diabetes programmes.SWOT analysis.One focus group with staff of MA and IB.
Service quality and participant satisfaction were very high in relation to multiple aspects (communication, punctuality, privacy, fit to needs). Staff training was also highly rated.Weaknesses in monitoring were identified, because it was difficult to enforce beneficiaries’ reporting obligations.Despite reaching less people than planned, the programme made a positive contribution to detection, as diagnoses led to 18.2% of additional cases identified (target was 20% increase).Stakeholders suggest rebalancing future measures towards prevention/education, continuing staff training, and widening eligibility to younger at-risk adults (e.g., 35+), given epidemiological trends.COVID-19 significantly constrained delivery (recruitment of 60+ people, ability to train staff, facility operations), prompting schedule extensions and online formats.
Problems with incomplete reporting by beneficiaries due to the absence of sanctions, which limited monitoring data. Baseline values for some indicators had to be estimated, reducing precision. Surveys of senior people proved challenging, and the response obtained by interviews with institutional stakeholders is limited.
Introduce enforceable reporting requirements (via call rules and grant agreements) to secure monitoring/evaluation data.Review and, if needed, adjust performance indicators so they better reflect the intervention logic and enable full monitoring of progress/effectiveness.Broaden the target group to include younger at-risk adults (e.g., 35+) and place greater emphasis on population education and patient self-management, while continuing staff training.Reduce bureaucracy and simplify recruitment. Make education a required component for screened participants.Strengthen follow-up through periodic post-programme surveys (every 6–12 months).Embed contingency arrangements for disruptions (e.g., remote/online training, flexible timelines) given pandemic-related risks to delivery.Diversify outreach channels to seniors and under-served areas to improve equity of access.
Evaluation of the Building Better Opportunities programme under the England ESF OP, 2014-2020. 6th annual report
Building Better Opportunities Evaluation. Annual report Spring 2022
This is the sixth annual report on the implementation of the Building Better Opportunities (BBO) programme supporting social inclusion and employment of people far from the labour market and with disadvantages. Data used go up to September 2021.
Analysis of documents.Analysis of monitoring and statistical data.Survey of 333 participants.6 case studies.
To assess the main results achieved by the projects and how the effects of Covid-19 have been tackled.
As of September 2021, around 145,000 participants have been engaged, 48% of them disabled and 63% from jobless households.Around 118,000 have exited the projects, of which 62% have reported their condition immediately after exiting. 36% of them moved to employment, 32% moved to education or training, and 15% moved from economic inactivity to job search.According to the survey, 10-12 months after the end of the project the share of those in employment and of those not working and claiming benefits decreased, whereas the share of those in volunteering activities and those in education/training increased. Most of participants reported increased confidence and wellbeing.After Covid-19, the BBO programme supported the development of digital skills of participants to tackle isolation, especially of those with disabilities or mental issues.Extensive partnerships have been created to reach targets, and this proved to be effective.
Ex-ante study of the Centro Regional Programme, 2021-2027, in Portugal
Avaliação Ex Ante e Avaliação Ambiental Estratégica do Programa Operacional Regional do Centro 2021-2027
António Manuel Figueiredo, Artur Costa, Carla Melo, Dinis Marques, Filipa Barreira, Maria Elisa Pérez Babo, Mariana Rodrigues
Quaternaire Portugal, www.quaternaire.pt
The report assesses the programme’s structure and objectives. The analysed version of the programme was made available to the evaluation team in November 2022.
To assess the programme’s relevance and coherence, the adequacy of the resources available, its consistency with environmental sustainability objectives, and whether expected impacts are achievable.
Desk research on programme documents and on regional, national and EU strategic documents.Exchanges with relevant stakeholders involved in the programming.
The structure of the Programme is quite rigid, but this can be mitigated by the MA during implementation, by prioritising regional needs.Some of the regional strategic priorities and needs – demographic decline, territorial resilience, training and cultural activities – are not adequately covered by the programme.The programme is internally coherent and is aligned with the major national and European strategic objectives.Compared to the OP of the previous programming period, there has been a reduction of resources, especially for PO1 and PO4. Resources allocated to cultural activities (PO5) are not proportionate to their importance for regional strategies. The areas of energy and climate transition, instead, are duly taken into account and are allocated a substantial amount of funds.The output and result indicators selected for the Programme are adequate, but not sufficient to capture the heterogeneity of the operations planned.
Ensure, through programme management and implementation, that the rigidity of the Programme is mitigated, with a view to better addressing regional priorities.Pay special attention to demographic decline and territorial resilience during the implementation phase.Use the Theory of the Programme as developed by the evaluation team as a support framework for the monitoring of the Programme.Ensure that the interventions concerning PO1 and PO4 are duly targeted and circumscribed, in order to maximise their effectiveness in light of the limited resources available.Adopt further indicators to improve the scope for the monitoring of the Programme and make use of strategic monitoring alongside the more traditional monitoring of results.
Evaluation of the implementation of the sustainable development principle
Ewaluacja wdrażania zasady zrównoważonego rozwoju w ramach PO IR 2014-2020
ul. Solec 38 lok. 105, 00-394 Warszawa, email: ecorys@ecorys.pl, biuro.rzeszow@ecorys.pl, Tel: +48 22 339 36 40
The Polish Smart Growth (SG) OP 2014-2020 is aimed at furthering the innovativeness and competitiveness of the Polish economy. The report assesses the implementation of the sustainable development principle and environmental-friendly activities.
To evaluate the application of the principle of sustainable development in the implementation of the SG OP 2014-2020 and to identify good examples of practice.
The evaluation uses quantitative and qualitative methods and is based on existing data (international, EU and national strategic documents) and data from CAWI/CATI surveys and case studies.
Environmental issues are not usually the factor determining whether or not an application is rejected.Indicators focus on outputs and results rather than implementation and are often not logically linked to the project implemented project; indicators reported by applicants are sometimes difficult to verify.The principle of sustainable development and its application is not always well understood by beneficiaries.
Projects had only just begun to be implemented.
Detailed information on the sustainable development principle should be provided to actual and potential beneficiaries.The project monitoring system should be reviewed to include more details of the implementation of the sustainable development principle.A list of indicators should be introduced to monitor the implementation of the sustainable development principle.
Evaluation of the impact of measures to improve energy efficiency and build a low-carbon economy, financed under PA4 of the Małopolskie OP, 2014-2020 - phase
Ewaluacja wpływu działań podejmowanych w ramach 4 osi RPO WM na lata 2014-2020 na poprawę efektywności energetycznej i budowanie gospodarki niskoemisyjnej w regionie – etap II
The evaluation examines the effects of projects under PA4 of the OP on the development of a modern energy sector, able to ensure the energy security of the region and improve the quality of life of the residents.
To assess the impact of support for the development of a modern energy sector in the region.
Desk research (programme and strategic documents, studies).Analysis of monitoring data and official statistics.14 interviews with representatives of institutions involved in the implementation of projects.3 case studies.Survey of 400 inhabitants, 114 municipalities, 65 beneficiaries and 79 Social Assistance Centres.Spatial correlation analysis.Evaluation workshop.
The measures have significantly increased the use of Renewable Energy Sources (RES) in the production of electricity, through photovoltaic installations.Energy efficiency improvements were significant in public and residential buildings and contribute to reductions in heating and electricity consumption.Modernisation of almost 340k inefficient heating sources was a key factor in air quality improvement, and helped reduce PM10 and PM2.5 emissions substantially.13% of households cannot access modern energy services. The Clean Air Programme has been an effective instrument for reducing energy poverty through various support measures.
The assessment of the impact was partly hampered by a lack of appropriate indicators at the project level and by a change in the methodology for estimating dust emissions in the region.
Further support photovoltaic installations and the development of renewable energy sources as well as energy efficiency projects.Maintain the focus on measures to improve air quality, including the modernisation of heating sources, given their proven impact on reducing emissions.Expand measures such as the Clean Air Programme to tackle energy poverty.
Third report on the Active inclusion priority in the Netherlands ESF OP, 2014-2020
Evaluatie ESF Actieve Inclusie, Derde verdiepende onderzoek over 2014-2017
Adriaan Oostveen, Miranda Witvliet, Marjolein Bouterse, Luuk Mallee
Bureau Bartels B.V., email: info@brtls.nl, internet: www.bureaubartels.nl - REGIOPLAN, email: info@regioplan.nl
The evaluation examines the ESF Active Inclusion programme, which started in 2014 and finances multiannual projects to increase integration into the labour market of those distant from it. Projects are implemented by labour market regions (AMRs), the Employee Insurance Agency (UWV) and the Ministry of Justice and Security (JenV).
To assess how effectively and efficiently funding has been used and the extent to which the expected results have been achieved. This is third report on the programme and is intended to investigate the drivers of labour market reintegration.
TBE, consisting of assumptions on instruments, target group, contextual factors, what works, for whom and in what circumstances;analysis of financial progress and output data;interviews with all 37 project coordinators;13 case studies on individual subprojects;analysis of data on labour market reintegration of participants in the programme.
By mid-September 2018, 92% of funding was allocated.The added value of projects consists of more intensive counselling to vulnerable people, in many cases going beyond the life of the projects.The cases studies shed light on the mechanisms which contribute to active inclusion and which are similar for different target groups: a strong focus on individual counselling and on finding a successful job match through job coaching.Targets for output and result indicators have been exceeded: 36% of participants in AMR projects found a job or improved their employment situation (target 20%), and 55% of those in UWV projects did so (in line with the target). Progress of projects managed by JenV is in line with the output indicator target for 2023.The projects are executed in line with assumptions in the Theory of change. They focus largely on people distant from the labour market; for refugees intense counselling and wage subsidies work effectively; for those with disabilities, direct work placement and job coaching are effective.Overall, training and job coaching are effective instruments if targeted at the right groups of participants and if deployed in a timely way.
Result indicators (participation in education) of JenV projects could not be calculated. Not all the assumptions of the theory of change could be tested. It was not possible to isolate the effects of other regular activities not financed by the ESF.
Simplify ESF measures in administrative terms.
Thematic evaluation of support for education infrastructures under the Campania ERDF OP, 2014-2020
Rapporto di valutazione tematica POR FESR 2014-2020: risultati degli investimenti sulle infrastrutture scolastiche
Nucleo di Valutazione e Verifica degli Investimenti Pubblici (NVVIP), Regione Campania
The evaluation aims to assess the implementation and impact of infrastructure investments in the education sector under the OP. Data used goes up to November 2023.
To assess the effectiveness of infrastructure investments in educational facilities and their impact on the socioeconomic context in Campania, as well as to identify lessons learned from implementation.
Analysis of monitoring data. Analysis of official statistical data. Analysis of literature.
The initial budget for Axis IX (educational infrastructure) was later reduced from €149.6 million to € 105.7 million also because of COVID-related reprogramming. As at November 2023, certified spending is below the target (€74.6 million). Most output targets remain unmet, such as digital aids for distance learning (44,000 vs 61,200) and the surface area addressed by interventions (25,350 sqm vs 47,850 sqm). Little progress of output indicators is also caused by COVID. The Polo Universitario San Giovanni a Teduccio is recognised as best practice in terms of integration of education infrastructure projects with broader regeneration measures.
Data available were not fully up-to-date.
Make sure that education interventions align with the 5 policy objectives of the 2021-2027 programming period and the RRF. Adopt a more integrated and collaborative approach with stakeholders to ensure cohesive planning, and effectiveness and efficiency of the measures. Replicate the model of “Polo Universitario San Giovanni a Teduccio” in future interventions, by integrating educational infrastructure projects with broader urban regeneration policies. Prioritise digital education and the provision of digital tools in future programmes.
Az Ifjúsági Foglalkoztatási Kezdeményezés elemzése
OP Economic Development and Innovation 2014-2020
Evaluation of the YEI implemented under the Economic Development and Innovation OP, initially targeted at young jobseekers registered for at least 6 months, but extended in June 2016 to those registered for at least 4 months and from January 2018 to all young jobseekers. Only 4 regions are eligible for YEI support: Southern Transdanubia, Northern Hungary and Northern Great Plain and Southern Great Plain. In the other regions, the Youth Guarantee is financed from national sources. The measures supported include training, allowances for travel costs and housing and support for business creation, a well as wage subsidies for employers hiring young jobseekers.
To assess the effectiveness and efficiency of the programme in 2015-2017.
The following methods were used based on monitoring data:Econometric estimation: a probit regression model to identify the factors affecting the successful completion of the programme.Counterfactual impact evaluation (regression discontinuity design).
Young people living in cities are more likely to be NEETs.Successful programme completion is greatly influenced by educational attainment level.Multiple participation in the programme reduces the chances of successful completion.By end-2017, over 40,000 participants were involved in YEI or around 115% of the commitment, and half completed the programme (84% of the 2023 target).The result indicator targets are also likely to be achieved. 8,456 were in employment 180 days after completing the programme (78% of the target).The cost per participant was very similar in YEI and Public works programmes (around EUR 3,000).The programme significantly contributes to the probability of being employed after 28 days but the impact diminishes over time.Wage subsidies and support to business creation have the most impact on the probability of remaining in employment after 180 days.
Ongoing interventions were assessed instead of finished programmes. Participants were not selected randomly and the composition of treated and control group differs.
Assessment of efficiency of ESF support within regional component of 2007-2013 OP HR
"Badanie skuteczNości wsparcia EFS realizowanego w ramach komponentu regionalnego PO KL 2007-2013"
Małgorzata Zub, Robert Kubajek, Paweł Penszko, Maciej Gajewski, Jan Szczucki
Policy & Action Group Uniconsult Sp. z o.o., ul. Kierbedzia 4 , 00-728 Warszawa, Tel: (022) 256 39 00, e-mail: biuro@pag-uniconsult.pl