The EFAS partner survey regarding the general satisfaction with the EFAS service, products and performance during the year 2021 was sent out to all partners in January 2022. A total of 54 responses were received, compared to 54 responses received in 2020, 36 in 2019, and 43 in 2018. The survey was, as in previous years, anonymous.

 

The survey responses are summarized according to the following categories:

• Demographics
• Overall Satisfaction
• Skill, model performance and trust
• EFAS Services
• EFAS Products and added value
• Future Developments

 

Demographics

The distribution of country origin for the survey respondents was similar in 2020 and 2021, but there was a higher percentage of respondents from the "others" category in 2021.

Pie-chart showing the country origin of the EFAS partners that answered the survey.

 

Most of the survey participants were both a data provider and an EFAS partner. 

Roles of survey respondents.

 

 

Overall Satisfaction

No major changes in the overall satisfaction were reported in comparison to the previous four years. However, it is worth noting that there was an increase in satisfaction in overall performance during the past 12 months in comparison to the 2020 survey, in which 7% of the participants rated the performance as “low”. 

 Average user response on overall satisfaction with the EFAS.

 

Skill, model performance and trust

Most of the participants (67%) rated the information about the EFAS model performance and forecast skill as high. However, 4% of participants indicated that they did not know this information was available. 

User response about the model performance and forecast skill.

 

EFAS services

The EFAS bulletin and the annual hydrological report are the most read, with 94% and 85% of participants respectively reading them at least sometimes. The detailed assessment report was read the least; 20% of participants did not read the report, and 11% of participants did not know it existed.

Average user response to EFAS informative resources.

 

Survey participants provided similar responses in 2021 as in 2020 when asked to identify their main benefits of being an EFAS partner.  Overall, the main benefits were the forecast, followed by the notifications, learning practices during annual meetings, and observed and forecasted precipitation. In 2020, the least voted benefit was "Training by DISS", and this option was not provided in the 2021 survey. New options for the 2021 survey were: (1) Europe-wide overview on ongoing and forecasted events, and (2) To be able to see/monitor the situation/forecast of neighboring basins. For both of these new options, 52% of the participants indicated that these were among their main benefits of being an EFAS partner. 

User responses on the main benefits of being an EFAS partner.

 

Participants were asked to comment on if they have noticed an improvement/worsening from the upgrade of the flash flood product in 2021. Their responses are included below. Overall, the responses indicate that the updates generally improved the flash flood product. However, survey participants identified several limitations of the current model and methodology and indicated that the flash flood notifications may be issued too often in some locations. It is also worth noting that many survey participants joined EFAS in 2021 and therefore did not have experience with the previous flash flood product.

Positive Feedback:

• I have noticed the improvement - a smaller amount of false notifications
• Yes, I have noticed an improvement of flash flood forecast product in 2021.
• A little improvement
• Yes
• We have noticed improvement
• Number of flash flood forecasts is more realistic since the upgrade
• The number of Flash FNs decreased - it is excellent the handling with triangles is very comfortable
• It appears to have improved slightly. The improvement appears to be better in predicting the locations of flash flooding.
• The forecast of a flash flood is very useful for us. Our warnings are based on our forecast and on EFAS products and notifications. Thanks to EFAS!
• Improvement for Croatia, especially mountain region and southern region (inner parts of Dalmatia near Cetina river). Also improvements for southern parts of Bosnia and Herzegovina (rivers Neretva and Trebižet).
• It has become better, more convenient.
• There is an improvement in that fewer notifications are issued now reducing the probability of false alarm.
• Better overview between the current Flash Flood reporting points on the basis of which notifications are issued.

Neutral Feedback:

• The performance is generally good, however we had an extreme event on 25th December in the South East of Ireland - up to 90mm of rain in less than 8 hours.  On review, this was predicted by the AROME model but not by others (ECWMF, Harmonie etc.).
• No special feedback. I however notice a steady reliable availability of the EFAS-product which is important. Forecast quality is relatively poor but for us the notification is more important than the exact measurement vs forecasted discharges.
• The number of false alarms has been reduced, but work still needs to be done to reduce the number of overestimated flash flood announcements.
• Any improvement. I think there were several Mediterranean events not included
• In general, not a clear change for our country area (Romania), we consider that flash flood forecast product have a similar performance as before.

Negative Feedback:

• The flash flood notifications are issued too often. The information in the e-mail is confusing / doesn't correspond to the expectation of many users (area susceptible to landslides isn't something often a relevant criteria / information for the user)
• Users of flash flood products should be aware of the methodology of the ERIC points and its limitations, e.g. ERIC calculates also 24h precipitation, which is much longer than usual perception of flash floods.
• I consider that the number of flash flood notifications issued for my country is extremely high. In some cases, there is an overlapping regarding the locations concerned. Moreover, I find the upgraded feedback as more complicated, it requires more time to be filled than the previous one.
• From my point of view, you have to introduce another layer into the EFAS portal, on which, within the framework of the forum, any issues related to the independent installation of the LISFLOOD model and its modernization can be discussed. Time is going very quickly. Without such action LISFLOOD will not be seriously improved in near future.

Other Feedback:

• We are new at EFAS in 2021. So the experience is not very big yet.
• I, personally, started to serve in 2021, thus, I can not compare the progress of 2021 with 2020.
• Actually, I don't have the experience yet, as we are partner only for a very short amount of time
• N/A, my organization is EFAS partner since Sept. 2021. In result no real experience is available at this time.
• We are not able to comment this since we are partners from 2021.
• Flash floods are not part of our work/agenda
• The flash floods are not characteristic feature of our rivers
• N/A
• Not sure.

 

For the question, I (my organisation) compare EFAS forecast with those from other provides, 39% answered no, 44% answered yes, and the remaining percent did not answer the question. Participants who answered yes were asked to specify which systems they use and how they compare with EFAS.  Many different systems were used:

• National System (3 participants) 
• Sava Flood Forecasting and Warning System (2 participants)
• We compare EFAS forecasts with the products of the Hungarian Hydrological Service for the Danube and the Tisza, as well as the products of the DELTARES FEWS platforms for the Sava, Velika, Zapadna and Južna Morava basins.
• All EFAS forecasts are compared with forecasts been worked out in regional national forecast centers.
• AEMet and other meteorogical services
• Italian CP hydrological models
• ICONPANDARHEI
• 16 competent flood services of the federal states of Germany (using different systems), usually they do compare well with EFAS
• Forecasts of DWD in Germany and governmental forecasts for Rivers.
• FFGS, SEE-MHEWS
• National system hydrological warning in Slovakia providing by SHMU
• Forecast precipitation is compared with Met Éireann's HARMONIE forecast and the AROME Forecast

 

Partners were asked to rank their preference on different forms of training with a value of 1 indicating their first choice and a value of 4 indicating their last choice. Overall, the partners prefer to receive training in the form of short online tutorials (average rank = 2.18), followed by regular webinars (average rank = 2.20). Online documentation and workshops during the annual meeting were the least preferred with average ranks of 2.79 and 2.82, respectively. These responses were similar to those received during the 2020 survey.

User responses about the preferences for receiving training. Note that in the 2020 survey partners were asked to check all that apply, but in the 2021 survey they had to rank them. 

 

The participants were asked in which topic(s) they would like to receive training, which resulted in the following responses:

• LISFLOOD (6 responses):
  • Installation of LISFLOOD model and main problems of this action
  • New achievements in the problem of modernization LISFLOOD model
  • Training in LISFLOOD also in native language, not only in English
• The Climate Data Store and downloading/using EFAS hydrological data (6 responses):
  • How can I download EFAS hydrological data?
  • Downloading real time EFAS simulations, both for discharge and models state parameters, in gridded format, and recommendations / best practice on how to read and process these raw data, in order to integrate / compare with the national system data and to generate new specific products for our country.
  • How to access and download EFAS hydrological data (simulations and forecasts 30 days after real-time) from the CDS (Climate Data Store)
  • Training in the Climate Data Store in native language, not only in English
  • Interaction and use of the EFAS data with other forecasting systems i.e. Sava Flood Forecasting and Warning System.
  • How to use EFAS Hydrological data from the CDS
• Jupyter notebooks (3 responses):
  • Training in Jupyter notebooks in native language, not only in English
• Use of the EFAS products (2 responses):
  • Handling /user guide of EFAS products for personnel (non-hydrologists) in our 24/7 situation centre of the federal civil protction authority - note that such a guide should have to focus on application and interpretation of EFAS by non-hydrologists / non-meteorologists, who quickly need to take decisions.
• Evaluation of forecasts and model skills (2 responses): 
  • Detailed information about forecasts skill (the methodology, examples, how this skill can be interpreted from the user's point of view) and planning developments in this area
• Seasonal and subseasonal forecasts
• EMS Mapping for satellite imagery support.
• Deactivation and reissue of notifications if the peak time is shifted
• Overall capabilities of the EFAS system for newly organized group of the employees in our organization and detailed one for the advanced ones.
• Training for beginners
• EFAS training for my institution in our offices.

 

EFAS products and added value

The ratings of the added value of the different EFAS notifications were similar in 2021 to those of the previous survey years. In general, most survey participants rate the added value of the different notifications as either high or very high, and there have not been major changes in the responses between survey years.

Average user response to the added value of notifications.

 

Satisfaction with each of the EFAS centres in 2021 was predominantly positive and similar to that of 2020. For the 2021 survey, the participants were not asked to rate the ease of providing feedback using the in-built form. 

Average user response about the satisfaction of the work of the different EFAS centers and the easiness/difficulties to give feedback.

 

In general, participants rated the usability or added value of the EFAS products higher in 2021 than in 2020. Participants were especially positive about the added value of the 6-hourly flood forecast information. The added value of the rapid risk assessment layer and of the sub-seasonal forecast were the least popular.

Average user response to the functionality/added value for some of the EFAS products/features.

 

Overall, awareness of the different EFAS products and features was similar in 2020 and 2021. For both years, the highest awareness was for the possibility to activate EMS (Emergency Management Service) Mapping for satellite imagery, that the LISFLOOD model is now open source, and about the training material on the EFAS-IS.  It is also worth noting that from 2020 to 2021 there were large increases in awareness about: (1) the possibility to request training of the EFAS Dissemination Centre, and (2) the EFAS wiki pages. However, awareness about the Jupyter notebooks and what can be done with them remained lowest among the products/features.

Average user response to awareness about some EFAS products/features.

 

From 2020 to 2021, there was an increase in usage of the simulated runoff, snow, and soil moisture data provided by the Sensor Observation Service (SOS) and the Climate Data Store (CDS). In addition, there were increases in the percentage of respondents who indicated that they think their organization will use the data in the future.

Average user response to the use of some EFAS products.

 

The survey participants were then asked to comment on: (1) what prevents them from using this data, and (2) what EFAS data they would like to have access to. These responses are summarized below:

• Did not know about these services. Lack of time and know how to download and integrate in information tools and decision support systems.
• I would like to have access to the following EFAS data: simulated river discharge (also forecast), simulated snow and simulated soil moisture.
• I think I have not yet reached the full potential of Copernicus CDS
• The main problem that up to now regional forecast centers have no any real support from official power organizations. First of all from Roshydromet.
• Technical equipment. Trained staff.
• Data of accumulated precipitation
• How to webinar
• We are in a process of establishing data downloads from EFAS to our national system
• I have access to data and I am satisfied with it. I have only one question concerning soil moisture. Is it any plan in future to add real time measurements, e.g. from some providers or from the satellite?
• We produce or get such data ourselves in finer spatial resolution
• 1. Systems to use data still at conceptual stage. 2. Recorded precipitation (24-hour accumulation from radar)

 

Overall, participants were less willing to provide feedback or analysis/articles for EFAS in 2021 than in 2020. However, 52% of the participants were willing to provide feedback for flash flood notifications, and 39% were willing to provide feedback for informal notifications.  In contrast, only 13% of participants were willing to write a short article, and 9% were willing to conduct a short verification analysis. 

Average user response about the willingness to provide feedback or analysis/articles for EFAS.

 

Future Developments

Survey respondents provided the following responses when asked what they are missing in EFAS:

• Special layer in EFAS portal where any EFAS partner can discuss the problems connected with LISFLOOD installation
• More detailed sub-seasonal and seasonal outlook products
• Accumulations of forecasted precipitation for more intervals, not just for the entire forecast period (e.g. 24, 48, 72 hours, 5 days)
• Maps with accumulated recorded precipitations in the last: 1, 3, 5, 10 days.
• During the period with snow cover, maps with accumulated simulated snowmelting contribution, both for the last 1, 2 and 3 days period, and forecasted for the next 24, 48, 72 hours, 5 days.
• Physical meetings

 

Between 2020 and 2021, there was a decrease in the percentage of survey participants who would like to receive both quality flags for hydrological/meteorological data and data in a standard format (harmonized time step and unit, quality controlled), which might also be due to the fact that a higher percentage of participants were no EFAS data providers. However, over 50% of the participants replied either "Yes" or "Maybe" to both questions.

 

Average user response about obtaining specific EFAS product/data.

 

The survey participants were also asked to indicate their thoughts on EFAS and open data. Responses were generally neutral, but more participants were in favor of making EFAS public without the flood notifications than of making the entire system publicly available.

Average user response about making publicly available the EFAS products/forecasts and the entire system.

 

 

Lastly, partners were also asked on any other comment they might have. These responses are included below:

• I wrote a lot. If 75% of written suggestions will be done it will be a very good result.
• Nice to work with all of you!!!!!
• On the EFAS and open data issue, I think that selected existing EFAS products but also new more specific and useful products dedicated for the public could be made publicly available.
• Thank you for hard work on improving the EFAS functionalities
• Thank you for organising the annual meetings and providing different webinars.
• About EFAS public availability: it would be nice to have some info available for everyone, but in some simple form, taking care not to turn general audience into self-proclaimed "scientist" and "forecasters" because real alerts should come from local authorities
• To make the data or system open, first run a pilot, then consider results carefully.
• N/A, my organization is EFAS partner since Sept. 2021. In result no real experience is available at this time.