BenchmarkDP consultation on software evaluation in digital preservation

JavaScript isn't enabled in your browser, so this file can't be opened. Enable and reload.

Welcome to the BenchmarkDP consultation on software evaluation in the digital preservation field !
With this consultation we intend to get insights into the current state of software evaluations in the digital preservation field. We are interested in the current practices of software evaluations, challenges faced, limitations and advantages they bring to the running businesses. Finally, we are interested in better community-driven evaluation initiatives and want to gauge community members’ willingness to participate in such initiatives.

Who are we ?
BenchmarkDP is a research project funded by the Vienna Science and Technology Fund (WWTF). It has started in November 2012 and is running until summer 2016. You can find more information at http://benchmark-dp.org

Why participate ?
By participating in this consultation, you can help us to get a better overview of the software evaluation practices and results in the digital preservation field. This will enable us to invest our limited resources in the place where they are most needed and will have the biggest impact.

Who should participate ?
This consultation is open to everybody who wants to participate. We are interested to cover categories that are related to software tools and evaluations of those tools. So if you are a practitioner working with software tools in a memory institution, or a software developer developing solutions for digital preservation problems, or a researcher in the digital preservation area doing research related to software tools, you are most welcome to join.

How long does it take ?
This is a short consultation. It shouldn’t take you more than 10 minutes to finish it.

Privacy
This consultation doesn’t collect any private data unless you decide to provide your contact details. The only data it collects are the answers to the questions. The collected data will be used for the research purposes and summarized results will be publicly available. The data is collected through Google Forms and after the consultation is closed it will be stored on a secure server.

This consultation is open until 30th of November 2015.

1) What organization do you work for ? *

Research funder

Big data science

Digital preservation vendor

Government agency

Publisher or content producer

Data intensive industry

Memory institution or content holder

Small or medium enterprise

University

Other:

2) What is your job responsibility ? *

Mark those that apply

Preservation of digital content

Research

Software development

Technical support (managing software and / or hardware components)

Management

Consulting

Other:

Required

3) What is your education background ? *

Mark those that apply

STEM - science, technology, engineering and maths (maths, computer science, physics, software engineering, …)

Library and information sciences

Humanities and social sciences (sociology, arts, literature, …)

Other:

Required

4) How important are software evaluations and evaluation results for your and the work of your organization ? *

not important at all

very important

5) How do you and/or your team evaluate software tools ?

Mark those that apply

Perform own experiments on a tool.

Reproduce results from experiments done by others (outside of my organization).

Review results from experiments produced by others (outside of my organization).

Other:

6) How are you and your organization affected by the current software evaluation practices and results in digital preservation? *

Mark those that apply

They have very little or no effect on our digital preservation activities.

The available results save resources for us as we don’t have to perform evaluations on our own.

Additional resources are needed for evaluations as the available results are not sufficient.

It is challenging to improve current software tools.

Our decisions around software are usually not based on rigorous evidence.

Other:

Required

7) What are you using evaluation results for ?

Mark those that apply

Results are used for making operations decisions in my organization

Results are used as a part of an exploratory research activity.

Results are used for a research output evaluation.

Results are used to enable improvements in certain tool.

Other:

8) When you have done experiments to evaluate software tools, how did you conduct them ?

Mark those that apply

Experiments were properly documented.

Input parameters were carefully varied to explore changes.

All the relevant outputs were carefully recorded.

Experiments were done as a part of the preservation planning process supported by the preservation planning tool Plato.

Experiments were done using the Planets Testbed web application.

Experiments were done with help of other tools and systems.

Publicly available datasets were used.

Internal private datasets were used.

All the experiments documentation and results were made publicly available.

Other:

9) Which publicly available results have you used or produced?

This is an optional field where you can provide author and the title or links to the documents containing used results.

10) Various challenges in software evaluation can make it difficult to get reliable results. Please rank the challenges according to how important they are for you. *

For each challenge assign a rank from 1 to 7. 1 is the highest rank. Each number can be used only once.

Lack of published evaluation results.

Lack of dedicated resources to conduct evaluations.

Lack of tools and systems to support evaluation.

Lack of properly annotated datasets (ground truth).

Lack of knowledge for doing proper evaluation.

Lack of collaboration in the community.

Lack of properly specified evaluation requirements.

11) Considering your work, rank the listed tool categories according to the need of having rigorous evaluation. *

For each category assign a rank from 1 to 6. 1 is the highest rank. Each rank can be used only once.

Migration

Emulation

Characterization

Web Harvesting

Identification

Validation

12) Rank the following aspects of software quality according to the importance for your work. *

For each aspect assign a rank from 1 to 3. 1 is the highest rank. Each rank can be used only once.

Resource efficiency: How efficiently does the tool perform what is meant to do?

Coverage: Does the tool perform all the operations that are required to for a certain scenario?

Correctness: Does the tool perform operations which it claims to perform in a correct way?

13) If the digital preservation community were to establish an initiative for software evaluation, would you be able to get involved? How?

Mark those that apply

By providing requirements for evaluation.

By providing annotated datasets.

By performing evaluations and validating results.

By hosting evaluation results.

By supporting the organization and coordination of the effort

By compiling lists of tools to be evaluated

Other:

Here you can provide any other opinions or facts you consider important for this topic but were not covered by the above questions.

If you are interested in this topic and would like to participate in possible future interviews you can provide your email address below.

Submit

Clear form

Never submit passwords through Google Forms.

This content is neither created nor endorsed by Google. Report Abuse - Terms of Service - Privacy Policy

Forms