Stephen Pinfield Affiliation:
The University of Sheffield Funder:
Part of RoRI (the Research on Research Institute) a collaboration of the Wellcome Trust, Digital Science, Leiden University and the University of Sheffield Project abstract:
Full project title: Transforming 'Excellence': Definitions, applications and alternatives to excellence in the evaluation, management and measurement of research Programme: The project is a part of the programme of work being undertaken by RoRI (the Research on Research Institute), a partnership founded by the Wellcome Trust, Digital Science and the Universities of Sheffield and Leiden leading at international consortium of funders, academics and technologists carrying out transformative and translational research on the global research system. The Transforming Excellence project aims to assess the ways in which the idea of excellence is currently used by funders and other key actors in the research ecosystem (including publishers, university managers, and research evaluation agencies, as well as researchers themselves), and to evaluate the functions it serves in specific practices (including funding decision making, publishing, quality evaluation etc), in order to explore its possible futures. Key research questions:
1. How is excellence perceived by research funders and policy agencies, taking account of different stages of the research process, disciplines, geographical regions etc? 2. How is the concept of excellence currently operationalised and used by research funders and policy agencies in different contexts (stages of the research process, disciplines, geographical regions etc)? 3. What is the purpose of the use of the concept of excellence as used by research funders and policy agencies? 4. What evidence is there of the impact of different operationalisations of excellence e.g. in terms of funding allocations, researcher behaviours etc? These questions will also be extended to other actors in the research system e.g. publishers, librarians as well as researchers themselves. Last modified:
08-01-2021 TRANSFORMING EXCELLENCE DEFINING YOUR DATA What data will you collect or create during the project?
Data collected during the research will consist of interview recordings and transcripts; policy, strategy and other documents; process descriptions and maps. These will be analysed using qualitative data analysis tools, which will include analytical coding and commenting. How will the data be collected or created, and over what time period?
Data will be collected working with case study sites in various forms during the project lifetime, currently planned from July 2020 to December 2021. It is possible that the project will be extended beyond the current expected end date. What formats will your digital data be in?
- Documents: Documents collected as data for the project (e.g. policy documents) may take a variety of forms e.g. PDF (.pdf), Word (.docx) or other formats. They may be converted into other formats for the purposes of analysis or preservation. Where necessary, documents are expected to be converted to Text (.txt) files for preservation.
- Audio files: Data from interviews will be captured as MP3 recordings. MP3 files will be used as working copies but for any longer-term retention necessary, files will be converted into the Ogg Vorbis (.ogg) file format.
- Transcripts: MP3 files will be used to transcribe fully the data for the purposes of analysis. Transcriptions will be produced and managed as Microsoft Word (.docx) and where necessary converted to Text (.txt) files for preservation.
Approximately how much digital data will be generated during the project?
- Videos files: Some videos may be part of the data analysed. In most cases, however, these are expected to be publicly available e.g. on websites, rather than managed by the project. Where the project does store videos they are expected to be in an MP4 format.
- Other documentation: Other documentation gathered for produced as part of the research meeting different formats to be decided. Consideration will be given to storage as formats are agreed.
Data generated by the project is expected to be no more than 10 GB, with the largest files being MP3 recordings (estimated to be no more than 100 MB per file). PDF documents are expected to vary in size, up to 2MB but may often be smaller. Are you using pre-existing datasets? Give details if possible, including conditions of use
Data is expected to be drawn from pre-existing documents and other qualitative data as described above, but not pre-existing assembled datasets. LOOKING AFTER YOUR DATA How will you make data easier to understand and use?
Data will be recorded in a spreadsheet designed to summarise data being otherwise for the project. Where will you store digital and physical data during the project?
Data will be stored on secure password protected drives managed by the University of Sheffield, Leiden or Wellcome Trust; and/or on password-protected machines of project team members with any files containing personal information also being individually password protected. How will you name and organise your data files?
TBC How will you ensure data is backed up?
Data will be stored on centrally-supported servers, provide by Sheffield, Leiden or Wellcome Trust, either in-house or cloud provision, with high-level security, environmental control, backup and incident management protocols in place. How often will you check your backup files?
Backup provision is provided automatically by the above providers. Will you use extra security precautions for any of your digital or physical data?
Any personal data will be kept to a minimum, expected to be personal name and organisational role. This will be protected as described above. ARCHIVING YOUR DATA What data will be archived (stored on a long-term basis) at the end of the project?
All of the data assembled and produced by the project is expected to be archived at the end of the project. How long will the data be stored for?
Data will be retained for three years following last use. Where will the archive be stored?
Data will be archived on the institutional research data servers of Sheffield or Leiden.
Who will archive the data?
Project team members will be responsible for managing data and preparing it for any archiving. If you plan to use storage other than a repository, who will be responsible for the data?
Not applicable. SHARING YOUR DATA How will you make your data available outside the research group after the project?
In reporting the research, we will name organisations involved in the study when describing our research methods. Individuals taking part will, however, remain anonymous. In reporting, we will not associate any individual or organisation with any particular quotations or other data, except where the quotations or data are already in the public domain or where we have explicit permission to do so.
Data from the interviews will not be shared publicly except in the form of anonymised short extracts from the interviews selected for illustrative purposes in reports of the research. Any extracts included in publications, conference presentations etc will be appropriately anonymised in order to ensure that confidentiality is maintained.
In specific exceptional cases, it may be necessary to attribute quotations to individuals or their organisations (e.g. if used to support the case studies in the previous phase of the research). In these cases, the research team will obtain explicit consent for this. Will you make all of your data available, or are there reasons you can't do this?
See above. How might you make more of your data available?
What licence might you attach to your data to say how it can be reused and shared?
Not applicable. IMPLEMENTING YOUR PLAN Who is responsible for making sure the plan is followed?
Project Sponsor, Stephen Pinfield, Project Manager, Thomas Franssen, reporting to the RoRI Founding Partners' Board. How often will the plan be reviewed and updated?
Quarterly What actions have you identified from the rest of this plan?
- Consideration of formats for various types of data
- Possible conversion of formats
- Agreement of repository or repositories for archiving (likely to be one or both of Leiden and Sheffield)