| Criteria | Type | Notes |
|---|---|---|
| User-facing | ||
| Installation instructions exist | binary | |
| Usage instructions exist | binary | |
| Tutorials exist | binary | Must have explanation of inputs, outputs, and test data set in a worked example |
| Test data sets and results available | binary | |
| Developer-facing | ||
| Open source | binary | |
| Has software tests | binary | Eg, unit tests |
| More than 90% code coverage reported | binary | |
| Clear channels for software maintenance and issues | binary | Eg, GitHub issues, author contact information |
Software standards
Framework for evaluating software standards in Plasmodium genomics
PGEforge aims to foster an ecosystem of high-quality, user-friendly tools that can be seamlessly integrated into genomic analysis workflows. One of the biggest challenges is the variability and lack of systematic assessment of existing tools, which often do not adhere to best practices in software development, including FAIR standards, maintenance, and usability.
Working towards this goal, a robust software standards evaluation framework was formulated to guide the development and assessment of tools used in Plasmodium genomic data analysis from both the end-user and developer perspective. This framework is crucial in addressing the variability and challenges associated with existing software tools but also to guide development of new tools, ensuring that they meet high standards of usability, accessibility, and reliability.
‘Ideal’ software practices
One of the primary objectives of this framework is to define ‘ideal’ software practices that are not tool-specific but applicable across a range of genomic analysis tools. These practices encompass:
- Comprehensive documentation
- Ease of installation
- Reliable and maintainable software
Additionally, during the 2023 RADISH23 hackathon, focused discussions highlighted the following software practices that are not necessarily essential, but are “nice-to-have’s”:
- Uses standard data input formats
- Computationally efficient
- Informative error handling
- Multiple languages for tutorials
- Minimal dependencies
- Modular code (eg split into functions)
- Well annotated code
These practices can guide development of new tools and/or improvement of existing tools.
Evaluation criteria
To implement these standards and facilitate tool evaluation and development, PGEforge has developed a set of measurable criteria that can be applied to evaluate the performance and usability of various tools. There are two categories:
- User-facing: criteria to evaluate the tool from an end-user perspective, for example whether installation instructions are available and easy-to-follow
- Developer-facing: criteria to evaluate the tool from a developer perspective, for example whether unit tests are implemented
The evaluation criteria encompass the following key themes, in line with the ‘ideal’ software practices for both end-users and developers:
- Quality and comprehensiveness of documentation
- Simplicity of installation processes
- Quality assurance and maintenance
Every criteria is scored on the following scale:
- 0: Criteria not fulfilled
- 1: Criteria fulfilled but not entirely
- 2: Criteria fulfilled
This is then translated to an end-user score and development score for the tool.
Tool evaluation
Every Plasmodium genomic analysis tool can be evaluated against these objective software standards to provide these scores. The resulting evaluation matrix and overview of each tool can be found here.