Key Points
- All papers submitted to the ISWC 2023 Resources Track must include a resource availability statement. Papers without a resource availability statement may be desk rejected.
- Resources must be available along with the initial submission, in anticipation of the review process.
- Reviewers are requested to review the resource as well as the paper.
- Papers should be self-contained.
Please find more details below.
Usability and Reproducibility
Usability of a resource as well as a clear understanding how a resource came about is a key component of progress in scientific research, to use such a resource competently in new research independently or to extend such a resource presented in scholarly publications. In the context of computer science, providing access to such material, such as ontologies, code, and datasets, greatly facilitates progress in research and reproducibility. This supports the independent verification of results, and can increase the impact of a work by allowing other researchers to reuse and build upon such resources. An important goal for ISWC is to continue improving the reproducibility of all papers, and usability and reusability of the resources described in the Resource track papers.
Resources vs. Paper Content
All papers submitted to the ISWC 2023 Resources Track need to be self-contained, meaning that the body of the paper should provide a clear statement of their claims and clear argumentation regarding how these claims are substantiated by evidence. This includes any details that are important for interpreting the resource, such as the specifications of the machines used for experiments on the resource, key statistics about datasets, important configurations or hyper-parameter settings, details about the metrics used, details about how data was collected, etc.
The resource must include the materials reported in the resources paper that cannot feasibly be included in the paper itself. Please refer to the “Resources of interest” section of the CfP for this track for a list of categories of resources that may be suitable for this track.
We strongly encourage authors to make the resource available under open licenses and in repositories that provide long-term availability and have a persistent URI (Zenodo, Figshare, institutional repository, etc.). See the “Availability” section of the Resources Track CfP for details.
Purpose of the Resource Availability Statement
While it has become increasingly common for papers to include links to supplemental material to help verify the results they present, often this is done in an ad hoc manner that differs from paper to paper, even in Resources Track papers.
To make it easier for readers to find the resource, to understand what is provided and what is not provided, and why, we ask all authors to include a standardized Resource Availability Statement at the end of their paper that:
- lists all resources presented in the paper;
- points to locations where those resources can be found;
- if applicable, justifies why certain resources cannot be made available (e.g., the privacy, technical or legal concerns involved), or indicates the conditions under which they can be made available (if any).
Format of the Resource Availability Statement
The Resource Availability Statement should be placed at the end of the paper, just before the References (and before Acknowledgements, if present). It counts within the 15 page limit. It should be formatted as an inline paragraph with an italicized heading:
Resource Availability Statement: Source code is available …
In LaTeX, this can be achieved with the markup:
\\paragraph*{Resource Availability Statement:} Source code is available …
In Word, authors can simply replicate the formatting shown in the box above.
Content of the Resource Availability Statement
The Statement should point to the location of all resources made available as presented in the paper. The authors must be careful to ensure that it is clear (either from the statement, or the resource linked from the statement) what versions, configurations, dependencies, steps, etc., are needed to use the resource. Some example templates of statements to include are:
- Source code for our System X is available from …
- Datasets X, Y, Z are available from …
- Query sets X, Y, Z are available from …
- Source code for Baselines X, Y, Z is available from …
- The raw data used to generate Figure X, Y and Tables A, B are available from …
Pointers to resources can be provided either directly in the text of the statement, as footnotes, or as bibliographical references; for example:
- Source code for OurX is available from Github.1
- Source code for OurX is available from Github [1].
- Source code for OurX is available from Github at https://github.com/ThisIsUs/OurX.
If multiple resources are available at one location, statements can be combined, for example: “Datasets A, B and query sets X, Y are available from …”. As per CfP, resources must not be submitted via EasyChair. In the unlikely case that a resource cannot be published openly, the authors must clarify this in the statement, summarizing why it cannot be published, and include details about the conditions under which the resource can be accessed; for example:
- Source code for our System X cannot be made available due to plans to commercialize the software.
- Dataset X cannot be made available as it incorporates private user data. However, a suitably anonymized version may be made available under a licence, upon contact with the authors.
Examples of Resource Availability Statements
Resource Availability Statement: Source code for OurX and the queries used in Section 4 are available from Github.1 The OurOntoX ontology is available from Zenodo [4]. The MyHealth dataset cannot be made available as it incorporates private user data.
______________________________________________________________
1 https://github.com/ThisIsUs/OurX
References
…
[4] This, I.U. OurOntoX Ontology. Zenodo doi:10.5281/zenodo.12345678. (2022) https://doi.org/10.5281/zenodo.4035223
…
Review of the Resource Availability Statement
Papers without a resource availability statement may be desk rejected.
Reviewers will be asked to review the Resource Availability Statement in order to verify that it does not omit any resources that are presented as novel resources in the paper, that the resources are indeed available at the locations indicated, and that all reasonable efforts have been made to make all relevant material available.
Reviewers are expected to review the content of the material. Nonetheless, it is important that the paper is self-contained, reporting on the resource and the context of its creation (refer to the full list of criteria in the CfP). Reviewers thus have to be able to access the resource during the review process, and are expected to make comments about the material. This means that authors must provide reviewers access to the resource from the original submission. Since this track is NOT anonymous, revealing the identity of the authors is expected.
