Semantic Web Challenges

Pretrained language models (LMs) have advanced a range of semantic tasks and have also shown promise for knowledge extraction from the models itself. Although several works have explored this ability in a setting called probing or prompting, the viability of knowledge base construction from LMs remains underexplored. In the 2nd edition of this challenge, participants are asked to build actual disambiguated knowledge bases from LMs, for given subjects and relations. In crucial difference to existing probing benchmarks like LAMA, we make no simplifying assumptions on relation cardinalities, i.e., a subject-entity can stand in relation with zero, one, or many object-entities. Furthermore, submissions need to go beyond just ranking predicted surface strings and materialize disambiguated entities in the output, which will be evaluated using established KB metrics of precision and recall. The challenge comes with multiple tracks depending on the language model choice and entities.

The Question Answering over Linked Data challenge, better known as QALD, ran ten times with a focus on factual Knowledge Graphs (KGs). This year, we will shift the focus on Knowledge Graph Question Answering (KGQA) over scholarly data. In particular, question answering (QA) over scholarly knowledge is due to the complex struc- ture significantly more challenging than encyclopedic or common-sense knowledge QA. Thus, KGQA systems increase the accessibility of the Web in general. This year, we will run two tasks. First, SciQA tests 100 complex question-answer pairs over the Open Research Knowledge Graph – ORKG. Second, DBLP-QUAD evaluates over 10.000 question-answer pairs on a version of the DBLP KG. We will leverage HuggingFace as the benchmarking platform.

SemTab 2023 is the fifth edition of the Semantic Web Challenge on Tabular Data to Knowledge Graph Matching that ran successfully for four years at ISWC 2019-2022: http://www.cs.ox.ac.uk/isg/challenges/sem-tab/. This year’s edition introduces a new task in the accuracy track, and will use new datasets.

Despite the development of several ontology reasoning optimizations, the traditional methods either do not scale well or only cover a subset of OWL 2 language constructs. As an alternative, neuro-symbolic approaches are gaining significant attention. However, the existing methods can not deal with very expressive ontology languages. Other than that, some SPARQL query engines also support reasoning, but their performance is still limited. To find and improve these performance bottlenecks of the reasoners, we ideally need several real-world ontologies that span the broad spectrum in terms of their size and expressivity. However, that is often not the case. One of the potential reasons for ontology developers not building ontologies that vary in terms of size and expressivity is the performance bottleneck of the reasoners. SemREC aims to deal with this chicken and egg problem.

The third edition of this challenge includes the following tasks.

Task-1. Ontologies

Submit a real-world ontology that is a challenge in terms of the reasoning time or memory consumed during reasoning. We expect a detailed description of the ontology along with the analysis of the reasoning performance, the workarounds if any, that were used to make the ontology less challenging (for example, dropping a few axioms, redesigning the ontology, etc.), and the (potential) applications in which the ontology could be used. We will be evaluating the submitted ontologies based on the time consumed for a reasoning task, such as classification, and the memory consumed during reasoning.

Task-2. Systems

Ontology/RDFS Reasoners. Submit an ontology/RDFS reasoner that uses neural-symbolic techniques for reasoning and optimization. We will evaluate the submitted systems on the test datasets for scalability (performance evaluation on large and expressive ontologies) and transfer capabilities (ability to reason over ontologies from different domains) based on precision and recall. We expect a detailed description of the system, including an evaluation of the system on the provided datasets.

SPARQL query engines that support entailment regimes such as RDF, RDFS, or OWL 2. We expect a detailed description of the system, including an evaluation of the system on the provided datasets.

The organizers: Gunjan Singh (Ph. D student, KRaCR Lab, IIIT Delhi), Raghava Mutharaju (Assistant Professor, KRaCR Lab, IIIT Delhi), Pavan Kapanipathi ( IBM T.J. Watson Research Center, USA)