Overview

Hybrid Intelligence (HI) is an emerging system design paradigm in which artificial intelligence (AI) augments, as opposed to replacing, human intelligence. Although there is an increasing emphasis on the idea of HI in the AI literature, there is a lack of systematic methods and metrics for developing HI systems.
The Hybrid Intelligence Competition (HI Comp) series aims to support the development of high-quality HI (Human-AI) teams, by exploring the possible benefits, risks, and consequences of collaboration between humans and AI systems.

HI Comp aims at pushing the state-of-the-art in Human-AI collaboration and teaming, and at generating a first repository of scenarios for researchers and practitioners to guide the development and evaluation of HI teams.

Call for participation

HI Comp delves into an interdisciplinary subject and requires minimal prerequisite technical expertise for participation.
We invite participation from all segments of the AAMAS community.
All participant teams can participate to the competition before and during the AAMAS 2024 conference in Auckland, New Zaeland, in May 2024, without prior registration. All the competition details are are available at the following link: Competition details.
See Submission section for more information on how to submit.

Challenge: Fundamental qualities of HI teams

The theme of this first iteration of the HI Comp is: fundamental qualities of HI teams.

The main task for the participants of the competition is to formulate HI scenarios that yield best and worst quality HI teams. The participants are presented with a set of use-cases involving HI teams. Analyzing these use cases, the participants are asked to develop best and worst case usage scenarios, based on the quality attributes from a recent Quality model for HI (Human-AI) Teams (see image below, more details are provided in the Competition details). The scenarios, reported in a specific template, are then evaluated by a jury of HI experts composed of the competition organizers. The participants are ranked based on the best and worst case scenarios they formulate.

All the competition details are are available at the following link: Competition details.

Figure: Quality attributes of HI Teams (From [1]).
[1] Dell'Anna, D.; Murukannaiah, P. K.; Dudzik, B.; Grossi, D.; Jonker, C. M.; Oertel, C.; and Yolum, P. (2024) “Toward a Quality Model for Hybrid Intelligence Teams.” In Proceedings of the 23rd International Conference on Autonomous Agents and MultiAgent Systems, AAMAS 2024, Preprint

Rules

• Participants form teams of two–three (preferably two, three as the exception).
• Participants are provided with a description of 4 HI use cases (Section 1 in the Competition details).
• Participants are provided with a number of HI quality attributes characterizing properties of quality HI Teams (Section 2 in the Competition details).
• Participants submit two Usage Scenarios (Scenario 1 and Scenario 2) in a given format (Section 3 in the Competition details).
  • For each scenario, participants choose one HI Use case from Section 1.
  • In Scenario 1, participants describe a Best Case Scenario highlighting the benefits for team collaboration stemming from at least one of the quality attributes from Section 2.
  • In Scenario 2, participants describe a Worst Case Scenario highlighting the risks and challenges stemming from the lack of at least one of the quality attributes from Section 2.
  • For each scenario, participants report, on a scale from 1 to 5, how much the quality attribute(s) affect the outcome of the described scenario.
• Participants present their scenarios during the competition day.
• Scenarios are evaluated based on the evaluation criteria (Section 4 in the Competition details)

Evaluation criteria

The scenarios will by a jury composed of the competition organizers, based on the following criteria, explained in Section 4 in the Competition details.

• Creativity
• Complexity
• Realism
• Potential impact on the use case
• Clarity of presentation

Prizes

Certificates and monetary prizes will be presented to the top three teams.

• 1st place: € 500
• 2nd place: € 400
• 3rd place: € 300

Prizes will be delivered only if more than three teams join the competition, if the teams adhere to the rules of the competition, and if minimum requirements (see Section 3 in the Competition details) are met.

Submission instructions

Submission can be done via the following online form (Qualtrics survey).
Participants are requested to directly write their answers in the provided form. Extra material can be attached but it is optional.

Before submitting, make sure to carefully read the Competition details, and to meet the minimum requirements for accessing the prize competition.

In the submission form, participants will be informed on which and how their data will be processed and managed, and they will be asked for consent before proceeding.
We intend to publish, on this website, names and affiliation of participants, alongside with their submitted scenarios.

Note: We strongly encourage you to submit only once, when the submission material is definitive. In exceptional cases, if you need to resubmit, we will consider the last submission before the deadline as the valid one. However, please try to avoid this.

Program and Important Dates

Competition Day and Location

Wednesday, May 8th, 2024, 14:00 - 16:30,Auckland, New Zaeland; jointly held with AAMAS 2024

Schedule (all times refer to the competition day local time)

14:00 - Introduction
14:15 - Competition time (1 hour)
14:45 - Latest starting time for participants
15:15 - Deadline for submission
15:15-16:00 - Coffee break and Jury reading time
16:00 - Announcing the winners and photo

Finalists and Participants

1st place

Team Zebra State University: Gilberto Antonio Marcon dos Santos (Zebra Technologies Corporation), Mahmoud Nasr (Zebra Technologies Corporation), John Meo (Applied Research Associates)

2nd place

Team Dream Team: Emre Erdogan (Utrecht University), ChatGPT4 (NA)

3rd place

Team AR_TEAM: Rui Prada (INESC-ID), Ann Nowé (VUB)

All participants teams (alphabetical order)

• AR_TEAM: Rui Prada (INESC-ID), Ann Nowé (VUB)
• Dream Team: Emre Erdogan (Utrecht University), ChatGPT4 (NA)
• Kawaii: Katsuhide Fujita (Tokyo University of Agriculture and Technology), Reyhan Aydogan (Ozyegin University)
• LouLu: Xingzhou Lou (Institute of Automation, Chinese Academy of Sciences), Vera Lu (Civilization Corp.)
• MarkBaz: Behrad Koohy (University of Southampton), Mark Towers (University of Southampton)
• Piha: Nathaniel Sauerberg (UT Austin), Emanuel Tewolde (CMU)
• Zebra State University: Gilberto Antonio Marcon dos Santos (Zebra Technologies Corporation), Mahmoud Nasr (Zebra Technologies Corporation), John Meo (Applied Research Associates)

Submitted Scenarios

In the following, a data set containing all the scenarios submitted by the competition participants is reported.

The columns in the dataset are described as follows:

• Date: The date of submission of the scenarios (Date).
• Event: The event where the scenario was submitted (String).
• Team: The name of the participants team that submitted the scenario (String).
• Best Case - Use Case HI Team: The HI Use Case (see Section 1 in the competition details) considered by the Team for the Best Case Scenario (String).
• Best Case - Scenario: The Best Case Scenario submitted by the Team (String).
• Worst Case - Use Case HI Team: The HI Use Case (see Section 1 in the competition details) considered by the Team for the Worst Case Scenario (String).
• Worst Case - Scenario: The Worst Case Scenario submitted by the Team (String).