Shared task @ AAAI 2024

Third Workshop on ​Multimodal Fact-Checking and Hate Speech Detection
February, 2024

ABOUT Factify5WQA

Important News Datasets Released:

Factify5WQA

Colab: https://codalab.lisn.upsaclay.fr/competitions/16053
Form: https://forms.gle/AeYMocP8cZGin1777

Please register here to access the dataset.

Contemporary automatic fact-checking systems focus on estimating truthfulness using numerical scores which are not human-interpretable. A human fact-checker generally follows several logical steps to verify a verisimilitude claim and conclude whether it's truthful or a mere masquerade. Therefore, it is necessary to have an aspect-based (delineating which part(s) are true and which are false) explainable system that can assist human fact-checkers in asking relevant questions related to a fact, which can then be validated separately to reach a final verdict. In this task, we will release a 5W framework (who, what, when, where, and why) for question-answer-based fact explainability. To that end, we present a semi-automatically generated dataset called FACTIFY-5WQA. The dataset is a fact verification dataset that aims to detect whether a claim is false or not based on a given evidence/document. With this task we provide questions such that the fact verification is done by checking the entailment of the answers from the claim and evidence respectively.

Some Samples from Data :

  • "claim": "Andre Agassi won seven titles.",
    "evidence": "Andre Kirk Agassi born April 29 , 1970 -RRB- is an American retired professional tennis player and former World No. 1 who was one of the sport 's most dominant players from the early 1990s to the mid-2000s . Generally considered by critics and fellow players to be one of the greatest tennis players of all time ...,
    "question": [ "How many titles did andre agassi win?", "Who won seven titles?" ],
    "claim_answer": [ "seven titles", "Andre Agassi" ],
    "evidence_answer": [ "eight-time Grand Slam champion", "Agassi" ],
    "label": "Refute"
  • "claim": "London police officer seriously injured in machete attack during vehicle stop. https://t.co/tnCa0MK6R9",
    "evidence": "By Julia Hollingsworth, CNNUpdated 0758 GMT (1558 HKT) August 8, 2019 (CNN)A London police officer is in a critical condition after a driver he pulled over attacked him with a machete. ",
    "question": [ "How was a london police officer seriously injured?", "Who was seriously injured in a machete attack?", "When was the london police officer attacked?" ],
    "claim_answer": [ ": in machete attack", "London police officer", "during vehicle stop" ],
    "evidence_answer": [ "a driver he pulled over attacked him with a machete", "A London police officer", "August 8, 2019" ],
    "label": "Support"
  • Dataset link : Please register in the Codalab page linked above to find details to access the dataset.
  • Evaluation metric : Since this task has three target variables, we will use an average BLEU score of the answers to the questions from the claim and evidence. Only if this score is greater than a set threshold and the label i.e. Support/Neutral/Refute is correct, do we consider that as a right prediction. The final accuracy is computed as the percentage of right predictions.
  • Submission instructions : Results are to be submitted in json file named "answer.json" in a zipped format. Refer to the starting kit for more details.
  • System description paper : All teams/participants will be invited to submit a paper describing their system. Accepted papers will be published in formal proceedings.
  • Paper submission instruction : To be updated.

IMPORTANT DATES

  • 13 October 2023 : Release of the training set.
  • 8 November 2023 : Release of the test set.
  • 30 November 2023 : Deadline for submitting the final results.
  • 12 December 2023 : Announcement of the results.
  • 05 January 2024 : System paper submission deadline (All teams are invited to submit a paper).
  • 20 January 2024 : Notification of system papers.
  • 25 January 2024 : Camera ready submission.

ORGANIZING COMMITTEE CHAIRS

Dr. Amitava Das:


Dr. Amitava Das is a Core Faculty & Research Associate Professor of the Artificial Intelligence Institute, at the University of South Carolina, and an Advisory Scientist to Wipro AI.

Research interests : Code-Mixing and Social Computing.

Organizing Activities [selective] : • Memotion @SemEval2020 • SentiMix @SemEval2020 • Computational Approaches to Linguistic Code-Switching @LREC 2020 • CONSTRAINT @AAAI2021

Dr. Amit Sheth:


Dr. Amit Sheth is the founding Director of the Artificial Intelligence Institute, and a CSE Professor at University of South Carolina.

Research interests : Knowledge Graph, NLP, Analysing Social Media

Organizing Activities [selective] : • Cysoc2021 @ ICWSM2021 • Emoji2021 @ICWSM2021 • KiLKGC 2021 @KGC21

Dr. Asif Ekbal:


Dr. Asif Ekbal is an Associate Professor of CSE at IIT Patna, India.

Research interests : NLP, CodeMixing and Social Computing.

Organizing Activities [selective] : • CONSTRAINT @AAAI2021

Aman Chadha


Aman Chadha is an Applied Sci- ence Manager at Amazon Alexa AI and a Researcher at Stanford AI.

Research interests : Multimodal AI, On-device AI, and Human-Centered AI.

ASSOCIATE ORGANIZERS


Parth Patwa
University of California Los Angeles (UCLA)

Megha Chakraborty
UNIVERSITY OF SOUTH CAROLINA, USA

Suryavardan Suresh
New York University

Anku Rani
University of South Carolina, USA

WEB CHAIR


Jinendra Malekar
UNIVERSITY OF SOUTH CAROLINA, USA

CONTACT US