@ AAAI 2025

Fourth Workshop on ​Multimodal Fact Checking and Hate Speech Detection
February, 2025


About : CT2 - AI Generated Text Detection

Important News Datasets Released:

CT2 - AI Generated Text Detection

Dataset:

To better understand which types of LLM-generated content are easier or harder to detect, we will include the following families of LLMs:

  • Encoder models: (e.g., BERT, DeBERTa)
  • Open-Source: (e.g., Llama 3.1, Yi-2)
  • Closed-Source: (e.g., Claude 3.5 Sonnet, GPT 4.0/mini)
  • SLMs: (e.g., Phi-3.5)
  • Mixture of Experts (MoEs) (e.g., Mixtral)
  • State Space Models (SSM) (e.g., Falcon-Mamba)

We will be releasing 50K data samples for this task. The data will be structured such that each prompt will have a human-written story and corresponding parallel generations from all the included LLMs. A snapshot of the data can be viewed here.

Tasks:

CT2 for text will consist of two sub-tasks:

  1. Task A: This is a binary classification task where the goal is to determine whether each given text document was generated by AI or created by a human.
  2. Task B: Building on Task A, this task requires participants to identify which specific LLM generated a given piece of AI-generated text. For this task, participants will know that the text is AI-generated and must predict whether it was produced by models such as GPT 4.0, DeBERTa, FalconMamba, Phi-3.5, or others.

Baseline:

Several systems for detecting AI-generated text, such as GPT-2 Output Detector, GLTR, and GPTZero, have recently emerged. However, these proprietary tools do not disclose their detection methods. In academic research, various techniques have been proposed to identify AI-generated text, including:

  • Perplexity estimation
  • Burstiness estimation
  • Negative log-likelihood curvature
  • Stylometric variation
  • Intrinsic property based approaches
  • Classifier-based approaches (RADAR, Conda)

A new interesting technique proposed at ICML 2024 finds that LLMs are more inclined to modify human-written text than AI-generated text when tasked with rewriting. This approach, named the geneRative AI Detection viA Rewriting method (Raidar), will be used as the baseline for both Tasks A and B.

Raidar Example

The authors highlight character deletions in red and character insertions in orange. Their findings indicate that human-generated text generally prompts more modifications compared to machine-generated text when rewritten.

Leaderboard

S.No Name Team Name Score for Task-A ▲▼ Score for Task-B ▲▼
1 Avinash Trivedi Sarang 1 0.9531
2 Duong Anh Kiet dakiet 0.9999 0.9082
3 Vijayasaradhi Indurthi tesla 0.9962 0.9218
4 Shrikant Malviya SKDU 0.9945 0.7615
5 Harika Abburi Drocks 0.9941 0.627
6 Manoj Saravanan Llama_Mamba 0.988 0.4551
7 Chinnappa Guggilla AI_Blues 0.9547 0.4698
8 Xinlong Zhang NLP_great 0.9157 0.1874
9 Shifali Agrahari Osint 0.8982 0.3072
10 Xiaoyu Xiaoyu 0.803 0.5696
11 Rohan R Rohan 0.7546 0.4053

Leaderboard-Subtask

Leaderboard for Task-A Scores

S.No Name Team Name Scores
1 Avinash Trivedi Sarang 1
2 Duong Anh Kiet dakiet 0.9999
3 Vijayasaradhi Indurthi tesla 0.9962
4 Shrikant Malviya SKDU 0.9945
5 Harika Abburi Drocks 0.9941
6 Manoj Saravanan Llama_Mamba 0.988
7 Chinnappa Guggilla AI_Blues 0.9547
8 Xinlong Zhang NLP_great 0.9157
9 Shifali Agrahari Osint 0.8982
10 Xiaoyu Xiaoyu 0.803
11 Rohan R Rohan 0.7546

Leaderboard for Task-B Scores

S.No Name Team Name Scores
1 Avinash Trivedi Sarang 0.9531
2 Vijayasaradhi Indurthi tesla 0.9218
3 Duong Anh Kiet dakiet 0.9082
4 Shrikant Malviya SKDU 0.7615
5 Harika Abburi Drocks 0.627
6 Xiaoyu Xiaoyu 0.5696
7 Chinnappa Guggilla AI_Blues 0.4698
8 Manoj Saravanan Llama_Mamba 0.4551
9 Rohan R Rohan 0.4053
10 Shifali Agrahari Osint 0.3072
11 Xinlong Zhang NLP_great 0.1874

IMPORTANT DATES

  • 31 October 2024: Release of the training set.
  • 10 November 2024: Release of the test set.
  • 2 December 2024: Release of the Updated test set.
  • 30 November 14 December 2024: Deadline for submitting the final results.
  • 20 December 2024: Announcement of the results.
  • 12 January 16 January 2025: System paper submission deadline (All teams are invited to submit a paper).
  • 20 January 2025: Notification of system papers.
  • 25 January 2025: Camera-ready submission.