Download

 

Diverse datasets are available for the different subtasks. All datasets are released as UTF-8 encoded comma separated “.csv” files, whose download is password-protected. Please write us [dankmemesevalita (AT) gmail.com] to get access to them.

 

General Guidelines

 

Development Datasets

 

Test Datasets

 

Annotated Test Datasets

 


Datasets

The whole dataset is composed of 2,361 images extracted from Instagram through a Python script aimed at the hashtag related to the Italian government crisis (“#crisidigoverno”). Posts have been extracted along with their related metadata and anonymized. For each meme, we will provide as input the image, the text transcription, the image vector representation and the following features:

  • Date: when the image has first been posted on the platform (Instagram)
  • Macro status: refers to meme layouts and their relation to diffused, conventionalised formats called macros. Binary: macro (1), not macro (0)
  • Picture manipulation: Non-manipulated or low impact changes are labeled 0 (e.g. the addition of a text or a logo). Heavily manipulated, impactful changes (e.g. images edited to include political actors) are labeled 1.
  • Visual actors: the political actors (i.e. politicians, parties’ logos) portrayed visually, as edited into the picture or portrayed in the original image.
  • Engagement: the sum of comments and likes on the artifact in its original setting (Instagram)
  • Meme: binary feature, where 0 represents non meme images and 1 meme images. This is the target label for the first subtask.
  • Hate Speech: binary feature only for memes. It differentiates memes with offensive language (1) from non offensive memes (0). This is the target label for the second subtask.
  • Event: feature only for meme images, categorizing them according to 4 events (described in \ref{sub 3}), plus a residual category labeled as 0. This is the target label for the third subtask.

Text Extraction. Texts have been extracted through optical character recognition (OCR) using Google’s Tesseract-OCR Engine, and further manually corrected by linguists.

Image Vector Representations. The dataset includes image embeddings. The vector representations are computed employing ResNet (He et al., 2016), a state-of-the-art model for image recognition based on Deep Residual Learning. Providing such image representations allows the participants to approach these multimodal tasks focusing primarily on its NLP aspects (Kiela and Bottou, 2014).

External Resources. Participants are allowed to use external resources, lexicons or independently annotated data. Given that, although we provide ResNet image embeddings, participants can use any other image representations.


 

Task 1: Meme Detection

The dataset counts 2000 images, half memes and half not. We split the dataset into training and test sets, in a proportion of 80-20% of items. The following table represents the format of the training dataset. The test dataset will be provided without gold labels (i.e. without the “Meme” attribute) for testing purposes.

Filename Engagement Date Manipulated Visual actors Text Meme
1.jpg 234 27/08/2019 1 Mattarella io che aspetto arrivino le 15 per assistere alla fine del governo 1
2.jpg 432 15/08/2019 0 Salvini cosa ha fatto a mosca la notte del 17 ottobre? 0

 


 

Task 2: Hate Speech Identification

The dataset counts 1,000 memes. We split the dataset into training and test sets, in a proportion of 80-20% of items. The following table represents the format of the training dataset. The test dataset will be provided without gold labels (i.e. without the “Hate Speech” attribute) for testing purposes.

Filename Engagement Manipulated Visual actors Text Hate Speech
1.jpg 114 0 Salvini, Conte, Di Maio aspetta manca ancora la parte in cui parlo del lavoro di tua sorella 1
2.jpg 412 1 Salvini merdman 1

 

Task 3: Event Clustering

The dataset counts 1000 memes. We split the dataset into training and test sets, in a proportion of 80-20% of items. A set of 800 labeled memes will be provided for development. In order to test and evaluate the model, the remaining 200 memes will be provided without labels.

Filename Date Engagement Macro Manipulation Visual actors Text Embedding ID Event
1.jpg 11/08/2019 324 0 1 Salvini, Berlusconi la crisi di governo è stata innescata mio signore hai lavorato bene lord salvini. tutto procede come avevo previsto 1 1
2.jpg 10/08/2019 123 0 0 Salvini, Di Maio, Conte, Mattarella il governo entra in crisi 5 secondi di penalità per vettel 1 1
3.jpg 20/08/2019 111 0 0 Salvini, Conte conte usa boato 2 2
4.jpg 20/08/2019 45 0 0 Salvini, Conte quando ti interroghi su chi abbia fatto affogare tutti i tuoi pacchetti udp io 2 2
5.jpg 29/08/2019 23 1 0 0 affogare i bambini con la lega fare l’elettroshock ai bambini con il pd 3 3
6.jpg 27/08/2019 987 1 0 0 di maio salvini ricordi come arrivare a palazzo chigi credo di sì, ma perché dovrei saperlo? 3 3
7.jpg 03/09/2019 65 0 0 0 quando provi a votare su rousseau ma il sito non carica 4 4
8.jpg 02/09/2019 112 0 0 Salvini quando perdi il posto al viminale ma sei ancora in tempo per andare a fare la chiusura della stagione ad Ibiza 4 4