CLTL at DIMEMEX Shared Task: Fine-Grained Detection of Hate Speech in Memes

We present the CLTL system developed for the DIMEMEX Shared Task on detecting fine-grained types of hate speech in Mexican Spanish memes. The competition consisted of two tasks. Task 1 involved classifying memes into hate speech, inappropriate, or harmless categories, while Task 2 required further classification of hateful memes into classism, sexism, racism, or other types. We explored the effectiveness of combining state-of-the-art language models with the Swin Transformer-based visual model to create a multimodal system using the Multilayer Perceptron fusion module for classification. Our experiments demonstrated that the XLM-T model combined with Swin Transformer V2 achieved the highest results, with an F1 score of 57.88 for Task 1 and 43.65 for Task 2, ranking 1st in both tasks in the competition.