Small Groups, Big Insights: Understanding the Crowd through Expressive Subgroup Analysis

When people gather, whether in small groups or large crowds, incidents such as fights or crushes can occur. Crowd management focuses on preventing such situations. Artificial Intelligence (AI) can assist in this by quickly processing large amounts of data, which can, for example, help security personnel analyse surveillance footage. For this it is important that AI systems are practically applicable, so they can function well in various situations. My dissertation contributes to these developments by researching automated crowd understanding. The focus is on identifying subgroups within these crowds. This provides insight into the dynamics within a crowd while helping maintain an overview. The emphasis is on recognising the emotions of these subgroups, where emotions are seen as predictors of behaviour. Analysing emotions could therefore enable security personnel to intervene timely to prevent escalation. I use various methods and data types to identify subgroups. For instance, I find that emotional subgroups in images are best recognised by grouping people based on facial expressions, proximity to each other, and gaze direction. Additionally, I develop a model that identifies violent subgroups in surveillance footage and shows where they are located in the frame. In another study, I work with social media posts, determining whether they are positive or negative and to what location the text refers. This way, a large area can be mapped, enabling for instance the tracking of riots' progression and location in a city. I furthermore work with audio fragments of crowds. My system automatically identifies whether the crowd is expressing positive or negative sentiments and whether it consists of one or multiple subgroups. Each of these data types has its pros and cons and addresses the need for more practically applicable systems in different ways. Ethics and data protection are crucial in this research field. Legislation such as the GDPR and the recently adopted AI Act provide legal frameworks for this. I analyse the implications of these laws for automated group analysis. Additionally, I investigate ethical concerns not addressed by these laws and propose various solutions. Key points include acknowledging that different situations require different approaches, individual responsibility, and the role of scientists. The AI Act provides ample space for conducting research, but I argue that researchers should not necessarily make full use of this. Instead, scientists can play a significant role in developing ethically responsible systems, ensuring that everyone can reap their benefits in the future.