AI is increasingly influencing every aspect of our lives. Automated Decision Making (ADM) systems use machine learning to decide who should receive a loan, who gets admitted to a prestigious university or who is invited for a job interview. With such systems influencing our lives at such a scale, it’s natural to ask whether these systems are fair and how we can check whether they are. Because of that the field of algorithmic fairness has seen a massive increase in popularity in recent years. Different disciplines, such as computer science, philosophy, mathematics and social sciences, have contributed to the debate. One central aspect of the field is the evaluation of systems through so-called fairness metrics. These metrics can be grouped into different categories. Group fairness metrics appear to be particularly popular, perhaps due to their apparently straightforward application in practice. However, their application still comes with many challenges. When should we opt for one fairness metric over another? Are some of the fairness metrics to some extent compatible? How can fairness metrics be balanced with other optimization goals, such as the utility of the model? What are the limitations of group fairness metrics? Can standard approaches for achieving group fairness metrics be implemented under EU law? This talk will provide an introduction to the fairness discussion and highlight ongoing work that attempts to answer the questions raised above, which are crucial for the practical usage of group fairness metrics.