Metrics for dataset demographic bias: a case study on facial expression recognition

Domínguez Catena, Iris; Paternain Dallo, Daniel; Galar Idoate, Mikel

Metrics for dataset demographic bias: a case study on facial expression recognition

dc.contributor.author	Domínguez Catena, Iris
dc.contributor.author	Paternain Dallo, Daniel
dc.contributor.author	Galar Idoate, Mikel
dc.contributor.department	Estadística, Informática y Matemáticas	es_ES
dc.contributor.department	Estatistika, Informatika eta Matematika	eu
dc.contributor.department	Institute of Smart Cities - ISC	en
dc.contributor.funder	Universidad Pública de Navarra - Nafarroako Unibertsitate Publikoa
dc.date.accessioned	2024-09-30T17:45:06Z
dc.date.available	2024-09-30T17:45:06Z
dc.date.issued	2024
dc.date.updated	2024-09-30T17:02:21Z
dc.description.abstract	Demographic biases in source datasets have been shown as one of the causes of unfairness and discrimination in the predictions of Machine Learning models. One of the most prominent types of demographic bias are statistical imbalances in the representation of demographic groups in the datasets. In this paper, we study the measurement of these biases by reviewing the existing metrics, including those that can be borrowed from other disciplines. We develop a taxonomy for the classification of these metrics, providing a practical guide for the selection of appropriate metrics. To illustrate the utility of our framework, and to further understand the practical characteristics of the metrics, we conduct a case study of 20 datasets used in Facial Emotion Recognition (FER), analyzing the biases present in them. Our experimental results show that many metrics are redundant and that a reduced subset of metrics may be sufficient to measure the amount of demographic bias. The paper provides valuable insights for researchers in AI and related fields to mitigate dataset bias and improve the fairness and accuracy of AI models.	en
dc.description.sponsorship	This work was supported in part by a predoctoral fellowship from the Research Service of the Universidad Pública de Navarra through open access funding, in part by the Spanish MICIN under Grants PID2019-108392GB-I00, PID2020-118014RB-I00, and PID2022-136627NB-I00/AEI/10.13039/501100011033 FEDER, UE, and in part by the Government of Navarre under Grant 0011-1411-2020-000079 - Emotional Films.
dc.format.mimetype	application/pdf	en
dc.identifier.citation	Dominguez-Catena, I., Paternain, D., Galar, M. (2024) Metrics for dataset demographic bias: a case study on facial expression recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1-18. https://doi.org/10.1109/TPAMI.2024.3361979.
dc.identifier.doi	10.1109/TPAMI.2024.3361979
dc.identifier.issn	0162-8828
dc.identifier.uri	https://academica-e.unavarra.es/handle/2454/51887
dc.language.iso	eng
dc.publisher	IEEE
dc.relation.ispartof	IEEE Transactions on Pattern Analysis and Machine Intelligence 46(8), 2024, 5209 - 5226
dc.relation.projectID	info:eu-repo/grantAgreement/AEI/Plan Estatal de Investigación Científica y Técnica y de Innovación 2017-2020/PID2019-108392GB-I00/ES/
dc.relation.projectID	info:eu-repo/grantAgreement/AEI/Plan Estatal de Investigación Científica y Técnica y de Innovación 2017-2020/PID2020-118014RB-I00/ES/
dc.relation.projectID	info:eu-repo/grantAgreement/AEI/Plan Estatal de Investigación Científica y Técnica y de Innovación 2021-2023/PID2022-136627NB-I00/ES/
dc.relation.projectID	info:eu-repo/grantAgreement/Gobierno de Navarra//0011-1411-2020-000079/
dc.relation.publisherversion	https://doi.org/10.1109/TPAMI.2024.3361979
dc.rights	© 2024 The Authors. This work is licensed under a Creative Commons Attribution 4.0 License
dc.rights.accessRights	info:eu-repo/semantics/openAccess
dc.rights.uri	https://creativecommons.org/licenses/by/4.0/
dc.subject	AI fairness	en
dc.subject	Artificial intelligence	en
dc.subject	Deep learning	en
dc.subject	Demographic bias	en
dc.subject	Facial expression recognition	en
dc.title	Metrics for dataset demographic bias: a case study on facial expression recognition	en
dc.type	info:eu-repo/semantics/article
dc.type.version	info:eu-repo/semantics/publishedVersion
dspace.entity.type	Publication
relation.isAuthorOfPublication	2ba22a83-13df-4d11-8e44-f9e97dbc13d0
relation.isAuthorOfPublication	ca16c024-51e4-4f8f-b457-dc5307be32d9
relation.isAuthorOfPublication	44c7a308-9c21-49ef-aa03-b45c2c5a06fd
relation.isAuthorOfPublication.latestForDiscovery	2ba22a83-13df-4d11-8e44-f9e97dbc13d0