EHLD: A General Data Framework for Harmful Language Detection and Evaluation

Purpose- Research into harmful language detection is hindered by fragmented datasets and incompatible label schemas, which significantly limit evaluation. This paper introduces EHLD: a general, extensible data framework supporting the integration, enrichment and evaluation of diverse harmful language resources, with a particular focus on Italian.

Design/Methodology/Approach- EHLD defines a unified schema with essential detection attributes (e.g., harmful_or_not and harm_subcategory), optional contextual dimensions (e.g., target, category, intersectionality), evaluation-oriented text complexity features, and metadata for provenance. We instantiate EHLD by integrating three data sources: curated datasets from the literature, large-scale LLM-based annotation from the Mappa dell’Intolleranza, and LLM generated data for inclusive language detection. We further introduce label_confidence to explicitly encode label reliability. Model selection follows an iterative cycle combining distributional analysis, linguistic diversity assessment, and performance evaluation.

Findings- The resulting dataset contains 237,956 instances with 18 features and exhibits substantial linguistic variety (Self-BLEU drops from 0.759 at 1-gram to 0.107 at 4-grams). After iterative, data-centric refinement, GPT-5-nano is selected as the reference model and achieves an F1-score of 0.813 on binary harmful language detection, outperforming reported baselines such as BERT-based multitask models and other LLMs on comparable Italian settings.

Research limitations/implications- Due to differences in datasets, domains, label definitions and evaluation protocols, comparisons across studies are not fully controlled. The framework partly relies on automatically generated labels, which, despite confidence-aware handling, may introduce residual noise.

Practical implications- By combining standardised labels, provenance tracking and complexity-oriented evaluation features, EHLD supports the construction of reproducible datasets and richer model auditing, enabling a more transparent deployment of harmful language detection systems.

Originality/value- EHLD contributes a reusable, confidence-aware integration framework that connects different harmful language resources and allows iterative, data-driven model selection and evaluation, with a focus on Italian.