Constraint based design of cementitious concrete analytics with multilingual SER for mining site and cross language generalizability performance
Sichen Pan,Jialing Li,Refka Ghodhbani,Rania M. Ghoniem,Raouf Feng Hassan,José Escorcia-Gutierrez
Abstract
The growing demand for sustainable construction materials has spurred extensive research to reduce cement consumption while maintaining adequate mechanical performance and durability. In parallel, cementitious concrete operations in mining sites require analytics that can support worker protection and decision making under language, acoustic, environmental, and task variability. However, most speech emotion recognition studies rely on controlled datasets and provide limited evidence for multilingual transfer, high noise robustness, worker disjoint validation, and mining site generalization. This study proposes a constraint-based cementitious concrete analytics framework using multilingual Speech Emotion Recognition (SER) for mining site worker assessment and cross language generalizability performance. The Bayesian Optimized Speech Emotion Model for Site Speech Emotion Recognition (BOSEM SiteSER) combines acoustic representation learning, multilingual pretraining, transcript semantics, mining site context fusion, noise adaptation, language alignment, Bayesian optimization, and uncertainty calibration. A structured corpus was developed across eight language strata, 48 mining operation sites, 1,440 workers, 6,480 worker days, 4,320 audio hours, and 259,200 utterances. Inputs included lapel and helmet microphone speech, environmental sound, thermal context, task logs, concrete batch records, mining activity records, transcript semantics, and self-report anchors. Seven worker condition classes were modeled, including neutral alertness, task pressure, frustration, cognitive fatigue, heat strain risk, acute stress, and recovery. BOSEM SiteSER was benchmarked against acoustic, Convolutional Neural Network (CNN), self-supervised speech, Automatic Speech Recognition (ASR) encoder, and SER foundation baselines under worker disjoint, day disjoint, leave one language out, leave one site out, high noise, unseen cementitious concrete operation, and unseen mining operation protocols. Results showed macro F1 of 0.902, balanced accuracy of 0.915, Expected Calibration Error of 0.031, high risk false negative rate of 0.046, and cross language macro F1 of 0.872 with an average F1 reduction of 0.040. The neural SER model demonstrated high predictive accuracy and robust generalizability, highlighting its potential as a decision support tool for sustainable cementitious composite design.
Sichen Pan — School of Computer Science and Technology, Guangdong University of Technology, Guangzhou 510006, Guangdong Province, China
Jialing Li — Chongqing Youth Vocational & Technical College, Chongqing 401320, China
Refka Ghodhbani — Center for Scientific Research and Entrepreneurship, Northern Border University, Arar 73213, Saudi Arabia
Rania M. Ghoniem — Department of Information Technology, College of Computer and Information Sciences, Princess Nourah bint Abdulrahman University, P.O. Box 84428, Riyadh 11671, Saudi Arabia
Raouf Feng Hassan — Department of Civil Engineering, Advanced Materials Research Group, Universiti Malaysia Kelantan, Malaysia; Civil Engineering Department, College of Engineering, Imam Mohammad Ibn Saud Islamic University (IMSIU),13318 Riyadh, Saudi Arabia
José Escorcia-Gutierrez — Department of Computational Science and Electronics, Universidad de la Costa, CUC, Barranquilla 080002, Colombia
PDF Viewer
Preview is limited to the first 3 pages. Sign in to access the full PDF.
