Impact of Missing Data on Big Data Analytics: A Case Study on Hospital Readmission Rates with Advanced Imputation Techniques

Abstract

The purpose of this study is to investigate the impact of missing data on big data analytics, specifically within the context of hospital readmission rates using the dataset "Impact of HbA1c Measurement on Hospital Readmission Rates" by Beata Strack et al. (2014). This dataset, comprising 70,000 clinical records, provides a rich foundation for exploring advanced data mining techniques. Initially, we analyze the dataset to identify and quantify missing data. We first create a machine learning (ML) model for classification by neglecting the rows with missing data and evaluate its performance using f1 score and ROC-AUC metrics.Subsequently, we categorize the missing data into Missing Completely at Random (MCAR), Missing at Random (MAR), and Missing Not at Random (MNAR). The data preprocessing phase includes outlier detection and normalization to ensure data quality and consistency. We then apply various imputation methods to handle the missing data and re-evaluate the classification models, comparing their performance against the initial models based on f1 score and ROC-AUC. To identify the most effective model, we evaluate multiple ML algorithms, including logistic regression, random forests, and gradient boosting. The findings of this study highlight the significant impact of missing data on model performance and the effectiveness of different imputation techniques in mitigating this impact. By focusing on the implications of missing data in big data analytics, this research provides valuable insights for data analysts and contributes to the development of more robust predictive models.