Evaluating the Efficacy of Machine Learning Algorithms for Customer Churn Prediction in The Insurance Industry

Abstract

In the insurance industry, large-scale data from customer transactions is generated daily, making it more cost-effective to retain existing clients than to acquire new ones due to customer churn. This study aims to identify the best classification algorithm for predicting customer churn, determine the characteristics of churning customers, and model the time until churn. After data preprocessing and addressing dataset imbalance using the Synthetic Minority Oversampling Technique (SMOTE), various predictive models, including logistic regression, naive Bayes, decision trees, random forests, neural networks, extra trees, boosting algorithms, and heterogeneous ensemble learning methods, were applied. K-fold cross-validation was utilized for hyperparameter tuning and preventing overfitting. The results, evaluated using the confusion matrix and Area Under Curve (AUC), showed that Extreme Gradient Boosting (XGBoost) and Light Gradient Boosting Machine (LGBM) classifiers achieved the highest accuracy at 88.57% and 88.53%, respectively, with precision, recall, and F-score also used for assessment. Survival analysis revealed that approximately 80% of policyholders survived the first 200 months, with “having children” being the most significant predictor of customer churn. Unlike standard classification, survival analysis enables modeling of both churn likelihood and expected time-to-churn, accounting for censored observations. This provides actionable insights for intervention timing, a critical advantage in customer retention planning