Unveiling Retail Customer Patterns: A Self-Taught Data Analyst's Journey

Introduction:

I embarked on my inaugural solo project as a budding data analyst, where I explored the intriguing realm of retail customer analytics. My objective was to glean insights, unravel patterns, and develop a predictive model that foretells customer preferences based on demographic segmentation.

Project Overview:

For this maiden project, I obtained a dataset comprising diverse customer demographics. The data pre-processing, exploratory data analysis (EDA), model creation, and performance evaluation were done using Python on Jupyter Notebook. The data contained relevant features that would be used to train the models. At the pre-processing and cleaning stage, the data was cleaned and prepared for exploratory data analysis [EDA].  The essence of the EDA was to reveal relationships, correlations and trends between the different variables. Then customer segmentation models were created utilizing different machine learning algorithms, including K-Nearest Neighbours, Random Forest, and Decision Tree. The models were trained using the training data so as to detect patterns, and then they were tested using the test data. Following this, the accuracy of each model was evaluated using the Precision, Recall, Accuracy and F1 Score metrics. This was done to show which of the models had the best performance and was most suitable for developing the customer segmentation model.

Data Collection:

For this project I made use of secondary data that was sourced from the Kaggle website in .csv (comma separate value) format. The dataset consisted of ten variables of the customer's demographic data, including the customer ID, gender, marital status, age, occupation, level of education, work experience, spending score, family size and segmentation for 10,695 customers who had been identified as frequent shoppers at a specific retailer.

Pre-processing and Cleaning:

The initial steps were quite like the polishing of a gem—a meticulous hunt for duplicates, missing values, and inconsistencies to present a dataset that was clean, consistent, and suitable for analysis. The datasset had no duplicate values or outliers; the missing values amongst the numerical variables were replaced with the mode value of that specific variable. Amongst the categorical variables, missing values were also replaced with the most frequently occurring values.  

Feature Selection:

Contingent upon the project objectives, certain features or variables were selected or excluded from the dataset. To enhance the accuracy of the models, it was necessary to remove correlated variables. I used pandas “.corr()” function to find correlations; and then used a heatmap in seaborn to visualize the correlation matrix. I selected crucial features for model training by means of the Relative Variable Importance in order to improve the models’ utility. Irrelevant variables were discarded, leaving a leaner, more impactful dataset.

Exploratory Data Analysis (EDA):

The EDA phase was done making use of statistical techniques such as measures of central tendency and measures of dispersion; and data visualisation techniques to examine and summarise the distinct features of the dataset. The EDA also involved some data visualisation. This visual exploration allowed the identification of patterns and associations that were not be readily apparent through statistical analysis alone. I did unravel some fascinating insights from my dataset. A greater portion of the customers (61.43%) are graduates; More than half of the customers were also low spenders who fell into the 20-40 years age group.

Model Training and Testing:

Following this, I split the data into an 80:20 ration with the 80% (8,068 customers) assigned to the train subset, and the 20% (2,627 customers), to the test subset. With machine learning as my guide, I trained and tested the data using multiple models—K-Nearest Neighbours, Decision Tree, and Random Forest which I imported from Scikit-learn. Before this, I converted the categorical data in the dataset to numerical data.

Python code for splitting data

Performance Evaluation:

At this stage, the effectiveness of each of the models was evaluated using the accuracy, precision, recall, and F1 score metrics. These metrics were also imported from Scikit-learn, and then displayed graphically using Matplotlib. 

Python code for Random Forest Accuracy Score and graph

Findings and Insights:

Amongst the three Machine Learning algorithms I used for this project, the Random Forest model had the highest level of accuracy at 53.04%, although the other two models were not too far behind. I came to the conclusion that the Random Forest model had a marginally superior performance in facilitating demographic customer segmentation and predictive analysis.

Conclusion:

In essence, this project helped to demystify the challenge of understanding customer preferences. The potential of machine learning in segmenting demographics and predicting purchasing behaviours bears a testament to its instrumental role in strategic marketing endeavours.