Predict Customer Churn with C# and ML.NET

Customer churn is one of the most familiar Predictive AI use cases.

A business wants to know which customers are likely to leave so it can intervene before the relationship ends. The basic question sounds simple:

Which current customers are at elevated risk of churning?

But building a useful churn model requires more than loading a CSV file, training an algorithm, and printing an accuracy score.

What does “churn” mean? When will the prediction be made? Which information will be available at that moment? What happens when the model misses a customer who eventually leaves? What happens when it incorrectly targets a satisfied customer? Should every customer with a model probability above 50% enter a retention campaign?

Those are not merely machine-learning questions. They are business and system-design questions.

To help .NET developers explore them, I created a complete C# and ML.NET teaching exercise:

GitHub repository: AInDotNet.MLNET.CustomerChurn

The purpose is not simply to demonstrate ML.NET binary-classification syntax. The exercise teaches developers and architects how to reason about a Predictive AI classification problem—from data profiling and baselines through model validation, threshold selection, and business action.

What Is Customer Churn Prediction?

Customer churn prediction is a binary-classification problem. For each customer, the model estimates which of two outcomes is more likely:

• The customer stays.
• The customer churns.

The exercise uses the Telco Customer Churn dataset published on Kaggle by Blastchar. It contains 7,043 customer records and 21 columns describing customer demographics, subscribed services, contract terms, billing choices, charges, and churn status.

The historical target is the Churn column, represented as Yes or No.

The model uses the historical relationship between customer characteristics and churn outcomes to score another customer. A business could then use that score to prioritize customers for a retention workflow.

That final step matters. A prediction does not create business value merely because it exists. Someone must use it to make a decision and take an appropriate action.

Start with a Prediction Contract

Before training a model, the exercise defines the prediction contract:

• Business question: Which current customers are at elevated risk of churning?
• Model output: A churn score or probability and a threshold-based classification.
• Historical label: Churn = Yes or Churn = No.
• Prediction point: A snapshot of an existing customer before a retention decision.
• Available information: Demographics, tenure, services, contract, billing, payment method, and charges.
• Possible action: Prioritize selected customers for a retention workflow.
• Success criterion: Detect enough true churners to justify intervention without targeting so many non-churners that the program becomes uneconomic.

The dataset also has an important limitation: it does not define a future prediction horizon such as “will this customer churn within the next 30 days?” It also does not establish precisely when every feature was captured relative to the churn outcome.

That means this is an excellent classification dataset, but it cannot prove prospective production performance. A production system would need time-indexed observations, a clearly defined future outcome, and proof that every model feature was available at the actual prediction point.

Recognizing that limitation is part of the lesson. Predictive AI starts by defining the prediction correctly—not by choosing an algorithm.

Profile the Data Before Modeling

The application profiles the raw CSV before loading it into an ML.NET training pipeline.

The initial analysis finds:

• 7,043 customers
• 5,174 customers who stayed
• 1,869 customers who churned
• A churn rate of 26.5%
• Four numeric candidate features
• Fifteen categorical candidate features
• Eleven missing TotalCharges values

It also identifies customerID as an identifier. The application retains it so developers can inspect individual incorrect predictions, but it excludes the identifier from the feature vector. A customer number describes a record; it should not be treated as a generalizable customer characteristic.

The missing TotalCharges values are converted to NaN and handled deliberately inside the ML.NET preprocessing pipeline. The replacement value is learned when the pipeline is fitted to the training data. This avoids calculating an imputation value from validation or final-test customers.

These details may seem less exciting than AutoML, but this is where many real predictive systems succeed or fail. A sophisticated trainer cannot rescue a badly framed target, unavailable features, contaminated validation data, or misleading identifiers.

The First Surprise: A Useless Model Is Already 73.5% Accurate

Because 73.5% of the customers stayed, a naive rule that predicts “stay” for every customer would be 73.5% accurate.

That sounds respectable until we examine what the rule actually accomplishes:

• Churn recall: 0%
• Churn F1 score: 0
• False negatives: 1,869

It fails to identify every customer who churned.

This is the exercise’s first major lesson:

Accuracy can make a useless classifier look successful when one outcome is more common than the other.

Customer churn is not as extremely imbalanced as credit-card fraud, but the imbalance is large enough to demonstrate the danger. If the business cares about finding churners, it must evaluate more than total accuracy.

Precision and Recall Answer Different Business Questions

Two metrics become especially important.

Precision asks: When the model predicts churn, how often is it correct?

Low precision means the retention program contacts many customers who would have stayed anyway. Those false positives consume staff time, incentives, discounts, and customer attention.

Recall asks: Of all customers who actually churned, how many did the model identify?

Low recall means the business misses customers it potentially could have retained. Those false negatives represent lost intervention opportunities.

Neither metric is universally more important. The appropriate balance depends on the economics and operational constraints of the retention program.

If contacting a customer costs almost nothing, the business may tolerate more false positives to achieve higher recall. If an intervention includes an expensive discount or a personal call from a limited account-management team, precision may matter more.

The model metric cannot make that choice by itself.

Five Controlled ML.NET Experiments

The application uses a 60/20/20 training, validation, and final-test design:

• Training data fits the models.
• Validation data compares feature sets, selects the AutoML winner, and selects a threshold.
• Final-test data evaluates the already-selected model and decision policy on untouched customers.

This separation is essential. If we select a model or threshold using the final test population, the result is no longer an independent estimate of generalization.

Run A: Four Numeric Features

Run A trains an SDCA logistic-regression model using:

• SeniorCitizen
• tenure
• MonthlyCharges
• TotalCharges

The validation results were:

• Accuracy: 77.2%
• Precision: 69.7%
• Recall: 22.3%
• F1: 0.338
• ROC-AUC: 0.724

The model exceeds the naive baseline’s accuracy, but it identifies only 83 of 372 validation churners. It misses 289.

That is why reporting accuracy alone would tell an incomplete—and potentially dangerous—story.

Run B: Add Selected Business Context

Run B keeps the same trainer and data populations but adds four categorical features:

• Contract
• Internet service
• Payment method
• Paperless billing

The result does not improve:

• Accuracy: 76.8%
• Recall: 20.4%
• F1: 0.315
• ROC-AUC: 0.708

This creates another useful lesson:

Adding plausible features does not guarantee a better model.

The result does not prove that these fields lack predictive signal. It tells us that adding this feature group did not improve this particular model configuration on this validation population.

Run C: Use the Full Feature Set

Run C adds the remaining customer and service categories while retaining the SDCA trainer.

The validation results improve slightly over Run B:

• Accuracy: 77.8%
• Precision: 72.4%
• Recall: 23.9%
• F1: 0.360
• ROC-AUC: 0.723

The full feature set produces the strongest SDCA F1 score, but the model still misses most churners at the default classification threshold.

Run D: Let ML.NET AutoML Select a Trainer

Run D keeps the full feature set and preprocessing pipeline but allows ML.NET AutoML to search for a stronger binary-classification trainer and hyperparameters.

In the documented run, AutoML selected FastTreeBinary.

Validation performance increased substantially:

• Accuracy: 79.5%
• Precision: 63.0%
• Recall: 51.3%
• F1: 0.566
• ROC-AUC: 0.841

AutoML more than doubled recall compared with the full-feature SDCA model while producing a much stronger ROC-AUC.

However, developers should not become attached to one winning trainer. In another run, AutoML selected L-BFGS logistic regression with almost the same validation AUC. Time-limited AutoML searches may produce different winners across runs or machines even when overall model quality remains similar.

The transferable lesson is not that FastTree always wins. It is that algorithm choice and hyperparameters can materially affect classification behavior—and that the winning model must be evaluated using a sound validation process.

Run E: Change the Decision Threshold Without Retraining the Model

Binary classifiers commonly use a default probability threshold of 0.50:

• Probability below 0.50: predict stay
• Probability at or above 0.50: predict churn

But 0.50 is a software default, not a law of business.

Run E holds the AutoML model and validation customers constant while testing thresholds from 0.50 down to 0.30.

At 0.50, validation recall was 51.3%. The model missed 181 of 372 churners.

At 0.30, validation recall increased to 75.8%, and false negatives fell to 90. The cost was lower precision and more false positives.

The 0.30 threshold produced the highest validation F1 score among the thresholds tested, so the application selected it as a transparent teaching rule. It then applied that already-selected threshold to the untouched final-test population.

Did the Model and Threshold Generalize?

Yes. Validation and final-test results were close.

Metric	Validation	Final Test
ROC-AUC	0.841	0.844
F1 at the 0.50 threshold	0.566	0.561
F1 at the selected 0.30 threshold	0.630	0.622
Recall at the selected 0.30 threshold	75.8%	71.8%
Precision at the selected 0.30 threshold	53.9%	54.9%

On the final test population, the default 0.50 threshold produced:

• 179 true positives
• 83 false positives
• 197 false negatives
• 47.6% recall
• 68.3% precision

The validation-selected 0.30 threshold produced:

• 270 true positives
• 222 false positives
• 106 false negatives
• 71.8% recall
• 54.9% precision

Lowering the threshold identified 91 additional churners, but it also created 139 additional false positives.

The model did not change. The decision policy changed.

That is one of the most important concepts in applied Predictive AI.

There Is No Universally Correct Churn Threshold

The exercise uses the highest validation F1 score as a simple, visible threshold-selection rule. That is useful for teaching, but it does not prove that 0.30 maximizes business value.

A real organization should consider:

• The cost of contacting a customer
• The value of the customer relationship
• The cost of a retention incentive
• The probability that an intervention will work
• The harm caused by unnecessary contact or discounts
• The number of customers the retention team can handle
• Different policies for different customer segments

The best policy may not be one threshold for every customer.

A company might use a lower threshold for high-value accounts, a higher threshold for expensive interventions, and capacity-based ranking when only a fixed number of customers can be contacted each week.

This leads to the broader architecture:

Prediction → Decision → Action → Business Outcome

Training the model completes only the first step.

Inspect Individual Model Failures

Aggregate metrics describe overall behavior, but they do not explain individual mistakes.

The application lists the highest-confidence incorrect predictions and retains the customer ID for diagnostic use. These examples can help developers form new hypotheses:

• Is an important feature missing?
• Does the customer belong to an unusual segment?
• Is the historical label noisy?
• Did the customer experience an event that is absent from the dataset?
• Is the model systematically failing on a particular contract or service combination?

The point is not to manually explain every error. It is to use failures as evidence for the next controlled experiment.

From Teaching Exercise to Production Churn System

This repository is intentionally a focused console application. A production implementation would need more.

At minimum, an enterprise churn system should address:

• A precise, future-looking churn definition
• A documented operational prediction point
• Proof that every feature is available at that moment
• Time-based or out-of-time validation
• Multiple validation periods or repeated-split sensitivity analysis
• Data and model versioning
• Probability calibration when business rules depend on probabilities
• A cost-based or capacity-based decision policy
• Batch or real-time scoring integration
• Security and privacy controls
• Data-quality and model-drift monitoring
• Outcome tracking after interventions
• An experiment demonstrating that retention actions actually reduce churn

A model can predict churn accurately without improving retention. The organization must still choose an intervention, execute it consistently, and measure whether customers behave differently because of it.

Try the Customer Churn Exercise

The complete C# and ML.NET source code is available on GitHub:

Download or clone AInDotNet.MLNET.CustomerChurn

The repository includes:

• Data-profiling code
• Culture-safe CSV parsing
• Numeric missing-value handling
• One-hot encoding for categorical features
• A naive majority-class baseline
• Three controlled SDCA experiments
• ML.NET AutoML model selection
• Precision, recall, F1, ROC-AUC, and confusion matrices
• Individual failure analysis
• Validation-based threshold selection
• Independent final-test evaluation
• Model saving and reloading
• A hypothetical customer prediction
• Additional experiments to try yourself

The dataset is not included in the repository. Download it from the Kaggle Telco Customer Churn page, place the CSV in the Data folder, and run the solution using Visual Studio 2026 or the .NET 10 SDK.

The Larger Lesson

Customer churn prediction is not primarily an exercise in finding the algorithm with the highest accuracy.

It is an exercise in defining the prediction, understanding the data, establishing a meaningful baseline, evaluating the right errors, separating model selection from final testing, choosing a decision policy, and measuring whether the resulting action improves a business outcome.

That is the difference between training a machine-learning model and engineering a useful Predictive AI system.

To explore additional business applications, architecture guidance, and hands-on .NET exercises, visit Forecasting and Predictive AI with Microsoft AI.

Want More?

• Our hub for Forecasting and Predictive AI has many other resources for you
• Hands-On Predictive AI with ML.NET: What a Taxi Fare Experiment Teaches About Data, Features, and Models
• Predict House Prices with C# and ML.NET: A Practical Predictive AI Lab

Frequently Asked Questions