Credit Card Fraud Detection with ML.NET: Why 99.8% Accuracy Can Still Fail

A machine learning model can achieve more than 99% accuracy and still fail completely at the problem it was designed to solve.

Credit card fraud detection is a perfect demonstration of this problem. Fraudulent transactions are extremely rare compared with legitimate transactions. A model can therefore appear highly accurate simply by predicting that every transaction is legitimate.

That is exactly what happens in Exercise 4 of the AInDotNet Predictive AI Lab series.

The new AInDotNet.MLNET.CreditCardFraud GitHub repository provides a complete C# and ML.NET teaching exercise covering:

• Severe class imbalance
• Majority-class baselines
• Precision, recall, F1, ROC-AUC, and PR-AUC
• Controlled training-data undersampling
• ML.NET AutoML
• Decision-threshold tuning
• Business-cost simulation
• Untouched final test data
• Model and decision-policy persistence
• Saved-artifact verification

The central lesson is simple:

The best predictive AI solution is not necessarily the model with the highest accuracy. It is the combination of model, threshold, and operational policy that produces the best business result.

The Credit Card Fraud Dataset

The exercise uses the public Credit Card Fraud Detection dataset available from Kaggle.

The original dataset contains:

• 284,807 transactions
• 284,315 legitimate transactions
• 492 fraudulent transactions
• 30 input features
• One binary target column named Class

A Class value of 0 identifies a legitimate transaction. A value of 1 identifies fraud.

Only approximately 0.173% of the original transactions are fraudulent. Put another way, legitimate transactions outnumber fraudulent transactions by roughly 578 to 1.

This imbalance makes ordinary classification accuracy dangerously misleading.

Why Accuracy Is the Wrong Starting Point

The exercise begins with a majority-class baseline. This baseline predicts that every transaction is legitimate.

On the validation population, that strategy produces:

• Accuracy: 99.817%
• Precision: 0%
• Recall: 0%
• F1 score: 0
• Fraudulent transactions detected: 0
• Fraudulent transactions missed: 104

The model is technically correct 99.817% of the time—but it fails to detect a single fraudulent transaction.

This is not a minor metric problem. It is a complete failure to answer the business question.

If the purpose of the system is to detect fraud, a model that detects no fraud is useless regardless of its overall accuracy.

That is why rare-event classification requires more than one headline metric.

The Metrics That Matter for Fraud Detection

Exercise 4 evaluates several complementary metrics.

Precision

Precision answers:

Of the transactions flagged as fraud, how many were actually fraudulent?

Low precision creates excessive false positives. In a real financial system, false positives may create unnecessary investigations, declined legitimate purchases, customer frustration, and additional operational expense.

Recall

Recall answers:

Of all fraudulent transactions, how many did the model detect?

Low recall means fraudulent transactions are being missed. Those false negatives may directly increase financial losses.

F1 score

F1 balances precision and recall at a particular decision threshold. It is useful when both missed fraud and false alerts matter, although it still does not assign a dollar value to either error.

ROC-AUC

ROC-AUC measures how well a model ranks positive examples above negative examples across decision thresholds.

It is useful, but it can appear optimistic when the positive class is extremely rare.

PR-AUC

Precision-recall area under the curve is generally more informative for highly imbalanced problems because it focuses directly on the model’s performance for the rare positive class.

Exercise 4 therefore configures AutoML to optimize PR-AUC rather than ordinary accuracy.

Remove Exact Duplicates Before Splitting the Data

The data profiler found 1,081 exact duplicate rows beyond their first occurrence:

• 1,062 duplicate legitimate transactions
• 19 duplicate fraudulent transactions

The exercise removes these duplicates before dividing the data into training, validation, and final test populations.

This order matters.

If duplicate observations are allowed to enter different splits, the model may be evaluated against transactions it effectively saw during training. That leakage can make validation and test results look better than performance on genuinely unseen data.

After duplicate removal, the exercise uses 283,726 unique observations:

• Training: 170,040 rows
• Validation: 56,764 rows
• Final test: 56,922 rows

The validation and final test populations retain the original class imbalance. Only specific training experiments rebalance the training data.

Run A: Establish the Majority-Class Baseline

The majority-class baseline predicts every transaction as legitimate.

It achieves 99.817% validation accuracy while producing zero fraud detections.

This baseline establishes an important minimum standard: a trained model must do more than generate an impressive-looking accuracy number. It must detect the rare class that matters.

Run B: Train FastTree on the Original Imbalanced Data

The first trained model uses the ML.NET FastTree binary classifier without resampling or class balancing.

Its validation results include:

• Positive precision: 57.664%
• Positive recall: 75.962%
• F1 score: 0.6556
• ROC-AUC: 0.9278
• PR-AUC: 0.4156
• Fraud detected: 79 of 104
• Fraud missed: 25
• False positives: 58

This model is dramatically more useful than the majority-class baseline. It detects approximately 76% of the fraudulent transactions.

However, the exercise does not stop with the first functional model.

Run C: Controlled Undersampling

The next experiment reduces the legitimate training examples while retaining every fraudulent training example.

The resulting training population contains approximately ten legitimate transactions for every fraudulent transaction:

• Legitimate training rows: 2,870
• Fraud training rows: 287
• Training fraud rate: 9.09%

The validation data remains untouched and retains its realistic class distribution.

This distinction is critical. Rebalancing may help the trainer learn the rare class, but evaluating against similarly rebalanced data would not represent the environment in which the model must operate.

The undersampled FastTree model produces:

• Positive precision: 45.545%
• Positive recall: 88.462%
• F1 score: 0.6013
• ROC-AUC: 0.9922
• PR-AUC: 0.7688
• Fraud detected: 92 of 104
• Fraud missed: 12
• False positives: 110

Undersampling materially improves fraud recall, but it also increases false positives.

That is a tradeoff—not an automatic improvement or failure.

Whether it is worthwhile depends on the relative costs of missed fraud, investigations, declined legitimate transactions, and customer friction.

Run D: Use AutoML to Search for a Stronger Model

The exercise then uses ML.NET AutoML to search for a better binary-classification pipeline while optimizing PR-AUC.

In the documented run, AutoML selected a LightGBM pipeline:

ReplaceMissingValues → Concatenate → LightGbmBinary

On the independent validation population, the winning AutoML model produced:

• Positive precision: 89.655%
• Positive recall: 75.000%
• F1 score: 0.8168
• ROC-AUC: 0.9520
• PR-AUC: 0.7792
• Fraud detected: 78 of 104
• Fraud missed: 26
• False positives: 9

The AutoML model delivered an excellent precision-versus-recall balance at its default threshold. It created far fewer false positives than either FastTree experiment.

But model selection is still not the same thing as deployment-policy selection.

A Model Score Is Not the Final Decision

A binary classifier produces a score or probability-like value. A decision threshold converts that value into a fraud or legitimate classification.

The default threshold is commonly 0.50, but there is nothing universally optimal about that value.

Changing the threshold changes the operational behavior of the system without retraining the model:

• Lower thresholds generally detect more fraud but create more false positives.
• Higher thresholds generally reduce false positives but may miss additional fraud.

Exercise 4 evaluates thresholds from 0.01 through 0.98 and reports every threshold where the confusion-matrix outcome changes.

For the AutoML model, the best validation F1 occurred at a threshold of 0.71:

• Precision: 92.857%
• Recall: 75.000%
• F1 score: 0.8298
• Fraud detected: 78
• Fraud missed: 26
• False positives: 6

Threshold tuning improved precision and F1 without reducing recall for this validation population.

However, the threshold with the best F1 is not automatically the threshold with the best financial outcome.

Turn Model Evaluation into a Business Decision

Technical metrics do not tell an organization how expensive each mistake is.

A false positive may require manual review, interrupt a transaction, or inconvenience a customer. A false negative may allow a fraudulent transaction and create a direct financial loss.

Exercise 4 therefore evaluates model-and-threshold combinations under different illustrative cost assumptions.

The simulator varies:

• Cost per false positive
• Missed-fraud loss multiplier
• Candidate model
• Decision threshold

Changing those assumptions changes the preferred solution.

For example, the documented sensitivity analysis selected different models or thresholds when:

• False-positive cost increased
• The financial effect of missed fraud increased
• The organization placed greater value on fraud recall
• The organization placed greater value on reducing unnecessary alerts

This is the real enterprise AI lesson.

There is no universally best fraud model independent of operational constraints and business economics.

The Selected Validation Policy

Using an illustrative cost of $25 for each false positive and one times the transaction amount for missed fraud, the validation process selected:

• Model: Undersampled FastTree
• Decision threshold: 0.98
• Fraud detected: 90 of 104
• Fraud missed: 14
• False positives: 31
• Precision: 74.380%
• Recall: 86.538%
• F1 score: 0.8000
• Estimated validation cost: $3,547.60

Notice that this was not the model with the highest validation F1.

The AutoML model produced the strongest F1 result, but the undersampled FastTree policy produced the lowest estimated cost under the selected economic assumptions.

That difference is precisely why model metrics and business policy must be evaluated separately.

Final Evaluation on Untouched Test Data

The selected model and threshold were then evaluated once against the untouched final test population.

The final results were:

• Transactions evaluated: 56,922
• Actual fraudulent transactions: 82
• Fraud detected: 65
• Fraud missed: 17
• False positives: 26
• Accuracy: 99.924%
• Precision: 71.429%
• Recall: 79.268%
• F1 score: 0.7514
• Fraud amount detected: $4,756.33
• Fraud amount missed: $1,925.02
• Estimated total cost: $2,575.02

The final test result is lower than the validation result, which is normal. The test population represents new data that was not used to choose the model, threshold, or business policy.

The purpose of an untouched test population is not to produce the most flattering metric. It is to provide a more honest estimate of performance on unseen data.

Save the Model and the Decision Policy Separately

The exercise generates two artifacts:

ModelArtifacts/CreditCardFraudModel.zip
ModelArtifacts/CreditCardFraudDecisionPolicy.json

The ML.NET model contains the learned prediction pipeline.

The JSON policy contains the selected operational threshold and the economic assumptions used during policy selection.

These are stored separately because the model alone does not define the business decision.

An organization may keep the same trained model while changing its threshold as fraud patterns, investigation capacity, customer-impact tolerance, or financial assumptions change.

The application reloads both artifacts and verifies:

• The saved model loads successfully.
• The reloaded model reproduces the expected prediction.
• The decision policy deserializes successfully.
• The reloaded policy matches the selected in-memory policy.

In the documented run, all artifact-verification checks passed.

What This Exercise Demonstrates

Credit card fraud detection is not simply a matter of training a binary classifier.

A defensible workflow must:

1. Profile the data before modeling.
2. Quantify the class imbalance.
3. Remove duplicate leakage before splitting.
4. Establish a majority-class baseline.
5. Preserve realistic validation and test distributions.
6. Evaluate precision, recall, F1, ROC-AUC, and PR-AUC.
7. Compare alternative class-imbalance strategies.
8. Inspect the model’s score distribution.
9. Tune the decision threshold separately from model training.
10. Translate prediction errors into business consequences.
11. Select the model and threshold using validation data.
12. Evaluate the selected policy once on untouched test data.
13. Save and verify both the model and its operational policy.

The result is not a production banking fraud system. The features are anonymized, the cost assumptions are illustrative, and transaction amount is only a proxy for potential fraud loss.

A production implementation would also require time-aware backtesting, original feature definitions, point-in-time validation, delayed-label handling, drift monitoring, investigation-capacity constraints, security controls, model governance, and institution-specific loss data.

Try the ML.NET Credit Card Fraud Exercise

The complete C# source code is available in the public GitHub repository:

AInDotNet.MLNET.CreditCardFraud

To run the exercise:

1. Clone or download the repository.
2. Download the dataset from Kaggle.
3. Place creditcard.csv in the project’s Data folder.
4. Restore the NuGet packages.
5. Build and run the .NET console application.
6. Review how each experiment changes precision, recall, false positives, false negatives, and estimated business cost.

Then modify the assumptions yourself.

Try a 5:1 or 20:1 undersampling ratio. Change the false-positive cost. Increase the missed-fraud multiplier. Add an analyst-review capacity limit. Replace the random split with chronological validation. Compare calibrated probabilities with the original model scores.

The important lesson is not that one trainer or threshold is always best.

The lesson is that predictive AI becomes useful only when model behavior is connected to a specific decision, an explicit cost structure, and an operational policy.

Want More?

Check out Predictive AI & Forecasting for Business