💰 Income Prediction ML Pipeline

End-to-End Machine Learning System for Income Classification

Live Demo • Report Bug • Request Feature

---

📋 Table of Contents

• Overview
• Key Features
• Architecture
• Tech Stack
• Project Structure
• Getting Started
  • Prerequisites
  • Installation
• Usage
  • Running the Pipeline
  • Web Application
  • API Endpoints
• Pipeline Components
• Model Performance
• Logging & Monitoring
• Future Roadmap
• Contributing
• Author
• License

---

🎯 Overview

An enterprise-grade, end-to-end machine learning system that predicts whether an individual's income exceeds $50K/year based on census data. This project demonstrates production-ready ML engineering practices including modular pipeline design, comprehensive logging, error handling, and deployment readiness.

Problem Statement: Classify individuals into income brackets (<=50K or >50K) using demographic and employment features to support financial planning and policy decisions.

---

✨ Key Features

🏗️ Production-Ready Architecture

• Modular Pipeline Design: Separate components for ingestion, transformation, and training
• Custom Exception Handling: Detailed error tracking with file names and line numbers
• Centralized Logging: Timestamped log directories for each execution instance
• Artifact Management: Organized storage of datasets, models, and preprocessors

🤖 Advanced ML Capabilities

• Automated Hyperparameter Tuning: GridSearchCV with 5-fold cross-validation
• Multi-Model Comparison: Logistic Regression, Decision Tree, Random Forest
• Robust Data Processing: Outlier handling (IQR method), imputation, scaling
• Model Serialization: Pickle-based model and preprocessor storage

🌐 Deployment & API

• Flask REST API: Production-ready web service
• Modern UI: Responsive web interface for predictions
• Cloud Deployment: Live on Render (free tier)
• Docker Ready: Containerization support (coming soon)

📊 Data Engineering

• Automated Train/Test Split: 70/30 stratified sampling
• Feature Engineering: Label encoding, target mapping, feature selection
• Data Validation: Schema validation and consistency checks
• Pipeline Persistence: Reusable transformation pipelines

---

🏛️ Architecture

┌─────────────────┐
│   Raw Data      │
│  (CSV Input)    │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│ Data Ingestion  │ ← Reads data, creates train/test splits
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│ Data Transform  │ ← Encoding, scaling, outlier handling
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│ Model Training  │ ← GridSearchCV, model selection
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  Model + Prep   │
│   Artifacts     │ ← Saved .pkl files
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  Flask API      │ ← Prediction endpoint
│  (Web Service)  │
└─────────────────┘

---

🛠️ Tech Stack

Category	Technologies
Language	Python 3.8+
ML/Data	scikit-learn, NumPy, Pandas
Visualization	Seaborn, Matplotlib
Web Framework	Flask
Serialization	Pickle, Joblib
Deployment	Render (Cloud Platform)
Version Control	Git, GitHub
Future	Docker, MLflow, DVC, CI/CD

---

📁 Project Structure

End-to-End-Income-Prediction/
│
├── 📂 artifacts/                    # Generated outputs
│   ├── data_ingestion/
│   │   ├── train.csv                # Training dataset (70%)
│   │   ├── test.csv                 # Test dataset (30%)
│   │   └── data.csv                 # Raw data
│   ├── data_transformation/
│   │   └── preprocessor.pkl         # Fitted preprocessing pipeline
│   └── model_trainer/
│       └── model.pkl                # Best trained model (~2.5 MB)
│
├── 📂 logs/                         # Timestamped execution logs
│   └── YYYY-MM-DD_HH-MM-SS/
│       └── YYYY-MM-DD_HH-MM-SS.log
│
├── 📂 notebook/                     # EDA and experimentation
│   └── data/
│       └── income_clean_data.csv
│
├── 📂 src/                          # Source code
│   ├── __init__.py
│   ├── logger.py                    # Logging configuration
│   ├── exceptions.py                # Custom exception classes
│   ├── utils.py                     # Helper functions
│   │
│   ├── 📂 components/               # Pipeline components
│   │   ├── data_ingestion.py       # Data loading & splitting
│   │   ├── data_transformation.py  # Feature engineering
│   │   └── model_trainer.py        # Model training & tuning
│   │
│   └── 📂 pipeline/                 # Inference pipelines
│       ├── predict_pipeline.py     # Production predictions
│       └── train_pipeline.py       # Training orchestration
│
├── 📂 templates/                    # HTML templates
│   ├── home.html                    # Input form
│   └── results.html                 # Prediction results
│
├── app.py                           # Flask application
├── requirements.txt                 # Python dependencies
├── setup.py                         # Package configuration
├── .gitignore
└── README.md

---

🚀 Getting Started

Prerequisites

• Python 3.8 or higher
• pip or conda package manager
• Git

Installation

Option 1: Virtual Environment (venv)

macOS/Linux:

# Clone the repository
git clone https://github.com/vansh-09/End-to-End-Income-Prediction.git
cd End-to-End-Income-Prediction

# Create and activate virtual environment
python3 -m venv venv
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt

Windows:

# Clone the repository
git clone https://github.com/vansh-09/End-to-End-Income-Prediction.git
cd End-to-End-Income-Prediction

# Create and activate virtual environment
python -m venv venv
venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

Option 2: Conda Environment

# Clone the repository
git clone https://github.com/vansh-09/End-to-End-Income-Prediction.git
cd End-to-End-Income-Prediction

# Create conda environment
conda create -n income_pred python=3.10 -y
conda activate income_pred

# Install dependencies
pip install -r requirements.txt

---

💻 Usage

Running the Pipeline

Execute the complete ML pipeline (data ingestion → transformation → training):

python3 -m src.components.data_ingestion

Pipeline Execution Flow:

1. ✅ Data Ingestion: Loads income_clean_data.csv from notebook/data/
2. ✅ Train/Test Split: Creates 70/30 stratified split
3. ✅ Data Transformation:
   • Label encodes categorical features
   • Maps target variable (<=50K → 0, >50K → 1)
   • Removes unnecessary columns
   • Handles outliers using IQR method
   • Applies median imputation + StandardScaler
4. ✅ Model Training:
   • Trains Logistic Regression, Decision Tree, Random Forest
   • Performs GridSearchCV with 5-fold CV
   • Saves best model to artifacts/model_trainer/model.pkl

Web Application

Start the Flask server:

python3 app.py

Access the application at: http://127.0.0.1:5001

Features:

• Modern, responsive UI
• Real-time predictions
• Form validation
• Error handling

API Endpoints

POST /predict

Request Body:

{
  "age": 39,
  "workclass": "State-gov",
  "education_num": 13,
  "marital_status": "Never-married",
  "occupation": "Adm-clerical",
  "relationship": "Not-in-family",
  "race": "White",
  "sex": "Male",
  "capital_gain": 2174,
  "capital_loss": 0,
  "hours_per_week": 40
}

Response:

{
  "prediction": "Income <=50K"
}

---

🔧 Pipeline Components

1. Data Ingestion (data_ingestion.py)

Responsibilities:

• Load raw CSV data
• Perform train/test split (70/30)
• Save datasets to artifacts/data_ingestion/

Key Features:

• Configurable data paths
• Automated directory creation
• Logging of dataset statistics

2. Data Transformation (data_transformation.py)

Preprocessing Steps:

# Categorical Encoding
features = ['workclass', 'education', 'marital-status', 'occupation', 
            'relationship', 'race', 'gender', 'native-country']

# Target Encoding
income: {'<=50K': 0, '>50K': 1}

# Feature Selection
Drop: ['education', 'fnlwgt', 'native-country']

# Outlier Handling
Method: IQR (Interquartile Range) capping

# Numeric Processing
Pipeline: MedianImputer → StandardScaler

Output:

• preprocessor.pkl: Fitted transformation pipeline

3. Model Training (model_trainer.py)

Models Evaluated:

Model	Hyperparameters	CV Folds
Logistic Regression	C, penalty, solver	5
Decision Tree	max_depth, min_samples_split	5
Random Forest	n_estimators, max_features	5

Selection Criteria:

• Best accuracy score on test set
• Model saved to artifacts/model_trainer/model.pkl

---

📊 Model Performance

Metric	Score
Training Accuracy	~85%
Test Accuracy	~83%
Precision	~0.80
Recall	~0.78
F1-Score	~0.79

Feature Importance (Top 5):

1. Capital Gain
2. Education Level
3. Age
4. Hours per Week
5. Relationship Status

---

📝 Logging & Monitoring

Centralized Logging System

Log Structure:

logs/
├── 2026-01-29_19-56-06/
│   └── 2026-01-29_19-56-06.log
├── 2026-01-29_19-57-23/
│   └── 2026-01-29_19-57-23.log

Log Entry Format:

[2026-01-29 19:56:06,123] - src.components.data_ingestion - INFO - Line 45 - Data ingestion started
[2026-01-29 19:56:07,456] - src.components.data_transformation - INFO - Line 78 - Preprocessing complete

Benefits:

• Timestamped execution tracking
• Module-level granularity
• Easy debugging and auditing
• Complete execution isolation

---

🔮 Future Roadmap

Phase 1: MLOps Integration

• Experiment Tracking: MLflow integration
• Data Versioning: DVC (Data Version Control)
• Model Registry: Centralized model management
• Performance Monitoring: Drift detection

Phase 2: DevOps & Deployment

• Containerization: Docker + Docker Compose
• CI/CD Pipeline: GitHub Actions
• Cloud Deployment: AWS/GCP/Azure
• Load Balancing: Horizontal scaling

---

🤝 Contributing

Contributions are welcome! Please follow these steps:

1. Fork the repository
2. Create a feature branch (git checkout -b feature/AmazingFeature)
3. Commit changes (git commit -m 'Add AmazingFeature')
4. Push to branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

Contribution Guidelines:

• Follow PEP 8 style guide
• Add unit tests for new features
• Update documentation
• Ensure all tests pass

---

👤 Author

Vansh Jain

• 📧 Email: vanshkjain09@gmail.com
• 💼 LinkedIn: Connect with me
• 🐙 GitHub: @vansh-09

---

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

---

🙏 Acknowledgments

• Dataset: UCI Machine Learning Repository (Adult Income Dataset)
• Inspiration: Real-world census data analysis
• Community: Open-source ML community

---

⭐ Star this repository if you find it helpful!

Report Issues • View Live Demo

Made with ❤️ by Vansh Jain