MarketMind — End-to-End MLOps Pipeline for Stock Prediction & AI Market Reports

An end-to-end automated system for weekly stock market analysis using Transfer Learning (LSTM) and Agentic AI (LangGraph).

---

Overview

MarketMind is a production-grade MLOps pipeline that automates the full lifecycle of stock price prediction and financial report generation. It moves beyond monolithic ML scripts into a modular, scalable, observable architecture built on modern MLOps tooling.

Key Capabilities

• Transfer Learning: A parent LSTM model is trained on the S&P 500 index and fine-tuned per ticker for accurate predictions with minimal data.
• Agentic AI: A multi-agent LangGraph system acts as financial analyst, market expert, and editor to generate analyst-quality reports.
• Real-time Serving: Low-latency predictions via FastAPI with Redis caching (24h TTL).
• Feature Store: Consistent feature management across training and serving via Feast.
• Observability: Full-stack monitoring with Prometheus, Grafana, and Evidently AI for data drift detection.
• Semantic Caching: Embedded reports stored in Qdrant — semantically similar queries (95%+ match) within 24h hit the cache.

---

Tech Stack

Component	Technology
Model	PyTorch (LSTM)
LLM Engine	Ollama (gpt-oss:20b-cloud)
Embeddings	Ollama (nomic-embed-text)
AI Agents	LangGraph, LangChain
Feature Store	Feast
Registry	MLflow (via DagsHub)
Vector DB	Qdrant
Cache	Redis Stack
Backend	FastAPI (async)
Frontend	Streamlit
Observability	Prometheus, Grafana, Evidently AI

---

Architecture

graph TB
    subgraph "User Layer"
        UI[Streamlit UI]
        MON_UI[Monitoring Dashboard]
    end

    subgraph "Logic Layer"
        API[FastAPI Orchestrator]
        TRAIN[Training Pipeline]
        AGENT[LangGraph Agents]
    end

    subgraph "Storage & Memory"
        REDIS[(Redis Cache)]
        QDRANT[(Qdrant Vector DB)]
        FEAST[(Feast Feature Store)]
        MLFLOW[DagsHub MLflow]
    end

    UI --> API
    API --> TRAIN
    API --> AGENT
    TRAIN --> FEAST
    TRAIN --> MLFLOW
    AGENT --> QDRANT
    API --> REDIS

Loading

---

Getting Started

1. Prerequisites

• Docker & Docker Compose
• Ollama running on the host
• UV Python package manager
• A free Finnhub API key
• A free DagsHub account for remote MLflow tracking

2. Install Ollama Models

ollama pull nomic-embed-text
ollama signin   # one-time, for the cloud-hosted gpt-oss:20b-cloud model

3. Clone & Configure

git clone https://github.com/RupinderSingh1313/MarketMind.git
cd MarketMind

Create a .env file in the project root:

DAGSHUB_USER_NAME=
DAGSHUB_REPO_NAME=
DAGSHUB_TOKEN=
MLFLOW_TRACKING_URI=https://dagshub.com/<user>/<repo>.mlflow
REDIS_HOST=localhost
REDIS_PORT=6379
GOOGLE_API_KEY=
FMI_API_KEY=

4. Launch the Stack

docker compose up --build -d

5. Access the Apps

• Streamlit UI: http://localhost:8501
• Monitoring Dashboard: http://localhost:8502
• FastAPI Docs: http://localhost:8000/docs
• Grafana: http://localhost:3000 (admin / admin)
• Prometheus: http://localhost:9090
• Qdrant: http://localhost:6333/dashboard

---

Agentic AI Workflow

The system uses 4 specialized agents coordinated by LangGraph:

1. Performance Analyst — interprets raw LSTM forecasts and technical indicators.
2. Market Expert — pulls latest news and sentiment using Yahoo Finance tooling.
3. Report Generator — synthesizes everything into a professional financial markdown report.
4. Critic — reviews the output for consistency and logic before final delivery.

---

MLOps Practices

• Auto-Healing: The API detects missing models and triggers training automatically.
• Model Registry: Every training run is logged to DagsHub with artifacts (scalers, plots, metrics).
• Drift Detection: Evidently AI runs scheduled checks for feature drift in stock data.
• Transfer Learning:
  • Parent: trained on ^GSPC (S&P 500)
  • Child: fine-tuned on individual tickers (e.g. NVDA, AAPL, GOOG)

---

License

Distributed under the MIT License.