Analyst Co-Pilot: Revolutionizing A/B Test Execution (powered by Human Intelligence & Antigravity AI)

📌 Project Overview & Business Impact

This repository showcases an elite-level analytical project completed at an unprecedented speed through a seamless symbiosis of human product intuition and AI execution. It explores an onboarding optimization feature in a chat application.

New users receive 20 bonus credits. Once their balance drops to 0, they encounter a hard paywall. We tested adding a "pinned chat reminder" when a user's balance drops to $\le 5$ credits to prompt earlier monetization.

Final Business Impact:

• Conversion Rate Lift: +5.65% increase in CR to the first payment.
• Statistical Significance: $p = 0.0123$ (statistically significant at $\alpha = 0.05$).
• Recommendation: 100% Rollout to the target audience.

🔗 Context & Original Requirements

• Candidate Profile: Artem Liakh
• Original Task Requirements: Notion Task Definition

📁 Repository Structure

├── .github/
│   └── assets/
│       ├── collab_banner.png
│       └── collab_banner_visual_plot.png
├── data/
│   ├── ab_test_data.csv
│   └── ab_test_historical_data.csv
├── notebooks/
│   ├── ab_test_analysis.ipynb
│   └── presentation_v2.html
├── AI_COLLABORATION.md
├── ANALYST_CO_PILOT_REPORT.md
├── LICENSE
└── README.md

🤖 Supercharged Productivity: The Analyst-AI Co-Pilot

This project represents a paradigm shift in data science workflows. By delegating boilerplate code, data wrangling, and statistical computation to the Antigravity AI Agent, the human Lead Product Scientist was freed up to focus entirely on high-level strategy:

• Defining the precise business hypotheses.
• Selecting the rigorous Intent-to-Treat (ITT) methodology.
• Translating pure statistical outputs into actionable, executive-level business recommendations.

This collaborative approach resulted in a 70% reduction in delivery time with zero compromise on analytical rigor. Read the deep-dive on our methodology and collaboration dynamics in AI_COLLABORATION.md.

🔬 Methodology Overview

1. Target Audience (Intent-to-Treat): Filtered historical and test data to only include users who genuinely reached a balance of $\le 5$ credits.
2. Power Analysis: Analyzed pre-pilot historical data to establish a baseline CR (~14.38%). Determined the required sample size (approx. 9,727 users per group) to detect a Minimum Detectable Effect (MDE) of 5% with 80% power.
3. Primary Metric Analysis: Applied a Z-test for Proportions to compare the CR between the Control and Test groups, calculating 95% Confidence Intervals (Wilson score).
4. Behavioral Analysis: Conducted a Mann-Whitney U test to analyze the Time-to-Payment distribution (see collab_banner_visual_plot.png), demonstrating that the pinned reminder successfully converted a highly hesitant segment of users.

🛠 Tech Stack

• Language: Python 3.x
• Data Manipulation: Pandas, NumPy
• Statistics: SciPy, Statsmodels
• Data Visualization: Seaborn, Matplotlib