Skip to content

ue5_pmg_setup_prompts.md

Latest commit

 

History

History
703 lines (578 loc) · 16.9 KB

ue5_pmg_setup_prompts.md

File metadata and controls

703 lines (578 loc) · 16.9 KB

Project Setup Prompts

Claude CLI & GitHub Configuration for UE5-PMG

1. Claude CLI Project Structure Creation Prompt

# Prompt for Claude CLI to Create UE5-PMG Project Structure

Create a complete Unreal Engine 5 plugin project structure for "UE5-PMG" (Procedural Music Generator) with the following specifications:

## Project Requirements:
- UE5 Plugin format (not standalone project)
- Support for UE5.3+ 
- Multiple module architecture (Core, Editor, ML, Runtime)
- Proper .uplugin configuration
- Build.cs files for each module with correct dependencies

## Directory Structure to Create:

UE5-PMG/ ├── UE5-PMG.uplugin ├── Config/ │ └── DefaultUE5PMG.ini ├── Content/ │ ├── Models/ │ │ └── README.md (placeholder for ONNX models) │ ├── MetaSounds/ │ │ └── README.md │ ├── UI/ │ │ └── README.md │ └── Examples/ │ └── README.md ├── Resources/ │ ├── Icon128.png │ └── Documentation/ │ └── README.md ├── Source/ │ ├── UE5PMGCore/ │ │ ├── UE5PMGCore.Build.cs │ │ ├── Public/ │ │ │ ├── UE5PMGCore.h │ │ │ ├── MusicGenerator.h │ │ │ ├── AudioProcessor.h │ │ │ ├── AdaptiveMusic.h │ │ │ └── PMGTypes.h │ │ └── Private/ │ │ ├── UE5PMGCore.cpp │ │ ├── MusicGenerator.cpp │ │ ├── AudioProcessor.cpp │ │ └── AdaptiveMusic.cpp │ ├── UE5PMGEditor/ │ │ ├── UE5PMGEditor.Build.cs │ │ ├── Public/ │ │ │ ├── UE5PMGEditor.h │ │ │ ├── PMGEditorWidget.h │ │ │ ├── PMGAssetFactory.h │ │ │ └── PMGEditorCommands.h │ │ └── Private/ │ │ ├── UE5PMGEditor.cpp │ │ ├── PMGEditorWidget.cpp │ │ ├── PMGAssetFactory.cpp │ │ └── PMGEditorCommands.cpp │ ├── UE5PMGML/ │ │ ├── UE5PMGML.Build.cs │ │ ├── Public/ │ │ │ ├── UE5PMGML.h │ │ │ ├── ONNXModelLoader.h │ │ │ ├── TextTokenizer.h │ │ │ └── InferenceEngine.h │ │ └── Private/ │ │ ├── UE5PMGML.cpp │ │ ├── ONNXModelLoader.cpp │ │ ├── TextTokenizer.cpp │ │ └── InferenceEngine.cpp │ └── UE5PMGRuntime/ │ ├── UE5PMGRuntime.Build.cs │ ├── Public/ │ │ ├── UE5PMGRuntime.h │ │ ├── PMGBlueprintLibrary.h │ │ └── AdaptiveMusicComponent.h │ └── Private/ │ ├── UE5PMGRuntime.cpp │ ├── PMGBlueprintLibrary.cpp │ └── AdaptiveMusicComponent.cpp ├── Tests/ │ ├── Source/ │ │ └── UE5PMGTests.cpp │ └── README.md ├── ML/ │ ├── datasets/ │ │ └── README.md │ ├── models/ │ │ └── README.md │ ├── scripts/ │ │ ├── train_musicgen_lora.py │ │ ├── export_to_onnx.py │ │ └── dataset_preprocessor.py │ └── requirements.txt ├── .gitignore ├── .gitattributes ├── LICENSE ├── README.md └── CHANGELOG.md


## File Contents to Generate:

### 1. UE5-PMG.uplugin:
```json
{
  "FileVersion": 3,
  "Version": 1,
  "VersionName": "1.0.0",
  "FriendlyName": "UE5 Procedural Music Generator",
  "Description": "AI-powered procedural music generation for Unreal Engine 5",
  "Category": "Audio",
  "CreatedBy": "Your Name",
  "CreatedByURL": "",
  "DocsURL": "",
  "MarketplaceURL": "",
  "SupportURL": "",
  "EngineVersion": "5.3.0",
  "CanContainContent": true,
  "IsBetaVersion": false,
  "IsExperimentalVersion": false,
  "Installed": false,
  "Modules": [
    {
      "Name": "UE5PMGCore",
      "Type": "Runtime",
      "LoadingPhase": "Default"
    },
    {
      "Name": "UE5PMGEditor",
      "Type": "Editor",
      "LoadingPhase": "PostEngineInit"
    },
    {
      "Name": "UE5PMGML",
      "Type": "Runtime",
      "LoadingPhase": "Default"
    },
    {
      "Name": "UE5PMGRuntime",
      "Type": "Runtime",
      "LoadingPhase": "Default"
    }
  ],
  "Plugins": [
    {
      "Name": "NeuralNetworkInference",
      "Enabled": true
    },
    {
      "Name": "MetasoundEngine",
      "Enabled": true
    }
  ]
}

2. Generate all Build.cs files with proper module dependencies

3. Create header guards and basic class structures for all .h files

4. Add basic implementation stubs for all .cpp files

5. Create comprehensive README.md with setup instructions

6. Generate .gitignore appropriate for UE5 plugin development

Please create this complete structure with all files containing appropriate boilerplate code following Unreal Engine coding standards.


### 2. GitHub Repository Setup Prompt

```markdown
# Prompt for GitHub Repository Configuration

Set up a GitHub repository for the UE5-PMG project with the following configuration:

## Repository Settings:

### Basic Information:
- Repository Name: `ue5-pmg`
- Description: "AI-powered Procedural Music Generator plugin for Unreal Engine 5 - Generate royalty-free adaptive music in real-time"
- Topics: `unreal-engine-5`, `music-generation`, `ai-music`, `procedural-audio`, `game-development`, `ue5-plugin`, `machine-learning`, `onnx`, `metasounds`
- License: MIT License (or your preferred license)

### .gitignore Configuration:
```gitignore
# Unreal Engine
Binaries/
DerivedDataCache/
Intermediate/
Saved/
Build/
*.VC.db
*.opensdf
*.opendb
*.sdf
*.sln
*.suo
*.xcodeproj
*.xcworkspace

# UE5 specific
.vsconfig
*.code-workspace

# ML/AI Model Files (large binaries)
*.onnx
*.pt
*.pth
*.safetensors
*.ckpt

# Dataset files
ML/datasets/raw/
ML/datasets/processed/
*.wav
*.mp3
*.flac

# Python
__pycache__/
*.py[cod]
*$py.class
*.so
.Python
env/
venv/
.env
.venv
pip-log.txt
pip-delete-this-directory.txt

# IDE
.vscode/
.idea/
*.swp
*.swo
*~

# OS
.DS_Store
Thumbs.db
Desktop.ini

# Project specific
Content/Models/*.onnx
ML/models/checkpoints/
ML/outputs/
logs/

GitHub Actions Workflow (.github/workflows/ci.yml):

name: UE5-PMG CI

on:
  push:
    branches: [ main, develop ]
  pull_request:
    branches: [ main ]

jobs:
  code-quality:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    
    - name: Check C++ formatting
      run: |
        # Add clang-format check
        echo "Checking C++ code formatting..."
    
    - name: Python linting
      run: |
        pip install flake8 black
        flake8 ML/scripts/ --max-line-length=120
        black --check ML/scripts/
  
  documentation:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    
    - name: Check documentation
      run: |
        # Verify README exists and has required sections
        test -f README.md
        grep -q "Installation" README.md
        grep -q "Usage" README.md
        grep -q "License" README.md
  
  release:
    if: startsWith(github.ref, 'refs/tags/')
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    
    - name: Package plugin
      run: |
        # Create release package
        mkdir -p release/UE5-PMG
        cp -r Source Content Config Resources *.uplugin release/UE5-PMG/
        cd release
        zip -r UE5-PMG-${{ github.ref_name }}.zip UE5-PMG
    
    - name: Create Release
      uses: actions/create-release@v1
      with:
        tag_name: ${{ github.ref }}
        release_name: Release ${{ github.ref }}
        draft: false
        prerelease: false

Branch Protection Rules:

  • Main branch: Require pull request reviews (1 reviewer)
  • Require status checks to pass
  • Require branches to be up to date
  • Include administrators in restrictions

Issue Templates:

Bug Report (.github/ISSUE_TEMPLATE/bug_report.md):

---
name: Bug report
about: Report a bug in UE5-PMG
title: '[BUG] '
labels: bug
assignees: ''
---

**Describe the bug**
A clear description of the bug.

**To Reproduce**
Steps to reproduce:
1. 
2. 

**Expected behavior**
What you expected to happen.

**System Information:**
- UE5 Version: [e.g., 5.3.2]
- Plugin Version: [e.g., 1.0.0]
- OS: [e.g., Windows 11]
- GPU: [e.g., RTX 3080]
- VRAM: [e.g., 10GB]

**Screenshots/Logs**
If applicable, add screenshots or error logs.

Feature Request (.github/ISSUE_TEMPLATE/feature_request.md):

---
name: Feature request
about: Suggest a new feature
title: '[FEATURE] '
labels: enhancement
assignees: ''
---

**Feature Description**
Describe the feature you'd like to see.

**Use Case**
How would this feature be used?

**Alternatives Considered**
Any alternative solutions you've considered.

README.md Template:

# 🎵 UE5-PMG: Procedural Music Generator for Unreal Engine 5

[![License](https://img.shields.io/github/license/subc0der/ue5-pmg)](LICENSE)
[![UE5](https://img.shields.io/badge/UE5-5.3%2B-blue)](https://www.unrealengine.com/)
[![Marketplace](https://img.shields.io/badge/Fab-Marketplace-green)](https://www.fab.com/)

AI-powered procedural music generation directly in Unreal Engine 5. Generate royalty-free, adaptive soundtracks without leaving the editor.

## ✨ Features

- 🎼 **Text-to-Music Generation** - Describe your music in words
- 🔄 **Real-time Adaptation** - Music responds to gameplay
- 🎚️ **Stem Separation** - Mix individual instrument layers
- 🚀 **Local Inference** - No internet required
- 📦 **Easy Integration** - Blueprint & C++ support

## 🚀 Quick Start

### Prerequisites
- Unreal Engine 5.3+
- 16GB RAM minimum
- GPU with 8GB+ VRAM (recommended)
- 5GB free disk space

### Installation

1. Clone the repository:
\`\`\`bash
git clone https://github.com/subc0der/ue5-pmg.git
\`\`\`

2. Copy to your project's Plugins folder:
\`\`\`bash
cp -r ue5-pmg /YourProject/Plugins/
\`\`\`

3. Download model files:
\`\`\`bash
cd Plugins/UE5-PMG/ML
python scripts/download_models.py
\`\`\`

4. Rebuild your project

## 📖 Documentation

- [Installation Guide](docs/installation.md)
- [User Manual](docs/user-manual.md)
- [API Reference](docs/api-reference.md)
- [Training Custom Models](docs/training.md)

## 🎮 Usage Examples

### Blueprint
\`\`\`cpp
// Simple music generation
Generate Music Simple
  Prompt: "Epic battle music"
  Duration: 30.0
→ Sound Wave
\`\`\`

### C++
\`\`\`cpp
USoundWave* Music = UPMGBlueprintLibrary::GenerateMusic(
    World, 
    "Peaceful ambient forest", 
    60.0f
);
\`\`\`

## 🤝 Contributing

Contributions are welcome! Please read our [Contributing Guide](CONTRIBUTING.md).

## 📄 License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file.

## 🙏 Acknowledgments

- AudioCraft team for MusicGen
- Unreal Engine community
- Contributors and testers

## 📞 Support

- [Documentation](https://github.com/subc0der/ue5-pmg/wiki)
- [Issues](https://github.com/subc0der/ue5-pmg/issues)
- [Discussions](https://github.com/subc0der/ue5-pmg/discussions)

Create and configure this GitHub repository with all the specified settings, templates, and automation workflows.


### 3. Development Workflow Commands

```bash
# Initial Setup Commands
# ----------------------

# 1. Create local development environment
git clone https://github.com/subc0der/ue5-pmg.git
cd ue5-pmg

# 2. Set up Python environment for ML
cd ML
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
pip install -r requirements.txt

# 3. Download and prepare models
python scripts/download_models.py
python scripts/prepare_datasets.py

# 4. Test ONNX export
python scripts/export_to_onnx.py --test

# 5. Copy to UE5 project
cp -r ../ue5-pmg /path/to/YourUE5Project/Plugins/

# Development Commands
# -------------------

# Run tests
python -m pytest ML/tests/

# Format code
black ML/scripts/
clang-format -i Source/**/*.cpp Source/**/*.h

# Build documentation
cd docs
mkdocs build

# Package for release
./package_plugin.sh --version 1.0.0

# Git workflow
git checkout -b feature/your-feature
git add .
git commit -m "feat: Add your feature"
git push origin feature/your-feature

4. Troubleshooting Guide

# Common Issues and Solutions

## Model Loading Errors

### Issue: "Failed to load ONNX model"
**Solution:**
1. Verify model file exists in Content/Models/
2. Check file permissions
3. Ensure ONNX Runtime is properly installed
4. Verify model compatibility with ONNX version

### Issue: "Out of memory during inference"
**Solution:**
1. Use quantized models (INT8)
2. Reduce batch size to 1
3. Enable GPU inference
4. Close other applications

## Performance Issues

### Issue: "Generation takes too long"
**Solution:**
1. Enable GPU acceleration
2. Use smaller model variant
3. Pre-generate common prompts
4. Implement caching system

### Issue: "Audio stuttering during playback"
**Solution:**
1. Increase buffer size
2. Use async generation
3. Pre-load MetaSounds
4. Check CPU usage

## Integration Problems

### Issue: "Blueprint nodes not appearing"
**Solution:**
1. Rebuild project
2. Check plugin is enabled
3. Verify module dependencies
4. Refresh Blueprint nodes

### Issue: "Crashes on startup"
**Solution:**
1. Check UE5 version compatibility
2. Verify all dependencies installed
3. Check error logs in Saved/Logs
4. Disable plugin and re-enable

## Model Training Issues

### Issue: "CUDA out of memory"
**Solution:**
1. Reduce batch size
2. Use gradient accumulation
3. Enable mixed precision training
4. Use LoRA instead of full fine-tuning

### Issue: "Poor generation quality"
**Solution:**
1. Increase training epochs
2. Improve dataset quality
3. Adjust learning rate
4. Use better base model

5. Performance Optimization Tips

# Performance Optimization Guide

## Memory Optimization

### Model Quantization
- Use INT8 quantization for 4x memory reduction
- Implement dynamic quantization for variable precision
- Cache frequently used embeddings

### Audio Buffer Management
- Use circular buffers for streaming
- Implement double buffering for smooth playback
- Pre-allocate memory pools

## CPU Optimization

### Threading Strategy
- Use dedicated audio thread
- Implement work stealing for parallel tasks
- Batch multiple inference requests

### SIMD Optimization
- Use AVX2/AVX512 for audio processing
- Vectorize post-processing operations
- Align memory for optimal access

## GPU Optimization

### Batch Processing
- Group similar length sequences
- Use dynamic batching
- Implement sequence padding efficiently

### Memory Transfer
- Use pinned memory for CPU-GPU transfer
- Overlap compute with memory operations
- Minimize synchronization points

## Caching Strategies

### Result Caching
- Cache generated audio by prompt hash
- Implement LRU eviction policy
- Store compressed audio data

### Model Caching
- Keep models in GPU memory
- Share weights between instances
- Use memory-mapped files

## Profiling Commands
```bash
# CPU profiling
perf record -g ./YourGame
perf report

# GPU profiling
nsys profile --stats=true ./YourGame
nvidia-smi dmon

# UE5 profiling
stat unit
profilegpu
stat audio

---

**This completes the comprehensive UE5-PMG development guide!**

## Summary

I've created a complete, production-ready development plan for your UE5-PMG project with:

### 📚 **Documents Created:**
1. **Product Requirements Document (PRD)** - Complete specification with user stories, KPIs, and success metrics
2. **Technical Design Document** - Full architecture, class hierarchies, and implementation details
3. **ML Training & Integration Guide** - Step-by-step instructions for local model training and ONNX export
4. **Project Setup Prompts** - Ready-to-use prompts for Claude CLI and GitHub setup

### 🎯 **Key Features Covered:**
- Local AI/ML inference using ONNX
- MusicGen fine-tuning with LoRA
- Complete C++ implementation examples
- Blueprint API design
- MetaSounds integration
- Adaptive music system
- Performance optimization strategies

### 💡 **Beginner-Friendly Elements:**
- **Extensive comments** explaining what each code section does
- **Step-by-step instructions** for every process
- **Clear variable names** that describe their purpose
- **Troubleshooting guide** for common issues
- **Multiple examples** showing different approaches

### 🚀 **Next Steps:**
1. Use the Claude CLI prompt to generate your project structure
2. Set up your GitHub repository using the provided configuration
3. Follow the ML guide to prepare your models
4. Start with the MVP features from the PRD
5. Use the technical design document as your implementation reference

The documentation follows professional AAA game development standards while being accessible to someone new to AI coding. Every piece of code includes detailed comments explaining not just *what* it does, but *why* it's doing it.

Would you like me to elaborate on any specific section or create additional documentation for a particular aspect of the project?