Skip to content

replit.md

Latest commit

 

History

History
46 lines (39 loc) · 4.89 KB

replit.md

File metadata and controls

46 lines (39 loc) · 4.89 KB
Raw

Comet Gemini - Streaming Chat Application

Overview

Comet Gemini is an AI chat application featuring real-time streaming capabilities and integration with an E2B desktop sandbox. Its primary purpose is to provide a highly interactive and responsive AI assistant experience, specifically designed for tasks requiring precise interaction with a desktop environment. The project aims to overcome common AI interaction challenges, such as incorrect coordinate generation and unresponsive UI, to deliver a seamless user experience.

User Preferences

  • Communication Style: I prefer simple language and clear, direct instructions.
  • Workflow: I want iterative development, where I see the effect of each action before the next is taken.
  • Interaction: Ask before making major changes. I need immediate feedback on actions.
  • Coding Style: I prefer a clean codebase with unused files and redundant code removed.
  • Agent Behavior:
    • AI should be able to click anywhere on the screen without limitations.
    • AI should perform actions sequentially, one at a time.
    • AI should not buffer events but process them immediately.
    • AI should not group message parts; each part should be a separate element.

System Architecture

UI/UX Decisions

  • Conditional Rendering: The application uses conditional rendering for desktop and mobile views to avoid dual DOM issues and ensure a single, optimized UI is present at any given time.
  • Real-time Rendering: flushSync() is used extensively to ensure immediate rendering of all updates, crucial for a real-time streaming experience.
  • GPU Optimizations: Messages are wrapped in RealtimeMessage components that include GPU optimizations for smoother performance.

Technical Implementations

  • AI Coordinate Validation: A three-level validation system for AI-generated coordinates includes:
    1. Coordinate Sanity Checklist: Critical rules are placed at the beginning of the prompt, reduced to four mandatory steps with clear examples. AI must define a target_region (e.g., "top-left") before providing coordinates.
    2. target_region Tool Parameter: The computer_use tool requires a target_region parameter, an enum of 9 screen regions (e.g., top-left, middle-center).
    3. Server-side Validation: Automatically calculates the actual region from [X, Y] coordinates and compares it with the declared target_region. Mismatches trigger a detailed error analysis, including correct ranges and examples.
  • Screenshot Metadata: Each screenshot includes detailed technical data (timestamp, resolution, aspect ratio, coordinate system, corner/edge coordinates) and a 3x3 ASCII grid visualization with coordinate ranges and examples for each region.
  • Clicking Behavior: Instructions explicitly state that there are no restrictions on where the AI can click, allowing clicks anywhere from (0,0) to (max_width-1, max_height-1). Coordinate validation has been removed from the server-side for left_click and right_click to ensure direct input to E2B.
  • Action Execution: AI actions are executed sequentially. Only the first tool call generated by the AI is processed, and its result is returned to the AI before it decides on the next action. This ensures controlled, step-by-step execution.
  • Typing Action: The type action now uses desktop.write(textToType) for correct text input, replacing the error-prone desktop.press() loop.
  • Continuous Action Loop: The while (true) loop for AI actions has been corrected to prevent premature stream closure, allowing the AI to perform an unlimited number of sequential actions until the task is complete.

System Design Choices

  • Real-time Streaming: Uses a custom JSON Lines format over fetch API for communication between the client and /api/chat/route.ts. Events (text-delta, tool-call-start, etc.) are streamed as JSON.stringify(event) + '\n'.
  • No Buffering/Grouping: ENABLE_BUFFERING and ENABLE_MESSAGE_GROUPING are explicitly set to false, and IMMEDIATE_PROCESSING to true in lib/streaming-config.ts to ensure immediate and granular event handling.
  • State Management: The lib/use-raw-streaming.ts hook (previously use-optimized-chat.ts) uses useState (instead of useReducer) and messagesRef for synchronous state tracking and immediate updates.
  • Cache Busting: Timestamps are included in every request to prevent caching issues.
  • Project Structure: Key components include app/page.tsx (main UI), app/api/chat/route.ts (OpenAI streaming endpoint), lib/use-raw-streaming.ts (streaming hook), and lib/streaming-config.ts.

External Dependencies

  • Next.js: Framework (version 15.2.1)
  • React: UI library (version 18+)
  • E2B Sandbox: Desktop environment integration for AI interactions (e.g., desktop.leftClick(x, y), desktop.write(textToType)).
  • OpenAI: Used for AI chat capabilities and generating tool calls.