Build Your Own Cuely-Style AI Assistant on Windows

2024 filed under: reflections

Cuely is an impressive AI assistant for macOS that lets you activate an AI helper with a simple keyboard shortcut, read your screen content, and provide contextual assistance. But what if you're on Windows? In this guide, I'll show you how I built a similar AI assistant for Windows using Electron, Tesseract.js for OCR, and the GPT-4o API.

What We're Building

Our Windows AI assistant will have these key features:

Prerequisites

Before we begin, make sure you have:

Technology Stack

Here's what powers our AI assistant:

Step 1: Project Setup

Create a new directory and initialize the project:

mkdir windows-ai-assistant
cd windows-ai-assistant
npm init -y

Install required dependencies:

npm install electron
npm install tesseract.js
npm install openai
npm install screenshot-desktop
npm install electron-store

Step 2: Basic Electron Structure

Create the main process file (main.js) that handles:

The window should be frameless, always on top, and positioned near the cursor when activated.

Step 3: Configure OpenAI API

Store your API key securely using electron-store or environment variables. Never hardcode API keys in your source code!

Create a settings panel where users can input their API key on first run. The key should be encrypted and stored locally.

Step 4: Implement Screen OCR

The OCR functionality involves:

  1. Capture screenshot  Use screenshot-desktop to grab the current screen
  2. Process with Tesseract  Extract text from the image
  3. Send to GPT-4o  Include the extracted text as context
  4. Display response  Show AI's answer in the interface

Tesseract.js can take a few seconds to process images, so include a loading indicator to show progress.

Step 5: Create the User Interface

Build a minimal, clean interface with:

Use CSS to create a modern, semi-transparent window with rounded corners and smooth animations.

Step 6: Hotkey Implementation

Register the global shortcut in the main process:

globalShortcut.register('CommandOrControl+\\', () => {
  mainWindow.show();
  mainWindow.focus();
});

Also implement:

Step 7: Integrate GPT-4o

Create an API wrapper that:

Example API call structure:

const messages = [
  { role: "system", content: "You are a helpful AI assistant." },
  { role: "user", content: userQuestion }
];

if (screenText) {
  messages.push({
    role: "user",
    content: `Screen content: ${screenText}`
  });
}

Step 8: Add Advanced Features

Conversation History

Store recent conversations so the AI can reference previous exchanges. Clear history when the window is closed or after a timeout.

Quick Actions

Implement shortcuts for common tasks:

Custom Prompts

Allow users to create custom prompt templates they use frequently.

Clipboard Integration

Option to automatically copy AI responses to clipboard for easy pasting.

Step 9: Optimize Performance

OCR Optimization

API Efficiency

Resource Management

Step 10: Package and Distribute

Use electron-builder to create an installer:

npm install --save-dev electron-builder

Configure in package.json:

"build": {
  "appId": "com.yourname.ai-assistant",
  "productName": "AI Assistant",
  "win": {
    "target": "nsis",
    "icon": "assets/icon.ico"
  }
}

Build the installer:

npm run build

Usage Tips

Screen Reading Best Practices

Effective Prompting

API Cost Management

Troubleshooting Common Issues

Hotkey Not Working

OCR Producing Gibberish

Slow Response Times

Privacy and Security Considerations

Consider adding:

Future Enhancements

Ideas to expand functionality:

Learning Resources

To dive deeper:

Conclusion

Building an AI assistant like Cuely for Windows is surprisingly achievable with modern web technologies. Electron makes desktop development accessible to web developers, Tesseract.js brings powerful OCR capabilities, and GPT-4o provides the intelligence.

This project taught me about:

The best part? Once you build it, you have a personalized AI assistant that works exactly how you want it to. You can customize the UI, add features specific to your workflow, and extend it indefinitely.

Start building, experiment, and create your own AI-powered productivity tool. The code is yoursmake it perfect for your needs.