Project overview
Explore the repository, its official documentation and the recorded Trending history.
Recorded Trending signals
Usagi-org/ai-goofish-monitor was recorded 2 times between 2026-11-04 and 2026-11-06. Its best recorded rank is #4. The latest saved snapshot reports 242 daily Stars and 17,498 total Stars. These are archive observations, not quality ratings or continuous activity measurements.
Official project documentation
The following is an excerpt from the project’s official README, fetched from GitHub. It may change independently of this archive. RepoWatch has not installed or functionally tested this project.
Xianyu Intelligent Monitor Bot
中文 | [English]
A Playwright and AI-powered multi-task real-time monitoring tool for Xianyu (闲鱼), featuring a complete web management interface.
Core Features
- Web Visual Management: Task management, account management, AI criteria editing, run logs, results browsing
- AI-Driven: Natural language task creation, multimodal model for in-depth product analysis
- Multi-Task Concurrency: Independent configuration for keywords, prices, filters, and AI prompts
- SQLite as Primary Storage: Tasks, results, and price history are persisted in one embedded database instead of repeatedly scanning
jsonl - Advanced Filtering: Free shipping, new listing time range, province/city/district filtering
- Instant Notifications: Supports ntfy.sh, WeChat Work (企业微信), Bark, Telegram, Webhook
- Scheduled Tasks: Cron expression configuration for periodic tasks
- Account & Proxy Rotation: Multi-account management, task-account binding, proxy pool rotation with failure retry
- Docker Deployment: One-click containerized deployment
Screenshots
Quick Start
Requirements
- Python 3.10+
- Node.js + npm (
Node v20.18.3has been verified to complete the frontend build) - Playwright CLI and Chromium. Before the first local run, install them with
python3 -m pip install playwright && python3 -m playwright install chromium - Chrome or Edge on desktop systems. On Linux, Chromium also works.
start.shchecks this prerequisite before continuing
git clone https://github.com/Usagi-org/ai-goofish-monitor
cd ai-goofish-monitor
cp .env.example .env
Minimum Configuration
| Variable | Description | Required |
|---|---|---|
OPENAI_API_KEY | AI model API key | Yes |
OPENAI_BASE_URL | OpenAI-compatible API base URL | Yes |
OPENAI_MODEL_NAME | Model name with image input support | Yes |
WEB_USERNAME / WEB_PASSWORD | Web UI login credentials, default admin/admin123 | No |
See "Configuration" below for the rest.
Start Locally
chmod +x start.sh
./start.sh
start.sh first validates the Playwright CLI and browser prerequisites. Once they are available, it installs project dependencies, builds the frontend, copies the artifacts, and starts the backend.
First-Time Setup
- Open the default Web UI at
http://127.0.0.1:8000and sign in. - Go to "Xianyu Account Management" and use the Chrome Extension to export and paste the Xianyu login-state JSON.
- Login-state files are stored in
state/, for examplestate/acc_1.json. - Go back to "Task Management", create a task, bind an account if needed, and run it.
Create Your First Task
AI mode: fill in the requirement description. Submission opens a separate progress dialog while the criteria are generated asynchronously.Keyword mode: provide keyword rules and the task is created immediately.Region filter: now uses a province / city / district selector backed by an embedded Xianyu page snapshot instead of manual text input.
🐳 Docker Deployment (Recommended)
git clone https://github.com/Usagi-org/ai-goofish-monitor && cd ai-goofish-monitor
cp .env.example .env
vim .env # fill in the required values
docker compose up -d
docker compose logs -f app
docker compose down
- Default Web UI:
http://127.0.0.1:8000 - The published Docker image already includes Chromium, so no extra browser install is required on the host.
- Update image:
docker compose pull && docker compose up -d - If you change
SERVER_PORTin.env, update theportsmapping indocker-compose.yamlas well. docker-compose.yamlnow mounts the primary SQLite database directory as./data:/app/data, with the default database file atdata/app.sqlite3- These paths are persisted by default:
data/for the SQLite primary store (tasks, results, price history)state/for login-state cookie filesprompts/for task prompt fileslogs/for runtime logsimages/for downloaded product images and per-task temporary image foldersconfig.json,jsonl/, andprice_history/as legacy sources for the first SQLite migration
Storage and Migration
- SQLite is now the online primary storage, with the default path
data/app.sqlite3 - You can override the database path with
APP_DATABASE_FILE; Docker sets it to/app/data/app.sqlite3 - On startup, the app initializes the schema and tries to import existing data once from legacy
config.json,jsonl/, andprice_history/ state/,prompts/,logs/, andimages/remain filesystem-based and are not stored in SQLite- Product images are temporarily downloaded to
images/task_images_<task_name>/and are normally cleaned up when the task finishes - After the first upgrade and after verifying the database contents in
data/app.sqlite3, you can decide whether to keep the legacyconfig.json,jsonl/, andprice_history/mounts
User Guide
Click to expand Web UI usage notes
Task Management
- Supports AI creation, keyword rules, price range, new listing filters, region filters, account binding, and cron scheduling.
- AI task creation runs as a background job and shows a dedicated progress dialog after submission.
- Region filtering can greatly reduce results, so leaving it empty is the safer default.
Account Management
- Import, update, and delete Xianyu login states.
- Each task can bind a specific account or leave account selection to the system.
Results and Logs
- The results page and export endpoints now query SQLite instead of directly scanning
jsonlfiles. - The logs page is the first place to inspect login-state expiry, anti-bot issues, or AI call failures.
System Settings
- View system status, edit prompts, and adjust proxy / rotation-related settings.
Developer Guide
Local Development
# backend
python -m src.app
# or
uvicorn src.app:app --host 0.0.0.0 --port 8000 --reload
# frontend
cd web-ui
npm install
npm run dev
- FastAPI initializes SQLite on startup and performs the one-time legacy import from
config.json/jsonl/price_historywhen needed spider_v2.pynow loads tasks from SQLite by default; JSON config is only used when--config <path>is passed explicitly- The default local database path is
data/app.sqlite3 - The Vite dev server proxies
/api,/auth, and/wstohttp://127.0.0.1:8000. npm run buildwritesweb-ui/dist/, andstart.shcopies it to the repository rootdist/.- FastAPI serves
dist/index.htmlanddist/assets/from the repository root. ./start.shprints the default app URLhttp://localhost:8000and API docs URLhttp://localhost:8000/docs.
Validation
PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 pytest
cd web-ui && npm run build
Task Creation API
Click to expand API behavior
POST /api/tasks/generatedecision_mode=ai: returns202with ajob; the client should poll for progress.decision_mode=keyword: returns the created task directly.
GET /api/tasks/generate-jobs/{job_id}: fetch AI task-generation progress.POST /auth/status: validate Web UI credentials.
Configuration
Click to expand common configuration items
AI and Runtime
OPENAI_API_KEY/OPENAI_BASE_URL/OPENAI_MODEL_NAME: required AI model settings.PROXY_URL: dedicated HTTP/SOCKS5 proxy for AI requests.RUN_HEADLESS: whether the scraper runs headless; keep ittruein Docker.SERVER_PORT: backend port, default8000.LOGIN_IS_EDGE: use Edge instead of Chrome locally; Docker images do not bundle Edge and always run with Chromium.PCURL_TO_MOBILE: convert desktop item URLs to mobile URLs.
Notifications
NTFY_TOPIC_URLGOTIFY_URL/GOTIFY_TOKENBARK_URLWX_BOT_URLTELEGRAM_BOT_TOKEN/TELEGRAM_CHAT_ID/TELEGRAM_API_BASE_URLWEBHOOK_*
Proxy Rotation and Failure Guard
PROXY_ROTATION_ENABLEDPROXY_ROTATION_MODEPROXY_POOLPROXY_ROTATION_RETRY_LIMITPROXY_BLACKLIST_TTLTASK_FAILURE_THRESHOLDTASK_FAILURE_PAUSE_SECONDSTASK_FAILURE_GUARD_PATH
See .env.example for the full list.
Web Authentication
Click to expand authentication notes
- The Web UI uses a login page and validates credentials through
POST /auth/status. - After login, the frontend stores local auth state for route guards and WebSocket startup.
- The default credentials are
admin/admin123; change them in production.
🚀 Workflow
The diagram below shows the core processing flow of a monitoring task. The main service runs in src.app and launches one or more task processes based on user actions or schedule triggers.
graph TD
A[Start Monitoring Task] --> B[Select Account/Proxy Configuration];
B --> C[Task: Search Products];
C --> D{Found New Products?};
D -- Yes --> E[Scrape Product Details & Seller Info];
E --> F[Download Product Images];
F --> G[Call AI for Analysis];
G --> H{AI Recommended?};
H -- Yes --> I[Send Notification];
H -- No --> J[Save Record to SQLite];
I --> J;
D -- No --> K[Next Page/Wait];
K --> C;
J --> C;
C --> L{Risk Control/Exception?};
L -- Yes --> M[Account/Proxy Rotation and Retry];
M --> C;
FAQ
Click to expand FAQ
Why does AI task creation take time?
In AI mode, the system generates analysis criteria before the task itself is created. This now runs as a background job with a separate progress dialog instead of blocking the task form.
Why is the region filter optional by default?
Region filtering can sharply reduce result volume. Leave it empty if you want a broader market scan first.
Why does the app say the frontend build artifacts are missing?
It means the repository root dist/ directory is missing. Run ./start.sh, or build the frontend in web-ui/ and make sure the artifacts are copied to the root dist/.
Why does ./start.sh complain about missing Playwright or a browser?
The script performs a prerequisite check before installing project dependencies. Install the Playwright CLI and Chromium first, then make sure Chrome, Edge, or Chromium is available on the system and rerun ./start.sh.
Acknowledgments
Click to expand acknowledgments
This project referenced the following excellent projects during development. Special thanks to:
Also thanks to LinuxDo contributors for script contributions:
And thanks to the LinuxDo community.
Also thanks to ClaudeCode/Gemini/Codex and other model tools for freeing our hands and experiencing the joy of Vibe Coding.
Notices
Click to expand notice details
- Please comply with Xianyu's user agreement and robots.txt rules. Do not make frequent requests to avoid burdening the server or having your account restricted.
- This project is for learning and technical research purposes only. Do not use it for illegal purposes.
- This project is released under the MIT License, provided "as is", without any form of warranty.
- The project authors and contributors are not responsible for any direct, indirect, incidental, or special damages or losses caused by the use of this software.
- For more details, please refer to the Disclaimer file.