Initial commit
This commit is contained in:
commit
57fc3525eb
7311 files changed
+1291398
No files matched your search
+236
@@ -0,0 +1,236 @@
|
||||
<p align="center">
|
||||
<img src="./static/images/logo.svg" alt="Universal Web API logo" width="128">
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<img src="./static/images/readme-wordmark.svg" alt="Universal Web API" width="720">
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<samp>Universal · Simple · Local</samp>
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="./README.en.md">English</a>
|
||||
/
|
||||
<a href="./README.zh-CN.md">简体中文</a>
|
||||
</p>
|
||||
|
||||
<br>
|
||||
|
||||
> Current release: **3.0.1** (see [`VERSION`](./VERSION); release notes in [`CHANGELOG-3.0.1.md`](./CHANGELOG-3.0.1.md))
|
||||
|
||||
**Universal Web API** is a **local API bridge & debugging tool** designed for developers. It converts AI web services (e.g., ChatGPT, DeepSeek, Claude, Gemini) logged in your local browser into local standard OpenAI/Anthropic-compatible APIs.
|
||||
|
||||
This project is dedicated to helping developers perform **workflow orchestration, client integration testing, and personal office automation locally**, ensuring data privacy and security without exposing API keys to third parties.
|
||||
|
||||
> ⚠️ **Compliance & Security Statement**: This tool runs entirely on the user's local system as a bridge helper. It **does not** provide any functionality to bypass authentication (login), crack security defenses (such as captcha solvers), or reverse-engineer encrypted APIs. Users must log into their own valid accounts in the controlled browser. Do not use this tool for high-frequency automated requests or commercial purposes.
|
||||
|
||||
---
|
||||
|
||||
## 📐 Project Architecture (Mermaid)
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
User([Client/User]) -->|OpenAI/Anthropic/Codex API| Route[1. API & Routing Layer app/api]
|
||||
Route -->|Session Dispatch/Concurrency| TabPool[2. Tab Pool & Lifecycle app/core/tab_pool]
|
||||
Route -->|Parse Tool Request| ToolCall[5. Tool Calling Adapter app/services/tool_calling]
|
||||
TabPool -->|Browser Control/Anti-detection| Browser[3. Automation & Workflow Engine app/core/workflow]
|
||||
Browser -->|Monitor Stream| Stream[4. Stream Monitor & Parsers app/core/parsers]
|
||||
Browser -->|Trigger Command Hook| CmdEng[6. Command Engine app/services/command_engine]
|
||||
|
||||
subgraph Core Utilities
|
||||
Config[7. Config & Presets app/services/config]
|
||||
Utils[8. Shared Utilities app/utils & app/models]
|
||||
end
|
||||
|
||||
Browser -.->|Depends| Config
|
||||
CmdEng -.->|Read/Write| Config
|
||||
Stream -.->|Helpers| Utils
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🌟 Highlights
|
||||
|
||||
* **⚡ Zero-config Standard Compatibility**: Fully compatible with OpenAI standard API (including `/v1/chat/completions` and `/v1/models`) with experimental compatibility support for third-party developer tools (such as `/v1/messages` connectivity testing for Claude Code and `/v1/responses` endpoint for Codex plugins).
|
||||
* **🛠️ Local Controlled Browser Drive**: Lightweight automation of Chromium-based browsers (Chrome / Edge, etc.) using DrissionPage. All data stays local for end-to-end privacy.
|
||||
* **🛡️ Human-like Interaction**: Built-in keyboard keypress simulations, focus emulation, and mouse path movements to minimize account detection.
|
||||
* **📦 Intelligent Tab Pooling**: Multi-tab concurrency with default, domain, fixed-tab, exact-URL, and URL-bound preset routes, plus first-idle, round-robin, and random allocation modes.
|
||||
* **📡 Dual-channel Stream Parsing**: Combined CDP network interception and DOM mutation monitoring to stream increments in real-time, regardless of the site's rendering technique.
|
||||
* **📎 Multimodal & Attachment Self-healing**:
|
||||
* Extract and download text, images, audio, and video content locally from web sessions.
|
||||
* Oversized prompts are automatically staged as local temporary files for upload (for sites that handle file-style inputs better).
|
||||
* **🧩 Robust Tool Calling**: Injects schema verification feedback loops into web sessions. If a web model produces invalid arguments, it automatically triggers local correction prompts, boosting tool-calling reliability.
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
### Prerequisites
|
||||
1. OS: Windows (Fully supported) / macOS or Linux (Core features supported)
|
||||
2. Requirements: **Python 3.10+** and a Chromium-based browser (Chrome, Edge, or Brave) installed.
|
||||
|
||||
### Setup Steps
|
||||
|
||||
1. **Download & Extract**: Download the latest release from [Releases](https://github.com/lumingya/universal-web-api/releases) and extract it to a path **without non-ASCII (e.g. Chinese) characters**.
|
||||
2. **Start the Service**:
|
||||
* **Windows**: Double-click **`start.bat`**.
|
||||
* **macOS / Linux**: Run **`python3 start.py`** in your terminal.
|
||||
3. **Initialization**: Once dependencies are validated and installed, a controlled browser window will pop up automatically, and the console will open in a normal browser at `http://127.0.0.1:8199`. Keep AI websites in the controlled browser, and use your normal browser for the dashboard and tutorial.
|
||||
4. **Log In**: In the controlled browser, log in to your own AI web accounts (e.g., chatgpt.com, claude.ai), then keep the target site on a real chat-ready page.
|
||||
5. **Configure Clients**: In any client, set the API configurations:
|
||||
* **Base URL**: `http://127.0.0.1:8199/v1`
|
||||
* **API Key**: If service API auth is disabled (`AUTH_ENABLED=false`), use any value (e.g., `sk-local`). If enabled, use `AUTH_TOKEN`. The dashboard uses the separate `DASHBOARD_AUTH_TOKEN`; do not share it with API users.
|
||||
|
||||
### Verify the first request
|
||||
|
||||
Run these checks from a terminal after at least one controlled-browser tab is logged in:
|
||||
|
||||
```bash
|
||||
# Service and model discovery
|
||||
curl http://127.0.0.1:8199/v1/models
|
||||
|
||||
# Non-streaming smoke test
|
||||
curl http://127.0.0.1:8199/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model":"web-api","messages":[{"role":"user","content":"Reply with OK"}],"stream":false}'
|
||||
```
|
||||
|
||||
For Windows PowerShell, use `Invoke-RestMethod` or keep the JSON in a file to avoid shell quoting issues. A real chat request is a better connectivity test than a client's short “test API” request, because some sites reject very short prompts.
|
||||
|
||||
### Supported API surfaces
|
||||
|
||||
| Endpoint | Compatibility | Notes |
|
||||
| :--- | :--- | :--- |
|
||||
| `POST /v1/chat/completions` | OpenAI Chat Completions | Streaming SSE, multimodal message parts, tool-calling repair |
|
||||
| `POST /v1/responses` | OpenAI Responses (experimental) | `input`, `instructions`, tools, and `previous_response_id` are translated to the browser workflow |
|
||||
| `POST /v1/messages` | Anthropic Messages (experimental) | Accepts `Authorization` or `x-api-key`; includes `count_tokens` compatibility |
|
||||
| `GET /v1/models` | OpenAI model listing | Returns generic models plus models discovered from active tabs |
|
||||
|
||||
All endpoints are local by default. Route a request to a specific site or tab with `/url/{domain}/v1`, `/tab/{index}/v1`, or `/tab-url/{token}/v1`; see the [routing guide](./static/tutorial/index.html#routing-methods) for precedence and examples.
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Supported Sites
|
||||
|
||||
Built-in automation rules are available for several mainstream AI websites. For unlisted sites, you can use the built-in AI assistant to analyze page DOM structures and generate adaptations. See [Add a New Site Guide](./static/tutorial/index.html#addsite).
|
||||
|
||||
| Site Name | URL | Notes |
|
||||
| :--- | :--- | :--- |
|
||||
| **ChatGPT** | chatgpt.com | Supports extremely long prompts via file uploads |
|
||||
| **DeepSeek** | chat.deepseek.com | Adapted for reasoning/thinking stream output extraction |
|
||||
| **Gemini** | gemini.google.com | Excellent for testing local multimodal workflows |
|
||||
| **Claude** | claude.ai | Comprehensive page interaction and attachment handling |
|
||||
| **Kimi** | www.kimi.com | Excellent long-context file-paste support |
|
||||
| **Qwen** | chat.qwen.ai | DOM rules adapted for domestic LLM web automation |
|
||||
| **Grok** | grok.com | Decodes native websocket/HTTP stream response data |
|
||||
| **Doubao** | www.doubao.com | Fully adapted for the latest page structures |
|
||||
| **AI Studio** | aistudio.google.com | High-throughput developers testing environment |
|
||||
| **Arena AI** | arena.ai | Comparative debugging (sensitive to IP quality) |
|
||||
|
||||
---
|
||||
|
||||
## 📖 Documentation
|
||||
|
||||
Detailed HTML guides are hosted locally and can be accessed via the dashboard after launch:
|
||||
|
||||
| Section | Description |
|
||||
| :--- | :--- |
|
||||
| 📖 [Full Tutorial](./static/tutorial/index.html#quickstart) | Installation, platform differences, and UI dashboard guides |
|
||||
| 🔗 [Connect API](./static/tutorial/index.html#connect) | Request parameters, routing modes (Default, Domain, Fixed Tab, Exact URL, URL-bound Preset), and code examples |
|
||||
| 🧩 [Function Calling](./static/tutorial/index.html#toolcalling) | Validation repair strategy and multi-turn prompt engineering explanations |
|
||||
| 🔄 [Tab Pool and Presets](./static/tutorial/index.html#tabpool) | Configuring concurrency, route methods, allocation modes, and custom task presets |
|
||||
| 📊 [Request Monitor](./static/tutorial/index.html#dashboard) | Inspect request history, failure details, per-site success rates, and debug stuck tasks |
|
||||
| 🧾 [API Endpoint Reference](./static/tutorial/index.html#api-reference) | Health, model, provider, OpenAI, Responses, and Anthropic endpoint quick reference |
|
||||
| 🛠️ [Core Selector Configuration](./static/tutorial/index.html#selectors) | CSS selector mapping, visual workflows, and streaming options |
|
||||
| 🛡️ [Stealth & Advanced Options](./static/tutorial/index.html#stealth) | Anti-detection settings, browser fingerprint overrides, and low-interference modes |
|
||||
| ❓ [Limitations & FAQ](./static/tutorial/index.html#faq) | Timeout troubleshooting, captcha handling, and OS differences |
|
||||
| 🔐 [Security Boundary](./static/tutorial/index.html#security-boundary) | Local binding, authentication, CORS, DevTools, and sensitive-data handling |
|
||||
|
||||
The tutorial is bundled as static HTML, so it also works without an internet connection. Open [`static/tutorial/index.html`](./static/tutorial/index.html) directly, or use the dashboard's Tutorial entry after startup.
|
||||
|
||||
## ⚙️ Configuration Essentials
|
||||
|
||||
Copy [`.env.example`](./.env.example) to `.env` only when you need to override defaults. Values are read at process start; restart `start.bat` / `start.py` after changing environment settings.
|
||||
|
||||
| Variable | Recommended local value | Purpose |
|
||||
| :--- | :--- | :--- |
|
||||
| `APP_HOST` | `127.0.0.1` | Bind to localhost. Use `0.0.0.0` only behind a trusted firewall/reverse proxy. A non-loopback bind without both API and dashboard auth refuses to start (override: `ALLOW_INSECURE_PUBLIC_BIND=true`). |
|
||||
| `APP_PORT` | `8199` | Dashboard and API port. |
|
||||
| `APP_DEBUG` | `false` | Enable `/docs`, `/redoc`, and detailed errors only for local debugging. |
|
||||
| `AUTH_ENABLED` | `false` | Require a Bearer token or `X-API-Key` for API calls. |
|
||||
| `AUTH_TOKEN` | a long random token | API credential when authentication is enabled. |
|
||||
| `DASHBOARD_AUTH_ENABLED` | `false` | Protect the dashboard independently from the API. |
|
||||
| `DASHBOARD_AUTH_TOKEN` | a separate token | Dashboard credential; never reuse it for client integrations. |
|
||||
| `CORS_ENABLED` | `false` | Keep disabled for same-origin local clients; enable only when a browser client needs cross-origin access. |
|
||||
| `CORS_ORIGINS` | empty | Empty = no cross-origin web page may call the service (the dashboard is same-origin). Browser requests from origins not listed here are rejected with 403. List explicit trusted origins instead of `*`. |
|
||||
| `BROWSER_PORT` | `9222` | Chromium remote-debugging port used by the controlled browser. |
|
||||
| `PROXY_ENABLED` | `false` | Send controlled-browser traffic through `PROXY_ADDRESS`. |
|
||||
| `PROXY_ADDRESS` | `http://127.0.0.1:7890` | HTTP or SOCKS5 proxy URL. |
|
||||
| `HELPER_API_KEY` | empty | Optional key for AI-assisted site analysis. |
|
||||
|
||||
Never commit `.env`, `chrome_profile/`, logs, or exported site configurations containing cookies or tokens. If the service is reachable from another machine, enable both API and dashboard authentication and restrict access at the network layer.
|
||||
|
||||
## 🧭 Routing and Presets at a Glance
|
||||
|
||||
The request body `model` is primarily a display/routing hint; the actual model is the one available in the selected web session. Presets control selectors, workflow, stream parsing, file handling, and media extraction for a site.
|
||||
|
||||
```text
|
||||
Default pool /v1/chat/completions
|
||||
Domain route /url/gemini.google.com/v1/chat/completions
|
||||
Fixed tab /tab/1/v1/chat/completions
|
||||
Exact URL token /tab-url/{token}/v1/chat/completions
|
||||
Domain + preset /url/gemini.google.com/pro/v1/chat/completions
|
||||
```
|
||||
|
||||
When several tabs match, allocation follows the configured `first_idle`, `round_robin`, or `random` mode. A request-level `preset_name` overrides the tab's default for that request only.
|
||||
|
||||
## 🧰 Troubleshooting Checklist
|
||||
|
||||
1. **The dashboard does not open:** confirm Python 3.10+, the Chromium browser, and that port `8199` (and debugging port `9222`) are free. Read the terminal output and `logs/` before restarting.
|
||||
2. **The tab is missing from the pool:** wait for the page to finish loading, close `chrome://newtab` and unrelated tabs, then use “Refresh tabs” in the dashboard.
|
||||
3. **Input or send fails:** the site likely changed its DOM. Test `input_box`, `send_btn`, and `result_container` separately in the selector workbench, then save the preset.
|
||||
4. **The response is empty or times out:** keep the controlled browser visible, avoid interacting with it during a request, check for captcha/rate-limit pages, and try DOM stream mode if network interception has no matching traffic.
|
||||
5. **Requests queue forever:** inspect Request Monitor for `acquire_timeout` or a stuck tab; cancel the request or use the force-release debug action only as a last resort.
|
||||
6. **Authentication returns 401/403:** send `Authorization: Bearer <AUTH_TOKEN>` (or `X-API-Key`) and verify `AUTH_ENABLED` is a boolean string (`true`/`false`), not a placeholder such as `your-secret-here`.
|
||||
|
||||
Do not attempt to bypass a site's login or human verification. Complete challenges manually in the controlled browser and lower request frequency when a site applies rate limits.
|
||||
|
||||
## 🧪 Development and Tests
|
||||
|
||||
```bash
|
||||
python -m venv venv
|
||||
# Windows: .\\venv\\Scripts\\Activate.ps1
|
||||
# macOS/Linux: source venv/bin/activate
|
||||
pip install -r requirements.txt
|
||||
pytest -q
|
||||
```
|
||||
|
||||
Most tests are browser-independent regression tests under [`tests/`](./tests). A live browser session is required for end-to-end site adaptation tests. Keep generated profiles and logs outside commits.
|
||||
|
||||
## 🔄 Upgrading Safely
|
||||
|
||||
Before upgrading, back up `config/`, `chrome_profile/`, `.env`, and any custom scripts. The updater preserves user data listed in `UPDATE_PRESERVE` and supports a configurable update whitelist in the dashboard. After upgrading, review the latest `CHANGELOG-<version>.md` (currently [`CHANGELOG-3.0.1.md`](./CHANGELOG-3.0.1.md)), rerun the DrissionPage patch when prompted, and send one real smoke-test request before resuming normal traffic.
|
||||
|
||||
---
|
||||
|
||||
## 🤝 Feedback & Discussion
|
||||
|
||||
* If you run into issues, join the QQ Group **1073037753**.
|
||||
* You can also open issues or suggest features in the GitHub [Issues](https://github.com/lumingya/universal-web-api/issues) tracker.
|
||||
|
||||
---
|
||||
|
||||
## ⚖️ Disclaimer
|
||||
|
||||
1. **Purpose**: This project is intended solely for personal technical research, educational demonstration, and testing. Do not deploy it in production environments or use it for commercial profit-making activities.
|
||||
2. **Compliance**: Before using this software, read and comply with the target sites' Terms of Service. Users are solely responsible for account limitations, suspensions, or disputes arising from using this automation tool.
|
||||
3. **No Hacking**: This software does not engage in network intrusion, cracking, reverse engineering of APIs, or bypassing payment walls. All interactions are achieved by automating actions in a legitimate browser owned and logged in by the user.
|
||||
4. **Liability**: The maintainers assume no liability for any direct or indirect damage or loss (including account bans, business losses, or data loss) resulting from using this software.
|
||||
|
||||
---
|
||||
|
||||
## 📄 License
|
||||
|
||||
This project is licensed under the [AGPL-3.0](./LICENSE).
|
||||
Reference in new issue
Block a user