gpt-researcher学习笔记(1)
https://github.com/assafelovic/gpt-researcher来自https://github.com/datawhalechina/Agent-Learning-Hub列出的开源项目参考项目从2023-05开始创建根据提交记录请claude code分析了一下其演变路径如下# GPT-Researcher: Evolution Key Change Points Analysed from 2,980 commits spanning May 2023 to May 2026.---## Phase 1 — Proof of Concept (May–July 2023)The project began with a single commit on **May 12, 2023**. The initial architecture was minimal:- A Python script driving a Selenium/Chrome browser to scrape web pages- DuckDuckGo as the only search backend- OpenAI (GPT-4) for synthesis- Three report types: outline_report, resource_report, detailed_report- A FastAPI server plain HTML/JS frontend added within the first two weeks**Key milestone:** WebSocket real-time streaming of agent progress (July 7), which gave the project its live thinking out loud feel that became its identity.---## Phase 2 — Community Bootstrap (July–September 2023)Growth exploded with community PRs almost immediately after launch. Key additions:- **Auto-agent selection** (Jul 20): LLM dynamically picks the right agent role instead of hardcoding it — Finance, Security Analyst, Business Analyst, etc.- **Docker support** (Jul 14–31): Containerized Chrome app for reproducible deployment- **LangChain integration** (Aug 18): Moved LLM/config to the LangChain abstraction layer- JS-rendered page scraping support via Selenium (Aug 24)The project gained ~150 GitHub stars in this phase.---## Phase 3 — Multi-Agent Architecture (Oct 2023–Sep 2024)This is where the projects research depth fundamentally changed.**~Oct 2023:** Introduction of a multi_agents module using **LangGraph** — a hierarchical system where a Master agent orchestrates Writer, Reviewer, Reviser, Editor, and Publisher sub-agents. This enabled long-form, structured research reports.**~Sep 2024 (PR #875):** The detailed_report type was rebuilt around multi-agent subtopic decomposition — each subtopic gets its own parallel research pass before being merged into a coherent final report.**~Oct 2024 (PR #941):** Introduction of a **Strategic LLM** concept — a separate, more powerful planning model distinct from the fast execution model — allowing cost/quality tradeoffs.Other expansions in this phase:- Azure OpenAI, AWS Bedrock, Ollama support- Multiple search retrievers: Google, Bing, SearxNG, Arxiv, PubMed- Vector store integration for hybrid localweb research (langchain_vectorstore report source)- Configurable embedding providers---## Phase 4 — Frontend Renaissance (Oct–Dec 2024)The plain HTML frontend was replaced with a full **Next.js application** (PR #898, Oct 2024):- TypeScript throughout- Image carousel in reports (Oct 2024, PR #925/#942)- Chat-with-research capability (post-report QA)- Mobile-responsive design- Source deduplication and favicon display- Structured logging system with downloadable logs (Dec 2024)**Language support** was added (Dec 2024, PR #1026) — reports can now be generated in any language via config.---## Phase 5 — Deep Research Agent Ecosystem (Feb–Jun 2025)**February 22, 2025** is the single biggest inflection point in the projects history.**Deep Research mode** (PR #1179/#1195) was introduced as a wholly new research paradigm: iterative, recursive search loops guided by a planning LLM that decides when to dig deeper vs. synthesize. This directly competed with OpenAIs Deep Research product.Simultaneously:- **Nodriver/Zendriver headless scraper** (Feb 2025) — a Chrome-automation library without Selenium, enabling stealthier and faster scraping- **FireCrawl scraper** (Feb 2025) — cloud-based scraping for JS-heavy sites- **MCP Server** (Mar 29, 2025) — GPT-Researcher exposed as a Model Context Protocol server, integrable into Claude, Cursor, etc.- **React NPM package** (Feb–Mar 2025) — embeddable GPTResearcher / component- **Domain filtering** (Feb 2025) — restrict research to specific domains- **Reasoning model support** (Feb–Mar 2025) — o1, o3, Gemini thinking mode, with reasoning_effort config---## Phase 6 — Provider Proliferation Observability (Jun–Nov 2025)The project became a true multi-provider platform:| Provider added | Date ||---|---|| GigaChat | Feb 2025 || OpenRouter | Mar 2025 || AI/ML API | May 2025 || vLLM (local) | May 2025 || DashScope (Alibaba) | Jun/Jul 2025 || Netmind | Jul 2025 || Avian | Feb 2026 || Forge | Feb 2026 || MiniMax | Mar 2026 |**LangChain dependency was removed** (Nov 7, 2025, PR #1547 migration/langchain) — a major architectural cleanup that reduced startup time and dependency bloat.**Hallucination evaluation framework** added (Jun 2025) with a judge-based LLM grader.**PWA support** (Sep 2025) — installable as a mobile app.**New UI redesign** (Apr 2025, Sep 2025) — two separate premium UI overhauls, the second introducing a sidebar, preferences modal, and polished report rendering.**LangSmith observability** (Jan 2026) — full tracing of LLM calls.---## Phase 7 — Modern Ecosystem (2026)The project shifted focus toward ecosystem integrations and reliability:- **AG2 partnership** (Mar 2026) — AutoGen2/AG2 as a multi-agent pipeline alternative- **Agent discovery endpoint** (Apr 2026) — REST API for external tools to enumerate available agents- **OpenAlex retriever** (Apr 2026) — academic literature search- **Xquik X/Twitter retriever** (Apr 2026)- **Smart context compression** (Feb 2026) — fast-path skips compression for small document sets- **URL deduplication** (Feb 2026) — prevents re-scraping identical sources- **Anthropic cost tracking** (May 2026) — real usage metadata from API responses- **Max tokens raised to 200k** (May 2026) — supporting modern long-output models- **Tokentoll CI integration** (May 2026) — GitHub Action that analyzes LLM cost impact of PRs---## Summary: Architectural Arc2023-05 Simple script: 1 search engine 1 LLM → report2023-07 Real-time WebSocket streaming2023-08 LangChain abstraction layer2023-10 Multi-agent LangGraph pipeline2024-09 Strategic LLM / planner separation2024-10 Next.js frontend images in reports2024-12 Language support, structured logging2025-02 Deep Research (iterative recursive loops)2025-03 MCP server / embeddable React component2025-06 MCP UI, FireCrawl, nodriver scrapers2025-07 Hallucination evals, domain filtering2025-11 LangChain fully removed2026-02 vLLM, cost tracking, smart compression2026-04 Agent discovery, academic retrievers2026-05 200k token cap, MiniMax, Anthropic cost trackingThe project evolved from a single-researcher curiosity in May 2023 into a production-grade, multi-provider autonomous research platform with 1,780 PRs merged across nearly 200 contributors in just over 3 years.