🎯 Major Rate Limiting Improvement - Research-Friendly Design
New Rate Limiting System:
✅ 12 calls per 5-minute window (vs 8 per session)
✅ 8 calls per 30-second burst protection
✅ Automatic time-based reset (no LM Studio restarts needed)
✅ Clear feedback with remaining calls and reset times
Key Benefits:
- Supports extended research sessions without interruption
- Prevents LLM spam while allowing legitimate research workflows
- Users can work continuously without restarting LM Studio
- Intelligent burst protection prevents overwhelming websites
Technical Implementation:
- Sliding window algorithm with timestamp tracking
- Dual-layer protection: burst + window limits
- Automatic cleanup of expired call history
- User-friendly error messages with precise wait times
This addresses the core user feedback that session-based limits were too restrictive for normal research use cases while maintaining responsible web scraping practices.
Breaking Change: Rate limiting behavior changed from session-based to time-based
Migration: No action needed - new system is more permissive
🚀 Production-ready MCP server for web search and content extraction
Core Features:
- Web search via local SearxNG instance with 70+ configurable engines
- Advanced web content extraction using Mozilla Readability
- Browser simulation with rotating user agents and realistic headers
- Rate limiting and responsible scraping practices (8 calls per session)
- Comprehensive error handling and detailed logging
- LM Studio integration with full MCP protocol compliance
Technical Highlights:
- JavaScript execution support via JSDOM for dynamic content
- Multi-layer content extraction with fallback strategies
- Retry logic for transient HTTP errors
- Binary content detection and proper handling
- Real-time log monitoring utility included
Author: Jay Leon (@manull)
License: MIT
Repository: https://github.com/manooll/webfetch-mcp
Jay