WAKIBBot, the WAKIB.ai crawler

User-Agent string

WAKIBBot identifies itself on every request with the exact string below — no browser spoofing.

WAKIBBot/1.0 (+https://wakib.ai; tech@wakib.ai)

What it does

• Fetches publicly available article pages and RSS/Atom feeds from sources WAKIB.ai has configured to follow.
• Uses the fetched text to produce short, attributed summaries that link back to the original article on the source's own site.
• Applies a seven-day freshness window: articles older than that are not fetched for summarization.
• Polls each source on its own cadence rather than continuously — typically every few hours, not a constant stream of requests.
• Reads and honours robots.txt before fetching anything else on a site.

What it does not do

• It does not train machine-learning models on publisher text. WAKIBBot is a reference-lane crawler: content is read to summarize and cite, never to train on.
• It does not republish articles in full — only short, attributed summaries with a link back to the source.
• It does not attempt to bypass paywalls, login walls, or bot challenges (e.g. CAPTCHAs). A blocked or gated request is treated as a signal to stop, not a puzzle to solve.

How to control it

WAKIBBot honours robots.txt rules addressed specifically to User-agent: WAKIBBot, and it also honours rules addressed to the wildcard User-agent: * group when no WAKIBBot-specific group exists. Changes to your robots.txtare picked up within 6 hours — that is how long WAKIBBot caches a site's robots policy before re-checking it.

Block WAKIBBot entirely

User-agent: WAKIBBot
Disallow: /

Block a single path

User-agent: WAKIBBot
Disallow: /subscriber-only/