What is specific to Meta AI
Distribution is the story. The assistant sits inside messaging, so questions are short and practical.
- meta-webindexer crawls to improve Meta AI search and helps it cite your content
- meta-externalagent crawls for training or indexing and can be blocked in robots.txt
- meta-externalfetcher fetches user-requested links and may bypass robots.txt
- facebookexternalhit builds link previews and is separate
- Robots.txt changes can take up to 24 hours