Mandelbot is Fractle's web crawler or spider; its continuing mission: to explore strange new websites, to seek out new and updated pages, to boldly go where no bot has gone before.
Mandelbot uses an algorithm to determine when, what, how often, and how many pages to crawl from each website. It then updates Fractle's indexes to reflect the new pages, updated pages, and dead pages it discovered.
So the pages on your website can be included in Fractle's index, thereby enabling the use of a Fractle search engine on your website.
If you're not yet a customer of Fractle, this is great news; it means we're hard at work preparing a personalized demo of Fractle just for you. We'll be in touch shortly to show you a demo of Fractle on your website. Better yet, simply let us know and we'll be in touch even sooner.
Yes, Mandelbot's policy is to be polite.
Currently Mandelbot should only access your website at most once every 10–15 seconds on average, unless you have allowed Mandelbot to access your website more frequently. However, the rate may be higher over some short periods due to network delays, crawler restarts, and testing.
Please contact us if you want Mandelbot to access your website more frequently or less frequently.
Mandelbot uses the following user agent:
Mozilla/5.0 (compatible; Mandelbot/1.0; +http://www.fractle.com/bot).
Spammers and other nefarious individuals can make themselves look like Mandelbot by changing their user agent to match. Unfortunately, we cannot stop them from doing this.
The IP addresses used by Mandelbot do vary, so if you spot unusual behaviour, please contact us with the relevant log entries so we can investigate.
Mandelbot implements the Robots Exclusion Protocol. This lets you block Mandelbot from crawling certain files by creating a robots.txt file and lets you block Mandelbot from indexing certain files using Robot Tags. For more details read how Mandelbot supports the Robots Exclusion Protocol.
If you want to stop Mandelbot from crawling your website and completely remove your website from Fractle's index, please contact us and we will honour your request as soon as possible. Please note, while crawling will stop promptly, it will take some time to completely remove your website from Fractle's index and we may require verification that you are authorized to act on behalf of the website.
You can fill out our contact form or send an email to mandelbot@fractle.com.
Fractle © 2024
