Skip to content
GetPicked

GetPickedBot

GetPickedBot is the crawler behind GetPicked. If you saw it in your logs, this page says what it fetched, what it keeps, and how to keep it out.

Last updated

What it does

GetPicked helps software companies find the “best tools” articles and directories where their competitors are listed and they aren't. GetPickedBot reads those public pages to record which products they list and link to.

It fetches two kinds of pages:

  • Articles and directory pages that came up in a search for a product category, or that link to pages we already know.
  • A single page someone asked about, for example when a person pastes a URL into our Link Checker to see whether it still links to them.

How it behaves

  • It obeys robots.txt when crawling.
  • It sends one request at a time to each site, waits at least 2 seconds between them, and slows down further when your server responds slowly.
  • It reads HTML only, stops after 5 MB, and doesn't run JavaScript.
  • It doesn't send cookies, log in or fill in forms.
  • Its user agent always starts with GetPickedBot/ and ends with a link to this page.

What it keeps

For each page: the links on it, their anchor text and attributes, the page title, its published or updated date, and when we checked it. We don't keep copies of pages. If a page GetPickedBot crawls asks not to be indexed (noindex), we don't store it at all. A page someone checks with the Link Checker is the exception: we keep the link and note the noindex, because whether search engines may index the page is part of the answer. We don't collect email addresses from pages.

Keep it off your site

Add these lines to the robots.txt file at the root of your site. GetPickedBot checks it before crawling, so the change takes effect from its next visit.

robots.txt that blocks GetPickedBot from the whole site
User-agent: GetPickedBot
Disallow: /

To keep it out of some folders only:

robots.txt that blocks GetPickedBot from two folders
User-agent: GetPickedBot
Disallow: /private/
Disallow: /drafts/

Checks a person asked for, like the free tools, link monitoring or verifying the links in a file they uploaded, fetch only the pages that check needs, much like opening them in a browser, and don't read robots.txt. If you'd like those stopped too, or GetPickedBot is causing you any trouble, email hello@getpicked.app with your domain. The operator reads every message and replies within 24 hours on weekdays (GMT+7).

More

How we use what the crawler finds is in the privacy policy.