Pink PostCrawling vs indexing
robots.txt controls crawling: whether a bot may fetch a URL. Meta robots and the X-Robots-Tag header control indexing: whether a page may appear in search. They interact in a way that catches people out: if robots.txt blocks a page, Google cannot see its noindex tag, so the URL can still be indexed (without content) from links. To remove a page from search, allow crawling and use noindex.
How robots.txt rules are matched
A bot uses the group of rules for its most specific user-agent, falling back to *. Within the group the longest matching path wins, and when an Allow and a Disallow match with the same length, Allow wins. * matches any characters and $ anchors the end of the URL. This checker follows those rules (RFC 9309), so a page shown as Allowed for Googlebot can still be blocked for others.
AI crawlers
Many sites now block AI crawlers such as GPTBot, ClaudeBot, CCBot and Google-Extended in robots.txt. Google states that Google-Extended does not affect Google Search. The table shows how this page treats each.
Why link builders check indexability
A link on a page that is noindexed or blocked from crawling passes little or no value. Before you pay for or pitch a placement, check the target page and a few existing posts on the site. After the link is live, the backlink checker runs these same checks for the specific link.