Definition
robots.txt is a file at the root of a site that declares which crawlers may fetch which paths. It is a declaration that well-behaved crawlers honour, not an enforcement mechanism, and it says nothing about what happens once a page has been fetched.
Why it matters
It is the cheapest thing to get wrong and the cheapest to fix. A single misplaced disallow can remove a site from every answer engine at once, and nothing on the site looks broken while it happens.
How to measure it
Parse the file per user agent rather than reading it top to bottom, because the most specific matching group wins and a later wildcard group does not apply to an agent that has its own. Then verify with real requests.