Google released its robots.txt parsing and matching library as open source in the hope of its now public code will help encourage web developers to agree on a standard way to spell out the proper etiquette for web crawlers.
https://www.theregister.co.uk/2019/07/02/google_robot_txt/
https://github.com/google/robotstxt
origin - https://www.pipiscrew.com/2019/07/google-open-sources-standardized-code-in-bid-to-become-mr-robots-txt/ google-open-sources-standardized-code-in-bid-to-become-mr-robots-txt