Posts Google open sources standardized code in bid to become Mr Robots.txt
Post
Cancel

Google open sources standardized code in bid to become Mr Robots.txt

Google released its robots.txt parsing and matching library as open source in the hope of its now public code will help encourage web developers to agree on a standard way to spell out the proper etiquette for web crawlers.

https://www.theregister.co.uk/2019/07/02/google_robot_txt/

https://github.com/google/robotstxt

origin - https://www.pipiscrew.com/2019/07/google-open-sources-standardized-code-in-bid-to-become-mr-robots-txt/ google-open-sources-standardized-code-in-bid-to-become-mr-robots-txt

This post is licensed under CC BY 4.0 by the author.
Contents

Trending Tags