Why Do I Need This?

Protect Your Data From Unauthorized Bots

"I have seen reports that it may or may not have been used. I have no information myself."
- YouTube CEO, Neal Mohan on OpenAI (ChatGPT) scraping and using their data

Thank you!

Aenean ornare velit lacus et varius enim proin aliquam facilisis ante sed etiam feugiat sed lorem consequat.

• Okay

What is a Robots.txt?

A Robots.txt tells bots what they are allowed to look at on your website

"Web Robots (also known as Web Wanderers, Crawlers, or Spiders), are programs that traverse the Web automatically. Search engines such as Google use them to index the web content, spammers use them to scan for email addresses, and they have many other uses."

Source:
https://www.robotstxt.org/robotstxt.html

Glossary background:
https://darkvisitors.com/agents/

More resources on creating and managing robots.txt files:

https://moz.com/learn/seo/robotstxt
http://www.robotstxt.org/about.html
https://moz.com/blog/interactive-guide-to-robots-txt
http://www.robotstxt.org/robotstxt.html

About Us

Futureproof TMT: Welcome to Robots Mapper!

RobotsMapper is a freemium tool built and developed by Futureproof TMT, the research and advisory firm of FIT Holdings.

Futureproof’s inaugural 2024 research report on the changing technical dynamics and market economics was released in April 2024, which includes an analysis of how the technology, media, and telecommunications sectors have evolved to serve hyperscale needs of the modern media and data landscape. The paper, which can be found here, also outlines how GenAI tools & LLM’s have caused an even further increase in the level and impact of non-human web traffic not just on analytics but on media intellectual property rights.

While researching current news coverage in preparation for releasing the research report, our team saw an interview with the YouTube CEO speaking about their stance on crawlers and scrapers from firms like OpenAI using YouTube’s hosted content to train their models. Surprisingly, we found that YouTube does not set even basic rules for crawlers from the likes of OpenAI and others, while navigating their sitemap on the open web!

If YouTube isn’t yet doing the basics to manage web robots, we created a tool to help others interested in this space to analyze what sites are managing their sitemaps properly for the age of AI.

Creators: Shailin Dhar, Scott Thomson and Arjun Krishna

Contact Us

Want more data? Let us know

List of Sites and Categories

Want access to our full dataset? Contact us