The Big List of Cities runs on a bot-based crawler developed by Dan Green and The Big List of Websites to discover and catalog U.S. cities. The goal of this site is to build a public index of information with short descriptions and links. It’s an exploratory, personal project that uses Generative AI.
The bot only visits publicly available domains that resolve successfully.
We do not scrape or store private data and respect robots.txt where applicable.
This bot identifies itself with the following HTTP User-Agent string:
Mozilla/5.0 (compatible; TheBigListBot/1.0; +https://thebiglistofwebsites.com/bot)