From 442f074bd70f9389ea3b489837a3e471dad8195c Mon Sep 17 00:00:00 2001 From: Nick Sweeting Date: Fri, 3 May 2024 18:34:01 -0700 Subject: [PATCH] Updated Publishing Your Archive (markdown) --- Publishing-Your-Archive.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Publishing-Your-Archive.md b/Publishing-Your-Archive.md index 473a08a..68a86e8 100644 --- a/Publishing-Your-Archive.md +++ b/Publishing-Your-Archive.md @@ -77,7 +77,7 @@ Archiving for personal backups, research, and some other use-cases are covered b Please modify the [`FOOTER_INFO`](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#footer_info) config variable to add your contact info to the footer of your index. -Note ArchiveBox prevents search engines from indexing your archives using `/robots.txt` by default, it's not recommended to disable this as it often leads to a flood of automated takedown requests from bots that check for cloned content online. +Note ArchiveBox prevents search engines from indexing your archives using [`/robots.txt`](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/templates/static/robots.txt#L2) by default, it's not recommended to [disable](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#custom_templates_dir) this as it often leads to a flood of automated takedown requests and abuse reports to your hosting provider (from bots that periodically scan for cloned content online). #### Further Reading