mirror/archivebox
1
0
Fork 0
mirror of https://github.com/pirate/ArchiveBox.git synced 2025-08-14 18:44:24 +02:00
Code Issues Releases Wiki Activity

Reformats document, fixes spelling and caps 

I know this is kind of uncool to do, but there's no need to have this indent on the list elements :)

- `+` secondary list bullets → `-`
- Github → GitHub
- ipfs → IPFS
- Heretrix → Heritrix
- warc → WARC (where appropriate)
- wayback machine → Wayback Machine (where appropriate, as used to relate to Internet Archive's tool)
Henry Wilkinson
2024-10-09 13:40:36 -04:00
committed by Nick Sweeting
parent 678856ed43
commit 74331e4082
1 changed files with 286 additions and 287 deletions
Download Patch File Download Diff File Expand all files Collapse all files

59
Web-Archiving-Community.md

@@ -16,7 +16,6 @@ The internet archiving community is surprisingly far-reaching and almost univers
Whether you want to learn which organizations are the big players in the web archiving space, want to find a specific open source tool for your web archiving need, or just want to see where archivists hang out online, this is my attempt at an index of the entire web archiving community. Whether you want to learn which organizations are the big players in the web archiving space, want to find a specific open source tool for your web archiving need, or just want to see where archivists hang out online, this is my attempt at an index of the entire web archiving community.
<img src="https://imgur.zervice.io/duS8Lm7.png" width="200px" align="right" style="float: right; margin: 5px"/> <img src="https://imgur.zervice.io/duS8Lm7.png" width="200px" align="right" style="float: right; margin: 5px"/>
- [The Master Lists](#The-Master-Lists) - [The Master Lists](#The-Master-Lists)
@@ -24,24 +23,24 @@ Whether you want to learn which organizations are the big players in the web arc
- [Web Archiving Software](#Web-Archiving-Projects) - [Web Archiving Software](#Web-Archiving-Projects)
*Open source tools and projects in the internet archiving space.* *Open source tools and projects in the internet archiving space.*
+ [Bookmarking Services](#bookmarking-services) - [Bookmarking Services](#bookmarking-services)
+ [Well-Known Open Source Projects](#from-the-archiveorg--archive-it-teams) - [Well-Known Open Source Projects](#from-the-archiveorg--archive-it-teams)
+ [Public Archiving Services](#other-public-archiving-services) - [Public Archiving Services](#other-public-archiving-services)
+ [ArchiveBox Alternatives](#other-archivebox-alternatives) - [ArchiveBox Alternatives](#other-archivebox-alternatives)
+ [Smaller Utilities](#smaller-utilities) - [Smaller Utilities](#smaller-utilities)
- [Reading List](#Reading-List) - [Reading List](#Reading-List)
*Articles, posts, and blogs relevant to ArchiveBox and web archiving in general.* *Articles, posts, and blogs relevant to ArchiveBox and web archiving in general.*
+ [Blogs](#Blogs) - [Blogs](#Blogs)
+ [Articles](#Articles) - [Articles](#Articles)
+ [ArchiveBox-Specific Posts, Tutorials, and Guides](#archivebox-specific-posts-tutorials-and-guides) - [ArchiveBox-Specific Posts, Tutorials, and Guides](#archivebox-specific-posts-tutorials-and-guides)
+ [ArchiveBox Discussions in News & Social Media](#archivebox-discussions-in-news--social-media) - [ArchiveBox Discussions in News & Social Media](#archivebox-discussions-in-news--social-media)
- [Communities](#Communities) - [Communities](#Communities)
*A collection of the most active internet archiving communities and initiatives.* *A collection of the most active internet archiving communities and initiatives.*
+ [Most Active Web-Archiving Communities](#most-active-communities) - [Most Active Web-Archiving Communities](#most-active-communities)
+ [Other Web Archiving Communities](#web-archiving-communities) - [Other Web Archiving Communities](#web-archiving-communities)
+ [General Archiving Foundations, Coalitions, Initiatives, and Institutes](#general-archiving-foundations-coalitions-initiatives-and-institutes) - [General Archiving Foundations, Coalitions, Initiatives, and Institutes](#general-archiving-foundations-coalitions-initiatives-and-institutes)
--- ---
@@ -89,15 +88,15 @@ Indexes of archiving institutions and software maintained by other people. If t
<img src="https://docs.monadical.com/uploads/9b9d97d4-eaa4-4399-b7e2-628f1ef81675.png" width="115px" align="right" style="float:right; margin: 5px"/> <img src="https://docs.monadical.com/uploads/9b9d97d4-eaa4-4399-b7e2-628f1ef81675.png" width="115px" align="right" style="float:right; margin: 5px"/>
<img src="https://upload.wikimedia.org/wikipedia/commons/thumb/8/84/Internet_Archive_logo_and_wordmark.svg/250px-Internet_Archive_logo_and_wordmark.svg.png" width="100px" align="right" style="float:right; margin: 5px"/> <img src="https://upload.wikimedia.org/wikipedia/commons/thumb/8/84/Internet_Archive_logo_and_wordmark.svg/250px-Internet_Archive_logo_and_wordmark.svg.png" width="100px" align="right" style="float:right; margin: 5px"/>
- **[Archive.org](https://archive.org) The O.G. wayback machine provided publicly by the Internet Archive (Archive.org)** - **[Archive.org](https://archive.org) The O.G. Wayback Machine provided publicly by the Internet Archive (Archive.org)**
- **[Archive.it](https://archive-it.org) commercial Wayback-Machine solution** - **[Archive.it](https://archive-it.org) commercial Wayback Machine solution**
- **[Heretrix](https://github.com/internetarchive/heritrix3) The king of internet archiving crawlers, powers the Wayback Machine** - **[Heritrix](https://github.com/internetarchive/heritrix3) The king of internet archiving crawlers, powers the Wayback Machine**
- **[Brozzler](https://github.com/internetarchive/brozzler) chrome headless crawler + WARC archiver maintained by Archive.org** - **[Brozzler](https://github.com/internetarchive/brozzler) chrome headless crawler + WARC archiver maintained by Archive.org**
- [WarcProx](https://github.com/internetarchive/warcprox) warc proxy recording and playback utility - [WarcProx](https://github.com/internetarchive/warcprox) WARC proxy recording and playback utility
- [WarcTools](https://github.com/internetarchive/warctools) utilities for dealing with WARCs - [WarcTools](https://github.com/internetarchive/warctools) utilities for dealing with WARCs
- [Grab-Site](https://github.com/ArchiveTeam/grab-site) An easy preconfigured web crawler designed for backing up websites - [Grab-Site](https://github.com/ArchiveTeam/grab-site) An easy preconfigured web crawler designed for backing up websites
- [WPull](https://github.com/ArchiveTeam/wpull) A pure python implementation of wget with WARC saving - [WPull](https://github.com/ArchiveTeam/wpull) A pure python implementation of wget with WARC saving
- [More on their Github...](https://github.com/internetarchive) - [More on their GitHub...](https://github.com/internetarchive)
--- ---
@@ -128,14 +127,14 @@ Indexes of archiving institutions and software maintained by other people. If t
<img src="https://avatars2.githubusercontent.com/u/4416806?s=280&v=4" width="130px" align="right" style="float: right; margin: 5px"/> <img src="https://avatars2.githubusercontent.com/u/4416806?s=280&v=4" width="130px" align="right" style="float: right; margin: 5px"/>
- **[ipwb](https://github.com/oduwsdl/ipwb) A distributed web archiving solution using pywb with ipfs for storage** - **[ipwb](https://github.com/oduwsdl/ipwb) A distributed web archiving solution using pywb with IPFS for storage**
- **[archivenow](https://github.com/oduwsdl/archivenow) tool that pushes urls into all the online archive services like Archive.is and Archive.org** - **[archivenow](https://github.com/oduwsdl/archivenow) tool that pushes urls into all the online archive services like Archive.is and Archive.org**
- [node-warc](https://github.com/N0taN3rd/node-warc) Parse And Create Web ARChive (WARC) files with node.js - [node-warc](https://github.com/N0taN3rd/node-warc) Parse And Create Web ARChive (WARC) files with node.js
- [WAIL](https://machawk1.github.io/wail/) Web archiver GUI using Heritrix and OpenWayback - [WAIL](https://machawk1.github.io/wail/) Web archiver GUI using Heritrix and OpenWayback
- [Squidwarc](https://github.com/N0taN3rd/Squidwarc) User-scriptable, archival crawler using Chrome - [Squidwarc](https://github.com/N0taN3rd/Squidwarc) User-scriptable, archival crawler using Chrome
- [WAIL (Electron)](https://github.com/n0tan3rd/wail) Electron app version of the original [wail](https://github.com/machawk1/wail) for creating and interacting with web archives - [WAIL (Electron)](https://github.com/n0tan3rd/wail) Electron app version of the original [wail](https://github.com/machawk1/wail) for creating and interacting with web archives
- **[warcreate](https://github.com/machawk1/warcreate) a Chrome extension for creating WARCs from any webpage** - **[warcreate](https://github.com/machawk1/warcreate) a Chrome extension for creating WARCs from any webpage**
- [More on their Github...](https://github.com/oduwsdl) - [More on their GitHub...](https://github.com/oduwsdl)
--- ---
@@ -145,7 +144,7 @@ Indexes of archiving institutions and software maintained by other people. If t
- [AUT](https://github.com/archivesunleashed/aut) Archives Unleashed Toolkit for analyzing web archives (formerly WarcBase) - [AUT](https://github.com/archivesunleashed/aut) Archives Unleashed Toolkit for analyzing web archives (formerly WarcBase)
- [Warclight](https://github.com/archivesunleashed/warclight) A Rails engine for finding and searching web archives - [Warclight](https://github.com/archivesunleashed/warclight) A Rails engine for finding and searching web archives
- [More on their Github...](https://github.com/archivesunleashed) - [More on their GitHub...](https://github.com/archivesunleashed)
--- ---
@@ -153,10 +152,10 @@ Indexes of archiving institutions and software maintained by other people. If t
### From the IIPC team ### From the IIPC team
- **[OpenWayback](https://github.com/iipc/openwayback/wiki) Open source project developing core Wayback-Machine components** - **[OpenWayback](https://github.com/iipc/openwayback/wiki) Open source project developing core Wayback Machine components**
- **[awesome-web-archiving](https://github.com/iipc/awesome-web-archiving) Large list of archiving projects and orgs** - **[awesome-web-archiving](https://github.com/iipc/awesome-web-archiving) Large list of archiving projects and orgs**
- [JWARC](https://github.com/iipc/jwarc) A Java library for reading and writing WARC files. - [JWARC](https://github.com/iipc/jwarc) A Java library for reading and writing WARC files.
- [More on their Github...](https://github.com/iipc) - [More on their GitHub...](https://github.com/iipc)
--- ---
@@ -188,6 +187,7 @@ Indexes of archiving institutions and software maintained by other people. If t
> *There are lots more projects listed here too: https://github.com/stars/pirate/lists/internet-archiving* > *There are lots more projects listed here too: https://github.com/stars/pirate/lists/internet-archiving*
- **[Browsertrix](https://webrecorder.net/browsertrix) + [ArchiveWeb.page](https://webrecorder.net/archivewebpage) + [ReplayWeb.page](https://webrecorder.net/replaywebpage) Webrecorder's archiving suite has the highest fidelity, and can flawlessly archive YouTube, X, Facebook, and other complex, JS-heavy SPAs**
- **[SingleFile](https://github.com/gildas-lormeau/SingleFile/) Web Extension / CLI util for Firefox and Chrome to save a web page as a single HTML file** - **[SingleFile](https://github.com/gildas-lormeau/SingleFile/) Web Extension / CLI util for Firefox and Chrome to save a web page as a single HTML file**
- **[Memex by Worldbrain.io](https://github.com/WorldBrain/Memex) a beautiful, user-friendly browser extension that archives all history with full-text search, annotation support, and more** - **[Memex by Worldbrain.io](https://github.com/WorldBrain/Memex) a beautiful, user-friendly browser extension that archives all history with full-text search, annotation support, and more**
- **[Hypothes.is](https://web.hypothes.is/) a web/pdf/ebook annotation tool that also archives content** - **[Hypothes.is](https://web.hypothes.is/) a web/pdf/ebook annotation tool that also archives content**
@@ -197,8 +197,7 @@ Indexes of archiving institutions and software maintained by other people. If t
- **[Polarized](https://getpolarized.io/) a desktop application for bookmarking, annotating, and archiving articles offline** - **[Polarized](https://getpolarized.io/) a desktop application for bookmarking, annotating, and archiving articles offline**
- **[LinkWarden](https://github.com/linkwarden/linkwarden) Link archival and curation web app, very similar to ArchiveBox** - **[LinkWarden](https://github.com/linkwarden/linkwarden) Link archival and curation web app, very similar to ArchiveBox**
- **[Photon](https://github.com/s0md3v/Photon) a fast crawler with archiving and asset extraction support** - **[Photon](https://github.com/s0md3v/Photon) a fast crawler with archiving and asset extraction support**
- **[Scoop](https://github.com/harvard-lil/scoop)** Create high-fidelity WARC/WACZ captures using a playwright browser, with support for signing, media extraction, PDFs, etc. ([by the perma.cc team](https://lil.law.harvard.edu/blog/2023/04/13/scoop-witnessing-the-web/)) - **[Scoop](https://github.com/harvard-lil/scoop)** Create high-fidelity WARC/WACZ captures using a playwright browser, with support for signing, media extraction, PDFs, etc. ([by the Perma.cc team](https://lil.law.harvard.edu/blog/2023/04/13/scoop-witnessing-the-web/))
- **[Browsertrix](https://webrecorder.net/browsertrix) + [ArchiveWeb.page](https://webrecorder.net/archivewebpage) + [ReplayWeb.page](https://webrecorder.net/replaywebpage) Webrecorder's archiving suite has the highest fidelity, and can flawlessly archive YouTube, X, Facebook, and other complex, JS-heavy SPAs**
Ones I haven't personally vetted: Ones I haven't personally vetted:
@@ -354,7 +353,7 @@ If any of these links are dead, you can find an archived version on https://arch
<img src="https://cdn.dribbble.com/users/896843/screenshots/2560608/news_media_icons-07.png" width="380px" align="right" style="float: right; margin: 5px"/> <img src="https://cdn.dribbble.com/users/896843/screenshots/2560608/news_media_icons-07.png" width="380px" align="right" style="float: right; margin: 5px"/>
- **Aggregators:** - **Aggregators:**
**[ProductHunt](https://www.producthunt.com/posts/archivebox)**, **[AlternativeTo](https://alternativeto.net/software/archivebox/)**, **[SaaSHub](https://www.saashub.com/archivebox)**, [Logiciels](https://www.logiciels.pro/logiciel-saas/archivebox/), [SteemHunt](https://steemhunt.com/@adnan556644/archivebox-the-open-source-self-hosted-internet-archiving-solution), [Recurse Center: The Joy of Computing](https://joy.recurse.com/posts/224-archivebox), [Github Changelog](https://changelog.com/news/archivebox-opensource-selfhosted-web-archive-6D0d), [Dev.To Ultra List](https://dev.to/teamxenox/-ultra-list-one-list-to-rule-them-all-march-19-4p4f), [O'Reilly 4 Short Links](https://www.oreilly.com/ideas/four-short-links-15-april-2019), [JaxEnter](https://jaxenter.com/github-trending-march-2019-157470.html) **[ProductHunt](https://www.producthunt.com/posts/archivebox)**, **[AlternativeTo](https://alternativeto.net/software/archivebox/)**, **[SaaSHub](https://www.saashub.com/archivebox)**, [Logiciels](https://www.logiciels.pro/logiciel-saas/archivebox/), [SteemHunt](https://steemhunt.com/@adnan556644/archivebox-the-open-source-self-hosted-internet-archiving-solution), [Recurse Center: The Joy of Computing](https://joy.recurse.com/posts/224-archivebox), [GitHub Changelog](https://changelog.com/news/archivebox-opensource-selfhosted-web-archive-6D0d), [Dev.To Ultra List](https://dev.to/teamxenox/-ultra-list-one-list-to-rule-them-all-march-19-4p4f), [O'Reilly 4 Short Links](https://www.oreilly.com/ideas/four-short-links-15-april-2019), [JaxEnter](https://jaxenter.com/github-trending-march-2019-157470.html)
- **Blog Posts & Podcasts:** - **Blog Posts & Podcasts:**
[Korben.info](https://korben.info/archivebox-un-clone-darchive-org-et-de-la-wayback-machine-a-auto-heberger.html), [Defining Desktop Linux Podcast #296 (0:55:00)](https://linuxunplugged.com/296), [Binärgewitter Podcast #221](http://blog.binaergewitter.de/2019/01/18/binaergewitter-talk-number-221-vertieft-in-die-andere-richtung/), [Schrankmonster.de](https://www.schrankmonster.de/2019/04/10/archive-your-slice-of-the-web/), [La Ferme Du Web](https://www.lafermeduweb.net/veille/archivebox-archivez-des-copies-de-sites-en-local-avec-tous-les-medias-lies) [Korben.info](https://korben.info/archivebox-un-clone-darchive-org-et-de-la-wayback-machine-a-auto-heberger.html), [Defining Desktop Linux Podcast #296 (0:55:00)](https://linuxunplugged.com/296), [Binärgewitter Podcast #221](http://blog.binaergewitter.de/2019/01/18/binaergewitter-talk-number-221-vertieft-in-die-andere-richtung/), [Schrankmonster.de](https://www.schrankmonster.de/2019/04/10/archive-your-slice-of-the-web/), [La Ferme Du Web](https://www.lafermeduweb.net/veille/archivebox-archivez-des-copies-de-sites-en-local-avec-tous-les-medias-lies)
- **Hacker News threads and comments:** - **Hacker News threads and comments:**
@@ -377,12 +376,12 @@ If any of these links are dead, you can find an archived version on https://arch
- **[International Internet Preservation Consortium (IIPC)](http://netpreserve.org/)** (International) - **[International Internet Preservation Consortium (IIPC)](http://netpreserve.org/)** (International)
- **[The Archive Team](https://www.archiveteam.org/), [URL Team](https://www.archiveteam.org/index.php?title=URLTeam), [r/ArchiveTeam](https://reddit.com/r/ArchiveTeam)** (International) - **[The Archive Team](https://www.archiveteam.org/), [URL Team](https://www.archiveteam.org/index.php?title=URLTeam), [r/ArchiveTeam](https://reddit.com/r/ArchiveTeam)** (International)
- **[Rhizome.org](http://archive.rhizome.org/)** The digital preservation group that works on [Conifer by Rhizome](https://conifer.rhizome.org/) formerly Webrecorder.io (USA) - **[Rhizome.org](http://archive.rhizome.org/)** The digital preservation group that works on [Conifer by Rhizome](https://conifer.rhizome.org/) formerly Webrecorder.io (USA)
- **[Webrecorder.net](https://webrecorder.net/)** Formerly known[¹](https://blog.conifer.rhizome.org/2020/06/11/webrecorder-conifer.html) as Webrecorder.io is a project Led by Ilya Kreymer, that researches and develops web archiving tools, widely used by the community. - **[Webrecorder](https://webrecorder.net/)** Formerly known[¹](https://blog.conifer.rhizome.org/2020/06/11/webrecorder-conifer.html) as Webrecorder.io is a company led by Ilya Kreymer, that researches and develops web archiving tools, widely used by the community.
- **[Old Dominion University: Web Science and Digital Libraries (WS-DL @ ODU)](https://ws-dl.cs.odu.edu)** (Virginia, USA) - **[Old Dominion University: Web Science and Digital Libraries (WS-DL @ ODU)](https://ws-dl.cs.odu.edu)** (Virginia, USA)
- **[r/DataHoarder](https://www.reddit.com/r/DataHoarder), [r/Archivists](https://www.reddit.com/r/Archivists/), [r/DHExchange](https://www.reddit.com/r/DHExchange/)** (International) - **[r/DataHoarder](https://www.reddit.com/r/DataHoarder), [r/Archivists](https://www.reddit.com/r/Archivists/), [r/DHExchange](https://www.reddit.com/r/DHExchange/)** (International)
- [The Eye](https://the-eye.eu) Non-profit working on content archival and long-term preservation (Europe) - [The Eye](https://the-eye.eu) Non-profit working on content archival and long-term preservation (Europe)
- [Digital Preservation Coalition](https://www.dpconline.org/about) & their [Software Tool Registry (COPTR)](http://coptr.digipres.org/Main_Page) (UK & Wales) - [Digital Preservation Coalition](https://www.dpconline.org/about) & their [Software Tool Registry (COPTR)](http://coptr.digipres.org/Main_Page) (UK & Wales)
- [Archives Unleashed Project](https://archivesunleashed.org/about-project/) and [UAP Github](https://github.com/archivesunleashed) (Canada) - [Archives Unleashed Project](https://archivesunleashed.org/about-project/) and [UAP GitHub](https://github.com/archivesunleashed) (Canada)
--- ---
@@ -405,7 +404,7 @@ Follow these technological and organizational archiving hubs for the latest arch
- [Stanford Libraries Web Archiving](https://library.stanford.edu/projects/web-archiving) (USA) - [Stanford Libraries Web Archiving](https://library.stanford.edu/projects/web-archiving) (USA)
- [Society of American Archivists: Electronic Records (SAA)](https://www2.archivists.org/groups/electronic-records-section) (USA) - [Society of American Archivists: Electronic Records (SAA)](https://www2.archivists.org/groups/electronic-records-section) (USA)
- [BitCurator Consortium (BCC)](https://bitcuratorconsortium.org/mission) (USA) - [BitCurator Consortium (BCC)](https://bitcuratorconsortium.org/mission) (USA)
- [Ethics & Archiving the Web Conference (Rhizome/Webrecorder.io)](https://eaw.rhizome.org/) (USA) - [Ethics & Archiving the Web Conference (Rhizome)](https://eaw.rhizome.org/) (USA)
- [Archivists Round Table of NYC](https://www.nycarchivists.org/) (USA) - [Archivists Round Table of NYC](https://www.nycarchivists.org/) (USA)
--- ---
@@ -475,7 +474,7 @@ You can find more organizations and initiatives on these other lists:
- [Official ArchiveBox Zulip Chat Server](https://zulip.archivebox.io) - [Official ArchiveBox Zulip Chat Server](https://zulip.archivebox.io)
- [Unofficial ArchiveBox Matrix chat room](https://matrix.to/#/#archivebox:matrix.org) (old) - [Unofficial ArchiveBox Matrix chat room](https://matrix.to/#/#archivebox:matrix.org) (old)
- [Github Discussions](https://github.com/ArchiveBox/ArchiveBox/discussions) - [GitHub Discussions](https://github.com/ArchiveBox/ArchiveBox/discussions)
### ArchiveBox on Social Media ### ArchiveBox on Social Media