crsociety.org FINALLY got back online after 4 months

Dean Pomerleau · Monday at 10:36 PM

Archive.org does a good job capturing format and linked images:

https://web.archive.org/web/20230921223251/https://www.crsociety.org/topic/11488-cold-exposure-other-mild-stressors-for-increased-health-longevity/

It even seems to do a good job with the video (!???) near the bottom of the page. It doesn't appear as the standard YouTube video player like it does on the original page, but I can't imagine archive.org is actually caching a copy of the video. Not sure though.

Alex K Chen · Monday at 11:43 PM

1 hour ago, Dean Pomerleau said:

This thread has a bunch of images mostly linked.

https://gyazo.com/24eed5461c7426ce00a271d35dd8ddfc [it didn't grab the other pages]

https://content.invisioncic.com/h253353/monthly_2022_12/732386800_Krasnoyarsk-31.jpg.035d5d63f747287ba9642d0a424ef698.jpg you need to auto-whitelist content.invisioncic.com/* content in order for images to d/l

====

Edited Tuesday at 12:06 AM by Alex K Chen

Alex K Chen · Tuesday at 12:25 AM

^still struggling to run, but it's fun to run in background and FINALLY try replit for once

====

https://www.crsociety.org/forum/8-members-only-area/page/5/

https://www.crsociety.org/forum/8-members-only-area/page/4

https://www.crsociety.org/forum/8-members-only-area/page/3/

https://www.crsociety.org/forum/8-members-only-area/page/2

https://www.crsociety.org/forum/8-members-only-area/page/1/

https://www.crsociety.org/forum/8-members-only-area/page/5

====

Generating URLs for thread 17649-vaccine-risks-injury-and-related-topics (pages 6-10):
https://www.crsociety.org/topic/17649-vaccine-risks-injury-and-related-topics/page/6/
https://www.crsociety.org/topic/17649-vaccine-risks-injury-and-related-topics/page/7/
https://www.crsociety.org/topic/17649-vaccine-risks-injury-and-related-topics/page/8/
https://www.crsociety.org/topic/17649-vaccine-risks-injury-and-related-topics/page/9/
https://www.crsociety.org/topic/17649-vaccine-risks-injury-and-related-topics/page/10/

Generating URLs for thread 17700-glucose-versus-cholesterol-optimization (pages 6-6):
https://www.crsociety.org/topic/17700-glucose-versus-cholesterol-optimization/page/6/

Edited Tuesday at 12:26 AM by Alex K Chen

Alex K Chen · Tuesday at 12:40 AM

^

https://www.gnu.org/software/wget/manual/html_node/Recursive-Retrieval-Options.html

(then use recursive and depth=2). Try depth=3 but it may take forever. This should be enough to archive almost all of crsociety

https://www.crsociety.org/

===

Try this below:

Quote

grep -E "https?://" urls.txt | wget -r -l 3 -k -p -D www.crsociety.org,content.invisioncic.com \
-A "*.html,*.php,*.htm" \
--convert-links \
-i -

or for httrack (which is way better)

httrack --depth=3 --ext-depth=3 --path "/path/to/save/location" \ --robots=0 --keep-alive \ --cookie "CookieName=CookieValue" \ --mirror \ -%v \ -iC2 \ -O "/path/to/save/location" \ --file-log \ -%L "urls.txt" \ "+*.content.invisioncic.com/*" \ "+www.crsociety.org/forum/6-general-health-and-longevity/*"

Updated Explanation

+*.content.invisioncic.com/*: This now includes all external links to content.invisioncic.com, ensuring that resources like images or assets from this domain are downloaded.
+www.crsociety.org/forum/6-general-health-and-longevity/*: Ensures that all forum pages in the specified directory are included in the download.

With this configuration, HTTrack will follow links to *.content.invisioncic.com while maintaining login status through the session cookie, allowing you to download content from both domains as specified.

Edited Tuesday at 01:02 AM by Alex K Chen

Alex K Chen · Tuesday at 01:18 AM

https://claude.site/artifacts/85ad24e5-cfcd-43bc-930f-14c8ebe861cc

https://claude.site/artifacts/3d68ae52-cd2a-4690-9da1-aa90a50e18ed

and iterate this algorithm for forum indexes for ALL subforum pages like https://www.crsociety.org/forum/5-cr-practice/page/28/, https://www.crsociety.org/forum/4-cr-science-theory/page/15/, https://www.crsociety.org/forum/9-chitchat/page/21/, and https://www.crsociety.org/forum/6-general-health-and-longevity/page/48/ [starting from page 1 for EACH of the subforums]

I'll create an enhanced script that first maps all subforums and then crawls each one completely.

Complete Forum Structure Crawler

Click to open code

Published

Let's add the URL export functionality:

URL Export Functions

Click to open code

Published

This complete system will:

Handle all subforums:
- General Health and Longevity (48 pages)
- CR Practice (28 pages)
- CR Science & Theory (15 pages)
- Chitchat (21 pages)
For each subforum:
- Crawl all index pages
- Extract all threads
- Parse thread metadata
- Generate URLs for all thread pages
Provide comprehensive output:
- Separate files for each subforum
- Combined URL list
- Detailed metadata and statistics

To use it:

python

Copy

# Initialize crawler crawler = ForumCrawler()  # Define HTML fetcher (implement with requests, etc.) def html_fetcher(url: str) -> str:  # Implement actual HTML fetching  pass  # Crawl everything all_threads = crawler.crawl_all_subforums(html_fetcher)  # Export all URLs and metadata export_structured_data(crawler, "crsociety_forum")

Would you like me to:

Add more specific parsing functionality?
Implement the HTML fetcher with rate limiting?
Add more export formats or metadata fields?

CopyRetry

AC

make sure all linked content from https://content.invisioncic.com* is saved

I'll modify the crawler to also capture and save all linked content from content.invisioncic.com, including images, avatars, and other media.

Enhanced Media Content Crawler

Click to open code

Published

This enhanced crawler:

Captures all media content from content.invisioncic.com:
- Images
- Avatars
- Attachments
- CSS/JS resources
- Other media files
Preserves directory structure:
- Maintains original URL paths
- Creates appropriate subdirectories
- Uses meaningful filenames when possible
Handles media properly:
- Sets correct content types
- Preserves file extensions
- Maintains referrer information
- Handles duplicates
Creates detailed manifests:
- Original URLs
- Local paths
- Content types
- Referrer pages

Directory structure example:

Copy

crsociety_archive/  ├── content.invisioncic.com/  │ ├── r123456/  │ │ ├── monthly_uploads/  │ │ ├── avatars/  │ │ └── attachments/  ├── media_manifest_6.txt  ├── media_manifest_5.txt  ├── subforum_6_threads.txt  └── metadata.txt

Would you like me to:

Add more specific media type handling?
Implement concurrent downloads?
Add resume/retry functionality?
Add more manifest formats?

Edited Tuesday at 01:36 AM by Alex K Chen

Alex K Chen · 2024-10-30T15:51:54Z

I asked someone to run via archivebot (#archiveteam) a day ago [as it is under high risk of near-deletion]. it's still running

[not sure how to access the warc file yet]

some sample output:

https://www.crsociety.org/ on 10-28; 18,096.0 MB in 103,289 resp. at 0.7/s, 357,911 in q.; 1 con. w/ 1000 ms delay; igoff9rrxic89n15t4tqb0f9qwaa8m

301 OK https://www.crsociety.org/topic/17850-are-we-exercising-too-much/?do=findComment&comment=42475

200 OK https://www.crsociety.org/topic/17850-are-we-exercising-too-much/?do=findComment&comment=42475

301 OK https://www.crsociety.org/topic/17850-are-we-exercising-too-much/?do=findComment&comment=42264

200 OK https://www.crsociety.org/topic/17850-are-we-exercising-too-much/?do=findComment&comment=42264

200 OK https://www.crsociety.org/topic/17850-are-we-exercising-too-much/?recommended=comments

301 OK https://www.crsociety.org/topic/17850-are-we-exercising-too-much/?do=findComment&comment=42467

200 OK https://www.crsociety.org/topic/17850-are-we-exercising-too-much/?do=findComment&comment=42467

301 OK https://www.crsociety.org/topic/17850-are-we-exercising-too-much/?do=findComment&comment=42553

200 OK https://www.crsociety.org/topic/17850-are-we-exercising-too-much/?do=findComment&comment=42553

301 OK https://www.crsociety.org/topic/17850-are-we-exercising-too-much/?do=findComment&comment=42566

200 OK https://www.crsociety.org/topic/17850-are-we-exercising-too-much/?do=findComment&comment=42395

200 OK https://www.theguardian.com/world/2022/jan/23/french-adventurer-75-attempting-to-row-across-the-atlantic-found-dead

301 OK https://www.crsociety.org/topic/17850-are-we-exercising-too-much/?do=findComment&comment=42394

200 OK https://www.crsociety.org/topic/17850-are-we-exercising-too-much/?do=findComment&comment=42394

301 OK https://www.crsociety.org/topic/17850-are-we-exercising-too-much/?do=findComment&comment=42464

200 OK https://www.crsociety.org/topic/17850-are-we-exercising-too-much/?do=findComment&comment=42464

301 OK https://www.crsociety.org/topic/17850-are-we-exercising-too-much/?do=findComment&comment=42527

200 OK https://www.crsociety.org/topic/17850-are-we-exercising-too-much/?do=findComment&comment=42527

301 OK https://www.crsociety.org/topic/17850-are-we-exercising-too-much/?do=findComment&comment=42457

200 OK https://www.crsociety.org/topic/17850-are-we-exercising-too-much/?do=findComment&comment=42457

200 OK https://jamanetwork.com/journals/jamainternalmedicine/fullarticle/2788473

301 OK https://www.crsociety.org/topic/17850-are-we-exercising-too-much/?do=findComment&comment=4247

Edited yesterday at 03:52 PM by Alex K Chen

Alex K Chen · 2024-10-30T16:13:40Z

also try this

https://www.firecrawl.dev/app/playground?url=https%3A%2F%2Fwww.crsociety.org%2Fforum%2F6-general-health-and-longevity%2F&mode=crawl&limit=10&excludes=&includes=&formats=markdown%2Chtml%2Clinks&onlyMainContent=true&excludeTags=&includeTags=&includeSubdomains=true&mapSearch=&uniqueKey=1730304786412

Alex K Chen · 2024-10-30T21:25:12Z

But firecrawl too complicated, I spun out a digitalocean droplet to run httrack on crsociety, hopefully it doesn't terminate when I close the window..

Alex K Chen · 2024-10-31T01:54:36Z

and it took forever.. and i figured out it had to do with the ext-depth not being limited to 1.

and then I kept on trying to restart httrack and to no avail even after rebooting the unix system, damnit.

nohup httrack --depth=3 --ext-depth=1 --path "./websites" \
--robots=0 --keep-alive \
--cookies=httrack_cookies.txt \
--mirror \
-%v \
-iC8 \
--timeout=60 \
--retries=3 \
-O "./websites" \
--file-log \
--error-log=httrack_errors.log \
-%L "urls.txt" \
"+*.content.invisioncic.com/*" \
"+*crsociety.org/*" \
"+*www.crsociety.org/*" \
> httrack.log 2>&1 &

echo $! > httrack.pid

Alex K Chen · 2024-10-31T02:14:45Z

I'll just do it on my own PC, winhttrack with depth = 3 [though depth=2 is much faster] and external-deoth=1. Maybe this will be quick enough

archivebot is still running

https://www.crsociety.org/ on 10-28; 22,145.0 MB in 130,046 resp. at 0.7/s, 370,562 in q.; 1 con. w/ 1000 ms delay; igoff9rrxic89n15t4tqb0f9qwaa8m

302 Connection closed. http://sci-hub.cc/10.1093/ije/dyw319

ERROR Fetching ‘http://ww99.sci-hub.cc/10.1093/ije/dyw319’ encountered an error: Connection closed.

301 OK http://onlinelibrary.wiley.com/doi/10.1111/jgs.14791/epdf

403 OK http://onlinelibrary.wiley.com/doi/10.1111/jgs.14791/epdf

302 OK http://sci-hub.cc/10.1002/mnfr.201400446

302 Connection closed. http://sci-hub.cc/10.1002/mnfr.201400446

ERROR Fetching ‘http://ww99.sci-hub.cc/10.1002/mnfr.201400446’ encountered an error: Connection closed.

302 OK http://sci-hub.cc/10.1080/21551197.2017.1299659

302 Connection closed. http://sci-hub.cc/10.1080/21551197.2017.1299659

ERROR Fetching ‘http://ww99.sci-hub.cc/10.1080/21551197.2017.1299659’ encountered an error: Connection closed.

301 OK https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5357610/

200 OK https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5357610/

301 OK http://www.the-scientist.com/?articles.view/articleNo/49006/title/Infographic--Circadian-Clock-Affects-Health-and-Disease/&utm_campaign=NEWSLETTER_TS_The-Scientist-Daily_2016&utm_source=hs_email&utm_medium=email&utm_content=50495832&_hsenc=p2ANqtz-8FtIMrvNoj1PaLlr6EDDStWItPUe66iWiPojpyxLlE3Bm7rDEhk1WCnDFa_u2s046Mn5I6oRkO_KGoSeQp0_qtViHodA&_hsmi=50495832

200 OK http://www.the-scientist.com/?articles.view/articleNo/49006/title/Infographic--Circadian-Clock-Affects-Health-and-Disease/&utm_campaign=NEWSLETTER_TS_The-Scientist-Daily_2016&utm_source=hs_email&utm_medium=email&utm_content=50495832&_hsenc=p2ANqtz-8FtIMrvNoj1PaLlr6EDDStWItPUe66iWiPojpyxLlE3Bm7rDEhk1WCnDFa_u2s046Mn5I6oRkO_KGoSeQp0_qtViHodA&_hsmi=50495832

302 OK http://sci-hub.cc/10.3945/ajcn.117.154294

302 Connection closed. http://sci-hub.cc/10.3945/ajcn.117.154294

ERROR Fetching ‘http://ww99.sci-hub.cc/10.3945/ajcn.117.154294’ encountered an error: Connection closed.

302 OK http://sci-hub.cc/doi/10.3945/an.116.014431

302 Connection closed. http://sci-hub.cc/doi/10.3945/an.116.014431

ERROR Fetching ‘http://ww99.sci-hub.cc/doi/10.3945/an.116.014431’ encountered an error: Connection closed.

404 OK http://jrms.mui.ac.ir/files/journals/1/articles/10516/public/10516-39461-1-PB.pdf

301 OK http://www.onlinejacc.org/content/69/9/1116

302 OK http://www.onlinejacc.org/content/69/9/1116

301 OK http://www.onlinejacc.org/content/69/9/1116

403 OK http://www.onlinejacc.org/content/69/9/1116

302 OK http://sci-hub.cc/10.1111/ger.12265

302 Connection closed. http://sci-hub.cc/10.1111/ger.12265

ERROR Fetching ‘http://ww99.sci-hub.cc/10.1111/ger.12265’ encountered an error: Connection closed.

301 OK http://www.cbc.ca/radio/thecurrent/the-current-for-february-22-2017-1.3992510/february-22-2017-full-episode-transcript-1.3994742

200 OK http://www.cbc.ca/radio/thecurrent/the-current-for-february-22-2017-1.3992510/february-22-2017-full-episode-transcript-1.3994742

301 OK https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5357610/pdf/jbm-24-31.pdf

200 OK https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5357610/pdf/jbm-24-31.pdf

302 OK https://academic.oup.com/ajh/article/30/3/279/2870260/Uric-Acid-and-New-Onset-Left-Ventricular

200 OK https://academic.oup.com/ajh/article/30/3/279/2870260/Uric-Acid-and-New-Onset-Left-Ventricular

Alex K Chen · 2024-10-31T13:54:08Z

it may be necessary to remove -*reportComment* and -*findComment* b/c these links force httrack to go over way way more URLs...

and -*getLastComment

===

https://poe.com/s/yer6HYKwjsFR6G1t6AXT

check https://www.archivebot.com/

===

AND

*&tab=comment* [damnit have to restart again]

so now scan rules are

Quote

-*
+*.png +*.gif +*.jpg +*.jpeg +*.css +*.js -ad.doubleclick.net/* -mime:application/foobar
+*.crsociety.org/*
+*.content.invisioncic.com/*
-*reportComment*
-*findComment*
-*&tab=comment*

Edited 1 hour ago by Alex K Chen

Sign In

crsociety.org FINALLY got back online after 4 months

Recommended Posts

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Updated Explanation

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Join the conversation