How to properly scrape the forum so we don't lose it again

[Alex K Chen]

Invisionforum gives admins the ability to download the entire forum free of change.

But there are other scraping services like Winhttrack which I used to scrape some entire forums (still need to get the options right for multi-page threads and too much is going on for me to properly run it for a while)

and some new ones like https://x.com/ericciarla/status/1847002234372215172

https://www.youtube.com/watch?v=V0d_Q3Gq3-Q

Edited October 19 by Alex K Chen

[Alex K Chen]

https://github.com/mendableai/firecrawl

https://forum.cloudron.io/topic/12140/firecrawl-on-cloudron-turn-any-site-into-llm-data-by-web-scraping/3

Claude Sonnet 3.5.1 has VERY interesting code I should try.

Edited October 25 by Alex K Chen