Crawling the Rust Book (Remote Server)

Crawl Results

Beam Us UpFreeGUI · 16 threads (auto)

00:05.91140114 / 01243 MB114 pages · 0 assets · 1243 MB peak

Beam Us Up (JS)FreeGUI · 16 threads (auto)

00:44.61140114 / 01905 MB114 pages · 0 assets · 1905 MB peak

Beam Us Up CLIFreeCLI · 10 threads

00:04.81140114 / 01283 MB114 pages · 0 assets · 1283 MB peak

Beam Us Up CLI (JS)FreeCLI · 10 threads

00:43.41140114 / 0912 MB114 pages · 0 assets · 912 MB peak

GNU WgetFreeCLI · Single thread

00:02.311462114 / 623.4 MB114 pages · 62 assets · 3.4 MB peak

InterroBotFreeGUI · Default threads

00:02.311450114 / 50476 MB114 pages · 50 assets · 476 MB peak

InterroBot (JS)PaidGUI · Default threads

00:39.611450114 / 50999 MB114 pages · 50 assets · 999 MB peak

InterroBot (Turbo JS)PaidGUI · Turbo threads

00:21.511450114 / 501077 MB114 pages · 50 assets · 1077 MB peak

InterroBot (Turbo)PaidGUI · Turbo threads

00:01.711450114 / 50439 MB114 pages · 50 assets · 439 MB peak

InterroBot CLIFreeCLI · Default threads

00:01.511450114 / 50156 MB114 pages · 50 assets · 156 MB peak

InterroBot CLI (JS)PaidCLI · Default threads

00:39.311450114 / 50613 MB114 pages · 50 assets · 613 MB peak

InterroBot CLI (Turbo JS)PaidCLI · Turbo threads

00:20.911450114 / 50742 MB114 pages · 50 assets · 742 MB peak

InterroBot CLI (Turbo)PaidCLI · Turbo threads

00:01.011450114 / 50160 MB114 pages · 50 assets · 160 MB peak

KatanaFreeCLI · 10 threads

00:12.411350113 / 5058.0 MB113 pages · 50 assets · 58.0 MB peak

Screaming FrogFreeGUI · Default threads

00:05.011548115 / 481351 MB115 pages · 48 assets · 1351 MB peak

Screaming Frog CLIFreeCLI · Default threads

00:05.311548115 / 48595 MB115 pages · 48 assets · 595 MB peak

SEO MacroscopeFreeGUI · Default threads

00:12.211551115 / 51270 MB115 pages · 51 assets · 270 MB peak

SEO Macroscope (High)FreeGUI · 10 threads

00:12.211551115 / 51254 MB115 pages · 51 assets · 254 MB peak

SiteOne CrawlerFreeGUI · Default threads

00:14.411461114 / 61353 MB114 pages · 61 assets · 353 MB peak

SiteOne Crawler (High)FreeGUI · 8 threads

00:06.411461114 / 61351 MB114 pages · 61 assets · 351 MB peak

Visual SEO StudioFreeGUI · Default threads

00:05.411428114 / 28157 MB114 pages · 28 assets · 157 MB peak

Xenu's Link SleuthFreeGUI · 30 threads

00:04.311561115 / 615.7 MB115 pages · 61 assets · 5.7 MB peak

Resource Usage

Watch in Realtime

Every crawl starts together at 0:07. Click a screen to enlarge it.

0:00.0 / 1:02.0

Beam Us UpReady

Beam Us Up (JS)Ready

Beam Us Up CLIReady

Beam Us Up CLI (JS)Ready

GNU WgetReady

InterroBotReady

InterroBot (JS)Ready

InterroBot (Turbo JS)Ready

InterroBot (Turbo)Ready

InterroBot CLIReady

InterroBot CLI (JS)Ready

InterroBot CLI (Turbo JS)Ready

InterroBot CLI (Turbo)Ready

KatanaReady

Screaming FrogReady

Screaming Frog CLIReady

SEO MacroscopeReady

SEO Macroscope (High)Ready

SiteOne CrawlerReady

SiteOne Crawler (High)Ready

Visual SEO StudioReady

Xenu's Link SleuthReady

Setups

Crawler	Version	Interface	Threads	Settings	Machine
Beam Us Up	2.0.0	GUI	16 threads (auto)	Default settings	Mac mini (M1)
Beam Us Up (JS)	2.0.0	GUI	16 threads (auto)	JavaScript rendering on	Mac mini (M1)
Beam Us Up CLI	2.0.0	CLI	10 threads	Default settings	Mac mini (M1)
Beam Us Up CLI (JS)	2.0.0	CLI	10 threads	JavaScript rendering on	Mac mini (M1)
GNU Wget	1.25.0	CLI	Single thread	Mirror preset	Mac mini (M1)
InterroBot	2.15.0	GUI	Default threads	Default settings	Mac mini (M1)
InterroBot (JS)	2.15.0	GUI	Default threads	JavaScript rendering on	Mac mini (M1)
InterroBot (Turbo JS)	2.15.0	GUI	Turbo threads	Turbo threads; JavaScript rendering on	Mac mini (M1)
InterroBot (Turbo)	2.15.0	GUI	Turbo threads	Turbo threads	Mac mini (M1)
InterroBot CLI	2.15.0	CLI	Default threads	Default settings	Mac mini (M1)
InterroBot CLI (JS)	2.15.0	CLI	Default threads	JavaScript rendering on	Mac mini (M1)
InterroBot CLI (Turbo JS)	2.15.0	CLI	Turbo threads	Turbo threads; JavaScript rendering on	Mac mini (M1)
InterroBot CLI (Turbo)	2.15.0	CLI	Turbo threads	Turbo threads	Mac mini (M1)
Katana	1.7.0	CLI	10 threads	Full depth; no subdomains; images kept	Mac mini (M1)
Screaming Frog	24.3.0	GUI	Default threads	Default settings	Mac mini (M1)
Screaming Frog CLI	24.3.0	CLI	Default threads	Default settings	Mac mini (M1)
SEO Macroscope	1.7.6.1	GUI	Default threads	Default settings	Windows 10 (VM)
SEO Macroscope (High)	1.7.6.1	GUI	10 threads	10 threads	Windows 10 (VM)
SiteOne Crawler	1.0.8	GUI	Default threads	Report upload off	Mac mini (M1)
SiteOne Crawler (High)	1.0.8	GUI	8 threads	8 threads; 50 requests/s; report upload off	Mac mini (M1)
Visual SEO Studio	2.5.3.1	GUI	Default threads	Max crawl depth 9	Windows 10 (VM)
Xenu's Link Sleuth	1.3.8	GUI	30 threads	Default settings	Windows 10 (VM)

How This Was Measured

Test Machines: Mac mini (M1/Tahoe)8 GB of RAM · macOS 26.6.2
Windows 10 (Hyper-V)15.8 GB of RAM · Windows 10 · virtual machine
Each crawler ran by itself. Crawl time, counts, memory, and CPU are captured consistently, per OS. See more on how it’s measured.
Sites: Rust Bookrust-book-remote.crawlerbench.com114 HTML pages176 files
GIMP 3.0 Manualgimp-remote.crawlerbench.com711 HTML pages2,616 files
Served from a remote server, reached over the internet.