ND Crawling a billion web pages in just over 24 hours, in 2025 (andrewkchan.dev)
197 points by pseudolus 10 days ago | 61 comments on HN ~lite vlite-2.0
Summary ~lite
The author crawled 1.005 billion web pages in 25.5 hours with a budget of $462.
Lite evaluation by llama-4-scout-wai-psq · editorial channel only · no per-section breakdown available
Longitudinal · 29 evals
+1 0 −1 HN
Audit Trail 49 entries
2026-03-05 10:07 eval_success PSQ evaluated: g-PSQ=0.280 (3 dims) - -
2026-03-05 10:07 eval Evaluated by llama-4-scout-wai-psq: +0.28 (Mild positive)
2026-03-05 09:58 eval_success PSQ evaluated: g-PSQ=0.320 (3 dims) - -
2026-03-05 09:58 eval Evaluated by llama-3.3-70b-wai-psq: +0.32 (Moderate positive)
2026-03-02 17:43 eval_success Lite evaluated: Neutral (0.00) - -
2026-03-02 17:43 eval Evaluated by llama-3.3-70b-wai: 0.00 (Neutral) 0.00
reasoning
tech tutorial no rights stance
2026-03-02 16:36 eval_success Lite evaluated: Neutral (0.00) - -
2026-03-02 16:36 eval Evaluated by llama-4-scout-wai: 0.00 (Neutral) 0.00
reasoning
ED, neutral tech tutorial
2026-03-02 16:25 eval_success Lite evaluated: Neutral (0.00) - -
2026-03-02 16:25 eval Evaluated by llama-3.3-70b-wai: 0.00 (Neutral) 0.00
reasoning
tech tutorial no rights stance
2026-03-01 17:29 eval_success Lite evaluated: Neutral (0.00) - -
2026-03-01 17:29 eval Evaluated by llama-4-scout-wai: 0.00 (Neutral) 0.00
reasoning
ED, neutral tech tutorial
2026-03-01 17:25 eval_success Lite evaluated: Neutral (0.00) - -
2026-03-01 17:25 eval Evaluated by llama-3.3-70b-wai: 0.00 (Neutral) 0.00
reasoning
tech tutorial no rights stance
2026-03-01 17:18 eval_success Lite evaluated: Neutral (0.00) - -
2026-03-01 17:18 eval Evaluated by llama-3.3-70b-wai: 0.00 (Neutral) 0.00
reasoning
tech tutorial no rights stance
2026-03-01 16:08 eval_success Lite evaluated: Neutral (0.00) - -
2026-03-01 16:08 eval Evaluated by llama-4-scout-wai: 0.00 (Neutral) 0.00
reasoning
ED, neutral tech tutorial
2026-03-01 16:02 eval_success Lite evaluated: Neutral (0.00) - -
2026-03-01 16:02 eval Evaluated by llama-4-scout-wai: 0.00 (Neutral) 0.00
reasoning
ED, neutral tech tutorial
2026-03-01 16:01 eval_success Lite evaluated: Neutral (0.00) - -
2026-03-01 16:01 eval Evaluated by llama-3.3-70b-wai: 0.00 (Neutral) 0.00
reasoning
tech tutorial no rights stance
2026-03-01 15:56 eval_success Lite evaluated: Neutral (0.00) - -
2026-03-01 15:56 eval Evaluated by llama-3.3-70b-wai: 0.00 (Neutral) 0.00
reasoning
tech tutorial no rights stance
2026-03-01 14:57 eval_success Lite evaluated: Neutral (0.00) - -
2026-03-01 14:57 eval Evaluated by llama-4-scout-wai: 0.00 (Neutral) 0.00
reasoning
ED, neutral tech tutorial
2026-03-01 14:51 eval_success Lite evaluated: Neutral (0.00) - -
2026-03-01 14:51 eval Evaluated by llama-3.3-70b-wai: 0.00 (Neutral) 0.00
reasoning
tech tutorial no rights stance
2026-03-01 14:15 eval_success Lite evaluated: Neutral (0.00) - -
2026-03-01 14:15 eval Evaluated by llama-4-scout-wai: 0.00 (Neutral) 0.00
reasoning
ED, neutral tech tutorial
2026-03-01 14:05 eval_success Lite evaluated: Neutral (0.00) - -
2026-03-01 14:05 eval Evaluated by llama-3.3-70b-wai: 0.00 (Neutral) 0.00
reasoning
tech tutorial no rights stance
2026-03-01 13:25 eval_success Lite evaluated: Neutral (0.00) - -
2026-03-01 13:25 eval Evaluated by llama-4-scout-wai: 0.00 (Neutral) 0.00
reasoning
ED, neutral tech tutorial
2026-03-01 13:20 eval_success Lite evaluated: Neutral (0.00) - -
2026-03-01 13:20 eval Evaluated by llama-4-scout-wai: 0.00 (Neutral) 0.00
reasoning
ED, neutral tech tutorial
2026-03-01 13:18 eval_success Lite evaluated: Neutral (0.00) - -
2026-03-01 13:18 eval Evaluated by llama-3.3-70b-wai: 0.00 (Neutral) 0.00
reasoning
tech tutorial no rights stance
2026-03-01 12:41 eval_success Lite evaluated: Neutral (0.00) - -
2026-03-01 12:41 eval Evaluated by llama-4-scout-wai: 0.00 (Neutral) 0.00
reasoning
ED, neutral tech tutorial
2026-03-01 12:40 eval Evaluated by llama-3.3-70b-wai: 0.00 (Neutral) 0.00
reasoning
tech tutorial no rights stance
2026-03-01 12:04 eval Evaluated by llama-4-scout-wai: 0.00 (Neutral) 0.00
reasoning
ED, neutral tech tutorial
2026-03-01 12:04 eval Evaluated by llama-3.3-70b-wai: 0.00 (Neutral) 0.00
reasoning
tech tutorial no rights stance
2026-02-28 16:22 eval Evaluated by llama-4-scout-wai: 0.00 (Neutral) 0.00
reasoning
ED, neutral tech tutorial
2026-02-28 16:22 eval Evaluated by llama-3.3-70b-wai: 0.00 (Neutral) 0.00
reasoning
tech tutorial no rights stance
2026-02-28 14:17 eval Evaluated by llama-4-scout-wai: 0.00 (Neutral) 0.00
reasoning
ED, neutral tech tutorial
2026-02-28 14:17 eval Evaluated by llama-3.3-70b-wai: 0.00 (Neutral) 0.00
reasoning
tech tutorial no rights stance
2026-02-28 07:04 eval Evaluated by llama-4-scout-wai: 0.00 (Neutral)
reasoning
ED, neutral tech tutorial
2026-02-28 06:52 eval Evaluated by llama-3.3-70b-wai: 0.00 (Neutral)
reasoning
tech tutorial no rights stance