GrowTech CentralSeven ways to cut scraping bandwidth
Runaway GB bills usually come from small things that are easy to fix.
Almost every customer who says "my quota vanished" turns out to be downloading things they never needed. Here are the seven fixes that pay off fastest.
1. Turn off images
This is the big one. On a marketplace product page, images are often the largest part of the total — on many product pages, more than half of it. If you only need the price and the product name, switch them off and your quota lasts five times longer.
2. Turn off fonts and video
Custom fonts are hundreds of kilobytes per page and you need none of them. Auto-playing video is worse still.
3. Skip the browser when you can
A full browser downloads everything the page asks for. If the data is already in the initial HTML, a plain HTTP request is far cheaper.
4. Use their API if one exists
Many sites have an internal data endpoint their own pages call. It returns raw data, at a tenth of the size, and changes shape less often.
5. Do not re-fetch unchanged pages
Keep the timestamp of your last fetch. If the site says nothing changed, you never need to download the body again.
6. Shrink the fake screen size
Some sites send high-resolution images for large screens. Set your browser window to phone size and the images shrink dramatically.
7. Check for a leak
If quota drains while your job is not even running, some other tool is usually still pointed at the same proxy. The usage page shows which hours the data moved.