What are /r/webscraping's favorite Products & Services?

From 3.5 billion Reddit comments

You could use this scraper here - https://www.scrapehero.com/how-to-scrape-historical-search-data-from-twitter/

It’s a scraper you can import to the web scraper chrome extension and run it. It doesn’t get blocked or banned for hundreds of pages - you just need to the computer up while it’s running.

Wax lets you run Python right from Google Sheets, nothing to install or download to your machine, just add it from the marketplace.

I replied in other thread but do not under any circumstances login to your instagram account and then try to scrape it. You will get banned. Instead, either scrape from public pages by using a residential proxy IP address like Scrapehero insta scraper or Specrom instagram scraper is doing, or create some throwaway insta accounts, login using that and try to scrape at a frequency of 25-100 pages/day.

I remember I was introduced to web scraping through a Udacity course. I think it was this:

https://www.udacity.com/course/data-wrangling-with-mongodb--ud032

I find the book “Web scraping with Python” by Mitchell to be also a pretty nice introduction to the subject.

Definitely possible. For more basic project you could use tampermonkey to run the code on the pages you visit. Nodejs with puppeteer would work better for a bigger project.

The harder part is writing the code. From my experience with Amazon, they are very inconsistent with specs. For example this page: https://www.amazon.com/dp/B07XWGWPH5 - "Memory Storage Capacity 4GB", they are talking about RAM, not storage space. Yet your Apple phone URL has "Memory Storage Capacity 128GB", where they mean storage space. Might not be a big deal and would still make the job easier, but some kind of supervision would be necessary.

It means that you need to inform where you have placed your web driver executable to the constructor.

You basically have two options:

1- Install the driver and update the PATH variable. If you do this, you do not have to specify executable_path

2- Install the driver and take a note of the full path, including the driver executable and send it to executable_path.

Web driver is a separate install from your browser and it is browser dependant. You can download it from this link.

If you are on mac, you can use home brew:

brew install geckodriver brew install chromedriver

If you are on Windows, you can use Chocolatey instead of homebrew.

These package managers will take care of keeping the driver compatible with your browser versions and update the path as well. Saves a lot of headaches.

I've tried it through postman and got the same error as yours even having all the same HTTP fields.

It seems that postman is not being detected in the application layer, but rather on the session layer since it does produce a very distinct fingerprint during the TLS handshake.

So, in order not to use a browser, you'd maybe need to implement a browser-like TLS handshake. You can use wireshark to see the differences(left = chrome, right=postman).

So that would be my next step If I were you and If you are really determined into trying to bypass that detection without using a headless browser.

Otherwise you can just use a headless browser to make the requests and let chromium produce the right fingerprint when requesting the API.

If you are really worried about how this will impact your computer's performance, you can always block all the requests. Even the mainframe one since all you need is the browser's capacity to make requests. That way you save brandwidth and don't waste your precious CPU power running the website's scripts.

Does it give an error as soon as you try to use the endpoint for the first time? or does it take a couple of tries before blocking you?

I've made a bunch of requests without getting blocked actually both from my browser and from a headless instance.

There doesn't seem to be much about those requests. The first request you make sends back a couple of cookies that your future requests needs to send along with them(if you send the request directly to the endpoint, you'll also receive those cookies).

Maybe that's what you are lacking.

The parameter you mentioned("_=") is just javascript's epoch(in miliseconds).

You can see it in this call.

But I don't believe it will work because there's a call to another API right after sending the event data. It probably is used by some server-side detection.

So, after one API request, it is likely that you'll get the captcha page.

In order not to use a regular browser you'd need to reverse the way they mount those other custom fields and then you'd need to grab your navigator's session cookies and use the same user agent.

That's a lot of work just to avoid CefSharp or pupeteer. It is much easier to just use one of those and then just block media requests in order to save brandwidth I'd say.

You are trying to scrape by only sending one http request. At that point you'll only receive linkedin's loading page. Only after that page loads the javascript kicks in and loads the rest of the page.

You have to reverse the way they use their own APIs in order to be able to get the job listing since they load them dynamically through javascript.

If you look at their requests, you can clearly see that they have a GET request that returns the job listing.

But that request has a bunch of custom http fields so just sending a regular get request won't do you any good.

​

So, my suggestion for you, if you are experienced enough would be to simply hit that "initiator" button on the request in order to see the callstack and try and make sense of where the CLRF token is located at and what informations this API returns.

But reversing those anti-bot counter measures takes time and patience. Way more than just developing the software you have already developed.

If you are not up to the task, then try and adapt your software to use job listings from other companies that return them in a static page. That way you'll be able to maintain the same architecture and still offer the same functionality.

Third option, if you still want to try and scrape linkedin and such, would be to try and use pupeteer+stealth in order to see if the page(linkedin) loads naturally and, after that, you can scrape the page. You just have to check when the page is fully loaded. But, generally, those chromeless instances are detected before the page even loads and you end up having to reverse a bunch of obfuscated code.

Not sure bit as per the documents i read, 'E' refers to day of the week. source

Lets say you just want to open a website and start grabbing the data. I haven't done it but it should be possible by using Javascript and the dev console of a browser.

Google came up with this https://www.freecodecamp.org/news/how-to-use-the-browser-console-to-scrape-and-save-data-in-a-file-with-javascript-b40f4ded87ef/

https://www.bitcoincash.org/privacy-policy.html for instance.

The email is only visible after JavaScript is executed but replaced with /cdn-cgi/l/email-protection#7606041f0017150f36141f0215191f181517051e58190411492503141c1315024b26041f0017150f before JS runtime.

Has someone reverse engineered this with an open source tool?

While this site usually caters to software, it appears to have a pretty good list of alternatives to YP, too: https://alternativeto.net/software/yellow-pages/ -- and then presumably you can chase some of those alternatives transitively to get the whole story

So that wasn't too hard for me to figure out, however, because I don't know if there were really any constants, it might not work on every page. Just find out if the class is the same on each and every page you're working with. Use this as a moment to learn that you need to find the constant and then work from there. I would be interested in seeing what you did to get up to this point before you just take my code and make it work for yourself.

​

import requests, from bs4 import BeautifulSoup

url = 'https://www.flipkart.com/search?q=iphone&otracker=search&otracker1=search&marketplace=FLIPKART&as'

page = requests.get(url)

soup = bs4.BeautifulSoup(page.text, 'lxml')

cls = soup.findAll('div', class_='_2rQ-NK')

for each in cls: print(each.text)

A few things to consider before you get too far into this.

First, these three websites make money on their data. They make it difficult to scrape data with dynamically generated class names for their css, and client side rendered data. Anything client side rendered is much more difficult to scrape. When you send a request through Python to a domain, you are requesting data from that domain’s server. These guys send back a basic HTML template and JavaScript that runs in your browser to create what the user sees. This means your request will probably not give you the data you want or need. They offer API’s... for a price to get their data seamlessly.

Second, these organizations aren’t as “boots on the ground” as you think they are. Typically, almost all weather data is free and open provided by NOAA. https://youtu.be/qMGn9T37eR8 here is a pretty good video to check out on this stuff. These organizations employ meteorologists to generate models and predictions. This would be the cause of any discrepancies in their forecasts.

I know of one free weather api if you are interested. https://darksky.net/dev Weather underground may also have a free API, not sure. Just google “anything API” before you go too far into scraping data. What you want may be provided much easier for free.

As for the basic programming stuff like running the script every hour:

A simple way would just be

from time import time

time.sleep(seconds in an hour)

Keep in mind that a simple solution like this is blocking. That means that your script is using a processor core and limiting your available resources at all times. There are plenty of other solutions for running cron-jobs correctly. Cron-jobs (chronological) is what you’re looking for.

Sorry for formatting, on mobile

Looks fantastic, I'll have to take this for a spin.

I'd be super interested to hear about how you've faced some of the scraping challenges, like circumventing anti-bot tech (e.g. hotels.com) or how you manage target-side deployments that change the markup of the page (and thus, potentially breaking some or all data selectors). Thanks!

Try this for free https://www.scrapehero.com/how-to-rotate-proxies-and-ip-addresses-using-python-3/ and see if works

I am not sure if this count but scrapinghub and diffbot have developed automatic extrction API to parse and extract data from news article and online product urls.

There is no simple way to do this. What I can suggest, for example if you use puppeteer or any manager for chrome debug protocol, you can prohibit certain requests. You can do this by request type or by name. You can intercept requests before they sent to server and prohibit them, it will look for page like network is down for certain requests. You need to find yourself what to block. Like this one: https://github.com/GoogleChrome/puppeteer/blob/master/examples/block-images.js You may need to block only couple and that would block all remained chain

You might check out puppeteer -- It's a nodejs library but pretty simple to pick up. I'm way more proficient at python than js and had no issues banging out what I wanted with some tutorials online. Can't comment on error code though 🤔

I think import.io should be in this list as well. Very nice UI. They are marketing towards larger clients it seems, but there are still freemium offers available. I am biased because I have used the tool for years, but it is gold standard IMO.

thanks for the recommendation. I tried playing around with it and it seems quite easy to use. the only problem i saw is that for similar pages with different ural i have to recreate all the selector from scratch. sample of 2 the pages i was refering to: - https://developers.google.com/speed/pagespeed/insights/?url=https%3A%2F%2Flife.com.by%2Fstore%2Fsmartphones%2Fsamsunggalaxya10-blue&tab=desktop - https://developers.google.com/speed/pagespeed/insights/?url=https%3A%2F%2Fwww.tele2.hr%2Fprivatni-korisnici%2Fmobiteli%2Fsamsung-galaxy-s10-dual-sim-128gb-prism-green%2Fd1195%2F&tab=desktop

Do you want to scrape websites that use old technologies or have specific markup in their code? If yes, then it is quite possible. You can also try BuiltWith.com to get list of sites using a specific tech stack (say, sites using WordPress)

I've never used selenium. Maybe it's just the user agent. Use fiddler to see what's going on the http request that changes when you use a headless browser.

But it is probably the user agent.

The Scrapy tutorial may be a fine place to start: https://docs.scrapy.org/en/1.7/intro/tutorial.html

Some of that discussion depends on exactly how "newbie" you're talking. The Scrapy tutorial assumes familiarity with programming, the python programming language, installing pip packages, how HTTP works, how to use the developer tools in your favorite browser, and so forth. While they don't specifically cover it, if you are going to work with Python you'll want to download the community edition of PyCharm, which is both free and open source, and the hands-down best Python IDE on the planet.

If you aren't already aware, Scrapy is beyond amazing for running scraping jobs, and there's a community here in r/scrapy

I'll take this opportunity to plug PyCharm, too, which is invaluable when working in python. The community edition is free and open source, and is plenty feature-filled for working with Scrapy.

Have you looked at services like this one? https://docparser.com

I have built a parser with code for extracting tables from documents I have but pdf is not a very good format for data processing so it might be tough.

Let me know if you need more help

You need to check the NordVPN ToS to see if scraping is allowed.

Based on previous experience, VPN has fixed number of server IP addresses easily detected by proxy detection services. You should try residential proxy.

As a teacher of mine once said, "it's barely legal". Check out HiQ vs LinkedIn for recent precedent. I'll also have a few sections on this topic in my book on data science coming out soon (https://www.amazon.com/Practical-Data-Science-Python-hands/dp/1801071977/ref=sr\_1\_1?dchild=1&keywords=data+science+nathan+george&qid=1630958650&sr=8-1). The short of it is yes, it's legal to scrape publicly-facing data. If you have to login, you've agreed to their TOS. But be respectful, ideally follow robots.txt and other guidance from sites like TOS.

Here I got it working, by clicking on the coordinates of the picture

const puppeteer = require('puppeteer')


puppeteer.launch({
  headless: false,
}).then(async browser => {
  var [page] = await browser.pages()

  await page.goto('https://www.amazon.com/Toshiba-HDTB410XK3AA-Canvio-Portable-External/dp/B079D359S6/ref=sr_1_4?crid=VSLZBRRN2ZAG&keywords=hard+drive&qid=1565674639&refinements=p_36%3A-10000%2Cp_n_feature_two_browse-bin%3A5446812011&rnid=562234011&s=pc&sprefix=hard%2Caps%2C205&sr=1-4')

  // Get coordinates of picture
  var pos = await page.evaluate(() => {
    var {x,y,width,height} = document.querySelector('#main-image-container').getBoundingClientRect()
    return {x,y,width,height}
  })

  await page.mouse.click(pos.x+pos.width/2, pos.y+pos.height/2)
})