What are /r/scrapy's favorite Products & Services?

From 3.5 billion Reddit comments

https://www.amazon.com/Learning-Scrapy-Dimitrios-Kouzis-Loukas/dp/1784399787

It’s one of the few resources out there that helps you understand that Scrapy isn’t about scraping. It’s about organizing a scraping project. And it shows you how to leverage the guts from the ground up. All the blog posts and whatnot you refer to teach you how to do something you might as well have done with requests and beautiful soup. The real power is in the things not directly related to requesting urls and parsing pages.

Sadly, it’s old and in Python 2. The examples don’t translate to modern executable code. But don’t let it deter you.

Direct answer is Yes Not knowing what issues you are facing you might want to check this article for scaling https://www.scrapehero.com/scalable-do-it-yourself-scraping-how-to-build-and-run-scrapers-on-a-large-scale/

Or google scalable scraping

I think yielding items from spider_idle signal could be suported, but this is not implemented. There are also ideas like https://github.com/scrapy/scrapy/issues/1395#issuecomment-203135634.

It is possible to get items directly (e.g. by storing them in a global variable). It is also possible to get items as-they-scraped using async API, it requires writing code in async way though (see https://stackoverflow.com/questions/40715369/how-to-save-the-data-from-a-scrapy-crawler-into-a-variable/40715544#40715544). An on-disk file has an advantage - it acts as a "checkpoint" and allows to debug scraping and processing separately.

To add on to /u/icypalms correct answer, PyCharm community edition is both free and open source, and would have stopped those mistakes from even leaving your editor (you wouldn't have needed to run them to find out).

Separately, JetBrains also has a neat Python Educational edition to walk you through some python coding exercises but inside the editor, which can be beneficial in both ways (language and editor).

The short version is yes, since it's a website; the medium version is to scrape all of a dictionary you'd really need an enumeration mechanism, whether via chasing all words in a thesaurus or if they offer a sitemap.xml (I checked, and their robots.txt doesn't declare one, sorry)

Pragmatically, you may be happier going after their Android app since there's a non-zero chance they ship the whole dictionary as a sqlite database in the app, or otherwise download one from somewhere

Thanks, i changed it from form to input and it prints out

['1', 'https://www.chess.com/home', 'nyFHR_bRC3VB72oWHKyt9Wckk90UspZDPHEDxXJYoO4']

I only want the third value, Is there a way to append an integer to the css selector to grab the specific value? Thanks again

Edit: nvrm i found it lol

Thank you! That was close to the initial concept actually, but I've figured it'll be excessive and not very flexible since:

• Target page parsing within a spider can be written in too many ways: using different preprocessors for each field, using simple dicts or Item class, using loaders or not using loaders.
• Crawling itself and following links is not covered here and it also differs in many ways from website to website (different spider.parse_* emitting different Request()'s and so on)

If I ever get to work on actual crawling capabilities I may give it more consideration. But that is a large task (mostly the visual editing part of all of it). Right now it would only be possible to generate a cookie cutter code like this, which is too simple a case and can be filled by hand with little trouble.

I use Scraperapi. They are cheap. They had some issues but its ok now. Also they have autoparse so you can get the results directly

Here is detailed explanation to create a spider.

https://dev.to/iankerins/scraping-millions-of-google-serps-the-easy-way-python-scrapy-spider-4hpc

I tried with COOKIES_ENABLED = False but I still get the same the same result:

2020-04-02 20:13:33 [scrapy.core.engine] DEBUG: Crawled (307) <GET https://search.yahoo.com/search?p=ip%3A23.227.38.64&gt; (referer: None)

2020-04-02 20:13:33 [scrapy.core.engine] INFO: Closing spider (finished)

MongoDB Atlas is awesome! I dunno how big your scrapes are but they have a free 512mb. They also have charts that I use free as well for quick data viz. The paid plans are not bad at all if you need to expand, but I've been on free for a while, and will need to transfer to paid in a year worth of scrapes if I don't offload old data. Just check it out https://www.mongodb.com/cloud/atlas no credit card on file or any of that funny business.

Which a real python editor would have immediately caught. Hell, even using PyDev (since you already have Eclipse installed) would be a significant improvement over whatever horrific editor you're using now.

> error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/ > > ---------------------------------------- > > ERROR: Failed building wheel for brotlipy

This is the actual error I wanted you to find.

Frankly, most of the content you find is pointed at beginners. Introduction to web scraping. The fact is, it turns out, web scraping is the easy part of a scraping project. Organizing a larger project, with multiple sources in the same domain however is extremely hard. There's an awful lot you need to consider. Project structure, logging, error handling, downloader throttling, item management, cleaning, exporting, etc. The content you are finding serves as an introduction to scraping, and these complexities distract from that goal. And frankly, it's much easier to produce that content.

In the end, if you're just doing a single-spurce throwaway project, maybe scrapy is overkill. But if you need a project that can be maintained over time, Scrapy adds an incredible amount of value that isn't obvious until you have experience maintaining such things.

For context, I've been doing retail price scraping for ten years. I started with a proprietary GUI tool, eventually learned Ruby to build my own frameworks, and had to abandon it all when I read the book "Learning Scrapy". As much as I prefer Ruby over Python for this kind of work, Scrapy has everything that was hard built-in. I learned Python and Scrapy in about a month and started writing our new scrapes on Scrapy. Today about 100 of the 200 scrapes I manage are on Scrapy, and it's my preferred starting point for any new data source, or project that might turn into something.

I still don't much care for Python, but I can't pass up Scrapy. It's way too powerful to pass up.

Also, don't let the above fool you into thinking it's only useful if you're doing a massive project. It's also fantastic for single source, simple scrapes too because it gives your project structure and item management.

https://www.amazon.com/Learning-Scrapy-Dimitrios-Kouzis-Loukas/dp/1784399787