What is Reddit's opinion of import.io?

From 3.5 billion Reddit comments

You can either start with Python, learn some programming and eventually start building scrapers (perhaps with the help of Scrapy).

Here is the sample exercise on scraping: Social Web Scraper at CodeAbbey

On the other hand for some tasks you can use online solutions, like this: https://import.io/

Probably you even may find it preferable.

Koliko tesko bi bilo napraviti neku vrstu analitickog engina.

Paresat njuskalo i index, jednom dvaput dnevno.

Za nekretnine bi ubacio i stranice agencija.

Vec sam to radio s import.io ali su mi ukinuli account jer je bilo samo 10000 req free, a ja sam ih imao po 50m mjesecno.

Probao sam sa octoparse i nisam dobivao dobre rezultate.

Prestao sam se bavit time jer mi je ideja postala dosadna, ali ak netko hoce, nek mi slobodno ukrade ideju.

Ja sam to sve radio s gore dostupnim alatima i excelom jer sam mazohist.

Featuri koji bi mi bili interesantni

• brza porcjena automobila na temelju godista,kilometraze i trenutnih cijena na oglasniku
• povijest cijena određenog modela automobila
• Zone nekretnina (razvrstano po krvartovima) i trend rasta/pada cijena po godinama.
• Usput i ugruba procjena nekretnine na temelju stanja (novogradnja/novija gradnja/starogradnja) lokacije i kvadrature.

f1db.de worked beautifully thanks; I got exactly what I needed. Annoyingly, I had seen f1db.de but hadn't found the entry list pages.

As a reminder for myself (and perhaps a help for others) what I did was:

• Created an Importio crawler to parse the web pages into just the table.
• The Importio crawler itself would not collect any data but they also turn the crawler into an API that you can use to return the parsed data.
• Created a list of URLs to be scraped (with Excel). The URLs are for the Importio API rather than the target site. I did not worry about the inevitable 404s (i.e. I did all seasons assuming 20 races per season).
• Used wget to pull the pages to a file.
• Used an online converter to change the HTML to Excel format (Excel, Chrome and IE couldn't open the file properly).
• Tidy up.

Thanks again for the tip...

As others have wisely noted, if you're going to roll your own implementation Python is probably the easiest choice. Scrapy is very powerful but has a bit of a learning curve. In the past I've just used urllib2 (plus TOR) to download massive tranches of sites, then regexes to extract the data. You may also want to load the HTML into BeautifulSoup for easy DOM traversal.

That all said, my new favorite solution for quick scraping is import.io. This might be the simplest option if you're short on time, but of course it's nowhere near as fun as writing your own tool.

That's a tall order :-(

Do you have filesystem access to the server this site is hosted on? Without something like that (or a developer at your side), this task is going to be quite difficult.

Take a look at import.io and see if that gets you any closer to the goal.

Since you didn't ask about scraping from within an app... Heres some software you can use to scrape it from your computer. https://import.io

By the way, consuming APIs, and scraping websites are completely different things.

See /u/asoruli answer for how to do what you're after with the url you posted.

Like most things, the answer is: it depends.

If the web service has an API of some sort (google the name of the data source your'e looking at + "API"), retrieving data from them will likely be easy or very easy. Otherwise, the range of difficulty varies a lot.

Before you embark on something like this, understand that in most cases, scraping web content for this sort of purpose (taking scores from ESPN and displaying them on your site) is a VERY murky legal area that often errs on the side of being illegal.

It also isn't as technically-straightforward as many might think these days for most sites that dynamically generate content (i.e. you can't simply curl the HTML for the stuff you need because it won't be there). While you can use something like Selenium to scrape data straight from the browser and use something like TOR to anonymize/randomize your activity, this is still stuff that will likely net you a shiny cease-and-desist from your data sources.

Additionally, there are services that remove a lot of the heavy lifting (https://import.io is one I can come up with offhand), but you (or your developer) might find that the datasets that they're providing you might not be a great fit for what you're trying to accomplish.

In any case, I highly recommend learning how to do this stuff yourself, since you'll have much more control over the direction of your concept and, if anything, will save money in the long run. Good luck!

import.io is a "Free Web Scraper and structured Data Collection Tool". It is incredibly easy to use and quite powerfull. It's a service that will cost you hundreds of dollars on other websites.

Examples of ways to use a web scraper:

• Collect prices of specific articles from an online shop
• Collect stock prices & economic news
• Collect and filter real estate ad's from craigslist
• Collect E-Mail adresses and other kinds of personal information

A free API may be hard to come by, as that's data that's worth serious money. Import.io may be of some help with scraping the content off an existing site.

A quick google search pointed me to Skyscanner. A polite email might get you access to their API.

Because we have always focused on creating an efficient and scalable infrastructure, we have been able to give you better data quality in larger datasets. Now we have users building data sets with almost 10,000 datasources. A lot free users are also doing hundreds of thousand of queries per day to multiple data sources, so we should be able to do what you need.

If you any tips/help us a email (also free) and checkout the site https://import.io/help

Data pulled together with Import.io and Python. Visualization in Tableau and matplotlib. More information and methodology available to read here https://www.import.io/post/using-web-data-to-see-how-the-uk-answered-marcus-rashfords-call-to-end-child-food-poverty/

Visualization created with Python. Source data created using Import.io. Data is available to download along with more information about methodology here https://www.import.io/post/trump-vs-biden-web-scraping-the-news-to-understand-media-coverage-and-sentiment/

I think import.io should be in this list as well. Very nice UI. They are marketing towards larger clients it seems, but there are still freemium offers available. I am biased because I have used the tool for years, but it is gold standard IMO.

I use Tweepy on a regular bases but I use it for scraping purposes (mainly with Import.io).

That said, it may be easier working directly with the Twitter API unless Tweepy has some specific functionality that would take too long to build. Here is a tutorial on how to build a scraper/analyzer with the Twitter API. That may help you with the first part (what you were going to use Tweepy for). I have never built a bot so I can't help you with the interaction portion of the program.

Hope that helps!

Easy, use https://import.io/ (can get away with free level) export your data as csv use https://wordpress.org/plugins/really-simple-csv-importer/ to import the csv, relax, enjoy your weekend and bask in the glory of how easy it all was. :)

I haven't seen an export option. If we can get the data out we can re-use a lot of the framework I've already laid out to get to similar insights. I've messed around briefly with import.io in the past. That tool might be able to help with getting it off the browser screen to an excel sheet faster than copy-paste + cleanup.

Haven't got the actual dataset, but it's pretty easy to make one.

• Select a list from here: http://skyscrapercenter.com/buildings
• Then use https://import.io/ to scrape the table.
• Download CSV of the data from Import.io.

It's not exactly the same thing, but check out Import.io. I've used it for data scraping & lead gen in the past, and it's robust but it depends on your project needs. You might have to play with it a bit, but it's worth looking into.

I found the best solution to this problem is to set up a web scraping service to monitor the sites you're watching and send you email or text alerts when new properties are posted. You don't need to write any code or pay any money for this, services like kimono labs and import.io will enable you to set up a web scraper in a couple minutes with no code. I used kimonolabs to successfully monitor new homepath listings and to monitor specific properties for status changes (i.e. falls out of contract, leaves first look, etc)

Another option is to use Import.io (which is much easier to learn) and then run all the permutations in Google Sheets or Excel. I'm not sure what your end goal is, but if you just need to complete this task, it may be more efficient to use a premade tool that is specific to web scraping.

Just my $0.02

If your real goal is to make a fantasy football algorithm, you can just use a tool like https://import.io/ to get the data - then do the actual algorithm in another language (including excel - formulas are a functional language!)

If this is a learning exercise, then go ahead and implement a web scraper of you own. I just wanted to check that this wasn't an XY problem and you weren't reinventing the wheel.

Hm, interesting. I don't think I have the tools to do this myself, but I can see how a program like this could be very useful. It looks like maybe import.io could do this based on a quick look at webscrapers. I haven't used these before, curious to hear your experience if you have.

Edit: Wow - I am super impressed with how easy import.io is to use. I can just go to the page of 5 star reviews and pull out every review and the user names of everyone who wrote them.

Edit2: Seems like you can only use import.io once - if you refresh it Amazon just gives you a captcha to make sure you're not a robot.

Woah man, you inputed everything manually? Nice work!

Anyways, in that case you might be interested in this: https://import.io/

Its a very nice automated tool that makes data extraction very easy.

You'll probably have to write a web scraper - or get someone to write one for you. Alternatively, you can learn how to use this site: https://import.io/. Their desktop app seems pretty good and they have a lot of tutorials and webinars.

The big stats sites like WhoScored, Squawka etc have the stats you want.

FYI this is the same value proposition as https://import.io/ (yes they do APIs as well)

But cool idea. I've always wanted something that can effectively capture Kickstarter data for my site CrowdLoot.com

Obviously the site's not done.. I load your demo and I can select fields but that's about it. There doesn't seem to be any ability to dig deeper (ie, follow links) or to do paging. Your demo also needs at least one sentence of text so I know wtf I'm meant to be doing :)

I'll join your mailing list though because I'd love to be in ground floor on this. Keep it up.

Keep in mind I literally have no clue... but have you looked into Kimono Labs or Import.io? Might be an efficient way to scrape the data from match stats page or something.

https://www.kimonolabs.com/ https://import.io/

Honestly, not sure what quantities you need, but would recommend import.io (https://import.io/) instead. Excel is okay for data tables.

If you must use Excel, go to Data -> under External Data choose From Web -> Navigate to Amazon and choose which tables you want to pull in.

I built a tool that does something similar: http://seo-analyser.import.io/ - Just stick all the URLs in there, and download as a CSV. You will get all the outbound links, title tags, headings etc...

If you wanted more specific data from a larger number of sites (and want it up to date); you should use https://import.io, for free, to collect all the data you want, and put it into a CSV or a Live Google sheet. It's stupidly easy and takes about <5 mins to set up a full crawl per site. You could easily do 50 websites, with all teh data you want in a an evening.

Another 'paid for' option is to look a SEO focused web index tools like Majestic SEO. I don't know if they include outlinks and what not, and you wont have any control over the data you get from the site (eg. if you wanted to get the page text you would struggle) but its worth a look.

I work at import and use it quite a bit, so i'd be happy to help. Just drop me a msg.

Good luck :)

If you don't have much experience of writing scripts, check out https://import.io. You just define what data you want to scrape from a website and then it extracts it for you. It's also free.

Give it a whirl on something like http://www.parkers.co.uk/cars/reviews/facts-and-figures/audi/a1/hatchback-2010/

Got it cheers. i'll have a little look into it.

I can share the code / source with you if you wanted to see how its done. It's actually super simple, albeit slightly bastardised, import.io scraper/extractor that uses regex/xPath instead of there visual training method to get common data across the URLs you put in.

https://import.io/data/v471/set/?mode=loadSource&amp;source=7f8a9f9f-f7f4-4ea2-8af7-477c28af3e62 (you need a free account to open it)

After that to make the site I just access the extractor via API and pass the URLs into it from the text box.

Hey Dude, if you still need em' here are 3 datasets full of beer reviews:

https://import.io/data/mine/?id=ba452517-d4d4-49b0-b4ad-233d49cf3888 https://import.io/data/mine/?id=ce11b80a-3621-4f69-94fa-0dfd6d23ebb9 https://import.io/data/mine/?id=ce11b80a-3621-4f69-94fa-0dfd6d23ebb9

You will need to login to get em, but then you can just crawl the sites, and download the CSVs.

I was doing a demo the other day and made this data source, which searches for stock quotes on Y! Finance and gets a load of data back using import.io: https://import.io/data/set/?mode=loadSource&amp;source=b2148884-cf56-41c9-825d-0f3ce49b4877

Note, I was doing a demo with it because I work there. Gimme a shout if you need a hand with it.