Scrapy
extracting the data you need from websites
Screenshots
Hunter's comment
'scraping' websites is pretty a common thing that's out there in the world. i knew companies many years back that did this at a micro level for people that just wanted information FAST that they could populate their CRM's with just to be able to cold call a bunch of execs and 'decision makers' in a business.
not cheap as well, they could run the reports and charge them quite a lot of money to deliver this data -- often the data was collected illegally or at least let's say it was a 'grey area' until laws came into place around it.
i'm sure it could be incredibly useful as well for someone wanting to build some web spiders that actually use this scraping technology in a productive way too, maybe for building exports or collecting together social media content to store away as legacy items.
Link
Contributors
Hunter: @teamhumble
This is posted on Steemhunt - A place where you can dig products and earn STEEM.
View on Steemhunt.com
Please read our posting guidelines. If you have any questions, please join our Discord Group.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
It's been useful actually, but i believe that many sites actually are blocking the accesses from those APIs. You may need to use it with VPN. Also im not sure if react-based websites nowadays would be working well with these types of scraping tools because they generate html tags after the server call (or only in a client page).
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Indeed a grey zone, but the website owner is in charge of the privacy in my opinion. Great product and hunt!
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
No doubt @teamhumble, "Scrapy" is very useful hunt you introduce here.
Steemhunt is great social media platform where we enjoy daily wonderful products, applications and other software.
Scrapy is very helpful scraping technology due to which we easily extract data we need from websites. Thanks a lot for always sharing useful hunts. stay blessed and keep sharing.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
hey, stop making silly comments.
so you just have your own f***ing comment format in SH like,
and just copied and pasted from the hunting post like this,
and again the format
Seriously, shame on you f***ing penny pickers.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
With some sites really loaded with information and ads, Scrapy can help a user to select valuable pieces of data. I will try it out soon.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
We all use data every day. Extracting it from website is really cool. I think I like this hunt.
It's really good hunt
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Great app written in Python and running on all systems to extract data from websites easily and quickly. Thanks for shating it @teamhumble, very useful.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
A great website scraping tool. Definately a tool to bookmark when you are looking for extracting data from different websites. Thanks for sharing.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Scrapy is a very good and innovative product through this product we can get data which type of material we want or required very fast. It is useful product and Great hunt.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
You really think this comment is helpful for SH? If you thought this hunt was cool, then you could just say "Cool hunt!". THB already mentioned all the info what you just repeated. You're clearly a penny pickers who constantly collecting f***** pennies from SH's comment voting pool. Shame on you.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
This is quite a good tool for extracting only relevant data from sites or other sources. The most important this is if it can get the delta, so I will need to take a look on this.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
No more writing Python script s to scrape websites. I can now simply use Scrape Application and Cron it to do the job. The best part is that it's open source. Thanks for sharing.
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Congratulations!
We have upvoted your post for your contribution within our community.
Thanks again and look forward to seeing your next hunt!
Want to chat? Join us on:
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit
Hi @teamhumble!
Your post was upvoted by @steem-ua, new Steem dApp, using UserAuthority for algorithmic post curation!
Your UA account score is currently 6.266 which ranks you at #217 across all Steem accounts.
Your rank has improved 1 places in the last three days (old rank 218).
In our last Algorithmic Curation Round, consisting of 265 contributions, your post is ranked at #20.
Evaluation of your UA score:
Feel free to join our @steem-ua Discord server
Downvoting a post can decrease pending rewards and make it less visible. Common reasons:
Submit