Ranter
Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Comments
-
@Arpit05 Yeah it only supports modern and common browsers right now. Such as chrome, firefox and safari
-
@Mitch377 thank you for your input. I am aware that the terms of use is not very good. It actually stopped me from going live with it for months because I might not like the consicvenses, but this weekend I though "what the heck, lets put it live"
-
@ianertson that's good that it follows robots.txt
I still hate the concept of scraping, but I'm biased because I worked at a place got a while that ran a bunch of "news" WordPress sites that just posted scraped content nonstop.
They had to call Google once a week to get sites back into search results. -
@NathanDoesDev I did also work for a company before, all they did was creating scrapers. That's actually where I got the idea from.
-
@ianertson that's my point. It's a terrible practice which is unethical (mass-plagerism and providing no valuable content to your users) and not to mention it's really bad for your site's SEO.
-
@NathanDoesDev I don't see how it is unethical though. It's just gathering data that is accesible to anyone anyways.
-
@ianertson taking somebody's content and posting it on your own site derives the actual author of traffic that should be theirs. It didn't provide unique content to users. It's plagerism if you don't attribute the content to the author. And once again, search engines like Google will penalize you for stealing content, and if you do it a lot then they will ban your site from the search engine.
Don't do it.
Related Rants
-
willbeddow23A client wanted me to make a website that compared the users face to that of a wrestler. We had done a lot of ...
-
jhole893My client is trying to force me to sign an ethics agreement that would allow them to sue me if found in breach...
-
Garbott7When your reworking a bot because they've realised your scraping their site and you spot this; GAME ON MF'ers
So I just launched a website where you can create web-scrapers with just the click of a button:
http://scrape.host
undefined
crawlers
scraping