SPIDERS, BOTS, HARVESTERS, AND DATA MINING ON THE DOLL FORUM

Ishtaria · Post by **Ishtaria** » Fri Feb 18, 2005 4:00 pm

I am making this announcement because there are some who do not care if their activities harm the ability of our membership to access TDF.

Today, a member of the Doll Forum ignored our request to not use automatic site copying tools and effectively brought our server down to its knees.

9 - No use of automated download programs (bots, web harvesters, etc.). Such programs effect the performance of our server by making too many concurrent requests.

In under one hour they had made over 12,000 requests to the server for image files. It took me another hour or so to extract their entries from our logs to gain a full appreciation of what they had done which took our process load from its normal 2.5 up to 27.9.

While they were "mining" images files at such a rapid rate, no one else was able to access our server with any degree of responsiveness. The server was very slow and sluggish for everyone else.

Anyone interested in seeing what I am talking about can look at the extraction file which I produced:
warning: this file is 3.5 megs in size:
http://www.dollforum.com/68.211.49.239.txt

I know who this was that did this and I have permanently banned them.

I am posting this because I have been finding that more and more people are trying to use image harvesters (bot, spiders, etc.) in an attempt to grab all image files.

At times, this has gotten so bad that we have now started discussing a very expensive solution to solving the problem. Funds which could have gone into improving our service will eventually be going into installing equipment in which to protect it from those greedy few within our membership.

I am making this announcement because starting with this member today, I will permanently ban anyone found to be using automated data mining tools to rapidly download images and other files.

I am sorry to be hard nosed about this, but I have tried every software solution to prevent this kind of selfish behavior and these people work to find a way around them. And it has brought us to a point where we will very likely be spending 3 to 4 thousand dollars on hardward just so some of our members do not abuse this free service.

Ish

Keithallen · Post by **Keithallen** » Fri Feb 18, 2005 4:33 pm

Ishtaria wrote

I am sorry to be hard nosed about this,

No need to be sorry, Ish. The rule is stated clearly, and we don't need selfish people jamming up the forum.

Ishtaria · Post by **Ishtaria** » Fri Feb 18, 2005 7:36 pm

Thanks keithallen!

I hate to go to such lengths, but I am at a loss as to what else can be done.

set · Post by **set** » Fri Feb 18, 2005 8:02 pm

hey, we all understand.

Keep up the good work Ish!

elvis921 · Post by **elvis921** » Sat Feb 19, 2005 4:41 am

Maybe that explain's why I couldn't even gain access to TDF last
Thursday nite / Friday morning about 4 AM EST.
In the short time I have been a member, ( 3.5 month's ) I HAVE NEVER experienced that before; While it is TRUE that TDF is a bit on the slow side, You have to take into consideration how much info there is
available to us all on this site. After a while, You learn to live with the
( slow ) speed. No big deal.....

BTW, Ish, Thank You for Your help / tip's concerning my PM to You.
I never FULLY comprehended Your role here at TDF...My problem was a
pimple on a fly's butt; compared to what YOU have to deal with on a daily
basis....And I will say now, THANK YOU for ALL You do here for all of us...

Elvis921

rentell · Post by **rentell** » Sat Feb 19, 2005 6:20 am

The minority always seem to f***k it up for the majority, selfish as always.
Thanks for saving the day again Ish.

Being none savvy about these matters does this mean the abuser managed to down load the images, if it slowed the server I assume they did?

regards Rentell

Inkling · Post by **Inkling** » Sat Feb 19, 2005 9:46 am

Yes - it's always the minority of A-holes who screw it all up for us all.

After all the hours of effort and multiple attempts, Ishtaria's post is nicer than what mine would have been if our roles were reversed.

Folks have to understand that it's very fine enough to get TDF service without abusing the service. Amin is going to want to keep TDF running, and members who enjoy the service should want TDF running as well.

mytime · Post by **mytime** » Sat Feb 19, 2005 9:54 am

Hi All, and especially Ish

Seems we need a donation compagne, to earn that 4 K soon.
Its not do-able to keep them manually out of this site!
I was under the impression that DR Jazz was going to install a firewall that cuold solve this kinda shit or am I wrong?

Further Ish, I haven't heard about your doll since long... will she come?!?

Mytime

PBShelley · Post by **PBShelley** » Sat Feb 19, 2005 3:36 pm

Bravo Ishtaria

Selfish p.o.s.'s need to be put down and removed because of their stupidity and blatant disregard for others. They'll keep doping it like the little insignificant ticks that they are.

To hell with them and their feelings about the matter, and thanks for removing these worthless self-serving greedy pricks away from our midst. If they can't be a part of the community, then they hare no place here, let alone d/l'ing our pictures that we post for our friends and Members' enjoyment.

You are awesome

Regards,
PBS & Lily

mytime · Post by **mytime** » Sat Feb 19, 2005 3:51 pm

Hi PB,

How Much We Agree With You!

.

Ish You're an Awesome Admin And We Hope You'll Get Your Doll Starr Soon!

Mytime & Helen

Ishtaria · Post by **Ishtaria** » Sat Feb 19, 2005 4:30 pm

Thanks, but I honestly don't feel that I deserve such praise. This is just the nature of providing a service on the internet today. And after the many commercial harvesting programs started becoming available, it because like making guns freely available to anyone who wanted one.

The companies who write such software should be held accountable as they proclaim they can "rip" all of the image files from a site within minutes. These clueless bastards know nothing about servers nor the impact that such large numbers of sustained requests can have on the servers they are raping.

This is one of the reasons why the Member Gallery is setup so that only those who are logged in as members can access it. It is because anonymous image harvesting is impossible (i.e., I can identify who is doing it) and to prevent those under 18 from accessing it (at least they would need to "agree" that they are 18 or over). From that standpoint, kids have no business in TDF anyway, and required registration provides an additional barrier for them.

Anyway, thanks again everyone, but I don't deserve it. I'm just trying to put out the fires and prevent new ones from effecting us.

Ish

jackrabbit · Post by **jackrabbit** » Sat Feb 19, 2005 5:35 pm

I blame the people who write the programs more than those who use them. The latter may just be typical ignorant fucks who have no idea what they are doing to the site. Admittedly, they are not without fault unless they look at every last picture they download.

I don't agree that the software writers are ignorant of the harm they cause. I'll bet they know exactly what they are doing to sites, and don't give a shit because they have a product to sell. Many of these download programs have throttles on them to slow down the access rate, so they obviously recognize that this is an issue. A downloader would be relatively benign this way, not that much different from manual downloads (again, if the person looks at every picture). But the throttle is not on by default, otherwise one could not claim to "rip every picture in minutes". Throttle use is probably buried in the documentation somewhere.

But again, the consumer may not understand all this. In some cases, it may not be as much selfishness as a combination of laziness and ignorance.