downloaders, players and searches, oh my!

[deepsepia]

Originally Posted by pelham456

before i go to that other thread, what do u mean "leech terabytes of pr0n"? what kind of sites are u scraping with it??

can't be the offerings HERE, b/c they all come off hosts with staged release and human verification tests. i don't see how JD is helpful on rapidgator, say, until the final page once uv gotten to a real link. so what can it do with MULTIPLE such offerings, e.g. "jenna jameson thread"??

You can scrape terabytes of links from JD if there are, for example, links to K2S or some other host. JD's "Deep link analyze" looks at at least some of the information on a forum page that it can crawl through-- works in some cases, not in others. Its not an issue of the filehost itself, usually -- its about how the forum exposes (or doesn't) the links.

I can find you Terabytes of RG links-- you'll probably want a premium account to download them as a practical matter -- but they work just fine.

There are forums that reject these kinds of techniques, needs a more sophisticated WGET approach-- but you'll find many that do permit this behavior . . . just have to experiment to find one which works for you. Many of these filehosts are more convenient with a premium account -- there are automated and crowd-sourced captcha solvers, but typically works better without. But even with a free account on some of the hosters, you can download a lot . . . and if you have a VPN, just pick a new IP, reboot, and you've got a new free quota (again, works for some, not others).

I posted a tutorial on another site a long time ago, but it still works pretty much the same

Originally Posted by Quick and Dirty webscraping with JDownloader

Have you ever wanted to get all the links from a thread on a board? Let's say you want to get all the links from a 200 page thread . . . how are you going to do it?

You could go to each page on and right click on the link . . . you'll get the links, and carpal tunnel syndrome as well. I've previously pointed out that you can grab all the links from a single page, with JD, by doing a Ctrl-A & Ctrl-C [Select all, Copy All] on the page, because JD will parse links from stuff you put on the clipboard. But there are some very big threads out there, hundreds of pages . . . that could still take some time.

There's gotta be an easier way to do this, you'd think-- and you'd be right.

If you write PHP or Python, there are a lot of ways to do it, but let's say you don't write code.

Jdownloader is your friend. You can take the address of any page from a thread and paste it into JD's Linkgrabber's "Analyze and add links" feature (the "plus" [+] button on the lower left corner of Linkgrabber), and JD will analyze the page and look for download links.

Ah, but what if you have a thread with a hundred pages? Suppose you have a thread:
[someforum]/Girlswithbigbooks-[Pagenumber].html -- where the pages are identified 1 through 100?

Simple. I used a little basic interpreter, but you can use whatever you know-- a spreadsheet would be fine-- to generate a text file that reads

[someforum]/Girlswithbigbooks-p1.html
[someforum]/Girlswithbigbooks-p2.html
[someforum]/Girlswithbigbooks-p3.html

and so on up to 100, or whatever number of pages there are. Note that there are number of different ways that different forums deal with page numbers, you just have to copy the format used in the thread that you're interested in.

Now copy it and paste into JD's "Add New Links" utility. JD will most likely pop up a screen saying that it couldn't find any links, and asks if you want it to perform "Deep Link Analysis" -- basically looking at each item on the page and seeing if it links through to a downloadable file. Click "Yes" to this.

JD will iterate through the pages, looking for all links & images (you can restrict JD to looking for links, to ignore images, and so on). Some sites hide links without thanks, or have other tricks . . . not everything will work at first, but with a little digging you can usually figure out a way. And some forums really, really, really don't want you doing this . . . so you don't and you find some site that permits it, no reason to break anyone's rules.

Its a bit slow -- for a massive thread its the sort of job to leave JD humming on overnight.

But it works very well, and if utilities like "WGET" leave you saying "huh?", then this approach -- which experienced web coders will tell you is amateur kludge-- is going to be the easiest.

[pelham456]

wow, this is all surreal to me. i can't even quite grasp how it can get the DL link from any of these sites where one doesn't even APPEAR until a final stage, but u clearly are well beyond that. and the idea that it can push thru CAPTCHAS?! several threads here indicate we humans cannot even manage that half the time!

i will certainly give all this a try. i don't have premium anything, but if, as u say, some of them work even without it, then i'll start with that. my wifi can only manage a few G per hour anyway, so it's not like site timeouts etc are a huge factor -- i DLed the entire marie luv thread once; i believe it took 8 months.

anyway, many many many thanks! if this all bears out, it will be the greatest thing in pr0n since mega packs!! (t0rr3nts of like 200 vids of same gal)

[deepsepia]

Originally Posted by pelham456

wow, this is all surreal to me. i can't even quite grasp how it can get the DL link from any of these sites where one doesn't even APPEAR until a final stage, but u clearly are well beyond that. and the idea that it can push thru CAPTCHAS?! several threads here indicate we humans cannot even manage that half the time!

i will certainly give all this a try. i don't have premium anything, but if, as u say, some of them work even without it, then i'll start with that. my wifi can only manage a few G per hour anyway, so it's not like site timeouts etc are a huge factor -- i DLed the entire marie luv thread once; i believe it took 8 months.

anyway, many many many thanks! if this all bears out, it will be the greatest thing in pr0n since mega packs!! (t0rr3nts of like 200 vids of same gal)

The essence of it is that there are two place that may stall you:

1) forum doesn't permit it. A forum like this one will permit you to capture links via a manual Ctrl-A, Ctrl-C on a loaded page, but won't automate ripping through other pages. That's their choice, and no reason to mess with it.

-- but here on PS, you can find individual pages with huge numbers of links, take for example
http://veterperementour.ru/t891623-p2...-hd-video.html

-- Just capturing the links from that page-- not the entire thread-- gives you



. . . and capturing an entire thread from a different place -- which does permit you to rip through many pages, rather than one by one, gets you to Terabytes very quickly.



Realistically, with most of the filehosts, you're going to need a Premium subscription, or you'll need to find one of the liberal sites, and then cycle through IPs on a VPN (basically they give you so many free gigs per 24 hours, but you can essentially get a new quota with a new IP).

Many more aspects to what JD can do . . .

. . . and at a higher level of sophistication, there's WGET, which gets you into web programming. WGET and some of the other scraping applications can get you pages which won't load otherwise-- they'll effectively fake a manual page load, get the token permitting access, and then rip it. This is much more complex than JD, and for me goes over the line. If a forum owner doesn't want something to be accessible without monkeying with it. . . I don't monkey with it.

So for me, JD is the "sweet spot" of "you can grab so much stuff, without being a jerk about it".

2) the second place that you can be tripped up is a captcha in a free version of a filehost-- a lot of them have them, but some don't. There are automated captcha solvers, which work . . . sometimes. See discussion among the JD cognoscenti for ideas on how to make that work.

Me, I find it worth it to pony up every so often for a month of premium-- given that JD can queue up those terabytes and download them 24/7, you can leach yourself a staggering amount -- and at much higher speeds than any torrent.

A VPN is worthwhile here, for all sorts of reasons.

NB Another nice feature of JD is that they detect mirrors, identical files. You can load up a backlog of thousands of files, many of them dupes, but from different filehosts (you'll have noticed that many folks upload the same file to many different hosts). With a big range of filehosts, JD just crunches its way through its backlog, looking for a file which it can dowload -- eg which has bandwidth. Once its done that file, it marks all the duplicates as mirror files, can delete them . . . that way you get the best opportunity to get the most free bandwidth, and also the most chances for a captcha solution to work.

[pelham456]

i have no trouble grasping that JD can go thru a forum and snag every link on a page or entire thread -- u keep repeating that like i do -- but THOSE links bear no resemblance whatsoever to the DL links which each of the hosts (eventually) produces! THAT is the part which is baffling me.

but i trust u that it DOES...somehow....

will play with it and report back.

tx again!

[deepsepia]

Originally Posted by pelham456

i have no trouble grasping that JD can go thru a forum and snag every link on a page or entire thread -- u keep repeating that like i do -- but THOSE links bear no resemblance whatsoever to the DL links which each of the hosts (eventually) produces! THAT is the part which is baffling me.

Pulls in Filehost links, the deep link analyse function is essentially a very limited web spider-- it looks for links on pages, so you get results that look like



You could do something similar with a true webspider or crawler, something like WebHarvy . . . which I've looked at but really seldom have a need for anything more complex than JD

What you're missing is that these applications can, if properly configured, find the links behind the pages . . .