I have tried online, programs (both free and $$$) and all seem to fail when indexing my website. They basically fail to capture all webpages.... especially those from photo albums (usually created using java).
Does anyone out there have some program etc that works for larger websites (hundreds of thousands of links) without crashing, slowing the computer down. i can run on apache, unix etc.
I am at my wits end with trying to find something that will work.
Something with options like not selecting images or certain folders etc.