Re: Pages with Specific URLS.

because?
you mean urls which contain a query part?

they can be crawled.
the default nutch configuration excludes them by this filter rule in
conf/crawl-urlfilter.txt

# skip URLs containing certain characters as probable queries, etc.
-[?*!@=]


Zaihan schrieb:
> Hi All,
>
> I'm sure I've read somewhere before that URLs that is made like
> http://www.site.com/categories.asp?cid=25&page=9 
>
> Can't be crawled. Is that true?
>
> Warmest Regards,
> Zaihan
>
>
>
>
>