Re: Host specific parsing

Koch Martina wrote:
> Hi,
>
> has anyone built a parsing plugin which decides on a per host basis how the 
> content of the document should be parsed?
>
> For example, if the title of a document is in the first <h1>-tag of a page for host1 
> , but the title for a document of host2 is in the third <h2>-tag, the plugin would 
> extract the title differently depending on the host.
>
> In my opinion something like a dispatcher plugin would be needed:
>
> -          Identify host of a document
>
> -          Read and cache instructions on how to get the information for that 
> host (database or config file)
>
> -          Execute host-specific plugin
>
> Do you have any suggestions on how to implement such a scenario efficiently? 
> Has anyone implemented something similiar and can point out possible 
> performance issues or other critical issues to be considered?

Yes, and yes. With the current plugin system you can create a new 
"dispatcher" plugin, and then add other necessary plugins as <import> 
elements. This way they will be accessible from the same classloader, so 
that you can instantiate them directly in your dispatcher plugin.

As for the lookup ... many solutions are possible. DB connections from 
map tasks may be problematic, both because of latency and the cost of 
setting up so many DB connections. OTOH, if you add local caching (using 
JCS or Ehcache) the hit/miss ratio should be decent enough. If the 
mapping of host names to plugins can be expressed by rules then maybe a 
simple rule set would be enough.

--
Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com