osdir.com


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: SQL Query Set Analyzer


Hi Zheng,

At Qubole, we are building something very similar to what you are looking
for. And from experience, I can tell you that it is a lot easy to build it
than what one may think.
We use Calcite parser to parse the SQL into Sqlnode and then use different
tree visitors to extract query attributes like  tables, filter columns,
joins, subqueries etc.,

Our approach is very similar to Uber's QueryParser project (
https://github.com/uber/queryparser ), but we go deeper in our analysis of
finding queries that are semantically similar to some canonicalized form.
If you intend to begin from scratch, I can give you some pointers to get
started.

Thanks,
Devjyoti


On Thu, Jul 26, 2018 at 9:37 AM, Zheng Shao <zshao9@xxxxxxxxx> wrote:

> Hi,
>
> We are thinking about starting a project to analyze huge number of SQL
> queries (think millions) to identify common patterns:
> * Common sub queries
> * Common filtering conditions (columns) for a table
> * Common join keys for table pairs
>
> Are there any existing projects on that direction using Calcite?  Would
> love to leverage instead of building from scratch.
>
> Zheng
>