OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Beam SQL Pipeline Options


Hi Arun,

If you are looking to connect to a SQL database from Beam Java code, then I you might want JdbcIO [1] with a PostgreSQL JDBC driver [2]. This is not currently wired up to Beam SQL specially; you have to connect in Java code. It would be a nice contribution.

Kenn

[1] https://beam.apache.org/documentation/sdks/javadoc/2.4.0/index.html?org/apache/beam/sdk/io/jdbc/JdbcIO.html
[2] https://jdbc.postgresql.org/

On Wed, Jun 6, 2018 at 9:49 PM arun kumar <sunke.arun@xxxxxxxxx> wrote:
Hi 

Thanks for the update.
Can you please share me if you have any documentation for connecting to postgres using beam code.

Thanks
Arun

On Wed, Jun 6, 2018, 9:54 PM Andrew Pilloud <apilloud@xxxxxxxxxx> wrote:
We are just about to the point of having a working pure SQL workflow for Beam! One of the last things that remains is how to configure Pipeline Options via a SQL shell. I have written up a proposal to use the set statement, for example "SET runner=DataflowRunner". I'm looking for feedback, particularly on what will make for the best user experience. Please take a look and comment: