osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Scio 0.5.0 released


Hi!

On Fri, Mar 9, 2018 at 1:22 PM Rafal Wojdyla <rav@xxxxxxxxxxx> wrote:
Hi all,

We have just released Scio 0.5.0. This is a major/breaking release - make sure to read the breaking changes section below.

- rav


"In ictu"

Breaking changes

  • BigQueryIO in JobTest#output now requires a type parameter. Explicit .map(T.toTableRow)of test data is no longer needed.
  • Typed AvroIO now accepts case classes instead of Avro records in JobTest. Explicit .map(T.toGenericRecord) of test data is no longer needed. See this change for more.
  • Package com.spotify.scio.extra.transforms is moved from scio-extra to scio-core, under com.spotify.scio.transforms.

See this section for more details.

Features

  • Support reading BigQuery as Avro #964, #992
  • BigQuery client now supports load and export #1060
  • Add TFRecordSpec support for Featran #1002
  • Add AsyncLookupDoFn #1012
This looks like a very useful general-purpose tool, Beam users have asked for something like this many times; and the implementation looks very high-quality as well. Any interest in backporting it into Beam?
 
  • Bump sparkey to 2.2.1, protobuf-generic to 0.2.4 #1028
  • Added ser/der support for joda DateTime #1038
  • Password is now optional for jdbc connection #1040
  • Add job cancellation option to ScioResult#waitUntilDone #1056 #1058 #1062 #1066

Bug fixes

  • Fix transform name in joins #1034 #1035
  • Add applyKvTransform #1020 #1032
  • Add helpers to initialize counters #1026 #1027
  • Fix SCollectionMatchers serialization #1001
  • Check runner version #1008 #1009
  • Log exception in AsyncLookupDoFn only if cache put fails #1039
  • ProjectId nonEmpty string check in BigQueryClient #1045
  • Fix SCollection#withSlidingWindows #1054