osdir.com


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Enrich testing doc with more unit test examples using AbstractStreamOperator


Hi all,

Does anyone want to share something about how to provide more better
testing tools or good documentations?

@Till, @Aljoscha
Do you have further suggestions about the improvements of testing tools and
what is the next step we can do?

Best,
Tony Wei


Tony Wei <tony19920430@xxxxxxxxx> 於 2018年9月29日 週六 上午12:20寫道:

> Hi Till,
>
> Thanks for your feedback. I didn't think about this before. It will be
> better if we can provide such tools instead of
> let user deal with testing in operator level directly. Because test
> harness was introduced for contributors, who
> may have more knowledge about internal design, to test their patches
> easily, it sometimes may not be intuitive
> to users to use test harness. And that's why I want to provide some
> examples in docs to ease the pain from such
> problem in the beginning.
>
> If Aljoscha has already made some efforts on it and let such tool be
> available in the near future, I will look forward
> to seeing this happen. And if there is anything I can help, I'm glad to
> give a hand.
>
> To follow up the discussion about users' requirements, I can provide some
> from my experience. What I want to
> achieve is to do unit testing on my operators with Flink features, such as
> states, event time and checkpoint.
>
> From the testing doc [1] I provided, I concluded some scenarios I would
> test in my flink application:
>
>    1. Easily control stream records to see the changes about the states
>    and outputs, no matter it is one input
>    stream operator or two.
>    2. Manually control watermark progress to test "out of order"
>    problem, "late data" problem or the behavior of
>    "event time service".
>    3. Preserve states from every version for testing states evolution and
>    compatibility or just to test the change
>    of Flink version.
>    4. Test customized stateful source function. In current version,
>    source function still need to be implemented
>    by a long running method. How to easily block it and verify the states
>    is helpful.
>    5. A simple way to expose states to verify the exactly value stored in
>    states, instead of testing it indirectly by
>    outputs. I archived this by getting state backend from test harness
>    and refer to some examples from Flink
>    project to access the keyed states I needed, but it is too deeper into
>    the internal implementations to make
>    it as an example in my testing doc [1].
>
> Best,
> Tony Wei
>
> [1]
> https://github.com/apache/flink/compare/master...tony810430:flink-testing-doc
>
> 2018-09-28 21:48 GMT+08:00 Till Rohrmann <trohrmann@xxxxxxxxxx>:
>
>> Hi Tony,
>>
>> I think this is a long sought-after feature to provide better testing
>> tools
>> for our users. Thus, I'm strongly in favour of adding something like this.
>> If I remember correctly Aljoscha already spend some brain cycles on this
>> and he also gave a training about the current state at FF SF 2018. I've
>> pulled him in to give more details.
>>
>> The first two things we can do is to collect requirements from our users
>> and to see what the current state is. Based on that we could plan which
>> things to add in which order.
>>
>> Cheers,
>> Till
>>
>> On Fri, Sep 28, 2018 at 3:13 PM Tony Wei <tony19920430@xxxxxxxxx> wrote:
>>
>> > Hi all,
>> >
>> > @Ken, Thanks for your positive feedback. I have a similar experience
>> with
>> > test harness and that's why
>> > I want to provide more contents on testing doc to prevent from this
>> kind of
>> > problems.
>> >
>> > Does anyone have any feedback and advice? I would like to collect more
>> > opinions from every developers
>> > and users. Please let me know what you think about this topic. All
>> > suggestions are welcome. Thank you.
>> >
>> > Best,
>> > Tony Wei
>> >
>> > 2018-09-26 2:20 GMT+08:00 Ken Krugler <kkrugler_lists@xxxxxxxxxxxx>:
>> >
>> > > Hi Tony,
>> > >
>> > > I think this would be great - we’ve been building out tests using
>> > > AbstractStreamOperator, and the lack of documentation has made it
>> > > challenging.
>> > >
>> > > For example, there was this exchange I had with Piotr about a month
>> ago:
>> > >
>> > > > You made a small mistake when restoring from state using test
>> harness,
>> > > that I myself have also done in the past. Problem is with an ordering
>> of
>> > > those calls:
>> > > >
>> > > >         result.open();
>> > > >         if (savedState != null) {
>> > > >             result.initializeState(savedState);
>> > > >         }
>> > > >
>> > > > Open is supposed to be called after initializeState, and if you look
>> > > into the code of AbstractStreamOperatorTestHarness#open, if it is
>> called
>> > > before initialize, it will initialize harness without any state.
>> > > >
>> > > > Unfortunate is that this is implicit behaviour that doesn’t throw
>> any
>> > > error (test harness is not part of a Flink’s public api). I will try
>> to
>> > fix
>> > > this: https://issues.apache.org/jira/browse/FLINK-10159 <
>> > > https://issues.apache.org/jira/browse/FLINK-10159>
>> > > — Ken
>> > >
>> > > > On Sep 25, 2018, at 3:30 AM, Tony Wei <tony19920430@xxxxxxxxx>
>> wrote:
>> > > >
>> > > > Hi all,
>> > > >
>> > > > It seems that there are more and more users from user mailing list
>> ask
>> > > how
>> > > > to do unit test with Flink
>> > > > features like states or timer. And the community usually tends to
>> > suggest
>> > > > them using
>> > > > `AbstractStreamOperator` and provide an example from Flink github
>> repo.
>> > > > Here I sort out some
>> > > > examples and write them down in the testing documentation [1]. And I
>> > > would
>> > > > link to contribute back
>> > > > to the Flink.
>> > > >
>> > > > The reason why I ask it first in dev mailing list is that
>> > > > `AbstractStreamOperator` is an internal API and
>> > > > could be changed at any time. I'm not sure if it is worth to provide
>> > > these
>> > > > examples on testing
>> > > > document, so I want to collect some feedbacks before I go to open a
>> > JIRA
>> > > > ticket.
>> > > >
>> > > > If this is feasible and valuable, then I will open the corresponding
>> > JIRA
>> > > > ticket and we can discuss
>> > > > more details of what examples are good to have in the document or
>> how
>> > to
>> > > > structure the content.
>> > > >
>> > > > I would really appreciate any feedback from you. Thanks in advance.
>> > > >
>> > > > Best Regards,
>> > > > Tony Wei
>> > > >
>> > > > [1]
>> > > > https://github.com/apache/flink/compare/master...
>> > > tony810430:flink-testing-doc
>> > >
>> > > --------------------------
>> > > Ken Krugler
>> > > +1 530-210-6378
>> > > http://www.scaleunlimited.com
>> > > Custom big data solutions & training
>> > > Flink, Solr, Hadoop, Cascading & Cassandra
>> > >
>> > >
>> >
>>
>
>