osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Proposal] Creating a reproducible environment for Beam Jenkins Tests


Thanks all. I am struggling with the missing buildscan reports when running jobs with containers. I believe it is a big disadvantage to use docker if the buildscan doesn't show up. I will keep updating my progress in this thread. In the meanwhile, any comments, suggestions and objections are still welcome.

Regards.
Yifan

On Tue, Sep 11, 2018 at 6:08 AM Alexey Romanenko <aromanenko.dev@xxxxxxxxx> wrote:
+1 Great feature that should help with complicated error cases.

On 11 Sep 2018, at 03:39, Henning Rohde <herohde@xxxxxxxxxx> wrote:

+1 Nice proposal. It will help eradicate some of the inflexibility and frustrations with Jenkins.

On Wed, Sep 5, 2018 at 2:30 PM Yifan Zou <yifanzou@xxxxxxxxxx> wrote:
Thank you all for making comments on this and I apologize for the late reply. 

To clarify the concerns of testing locally, it is still able to run tests without Docker. One of the purposes of this is to create an identical environment as we are running in Jenkins that would be helpful to reproduce strange errors. Contributors could choose starting a container and run tests in there, or just run tests directly. 



On Wed, Sep 5, 2018 at 6:37 AM Ismaël Mejía <iemejia@xxxxxxxxx> wrote:
BIG +1, the previous work on having docker build images [1] had a
similar goal (to have a reproducible build environment). But this is
even better because we will guarantee the exact same environment in
Jenkins as well as any further improvements. It is important to
document the setup process as part of this (for future maintenance +
local reproducibility).

Just for clarification this is independent of running the tests
locally without docker, it is more to improve the reproducibility of
the environment we have on jenkins locally for example to address some
weird Heissenbug.

I just added BEAM-5311 to track the removal of the docker build images
when this is ready (of course if there are no objections to this
proposal).

[1] https://beam.apache.org/contribute/docker-images/
On Thu, Aug 30, 2018 at 3:58 PM Jean-Baptiste Onofré <jb@xxxxxxxxxxxx> wrote:
>
> Hi,
>
> That's interesting, however, it's really important to still be able to
> easily run test locally, without any VM/Docker required. It should be
> activated by profile or so.
>
> Regards
> JB
>
> On 27/08/2018 19:53, Yifan Zou wrote:
> > Hi,
> >
> > I have a proposal for creating a reproducible environment for Jenkins
> > tests by using docker container. The thing is, the environment
> > configurations on Beam Jenkins slaves are sometimes different from
> > developer's machines. Test failures on Jenkins may not be easy to
> > reproduce locally. Also, it is not convenient for developers to add or
> > modify underlying tools installed on Jenkins VMs, since they're managed
> > by Apache Infra. This proposal is aimed to address those problems.
> >
> > https://docs.google.com/document/d/1y0YuQj_oZXC0uM5-gniG7r9-5gv2uiDhzbtgYYJW48c/edit#heading=h.bg2yi0wbhl9n
> >
> > Any comments are welcome. Thank you.
> >
> > Regards.
> > Yifan
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@xxxxxxxxxx
> http://blog.nanthrax.net
> Talend - http://www.talend.com