OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: A "Kamel" crazy idea


Of course you need to port them to go lang, the project is not intended to
be a drop in replacement for camel but a companion for integration that do
not require all the power and flexibility the jvm implementation provide

On Fri, 13 Jul 2018 at 08:55, Willem Jiang <willem.jiang@xxxxxxxxx> wrote:

> Hi Luca,
>
> You are building Go version of Camel by implement the DSL in Go.
> Now we can start the camel context by using camel-go to load the route,
> but here are some missing points that I want fill such as  how to reuse the
> Camel components?
>
>
>
>
>
> Willem Jiang
>
> Twitter: willemjiang
> Weibo: 姜宁willem
>
> On Fri, Jul 13, 2018 at 2:29 PM, Luca Burgazzoli <lburgazzoli@xxxxxxxxx>
> wrote:
>
> > Love it so ++1
> >
> > Btw, I did start some (early) experiments around alternative canel
> runtime
> > in my spare time:
> >
> > - https://github.com/lburgazzoli/camel-go
> > - https://github.com/lburgazzoli/camel-go-examples/
> > tree/master/example-yaml
> >
> > Happy to make it part of the kamel initiative if you find it interesting.
> >
> > On Fri, 13 Jul 2018 at 08:14, Sascha Dirbach <
> > sascha.dirbach@xxxxxxxxxxxxxxxxxxxxxx> wrote:
> >
> > > Hi Nicola,
> > >
> > > +1
> > >
> > > Great idea, do you already have a concept/idea/sketch on how to deal
> > > with complex configurations? i.e. SSL certs for outgoing calls, XSLT
> > > transformations, etc.
> > >
> > > When I think about it, you could probably use configmaps/secrets to
> > > mount these config in the container an then reference them in the
> route.
> > >
> > > Best regards,
> > >
> > > Sascha
> > >
> > > Am 13.07.2018 um 01:30 schrieb Nicola Ferraro:
> > > > Hi Cameleers,
> > > > it's now passed some time since I started thinking about a new
> project
> > > that
> > > > we can begin here at Apache Camel, and I'd like to have your opinion.
> > > >
> > > > We've already been targeting cloud-native applications with Camel,
> > > > especially on top of Kubernetes, that is becoming "the standard"
> cloud
> > > > platform. But writing a Camel integration and running it on
> Kubernetes
> > > > requires some effort: choosing the base platform (spring-boot, karaf,
> > > > simple main?), adding health checks (actuator?), packaging a docker
> > image
> > > > and creating the Kubernetes resources (fabric8-maven-plugin, helm?),
> > > > publishing the image on a docker registry, then finally deploying the
> > > > resources on a Kubernetes cluster.
> > > >
> > > > The resulting integration container is then far from being optimal
> > from a
> > > > resource consumption point of view: it is likely that a Camel
> > Spring-Boot
> > > > application will require at least 200MB of RAM and also some CPU
> shares
> > > > because of polling threads used by many components.
> > > >
> > > > In case people use a CI/CD pipeline, it will take also a long time to
> > get
> > > > from a code update to having a Kubernetes POD up and running.
> > > > Apart from compilation and image push/pull time, also startup time is
> > > often
> > > > ~10 seconds for Camel + Spring-Boot in a container with standard
> limits
> > > on
> > > > resources, making it difficult to propose this combination for
> > > "serverless
> > > > integration" (this term is becoming increasingly more popular).
> > > >
> > > > So, my proposal is to start to investigate a "more cloud-native"
> > approach
> > > > to integration: *making Camel integrations first-class citizens in
> > > > Kubernetes, and making them super fast and lightweight.*
> > > >
> > > > We can base the project on Kubernetes Custom Resource Definitions
> (CRD)
> > > > <
> > > https://kubernetes.io/docs/concepts/extend-kubernetes/
> > api-extension/custom-resources/
> > > >,
> > > > for example a Integration CRD and have a Kubernetes "operator"
> > > > <https://coreos.com/operators/> taking care of:
> > > > - Optimizing the integration that we want to run
> > > > - Packaging in a container
> > > > - Running it on Kubernetes
> > > > - Managing its entire lifecycle
> > > >
> > > > A Kubernetes-native integration may look like:
> > > >
> > > > -------------------
> > > > kind: "Integration"
> > > > apiVersion: "camel.apache.org/v1alpha1"
> > > > metadata:
> > > >  name: "example"
> > > > spec:
> > > >  replicas: 1
> > > >  routes:
> > > >   - id: timer
> > > >     route:
> > > >      - type: endpoint
> > > >        uri: timer:tick
> > > >      - type: endpoint
> > > >        uri: log:info
> > > > -------------------
> > > >
> > > > For those who are not familiar with Kubernetes resources, this kind
> of
> > > > YAML/JSON resource definitions are really common.
> > > > The example route is embedded in the Kubernetes resource declaration
> > and
> > > > follows a basic "flow DSL". We may start from a basic one and evolve
> it
> > > as
> > > > new requirements arrive from the community.
> > > >
> > > > I've made a very simple (but working) POC here:
> > > > https://github.com/nicolaferraro/integration-operator.
> > > >
> > > > This idea of a "Cloud-Native Camel" on Kubernetes (project codename
> can
> > > be "
> > > > *Kamel*", if you like it :D), will be an enabler for a lot of nice
> > > features.
> > > >
> > > > For example, we can propose "Kamel" as "ideal" platform for
> "serverless
> > > > integration" (I see many people reinventing the wheel out there): the
> > > > operator can reduce resource consumption of a single integration by
> > > > optimizing the runtime and also pause/resume integrations when they
> are
> > > not
> > > > used, that is the basic idea behind "serverless" (e.g. think to
> > > > HTTP-triggered integrations, but not only).
> > > > Focusing on serverless will bring more emphasis on push-based
> > > notifications
> > > > (webhooks, cloud events <https://cloudevents.io/>), that are rarely
> > > used in
> > > > Camel components, that prefer a poll based approach being it simpler
> to
> > > use
> > > > in classic deployments, but not so good in the cloud, where more
> > > resources
> > > > become higher direct costs for the users.
> > > >
> > > > The presence of the simplified DSL enables also experimenting on
> > > "*reduced*
> > > > subsets of Camel" implemented in languages other than Java, for
> example
> > > one
> > > > language that has a reactive approach on thread scheduling and a
> really
> > > low
> > > > memory footprint, like Go.
> > > >
> > > > But apart from this kind of experiments (that are valid IMO), the
> > "Kamel"
> > > > optimizer will have free room to choose the right platform for the
> > > > integration that the user wants to run, including, in the future,
> doing
> > > AOT
> > > > compilation using Graal/VM (less memory, faster startup) if the
> > features
> > > > (components) used in the integration are supporting it (maybe we can
> > add
> > > > AOT compilation in the roadmap for Camel 3).
> > > > A silly optimization: integrations starting from "timer:..." may be
> > > > scheduled directly with Kubernetes CronJobs, so they will consume
> > > resources
> > > > only when actually running.
> > > >
> > > > Being the final integrations lightweight and being the DSL
> > > > language-independent, we may see a increased adoption of Camel also
> as
> > > > agile integration layer for not-only-java applications (both "cloud"
> > and
> > > > "serverless" applications).
> > > >
> > > > I'm the first one that would like to work on a project ilke this.
> I've
> > > > worked on many Kubernetes/Openshift based applications and frameworks
> > in
> > > > the past years, also on operators and CRDs, and I think this way of
> > > > redesigning integrations has a lot of potential.
> > > >
> > > > Integrations will not be necessarily limited to the simplified DSL,
> but
> > > we
> > > > can add extension points for scripting and even custom libraries
> > > (although
> > > > limiting the freedom of the optimizer).
> > > >
> > > > The most important thing: it may become a great project, since it's
> > > driven
> > > > by a great community.
> > > >
> > > > So, what do you think? Is it crazy enough?
> > > >
> > > > Nicola
> > > >
> > >
> > > --
> > > Sascha Dirbach
> > >
> > > Inhaber
> > >
> > > endless webservices
> > > Marco Paetschke & Sascha Dirbach GbR
> > >
> > > Kirchweg 113
> > > 28201 Bremen
> > >
> > > Mobil: +49 (0)160-94182103
> > > Mail: sascha.dirbach@xxxxxxxxxxxxxxxxxxxxxx
> > > Web: www.endless-webservices.de
> > >
> > > USt-IdNr.: DE310969215
> > >
> > >
> > > --
> > --
> > Luca Burgazzoli
> >
>
-- 
--
Luca Burgazzoli