Re: A "Kamel" crazy idea
Great idea and love the Kamel name.
On Fri, Jul 13, 2018 at 1:30 AM, Nicola Ferraro <ni.ferraro@xxxxxxxxx> wrote:
> Hi Cameleers,
> it's now passed some time since I started thinking about a new project that
> we can begin here at Apache Camel, and I'd like to have your opinion.
> We've already been targeting cloud-native applications with Camel,
> especially on top of Kubernetes, that is becoming "the standard" cloud
> platform. But writing a Camel integration and running it on Kubernetes
> requires some effort: choosing the base platform (spring-boot, karaf,
> simple main?), adding health checks (actuator?), packaging a docker image
> and creating the Kubernetes resources (fabric8-maven-plugin, helm?),
> publishing the image on a docker registry, then finally deploying the
> resources on a Kubernetes cluster.
> The resulting integration container is then far from being optimal from a
> resource consumption point of view: it is likely that a Camel Spring-Boot
> application will require at least 200MB of RAM and also some CPU shares
> because of polling threads used by many components.
> In case people use a CI/CD pipeline, it will take also a long time to get
> from a code update to having a Kubernetes POD up and running.
> Apart from compilation and image push/pull time, also startup time is often
> ~10 seconds for Camel + Spring-Boot in a container with standard limits on
> resources, making it difficult to propose this combination for "serverless
> integration" (this term is becoming increasingly more popular).
> So, my proposal is to start to investigate a "more cloud-native" approach
> to integration: *making Camel integrations first-class citizens in
> Kubernetes, and making them super fast and lightweight.*
> We can base the project on Kubernetes Custom Resource Definitions (CRD)
> for example a Integration CRD and have a Kubernetes "operator"
> <https://coreos.com/operators/> taking care of:
> - Optimizing the integration that we want to run
> - Packaging in a container
> - Running it on Kubernetes
> - Managing its entire lifecycle
> A Kubernetes-native integration may look like:
> kind: "Integration"
> apiVersion: "camel.apache.org/v1alpha1"
> name: "example"
> replicas: 1
> - id: timer
> - type: endpoint
> uri: timer:tick
> - type: endpoint
> uri: log:info
> For those who are not familiar with Kubernetes resources, this kind of
> YAML/JSON resource definitions are really common.
> The example route is embedded in the Kubernetes resource declaration and
> follows a basic "flow DSL". We may start from a basic one and evolve it as
> new requirements arrive from the community.
> I've made a very simple (but working) POC here:
> This idea of a "Cloud-Native Camel" on Kubernetes (project codename can be "
> *Kamel*", if you like it :D), will be an enabler for a lot of nice features.
> For example, we can propose "Kamel" as "ideal" platform for "serverless
> integration" (I see many people reinventing the wheel out there): the
> operator can reduce resource consumption of a single integration by
> optimizing the runtime and also pause/resume integrations when they are not
> used, that is the basic idea behind "serverless" (e.g. think to
> HTTP-triggered integrations, but not only).
> Focusing on serverless will bring more emphasis on push-based notifications
> (webhooks, cloud events <https://cloudevents.io/>), that are rarely used in
> Camel components, that prefer a poll based approach being it simpler to use
> in classic deployments, but not so good in the cloud, where more resources
> become higher direct costs for the users.
> The presence of the simplified DSL enables also experimenting on "*reduced*
> subsets of Camel" implemented in languages other than Java, for example one
> language that has a reactive approach on thread scheduling and a really low
> memory footprint, like Go.
> But apart from this kind of experiments (that are valid IMO), the "Kamel"
> optimizer will have free room to choose the right platform for the
> integration that the user wants to run, including, in the future, doing AOT
> compilation using Graal/VM (less memory, faster startup) if the features
> (components) used in the integration are supporting it (maybe we can add
> AOT compilation in the roadmap for Camel 3).
> A silly optimization: integrations starting from "timer:..." may be
> scheduled directly with Kubernetes CronJobs, so they will consume resources
> only when actually running.
> Being the final integrations lightweight and being the DSL
> language-independent, we may see a increased adoption of Camel also as
> agile integration layer for not-only-java applications (both "cloud" and
> "serverless" applications).
> I'm the first one that would like to work on a project ilke this. I've
> worked on many Kubernetes/Openshift based applications and frameworks in
> the past years, also on operators and CRDs, and I think this way of
> redesigning integrations has a lot of potential.
> Integrations will not be necessarily limited to the simplified DSL, but we
> can add extension points for scripting and even custom libraries (although
> limiting the freedom of the optimizer).
> The most important thing: it may become a great project, since it's driven
> by a great community.
> So, what do you think? Is it crazy enough?
Camel in Action 2: https://www.manning.com/ibsen2