Kubernetes on your Platform

As of Apprenda 7.0.0, you are able to add a Kubernetes cluster to your Platform and use Platform features to deploy and manage applications on the cluster. If you already have a cluster added to your Platform, see more creating applications with Kubernetes components.

Why add a Kubernetes cluster?

The Platform acts as a governing layer for your Kubernetes clusters, allowing you to leverage the native behavior of the Platform to manage application settings before deploying the application onto the cluster. Using the Platform to manage both your new Kubernetes applications and existing applications will simplify interactions between the two groups. The Platform also allows you to separate the concerns of developers and cluster manages allowing developers to focus on developing better applications and operators on efficiently managing infrastructure.

Utilizing Platform features like Resource Policies, Custom Properties, or Platform Registry Settings, Platform Operators can establish rules for the behavior of Pods on the cluster. The Platform will automatically inject your desired configurations into into spec files at deploy time.

The Platform also abstracts away routing concerns by creating Services for all your applications by default. Developers are able to focus on developing their applications, without having to be concerned with the internal workings of Kubernetes. Once a spec file is uploaded, the Platform handles the necessarily configurations for things like aligning with resource management policies or exposing your application to a URL.

Multi-Cluster Support

As of Platform version 8.2.0, Platform Operators are able to add multiple clusters to a Platform. The addition of multi-cluster support to the Platform enables management of all clusters being used through a single shared service. Development teams can continue to use their custom configured clusters and operators gains the ability to set consistent policies across all clusters. Using the Platform to manage your clusters also allows Platform Operators a way to view an inventory of applications running on all clusters in your organization.

With multi-cluster support, Application Deployment Polices are now enabled to help you manage Pod deployments to clusters. Using these polices, you can isolate development teams to a set of clusters and remove concerns about clashing configuration requirements from other teams or applications. Deployment polices also enable to use your attached clusters as separate development, test, and production environments.

Cluster Requirements

Any cluster that meets the minimum requirements detailed below can be added to a Platform. Cluster support is infrastructure agnostic, meaning the Platform will treat both public and private hosted clusters the same.

Clusters added to the Platform must meet the following requirements:

  • Running Kubernetes 1.4 or later
  • The cluster URL is resolvable to a IP address
  • A user that is able to interact with the Kubernetes API, or as of Platform version 7.2.0, a X.509 Certificate signed by a Certificate Authority that is trusted by the Kubernetes API server. See more about setting up cluster authentication for the Platform
  • Cluster nodes are accessible to the Platform. This can be a subset of nodes on the cluster that the Platform can use as gateways to access the cluster

If your cluster meets these requirements, add it to the Platform on the Infrastructure>Clusters page in the SOC. If you are using a Platform version before 8.2.0, add a cluster on the Infrastructure>Clouds page of the SOC.

Creating a cluster

If you do not already have a cluster, we recommend using the Kismatic Enterprise Toolkit (KET) to set up your cluster. KET is Apprenda’s open source cluster installer that simplifies the process of creating a Kubernetes cluster for enterprise environments.

See the KET docs for more about installing Kubernetes with KET.

How does the Platform connect to a cluster?

Gateway Nodes

Gateway nodes are how the Platform is able to access the cluster. Assigned when the cluster is added to the Platform, they allow the Platform to connect to the cluster to route traffic to deployed Pods.

The Platform must be able to reach all nodes assigned as a gateway node and you must have at least one node assigned as a gateway node in order for the Platform to be able to interact with the cluster. Its recommended that you assign more than one node as a gateway to handle traffic loads and make sure that at least one node is always available for the Platform to connect to the cluster.

Accessing the Kubernetes API

The Kubernetes REST API is the main entrypoint for any Kubernetes cluster. It is used by operators and developers to manage all the resources that are deployed on the cluster. To access the API, the user has to authenticate using one of the authentication strategies supported by Kubernetes. Once the user is authenticated, the Authorization module kicks in to verify that the user has permissions to perform the requested action. See the Kubernetes documentation for more on authentication to a cluster.

In the same way, the Platform interacts with the Kubernetes API to deploy and manage the Kubernetes components of the applications you create on the Platform. To communicate with the cluster, the Platform’s Cluster Manager service needs to authenticate with the Kubernetes API server. As of Platform version 7.2.0, the Platform supports different authentication strategies, and allows the operator to select the strategy that should be used.

Basic Authentication uses HTTP Basic Authentication, in which the Platform Cluster Manager sends a username and password when issuing a request to the API server. In Platform versions before 7.2.0, only Basic Authentication is supported for clusters.

Client Certificate Authentication relies on X.509 certificates signed by a Certificate Authority that is trusted by the Kubernetes API server. The Platform Cluster Manager presents this certificate to the Kubernetes API server when issuing requests. If the API server trusts the signing CA of the certificate presented by the Platform Cluster Manager, the request is authenticated. Once authenticated, the API server will derive the username and groups from the certificate itself. The username is derived from the Common Name of the Subject, and the groups are obtained from the Organization fields of the certificate.

To create a PFX file from an existing certificate and private key, you may use OpenSSL.


  • OpenSSL
  • Client certificate
  • Client certificate’s corresponding private key
  • Certificate Authority’s certificate or public key

The following OpenSSL command will create an admin.pfx file using the client certificate file named admin.pem, the corresponding private key file named admin-key.pem and the CA cert file named ca.pem.

openssl pkcs12 -export -out admin.pfx -inkey admin-key.pem -in admin.pem -certfile ca.pem

Viewing cluster nodes on the Platform

When the Platform first connects to a cluster it asks Kubernetes about the topology of the cluster and then adds each of the reported nodes as a Kubernetes Linux Host. The reported nodes are visible on the Infrastructure>Servers page the SOC. You will be able to view all nodes on the cluster and the Pods deployed to each cluster through the SOC.

Platform Namespace

The Platform uses its own Namespace on an attached cluster to manage and deploy applications. When you attach a cluster to the Platform, the Platform will create the Namespace it will use on the cluster, if it isn’t already created. The default Namesace is defined in the Apprenda.Kubernetes.Namespace Platform Registry Setting and set to acp at Platform installation. In Platform version 8.2.0, you are able to set a non-default Namespace for a cluster when you attach it to the Platform.

Its recommended to use a new dedicated Namespace for the Platform when attaching a cluster. If you use an existing Namespace, note that the Platform may remove Pods that are not associated with the Platform. If you want to re-attach a removed cluster, you should use the same Namespace the Platform previously managed if it still exists on the cluster.

The Platform will not know about applications in other Namespaces of an attached cluster. Its recommended, though not required, that you use a cluster that is solely dedicated to applications from the Platform as you may experience over allocation of cluster resources.

Cluster Targeting and Deployment Policies

As of Platform version 8.2.0, you can attach multiple Kubernetes clusters to the same Cloud, providing you with a single operational portal for managing all the clusters in your organization. To help manage Pod deployments to the right cluster, you can use Application Deployment Policies and Custom Properties to set deployment rules for your clusters. These policies are set at the cluster level, meaning that you can manage which cluster the Platform deploys a certain component, but Kubernetes will handle choosing the specific node the Pod is deployed to on a cluster.

When deploying, The Platform will not split Pods across different clusters. When the first instance of a Kubernetes Pod component is deployed, the Platform decides which cluster to use and the same cluster is used for all future deployments of the same component.

The Platform will use the following rules to determine which cluster to deploy a component:

  1. If there is at least one component instance deployed on a cluster, the Platform will deploy other instances to the same cluster
  2. If this is the first instance of the Kubernetes component, the Platform will use Application Deployment Policies to filter the list of potential clusters for deployment
  3. If there are multiple potential clusters after filtering, the Platform will use the configured Distribution Strategies to determine which cluster to deploy to

Distribution Strategies for Clusters

If multiple clusters are applicable for deploying a Kubernetes component, the Platform will choose the deployment cluster according to the configured Distribution Strategies in the Hosting.Pods.Packing and Hosting.Pods.PackingType Registry Settings.

Hosting.Pods.Packing determines how the Platform should distribute across multiple cluster options. It can be set to either dense, or sparse (default). When compressed (dense), the Platform will choose to deploy workloads to clusters that already have deployed workloads if space is available. If balanced (sparse), the Platform will choose to deploy to a cluster with the fewest workloads. This only applies when Resource Throttling is enabled, and does not apply to Pods components which have a Resource Policy with no limits for the corresponding packing type.

Hosting.Pods.PackingType determines if the Platform evaluates Hosting.Pods.Packing based on CPU or Memory usage when deciding which cluster to deploy a workload.

For example, if Hosting.Pods.Packing is set to sparse and Hosting.Pods.PackingType is set to CPU, the Platform will chose the cluster with the most available CPU to deploy the next new Pod component.

Business Continuity and Failover

In the event that a cluster is unavailable, the Platform will try to redeploy all Pods to a new cluster. It is up to Platform Operators to make sure that there are back-up available deployment hosts and that any potential deployment host is configured to host the workloads.

Resource Management

You can create and assign Resource Policies to Kubernetes components to help manage resource consumption on your cluster. Resource Policies are translated into resource requests and limits in a PodTemplate for a Kubermetes component by the Platform at deploy time. When the Pods are deployed to the cluster, Kubernetes handles throttling resources. See the Kubernetes documentation more about resource management on Kubernetes.

An important difference in resource management between Kubernetes and the Platform is that Kubernetes applies resource limits at the container level and the Platform applies at the Pod level. This means that all containers within the same Pod are each given the same resource limits as the assigned Resource Policy making the total allotment n * CPU/Memory Limit (where n is the number of containers in a Pod). For example, if a Pod is defined with 2 containers and assigned a Resource Policy that sets a limit of .1 CPU, both containers will be assigned a limit of .1, making the total consumption limit .2 fraction of cores. Heapster

The Platform utilizes Heapster to collect resource utilization information for the cluster. In order to view this information in the on the Platform, you much configure your cluster to use Heapster. This is not required to add the cluster to the Platform, but is recommended so that Platform Operators and Developers can view accurate information in the SOC and Development Portal. See Heapster’s documentation for more on how to configure it on your cluster.

Note that utilization information will be reported in fractions of cores instead of MHZ to more accurately reflect the resource consumption on your cluster to how Kubernetes will track it. The Platform will poll your cluster in intervals defined in the Kubernetes.PodUtilizationUpdatePeriodSeconds (default: 60 seconds) to collect utilization data if Kubernetes.CapturePodUtilization is enabled (default: True). If you do not set up Heapster, no utilization information will be available on the Platform.