Kubernetes Cluster k8s_cluster

The kubernetes_cluster resource allows you to create immutable Kubernetes clusters running in Docker containers using K3s.

Properties

  • Name
    network
    Type
    (network_attachment: {})
    Required
    required
    Readonly
    Description

    Network attaches the container to an existing network defined in a separate stanza. This block can be specified multiple times to attach the container to multiple networks.

  • Name
    image
    Type
    (image: {})
    Required
    Readonly
    Description

    Image defines a Docker image to use when creating the container. By default the nomad cluster resource will be created using the latest Jumppad container image.

  • Name
    environment
    Type
    (map[string]string: map[]{})
    Required
    Readonly
    Description

    An env stanza allows you to set environment variables in the container. This stanza can be specified multiple times.

    env {
  key   = "PATH"
  value = "/usr/local/bin"
}
  • Name
    volume
    Type
    (volume: {})
    Required
    Readonly
    Description

    Additional volume to mount to the server and client nodes.

    volume {
  source = "./mydirectory"
  destination = "/path_in_container"
}
  • Name
    copy_image
    Type
    (image: {})
    Required
    Readonly
    Description

    Docker image in the local Docker image cache to copy to the cluster on creation. This image is added to the Nomad clients docker cache enabling jobs to use images that may not be in the local registry.

    Jumppad tracks changes to copied images, should the image change running jumppad up would push any changes to the cluster automatically.

    copy_image {
  name = "mylocalimage:versoin"
}
  • Name
    port
    Type
    (port: {})
    Required
    Readonly
    Description

    A port stanza allows you to expose container ports on the local network or host. This stanza can be specified multiple times.

    port {
  local = 80
  host  = 8080
}
  • Name
    port_range
    Type
    (port_range: {})
    Required
    Readonly
    Description

    A port_range stanza allows you to expose a range of container ports on the local network or host. This stanza can be specified multiple times.

    The following example would create 11 ports from 80 to 90 (inclusive) and expose them to the host machine.

    port {
  range       = "80-90"
  enable_host = true
}
  • Name
    api_port
    Type
    (int: 443)
    Required
    Readonly
    Description

    Port to expose the Kubernetes API on the host. By default this uses the standard api port 443; however, if you are running multiple kubernetes instances you will need to override this value.

  • Name
    connector_port
    Type
    (int: )
    Required
    Readonly
    readonly
    Description

    The port where the Jumppad connector is exposed to the host, this property is requied by the ingress resource and is not generally needed when building blueprints.

  • Name
    kube_config
    Type
    (kube_config: {})
    Required
    Readonly
    readonly
    Description

    Details for the Kubenetes config file that can be used to interact with the cluster.

  • Name
    container_name
    Type
    (string: "")
    Required
    Readonly
    readonly
    Description

    The fully qualified resource name for the Kubernetes cluster, this value can be used to address the server from the Docker network. It is also the name of the Docker container.

      server.name.k8s-cluster.jumpad.dev
  • Name
    external_ip
    Type
    (string: "")
    Required
    Readonly
    readonly
    Description

    Local IP address of the Nomad server, this property can be used to set the NOAMD_ADDR on the Jumppad client.

      output "K8S_ADDR" {
    value = "https://${resource.k8s_cluster.dev.external_ip}:${resource.k8s_cluster.dev.api_port}"
  }

config

Specifies the configuration for the Kubernetes cluster.

  • Name
    docker
    Type
    (docker_config: {})
    Required
    Readonly
    Description

    Docker configuration for the Kubernetes cluster.

  config {
    docker {
      no_proxy            = ["insecure.container.local.jmpd.in"]
      insecure_registries = ["insecure.container.local.jmpd.in:5003"]
    }
  }

docker_config

Specifies the configuration for the Docker engine in the cluster.

  • Name
    no_proxy
    Type
    ([]string: [])
    Required
    Readonly
    Description

    A list of docker registries that should not be proxied.

  • Name
    insecure_registries
    Type
    ([]string: [])
    Required
    Readonly
    Description

    A list of insecure docker registries.

network_attachment

Network attachment defines a network to which the container is attached.

  • Name
    id
    Type
    (string: "")
    Required
    required
    Readonly
    Description

    ID of the network to attach the container to, specified in reference format. e.g. to attach to a network called cloud

    network {
  id = "network.cloud"
}
  • Name
    ip_address
    Type
    (string: "")
    Required
    Readonly
    Description

    Static IP address to assign container for the network, the ip address must be within range defined by the network subnet. If this parameter is omitted an IP address will be automatically assigned.

  • Name
    aliases
    Type
    ([]string: [])
    Required
    Readonly
    Description

    Aliases allow alternate names to specified for the container. Aliases can be used to reference a container across the network, the container will respond to ping and other network resolution using the primary assigned name [name].container.shipyard.run and the aliases.

    network {
  name    = "network.cloud"
  aliases = [
    "alt1.container.local.jmpd.in", 
    "alt2.container.local.jmpd.in"
  ]
}
  • Name
    name
    Type
    (string: "")
    Required
    Readonly
    readonly
    Description

    Name will equal the name of the network as created by jumppad.

  • Name
    assigned_address
    Type
    (string: "")
    Required
    Readonly
    readonly
    Description

    assigned_address will equal the assigned IP address for the network. This will equal ip_address if set; otherwise, this is the automatically assigned ip_address.

image

Image defines a Docker image used when creating this container. An Image can be stored in a public or a private repository.

  • Name
    name
    Type
    (string: "")
    Required
    required
    Readonly
    Description

    Name of the image to use when creating the container, can either be the full canonical name or short name for Docker official images. e.g. consul:v1.6.1 or docker.io/consul:v1.6.1.

  • Name
    username
    Type
    (string: "")
    Required
    Readonly
    Description

    Username to use when connecting to a private image repository

  • Name
    password
    Type
    (string: "")
    Required
    Readonly
    Description

    Password to use when connecting to a private image repository, for both username and password interpolated environment variables can be used in place of static values.

    image {
  name = "myregistry.io/myimage:latest"
  username = env("REGISTRY_USERNAME")
  password = env("REGISTRY_PASSWORD")
}

volume

A volume type allows the specification of an attached volume.

  • Name
    source
    Type
    (string: "")
    Required
    required
    Readonly
    Description

    The source volume to mount in the container, can be specified as a relative ./ or absolute path /usr/local/bin. Relative paths are relative to the file declaring the container.

  • Name
    destination
    Type
    (string: "")
    Required
    required
    Readonly
    Description

    The destination in the container to mount the volume to, must be an absolute path.

  • Name
    type
    Type
    (string: bind)
    Required
    Readonly
    Description

    The type of the mount, can be one of the following values:

    • bind: bind the source path to the destination path in the container
    • volume: source is a Docker volume
    • tmpfs: create a temporary filesystem
  • Name
    bind_propagation
    Type
    (string: rprivate)
    Required
    Readonly
    Description

    Configures bind propagation for Docker volume mounts, only applies to bind mounts, can be one of the following values:

    • shared
    • slave
    • private
    • rslave
    • rprivate

    For more information please see the Docker documentation https://docs.docker.com/storage/bind-mounts/#configure-bind-propagation

  • Name
    bind_propagation_non_recursive
    Type
    (boolean: false)
    Required
    Readonly
    Description

    Configures recursiveness of the bind mount. By default Docker mounts with the equivalent of mount --rbind meaning that mounts below the the source directory are visible in the container. For instance running docker run --rm --mount type=bind,src=/,target=/host,readonly busybox will make /run of the host available as /host/run in the container -- and to make matters even worse it will be writable (since only the toplevel bind is set readonly, not the children). If bind_propagation_non_recursive is set to true then the container will only see an empty /host/run, meaning the tmpfs which is typically mounted to /run on the host is not propagated into the container.

  • Name
    selinux_relabel
    Type
    (string: "")
    Required
    Readonly
    Description

    Configures Selinux relabeling for the container (usually specified as :z or :Z) and can be one of the following values:

    • shared (Equivalent to :z)
    • private (Equivalent to :Z)

port

A port stanza defines host to container communications

  • Name
    local
    Type
    (string: "")
    Required
    required
    Readonly
    Description

    The local port in the container.

  • Name
    host
    Type
    (string: "")
    Required
    required
    Readonly
    Description

    The host port to map the local port to.

  • Name
    protocol
    Type
    (string: "")
    Required
    Readonly
    Description

    The protocol to use when exposing the port, can be "tcp", or "udp".

  • Name
    open_in_browser
    Type
    (string: /)
    Required
    Readonly
    Description

    Should a browser window be automatically opened when this resource is created. Browser windows will open at the path specified by this property.

port_range

A port_range stanza defines host to container communications by exposing a range of ports for the container.

  • Name
    range
    Type
    (string: "")
    Required
    required
    Readonly
    Description

    The port range to expose, e.g, 8080-8082 would expose the ports 8080, 8081, 8082.

  • Name
    enable_host
    Type
    (bool: false)
    Required
    Readonly
    Description

    The host port to map the local port to.

  • Name
    protocol
    Type
    (string: tcp)
    Required
    Readonly
    Description

    The protocol to use when exposing the port, can be "tcp", or "udp".

kube_config

The kube_config stanza provides the the details that can be used to interact with the cluster.

  • Name
    path
    Type
    (string: "")
    Required
    Readonly
    readonly
    Description

    The path to the kubeconfig file.

  • Name
    ca
    Type
    (string: "")
    Required
    Readonly
    readonly
    Description

    The certificate authority used to secure the cluster API. This value is base64 encoded.

  • Name
    client_certificate
    Type
    (string: "")
    Required
    Readonly
    readonly
    Description

    The client certificate authority used authenticate to the cluster API. This value is base64 encoded.

  • Name
    client_key
    Type
    (string: "")
    Required
    Readonly
    readonly
    Description

    The private key used authenticate to the cluster API. This value is base64 encoded.

Meta Properties

In addition to the main properties, all resources have meta properties, such as the id of the resource. To see the list of these properties please see the Meta Properties section in the documentation /docs/resources/meta.

Examples

Simple cluster

The following example creates a single node Kubernetes cluster and sets an output to the config path.

resource "network" "cloud" {
  subnet = "10.5.0.0/16"
}

resource "k8s_cluster" "k3s" {
  network {
    id = resource.network.cloud.meta.id
  }
}

output "KUBECONFIG" {
  value = resource.k8s_cluster.k3s.kube_config.path
}

Full Example

The following example creates a Kubernetes cluster and applies a Helm chart from a remote repository, a local Kubernetes deployment, and configures ingress resources for both the Helm chart and local deployment.

resource "network" "cloud" {
  subnet = "10.5.0.0/16"
}

resource "k8s_cluster" "k3s" {
  network {
    id = resource.network.cloud.meta.id
  }

  copy_image {
    name = "shipyardrun/connector:v0.1.0"
  }
}

resource "k8s_config" "fake_service" {
  cluster = resource.k8s_cluster.k3s

  paths = ["./fake_service.yaml"]

  health_check {
    timeout = "240s"
    pods    = ["app.kubernetes.io/name=fake-service"]
  }
}

resource "helm" "vault" {
  cluster = resource.k8s_cluster.k3s

  repository {
    name = "hashicorp"
    url  = "https://helm.releases.hashicorp.com"
  }

  chart   = "hashicorp/vault"
  version = "v0.18.0"

  values = "./helm/vault-values.yaml"

  health_check {
    timeout = "240s"
    pods    = ["app.kubernetes.io/name=vault"]
  }
}

resource "ingress" "vault_http" {
  port = 18200

  target {
    resource = resource.k8s_cluster.k3s
    port = 8200

    config = {
      service   = "vault"
      namespace = "default"
    }
  }
}

resource "ingress" "fake_service" {
  port = 19090

  target {
    resource = resource.k8s_cluster.k3s
    port = 9090

    config = {
      service   = "fake-service"
      namespace = "default"
    }
  }
}

output "VAULT_ADDR" {
  value = "http://${resource.ingress.vault_http.local_address}"
}

output "KUBECONFIG" {
  value = resource.k8s_cluster.k3s.kube_config.path
}

Image Caching

Kubernetes clusters do not share the local machines Docker image cache. Each node in a cluster has it's own unqiue cache.

To save bandwidth all containers launched in the Kubernetes cluster pulled through an image cache that runs in Docker. After the first pull all images are subsequently pulled from the image cache not the public internet. This cache is global to all Nomad and Kubernetes clusters created with Jumppad.

For more information on the image cache see the container_registry resource.