Skip to content

Functions

Kanister Functions are written in go and are compiled when building the controller. They are referenced by Blueprints phases. A Kanister Function implements the following go interface:

go
// Func allows custom actions to be executed.
type Func interface {
    Name() string
    Exec(ctx context.Context, args ...string) (map*string*interface{}, error)
    RequiredArgs() []string
    Arguments() []string
}

Kanister Functions are registered by the return value of Name(), which must be static.

Each phase in a Blueprint executes a Kanister Function. The Func field in a BlueprintPhase is used to lookup a Kanister Function. After BlueprintPhase.Args are rendered, they are passed into the Kanister Function's Exec() method.

The RequiredArgs method returns the list of argument names that are required. And Arguments method returns the list of all the argument names that are supported by the function.

Existing Functions

The Kanister controller ships with the following Kanister Functions out-of-the-box that provide integration with Kubernetes:

KubeExec

KubeExec is similar to running

bash
kubectl exec -it --namespace <NAMESPACE> <POD> -c <CONTAINER> [CMD LIST...]
ArgumentRequiredTypeDescription
namespaceYesstringnamespace in which to execute
podYesstringname of the pod in which to execute
containerNostring(required if pod contains more than 1 container) name of the container in which to execute
commandYes[]stringcommand list to execute

Example:

yaml
- func: KubeExec
  name: examplePhase
  args:
    namespace: "{{ .Deployment.Namespace }}"
    pod: "{{ index .Deployment.Pods 0 }}"
    container: kanister-sidecar
    command:
      - sh
      - -c
      - |
        echo "Example"

KubeExecAll

KubeExecAll is similar to running KubeExec on specified containers of given pods (all specified containers on given pods) in parallel. In the below example, the command is going to be executed in both the containers of the given pods.

ArgumentRequiredTypeDescription
namespaceYesstringnamespace in which to execute
podsYesstringspace separated list of names of pods in which to execute
containersYesstringspace separated list of names of the containers in which to execute
commandYes[]stringcommand list to execute

Example:

yaml
- func: KubeExecAll
  name: examplePhase
  args:
    namespace: "{{ .Deployment.Namespace }}"
    pods: "{{ index .Deployment.Pods 0 }} {{ index .Deployment.Pods 1 }}"
    containers: "container1 container2"
    command:
      - sh
      - -c
      - |
        echo "Example"

KubeTask

KubeTask spins up a new container and executes a command via a Pod. This allows you to run a new Pod from a Blueprint.

ArgumentRequiredTypeDescription
namespaceNostringnamespace in which to execute (the pod will be created in controller's namespace if not specified)
imageYesstringimage to be used for executing the task
commandYes[]stringcommand list to execute
podOverrideNomap[string]interface{}specs to override default pod specs with
podAnnotationsNomap[string]stringcustom annotations for the temporary pod that gets created
podLabelsNomap[string]stringcustom labels for the temporary pod that gets created

Example:

yaml
- func: KubeTask
  name: examplePhase
  args:
    namespace: "{{ .Deployment.Namespace }}"
    image: busybox
    podOverride:
      containers:
      - name: container
        imagePullPolicy: IfNotPresent
    podAnnotations:
      annKey: annValue
    podLabels:
      labelKey: labelValue
    command:
      - sh
      - -c
      - |
        echo "Example"

ScaleWorkload

ScaleWorkload is used to scale up or scale down a Kubernetes workload. It also sets the original replica count of the workload as output artifact with the key originalReplicaCount. The function only returns after the desired replica state is achieved:

  • When reducing the replica count, wait until all terminating pods complete.
  • When increasing the replica count, wait until all pods are ready.

Currently the function supports Deployments, StatefulSets and DeploymentConfigs.

It is similar to running

bash
kubectl scale deployment <DEPLOYMENT-NAME> --replicas=<NUMBER OF REPLICAS> --namespace <NAMESPACE>

This can be useful if the workload needs to be shutdown before processing certain data operations. For example, it may be useful to use ScaleWorkload to stop a database process before restoring files. See Using ScaleWorkload function with output artifact for an example with new ScaleWorkload function.

ArgumentRequiredTypeDescription
namespaceNostringnamespace in which to execute
nameNostringname of the workload to scale
kindNostring[deployment] or [statefulset]
replicasYesintThe desired number of replicas
waitForReadyNoboolWhether to wait for the workload to be ready before executing next steps. Default Value is true

Example of scaling down:

yaml
- func: ScaleWorkload
  name: examplePhase
  args:
    namespace: "{{ .Deployment.Namespace }}"
    name: "{{ .Deployment.Name }}"
    kind: deployment
    replicas: 0

Example of scaling up:

yaml
- func: ScaleWorkload
  name: examplePhase
  args:
    namespace: "{{ .Deployment.Namespace }}"
    name: "{{ .Deployment.Name }}"
    kind: deployment
    replicas: 1
    waitForReady: false

PrepareData

This function allows running a new Pod that will mount one or more PVCs and execute a command or script that manipulates the data on the PVCs.

The function can be useful when it is necessary to perform operations on the data volumes that are used by one or more application containers. The typical sequence is to stop the application using ScaleWorkload, perform the data manipulation using PrepareData, and then restart the application using ScaleWorkload.

NOTE

It is extremely important that, if PrepareData modifies the underlying data, the PVCs must not be currently in use by an active application container (ensure by using ScaleWorkload with replicas=0 first). For advanced use cases, it is possible to have concurrent access but the PV needs to have RWX mode enabled and the volume needs to use a clustered file system that supports concurrent access.

ArgumentRequiredTypeDescription
namespaceYesstringnamespace in which to execute
imageYesstringimage to be used the command
volumesNomap[string]stringMapping of pvcName to mountPath under which the volume will be available
commandYes[]stringcommand list to execute
serviceaccountNostringservice account info
podOverrideNomap[string]interface{}specs to override default pod specs with
podAnnotationsNomap[string]stringcustom annotations for the temporary pod that gets created
podLabelsNomap[string]stringcustom labels for the temporary pod that gets created

NOTE

The volumes argument does not support subPath mounts so the data manipulation logic needs to be aware of any subPath mounts that may have been used when mounting a PVC in the primary application container. If volumes argument is not specified, all volumes belonging to the protected object will be mounted at the predefined path /mnt/prepare_data/<pvcName>

Example:

yaml
- func: ScaleWorkload
  name: ShutdownApplication
  args:
    namespace: "{{ .Deployment.Namespace }}"
    name: "{{ .Deployment.Name }}"
    kind: deployment
    replicas: 0
- func: PrepareData
  name: ManipulateData
  args:
    namespace: "{{ .Deployment.Namespace }}"
    image: busybox
    volumes:
      application-pvc-1: "/data"
      application-pvc-2: "/restore-data"
    podAnnotations:
      annKey: annValue
    podLabels:
      labelKey: labelValue
    command:
      - sh
      - -c
      - |
        cp /restore-data/file_to_replace.data /data/file.data

BackupData

This function backs up data from a container into any object store supported by Kanister.

WARNING

The BackupData will be deprecated soon. We recommend using CopyVolumeData instead. However, RestoreData and DeleteData will continue to be available, ensuring you retain control over your existing backups.

NOTE

It is important that the application includes a kanister-tools sidecar container. This sidecar is necessary to run the tools that capture path on a volume and store it on the object store.

Arguments:

ArgumentRequiredTypeDescription
namespaceYesstringnamespace in which to execute
podYesstringpod in which to execute
containerYesstringcontainer in which to execute
includePathYesstringpath of the data to be backed up
backupArtifactPrefixYesstringpath to store the backup on the object store
encryptionKeyNostringencryption key to be used for backups
insecureTLSNoboolenables insecure connection for data mover

Outputs:

OutputTypeDescription
backupTagstringunique tag added to the backup
backupIDstringunique snapshot id generated during backup

Example:

yaml
actions:
  backup:
    outputArtifacts:
      backupInfo:
        keyValue:
          backupIdentifier: "{{ .Phases.BackupToObjectStore.Output.backupTag }}"
    phases:
      - func: BackupData
        name: BackupToObjectStore
        args:
          namespace: "{{ .Deployment.Namespace }}"
          pod: "{{ index .Deployment.Pods 0 }}"
          container: kanister-tools
          includePath: /mnt/data
          backupArtifactPrefix: s3-bucket/path/artifactPrefix

BackupDataAll

This function concurrently backs up data from one or more pods into an any object store supported by Kanister.

WARNING

The BackupDataAll will be deprecated soon. However, RestoreDataAll and DeleteDataAll will continue to be available, ensuring you retain control over your existing backups.

NOTE

It is important that the application includes a kanister-tools sidecar container. This sidecar is necessary to run the tools that capture path on a volume and store it on the object store.

Arguments:

ArgumentRequiredTypeDescription
namespaceYesstringnamespace in which to execute
podsNostringpods in which to execute (by default runs on all the pods)
containerYesstringcontainer in which to execute
includePathYesstringpath of the data to be backed up
backupArtifactPrefixYesstringpath to store the backup on the object store appended by pod name later
encryptionKeyNostringencryption key to be used for backups
insecureTLSNoboolenables insecure connection for data mover

Outputs:

OutputTypeDescription
BackupAllInfostringinfo about backup tag and identifier required for restore

Example:

yaml
actions:
  backup:
    outputArtifacts:
      params:
        keyValue:
          backupInfo: "{{ .Phases.backupToObjectStore.Output.BackupAllInfo }}"
    phases:
      - func: BackupDataAll
        name: BackupToObjectStore
        args:
          namespace: "{{ .Deployment.Namespace }}"
          container: kanister-tools
          includePath: /mnt/data
          backupArtifactPrefix: s3-bucket/path/artifactPrefix

RestoreData

This function restores data backed up by the BackupData function. It creates a new Pod that mounts the PVCs referenced by the specified Pod and restores data to the specified path.

NOTE

It is extremely important that, the PVCs are not be currently in use by an active application container, as they are required to be mounted to the new Pod (ensure by using ScaleWorkload with replicas=0 first). For advanced use cases, it is possible to have concurrent access but the PV needs to have RWX mode enabled and the volume needs to use a clustered file system that supports concurrent access.

ArgumentRequiredTypeDescription
namespaceYesstringnamespace in which to execute
imageYesstringimage to be used for running restore
backupArtifactPrefixYesstringpath to the backup on the object store
backupIdentifierNostring(required if backupTag not provided) unique snapshot id generated during backup
backupTagNostring(required if backupIdentifier not provided) unique tag added during the backup
restorePathNostringpath where data is restored
podNostringpod to which the volumes are attached
volumesNomap[string]stringMapping of [pvcName] to [mountPath] under which the volume will be available
encryptionKeyNostringencryption key to be used during backups
insecureTLSNoboolenables insecure connection for data mover
podOverrideNomap[string]interface{}specs to override default pod specs with
podAnnotationsNomap[string]stringcustom annotations for the temporary pod that gets created
podLabelsNomap[string]stringcustom labels for the temporary pod that gets created

NOTE

The image argument requires the use of ghcr.io/kanisterio/kanister-tools image since it includes the required tools to restore data from the object store. Between the pod and volumes arguments, exactly one argument must be specified.

Example:

Consider a scenario where you wish to restore the data backed up by the BackupData function. We will first scale down the application, restore the data and then scale it back up. For this phase, we will use the backupInfo Artifact provided by backup function.

yaml
- func: ScaleWorkload
  name: ShutdownApplication
  args:
    namespace: \"{{.Deployment.Namespace }}\"
    name: \"{{ .Deployment.Name }}\"
    kind: Deployment
    replicas: 0
- func: RestoreData
  name: RestoreFromObjectStore
  args:
    namespace: \"{{.Deployment.Namespace }}\"
    pod: \"{{ index .Deployment.Pods 0 }}\"
    image: ghcr.io/kanisterio/kanister-tools:0.110.0
    backupArtifactPrefix: s3-bucket/path/artifactPrefix
    backupTag: \"{{.ArtifactsIn.backupInfo.KeyValue.backupIdentifier }}\"
    podAnnotations:
      annKey: annValue
    podLabels:
      labelKey: labelValue
- func: ScaleWorkload
  name: StartupApplication
  args:
    namespace: \"{{.Deployment.Namespace }}\"
    name: \"{{ .Deployment.Name }}\"
    kind: Deployment
    replicas: 1

RestoreDataAll

This function concurrently restores data backed up by the BackupDataAll function, on one or more pods. It concurrently runs a job Pod for each workload Pod, that mounts the respective PVCs and restores data to the specified path.

NOTE

It is extremely important that, the PVCs are not be currently in use by an active application container, as they are required to be mounted to the new Pod (ensure by using ScaleWorkload with replicas=0 first). For advanced use cases, it is possible to have concurrent access but the PV needs to have RWX mode enabled and the volume needs to use a clustered file system that supports concurrent access.

ArgumentRequiredTypeDescription
namespaceYesstringnamespace in which to execute
imageYesstringimage to be used for running restore
backupArtifactPrefixYesstringpath to the backup on the object store
restorePathNostringpath where data is restored
podsNostringpods to which the volumes are attached
encryptionKeyNostringencryption key to be used during backups
backupInfoYesstringsnapshot info generated as output in BackupDataAll function
insecureTLSNoboolenables insecure connection for data mover
podOverrideNomap[string]interface{}specs to override default pod specs with
podAnnotationsNomap[string]stringcustom annotations for the temporary pod that gets created
podLabelsNomap[string]stringcustom labels for the temporary pod that gets created

NOTE

The image argument requires the use of [ghcr.io/kanisterio/kanister-tools] image since it includes the required tools to restore data from the object store. Between the pod and volumes arguments, exactly one argument must be specified.

Example:

Consider a scenario where you wish to restore the data backed up by the BackupDataAll function. We will first scale down the application, restore the data and then scale it back up. We will not specify pods in args, so this function will restore data on all pods concurrently. For this phase, we will use the params Artifact provided by BackupDataAll function.

yaml
- func: ScaleWorkload
  name: ShutdownApplication
  args:
    namespace: "{{ .Deployment.Namespace }}"
    name: "{{ .Deployment.Name }}"
    kind: Deployment
    replicas: 0
- func: RestoreDataAll
  name: RestoreFromObjectStore
  args:
    namespace: "{{ .Deployment.Namespace }}"
    image: ghcr.io/kanisterio/kanister-tools:0.110.0
    backupArtifactPrefix: s3-bucket/path/artifactPrefix
    backupInfo: "{{ .ArtifactsIn.params.KeyValue.backupInfo }}"
    podAnnotations:
      annKey: annValue
    podLabels:
      labelKey: labelValue
- func: ScaleWorkload
  name: StartupApplication
  args:
    namespace: "{{ .Deployment.Namespace }}"
    name: "{{ .Deployment.Name }}"
    kind: Deployment
    replicas: 2

CopyVolumeData

This function copies data from the specified volume (referenced by a Kubernetes PersistentVolumeClaim) into an object store. This data can be restored into a volume using the RestoreData function

NOTE

The PVC must not be in-use (attached to a running Pod)

If data needs to be copied from a running workload without stopping it, use the BackupData function

Arguments:

ArgumentRequiredTypeDescription
namespaceYesstringnamespace the source PVC is in
volumeYesstringname of the source PVC
dataArtifactPrefixYesstringpath on the object store to store the data in
encryptionKeyNostringencryption key to be used during backups
insecureTLSNoboolenables insecure connection for data mover
podOverrideNomap[string]interface{}specs to override default pod specs with
podAnnotationsNomap[string]stringcustom annotations for the temporary pod that gets created
podLabelsNomap[string]stringcustom labels for the temporary pod that gets created

Outputs:

OutputTypeDescription
backupIDstringunique snapshot id generated when data was copied
backupRootstringparent directory location of the data copied from
backupArtifactLocationstringlocation in objectstore where data was copied
backupTagstringunique string to identify this data copy

Example:

If the ActionSet Object is a PersistentVolumeClaim:

yaml
- func: CopyVolumeData
  args:
    namespace: "{{ .PVC.Namespace }}"
    volume: "{{ .PVC.Name }}"
    dataArtifactPrefix: s3-bucket-name/path
    podAnnotations:
      annKey: annValue
    podLabels:
      labelKey: labelValue

DeleteData

This function deletes the snapshot data backed up by the BackupData function.

ArgumentRequiredTypeDescription
namespaceYesstringnamespace in which to execute
backupArtifactPrefixYesstringpath to the backup on the object store
backupIDNostring(required if backupTag not provided) unique snapshot id generated during backup
backupTagNostring(required if backupID not provided) unique tag added during the backup
encryptionKeyNostringencryption key to be used during backups
insecureTLSNoboolenables insecure connection for data mover
podOverrideNomap[string]interface{}specs to override default pod specs with
podAnnotationsNomap[string]stringcustom annotations for the temporary pod that gets created
podLabelsNomap[string]stringcustom labels for the temporary pod that gets created

Example:

Consider a scenario where you wish to delete the data backed up by the BackupData function. For this phase, we will use the backupInfo Artifact provided by backup function.

yaml
- func: DeleteData
  name: DeleteFromObjectStore
  args:
    namespace: "{{ .Namespace.Name }}"
    backupArtifactPrefix: s3-bucket/path/artifactPrefix
    backupTag: "{{ .ArtifactsIn.backupInfo.KeyValue.backupIdentifier }}"
    podAnnotations:
      annKey: annValue
    podLabels:
      labelKey: labelValue

DeleteDataAll

This function concurrently deletes the snapshot data backed up by the BackupDataAll function.

ArgumentRequiredTypeDescription
namespaceYesstringnamespace in which to execute
backupArtifactPrefixYesstringpath to the backup on the object store
backupInfoYesstringsnapshot info generated as output in BackupDataAll function
encryptionKeyNostringencryption key to be used during backups
reclaimSpaceNoboolprovides a way to specify if space should be reclaimed
insecureTLSNoboolenables insecure connection for data mover
podOverrideNomap[string]interface{}specs to override default pod specs with
podAnnotationsNomap[string]stringcustom annotations for the temporary pod that gets created
podLabelsNomap[string]stringcustom labels for the temporary pod that gets created

Example:

Consider a scenario where you wish to delete all the data backed up by the BackupDataAll function. For this phase, we will use the params Artifact provided by backup function.

yaml
- func: DeleteDataAll
  name: DeleteFromObjectStore
  args:
    namespace: "{{ .Namespace.Name }}"
    backupArtifactPrefix: s3-bucket/path/artifactPrefix
    backupInfo: "{{ .ArtifactsIn.params.KeyValue.backupInfo }}"
    reclaimSpace: true
    podAnnotations:
      annKey: annValue
    podLabels:
      labelKey: labelValue

LocationDelete

This function uses a new Pod to delete the specified artifact from an object store.

ArgumentRequiredTypeDescription
artifactYesstringartifact to be deleted from the object store

NOTE

The Kubernetes job uses the ghcr.io/kanisterio/kanister-tools image, since it includes all the tools required to delete the artifact from an object store.

Example:

yaml
- func: LocationDelete
  name: LocationDeleteFromObjectStore
  args:
    artifact: s3://bucket/path/artifact

CreateVolumeSnapshot

This function is used to create snapshots of one or more PVCs associated with an application. It takes individual snapshot of each PVC which can be then restored later. It generates an output that contains the Snapshot info required for restoring PVCs.

NOTE

Currently we only support PVC snapshots on AWS EBS. Support for more storage providers is coming soon!

Arguments:

ArgumentRequiredTypeDescription
namespaceYesstringnamespace in which to execute
pvcsNo[]stringlist of names of PVCs to be backed up
skipWaitNoboolinitiate but do not wait for the snapshot operation to complete

When no PVCs are specified in the pvcs argument above, all PVCs in use by a Deployment or StatefulSet will be backed up.

Outputs:

OutputTypeDescription
volumeSnapshotInfostringSnapshot info required while restoring the PVCs

Example:

Consider a scenario where you wish to backup all PVCs of a deployment. The output of this phase is saved to an Artifact named backupInfo, shown below:

yaml
actions:
  backup:
    outputArtifacts:
      backupInfo:
        keyValue:
          manifest: "{{ .Phases.backupVolume.Output.volumeSnapshotInfo }}"
    phases:
    - func: CreateVolumeSnapshot
      name: backupVolume
      args:
        namespace: "{{ .Deployment.Namespace }}"

WaitForSnapshotCompletion

This function is used to wait for completion of snapshot operations initiated using the CreateVolumeSnapshot function. function.

Arguments:

ArgumentRequiredTypeDescription
snapshotsYesstringsnapshot info generated as output in CreateVolumeSnapshot function

CreateVolumeFromSnapshot

This function is used to restore one or more PVCs of an application from the snapshots taken using the CreateVolumeSnapshot function. It deletes old PVCs, if present and creates new PVCs from the snapshots taken earlier.

Arguments:

ArgumentRequiredTypeDescription
namespaceYesstringnamespace in which to execute
snapshotsYesstringsnapshot info generated as output in CreateVolumeSnapshot function

Example:

Consider a scenario where you wish to restore all PVCs of a deployment. We will first scale down the application, restore PVCs and then scale up. For this phase, we will make use of the backupInfo Artifact provided by the CreateVolumeSnapshot function.

yaml
- func: ScaleWorkload
  name: shutdownPod
  args:
    namespace: "{{ .Deployment.Namespace }}"
    name: "{{ .Deployment.Name }}"
    kind: Deployment
    replicas: 0
- func: CreateVolumeFromSnapshot
  name: restoreVolume
  args:
    namespace: "{{ .Deployment.Namespace }}"
    snapshots: "{{ .ArtifactsIn.backupInfo.KeyValue.manifest }}"
- func: ScaleWorkload
  name: bringupPod
  args:
    namespace: "{{ .Deployment.Namespace }}"
    name: "{{ .Deployment.Name }}"
    kind: Deployment
    replicas: 1

DeleteVolumeSnapshot

This function is used to delete snapshots of PVCs taken using the CreateVolumeSnapshot function.

Arguments:

ArgumentRequiredTypeDescription
namespaceYesstringnamespace in which to execute
snapshotsYesstringsnapshot info generated as output in CreateVolumeSnapshot function

Example:

yaml
- func: DeleteVolumeSnapshot
  name: deleteVolumeSnapshot
  args:
    namespace: "{{ .Deployment.Namespace }}"
    snapshots: "{{ .ArtifactsIn.backupInfo.KeyValue.manifest }}"

BackupDataStats

This function get stats for the backed up data from the object store location

NOTE

It is important that the application includes a kanister-tools sidecar container. This sidecar is necessary to run the tools that get the information from the object store.

Arguments:

ArgumentRequiredTypeDescription
namespaceYesstringnamespace in which to execute
backupArtifactPrefixYesstringpath to the object store location
backupIDYesstringunique snapshot id generated during backup
modeNostringmode in which stats are expected
encryptionKeyNostringencryption key to be used for backups
podAnnotationsNomap[string]stringcustom annotations for the temporary pod that gets created
podLabelsNomap[string]stringcustom labels for the temporary pod that gets created

Outputs:

OutputTypeDescription
modestringmode of the output stats
fileCountstringnumber of files in backup
sizestringsize of the number of files in backup

Example:

yaml
actions:
  backupStats:
    outputArtifacts:
      backupStats:
        keyValue:
          mode: "{{ .Phases.BackupDataStatsFromObjectStore.Output.mode }}"
          fileCount: "{{ .Phases.BackupDataStatsFromObjectStore.Output.fileCount }}"
          size: "{{ .Phases.BackupDataStatsFromObjectStore.Output.size }}"
    phases:
      - func: BackupDataStats
        name: BackupDataStatsFromObjectStore
        args:
          namespace: "{{ .Deployment.Namespace }}"
          backupArtifactPrefix: s3-bucket/path/artifactPrefix
          mode: restore-size
          backupID: "{{ .ArtifactsIn.snapshot.KeyValue.backupIdentifier }}"
          podAnnotations:
            annKey: annValue
          podLabels:
            labelKey: labelValue

CreateRDSSnapshot

This function creates RDS snapshot of running RDS instance.

Arguments:

ArgumentRequiredTypeDescription
instanceIDYesstringID of RDS instance you want to create snapshot of
dbEngineNostringRequired in case of RDS Aurora instance. Supported DB Engines: aurora aurora-mysql and aurora-postgresql

Outputs:

OutputTypeDescription
snapshotIDstringID of the RDS snapshot that has been created
instanceIDstringID of the RDS instance
securityGroupID[]stringAWS Security Group IDs associated with the RDS instance
allocatedStoragestringSpecifies the allocated storage size in gibibytes (GiB)
dbSubnetGroupstringSpecifies the DB Subnet group associated with the RDS instance

Example:

yaml
actions:
  backup:
    outputArtifacts:
      backupInfo:
        keyValue:
          snapshotID: "{{ .Phases.createSnapshot.Output.snapshotID }}"
          instanceID: "{{ .Phases.createSnapshot.Output.instanceID }}"
          securityGroupID: "{{ .Phases.createSnapshot.Output.securityGroupID }}"
          allocatedStorage: "{{ .Phases.createSnapshot.Output.allocatedStorage }}"
          dbSubnetGroup: "{{ .Phases.createSnapshot.Output.dbSubnetGroup }}"
    configMapNames:
    - dbconfig
    phases:
    - func: CreateRDSSnapshot
      name: createSnapshot
      args:
        instanceID: '{{ index .ConfigMaps.dbconfig.Data "postgres.instanceid" }}'

ExportRDSSnapshotToLocation

This function spins up a temporary RDS instance from the given snapshot, extracts database dump and uploads that dump to the configured object storage.

Arguments:

ArgumentRequiredTypeDescription
instanceIDYesstringRDS db instance ID
namespaceYesstringnamespace in which to execute the Kanister tools pod for this function
snapshotIDYesstringID of the RDS snapshot
dbEngineYesstringone of the RDS db engines. Supported engine(s): PostgreSQL
usernameNostringusername of the RDS database instance
passwordNostringpassword of the RDS database instance
backupArtifactPrefixNostringpath to store the backup on the object store
databasesNo[]stringlist of databases to take backup of
securityGroupIDNo[]stringlist of securityGroupID to be passed to temporary RDS instance
dbSubnetGroupNostringDB Subnet Group to be passed to temporary RDS instance
imageNostringkanister-tools image to be used for running export job
podAnnotationsNomap[string]stringcustom annotations for the temporary pod that gets created
podLabelsNomap[string]stringcustom labels for the temporary pod that gets created

NOTE

- If databases argument is not set, backup of all the databases will be taken. - If securityGroupID argument is not set, ExportRDSSnapshotToLocation will find out Security Group IDs associated with instance with instanceID and will pass the same. - If backupArtifactPrefix argument is not set, instanceID will be used as backupArtifactPrefix. - If dbSubnetGroup argument is not set, default DB Subnet group will be used.

Outputs:

OutputTypeDescription
snapshotIDstringID of the RDS snapshot that has been created
instanceIDstringID of the RDS instance
backupIDstringunique backup id generated during storing data into object storage
securityGroupID[]stringAWS Security Group IDs associated with the RDS instance

Example:

yaml
actions:
  backup:
    outputArtifacts:
      backupInfo:
        keyValue:
          snapshotID: "{{ .Phases.createSnapshot.Output.snapshotID }}"
          instanceID: "{{ .Phases.createSnapshot.Output.instanceID }}"
          securityGroupID: "{{ .Phases.createSnapshot.Output.securityGroupID }}"
          backupID: "{{ .Phases.exportSnapshot.Output.backupID }}"
          dbSubnetGroup: "{{ .Phases.createSnapshot.Output.dbSubnetGroup }}"
    configMapNames:
    - dbconfig
    phases:

    - func: CreateRDSSnapshot
      name: createSnapshot
      args:
        instanceID: '{{ index .ConfigMaps.dbconfig.Data "postgres.instanceid" }}'

    - func: ExportRDSSnapshotToLocation
      name: exportSnapshot
      objects:
        dbsecret:
          kind: Secret
          name: '{{ index .ConfigMaps.dbconfig.Data "postgres.secret" }}'
          namespace: "{{ .Namespace.Name }}"
      args:
        namespace: "{{ .Namespace.Name }}"
        instanceID: "{{ .Phases.createSnapshot.Output.instanceID }}"
        securityGroupID: "{{ .Phases.createSnapshot.Output.securityGroupID }}"
        username: '{{ index .Phases.exportSnapshot.Secrets.dbsecret.Data "username" | toString }}'
        password: '{{ index .Phases.exportSnapshot.Secrets.dbsecret.Data "password" | toString }}'
        dbEngine: "PostgreSQL"
        databases: '{{ index .ConfigMaps.dbconfig.Data "postgres.databases" }}'
        snapshotID: "{{ .Phases.createSnapshot.Output.snapshotID }}"
        backupArtifactPrefix: test-postgresql-instance/postgres
        dbSubnetGroup: "{{ .Phases.createSnapshot.Output.dbSubnetGroup }}"
        podAnnotations:
          annKey: annValue
        podLabels:
          labelKey: labelValue

RestoreRDSSnapshot

This function restores the RDS DB instance either from an RDS snapshot or from the data dump (if [snapshotID] is not set) that is stored in an object storage.

NOTE

- If [snapshotID] is set, the function will restore RDS instance from the RDS snapshot. Otherwise backupID needs to be set to restore the RDS instance from data dump. - While restoring the data from RDS snapshot if RDS instance (where we have to restore the data) doesn't exist, the RDS instance will be created. But if the data is being restored from the Object Storage (data dump) and the RDS instance doesn't exist new RDS instance will not be created and will result in an error.

Arguments:

ArgumentRequiredTypeDescription
instanceIDYesstringRDS db instance ID
snapshotIDNostringID of the RDS snapshot
usernameNostringusername of the RDS database instance
passwordNostringpassword of the RDS database instance
backupArtifactPrefixNostringpath to store the backup on the object store
backupIDNostringunique backup id generated during storing data into object storage
securityGroupIDNo[]stringlist of securityGroupID to be passed to restored RDS instance
namespaceNostringnamespace in which to execute. Required if snapshotID is nil
dbEngineNostringone of the RDS db engines. Supported engines: PostgreSQL, aurora, aurora-mysql and aurora-postgresql. Required if snapshotID is nil or Aurora is run in RDS instance
dbSubnetGroupNostringDB Subnet Group to be passed to restored RDS instance
imageNostringkanister-tools image to be used for running restore, only relevant when restoring from data dump (if snapshotID is empty)
podAnnotationsNomap[string]stringcustom annotations for the temporary pod that gets created
podLabelsNomap[string]stringcustom labels for the temporary pod that gets created

NOTE

- If snapshotID is not set, restore will be done from data dump. In that case backupID [arg] is required. - If securityGroupID argument is not set, RestoreRDSSnapshot will find out Security Group IDs associated with instance with instanceID and will pass the same. - If dbSubnetGroup argument is not set, default DB Subnet group will be used.

Outputs:

OutputTypeDescription
endpointstringendpoint of the RDS instance

Example:

yaml
restore:
  inputArtifactNames:
  - backupInfo
  kind: Namespace
  phases:
  - func: RestoreRDSSnapshot
    name: restoreSnapshots
    objects:
      dbsecret:
        kind: Secret
        name: '{{ index .ConfigMaps.dbconfig.Data "postgres.secret" }}'
        namespace: "{{ .Namespace.Name }}"
    args:
      namespace: "{{ .Namespace.Name }}"
      backupArtifactPrefix: test-postgresql-instance/postgres
      instanceID:  "{{ .ArtifactsIn.backupInfo.KeyValue.instanceID }}"
      backupID:  "{{ .ArtifactsIn.backupInfo.KeyValue.backupID }}"
      securityGroupID:  "{{ .ArtifactsIn.backupInfo.KeyValue.securityGroupID }}"
      username: '{{ index .Phases.restoreSnapshots.Secrets.dbsecret.Data "username" | toString }}'
      password: '{{ index .Phases.restoreSnapshots.Secrets.dbsecret.Data "password" | toString }}'
      dbEngine: "PostgreSQL"
      dbSubnetGroup: "{{ .ArtifactsIn.backupInfo.KeyValue.dbSubnetGroup }}"
      podAnnotations:
        annKey: annValue
      podLabels:
        labelKey: labelValue

DeleteRDSSnapshot

This function deletes the RDS snapshot by the [snapshotID].

Arguments:

ArgumentRequiredTypeDescription
snapshotIDNostringID of the RDS snapshot

Example:

yaml
actions:
  delete:
  kind: Namespace
  inputArtifactNames:
  - backupInfo
  phases:
  - func: DeleteRDSSnapshot
    name: deleteSnapshot
    args:
      snapshotID: "{{ .ArtifactsIn.backupInfo.KeyValue.snapshotID }}"

KubeOps

This function is used to create or delete Kubernetes resources.

Arguments:

ArgumentRequiredTypeDescription
operationYesstringcreate or delete Kubernetes resource
namespaceNostringnamespace in which the operation is executed
specNostringresource spec that needs to be created
objectReferenceNomap[string]interface{}object reference for delete operation

Example:

yaml
- func: KubeOps
  name: createDeploy
  args:
    operation: create
    namespace: "{{ .Deployment.Namespace }}"
    spec: |-
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: "{{ .Deployment.Name }}"
      spec:
        replicas: 1
        selector:
          matchLabels:
            app: example
        template:
          metadata:
            labels:
              app: example
          spec:
            containers:
            - image: busybox
              imagePullPolicy: IfNotPresent
              name: container
              ports:
              - containerPort: 80
                name: http
                protocol: TCP
- func: KubeOps
  name: deleteDeploy
  args:
    operation: delete
    objectReference:
      apiVersion: "{{ .Phases.createDeploy.Output.apiVersion }}"
      group: "{{ .Phases.createDeploy.Output.group }}"
      resource: "{{ .Phases.createDeploy.Output.resource }}"
      name: "{{ .Phases.createDeploy.Output.name }}"
      namespace: "{{ .Phases.createDeploy.Output.namespace }}"

WaitV2

This function is used to wait on a Kubernetes resource until a desired state is reached. The wait condition is defined in a Go template syntax.

Arguments:

ArgumentRequiredTypeDescription
timeoutYesstringwait timeout
conditionsYesmap[string]interface{}keys should be allOf and/or anyOf with value as []Condition

Condition struct:

yaml
condition: "Go template condition that returns true or false"
objectReference:
  apiVersion: "Kubernetes resource API version"
  resource: "Type of resource to wait for"
  name: "Name of the resource"

The Go template conditions can be validated using kubectl commands with -o go-template flag. E.g. To check if the Deployment is ready, the following Go template syntax can be used with kubectl command

bash
kubectl get deploy -n $NAMESPACE $DEPLOY_NAME \
  -o go-template='{{ $available := false }}{{ range $condition := $.status.conditions }}{{ if and (eq .type "Available") (eq .status "True")  }}{{ $available = true }}{{ end }}{{ end }}{{ $available }}'

The same Go template can be used as a condition in the WaitV2 function.

Example:

yaml
- func: WaitV2
  name: waitForDeploymentReady
  args:
    timeout: 5m
    conditions:
      anyOf:
      - condition: '{{ $available := false }}{{ range $condition := $.status.conditions }}{{ if and (eq .type "Available") (eq .status "True") }}{{ $available = true }}{{ end }}{{ end }}{{ $available }}'
        objectReference:
          apiVersion: "v1"
          group: "apps"
          name: "{{ .Object.metadata.name }}"
          namespace: "{{ .Object.metadata.namespace }}"
          resource: "deployments"

Wait (deprecated)

This function is used to wait on a Kubernetes resource until a desired state is reached.

Arguments:

ArgumentRequiredTypeDescription
timeoutYesstringwait timeout
conditionsYesmap[string]interface{}keys should be allOf and/or anyOf with value as []Condition

Condition struct:

yaml
condition: "Go template condition that returns true or false"
objectReference:
  apiVersion: "Kubernetes resource API version"
  resource: "Type of resource to wait for"
  name: "Name of the resource"

NOTE

We can refer to the object key-value in Go template condition with the help of a $ prefix JSON-path syntax.

Example:

yaml
- func: Wait
  name: waitNsReady
  args:
    timeout: 60s
    conditions:
      allOf:
        - condition: '{{ if (eq "{ $.status.phase }" "Invalid")}}true{{ else }}false{{ end }}'
          objectReference:
            apiVersion: v1
            resource: namespaces
            name: "{{ .Namespace.Name }}"
        - condition: '{{ if (eq "{ $.status.phase }" "Active")}}true{{ else }}false{{ end }}'
          objectReference:
            apiVersion: v1
            resource: namespaces
            name: "{{ .Namespace.Name }}"

CreateCSISnapshot

This function is used to create CSI VolumeSnapshot for a PersistentVolumeClaim. By default, it waits for the VolumeSnapshot to be ReadyToUse.

Arguments:

ArgumentRequiredTypeDescription
nameNostringname of the VolumeSnapshot. Default value is <pvc>-snapshot-<random-alphanumeric-suffix>
pvcYesstringname of the PersistentVolumeClaim to be captured
namespaceYesstringnamespace of the PersistentVolumeClaim and resultant VolumeSnapshot
snapshotClassYesstringname of the VolumeSnapshotClass
labelsNomap[string]stringlabels for the VolumeSnapshot

Outputs:

OutputTypeDescription
namestringname of the CSI VolumeSnapshot
pvcstringname of the captured PVC
namespacestringnamespace of the captured PVC and VolumeSnapshot
restoreSizestringrequired memory size to restore PVC
snapshotContentstringname of the VolumeSnapshotContent

Example:

yaml
actions:
  backup:
    outputArtifacts:
      snapshotInfo:
        keyValue:
          name: "{{ .Phases.createCSISnapshot.Output.name }}"
          pvc: "{{ .Phases.createCSISnapshot.Output.pvc }}"
          namespace: "{{ .Phases.createCSISnapshot.Output.namespace }}"
          restoreSize: "{{ .Phases.createCSISnapshot.Output.restoreSize }}"
          snapshotContent: "{{ .Phases.createCSISnapshot.Output.snapshotContent }}"
    phases:
    - func: CreateCSISnapshot
      name: createCSISnapshot
      args:
        pvc: "{{ .PVC.Name }}"
        namespace: "{{ .PVC.Namespace }}"
        snapshotClass: do-block-storage

CreateCSISnapshotStatic

This function creates a pair of CSI VolumeSnapshot and VolumeSnapshotContent resources, assuming that the underlying real storage volume snapshot already exists. The deletion behavior is defined by the deletionPolicy property (Retain, Delete) of the snapshot class.

For more information on pre-provisioned volume snapshots and snapshot deletion policy, see the Kubernetes documentation.

Arguments:

ArgumentRequiredTypeDescription
nameYesstringname of the new CSI VolumeSnapshot
namespaceYesstringnamespace of the new CSI VolumeSnapshot
driverYesstringname of the CSI driver for the new CSI VolumeSnapshotContent
handleYesstringunique identifier of the volume snapshot created on the storage backend used as the source of the new VolumeSnapshotContent
snapshotClassYesstringname of the VolumeSnapshotClass to use

Outputs:

OutputTypeDescription
namestringname of the new CSI VolumeSnapshot
namespacestringnamespace of the new CSI VolumeSnapshot
restoreSizestringrequired memory size to restore the volume
snapshotContentstringname of the new CSI VolumeSnapshotContent

Example:

yaml
actions:
  createStaticSnapshot:
    phases:
    - func: CreateCSISnapshotStatic
      name: createCSISnapshotStatic
      args:
        name: volume-snapshot
        namespace: default
        snapshotClass: csi-hostpath-snapclass
        driver: hostpath.csi.k8s.io
        handle: 7bdd0de3-aaeb-11e8-9aae-0242ac110002

RestoreCSISnapshot

This function restores a new PersistentVolumeClaim using CSI VolumeSnapshot.

Arguments:

ArgumentRequiredTypeDescription
nameYesstringname of the VolumeSnapshot
pvcYesstringname of the new PVC
namespaceYesstringnamespace of the VolumeSnapshot and resultant PersistentVolumeClaim
storageClassYesstringname of the StorageClass
restoreSizeYesstringrequired memory size to restore PVC. Must be greater than zero
accessModesNo[]stringaccess modes for the underlying PV (Default is ["ReadWriteOnce"])
volumeModeNostringmode of volume (Default is "Filesystem")
labelsNomap[string]stringoptional labels for the PersistentVolumeClaim

NOTE

Output artifact snapshotInfo from CreateCSISnapshot function can be used as an input artifact in this function.

Example:

yaml
actions:
  restore:
    inputArtifactNames:
    - snapshotInfo
    phases:
    - func: RestoreCSISnapshot
      name: restoreCSISnapshot
      args:
        name: "{{ .ArtifactsIn.snapshotInfo.KeyValue.name }}"
        pvc: "{{ .ArtifactsIn.snapshotInfo.KeyValue.pvc }}-restored"
        namespace: "{{ .ArtifactsIn.snapshotInfo.KeyValue.namespace }}"
        storageClass: do-block-storage
        restoreSize: "{{ .ArtifactsIn.snapshotInfo.KeyValue.restoreSize }}"
        accessModes: ["ReadWriteOnce"]
        volumeMode: "Filesystem"

DeleteCSISnapshot

This function deletes a VolumeSnapshot from given namespace.

Arguments:

ArgumentRequiredTypeDescription
nameYesstringname of the VolumeSnapshot
namespaceYesstringnamespace of the VolumeSnapshot

NOTE

Output artifact snapshotInfo from CreateCSISnapshot function can be used as an input artifact in this function.

Example:

yaml
actions:
  delete:
    inputArtifactNames:
    - snapshotInfo
    phases:
    - func: DeleteCSISnapshot
      name: deleteCSISnapshot
      args:
        name: "{{ .ArtifactsIn.snapshotInfo.KeyValue.name }}"
        namespace: "{{ .ArtifactsIn.snapshotInfo.KeyValue.namespace }}"

DeleteCSISnapshotContent

This function deletes an unbounded VolumeSnapshotContent resource. It has no effect on bounded VolumeSnapshotContent resources, as they would be protected by the CSI controller.

Arguments:

ArgumentRequiredTypeDescription
nameYesstringname of the VolumeSnapshotContent

Example:

yaml
actions:
  deleteVSC:
    phases:
    - func: DeleteCSISnapshotContent
      name: deleteCSISnapshotContent
      args:
        name: "test-snapshot-content-content-dfc8fa67-8b11-4fdf-bf94-928589c2eed8"

BackupDataUsingKopiaServer

This function backs up data from a container into any object store supported by Kanister using Kopia Repository Server as data mover.

NOTE

It is important that the application includes a kanister-tools sidecar container. This sidecar is necessary to run the tools that back up the volume and store it on the object store.

Additionally, in order to use this function, a RepositoryServer CR is needed while creating the ActionSets.

Arguments:

ArgumentRequiredTypeDescription
namespaceYesstringnamespace of the container that you want to backup the data of
podYesstringpod name of the container that you want to backup the data of
containerYesstringname of the kanister sidecar container
includePathYesstringpath of the data to be backed up
snapshotTagsNostringcustom tags to be provided to the kopia snapshots
repositoryServerUserHostnameNostringuser's hostname to access the kopia repository server. Hostname would be available in the user access credential secret

Outputs:

OutputTypeDescription
backupIDstringunique snapshot id generated during backup
sizestringsize of the backup
phySizestringphysical size of the backup

Example:

yaml
actions:
backup:
  outputArtifacts:
    backupIdentifier:
      keyValue:
        id: "{{ .Phases.backupToS3.Output.backupID }}"
  phases:
  - func: BackupDataUsingKopiaServer
    name: backupToS3
    args:
      namespace: "{{ .Deployment.Namespace }}"
      pod: "{{ index .Deployment.Pods 0 }}"
      container: kanister-tools
      includePath: /mnt/data

RestoreDataUsingKopiaServer

This function restores data backed up by the BackupDataUsingKopiaServer function. It creates a new Pod that mounts the PVCs referenced by the Pod specified in the function argument and restores data to the specified path.

NOTE

It is extremely important that, the PVCs are not currently in use by an active application container, as they are required to be mounted to the new Pod (ensure by using ScaleWorkload with replicas=0 first). For advanced use cases, it is possible to have concurrent access but the PV needs to have RWX access mode and the volume needs to use a clustered file system that supports concurrent access.

ArgumentRequiredTypeDescription
namespaceYesstringnamespace of the application that you want to restore the data in
imageYesstringimage to be used for running restore job (should contain kopia binary)
backupIdentifierYesstringunique snapshot id generated during backup
restorePathYesstringpath where data to be restored
podNostringpod to which the volumes are attached
volumesNomap[string]stringmapping of [pvcName] to [mountPath] under which the volume will be available
podOverrideNomap[string]interface{}specs to override default pod specs with
repositoryServerUserHostnameNostringuser's hostname to access the kopia repository server. Hostname would be available in the user access credential secret
podAnnotationsNomap[string]stringcustom annotations for the temporary pod that gets created
podLabelsNomap[string]stringcustom labels for the temporary pod that gets created

NOTE

The image argument requires the use of ghcr.io/kanisterio/kanister-tools image since it includes the required tools to restore data from the object store.

Either pod or the volumes arguments must be specified to this function based on the function that was used to backup the data. If [BackupDataUsingKopiaServer] is used to backup the data we should specify pod and for [CopyVolumeDataUsingKopiaServer], volumes should be specified.

Additionally, in order to use this function, a RepositoryServer CR is required.

Example:

Consider a scenario where you wish to restore the data backed up by the BackupDataUsingKopiaServer function. We will first scale down the application, restore the data and then scale it back up. For this phase, we will use the backupIdentifier Artifact provided by backup function.

yaml

- func: ScaleWorkload
  name: shutdownPod
  args:
    namespace: \"{{.Deployment.Namespace }}\"
    name: \"{{ .Deployment.Name }}\"
    kind: Deployment
    replicas: 0
- func: RestoreDataUsingKopiaServer
  name: restoreFromS3
  args:
    namespace: \"{{ .Deployment.Namespace }}\"
    pod: \"{{ index.Deployment.Pods 0 }}\"
    backupIdentifier: \"{{.ArtifactsIn.backupIdentifier.KeyValue.id }}\"
    restorePath: /mnt/data
    podAnnotations:
      annKey: annValue
    podLabels:
      labelKey: labelValue
- func: ScaleWorkload
  name: bringupPod
  args:
    namespace: \"{{.Deployment.Namespace }}\"
    name: \"{{ .Deployment.Name }}\"
    kind: Deployment
    replicas: 1

DeleteDataUsingKopiaServer

This function deletes the snapshot data backed up by the BackupDataUsingKopiaServer function. It creates a new Pod that runs delete snapshot command.

NOTE

The image argument requires the use of ghcr.io/kanisterio/kanister-tools image since it includes the required tools to delete snapshot from the object store.

Additionally, in order to use this function, a RepositoryServer CR is required.

ArgumentRequiredTypeDescription
namespaceYesstringnamespace in which to execute the delete job
backupIDYesstringunique snapshot id generated during backup
imageYesstringimage to be used for running delete job (should contain kopia binary)
repositoryServerUserHostnameNostringuser's hostname to access the kopia repository server. Hostname would be available in the user access credential secret
podAnnotationsNomap[string]stringcustom annotations for the temporary pod that gets created
podLabelsNomap[string]stringcustom labels for the temporary pod that gets created

Example:

Consider a scenario where you wish to delete the data backed up by the BackupDataUsingKopiaServer function. For this phase, we will use the backupIdentifier Artifact provided by backup function.

yaml
- func: DeleteDataUsingKopiaServer
  name: DeleteFromObjectStore
  args:
    namespace: "{{ .Deployment.Namespace }}"
    backupID: "{{ .ArtifactsIn.backupIdentifier.KeyValue.id }}"
    image: ghcr.io/kanisterio/kanister-tools:0.89.0
    podAnnotations:
      annKey: annValue
    podLabels:
      labelKey: labelValue

Registering Functions

Kanister can be extended by registering new Kanister Functions.

Kanister Functions are registered using a similar mechanism to database/sql drivers. To register new Kanister Functions, import a package with those new functions into the controller and recompile it. -->