Installing the edge cluster agent

Introduction

This content provides instructions on installing the cluster agent on common Kuberentes distributions. It applies to both single worker node clusters as well as multi-node clusters.

Pre-requisites

Architecture must be either x86_64 or arm64
Operating system must be modern Linux variant with 64-bit and systemd support

Note: Open Horizon cluster agent installation requires cluster admin access on the edge cluster. Additionally, the “jq” command-line JSON processor must be installed prior to running the agent install script.

Installing

Begin by installing the Open Horizon agent on one of these types of Kubernetes edge clusters:

Installing agent on OCP Kubernetes edge cluster
Installing agent on k3s and microk8s edge clusters

Then, deploy an edge service to your edge cluster:

Deploying services to your edge cluster

If you need to remove the agent:

Removing agent from edge cluster

Installing agent on OCP Kubernetes edge cluster

This content describes how to install the Open Horizon agent on your OCP edge cluster. Follow these steps on a host that has admin access to your edge cluster:

oc login https://<api_endpoint_host>:<port> -u <admin_user> -p <admin_password> --insecure-skip-tls-verify=true

If you have not completed the steps in Creating your API key, do that now. This process creates an API key, locates some files, and gathers environment variable values that are needed when you set up edge nodes. Set the same environment variables for this edge cluster. Set the HZN_NODE_ID of the edge cluster.
```
export HZN_EXCHANGE_USER_AUTH=iamapikey:<api-key>
export HZN_ORG_ID=<your-exchange-organization>
export HZN_FSS_CSSURL=https://<management-hub-ingress>/edge-css/
export HZN_NODE_ID=<edge-cluster-node-name>
```
Set the agent namespace variable to its default value (or whatever namespace you want to explicitly install the agent into):
```
export AGENT_NAMESPACE=openhorizon-agent
```
Set the storage class that you want the agent to use - either a built-in storage class or one that you created. You can view the available storage classes with the first of the following two commands, then substitute the name of the one you want to use into the second command. One storage class should be labeled (default):
```
oc get storageclass
export EDGE_CLUSTER_STORAGE_CLASS=<rook-ceph-cephfs-internal>
```
Determine whether a default route for the OpenShift image registry has been created so that it is accessible from outside of the cluster:
```
oc get route default-route -n openshift-image-registry --template=''
```
If the command response indicates the default-route is not found, you need to expose it (see Exposing the registry for details):
```
oc patch configs.imageregistry.operator.openshift.io/cluster --patch '{"spec":{"defaultRoute":true}}' --type=merge
```

Retrieve the repository route name that you need to use:

export OCP_IMAGE_REGISTRY=`oc get route default-route -n openshift-image-registry --template=''`

Create a new project to store your images:

export OCP_PROJECT=$AGENT_NAMESPACE
oc new-project $OCP_PROJECT

Create a service account with a name of your choosing:

export OCP_USER=<service-account-name>
oc create serviceaccount $OCP_USER

Add a role to your service account for the current project:

oc policy add-role-to-user edit system:serviceaccount:$OCP_PROJECT:$OCP_USER

Set your service account token to the following environment variable:
```
export OCP_TOKEN=`oc serviceaccounts get-token $OCP_USER`
```

Get the OpenShift certificate and configure Docker to trust it:

echo | openssl s_client -connect $OCP_IMAGE_REGISTRY:443 -showcerts | sed -n "/-----BEGIN CERTIFICATE-----/,/-----END CERTIFICATE-----/p" > ca.crt

On Linux:

mkdir -p /etc/docker/certs.d/$OCP_IMAGE_REGISTRY
cp ca.crt /etc/docker/certs.d/$OCP_IMAGE_REGISTRY
systemctl restart docker.service

On macOS:

mkdir -p ~/.docker/certs.d/$OCP_IMAGE_REGISTRY
cp ca.crt ~/.docker/certs.d/$OCP_IMAGE_REGISTRY

On macOS, use the Docker Desktop icon on the right side of the desktop menu bar to restart Docker by clicking Restart in the dropdown menu.

echo "$OCP_TOKEN" | docker login -u $OCP_USER --password-stdin $OCP_IMAGE_REGISTRY

Configure additional trust stores for image registry access:

oc create configmap registry-config --from-file=$OCP_IMAGE_REGISTRY=ca.crt -n openshift-config

Edit the new registry-config:

oc edit image.config.openshift.io cluster

Update the spec: section:

spec:
  additionalTrustedCA:
    name: registry-config

The agent-install.sh script stores the Open Horizon agent in the edge cluster container registry. Set the registry user, password, and full image path (minus the tag):

export EDGE_CLUSTER_REGISTRY_USERNAME=$OCP_USER
export EDGE_CLUSTER_REGISTRY_TOKEN="$OCP_TOKEN"
export IMAGE_ON_EDGE_CLUSTER_REGISTRY=$OCP_IMAGE_REGISTRY/$OCP_PROJECT/amd64_anax_k8s

Note: The Open Horizon agent image is stored in the local edge cluster registry because the edge cluster Kubernetes needs ongoing access to it, in case it needs to restart it or move it to another pod.

Download the agent-install.sh script from the Cloud Sync Service (CSS) and make it executable:

curl -u "$HZN_ORG_ID/$HZN_EXCHANGE_USER_AUTH" -k -o agent-install.sh $HZN_FSS_CSSURL/api/v1/objects/IBM/agent_files/agent-install.sh/data
chmod +x agent-install.sh

Run agent-install.sh to get the necessary files from CSS, install and configure the Horizon agent, and register your edge cluster with policy:
```
./agent-install.sh -D cluster -i 'css:'
```

Notes:

To see all of the available flags, run: ./agent-install.sh -h
If an error causes agent-install.sh to fail, correct the error and run agent-install.sh again. If that does not work, run agent-uninstall.sh (see Removing agent from edge cluster) before running agent-install.sh again.

Change to the agent namespace (also known as project) and verify that the agent pod is running:
```
oc project $AGENT_NAMESPACE
oc get pods
```

Now that the agent is installed on your edge cluster, you can run these commands if you want to familiarize yourself with the Kubernetes resources associated with the agent:

oc get namespace $AGENT_NAMESPACE
oc project $AGENT_NAMESPACE   # ensure this is the current namespace/project
oc get deployment -o wide
oc get deployment agent -o yaml   # get details of the deployment
oc get configmap openhorizon-agent-config -o yaml
oc get secret openhorizon-agent-secrets -o yaml
oc get pvc openhorizon-agent-pvc -o yaml   # persistent volume

Often, when an edge cluster is registered for policy, but does not have user-specified node policy, none of the deployment policies will deploy edge services to it. That is the case with the Horizon examples. Proceed to Deploying services to your edge cluster to set node policy so that an edge service will be deployed to this edge cluster.

Installing agent on K3s and microk8s edge clusters

This content describes how to install the Open Horizon agent on K3s or microk8s , lightweight and small Kubernetes clusters:

Log in to your edge cluster as root.
If you have not completed the steps in Creating your API key, do that now. This process creates an API key, locates some files, and gathers environment variable values that are needed when setting up edge nodes. Set the same environment variables on this edge cluster:
```
export HZN_EXCHANGE_USER_AUTH=iamapikey:<api-key>
export HZN_ORG_ID=<your-exchange-organization>
export HZN_FSS_CSSURL=https://<ieam-management-hub-ingress>/edge-css/
```
Copy the agent-install.sh script to your new edge cluster.
The agent-install.sh script will store the Open Horizon agent in the edge cluster image registry. Set the full image path (minus the tag) that should be used. For example:
- On k3s:
```
 REGISTRY_ENDPOINT=$(kubectl get service docker-registry-service | grep docker-registry-service | awk '{print $3;}'):5000
 export IMAGE_ON_EDGE_CLUSTER_REGISTRY=$REGISTRY_ENDPOINT/openhorizon-agent/amd64_anax_k8s
```
- On microk8s:
```
 export IMAGE_ON_EDGE_CLUSTER_REGISTRY=localhost:32000/openhorizon-agent/amd64_anax_k8s
```
Note: The Open Horizon agent image is stored in the local edge cluster registry because the edge cluster Kubernetes needs ongoing access to it, in case it needs to restart it or move it to another pod.

Instruct agent-install.sh to use the default storage class:

On k3s:

 export EDGE_CLUSTER_STORAGE_CLASS=local-path

On microk8s:

 export EDGE_CLUSTER_STORAGE_CLASS=microk8s-hostpath

Run agent-install.sh to get the necessary files from CSS (Cloud Sync Service), install and configure the Horizon agent, and register your edge cluster with policy:
```
./agent-install.sh -D cluster -i 'css:'
```
Notes:
- To see all of the available flags, run: ./agent-install.sh -h
- If an error occurs causing agent-install.sh to not complete successfully, correct the error that is displayed, and run agent-install.sh again. If that does not work, run agent-uninstall.sh (see Removing agent from edge cluster) before running agent-install.sh again.

Verify that the agent pod is running:

kubectl get namespaces
kubectl -n openhorizon-agent get pods

Usually, when an edge cluster is registered for policy, but does not have any user-specified node policy, none of the deployment policies deploy edge services to it. This is expected. Proceed to Deploying services to your edge cluster to set node policy so that an edge service will be deployed to this edge cluster.

Deploying services to your edge cluster

Setting node policy on this edge cluster can cause deployment policies to deploy edge services here. This content shows an example of doing that.

Set some aliases to make it more convenient to run the hzn command. (The hzn command is inside the agent container, but these aliases make it possible to run hzn from this host.)

cat << 'END_ALIASES' >> ~/.bash_aliases
alias getagentpod='kubectl -n openhorizon-agent get pods --selector=app=agent -o jsonpath={.items[].metadata.name}'
alias hzn='kubectl -n openhorizon-agent exec -i $(getagentpod) -- hzn'
END_ALIASES
source ~/.bash_aliases

Verify that your edge node is configured (registered with the Open Horizon management hub):
```
hzn node list
```
To test your edge cluster agent, set your node policy with a property that deploys the example helloworld operator and service to this edge node:
```
cat << 'EOF' > operator-example-node.policy.json
{
  "properties": [
    { "name": "openhorizon.example", "value": "nginx-operator" }
  ]
}
EOF

cat operator-example-node.policy.json | hzn policy update -f-
hzn policy list
```
Note:
- Because the real hzn command is running inside the agent container, for any hzn commands that require an input file, you need to pipe the file into the command so its content will be transferred into the container.
After a minute, check for an agreement and the running edge operator and service containers:
```
hzn agreement list
kubectl -n openhorizon-agent get pods
```

Using the pod IDs from the previous command, view the log of edge operator and service:

kubectl -n openhorizon-agent logs -f <operator-pod-id>
# control-c to get out
kubectl -n openhorizon-agent logs -f <service-pod-id>
# control-c to get out

You can also view the environment variables that the agent passes to the edge service:
```
kubectl -n openhorizon-agent exec -i <service-pod-id> -- env | grep HZN_
```

Changing what services are deployed to your edge cluster

To change what services are deployed to your edge cluster, change the node policy:
```
cat <new-node-policy>.json | hzn policy update -f-
hzn policy list
```
After a minute or two the new services will be deployed to this edge cluster.
Note: On some VMs with microk8s, the service pods that are being stopped (replaced) might stall in the Terminating state. If that happens, run:
```
kubectl delete pod <pod-id> -n openhorizon-agent --force --grace-period=0
pkill -fe <service-process>
```
If you want to use a pattern instead of a policy to run services on your edge cluster:
```
hzn unregister -f
hzn register -n $HZN_EXCHANGE_NODE_AUTH -p <pattern-name>
```