Deploy 4PointX on K8s¶

Log in to the K8S Master VM and switch to 4px Linux User:

sudo su - 4px

Modify the containerd.service configuration to allow Kubernetes to access the Docker registry.¶

Containerd Changes to connect with 4PX Image Repository

Open containerd Configuration file:

sudo vi /etc/containerd/config.toml

In case the file is blank, run:

sudo mkdir -p /etc/containerd/
containerd config default | sudo tee /etc/containerd/config.toml
sudo sed -i 's/SystemdCgroup \= false/SystemdCgroup \= true/g' /etc/containerd/config.toml

Edit/replace the configuration to access the insecure registry registry.4pointx.com:5000:*Depreciating Soon* Note: The line number could be different depending on the OS & Other configurations. You can try finding “config_path“ and matching the indentation.

[plugins."io.containerd.grpc.v1.cri".registry]
  config_path = ""

  [plugins."io.containerd.grpc.v1.cri".registry.auths]

  [plugins."io.containerd.grpc.v1.cri".registry.configs]
    [plugins."io.containerd.grpc.v1.cri".registry.configs."registry.4pointx.com:5000".tls]
      insecure_skip_verify = true
    [plugins."io.containerd.grpc.v1.cri".registry.configs."localhost:5000".tls]
      insecure_skip_verify = true
  [plugins."io.containerd.grpc.v1.cri".registry.headers]

  [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
    [plugins."io.containerd.grpc.v1.cri".registry.mirrors."registry.4pointx.com:5000"]
      endpoint = ["https://registry.4pointx.com:5000"]

Restart containerd
sudo systemctl restart containerd
Repeat the above steps on all Kubernetes nodes. (You can reuse the same containerd file in all nodes)

Configuring Worker Nodes For OpenSearch.¶

Note

Note: If Pod scheduling is enabled on the master. This step has to be applied to the Master node too.

Run the following command on each worker node:Ref Link: OS Installation Important Settings
Open the /etc/sysctl.conf file in your preferred text editor with superuser privileges. For example, using vi
sudo vi /etc/sysctl.conf
Add the following line to the file to set vm.max_map_count to 262144:
vm.max_map_count=262144
Save the file and reload the sysctl configuration to apply the changes:
sudo sysctl -p

Preparing SSL Certificate (in master only).¶

Before deploying 4pointX, ensure there is an SSL certificate. If not, create it using the following command, or rename .crt to __4pointx_com.crt and .key to server.key:

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -subj "/C=IN/ST=KN/L=BN/O=EV/OU=4PX/CN=*.4pointx.com" -keyout server.key -out __4pointx_com.crt

Create a 4px Namespace in the K8S Cluster.¶

Create the namespace from the K8S Master Node (VM):

kubectl create namespace 4px

Verify the namespace creation:

kubectl get namespace

OSD Secret.¶

Create OSD secret using the below command where __4pointx_com.crt & server.key is present.

kubectl create secret generic osd --from-file=__4pointx_com.crt --from-file=server.key -n 4px

Docker Registry Secret¶

kubectl create secret docker-registry 4px-registry-creds  --docker-server=registry.4pointx.com:5000  --docker-username=""  --docker-password="" -n 4px

*Contact the 4pointx admin for Docker username and password

Kafka TLS Certificate Setup¶

Kafka requires external clients and connectors to use SASL_SSL authentication for encrypted communication.

Generate Private Key & Certificate Signing Request (CSR):

openssl req -new -newkey rsa:4096 -keyout kafka-broker.key -out kafka-broker.csr -nodes -subj "/CN=*.4pointx.com"

Create Subject Alternative Name File (update IP accordingly):

cat > kafka-broker.ext <<EOF
subjectAltName = IP:34.125.17.198, DNS:*.4pointx.com, DNS:kafka-broker, DNS:localhost
EOF

Generate the Self-Signed Certificate

openssl x509 -req -days 365 -in kafka-broker.csr -signkey kafka-broker.key -out kafka-broker.crt -extfile kafka-broker.ext

Create JKS Keystore and TruststoreFirst, install Java (required for keytool command)

sudo dnf install java-11-openjdk-devel

openssl pkcs12 -export -in kafka-broker.crt -inkey kafka-broker.key -out kafka-broker.p12 -name kafka-broker -password pass:KafkaSecret123!
keytool -importkeystore -destkeystore kafka.keystore.jks -srckeystore kafka-broker.p12 -srcstoretype PKCS12 -alias kafka-broker -deststorepass KafkaSecret123! -srcstorepass KafkaSecret123!

Create a Truststore for Clients

keytool -import -trustcacerts -file kafka-broker.crt -alias CARoot -keystore kafka.truststore.jks -storepass KafkaSecret123! -noprompt

Convert the truststore file to PEM format for Python clients.

keytool -exportcert -alias caroot -keystore kafka.truststore.jks -file ca-cert.der
openssl x509 -inform DER -in ca-cert.der -out ca-cert.pem

Create Kubernetes secrets from the ca-cert.pem file. This file mounts to the config pod and is shared with the Gateway during the handshake.
1. kubectl create secret generic client-pem-secret –from-file=ca-cert.pem=ca-cert.pem -n 4px
Create Kubernetes TLS Secrets
1. kubectl create secret generic kafka-tls-secret
–from-file=kafka.keystore.jks=kafka.keystore.jks –from-file=kafka.truststore.jks=kafka.truststore.jks -n 4px
1. kubectl create secret generic kafka-passwords
–from-literal=keystore-password=KafkaSecret123! –from-literal=truststore-password=KafkaSecret123! –from-literal=key-password=dummy -n 4px

Deploying 4PX Setup¶

Create an Object Definition File 4px_setup.yaml
vi 4px_setup.yaml
Add relevant GitLab token, Kafka Proxy, image name(master v/s test)
Example content for 4px_setup.yaml:Note: Change the Version and Persisted Volume size for the respective module.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: k8s-setup
  namespace: 4px
spec:
  replicas: 1
  selector:
    matchLabels:
      app: k8s-setup
  template:
    metadata:
      labels:
        app: k8s-setup
    spec:
      serviceAccount: k8s-setup
      serviceAccountName: k8s-setup
      containers:
        - name: setup-k8s
          image: registry.4pointx.com:5000/4pointx/setup_kubernetes:master
          imagePullPolicy: Always
          env:
            - name: SUBDOMAIN
              value: "demo"
            - name: KI_HOST
              value: "osd-opensearch-dashboards"
            - name: REGISTRY_URL
              value: "registry.4pointx.com:5000"
            - name: VERSION
              value: "master"
            - name: KAFKA_BROKER_DATA_VOLUME_SIZE
              value: "4Gi"
            - name: KAFKA_BROKER_LOGS_VOLUME_SIZE
              value: "4Gi"
            - name: KAFKA_BROKER_REPLICAS
              value: "2"
            #The number of nodeports should be equal to the number of replicas.
            #The values should be separated by comma and in "".
            - name: KAFKA_BROKER_NODEPORT
              value: "32005, 32006"
            - name: KAFKA_BROKER_NODE_SERVICE_ADDRESS
              # Initiate value from kafka-broker-0 to kafka-broker-(numberOfKafkaBrokerReplicas-1).
              # Example value: "kafka-broker-0.kafka-broker-headless.4px.svc.cluster.local:9094, kafka-broker-1.kafka-broker-headless.4px.svc.cluster.local:9094"
              value: "kafka-broker-0.kafka-broker-headless.4px.svc.cluster.local:9094, kafka-broker-1.kafka-broker-headless.4px.svc.cluster.local:9094"
            - name: KAFKA_BROKER_HOST_NAME
              value: "34.125.120.4"
            - name: KAFKA_ZOOKEEPER_VOLUME_SIZE
              value: "4Gi"
            - name: KAFKA_ZOOKEEPER_REPLICAS
              value: "1"
            - name: OPENSEARCH_VOLUME_SIZE
              value: "35Gi"
            - name: OPENSEARCH_REPLICAS
              value: "2"
            - name: OPENSEARCH_HEAP_SIZE_MB
              value: "4096"
            - name: STORAGE_CLASS
              value: "longhorn"
            - name: GITLAB_TOKEN
              value: ""
            - name: OPENSEARCH_HOST
              value: opensearch-cluster-master
            - name: OPENCSEARCH_PORT
              value: "9200"
            - name: SLEEP_INTERVAL
              value: "60"
            - name: KAFKA_TOPIC_PARTITIONS
              value: "2"
            - name: KAFKA_TOPIC_REPLICAS
              value: "2"
          command: ["/bin/bash"]
          args:
            - "-c"
            - "/app/run.sh"
      imagePullSecrets:
        - name: 4px-registry-creds
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: k8s-setup
  namespace: 4px
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: k8s-setup-role
  namespace: 4px
rules:
  - apiGroups: ["*"]
    resources: ["*"]
    verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: k8s-setup-role-binding
  namespace: 4px
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: k8s-setup-role
subjects:
  - kind: ServiceAccount
    name: k8s-setup
    namespace: 4px
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: k8s-setup-cluster-role
rules:
  - apiGroups: ["*"]
    resources: ["*"]
    verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: k8s-setup-cluster-role-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: k8s-setup-cluster-role
subjects:
  - kind: ServiceAccount
    name: k8s-setup
    namespace: 4px
---
apiVersion: v1
kind: Service
metadata:
  name: k8s-setup
  namespace: 4px
spec:
  selector:
    app: k8s-setup
  type: NodePort
  ports:
    - protocol: TCP
      port: 80
      nodePort: 32700
      targetPort: 8090

Commands:

Create the 4px_setup.yaml file and paste the above content on the K8S Master Node.

Apply the 4px_setup.yaml file using kubectl:.
kubectl apply -f 4px_setup.yaml

Check the status of all modules.

kubectl get pods -n 4px
#Add -w to the above command to see live status

Note: By default, there will be two Kafka brokers and Opensearch, one Zookeeper, and Opensearch-Dashboard. To scale up and scale down, Refer To the Resize 4PX Modules Page under the Administration Tab.

Testing the 4PX OSD¶

Test 4px OpenSearch Dashboards is deployed and running.

Run the following command on the K8S Master Node to check if it's accessible:

curl -vk https://localhost:32000