This is the minimum list of hardening and other steps that need to be performed to secure the Linux server containing the DIVOC platform. The assumption is that the installation happens on a bare metal setup. While the concepts remain the same, the methodology might differ for commercial cloud setups.

Check for open ports

• Identifying open connections to the internet is a critical mission.

• Once you have identified the open ports, you can stop/purge the applications which keep unnecessary ports open.

• The only acceptable open ports are 22 and 443. Access to the other ports outside the Kubernetes network should be prohibited.

• All ingress should be routed through the 443 port only as needed.

Secure SSH

SSH is secure, but we need to harden this service as well. If we can disable SSH, then the problem is solved. However, if we want to use it, we have to change the default configuration of SSH. Password-based authentication should be disabled and only key-based authentication should be allowed. The steps for creating sudo users with public and private keys are as follows:

• Create a non-root sudo user

• Add the user to sudo users group

• Login to the machine as the user

• Create SSH directory with appropriate permissions

• Generate a key pair for the protocol, and run:

• Share the public key created with users you expect to connect to the server

• You can modify your SSH configuration to be more secure by performing the following changes to the configuration file:

• Make sure that root cannot login remotely through SSH

• Allow some specific users

• Enable public key-based authentication and disable password-based authentication

• There are some additional options that must exist in the “sshd_config” file: MaxAuthTries 5

• Finally, set the permissions on the sshd_config file so that only root users can change its contents:

Kubernetes

Recovering control plane

• To recover from broken nodes in the control plane, use the "recover-control-plane.yml" playbook.

• Back up what you can.

• Provision new nodes to replace the broken ones.

• Place the surviving nodes of the control plane first in the "etcd" and "kube_control_plane" groups.

• Add the new nodes below the surviving control plane nodes in the "etcd" and "kube_control_plane" groups.

Examples of what broken means in this context:

• One or more bare metal node(s) suffer from unrecoverable hardware failure.

• One or more node(s) fail during patching or upgrading.

• Etcd database corruption.

• Other node-related failures that leave your control plane degraded or nonfunctional.

• Note: You need at least one functional control plane node to be able to recover using this method. If all control planes go down, there is no scope of recovery, and you will have to reinstall Kubernetes. Typically, even if the control plane goes down, the application still functions. Kubernetes functions like scaling, creating new pods, upgrading deployments, etc. will not work. The application, as is available, will continue to function.

Runbook

• Move any broken etcd nodes into the "broken_etcd" group, make sure the "etcd_member_name" variable is set.

• Move any broken control plane nodes into the "broken_kube_control_plane" group.

• Run the playbook with --limit etcd,kube_control_plane, and increase the number of etdc retries by setting -e etcd_retries=10 or something even larger. The amount of retries required is difficult to predict.

• Once you are done, you should have a fully working control plane again.

Recover from the last quorum

• The playbook attempts to figure out if the etcd quorum is intact. If the quorum is lost, it will attempt to take a snapshot from the first node in the "etcd" group and restore from that.

• To restore from an alternate snapshot, set the path to that snapshot in the "etcd_snapshot" variable: -e etcd_snapshot=/tmp/etcd_snapshot.

Adding or replacing a node

Removal of first kube_control_plane and etcd-master

Currently, you cannot remove the first node in your kube_control_plane and etcd-master list. If you still want to remove this node, you have to do the following:

1. Modify the order of your control plane list by pushing your first entry to any other position, such as if you want to remove node-1 of the following example:

2. Run upgrade-cluster.yml or cluster.yml. After this, you are good to go on with the removal.

Adding or replacing a worker node

1. Add a new node to the inventory.

2. Run scale.yml. You can use --limit=NODE_NAME to limit Kubespray to avoid disturbing other nodes in the cluster. Before using --limit, run playbook facts.yml without the limit to refresh facts cache for all nodes.

3. Remove an old node with remove-node.yml. With the old node still in the inventory, run remove-node.yml. You need to pass -e node=NODE_NAME to the playbook to limit the execution to the node being removed. If the node you want to remove is not online, you should add reset_nodes=false and allow_ungraceful_removal=true to your extra-vars: -e node=NODE_NAME -e reset_nodes=false -e allow_ungraceful_removal=true. Use this flag even when you remove other types of nodes like a control plane or etcd nodes.

4. Remove node from the inventory.

Adding or replacing a control plane node

1. Append the new host to the inventory and run cluster.yml. You cannot use scale.yml for that.

2. In all hosts, restart nginx-proxy pod. This pod is a local proxy for the apiserver. Kubespray will update its static config, but it needs to be restarted to reload:

3. With the old node still in the inventory, run remove-node.yml. You need to pass -e node=NODE_NAME to the playbook to limit the execution to the node being removed. If the node you want to remove is not online, you should add reset_nodes=false and allow_ungraceful_removal=true to your extra-vars.

Assumptions

• Use a Debian-based Linux distribution (preferably Ubuntu)

• Experience in running simple shell and bash commands

Pre-requisites

• Debian-based OS (Ubuntu)

• sshpass

• Ansible

• GIT

• kubectl

• List of servers and ability to access them using key-based authentication

• Server map to list servers against software

• Access to the DIVOC installer repository

• Access to the implementation-specific DIVOC code

Suggested servers for HA setup

The sizing and count of the servers can change based on the load requirements. However, for a truly HA setup, the following are the minimum requirements:

Postgres and etcd

3 servers for HA setup: one master and 2 replicas. The etcd cluster can also be set up on the same servers.

Kubernetes

6 servers: 3 for control plane (or master node can be relatively smaller sized instances) and 3 worker nodes (for deploying the application).

Kafka and Zookeeper

3 servers containing both Zookeeper and Kafka (Ideally Zookeeper and Kafka need to be installed on separate servers but we should be fine to install both on the same machine).

Redis

3 servers

Elasticsearch

3 servers

Docker-registry

1 server

Overview

There are three scripts that need to be run to complete the DIVOC installation process:

1. Installing the prerequisites and setting various hardware clusters as detailed above.

2. Building the pushing the docker images to the appropriate registry.

3. Deploying code from the registry into Kubernetes cluster.

Installation dependencies

1. Clone the repository available at https://github.com/egovernments/divoc-installer.

2. Create an inventory file from the sample inventory file located at https://github.com/egovernments/divoc-installer/blob/master/inventory.example.ini.

3. Add the inventory details as per the comments present in the file.

4. Run the install.sh present within the divoc-installer with the elevated privileges (we can also use nohup for running in the background):

• It will install the dependencies like python3, ansible, etc.

• It will install the applications and configure them on the servers mentioned in the inventory file.

Build Docker images

• Run the build.sh file with elevated privileges.

• Default values for the Docker repository are from dockerhub.

• The Default value for the GIT repo is the master branch of the https://github.com/egovernments/DIVOC.git.

Install DIVOC application

1. The sample default Kubernetes deployment files are available at https://github.com/egovernments/divoc-installer/tree/master/kube-deployment-config-example.

2. Make a copy of the folder and change the internal script files to have the following configurations. It is recommended that you maintain your own configuration in a separate Github repository so that you have version control and backup (you require only the example folder, not the full repository).

a. Within the divoc-installer director, open the divoc-config.yaml file present within the deployment configuration directory and make the following changes:

- DB_HOST

- DB_USER

- DB_PASS

- DB_PORT

- KAFKA_BOOTSTRAP_SERVERS

- REDIS_URL

- CLICKHOUSE_URL

- AUTH_PRIVATE_KEY

- AUTH_PUBLIC_KEY

- CERTIFICATE_NAMESPACE

- CERTIFICATE_NAMESPACE_V2

- CERTIFICATE_CONTROLLER_ID

- CERTIFICATE_PUBKEY_ID

- CERTIFICATE_DID

- CERTIFICATE_ISSUER

- CERTIFICATE_BASE_URL

- CERTIFICATE_FEEDBACK_BASE_URL

- CERTIFICATE_INFO_BASE_URL

- CERTIFICATE_PUBLIC_KEY

- CERTIFICATE_PRIVATE_KEY

- CITIZEN_PORTAL_URL

b. Modify registry-deployment.yaml to change the following:

- connectionInfo_password

- connectionInfo_uri

- connectionInfo_username

- elastic_search_enabled

- registry_base_apis_enable

- taskExecutor_index_queueCapacity

- auditTaskExecutor_queueCapacity

- Signature_enabled

c. Modify keycloak-deployment.yaml to add the following information:

- DB_ADDR

- DB_DATABASE

- DB_PASSWORD

- DB_PORT

- DB_USER

- DB_VENDOR

- KEYCLOAK_USER

- KEYCLOAK_PASSWORD

- ENABLE_OTP_MESSAGE

- KAFKA_BOOTSTRAP_SERVERS

3. Run the deploy script to deploy the application on Kubernetes.

Post installation

Create indexes on database tables

The indexes for efficient querying of the database tables do not get automatically created and hence need to be created manually. Execute registry_index.sql is present within the DIVOC codebase on the database. A restart of the registry service is required for this change to reflect.

Note: Database tables are only created when the first API request is received.

Overview

This is the generic solution for backup and restore. Depending on the backup strategy used, the tools might change.

Backing up and restoring Postgres

• The Ansible script will automatically configure pg_basebackup, pgbackrest, wal-g and other recovery tools. For the sake of simplicity, we can use pg_dump and pg_restore.

• The following command takes a backup. This will create a compressed tarball backup in the directory mentioned:

• This can be scheduled using a cron as shown below:

• Pg_basebackup is installed along with psql client

• You can restore from pg_dump as follows:

Backing up and restoring Clickhouse

The plan is to use clickhouse-backup, which is open sourced under the liberal MIT license. This tool has the ability to create archived backups and upload them to NFS, S3, GCS, AZBlob, SFTP and other remote data repositories.

• Download the latest release from https://github.com/AlexAkulov/clickhouse-backup/releases

• Untar the archive

• Create a configuration file as follows, call it config.ini

• Ensure configuration under clickhouse and general section of the configuration file. The rest are not mandatory.

• If automated remote upload functionality is needed, the appropriate section needs to be filled in: sftp, ftp, s3, GCS, AZBlob, etc.

• The following command can be run:

• The following is the list of possible commands which can be executed:

Backing up and restoring Kafka

• Backup zookeeper state data

- Go to file

- Copy location of dataDir property (typically, /tmp/zookeeper)

- Run the following command:

• Backup Kafka topics and messages

- Go to the file kafka/config/server.properties

- Copy location of log.dirs (typically, /tmp/kafka-logs)

- Stop Kafka:

- Login as kafka user:

- Run the following command:

• Restore zookeeper

- sudo systemctl stop kafka

- sudo systemctl stop zookeeper

- sudo -iu kafka

- rm -r /tmp/zookeeper/*

- tar -C /tmp/zookeeper -xzf /home/kafka/zookeeper-backup.tar.gz

--strip-components 2

• Restore kafka

- rm -r /tmp/kafka-logs/*

- tar -C /tmp/kafka-logs -xzf /home/kafka/kafka-backup.tar.gz --strip-components 2

- sudo systemctl start kafka

- sudo systemctl start zookeeper

• Verification of Restoration

- ~/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic

BackupTopic --from-beginning

Backing up and restoring Redis

Redis provides an in-built command to save a backup.

• Install redis-cli using

• The following command takes a backup of the redis-server:

• This will save the backup as dump.rdb within:

Restoration can be done in the following way:

• Locate the redis data directory, typically:

• Move the dump.rdb file into this folder

• Start redis server

• This will ensure that data is restored automatically

This section will cover the following:

• How to install DIVOC

• Backup & Restore: Postgres, Clickhouse, Kafka, & Redis

• Infrastructure Recovery

• Server Hardening

All content on this page by eGov Foundation is licensed under a Creative Commons Attribution 4.0 International License.