# Kubernetes on AWS (via Terraform) Documentation

Andrew Lee (andrew@voiceit.io)

# Abstract

Please use the following documentation to set up a VoiceIt Private Cloud Mode compatible Kubernetes installation on AWS.

Much of the set up is automated via Terraform scripting, but there are still some manual steps involved.

The following steps will need to create the following types of resources in AWS (not an exhaustive list):

  • Elastic Kubernetes Service (EKS)
  • Relational Database Service (RDS)
  • EC2
  • Elastic Block Storage (EBS)
  • Virtual Private Cloud (VPC)/Subnets
  • Elastic Load Balancer (ELB)
  • Security Groups

# Assumptions

You will also need an account on voiceit.io in order to access the Licenses and Files section of Dashboard to retrieve files. This scripting only works on Linux or macOS.

# Obtaining a License Key

Please contact VoiceIt to receive a license key, making sure that you notify us of any VoiceIt developer on your team (i.e. accounts that have logins to voiceit.io) you would like to have access to the license information on Dashboard. Once the license key gets created, you should be able to view and download the initial Kubernetes configuration files needed to start the system directly from Dashboard.

For the most part, most security sensitive environment variable fields (such as license key) have already been filled out for your convenience in these Kubernetes configuration files.

dashboard_licenses dashboard_files

# Dependencies

Please install the following programs from their respective official installation documentation.

Name Command Installation Instructions Special Notes
Terraform CLI terraform https://learn.hashicorp.com/tutorials/terraform/install-cli
CLI for Amazon EKS eksctl https://docs.aws.amazon.com/eks/latest/userguide/eksctl.html
AWS CLI aws https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html Depends on python3
Kubernetes CLI kubectl https://kubernetes.io/docs/tasks/tools/
Helm 3 helm https://helm.sh/docs/intro/install/

# Docker Hub

Please sign up for a Docker Hub user account (rather than an organization account) and please send us the username via email to support@voiceit.io.

We can then grant you the pull access to all our private containers we developed for VoiceIt PCM.

# Grab a Domain name from Route 53

This domain will be used to host VoiceIt resources to public endpoints.

In AWS Web Console, please go search for "Route 53" in the top of the page, then go to "Domains -> Registered domains" on the left hand column. then the blue button "Register Domain".

# Get a Public TLS Cert (Please wait until after your Route 53 domain registers with the registrar.)

This will be used by Elastic Load Balancers to create TLS encrypted endpoints.

In AWS Web Console, please go search for "ACM" (Certificate Manager), in the top of the page, then click on the orange button on the top right of the page "Request".

  • Request a public certificate
  • Next
  • In Domain names, please add the root cert and the wildcard cert (Add another name to this certificate). For example, for voiceit.io, I would need to add the fully qualified domain names "voiceit.io" and "*.voiceit.io".
  • Select validation method "Use DNS validation" (default option).

You should now see the created certificate in the Pending state. Please go ahead in click into the cert by clicking the blue text for "Certificate ID", and then click on "Create records in Route 53". The certificate should then get validated.

# IAM

The following are the least-privileged access policy that is needed to create the AWS resources for Voiceit PCM:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "sts:GetCallerIdentity",
                "sts:AssumeRoleWithWebIdentity",
                "sts:AssumeRole",
                "rds:AddTagsToResource",
                "eks:CreateCluster",
                "eks:AssociateAccessPolicy",
                "eks:CreateAccessEntry",
                "eks:DescribeCluster",
                "eks:ListAssociatedAccessPolicies",
                "eks:CreateFargateProfile",
                "eks:CreateNodegroup",
                "eks:CreateAddon",
                "eks:DescribeFargateProfile",
                "eks:DescribeAddon",
                "eks:DescribeNodegroup",
                "eks:TagResource",
                "eks:DescribeAccessEntry",
                "ec2:*",
                "iam:PassRole",
                "rds:CreateDBClusterParameterGroup",
                "rds:ModifyDBClusterParameterGroup",
                "rds:DescribeDBClusterParameterGroups",
                "rds:DescribeDBClusterParameters",
                "iam:CreateRole",
                "iam:GetRole",
                "iam:ListRolePolicies",
                "iam:AttachRolePolicy",
                "logs:CreateLogGroup",
                "logs:PutRetentionPolicy",
                "logs:ListTagsLogGroup",
                "iam:CreatePolicy",
                "iam:GetPolicy",
                "iam:GetPolicyVersion",
                "logs:DescribeLogGroups",
                "iam:PutRolePolicy",
                "iam:GetRolePolicy",
                "iam:ListAttachedRolePolicies",
                "rds:CreateDBSubnetGroup",
                "rds:DescribeDBSubnetGroups",
                "rds:ListTagsForResource",
                "rds:CreateDBCluster",
                "rds:DescribeDBClusters",
                "rds:DescribeGlobalClusters",
                "rds:CreateDBInstance",
                "rds:DescribeDBInstances",
                "iam:CreateOpenIDConnectProvider",
                "iam:TagOpenIDConnectProvider",
                "elasticloadbalancing:*",
                "iam:GetOpenIDConnectProvider",
                "ssm:GetParameter",
                "iam:DeleteOpenIDConnectProvider",
                "cloudformation:ListStacks",
                "cloudformation:CreateStack",
                "cloudformation:DescribeStacks",
                "kms:CreateKey",
                "kms:EnableKeyRotation",
                "kms:DescribeKey",
                "kms:GetKeyPolicy",
                "kms:GetKeyRotationStatus",
                "kms:ListResourceTags",
                "kms:ScheduleKeyDeletion",
                "kms:CreateGrant",
                "kms:CreateAlias",
                "iam:CreateServiceLinkedRole",
                "rds:ModifyDBCluster",
                "logs:TagResource",
                "iam:TagRole"
            ],
            "Resource": "*"
        }
    ]
}

Please create a custom IAM Policy with these permissions, and then attach this policy to an IAM user who will be used to create the AWS resources.

# Run Terraform

# Export correct environment variables for AWS

Please export the IAM user information with the above attached policy as follows:

export AWS_ACCESS_KEY_ID="AK111111111111111111"
export AWS_SECRET_ACCESS_KEY="11111111111111111111111111111111+2222222"

# Run Terraform

Please get the file Credeentials.pdf from the VoiceIt Dashboard Files section which will become available after VoiceIt creates your License. If you don't see this file, please contact VoiceIt Support.

curl https://drive.voiceit.io/files/terraform_aws.zip > terraform_aws.zip
unzip terraform_aws.zip
cd terraform_aws

# MongoDB Pod vs DocumentDB

We suggest using AWS DocumentDB to provide MongoDB services as it is highly reliable and performant. However, if for some reason you opt to use VoiceIt's MongoDB Pod (based on Mongo Server 4.4), please delete the file documentdb.tf before the following steps.

If you are using DocumentDB, please add the following variables when running terraform apply in addition to the variables defined below:

  1. -var="documentdb_username=(Please fill this from the "MongoDB Username" field inCredentials.pdf)"
  2. -var="documentdb_password=(Please fill this from the "MongoDB Password" field inCredentials.pdf)"
terraform init -upgrade
terraform apply -auto-approve \
            -var="mysql_username=(Please fill this from the "MySQL Username" field in `Credentials.pdf`)" \
            -var="mysql_password=(Please fill this from the "MySQL Password" field in `Credentials.pdf`)" \
            -var="docker_hub_email=dockerhubemail@yourdomain.com" \
            -var="docker_hub_username=insert_docker_hub_username" \
            -var="docker_hub_password=insert_docker_hub_password" \
            -var="region=us-east-2-for-example" \
            -var="aws_account_id=111122223333"

PLEASE NOTE

Creation of Docker Secrets in the (secrets.tf) fails about half the time with the message Error: Unauthorized. In this case, just run terraform apply -auto-approve with the same arguments one more time, and it should work the second time around.

# Add your other AWS IAM users in Kubernetes if needed

Please see https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html

Move onto setting up the Rest of VoiceIt PCM/OPM

# Destroy Infrastructure

The following notes should help you if you are trying to destroy all EKS resources.

If you went through setting up VoiceIt PCM resources by following the documentation at https://pcm.voiceit.io or https://opm.voiceit.io, you will need to ensure you delete all deployments, services, and especially ingresses (which also correspond to a AWS Application Load Balancer).

Use admin permissions to bring down all resources.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "*",
            "Resource": "*"
        }
    ]
}

Once the policy is created and attached to the IAM user, please run the following:

terraform destroy -auto-approve

Please note that it will still ask for variables, but you can leave them all blank here by pressing "Return" when prompted.

After running destroy, you will want to manually ensure the following resources are deleted by going to AWS Web Console (they may already be deleted):

  • CloudFormation > eksctl-voiceit-eks-addon-iamserviceaccount-kube-system-aws-load-balancer-controller
  • IAM Role > AmazonEKSLoadBalancerControllerRole
  • IAM Role > t3a-large-eks-node-group...
  • IAM Role > ebs-csi-driver-controller...
  • IAM Role > eks-fargate-profile
  • IAM Role > voiceit-eks-cluster-....
  • IAM Policy > AWSLoadBalancerControllerIAMPolicy
  • IAM Policy > ebs-csi-driver-policy
  • IAM Policy > voiceit-eks-cluster-ClusterEncryption...