Terraform

What is Terraform exactly and why do we need Terraform ?

1. Terraform is an open-source infrastructure as code tool that allows you to define, provision, and manage cloud infrastructure using a simple, declarative configuration language HCL.

2. It supports multiple cloud providers, such as AWS, Azure, and Google Cloud, enabling you to automate the setup and management of your infrastructure consistently and efficiently.

3. Terraform simplifies and automates the process of managing cloud infrastructure, providing consistency, repeatability, and collaboration. By using Terraform, you can ensure that your infrastructure is defined, version-controlled, and maintained efficiently across multiple cloud providers.

Benefits of Infrastructure as Code (IaC)

1. Consistency: Ensure the same configurations are applied every time, reducing configuration drift and human error.

2. Speed and Efficiency: Automate repetitive tasks, enabling faster provisioning and updates to infrastructure.

3. Collaboration: Teams can work together on infrastructure code, leveraging version control for reviews and collaboration.

4. Scalability: Easily scale resources up or down based on needs without manual configuration.

5. Disaster Recovery: Quickly recreate infrastructure in case of failures by reapplying the code.

Install Terraform

• Follow the official guide to install Terraform on your local machine.

• Terraform Installation Guide --> Terraform Installation Guide

Terraform Core Concepts

1. Providers: Providers in infrastructure as code (IaC) tools like Terraform are responsible for managing the lifecycle of resources. They interact with APIs of cloud providers (like AWS, Azure, GCP) to create, update, and delete resources. Providers abstract away the complexity of interacting with these APIs.

2. Resources: Resources represent the infrastructure components you want to manage using Terraform. These could be virtual machines, databases, networks, etc. You define resources in your Terraform configuration files, specifying their type and configuration parameters. Terraform then uses the provider to create, modify, or delete these resources as needed.

3. Variables: Variables in Terraform allow you to parameterize your configurations, making them reusable and adaptable to different environments. You can define variables in Terraform configuration files or in separate variable files. They can be used to customize resource configurations, such as specifying instance sizes, region names, or network configurations.

4. State: Terraform keeps track of the state of your infrastructure in a state file. This state file is crucial for Terraform to understand the current state of your resources and to plan and execute changes effectively. It stores information about the resources managed by Terraform, their dependencies, and their attributes. Terraform uses this state to determine what changes need to be applied to bring your infrastructure in line with your configuration.

Setup LAB environment:

1. Navigate to the Git repository --> Terraform

2. Click on Code --> Codespaces --> Click on "create codespace"

3. Codespace is free for 60 hours/month it's a container sandbox env ( 2CPU, 4 GB RAM)

4. To install any tools, navigate to search bar and search ">Dev Container Configuration"

   

5. Click on "Modify the active configuration" and then search for "Terraform" with verified devcontainer configuration check mark.

   

6. We will do the same step to install aws-cli as well and once that is done our devcontainer configuration will look like this.

   

7. Lastly we just need to rebuild the container --> ">rebuild container"

   

8. Now we can verify the installation (it will take some time 5-10 minutes to start container

9. Create an AWS account if you don't have you can create --> here

10. Go to AWS console --> Your profile --> Security Credentials --> Access Keys

11. Now setup the AWS configuration with terraform provider

aws configure 

12. Setup credentials and once that is done we are ready to perform labs.

SCENARIO-1: Basic EC2 Instance creation:

• Create a terraform configuration to provision a (EC2 instance on AWS)

• main.tf This file contains the main Terraform configuration.

• GitHub Page --> LAB-1:EC2

SCENARIO-2: Infrastructure Provisioning:

Problem Statement: A company wants to deploy a web server on AWS to serve static content. Provision an EC2 instance with Terraform and configure it to run a web server.

Solution:

• Configure the AWS provider.

• Create an EC2 instance.

• Set up a security group to allow HTTP access.

• Use a user data script to start a web server on the EC2 instance.

• GitHub Page for solution implementation --> LAB-2:Infra

Terraform Terminology:

• Provider: A provider is a plugin for Terraform that defines and manages resources for a specific cloud or infrastructure platform. Examples of providers include AWS, Azure, Google Cloud, and many others. You configure providers in your Terraform code to interact with the desired infrastructure platform.

• GitHub Page for Providers Demo --> LAB-3:Infra

• Input variables are used to customize Terraform configurations. You define them using the variable block.

• Output variables are used to display useful information about the infrastructure after Terraform applies the changes.

  
  

Terraform .tfvars:

In Terraform, .tfvars files (typically with a .tfvars extension) are used to set specific values for input variables defined in your Terraform configuration.

They allow you to separate configuration values from your Terraform code, making it easier to manage different configurations for different environments (e.g., development, staging, production) or to store sensitive information without exposing it in your code.

Here's the purpose of .tfvars files:

1. Separation of Configuration from Code: Input variables in Terraform are meant to be configurable so that you can use the same code with different sets of values. Instead of hardcoding these values directly into your .tf files, you use .tfvars files to keep the configuration separate. This makes it easier to maintain and manage configurations for different environments.

2. Sensitive Information: .tfvars files are a common place to store sensitive information like API keys, access credentials, or secrets. These sensitive values can be kept outside the version control system, enhancing security and preventing accidental exposure of secrets in your codebase.

3. Reusability: By keeping configuration values in separate .tfvars files, you can reuse the same Terraform code with different sets of variables. This is useful for creating infrastructure for different projects or environments using a single set of Terraform modules.

4. Collaboration: When working in a team, each team member can have their own .tfvars file to set values specific to their environment or workflow. This avoids conflicts in the codebase when multiple people are working on the same Terraform project.

Modules:

The advantage of using Terraform modules in your infrastructure as code (IaC) projects lies in improved organization, reusability, and maintainability. Here are the key benefits:

1. Modularity: Terraform modules allow you to break down your infrastructure configuration into smaller, self-contained components. This modularity makes it easier to manage and reason about your infrastructure because each module handles a specific piece of functionality, such as an EC2 instance, a database, or a network configuration.

2. Reusability: With modules, you can create reusable templates for common infrastructure components. Instead of rewriting similar configurations for multiple projects, you can reuse modules across different Terraform projects. This reduces duplication and promotes consistency in your infrastructure.

3. Simplified Collaboration: Modules make it easier for teams to collaborate on infrastructure projects. Different team members can work on separate modules independently, and then these modules can be combined to build complex infrastructure deployments. This division of labor can streamline development and reduce conflicts in the codebase.

4. Versioning and Maintenance: Modules can have their own versioning, making it easier to manage updates and changes. When you update a module, you can increment its version, and other projects using that module can choose when to adopt the new version, helping to prevent unexpected changes in existing deployments.

5. Abstraction: Modules can abstract away the complexity of underlying resources. For example, an EC2 instance module can hide the details of security groups, subnets, and other configurations, allowing users to focus on high-level parameters like instance type and image ID.

6. Testing and Validation: Modules can be individually tested and validated, ensuring that they work correctly before being used in multiple projects. This reduces the risk of errors propagating across your infrastructure.

7. Documentation: Modules promote self-documentation. When you define variables, outputs, and resource dependencies within a module, it becomes clear how the module should be used, making it easier for others (or your future self) to understand and work with.

8. Scalability: As your infrastructure grows, modules provide a scalable approach to managing complexity. You can continue to create new modules for different components of your architecture, maintaining a clean and organized codebase.

9. Security and Compliance: Modules can encapsulate security and compliance best practices. For instance, you can create a module for launching EC2 instances with predefined security groups, IAM roles, and other security-related configurations, ensuring consistency and compliance across your deployments.

Terraform State File:

1. State file in Terraform is where Terraform records the information of the infrastructure it has created.

2. State file is important because it helps Terraform update and manage existing infrastructure instead of creating new instances or resources.

3. Advantages of using the state file include updating existing infrastructure and destroying infrastructure in a controlled manner.

4. One drawback of the state file is that it records sensitive information, such as passwords, by default.

5.  Storing the state file on a personal laptop or any vulnerable device can compromise the security of sensitive information in the state file.

6.  If the state file is not pushed to the Version Control System, Terraform may not know about updates and can delete the corresponding infrastructure.

7. Using a remote backend solves the drawbacks of storing the state file in a Version Control System.
   

Remote Backend:

1. With a remote backend, the state file is automatically updated in the external resource (e.g., S3 bucket) when applying Terraform configurations.

2.  Storing the state file in a separate remote backend, such as an S3 bucket, enhances security and prevents sensitive information from being exposed in a Version Control System.

3. With a remote backend workflow, DevOps engineers can clone the GitHub repository, make changes locally, verify them with `terraform apply`, and raise a pull request to update the remote state file stored in the S3 bucket.

4. Before running `terraform init`, there will be no state file available. After running `terraform apply`, the state file will be created and populated with resource information.

5. The populated state file contains information about the created resources, such as the instance ID, public and private IP addresses.

6.  Terraform can show the state file using the `terraform show` command.

7. Deleting the state file can cause Terraform to lose track of the created resources.

8. Using a remote backend like S3 requires creating a `backend.tf` file with the necessary backend configuration.

9. The S3 bucket name and region are required in the backend configuration.

10.  Initializing the project with the backend configuration will create the S3 bucket.

11. Modifying the code no longer requires handling the state file manually.

12. Now if we delete the terraform.tfvars from local and try terraform init it will setup remote backend and after terraform apply it will create the resources managing state files in s3.

    
    

Locking Mechanism in Terraform:

1. Terraform uses locking mechanism to prevent conflicts when multiple people try to update the same project simultaneously.

2. DynamoDB can be used as a locking mechanism for Terraform projects.

3. Adding the DynamoDB lock details to the backend configuration enables Terraform to handle locking.

Terraform Workspaces:

1. Terraform workspaces allow us to manage multiple environments (like dev, staging, prod) in the same Terraform configuration directory.

2. Each workspace maintains its own state file.

3. GitHub Page for all the workspace commands - LAB-6:Infra

Terraform Capstone Project:

                     +------------------+
                     |  Terraform Code  |
                     +------------------+
                              |
        +---------------------+--------------------+
        |                                          |
+------------------+                  +------------------+
|   Environments   |                  |     Modules      |
+------------------+                  +------------------+
        |                                      |
                         +----------------+------+----------------
                         |       Modules for AWS Resources        |
+------+  +-------+      +---------+   +----+   +-----+   +-------+   
|common|  |  dev  |      | API GW  |   |ECS |   | EC2 |   | Route |   
|nonprod| |  prod |      +---------+   +----+   +-----+   +-------+   
+------+  +-------+                                                  
                             

AWS resources blocks for infra automation:

infra
├── environments
│   ├── common-nonprod
│   │   ├── backend.tf
│   │   ├── dns.tf
│   │   ├── ecr.tf
│   │   ├── providers.tf
│   │   ├── terraform.tfvars
│   │   └── variables.tf
│   ├── dev
│   │   ├── certs
│   │   │   └── private-cloudfront.pub
│   │   ├── backend.tf
│   │   ├── credentials.yaml
│   │   ├── id_rsa.pub
│   │   ├── local.tf
│   │   ├── main.tf
│   │   ├── outputs.tf
│   │   ├── providers.tf
│   │   ├── securitygroups.tf
│   │   ├── terraform.tfvars
│   │   └── variables.tf
│   ├── staging
│   │   ├── certs
│   │   │   ├── ca.crt
│   │   │   ├── client1.domain.tld.crt
│   │   │   ├── client1.domain.tld.key
│   │   │   ├── id_rsa.pub
│   │   │   ├── private-cloudfront.pem
│   │   │   ├── private-cloudfront.pub
│   │   │   ├── server.crt
│   │   │   └── server.key
│   │   ├── backend.tf
│   │   ├── local.tf
│   │   ├── main.tf
│   │   ├── outputs.tf
│   │   ├── providers.tf
│   │   ├── securitygroups.tf
│   │   ├── terraform.tfvars
│   │   └── variables.tf
│   ├── production
│   │   ├── certs
│   │   │   ├── ca.crt
│   │   │   ├── client1.domain.tld.crt
│   │   │   ├── client1.domain.tld.key
│   │   │   ├── id_rsa.pub
│   │   │   ├── private-cloudfront.pem
│   │   │   ├── private-cloudfront.pub
│   │   │   ├── server.crt
│   │   │   └── server.key
│   │   ├── backend.tf
│   │   ├── ec2.tf
│   │   ├── common-ecr.tf
│   │   ├── local.tf
│   │   ├── main.tf
│   │   ├── outputs.tf
│   │   ├── providers.tf
│   │   ├── securitygroups.tf
│   │   ├── terraform.tfvars
│   │   └── variables.tf
├── modules
│   ├── api-gateway
│   │   ├── main.tf
│   │   ├── outputs.tf
│   │   └── variables.tf
│   ├── aws_client_vpn
│   │   ├── certificates.tf
│   │   ├── main.tf
│   │   └── variables.tf
│   ├── certificates
│   │   ├── certificates.tf
│   │   ├── output.tf
│   │   └── variables.tf
│   ├── dns
│   │   ├── main.tf
│   │   ├── output.tf
│   │   └── variables.tf
│   ├── ec2
│   │   ├── alb.tf
│   │   ├── local.tf
│   │   ├── main.tf
│   │   ├── target_group.tf
│   │   └── variables.tf
│   ├── ecr
│   │   ├── main.tf
│   │   └── variables.tf
│   ├── ecs
│   │   ├── alb.tf
│   │   ├── alb_target_groups.tf
│   │   ├── ecs_service.tf
│   │   ├── ecs_task_definition.tf
│   │   ├── iam.tf
│   │   ├── local.tf
│   │   ├── main.tf
│   │   ├── output.tf
│   │   ├── route53.tf
│   │   ├── s3.tf
│   │   ├── ssl_certificates.tf
│   │   └── variables.tf
│   ├── ecs_api_gateway
│   │   ├── alb.tf
│   │   ├── alb_target_groups.tf
│   │   ├── api_gateway.tf
│   │   ├── api_gateway_custom_domain.tf
│   │   ├── api_gateway_integrations.tf
│   │   ├── api_gateway_route53.tf
│   │   ├── api_gateway_routes.tf
│   │   ├── api_gateway_stages.tf
│   │   ├── api_gateway_vpc_link.tf
│   │   ├── ecs_service.tf
│   │   ├── ecs_task_definition.tf
│   │   ├── iam.tf
│   │   ├── local.tf
│   │   ├── main.tf
│   │   ├── output.tf
│   │   ├── ssl_certificates.tf
│   │   └── variables.tf
│   ├── rds
│   │   ├── groups.tf
│   │   ├── iam.tf
│   │   ├── local.tf
│   │   ├── main.tf
│   │   ├── route53.tf
│   │   ├── secret.tf
│   │   └── variables.tf
│   ├── redis_valkey
│   │   ├── main.tf
│   │   └── variable.tf
│   ├── route53
│   │   ├── main.tf
│   │   ├── output.tf
│   │   └── variables.tf
│   ├── s3
│   │   ├── main.tf
│   │   └── variables.tf
│   ├── secret-manager
│   │   ├── main.tf
│   │   ├── output.tf
│   │   └── variables.tf
│   ├── security-groups
│   │   ├── local.tf
│   │   ├── main.tf
│   │   ├── output.tf
│   │   └── variables.tf
│   ├── static_website
│   │   ├── cdn.tf
│   │   ├── local.tf
│   │   ├── main.tf
│   │   ├── private_cdn_key.tf
│   │   ├── route53.tf
│   │   └── variables.tf
│   ├── vpc
│   │   ├── local.tf
│   │   ├── main.tf
│   │   ├── nacl.tf
│   │   ├── output.tf
│   │   ├── routetables.tf
│   │   ├── subnets.tf
│   │   └── variables.tf

1. Terraform Code: Root of the project.

2. Environments:

   • common-nonprod: Shared configurations for non-production.

   • dev: Development-specific configurations.

   • staging: Staging environment.

   • production: Production environment.

3. Modules: Reusable Terraform components for AWS services:

   • API Gateway, ECS, EC2, Route 53, RDS, etc.

TO BE CONTINUED ...