Terraform 시작하기
The One Million Container Challenge
The Two Million Container Challenge
작업용 Terraform 서버 생성
Terraform 설치
AWS AMI
인스턴스를 생성합니다.
메모리는 500M 면 충분합니다.
Terraform 을 설치합니다.
sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/AmazonLinux/hashicorp.repo
sudo yum -y install terraform
terraform -help
IAM 권한 설정
여기 를 참고해서 IAM 계정을 생성합니다.
AmazonEC2FullAccess
권한을 부여합니다.
다운받은 CSV 파일을 참조해서 aws configure
을 실행합니다.
aws configure
AWS Access Key ID [None]: AKIA3VXXXXXXXXXX
AWS Secret Access Key [None]: sDSWqBzAyunFXXXXXXXXXXXXXXXXXXXXXX
Default region name [None]: ap-northeast-2
mkdir mywork
cd mywork
EC2 인스턴스 생성
vi main.tf
-------------------------------------
provider "aws" {
profile = "default"
region = "ap-northeast-2"
}
resource "aws_instance" "my_server" {
ami = "ami-0a0de518b1fc4524c"
instance_type = "t2.nano"
tags = {
Name = "MyExampleAppServerInstance"
}
}
-------------------------------------
AMI 이미지 ID 는 아래에서 찾을 수 있습니다.
설정을 검증합니다.
# 테라폼 초기화
terraform init
# 설정파일 검증
terraform validate
# 실행시 결과물 출력
terraform plan
실제로 인스턴스를 생성합니다.
terraform apply
아래 명령으로 생성된 상태를 확인할 수 있습니다.
EC2 인스턴스는 생성되었지만, 아무 서비스도 없고,
심지어 EC2 인스턴스에 접속할 ssh 접근 권한도 없습니다.
terraform show
생성된 인스턴스를 삭제합니다.
terraform destroy
EC2 인스턴스에 도커 이미지 실행
목표
EC2 인스턴스를 생성하고, nginx 이미지를 실행하고, 퍼블릭으로 8080 포트를 오픈합니다.
EC2 인스턴스 생성
vi main.tf
-------------------------------------
provider "aws" {
profile = "default"
region = "ap-northeast-2"
}
resource "aws_instance" "my_server" {
ami = "ami-0a0de518b1fc4524c"
instance_type = "t2.nano"
tags = {
Name = "my webserver"
}
}
-------------------------------------
nginx 이미지 실행
vi main.tf
-------------------------------------
provider "aws" {
profile = "default"
region = "ap-northeast-2"
}
resource "aws_instance" "my_server" {
ami = "ami-0a0de518b1fc4524c"
instance_type = "t2.nano"
user_data = <<-EOF
#! /bin/bash
sudo yum update -y
sudo yum install docker -y
sudo service docker start
sudo systemctl enable docker.service
sudo mkdir -p /var/web
echo "<h1>Hello, World.</h1>" | sudo tee /var/web/index.html
sudo docker run --name nginx \
-v /var/web:/usr/share/nginx/html \
-d \
-p 8080:80 \
nginx
EOF
tags = {
Name = "my webserver"
}
}
-------------------------------------
8080 포트 오픈
sg_ssh
, key_name
는 테스트 용도로만 사용하고 이후에는 삭제합니다.
vi main.tf
-------------------------------------
provider "aws" {
profile = "default"
region = "ap-northeast-2"
}
resource "aws_instance" "my_server" {
ami = "ami-0a0de518b1fc4524c"
instance_type = "t2.nano"
key_name = "skyer9" # 자신의 키 페어입력
vpc_security_group_ids = [
aws_security_group.sg_web.id,
aws_security_group.sg_ssh.id
]
user_data = <<-EOF
#! /bin/bash
sudo yum update -y
sudo yum install docker -y
sudo service docker start
sudo systemctl enable docker.service
sudo mkdir -p /var/web
echo "<h1>Hello, World.</h1>" | sudo tee /var/web/index.html
sudo docker run --name nginx \
-v /var/web:/usr/share/nginx/html \
-d \
-p 8080:80 \
nginx
EOF
tags = {
Name = "my webserver"
}
}
resource "aws_security_group" "sg_web" {
name = "sg_web"
ingress {
from_port = "8080"
to_port = "8080"
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
# 테라폼이 아웃바운드 허용을 삭제하므로, 다시 추가해야 합니다.
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
}
resource "aws_security_group" "sg_ssh" {
name = "sg_ssh"
ingress {
from_port = "22"
to_port = "22"
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
}
-------------------------------------
생성
terraform init
terraform validate
terraform plan
terraform apply
nginx 가 실행될 때까지, 인스턴스 생성 후에도 30~60초 정도 추가로 소요됩니다.
terraform destroy
nomad 클러스터 생성
목표
Consul
서버 1개 생성 후, 할당받은 프라이빗 아이피로 nomad 서버 1개를 생성합니다.
다시 nomad 서버의 프라이빗 아이피를 파라미터로 nomad 클라이언트를 1개 생성합니다.
Security Group 생성
- sg_outbound : 아웃 바운드 허용
- sg_allow_nomad/sg_protect_nomad : nomad 서버 허용
- sg_allow_me : 내 아이피 허용
resource "aws_security_group" "sg_outbound" {
name = "sg_outbound"
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
}
resource "aws_security_group" "sg_allow_nomad" {
name = "sg_allow_nomad"
}
resource "aws_security_group" "sg_protect_nomad" {
name = "sg_protect_nomad"
ingress {
from_port = 0
to_port = 0
protocol = "-1"
security_groups = [ "${aws_security_group.sg_allow_nomad.id}" ]
}
}
resource "aws_security_group" "sg_allow_me" {
name = "sg_allow_me"
ingress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["183.101.XXX.XXX/32"] # 내 아이피 허용
}
}
Consul 서버 생성
http://<Consul 서버 퍼블릭 아이피>:8500 에 접속해서 생성을 확인할 수 있습니다.
생성확인 후에는 key_name
과 aws_security_group.sg_allow_me.id
는 제거해 줍니다.
resource "aws_instance" "consul_server" {
ami = "ami-0a0de518b1fc4524c"
instance_type = "t2.nano"
key_name = "skyer9" # 자신의 키 페어입력
vpc_security_group_ids = [
aws_security_group.sg_outbound.id,
aws_security_group.sg_allow_nomad.id,
aws_security_group.sg_protect_nomad.id,
aws_security_group.sg_allow_me.id
]
user_data = <<-EOF
#! /bin/bash
sudo yum update -y
sudo yum install docker -y
sudo service docker start
sudo systemctl enable docker.service
sudo docker run -d --name consul -p 8500:8500 -p 8600:8600/udp consul
EOF
tags = {
Name = "my consul server"
}
}
nomad 서버 생성
vi aws/userdata_nomad_server.sh
-------------------------------------
#! /bin/bash
sudo yum update -y
# sudo yum install docker -y
# sudo service docker start
# sudo systemctl enable docker.service
wget https://releases.hashicorp.com/nomad/1.1.3/nomad_1.1.3_linux_amd64.zip
unzip nomad_1.1.3_linux_amd64.zip
sudo chown root:root nomad
sudo chmod 777 nomad
sudo mv nomad /usr/bin/
sudo mkdir /etc/nomad.d
sudo bash -c 'cat << EOF > /etc/nomad.d/server.hcl
datacenter = "dc1"
data_dir = "/var/lib/nomad/"
bind_addr = "0.0.0.0"
server {
enabled = true
bootstrap_expect = 1
}
consul {
address = "CONSUL_SERVER_IP:8500"
}
EOF'
sudo bash -c 'cat << EOF > /lib/systemd/system/nomad.service
[Unit]
Description=Nomad
Documentation=https://nomadproject.io/docs/
Wants=network-online.target
After=network-online.target
# When using Nomad with Consul it is not necessary to start Consul first. These
# lines start Consul before Nomad as an optimization to avoid Nomad logging
# that Consul is unavailable at startup.
#Wants=consul.service
#After=consul.service
[Service]
ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/usr/bin/nomad agent -config /etc/nomad.d
KillMode=process
KillSignal=SIGINT
LimitNOFILE=65536
LimitNPROC=infinity
Restart=on-failure
RestartSec=2
StartLimitBurst=3
StartLimitIntervalSec=10
TasksMax=infinity
OOMScoreAdjust=-1000
[Install]
WantedBy=multi-user.target
EOF'
sudo systemctl daemon-reload
sudo systemctl enable nomad
sudo systemctl start nomad
-------------------------------------
depends_on
으로 의존성(생성순서) 을 강제합니다.
aws_instance.consul_server.private_ip
을 이용해 Consul Server 의 프라이빗 아이피를 가져옵니다.
resource "aws_instance" "nomad_server" {
ami = "ami-0a0de518b1fc4524c"
instance_type = "t2.nano"
key_name = "skyer9" # 자신의 키 페어입력
# count = 5 # 인스턴스를 5개 생성
vpc_security_group_ids = [
aws_security_group.sg_outbound.id,
aws_security_group.sg_allow_nomad.id,
aws_security_group.sg_protect_nomad.id,
aws_security_group.sg_allow_me.id
]
user_data = "${replace(file("./aws/userdata_nomad_server.sh"), "CONSUL_SERVER_IP", aws_instance.consul_server.private_ip)}"
tags = {
Name = "my nomad server"
}
depends_on = [ aws_instance.consul_server ]
}
nomad 클라이언트 생성
vi aws/userdata_nomad_client.sh
-------------------------------------
#! /bin/bash
sudo yum update -y
sudo yum install docker -y
sudo service docker start
sudo systemctl enable docker.service
sudo yum install java-11-amazon-corretto-headless -y
wget https://releases.hashicorp.com/nomad/1.1.3/nomad_1.1.3_linux_amd64.zip
unzip nomad_1.1.3_linux_amd64.zip
sudo chown root:root nomad
sudo chmod 777 nomad
sudo mv nomad /usr/bin/
sudo mkdir /etc/nomad.d
sudo bash -c 'cat << EOF > /etc/nomad.d/client.hcl
# bind_addr = "127.0.0.1"
datacenter = "dc1" # 클러스터명
data_dir = "/var/lib/nomad/"
client {
enabled = true
servers = ["NOMAD_SERVER_IP"] # 노마드 서버 프라이빗 아이피
}
EOF'
sudo bash -c 'cat << EOF > /lib/systemd/system/nomad.service
[Unit]
Description=Nomad
Documentation=https://nomadproject.io/docs/
Wants=network-online.target
After=network-online.target
# When using Nomad with Consul it is not necessary to start Consul first. These
# lines start Consul before Nomad as an optimization to avoid Nomad logging
# that Consul is unavailable at startup.
#Wants=consul.service
#After=consul.service
[Service]
ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/usr/bin/nomad agent -config /etc/nomad.d
KillMode=process
KillSignal=SIGINT
LimitNOFILE=65536
LimitNPROC=infinity
Restart=on-failure
RestartSec=2
StartLimitBurst=3
StartLimitIntervalSec=10
TasksMax=infinity
OOMScoreAdjust=-1000
Environment="HOME=/root"
[Install]
WantedBy=multi-user.target
EOF'
sudo systemctl daemon-reload
sudo systemctl enable nomad
sudo systemctl start nomad
-------------------------------------
resource "aws_instance" "nomad_client" {
ami = "ami-0a0de518b1fc4524c"
instance_type = "t2.nano"
key_name = "skyer9" # 자신의 키 페어입력
vpc_security_group_ids = [
aws_security_group.sg_outbound.id,
aws_security_group.sg_allow_nomad.id,
aws_security_group.sg_protect_nomad.id,
aws_security_group.sg_allow_me.id
]
# nomad 서버가 1대인 경우
user_data = "${replace(file("./aws/userdata_nomad_client.sh"), "NOMAD_SERVER_IP", aws_instance.nomad_server.private_ip)}"
# 2대 이상인 경우
# user_data = "${replace(file("./aws/userdata_nomad_client.sh"), "NOMAD_SERVER_IP", join("\",\"", aws_instance.nomad_server.*.private_ip))}"
tags = {
Name = "my nomad client"
}
depends_on = [ aws_instance.nomad_server ]
}
Auto Scaling
목표
nomad client 노드에 대해 Auto Scaling 구현합니다.
nomad 에서의 Auto Scaling 은 Application Level
, Cluster Level
두가지의 Auto Scaling 을 지원합니다.
Nomad Autoscaler 설치
Nomad Autoscaler 독립서버로 설치할 수도 있지만, Job 으로 등록하는 것이 일반적입니다.
vi prometheus.nomad
------------------------------
job "prometheus" {
datacenters = ["dc1"]
group "prometheus" {
count = 1
network {
port "prometheus_ui" {}
}
task "prometheus" {
driver = "docker"
config {
image = "prom/prometheus:v2.25.0"
ports = ["prometheus_ui"]
args = [
"--config.file=/etc/prometheus/config/prometheus.yml",
"--storage.tsdb.path=/prometheus",
"--web.listen-address=0.0.0.0:${NOMAD_PORT_prometheus_ui}",
"--web.console.libraries=/usr/share/prometheus/console_libraries",
"--web.console.templates=/usr/share/prometheus/consoles",
]
volumes = [
"local/config:/etc/prometheus/config",
]
}
template {
data = <<EOH
---
global:
scrape_interval: 1s
evaluation_interval: 1s
scrape_configs:
- job_name: nomad
scrape_interval: 10s
metrics_path: /v1/metrics
params:
format: ['prometheus']
consul_sd_configs:
- server: '{{ env "attr.unique.network.ip-address" }}:8500'
services: ['nomad','nomad-client']
relabel_configs:
- source_labels: ['__meta_consul_tags']
regex: .*,http,.*
action: keep
- job_name: traefik
metrics_path: /metrics
consul_sd_configs:
- server: '{{ env "attr.unique.network.ip-address" }}:8500'
services: ['traefik-api']
EOH
change_mode = "signal"
change_signal = "SIGHUP"
destination = "local/config/prometheus.yml"
}
resources {
cpu = 100
memory = 256
}
service {
name = "prometheus"
port = "prometheus_ui"
tags = [
"traefik.enable=true",
"traefik.tcp.routers.prometheus.entrypoints=prometheus",
"traefik.tcp.routers.prometheus.rule=HostSNI(`*`)"
]
check {
type = "http"
path = "/-/healthy"
interval = "10s"
timeout = "2s"
}
}
}
}
}
------------------------------
nomad run prometheus.nomad
vi autoscaler.nomad
------------------------------
job "autoscaler" {
datacenters = ["dc1"]
group "autoscaler" {
count = 1
task "autoscaler" {
driver = "docker"
config {
image = "hashicorp/nomad-autoscaler:0.3.3"
command = "nomad-autoscaler"
args = ["agent", "-config", "${NOMAD_TASK_DIR}/config.hcl"]
}
template {
data = <<EOF
plugin_dir = "/plugins"
nomad {
address = "http://{{env "attr.unique.network.ip-address" }}:4646"
}
apm "nomad" {
driver = "nomad-apm"
config = {
address = "http://{{env "attr.unique.network.ip-address" }}:4646"
}
}
apm "prometheus" {
driver = "prometheus"
config = {
address = "http://{{ env "attr.unique.network.ip-address" }}:9090"
}
}
strategy "target-value" {
driver = "target-value"
}
EOF
destination = "${NOMAD_TASK_DIR}/config.hcl"
}
}
}
}
------------------------------
nomad run autoscaler.nomad
Application Level
vi hello.nomad
job "hello" {
datacenters = ["dc1"]
type = "service"
group "helloGroup" {
network {
port "http" {}
port "https" {}
# port "lb" { static = 8080 }
}
count = 1
scaling {
enabled = true
min = 1
max = 2
policy {
cooldown = "1m"
evaluation_interval = "30s"
check "avg_sessions" {
source = "prometheus"
query = "sum(traefik_entrypoint_open_connections{entrypoint=\"webapp\"})/scalar(nomad_nomad_job_summary_running{task_group=\"demo\"})"
strategy "target-value" {
target = 10
}
}
}
}
# Define a task to run
task "helloTask" {
driver = "java"
config {
jar_path = "local/TestPublic-0.0.2-SNAPSHOT.jar"
jvm_options = ["-Xmx128m","-Xms128m"]
}
env {
PORT = "${NOMAD_PORT_http}"
NODE_IP = "${NOMAD_IP_http}"
}
service {
name = "helloService"
# port = "lb"
port = "http"
check {
type = "http"
path = "/hello" # health check 용 url
interval = "2s"
timeout = "2s"
}
}
resources {
cpu = 500 # 500 Mhz
memory = 200 # 200 MB
}
# 원격에서 다운받아야 합니다.
artifact {
source = "https://github.com/skyer9/TestPublic/raw/master/TestPublic-0.0.2-SNAPSHOT.jar"
}
}
}
}
nomad run hello.nomad
Cluster Level
Auto Scaling + Load Balancing
목표
nomad client 에 Auto Scaling + Load Balancing
기능을 부여합니다.
data "aws_availability_zones" "available" {
state = "available"
}
```
```terraform