# timotyenservice

[![Build Status](https://jenkins.internal.justin.tv/buildStatus/icon?job=timotyenorg/timotyenservice/master)](https://jenkins.internal.justin.tv/job/timotyenorg/job/timotyenservice/job/master)

Include an elevator pitch summary of what your app does.


## Bootstrap of your application
Remove the bootstrap section when you're done

### Account prereqs
These need to be done to your account before you spin up your microservice.

1. Setup aws cli for your account profiles.  The recommended way is [isengard credentails](https://git-aws.internal.justin.tv/twitch/isengard_credentials).
2. Create a [jenkins user](https://git-aws.internal.justin.tv/twitch/jenkins_cf_user) for production/pipeline account and add the user credentials to jenkins.
4. Setup a VPC for your account.  You should probably [make your own](https://git-aws.internal.justin.tv/twitch/basic_vpc).  Or, you can have #systems [setup one for you](https://wiki.twitch.com/display/AD/How+to+Set+Up+a+New+AWS+Account#HowtoSetUpaNewAWSAccount-IfyouwouldlikeCoreSystemstosetupyourVPC:), but that will take a few days.
3. Create a [bastion host](https://git-aws.internal.justin.tv/twitch/bastion_ldap_host) in development inside your VPC.
5. [Isengard](https://isengard.amazon.com/) access configured with an "admin" role that you are part of for staging and production.
    - If you have created the accounts with administrator roles that are spelled differently ("Admin", "ADMIN", etc.), you should update the developer role parameter value in the staging and production json files under cloudformation/infra like so:
    ```
    {
      "ParameterKey": "DeveloperRole",
      "ParameterValue": "arn:aws:iam::<account_id>:role/Admin"
    }
    ```
### Microservice spinup prereqs
1. [Docker for mac](https://download.docker.com/mac/stable/Docker.dmg) stable.  You *need* docker version >= 18.03. `docker version`
2. [mwinit](https://w.amazon.com/index.php/NextGenMidway/UserGuide#Mac) >= 1.24 `mwinit --version` (Note: Older versions work if you're on WPA2)
3. [teleport-bastion](https://wiki.twitch.com/display/SEC/Teleport+Bastion) `teleport-bastion status`
4. [aws cli](https://docs.aws.amazon.com/cli/latest/userguide/cli-install-macos.html) >= [1.11.91](https://github.com/aws/aws-cli/blob/develop/CHANGELOG.rst#11191): Only required to download the builder from ECR. `aws --version`
5. [jq](https://stedolan.github.io/jq/download/): Required to parse JSON in some of the [jenkins_cf_deployer](https://git-aws.internal.justin.tv/chat/copo/tree/master/scripts/jenkins_cf_deployer) scripts.
6. Be an [Isengard](https://isengard.amazon.com/) admin for the production and staging accounts you want to deploy into.

### Microservice spinup prereqs install commands

Use the commands if you cannot figure out the recommended way to install a prereq

#### How to install docker (if you do not have docker)

```bash
Click the `dmg`, open it, and move it to applications.  Then run the docker daemon.
```

#### How to install mwinit (if you do not have mwinit)
```bash
curl --fail -O https://s3.amazonaws.com/com.amazon.aws.midway.software/mac/mwinit \
&& chmod u+x mwinit && sudo mv mwinit /usr/local/bin/mwinit
which mwinit
mwinit --version
```

#### How to install teleport bastion (if you do not have teleport bastion)
```bash
brew tap --full twitch/security https://twitch-security-packages.s3.amazonaws.com/homebrew.git
brew update
brew install twitch-bastion-util
teleport-bastion enable
```

#### How to install aws-cli (if you do not have aws-cli)
```bash
curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip"
unzip awscli-bundle.zip
sudo ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws
which aws
aws --version
```

#### How to install jq (if you do not have jq)
```bash
brew install jq
```


### Create a new git repository

1. Visit https://git-aws.internal.justin.tv/organizations/discovery/repositories/new
2. Give it a name with all lowercase characters of length 3-12 characters. This isn't a hard requirement, but will help
the bootstrapping happen without strange issues.

### Replace your git repository with example app
Setup some environment variables
```bash
# Replace these two
export GHE=discovery
export REPO=timotyenservice
```

Then check out the code

```bash
mkdir -p ~/workspace
cd ~/workspace
git clone git@git-aws.internal.justin.tv:${GHE}/${REPO}.git
cd ${REPO}
git remote add timotyenservice git@git-aws.internal.justin.tv:timotyenorg/timotyenservice.git
git fetch -av --all
git checkout -b from_example timotyenservice/master
```

### Modify service variables and verify them 
```bash
vim vars.sh # And edit all variables at the top, especially SERVICE=
teleport-bastion login && mwinit && ./make.sh bootstrap precheck
```

### Bootstrap everything

```bash
# turn up the speakers on your mac :)
teleport-bastion login && mwinit && ./make.sh bootstrap everything
```

### Verify everything works

1. Open https://jenkins.internal.justin.tv/job/discovery/ and click on "Scan Organization Now".  Wait a bit for the scan job to run.
2. Once detected, visit your pipeline in Jenkins
3. Check out your code on github
4. Skip ahead to the development section below!

### Expose your service to others and optional setups

To allow other accounts to access your service, you will need to use [AWS Peering](https://wiki.twitch.com/display/SYS/AWS+VPC+Peering+at+Twitch).

To setup SSL
1. Set UseHTTPS=true in [staging.json](./cloudformation/infra/staging.json) and [production.json](./cloudformation/infra/production.json)
2. Run `./make.sh infra update staging`.  This script will eventually pause at ACM certification
3. Visit  [DNS](https://dashboard.internal.justin.tv/dns) and add the DNS entry ACM needs.  Here is an [example](https://dashboard.internal.justin.tv/dns/?entry=_902eb7c06af2fb3c9fb51adfdee3c956.staging.anotherapp.internal.justin.tv).
4. The command in step (3) will then eventually finish.

To setup Sandstorm
1. create a [role](https://dashboard.internal.justin.tv/sandstorm/manage-secrets) for your
service (both staging and production)
2. Modify [deploy.yaml](./cloudformation/deploy/deploy.yaml) to set the sandstorm ARN.
3. modify [populate_app_env](./make.sh) to set your sandstorm env variable when running code locally.

To setup S2S auth:
1. create a staging and production [S2S service](https://dashboard.internal.justin.tv/s2s/services) and
allow your task ARN.
2. Modify [deploy.yaml](./cloudformation/deploy/deploy.yaml) to set the environment variable s2s_auth
 to the name of your S2S service.
3. Modify [populate_app_env](./make.sh) to set your s2s env variable when running code locally.

To onboard onto autoprof:
1. Open the [production cloudformation infra](https://isengard.amazon.com/federate?account=945359790842&role=admin&destination=cloudformation%2Fhome%3Fregion%3Dus-west-2%23%2Fstack%2Fdetail%3FstackId%3Dtimotyenservice-production-infra) outputs. Copy the `AutoprofBucketARN` and `AutoprofReaderRoleARN` values.
2. Visit the [autoprof onboarding UI](https://autoprof.video.xarth.tv/onboard) and enter the values from the previous step

### Timotyenservice bootcamp

You don't need to do this, but doing it could familiarize yourself with how things are setup.

#### Debug a command

Debug a make command.  Let's try a simple one, like one that formats all our go code.

```bash
DEBUG=true ./make.sh go format
```

#### Open a bastion shell

Go onto the bastion host inside the development environment.  From there you could curl your service's ALB.

```bash
> ./make.sh bastion execute
Last login: Wed Nov  7 20:01:33 2018 from 10.204.93.204

       __|  __|_  )
       _|  (     /   Amazon Linux 2 AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-2/
4 package(s) needed for security, out of 16 available
Run "sudo yum update" to apply all updates. 
[jlindamo@ip-10-204-94-154 ~]$ curl internal-timotyenservice-staging-ALB-668710864.us-west-2.elb.amazonaws.com
404 page not found
```

#### Customize the builder

Add a custom binary to the builder.  Let's use ruby.  Modify [the builder](./builder/Dockerfile) and
add a ruby installation to the apt-get command.

```dockerfile
   # For protoc install
   unzip=6.0\* \
+  # For ruby
+  ruby=1:2\* \ 
   # For aws-cli
   python-setuptools=33.1\* python-pip=9.0\* \
```

Now modify [vars.sh](./vars.sh) to signal you have a new version of the builder.

```bash
BUILDER_TAG=${SERVICE}-builder:builder-666 # Or whatever is the next number
```

Now push this commit and watch Jenkins make and push your builder.

```bash
git commit -am 'A new builder'
git push origin HEAD:testing_jenkins
```

When the jenkins job is finished, you should be able to download the builder it created.  You can optionally build
the builder yourself with `./make.sh builder make`.

```bash
# Once jenkins has built and pushed the builder
./make.sh builder download
```

#### Add your own make command

Modify [make.sh](./make.sh) and add a bash function that now uses ruby.

```bash
function root__sayhi() {
  if [[ "${1-}" == "help" ]]; then
    echo "Say hello to the user"
    exit 0
  fi
  if [[ "${USE_BUILDER}" == "true" ]]; then
    make_exec builder run sayhi $@
  fi
  ruby -e "puts 'Hello from ruby'"
}
```

Now run your new command.  Since USE_BUILDER is the default, it should run inside your
docker builder.

```bash
> ./make.sh sayhi
Hello from ruby
```

#### Try skipping the docker container

If you have ruby on your mac, try skipping the docker container.  This may or may not work

```bash
> USE_BUILDER=false ./make.sh sayhi
Hello from ruby
```

#### Put your make command inside the pipeline

Modify [Jenkinsfile](./Jenkinsfile) and add a stage for your custom command.  I suggest right after stage("Lint").

```groovy
   stage("Lint") {
     steps {
       sh './make.sh go lint'
       sh './make.sh docker lint cmd/timotyenservice/Dockerfile cmd/timotyenservice/Dockerfile_integration_test builder/Dockerfile'
     }
   }
+  stage("Talk") {
+    steps {
+      sh './make.sh sayhi'
+    }
+  }
```

Push your modification to a new branch:

```bash
git commit -am 'Testing stuff'
git push origin HEAD:testing_jenkins
```

[Visit your build pipeline](https://jenkins.internal.justin.tv/job/timotyenorg/job/timotyenservice/job/testing_jenkins/) and watch it run your custom step

#### Remove existing make commands

Modify [make.sh](./make.sh) and remove the default make command for the bootstrapping.

```bash
unset -f root__bootstrap
``` 

Now try running bootstraps and it should fail

```bash
> ./make.sh bootstrap
Undefined subcommand bootstrap
```

#### Run the app without using Docker

You can try running your application without using Docker.  This may decrease start-up time since docker for mac's file
system driver is very slow.

```bash
< USE_BUILDER=false ./make.sh go run
```

## Timotyenservice's big table of links

| Thing | Link |
|-------------|-----|
| Staging endpoint | http://www.staging.timotyenservice.internal.justin.tv |
| Production endpoint | http://www.production.timotyenservice.internal.justin.tv |
| Grafana dashboard | https://grafana.internal.justin.tv/dashboard/db/timotyenservice |
| Alerting (also via grafana) | https://grafana.internal.justin.tv/dashboard/db/timotyenservice |
| Source code | https://git-aws.internal.justin.tv/timotyenorg/timotyenservice |
| Code build | https://jenkins.internal.justin.tv/job/timotyenorg/job/timotyenservice/job/master |
| Dev AWS account | https://isengard.amazon.com/account/960608786107 |
| Production AWS account | https://isengard.amazon.com/account/945359790842 |
| Build deployment | https://jenkins.internal.justin.tv/job/timotyenorg/job/timotyenservice/job/master |
| Container logs (staging) | https://isengard.amazon.com/federate?account=960608786107&role=admin&destination=cloudwatch%2Fhome%3Fregion%3Dus-west-2%23logStream%3Agroup%3Dtimotyenservice-staging-ECSLogGroup |
| Container logs (production) | https://isengard.amazon.com/federate?account=945359790842&role=admin&destination=cloudwatch%2Fhome%3Fregion%3Dus-west-2%23logStream%3Agroup%3Dtimotyenservice-production-ECSLogGroup |
| Secrets management | https://dashboard.internal.justin.tv/sandstorm/manage-secrets?nameFilter=timotyenservice |
| Staging DNS management | https://dashboard.internal.justin.tv/dns/?entry=www.staging.timotyenservice.internal.justin.tv |
| Production DNS management | https://dashboard.internal.justin.tv/dns/?entry=www.production.timotyenservice.internal.justin.tv |
| Error logs (staging) | https://isengard.amazon.com/federate?account=960608786107&role=admin&destination=cloudwatch%2Fhome%3Fregion%3Dus-west-2%23logs-insights%3AqueryDetail%3D~(end~0~start~-3600~timeType~'RELATIVE~unit~'seconds~editorString~'fields*20*40timestamp*2c*20*40message*2c*20ispresent*28err*29*20as*20exists*0a*7c*20filter*20exists*20*21*3d*200*0a*7c*20sort*20*40timestamp*20desc*0a*7c*20limit*2020~isLiveTail~false~queryId~'timotyenservice-staging-ECSLogGroup~source~'timotyenservice-staging-ECSLogGroup) |
| Error logs (production) | https://isengard.amazon.com/federate?account=945359790842&role=admin&destination=cloudwatch%2Fhome%3Fregion%3Dus-west-2%23logs-insights%3AqueryDetail%3D~(end~0~start~-3600~timeType~'RELATIVE~unit~'seconds~editorString~'fields*20*40timestamp*2c*20*40message*2c*20ispresent*28err*29*20as*20exists*0a*7c*20filter*20exists*20*21*3d*200*0a*7c*20sort*20*40timestamp*20desc*0a*7c*20limit*2020~isLiveTail~false~queryId~'timotyenservice-production-ECSLogGroup~source~'timotyenservice-production-ECSLogGroup) |
| Slack channel | [#timotyenservice](https://twitch.slack.com/messages/timotyenservice) |
| Staging infra cloudformation | https://isengard.amazon.com/federate?account=960608786107&role=admin&destination=cloudformation%2Fhome%3Fregion%3Dus-west-2%23%2Fstack%2Fdetail%3FstackId%3Dtimotyenservice-staging-infra |
| Staging deploy cloudformation | https://isengard.amazon.com/federate?account=960608786107&role=admin&destination=cloudformation%2Fhome%3Fregion%3Dus-west-2%23%2Fstack%2Fdetail%3FstackId%3Dtimotyenservice-staging-deploy |
| Production infra cloudformation | https://isengard.amazon.com/federate?account=945359790842&role=admin&destination=cloudformation%2Fhome%3Fregion%3Dus-west-2%23%2Fstack%2Fdetail%3FstackId%3Dtimotyenservice-production-infra |
| Production deploy cloudformation | https://isengard.amazon.com/federate?account=945359790842&role=admin&destination=cloudformation%2Fhome%3Fregion%3Dus-west-2%23%2Fstack%2Fdetail%3FstackId%3Dtimotyenservice-production-deploy |
| Canary deploy cloudformation | https://isengard.amazon.com/federate?account=945359790842&role=admin&destination=cloudformation%2Fhome%3Fregion%3Dus-west-2%23%2Fstack%2Fdetail%3FstackId%3Dtimotyenservice-canary-deploy |
| Oncall rotation | https://twitchoncall.pagerduty.com/services/PAKT4XA |
| Service Catalog | https://status.internal.justin.tv/services/392 |


## Development

### Verify prereqs

1. [Docker for mac](https://download.docker.com/mac/stable/Docker.dmg) stable.  You *need* docker version >= 18.03.
2. [mwinit](https://w.amazon.com/index.php/NextGenMidway/UserGuide#Mac) >= 1.24 `mwinit --version` (Note: Older versions work if you're on WPA2)
3. [teleport-bastion](https://wiki.twitch.com/display/SEC/Teleport+Bastion) `teleport-bastion status`
4. [aws cli](https://docs.aws.amazon.com/cli/latest/userguide/cli-install-macos.html) >= [1.11.91](https://github.com/aws/aws-cli/blob/develop/CHANGELOG.rst#11191): Only required to download the builder from ECR. `aws --version`

### Verify your setup is ready for development

```bash
# Run this in one terminal
./make.sh bastion tunnel
# Run this in *ANOTHER* terminal
./make.sh extra ok_for_development
```

### Write some code

Write code and make changes.  Feel free to push your changes to jenkins in a
branch to verify lints/checks.

### Run your application locally

```bash
./make.sh go run
```

### Check locally your code works

It's ok to offload this to a jenkins branch during normal development, but you
should verify at least once you can check your code locally.

```bash
./make.sh go lint
./make.sh go test
./make.sh go integration_test
./make.sh docker build
```

### Submit pull request to github and wait for review
```bash
git push origin HEAD:pull_request
# Create pull request on https://git-aws.internal.justin.tv/timotyenorg/timotyenservice
```

### Change cloudformation infrastructure 

To deploy changes to [infra.yaml](./cloudformation/infra/infra.yaml) you can use `cfmanage`.  First inspect your
cloudformation state

```bash
./make.sh infra manage inspect infra staging
##
## ... lots of output here
##
```

Next, deploy your cloudformation file

```bash
./make.sh infra manage execute infra staging
##
## ... lots of output here
##
```

### Merge to master to deploy 

Deployment is controlled via [Jenkinsfile](./Jenkinsfile).  Every push to master will run the deployment pipeline and
auto deploy to staging.  Manual confirmation is required for canary then production pushes.

### Code revert

Master *should* be what you want production to be, and a reverted push should be a `git revert` and push to master.
If however, you are in a rush all the jenkins commands, since they use a builder, are also runnable from the CLI!

```bash
GIT_COMMIT=abcdefg AWS_PROFILE=production-deploy ./make.sh infra manage execute deploy canary -a
GIT_COMMIT=abcdefg AWS_PROFILE=production-deploy ./make.sh infra manage execute deploy production -a
```

### Code profile

If you have go >= 1.11 installed locally, you can profile the service on port 6060.  A simple helper exists
to profile a random host in a ECS cluster.

```bash
# Requires teleport-bastion socks proxy running in the background
# If it hangs, stop and restart your tunnel
AWS_PROFILE=staging-infra ./make.sh ecs go_dump_profile staging
```

## Operation

This section includes operational tasks that an on-call would want to do for the service.  It should also include
a section on how to debug the service (Go here, try this, look for this, etc).

