# Aperture Runbook

This describes the different components of Aperture and how to operate them.

[twitch-cb-aws account](https://twitch-cb-aws.signin.aws.amazon.com/console)

[Pagerduty](https://twitchoncall.pagerduty.com/services/PZA78Q0)

[Service Catalog](https://servicecatalog.internal.justin.tv/services/487)

[Github](https://git.xarth.tv/businessviewcount/aperture)

[clean-deploy](https://clean-deploy.internal.justin.tv/#/businessviewcount/aperture)

## Table of Contents
1. [Deploying and Rollbacks](#deploying-and-rollbacks)
2. [Monitoring and Errors](#monitoring-and-errors)
3. [Pagerduty](#pagerduty)
4. [Infrastructure](#infrastructure)
    1. [Terraform](#terraform)
    2. [Elastic Beanstalk](#elastic-beanstalk)
    3. [Kinesis Data Streams](#kinesis-data-streams)
    4. [Kinesis Analytics](#kinesis-analytics)
    5. [Elasticache](#elasticache)
    6. [Lambdas](#lambdas)
    7. [Parameters and Configuration](#parameters-and-configuration)
5. [Past Alarms](#past-alarms)

### Deploying and Rollbacks
See the deployment [docs](/docs/deployment.md) for help with deploying and rolling back.

### Monitoring and Errors
Errors are logged to [rollbar](https://rollbar.com/Twitch/Aperture/items/).
There is a Grafana dashboard [here](https://grafana.internal.justin.tv/d/K0W6FxXiz/business-view-count-aperture).
Beanstalk apps can also be monitored from the AWS console:

1. [Production](https://us-west-2.console.aws.amazon.com/elasticbeanstalk/home?region=us-west-2#/application/overview?applicationName=cb-aperture-production)
2. [Staging](https://us-west-2.console.aws.amazon.com/elasticbeanstalk/home?region=us-west-2#/application/overview?applicationName=cb-aperture-staging)

Note that you need to request access to Rollbar and to the AWS account where the application lives (but grafana should be accessible). For access, ask in #business-view-count on slack
or ping Jeffrey Lym.

### Pagerduty
[Pagerduty](https://twitchoncall.pagerduty.com/services/PZA78Q0)
Pagerduty is integrated with both Cloudwatch and rollbar. Pages are sent based on error rates and application health. See [alarms](/docs/alarms.md) for details.

## Infrastructure

### Terraform
Infrastructure configuration is done using terraform. All these files live in the git rep, in the `terraform` folder. Changes should be applied to both staging and production, and committed to the repo using a pull request. If you need assistance with terraform, ask in #business-view-count on slack.

### Elastic Beanstalk
Aperture's API exists as an elastic beanstalk application, for both staging and production environments.

1. [Production](https://us-west-2.console.aws.amazon.com/elasticbeanstalk/home?region=us-west-2#/application/overview?applicationName=cb-aperture-production)
2. [Staging](https://us-west-2.console.aws.amazon.com/elasticbeanstalk/home?region=us-west-2#/application/overview?applicationName=cb-aperture-staging)

### Kinesis Data Streams
Aperture processes minute watched data using a kinesis stream provided by Data Infrastructure.

1. [Production](https://us-west-2.console.aws.amazon.com/kinesis/home?region=us-west-2#/streams/details?streamName=spade-downstream-prod-cb-aperture-prod-mw&tab=details)
2. [Staging](https://us-west-2.console.aws.amazon.com/kinesis/home?region=us-west-2#/streams/details?streamName=spade-downstream-prod-cb-aperture-staging-mw&tab=details)

### Kinesis Analytics
A kinesis analytics application runs on the minute-watched stream. This will run some SQL on the stream, which calculates our squad stream secondary player ratio.
The result is then output to a lambda, for writing to our elasticache cluster.

The SQL that the kinesis analytics app runs can be found in the repo [here](https://git.xarth.tv/businessviewcount/aperture/blob/master/kinesis_analytics/mw_ratio.sql).

1. [Production](https://us-west-2.console.aws.amazon.com/kinesisanalytics/home?region=us-west-2#/wizard/hub?applicationName=cb-aperture-production-minute-watched-ratio-application)
2. [Staging](https://us-west-2.console.aws.amazon.com/kinesisanalytics/home?region=us-west-2#/wizard/hub?applicationName=cb-aperture-staging-minute-watched-ratio-application)

### Elasticache
Minute watched ratios are stored in a memcached cluster:

1. [Production](https://us-west-2.console.aws.amazon.com/elasticache/home?region=us-west-2#memcached-nodes:id=cb-aperture-prod;nodes)
2. [Staging](https://us-west-2.console.aws.amazon.com/elasticache/home?region=us-west-2#memcached-nodes:id=cb-aperture-staging;nodes)

### Lambdas
There are a few different lambdas used by Aperture:

1. output_mw_ratio_to_elasticache
    - The Kinesis Analytics application outputs the results of its query to this lambda, which is responsible for writing the output of the query to elasticache
    - [Production](https://us-west-2.console.aws.amazon.com/lambda/home?region=us-west-2#/functions/cb-aperture-production-output-mw-ratio-lambda?tab=graph)
    - [Staging](https://us-west-2.console.aws.amazon.com/lambda/home?region=us-west-2#/functions/cb-aperture-staging-output-mw-ratio-lambda?tab=graph)

2. error_stream_logger
    - The Kinesis Analytics application outputs the errors of its query to this lambda, which is responsible for writing the output of the query to rollbar
    - [Production](https://us-west-2.console.aws.amazon.com/lambda/home?region=us-west-2#/functions/cb-aperture-production-error-stream-logger-lambda?tab=graph)
    - [Staging](https://us-west-2.console.aws.amazon.com/lambda/home?region=us-west-2#/functions/cb-aperture-staging-error-stream-logger-lambda?tab=graph)

3. spade_logger
    - Calls the aperture API in order to write recent channel concurrents data to spade. This lambda runs on a 1 minute schedule, based on a cloudwatch rule.
    - [Production](https://us-west-2.console.aws.amazon.com/lambda/home?region=us-west-2#/functions/cb-aperture-production-spade-logger-lambda?tab=graph)
    - [Staging](https://us-west-2.console.aws.amazon.com/lambda/home?region=us-west-2#/functions/cb-aperture-staging-spade-logger-lambda?tab=graph)

4. pubsub_sender
    - Like spade_logger, calls the aperture API to send recent channel concurrents to pubsub. Runs on a 1 minute schedule, based on a cloudwatch rule.
    - [Production](https://us-west-2.console.aws.amazon.com/lambda/home?region=us-west-2#/functions/cb-aperture-production-pubsub-sender-lambda?tab=graph)
    - [Staging](https://us-west-2.console.aws.amazon.com/lambda/home?region=us-west-2#/functions/cb-aperture-staging-pubsub-sender-lambda?tab=graph)

### Parameters and Configuration
Configuration variables are stored using AWS Systems Manager. Values for Aperture are stored with the prefix `/configurations/cb/<environment>/aperture`.

1. [Production/Staging](https://us-west-2.console.aws.amazon.com/systems-manager/parameters/?region=us-west-2#)

### Cloudwatch Rules
The spade and pubsub lambdas are run on cloudwatch event rules. These can be enabled/disabeld from the console:

1. Spade logger
    - [Production](https://us-west-2.console.aws.amazon.com/cloudwatch/home?region=us-west-2#rules:name=cb-aperture-production-spade-logger-lambda-event)
    - [Staging](https://us-west-2.console.aws.amazon.com/cloudwatch/home?region=us-west-2#rules:name=cb-aperture-staging-spade-logger-lambda-event)

2. Pubsub sender
    - [Production](https://us-west-2.console.aws.amazon.com/cloudwatch/home?region=us-west-2#rules:name=cb-aperture-production-pubsub-sender-lambda-event)
    - [Staging](https://us-west-2.console.aws.amazon.com/cloudwatch/home?region=us-west-2#rules:name=cb-aperture-staging-pubsub-sender-lambda-event)

### Past Alarms
See [alarms](/docs/alarms.md) for details.
