<!-- TOC -->
# Quicklinks
<table>
<tbody align="center">
<tr class="odd">
<td>

[Contribution Guide](contributing.md)

</td>
<td>

[VPC Endpoints (Private Links) Guide](./terraform/modules/vpc-endpoint/README.md)

</td>
<td>

[Local Development Guide](#local-development)

</td>
</tr>
<tr class="even">
<td>

[Architecture Document](https://docs.google.com/document/d/1JB_f3QQw9Bf0PY1cymhn4j98jmXSmWMzpCpf_xZ19s8/edit)

</td>
<td>

#[admin-services] on Slack

</td>
<td>

[ADMIN] on JIRA

</td>
</tr>
<tr class="even">
<td>

Production  
[admin-panel.xarth.tv](https://admin-panel.internal.justin.tv)

</td>
<td>

Staging  
[**staging**.admin-panel.xarth.tv](https://staging.admin-panel.xarth.tv)

</td>
<td>

Canary  
[**canary**.admin-panel.xarth.tv](https://canary.admin-panel.xarth.tv)

</td>
</tr>
</tbody>
</table>

# Table of Contents

| [First Time Setup](#first-time-setup)                                                                                                                                                                                                                                                                                                                                                                                                     | [Running Server Locally](#running-server-locally)                                                                                                                  | [Utilities](#utilities)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | [Troubleshooting](#troubleshooting)                                      |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |   | ------------------------------------------------------------------------ |
| <ol start="1"><li><a href="#developer-tools">Developer tools</a></li><li><a href="#ldap-group-access">LDAP Group Access</a></li><li><a href="#s2s-and-sandstorm-onboarding">S2S and Sandstorm onboarding</a></li><li><a href="#ldap-hosts-entry">LDAP hosts entry</a></li><li><a href="#docker-network-connectivity">Docker Network Connectivity</a></li><li><a href="#install-ruby-dependencies">Install ruby dependencies</a></li></ol> | <ol start="1"><li><a href="#credential-injection">Credential Injection</a></li><li><a href="#start-the-server-in-the-background">Starting the server</a></li></ol> | <ul><li><a href="#viewing-logs">Viewing logs</a><ul><li><a href="#console-logs">Console</a></li><li><a href="#rails-logger">Rails</a></li></ul></li><li><a href="#running-bundler-scripts">Running Bundler Scripts</a><ul><li><a href="#linting">Linting</a></li><li><a href="#unit-tests">Unit Testing</a></li><li><a href="#integration--feature-tests">Integration / Feature Testing</a></li><li><a href="#generating-templated-admin-pages">Generating Templated Pages</a></li><li><a href="#workers">Using Workers</a></li></ul></li><li><a href="#twirp-clients">Generating Twirp Clients</a></li></ul> | <ul><li><a href="#file-cache-stucking">File cache stucking</a></li></ul> |

<!-- /TOC -->

# Local Development

## First Time Setup

#### Developer tools

You'll need to install a couple of developer tools.

```bash
    brew tap twitch/internal git@git-aws.internal.justin.tv:common/homebrew-custom.git
    brew tap twitch/security https://twitch-security-packages.s3.amazonaws.com/homebrew.git
    brew update
    brew install twitch-bastion-util twinit
```

- [docker]
- [docker-compose]

#### LDAP Group Access

Contact someone in the [team-admin-panel-contributors-admin] LDAP group via #[admin-services] to be added to the [team-admin-panel-contributors] group.  
**You need to be in this group to access the bastion jump host required to create a proxy to the staging network stack, as well as to onboard yourself onto s2s and Sandstorm.**

It can take up to **two hours** for the bastion system to register your permissions. You also may need to explictly log out (go to https://login.xarth.tv/ext/logout) and log back in to SSO for permissions to kick in.

#### S2S and Sandstorm onboarding

You'll need to whitelist an IAM ARN for:

- the [admin-panel-staging] s2s role.
- the [admin-panel-development] sandstorm role.

This should ideally be the ARN of your team's **dev/staging account Isengard role**, but can be a local IAM user ARN if this is not feasible.  
*Using a common role via Isengard allows for all team members to be whitelisted in one go, and removes access automatically when members leave your team.*

#### LDAP hosts entry

For local development, the LDAP hosts are accessed via SSH port forwarding with
a TLS secured connection. You will need to add a DNS entry if you are developing
on OSX to map LDAP to the loopback device.

Open your `/etc/hosts` file and add the following entry.

```bash
    127.0.0.1 ldap-usw2.internal.justin.tv
```

#### Docker Network Connectivity

All of the services that admin panel needs to talk to are only accessible from the VPC that it runs out of. We use [teleport remote] to create a SOCKS5 proxy and a couple of port forwarding rules to gain connectivity.

Make sure you have `teleport-bastion` enabled.

You should be able to access Admin Panel's bastion host automatically if you gained `team-admin-panel-contributors` LDAP group access above.

```bash
    teleport-bastion enable
```

In a separate long-running shell, run:

```bash
    make localdev-proxy
```

The rails app's `development` environment is configured to use this proxy for LDAP access.

**If you encounter issues using teleport-bastion or related proxies**
- reference the [teleport documentation] as it covers many common failure states, such as 
    - ssh key invalidation
    - mismatched usernames
    - bastion access cache stucking
    - etc.  
If you then still require assistance, talk to the Teleport managing team in the #[teleport_bastion slack] channel.

**Before you can install dependencies**, you'll need to start a separate `twinit` proxy to create an HTTP proxy to allow docker to communicate with the internal gem server (gem.internal.justin.tv).

Twinit can either be run inside a shell, or as a daemon
- `twinit foreground` runs the proxy inside the current shell. This is typically useful for short-term access such as installing dependencies, or for debugging general bastion SSH access issues.
- `twinit enable` is used to run the proxy in the background as a daemon. It includes a built-in watchdog to reconnect if disconnected.  
    note: `teleport-bastion` SSO login windows will spawn in your primary browser when `twinit` connects initially. Teleport certificates live for 30 minutes post-generation, so this may also occur during reconnection after network connectivity loss ( network changes, system sleep, etc. )

For *general* resource access inside Admin Panel, you can either: 

- Remain on the Twitch VPN while running local dev
- Configure Docker to utilize `twinit` as its default proxy: 
    - run `twinit enable` to run twinit in daemon mode
    - configure Docker Desktop preferences to use the local `twinit` proxy  
        ```
        Docker Desktop > Preferences
        Resources > Proxies
            [☑️] Manual proxy configuration
            Web Server (HTTP)
                http://docker.for.mac.host.internal:12345
            Secure Web Server
                http://docker.for.mac.host.internal:12345
            Bypass proxy settings for these hosts & domains
                admin-panel.us-west-2.beta.twitchbeefcake.services.twitch.a2z.com
        ```

#### Install ruby dependencies

Running the following will install all required bundle files locally:

```
docker-compose up bundle-install
```

*This will only need to be done once to begin with, and again any time the [Gemfile](./Gemfile) is updated.*

If you encounter issues installing gems, please refer to the **Network Connectivity** segment above.

## Running Server Locally

### Credential Injection

Source `docker-env.sh` in one of the following ways to inject your AWS credentials into your env vars for Docker:

> #### \[default\] AWS Credentials:
>
> ```bash
>       . ./docker-env.sh
> ```
>
> #### Profile AWS Credentials:
>
> ```bash
>       AWS_PROFILE=twitch-admin-panel-dev/admin . ./docker-env.sh
> ```
>
> #### Environment Credentials:
>
> ```bash
>       AWS_ACCESS_KEY_ID=ASIAGOL2WVIOHLATTE AWS_SECRET_ACCESS_KEY=ayylmao . ./docker-env.sh
> ```
>
> #### Usage / Configuration Priority
>
> ```
> ./docker-env.sh
>
> docker-env:
>
> === Usage:
>
>     Profile AWS Credentials:
>       AWS_PROFILE=twitch-admin-panel-dev/admin . ./docker-env.sh
>
>     Default AWS Credentials:
>       . ./docker-env.sh
>
>     Environment Credentials:
>       AWS_ACCESS_KEY_ID=ASIAGOL2WVIOHLATTE AWS_SECRET_ACCESS_KEY=ayylmao . ./docker-env.sh
>
> === Config Status & Priority Order:
>     Using AWS_PROFILE         : False
>     Using AWS_ACCESS_KEY_ID   : False
>     Using AWS CLI Default key : True
>
>
> === Please source me as above next time; Have a good day!
>
> ```

### Start the server in the background:

```
docker-compose up -d
```

## Utilities

### Viewing logs

#### Console logs

```
docker-compose logs -f app
```

```
docker-compose logs -f worker
```

#### Rails Logger

*Rails logging is currently painful to parse. We're working on legibility updates.*

While cd'd into the Admin Panel directory:

- Rails Server log
    ```
    tail -F log/development-docker.log
    ```
- RSpec Test log
    ```
    tail -F log/test.log
    ```

### Running bundler scripts:

#### Linting
```
docker-compose run --rm -e RAILS_ENV="test" app bundle exec rubocop
```

#### Unit Tests
```
docker-compose run --rm -e RAILS_ENV="test" app bundle exec rspec
```

#### Integration / Feature Tests

To perform feature testing *locally*:

1. Start your local development environment and verify it's accepting new requests
2. Tell RSpec to run specs of `type:feature`
    ```
    docker-compose run --rm -e RAILS_ENV="test" app bundle exec rspec -t type:feature
    ```

Integration / Feature tests are run against the Admin Panel Staging stack, so please set expectations on data access accordingly; Compliant services in staging environments return valid but *scrubbed* user information, and as such cannot be used as a direct environment comparison for data integrity - just data formats.  
As an immediate example, Users Service Staging returns fake email addresses and phone numbers for users.

For examples of existing feature tests, look within the [spec/features](./spec/features) folder.  
*We're consistently working on simplifying the process and grammar of feature testing; At time of writing, feature tests are relatively simple to create, but as they proxy "real" user interaction, they require permissions to be added to the `srvc-adminpanel-integrationtest` user for the functionality being tested.*

> During builds, Jenkins runs these tests to validate critical functionality, as well as to provide much higher confidence in the consistent functionality of existing tooling. As such, ensure that your feature tests do not perform activities that may impact your service negatively, and provide fallbacks in the code you're testing against to prevent service failures from failing the test ( unless business-critical ), build, and ( eventually, on production ) crashing the page.
>
> Jenkins uses the following roles to authenticate S2S calls and get staging secrets. These may differ from the staging secrets gathered by your local development environment. 
>
> - Sandstorm: [admin-panel-staging-integration-testing]
> - S2S: [admin-panel-staging]

#### Generating templated admin pages

Use admin panel's custom generators to generate a sample admin page as a starting point:

    docker-compose run --rm -e RAILS_ENV="test" app bundle exec rails g admin:scaffold <ServiceName> <attribute1> <attribute2> ...

**admin:scaffold** invokes:

    rails g admin:service <ServiceName> <attribute1> <attribute2> ...
    rails g admin:scaffold_controller <ServiceName> <attribute1> <attribute2> ...

**admin:service** generates service classes under `app/services/`

**admin:scaffold_controller** creates controllers under `app/controllers/`. It also invokes:

    rails g admin:views <ServiceName> <attribute1> <attribute2> ...
    rails g admin:resource_route <ServiceName>
    rails g helper <ServiceName>

**admin:views** generates the views for the controller under `app/views/`. It also adds a sidebar entry into `app/views/layouts/_sidebar.html.haml`

**admin:resource_route** Adds resource routes in `config/routes.rb`

**helper** is generic rails helper generator

**Example:**

    docker-compose run --rm -e RAILS_ENV="test" app bundle exec rails g admin:scaffold FoobarService key value

generates:

    invoke  scaffold_controller
    create    app/controllers/app_settings/items_controller.rb
    invoke    resource_route
     route      namespace :app_settings do
                  resources :items
                end
    invoke    views
    create      app/views/app_settings/items/index.html.haml
    create      app/views/app_settings/items/edit.html.haml
    create      app/views/app_settings/items/show.html.haml
    create      app/views/app_settings/items/new.html.haml
    create      app/views/app_settings/items/_form.html.haml
    create      app/views/app_settings/items/_list.html.haml
    create      app/views/app_settings/items/_filters.html.haml
    append      app/views/layouts/_sidebar.html.haml
    create      app/views/app_settings/_tabs.html.haml
    append      app/views/app_settings/_tabs.html.haml
    invoke    helper
    create      app/helpers/app_settings/item_helper.rb
    invoke  service
    create    app/services/app_settings/base.rb
    create    app/services/app_settings/item.rb

**Note:** By default, Generate creates a service class named `Item` namespaced under `FoobarService` with enough support to display, modify, and delete app settings. A slight modification is required if you require a custom class names:

e.g., to add a sub-menus/sub-classes named `ArbKeys` instead of `Item` under `FoobarService`:

    docker-compose run --rm -e RAILS_ENV="test" app bundle exec rails g admin:scaffold FoobarService::ArbKeys

#### Workers

We use the [resque](https://github.com/resque/resque) library with the [resque-status](https://github.com/quirkey/resque-status) plugin for managing workers. 

To start a worker, run:

    $ docker-compose run --rm -e RAILS_ENV="test" -e QUEUE="*" app bundle exec rake resque:work

If you would like to add your own jobs to admin-panel, please make sure to first read through the resque-status README, then add new jobs to `app/jobs`.

A web interface to the job queue is available at `/jobs`, where you can see the status of any currently running jobs, as well as any failed jobs.

#### Twirp Clients

1. Install `protoc` at version 3.6.1. You can validate using `protoc --version`.
2. Add your service.proto to `./twirp/<service-name>/service.proto`.  
    **Currently, we only support having all of your twirp definitions in a single proto file**;  
    Multiple files may cause issues with the ruby generated clients.
3. Run `make twirp-gen` to generate your client.

#### Autheticating With S2S2

All client authenticate via legacy [S2Sv0] by default. If you want to authenticate via
[S2S2], do the following.

Add your service URL to the `s2s2` keys in `./config/settings.yml`. Make sure to
update `development`, `staging`, and `production`.

```
  s2s2:
    hosts:
      - url: https://haberdashher.services.twitch.a2z.com
        scopes: []
```

Next, when you create your connection, use the `s2s2_middleware` as an
additional client middleware.

```
module BeefCake
    class Base < Service::Base

        def self.client
            @client ||= Twitch::Fulton::Beefcake::TwitchBeefcakeClient.new(
                connection(additional_middleware: [s2s2_middleware])
            )
        end
    end
end
```

In your service unit tests, you'll need to use this context to stub out calls to
retrieve AWS credentials.

```
describe TwitchZuko::RedemptionsHistory do
  include ServiceHelper

  include_context 'with s2s2'

  describe "#call" do
  end
end
```

## Troubleshooting

#### File cache stucking

If you make a code change and the running app doesn't automatically reflect it on-refresh, force Rails to reload the environment from disk by restarting either the App container:

```
docker-compose restart app && docker-compose logs -f app
```

or the Worker container, if the change is related to workers:

```
docker-compose restart worker && docker-compose logs -f worker
```

[ADMIN]: http://jira.twitch.com/browse/ADMIN
[admin-services]: https://slack.com/app_redirect?channel=C1J3275MG
[docker]: https://docs.docker.com/docker-for-mac/install/
[docker-compose]: https://docs.docker.com/compose/install/
[S2Sv0]: https://wiki.twitch.com/x/h4SHCg
[S2S2]: https://wiki.twitch.com/x/1pxkDQ

[admin-panel-staging]: https://dashboard.xarth.tv/s2s/services/YTgxOWIxYzctNGQzNi00OThmLWI2NWUtNWRlYmY3OThkYzNmOmFkbWluLXBhbmVsLXN0YWdpbmc
[admin-panel-development]: https://dashboard.xarth.tv/sandstorm/manage-roles?action=editrole&role=admin-panel-development
[admin-panel-staging-integration-testing]: https://dashboard.xarth.tv/sandstorm/manage-roles?action=editrole&role=admin-panel-staging-integration-testing

[teleport remote]: https://wiki.twitch.com/display/SEC/Teleport+Remote
[teleport documentation]: https://wiki.twitch.com/display/SEC/Teleport+Bastion
[teleport_bastion slack]: https://slack.com/app_redirect?channel=C7YUZR6SJ

[team-admin-panel-contributors]: https://dashboard.xarth.tv/ldaptools/group/team-admin-panel-contributors
[team-admin-panel-contributors-admin]: https://dashboard.xarth.tv/ldaptools/group/team-admin-panel-contributors-admin
