MozMEAO SRE Status Report - 5/9/2017
Here’s what happened on the MozMEAO SRE team from May 3rd - May 9th.
Current work
Bedrock (mozilla.org)
Bedrock multi-region RDS provisioning
Work continues to move Bedrock from Deis 1/Fleet to Kubernetes. The team has implemented Terraform automation to provision RDS instances in multiple regions.
Demo deployments
Jenkins deployments have been restructured, and demos now build in main pipeline. This was a meaty PR from pmac, and a motivation to upgrade Deis Workflow to the latest version (more info below).
Next actions:
- create persistent development, staging, and production applications using RDS (Postgres)
- enable deployments to new apps in Jenkins
- Cloudfront distribution and integration testing
MDN
We’re working on migrating custom Apache config for MDN directly in Kuma/Django for the eventual move to AWS. Most of the Apache rewrites/redirects have been implemented in Kuma, with only a few remaining.
Basket
The FxA team would like to send events (FXA_IDs) to Basket and Salesforce, and need SQS queues in order to move forward. We automated the provisioning of dev/stage/prod SQS queues, and passed off credentials to the appropriate engineers.
Kubernetes / Deis Workflow
Deis Workflow has been upgraded to latest version (2.14.20) in Virginia and Tokyo. We hit a few snags during the first upgrade, as our Workflow install has some customization that wasn’t applied. Subsequent upgrades should be easier, as we’ve automated the process via a script (with minor tweaks in this PR).
Snippets
Snippets-stats is running in Tokyo and Virginia.
snippet-stats was already running on our Deis 1 clusters in Oregon and Ireland, however Giorgos enabled it on our Virginia and Tokyo Kubernetes clusters.
- Metrics have been validated for snippets-stats in Virginia and Tokyo.
- Application memory/CPU limits and autoscaling have been configured in Tokyo and Virginia.
Issues with HTTP_X_FORWARDED_PROTO header not set for for snippets-*.virginia.moz.works
We created a generic http to https redirector service that runs in Kubernetes. This allows Kubernetes to handle forwarding http to https for us without having custom implementations in each application. However, there remained an issue in our current ELB setup where HTTP_X_FORWARDED_PROTO was not set, and thus Django cannot be aware whether a connection is secure or not.
pmac has implemented an alternative to X-Forwarded-Proto using an HTTPS env var and a SWGIRequest subclass.
Thanks to Giorgos and pmac for their hard work on this!
Decommission webwewant.mozilla.org
webwewant.mozilla.org has been decommissioned. All requests to webwewant.mozilla.org are now being forwarded to https://www.mozilla.org.
Future work
Decommission openwebdevice.org
Waiting for some internal communications before moving forward.
Nucleus
We’re planning to move nucleus to Kubernetes, and then proceed to decommissioning current nucleus infra.
Basket
We’re planning to move basket to Kubernetes shortly after the nucleus migration, and then proceed to decommissioning existing infra.
New Kubernetes cluster
We’ll be creating a new Kubernetes cluster in Portland so we can take advantage of EFS to support MDN in that region. We currently run many of our services from Portland, Virginia, and Ireland. The new cluster will be created in an entirely new VPC, and existing resources will not be shared.