MozMEAO SRE Status Report - 5/23/2017

23 May 2017, by Dave Parfitt

Here’s what happened on the MozMEAO SRE team from May 17th - May 23rd.

Current work

Bedrock has been deployed to our Virginia Kubernetes cluster, with our Tokyo cluster closely following shortly later today or Wednesday.

Gunicorn was upgraded and the Meinheld worker is now used to help prevent timeouts.
- Meinheld helped to resolve timeout issues in the snippets app as well.
- Thanks to pmac for the assistance!
We’ve cleaned up some Bedrock DNS entries
Some decom work on unused Bedrock prod/stage ELBs was performed.
ELB certificates and DNS entries were created as part of the move to K8s.
App limits and requests set: Dev, stage and production environment limits and requests have been set in Kubernetes in preparation for the production push. This allows Bedrock to take advantage of cluster and pod autoscaling, which is documented here.
- Since we’ve added K8s to the www.mozilla.org Route 53 Traffic Policy, we’ve had to fine tune the memory and cpu limits for better performance.

Multi-region EFS replication progress is in this WIP branch.
- the work in this branch will also be used in MySQL backup solution for AWS hosted MDN.
The last remaining Apache rewrites have been implemented in Django.
More work is needed to ensure the mime types are correct and consistent in Django.
httpd Alias directives need to be implemented in Django.

File storage
- store all additional media (including samples) on EFS.
Serving HTTP
- Django hosts everything, completely removing httpd.
Database
- We’re sticking with self-hosted MySQL running per-region in Kubernetes.
- We’ll need a custom backup solution
Elasticsearch.
- (waiting for confirmation) for the initial migration, use AWS hosted ES, keeping in mind that we can always self-host.

Waiting for some internal communications before moving forward.

We’re planning to move nucleus to Kubernetes, and then proceed to decommissioning current nucleus infra.

We’re planning to move basket to Kubernetes shortly after the nucleus migration, and then proceed to decommissioning existing infra.