Here’s what happened on the MozMEAO SRE team from August 8th - August 15th.
Current work
MDN Migration to AWS
We’ve setup a few cronjobs to periodically sync static files from the current SCL3 datacenter to an S3 bucket. Our Kubernetes development environment runs a cronjobs that pulls these files from S3 to a local EFS mount.
There was some additional work needed to deal with files in SCL3 that contained unicode characters in their names.
A cronjob in Kubernetes has been implemented to backup new files uploaded to our shared EFS volume.
We’ve finished our evaluation of hosted Elasticsearch from elastic.co, which we’ll be using for our initial migration in production.
Here’s what happened on the MozMEAO SRE team from August 1st - August 8th.
Current work
MDN Migration to AWS
Our goal is to pilot a read-only maintenance mode with Kubernetes-hosted MySQL. For production, there’s some work related to MySQL custom collation that needs to be resolved before we move to AWS RDS. More on this in coming weeks.
We’re going to be storing static samples, diagrams, and presentations in a shared EFS persistent volume, and there’s work in progress on an automated backup solution to Amazon S3.
Here’s what happened in July in
Kuma,
the engine of
MDN Web Docs:
Shipped the new design to all users
Shipped the sample database
Shipped tweaks and fixes
Here’s the plan for August:
Continue the redesign and interactive examples
Update localization of macros
Establish maintenance mode in AWS
Done in July
Shipped the New Design to All Users
In June, we revealed the new MDN web docs design to beta testers. In July,
Stephanie Hobson and Schalk
Neethling fixed many bugs, adjusted
styles, shipped the homepage redesign, and answered a lot of feedback.
The new design was shipped to all MDN Web Docs users on July 25, and the old
design files were retired.
The redesign was a big change, with some interesting problems that called for
creative solutions. For details, see Stephanie’s blog post,
The MDN Redesign “Behind the Scenes”.
Shipped the Sample Database
The sample database project, started in May 2016, finally shipped in July.
Data is an important part of Kuma development. With the code and backing
services you get the home page, and not much else. To develop features or test
changes, you often need wiki pages, historical revisions, waffle flags,
constance settings, tags, search topics, users and groups. Staff developers
could download a 2 GB anonymized production database, wait 30 minutes for it to
load, and then they would have a useful dev environment. Contributors had to
manually copy data from production, and usually didn’t bother. The sample
database has a small but representative data set, suitable for 90% of
development tasks, and takes less than a minute to download and install.
The sample database doesn’t have all the data on MDN, to keep it small.
There are now
scraping tools
for adding more production data to your development database.
This is especially useful for development and testing of
KumaScript macros,
which often require specific pages.
Finally, integration testing is challenging because non-trivial testing
requires some known data to be present, such as specific pages and editor
accounts. Now, a testing deployment can combine new code with the sample
database, and automated browser-based tests can verify new and old
functionality. Some tests can change the data, and the sample data can
be reloaded to a known state for the next test.
Kuma PR 4338:
Move the locale files from kuma to their own repository,
mozilla-l10n/mdn-l10n,
from John Whitlock.
This will allow for the higher volume of locale changes when KumaScript
string are added, without requiring unnecessary source builds.
Kumascript PR 241:
Fix API calls such as page.subpagesExpand, making more macros
work in the Docker development and production environments, from
Ryan Johnson
Kumascript PR 248:
The first of many PRs adding Brazilian Portuguese (pt-BR) from
first time contributor
Fernandolrs.
Continue the redesign and the interactive examples
We’ve established the new look-and-feel of MDN on the homepage and article
pages, and will continue to tweak the design for corner cases and bugs. For the
next phase, we’ll look at the content of article pages, and consider better
ways to display information and to navigate within and between pages. It is
harder to change these aspects than global headers and footers, so it may be
a while before you see the fruits of this design process.
Work continues on the interactive examples. They have gone through several
review and bug fix cycles, and have a working production deployment system.
There’s been interest and work to enable contributions
(Issue 99).
In August, we’ll launch user testing, and enable the new examples for beta
testers. See the
projects page
for the remaining work.
Update Localization of Macros
Currently, KumaScript macros use in-macro localization strings and
utility functions like
getLocalString
to localize output for three to five languages. Meanwhile, user interface strings in
Kuma are translated in
Pontoon into 57 languages. We’d
like to use a similar workflow for strings in macros.
In August, we’ll assemble the toolchain for localizing strings at render time,
and for extracting the localizable strings for translation in Pontoon.
Converting the macros to use localizable strings will be a long process, but
there’s a lot of community interest in translations, so we should get some
help.
Establish Maintenance Mode in AWS
Over the past 12 months, we’ve made some changes to MDN development, such as
switching to a Docker development environment, moving Kumascript macros to
Github, and getting our browser-based integration tests working. There are
benefits to each of these, but they were chosen because they move us closer to
our long term goal of serving MDN from AWS. We’ve slowly filled out our
tech tree from our
AWS plan:
In August, we plan to prepare a maintenance mode deployment in AWS, and send
some production traffic to it. This will allow us to model the resources
needed when the production environment is hosted in AWS. It will also keep MDN
data available while the production database is transferred, when we finalize
the transition.
MDN interactive examples can now be hosted in S3/Cloudfront. This allows us to deploy interactive examples to an S3 bucket via Jenkins, viewable at interactive-examples.mdn.mozilla.net.
Decommissioning the Virginia cluster
The last remaining services running on the Virginia cluster have been moved to other regions or hosting options:
