MozMEAO SRE Status Report - July 18, 2017

Here’s what happened on the MozMEAO SRE team from July 11th - July 18th.

Current work


Decommissioning old infrastructure

We’re planning on decommissioning our Deis 1 infrastructure starting with Ireland, as our apps are all running on Kubernetes in multiple regions. Once the Ireland cluster has been shut down, we’ll continue on to our Portland cluster.

Additionally, we’ll be scaling down our Virginia cluster, as our apps are being moved to regions with lower latencies for the majority of our users.

MozMEAO SRE Status Report - July 11, 2017

Here’s what happened on the MozMEAO SRE team from July 5th - July 11th.

This weeks report is brief as the team is returning from the Mozilla San Francisco All Hands and vacation.

Current work

Static site hosting


  • Our main applications are being moved to our new Frankfurt Kubernetes cluster.

Kuma Report, June 2017

Here’s what happened in June in Kuma, the engine of MDN Web Docs:

  • Shipped the New Design to Beta Testers
  • Added KumaScript macro tests
  • Continued MDN data projects
  • Shipped tweaks and fixes

Here’s the plan for July:

  • Continue the redesign
  • Experiment with on-site interactive examples
  • Update localization of macros
  • Ship the sample database

Done in June

Shipped the new design to beta testers

This month, we revealed some long-planned changes. First, MDN is focusing on web docs, which includes changing our identity from “Mozilla Developer Network” to “MDN Web Docs”. Second, we’re shipping a new design to beta users, to reflect Mozilla’s new brand identity as well as the MDN Web Docs brand.

Stephanie Hobson did a tremendous amount of work over 26 Kuma PRs and 2 KumaScript PRs to launch a beta of the updated design on wiki pages. A lot of dead code has been removed, and non-beta users continue to get the current design. Schalk Neethling reviewed the PRs as fast as they were created, including checking the rendering in supported browsers. Our beta users have provided a lot of feedback and found some bugs, which Stephanie has been triaging, tracking, and fixing.

This work continues in July, with an update to the homepage and other pages. When we’ve completed the redesign, we’ll ship the update to all users. If you want to see it early, opt-in as a beta tester.

Added KumaScript Macro Tests

Macros used to be tested manually, in production. After moving the macros to GitHub, they were still tested manually, but in the development environment. In June, Ryan Johnson added an automated testing framework, and tests for five macros, in PR 204. This allows us to mock the Kuma APIs needed for rendering, and to test macros in different locales and situations. This will help us refine and refactor macros in the future.

Continued MDN Data Projects

The MDN data projects were very busy in June, with 48 browser-compat-data PRs and 10 data PRs merged. MDN “writers” Florian Scholz, wbamberg, and Eric Shepherd have been converting MDN browser compatibility tables to JSON data, refining the schema and writing documentation. This is already becoming a community project, with almost half of the PRs coming from contributors such as Andy McKay (1 PR). Dominik Moritz (2 PRs), Roman Dvornov (6 PRs), Ng Yik Phang (2 PRs), and Sebastian Noack (16 PRs!).

The tools and processes are updating as well, to keep up with the activity. browser-compat-data gained a linter in PR 240. mdn-browser-compat-data’s npm package was bumped to version 0.0.2, and then 0.0.3. mdn-data was released as 1.0.0. We’re loading the browser-compat-data from the NPM package in production, and hope to start loading the data NPM package soon.

We’re happy with the progress on the data projects. There’s a lot of work remaining to convert the data on MDN, and also a lot of work to automate the process so that changes are reflected in production as quickly as possible.

Shipped Tweaks and Fixes

Here’s some other highlights from the 44 merged Kuma PRs in June:

Here’s some other highlights from the 31 merged KumaScript PRs in June:

Planned for July

Continue the Redesign

Some of the styling for the article pages is shared across other pages, but there is more work to do to complete the redesign. Up next is the homepage, which will change to reflect our new focus on documenting the open web. Other pages will need further work to make the site consistent. When we and the beta testers are mostly happy, we’ll ship the design to all MDN visitors, and then remove the old design code.

Experiment with On-site Interactive Examples

We’re preparing some interactive examples, so that MDN readers can learn by adjusting the code without leaving the site. We’re still working out the details of serving these examples at production scale, so we’re limiting the July release to beta users. You can follow the work at mdn/interactive-examples.

Update Localization of Macros

Currently, KumaScript macros use in-macro localization strings and utility functions like getLocalString to localize output for three to five languages. Meanwhile, user interface strings in Kuma are translated in Pontoon into 57 languages. We’d like to use a similar workflow for strings in macros, and will get started on this process in July.

Ship the Sample Database

The Sample Database has been promised every month since October 2016, and has slipped every month. We almost had to break the tradition, but we can say again it will ship next month. PR 4248, adding the scrape_document command, shipped in June. The final code, PR 4076, is in review, and should be merged in July.


We enabled HTTP/2 on MDN’s CDN.

We didn’t do anything to optimize for HTTP/2, we just enabled it.

We’re seeing performance improvements.

You don’t have to get ready before you start using HTTP/2

While doing research to see if turning it on without doing any optimizations was a good idea I read things like:

“It also means that all of those HTTP1 performance techniques are harmful. They will make a HTTP2 website slower, not faster - don’t use them.” - HTTP2 for front-end web developers


“However, many of the things you think of as being best practices can be detrimental to performance under HTTP/2.” - Getting Ready For HTTP2: A Guide For Web Designers And Developers

Which suggest that enabling HTTP/2 on a site optimized for HTTP/1.1 could result in a slower site.

A better way to interpret those quotes is:

If you optimize for HTTP/1.1 and turn on HTTP/2 your site will not be as fast as it could be - but it might still be faster!

On MDN we concatenate a lot of our files but we don’t concatenate all of them. For example, our article pages have 9 different files coming from our CDN. I thought we could benefit from a bit of HTTP/2’s multiplexing and header compression. And we did. You can see the DNS lookup time drop off in this waterfall from Pingdom:

Waterfall showing over 0.3s of DNS look up for each request.

Waterfall showing DNS lookup for only first asset requested.

Some numbers

Overall, our tests don’t show a huge improvement in page load speed but there are small improvements for everyone, and a real improvement for users located far away from our servers. (Hi Australia and China!)

Service Location Browser HTTP/1.1 HTTP/2 Change
Pingdom Dallas Chrome 1.54s 1.34s 0.2s
Pingdom Melbourne Chrome 2.94s 2.80s 0.14s
WebPageTest London IE11 2.39s 2.37s 0.02s
WebPageTest Australia Firefox 5.61s 5.17s 0.44s
Google Analytics All Chrome 3.74s 3.04s 0.7s
Google Analytics All Firefox 3.99s 3.71s 0.28s
Google Analytics Australia All 3.01s 1.69s 1.32s
Google Analytics China All 8.10s 6.69s 1.41s

I tried to segment our users in Google Analytics to make sure we did not have a negative impact on users relying on HTTP/1.1 and… I couldn’t find enough users to draw any conclusions. MDN is lucky like that. (It’s possible the IE11 test in the table above is on Windows 7 and does not support HTTP/2, but WebPageTest doesn’t identify the OS.) In theory, older browsers should not be affected because the protocol falls back to HTTP/1.1.

There was a lot of variation in the page speed data I examined. I recommend running your before and after benchmark tests multiple times on multiple days so you can take an average. Try to wait a week before drawing conclusions from your analytics data as well.

In a perfect world you don’t increase the amount of code on your site or invalidate anyone’s caches in the sample time period, but we don’t develop in a perfect world.

Read more on HTTP/2


Get our pages into data centres around the world.

This involves changing our hosting services, not a small task, and changing our pages to serve the same content to all logged out users.

Decrease asset size by removing support for older browsers.

If you think working on MDN was a great job because we have very modern browser support requirements, remember we’re also working on a 10 year old code base.

Thanks for using MDN!

MozMEAO SRE Status Report - June 20, 2017

Here’s what happened on the MozMEAO SRE team from June 13th - June 20th.

Current work

Static site hosting

  • The irlpodcast site now has a staging environment also hosted in S3 with CloudFront. Additionally, Jenkins has been updated to deploy to staging and production via git push.

  • We’re going to move from Kubernetes to S3 and CloudFront hosting. Production and staging environments have been provisioned, but we’ll need to update Jenkins to push changes to these new environments.

Basket move to Kubernetes

Kubernetes (general)

Our DataDog, New Relic and MIG DaemonSets have been configured to use Kubernetes tolerations to schedule pods on master nodes. This allows us to capture metrics from K8s master nodes in additional to worker nodes.

Frankfurt Kubernetes cluster provisioning

Work continues to enable our apps in the new Frankfurt Kubernetes cluster. In addition, we’re working on automating our app installs as must as possible.


  • ElasticSearch will be upgraded to 2.4 in SCL3 production, June 21 11 AM PST

  • We may reconsider self-hosting ElasticSearch.