Quality Standards

Version 1, last updated on 2023-10-19

Hello, this doc is about explicitly defining quality in the world of services software at Mozilla.

Definition: quality refers to the degree to which a software and its related deliverables meet or exceed the expectations of its stakeholders. However, what are those expectations and does the team share the same expectations? This document defines what the expectations are and provides a way to measure the quality of a software.

Quality guarantees reliability and maintainability of the software, as well as efficiency of development.

The goal of this initiative is to increase the familiarity of codebases, so we can be effective sooner. Ultimately, this means more of our time can be spent on innovation and less on just keeping the lights on.

More projects than people is a common situation for service engineers at Mozilla. As priorities shift, reorgs happen, projects are moved around, at some point we are maintaining several codebases. Most of these projects were not developed to a shared set of quality criteria so there is little familiarity between them.

Note: Instead of enforcing a limited set of rules, we made the choice to have many rules and a global score. Scorecards show complicance in each category, which allows progress in iterations.
A messy workbench © Dan Larson A professional workbench © mtneer_man

Objectives:

Non Goals:

For comments and questions, don't hesitate to get in touch in the #coreservices Slack channel, or open issues and pull-requests in the source repo.

Table of Contents

Rules

Score Cards

Rules

Repository Ergonomics

Ergonomics (ûr″gə-nŏm′ĭks): Design factors, as for the workplace, intended to maximize productivity by minimizing operator fatigue and discomfort.

Rule Description Details Popularity
repo-standard-files
  • README.(md,rst)
  • LICENSE
  • CODE_OF_CONDUCT.md
  • CONTRIBUTING.md
  • SECURITY.md
  • SUPPORT.md
 See https://en.wikipedia.org/wiki/Contributing_guidelines and https://mozillascience.github.io/working-open-workshop/contributing/ templates at https://contributing.md
repo-standard-folders
  • docs/
  • bin/
repo-readme-sections

Readme contains the following:

  • Project name as main title
  • Status (invest, sustain, sunset, abandoned)
  • Usage
  • Link to documentation

Our goals here are:

  • avoid empty README
  • expose maintenance status
  • basic usage is detailed
  • standard sections will help reading/scanning for newcomers
repo-makefile-targets

Make has the following targets:

  • help
  • test
  • start
  • lint
  • format
repo-main-branch-name Main branch is called main
repo-main-branch-protected Main branch is protected without exceptions
repo-version-tags-protected Tags starting with v* can only be pushed by maintainers/admins Github docs
repo-issue-labels

Standard issue labels exist:

  • bug
  • dependencies
  • enhancement
  • documentation
  • wontfix
  • good-first-issue
repo-team-reviewers-permissions Team of reviewers has at least “Maintain” permissions
repo-pull-request-merge Only “squash and merge” and “rebase and merge” are allowed
repo-pre-commit-optional Pre-commit hooks are optional Example in Taskcluster

Repository Automation

Rule Description Details Popularity
repo-release-notes-auto Release Notes are generated from commits using .github/release.yml Sections titles contain at least “Breaking changes”, “Bug fixes”, “New features”, “Documentation”, and “Other changes”
repo-dependencies-auto Pull-requests are automatically opened for dependencies File .github/dependabot.yml and/or renovate.json exist
repo-automerge-enabled Automerge and merge queues are enabled
repo-head-branch-auto-delete Auto-delete head branches is enabled
repo-code-owners Code owners are configured At least with team of reviewers for everything (eg. @syseng-reviewers)

Testing and linting

Rule Description Details Popularity
test-lint-secrets A test verifies that no secret is committed to the sources Eg. Yelp/detect-secrets
test-lint-code-format A test verifies that code format respects standard Eg. prettier, black+isort, cargo fmt
test-lint-static-analysis A test verifies that code respects idioms and good practices Eg. eslint, bandit+pylint, cargo clippy
test-lint-type-checking A test verifies that types are used consistently Eg. typescript, mypy
test-unit-coverage Unit tests fail if coverage is under 90% Better if 100% with branching option
test-documentation-build A test verifies that documentation builds successfully without warnings If the project does not have built documentation, a lint check for markdown files is recommended
test-container-build A test verifies that container builds successfully
test-container-integration A test executes integration tests on the built container
test-pull-requests-labels A test verifies that pull-requests have labels
dependencies-reproducibility Dependencies are locked between local clone, tests, and container Eg. use poetry, cargo, …

Releasing

These checks apply to services that still rely on tags and releases. If a service has moved to Continuous Delivery, without tagging or releasing at all, they can all be considered as compliant.

Rule Description Details Popularity
release-semver-calver Project follows SemVer or Calver version scheme
release-tag-v-prefix The release tags have v(.+) format
release-tag-only Project version has a single source of truth. For example, it is read from version.json which is automatically built from current git tag Does not apply to libraries, where a version number usually appears in package metadata
release-publish Project container or package is published on git tag

Deployment

Rule Description Details Popularity
deployment-dockerflow Container follows Dockerflow spec
deployment-integration-test Integration tests are part of deployment pipeline Might be as well end-to-end (browser tests)
deployment-auto-main Automatic deployment on DEV/STAGE environment on pull-requests merge
deployment-auto-release Automatic deployment on STAGE/PROD environment on version tag If they pass through a quality gate like an automated integration test
deployment-auto-pull-requests Automatic deployment on temporary instances on pull-requests Should be allowed to repo owners only?

Monitoring

Rule Description Details Popularity
monitoring-heartbeat-backends Heartbeat endpoint checks that backends can executed desired operations (eg. DB, cache, filesystem etc.) to support troubleshooting
monitoring-heartbeat-error Heartbeat checks throw errors that will be reported
monitoring-requests-duration API requests processing duration is monitored
monitoring-requests-id A unique ID is associated to each request and logged
monitoring-backends-duration Backends calls durations are monitored Existing standards
monitoring-logging-level Logging has INFO level by default
monitoring-logging-mozlog Logging format is MozLog by default
monitoring-logging-env-configuration Logging level is set using LOG_LEVEL env variable. Logging format is set using LOG_FORMAT env variable
monitoring-error-reporting Sentry is setup by environment

Telemetry

Rule Description Details Popularity
telemetry-metrics Project sends live telemetry StatsD with tags
telemetry-naming Probes start with project name (eg. ctms.*)
telemetry-documented Probes are listed in documentation
telemetry-query-examples BigQuery example queries are given in documentation

Notifications

Rule Description Details Popularity
notifications-error-chat Sentry reports to Slack channel Sentry creates Github issues?

Documentation Content

Rule Description Details Popularity
documentation-decisions-records Enumerate evaluation criteria. Enumerate considered options with score/comment for each criteria.
documentation-security-assessment Rapid Risk Assessment, either a link or its content
documentation-dev Instructions for a developper: how to run the tests, procedures to obtain credentials on dev/stage instances, contribute, contact the team, etc.
documentation-test-args Instructions on how to run test suite with custom arguments (eg. test selection, verbosity, etc.)
documentation-operator Instructions for an operator: how to deploy and run the application, how to configure it for production, etc. Can also be called “Deployment docs”
documentation-troubleshooting Common troubleshooting steps

Score Cards

CTMS

81% (89 / 110), 2023-06-30

  • Deployment (6/10)
  • Documentation Content (8/12)
  • Monitoring (17/18)
  • Notifications (2/2)
  • Releasing (8/8)
  • Repository Automation (10/10)
  • Repository Ergonomics (22/22)
  • Telemetry (8/8)
  • Testing and linting (19/20)

Missing from standard v1:

  • test-documentation-build: Partially (Docs are only markdown files)
  • deployment-integration-test: No
  • deployment-auto-pull-requests: No
  • monitoring-logging-env-configuration: Partially (USE_MOZLOG instead of LOG_FORMAT, LOGGING_LEVEL instead of LOG_LEVEL)
  • documentation-security-assessment: No
  • documentation-troubleshooting: No

73% (80 / 110), 2023-02-01

  • Deployment (6/10)
  • Documentation Content (6/12)
  • Monitoring (15/18)
  • Notifications (2/2)
  • Releasing (8/8)
  • Repository Automation (10/10)
  • Repository Ergonomics (18/22)
  • Telemetry (6/8)
  • Testing and linting (19/20)

Missing from standard v1:

  • repo-standard-files: No (Missing CODE_OF_CONDUCT.md, CONTRIBUTING.md, SECURITY.md)
  • repo-readme-sections: No (README has only links to docs/)
  • test-documentation-build: Partially (Docs are only markdown files)
  • deployment-integration-test: No
  • deployment-auto-pull-requests: No
  • monitoring-logging-env-configuration: Partially (USE_MOZLOG instead of LOG_FORMAT, LOGGING_LEVEL instead of LOG_LEVEL)
  • monitoring-requests-duration: No
  • telemetry-query-examples: No (Until now, logs/metrics are not easily accessible)
  • documentation-security-assessment: No
  • documentation-test-args: No
  • documentation-troubleshooting: No

JBI

95% (105 / 110), 2023-07-04

  • Deployment (6/10)
  • Documentation Content (12/12)
  • Monitoring (18/18)
  • Notifications (2/2)
  • Releasing (8/8)
  • Repository Automation (10/10)
  • Repository Ergonomics (22/22)
  • Telemetry (8/8)
  • Testing and linting (19/20)

Missing from standard v1:

  • test-documentation-build: Partially
  • deployment-integration-test: No
  • deployment-auto-pull-requests: No

78% (86 / 110), 2023-02-04

  • Deployment (6/10)
  • Documentation Content (6/12)
  • Monitoring (15/18)
  • Notifications (2/2)
  • Releasing (8/8)
  • Repository Automation (10/10)
  • Repository Ergonomics (14/22)
  • Telemetry (8/8)
  • Testing and linting (17/20)

Missing from standard v1:

  • repo-standard-files: No (No LICENSE, CONTRIBUTING, files)
  • repo-standard-folders: No (No docs/ folder, bin/ instead of scripts/ )
  • repo-readme-sections: No (No status section in readme)
  • repo-makefile-targets: No (No make clean)
  • test-documentation-build: Partially (No built documentation, no markdown lint)
  • test-container-integration: No (Built container is not tested)
  • deployment-integration-test: No (No integration test available.)
  • deployment-auto-pull-requests: No (No temporary instance deployed on pull-requests. THis would improve productivity since we could test configuration changes before merging/deploying to STAGE.)
  • monitoring-logging-level: No (DEBUG by default instead of INFO)
  • monitoring-logging-mozlog: Partially (Log level via LOG_LEVEL but log format is not configurable.)
  • documentation-decisions-records: No
  • documentation-security-assessment: No (No link to RRA.)
  • documentation-test-args: No

Telescope

85% (93 / 110), 2023-09-04

  • Deployment (6/10)
  • Documentation Content (8/12)
  • Monitoring (17/18)
  • Notifications (0/2)
  • Releasing (8/8)
  • Repository Automation (10/10)
  • Repository Ergonomics (22/22)
  • Telemetry (3/8)
  • Testing and linting (18/20)

Missing from standard v1:

  • test-documentation-build: Partially
  • test-container-integration: Partially
  • deployment-integration-test: No
  • deployment-auto-pull-requests: No
  • monitoring-backends-duration: Partially
  • telemetry-metrics: No
  • telemetry-naming: Partially
  • telemetry-documented: Partially
  • telemetry-query-examples: Partially
  • notifications-error-chat: No
  • documentation-security-assessment: No
  • documentation-operator: Partially
  • documentation-troubleshooting: Partially

71% (78 / 110), 2023-01-01

  • Deployment (6/10)
  • Documentation Content (6/12)
  • Monitoring (13/18)
  • Notifications (0/2)
  • Releasing (8/8)
  • Repository Automation (10/10)
  • Repository Ergonomics (15/22)
  • Telemetry (3/8)
  • Testing and linting (16/20)

Missing from standard v1:

  • repo-standard-files: No (Missing CODE_OF_CONDUCT.md, CONTRIBUTING.md, SECURITY.md)
  • repo-standard-folders: No (scripts/ instead of bin/)
  • repo-readme-sections: Partially (Missing status badge, documentation within README)
  • repo-makefile-targets: No (Missing help, serve instead of start)
  • test-lint-secrets: No
  • test-documentation-build: Partially
  • test-container-integration: Partially (No integration tests (Telescope has no backend))
  • deployment-integration-test: No
  • deployment-auto-pull-requests: No (No DEV environment and no temporary instance deployed on pull-requests.)
  • monitoring-heartbeat-backends: Partially (No backend)
  • monitoring-heartbeat-error: Partially (No check in heartbeat)
  • monitoring-requests-id: No
  • monitoring-backends-duration: Partially (No backend)
  • telemetry-metrics: No (No telemetry)
  • telemetry-naming: Partially (No telemetry)
  • telemetry-documented: Partially (No telemetry)
  • telemetry-query-examples: Partially (No telemetry)
  • notifications-error-chat: No (Would be too verbose)
  • documentation-security-assessment: No
  • documentation-test-args: No
  • documentation-operator: Partially (Only configuration)
  • documentation-troubleshooting: Partially (Troubleshooting docs are on Mana but not about Telescope itself)

Remote Settings

84% (92 / 110), 2023-09-04

  • Deployment (6/10)
  • Documentation Content (9/12)
  • Monitoring (16/18)
  • Notifications (2/2)
  • Releasing (6/8)
  • Repository Automation (10/10)
  • Repository Ergonomics (22/22)
  • Telemetry (4/8)
  • Testing and linting (17/20)

Missing from standard v1:

  • test-lint-static-analysis: Partially
  • test-lint-type-checking: No
  • release-tag-v-prefix: No
  • deployment-integration-test: No
  • deployment-auto-pull-requests: No
  • monitoring-logging-env-configuration: No
  • telemetry-documented: No
  • telemetry-query-examples: No
  • documentation-test-args: No
  • documentation-operator: Partially

66% (73 / 110), 2023-03-04

  • Deployment (6/10)
  • Documentation Content (7/12)
  • Monitoring (14/18)
  • Notifications (0/2)
  • Releasing (4/8)
  • Repository Automation (8/10)
  • Repository Ergonomics (17/22)
  • Telemetry (4/8)
  • Testing and linting (13/20)

Missing from standard v1:

  • repo-standard-files: Partially (Missing CONTRIBUTING.md, SECURITY.md)
  • repo-readme-sections: No (Missing status badge, Usage is called “Run”)
  • repo-makefile-targets: Partially (Missing help)
  • repo-pre-commit-optional: Partially (No pre-commit :))
  • repo-release-notes-auto: No (We use a CHANGELOG.rst file.)
  • test-lint-secrets: No
  • test-lint-static-analysis: Partially (Only flake8)
  • test-lint-type-checking: No
  • test-pull-requests-labels: No
  • release-tag-v-prefix: No
  • release-tag-only: No (Requires a pull-request to update VERSION and CHANGELOG.rst files)
  • deployment-integration-test: No (This is planned for H1/H2 2023)
  • deployment-auto-pull-requests: No (No temporary instance deployed on pull-requests.)
  • monitoring-requests-id: No
  • monitoring-logging-env-configuration: No (via .ini)
  • telemetry-documented: No
  • telemetry-query-examples: No
  • notifications-error-chat: No
  • documentation-security-assessment: No (No link)
  • documentation-test-args: No
  • documentation-operator: Partially (Only via Kinto)

Tecken

49% (54 / 110), 2023-11-06

  • Deployment (2/10)
  • Documentation Content (4/12)
  • Monitoring (12/18)
  • Notifications (0/2)
  • Releasing (6/8)
  • Repository Automation (2/10)
  • Repository Ergonomics (11/22)
  • Telemetry (6/8)
  • Testing and linting (11/20)

Missing from standard v1:

  • repo-standard-files: Partially (Missing SECURITY and SUPPORT files)
  • repo-readme-sections: Partially (Missing status)
  • repo-makefile-targets: Partially (run instead of start and lintfix instead of format)
  • repo-version-tags-protected: No
  • repo-release-notes-auto: No (Changelog is put in tag annotation)
  • repo-code-owners: No
  • test-lint-secrets: No
  • test-lint-type-checking: No
  • test-unit-coverage: No (No coverage mesurement)
  • test-container-integration: Partially (Frontend tests are ran. System tests exist.)
  • test-pull-requests-labels: No
  • release-tag-v-prefix: No
  • deployment-auto-pull-requests: No
  • monitoring-backends-duration: Partially (metrics are used to monitor uploads/downloads, but apparently not for database/cache calls)
  • monitoring-logging-env-configuration: Partially (human format only in DEV, not configurable, and default level using LOGGING_DEFAULT_LEVEL env var source)
  • monitoring-error-reporting: No (No environment / version passed to init source)
  • telemetry-query-examples: No
  • documentation-security-assessment: No
  • documentation-test-args: No
  • documentation-operator: No
  • documentation-troubleshooting: No
  • monitoring-heartbeat-error: No

Could not be verified:

  • repo-main-branch-protected
  • repo-team-reviewers-permissions
  • repo-pull-request-merge
  • repo-automerge-enabled
  • repo-head-branch-auto-delete
  • deployment-integration-test
  • deployment-auto-main
  • deployment-auto-release
  • notifications-error-chat

Experimenter

65% (72 / 110), 2023-12-07

  • Deployment (7/10)
  • Documentation Content (6/12)
  • Monitoring (12/18)
  • Notifications (2/2)
  • Releasing (8/8)
  • Repository Automation (4/10)
  • Repository Ergonomics (17/22)
  • Telemetry (4/8)
  • Testing and linting (12/20)

Missing from standard v1:

  • repo-standard-folders: Partially
  • repo-issue-labels: No
  • repo-release-notes-auto: No
  • test-lint-secrets: No
  • test-unit-coverage: No
  • test-documentation-build: No
  • test-pull-requests-labels: No
  • deployment-dockerflow: Partially
  • deployment-auto-pull-requests: No
  • monitoring-heartbeat-error: No
  • monitoring-backends-duration: No
  • monitoring-logging-level: No
  • telemetry-documented: No
  • telemetry-query-examples: No
  • documentation-security-assessment: No
  • documentation-operator: No
  • documentation-troubleshooting: No

Could not be verified:

  • repo-pull-request-merge
  • repo-automerge-enabled
  • repo-head-branch-auto-delete

59% (65 / 110), 2023-11-06

  • Deployment (7/10)
  • Documentation Content (6/12)
  • Monitoring (9/18)
  • Notifications (2/2)
  • Releasing (8/8)
  • Repository Automation (4/10)
  • Repository Ergonomics (13/22)
  • Telemetry (4/8)
  • Testing and linting (12/20)

Missing from standard v1:

  • repo-standard-files: Partially (Missing SECURITY.md and SUPPORT.md)
  • repo-standard-folders: Partially (scripts/ instead of bin/)
  • repo-readme-sections: Partially (Missing Status)
  • repo-makefile-targets: No (up instead of start, no help, code_format instead of format, check instead of lint, pytest instead of test, ...)
  • repo-issue-labels: No (No documentation, defect instead of bug)
  • repo-release-notes-auto: No
  • test-lint-secrets: No
  • test-unit-coverage: No
  • test-documentation-build: No
  • test-pull-requests-labels: No
  • deployment-dockerflow: Partially (No readonly user apparently, /experimenter instead of /app, not ENTRYPOINT and CMD, ...)
  • deployment-auto-pull-requests: No
  • monitoring-logging-level: No (DEBUG source)
  • monitoring-logging-env-configuration: No (LOGGING_CONSOLE_LEVEL and LOGGING_USE_JSON)
  • monitoring-error-reporting: Partially (Without env/version source)
  • telemetry-documented: No
  • telemetry-query-examples: No
  • documentation-security-assessment: No
  • documentation-operator: No
  • documentation-troubleshooting: No
  • monitoring-heartbeat-error: No
  • monitoring-backends-duration: No

Could not be verified:

  • repo-pull-request-merge
  • repo-automerge-enabled
  • repo-head-branch-auto-delete

Balrog

31% (34 / 110), 2023-10-18

  • Deployment (1/10)
  • Documentation Content (2/12)
  • Monitoring (8/18)
  • Notifications (0/2)
  • Releasing (6/8)
  • Repository Automation (4/10)
  • Repository Ergonomics (7/22)
  • Telemetry (4/8)
  • Testing and linting (2/20)

Missing from standard v1:

  • repo-standard-files: Partially (Missing CONTRIBUTING, SECURITY, SUPPORT)
  • repo-standard-folders: Partially (scripts/ instead of bin/)
  • repo-readme-sections: Partially (No title, no status, no usage)
  • repo-makefile-targets: No (No Makefile)
  • repo-issue-labels: No (bug, no enhancement, no documentation, no wontfix, first-bug instead of good-first-issue)
  • repo-release-notes-auto: No (No changelog)
  • repo-automerge-enabled: No
  • test-lint-secrets: No
  • test-lint-code-format: No
  • test-lint-static-analysis: No
  • test-lint-type-checking: No
  • test-unit-coverage: No
  • test-documentation-build: No
  • test-pull-requests-labels: No
  • deployment-dockerflow: Partially (No dedicated user, not exposing 8000, ...)
  • deployment-auto-pull-requests: No
  • monitoring-error-reporting: No (No env Sentry SDK init)
  • telemetry-documented: No (Not listed in infra docs)
  • telemetry-query-examples: No
  • documentation-decisions-records: No
  • documentation-security-assessment: No
  • documentation-test-args: No
  • documentation-troubleshooting: No

Could not be verified:

  • repo-main-branch-protected
  • repo-version-tags-protected
  • repo-team-reviewers-permissions
  • repo-pull-request-merge
  • repo-head-branch-auto-delete
  • test-container-build : (Could not find in taskcluster)
  • test-container-integration : (Could not find it)
  • release-publish
  • deployment-integration-test
  • deployment-auto-main
  • deployment-auto-release
  • monitoring-heartbeat-error : (Could not find heartbeat checks implementation)
  • monitoring-requests-duration
  • monitoring-requests-id
  • monitoring-backends-duration
  • notifications-error-chat
  • documentation-operator

addons-server

57% (63 / 110), 2023-10-19

  • Deployment (7/10)
  • Documentation Content (4/12)
  • Monitoring (13/18)
  • Notifications (0/2)
  • Releasing (6/8)
  • Repository Automation (6/10)
  • Repository Ergonomics (10/22)
  • Telemetry (4/8)
  • Testing and linting (14/20)

Missing from standard v1:

  • repo-standard-files: Partially (Missing SUPPORT.md)
  • repo-standard-folders: No (scripts instead of bin/)
  • repo-readme-sections: Partially (No status nor usage)
  • repo-makefile-targets: Partially (No start)
  • repo-main-branch-name: No (master instead of main)
  • repo-version-tags-protected: No
  • repo-issue-labels: Partially (labels have prefixes, no enhancement, good_first_bug instead of good-first-issue)
  • repo-team-reviewers-permissions: No (No but only 2 engineers in the team currently, all with write access)
  • repo-release-notes-auto: No (No changelog)
  • repo-automerge-enabled: No
  • test-unit-coverage: No (No coverage reporting)
  • test-container-integration: No
  • test-pull-requests-labels: No
  • release-tag-v-prefix: No
  • deployment-dockerflow: Partially (No entrypoint, cmd, not exposing 8000, ...)
  • deployment-auto-pull-requests: No
  • monitoring-heartbeat-error: Partially (Not everything is logged source)
  • monitoring-requests-duration: Partially (In Grafana (not in MozLog field))
  • monitoring-backends-duration: Partially (Via heartbeat)
  • monitoring-logging-env-configuration: No (source)
  • telemetry-documented: No
  • telemetry-query-examples: No
  • notifications-error-chat: No (Too verbose. Sentry tracebacks can automatically comment in GitHub and can be linked to GitHub issues though.)
  • documentation-decisions-records: No
  • documentation-security-assessment: No
  • documentation-operator: No
  • documentation-troubleshooting: No

kitsune

50% (55 / 110), 2023-12-07

  • Deployment (5/10)
  • Documentation Content (8/12)
  • Monitoring (14/18)
  • Notifications (1/2)
  • Releasing (2/8)
  • Repository Automation (2/10)
  • Repository Ergonomics (15/22)
  • Telemetry (0/8)
  • Testing and linting (8/20)

Missing from standard v1:

  • repo-version-tags-protected: No
  • repo-issue-labels: Partially
  • repo-pull-request-merge: No
  • repo-pre-commit-optional: No
  • repo-release-notes-auto: No
  • repo-automerge-enabled: No
  • repo-head-branch-auto-delete: No
  • repo-code-owners: No
  • test-lint-secrets: No
  • test-lint-type-checking: No
  • test-unit-coverage: No
  • test-documentation-build: No
  • test-container-integration: No
  • test-pull-requests-labels: No
  • release-tag-v-prefix: No
  • release-tag-only: No
  • release-publish: No
  • deployment-integration-test: No
  • deployment-auto-release: Partially
  • deployment-auto-pull-requests: No
  • monitoring-backends-duration: No
  • monitoring-logging-mozlog: No
  • telemetry-metrics: No
  • telemetry-naming: No
  • telemetry-documented: No
  • telemetry-query-examples: No
  • notifications-error-chat: Partially
  • documentation-security-assessment: No
  • documentation-troubleshooting: No

39% (43 / 110), 2023-10-19

  • Deployment (5/10)
  • Documentation Content (8/12)
  • Monitoring (5/18)
  • Notifications (1/2)
  • Releasing (2/8)
  • Repository Automation (2/10)
  • Repository Ergonomics (12/22)
  • Telemetry (0/8)
  • Testing and linting (8/20)

Missing from standard v1:

  • repo-standard-files: Partially (No SUPPORT, no SECURITY, no CONTRIBUTING)
  • repo-readme-sections: Partially (No Status, no Usage)
  • repo-makefile-targets: Partially (No format, run instead of start)
  • repo-issue-labels: Partially (No bug, no good-first-issue, ...)
  • repo-pre-commit-optional: No (source)
  • repo-release-notes-auto: No
  • repo-code-owners: No
  • test-lint-secrets: No
  • test-lint-type-checking: No (Only flake8)
  • test-unit-coverage: No
  • test-documentation-build: No
  • test-container-integration: No (No integration test apparently (only unit and js))
  • test-pull-requests-labels: No
  • release-tag-v-prefix: No
  • deployment-integration-test: No (No integration tests apparently)
  • deployment-auto-pull-requests: No
  • monitoring-heartbeat-backends: No (Could not find mention of heartbeat or dockerflow)
  • monitoring-heartbeat-error: No
  • monitoring-requests-duration: No
  • monitoring-requests-id: No
  • monitoring-backends-duration: No (statsd was removed)
  • monitoring-logging-mozlog: No (No mention of Json formatter)
  • monitoring-logging-env-configuration: Partially (Not LOG_FORMAT)
  • telemetry-metrics: No (statsd was removed)
  • telemetry-naming: No
  • telemetry-documented: No
  • telemetry-query-examples: No
  • documentation-security-assessment: No
  • documentation-troubleshooting: No
  • repo-version-tags-protected: No
  • repo-pull-request-merge: No
  • repo-automerge-enabled: No
  • repo-head-branch-auto-delete: No
  • release-tag-only: No
  • release-publish: No
  • deployment-auto-release: Partially (Automatic releases but without tags. We are on the process of switching to tags)
  • notifications-error-chat: Partially (We do get notification but it's from NewRelic and from the pipelines. We do use Sentry but we don't have slack automation)

Taskcluster

52% (57 / 110), 2023-11-07

  • Deployment (4/10)
  • Documentation Content (6/12)
  • Monitoring (6/18)
  • Notifications (1/2)
  • Releasing (8/8)
  • Repository Automation (8/10)
  • Repository Ergonomics (12/22)
  • Telemetry (2/8)
  • Testing and linting (10/20)

Missing from standard v1:

  • repo-standard-files: Partially (Missing SUPPORT.md)
  • repo-standard-folders: No (No bin/, no docs/)
  • repo-readme-sections: Partially (Missing status)
  • repo-makefile-targets: No (No Makefile)
  • repo-version-tags-protected: No (No tag protection rules)
  • repo-issue-labels: Partially (No wontfix, docs instead of documentation)
  • repo-pull-request-merge: Partially (Merge commits are also allowed)
  • repo-release-notes-auto: Partially (Using a command yarn release)
  • repo-automerge-enabled: Partially (Merge queues are not enabled)
  • test-lint-secrets: No
  • test-lint-type-checking: No
  • test-unit-coverage: No
  • test-pull-requests-labels: No
  • deployment-dockerflow: No (Does not expose 8000, not using specific user, ...)
  • deployment-integration-test: Partially (A simple smoketest is run after deployment)
  • deployment-auto-release: Partially (No quality gate to pass through before DEV deployment)
  • deployment-auto-pull-requests: No
  • monitoring-heartbeat-backends: No
  • monitoring-heartbeat-error: No (Heartbeat is a simple Nginx endpoint that returns 200)
  • monitoring-logging-mozlog: No (only if ENV==production source)
  • monitoring-logging-env-configuration: No
  • monitoring-error-reporting: No (Sentry init does not set environment)
  • notifications-error-chat: Partially (Notifications of elevated Sentry errors are sent to #taskcluster-services-notifications and no GitHub issues are created automatically)
  • documentation-decisions-records: No (There are design docs though)
  • documentation-test-args: No (dev documentation does not mention it)
  • test-container-integration: No (CI doesn’t build containers on every commit, only for release events)
  • monitoring-backends-duration: No (No statsd / live metrics)
  • telemetry-naming: No (No statsd / live metrics)
  • telemetry-documented: No (No statsd / live metrics)
  • telemetry-query-examples: No (No statsd / live metrics)
  • documentation-security-assessment: No