glean_parser package

Submodules

glean_parser.coverage module

Produce coverage reports from the raw information produced by the GLEAN_TEST_COVERAGE feature.

glean_parser.coverage.coverage(coverage_reports: ~typing.List[~pathlib.Path], metrics_files: ~typing.Sequence[~pathlib.Path], output_format: str, output_file: ~pathlib.Path, parser_config: ~typing.Dict[str, ~typing.Any] | None = None, file=<_io.TextIOWrapper name='<stderr>' mode='w' encoding='utf-8'>) → int

Commandline helper for coverage.

Parameters:

• coverage_reports – List of coverage report files, output from the Glean SDK when the GLEAN_TEST_COVERAGE environment variable is set.

• metrics_files – List of Path objects to load metrics from.

• output_format – The coverage output format to produce. Must be one of OUTPUTTERS.keys().

• output_file – Path to output coverage report to.

• parser_config – Parser configuration object, passed to parser.parse_objects.

Returns:

Non-zero if there were any errors.

glean_parser.data_review module

Produce skeleton Data Review Requests.

glean_parser.data_review.generate(bug: str, metrics_files: Sequence[Path]) → int

Commandline helper for Data Review Request template generation.

Parameters:

• bug – pattern to match in metrics’ bug_numbers lists.

• metrics_files – List of Path objects to load metrics from.

Returns:

Non-zero if there were any errors.

glean_parser.go_server module

Outputter to generate server Go code for collecting events.

This outputter is different from the rest of the outputters in that the code it generates does not use the Glean SDK. It is meant to be used to collect events in server-side environments. In these environments SDK assumptions to measurement window and connectivity don’t hold. Generated code takes care of assembling pings with metrics, and serializing to messages conforming to Glean schema.

Warning: this outputter supports limited set of metrics, see SUPPORTED_METRIC_TYPES below.

Generated code creates two methods for each ping (RecordPingX and RecordPingXWithoutUserInfo) that are used for submitting (logging) them. If pings have event metrics assigned, they can be passed to these methods.

glean_parser.go_server.clean_string(s: str) → str

glean_parser.go_server.generate_event_type_name(metric: Metric) → str

glean_parser.go_server.generate_extra_name(extra: str) → str

glean_parser.go_server.generate_metric_argument_name(metric: Metric) → str

glean_parser.go_server.generate_metric_name(metric: Metric) → str

glean_parser.go_server.generate_metric_type(metric_type: str) → str

glean_parser.go_server.generate_ping_events_type_name(ping_name: str) → str

glean_parser.go_server.generate_ping_type_name(ping_name: str) → str

glean_parser.go_server.output_go(objs: Dict[str, Dict[str, Metric | Ping | Tag]], output_dir: Path, options: Dict[str, Any] | None) → None

Given a tree of objects, output Go code to output_dir.

The output is a single file containing all the code for assembling pings with metrics, serializing, and submitting.

Parameters:

• objects – A tree of objects (metrics and pings) as returned from parser.parse_objects.

• output_dir – Path to an output directory to write to.

glean_parser.javascript module

Outputter to generate Javascript code for metrics.

glean_parser.javascript.args(obj_type: str) → Dict[str, object]

Returns the list of arguments for each object type.

glean_parser.javascript.class_name_factory(platform: str) → Callable[[str], str]

Returns a function that receives an obj_type and returns the correct class name for that type in the current platform.

glean_parser.javascript.extra_type_name(extra_type: str) → str

Returns the equivalent TypeScript type to an extra type.

glean_parser.javascript.generate_build_date(date: str | None) → str

Generate the build Date object.

glean_parser.javascript.import_path(obj_type: str) → str

Returns the import path of the given object inside the @mozilla/glean package.

glean_parser.javascript.javascript_datatypes_filter(value: list | dict | str | int | float | None) → str

A Jinja2 filter that renders Javascript literals.

Based on Python’s JSONEncoder, but overrides:

• lists to use listOf

• sets to use setOf

• Rate objects to a CommonMetricData initializer (for external Denominators’ Numerators lists)

glean_parser.javascript.output(lang: str, objs: Dict[str, Dict[str, Metric | Ping | Tag]], output_dir: Path, options: Dict[str, Any] | None = None) → None

Given a tree of objects, output Javascript or Typescript code to output_dir.

Parameters:

• lang – Either “javascript” or “typescript”;

• objects – A tree of objects (metrics and pings) as returned from parser.parse_objects.

• output_dir – Path to an output directory to write to.

• options –

  options dictionary, with the following optional keys: - platform: Which platform are we building for. Options are webext and qt.

  Default is webext.

  • version: The version of the Glean.js Qt library being used.

    This option is mandatory when targeting Qt. Note that the version string must only contain the major and minor version i.e. 0.14.

  • with_buildinfo: If “true” a gleanBuildInfo.(js|ts) file is generated.

    Otherwise generation of that file is skipped. Defaults to “false”.

  • build_date: If set to 0 a static unix epoch time will be used.

    If set to a ISO8601 datetime string (e.g. 2022-01-03T17:30:00) it will use that date. Other values will throw an error. If not set it will use the current date & time.

glean_parser.javascript.output_javascript(objs: Dict[str, Dict[str, Metric | Ping | Tag]], output_dir: Path, options: Dict[str, Any] | None = None) → None

Given a tree of objects, output Javascript code to output_dir.

Parameters:

• objects – A tree of objects (metrics and pings) as returned from parser.parse_objects.

• output_dir – Path to an output directory to write to.

• options –

  options dictionary, with the following optional keys:

  • namespace: The identifier of the global variable to assign to.

    This will only have and effect for Qt and static web sites. Default is Glean.

  • platform: Which platform are we building for. Options are webext and qt.

    Default is webext.

glean_parser.javascript.output_typescript(objs: Dict[str, Dict[str, Metric | Ping | Tag]], output_dir: Path, options: Dict[str, Any] | None = None) → None

Given a tree of objects, output Typescript code to output_dir.

# Note

The only difference between the typescript and javascript templates, currently is the file extension.

Parameters:

• objects – A tree of objects (metrics and pings) as returned from parser.parse_objects.

• output_dir – Path to an output directory to write to.

• options –

  options dictionary, with the following optional keys:

  • namespace: The identifier of the global variable to assign to.

    This will only have and effect for Qt and static web sites. Default is Glean.

  • platform: Which platform are we building for. Options are webext and qt.

    Default is webext.

glean_parser.javascript_server module

Outputter to generate server Javascript code for collecting events.

This outputter is different from the rest of the outputters in that the code it generates does not use the Glean SDK. It is meant to be used to collect events in server-side environments. In these environments SDK assumptions to measurement window and connectivity don’t hold. Generated code takes care of assembling pings with metrics, serializing to messages conforming to Glean schema, and logging with mozlog. Then it’s the role of the ingestion pipeline to pick the messages up and process.

Warning: this outputter supports limited set of metrics, see SUPPORTED_METRIC_TYPES below.

There are two patterns for event structure supported in this environment: * Events as Event metric type, where we generate a single class per ping with

record{event_name} method for each event metric. This is recommended to use for new applications as it allows to fully leverage standard Data Platform tools post-ingestion.

• Custom pings-as-events, where for each ping we generate a class with a single record method, usually with an event_name string metric.

Therefore, unlike in other outputters, here we don’t generate classes for each metric.

glean_parser.javascript_server.clean_string(s: str) → str

glean_parser.javascript_server.event_class_name(ping_name: str, metrics_by_type: Dict[str, List[Metric]]) → str

glean_parser.javascript_server.generate_event_metric_record_function_name(metric: Metric) → str

glean_parser.javascript_server.generate_js_metric_type(metric: Metric) → str

glean_parser.javascript_server.generate_metric_argument_name(metric: Metric) → str

glean_parser.javascript_server.generate_metric_name(metric: Metric) → str

glean_parser.javascript_server.generate_ping_factory_method(ping: str, metrics_by_type: Dict[str, List[Metric]]) → str

glean_parser.javascript_server.output(lang: str, objs: Dict[str, Dict[str, Metric | Ping | Tag]], output_dir: Path, options: Dict[str, Any] | None = None) → None

Given a tree of objects, output Javascript or Typescript code to output_dir.

The output is a single file containing all the code for assembling pings with metrics, serializing, and submitting.

Parameters:

• lang – Either “javascript” or “typescript”;

• objects – A tree of objects (metrics and pings) as returned from parser.parse_objects.

• output_dir – Path to an output directory to write to.

glean_parser.javascript_server.output_javascript(objs: Dict[str, Dict[str, Metric | Ping | Tag]], output_dir: Path, options: Dict[str, Any] | None = None) → None

Given a tree of objects, output Javascript code to output_dir.

Parameters:

• objects – A tree of objects (metrics and pings) as returned from parser.parse_objects.

• output_dir – Path to an output directory to write to.

• options –

  options dictionary, with the following optional keys:

  • module_spec: Module specification to use. Options are es, commonjs.

    Default is es.

glean_parser.javascript_server.output_typescript(objs: Dict[str, Dict[str, Metric | Ping | Tag]], output_dir: Path, options: Dict[str, Any] | None = None) → None

Given a tree of objects, output Typescript code to output_dir.

Parameters:

• objects – A tree of objects (metrics and pings) as returned from parser.parse_objects.

• output_dir – Path to an output directory to write to.

glean_parser.kotlin module

Outputter to generate Kotlin code for metrics.

glean_parser.kotlin.class_name(obj_type: str) → str

Returns the Kotlin class name for a given metric or ping type.

glean_parser.kotlin.extra_type_name(typ: str) → str

Returns the corresponding Kotlin type for event’s extra key types.

glean_parser.kotlin.generate_build_date(date: str | None) → str

Generate the build timestamp.

glean_parser.kotlin.kotlin_datatypes_filter(value: list | dict | str | int | float | None) → str

A Jinja2 filter that renders Kotlin literals.

Based on Python’s JSONEncoder, but overrides:

• lists to use listOf

• dicts to use mapOf

• sets to use setOf

• enums to use the like-named Kotlin enum

• Rate objects to a CommonMetricData initializer (for external Denominators’ Numerators lists)

glean_parser.kotlin.output_kotlin(objs: Dict[str, Dict[str, Metric | Ping | Tag]], output_dir: Path, options: Dict[str, Any] | None = None) → None

Given a tree of objects, output Kotlin code to output_dir.

Parameters:

• objects – A tree of objects (metrics and pings) as returned from parser.parse_objects.

• output_dir – Path to an output directory to write to.

• options –

  options dictionary, with the following optional keys:

  • namespace: The package namespace to declare at the top of the generated files. Defaults to GleanMetrics.

  • glean_namespace: The package namespace of the glean library itself. This is where glean objects will be imported from in the generated code.

  • with_buildinfo: If “true” a GleanBuildInfo.kt file is generated. Otherwise generation of that file is skipped. Defaults to “true”.

  • build_date: If set to 0 a static unix epoch time will be used.

    If set to a ISO8601 datetime string (e.g. 2022-01-03T17:30:00) it will use that date. Other values will throw an error. If not set it will use the current date & time.

glean_parser.kotlin.structure_type_name(typ: str) → str

Returns the corresponding Kotlin type for structure items.

glean_parser.kotlin.type_name(obj: Metric | Ping) → str

Returns the Kotlin type to use for a given metric or ping object.

glean_parser.lint module

class glean_parser.lint.CheckType(value)

Bases: Enum

An enumeration.

error = 1

warning = 0

class glean_parser.lint.GlinterNit(check_name: str, name: str, msg: str, check_type: CheckType)

Bases: object

format()

glean_parser.lint.check_bug_number(metric: Metric | Ping, parser_config: Dict[str, Any]) → Generator[str, None, None]

glean_parser.lint.check_category_generic(category_name: str, metrics: Iterable[Metric]) → Generator[str, None, None]

The category name is too generic.

glean_parser.lint.check_common_prefix(category_name: str, metrics: Iterable[Metric]) → Generator[str, None, None]

Check if all metrics begin with a common prefix.

glean_parser.lint.check_empty_datareview(metric: Metric, parser_config: Dict[str, Any]) → Generator[str, None, None]

glean_parser.lint.check_expired_date(metric: Metric, parser_config: Dict[str, Any]) → Generator[str, None, None]

glean_parser.lint.check_expired_metric(metric: Metric, parser_config: Dict[str, Any]) → Generator[str, None, None]

glean_parser.lint.check_metric_on_events_lifetime(metric: Metric, parser_config: Dict[str, Any]) → Generator[str, None, None]

A non-event metric on the Events ping only makes sense if its value is immutable over the life of the ping.

glean_parser.lint.check_misspelled_pings(metric: Metric, parser_config: Dict[str, Any]) → Generator[str, None, None]

glean_parser.lint.check_old_event_api(metric: Metric, parser_config: Dict[str, Any]) → Generator[str, None, None]

glean_parser.lint.check_redundant_ping(pings: Ping, parser_config: Dict[str, Any]) → Generator[str, None, None]

Check if the pings contains ‘ping’ as the prefix or suffix, or ‘ping’ or ‘custom’

glean_parser.lint.check_tags_required(metric_or_ping: Metric | Ping, parser_config: Dict[str, Any]) → Generator[str, None, None]

glean_parser.lint.check_unexpected_unit(metric: Metric, parser_config: Dict[str, Any]) → Generator[str, None, None]

unit was allowed on all metrics and recently disallowed. We now warn about its use on all but quantity and custom distribution metrics.

glean_parser.lint.check_unit_in_name(metric: Metric, parser_config: Dict[str, Any]) → Generator[str, None, None]

The metric name ends in a unit.

glean_parser.lint.check_unknown_ping(check_name: str, check_type: CheckType, all_pings: Dict[str, Ping], metrics: Dict[str, Metric], parser_config: Dict[str, Any]) → Generator[GlinterNit, None, None]

Check that all pings in send_in_pings for all metrics are either a builtin ping or in the list of defined custom pings.

glean_parser.lint.check_user_lifetime_expiration(metric: Metric, parser_config: Dict[str, Any]) → Generator[str, None, None]

glean_parser.lint.check_valid_in_baseline(metric: Metric, parser_config: Dict[str, Any]) → Generator[str, None, None]

glean_parser.lint.glinter(input_filepaths: ~typing.Iterable[~pathlib.Path], parser_config: ~typing.Dict[str, ~typing.Any] | None = None, file=<_io.TextIOWrapper name='<stderr>' mode='w' encoding='utf-8'>) → int

Commandline helper for glinter.

Parameters:

• input_filepaths – List of Path objects to load metrics from.

• parser_config – Parser configuration object, passed to parser.parse_objects.

• file – The stream to write the errors to.

Returns:

Non-zero if there were any glinter errors.

glean_parser.lint.lint_metrics(objs: ~typing.Dict[str, ~typing.Dict[str, ~glean_parser.metrics.Metric | ~glean_parser.pings.Ping | ~glean_parser.tags.Tag]], parser_config: ~typing.Dict[str, ~typing.Any] | None = None, file=<_io.TextIOWrapper name='<stderr>' mode='w' encoding='utf-8'>) → List[GlinterNit]

Performs glinter checks on a set of metrics objects.

Parameters:

• objs – Tree of metric objects, as returns by parser.parse_objects.

• file – The stream to write errors to.

Returns:

List of nits.

glean_parser.markdown module

Outputter to generate Markdown documentation for metrics.

glean_parser.markdown.data_sensitivity_numbers(data_sensitivity: List[DataSensitivity] | None) → str

glean_parser.markdown.extra_info(obj: Metric | Ping) → List[Tuple[str, str]]

Returns a list of string to string tuples with extra information for the type (e.g. extra keys for events) or an empty list if nothing is available.

glean_parser.markdown.if_empty(ping_name: str, custom_pings_cache: Dict[str, Ping] | None = None) → bool

glean_parser.markdown.metrics_docs(obj_name: str) → str

Return a link to the documentation entry for the Glean SDK metric of the requested type.

glean_parser.markdown.output_markdown(objs: Dict[str, Dict[str, Metric | Ping | Tag]], output_dir: Path, options: Dict[str, Any] | None = None) → None

Given a tree of objects, output Markdown docs to output_dir.

This produces a single metrics.md. The file contains a table of contents and a section for each ping metrics are collected for.

Parameters:

• objects – A tree of objects (metrics and pings) as returned from parser.parse_objects.

• output_dir – Path to an output directory to write to.

• options – options dictionary, with the following optional key: - project_title: The projects title.

glean_parser.markdown.ping_bugs(ping_name: str, custom_pings_cache: Dict[str, Ping] | None = None) → List[str] | None

glean_parser.markdown.ping_data_reviews(ping_name: str, custom_pings_cache: Dict[str, Ping] | None = None) → List[str] | None

glean_parser.markdown.ping_desc(ping_name: str, custom_pings_cache: Dict[str, Ping] | None = None) → str

Return a text description of the ping. If a custom_pings_cache is available, look in there for non-reserved ping names description.

glean_parser.markdown.ping_docs(ping_name: str) → str

Return a link to the documentation entry for the requested Glean SDK built-in ping.

glean_parser.markdown.ping_include_client_id(ping_name: str, custom_pings_cache: Dict[str, Ping] | None = None) → bool

glean_parser.markdown.ping_reasons(ping_name: str, custom_pings_cache: Dict[str, Ping]) → Dict[str, str]

Returns the reasons dictionary for the ping.

glean_parser.markdown.ping_review_title(data_url: str, index: int) → str

Return a title for a data review in human readable form.

Parameters:

• data_url – A url for data review.

• index – Position of the data review on list (e.g: 1, 2, 3…).

glean_parser.metrics module

Classes for each of the high-level metric types.

class glean_parser.metrics.Boolean(type: str, category: str, name: str, bugs: List[str], description: str, notification_emails: List[str], expires: Any, metadata: Dict | None = None, data_reviews: List[str] | None = None, version: int = 0, disabled: bool = False, lifetime: str = 'ping', send_in_pings: List[str] | None = None, unit: str | None = None, gecko_datapoint: str = '', no_lint: List[str] | None = None, data_sensitivity: List[str] | None = None, defined_in: Dict | None = None, telemetry_mirror: str | None = None, _config: Dict[str, Any] | None = None, _validated: bool = False)

Bases: Metric

typename: str = 'boolean'

class glean_parser.metrics.Counter(type: str, category: str, name: str, bugs: List[str], description: str, notification_emails: List[str], expires: Any, metadata: Dict | None = None, data_reviews: List[str] | None = None, version: int = 0, disabled: bool = False, lifetime: str = 'ping', send_in_pings: List[str] | None = None, unit: str | None = None, gecko_datapoint: str = '', no_lint: List[str] | None = None, data_sensitivity: List[str] | None = None, defined_in: Dict | None = None, telemetry_mirror: str | None = None, _config: Dict[str, Any] | None = None, _validated: bool = False)

Bases: Metric

typename: str = 'counter'

class glean_parser.metrics.CowString(val: str)

Bases: str

Wrapper class for strings that should be represented as a Cow<’static, str> in Rust, or String in other target languages.

This wraps str, so unless CowString is specifically handled it acts (and serializes) as a string.

class glean_parser.metrics.CustomDistribution(*args, **kwargs)

Bases: Metric

typename: str = 'custom_distribution'

class glean_parser.metrics.DataSensitivity(value)

Bases: Enum

An enumeration.

highly_sensitive = 4

interaction = 2

stored_content = 3

technical = 1

web_activity = 3

class glean_parser.metrics.Datetime(*args, **kwargs)

Bases: TimeBase

typename: str = 'datetime'

class glean_parser.metrics.Denominator(type: str, category: str, name: str, bugs: List[str], description: str, notification_emails: List[str], expires: Any, metadata: Dict | None = None, data_reviews: List[str] | None = None, version: int = 0, disabled: bool = False, lifetime: str = 'ping', send_in_pings: List[str] | None = None, unit: str | None = None, gecko_datapoint: str = '', no_lint: List[str] | None = None, data_sensitivity: List[str] | None = None, defined_in: Dict | None = None, telemetry_mirror: str | None = None, _config: Dict[str, Any] | None = None, _validated: bool = False)

Bases: Counter

numerators: List[Rate] = []

typename: str = 'denominator'

class glean_parser.metrics.Event(*args, **kwargs)

Bases: Metric

property allowed_extra_keys

property allowed_extra_keys_with_types

default_store_names: List[str] = ['events']

typename: str = 'event'

static validate_extra_keys(extra_keys: Dict[str, str], config: Dict[str, Any]) → None

class glean_parser.metrics.HistogramType(value)

Bases: Enum

An enumeration.

exponential = 1

linear = 0

class glean_parser.metrics.Jwe(*args, **kwargs)

Bases: Metric

typename: str = 'jwe'

class glean_parser.metrics.Labeled(*args, **kwargs)

Bases: Metric

labeled = True

serialize() → Dict[str, list | dict | str | int | float | None]

Serialize the metric back to JSON object model.

class glean_parser.metrics.LabeledBoolean(*args, **kwargs)

Bases: Labeled, Boolean

typename: str = 'labeled_boolean'

class glean_parser.metrics.LabeledCounter(*args, **kwargs)

Bases: Labeled, Counter

typename: str = 'labeled_counter'

class glean_parser.metrics.LabeledCustomDistribution(*args, **kwargs)

Bases: Labeled, CustomDistribution

typename: str = 'labeled_custom_distribution'

class glean_parser.metrics.LabeledMemoryDistribution(*args, **kwargs)

Bases: Labeled, MemoryDistribution

typename: str = 'labeled_memory_distribution'

class glean_parser.metrics.LabeledQuantity(*args, **kwargs)

Bases: Labeled, Quantity

typename: str = 'labeled_quantity'

class glean_parser.metrics.LabeledString(*args, **kwargs)

Bases: Labeled, String

typename: str = 'labeled_string'

class glean_parser.metrics.LabeledTimingDistribution(*args, **kwargs)

Bases: Labeled, TimingDistribution

typename: str = 'labeled_timing_distribution'

class glean_parser.metrics.Lifetime(value)

Bases: Enum

An enumeration.

application = 1

ping = 0

user = 2

class glean_parser.metrics.MemoryDistribution(*args, **kwargs)

Bases: Metric

typename: str = 'memory_distribution'

class glean_parser.metrics.MemoryUnit(value)

Bases: Enum

An enumeration.

byte = 0

gigabyte = 3

kilobyte = 1

megabyte = 2

class glean_parser.metrics.Metric(type: str, category: str, name: str, bugs: List[str], description: str, notification_emails: List[str], expires: Any, metadata: Dict | None = None, data_reviews: List[str] | None = None, version: int = 0, disabled: bool = False, lifetime: str = 'ping', send_in_pings: List[str] | None = None, unit: str | None = None, gecko_datapoint: str = '', no_lint: List[str] | None = None, data_sensitivity: List[str] | None = None, defined_in: Dict | None = None, telemetry_mirror: str | None = None, _config: Dict[str, Any] | None = None, _validated: bool = False)

Bases: object

default_store_names: List[str] = ['metrics']

glean_internal_metric_cat: str = 'glean.internal.metrics'

identifier() → str

Create an identifier unique for this metric. Generally, category.name; however, Glean internal metrics only use name.

is_disabled() → bool

is_expired() → bool

is_internal_metric() → bool

classmethod make_metric(category: str, name: str, metric_info: Dict[str, list | dict | str | int | float | None], config: Dict[str, Any] | None = None, validated: bool = False)

Given a metric_info dictionary from metrics.yaml, return a metric instance.

Param:

category The category the metric lives in

Param:

name The name of the metric

Param:

metric_info A dictionary of the remaining metric parameters

Param:

config A dictionary containing commandline configuration parameters

Param:

validated True if the metric has already gone through jsonschema validation

Returns:

A new Metric instance.

metric_types: Dict[str, Any] = {'ERROR': <class 'glean_parser.metrics.Labeled'>, 'boolean': <class 'glean_parser.metrics.Boolean'>, 'counter': <class 'glean_parser.metrics.Counter'>, 'custom_distribution': <class 'glean_parser.metrics.CustomDistribution'>, 'datetime': <class 'glean_parser.metrics.Datetime'>, 'denominator': <class 'glean_parser.metrics.Denominator'>, 'event': <class 'glean_parser.metrics.Event'>, 'jwe': <class 'glean_parser.metrics.Jwe'>, 'labeled_boolean': <class 'glean_parser.metrics.LabeledBoolean'>, 'labeled_counter': <class 'glean_parser.metrics.LabeledCounter'>, 'labeled_custom_distribution': <class 'glean_parser.metrics.LabeledCustomDistribution'>, 'labeled_memory_distribution': <class 'glean_parser.metrics.LabeledMemoryDistribution'>, 'labeled_quantity': <class 'glean_parser.metrics.LabeledQuantity'>, 'labeled_string': <class 'glean_parser.metrics.LabeledString'>, 'labeled_timing_distribution': <class 'glean_parser.metrics.LabeledTimingDistribution'>, 'memory_distribution': <class 'glean_parser.metrics.MemoryDistribution'>, 'object': <class 'glean_parser.metrics.Object'>, 'quantity': <class 'glean_parser.metrics.Quantity'>, 'rate': <class 'glean_parser.metrics.Rate'>, 'string': <class 'glean_parser.metrics.String'>, 'string_list': <class 'glean_parser.metrics.StringList'>, 'text': <class 'glean_parser.metrics.Text'>, 'timespan': <class 'glean_parser.metrics.Timespan'>, 'timing_distribution': <class 'glean_parser.metrics.TimingDistribution'>, 'url': <class 'glean_parser.metrics.Url'>, 'uuid': <class 'glean_parser.metrics.Uuid'>}

serialize() → Dict[str, list | dict | str | int | float | None]

Serialize the metric back to JSON object model.

typename: str = 'ERROR'

validate_expires()

class glean_parser.metrics.Object(*args, **kwargs)

Bases: Metric

ALLOWED_TOPLEVEL = {'items', 'properties', 'type'}

ALLOWED_TYPES = ['object', 'array', 'number', 'string', 'boolean']

typename: str = 'object'

static validate_structure(structure)

class glean_parser.metrics.Quantity(type: str, category: str, name: str, bugs: List[str], description: str, notification_emails: List[str], expires: Any, metadata: Dict | None = None, data_reviews: List[str] | None = None, version: int = 0, disabled: bool = False, lifetime: str = 'ping', send_in_pings: List[str] | None = None, unit: str | None = None, gecko_datapoint: str = '', no_lint: List[str] | None = None, data_sensitivity: List[str] | None = None, defined_in: Dict | None = None, telemetry_mirror: str | None = None, _config: Dict[str, Any] | None = None, _validated: bool = False)

Bases: Metric

typename: str = 'quantity'

class glean_parser.metrics.Rate(*args, **kwargs)

Bases: Metric

typename: str = 'rate'

class glean_parser.metrics.String(type: str, category: str, name: str, bugs: List[str], description: str, notification_emails: List[str], expires: Any, metadata: Dict | None = None, data_reviews: List[str] | None = None, version: int = 0, disabled: bool = False, lifetime: str = 'ping', send_in_pings: List[str] | None = None, unit: str | None = None, gecko_datapoint: str = '', no_lint: List[str] | None = None, data_sensitivity: List[str] | None = None, defined_in: Dict | None = None, telemetry_mirror: str | None = None, _config: Dict[str, Any] | None = None, _validated: bool = False)

Bases: Metric

typename: str = 'string'

class glean_parser.metrics.StringList(type: str, category: str, name: str, bugs: List[str], description: str, notification_emails: List[str], expires: Any, metadata: Dict | None = None, data_reviews: List[str] | None = None, version: int = 0, disabled: bool = False, lifetime: str = 'ping', send_in_pings: List[str] | None = None, unit: str | None = None, gecko_datapoint: str = '', no_lint: List[str] | None = None, data_sensitivity: List[str] | None = None, defined_in: Dict | None = None, telemetry_mirror: str | None = None, _config: Dict[str, Any] | None = None, _validated: bool = False)

Bases: Metric

typename: str = 'string_list'

class glean_parser.metrics.Text(type: str, category: str, name: str, bugs: List[str], description: str, notification_emails: List[str], expires: Any, metadata: Dict | None = None, data_reviews: List[str] | None = None, version: int = 0, disabled: bool = False, lifetime: str = 'ping', send_in_pings: List[str] | None = None, unit: str | None = None, gecko_datapoint: str = '', no_lint: List[str] | None = None, data_sensitivity: List[str] | None = None, defined_in: Dict | None = None, telemetry_mirror: str | None = None, _config: Dict[str, Any] | None = None, _validated: bool = False)

Bases: Metric

typename: str = 'text'

class glean_parser.metrics.TimeBase(*args, **kwargs)

Bases: Metric

class glean_parser.metrics.TimeUnit(value)

Bases: Enum

An enumeration.

day = 6

hour = 5

microsecond = 1

millisecond = 2

minute = 4

nanosecond = 0

second = 3

class glean_parser.metrics.Timespan(*args, **kwargs)

Bases: TimeBase

typename: str = 'timespan'

class glean_parser.metrics.TimingDistribution(*args, **kwargs)

Bases: TimeBase

typename: str = 'timing_distribution'

class glean_parser.metrics.Url(type: str, category: str, name: str, bugs: List[str], description: str, notification_emails: List[str], expires: Any, metadata: Dict | None = None, data_reviews: List[str] | None = None, version: int = 0, disabled: bool = False, lifetime: str = 'ping', send_in_pings: List[str] | None = None, unit: str | None = None, gecko_datapoint: str = '', no_lint: List[str] | None = None, data_sensitivity: List[str] | None = None, defined_in: Dict | None = None, telemetry_mirror: str | None = None, _config: Dict[str, Any] | None = None, _validated: bool = False)

Bases: Metric

typename: str = 'url'

class glean_parser.metrics.Uuid(type: str, category: str, name: str, bugs: List[str], description: str, notification_emails: List[str], expires: Any, metadata: Dict | None = None, data_reviews: List[str] | None = None, version: int = 0, disabled: bool = False, lifetime: str = 'ping', send_in_pings: List[str] | None = None, unit: str | None = None, gecko_datapoint: str = '', no_lint: List[str] | None = None, data_sensitivity: List[str] | None = None, defined_in: Dict | None = None, telemetry_mirror: str | None = None, _config: Dict[str, Any] | None = None, _validated: bool = False)

Bases: Metric

typename: str = 'uuid'

glean_parser.parser module

Code for parsing metrics.yaml files.

glean_parser.parser.parse_objects(filepaths: Iterable[Path], config: Dict[str, Any] | None = None) → Generator[str, None, Dict[str, Dict[str, Metric | Ping | Tag]]]

Parse one or more metrics.yaml and/or pings.yaml files, returning a tree of metrics.Metric, pings.Ping, and tags.Tag instances.

The result is a generator over any errors. If there are no errors, the actual metrics can be obtained from result.value. For example:

result = metrics.parse_metrics(filepaths)
for err in result:
    print(err)
all_metrics = result.value

The result value is a dictionary of category names to categories, where each category is a dictionary from metric name to metrics.Metric instances. There are also the special categories pings and tags containing all of the pings.Ping and tags.Tag instances, respectively.

Parameters:

• filepaths – list of Path objects to metrics.yaml, pings.yaml, and/or tags.yaml files

• config –

  A dictionary of options that change parsing behavior. Supported keys are:

  • allow_reserved: Allow values reserved for internal Glean use.

  • do_not_disable_expired: Don’t mark expired metrics as disabled. This is useful when you want to retain the original “disabled” value from the metrics.yaml, rather than having it overridden when the metric expires.

  • allow_missing_files: Do not raise a FileNotFoundError if any of the input filepaths do not exist.

glean_parser.parser.validate(content: Dict[str, list | dict | str | int | float | None], filepath: str | Path = '<input>') → Generator[str, None, None]

Validate the given content against the appropriate schema.

glean_parser.pings module

Classes for managing the description of pings.

class glean_parser.pings.Ping(name: str, description: str, bugs: List[str], notification_emails: List[str], metadata: Dict | None = None, data_reviews: List[str] | None = None, include_client_id: bool = False, send_if_empty: bool = False, reasons: Dict[str, str] | None = None, defined_in: Dict | None = None, no_lint: List[str] | None = None, enabled: bool | None = None, _validated: bool = False)

Bases: object

identifier() → str

Used for the “generated from …” comment in the output.

property reason_codes: List[str]

serialize() → Dict[str, list | dict | str | int | float | None]

Serialize the metric back to JSON object model.

property type: str

glean_parser.python_server module

Outputter to generate server Python code for collecting events.

This outputter is different from the rest of the outputters in that the code it generates does not use the Glean SDK. It is meant to be used to collect events in server-side environments. In these environments SDK assumptions to measurement window and connectivity don’t hold. Generated code takes care of assembling pings with metrics, and serializing to messages conforming to Glean schema.

Warning: this outputter supports limited set of metrics, see SUPPORTED_METRIC_TYPES below.

The generated code creates a ServerEventLogger class for each ping that has at least one event metric. The class has a record method for each event metric.

glean_parser.python_server.camelize(s: str) → str

glean_parser.python_server.clean_string(s: str) → str

glean_parser.python_server.generate_event_record_function_name(event_metric: Metric) → str

glean_parser.python_server.generate_metric_type(metric_type: str) → str

glean_parser.python_server.generate_ping_factory_method(ping: str) → str

glean_parser.python_server.output_python(objs: Dict[str, Dict[str, Metric | Ping | Tag]], output_dir: Path, options: Dict[str, Any] | None) → None

Given a tree of objects, output Python code to output_dir.

The output is a file containing all the code for assembling pings with metrics, serializing, and submitting, and an empty __init__.py file to make the directory a package.

Parameters:

• objects – A tree of objects (metrics and pings) as returned from parser.parse_objects.

• output_dir – Path to an output directory to write to.

glean_parser.ruby_server module

Outputter to generate server ruby code for collecting events.

This outputter is different from the rest of the outputters in that the code it generates does not use the Glean SDK. It is meant to be used to collect events using “events as pings” pattern in server-side environments. In these environments SDK assumptions to measurement window and connectivity don’t hold. Generated code takes care of assembling pings with metrics, serializing to messages conforming to Glean schema, and logging using a standard Ruby logger. Then it’s the role of the ingestion pipeline to pick the messages up and process.

Warning: this outputter supports a limited set of metrics, see SUPPORTED_METRIC_TYPES below.

glean_parser.ruby_server.event_class_name(metric: Metric) → str

glean_parser.ruby_server.generate_metric_argument_description(metric: Metric) → str

glean_parser.ruby_server.generate_metric_argument_name(metric: Metric) → str

glean_parser.ruby_server.generate_metric_name(metric: Metric) → str

glean_parser.ruby_server.output_ruby(objs: Dict[str, Dict[str, Metric | Ping | Tag]], output_dir: Path, options: Dict[str, Any] | None) → None

Given a tree of objects, output ruby code to output_dir.

The output is a single file containing all the code for assembling pings with metrics, serializing, and submitting.

Parameters:

• objects – A tree of objects (metrics and pings) as returned from parser.parse_objects.

• output_dir – Path to an output directory to write to.

glean_parser.ruby_server.ping_class_name(pingName: str) → str

glean_parser.rust module

Outputter to generate Rust code for metrics.

class glean_parser.rust.Category(name: str, objs: Dict[str, Metric | Ping | Tag], contains_pings: bool)

Bases: object

Data struct holding information about a metric to be used in the template.

glean_parser.rust.class_name(obj_type)

Returns the Rust class name for a given metric or ping type.

glean_parser.rust.ctor(obj)

Returns the scope and name of the constructor to use for a metric object. Necessary because LabeledMetric<T> is constructed using LabeledMetric::new not LabeledMetric<T>::new

glean_parser.rust.extra_keys(allowed_extra_keys)

Returns the &’static [&’static str] ALLOWED_EXTRA_KEYS for impl ExtraKeys

glean_parser.rust.extra_type_name(typ: str) → str

Returns the corresponding Rust type for event’s extra key types.

glean_parser.rust.output_rust(objs: Dict[str, Dict[str, Metric | Ping | Tag]], output_dir: Path, options: Dict[str, Any] | None = None) → None

Given a tree of objects, output Rust code to output_dir.

Parameters:

• objs – A tree of objects (metrics and pings) as returned from parser.parse_objects.

• output_dir – Path to an output directory to write to.

• options – options dictionary, not currently used for Rust

glean_parser.rust.rust_datatypes_filter(value)

A Jinja2 filter that renders Rust literals.

Based on Python’s JSONEncoder, but overrides:

• dicts and sets to raise an error

• sets to vec![] (used in labels)

• enums to become Class::Value

• lists to vec![] (used in send_in_pings)

• null to None

• strings to “value”.into()

• Rate objects to a CommonMetricData initializer (for external Denominators’ Numerators lists)

glean_parser.rust.structure_type_name(typ: str) → str

Returns the corresponding Rust type for structure items.

glean_parser.rust.type_name(obj)

Returns the Rust type to use for a given metric or ping object.

glean_parser.swift module

Outputter to generate Swift code for metrics.

class glean_parser.swift.BuildInfo(build_date)

Bases: object

class glean_parser.swift.Category

Bases: object

Data struct holding information about a metric to be used in the template.

contains_pings: bool

name: str

objs: Dict[str, Metric | Ping | Tag]

glean_parser.swift.class_name(obj_type: str) → str

Returns the Swift class name for a given metric or ping type.

glean_parser.swift.extra_type_name(typ: str) → str

Returns the corresponding Swift type for event’s extra key types.

glean_parser.swift.generate_build_date(date: str | None) → str

Generate the build timestamp.

glean_parser.swift.output_swift(objs: Dict[str, Dict[str, Metric | Ping | Tag]], output_dir: Path, options: Dict[str, Any] | None = None) → None

Given a tree of objects, output Swift code to output_dir.

Parameters:

• objects – A tree of objects (metrics and pings) as returned from parser.parse_objects.

• output_dir – Path to an output directory to write to.

• options –

  options dictionary, with the following optional keys: - namespace: The namespace to generate metrics in - glean_namespace: The namespace to import Glean from - allow_reserved: When True, this is a Glean-internal build - with_buildinfo: If “true” the GleanBuildInfo is generated.

  Otherwise generation of that file is skipped. Defaults to “true”.

  • build_date: If set to 0 a static unix epoch time will be used.

    If set to a ISO8601 datetime string (e.g. 2022-01-03T17:30:00) it will use that date. Other values will throw an error. If not set it will use the current date & time.

glean_parser.swift.structure_type_name(typ: str) → str

Returns the corresponding Swift type for structure items.

glean_parser.swift.swift_datatypes_filter(value: list | dict | str | int | float | None) → str

A Jinja2 filter that renders Swift literals.

Based on Python’s JSONEncoder, but overrides:

• dicts to use [key: value]

• sets to use […]

• enums to use the like-named Swift enum

• Rate objects to a CommonMetricData initializer (for external Denominators’ Numerators lists)

glean_parser.swift.type_name(obj: Metric | Ping) → str

Returns the Swift type to use for a given metric or ping object.

glean_parser.swift.variable_name(var: str) → str

Returns a valid Swift variable name, escaping keywords if necessary.

glean_parser.tags module

class glean_parser.tags.Tag(name: str, description: str, defined_in: Dict | None = None, no_lint: List[str] | None = None, _validated: bool = False)

Bases: object

serialize() → Dict[str, list | dict | str | int | float | None]

Serialize the tag back to JSON object model.

property type: str

glean_parser.translate module

High-level interface for translating metrics.yaml into other formats.

class glean_parser.translate.Outputter(output_func: Callable[[Dict[str, Dict[str, Metric | Ping | Tag]], Path, Dict[str, Any]], None], clear_patterns: List[str] | None = None)

Bases: object

Class to define an output format.

Each outputter in the table has the following member values:

• output_func: the main function of the outputter, the one which does the actual translation.

• clear_patterns: A list of glob patterns to clear in the directory before writing new results to it.

glean_parser.translate.transform_metrics(objects)

Transform the object model from one that represents the YAML definitions to one that reflects the type specifics needed by code generators.

e.g. This will transform a rate to be a numerator if its denominator is external.

glean_parser.translate.translate(input_filepaths: Iterable[Path], output_format: str, output_dir: Path, options: Dict[str, Any] | None = None, parser_config: Dict[str, Any] | None = None)

Translate the files in input_filepaths to the given output_format and put the results in output_dir.

Parameters:

• input_filepaths – list of paths to input metrics.yaml files

• output_format – the name of the output format

• output_dir – the path to the output directory

• options – dictionary of options. The available options are backend format specific.

• parser_config – A dictionary of options that change parsing behavior. See parser.parse_metrics for more info.

glean_parser.translate.translate_metrics(input_filepaths: Iterable[Path], output_dir: Path, translation_func: Callable[[Dict[str, Dict[str, Metric | Ping | Tag]], Path, Dict[str, Any]], None], clear_patterns: List[str] | None = None, options: Dict[str, Any] | None = None, parser_config: Dict[str, Any] | None = None)

Translate the files in input_filepaths by running the metrics through a translation function and writing the results in output_dir.

Parameters:

• input_filepaths – list of paths to input metrics.yaml files

• output_dir – the path to the output directory

• translation_func –

  the function that actually performs the translation. It is passed the following arguments:

  • metrics_objects: The tree of metrics as pings as returned by parser.parse_objects.

  • output_dir: The path to the output directory.

  • options: A dictionary of output format-specific options.

  Examples of translation functions are in kotlin.py and swift.py.

• clear_patterns – a list of glob patterns of files to clear before generating the output files. By default, no files will be cleared (i.e. the directory should be left alone).

• options – dictionary of options. The available options are backend format specific. These are passed unchanged to translation_func.

• parser_config – A dictionary of options that change parsing behavior. See parser.parse_metrics for more info.

glean_parser.translation_options module

glean_parser.translation_options.translate_options(ctx, param, value)

glean_parser.util module

glean_parser.util.Camelize(value: str) → str

Convert the value to CamelCase (with an upper case first letter).

This is a thin wrapper around inflection.camelize that handles dots in addition to underscores.

class glean_parser.util.DictWrapper

Bases: dict

glean_parser.util.JSONType

The types supported by JSON.

This is only an approximation – this should really be a recursive type.

alias of list | dict | str | int | float | None

glean_parser.util.build_date(date: str | None) → datetime

Generate the build timestamp.

If date is set to 0 a static unix epoch time will be used. If date it is set to a ISO8601 datetime string (e.g. 2022-01-03T17:30:00) it will use that date. Note that any timezone offset will be ignored and UTC will be used. Otherwise it will throw an error.

If date is None it will use the current date & time.

glean_parser.util.camelize(value: str) → str

Convert the value to camelCase (with a lower case first letter).

This is a thin wrapper around inflection.camelize that handles dots in addition to underscores.

glean_parser.util.date_fromisoformat(datestr: str) → date

glean_parser.util.datetime_fromisoformat(datestr: str) → datetime

glean_parser.util.ensure_list(value: Any) → Sequence[Any]

Ensures that the value is a list. If it is anything but a list or tuple, a list with a single element containing only value is returned.

glean_parser.util.fetch_remote_url(url: str, cache: bool = True)

Fetches the contents from an HTTP url or local file path, and optionally caches it to disk.

glean_parser.util.format_error(filepath: str | Path, header: str, content: str, lineno: int | None = None) → str

Format a jsonshema validation error.

glean_parser.util.get_jinja2_template(template_name: str, filters: Iterable[Tuple[str, Callable]] = ())

Get a Jinja2 template that ships with glean_parser.

The template has extra filters for camel-casing identifiers.

Parameters:

• template_name – Name of a file in glean_parser/templates

• filters – tuple of 2-tuple. A tuple of (name, func) pairs defining additional filters.

glean_parser.util.get_null_resolver(schema)

Returns a JSON Pointer resolver that does nothing.

This lets us handle the moz: URLs in our schemas.

glean_parser.util.is_expired(expires: str, major_version: int | None = None) → bool

Parses the expires field in a metric or ping and returns whether the object should be considered expired.

glean_parser.util.keep_value(f)

Wrap a generator so the value it returns (rather than yields), will be accessible on the .value attribute when the generator is exhausted.

glean_parser.util.load_yaml_or_json(path: Path)

Load the content from either a .json or .yaml file, based on the filename extension.

Parameters:

path – pathlib.Path object

Rtype object:

The tree of objects as a result of parsing the file.

Raises:

• ValueError – The file is neither a .json, .yml or .yaml file.

• FileNotFoundError – The file does not exist.

glean_parser.util.ordered_yaml_dump(data, **kwargs)

glean_parser.util.parse_expiration_date(expires: str) → date

Parses the expired field date (yyyy-mm-dd) as a date. Raises a ValueError in case the string is not properly formatted.

glean_parser.util.parse_expiration_version(expires: str) → int

Parses the expired field version string as an integer. Raises a ValueError in case the string does not contain a valid positive integer.

glean_parser.util.pprint_validation_error(error) → str

A version of jsonschema’s ValidationError __str__ method that doesn’t include the schema fragment that failed. This makes the error messages much more succinct.

It also shows any subschemas of anyOf/allOf that failed, if any (what jsonschema calls “context”).

glean_parser.util.remove_output_params(d, output_params)

Remove output-only params, such as “defined_in”, in order to validate the output against the input schema.

glean_parser.util.report_validation_errors(all_objects)

Report any validation errors found to the console.

Returns the number of errors reported.

glean_parser.util.screaming_case(value: str) → str

Convert the value to SCREAMING_SNAKE_CASE.

glean_parser.util.snake_case(value: str) → str

Convert the value to snake_case.

glean_parser.util.to_camel_case(input: str, capitalize_first_letter: bool) → str

Convert the value to camelCase.

This additionally replaces any ‘.’ with ‘_’. The first letter is capitalized depending on capitalize_first_letter.

glean_parser.util.validate_expires(expires: str, major_version: int | None = None) → None

If expiration by major version is enabled, raises a ValueError in case expires is not a positive integer. Otherwise raises a ValueError in case the expires is not ISO8601 parseable, or in case the date is more than 730 days (~2 years) in the future.

glean_parser.util.yaml_load(stream)

Map line number to yaml nodes, and preserve the order of metrics as they appear in the metrics.yaml file.

glean_parser.validate_ping module

Validates the contents of a Glean ping against the schema.

glean_parser.validate_ping.validate_ping(ins, outs=None, schema_url=None)

Validates the contents of a Glean ping.

Parameters:

• ins – Input stream or file path to the ping contents to validate

• outs – Output stream to write errors to. (Defaults to stdout)

• schema_url – HTTP URL or local filesystem path to Glean ping schema. Defaults to the current version of the schema in mozilla-pipeline-schemas.

Return type:

int 1 if any errors occurred, otherwise 0.

Module contents

Top-level package for Glean parser.