Monitoring

User-facing views for pipeline monitoring, including views on payload_bytes tables

Source Directory

airflow_dag

Airflow Dag

monitoring.airflow_dag

Please provide a description for the query

Schema

Column	Description	Type	Nullable
dag_id	Airflow DAG id (.)	String	Yes
root_dag_id	Airflow Root DAG id	String	Yes
subdag_id	Airflow subdag id	String	Yes
is_subdag	Indicates if this DAG is a subdag	Boolean	Yes
description	DAG's description	String	Yes
schedule_interval	DAG's scheduling settings	String	Yes
owners	DAG owners (comma separated)	String	Yes
is_active	Indicates if the DAG is active	Boolean	Yes
is_paused	Indicates if the DAG is currently paused	Boolean	Yes
has_task_concurrency_limits	Indicates if DAG has concurrency limit set	Boolean	Yes
max_active_runs	Max number of active DAG runs allowed	Integer	Yes
max_active_tasks	Max number of active task instances in the DAG allowed	Integer	Yes
next_dagrun		Timestamp	Yes
next_dagrun_create_after		Timestamp	Yes
next_dagrun_data_interval_start		Timestamp	Yes
next_dagrun_data_interval_end		Timestamp	Yes
last_parsed_time	The last time the DAG was parsed	Timestamp	Yes

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	monitoring_derived	airflow_dag_v1

Source Directory | View Definition | Metadata File

---

airflow_dag_note

Airflow Dag Note

monitoring.airflow_dag_note

Please provide a description for the query

Schema

Column	Description	Type	Nullable
dag_run_id		Integer	Yes
content		String	Yes
created_at		Timestamp	Yes
updated_at		Timestamp	Yes
user_id		Integer	Yes

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	monitoring_derived	airflow_dag_note_v1

Source Directory | View Definition | Metadata File

---

airflow_dag_owner_attributes

Airflow Dag Owner Attributes

monitoring.airflow_dag_owner_attributes

Please provide a description for the query

Schema

Column	Description	Type	Nullable
dag_id		String	Yes
owner		String	Yes
link		String	Yes

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	monitoring_derived	airflow_dag_owner_attributes_v1

Source Directory | View Definition | Metadata File

---

airflow_dag_run

Airflow Dag Run

monitoring.airflow_dag_run

Please provide a description for the query

Schema

Column	Description	Type	Nullable
dag_id	Airflow DAG id	String	Yes
run_type	manual, scheduled or backfill	String	Yes
external_trigger	Indicates if the DAG was triggered externally	Boolean	Yes
state	State of the DAG run instance (e.g. success)	String	Yes
execution_date	Execution date of the DAG Run instance	Timestamp	Yes
start_date	When the DAG run started	Timestamp	Yes
end_date	When the DAG run finished	Timestamp	Yes

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	monitoring_derived	airflow_dag_run_v1

Source Directory | View Definition | Metadata File

---

airflow_dag_tag

Airflow Dag Tag

monitoring.airflow_dag_tag

Please provide a description for the query

Schema

Column	Description	Type	Nullable
dag_id	Airflow DAG id	String	Yes
tags		String	No

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	monitoring_derived	airflow_dag_tag_v1

Source Directory | View Definition | Metadata File

---

airflow_dag_warning

Airflow Dag Warning

monitoring.airflow_dag_warning

Please provide a description for the query

Schema

Column	Description	Type	Nullable
timestamp		Timestamp	Yes
dag_id		String	Yes
warning_type		String	Yes
message		String	Yes

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	monitoring_derived	airflow_dag_warning_v1

Source Directory | View Definition | Metadata File

---

airflow_import_error

Airflow Import Error

monitoring.airflow_import_error

Please provide a description for the query

Schema

Column	Description	Type	Nullable
timestamp		Timestamp	Yes
id		Integer	Yes
filename		String	Yes
stacktrace		String	Yes

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	monitoring_derived	airflow_import_error_v1

Source Directory | View Definition | Metadata File

---

airflow_job

Airflow Job

monitoring.airflow_job

Please provide a description for the query

Schema

Column	Description	Type	Nullable
id		Integer	Yes
dag_id		String	Yes
end_date		Timestamp	Yes
executor_class		String	Yes
hostname		String	Yes
job_type		String	Yes
latest_heartbeat		Timestamp	Yes
start_date		Timestamp	Yes
state		String	Yes
unixname		String	Yes

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	monitoring_derived	airflow_job_v1

Source Directory | View Definition | Metadata File

---

airflow_slot_pool

Airflow Slot Pool

monitoring.airflow_slot_pool

Please provide a description for the query

Schema

Column	Description	Type	Nullable
id	Pool id	Integer	Yes
pool	Pool name	String	Yes
description	Pool description	String	Yes
slots	Number of slots available in the pool	Integer	Yes

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	monitoring_derived	airflow_slot_pool_v1

Source Directory | View Definition | Metadata File

---

airflow_task_fail

Airflow Task Fail

monitoring.airflow_task_fail

Please provide a description for the query

Schema

Column	Description	Type	Nullable
dag_id	Airflow DAG id	String	Yes
task_id	Airflow task id	String	Yes
duration		Integer	Yes
start_date		Timestamp	Yes
end_date		Timestamp	Yes

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	monitoring_derived	airflow_task_fail_v1

Source Directory | View Definition | Metadata File

---

airflow_task_instance

Airflow Task Instance

monitoring.airflow_task_instance

Please provide a description for the query

Schema

Column	Description	Type	Nullable
dag_id		String	Yes
run_id		String	Yes
task_id		String	Yes
state		String	Yes
try_number		Integer	Yes
start_date		Timestamp	Yes
end_date		Timestamp	Yes
duration		Float	Yes
job_id		Integer	Yes
executor_config		Bytes	Yes
external_executor_id		String	Yes
hostname		String	Yes
max_tries		Integer	Yes
next_kwargs		String	Yes
next_method		String	Yes
operator		String	Yes
pid		Integer	Yes
pool		String	Yes
pool_slots		Integer	Yes
priority_weight		Integer	Yes
queue		String	Yes
queued_by_job_id		Integer	Yes
queued_dttm		Timestamp	Yes
trigger_id		Integer	Yes
trigger_timeout		Datetime	Yes
unixname		String	Yes
map_index		Integer	Yes

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	monitoring_derived	airflow_task_instance_v1

Source Directory | View Definition | Metadata File

---

airflow_task_instance_note

Airflow Task Instance Note

monitoring.airflow_task_instance_note

Please provide a description for the query

Schema

Column	Description	Type	Nullable
dag_id		String	Yes
map_index		Integer	Yes
run_id		String	Yes
task_id		String	Yes
content		String	Yes
created_at		Timestamp	Yes
updated_at		Timestamp	Yes
user_id		Integer	Yes

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	monitoring_derived	airflow_task_instance_note_v1

Source Directory | View Definition | Metadata File

---

airflow_task_reschedule

Airflow Task Reschedule

monitoring.airflow_task_reschedule

Please provide a description for the query

Schema

Column	Description	Type	Nullable
dag_id	Airflow DAG id	String	Yes
task_id	Airflow task id	String	Yes
run_id	Unique identifier of the task run instance	String	Yes
try_number	Attempt number to execute the task	Integer	Yes
reschedule_date	Datetime when task has been rescheduled	Timestamp	Yes
start_date	When the DAG run started	Timestamp	Yes
end_date	When the DAG run finished	Timestamp	Yes
duration	Time elapsed since start until end of task's exeuction (seconds)	Integer	Yes

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	monitoring_derived	airflow_task_reschedule_v1

Source Directory | View Definition | Metadata File

---

airflow_trigger

Airflow Trigger

monitoring.airflow_trigger

Please provide a description for the query

Schema

Column	Description	Type	Nullable
id		Integer	Yes
classpath		String	Yes
created_date		Timestamp	Yes
kwargs		String	Yes
triggerer_id		Integer	Yes
trigger_id		Integer	Yes
created_at		Timestamp	Yes

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	monitoring_derived	airflow_trigger_v1

Source Directory | View Definition | Metadata File

---

airflow_user

Airflow User

monitoring.airflow_user

Please provide a description for the query

Schema

Column	Description	Type	Nullable
user_role		String	Yes
user_id	Airflow user id	Integer	Yes
created_on		Datetime	Yes
changed_on		Datetime	Yes
last_login		Datetime	Yes

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	monitoring_derived	airflow_user_v1

Source Directory | View Definition | Metadata File

---

average_ping_sizes

Average Ping Sizes

monitoring.average_ping_sizes

Please provide a description for the query

Schema

Column	Description	Type	Nullable
submission_date		Date	Yes
dataset_id		String	Yes
table_id		String	Yes
average_byte_size		Float	Yes
total_byte_size		Integer	Yes
row_count		Integer	Yes

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	monitoring_derived	average_ping_sizes_v1

Source Directory | View Definition | Metadata File

---

bigquery_etl_scheduled_queries_cost

Bigquery Etl Scheduled Queries Cost

monitoring.bigquery_etl_scheduled_queries_cost

Please provide a description for the query

Schema

Column	Description	Type	Nullable
submission_date		Date	Yes
creation_time		Timestamp	Yes
dataset		String	Yes
table		String	Yes
cost_usd		Float	Yes
total_bytes_processed		Integer	Yes

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	monitoring_derived	bigquery_etl_scheduled_queries_cost_v1

Source Directory | View Definition | Metadata File

---

bigquery_etl_scheduled_query_usage

Bigquery Etl Scheduled Query Usage

monitoring.bigquery_etl_scheduled_query_usage

Please provide a description for the query

Schema

Column	Description	Type	Nullable
submission_date		Date	Yes
project_id		String	Yes
table		String	Yes
number_of_accesses		Integer	Yes

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	monitoring_derived	bigquery_etl_scheduled_query_usage_v1

Source Directory | View Definition | Metadata File

---

bigquery_table_storage

Bigquery Table Storage

monitoring.bigquery_table_storage

Please provide a description for the query

Schema

Column	Description	Type	Nullable
creation_date	The table's creation date	Date	Yes
project_id	The ID of the project that contains the dataset	String	Yes
dataset_id	The name of the dataset that contains the table	String	Yes
table_id	The name of the table	String	Yes
total_rows	The total number of rows in the table	Integer	Yes
total_partitions	The total number of partitions present in the table	Integer	Yes
total_logical_bytes	Total number of uncompressed bytes in the table	Integer	Yes
active_logical_bytes	Number of uncompressed bytes less than 90 days old	Integer	Yes
long_term_logical_bytes	Number of uncompressed bytes more than 90 days old	Integer	Yes
total_physical_bytes	Total number of compressed bytes used for storage	Integer	Yes
active_physical_bytes	Number of compressed bytes less than 90 days old	Integer	Yes
long_term_physical_bytes	Number of compressed bytes more than 90 days old	Integer	Yes
time_travel_physical_bytes	Number of compressed bytes for deleted or changed data	Integer	Yes

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	monitoring_derived	bigquery_table_storage_v1

Source Directory | View Definition | Metadata File

---

bigquery_table_storage_timeline_daily

Bigquery Table Storage Timeline Daily

monitoring.bigquery_table_storage_timeline_daily

Please provide a description for the query

Schema

Column	Description	Type	Nullable
change_date	Date of change to the data in the table.	Date	Yes
project_id	The ID of the project that contains the dataset	String	Yes
dataset_id	The name of the dataset that contains the table	String	Yes
table_id	The name of the table	String	Yes
deleted	Indicates whether or not the table is deleted	Boolean	Yes
creation_date	The table's creation date	Date	Yes
change_count	The total number of changes to a table in one day	Integer	Yes
avg_total_rows	Average total number of rows in the table	Float	Yes
avg_total_partitions	Average total number of partitions in the table	Float	Yes
avg_total_logical_bytes	Average total number of uncompressed bytes in the table	Float	Yes
avg_active_logical_bytes	Average number of uncompressed bytes less than 90 days old	Float	Yes
avg_long_term_logical_bytes	Average number of uncompressed bytes more than 90 days old	Float	Yes
avg_total_physical_bytes	Average total number of compressed bytes used for storage	Float	Yes
avg_active_physical_bytes	Average number of compressed bytes less than 90 days old	Float	Yes
avg_long_term_physical_bytes	Average number of compressed bytes more than 90 days old	Float	Yes
avg_time_travel_physical_bytes	Average number of compressed bytes for deleted or changed data	Float	Yes

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	monitoring_derived	bigquery_table_storage_timeline_daily_v1

Source Directory | View Definition | Metadata File

---

bigquery_tables_inventory

Bigquery Tables Inventory

monitoring.bigquery_tables_inventory

Please provide a description for the query

Schema

Column	Description	Type	Nullable
submission_date	The date when data was captured	Date	Yes
creation_date	The table's creation date	Date	Yes
project_id	The ID of the project that contains the dataset	String	Yes
dataset_id	The name of the dataset that contains the table or view	String	Yes
table_id	The name of the table or view	String	Yes
table_type	The table type	String	Yes
last_modified_date	The table's last modified date	Date	Yes
owners	The owner of the table listed in metadata yaml file	String	No
last_used_date	The date when the tables was last queried	Date	Yes

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	monitoring_derived	bigquery_tables_inventory_v1

Source Directory | View Definition | Metadata File

---

bigquery_usage

Bigquery Usage

monitoring.bigquery_usage

Please provide a description for the query

Schema

Column	Description	Type	Nullable
source_project	Project used for executing job	String	Yes
creation_date	Job creation date	Date	Yes
job_id	ID of job	String	Yes
job_type	The type of the job	String	Yes
state	Running state of the job	String	Yes
reference_project_id	The ID of the project referenced by the job	String	Yes
reference_dataset_id	The ID of the dataset referenced by the job	String	Yes
reference_table_id	The ID of the table referenced by the job	String	Yes
destination_project_id	The ID of the project containing the destination table	String	Yes
destination_dataset_id	The ID of the dataset containing the destination table	String	Yes
destination_table_id	The ID of the destination table	String	Yes
user_email	Email address or service account of the user who ran the job	String	Yes
username	Email address of the user who ran the query	String	Yes
query_id	query id of the job	String	Yes
task_duration	The time it took to run the job	Interval	Yes
total_terabytes_processed	Total terabytes processed by the job	Float	Yes
error_location	Specifies where the error occurred, if present	String	Yes
error_reason	A short error code that summarizes the error	String	Yes
error_message	A human-readable description of the error	String	Yes
reservation_id	Name of the primary reservation (slots for distributing resources) assigned to this job.	String	Yes
cache_hit	Whether the query results of this job were from a cache	Boolean	Yes

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	monitoring_derived	bigquery_usage_v1

Source Directory | View Definition | Metadata File

---

bigquery_usage_costs

Bigquery Usage Costs

monitoring.bigquery_usage_costs

Please provide a description for the query

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	monitoring	bigquery_usage

Source Directory | View Definition | Metadata File

---

bigquery_usage_jobs_by_org

Bigquery Usage Jobs By Org

monitoring.bigquery_usage_jobs_by_org

Please provide a description for the query

Schema

Column	Description	Type	Nullable
source_project	Project used for executing job	String	Yes
creation_date	Job creation date	Date	Yes
job_id	ID of job	String	Yes
job_type	The type of the job	String	Yes
reservation_id	Name of the primary reservation (slots for distributing resources) assigned to this job	String	Yes
cache_hit	Whether the query results of this job were from a cache	Boolean	Yes
state	Running state of the job	String	Yes
statement_type	The type of query statement	String	Yes
query_id	The id of the query	String	Yes
reference_project_id	The ID of the project referenced by the job	String	Yes
reference_dataset_id	The ID of the dataset referenced by the job	String	Yes
reference_table_id	The ID of the table referenced by the job	String	Yes
destination_project_id	The ID of the project containing the destination table	String	Yes
destination_dataset_id	The ID of the dataset containing the destination table	String	Yes
destination_table_id	The ID of the destination table	String	Yes
user_email	Email address or service account of the user who ran the job	String	Yes
username	The name of the user who ran the job	String	Yes
task_duration	The time it took to run the job	Interval	Yes
total_terabytes_processed	Total terabytes processed by the job	Float	Yes
total_terabytes_billed	Total terabytes billed by the job if on-demand pricing is used	Float	Yes
total_slot_ms	Slot milliseconds for the job over its entire duration	Integer	Yes
error_location	Specifies where the error occurred, if present	String	Yes
error_reason	A short error code that summarizes the error	String	Yes
error_message	A human-readable description of the error	String	Yes
resource_warning	The warning message that appears if the resource usage is above the internal threshold of the system	String	Yes
submission_date	Date Airflow DAG is executed, and partition date	Date	Yes

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	monitoring_derived	bigquery_usage_v2

Source Directory | View Definition | Metadata File

---

column_size

Column Size

monitoring.column_size

Please provide a description for the query

Schema

Column	Description	Type	Nullable
submission_date		Date	Yes
dataset_id		String	Yes
table_id		String	Yes
column_name		String	Yes
byte_size		Integer	Yes

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	monitoring_derived	column_size_v1

Source Directory | View Definition | Metadata File

---

deletion_request_volume

Deletion Request Volume

monitoring.deletion_request_volume

Please provide a description for the query

Schema

Column	Description	Type	Nullable
submission_date	Date for which deletion request pings were counted	Date	Yes
dataset_id	Dataset for which deletion request pings were counted	String	Yes
num_rows	Total number of deletion request pings	Integer	Yes

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	monitoring_derived	deletion_request_volume_v2

Source Directory | View Definition | Metadata File

---

event_monitoring_live

Event Monitoring Live

monitoring.event_monitoring_live

View that combines live and aggregated event monitoring data for

• Owners:

  - ascholtz@mozilla.com

  - akomar@mozilla.com

Schema

Column	Description	Type	Nullable
window_start		Timestamp	Yes
window_end		Timestamp	Yes
event_category		String	Yes
event_name		String	Yes
event_extra_key		String	Yes
country		String	Yes
normalized_app_name		String	Yes
channel		String	Yes
version		String	Yes
experiment		String	Yes
experiment_branch		String	Yes
total_events		Integer	Yes

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	accounts_backend_derived	event_monitoring_live_v1
moz-fx-data-shared-prod	accounts_frontend_derived	event_monitoring_live_v1
moz-fx-data-shared-prod	bedrock_derived	event_monitoring_live_v1
moz-fx-data-shared-prod	burnham_derived	event_monitoring_live_v1
moz-fx-data-shared-prod	debug_ping_view_derived	event_monitoring_live_v1
moz-fx-data-shared-prod	firefox_desktop_background_defaultagent_derived	event_monitoring_live_v1
moz-fx-data-shared-prod	firefox_desktop_background_tasks_derived	event_monitoring_live_v1
moz-fx-data-shared-prod	firefox_desktop_background_update_derived	event_monitoring_live_v1
moz-fx-data-shared-prod	firefox_desktop_derived	event_monitoring_live_v1
moz-fx-data-shared-prod	firefox_translations_derived	event_monitoring_live_v1
moz-fx-data-shared-prod	glean_dictionary_derived	event_monitoring_live_v1
moz-fx-data-shared-prod	mdn_yari_derived	event_monitoring_live_v1
moz-fx-data-shared-prod	monitor_cirrus_derived	event_monitoring_live_v1
moz-fx-data-shared-prod	monitor_frontend_derived	event_monitoring_live_v1
moz-fx-data-shared-prod	monitoring_derived	event_monitoring_aggregates_v1
moz-fx-data-shared-prod	moso_mastodon_backend_derived	event_monitoring_live_v1
moz-fx-data-shared-prod	moso_mastodon_web_derived	event_monitoring_live_v1
moz-fx-data-shared-prod	mozilla_lockbox_derived	event_monitoring_live_v1
moz-fx-data-shared-prod	mozilla_mach_derived	event_monitoring_live_v1
moz-fx-data-shared-prod	mozillavpn_derived	event_monitoring_live_v1
moz-fx-data-shared-prod	mozphab_derived	event_monitoring_live_v1
moz-fx-data-shared-prod	org_mozilla_bergamot_derived	event_monitoring_live_v1
moz-fx-data-shared-prod	org_mozilla_connect_firefox_derived	event_monitoring_live_v1
moz-fx-data-shared-prod	org_mozilla_fenix_derived	event_monitoring_live_v1
moz-fx-data-shared-prod	org_mozilla_fenix_nightly_derived	event_monitoring_live_v1
moz-fx-data-shared-prod	org_mozilla_fennec_aurora_derived	event_monitoring_live_v1
moz-fx-data-shared-prod	org_mozilla_firefox_beta_derived	event_monitoring_live_v1
moz-fx-data-shared-prod	org_mozilla_firefox_derived	event_monitoring_live_v1
moz-fx-data-shared-prod	org_mozilla_firefox_vpn_derived	event_monitoring_live_v1
moz-fx-data-shared-prod	org_mozilla_firefoxreality_derived	event_monitoring_live_v1
moz-fx-data-shared-prod	org_mozilla_focus_beta_derived	event_monitoring_live_v1
moz-fx-data-shared-prod	org_mozilla_focus_derived	event_monitoring_live_v1
moz-fx-data-shared-prod	org_mozilla_focus_nightly_derived	event_monitoring_live_v1
moz-fx-data-shared-prod	org_mozilla_ios_fennec_derived	event_monitoring_live_v1
moz-fx-data-shared-prod	org_mozilla_ios_firefox_derived	event_monitoring_live_v1
moz-fx-data-shared-prod	org_mozilla_ios_firefoxbeta_derived	event_monitoring_live_v1
moz-fx-data-shared-prod	org_mozilla_ios_firefoxvpn_derived	event_monitoring_live_v1
moz-fx-data-shared-prod	org_mozilla_ios_firefoxvpn_network_extension_derived	event_monitoring_live_v1
moz-fx-data-shared-prod	org_mozilla_ios_focus_derived	event_monitoring_live_v1
moz-fx-data-shared-prod	org_mozilla_ios_klar_derived	event_monitoring_live_v1
moz-fx-data-shared-prod	org_mozilla_ios_lockbox_derived	event_monitoring_live_v1
moz-fx-data-shared-prod	org_mozilla_klar_derived	event_monitoring_live_v1
moz-fx-data-shared-prod	org_mozilla_mozregression_derived	event_monitoring_live_v1
moz-fx-data-shared-prod	org_mozilla_reference_browser_derived	event_monitoring_live_v1
moz-fx-data-shared-prod	org_mozilla_tv_firefox_derived	event_monitoring_live_v1
moz-fx-data-shared-prod	org_mozilla_vrbrowser_derived	event_monitoring_live_v1
moz-fx-data-shared-prod	pine_derived	event_monitoring_live_v1
moz-fx-data-shared-prod	treeherder_derived	event_monitoring_live_v1
moz-fx-data-shared-prod	viu_politica_derived	event_monitoring_live_v1

Source Directory | View Definition | Metadata File

---

payload_bytes_decoded_all

All Decoded Tables

monitoring.payload_bytes_decoded_all

A view on top of the payload_bytes_decoded.* tables that includes all fields except the payload.

This view is useful for overall pipeline monitoring.

Clustering fields: submission_timestamp

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	monitoring	payload_bytes_decoded_structured
moz-fx-data-shared-prod	monitoring	payload_bytes_decoded_stub_installer
moz-fx-data-shared-prod	monitoring	payload_bytes_decoded_telemetry

Source Directory | View Definition | Metadata File

---

payload_bytes_decoded_structured

Structured Pipeline Family Decoded

monitoring.payload_bytes_decoded_structured

A view on top of the payload_bytes_decoded.structured* tables that includes all fields except the payload.

This view is useful for overall pipeline monitoring.

Clustering fields: submission_timestamp

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	payload_bytes_decoded	structured*

Source Directory | View Definition | Metadata File

---

payload_bytes_decoded_stub_installer

Stub Installer Pipeline Family Decoded

monitoring.payload_bytes_decoded_stub_installer

A view on top of the payload_bytes_decoded.stub_installer* tables that includes all fields except the payload.

This view is useful for overall pipeline monitoring.

Clustering fields: submission_timestamp

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	payload_bytes_decoded	stub_installer*

Source Directory | View Definition | Metadata File

---

payload_bytes_decoded_telemetry

Telemetry Pipeline Family Decoded

monitoring.payload_bytes_decoded_telemetry

A view on top of the payload_bytes_decoded.telemetry* tables that includes all fields except the payload.

This view is useful for overall pipeline monitoring.

Clustering fields: submission_timestamp

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	payload_bytes_decoded	telemetry*

Source Directory | View Definition | Metadata File

---

payload_bytes_error_all

All Errors

monitoring.payload_bytes_error_all

A view on top of the payload_bytes_error.* tables that includes all fields except the payload.

This view is useful for overall monitoring of pipeline errors.

Clustering fields: submission_timestamp

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	monitoring	payload_bytes_error_structured
moz-fx-data-shared-prod	payload_bytes_error	stub_installer
moz-fx-data-shared-prod	payload_bytes_error	telemetry

Source Directory | View Definition | Metadata File

---

payload_bytes_error_structured

Structured Pipeline Family Errors

monitoring.payload_bytes_error_structured

A view on top of the payload_bytes_error.structured table that includes all fields except the payload.

This view is useful for overall monitoring of pipeline errors.

Clustering fields: submission_timestamp

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	payload_bytes_error	structured

Source Directory | View Definition | Metadata File

---

schema_error_counts

Schema Error Counts

monitoring.schema_error_counts

Please provide a description for the query

Schema

Column	Description	Type	Nullable
submission_date		Date	Yes
document_namespace		String	Yes
document_type		String	Yes
document_version		String	Yes
hour		Timestamp	Yes
job_name		String	Yes
path		String	Yes
error_count		Integer	Yes
sample_error_messages	concatenated error messages, limited to 300 characters	String	Yes

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	monitoring_derived	schema_error_counts_v2

Source Directory | View Definition | Metadata File

---

shredder_progress

Shredder Progress

monitoring.shredder_progress

Please provide a description for the query

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	monitoring_derived	shredder_progress_v1

Source Directory | View Definition | Metadata File

---

shredder_rows_deleted

Shredder Rows Deleted

monitoring.shredder_rows_deleted

Please provide a description for the query

Schema

Column	Description	Type	Nullable
end_time		Timestamp	Yes
job_id		String	Yes
project_id		String	Yes
dataset_id		String	Yes
table_id		String	Yes
partition_id		String	Yes
deleted_row_count		Integer	Yes

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	monitoring_derived	shredder_rows_deleted_v1

Source Directory | View Definition | Metadata File

---

stable_and_derived_table_sizes

Stable And Derived Table Sizes

monitoring.stable_and_derived_table_sizes

Please provide a description for the query

Schema

Column	Description	Type	Nullable
submission_date		Date	Yes
dataset_id		String	Yes
table_id		String	Yes
byte_size		Integer	Yes

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	monitoring_derived	stable_and_derived_table_sizes_v1

Source Directory | View Definition | Metadata File

---

stable_table_column_counts

Stable Table Column Counts

monitoring.stable_table_column_counts

Please provide a description for the query

Schema

Column	Description	Type	Nullable
submission_date		Date	Yes
dataset		String	Yes
table_name		String	Yes
total_columns		Integer	Yes

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	monitoring_derived	stable_table_column_counts_v1

Source Directory | View Definition | Metadata File

---

stable_table_sizes

Stable Table Sizes

monitoring.stable_table_sizes

Please provide a description for the query

Schema

Column	Description	Type	Nullable
submission_date		Date	Yes
dataset_id		String	Yes
table_id		String	Yes
byte_size		Integer	Yes

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	monitoring_derived	stable_table_sizes_v1

Source Directory | View Definition | Metadata File

---

structured_detailed_error_counts

Structured Detailed Error Counts

monitoring.structured_detailed_error_counts

Please provide a description for the query

Schema

Column	Description	Type	Nullable
hour		Timestamp	Yes
document_namespace		String	Yes
document_type		String	Yes
document_version		String	Yes
error_type		String	Yes
ping_count		Integer	Yes
error_count		Integer	Yes
error_message		String	Yes
sample_payload		String	Yes
error_ratio		Float	Yes

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	monitoring_derived	structured_detailed_error_counts_v1

Source Directory | View Definition | Metadata File

---

structured_distinct_docids

Structured Distinct Docids

monitoring.structured_distinct_docids

Please provide a description for the query

Schema

Column	Description	Type	Nullable
submission_date		Date	Yes
namespace		String	Yes
doc_type		String	Yes
decoded		Integer	Yes
live		Integer	Yes
stable		Integer	Yes
decoded_nondistinct		Integer	Yes
live_nondistinct		Integer	Yes

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	monitoring_derived	structured_distinct_docids_v1

Source Directory | View Definition | Metadata File

---

structured_error_counts

Structured Error Counts

monitoring.structured_error_counts

Please provide a description for the query

Schema

Column	Description	Type	Nullable
hour		Timestamp	Yes
document_namespace		String	Yes
document_type		String	Yes
document_version		String	Yes
error_type		String	Yes
ping_count		Integer	Yes
error_count		Integer	Yes
error_ratio		Float	Yes

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	monitoring_derived	structured_error_counts_v1

Source Directory | View Definition | Metadata File

---

structured_missing_columns

Structured Missing Columns

monitoring.structured_missing_columns

Please provide a description for the query

Schema

Column	Description	Type	Nullable
submission_date		Date	Yes
document_namespace		String	Yes
document_type		String	Yes
document_version		String	Yes
path		String	Yes
path_count		Integer	Yes

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	monitoring_derived	structured_missing_columns_v1

Source Directory | View Definition | Metadata File

---

suggest_click_rate_live

Live quicksuggest-click Ping Count Per Minute

monitoring.suggest_click_rate_live

A live view of submissions intended for operational monitoring use cases.

Schema

Column	Description	Type	Nullable
submission_minute		Timestamp	Yes
n		Integer	Yes
n_release		Integer	Yes
n_beta		Integer	Yes
n_nightly		Integer	Yes
n_merino		Integer	Yes
n_merino_release		Integer	Yes
n_merino_beta		Integer	Yes
n_merino_nightly		Integer	Yes

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	monitoring_derived	suggest_click_rate_live_v1

Source Directory | View Definition | Metadata File

---

suggest_impression_rate_live

Live quicksuggest-impression Ping Count Per Minute

monitoring.suggest_impression_rate_live

A live view of submissions intended for operational monitoring use cases.

Schema

Column	Description	Type	Nullable
submission_minute		Timestamp	Yes
n		Integer	Yes
n_release		Integer	Yes
n_beta		Integer	Yes
n_nightly		Integer	Yes
n_merino		Integer	Yes
n_merino_release		Integer	Yes
n_merino_beta		Integer	Yes
n_merino_nightly		Integer	Yes

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	monitoring_derived	suggest_impression_rate_live_v1

Source Directory | View Definition | Metadata File

---

telemetry_distinct_docids

Telemetry Distinct Docids

monitoring.telemetry_distinct_docids

Please provide a description for the query

Schema

Column	Description	Type	Nullable
submission_date		Date	Yes
doc_type		String	Yes
decoded		Integer	Yes
live		Integer	Yes
stable		Integer	Yes
namespace		String	Yes
decoded_nondistinct		Integer	Yes
live_nondistinct		Integer	Yes

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	monitoring_derived	telemetry_distinct_docids_v1

Source Directory | View Definition | Metadata File

---

telemetry_missing_columns

Telemetry Missing Columns

monitoring.telemetry_missing_columns

Please provide a description for the query

Schema

Column	Description	Type	Nullable
submission_date		Date	Yes
document_namespace		String	Yes
document_type		String	Yes
document_version		String	Yes
path		String	Yes
path_count		Integer	Yes

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	monitoring_derived	telemetry_missing_columns_v3

Source Directory | View Definition | Metadata File

---

topsites_click_rate_live

Live topsites-click Ping Count Per Minute

monitoring.topsites_click_rate_live

A live view of submissions intended for operational monitoring use cases.

Schema

Column	Description	Type	Nullable
submission_minute		Timestamp	Yes
n		Integer	Yes
n_contile		Integer	Yes
n_remotesettings		Integer	Yes

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	monitoring_derived	topsites_click_rate_live_v1

Source Directory | View Definition | Metadata File

---

topsites_impression_rate_live

Live topsites-impression Ping Count Per Minute

monitoring.topsites_impression_rate_live

A live view of submissions intended for operational monitoring use cases.

Schema

Column	Description	Type	Nullable
submission_minute		Timestamp	Yes
n		Integer	Yes
n_contile		Integer	Yes
n_remotesettings		Integer	Yes

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	monitoring_derived	topsites_impression_rate_live_v1

Source Directory | View Definition | Metadata File

---

topsites_rate_fenix_live

Live topsites Ping Counts Per Minute for Fenix

monitoring.topsites_rate_fenix_live

A live view of impressions and clicks intended for operational monitoring use cases.

Schema

Column	Description	Type	Nullable
submission_minute		Timestamp	Yes
n_click		Integer	Yes
n_impression		Integer	Yes
n_click_release		Integer	Yes
n_impression_release		Integer	Yes

Referenced Tables

Project	Dataset	Table
moz-fx-data-shared-prod	monitoring_derived	topsites_rate_fenix_beta_live_v1
moz-fx-data-shared-prod	monitoring_derived	topsites_rate_fenix_nightly_live_v1
moz-fx-data-shared-prod	monitoring_derived	topsites_rate_fenix_release_live_v1

Source Directory | View Definition | Metadata File

---