mozetl package

Subpackages

• mozetl.addon_aggregates package
  • Submodules
  • mozetl.addon_aggregates.addon_aggregates module
  • Module contents
• mozetl.basic package
  • Submodules
  • mozetl.basic.transform module
  • Module contents
• mozetl.clientsdaily package
  • Submodules
  • mozetl.clientsdaily.fields module
  • mozetl.clientsdaily.rollup module
  • Module contents
• mozetl.engagement package
  • Subpackages
    • mozetl.engagement.churn package
      • Submodules
      • mozetl.engagement.churn.job module
      • mozetl.engagement.churn.release module
      • mozetl.engagement.churn.schema module
      • mozetl.engagement.churn.utils module
      • Module contents
    • mozetl.engagement.churn_to_csv package
      • Submodules
      • mozetl.engagement.churn_to_csv.job module
      • Module contents
    • mozetl.engagement.retention package
      • Submodules
      • mozetl.engagement.retention.job module
      • mozetl.engagement.retention.schema module
      • Module contents
  • Module contents
• mozetl.experimentsdaily package
  • Submodules
  • mozetl.experimentsdaily.rollup module
  • Module contents
• mozetl.hardware_report package
  • Submodules
  • mozetl.hardware_report.check_output module
  • mozetl.hardware_report.hardware_dashboard module
  • mozetl.hardware_report.summarize_json module
  • Module contents
• mozetl.landfill package
  • Submodules
  • mozetl.landfill.sampler module
  • Module contents
• mozetl.search package
  • Submodules
  • mozetl.search.aggregates module
  • Module contents
• mozetl.shield package
  • Submodules
  • mozetl.shield.privacy_prefs module
  • mozetl.shield.utils module
  • Module contents
• mozetl.sync package
  • Submodules
  • mozetl.sync.bookmark_validation module
  • Module contents
• mozetl.taar package
  • Submodules
  • mozetl.taar.taar_amodump module
  • mozetl.taar.taar_amowhitelist module
  • mozetl.taar.taar_dynamo module
  • mozetl.taar.taar_lite_guidguid module
  • mozetl.taar.taar_lite_guidranking module
  • mozetl.taar.taar_locale module
  • mozetl.taar.taar_similarity module
  • mozetl.taar.taar_utils module
  • Module contents
• mozetl.testpilot package
  • Submodules
  • mozetl.testpilot.containers module
  • mozetl.testpilot.pulse module
  • mozetl.testpilot.txp_mau_dau module
  • mozetl.testpilot.utils module
  • Module contents
• mozetl.topline package
  • Submodules
  • mozetl.topline.historical_backfill module
  • mozetl.topline.schema module
  • mozetl.topline.topline_dashboard module
  • mozetl.topline.topline_summary module
  • Module contents

Submodules

mozetl.cli module

mozetl.constants module

mozetl.main module

mozetl.main.etl_job(sc, sqlContext)

This is the function that will be executed on the cluster

mozetl.main.get_data(sc)

mozetl.main.ping_to_row(ping)

mozetl.main.transform_pings(pings)

Take a dataframe of main pings and summarize OS share

mozetl.schemas module

mozetl.utils module

mozetl.utils.delete_from_s3(bucket_name, keys_to_delete)

mozetl.utils.extract_submission_window_for_activity_day(frame, date, lag_days)

Extract rows with an activity_date of date minus lag_days and a submission_date between activity_date and date (inclusive).

:date ‘Y-m-d’ of the end of the target period :lag_days number of days after date in the target period :frame DataFrame homologous with main_summary

Note that the start_date will be lag-days days before date. In other words, if you pass in 2017-01-20 and set lag-days to 5, the aggregation will be processed for day 2017-01-15 (the resulting data will cover submission dates including the activity day itself plus 5 days of lag for a total of 6 days).

mozetl.utils.format_as_submission_date(date)

mozetl.utils.format_spark_path(bucket, prefix)

mozetl.utils.generate_filter_parameters(end_date, days_back)

mozetl.utils.parse_as_submission_date(date_string)

mozetl.utils.send_ses('me@example.com, 'greetings', "Hi!", 'you@example.com)

```

Raises a RuntimeError if the message did not send correctly.

mozetl.utils.stop_session_safely(spark_session)

Safely stops Spark session. This is no-op if running on Databricks - we shouldn’t stop session there since it’s managed by the platform, doing so fails the job.

mozetl.utils.upload_file_to_s3(client, filepath, bucket, key, ACL='bucket-owner-full-control')

mozetl.utils.write_csv(dataframe, path, header=True)

Write a dataframe to local disk.

Disclaimer: Do not write csv files larger than driver memory. This is ~15GB for ec2 c3.xlarge (due to caching overhead).

mozetl.utils.write_csv_to_s3(dataframe, bucket, key, header=True)

Module contents