LSST Documentation Hub

The Documentation Hub is a service in preview by LSST SQuaRE. Updated on July 30, 2019.

LSST Project Management

Proposed Policy for Independent Data Access Centers

LPM-251

lpm-251.lsst.io

  • William O’Mullane
  • Beth Willman
  • Melissa Graham
  • Leanne Guy
  • Robert Blum

This document describes the proposed policies for groups that are independent from the LSST Project and Operations (i.e. LSST Data Facility) and would like to stand up an independent Data Access Center (iDAC; existing data centers that could serve LSST data products are considered iDACs for purposes of this document). Some iDACs may want to serve only a subset of the LSST data products: this document proposes three portion sizes, from full releases to a “light” catalog without posteriors. Guidelines and requirements for iDACs in terms of data storage, computational resources, dedicated personnel, and user authentication are described, as well as a preliminary assessment of the cost impacts. Some institutions, even those inside the US and Chile, may serve LSST data products locally to their research community. Requirements and responsibilities for such institutional bulk data transfers are also described here. THE PURPOSE OF THIS DRAFT DOCUMENT IS TO SERVE AS A PRELIMINARY RESOURCE FOR PARTNER INSTITUTIONS WISHING TO ASSESS THE FEASIBILITY OF HOSTING AN IDAC.

LSST Systems Engineering

LSST Data Quality Assurance Plan

LSE-63

lse-63.lsst.io

  • Tony Tyson
  • DQA Team
  • Science Collaboration

LSST must supply trusted petascale data products. The mechanisms by which the LSST project achieve this unprecedented level of data quality will have spinoff to data-enabled science generally. This document specifies high-level requirements for a LSST Data Quality Assessment Framework, and defines the four levels of quality assessment (QA) tools. Because this process involves system-wide hardware and software, data QA must be defined at the System level. The scope of this document is limited to the description of the overall framework and the general requirements. It derives from the LSST Science Requirements Document [LPM-17]. A flow-down document will describe detailed implementation of the QA, including the algorithms. In most cases the monitoring strategy, the development path for these tools or the algorithms are known. Related documents are: LSST System Requirements [LSE-29], Optimal Deployment Parameters Document-11624, Observatory System Specifications [LSE-30], Configuration Management Plan [LPM-19], Project Quality Assurance Plan [LPM-55], Software Development Plan [LSE-16], Camera Quality implementation Plan [LCA-227], System Engineering Management Plan [LSE-17], and the Operations Plan [LPM-73].

Defining the Transformation Between Camera Engineering Coordinates and Camera Data Visualization Coordinates

LSE-349

lse-349.lsst.io

  • K. Simon Krughoff

There are various reasons for wanting to produce a representation of the camera focal plane in a physical coordinate system with the sensors oriented as they reside in the assembled instrument. The first reason is in engineering diagrams used for fabrication and assembly of the camera. The second is in visualizations for quality assesment, data analysis, or science verification purposes. For reasons motivated herein, these two corrdinates systems cannot be the same for LSST. This document presents the coordinate transform to be applied in translating between the two coordinate systems: 1) the camera engineering diagram coordinate system and 2) the camera data visualization coordinate system.

LSST Science Platform Vision Document

LSE-319

lse-319.lsst.io

  • M. Jurić
  • D. Ciardi
  • G.P. Dubois-Felsmann
  • L.P. Guy

This document defines and describes the “LSST Science Platform,” a set of integrated web applications and services deployed at the LSST Data Access Centers (DACs) through which the scientific community will access, visualize, subset, and perform next-to-the-data analysis of the data collected by the Large Synoptic Survey Telescope (LSST). These services can be broken down to three different “Aspects”: a web PORTAL, designed to provide essential data access and visualization services through a simple-to-use website, a NOTEBOOK environment, that will provide a Jupyter Notebook-like interface, based on JupyterLab, enabling next-to-the-data analysis, and an extensive set of WEB APIS that the users will be able to use to remotely examine the LSST data set using tools they’re already familiar with. This document lays out the high-level vision for the aforementioned Aspects and some associated backend services. It is intentionally brief, and meant to generally guide the flow-down of requirements and development of product specifications, prioritization, and plans for the Agile development of the relevant elements of the DM system.

Data Products Definition Document

LSE-163

lse-163.lsst.io

  • M. Jurić
  • T. Axelrod
  • A.C. Becker
  • J. Becla
  • E. Bellm
  • J.F. Bosch
  • D. Ciardi
  • A.J. Connolly
  • G.P. Dubois-Felsmann
  • F. Economou
  • M. Freemon
  • M. Gelman
  • R. Gill
  • M. Graham
  • L.P. Guy
  • Ž. Ivezić
  • T. Jenness
  • J. Kantor
  • K.S. Krughoff
  • K-T Lim
  • R.H. Lupton
  • F. Mueller
  • D. Nidever
  • W. O’Mullane
  • M. Patterson
  • D. Petravick
  • D. Shaw
  • C. Slater
  • M. Strauss
  • J. Swinbank
  • J.A. Tyson
  • M. Wood-Vasey
  • X. Wu

This document describes the data products and processing services to be delivered by the Large Synoptic Survey Telescope (LSST).

LSST will deliver three levels of data products and services. PROMPT (Level 1) data products will include images, difference images, catalogs of sources and objects detected in difference images, and catalogs of Solar System objects. Their primary purpose is to enable rapid follow-up of time-domain events. DATA RELEASE (Level 2) data products will include well calibrated single-epoch images, deep coadds, and catalogs of objects, sources, and forced sources, enabling static sky and precision time-domain science. The SCIENCE PLATFORM will allow for the creation of USER GENERATED (Level 3) data products and will enable science cases that greatly benefit from co-location of user processing and/or data within the LSST Archive Center. LSST will also devote 10% of observing time to programs with special cadence. Their data products will be created using the same software and hardware as Prompt (Level 1) and Data Release (Level 2) products. All data products will be made available using user-friendly databases and web services. Note, prior to 2018 Data products were referred to as Level 1, Level 2, and Level 3; this nomenclature was updated in 2018 to Prompt Products, Data Release Products and User Generated Products respectively [LPM-231]. In the abstract of this document both nomenclatures are used but throughout the remainder of this document only the new terminology is used. In other project and requirements documentation, the old terminology will likely persist.

LSST Project Science Team Technical Notes

Discussion of Object vs. Source table queries and data distribution

PSTN-003

pstn-003.lsst.io

  • William O’Mullane
  • Fritz Mueller

AMCL asked about distribution of various catalog products. They also suggested potential usage levels for these products. This brings questions such as: Qserv is built to allow SQL queries on sources, but will most queries be on object? This note discuses data access, interaction, and distribution.

On the Choice of LSST Flux Units

PSTN-001

pstn-001.lsst.io

  • Željko Ivezić
  • the LSST Project Science Team

A linear measure of flux is preferred for LSST catalogs. This document provides technical details about this preference in support of the LSST Project Science Team’s decision to adopt nano-jansky (1 nJy =10−35  Wm−2Hz−1) as the standard LSST flux unit. Difficulties associated with homogenizing broad-band flux measurements to a uniform system are also discussed in some detail.

LSST Data Management

Call for Letters of Intent for Community Alert Brokers

LDM-682

ldm-682.lsst.io

  • Eric Bellm
  • Robert Blum
  • Melissa Graham
  • Leanne Guy
  • Željko Ivezić
  • William O’Mullane
  • John Swinbank _for the LSST Project_

A major product of the nightly processing of LSST images is a world-public stream of alerts from transient, variable, and moving sources. Science users may access these alerts through third-party community brokers, which will receive the LSST alerts, add scientific value, and redistribute them to the scientific community.

This document is a call for Letters of Intent (LOIs) to propose a community broker, as described in “Plans and Policies for LSST Alert Distribution” [LDM-612].

LSST Software Release Management

LDM-672

ldm-672.lsst.io

  • G. Comoretto
  • L. P. Guy
  • W. O’Mullane
  • K.-T. Lim
  • M. Butler
  • M. Gelman
  • J.D. Swinbank

This document outlines the policies and high level management approach for software product releases.

LSST Science Platform Final Design Review

LDM-652

ldm-652.lsst.io

  • Leanne P. Guy

This document provides the charge to the review committee for the LSST Science Platform (LSP) Final Design Review (FDR). The review will be a formal and internal review of the planned LSP capabilities in the LSST operations era as per the guidelines outlined in LDM-294 and LSE-159.

LSST Data Management Acceptance Test Specification

LDM-639

ldm-639.lsst.io

  • L.P. Guy
  • W.M. Wood-Vasey
  • E. Bellm
  • J.F. Bosch
  • J.L. Carlin
  • G.P. Dubois-Felsmann
  • M.L. Graham
  • R. Gruendl
  • K.S. Krughoff
  • K.-T. Lim
  • R.H. Lupton
  • C. Slater
  • G. Comoretto

This document describes the detailed acceptance test specification for the LSST Data Management System.

Plans and Policies for LSST Alert Distribution

LDM-612

ldm-612.lsst.io

  • Eric Bellm
  • Robert Blum
  • Melissa Graham
  • Leanne Guy
  • Željko Ivezić
  • William O’Mullane
  • Maria Patterson
  • John Swinbank
  • Beth Willman _for the LSST Project_

A major product of the nightly processing of LSST images is a world-public stream of alerts from transient, variable, and moving sources. Science users will access these alerts through community brokers or through a simple filtering service provided by LSST. This document provides a guide to the plans and policies for the alert distribution system to aid science users, broker developers, funding agencies, and LSST Project personnel. It describes the components of the alert distribution system and the data rights required to access specific scientific products. It provides guidelines for organizations developing community brokers and describes the planned capabilities of the LSST simple alert filtering service and Science Platform to aid science users in planning for LSST science.

Data Access Use Cases

LDM-592

ldm-592.lsst.io

  • Tim Jenness
  • Jim Bosch
  • Michelle Gower
  • Simon Krughoff
  • Russell Owen
  • Pim Schellart
  • Brian van Klaveren
  • Dominique Boutigny

Use Cases written by the Butler Working Group covering data discovery, data storage, and data retrieval.

Science Platform Design

LDM-542

ldm-542.lsst.io

  • Gregory Dubois-Felsmann
  • Frossie Economou
  • Kian-Tat Lim
  • Fritz Mueller
  • Stephen R. Pietrowicz
  • Xiuqin Wu

This document describes the design of the LSST Science Platform, the primary user-facing interface of the LSST Data Management System.

LSST Science Platform Test Specification

LDM-540

ldm-540.lsst.io

  • G. P. Dubois-Felsmann
  • L.P. Guy
  • J. Carlin
  • K.S. Krughoff
  • C. Slater
  • M. Wood-Vasey

This document describes the detailed test specification for the LSST Science Platform. It is a work in progress; the current version provides Test Cases covering all requirements on the LSST Science Platform, however only ≈ 10% are currently fully specified. This document will be updated as work continues on completing Test Cases.

LSST DM Raw Image Archiving Service Test Specification

LDM-538

ldm-538.lsst.io

  • Michelle Butler
  • Jim Parsons
  • Michelle Gower

This document describes the detailed test specification for the LSST DM Raw Image Archiving Service. This is a specific DM test, and will grow as more tests are needed for the entire environment. This includes two individual tests for the overall raw image creation and ingest into the permanent record of the survey.

Data Management  Test Plan

LDM-503

ldm-503.lsst.io

  • William O’Mullane
  • John Swinbank
  • Mario Juric
  • Frossie Economou
  • Gabriele Comoretto

This is the Test Plan for Data Management. In it we define terms associated with testing and further test specifications for specific items.

Data Management Organization and Management

LDM-294

ldm-294.lsst.io

  • William O’Mullane
  • John Swinbank
  • Mario Juric
  • Leanne Guy
  • DMLT

This management plan covers the organization and management of the Data Management (DM) subsystem during the development, construction, and commissioning of LSST. It sets out DM goals and lays out the management organization roles and responsibilities to achieve them. It provides a high level overview of DM architecture, products and processes. It provides a structured starting point for understanding DM and pointers to further documentation.

Concept of Operations for the LSST Data Facility Services

LDM-230

ldm-230.lsst.io

  • D. Petravick
  • M. Butler
  • M. Gelman

This document describes the operational concepts for the emerging LSST Data Facility, which will operate the system that will be delivered by the LSST construction project. The services will be incrementally deployed and operated by the construction project as part of verification and validation activities within the construction project.

Data Management Middleware Design

LDM-152

ldm-152.lsst.io

  • K.-T. Lim
  • G. Dubois-Felsmann
  • M. Johnson
  • M. Juric
  • D. Petravick

The LSST middleware is designed to isolate scientific application pipelines and payloads, including the Alert Production, Data Release Production, Calibration Products Productions, and science user pipelines executed within the LSST Science Platform, from details of the underlying hardware and system software. It enables flexible reuse of the same code in multiple environments ranging from offline laptops to shared-memory multiprocessors to grid-accessed clusters, with a common I/O and logging model. It ensures that key scientific and deployment parameters controlling execution can be easily modified without changing code but also with full provenance to understand what environment and parameters were used to produce any dataset. It provides flexible, high-performance, low-overhead persistence and retrieval of datasets with data repositories and formats selected by external parameters rather than hard-coding.

Data Management Science Pipelines Design

LDM-151

ldm-151.lsst.io

  • J.D. Swinbank
  • T. Axelrod
  • A.C. Becker
  • J. Becla
  • E. Bellm
  • J.F. Bosch
  • H. Chiang
  • D.R. Ciardi
  • A.J. Connolly
  • G.P. Dubois-Felsmann
  • F. Economou
  • M. Fisher-Levine
  • M. Graham
  • Ž. Ivezić
  • M. Jurić
  • T. Jenness
  • R.L. Jones
  • J. Kantor
  • S. Krughoff
  • K-T. Lim
  • R.H. Lupton
  • F. Mueller
  • D. Petravick
  • P.A. Price
  • D.J. Reiss
  • D. Shaw
  • C. Slater
  • M. Wood-Vasey
  • X. Wu
  • P. Yoachim
  • _for the LSST Data Management_

The LSST Science Requirements Document (the LSST SRD) specifies a set of data product guidelines, designed to support science goals envisioned to be enabled by the LSST observing program. Following these guidlines, the details of these data products have been described in the LSST Data Products Definition Document (DPDD), and captured in a formal flow-down from the SRDvia the LSST System Requirements (LSR), Observatory System Specifications (OSS), to the Data Management System Requirements (DMSR). The LSST Data Management subsystem’s responsibilities include the design, implementation, deployment and execution of software pipelines necessary to generate these data products. This document describes the design of the scientific aspects of those pipelines.

Data Management System Design

LDM-148

ldm-148.lsst.io

  • K.-T. Lim
  • J. Bosch
  • G. Dubois-Felsmann
  • T. Jenness
  • J. Kantor
  • W. O’Mullane
  • D. Petravick
  • G. Comoretto
  • the DM Leadership Team

The LSST Data Management System (DMS) is a set of services employing a variety of software components running on computational and networking infrastructure that combine to deliver science data products to the observatory’s users and support observatory operations. This document describes the components, their service instances, and their deployment environments as well as the interfaces among them, the rest of the LSST system, and the outside world.

Data Management Database Design

LDM-135

ldm-135.lsst.io

  • Jacek Becla
  • Daniel Wang
  • Serge Monkewitz
  • K-T Lim
  • John Gates
  • Andy Salnikov
  • Andrew Hanushevsky
  • Douglas Smith
  • Bill Chickering
  • Michael Kelsey
  • Fritz Mueller

This document discusses the LSST database system architecture.

LSST Data Management Test Reports

LDM-503-1 (WISE Data Loaded in PDAC) Test Report

DMTR-52

dmtr-52.lsst.io

  • Gregory P. Dubois-Felsmann
  • Xiuqin Wu

This is the test report for LDM-503-1 (WISE Data Loaded in PDAC), an LSST DM level 2 milestone pertaining to the LSST Science Platform, with tests performed according to LSP-00, Portal and API Aspect Deployment of a Wide-Area Dataset.

LDM-503-2 (HSC Reprocessing) Test Report

DMTR-51

dmtr-51.lsst.io

  • Jim Bosch
  • Hsin-Fang Chiang
  • Michelle Gower
  • Mikolaj Kowalik
  • Timothy Morton
  • John D. Swinbank

This is the test report for LDM-503-2 (HSC Reprocessing), an LSST DM level 2 milestone pertaining to the LSST Level 2 System.

LSST Data Management Technical Notes

Image Display WG

DMTN-126

dmtn-126.lsst.io

  • Yusra AlSayyad (chair)
  • Scott Daniel
  • Gregory Dubois-Felsmann
  • Simon Krughoff
  • Lauren MacArthur
  • John Swinbank

This document describes the findings of the LSST DM Image Display Working Group.

Google Cloud Engagement Results

DMTN-125

dmtn-125.lsst.io

  • Kian-Tat Lim

From September 2018 to March 2019, LSST Data Management (DM) and Google Cloud conducted a proof of concept engagement to investigate the utility, applicability, and performance of Google Cloud services with regard to various DM needs. This document describes what was accomplished in each area, including measurements obtained, and presents some conclusions about how the engagement went overall.

Automated Quality Control Systems

DMTN-124

dmtn-124.lsst.io

  • Frossie Economou
  • Simon Krughoff
  • Angelo Fausti
  • Joshua Hoblitt
  • Jonatha Sick
  • Adam Thornton

A description of services that support quality control activities in LSST Data Management.

Batch Production Services Design

DMTN-123

dmtn-123.lsst.io

  • Kian-Tat Lim

The LSST DM Batch Production Services are designed to allow large-scale workflows, including the Data Release and Calibration Product Productions, to execute in well-managed fashion, potentially in multiple environments. They ensure that provenance is captured and recorded to understand what environment and parameters were used to produce any dataset.

Data Backbone Design

DMTN-122

dmtn-122.lsst.io

  • Michelle Gower
  • Kian-Tat Lim

The Data Backbone (DBB) is a key component that provides for data storage, transport, and replication, allowing data products to move between enclaves. This service provides policy-based replication of files (including images and flat files to be loaded into databases as well as other raw and intermediate files) and databases (including metadata about files as well as other miscellaneous databases but not including the large Data Release catalogs) across multiple physical locations, including the Base, Commissioning Cluster, NCSA, and Data Access Centers. It manages caches of files at each endpoint as well as persistence to long-term archival storage (e.g. tape). It provides a registration mechanism for new datasets and database entries and a retrieval mechanism compatible with the Data Butler.

Impact of variable seeing on DCR coadd generation

DMTN-121

dmtn-121.lsst.io

  • Ian Sullivan

A summary of the tests performed in the Spring of 2019 to quantify the impact of variable seeing on the quality of the Differential Chromatic Refraction (DCR) correction algorithm. The current state of the DCR algorithm naively assumes that all of the input exposures used to construct the template have identical PSFs, and there has been some concern that the quality of the model will degrade when those input exposures have variable seeing. In this note I compare the performance of templates created with DcrAssembleCoaddTask with our current standard, CompareWarpAssembleCoaddTask, and estimate the range of observing conditions where the current algorithm is sufficient.

Improving Extensibility in afw.image.Exposure and Replacing afw.table.io

DMTN-120

dmtn-120.lsst.io

  • Krzysztof Findeisen
  • Jim Bosch

A design study for improving the persistence of Stack classes, particularly lsst.afw.image.Exposure and its components. This note describes the design decisions behind the lsst.afw.typehandling.GenericMap class template and explores options for an off-the-shelf persistence framework to replace lsst.afw.table.io.

Review of Timeseries Features

DMTN-118

dmtn-118.lsst.io

  • Eric Bellm

The DPDD allocates space for pre-computed timeseries features, and a sample set is baselined in LDM-151. However, other features have been developed. This technote reviews the relevant literature, grouping related features where possible, and discusses potential concerns.

LSST + Amazon Web Services Proof of Concept

DMTN-114

dmtn-114.lsst.io

  • Kian-Tat Lim
  • Leanne Guy
  • Hsin-Fang Chiang

This document is a proposal for an engagement to verify that a cloud deployment of the LSST Data Management (DM) Data Release Production (DRP) is feasible, measure its performance, determine its final discounted cost, and investigate more-native cloud options for handling system components that may be costly to develop or maintain.

DM Usage in Observatory Operations

DMTN-111

dmtn-111.lsst.io

  • Kian-Tat Lim

This document describes how DM services and software are used to support Observatory Operations, includingg critical Commandable SAL Components (CSCs) at the Summit and the Commissioning Team at the Summit and Base.

Conda Environment Proposal for Science Pipelines

DMTN-110

dmtn-110.lsst.io

  • Gabriele Comoretto

This proposal is an improved way to deal with Conda environments when building the Science Pipelines software. The benefits of implementing it can be seen in both the development and release processes. The effort required for its implementation is limited since it reuses all tooling already in place.

Security of prompt imaging on LSST

DMTN-108

dmtn-108.lsst.io

  • William O’Mullane

This is a brief summary of on the transmission of images for alert processing . Specifically it discusses how secure the transfer is and if something more could be done in that area.

Options for Alert Production in LSST Operations Year 1

DMTN-107

dmtn-107.lsst.io

  • M. L. Graham
  • E. C. Bellm
  • C. T. Slater
  • L. P. Guy
  • the DM System Science Team

This document reviews five options for alert production in LSST Operations Year 1 (LOY1), taking into account any implications on LSST formal requirements including up-scopes, down-scopes or explicit violations. The Data Management System Science Team’s preferred option for maximizing LSST science is to generate template images from as much of the commissioning data as possible (∼2000 $\rm deg^2$) and use them to run Difference Image Analysis and alert production during LOY1. A proposal to increase the sky area covered by commissioning-data templates in at least a single filter to ≳10,000 $\rm deg^2$ via a “filler” scheduler program is also presented. As a potential moderate up-scope, this study presents an option to build interim templates on a ∼monthly basis during LOY1, which could increase the accessible sky area by ∼1000-2000 $\rm deg^2$ per month, which can be reconsidered closer to the start of Operations.

LSP Capabilities for AuxTel, Commissioning, and Early Operations

DMTN-105

dmtn-105.lsst.io

  • Gregory Dubois-Felsmann

Narrative summary of the LSST Science Platform capabilities that will be available during the remainder of Construction (FY19+) and, specifically, for initial AuxTel commissioning, main telescope commissioning with ComCam and with the full camera, and for any early data releases prior to the start of formal operations.

LSST Alerts: Key Numbers

DMTN-102

dmtn-102.lsst.io

  • M. L. Graham
  • E. Bellm
  • L. Guy
  • C. T. Slater
  • G. Dubois-Felsmann
  • the Data Management System Science Team

A quantitative review of the key numbers associated with the LSST Alert Stream.

As-is HSC Reprocessing

DMTN-088

dmtn-088.lsst.io

  • Hsin-Fang Chiang
  • Margaret W. G. Johnson

This document summarizes the status and procedures of the HSC data reprocessing campaigns done by LDF as of early Fall 2018 cycle.

Proposed Modifications to Solar System Processing and Data Products

DMTN-087

dmtn-087.lsst.io

  • Mario Juric
  • Siegfried Eggl
  • Joachim Moeyens
  • Lynne Jones

A high-level presentation of proposed changes to Solar System processing and data products baseline, to bring LSST Solar System processing closer to common asteroid-survey workflows, and help the on-schedule delivery of a scientifically useful set of Solar System data products.

QA Strategy Working Group Report

DMTN-085

dmtn-085.lsst.io

  • Bellm
  • E.C.
  • Chiang
  • H.-F.
  • Fausti
  • A.
  • Krughoff
  • K.S.
  • MacArthur
  • L.A.
  • Morton
  • T.D.
  • Swinbank
  • J.D. (chair)
  • Roby
  • T.

This document describes the work undertaken by the QA Strategy Working Group and presents its conclusions as a series of recommendations to DM Project Management.

On accessing EFD data in the Science Platform

DMTN-082

dmtn-082.lsst.io

  • Simon Krughoff
  • Frossie Economou

The EFD is a powerful tool for correlating pipelines behavior with the state of the observatory. Since it contains the logging information published to the service abstraction layer (SAL) for all sensors and devices on the summit, it is a one stop shop for looking up the state of the observatory at a given moment. The expectation is that the science pipelines validation, science verification, and commissioning teams will all need, at one time or another, to get information like this for measuring the sensitivity of various parts of the system on observatory state: e.g. temperature, dome orientation, wind speed and direction, gravity vector. This leads to the further expectation that the various DM and commissioning teams will want to work with a version of the EFD that is accessed like a traditional relational database, possibly with linkages between individual exposures and specific pieces of observatory state. This implies the need for another version of the EFD (called the DM-EFD here) that has had transforms applied to the raw EFD that make it more immediately applicable to questions the DM team will want to ask.

Coaddition Artifact Rejection and CompareWarp

DMTN-080

dmtn-080.lsst.io

  • Yusra AlSayyad

LSST images will be contaminated with transient artifacts, such as optical ghosts, satellite trails, and cosmic rays, and with transient astronomical sources, such as asteroid ephemerides. We developed and tested an algorithm to find and reject these artifacts during coaddition, in order to produce clean coadds to be used for deep detection and preliminary object characterization. This algorithm, CompareWarpAssembleCoadd, uses the time-series of PSF-matched warped images to identify transient artifacts. It detects artifact candidates on the image differences between each PSF-matched warp and a static sky model. These artifact candidates include both true transient artifacts and difference-image false positives such as difficult-subtract-sources and variable sources such as stars and quasars. We use the feature that true transients appear at a given position in the difference images in only a small fraction (configurable) of visits, whereas variable sources and difficult-to-subtract sources appear in most difference images. In this report, we present a description of the method and an evaluation using Hyper SuprimeCam PDR1 data.

LSST Fall 2017 Crowded Fields Testing

DMTN-077

dmtn-077.lsst.io

  • K. Suberlak
  • C. Slater
  • Ž. Ivezić

We quantify the performance of the LSST pipeline for processing crowded fields, using images obtained from DECam and comparing to a specialized crowded field analysis performed as part of the DECAPS survey.

Considering single-visit LSST depth, in an example field of roughly the highest density seen in the LSST Wide-Fast-Deep area, DECAPS detects 200 000 sources per sq. deg. to a limiting depth of 23rd magnitude. At this source density the mean LSST-DECAPS completeness between 18th and 20th mag is 80%, and it drops to 50% at 21.5 mag.

For fields inside the Galactic plane cadence zone, source density rapidly increases. For instance, in a field in which DECAPS detects 500 000 sources per sq. deg. (5σ depth of 23.2), the mean completeness between 18th and 20th mag is 78%, and it drops to 50% at 20.2 mag.

In terms of photometric repeatability, above 19th mag LSST and DECAPS are in a systematics-dominated regime, and there is only a slow dependence on source density. At fainter magnitudes, the scatter between LSST and DECAPS is less than the uncertainty from photon noise for source densities up to 100 000 per sq. deg, but the scatter grows to twice the photon noise at densities of 300 000 per sq. deg. and above.

For repeat measurements of the same field with LSST, the astrometric scatter per source is at the level of 10-30 milliarcseconds for bright stars (g < 19), and is not strongly dependent on stellar crowdedness.

DM QA Status & Plans

DMTN-074

dmtn-074.lsst.io

  • Simon Krughoff
  • John Swinbank

This document will:

  • Describe the current status of “” tools, in the broadest sense, currently provided by Data Management;

  • Sketch out a set of common use cases and requirements for future QA tool and service development across the subsystem.

It is intended to serve as input to planning for currently being undertaken by the DM Leadership Team, the DM System Science Team, and the DM QA Strategy Working Group (LDM-622).

Lossy Compression WG Report

DMTN-068

dmtn-068.lsst.io

  • R. A. Gruendl

We report on the investigation into the use of lossy compression algorithms on LSST images that otherwise could not be stored for general retrieval and use by scientists. We find that modest quantization of images coupled with lossless compression algorithms can provide a factor of ∼6 savings in storage space while still providing images useful for followup scientific investigations. Given that this is only means that some products could be made quickly available to users and would free resources for community ues that would otherwise be necessary to re-compute these products, we recommend that LSST consider using a lossy compression to archive and serve image products where appropriate.

Memory Needs of Pipeline tasks

DMTN-066

dmtn-066.lsst.io

  • Samantha Thrush

Pipeline driver tasks such as singleFrameDriver, coaddDriver, and multiBandDriver have been tested to see what their memory usage is. This technote will detail how these memory tests were ran and what results were found.

Data Management and LSST Special Programs

DMTN-065

dmtn-065.lsst.io

  • M. L. Graham
  • M. Jurić
  • K.-T. Lim
  • E. Bellm

This document provides an in-depth description of the role of the LSST Project in preparing software and providing computational resources to process the data from Special Programs (deep drilling fields and/or mini-surveys). The plans and description in this document flow down from the requirements in LSE-61 regarding processing for Special Programs. The main target audience is the LSST Data Management (DM) team, but members of the community who are preparing white papers on science-driven observing strategies may also find this document useful. The potential diversity of data from Special Programs is summarized, including boundaries imposed by technical limitations of the LSST hardware. The capability of the planned Data Management system to processes this diversity of Special Programs data is the main focus of this document. Case studies are provided as examples of how the LSST software and/or user-generated pipelines may be combined to process the data from Special Programs.

Testing the LSST DM Stack on Deep Lens Survey Data

DMTN-063

dmtn-063.lsst.io

  • Imran Hasan
  • Perry Gee
  • Tony Tyson

We use version 13.0.9 of the LSST DM stack to reduce optical R band data taken with the KPNO 4-meter telescope for the Deep Lens Survey (DLS). Because this data set achieves an LSST like depth and has been studied and characterized exhaustively over the past decade, it provides an ideal setting to evaluate the performance of the LSST DM stack. In this report we examine registration, WCS fitting, and image co-addition of DLS data with the LSST DM stack. Aside from using a customized Instrument Signature Removal package, we are successful in using the DM stack to process imaging data of a 40 x 40 square arcminute subset of the DLS data, ultimately creating a coadd image. We find the astrometric solutions on individual chips have typical errors <15 miliarcseconds, demonstrating the effectiveness of the Jointcal package. Indeed, our findings in this regard on the DLS data are consistent with similar investigations on HSC and CFHT data.

A closer look at the astrometry data set shows it contains larger errors in Right Ascension than Declination. Further examination indicates these errors are likely due to a guider problem with the telescope, and not the result of proper motions of stars, or a problem with the DM stack itself.

Finally, we produce a coadd using the reduced data. Our coadd is approximately 40 square arcminutes-much larger than the coadds typically created with the stack. Creating a large image stretched our machines to their limits, and we believe a dearth of system resources lead to coadd creation Task not finishing. In spite of this, the coadd produced by the stack is of comparable quality to its counterpart produced by the DLS team in previous analysis in terms of depth, and ability to remove artifacts which do not correspond to true astrophysical objects. However issues were encountered with SafeClip.

Initial Installation of a DAQ Test Stand at NCSA

DMTN-052

dmtn-052.lsst.io

  • K-T Lim

Report on the delivery, installation, and initial use of an LSST Camera data acquisition (DAQ) test stand at NCSA, July 18-20, 2017. Includes notes from a discussion of future plans for DAQ work that was held following the installation.

LDF File Systems Baseline Overview

DMTN-051

dmtn-051.lsst.io

  • Don Petravick

This is a descriptive and explanatory document, not a normative document. This document explains the proposed baseline as presented in the DM replan in July, 2017, referred to just “baseline” in the prose that follows.

LSST Data Release Processing: Object Catalog Photometric Redshifts

DMTN-049

dmtn-049.lsst.io

  • M. L. Graham

The purpose of this document is to begin to assemble the diversity of motivations driving the inclusion of photometric redshifts in the LSST Data Release Object Catalog, and prepare to make a decision on what kind of photo-z products will be used. The roadmap for this process is described in Section [sec:intro]. We consider the photo-z use-cases in order to validate that the type of photo-z incorporated into the DRP catalog, and the format in which it is stored, meets the needs of both DM and the community. We also compile potential evaluation methods for photo-z algorithms, and demonstrate these options by applying them to the photo-z results of two off-the-shelf photo-z estimators. The long-term plan is for this document to develop over time and eventually describe the decision-making process and the details of the selected algorithm(s) and products. PRELIMINARY RECOMMENDATIONS CAN BE FOUND IN SECTION [SEC:INTRO].

Tests with InfiniDB

DMTN-047

dmtn-047.lsst.io

  • Jim Tommaney
  • Jacek Becla
  • Kian-Tat Lim
  • Daniel Wang

Tests performed with InfiniDB in late 2010. Testing involved executing the most complex queries such as near neighbor on 1 billion row USNOB catalog

LSST DM Software Release Considerations

DMTN-044

dmtn-044.lsst.io

  • John D. Swinbank

This attempts to summarise the debate around, and suggest a path forward, for LSST software releases. Although some recommendations are made, they are intended to serve as the basis of discussion, rather than as a complete solution.

This material is based on discussions with several team members over a considerable period. Errors are to be expected; apologies are extended; corrections are welcome.

A Prototype AP Pipeline

DMTN-039

dmtn-039.lsst.io

  • Meredith Rawls

This note describes work done for DM-7295. It includes instructions for using the LSST Stack to process a set of raw DECam images from ISR through Difference Imaging.

Measurement of Blended Objects in LSST

DMTN-038

dmtn-038.lsst.io

  • Jim Bosch

Most LSST objects will overlap one or more of its neighbors enough to affect naive measurements of their properties. One of the major challenges in the deep processing pipeline will be measuring these sources in a way that corrects for and/or characterizes the effect of these blends.

Pessimistic Pattern Matching for LSST

DMTN-031

dmtn-031.lsst.io

  • Christopher B. Morrison

The current reference catalog matcher used by LSST for astrometry has be found to not be adequately robust and fails to find matches on serveral current datasets. This document describes a potential replacement algorithm, and compares its performance with the current implementation.

Science Pipelines Documentation Design

DMTN-030

dmtn-030.lsst.io

  • Jonathan Sick
  • Mandeep S. S. Gill
  • Simon Krughoff
  • John Swinbank

A design discussion and implementation plan for the pipelines.lsst.io documentation project, including information design and topic templates.

Pybind11 wrapping step-by-step

DMTN-026

dmtn-026.lsst.io

  • Fred Moolekamp
  • Pim Schellart

This document describes how to wrap an LSST package with pybind11. It does this by following the process of wrapping a single header file in afw step-by-step.

Data Management Project Management Guide

DMTN-020

dmtn-020.lsst.io

  • Jacek Becla
  • Frossie Economou
  • Margaret Gelman
  • Jeff Kantor
  • Simon Krughoff
  • Kevin Long
  • Fritz Mueller
  • William O’Mullane
  • John D. Swinbank
  • Xiuqin Wu

This is the DM guide for T/CAMs implementing the earned value system.

Dipoles in difference imaging from DCR

DMTN-019

dmtn-019.lsst.io

  • Ian Sullivan

I use the StarFast simulator to generate many simulated observations of a field at a range of airmasses from 1.0 to 2.0, and at several LSST bands. After differencing each image from the observation in each band closest to zenith, I generate a metric to characterize the number and size of dipoles in the residual.

Flavors of Coadds

DMTN-015

dmtn-015.lsst.io

  • Jim Bosch

A glossary of different kinds of coadded images, with brief descriptions of the algorithms behind them.

Testing Shear Bias Using Galsim Galaxy Simulations

DMTN-011

dmtn-011.lsst.io

  • Perry Gee

Writeup of work done in Summer 2015 Winter 2016 to test for shear bias in measurements done using CModel and ShapeletPsfApprox from the DM stack. Tests were done on galaxies of known shape in the style of great3sims using constant shear. Psfs applied were produced by PhoSim.

Current LSST stack WCS usage

DMTN-005

dmtn-005.lsst.io

  • John Parejko

This document describes our current World Coordinate System usage and implementation, in preparation for either a significant refactor or complete reimplementation using another public library.

LSST SQuaRE Technical Notes

QA Strategy Working Group recommendations for SQuaSH

SQR-033

sqr-033.lsst.io

  • Angelo Fausti

Recently, in DMTN-085, the QA Strategy Working Group (QAWG) made specific recommendations to improve the SQuaSH metrics dashboard. This technote presents a technical overview of the current implementation, and a plan to implement what is missing.

Rendering and testing examples and tutorials in LSST documentation

SQR-032

sqr-032.lsst.io

  • Jonathan Sick

Examples and tutorials are important elements of successful documentation. They show the reader how something can be accomplished, which is often more powerful and effective than a mere description. Such examples are only useful if they are correct, though. Automated testing infrastructure is the best way to ensure that examples are correct when they are committed, and remain correct as a project evolves. Recently, in DMTN-085, the QA Strategy Working Group (QAWG) issued specific recommendations to improve how examples are managed and tested in LSST's documentation. This technote analyzes these recommendations and translates them into technical requirements. Subsequently, this technote also provides an overview of how example code management and testing has been implemented.

DM-EFD prototype implementation

SQR-029

sqr-029.lsst.io

  • Frossie Economou
  • Simon Krughoff
  • Jonathan Sick
  • Angelo Fausti
  • Adam Thornton
  • Joshua Hoblitt

Report on the DM-EFD prototype implementation based on Kafka and InfluxDB

Design of the notebook-based report system

SQR-023

sqr-023.lsst.io

  • Jonathan Sick

The notebook-based test report system provides a way for LSST to generate and publish data-driven reports with an automated system. This technote describes the technical design behind the notebook-based report system.

An Example JupyterLab Development Workflow

SQR-021

sqr-021.lsst.io

  • Simon Krughoff

The JupyterLab environment is becoming a powerful tool for all sorts of tasks that LSST team members commonly undertake. Data exploration and analysis are obvious cases where the distributed notebook environment is useful. This notebook will show how to use the notebook environment in conjunction with other aspects of JupyterLab: shell access and git authentication, to produce a meaningful development workflow.

Expressing LSST Project Metadata with JSON-LD

SQR-020

sqr-020.lsst.io

  • Jonathan Sick

This technote explores how JSON-LD (Linked Data) can be used to describe a variety of LSST project artifacts, including source code and documents. We provide specific examples using standard vocabularies (http://schema.org and CodeMeta) and explore whether custom terms are needed to support LSST use cases.

LSST Verification Framework API Demonstration

SQR-019

sqr-019.lsst.io

  • Jonathan Sick
  • Angelo Fausti

This technote describes, in a tutorial style, the lsst.verify API. This Verification Framework enables the LSST organization to define performance metrics, measure those metrics in Pipeline code, export metrics to a monitoring dashboard, and test performance against specifications.

LSST DocHub Design

SQR-013

sqr-013.lsst.io

  • Jonathan Sick

Research and design of a documentation metadata database and API for LSST based on JSON-LD metadata.

The SQuaSH metrics dashboard

SQR-009

sqr-009.lsst.io

  • Angelo Fausti

The SQuaSH metrics dashboard is used to track metrics collected from the Science Pipelines. This technote describes its current design and vision.

LSST Simulations Technical Notes