LSST Documentation Hub

LSST Project Management

LSST Systems Engineering

LSST Data Quality Assurance Plan

LSE-63

lse-63.lsst.io

  • Tony Tyson
  • DQA Team
  • Science Collaboration

LSST must supply trusted petascale data products. The mechanisms by which the LSST project achieve this unprecedented level of data quality will have spinoff to data-enabled science generally. This document specifies high-level requirements for a LSST Data Quality Assessment Framework, and defines the four levels of quality assessment (QA) tools. Because this process involves system-wide hardware and software, data QA must be defined at the System level. The scope of this document is limited to the description of the overall framework and the general requirements. It derives from the LSST Science Requirements Document [LPM-17]. A flow-down document will describe detailed implementation of the QA, including the algorithms. In most cases the monitoring strategy, the development path for these tools or the algorithms are known. Related documents are: LSST System Requirements [LSE-29], Optimal Deployment Parameters Document-11624, Observatory System Specifications [LSE-30], Configuration Management Plan [LPM-19], Project Quality Assurance Plan [LPM-55], Software Development Plan [LSE-16], Camera Quality implementation Plan [LCA-227], System Engineering Management Plan [LSE-17], and the Operations Plan [LPM-73].

Data Products Definition Document

LSE-163

lse-163.lsst.io

  • M. Jurić
  • T. Axelrod
  • A.C. Becker
  • J. Becla
  • E. Bellm
  • J.F. Bosch
  • D. Ciardi
  • A.J. Connolly
  • G.P. Dubois-Felsmann
  • F. Economou
  • M. Freemon
  • M. Gelman
  • R. Gill
  • M. Graham
  • Ž. Ivezić
  • T. Jenness
  • J. Kantor
  • K.S. Krughoff
  • K-T Lim
  • R.H. Lupton
  • F. Mueller
  • D. Nidever
  • W. O’Mullane
  • M. Patterson
  • D. Petravick
  • D. Shaw
  • C. Slater
  • M. Strauss
  • J. Swinbank
  • J.A. Tyson
  • M. Wood-Vasey
  • X. Wu

This document describes the data products and processing services to be delivered by the Large Synoptic Survey Telescope (LSST).

LSST will deliver three levels of data products and services. PROMPT (Level 1) data products will include images, difference images, catalogs of sources and objects detected in difference images, and catalogs of Solar System objects. Their primary purpose is to enable rapid follow-up of time-domain events. DATA RELEASE (Level 2) data products will include well calibrated single-epoch images, deep coadds, and catalogs of objects, sources, and forced sources, enabling static sky and precision time-domain science. The SCIENCE PLATFORM will allow for the creation of USER GENERATED (Level 3) data products and will enable science cases that greatly benefit from co-location of user processing and/or data within the LSST Archive Center. LSST will also devote 10% of observing time to programs with special cadence. Their data products will be created using the same software and hardware as Prompt (Level 1) and Data Release (Level 2) products. All data products will be made available using user-friendly databases and web services. Note, prior to 2018 Data products were referred to as Level 1, Level 2, and Level 3; this nomenclature was updated in 2018 to Prompt Products, Data Release Products and User Generated Products respectively [LPM-231]. In the abstract of this document both nomenclatures are used but throughout the remainder of this document only the new terminology is used. In other project and requirements documentation, the old terminology will likely persist.

LSST Data Management

LSST Data Management Acceptance Test Specification

LDM-639

ldm-639.lsst.io

  • L.P. Guy
  • W.M. Wood-Vasey
  • E. Bellm
  • J.F. Bosch
  • G.P. Dubois-Felsmann
  • M.L. Graham
  • R. Gruendl
  • K.S. Krughoff
  • K.-T. Lim
  • R.H. Lupton
  • C. Slater
  • G. Comoretto

This document describes the detailed acceptance test specification for the LSST Data Management System.

Data Access Use Cases

LDM-592

ldm-592.lsst.io

  • Tim Jenness
  • Jim Bosch
  • Michelle Gower
  • Simon Krughoff
  • Russell Owen
  • Pim Schellart
  • Brian van Klaveren
  • Dominique Boutigny

Use Cases written by the Butler Working Group covering data discovery, data storage, and data retrieval.

Science Platform Design

LDM-542

ldm-542.lsst.io

  • Gregory Dubois-Felsmann
  • Frossie Economou
  • Kian-Tat Lim
  • Fritz Mueller
  • Xiuqin Wu

This document describes the design of the LSST Science Platform, the primary user-facing interface of the LSST Data Management System.

LSST DM Raw Image Archiving Service Test Specification

LDM-538

ldm-538.lsst.io

  • Michelle Butler
  • Jim Parsons
  • Michelle Gower

This document describes the detailed test specification for the LSST DM Raw Image Archiving Service. This is a specific DM test, and will grow as more tests are needed for the entire environment. This includes two individual tests for the overall raw image creation and ingest into the permanent record of the survey.

Data Management  Test Plan

LDM-503

ldm-503.lsst.io

  • William O’Mullane
  • John Swinbank
  • Mario Juric
  • Frossie Economou

This is the Test Plan for Data Management. In it we define terms associated with testing and further test specifications for specific items.

Data Management Organization and Management

LDM-294

ldm-294.lsst.io

  • William O’Mullane
  • John Swinbank
  • Mario Juric
  • DMLT

This management plan covers the organization and management of the Data Management (DM) subsystem during the development, construction, and commissioning of LSST. It sets out DM goals and lays out the management organization roles and responsibilities to achieve them. It provides a high level overview of DM architecture, products and processes. It provides a structured starting point for understanding DM and pointers to further documentation.

Concept of Operations for the LSST Data Facility Services

LDM-230

ldm-230.lsst.io

  • D. Petravick
  • M. Butler
  • M. Gelman

This document describes the operational concepts for the emerging LSST Data Facility, which will operate the system that will be delivered by the LSST construction project. The services will be incrementally deployed and operated by the construction project as part of verification and validation activities within the construction project.

Data Management Middleware Design

LDM-152

ldm-152.lsst.io

  • K.-T. Lim
  • G. Dubois-Felsmann
  • M. Johnson
  • M. Juric
  • D. Petravick

The LSST middleware is designed to isolate scientific application pipelines and payloads, including the Alert Production, Data Release Production, Calibration Products Productions, and science user pipelines executed within the LSST Science Platform, from details of the underlying hardware and system software. It enables flexible reuse of the same code in multiple environments ranging from offline laptops to shared-memory multiprocessors to grid-accessed clusters, with a common I/O and logging model. It ensures that key scientific and deployment parameters controlling execution can be easily modified without changing code but also with full provenance to understand what environment and parameters were used to produce any dataset. It provides flexible, high-performance, low-overhead persistence and retrieval of datasets with data repositories and formats selected by external parameters rather than hard-coding. Middleware services enable efficient, managed replication of data over both wide area networks and local area networks.

Data Management Science Pipelines Design

LDM-151

ldm-151.lsst.io

  • J.D. Swinbank
  • T. Axelrod
  • A.C. Becker
  • J. Becla
  • E. Bellm
  • J.F. Bosch
  • H. Chiang
  • D.R. Ciardi
  • A.J. Connolly
  • G.P. Dubois-Felsmann
  • F. Economou
  • M. Fisher-Levine
  • M. Graham
  • Ž. Ivezić
  • M. Jurić
  • T. Jenness
  • R.L. Jones
  • J. Kantor
  • S. Krughoff
  • K-T. Lim
  • R.H. Lupton
  • F. Mueller
  • D. Petravick
  • P.A. Price
  • D.J. Reiss
  • D. Shaw
  • C. Slater
  • M. Wood-Vasey
  • X. Wu
  • P. Yoachim
  • _for the LSST Data Management_

The LSST Science Requirements Document (the LSST SRD) specifies a set of data product guidelines, designed to support science goals envisioned to be enabled by the LSST observing program. Following these guidlines, the details of these data products have been described in the LSST Data Products Definition Document (DPDD), and captured in a formal flow-down from the SRDvia the LSST System Requirements (LSR), Observatory System Specifications (OSS), to the Data Management System Requirements (DMSR). The LSST Data Management subsystem’s responsibilities include the design, implementation, deployment and execution of software pipelines necessary to generate these data products. This document describes the design of the scientific aspects of those pipelines.

Data Management System Design

LDM-148

ldm-148.lsst.io

  • K.-T. Lim
  • J. Bosch
  • G. Dubois-Felsmann
  • T. Jenness
  • J. Kantor
  • W. O’Mullane
  • D. Petravick
  • G. Comoretto
  • the DM Leadership Team

The LSST Data Management System (DMS) is a set of services employing a variety of software components running on computational and networking infrastructure that combine to deliver science data products to the observatory’s users and support observatory operations. This document describes the components, their service instances, and their deployment environments as well as the interfaces among them, the rest of the LSST system, and the outside world.

Data Management Database Design

LDM-135

ldm-135.lsst.io

  • Jacek Becla
  • Daniel Wang
  • Serge Monkewitz
  • K-T Lim
  • John Gates
  • Andy Salnikov
  • Andrew Hanushevsky
  • Douglas Smith
  • Bill Chickering
  • Michael Kelsey
  • Fritz Mueller

This document discusses the LSST database system architecture.

LSST Data Management Test Reports

LDM-503-1 (WISE Data Loaded in PDAC) Test Report

DMTR-52

dmtr-52.lsst.io

  • Gregory P. Dubois-Felsmann
  • Xiuqin Wu

This is the test report for LDM-503-1 (WISE Data Loaded in PDAC), an LSST DM level 2 milestone pertaining to the LSST Science Platform, with tests performed according to LSP-00, Portal and API Aspect Deployment of a Wide-Area Dataset.

LDM-503-2 (HSC Reprocessing) Test Report

DMTR-51

dmtr-51.lsst.io

  • Jim Bosch
  • Hsin-Fang Chiang
  • Michelle Gower
  • Mikolaj Kowalik
  • Timothy Morton
  • John D. Swinbank

This is the test report for LDM-503-2 (HSC Reprocessing), an LSST DM level 2 milestone pertaining to the LSST Level 2 System.

LSST Data Management Technical Notes

As-is HSC Reprocessing

DMTN-088

dmtn-088.lsst.io

  • Hsin-Fang Chiang
  • Margaret W. G. Johnson

This document summarizes the status and procedures of the HSC data reprocessing campaigns done by LDF as of early Fall 2018 cycle.

On accessing EFD data in the Science Platform

DMTN-082

dmtn-082.lsst.io

  • Simon Krughoff
  • Frossie Economou

The EFD is a powerful tool for correlating pipelines behavior with the state of the observatory. Since it contains the logging information published to the service abstraction layer (SAL) for all sensors and devices on the summit, it is a one stop shop for looking up the state of the observatory at a given moment. The expectation is that the science pipelines validation, science verification, and commissioning teams will all need, at one time or another, to get information like this for measuring the sensitivity of various parts of the system on observatory state: e.g. temperature, dome orientation, wind speed and direction, gravity vector. This leads to the further expectation that the various DM and commissioning teams will want to work with a version of the EFD that is accessed like a traditional relational database, possibly with linkages between individual exposures and specific pieces of observatory state. This implies the need for another version of the EFD (called the DM-EFD here) that has had transforms applied to the raw EFD that make it more immediately applicable to questions the DM team will want to ask.

Coaddition Artifact Rejection and CompareWarp

DMTN-080

dmtn-080.lsst.io

  • Yusra AlSayyad

LSST images will be contaminated with transient artifacts, such as optical ghosts, satellite trails, and cosmic rays, and with transient astronomical sources, such as asteroid ephemerides. We developed and tested an algorithm to find and reject these artifacts during coaddition, in order to produce clean coadds to be used for deep detection and preliminary object characterization. This algorithm, CompareWarpAssembleCoadd, uses the time-series of PSF-matched warped images to identify transient artifacts. It detects artifact candidates on the image differences between each PSF-matched warp and a static sky model. These artifact candidates include both true transient artifacts and difference-image false positives such as difficult-subtract-sources and variable sources such as stars and quasars. We use the feature that true transients appear at a given position in the difference images in only a small fraction (configurable) of visits, whereas variable sources and difficult-to-subtract sources appear in most difference images. In this report, we present a description of the method and an evaluation using Hyper SuprimeCam PDR1 data.

LSST Fall 2017 Crowded Fields Testing

DMTN-077

dmtn-077.lsst.io

  • K. Suberlak
  • C. Slater
  • Ž. Ivezić

We quantify the performance of the LSST pipeline for processing crowded fields, using images obtained from DECam and comparing to a specialized crowded field analysis performed as part of the DECAPS survey.

Considering single-visit LSST depth, in an example field of roughly the highest density seen in the LSST Wide-Fast-Deep area, DECAPS detects 200 000 sources per sq. deg. to a limiting depth of 23rd magnitude. At this source density the mean LSST-DECAPS completeness between 18th and 20th mag is 80%, and it drops to 50% at 21.5 mag.

For fields inside the Galactic plane cadence zone, source density rapidly increases. For instance, in a field in which DECAPS detects 500 000 sources per sq. deg. (5σ depth of 23.2), the mean completeness between 18th and 20th mag is 78%, and it drops to 50% at 20.2 mag.

In terms of photometric repeatability, above 19th mag LSST and DECAPS are in a systematics-dominated regime, and there is only a slow dependence on source density. At fainter magnitudes, the scatter between LSST and DECAPS is less than the uncertainty from photon noise for source densities up to 100 000 per sq. deg, but the scatter grows to twice the photon noise at densities of 300 000 per sq. deg. and above.

For repeat measurements of the same field with LSST, the astrometric scatter per source is at the level of 10-30 milliarcseconds for bright stars (g < 19), and is not strongly dependent on stellar crowdedness.

DM QA Status & Plans

DMTN-074

dmtn-074.lsst.io

  • Simon Krughoff
  • John Swinbank

This document will:

  • Describe the current status of “” tools, in the broadest sense, currently provided by Data Management;

  • Sketch out a set of common use cases and requirements for future QA tool and service development across the subsystem.

It is intended to serve as input to planning for currently being undertaken by the DM Leadership Team, the DM System Science Team, and the DM QA Strategy Working Group (LDM-622).

Lossy Compression WG Report

DMTN-068

dmtn-068.lsst.io

  • R. A. Gruendl

We report on the investigation into the use of lossy compression algorithms on LSST images that otherwise could not be stored for general retrieval and use by scientists. We find that modest quantization of images coupled with lossless compression algorithms can provide a factor of ∼6 savings in storage space while still providing images useful for followup scientific investigations. Given that this is only means that some products could be made quickly available to users and would free resources for community ues that would otherwise be necessary to re-compute these products, we recommend that LSST consider using a lossy compression to archive and serve image products where appropriate.

Memory Needs of Pipeline tasks

DMTN-066

dmtn-066.lsst.io

  • Samantha Thrush

Pipeline driver tasks such as singleFrameDriver, coaddDriver, and multiBandDriver have been tested to see what their memory usage is. This technote will detail how these memory tests were ran and what results were found.

Data Management and LSST Special Programs

DMTN-065

dmtn-065.lsst.io

  • M. L. Graham
  • M. Jurić
  • K.-T. Lim
  • E. Bellm

This document provides an in-depth description of the role of the LSST Project in preparing software and providing computational resources to process the data from Special Programs (deep drilling fields and/or mini-surveys). The plans and description in this document flow down from the requirements in LSE-61 regarding processing for Special Programs. The main target audience is the LSST Data Management (DM) team, but members of the community who are preparing white papers on science-driven observing strategies may also find this document useful. The potential diversity of data from Special Programs is summarized, including boundaries imposed by technical limitations of the LSST hardware. The capability of the planned Data Management system to processes this diversity of Special Programs data is the main focus of this document. Case studies are provided as examples of how the LSST software and/or user-generated pipelines may be combined to process the data from Special Programs.

Testing the LSST DM Stack on Deep Lens Survey Data

DMTN-063

dmtn-063.lsst.io

  • Imran Hasan
  • Perry Gee
  • Tony Tyson

We use version 13.0.9 of the LSST DM stack to reduce optical R band data taken with the KPNO 4-meter telescope for the Deep Lens Survey (DLS). Because this data set achieves an LSST like depth and has been studied and characterized exhaustively over the past decade, it provides an ideal setting to evaluate the performance of the LSST DM stack. In this report we examine registration, WCS fitting, and image co-addition of DLS data with the LSST DM stack. Aside from using a customized Instrument Signature Removal package, we are successful in using the DM stack to process imaging data of a 40 x 40 square arcminute subset of the DLS data, ultimately creating a coadd image. We find the astrometric solutions on individual chips have typical errors <15 miliarcseconds, demonstrating the effectiveness of the Jointcal package. Indeed, our findings in this regard on the DLS data are consistent with similar investigations on HSC and CFHT data.

A closer look at the astrometry data set shows it contains larger errors in Right Ascension than Declination. Further examination indicates these errors are likely due to a guider problem with the telescope, and not the result of proper motions of stars, or a problem with the DM stack itself.

Finally, we produce a coadd using the reduced data. Our coadd is approximately 40 square arcminutes-much larger than the coadds typically created with the stack. Creating a large image stretched our machines to their limits, and we believe a dearth of system resources lead to coadd creation Task not finishing. In spite of this, the coadd produced by the stack is of comparable quality to its counterpart produced by the DLS team in previous analysis in terms of depth, and ability to remove artifacts which do not correspond to true astrophysical objects. However issues were encountered with SafeClip.

Initial Installation of a DAQ Test Stand at NCSA

DMTN-052

dmtn-052.lsst.io

  • K-T Lim

Report on the delivery, installation, and initial use of an LSST Camera data acquisition (DAQ) test stand at NCSA, July 18-20, 2017. Includes notes from a discussion of future plans for DAQ work that was held following the installation.

LDF File Systems Baseline Overview

DMTN-051

dmtn-051.lsst.io

  • Don Petravick

This is a descriptive and explanatory document, not a normative document. This document explains the proposed baseline as presented in the DM replan in July, 2017, referred to just “baseline” in the prose that follows.

LSST DRP (Level 2) Catalog Photometric Redshifts

DMTN-049

dmtn-049.lsst.io

  • M. L. Graham

The purpose of this document is to begin to assemble the diversity of motivations driving the inclusion of photometric redshifts in the LSST Level 2 Object Catalog, and prepare to make a decision on what kind of photo-z products will be used. The roadmap for this process is described in Section [sec:intro]. We consider the photo-z use-cases in order to validate that the type of photo-z incorporated into the Level 2 DRP catalog, and the format in which it is stored, meets the needs of both DM and the community. We also compile potential evaluation methods for photo-z algorithms, and demonstrate these options by applying them to the photo-z results of two off-the-shelf photo-z estimators. The long-term plan is for this document to develop over time and eventually describe the decision-making process and the details of the selected algorithm(s) and products. PRELIMINARY RECOMMENDATIONS CAN BE FOUND IN SECTION [SEC:INTRO].

Tests with InfiniDB

DMTN-047

dmtn-047.lsst.io

  • Jim Tommaney
  • Jacek Becla
  • Kian-Tat Lim
  • Daniel Wang

Tests performed with InfiniDB in late 2010. Testing involved executing the most complex queries such as near neighbor on 1 billion row USNOB catalog

LSST DM Software Release Considerations

DMTN-044

dmtn-044.lsst.io

  • John D. Swinbank

This attempts to summarise the debate around, and suggest a path forward, for LSST software releases. Although some recommendations are made, they are intended to serve as the basis of discussion, rather than as a complete solution.

This material is based on discussions with several team members over a considerable period. Errors are to be expected; apologies are extended; corrections are welcome.

A Prototype AP Pipeline

DMTN-039

dmtn-039.lsst.io

  • Meredith Rawls

This note describes work done for DM-7295. It includes instructions for using the LSST Stack to process a set of raw DECam images from ISR through Difference Imaging.

Measurement of Blended Objects in LSST

DMTN-038

dmtn-038.lsst.io

  • Jim Bosch

Most LSST objects will overlap one or more of its neighbors enough to affect naive measurements of their properties. One of the major challenges in the deep processing pipeline will be measuring these sources in a way that corrects for and/or characterizes the effect of these blends.

Pessimistic Pattern Matching for LSST

DMTN-031

dmtn-031.lsst.io

  • Christopher B. Morrison

The current reference catalog matcher used by LSST for astrometry has be found to not be adequately robust and fails to find matches on serveral current datasets. This document describes a potential replacement algorithm, and compares its performance with the current implementation.

Science Pipelines Documentation Design

DMTN-030

dmtn-030.lsst.io

  • Jonathan Sick
  • Mandeep S. S. Gill
  • Simon Krughoff
  • John Swinbank

A design discussion and implementation plan for the pipelines.lsst.io documentation project, including information design and topic templates.

Pybind11 wrapping step-by-step

DMTN-026

dmtn-026.lsst.io

  • Fred Moolekamp
  • Pim Schellart

This document describes how to wrap an LSST package with pybind11. It does this by following the process of wrapping a single header file in afw step-by-step.

Data Management Project Management Guide

DMTN-020

dmtn-020.lsst.io

  • Jacek Becla
  • Frossie Economou
  • Margaret Gelman
  • Jeff Kantor
  • Simon Krughoff
  • Kevin Long
  • Fritz Mueller
  • William O’Mullane
  • John D. Swinbank
  • Xiuqin Wu

This is the DM guide for T/CAMs implementing the earned value system.

Dipoles in difference imaging from DCR

DMTN-019

dmtn-019.lsst.io

  • Ian Sullivan

I use the StarFast simulator to generate many simulated observations of a field at a range of airmasses from 1.0 to 2.0, and at several LSST bands. After differencing each image from the observation in each band closest to zenith, I generate a metric to characterize the number and size of dipoles in the residual.

Flavors of Coadds

DMTN-015

dmtn-015.lsst.io

  • Jim Bosch

A glossary of different kinds of coadded images, with brief descriptions of the algorithms behind them.

Testing Shear Bias Using Galsim Galaxy Simulations

DMTN-011

dmtn-011.lsst.io

  • Perry Gee

Writeup of work done in Summer 2015 Winter 2016 to test for shear bias in measurements done using CModel and ShapeletPsfApprox from the DM stack. Tests were done on galaxies of known shape in the style of great3sims using constant shear. Psfs applied were produced by PhoSim.

Current LSST stack WCS usage

DMTN-005

dmtn-005.lsst.io

  • John Parejko

This document describes our current World Coordinate System usage and implementation, in preparation for either a significant refactor or complete reimplementation using another public library.

LSST SQuaRE Technical Notes

DMS end-of-night report

SQR-026

sqr-026.lsst.io

  • Angelo Fausti
  • Frossie Economou
  • Simon Krughoff

The purpose of this technote is to describe the software components that will produce the Data Quality report for the Prompt Processing Pipeline.

Design of the notebook-based report system

SQR-023

sqr-023.lsst.io

  • Jonathan Sick

The notebook-based test report system provides a way for LSST to generate and publish data-driven reports with an automated system. This technote describes the technical design behind the notebook-based report system.

An Example JupyterLab Development Workflow

SQR-021

sqr-021.lsst.io

  • Simon Krughoff

The JupyterLab environment is becoming a powerful tool for all sorts of tasks that LSST team members commonly undertake. Data exploration and analysis are obvious cases where the distributed notebook environment is useful. This notebook will show how to use the notebook environment in conjunction with other aspects of JupyterLab: shell access and git authentication, to produce a meaningful development workflow.

Expressing LSST Project Metadata with JSON-LD

SQR-020

sqr-020.lsst.io

  • Jonathan Sick

This technote explores how JSON-LD (Linked Data) can be used to describe a variety of LSST project artifacts, including source code and documents. We provide specific examples using standard vocabularies (http://schema.org and CodeMeta) and explore whether custom terms are needed to support LSST use cases.

LSST Verification Framework API Demonstration

SQR-019

sqr-019.lsst.io

  • Jonathan Sick
  • Angelo Fausti

This technote describes, in a tutorial style, the lsst.verify API. This Verification Framework enables the LSST organization to define performance metrics, measure those metrics in Pipeline code, export metrics to a monitoring dashboard, and test performance against specifications.

LSST DocHub Design

SQR-013

sqr-013.lsst.io

  • Jonathan Sick

Research and design of a documentation metadata database and API for LSST based on JSON-LD metadata.

The SQuaSH metrics dashboard

SQR-009

sqr-009.lsst.io

  • Angelo Fausti

The SQuaSH dashboard is used to monitor the KPM metrics computed by the LSST verification framework, here we describe its design and implementation details.

LSST Simulations Technical Notes