The Spectroscopic Pipeline

Introduction and Resources

The Early Data Release (EDR) paper is the fundamental resource for understanding the processing and data products from the SDSS, describing the pipelines and contents of generated data products. Successive data release papers: DR1, DR2, DR3, DR4, DR5, DR6, DR7 and DR8, describe changes between data releases. The technical summary provides more general information on the SDSS-I survey. The SDSS Project Book, written for a funding proposal early in the survey, is the most exhaustive description of the survey but was last updated in 1997. The SDSS and SDSS-III web pages list the technical publications.

The algorithms page includes links to pages describing algorithms used by the spectroscopic data reduction pipelines. This page provides a summary of those steps and the associated output files.

The remainder of this page starts with a brief overview of spectroscopic data processing, followed by sections that describe the steps in data processing in detail. In addition to the more detailed description, each section provides references to papers that give additional details, and a table of the files associated with that step of the pipeline that can be found in the SAS. These tables include links to the file format documentation (the "data model") and templates which can be used to generate SAS URLs for those files. The templates are in "C printf" format, and can be used in C, bash, Python, and many other languages to automatically generate URLs.

Most of the catalog data (but not the spectra themselves) have been loaded into the Catalog Archive Server (CAS) database. Users are often better off obtaining SDSS data through a carefully constructed CAS query than they are downloading the data files from the SAS. Simple queries can be used to select just the objects and parameters of interest, while more complex queries can be used to do complex calculations on many objects, thereby avoiding the need to download the data on them at all.

Overview

Spectroscopic Observing
The spectrographs mounted on the primary 2.5m telescope collected spectra from each plate. There are two spectrographs, each of which collects data from 320 (SDSS) or 500 (BOSS) fibers. Each spectrograph has a dichroic that sends light to red and blue cameras, so the instrument produces a total of four images for each exposure.
Spectroscopic Data Reduction
The spectroscopic pipelines extract one dimensional spectra from the raw exposures produced by the spectrographs, calibrate them in wavelength and flux, combine the red and blue halves of the spectra, measure features in these spectra, measure redshifts from these features, and classify the objects as galaxies, stars, or quasars.

Notes

In the tables below, prepend 'http://data.sdss3.org/sas/dr10/' to all 'URL format' values to get the full URL.
URL suffixes are listed for the original SDSS-I/-II survey. Replace "sdss/" with "boss/" to get the equivalent BOSS survey files.
Data reduction could occur multiple times for both images and spectra. Each time we repeated a data reduction, we labeled the output from that reduction process with a distinct rerun number.
The target selection algorithms page describes how the pipeline performs target selection, including selection of Main Galaxy Sample, Luminous Red Galaxies (LRG), Quasars, Stars, ROSAT All-Sky Survey sources, Serendipity, and SEGUE targets. The target selection pipeline extracts calibrated catalogs of objects and corresponding field by field statistics from the operations database.
The tiling algorithms page describes the process by which the spectroscopic plates are designed and placed relative to each other.

Spectroscopic Observing

Plate Plugging (plug)

When the observatory is ready to observe a plate, the observatory staff plugs optical fibers into the holes drilled into the plates, and maps which fiber corresponds to which hole (and therefore which object) by shining light through each fiber. This data is incorporated into one of the HDUs of the spPlate file described below.

Raw Data Collection

Observers mount cartridges containing the drilled, plugged plates on the telescope, and collect a series of 15-minute exposures on each plate until it reached a threshold estimated signal to noise ratio and at least three exposures had been collected.

SAS files generated in spectroscopic data collection
File Type	in/out	Description	URL format	format parameters
sdReport	out	records exposures collected on a night	Not public
sdR	out	raw spectroscopic data frames	sdss/spectro/data/%d/sdR-%c%d-%08d.fit.gz	mjd, CCD (r or b), camera (1 or 2), exposure id

Spectroscopic Data Reduction

The idlspec2d software has two major pipeline steps:

spec2d: Extract and calibrate 1-dimensional spectra from 2-dimensional raw CCD data
spec1d: Measure object classifications and redshift from those 1D spectra.

Two-dimensional Pipeline (spec2d)

References: 2002AJ....123..485S, section 4.10.1

The spec2d pipeline reads science and calibration exposures from the spectrographs, reduces and calibrates the science exposures, extracts the one dimensional spectra from the two dimensional exposures, stacks multiple exposures into combined spectra, and produces corresponding masks and noise estimates.

SAS files used or generated by the spectro2d pipeline
File Type	in/out	Description	URL format	format parameters
spPlan2d	in	the spectro2d processing plan	sdss/spectro/redux/%d/%04d/spPlan2d-%04d-%d.par	rerun, plate, plate, mjd
spPlancomb	in	the processing plan for combining spectra	sdss/spectro/redux/%d/%04d/spPlancomd-%04d-%d.par	rerun, plate, plate, mjd
plPlugMapM	in	records which fiber corresponds to which hole in a plate (and therefore objects, and what coordinates on the sky)	Not public
sdReport	in	records exposures collected on a night	Not public
sdR	in	raw spectroscopic data frames	sdss/spectro/data/%d/sdR-%c%d-%08d.fit.gz	mjd, CCD (r or b), camera (1 or 2), exposure id
spCFrame	out	calibrated spectra for a single CCD and exposure	sdss/spectro/redux/%d/%04d/spCFrame-%c%d-%08d.par	rerun, plate, CCD (r or b), camera, exposure id
spPlate	out	the 640 (SDSS) or 1000 (BOSS) combined flux- and wavelength-calibrated spectra over all exposures (potentially spanning multiple nights) for a given mapped plate	sdss/spectro/redux/%s/%04d/spPlate-%04d-%d.fits	run2d, plate, plate, mjd

One-dimensional Pipeline (spec1d)

The spec1d pipeline produces the following files:

SAS files generated by the Princeton 1D pipeline
File Type	in/out	Description	URL format	format parameters
spZline	out	emission line fits	sdss/spectro/redux/%d/%04d/spZline-%04d-%d.fits	rerun, plate, plate, mjd
spZall	out	all spectroscopic classifications and redshifts	sdss/spectro/redux/%d/%04d/spZall-%04d-%d.fits	rerun, plate, plate, mjd
spZbest	out	spectroscopic classifications and redshifts	sdss/spectro/redux/%d/%04d/spZbest-%04d-%d.fits	rerun, plate, plate, mjd

Per-object spec files

NEW with DR9: The pipeline also provides a reformatting of the same spectral data into one file per PLATE-MJD-FIBER, including the coadded spectra from spPlate, the emission line fits from spZline, the redshifts and classifications from spZall and spZbest, and optionally the individual exposure spectra from spCFrame. These are useful when you need all of the information for a small subset of objects.

File Type	in/out	Description	URL format	format parameters
spec	out	All spectral information for a single PLATE-MJD-FIBER	sdss/spectro/redux/%d/spectra/%04d/spec-%04d-%05d-%04d.fits	run2d, plate, plate, mjd, fiber
speclite	out	All spectral information for a single PLATE-MJD-FIBER except the individual exposures	sdss/spectro/redux/%d/spectra/lite/%04d/spec-%04d-%05d-%04d.fits	run2d, plate, plate, mjd, fiber

Stellar Parameters Pipeline (sspp)

References: 2008AJ....136.2022L, 2008AJ....136.2050L, 2008AJ....136.2070A

The SEGUE stellar parameters pipeline produces a number of files, stored together:

SAS files used or generated by the SSPP pipeline
File Type	in/out	Description	URL format	format parameters
ssppOut	out	SSPP stellar parameters ([Fe/H], log g, etc.)	sdss/sspp/%d/%04d/output/param/ssppOut-%04d-%5d.fit	rerun, plate, plate, mjd
ssppOut_lineindex	out	SSPP line indices	sdss/sspp/%d/%04d/output/param/ssppOut-%04d-%5d.lineindex.fit	rerun, plate, plate, mjd