![[EXT]](images/offsite.png)
The Imaging Pipeline
Introduction and Resources
The Early Data Release (EDR) paper is the fundamental resource for understanding the processing and data products from the SDSS, describing the pipelines and contents of generated data products. Successive data release papers: DR1, DR2, DR3, DR4, DR5, DR6 and DR7, describe changes between data releases. The technical summary provides more general information on the survey. The SDSS Project Book, written for a funding proposal early in the survey, is the most exhaustive description of the survey but was last updated in 1997.
The algorithms page includes links to pages describing algorithms used by the data reduction pipelines, and the SAS datamodel has a table of the most commonly useful files in the SAS. Truly gory details can be found on Robert Lupton's page of photo documents, including the draft photometric reduction paper as well as a treasure trove of photometric data reduction wisdom.
The remainder of this page starts with a brief overview of imaging data processing, followed by sections that describe the steps in data processing in detail. In addition to the more detailed description, each detail section privedes references to papers that give additional details, and a table of the files associated with that step of the pipeline that can be found in the SAS. These tables include links to descriptions of the formats of those files and templates that can be used to generate SAS URLs for those files. The templates are in "C printf" format, and can be used in C, bash, Python, and many other languages to automatically generate URLs.
Most of the catalog data (but not the images themselves) have been loaded into the Catalog Archive Server (CAS) database. Users are often better off obtaining SDSS-III data through a carefully constructed CAS query than they are downloading the data files from the SAS. Simple queries can be used to select just the objects and parameters of interest, while more complex queries can be used to do complex calculations on many objects, thereby avoiding the need to download the data on them at all.
Overview
- Imaging Observing
We used three instruments when collecting imaging data: the imaging camera mounted on the primary 2.5m SDSS telescope, which collected the imaging data themselves; the 0.5m photometric telescope, which collected images of photometric standard stars and reference fields; and a 10 micron all sky scanner (and later camera), used to detect clouds. The photometric telescope is no longer used in imaging data reduction and its data is not described here. - Imaging Data Reduction
The SDSS data processing factory used a collection of pipelines to process and calibrate the data from the imaging camera, ultimately producing a variety of data products including images with instrumental signatures removed, a photometric solution for the night, and a catalog of objects found in the data. - Resolution of Multiple Detections
The resolve step declares one observation on an object the primary observation, and others secondary, thereby avoiding unintentional duplication of objects. This step ultimately produces an astrometrically and photometrically calibrated catalog of objects found in the data from the imaging camera. - Recalibration
Improved photometric calibrations became possible at the end of the survey; we used a separate pipeline to refine the photometric calibrations. - Final Photometric Catalog
The results of the resolve and calibration stages are combined with the uncalibrated object catalogs to produce final catalogs.
Notes
- In the tables below, add 'http://data.sdss3.org/sas/dr8/' to all 'URL format' values to get the full URL.
- Data reduction could occur multiple times for both images and spectra. Each time we repeated a data reduction, we labeled the output from that reduction process with a distinct rerun number.
- In the case of imaging rerun numbers, different decades in the rerun number designate significant differences in the data reduction software. For example, rerun 1 and rerun 2 of a given run would have been processed by identical (or at least very similar) versions of the data reduction software, while rerun 40 would have been reduced with a significantly different version of the pipeline. However, the only relevant reruns for DR8 are 157 and 301, with the bulk of the data in rerun 301.
Imaging Observing
Imager
The imager collected survey data from the imager in drift scan (also called TDI) mode. We use the term "run" to designate a single scan along a great circle. The data acquisition system system divided the data from each CCD into frames, stored as FITS files; the observatory sent data to be processed as a collection of FITS files, each corresponding to an arc along the great circle 1361 rows (539 arcseconds) long from a single CCD. In addition to the images themselves, the data acquisition system also generated a variety of metadata and other engineering files, which included pixel statistics and a catalog of bright stars for each field.
| File Type | in/out | Description | URL format | format parameters |
|---|---|---|---|---|
| idReport | out | records runs collected on a night | groups/boss/photo/redux/%d/%d/logs/idReport-%05d.par | rerun, run, mjd |
| idR | out | raw imaging frames | groups/boss/photo/data/%d/fields/%d/idR-%06d-%s%d-%04d.fit.Z | run, camcol, run, filter, camcol, field |
| idGang | out | assorted data gathered by the data acquisition system | groups/boss/photo/data/%d/gangs/idGang-%06d-%s%d-%04d.fit | run, run, r(ow) or c(ol), row number or column number, frame |
| idFrameLog | out | camera position information for each frame | groups/boss/photo/redux/%d/%d/logs/idFrameLog-%06d-%d.par | rerun, run, run, crate |
| opCamera | out | imaging camera geometry | groups/boss/photo/redux/%d/%d/logs/opCamera-%05d.par | rerun, run, mjd (of generation) |
| opConfig | out | CCD readout configurations | groups/boss/photo/redux/%d/%d/logs/opConfig-%05d.par | rerun, run, mjd (of generation) |
| opECalib | out | CCD electronic calibrations | groups/boss/photo/redux/%d/%d/logs/opECalib-%05d.par | rerun, run, mjd (of generation) |
Infrared All-Sky Camera
Clouds can be clearly seen at night in a 10 micron all-sky camera. We used two such cameras at different times in the survey to monitor the sky for cloud (and therefore non-photometric conditions).
| File Type | in/out | Description | URL format | format parameters |
|---|---|---|---|---|
| cR | out | A single image from the infrared camera | apo_staging/ircam/%05d/cR%06d.fit | mjd, UTC time (HHMMSS) |
| irsc | out | A log of statistics on ircam frames | apo_staging/ircam/%05d/irsc.log | mjd |
Other metadata
The observatory produces an assortment of engineering and other metadata.
| File Type | in/out | Description | URL format | format parameters |
|---|---|---|---|---|
| night log | out | A prose account of the night | apo_staging/astrolog/%05d/manualLog-full.txt | mjd |
| idWeather | out | A log of weather data | groups/boss/photo/redux/%d/%d/logs/idWeather-%05d.par | rerun, run, mjd |
Imaging Data Reduction
Serial Stamp Collecting Pipeline (SSC)
The SSC repackages some of the data produced by the data acquisition system and produces postage stamps of bright reference stars.
| File Type | in/out | Description | URL format | format parameters |
|---|---|---|---|---|
| scPlan | in | the SSC processing plan | groups/boss/photo/redux/%d/%d/ssc/scPlan.par | run, rerun |
| idReport | in | records runs collected on a night | groups/boss/photo/redux/%d/%d/logs/idReport-%05d.par | rerun, run, mjd |
| opCamera | in | imaging camera geometry | groups/boss/photo/redux/%d/%d/logs/opCamera-%05d.par | rerun, run, mjd (of generation) |
| opConfig | in | CCD readout configurations | groups/boss/photo/redux/%d/%d/logs/opConfig-%05d.par | rerun, run, mjd (of generation) |
| opECalib | in | CCD electronic calibrations | groups/boss/photo/redux/%d/%d/logs/opECalib-%05d.par | rerun, run, mjd (of generation) |
| opBC | in | CCD bad pixel file | groups/boss/photo/redux/%d/%d/log/opBC-%d.par | rerun, run, mjdbp (see fpPlan) |
| idR | in | raw imaging frames | groups/boss/photo/data/%d/fields/%d/idR-%06d-%s%d-%04d.fit.Z | run, camcol, run, filter, camcol, field |
| idGang | in | assorted data gathered by the data acquisition system | groups/boss/photo/data/%d/gangs/idGang-%06d-%s%d-%04d.fit | run, run, r(ow) or c(ol), row number or column number, frame |
| koCat | in | the catalog of known bright stars | groups/boss/photo/redux/%d/%d/ssc/koCat-%06d.fit | run, rerun, run |
Postage Stamp Pipeline (PSP)
References: 2002AJ....123..485S, section 4.3
The PSP measures the bias and global sky level, and fits a model for the point spread function for each field.
| File Type | in/out | Description | URL format | format parameters |
|---|---|---|---|---|
| psParam | in | PSP tunable parameters | groups/boss/photo/redux/%d/%d/photo/psParam.par | run, rerun |
| psPlan | in | the PSP processing plan | groups/boss/photo/redux/%d/%d/photo/psPlan.par | run, rerun |
| opCamera | in | imaging camera geometry | groups/boss/photo/redux/%d/%d/logs/opCamera-%05d.par | rerun, run, mjd (of generation) |
| opConfig | in | CCD readout configurations | groups/boss/photo/redux/%d/%d/logs/opConfig-%05d.par | rerun, run, mjd (of generation) |
| opECalib | in | CCD electronic calibrations | groups/boss/photo/redux/%d/%d/logs/opECalib-%05d.par | rerun, run, mjd (of generation) |
| opBC | in | CCD bad pixel file | groups/boss/photo/redux/%d/%d/log/opBC-%d.par | rerun, run, mjdbp (see fpPlan) |
| idB | in | bias vector | groups/boss/photo/redux/%d/%d/photo/calib/idB-%06d-%c%d.fit | rerun, run, calibration run, filter, camcol |
| psField | out | the initial photometric calibration and point spread function fit by field (see the extracting PSF images page) | groups/boss/photo/redux/%d/%d/objcs/%d/psField-%06d-%d-%04d.fit | rerun, run, camcol, run, camcol, field |
Astrometric Pipeline (astrom)
References: 2003AJ....125.1559P
The astrometric pipeline calculates the astrometric solution. The astrometry and astrometry QA (UPDATE LINK) web pages provide more information on the algorithms used and the proper interpretation of the output.
| File Type | in/out | Description | URL format | format parameters |
|---|---|---|---|---|
| asParam | in | astrom tunable parameters | groups/boss/photo/redux/%d/%d/astrom/asParam.par | rerun, run |
| asPlan | in | the astrom processing plan | groups/boss/photo/redux/%d/%d/astrom/asPlan.par | rerun, run |
| opCamera | in | imaging camera geometry | groups/boss/photo/redux/%d/%d/logs/opCamera-%05d.par | rerun, run, mjd (of generation) |
| opConfig | in | CCD readout configurations | groups/boss/photo/redux/%d/%d/logs/opConfig-%05d.par | rerun, run, mjd (of generation) |
| idReport | in | records runs collected on a night | groups/boss/photo/redux/%d/%d/logs/idReport-%05d.par | rerun, run, mjd |
| idWeather | in | A log of weather data | groups/boss/photo/redux/%d/%d/logs/idWeather-%05d.par | rerun, run, mjd |
| asTrans | out | tranformation coefficients from row, column to great circle coordinates | groups/boss/photo/redux/%d/%d/astrom/asTrans-%06d.fit | rerun, run, run |
Frames Pipeline (frames)
References: 2002AJ....123..485S, section 4.4
The frames pipeline applies flat field and bias corrections to each frame, and interpolates values for pixels in bad columns and bleed trails and those corrupted by cosmic rays. In previous data releases, this pipeline would produce a "corrected frame" or fpC file. These files are no longer produced. Instead, a frame file is produced by a different method, though it contains substantially the same information. The frames pipeline described here also detects bright objects and estimates the varying sky background, detects fainter objects, and measures a variety of properties for each object. Several algorithms pages describe details of the processing performed by frames.
- Measures of flux and magnitudes discribes the variety of methods frames uses to measure the magnitude of each object, including radial profiles, fiber, model, cModel, Petrosian and PSF magnitudes and when to use each.
- Classification and Morphology describes other measurements of objects, including star/galaxy separation, model fits, surface brightnesses, ellipticities, adaptive moments and isophotal quantities.
- IMPORTANT! Photometric processing flags describes the variety of flags frames sets for each object, essential for properly interpreting the catalog.
- Flat field creation and quality. See the Ubercal paper for a more detailed discussion of flat-fielding.
- Sky measurement
- Deblending describes how the frames pipeline decides if an initial single detection is in fact a blend of multiple overlapping objects, and, if so, how it separates (or "deblends") them.
- Creation of imaging masks
| File Type | in/out | Description | URL format | format parameters |
|---|---|---|---|---|
| fpParam | in | frames tunable parameters | groups/boss/photo/redux/%d/%d/photo/fpParam.par | rerun, run |
| fpPlan | in | frames processing plan | groups/boss/photo/redux/%d/%d/photo/fpPlan.par | rerun, run |
| idReport | in | records runs collected on a night | groups/boss/photo/redux/%d/%d/logs/idReport-%05d.par | rerun, run, mjd |
| koCat | in | the catalog of known bright stars | groups/boss/photo/redux/%d/%d/ssc/koCat-%06d.fit | run, rerun, run |
| opConfig | in | CCD readout configurations | groups/boss/photo/redux/%d/%d/logs/opConfig-%05d.par | rerun, run, mjd (of generation) |
| opECalib | in | CCD electronic calibrations | groups/boss/photo/redux/%d/%d/logs/opECalib-%05d.par | rerun, run, mjd (of generation) |
| opBC | in | CCD bad pixel file | groups/boss/photo/redux/%d/%d/log/opBC-%d.par | rerun, run, mjdbp (see fpPlan) |
| idB | in | bias vector | groups/boss/photo/redux/%d/%d/photo/calib/idB-%06d-%c%d.fit | rerun, run, calibration run, filter, camcol |
| psField | in | the initial photometric calibration and point spread function fit by field (see the extracting PSF images page) | groups/boss/photo/redux/%d/%d/objcs/%d/psField-%06d-%d-%04d.fit | rerun, run, camcol, run, camcol, field |
| asTrans | in | tranformation coefficients from row, column to great circle coordinates | groups/boss/photo/redux/%d/%d/astrom/asTrans-%06d.fit | rerun, run, run |
| fpFieldStat | out | field statistics | groups/boss/photo/redux/%d/%d/objcs/%d/fpFieldStat-%06d-%d-%04d.fit | rerun, run, camcol, run, camcol, field |
| fpObjc | out | the (uncalibrated) object catalog | groups/boss/photo/redux/%d/%d/objcs/%d/fpObjc-%06d-%d-%04d.fit | rerun, run, camcol, run, camcol, field |
| fpBIN | out | a 4x4 binned version of the corrected image after masking of objects and subtraction of sky, an image of the estimate of the sky, an image of the uncertainty in the sky estimate, and a table of bright star wings | groups/boss/photo/redux/%d/%d/objcs/%d/fpBIN-%06d-%c%d-%04d.fit.gz | rerun, run, camcol, run, filter, camcol, field |
| fpM | out | the frame masks (read using readAtlasImages) | groups/boss/photo/redux/%d/%d/objcs/%d/fpM-%06d-%c%d-%04d.fit.gz | rerun, run, camcol, run, filter, camcol, field |
| fpAtlas | out | the atlas images for all objects detected (read using readAtlasImages) | groups/boss/photo/redux/%d/%d/objcs/%d/fpAtlas-%06d-%d-%04d.fit | rerun, run, camcol, run, camcol, field |
Resolve Multiple Detections
A given area on the sky may be observed by multiple runs. The final calibrated catalogs contain a field declaring whether the the specific observation of the object is "primary," and the canonical measurement of the object, or "secondary," a duplicate. These catalogs are generated twice, once based on the single run in isolation by exportChunk, and again in the resolve step. The catalogs produced by exportChunk declare an object primary or secondary based on position alone, introducing the possibility of inconsistencies due to errors in position; it is possible, for example, that two separate observation of an object be declared primary if the position measured for the object is slightly different in each detection.
The catalog generated by the resolve step, on the other hand, uses an operational database to search for other detections, and guarantees that one and only one detection be declared primary.
See the EDR paper, 2002AJ....123..485S, section 4.7 for details on how objects are resolved.
The catalogs produced by exportChunk can be found in the run directory of each run:
| File Type | in/out | Description | URL format | format parameters |
|---|---|---|---|---|
| reObjGlobal | out | Global resolve status for every object in a field | groups/boss/resolve/%s/%d/%d/resolve/%d/reObjGlobal-%06d-%d-%04d.fit | resolve rerun, rerun, run, camcol, run, camcol, field |
| thingList | out | The full list of catalog entries for the resolved survey | groups/boss/resolve/%s/thingList.fits | resolve rerun |
| thingIndex | out | The full list of unique primary objects, gives position of an object in the thingList | groups/boss/resolve/%s/thingIndex.fits | resolve rerun |
| window_flist | out | The full list of fields used to determine the window function | groups/boss/resolve/%s/window_flist.fits | resolve rerun |
| window_unified | out | The full list of polygons determining the primary area of the window function | groups/boss/resolve/%s/window_unified.fits | resolve rerun |
| window_blist | out | The full list of balkans determining the primary area of the window function | groups/boss/resolve/%s/window_blist.fits | resolve rerun |
| window_bcaps | out | The full list of caps determining the primary area of the window function | groups/boss/resolve/%s/window_bcaps.fits | resolve rerun |
| window_bindx | out | Matches balkans to fields in the window function, sorted by balkans | groups/boss/resolve/%s/window_bindx.fits | resolve rerun |
| window_findx | out | Matches balkans to fields in the window function, sorted by sorted by fields | groups/boss/resolve/%s/window_findx.fits | resolve rerun |
Photometric Recalibration (Ubercal)
References: 2008ApJ...674.1217P
The "ubercal" recalibration recalibrates the survey by simultaneously solving for the calibration parameters and relative stellar fluxes using overlapping observations. The algorithm used decouples the problem of relative calibrations from that of absolute calibrations; the absolute calibration is reduced to determining a few numbers for the entire survey. See the photometric calibration algorithms page.
| File Type | in/out | Description | URL format | format parameters |
|---|---|---|---|---|
| calibPhotomGlobal | out | Ubercal results (nanomaggies/count) and associated flat fields | groups/boss/calib/%s/%d/%d/nfcalib/calibPhotomGlobal-%06d-%d.fits | calibration rerun, rerun, run, run, camcol |
| calib | out | Uberal quality assurance plots | groups/boss/calib/%s/plots/calib-%06d-%c-%s.png | calibration rerun, run, filter, type (flat, hist, run) |
Photometric Catalogs
The final photometric catalogs contain the raw data that will be loaded into the CAS database.
| File Type | in/out | Description | URL format | format parameters |
|---|---|---|---|---|
| photoObj | out | Full, calibrated outputs of the imager photometric pipeline | groups/boss/photoObj/%d/%d/%d/photoObj-%06d-%d-%04d.fits | rerun, run, camcol, run, camcol, field |
| photoField | out | Summary outputs of the properties of every field | groups/boss/photoObj/%d/%d/photoField-%06d-%d.fits | rerun, run, run, camcol |
| photoRunAll | out | Summary information of the properties of every photometric run | groups/boss/photoObj/photoRunAll-%s.fits | release (dr8) |