Classifying Objects by Medium-Band Observations
- a spectrophotometric 17-filter survey -

The COMBO-17 data reduction cookbook


0. Contents

  1. Data reduction Overview
  2. Preparation of WFI reduction environment
  3. Getting started, getting help
  4. MPIAPHOT catalog and 'frame list' handling
  5. Pipeline, part I:
    1. Input data: INPUT/WFI
    2. Restore filenames: RESTORE/WFI
    3. Alternative Restore after 2001: @@ WFI:restore_2001
    4. Debiasing only: DEBIAS/WFI
    5. Mosaicing only: MOSAIC/WFI
    6. Creation of flatfield: FLAT/WFI
    7. Correction by flatfield only: CORRECT/WFI
    8. Construction of superflat: FLAT/WFI, SMOOTH/WFI
    9. Application of superflat: CORRECT/WFI, CORRECT/ADD
    10. Subtraction of radially symmetric background: RINGEX/WFI
    11. Detection of objects in frame: FIOBJECT/WFI
    12. Determination of positional offsets between frames: FIMOVE/WFI
    13. Stacking of frames for cosmic removal and summed image: STACK/MOSAIC
  6. Pipeline, part II:
    1. Extract objects from summed frame: SEXTRACT/WFI
    2. Prepare frames for projection of object positions: PREPARE/PROJECT
    3. Project extracted objects onto individual frames: PROJECT/WFI
    4. Perform photometry on individual frames: EVALUATE/CADIS
    5. Select brightest objects in each frame for normalisation: CADIS/SETNORM
    6. Set up master reference table for photometry: CADIS/REFER
    7. Combine photometry of individual frames into one: CADIS/UNITE
    8. Identify standard stars in each frame: CADIS/STD
    9. Convert count rates into physical fluxes using standard: CADIS/INTEG
    10. Combine all unite tables into one master flux table for all filters: CADIS/FLUX
  7. Naming conventions


I. Data reduction overview:

We recommend using the MIDAS image processing package with our context environments (WFI, MPIAPHOT, CADIS) for the reduction of dithered WFI survey images. New users should first familiarize themselves with MIDAS and the data structures of commands, images, tables and especially descriptors and catalogs. Before starting any WFI reduction on a new user account, the WFI environment needs to be properly set up by the user (see Section II). MPIAPHOT provides advanced catalog handling which is more flexible and easier to use than the standard MIDAS loop EXEC/CAT (please read Section IV).

The following data reduction from raw frames to final object catalogs can be considered a two-step process. Part 1 of the pipeline deals with processing raw images to final images, following these basic steps (see Section V for details):
  • Loading raw frames from FITS tapes (DAT or DLT) or from disk and restoring original filenames
  • Creating a proper flatfield
  • Processing science frames (debiasing, mosaic composition, flat-fielding)
  • Optional: Applying a superflat correction or removing a fringe pattern or scattered light
  • Measuring coordinate transformations between frames
  • Cosmic Correction and stacking of corrected frames to yield sumframes
Part 2 of the pipeline deals with deriving a table of objects from the sumframes and measuring their photometric properties. The principal steps are (see Section VI for details):
  • Object search on the high-sensitivity, high-resolution R-band sum frame (using SExtractor)
  • Projection of the object table into coordinate systems of individual images
  • Photometry at the object location in images
  • Combination of all measurements to final object catalog with positions and photometry
  • Calibration check and multi-color classification of object catalog


II. Preparation of WFI reduction environment:

We assume a user account where MIDAS has already been used successfully. For setting up the WFI data reduction environment a few steps have to be taken:
  • The following files have to copied into your $MID_WORK-directory:

    cp  /photo/user/exe/mpiaphot.ctx  $MID_WORK/
    cp  /photo/user/exe/cadis.ctx  $MID_WORK/
    cp  /photo/user/exe/WFI/DETECTOR.PAR  $MID_WORK/
    cp  /photo/user/exe/WFI/INSTRUMENT.PAR  $MID_WORK/ 
    cp  /photo/user/exe/WFI/wfi.ctx  $MID_WORK/ 
  • The following environment variables have to be set in your ~/.cshrc file:

    setenv WFI /photo/user/exe/WFI
    setenv PM /photo/user/exe
    setenv CADIS /photo/user/exe/CADIS
    setenv PHOTDATA /photo/user/tbl/PHOTDATA
    setenv PHOTSTDS /photo/user/tbl/PHOTSTDS/CADIS 
    The last two directories contain the filter efficiency curves and the standard star spectra, respectively. If you use your own, you have to change the path definitions accordingly. If you do not intend to run part II of the pipeline, you can skip the last two definitions altogether.


III. Getting started, getting help:

You can start working in the WFI environment by setting the context WFI from a MIDAS session prompt:
  set/cont WFI
After a large display window opens the user will be prompted with six setup questions, which can usually be all answered by the 'enter' key. The context WFI will automatically call the context MPIAPHOT and report on its succesful initialization by outputting:

           ... DETECTOR found
and a few lines below
       ... context level MPIAPHOT activated
Without printed comments it will also call the context CADIS. The WFI reduction pipeline will report its software version with a list of available commands including short descriptions. This list can be printed at any time with:
  help/wfi
At any time sending a command without any parameters attached will print a help text explaining syntax and function of that command.


IV. MPIAPHOT catalog and 'frame list' handling:

You can work with standard MIDAS catalogs in the WFI environment using CREA/ICAT, CREA/TCAT for catalog creation and EXEC/CAT for loop execution. However, MPIAPHOT allows you advanced use of catalogs and frame lists in loop commands. Many commands in MPIAPHOT, CADIS and WFI use a 'frame list' parameter, which can either be a list of frames, a standard catalog or a selection of frames from a catalog. In addition, the MPIAPHOT loop commands PROCESS/IMAGE and PROCESS/TABLE can be used to run a loop on any command as long as this command expects the input frame as its first parameter. Examples for the command syntax are:
  RESTORE/WFI  'frame list'
  STACK/MOSAIC  'outframe' = 'frame list'
  PROCESS/TABLE CADIS/INTEG  'frame list'
The string specifying the 'frame list' may contain one of the following:
  • catalog name followed by a colon (frame list will contain all frames in catalog)
  • catalog name followed by a colon and a list of numbers or ranges of numbers
  • an explicit list of frame names separated by commata
Examples for valid 'frame lists':
  raw_frames:
  raw_frames:1,5-7,10
  raw_45678
  raw_32199,raw_32301,raw_32302,raw_40000


V. Pipeline, part I:

A typical reduction session would proceed like the following example, where we present in order the action taken, the command used and the explanatory command help text as available at the prompt.
  1. Use a tape listing to identify the file positions of the FITS raw frames on tape and load these frames from tape to disk, where they become MIDAS bdf files, using INPUT/WFI.
    call: INPUT/WFI  'file-list' 'out-initial' 'source' 'in-initial'
    
          This command reads FITS files from DAT or DLT tape or
          from folder 'source' on disk. The file-list is a list
          of numbers pointing at the individual files (position
          on the tape or number in the filename if on disk). If
          the files are retrieved from disk, the initial of the
          FITS file has to be provided.
          The output name starts with the given out-initial and
          is followed by the file identification number.
          The command writes nine MIDAS bdf frames, one for the
          header with descriptors and eight frames for the data
          images receiving the same name, where a,b,c,d,e,f,g,h
          is appended for the respective subimage.
          
    e.g.: INPUT/WFI  7,14-17,21  raw_  tape
            reads frames number 7,14 to 17 and 21 from tape and
            names the header files raw_0007, raw_0014...as well
            as the frames raw_0007a, raw_0007b, raw_0007c, etc.
          INPUT/WFI  7,14  raw_  dlt
            reads frames number 7 and 14 from the DLT drive and
            names the header files raw_0007, raw_0014...as well
            as the frames raw_0007a, raw_0007b, raw_0007c, etc.
          INPUT/WFI  7528  rawa  /data/mine/  fram
            reads FITS frame fram7528.fits from the directory
            /data/mine/ and calls the header file rawa7528
    
    
  2. Create a catalog of the FITS header files, in this example with
      CREA/ICAT  allraw  raw_????.bdf  
    and restore the original unique filenames from the observing run with RESTORE/WFI.
    call: RESTORE/WFI  'in-list'
    
          'in-list' denotes a header file of a frame, or a list
          or a catalog of header files, for which the subimages 
          a,b,c,d,e,f,g,h do exist. The command uses the unique
          number from descriptor ESO.DET.EXP.ID (originally NO)
          for the name wfi_'#' and copies the descriptor object
          into the descriptor ident.
    
          Note: The catalog should contain only header files!
    
    e.g.: RESTORE/WFI  raw_27583
          RESTORE/WFI  allraw:
    
    
  3. ALTERNATIVE Restore command for frames taken after mid-2001 when the numbering scheme of WFI images was changed at the telescope.
    call: @@ WFI:restore_2001  'in-list' 'rootname'
    
          'in-list' denotes a header file of a frame, or a list
          or a catalog of header files, for which the subimages 
          a,b,c,d,e,f,g,h do exist. The command uses a rootname
          as a prefix and a serial frame number from within the
          'in_list' for the complete target filename 'root'+'#'
          and copies the descriptor object into the descriptor
          ident. The rootname should be 'wfi_##' with number ##
          so that existing images are not overwritten, careful!
    
          Note: The catalog should contain only header files!
    
    
     
    e.g.: RESTORE/WFI  allraw: wfi_91
          RESTORE/WFI  allraw: wfi_77
     
    
  4. In principle, you have the choice of

    1. reducing the flatfield by the steps of debiasing, mosaicing, averaging and then applying the DEMOCO/WFI command to the science frames with the established flatfield as shown in step (3) or
    2. reducing flatfields and science frames together by the steps debiasing and mosaicing, and after averaging the flatfields just using the CORRECT/WFI command to the science frames for the flatfielding itself.

    In the following, we demonstrate the procedure of choice (b). Therefore, create a catalog of all restored header files with
      CREA/ICAT  allframes  wfi_?????.bdf  
    and use the command DEBIAS/WFI to subtract the bias from the frames and remove non-linearities of some mosaic chips.
    call: DEBIAS/WFI  'out-initials-4' = 'in-list'
    
          'in-list' denotes a header file of a frame or a list
          or a catalog of header files for which the subimages 
          a,b,c,d,e,f,g,h do exist. The command takes the bias
          from the overscan of each CCD, subtracts it, removes
          non-linearities in CCD responses and converts counts
          to electrons.
          The output name starts with the given initials and is
          followed by the 5th to last letter of the input name.
          The given initials should be four letters long.
     
          Note: The catalog should contain only header files!
          
    e.g.: DEBIAS/WFI  prp_  =  wfi_7583
            produces the nine files of a frame called prp_7583
          DEBIAS/WFI  prp_  =  allframes:3-6
     
    
  5. Now create a catalog of all header files of debiased frames with
      CREA/ICAT  prepframes  prp_?????.bdf  
    and use the command MOSAIC/WFI to generate a single bdf frame with the eight individual mosaic chips placed correctly and skewed to remove chip rotations. In this step also hot pixels with more than 50.000 e- dark level and bad columns are masked out. If you ever needed to, you could split a mosaiced frame back into its eight individual CCDs and its header file using DEMOSAIC/WFI.
    call: MOSAIC/WFI  'in-list'  MASK/NOMASK
    
          'in-list' denotes a header file of a frame, or a list
          or a catalog of header files, for which the subimages 
          a,b,c,d,e,f,g and h do exist. The command inserts the
          eight individual images of the mosaic into one single
          MIDAS frame called 'inframe'_all which gets a size of
          8574 by 8256 pixels.
          Areas of lost information (gaps and some bad columns)
          can be masked out and set to intensity -100. For this
          the 2nd parameter is defaulted to MASK. To switch off
          masking use NOMASK instead. The descriptor DATA_LIMIT
          is created and set to -99 to label areas with -100 as
          being of "no information".
    
          Note: The catalog should contain only header files!
    
    e.g.: MOSAIC/WFI  prp_7583  
            produces a frame called prp_7583_all
          MOSAIC/WFI  prp_frames:  NOMASK
    
    
  6. At this time, you have to select the suitable input flatfield frames for the creation of an average flatfield frame. A minimum of three frames are necessary for the median algorithm to eliminate outlier pixels, four are recommended and five are the optimum case. These flatfields need to have consistent global intensity patterns, otherwise the averaging procedure will fail to produce a correct result. Select your flatfields for an input frame list into the routine and run FLAT/WFI, which takes an appreciable amount of time on the order of 1-2 hours.
    call: FLAT/WFI  'outname' = 'in-list' 'kappa,Dm'
    
          'in-list' is a catalog or list of mosaiced frames where
          debiasing has been done before. This command normalizes
          and averages the frames using a kappa-sigma-clipping to
          eliminate values deviating significantly from the pixel
          value distribution as caused e.g. by cosmics and stars.
          The value of the cut-off 'kappa' is defaulted to 5.0 in
          units of sigma. 'Dm' denotes the diameter of the median
          filter used in the sub-routine AVERGOOD. The descriptor
          FF_NORM contains the average pixel value of the result
          flatfield.
     
    e.g.: FLAT/WFI  flat_R_ver01  =  mflat_R:
          FLAT/WFI  flat_R_ver01  =  mflat_R:3,6-8,11
     
    
    The flatfield obtained should be checked for quality by inspecting the flatfield itself and the corresponding 'flatfield'.rms frame. The latter should be rather uniform and show low values. It is possible that different flatfields are needed for data taken in different nights although the WFI flatfields appear rather stable within a short observing run. However, flatfields taken in suboptimal fashion can differ significantly due to scattered light. In this case, the question is simply phrased ''what corrects the science frames best?'', but not easily answered.
    
    
  7. The last part in steps (4)-(7) is the actual flatfielding of your science frames using the command CORRECT/WFI. Do not be surprised if the resulting corrected science frames do not appear entirely flat, since scattered light can enter the WFI at night and change the global intensity pattern of science frames as well.
    call: CORRECT/WFI  'out-initials-4' = 'in-list' 'flatfield'
    
          'in-list' denotes mosaic frames, where bias subtraction
          has been done before. The command divides the 'inframe'
          by an internally normalised version of the 'flatfield',
          in order to roughly conserve the electron flux units of
          'inframe'. Also, the frame statistics (i.e. background, 
          noise) are calculated and stored in descriptor FR_STAT.
    
          The output name starts with the given initials (have to
          be FOUR letters long!), and is followed by the fifth to
          last letter of the input name.
           
    e.g.: CORRECT/WFI  corr  =  prep47583_all flat_B_45690
            produces a frame called corr47583_all
          CORRECT/WFI  corr  =  allframes: myflat
     
    
  8. Now you have reduced your data to supposedly flat science frames. If you are satisfied with the frames, you can directly go to step (11). However, if you suspect that something is wrong with your frames, you have to decide what:
    1. Your frames might show radially symmetric background variations due to a night-sky emission line located in the wings of the filter transmission curve, especially when you are using a narrow filter. In this case go to step (10) and come back here afterwards.
    2. The frames might contain additive scattered light or additive fringes, which you would like to subtract to obtain a flat background. In this case you have to obtain a residual flatfield by averaging the science frames themselves (''superflat'') and correct the frames in an additive fashion.
    3. Or the frames could have a multiplicative error due to a wrong flatfield in the first place, which was affected by obscuration or scattered light itself. Again, you may want to obtain a residual flatfield (''superflat'') by averaging the science frames and correct in a multiplicative way.
    When constructing a superflat, the idea is to obtain an image of the non-flat background or the fringe pattern while eliminating the celestial objects contained in the frames. This can only be done properly with images taken at as different locations on the sky as possible. It is especially advantageous to mix images from different fields. Among a single field use preferably those which are as far apart from each other as possible due to the dithering.
    If you want to eliminate a global background shape or a multiplicative flatfield error, you would prefer a superflat with very little pixel-to-pixel variation and smoothed shapes. But if you want to correct for fringes, you need the initial superflat with all small-scale variations being preserved. Technically, the superflat is obtained by using the known command FLAT/WFI to average the science frames, but only frames with similar background shapes can be averaged properly. It is recommended to use no less than five, better ten, frames. Ideally, one would like to mix frames from different fields to better eliminate the halos of bright foreground stars, but if the background shape differs from field to field this will not be possible. Images containing moonlight or dusk skylight should only be included after adjusting their FLAT_BKG descriptor to a suitable value accounting for the extra illumination. The process needs 1-3 hours of time depending on the number of frames involved.
    Optionally, a second step could be to smooth the flat with SMOOTH/WFI:
    call: SMOOTH/WFI  'outframe' = 'inframe' 'xw,yw,bins' 'Lo,Hi'
    
          'inframe' denotes a mosaic frame that is to be smoothed
          chipwise by a median filter having a size of 'xw' times
          'yw' pixels and 'bins' intensity bins ranging from 'Lo'
          to 'Hi". The command uses MEDIAN/FAST and calculates an 
          'outframe' to be used as a superflat later. The default
          values for 'xw,yw,bins' are 80,80,5001 while 'Lo,Hi' is
          set to 0.9,1.1 relative to fr_stat(1). The latter needs
          adjustment if a wider range of values is present in the
          inframe.
     
    e.g.: SMOOTH/WFI  sfsmAXAF_486  =  sf__AXAF_486 
            produces the smoothed frame sfsmAXAF_486
     
    
  9. Now, the science flat should be checked for quality including its rms-frame and the frames already flatfielded once should be flatfielded once more with the residual science flat, either in a multiplicative fashion by
    e.g.  CREA/ICAT corr corr*all.bdf
          CORRECT/WFI  cor2  =  corr: sciflat_Oct99_R
    
    
    or in an additive fashion by
    e.g.  CREA/ICAT corr corr*all.bdf
          CORRECT/ADD  cor2  =  corr: sciflat_Oct99_R  MASK  WFI
    
    
  10. If your frames contain background variations which are radially symmetric, it might be due to a night-sky emission line located in the wings of the filter transmission curve, especially when you are using a narrow filter. This can be eliminated with
    call: RINGEX/WFI  'out-4' = 'in-list' 'ring-4' 'xc,yc' 'k,tw,rw'
    
          'in-list' denotes flatfielded mosaic frames, which show
          radially symmetric background levels (usually night-sky
          emission lines loated in the wings of the filter curve.
    
          The command fits a smoothed function to the frame which
          is named with the initials 'ring-4' and followed by the
          fifth to last letter of the input name. The extra light
          in this function is subtracted from the input frame and
          yields an output frame. The smoothed function is rather
          symmetric around the image center 'xc,yc' (WFI-defaults
          are 4287,4128). Objects are clipped according to 'k' in
          units of sigma, and the function is smoothed by 'tw,rw'
          bins in polar coordinates (t=theta, r=radius).
    
          The output name starts with the given initials (have to
          be FOUR letters long!), and is followed by the fifth to
          last letter of the input name.
           
    e.g.: RINGEX/WFI  cor2  =  corr47583_all ring
            produces a frame called cor247583_all
          RINGEX/WFI  corr  =  allframes: ri2_
    
    
  11. At this point, properly flatfielded images should exist which would only need cosmic correction and stacking to a deep sumframe. Before that, a coordinate transformation between the individual frames needs to be determined, which is accurate to one pixel and takes only translation into account, but explicitely no rotation or scaling, since the cosmic correction works on the original pixel grids without any interpolation or sub-pixel operations. The first step is now to generate a table of bright objects with good positions for each frame using the command FIOBJECT/WFI. The table will carry the same name as the bdf frame it belongs to.
    call: FIOBJECT/WFI  'in-list'
    
          'in-list' denotes flatfield-corrected mosaic frames.
          The command searches for bright objects in each of 16
          subareas and stores them in a table. They can be used
          to determine the positional offsets between different
          frames later (see FIMOVE/WFI). At least some 25 stars
          should be found per subarea for best performance.
     
       
    e.g.: FIOBJECT/WFI  corr34567_all
          FIOBJECT/WFI  allcorrs:
     
    
  12. After the object tables have been generated, the relative positional offsets are determined with respect to a reference frame, that should not be chosen to be an unusually short exposure. Otherwise only few common objects are found among a deep frame and the shallow reference frame, and all longer exposures will suffer from small object statistics. In fact, the solutions found and the number of common objects used are monitored on the screen. The command to be used here is FIMOVE/WFI.
    call: FIMOVE/WFI  'reference frame' 'in-list' 'switch' 'pars'
    
          'in-list' and the 'reference frame' denote corrected
          mosaic frames, for which the object tables have been
          generated already using FIOBJ/WFI. This command here
          determines the positional offset of the 'inframe' in
          comparison to the 'reference frame'. For 16 subareas
          independent offset values are obtained and listed at
          the end of the command procedure. All offsets should
          turn out the same within +-2 pixel of their average.
     
          If the 'switch' is set to all, all available objects
          are used. In default case only stars are used. Also,
          this command determines an average PSF value for the
          whole mosaic and saves it to the inframe. 'pars' are
          optional parameters for constraining the solution of
          the coordinate transformation and have the structure
          'mean scale, +- limits, mean angle, +- limits'. This
          parameter should not be used before the cosmics have
          been corrected, which need the default setting where
          no scaling or rotation is allowed, only translation.
          If one or more sectors of a frame are not found with
          the correct offset measured for the vast majority of 
          sectors, try setting 'switch' to all to increase the 
          number of objects used to derive the offset.
    
    e.g.: FIMOVE/WFI  corr39096_all corr39234_all all
          FIMOVE/WFI  corr39096_all allcorrs:
          FIMOVE/WFI  corr39096_all corr39234_all ? 1,0.01,0,0.1
     
    
  13. Now the cosmic correction and the stacking can be done, which are performed in a single step. Cosmics are identified as outliers when comparing a pixel value at a given position on the sky among different images. We recommend to use always a minimum of five images on any field to have the cosmic correction even working in sections of the field which are obstructed by the mosaic gaps in one or two of the frames. The command STACK/MOSAIC needs 3-7 hours of time depending on the number of frames involved. Also depending on number of frames (e.g. 10 to 40), 5-25 GB of scratch disk space are needed for storing intermediate results and output frames.
    call: STACK/MOSAIC  'outframe' = 'inlist' 'cosm-TH,rad' 'stack'
    
          'inlist' denotes a list/catalog of corr-frames to which
          FIMOVE/WFI has been applied before. This command stacks
          all frames from 'inlist' into the deep 'outframe' after
          performing a cosmic correction.  For each corr-frame it
          creates a cosmic corrected cosm-frame. In 'cosm-TH,rad'
          the cosm-TH defines a kappa-clipping-threshold in units
          of sigma, and rad the radius for cosmic identification,
          where the default values are 5.0,3.0. In addition, each
          single cosm-frame is accompanied by a cosmic mask frame
          having the name initials cmsk. The outframe gets a STEP
          descriptor value of 0.238, 0.238 (arcsec/pixel) and the
          descriptors FR_STAT, EXP_TIME and CCD_PARA are updated.
     
          If cosm-TH is set to  0  or below, no cosmic correction
          is applied. If 'stack' is set to NO, the frames are not
          stacked into a deep outframe.
     
    e.g.: STACK/MOSAIC  sum_R_Feb2000  =  corrXY:
          STACK/MOSAIC  sum_R_Feb2000  =  corrXY: 6.5,3.0
          STACK/MOSAIC  sum_R_test     =  corrXY: 6.5,3.0 no
          STACK/MOSAIC  sum_R_nocos    =  corrXY: 0
    
    
    The cosm-frames should be compared with the corr-frames and the cmsk-frames to make sure that the cosmic correction has worked properly. Frames with vastly different seeing should not be used together, since then the cosmic correction can easily consider the PSF peaks of stars in good seeing and their wings in bad seeing as cosmics and remove them. So, the quality check should focus investigate, whether the latter has happened and whether faint cosmics are removed successfully. Also, the sumframe should be checked for quality regarding its achieved depth and the uniformity and noise-properties of the background. Especially, borders of the mosaic gaps should not be visible any more.

    Of course, only survey quality science frames should be used for the sumframe. Frames with too high background resulting from scattered moonlight, dusk or dawn light etc. should be omitted. Also, frames with seeing below survey quality should be left out.
Now you are done with the basic image reduction. You have cosm-, cmsk- and sumframes available for photometric and morphologic analysis.


VI. Pipeline, part II:

Part II of the pipeline deals with the photomtetric analysis of reduced frames. The product are only tables and therefore only relatively little extra disk space is needed here. Maybe with the exception of the first step, all steps could be combined in a master-prg which processes the existing reduced and archived frames. Again, we proceed through the steps one by one.
  1. Use the SExtractor software to generate a list of sources detected in your deepest image and choose an appropriate detection threshold depending on how deep into the noise you would like to go. In COMBO-17 applications, use the deepest R-band sumframe and a default threshold. The resulting table is also called a master table, since it is the starting point for the work in pipeline part II. You can check the depth of found objects by plotting all objects from the master table (found at positions :x_mark and :y_mark) into an overlay of the sumframe.
    call: SEXTRACT/WFI  'out-init' = 'sum-list' 'Minpix' 'TH' 'C' 'check'
    
          'sum-list' denotes sum images with flat backgrounds.
          The command searches for objects on the sum-frame and
          stores them in a table. The name of this table begins
          with 'out-init', and is followed by the fifth to last
          letter in the name 'sum-frame'. Objects need to cover
          at least 'Minpix' pixels with flux values being above
          the threshold 'TH'. Default values are 0 for 'Minpix'
          which means that 'Minpix' is adjusted to the value of
          seeing, and 4 for 'TH'.
          
    e.g.: SEXTRACT/WFI  sext = sum_R_feb2000 ? 3
          SEXTRACT/WFI  sext = allsums:
     
    
  2. Now you need to project the coordinates of the objects contained in the master table onto the individual cosm-frames, so that they can be found there for the photometric analysis. As a reference you need the sumframe and its corresponding FIOBJ tables. Therefore, you need to move the FIOBJ tables together with the sumimage and the master table into the directory containing the cosm-frames. You first need to the prepare the cosm-frames and cmsk-frames by having a number of descriptors properly set using PREPARE/PROJECT. In particular, you need to identify the table holding the transmission curve of the filter used and also choose a 4-letter name for the later filter specific columns in the unite- and final flux-tables.
    call: PREPARE/PROJECT  'sum' 'cosm-list' 'filter-tbl' 'fluxname'
    
          This command prepares a list or catalog of cosm-frames
          for the application of PROJECT/WFI command by
    
          (1) setting the step descriptor of cosm-frame and the
              corresponding cmsk-frame to 0.238,0.238
          (2) setting further descriptors properly
          (3) running FIOBJ/WFI on the cosm-frame
          (4) running FIMOV/WFI with reference to the sum-frame
              on which the SExtractor Master table is based.
     
          The first time this command is applied to a cosm frame
          the descriptors filter, epsilist and lambda need to be
          written. 'filter-tbl' contains a transmission function
          and is located on the PHOTDATA area. The 'fluxname' is
          used for the flux columns in the unite and flux table.
          The default is not to change the descriptors, which is
          used e.g. when rerunning the analysis.
     
    e.g.: PREPARE/PROJECT  SGP_R  cosm49480_all wfi_R_full R_99
          PREPARE/PROJECT  AXAF_R_Jan00  cosm_R_ax:
    
    
  3. The actual coordinate transformation of the master table into the coordinate system of each single cosm-frame is performed with PROJECT/WFI. The transformation takes translation, rotation and 2-D independent scaling for each of 16 independent subareas into account. The command also performs a position detection for about 300 bright, unsaturated stars and compares the position found with the projected solution to assess the quality of the transformation. The 2-D-rms of the positional error is stored in the descriptor PROJ_RMS and reported on the screen as well as the number of stars used to determine the scatter. In COMBO-17 applications the 2-D-rms value should stay below 0.13 arcsec for long exposures and 0.28 arcsec for short exposures. The number of stars used should be in the range of 50 to 500. Also the best-guess PSF shape is determined from these bright stars, reported on the screen and stored in the descriptor PSF_MEAN. It is of crucial importance for the EVALUATE procedure and should be checked for plausibility.
    call: PROJECT/WFI  'out-initials-4' = 'cosm-list' 'Master'
    
          This command projects the 'Master' object list into the
          coordinate system of each frame in 'cosm-list' yielding
          a frame specific object table for later use in EVALUATE. 
          PREPARE/PROJECT must be applied to 'cosm frame' before.
          The name of the output table starts with given initials
          (have to be four letters long!), and is followed by the
          fifth to last letter of the name of the cosm frame. The
          select flag of the 'Master' table is used (set to all)!
     
    e.g.: PROJECT/WFI  proj  =  cosm49460_all  sextSGP_1
          PROJECT/WFI  proj  =  cosm_R_ax:  sextAXAF_R00
      
    
  4. From this point on, the pipeline software from the context CADIS is used, but still described for the example of a WFI data application in the COMBO-17 survey. CADIS commands do not have the same help files available as WFI commands, therefore the layout of this cookbook changes slightly in the following.

    For running EVALUATE on a set of cosm-frames you also need a setup file in the same directory, which defines aperture sizes and other parameters of the photometric procedure. For COMBO-17, get the proper file by

    cp /photo/user/exe/e15w.eval ./
    Then run the photometric analysis using EVALUATE/CADIS.
    call: EVALUATE/CADIS  'out-init-4' = 'in-list' 'proj-init-4'
    
          where 'in-list' is a catalog or list of cosm-frames and
          'proj-init-4' is the initial of the tables generated in
          the previous step and containing projected coordinates.
          'out-init-4' needs to be the name of the setup file for
          the EVALUATE photometry. For each cosm-frame a table is
          generated with the initials 'out-init-4', that contains
          the photometry of all objects as measured in the frame,
          also called an eval-table.
    
    e.g.: EVAL/CADIS  e15w = cosm: proj
    
    
  5. Select a sample of around 1000 bright but unsaturated stars in each frame which are used for a flux normalisation among the different frames using CADIS/SETNORM.
    call: CADIS/SETNORM  'in-list' 'z0_min' N 1 'satlimit' 'ideal'
    
          where 'in-list' is a catalog or list of eval-tables, and
          'z0_min' is the required minimum peak intensity required
          for a star to enter the sample. The maximum level is set
          by 'satlimit' (leave default at 90.000 electrons). E.g., 
          a 'z0_min' value of 2000 works well with 5 minute R-band
          exposures, but use values as low as 100 (default) may be
          used. If an 'ideal' number of stars is given the program
          tries to find itself a minimum intensity, giving roughly
          the 'ideal' number, but never goes below 'z0_min'. 
    
          In practice, please use MIDAS catalog loops and not the
          MPIAPHOT loops! Also, for WFI data use 50.000 electrons
          as a saturation value and ideally 1000 stars.
    
    e.g.: CADIS/SETNORM  e15w49222_all  200  N  1  50000  1000
          EXEC/CAT CADIS/SETNORM  e15w.cat  200  N  1  50000  1000
    
    
  6. Tell the context CADIS the name of the SExtractor master table that is relevant for the unite-tables you are going to generate afterwards. The unite- and flux tables will take basic information on every object (position, morphology etc.) from this reference master table.
    call: CADIS/REFER  'master'
    
    e.g.: CADIS/REFER  sextAXAFR_99
    
    
  7. The next step is to combine all flux measurements from an individual filter as contained in the eval-tables into a single deeper filter measurement using CADIS/UNITE. Here it is very important, that the first table in the list does not originate from a short exposure, since it is used as the reference for the flux normalisation. One possible approach is to rename the table prefixes of short exposures such, that they are sorted towards the end of the catalog, e.g. by changing them from e15w#####_all.tbl into e15z#####_all.tbl
    call: CADIS/UNITE  'unite-table' = 'in-list'
    
          where 'in-list' is a catalog or list of eval-tables, to
          which CADIS/SETNORM has successfully been applied. 
    
    e.g.: CADIS/UNITE  unitR_99 = e15wR_99:
    
    
  8. In each unite-table you need to identify the standard stars using CADIS/STD. You need to know the name of the standard star as defined in $PHOTSTDS/standards.tbl and the number of the star in the master table.
    call: CADIS/STD  'unite-table'  'number'  'standard-name'
    
    e.g.: CADIS/STD  unitR_99  23499  AXAF_3
    
    
  9. Convert the count rates into physical fluxes by calibrating the counts with standard stars and filter transmission curves using CADIS/INTEG.
    call: CADIS/INTEG  'unite-table' 
    
    e.g.: CADIS/INTEG  unitR_99
    
    
  10. Now move all unite-tables into a single directory and combine them all into the final flux table using CADIS/FLUX. Here, you should give a version name, which is stored in a descriptor, especially if you intend not to use a date in the name of the flux table.
    call: CADIS/FLUX  'flux-tbl' = 'unite-in-list'  'version'
    
    e.g.: CADIS/FLUX  fluxAXAF_31Dec00  =  unit:  test
          CADIS/FLUX  fluxS11__17Feb99  =  allunites:  V1.0
    
    
    Now you have an object table containing fluxes and errors from all filters you have included. The error distribution should be checked and not show many bright objects with high errors deviating largely from the scatter parabola defined by ideal photon noise. The next steps would be calculating colors, checking the calibration and running a multi-color classification.

VI. Naming conventions:

still to be written...

Back to COMBO-17 index page


Last update July 7, 2005, CW