How to reduce my data?

Usually, the newly observed data is directly reduced on-site at the telescope and can be copied to MPIA e.g. via dropbox.
Nevertheless, sometimes the data is not copied, is lost or older data from the ESO archive is in need to be reprocessed. To do so, one needs the offline twin of the on-site data reduction software written in MIDAS. It can be obtained here.
The first step is to unpack the software. The folder, which will be used most is the "midwork" folder.

Download raw frames from the ESO archive

First of all it is necessary to download the correct files from the ESO raw data archive. Doing so, one has to remember that only targets observed within the same night can be reduced together. The reason for this is that the daytime calibration may change from one day to the other and may falsify the results when applied to different days. So even if several frames of the same object taken within different nights are in need to be re-reduced, one has to proceed day-by-day.
By using the search interface, it is possible to get a list of a single night taken with FEROS. By sorting the exposures by observation time, we get a list of first calibration files from the afternoon (bias, flats and waves) and afterwards the target frames.
In order to reduce some (or all) object frames, we have to download the following frames from the afternoon calibration:
  • 5 Bias
  • 10 Flats
  • 6 Waves
Doing so, one has to take care that one takes the last frames taken before the object exposures start. In case more than e.g. 6 Waves have been taken (this is usually the case), one takes the last 6 taken before the science observations startet. These are the ThAr+Ne exposures that are usually taken for the data reduction. The same goes for Bias and flats.
As soon as the 21 calibration files have been marked (checkbox on the left side, which has to be green, otherwise the data is still for restricted use of the PI), the desired object frames can be selected the same way.
Once all frames are marked, they can be requested from the archive by clicking the button "Request marked datasets" and afterwards "Submit". As soon as the frames are available, a Shell script can be downloaded named "downloadRequestXXXXXscript.sh".
The next step is to create a folder in which both the raw files and later the reduced files shall be copied, e.g. /home/user/FEROS/YYYY-MM-DD. By copying the just downloaded shell script, it can be made executable with "chmod +x downloadRequestXXXXXscript.sh" from a terminal. By executing the script (./downloadRequestXXXXXscript.sh) the ESO archive is automatically contacted and the requested files are retrieved.
The result will be 21 calibration files and the desired number of object frames with the cryptic names FEROS.YYYY-MM-DDTHH:MM:SS.SSS.fits. Object and calibration files can not be distinguished without a look in the fits-header.

Preparing the raw frames for MIDAS

Unfortunately, MIDAS does not work with .fits files, but one has to follow a certain naming convention so that the downloaded pipeline can be executed. This pipeline simply works with files named feroXXXX.mt with 4 digits, which can be freely chosen. In order to rename the downloaded files and make them distinguishable, you can use the IDL script re_name.pro, which does the job. You will be able to take part in the renaming process by specifying the first 2 of the 4 digits freely. The program will ask for it at the end for you input. The last 2 digits will make the files according to their content distinguishable:
  • Numbers 01-05: the 5 Bias frames
  • Numbers 06-15: the 10 Flatfield frames
  • Numbers 16-21: the 6 Wave frames
  • Numbers 51 upwards: the object frames
If the keyword "mv" is set during the execution of the IDL script re_name.pro, the originally downloaded files from the archive will be deleted.
Simultaniously to the file renaming, the script creates 2 MIDAS programs, which will be called "test_calibration_XX.prg" and "reduce_all_XX.prg". They will have the same 2 digits "XX" you specified before for the file renaming. Copying these 2 files to the midwork directory of the data reduction pipeline, everything has been prepared to reduce the raw files. An example for the two MIDAS files being created can be seen here: test_calibration_XX.prg and reduce_all_XX.prg

Reduce raw frames with MIDAS - Initial calibration

Once the .prg files created with re_name.pro are copied to the midwork folder of the data reduction software, one starts MIDAS in that very midwork folder using the terminal and the command "inmidas". The immediate result are 6 empty windows (5 white ones and 1 black one. Don't EVER close them! The program will crash!). If these windows don't open but MIDAS runs, you are not in the midwork folder. In case MIDAS tells you that the key has expired, see below for the fairly easy solution.
The next step is to execute the program "text_calibration_XX.prg" via "@@ test_calibration_XX.prg". This program copies the renamed files from your object directory to the midwork folder and feeds the information to the MIDAS catalog. Then it executes the calibration procedure. As soon as it is finished, the previously opened windows are not empty anymore. They will show the following:
  • MIDAS_01 display_0: A picture of the ThAr+Ne
  • MIDAS_01 graph_0: A picture of the detected orders
  • MIDAS_01 graph_1: A picture of the residuals for the object fibre
  • MIDAS_01 graph_2: A picture of the residuals for the sky/calibration fibre
  • MIDAS_01 graph_3: A picture of the average order shape for the object fibre
  • MIDAS_01 graph_4: A picture of the average order shape for the sky/calibration fibre
In order to check the quality of the calibration, one has to consider different parameters:
  • MIDAS_01 graph_0: Do the detected orders overlap? Is apparently one missing or it has been detected one at a position that is far off the others?
    • Does the graph look like this?
      Order position
      If yes, then this part of the calibration is ok. In case you see any overlap etc., this is most likely the fault of a flatfield image. What to do against this: see below
  • MIDAS_01 graph_1 and graph_2: Is the scatter given as rms in both plots larger than about 3x10-3 or less? Does the pattern show any structure like a sine shape?
    • Do the graphs look similar to this with an rms<3x10-3?
      Scatter
      If yes, then this part of the calibration is ok. In case the rms is larger than about 3x10-3 and/or the pattern shows a structure this is most likely the fault of the ThAr calibration files. What to do against this: see below
  • MIDAS_01 graph_3 and graph_4: Is the average order shape not approximately Gaussian shape?
    • Do the plots look similar to this?
      Scatter
      If yes, then this part of the calibration is ok. In case the order shapes do not look like this (what happens not very often!), the reason is not easy to figure out.

Reduce raw frames with MIDAS - Optimize the calibration

In case the order locations, the rms, the scatter etc was not as expected, the reasons are either some bad flatfields and/or ThAr calibration frames. In order to fix this, it is possible to edit the initial MIDAS program "test_calibration_XX.prg" ro get rid of files that could cause the problems. This works by just commenting the individual lines out with an exclamation mark, for example:
!add/icat ThAr fero2521.mt
It is important not to simply delete the line, because you might find out that this one frame was not the origin of the problems and you might want to include it again within further steps.
In case the orders were not correctly detected (overlaps, gaps etc) the reason is most likely one or more bad flatfield frames. To fix this is lots of work, because the bad frames can not easily be identified. It is a try-and-error task. Since it can also be a combination of two or more flatfield frames, to fix this will take a while!
To fix a too large rms and/or a pattern in the residuals (e.g. sine-shape) is much easier. Usually the problem is an under- or overexposed ThAr frame. Since the ThAr frames are sorted by exposure time in the test_calibration_XX.prg file, it is easy to comment out the maybe underexposed short exposures (feroXX16.mt and feroXX17.mt) or the long ones (feroXX20.mt and feroXX21.mt).
For the whole optimizing procedure: it is a large matter of experience (oh, I recognize this patter, the origin was last time the short exposures of ThAr) and luck. So don't give up!

Reduce raw frames with MIDAS - Final reduction

Once a satisfying calibration has been achieved (= all calibration frames look similar to the ones above), the final reduction of the object files can take place. To do so, simply execute the MIDAS program reduce_all_XX.prg by typing
"@@ reduce_all_XX.prg"
All object frames will then automatically be reduced and the reduction products will be copied to your initial folder in which the raw frames are located. You'll find a few new folders there:
  • fibre1: this folder contains the merged object spectrum (all 39 orders glued together) in .mt-format, which is simply a .fits file and can be treated accordingly
  • fibre1/singleOrders: this folder contains all single orders of the object spectrum from 01 to 39 in .mt format
  • fibre2: this folder contains the merged sky/calibration spectrum (all 39 orders glued together) in .mt-format
  • fibre1/singleOrders: this folder contains all single orders of the sky/calibration spectrum from 01 to 39 in .mt format
  • extracted: this folders contains small parts of the object spectrum like the H-alpha line including a .ps plot for a quick look of the quality of the spectrum

Working with the data

In principle, the pipeline products sorted in the various folders (see above) are ready to be used for lots of different purposes: determination of stellar parameters, projected rotational velocities, radial velocity etc.
Nevertheless, it can be of use for some programs (especially plot programs) to have the data in ascii (wavelength vs. flux) instead of .mt (=.fits). Since the .mt files simply contain the flux, the wavelength has to be created from informations given in the header.
To do so, you can use the IDL script make_ascii.pro.

Troubleshooting

Once in a while the following error can appear while working with the data reduction pipeline in MIDAS:

"Key has expired"

To solve this, you leave MIDAS and enter the following commands (one after the other) in the terminal:
unalias inmidas
unalias gomidas
unset MIDASHOME
unset MIDVERS

PLEASE NOTE!

The barycentric correction of the MIDAS pipeline is not very accurate! This problem has been described in Mueller, A. et al., 2013. The more accurate value can be calculated using the IDL program baryvel.pro. The unprecise value given in the header of the processed data looks like this (given in km/s):
HISTORY 'BARY_CORR','R*4',1,1,'5E14.7'
HISTORY -1.6480301E+01
By applying this correction backwards and use the newly calculated one with baryvel.pro it is possible to correct the data properly in terms of Earth's and Solar System's movement.