How to reduce my data?
Usually, the newly observed data is directly reduced on-site at the telescope and can be copied to MPIA e.g. via dropbox.Nevertheless, sometimes the data is not copied, is lost or older data from the ESO archive is in need to be reprocessed. To do so, one needs the offline twin of the on-site data reduction software written in MIDAS. It can be obtained here.
The first step is to unpack the software. The folder, which will be used most is the "midwork" folder.
Download raw frames from the ESO archive
First of all it is necessary to download the correct files from the ESO raw data archive. Doing so, one has to remember that only targets observed within the same night can be reduced together. The reason for this is that the daytime calibration may change from one day to the other and may falsify the results when applied to different days. So even if several frames of the same object taken within different nights are in need to be re-reduced, one has to proceed day-by-day.By using the search interface, it is possible to get a list of a single night taken with FEROS. By sorting the exposures by observation time, we get a list of first calibration files from the afternoon (bias, flats and waves) and afterwards the target frames.
In order to reduce some (or all) object frames, we have to download the following frames from the afternoon calibration:
- 5 Bias
- 10 Flats
- 6 Waves
As soon as the 21 calibration files have been marked (checkbox on the left side, which has to be green, otherwise the data is still for restricted use of the PI), the desired object frames can be selected the same way.
Once all frames are marked, they can be requested from the archive by clicking the button "Request marked datasets" and afterwards "Submit". As soon as the frames are available, a Shell script can be downloaded named "downloadRequestXXXXXscript.sh".
The next step is to create a folder in which both the raw files and later the reduced files shall be copied, e.g. /home/user/FEROS/YYYY-MM-DD. By copying the just downloaded shell script, it can be made executable with "chmod +x downloadRequestXXXXXscript.sh" from a terminal. By executing the script (./downloadRequestXXXXXscript.sh) the ESO archive is automatically contacted and the requested files are retrieved.
The result will be 21 calibration files and the desired number of object frames with the cryptic names FEROS.YYYY-MM-DDTHH:MM:SS.SSS.fits. Object and calibration files can not be distinguished without a look in the fits-header.
Preparing the raw frames for MIDAS
Unfortunately, MIDAS does not work with .fits files, but one has to follow a certain naming convention so that the downloaded pipeline can be executed. This pipeline simply works with files named feroXXXX.mt with 4 digits, which can be freely chosen. In order to rename the downloaded files and make them distinguishable, you can use the IDL script re_name.pro, which does the job. You will be able to take part in the renaming process by specifying the first 2 of the 4 digits freely. The program will ask for it at the end for you input. The last 2 digits will make the files according to their content distinguishable:- Numbers 01-05: the 5 Bias frames
- Numbers 06-15: the 10 Flatfield frames
- Numbers 16-21: the 6 Wave frames
- Numbers 51 upwards: the object frames
Simultaniously to the file renaming, the script creates 2 MIDAS programs, which will be called "test_calibration_XX.prg" and "reduce_all_XX.prg". They will have the same 2 digits "XX" you specified before for the file renaming. Copying these 2 files to the midwork directory of the data reduction pipeline, everything has been prepared to reduce the raw files. An example for the two MIDAS files being created can be seen here: test_calibration_XX.prg and reduce_all_XX.prg
Reduce raw frames with MIDAS - Initial calibration
Once the .prg files created with re_name.pro are copied to the midwork folder of the data reduction software, one starts MIDAS in that very midwork folder using the terminal and the command "inmidas". The immediate result are 6 empty windows (5 white ones and 1 black one. Don't EVER close them! The program will crash!). If these windows don't open but MIDAS runs, you are not in the midwork folder. In case MIDAS tells you that the key has expired, see below for the fairly easy solution.The next step is to execute the program "text_calibration_XX.prg" via "@@ test_calibration_XX.prg". This program copies the renamed files from your object directory to the midwork folder and feeds the information to the MIDAS catalog. Then it executes the calibration procedure. As soon as it is finished, the previously opened windows are not empty anymore. They will show the following:
- MIDAS_01 display_0: A picture of the ThAr+Ne
- MIDAS_01 graph_0: A picture of the detected orders
- MIDAS_01 graph_1: A picture of the residuals for the object fibre
- MIDAS_01 graph_2: A picture of the residuals for the sky/calibration fibre
- MIDAS_01 graph_3: A picture of the average order shape for the object fibre
- MIDAS_01 graph_4: A picture of the average order shape for the sky/calibration fibre
- MIDAS_01 graph_0: Do the detected orders overlap? Is apparently one missing or it has been detected one at a position that is far off the others?
- Does the graph look like this?
If yes, then this part of the calibration is ok. In case you see any overlap etc., this is most likely the fault of a flatfield image. What to do against this: see below
- Does the graph look like this?
- MIDAS_01 graph_1 and graph_2: Is the scatter given as rms in both plots larger than about 3x10-3 or less? Does the pattern show any structure like a sine shape?
- Do the graphs look similar to this with an rms<3x10-3?
If yes, then this part of the calibration is ok. In case the rms is larger than about 3x10-3 and/or the pattern shows a structure this is most likely the fault of the ThAr calibration files. What to do against this: see below
- Do the graphs look similar to this with an rms<3x10-3?
- MIDAS_01 graph_3 and graph_4: Is the average order shape not approximately Gaussian shape?
- Do the plots look similar to this?
If yes, then this part of the calibration is ok. In case the order shapes do not look like this (what happens not very often!), the reason is not easy to figure out.
- Do the plots look similar to this?
Reduce raw frames with MIDAS - Optimize the calibration
In case the order locations, the rms, the scatter etc was not as expected, the reasons are either some bad flatfields and/or ThAr calibration frames. In order to fix this, it is possible to edit the initial MIDAS program "test_calibration_XX.prg" ro get rid of files that could cause the problems. This works by just commenting the individual lines out with an exclamation mark, for example:!add/icat ThAr fero2521.mt
It is important not to simply delete the line, because you might find out that this one frame was not the origin of the problems and you might want to include it again within further steps.
In case the orders were not correctly detected (overlaps, gaps etc) the reason is most likely one or more bad flatfield frames. To fix this is lots of work, because the bad frames can not easily be identified. It is a try-and-error task. Since it can also be a combination of two or more flatfield frames, to fix this will take a while!
To fix a too large rms and/or a pattern in the residuals (e.g. sine-shape) is much easier. Usually the problem is an under- or overexposed ThAr frame. Since the ThAr frames are sorted by exposure time in the test_calibration_XX.prg file, it is easy to comment out the maybe underexposed short exposures (feroXX16.mt and feroXX17.mt) or the long ones (feroXX20.mt and feroXX21.mt).
For the whole optimizing procedure: it is a large matter of experience (oh, I recognize this patter, the origin was last time the short exposures of ThAr) and luck. So don't give up!
Reduce raw frames with MIDAS - Final reduction
Once a satisfying calibration has been achieved (= all calibration frames look similar to the ones above), the final reduction of the object files can take place. To do so, simply execute the MIDAS program reduce_all_XX.prg by typing"@@ reduce_all_XX.prg"
All object frames will then automatically be reduced and the reduction products will be copied to your initial folder in which the raw frames are located. You'll find a few new folders there:
- fibre1: this folder contains the merged object spectrum (all 39 orders glued together) in .mt-format, which is simply a .fits file and can be treated accordingly
- fibre1/singleOrders: this folder contains all single orders of the object spectrum from 01 to 39 in .mt format
- fibre2: this folder contains the merged sky/calibration spectrum (all 39 orders glued together) in .mt-format
- fibre1/singleOrders: this folder contains all single orders of the sky/calibration spectrum from 01 to 39 in .mt format
- extracted: this folders contains small parts of the object spectrum like the H-alpha line including a .ps plot for a quick look of the quality of the spectrum
Working with the data
In principle, the pipeline products sorted in the various folders (see above) are ready to be used for lots of different purposes: determination of stellar parameters, projected rotational velocities, radial velocity etc.Nevertheless, it can be of use for some programs (especially plot programs) to have the data in ascii (wavelength vs. flux) instead of .mt (=.fits). Since the .mt files simply contain the flux, the wavelength has to be created from informations given in the header.
To do so, you can use the IDL script make_ascii.pro.
Troubleshooting
Once in a while the following error can appear while working with the data reduction pipeline in MIDAS:"Key has expired"
To solve this, you leave MIDAS and enter the following commands (one after the other) in the terminal:
unalias inmidas
unalias gomidas
unset MIDASHOME
unset MIDVERS
PLEASE NOTE!
The barycentric correction of the MIDAS pipeline is not very accurate! This problem has been described in Mueller, A. et al., 2013. The more accurate value can be calculated using the IDL program baryvel.pro. The unprecise value given in the header of the processed data looks like this (given in km/s):HISTORY 'BARY_CORR','R*4',1,1,'5E14.7'
HISTORY -1.6480301E+01
By applying this correction backwards and use the newly calculated one with baryvel.pro it is possible to correct the data properly in terms of Earth's and Solar System's movement.