Fermi Gamma-ray Space Telescope

Data Preparation

Photon and spacecraft data are all that a user needs for the analysis. For the definition of LAT data products see the information in the Cicerone.

The LAT data can be extracted from the Fermi Science Support Center web site, as described in section Extract LAT data. Preparing these data for analysis depends on the type of analysis you wish to perform (e.g. point source, extended source, GRB spectral analysis, timing analysis, etc). The different cuts to the data are described in detail in the Cicerone.

Data preparation consists of two steps.

  • (gtselect): Used to make cuts based on columns in the event data file such as time, energy, position, zenith angle, instrument coordinates, event class, and event type (new in Pass 8).
  • (gtmktime): In addition to cutting the selected events, gtmktime makes cuts based on the spacecraft file and updates the GTI extension.

Here we give an example of how to prepare the data for the analysis of a point source. For your particular source analysis you have to prepare your data performing similar steps, but with the cuts suggested in Cicerone for your case.

You can download this tutorial as a Jupyter notebook and run it interactively. Please see the instructions for using the notebooks with the Fermitools.

1. Event selection with gtselect

In this section, we look at making basic data cuts using gtselect. By default, gtselect prompts for cuts on:

  • Time
  • Energy
  • Position (RA,Dec,radius)
  • Maximum Zenith Angle

However, by using hidden parameters defined on the command line (or using the 'Show Advanced Parameters' check box in GUI mode), you can also make cuts on:

  • Event class ID
  • Event type ID (selects on conversion type, angular or energy reconstruction quality)
  • Minimum pulse phase
  • Maximum pulse phase

For this example we use data that was extracted using the procedure described in the Extract LAT Data tutorial. The original selection used the following information:

  • Search Center (RA,DEC) = (193.98,-5.82)
  • Radius = 20 degrees
  • Start Time (MET) = 239557417 seconds (2008-08-04 T15:43:37)
  • Stop Time (MET) = 255398400 seconds (2009-02-04 T00:00:00)
  • Minimum Energy = 100 MeV
  • Maximum Energy = 500000 MeV

The LAT operated in survey mode for that period of time. We provide the user with the photon and spacecraft data files extracted in the same method as described in the Extract LAT data tutorial:

  1. L1506091032539665347F73_PH00.fits
  2. L1506091032539665347F73_PH01.fits
  3. L1506091032539665347F73_SC00.fits

If more than one file was generated by the LAT Data Server, we will need to provide an input file list, in order to use all the event data files in the same analysis. This text file can be generated by typing:

ls *_PH* > events.txt

This input file can be used in place of a single input events (or FT1) file by placing an @ symbol before the text filename. The output from gtselect will be a single file containing all events from the combined file list that satisfy the other specified cuts.

For a simple point source analysis, it is recommended that you only include events with a high probability of being photons. This cut is performed by selecting "source" class events with the the gtselect tool by including the hidden parameter evclass on the command line. For LAT Pass 8 data, "source" events are specified as event class 128 (the default value).

Additionally, in Pass 8 you can supply the hidden parameter evtype (event type) which is a sub-selection on evclass. For a simple analysis, we wish to include all front+back converting events within all PSF and Energy subclasses. This is specified as evtype 3 (the default value).

The recommended values for both evclass and evtype may change as LAT data processing develops.

Now run gtselect to select the data you wish to analyze. For this example, we consider the source class photons within a 20 degree acceptance cone of the blazar 3C 279. We apply the gtselect tool to the data file as follows:

prompt> gtselect evclass=128 evtype=3
Input FT1 file[] @events.txt
Output FT1 file[] 3C279_region_filtered.fits
RA for new search center (degrees) (0:360) [0] 193.98
Dec for new search center (degrees) (-90:90) [0] -5.82
radius of new search region (degrees) (0:180) [180] 20
start time (MET in s) (0:) [0] INDEF
end time (MET in s) (0:) [0] INDEF
lower energy limit (MeV) (0:) [30] 100
upper energy limit (MeV) (0:) [300000] 500000
maximum zenith angle value (degrees) (0:180) [180] 90
Done.
prompt>

The filtered data is provided here.

If you don't want to make a selection on a given parameter, just enter a zero (0) as the value.

In this step we also selected the maximum zenith angle value as suggested in the Cicerone. Photons coming from the Earth limb are a strong source of background. You can minimize this effect with a zenith angle cut. The value of 90 degrees is suggested for reconstructing events above 100 MeV and provides a sufficient buffer between your region of interest (ROI) and the Earth's limb. In the next step, gtmktime will remove any time period that our ROI overlaps this buffer region. While increasing the buffer (reducing zmax) may decrease the background rate from albedo gammas, it will also reduce the amount of time your ROI is completely free of the buffer zone and thus reduce the livetime on the source of interest.

Notes:

  • The RA and Dec of the search center must exactly match that used in the dataserver selection. If they are not the same, multiple copies of the source position will appear in your prepared data file which will cause later stages of analysis to fail. See "DSS Keywords" below.
  • The radius of the search region selected here must lie entirely within the region defined in the dataserver selection. They can be the same values, with no negative effects.
  • The time span selected here must lie within the time span defined in the dataserver selection. They can be the same values with no negative effects.
  • The energy range selected here must lie within the time span defined in the dataserver selection. They can be the same values with no negative effects.

BE AWARE: gtselect writes descriptions of the data selections to a series of `Data Sub-Space' (DSS) keywords in the EVENTS extension header. These keywords are used by the exposure-related tools and by gtlike for calculating various quantities, such as the predicted number of detected events given by the source model. These keywords MUST be same for all of the filtered event files considered in a given analysis. gtlike will check to ensure that all of the DSS keywords are the same in all of the event data files. For a discussion of the DSS keywords see the Data Sub-Space Keywords page.

There are multiple ways to view information about your data file. For example:

  • You may obtain the value of start and end time of your file by using the fkeypar tool. This tool is part of the FTOOLS software package, and is used to read the value of a FITS header keyword and write it to an output parameter file. For more information on the fkeypar tool, type:
    fhelp fkeypar
  • The gtvcut tool can be used to view the DSS keywords in a given extension, where the EVENTS extension is assumed by default. This is an excellent way to to find out what selections have been made already on your data file (by either the dataserver, or previous runs of gtselect). NOTE: If you wish to view the (very long) list of good time intervals (GTIs), you can use the hidden parameter `suppress_gtis=no' on the command line. The full list of GTIs is suppressed by default.
    prompt> gtvcut suppress_gtis=no
    Input FITS file[] 3C279_region_filtered.fits
    Extension name[EVENTS]
    DSTYP1: BIT_MASK(EVENT_CLASS,128,P8R3)
    DSUNI1: DIMENSIONLESS
    DSVAL1: 1:1

    DSTYP2: POS(RA,DEC)
    DSUNI2: deg
    DSVAL2: CIRCLE(193.98,-5.82,20)

    DSTYP3: TIME
    DSUNI3: s
    DSVAL3: TABLE
    DSREF3: :GTI

    GTIs:
    239557417.494 239558069.093
    239559567.98 239563954.084
    239565647.986 239569844.085
    ...
    255384024.931 255388191.087
    255390039.929 255394247.086
    255396028.929 255398400

    DSTYP4: BIT_MASK(EVENT_TYPE,3,P8R3)
    DSUNI4: DIMENSIONLESS
    DSVAL4: 1:1

    DSTYP4: ENERGY
    DSUNI4: MeV
    DSVAL4: 100:500000

    DSTYP5: ZENITH_ANGLE
    DSUNI5: deg
    DSVAL5: 0:90

    prompt>

2. Time Selection with gtmktime

You may have noticed that all of these files have a GTI extension in them. Before we look at making selections with the gtmktime tool, we should probably clarify what at Good Time Interval (GTI) is:

  • Simply stated, a GTI is a time range when the data can be considered valid. The GTI extension contains a list of these GTI's for the file. Thus the sum of the entries in the GTI extension of a file corresponds to the time when the data in the file is "good".

How are these interpreted for Fermi?

  • The initial list of GTI's are the times that the LAT was collecting data over the time range you selected. The LAT does not collect data while the observatory is transiting the Southern Atlantic Anomaly (SAA), or during rare events such as software updates or spacecraft maneuvers.

    Notes:

    • Your object will most likely not be in the field of view during the entire time that the LAT was taking data.
    • Additional data cuts made with gtmktime will update the GTI's based on the cuts specified in both gtmktime and gtselect.
    • The Fermitools use the GTI's when calculating exposure. If these have not been properly updated, the exposure correction made during science analysis may be incorrect.

gtmktime is used to update the GTI extension and make cuts based on spacecraft parameters contained in the spacecraft (pointing and livetime history) file. It reads the spacecraft file and, based on the filter expression and specified cuts, creates a set of GTIs. These are then combined (logical and) with the existing GTIs in the Event data file, and all events outside this new set of GTIs are removed from the file. New GTIs are then written to the GTI extension of the new file.

Cuts can be made on any field in the spacecraft file by adding terms to the filter expression using C-style relational syntax:

!->not, &&-> and, -> or, ==, !=, >, <, >=, <=

ABS(), COS(), SIN(), etc., also work

NOTE: Every time you specify an additional cut on time, ROI, zenith angle, event class, or event type using gtselect, you must run gtmktime to reevaluate the GTI selection.

Several of the cuts made above with gtselect will directly affect the exposure. Running gtmktime will select the correct GTIs to handle these cuts.

It is also especially important to apply a zenith cut for small ROIs (< 20 degrees), as this brings your source of interest close to the Earth's limb.There are two different methods for handling the complex cut on zenith angle. One method involves excluding time intervals where the buffer zone defined by the zenith cut intersects the ROI from the list of GTIs. In order to do that, run gtmktime and answer "yes" at the prompt:

Apply ROI-based zenith angle cut [ ] yes
  • NOTE: If you are studying a very broad region (or the whole sky) you would lose most (all) of your data when you implement the ROI-based zenith angle cut in gtmktime. In this case you can allow all time intervals where the cut intersects the ROI, but the intersection lies outside the FOV. To do this, run gtmktime specifying a filter expression defining your analysis region, and answer "no" to the question regarding the ROI-based zenith angle cut:
    Filter expression [ ] ( ANGSEP(RA_ZENITH,DEC_ZENITH,RA_SCZ,DEC_SCZ) < limb_angle_minus_FOV ) || ( ANGSEP(RA_ZENITH,DEC_ZENITH,RA_of_center_ROI,DEC_of_center_ROI) + radius_ROI < zenith_cut )
    Apply ROI-based zenith angle cut [ ] no

    Here, RA_of_center_ROI, DEC_of_center_ROI and radius_ROI correspond to the ROI selection made with gtselect, zenith_cut is defined as 90 degrees (as above), and limb_angle_minus_FOV is (zenith angle of horizon - FOV radius) where the zenith angle of the horizon is 113 degrees.

You can, instead, apply the zenith cut to the livetime calculation while running gtltcube. This is the method that is currently recommended by the LAT team (see the Livetimes and Exposure section of the Cicerone), and is the method we will use most commonly in these analysis threads. To do this, answer "no" at the gtmktime prompt:

Apply ROI-based zenith angle cut [ ] no

And instead, remember to use the hidden "zmax" parameter later when calculating the livetime cube:

prompt> gtltcube zmax=90

gtmktime also provides the ability to exclude periods when some event has negatively affected the quality of the LAT data. To do this, we select good time intervals (GTIs) by using a logical filter for any of the quantities in the spacecraft file. Some possible quantities for filtering data are:

  • DATA_QUAL - quality flag set by the LAT instrument team (0 = Bad data, 1 = Good data, 2 = GPS timing anomaly, see the detailed definition in the following link)
  • LAT_CONFIG - instrument configuration (0 = not recommended for analysis, 1 = science configuration)
  • ROCK_ANGLE - can be used to eliminate pointed observations from the dataset.

    NOTE: A history of the rocking profiles that have been used by the LAT can be found in the observations section.


NOTE: For standard analyses that do not involve pulsar timing, the current gtmktime filter expression recommended by the LAT team is: (DATA_QUAL>0)&&(LAT_CONFIG==1). For pulsar timing, the expression should be: (DATA_QUAL==1)&&(LAT_CONFIG==1).

Here is an example of running gtmktime on the 3C 279 filtered events file. It is useful to rename the spacecraft file to something easier. Here, we have renamed L14043015051399489F7F38_SC00.fits to spacecraft.fits.

prompt> gtmktime
Spacecraft data file[] spacecraft.fits
Filter expression[] (DATA_QUAL>0)&&(LAT_CONFIG==1)
Apply ROI-based zenith angle cut[] no
Event data file[] 3C279_region_filtered.fits
Output event file name[] 3C279_region_filtered_gti.fits

The data with all the cuts described above is provided in this link.

After the data preparation it is advisable to take a look at your data before beginning the detailed analysis. The Explore LAT data tutorial has suggestions on method of getting a quick preview of your data.

Last updated by: N. Mirabal 10/04/2018