Data Preparation

Photon and spacecraft data are all that a user needs for the analysis. For the definition of LAT data products see the information in the Cicerone.

The LAT data can be extracted from the Fermi Science Support Center web site, as described in section Extract LAT data. Preparing these data for analysis depends on the type of analysis you wish to perform (e.g. point source, extended source, GRB spectral analysis, timing analysis, etc). The different cuts to the data are described in detail in the Cicerone.

Data preparation consists of two steps.

Event selection (gtselect) -- Used to make cuts based on columns in the event data file such as time, energy, position, zenith angle, instrument coordinates and event class.
Time selection (gtmktime) -- In addition to cutting the selected events, gtmktime makes cuts based on the spacecraft file and updates the GTI extension.

Here we give an example of how to prepare the data for the analysis of a point source. For your particular source analysis you have to prepare your data performing similar steps, but with the cuts suggested in Cicerone for your case.

1. Event selection with gtselect

In this section, we look at making basic data cuts using gtselect. By default, gtselect prompts for cuts on:

Time
Energy
Position (RA,Dec,radius)
Maximum Zenith Angle

However, by using hidden parameters defined on the command line (or using the 'Show Advanced Parameters' check box in GUI mode), you can also make cuts on:

Minimum event class ID
Maximum event class ID
Minimum pulse phase
Maximum pulse phase
Conversion type (-1=both, 0=Front, 1=Back)

For this example we use data that was extracted using the procedure described in the Extract LAT Data tutorial. The original selection used the following information:

Search Center (RA, DEC) =(193.98, -5.82)
Radius = 20 degrees
Start Time (MET) = 239557417 seconds (2008-08-04 T15:43:37)
Stop Time (MET) = 255398400 seconds (2009-02-04 T00:00:00)
Minimum Energy = 100 MeV
Maximum Energy = 100000 MeV

The LAT operated in survey mode for that period of time. We provide the user with the original photon and spacecraft data files extracted in the Extract LAT data tutorial:

In order to use the two event data files in the same analysis, we will need to provide an input file list. This text file can be generated by typing:


ls *_PH* > events.txt

This input file can be used in place of a single input events (or FT1) file by placing an @ symbol before the text filename. The output from gtselect will be a single file containing all events from the combined file list that satisfy the other specified cuts.

For point source analysis, it is recommended that you include events with a high probability of being photons. This cut is performed by selecting diffuse class events with the gtselect tool by including the hidden parameters evclsmin and evclsmax on the command line. For LAT Pass 6 data, diffuse events are specified as event classes 3 and 4. This value may change as LAT data processing develops.

Now run gtselect to select the data you wish to analyze. For this example, we consider the diffuse class photons within a 20 degree acceptance cone of the blazar 3C 279. We apply the gtselect tool to the data file as follows:


prompt> gtselect evclsmin=3 evclsmax=4 

Input FT1 file[] @events.txt

Output FT1 file[] 3C279_region_filtered.fits

RA for new search center (degrees) (0:360) [0] 193.98

Dec for new search center (degrees) (-90:90) [0] -5.82

radius of new search region (degrees) (0:180) [180] 20

start time (MET in s) (0:) [0] 239557417 

end time (MET in s) (0:) [0] 255398400 

lower energy limit (MeV) (0:) [30] 100

upper energy limit (MeV) (0:) [300000] 100000

maximum zenith angle value (degrees) (0:180) [180] 105

Done.

prompt>

If you don't want to make a selection on a given parameter, just enter a zero (0) as the value.

In this step we also selected the maximum zenith angle value (105 degrees) as suggested in the Cicerone. Photons coming from the Earth limb are a strong source of background. You can minimize this effect with a zenith-angle cut. The value of 105 degrees is the one most commonly used, and provides an 8 degree buffer between your region of interest (ROI) and the Earth's limb. In the next step, gtmktime will remove any time period that our ROI overlaps this buffer region. While increasing the buffer (reducing zmax) may decrease the background rate from albedo gammas, it will also reduce the amount of time your ROI is completely free of the buffer zone and thus reduce the livetime on the source of interest.

Notes:

The RA and Dec of the search center should match that used in the dataserver selection.
The radius of the search region selected here must lie entirely within the region defined in the dataserver selection. They can be the same values, with no negative effects.
The time span selected here must lie within the time span defined in the dataserver selection. They can be the same values with no negative effects.
The energy range selected here must lie within the time span defined in the dataserver selection. They can be the same values with no negative effects.

The filtered data is provided here.

gtselect writes descriptions of the data selections to a series of 'Data Sub-Space' (DSS) keywords in the EVENTS extension header. These keywords are used by the exposure-related tools and by gtlike for calculating various quantities, such as the predicted number of detected events given by the source model. These keywords must be same for all of the filtered event files considered in a given analysis. gtlike will check to ensure that all of the DSS keywords are the same in all of the event data files. For a discussion of the DSS keywords see the Data Sub-Space Keywords page.

There are multiple ways to view information about your data file. For example:

You may obtain the value of start and end time of your file by using the fkeypar tool. This tool is part of the FTOOLS software package, and is used to read the value of a FITS header keyword and write it to an output parameter file. For more information on the fkeypar tool, type: fhelp fkeypar
The gtvcut tool can be used to view the DSS keywords in a given extension, where the EVENTS extension is assumed by default. This is an excellent way to to find out what selections have been made already on your data file (by either the dataserver, or previous runs of gtselect). NOTE: You may wish to use the hidden parameter 'suppress_gtis=yes' on the command line to suppress the long output of good time intervals (GTIs) produced by this tool.


prompt> gtvcut

Input FITS file[] 3C279_region_filtered.fits 

Extension name[EVENTS] 

DSTYP1: POS(RA,DEC)

DSUNI1: deg

DSVAL1: CIRCLE(193.98,-5.82,20)





DSTYP2: TIME

DSUNI2: s

DSVAL2: TABLE

DSREF2: :GTI



GTIs:

239557417.494  239558069.093

239559567.98  239563954.084

239565647.986  239569844.085

...

255384024.931  255388191.087

255390039.929  255394247.086

255396028.929  255398400



DSTYP3: ENERGY

DSUNI3: MeV

DSVAL3: 100:100000



DSTYP4: EVENT_CLASS

DSUNI4: dimensionless

DSVAL4: 3:3



DSTYP5: ZENITH_ANGLE

DSUNI5: deg

DSVAL5: 0:105



prompt>

2. Time Selection with gtmktime

You may have noticed that all of these files have a GTI extension in them. Before we look at making selections with the gtmktime tool, we should probably clarify what at Good Time Interval (GTI) is:

Simply stated, a GTI is a time range when the data can be considered valid. The GTI extension contains a list of these GTI's for the file. Thus the sum of the entries in the GTI extension of a file corresponds to the time when the data in the file is "good".

How are these interpreted for Fermi?

The initial list of GTI's are the times that the LAT was collecting data over the time range you selected. The LAT does not collect data while the observtory is transiting the Southern Atlantic Anomoly (SAA), or during rare events such as software updates or spacecraft maneuvers.
Notes:
- Your object will most likely not be in the field of view during the entire time that the LAT was taking data.
- Additional data cuts made with gtmktime will update the GTI's based on the cuts specified in both gtmktime and gtselect.
- The Science Tools use the GTI's when calculating exposure. If these have not been properly updated, the exposure correction made during science analysis may be incorrect.

gtmktime is used to update the GTI extension and make cuts based on spacecraft parameters contained in the spacecraft (pointing and livetime history) file. It reads the spacecraft file and, based on the filter expression and specified cuts, creates a set of GTIs. These are then combined (logical and) with the existing GTIs in the Event data file, and all events outside this new set of GTIs are removed from the file. New GTIs are then written to the GTI extension of the new file.

Cuts can be made on any field in the spacecraft file by adding terms to the filter expression using C-style relational syntax:

!->not, &&-> and, ||-> or, ==, !=, >, <, >=, <=

ABS(), COS(), SIN(), etc., also work

NOTE: Every time you specify an additional cut on time, ROI, zenith angle, or event class using gtselect, you must run gtmktime to reevaluate the GTI selection.

Several of the cuts made above with gtselect will directly affect the exposure. gtmktime will select the correct GTIs to handle these cuts, including the complex cut on zenith angle. The easy way to implement this cut is to exclude time intervals where the buffer zone defined by the zenith cut intersects the ROI from the list of GTIs. In order to do that, run gtmktime and answer "yes" at the prompt:


Apply ROI-based zenith angle cut [ ] yes

Applying this cut is especially important if your ROI is small (< 20 degrees) bringing your source of interest close to the Earth's limb.

Note: If you are studing a very broad region (or the whole sky) you would lose most (all) of your data when you implement the zenith angle cut. In this case you can allow all time intervals where the cut intersects the ROI, but the intersection lies outside the FOV. To do this, run gtmktime specifying a filter expression defining your analysis region, and answer "no" to the question regarding the ROI-based zenith angle cut:
Filter expression [ ] ( ANGSEP(RA_ZENITH,DEC_ZENITH,RA_SCZ,DEC_SCZ) < limb_angle_minus_FOV ) || ( ANGSEP(RA_ZENITH,DEC_ZENITH,RA_of_center_ROI,DEC_of_center_ROI) + radius_ROI < zenith_cut ) Apply ROI-based zenith angle cut [ ] no
Here, RA_of_center_ROI, DEC_of_center_ROI and radius_ROI correspond to the ROI selection made with gtselect, zenith_cut is defined as 105 degrees (as above), and limb_angle_minus_FOV is (zenith angle of horizon - FOV radius) where the zenith angle of the horizon is 113 degrees.

gtmktime also provides the ability to exclude periods when some event has negatively affected the quality of the LAT data. To do this, we select good time intervals (GTIs) by using a logical filter for any of the quantities in the spacecraft file. Some possible quantities for filtering data are:

DATA_QUAL - quality flag set by the LAT instrument team (0 = bad, 1 = ok)
LAT_CONFIG - instrument configuration (0 = not recommended for analysis, 1 = science configuration)
ROCK_ANGLE - can be used to eliminate pointed observations from the dataset.
NOTE: A history of the rocking profiles that have been used by the LAT can be found in the observations section.

The current gtmktime filter expression recommended by the LAT team is:
DATA_QUAL==1 && LAT_CONFIG==1 && ABS(ROCK_ANGLE)<52.

Here it is an example of running gtmktime on the 3C 279 filtered events file. It is useful to rename the spacecraft file to something easier. Here, we have renamed L100806104523E0D2F37E54_SC00.fits to spacecraft.fits.


prompt> gtmktime

Spacecraft data file[] spacecraft.fits

Filter expression[] DATA_QUAL==1 && LAT_CONFIG==1 && ABS(ROCK_ANGLE)<52

Apply ROI-based zenith angle cut[yes] 

Event data file[] 3C279_region_filtered.fits

Output event file name[] 3C279_region_filtered_gti.fits

The data with all the cuts described above is provided in this link.
The renamed spacecraft file is also available.

After the data preparation it is advisable to take a look at your data before beginning the detailed analysis. The Explore LAT data tutorial has suggestions on method of getting a quick preview of your data.

Last updated by: Elizabeth Ferrara 08/06/2010