Fermi Gamma-ray Space Telescope

Using Fermi-LAT All-sky Weekly Files

Some types of data analysis lend themselves to the use of all-sky data files, rather than repeatedly downloading overlapping regions of sky for repeat analysis. Examples are:

  • Light curves and spectra for a large number of sources, whether galactic, extragalactic, or both
  • Characterization of large-scale diffuse emission
  • Analysis of moving sources, like the Sun
  • Searches for transient events (like GRBs) that have an isotropic distribution

This analysis thread describes how to generate an all-sky data set for use in other types of analysis. We will then create an exposure-corrected all-sky image as an example of how to use the weekly files.

Download the weekly LAT data files

The weekly LAT data files are available from the FSSC FTP Server. These files contain the LAT data acquired during each week of Fermi's science mission. (Mission weeks run Thursday through Wednesday, UTC time.) The file for the current mission week will also be listed, even though the week may not yet be complete. The current mission week file is regenerated by the FSSC after every data delivery from the LAT instrument team. This guarantees that the FTP Server and the LAT data server contain the same data, and both are up-to-date.

You can download the LAT weekly files one at a time from the FTP site. Or you can use wget to download a full set of files. Once you have the current set of files, you can use the same wget command (from the same directory) to download only the newest files. In the event that the LAT team reprocesses data for the full mission, the filenames will change, and wget will download a full set of the new, reprocessed, data.

The command to download the full set of weekly photon files is:

> wget -m -P . -nH --cut-dirs=4 -np -e robots=off ftp://legacy.gsfc.nasa.gov/fermi/data/lat/weekly/photon/

LAT data has been classified based on the quality of the event reconstruction. This classification can then be used to reduce background and improve performance by filtering out poorly reconstructed events. The clases, in order of improved event reconstruction are: TRANSIENT (event class = 0), SOURCE (class = 2), CLEAN (class = 3), and ULTRACLEAN (class = 4). Event class 1 is not used. The clean classes are primarily used for analysis of large-scale diffuse structures, that require a lot of time to gain sufficient statistics for analysis, but require very low background. The transient class has a lot of background, but is useful for very short timescale events (like GRBs) where the flux of the event is much brighter than the background for a short period of time.

If your goal is to search for rapid transient events, you will need to download the weekly extended files rather than the photon files. The extended files contain all the events, including the TRANSIENT class; while the photon files have already been filtered to be SOURCE class events or better. Extended files also contain additional information about each photon that may be useful if you are trying to characterize the quality of a particular signal, such as for dark matter line searches.

The command to download the extended weekly files is:

> wget -m -P . -nH --cut-dirs=4 -np -e robots=off ftp://legacy.gsfc.nasa.gov/fermi/data/lat/weekly/extended/

In order to do analysis for the full mission, you will also need the mission-long spacecraft file. This file contains an entries at 30-second intervals during all periods of active data-taking by the LAT instrument. As a result, this file can be rather large. There are also weekly spacecraft files that contain the same information. However, some users have found the pre-generated full mission file to be more robust during analysis than a concatenated set of weekly files. As a result, we recommend re-downloading the full mission file when you update your all-sky data set.

The command to download the mission spacecraft file is:

> wget -m -P . -nH --cut-dirs=4 -np -e robots=off ftp://legacy.gsfc.nasa.gov/fermi/data/lat/mission/spacecraft/

It may be useful to rename the spaccraft file to "spacecraft.fits" for ease of use in other analyses.

Combine the data files

You should now have a full set of weekly files and a mission-long spacecraft file. The next step is to combine the weekly files into a single file. We will use the gtselect tool, which can also be used for filtering data. But we want to combine the full data set without removing any events, so we will use slightly different parameters. To combine files, we first create a file list:

> ls lat_photon_weekly* > filelist.txt

If you look at filelist.txt, you will see it is simply a list of all the files in the directory that start with the string "lat_photon_weekly". You will also want to clear any parameters in gtselect from previous analysis that could inadvertently remove events from the dataset. Clear the gtselect parameter file back to defaults with the command:

> punlearn gtselect

Now you're ready to run gtselect to combine the data files. Here we will use "evclass=0" to insure that gtselect does not filter out transient events, in case they are needed for your later analysis.

> gtselect evclass=0
Input FT1 file [] @filelist.txt
Output FT1 file [] lat_alldata.fits
RA for new search center (degrees) (0:360) [] 0
Dec for new search center (degrees) (-90:90) [] 0
radius of new search region (degrees) (0:180) [] 180
start time (MET in s) (0:) [INDEF] INDEF
end time (MET in s) (0:) [INDEF] INDEF
lower energy limit (MeV) (0:) [100] 30
upper energy limit (MeV) (0:) [300000] 300000
maximum zenith angle value (degrees) (0:180) [] 180

Combining the files takes 10-15 minutes for the full-5-year dataset.

Now that you have mission-long data and spacecraft files, you are ready to generate light curves, search for GRBs, or perform any other type of analysis that requires using data from many regions on the sky. A simple example is described below.

Using the LAT data to produce an all-sky image

Step 1: Remove limb photons

First, filter the data for the proper event class and to remove the earth limb contamination. We will use the SOURCE class, as it has been tuned to balance statistics with background for long-duration point source analysis. We will remove contamination by limiting the reconstructed zenith angle to events at an angle of 100° or less. We will also improve the PSF by excluding events with reconstructed energies below 1 GeV. Run gtselect to filter the data and apply a zenith cut.

> gtselect evclass=2
Input FT1 file[] lat_alldata.fits
Output FT1 file[] lat_source_zmax100_gt1gev.fits
RA for new search center (degrees) (0:360) [] 0
Dec for new search center (degrees) (-90:90) [] 0
radius of new search region (degrees) (0:180) [] 180
start time (MET in s) (0:) [] INDEF
end time (MET in s) (0:) [] INDEF
lower energy limit (MeV) (0:) [] 1000
upper energy limit (MeV) (0:) [] 300000
maximum zenith angle value (degrees) (0:180) [] 100

Next, correct the exposure for the events you filtered out. The way the Fermi Science Tools account for exposure is computed based on the Good Time Intervals (GTIs) recorded in the event file. Because we have eliminated some events, we need to update the GTIs for the filtered file. Even though we have used a zenith cut on the data, we will not correct for it here. For an all-sky analysis, an ROI-based zenith cut would eliminate the entire dataset. Instead, we will later correct the livetime calculation for the zenith angle cut made with gtselect.

The gtmktime tool updates the GTI list based on events we eliminated in the previous step. It also provides the ability to filter on any of the parameters stored in the spacecraft pointing history file. We will use the most basic filter recommended by the LAT team.

> gtmktime
Spacecraft data file [] spacecraft.fits
Filter expression [] (DATA QUAL==1)&&(LAT CONFIG==1)
Apply ROI-based zenith angle cut [] no
Event data file [] lat_source_zmax100_gt1gev.fits
Output event file name[] lat_source_zmax100_gt1gev_gti.fits

Step 2: Bin the data

Now you will bin the data in preparation for exposure correction. In order to correct the map for exposure, you have to account for the changing response with energy. To do this, bin the data into a counts cube using gtbin, but with a single energy bin.

> gtbin
Type of output file (CCUBE|CMAP|LC|PHA1|PHA2|HEALPIX) [] CCUBE
Event data file name [] lat_source_zmax100_gt1gev_gti.fits
Output file name [] lat_source_zmax100_gt1gev_ccube.fits
Spacecraft data file name [] spacecraft.fits
Size of the X axis in pixels [] 3600
Size of the Y axis in pixels [] 1800
Image scale (in degrees/pixel) [] 0.1
Coordinate system (CEL - celestial, GAL - galactic) (CEL|GAL) [] GAL
First coordinate of image center in degrees (RA or galactic l) [] 0
Second coordinate of image center in degrees (DEC or galactic b) [] 0
Rotation angle of image axis, in degrees [] 0
Projection method e.g. AIT|ARC|CAR|GLS|MER|NCP|SIN|STG|TAN: [] AIT
Algorithm for defining energy bins (FILE|LIN|LOG) [] LOG
Start value for first energy bin in MeV [] 1000
Stop value for last energy bin in MeV [] 300000
Number of logarithmically uniform energy bins [] 1

Step 3: Calculate exposure map

Because Fermi data are sparse, the accumulated exposure time for any given point in the sky is based on the observatory's pointing history and the response of the instrument to events at various incidence angles and different energies. The LAT team has calculated how the instrument responds by modeling the entire instrument and then running a MonteCarlo simulation of many millions of events of different energies and directions. These simulations characterize how the gamma rays propagate through an ideal instrument, and the information is recorded in the Instrument Response Functions (IRF). Since the response depends on how much data filtering is performed, there are different IRFs for the different event classes.

The current weekly files use the P7REP_V15 IRFs, and we have selected SOURCE class data. To see which IRFs are available from within the Fermi Science Tools, run the gtirfs command.

> gtirfs
P6_V11_DIFFUSE ( = P6_V11_DIFFUSE::FRONT + P6_V11_DIFFUSE::BACK )
P6_V11_DIFFUSE::BACK
P6_V11_DIFFUSE::FRONT

(...Some output suppressed...)

P7REP_CLEAN_V10 ( = P7REP_CLEAN_V10::FRONT + P7REP_CLEAN_V10::BACK )
P7REP_CLEAN_V10::BACK
P7REP_CLEAN_V10::FRONT
P7REP_CLEAN_V15 ( = P7REP_CLEAN_V15::FRONT + P7REP_CLEAN_V15::BACK )
P7REP_CLEAN_V15::BACK
P7REP_CLEAN_V15::FRONT
P7REP_SOURCE_V10 ( = P7REP_SOURCE_V10::FRONT + P7REP_SOURCE_V10::BACK )
P7REP_SOURCE_V10::BACK
P7REP_SOURCE_V10::FRONT
P7REP_SOURCE_V15 ( = P7REP_SOURCE_V15::FRONT + P7REP_SOURCE_V15::BACK )
P7REP_SOURCE_V15::BACK
P7REP_SOURCE_V15::FRONT
P7REP_TRANSIENT_V10 ( = P7REP_TRANSIENT_V10::FRONT + P7REP_TRANSIENT_V10::BACK )
P7REP_TRANSIENT_V10::BACK
P7REP_TRANSIENT_V10::FRONT
P7REP_TRANSIENT_V15 ( = P7REP_TRANSIENT_V15::FRONT + P7REP_TRANSIENT_V15::BACK )
P7REP_TRANSIENT_V15::BACK
P7REP_TRANSIENT_V15::FRONT
P7REP_ULTRACLEAN_V10 ( = P7REP_ULTRACLEAN_V10::FRONT + P7REP_ULTRACLEAN_V10::BACK )
P7REP_ULTRACLEAN_V10::BACK
P7REP_ULTRACLEAN_V10::FRONT
P7REP_ULTRACLEAN_V15 ( = P7REP_ULTRACLEAN_V15::FRONT + P7REP_ULTRACLEAN_V15::BACK )
P7REP_ULTRACLEAN_V15::BACK

(...Some output suppressed...)

P7ULTRACLEAN_V6 ( = P7ULTRACLEAN_V6::FRONT + P7ULTRACLEAN_V6::BACK )
P7ULTRACLEAN_V6::BACK
P7ULTRACLEAN_V6::FRONT

The appropriate IRF for this data set is P7REP_SOURCE_V154.

Calculating exposure for an all-sky map first requires calculating the instrument livetime for the entire sky (using gtltcube) and then convolving the livetime with the IRF (using gtexpmap2). The pixel size for the livetime does not need to match the binned data, but you do need to provide the event file as gtltcube uses the good time intervals to calculate the livetime at each position.

You will also need to correct the livetime for the zenith angle cut made with gtselect earlier. To do this, use the "zmax" option on the command line and match the value used with gtselect. This modifies the livetime calculation to account for the events you removed earlier.

Here is an example of the input for gtltcube:

> gtltcube zmax=100
Event data file [] lat_source_zmax100_gt1gev_gti.fits
Spacecraft data file [] spacecraft.fits
Output file [] lat_source_zmax100_gt1gev_ltcube.fits
Step size in cos(theta) (0.:1.) [0.025]
Pixel size (degrees) [1]
Working on file spacecraft.fits
.....................!

Now you can now calculate the exposure at each point in the sky in each energy bin (we used only one bin) using gtexpcube. You will need to enter the pixel geometry and energy binning to match what you used in the counts cube. Also, e sure to use "CENTER" for the energy layer reference. If you use "EDGE" you will get an additional energy plane in the output file that causes complications later.

> gtexpcube
Exposure, or Livetime cube input file name [] lat_source_zmax100_gt1gev_ltcube.fits
FT1 events input file name [] lat_source_zmax100_gt1gev_gti.fits
Count map input file name (NONE for manual input of map geometry) [] NONE
Exposure map output file name [] lat_source_zmax100_gt1gev_expcube1.fits
Response function to use. Run gtirfs for a list [] P7REP_SOURCE_V15
Size of the X axis in pixels (leave at 1 for auto full sky (1:) [1] 3600
Size of the Y axis in pixels (leave at 1 to copy nxpix or auto full sky) (1:) [1] 1800
Image scale (in degrees/pixel) [1] 0.1
Coordinate system (CEL - celestial, GAL -galactic) (CEL|GAL) [] GAL
First coordinate of image center in degrees (RA or galactic l) [] 0
Second coordinate of image center in degrees (DEC or galactic b) [] 0
Rotation angle of image axis, in degrees [] 0
Projection method (AIT|ARC|CAR|ZEA|GLS|MER|NCP|SIN|STG|TAN) [] AIT
Start value for first energy bin [] 1000
Stop value for last energy bin [] 300000
Number of logarithmically uniform energy bins [] 1
How are energy layers computed from count map ebounds? (CENTER|EDGE) [] CENTER
Creating an Exposure object from file lat_source_zmax100_gt1gev_ltcube.fits
==> no phi dependence found in ltcube
Using Aeff(s)
Combining exposure from the response function(s), specified by "P7REP_SOURCE_V15":
P7REP_SOURCE_V15::FRONT
P7REP_SOURCE_V15::BACK

==> disabling phi dependence in irfs
==> disabling phi dependence in irfs
cos theta cutoff used: 0
Creating an Image, will write to file lat_source_zmax100_gt1gev_expcube.fits
Layer energy etendue miniumum mean maximum
0 150500 8927 2.89e+10 3.39e+10 3.91e+10

If you open the exposure map with ds9, you will see that the image has two energy planes. The first plane is for 1-300 GeV, the second is for 300 GeV and above. We will only use the first image plane to correct our all-sky map.

Step 4: Correct the all-sky image for exposure and scale

To complete the exposure correction, you must perform a set of arithmetic functions on the all-sky counts cube using the FTOOLS fextract, faith, fimgtrim, and fcarith. First, extract the first of the two exposure planes:

Next, correct the value of each pixel for the exposure:

> farith
Name of 1st FITS file and [ext#] [] lat_source_zmax100_gt1gev_ccube.fits
Name of 2nd FITS file and [ext#] [] lat_source_zmax100_gt1gev_expcube.fits
Name of OUTFIL FITS file [] lat_source_zmax100_gt1gev_corrmap.fits
Operation to perform (ADD,SUB,DIV,MUL(or +,-,/,*),MIN,MAX) [] DIV

Then trim the pixels that were outside the Aitoff projection to zero, so they don't affect your display.

> fimgtrim
Input image file: [] lat_source_zmax100_gt1gev_corrmap.fits
Lower threshold value: [] 0
Lower constant value: [] 0
Upper threshold value: [] INDEF
Output image file: [] lat_source_zmax100_gt1gev_corrmap2.fits

****** successfully exited ******

Finally, scale the image so that the maximum pixel is equal to 255 (the typical maximum dynamic range for graphics). There are a number of ways to find the maximum pixel value. Here we have used ds9 to find a maximum pixel value of 4.143x10-8. This means our scale factor should be 255/4.143x10-8, or 6.155x109.

> fcarith
Name of FITS file and [ext#] [] lat_source_zmax100_gt1gev_corrmap2.fits
Value of the CONSTANT [] 6.155e9
Name of OUTFILE FITS file [] lat_source_zmax100_gt1gev_corrmap3.fits
Enter Operation ADD,SUB,DIV,MUL(or +,-,/,*) [] MUL

You can look at the final product in ds9, using log scaling. This process is approximately the same as is used to create the LAT all-sky images released by the instrument team. This method can be used for different energy ranges, different event class selections, or even to compare different data sets.

Last updated by: Elizabeth Ferrara, 12/29/2013