Fermi Gamma-ray Space Telescope

Using LAT's New Pass 8 Data

The FSSC is now serving Pass 8 LAT data for analysis. The new version of LAT data provides a number of improvements over the reprocessed Pass 7 data, and is considered the best dataset for all types of LAT analysis. As of the release date (June 24, 2015) reprocessed Pass 7 data is no longer being served. However, existing Pass 7 reprocessed data has been archived and is available from the FSSC's FTP server.

Pass 8 provides a full reprocessing of the entire mission dataset, including improved event reconstruction, a wider energy range, better energy measurements, and significantly increased effective area. In addition, the events have been evaluated for their measurement quality in both position and energy. This allows the user to select a subset of the events if appropriate to improve analysis results. To support the use of these data selections, there have been some structural changes to the Fermitools.

Here we discuss the changes to the data and tools, and how they affect your analysis.

Pass 8 Bottom Line

square inscribed in a circle Pass 8 contains a lot of changes, and the rest of this page may seem overwhelming. If you just want to get started doing a standard LAT analysis, here's the bottom line:

  • Recommended event class for source analysis is "P8 Source" class (evclass=128).
  • Add evtype parameter to your gtselect call (convtype parameter is deprecated). Recommended event type for source analysis is "FRONT+BACK" (evtype=3).
  • Recommended zenith angle cut to eliminate Earth limb events ("zmax") is 90 degrees for events at 100 MeV and above.
  • Recommended source list for analysis is the 3FGL Catalog. A python script is available at the User-Contributed Tools page that creates XML model files using the 3FGL catalog FITS file.

Changes to Data files

  • New data set
    The raw LAT data have been reprocessed using completely new reconstruction code that achieves better performance. Originally motivated by the issue of instrumental pile-up (referred to as ghost events), the new event reconstruction features improvements in many key areas beyond that:
    • New pattern recognition in the tracker reconstruction that does not depend on the calorimeter information and is less sensitive to the track confusion induced by backsplash from the calorimeter. This improvement is significantly more effective for events at high energy and large off-axis angles.
    • Identification of clustering in the calorimeter improves the separation of charge deposition for multiple-particle events.
    • Better energy reconstruction, with improved handling of energy leakage and crystal saturation
    • Improved algorithm for associating tracks with ACD tiles for charged particle rejection.

    The instrument Monte Carlo model has been revised based on the improved understanding of the instrument. From this, the LAT team has generated new event selection cuts that increase the acceptance over the entire LAT energy band. In addition, they no longer eliminate events that have tracks, but do not deposit energy in the calorimeter. This creates a significant increase in acceptance at lower energies. A brief summary of these changes is available in this proceedings paper.

    As a result Pass 8 contains many more events than Pass 7 for a given time span. This increased effective area also corresponds to an increase in absolute background levels, but the signal-to-background ratios have been improved so that the point source sensitivity is improved over the whole LAT energy range. Details are available on the LAT Performance Page.

    The data now include information about position measurement quality and energy measurement quality on an event-by-event basis. This information is encoded in the new EVENT_TYPE column of each Photon or Extended event file.
  • Event classes expanded
    The EVENT_CLASS parameter has been changed from an integer value to a bitfield which can support many more event classes. TRANSIENT classes are now available for several levels of background contamination (IGRB rate, as discussed in the Cicerone). In addition, a new class has been added for analysis of prompt emission from solar flares during periods when there is pile-up in the detector.

    The TRANSIENT classes are:
    • TRANSIENT010 - Background levels at 1x the IGRB rate (from Pass 6 analysis)
    • TRANSIENT020 - Background levels at 2x the IGRB rate (from Pass 6 analysis)
    • TRANSIENT015S - Background levels at 1.5x the Pass 6 IGRB rate, with reduced pileup effects.

    The TRANSIENT classes utilize bits 1-6 of the EVENT_CLASS bitfield, while TRANSIENT015S uses bit 16. The classes for source and diffuse analysis are: SOURCE (bit 7), CLEAN (bit 8), ULTRACLEAN (bit 9), ULTRACLEANVETO (bit 10) and SOURCEVETO (bit 11). As before, the main analysis classes (Source through Ultracleanveto; SourceVeto is nested only to Source) are hierarchical. (Related Cicerone discussion)

    The ULTRACLEANVETO class has been added, and is the recommended class for investigation of the IGRB and diffuse emission at intermediate to high latitudes (b> 10 deg). This class has additional cuts to further reduce residual background, resulting in very low cosmic ray contamination. The SOURCEVETO class has also been added. It is almost identical to SOURCE class below 10 GeV, but above 50 GeV it achieves the same level of background rejection than ULTRACLEAN but with 15% more acceptance.
  • Event types added
    Event types partition the data based on a specific characteristic (e.g., the quality of the direction or energy reconstruction). This allows for analysis methods that take advantage of potentially useful extra information that the reconstruction makes available on an event-by-event basis, information that is effectively lost when averaging over the entire data set.

    In previous LAT data, each event class has been partitioned by conversion type into FRONT and BACK, depending on the location of the tracker layer where the photon-to-pair conversion occurred. Instrument Response Functions for each conversion type were determined separately. This partition was introduced because the PSF of front converting events is intrisically better than the PSF of back converting events. The FRONT/BACK division allows the analysis to treat these two partitions separately, each with its appropriate response functions.

    Pass 8 introduces two additional event type partitions:
    • PSF event type: using an event-level quantity indicating the quality of the reconstructed direction, the data is divided into quartiles, from the lowest quality quartile (PSF0) to the best quality quartile (PSF3)
    • EDISP event type: same as PSF except that the event-level quantity used to partition the data indicates the quality of the energy reconstruction.

    As a result, each event class is partioned in 3 ways:
    • FRONT/BACK
    • PSF0/PSF1/PSF2/PSF3
    • EDISP0/EDISP1/EDISP2/EDISP3

    and the IRFs have been determined for and only for these event types. This means that it is not possible to mix event type selections. (I.e. there are no IRFs corresponding to events that are only FRONT and PSF3, or to events that are only PSF3 and EDISP3.)
  • No diffuse responses in standard data files
    In previous data releases, the diffuse response columns in the events files have been populated with pre-calculated diffuse responses for the most commonly used event classes. The Pass 8 data have been released without filling these response columns, to make the data set more rapidly available to the public. As a result, any unbinned analysis that uses diffuse components will require that you run gtdiffrsp as part of the analysis chain.
    Be aware that running gtdiffrsp is a computationally-intensive process. This will take a significant amount of time to complete. However, the FSSC is providing filtered weekly files with diffuse response columns that may be suitable for some types of analyses.
  • New weekly files
    The LAT Data Server provides Pass 8 photon files for the SOURCE, CLEAN, ULTRACLEAN and ULTRACLEANVETO event classes. The data Server will also provide TRANSIENT events by selecting extended data on the query page. This is the only way to access TRANSIENT events.

    Both photon and extended weekly data files, plus weekly spacecraft files, can be downloaded from the ftp site, or retrieved with wget:

    wget -m -P . -nH --cut-dirs=4 -np -e robots=off http://fermi.gsfc.nasa.gov/FTP/fermi/data/lat/weekly/extended/
    wget -m -P . -nH --cut-dirs=4 -np -e robots=off http://fermi.gsfc.nasa.gov/FTP/fermi/data/lat/weekly/photon/
    wget -m -P . -nH --cut-dirs=4 -np -e robots=off http://fermi.gsfc.nasa.gov/FTP/fermi/data/lat/weekly/spacecraft/

    Pass 7 weekly files created prior to June 25, 2015 have been archived and are available via ftp:
    Pass 7 Reprocessed: http://heasarc.gsfc.nasa.gov/FTP/fermi/data/lat/weekly/p7v6d/
    Pass 7 Updated: http://heasarc.gsfc.nasa.gov/FTP/fermi/data/lat/weekly/p7v6b/
    Pass 7 Original: http://heasarc.gsfc.nasa.gov/FTP/fermi/data/lat/weekly/p7v6/

    No new data (acquired since June 24, 2015) will be processed using Pass 7 reconstruction and classification methods.

Changes to Fermitools and Analysis Methods

  • Fermitools
    A new version of the Fermitools has been released and is available for download. You will need these tools to be able to properly analyze Pass 8 data.
  • Changes to the evclass parameter
    The change to the EVENT_CLASS bitfield necessitates a change to the value of evclass in the gtselect command. The evclass argument accepts the integer representation of the event class bit number (2n, where n is the value of the event class bit). For a standard, P7REP-like analysis, the gtselect command should be:

    prompt> gtselect evclass=128 evtype=3 A list of the bitfield values available for the gtselect command is available in the Cicerone.
  • Addition of the evtype parameter
    Selections on the EVENT_TYPE bitfield can be applied with the evtype parameter. The evtype argument accepts the integer representation of a single event type bit (2n, where n is the value of the event type bit) or set of event type bits (2^n1 + 2^n2 + ...).
    NOTE: With the introduction of the evtype parameter, the use of the convtype parameter in gtselect is now deprecated. Selections on conversion type should be made exclusively with the evtype parameter (using evtype=1 and evtype=2 to select FRONT and BACK events, respectively).
    For a standard, SOURCE-class analysis using both FRONT and BACK events, the gtselect command should be:
    prompt> gtselect evclass=128 evtype=3 A list of the bitfield values available for the gtselect command is available in the Cicerone. Three other tools now also require the use of the evtype parameter:
    • gtdiffrsp - The parameter is only needed if gtselect has not been run prior to running this tool.
    • gtexpcube2 - The parameter is only needed when generating an all-sky exposure map (where no cmap file is specified).
    • gtobssim - The parameter specifies what event types should be simulated. Possible values are "none" for FRONT+BACK, "PSF" for PSF0+PSF1+PSF2+PSF3, or "EDISP" for EDISP0+EDISP1+EDISP2+EDISP3. It is not possible to simulate a single event type.
  • Reduction in backward compatibility
    Changes required for Pass 8 analysis have affected many areas of the Fermitools, resulting in certain limitations when using the new toolset with Pass 7 Reprocessed (or earlier) data sets. Specifically:
    • Pass 6 analysis is completely unsupported.
    • Automatic resolution of IRFs (setting irfs=CALDB) is not supported for Pass 7 analysis. When analyzing Pass 7 data, the IRF name must be given explicitly when running a tool that accepts the "irfs" parameter (for example, set irfs=P7REP_SOURCE_V15 when analyzing source-class data).
    • Pass 7 analysis using FRONT-only or BACK-only events is not supported, as the mechanism for selecting these types has changed. Pass 7 analysis using all events has not been affected.
  • Disabling of energy dispersion
    If you are using data below 300 MeV, you may wish to enable the use of the energy dispersion correction. Be aware, while enabling this correction will provide more accurate results, there is a significant incrase in the amount of processor time required for the analysis. (To understand if this correction may be needed for your analysis, see the discussion of LAT energy dispersion in the Cicerone.) Energy dispersion correction is currently only possible when performing a Binned Likelihood analysis using python (see this analysis thread). PyLikelihood currently supports the following commands for manipulating the application of the energy dispersion correction:
    • Turn on energy dispersion handling - like.logLike.set_edisp_flag(True)
    • Check if edisp handling is turned on - print like.logLike.use_edisp()
    • Turn off energy dispersion handling - like.logLike.set_edisp_flag(False)
    By default, once enabled, energy dispersion correction is enabled for all the components in the XML model. However, energy dispersion correction should not be enabled for components that have already been corrected for energy dispersion, or components that were derived from a fit to the data without energy dispersion correction. This includes the current Pass 8 Galactic diffuse model and the isotropic templates, as well as any point-like or extended sources for which the spectral parameters are fixed to the 3FGL values. This page discusses in detail the reasoning behind the application of an energy dispersion correction in Pass 8 analysis.

    To turn off energy dispersion for a specific source, include an option (apply_edisp="false") in the <spectrum> attribute for that source in your XML model. Here is an example of how to include that option for the Galactic diffuse model:
    <source name="gll_iem_v06" type="DiffuseSource">
    <spectrum type="ConstantValue" apply_edisp="false">
    <parameter free="1" max="10.0" min="0.0" name="Value" scale="1.0" value= "1.0"/>
    </spectrum>
    <spatialModel file="$(FERMI_DIR)/refdata/fermi/diffuseModels/gll_iem_v06.fit" type="MapCubeFunction">
    <parameter free="0" max="1000.0" min="0.001" name="Normalization" scale= "1.0" value="1.0"/>
    </spatialModel>
    </source>
     In gtobssim, the simulation of energy dispersion is enabled by default. The ability to turn off that simulation for a specific source has been added to this version of the Fermitools. Simply add the same option (apply_edisp="false") to the <spectrum> attribute for that source in your gtobssim XML model. <source name="GalacticDiffuse_v06">
    <spectrum escale="MeV" apply_edisp="false">
    <SpectrumClass name="MapCube" params="6.6267, $(FERMI_DIR)/refdata/fermi/diffuseModels/gll_iem_v06.fit"/>
    <use_spectrum frame="galaxy"/>
    </spectrum>
    </source>
  • Removing Earth limb emission
    The LAT team has changed their recommendation for removing contamination due to Earth limb emission from the data. They no longer recommend using a spatial template to accommodate the residual emission. Instead, the LAT team recommends the following:
    • gtselect: Set zmax to 90 degrees
    • gtmktime: Set roicut=no
    • gtltcube: Add zmax=90 as a command line option
    For more information, see the Data Preparation section of the Cicerone.

Changes to Analysis Products

  • Rescaled Galactic diffuse model
    The LAT team has provided a version of the Pass 7 reprocessed Galactic Diffuse model that has been scaled to account for differences in energy dispersion between Pass 7 reprocessed data and Pass 8 data. As always, this model is designed to be used for point source analysis, and is not appropriate for the investigation of medium or large scale diffuse structures within the LAT data. A description of the unscaled Galactic diffuse model is available here. The rescaled model is included in the current release of the Fermitools.
  • Optimization of other analysis products
    As is done for every data release, the LAT team has optimized the Instrument Response Functions (IRFs) and has revised the isotropic diffuse model to accurately reflect the performance of Pass 8 data. The recommended IRFs for Pass 8 data analysis are P8R3_V2, and these have been released with the new version of the Fermitools. The updated isotropic models are also included in the new tools. In addition, you can download the new files from the LAT Background Models web page.