How does Fermi LAT data from flaring galaxies get to scientists so fast?

You too can see the latest results and even analyze Fermi Large Area Telescope (LAT) data for yourself - the tools and the data are all public! Now to tell you about some of the steps involved in getting those LAT data from low earth orbit into your computer, ready for you to discover the next big flare from an Active Galactic Nucleus (AGN).

The journey from the spacecraft to your computer

The LAT detects gamma rays almost continuously, with short breaks for a few minutes every orbit. The data are temporarily stored on the spacecraft and every few hours transmitted to the ground via NASA's Tracking and Data Relay Satellite System (TDRSS). TDRSS relays the data to White Sands, New Mexico and from there to NASA's Fermi Mission Operations Center at Goddard Space Flight Center (GSFC) in Greenbelt MD. The LAT data and associated information needed for processing are extracted and sent across the country to the SLAC National Accelerator Laboratory at Stanford University. At this point the intense data processing described below begins. As soon as they are ready, gamma-ray data and the latest results on the gamma-ray sky are made publicly available back at the Fermi Science Support Center at Goddard.

Fermi data path from spacecraft to the public.
Credit: Fermi LAT Collaboration

Large volumes of data to crunch

The LAT collects data at the rate of several thousand frames per second, each one produced by either a gamma-ray or a charged particle. Each frame contains information from close to a million sensors in the instrument. This amounts to a huge data volume. Bandwidth to the ground is limited, so we need to minimize the amount of data to be transferred. Sophisticated processing on-board the spacecraft is used to retain only those frames most likely to be caused by gamma rays. This reduces the rate to a few hundred frames per second. Finally, custom algorithms on-board the spacecraft then compress the data as much as possible before transmission to the ground.

Upon arrival at SLAC, over 1000 computer cores immediately leap into action (pushing other work out of the way if need be!) to turn the raw data into lists of gamma rays and their properties - arrival time, direction and energy - suitable for science. The workflow for this process is very complex as illustrated in the diagram:

Processing chain for Fermi-LAT science data.
Credit: Fermi LAT Collaboration

A day's raw data from the LAT totals about 15 GB. "Data reduction" takes the 15 GB of raw, compressed LAT data and unpacks and expands it by a factor of 50 to a whopping 750 GB of data each day. Further processing identifies the few frames per second that make the final cut as gamma rays. The final information about each of these gamma rays is delivered back to the Fermi Science Support Center, amounting to a mere 200 MB per day.

Exploring the gamma-ray sky

As soon as we have the information about the individual gamma rays, our attention immediately turns to the next stage of data processing - to find out what they can tell us about the recent behavior of the gamma-ray sky. Fermi surveys the sky every few hours, so can spot new or changing gamma-ray sources. It is critically important to identify these as quickly as possible so that telescopes that don't have the luxury of observing the whole sky can be told where to point. The map below shows the most recent (at the time of writing) daily snapshot of the gamma-ray sky, with circles marking new and flaring objects - each one marking the location of extreme astrophysics at work.

Automated software searches for bright sources.
Credit: Fermi LAT Collaboration

Data processing is a 24x7 operation and the goal at SLAC is to "turn around" a data downlink in about an hour. The system is very resilient, but things occasionally can go wrong. A team is charged with monitoring the data processing and fixing problems, and spans the globe with key members at SLAC on the U.S. West Coast, GSFC on the U.S East Coast, Pisa, Italy and now Hiroshima, Japan. The sun doesn't set on Fermi LAT science operations.

Interested in doing your own analysis? Go to https://fermi.gsfc.nasa.gov/ssc/data/ to find the instructions and all the latest Fermi data.

More Links:

Recent announcements from the Fermi-LAT team can be found at https://www-glast.stanford.edu/cgi-bin/pub_rapid.