ACIS Recipes: Clean the Data

Clean the data (0.5" randomization, low-quality events, bad columns/grades, high background times, and flaring pixels

A variety of improvements to the ACIS data can be made to significantly lower contaminations by cosmic ray events and otherwise improve the dataset. Many (but not all) of these functions are currently performed by CXC in the production of Level 2 event files, however L2 files cannot be used if CTI correction is desired. This section also includes the critical choices of event grades and energies to be used in science analysis. Note: This page does not include recipes to detect telemetry saturation, FEP0 problems, and timestamp errors which may also affect ACIS data.

The most basic cleaning steps are to remove events with non-zero status, undesired grades, and non-physical energies:


dmcopy

Input: acis.evts[EVENTS][status=0, grade=0,2,3,4,6, energy=500:8000]

Output: acis1.evts

Note 1: If an energy range is not specified, be sure to filter the data to exclude events with underflow and overflow energies -- PI=0, 1 and 1024, as recommended in a CXC caveat page.
Note 2: The meaning of the bits in the STATUS column of Level 1 event FITS files are described in pp. 18-20 of the following; ACIS document. Observers should check further if their dataset has a significant number of non-zero STATUS events; this would be nonstandard but is occasionally seen.

An important but oft ignored functionality in the CXC Level 1-2 processing is the addition of a uniform random number ranging from -0.25" to +0.25" to the location of each event in each of CHIPX and CHIPY coordinates. While a technical issue motivated this positional randomization, we believe in most or all cases it unnecessarily blurs the superb point spread function of the Chandra mirrors. The randomization can be removed by running the CIAO program acis_process_events as describe by the CXC thread.
Note 1: It is wise to examine the changes in position produced here (e.g. using IDL) as acis_process_events is a complex program with many parameters, and erroneous results have occasionally been found . Recall that uniform random offsets in chip coordinates does not produce uniform random offsets in sky coordinates, but the mean offset should be zero, the maximum offset should always be sqrt{2}*0.5 pixel, and the same number of events should be present.
Note 2: The acis_process_events program generally recomputes the ENERGY and PI columns and thus must be run BEFORE the CTI corrector.

L. Townsley informs us that forbidden "Bev" grades sometimes show up in ACIS datasets. These are grades 22, 66, 107, 214 and 255 which are supposed to have been rejected on-board prior to telemetry. Their presence may suggest an error in processing (e.g. use of wrong bias file), but the cause is currently unknown. If you find a dataset with many BeV grades, please inform the CXC Helpdesk. A Perl script to search for Bev grades has been written by S. Koch tells the number and fraction of events in the Bev grades:

bev_grades.pl Events.fits THE SEARCH FOR "Bev grades" MUST OCCUR BEFORE CTI CORRECTION, SINCE GRADES ARE RECOMPUTED IN THAT PROCESS!

We remove stripes along Y axis along due amplifier boundaries (mainly Flight Grade=64 events) and hot columns. Most of these selection criteria are presented in a summary of bad events page. We use FTOOL fselect rather than CIAO dmcopy because of the complexity of the selection criteria. Repeated use of dmcopy should also work. We give two versions here: for the four ACIS-I chips and or the single ACIS-S3 chip. New selections are needed for remaining ACIS-S chips, and it is possible that the hot columns and stripes may depend on focal plane temperature and cosmic ray intensity. New hot columns may emerge in the future. Therefore, examine the image carefully and make adjustments in the event selection as needed.

ccd_id==7&&(chipx!=512||(fltgrade!=2&&fltgrade!=64))&&(chipx!=513||(fltgrade!=2&&fltgrade!=64))

Note 1: The selection criteria must be in a continuous line without line-breaks.
Communications with CXC suggest that L1 to L2 processing at the CXC includes the application of GTI tables in the flight timeline (FLT) file. A precise definition of the "bad" observation characteristics that are represented by the FLT file's GTI tables has never been seen by this author.

Note 1

Note 2

WARNING! Until August 2000, dmcopy had a serious bug related to filtering with a GTI table. Attempting to apply the "flt1" GTI tables as described above may be unwise.

In data processed before late-2000, there may be bad time intervals due to brief (10-60 sec) glitches in the aspect offsets. If these are present, these bad time periods can be filtered out using steps 16-18 in the CXC HETG thread. Aspect offsets should be removed in CXC reprocessing. To look for aspect offset glitches in IDL:

fn='acis_aoff1.fits' a = mrdfits(fn,1,COLUMNS=['TIME', 'X_OFFSETS', 'Y_OFFSETS', 'ROLL_OFFSETS']) function_1d,id,a.TIME,a.X_OFFSETS function_1d,id,a.TIME,a.Y_OFFSETS function_1d,id,a.TIME,a.ROLL_OFFSETS

If aspect glitches are found, then a new GTI file generated using dmgti is needed; e.g. with a boolean expression involving X_OFFSETS, Y_OFFSETS, ROLL_OFFSETS, or TIME that describes the "good" time intervals. This expression will be required later in the create_exposure_map script which removes glitches from the aspect offsets file prior to constructing exposure maps. Three Web pages that give examples of using dmgti are here, here, and here.

punlearn dmgti dmgti infile=acis_aoff1.fits outfile=temp.gti userlimit="your boolean expression" clobber=yes

Note 1: Due to a bug in dmgti, it may be necessary to subtract a very small time interval from all STOP times in the resulting file. This can be done as follows:

punlearn fcalc fcalc infile=temp.gti+2 outfile=new.gti clname=STOP expr="STOP-0.00001" clobber=yes

Note 2: It is not clear that sky photons positions are affected by the aspect glitches, in which case it is not necessary (or desirable) to remove these time intervals. This can be checked only with a very strong source in the field.
Note 3: Glitches in aspect are not always outliers with values outside the range of normal values. Sometimes the glitches are when X_OFFSETS, Y_OFFSETS, or ROLL_OFFSETS jump to the value 0.0.
Note 4: Some glitches are of very brief duration (perhaps one sample) and are very hard to spot visually on a plot that covers many kiloseconds!

A simple script, cleanit, has been used at PSU to perform the status, grade, and energy filtering described above, some of the cosmetic filtering described above, and to optionally apply one or two GTI tables (e.g. the flt1 tables and a user-defined aspect glitch GTI table).

USAGE: cleanit acis.evt1.fits [gti1.fits] [gti2.fits]

Some datasets have periods of high background count rates due to solar activity (or proximity to the Earth's magnetosphere?). Several procedures for finding and removing these have been developed:

IDL's Event Browser developed by P. Broos (see the timing recipe page)
CIAO's lightcurve and Prism tools following a CIAO science thread
acisbg, a collection of CIAO tools (lc_clean, make_acisbg, badpixfilter) with Cookbook documentation developed by M. Markevich (CXC).

Note that some observers prefer to examine the light curve of each CCD and declare different good time intervals for each CCD, rather than apply a single set of GTIs to the entire observation (as is done in the CIAO thread above).

A critical type of screening involves removal of flaring pixel events, which are sequences of 2-7 events occurring in sequential readout frames in a single (CHIPX,CHIPY) pixel. They are thought to occur when a particularly energetic (non-solar) cosmic ray particle creates an electron cloud that fills traps that take several seconds to decay. (Note that the initial CR impact was rejected on-board; only the slower release of charge may resemble real X-rays and be included in telemetry.) Experience shows that 2-4% of background events in ACIS-I may be flaring pixels. If they are not removed, they can combine with random background events to produce faint spurious astronomical sources in wavdetect. Fortunately, these spurious sources can be quickly recognized by their physically-implausible variability characteristics (~half of counts occurring within ~20 seconds).

T. Miyaji provides a stand-alone C program flagflare.c which produces a new event FITS file with an additional column flagging flaring pixels It is available in a small package with a Makefile, C program and README file as Flagflare.tar.gz.
Note 1: The observer must decide what criteria should be used to flag and remove flaring pixels. For instance, 2 or 3 events at the same (CHIPX,CHIPY) coordinates in sequential readout frames; 3 or 4 events in non-sequential frames within 10-20 seconds; etc. See 'goflagflare.csh -h' for several flags implemented in Miyaji's code. We have not studied the consequences of various flag levels on real data.
Note 2: Indiscriminate removal of flaring pixels from cosmic ray hits will also remove real source photons from strong sources. Y. Tsuboi has studied this effect in the Orion Nebula field, which has ~1000 sources of different brightnesses. She finds that, using Miyaji's standard flagging on a 50ks exposure, a source with 0.002 cts/s will have 0-4 events flagged, a source with 0.02 cts/s will have 3-100 events flagged, and a source with 0.1 cts/s will have ~1000 events flagged. This is consistent with expectations from the Poisson statistics of photon arrival times. No organized software has been developed to separate the background flaring pixels from these real source events flagged by the flaring pixel algorithm.

It is important to realize that the removal of flaring pixels from a event list will remove real photons from sources as well as spurious events from the cosmic-ray-induced background. A study by Y. Tsuboi indicates that roughly 5% of source photons, even at relatively low flux levels. We therefore recommend that source detection (e.g. wavdetect) be performed on the dataset cleaned of flaring pixels, but source extraction be performed on the dataset prior to flaring pixel removal.

A simple procedure for implementing the removal of flaring pixels is:
/bulk/pkg/xray/bin/goflagflare.csh -h ; read instructions cp *.evt1.fits *_noflarecorr.evt1.fits ; keep uncorrected file mv *.evt1.fits temp_flagflare.fits ; this file acquire new 'flare' column /bulk/pkg/xray/bin/goflagflare.csh temp_flarflare.fits ; run the program fselect ; remove flagged events Input: temp_flagflare.fits Output: *_flarecorr.evt1.fits Selection: flare<1 Run dmstat and ds9 on before/after files to examine changes