An empirical examination of WISE/NEOWISE asteroid analysis and results
Abstract
Asteroid observations by the WISE space telescope and the analysis of those observations by the NEOWISE project have provided more information about the diameter, albedo, and other properties of approximately 164,000 asteroids, more than all other sources combined. The raw data set from this mission will likely be the largest and most important such data on asteroids available for many years. To put this trove of data to productive use, we must understand its strengths and weaknesses, and we need clear and reproducible methods for analyzing the data set. This study critically examines the WISE observational data and the NEOWISE results published in both the original papers and the NASA Planetary Data System (PDS). There seem to be multiple areas where the analysis might benefit from improvement or independent verification. The NEOWISE results were obtained by the application of 10 different modeling methods, many of which are not adequately explained or even defined, to 12 different combinations of WISE band data. More than half of NEOWISE results are based on a single band of data. The majority of curve fits to the data in the NEOWISE results are of poor quality, frequently missing most or all of the data points on which they are based. Complete misses occur for about 30% of single-band results, and among the results derived from the most common multiple-band combinations, about 43% miss all data points in at least one band. The NEOWISE data analysis relies on assumptions that are in many cases inconsistent with each other. A substantial fraction of WISE data was systematically excluded from the NEOWISE analysis. Building on methods developed by Hanuš et al. (2015), I show that error estimates for the WISE observational data were not well characterized, and all observations have true uncertainty at least a factor of 1.3-2.5 times larger than previously described, depending on the band. I also show that the error distribution is not well fit by a normal distribution. These findings are important because the Monte Carlo error-analysis method used by the NEOWISE project depends on both the observational errors and the normal distribution. An empirical comparison of published NEOWISE diameters to those in the literature that were estimated by using radar, occultation, or spacecraft (ROS) measurements shows that, for 129 results involving 105 asteroids, the NEOWISE diameters presented in tables of thermal-modeling results exactly match prior ROS results from the literature. While these are only a tiny fraction (0.06%) of the asteroids analyzed, they are important because they represent the only independent check on NEOWISE diameter accuracy. After removing the exact matches and adding additional ROS results, I find that the accuracy of diameter estimates for NEOWISE results depends strongly on the choice of data bands and on which of the 10 models was used. I show that systematic errors in the diameter estimates are much larger than previously described and range from - 5% to + 23%. In addition, random errors range from - 15% to + 19% when all four WISE bands were used, and from - 39% to + 57% in cases employing only the W2 band. The empirical results presented here show that much work remains to be done in analyzing data from the WISE/NEOWISE mission and interpreting it for asteroid science.
</ce:displayed-quote></ce:para>These assertions imply that analysis of the WISE/NEOWISE data is complete, at least with respect to diameter, because it is already within or close to the tolerance of the best available comparison data (measurements from radar, occultation and spacecraft observations, denoted here as ROS).</ce:para>Work outside the NEOWISE group to fully understand and exploit the data is still in its early stages, however. Seven years after the initial publication of the NEOWISE calculations, the results have yet to be replicated (i.e., physical properties obtained by model fits performed on the observational data) by any independent group. The question of replicability is important because numerous astronomers have relied on the NEOWISE results to draw conclusions about many salient topics in solar-system science (Bauer et al., 2013; Faherty et al., 2015; Mainzer et al., 2012b, 2012c, 2011e; Masiero et al., 2015a; 2015b, 2013; Nugent et al., 2012; Sonnett et al., 2015). The initial paper by Mainzer et al. (2011a) has been cited at least 270 times, and at least 1400 papers mention or reference NEOWISE, according to a recent query of Google Scholar.</ce:para>NEOWISE results have been applied to study the distribution of asteroid diameter and/or visible-band albedo values (Wright et al., 2016), including the distributions of the size and albedo of near-Earth objects (NEO). Those distributions in turn have strongly influenced discussion about a search for potentially hazardous Earth-impacting asteroids (Grav et al., 2015; Mainzer et al., 2015a; 2015b; Myhrvold, 2016), underscoring the importance of both minimizing and properly characterizing the inevitable error in such estimates.</ce:para>The analysis of broad frequency distributions of asteroid sizes or albedos can tolerate a degree of uncertainty in individual estimates, but systematic biases pose greater problems. Studies of asteroid families or individual asteroids-such as those that apply more detailed and sophisticated thermophysical modeling than the simple thermal models considered here (Alí-Lagoa et al., 2013; Hanuš et al., 2015; Koren et al., 2015)-require higher accuracy and thoroughly characterized errors.</ce:para>In the course of a project to find asteroids within the NEOWISE results which might be fast-tumbling asteroids and thus have a different spectrum than that predicted by the NEATM (Myhrvold, 2016), I attempted to understand and replicate the NEOWISE results. It soon became apparent that the NEOWISE data analysis procedure has not yet been documented in sufficient detail to allow replication. Other inconsistencies were also noticed, leading to the empirical examination presented here.</ce:para>This study is organized into nine principal sections. Section 2 presents a summary of the WISE and NEOWISE data sources used in this study. Section 3 gives an overview of the NEOWISE results, which the NEOWISE group obtained by applying 10 very different modeling methods to 12 distinct combinations of WISE data bands. The accuracy of these methods varies markedly, both among methods and within each method as it is applied to the various band combinations. The NEOWISE results thus must be viewed as a collection of results from disparate methods applied to disparate data resources.</ce:para>Of the 165,794 asteroid physical property results summarized here, more than half of them are based on data from a single WISE band; just 2% make use of all four bands of data, in large part because the NEOWISE analysis systematically discarded large amounts of data.</ce:para>Section 4 reports the discovery that, in 129 cases involving 105 asteroids, the diameter estimates presented in NEOWISE papers as the result of thermal-model fitting exactly match diameter estimates from previous ROS sources. Most of these cases have been omitted from the more recent PDS archive. However, nine such cases, involving six asteroids, remain in the PDS and bear a model code describing them as having a diameter not determined by fitting (i.e., a diameter set as an input to the modeling rather than being an output).</ce:para>Section 5 further develops work by Hanuš et al., (2015) that examined the small percentage of NEOWISE asteroid observations that occur in the regions where successive WISE frames overlap. The successively imaged asteroids provide an ideal data set for measuring the quality of estimated standard error in the resulting flux observations. I expand the analysis of Hanuš et al., which was limited to two WISE bands and 400 cases, to include all four bands and more than 100,000 cases. I find that the WISE pipeline systematically underestimated observational error by a large factor, which varies in magnitude by band. Moreover, the flux errors do not follow a Gaussian (normal) distribution. Examination of WISE observations of ≈ 1 million stars finds that this effect is not confined to asteroids and is not related to detector-edge effects.</ce:para>Section 6 is an analysis of the accuracy of NEOWISE model fits to the WISE data. Surprisingly, I find that nearly a third of single-band NEOWISE curves miss every data point they are supposed to fit in the sense that the residuals for the curve are all the same sign, so the curve passes either above all of the data points or below all of the data points. Of the multiband modeling results, only 48% to 58% pass among data points in each band used (i.e., have residuals that include positive and negative signs or are zero). Those fits that do pass among the data points include many that appear to be of poor quality.</ce:para>Section 7 shows that nearly 15,000 NEOWISE results violate the mathematical definition relating diameter and visible-band geometric albedo to absolute visible-band magnitude H, to a degree greater than can be explained by numerical issues such as rounding. Some of the violations would cause more than a tenfold change in pv or as much as a fourfold change in D-evidence of serious mathematical inconsistency among the NEOWISE models that fit pv. I also show that updating the values of H used in the NEOWISE analysis (presumably from 2010) to 2017 values from the Minor Planet Center (MPC) alters the estimate of pv by 10% or more for over half of the NEOWISE results. Considering that values of H from MPC have a dispersion of ∼0.3 magnitude when compared to precisely measured values (Pravec et al., 2012), this may be the largest source of uncertainty in derived albedo values.</ce:para>Section 8 examines the error analysis used as the basis for the claim in the NEOWISE papers that results are accurate and finds systematic flaws at multiple levels of the analysis.</ce:para>Section 9 compares the accuracy of NEOWISE diameter estimates to ROS results and two prior sets of thermal-modeling results from IRAS and Ryan and Woodward (2010) (abbreviated here as RW). Concordance among the estimates is found to vary strongly with the data bands analyzed and the modeling methods used. Across the various combinations of models and bands included in the PDS, the systematic error ranges from - 7% to + 23%. In addition, the 68.27% confidence interval that accounts for random errors is calculated to span - 13.8% to + 18.8% when all four WISE bands were used, with wider confidence intervals (- 23% to + 27%) when fewer bands were employed.</ce:para>Section 10 discusses these findings and their implications on work that has been based on the NEOWISE results. Details of selected points and methods are presented in the Appendix and in the Supplementary Information (SI) document.</ce:para></ce:section> </ce:displayed-quote></ce:para>That overview omits crucial details necessary to replicate the fits. None of the NEOWISE papers indicate whether ordinary least-squares (OLS) fitting or a χ2 or weighted least-squares (WLS) approach was used, for example. The PDS model codes reveal the parameters that were fit but also raise important questions, discussed below.</ce:para> </ce:displayed-quote></ce:para>While low-level noise and cosmic rays are important to reject, the method described would seem to be ill-suited to either task. Noise and cosmic rays are intended to be handled by the WISE pipeline quality and artifact flags, as well as by the pipeline error estimates. No discussion is offered in Grav/JT:Pre of why those mechanisms are insufficient.</ce:para>Indeed, intrinsic sensitivity issues limit the number of observations in some bands. Taken across the entire FC mission, the W3 band includes the most observations of asteroids. The W1 band has only 21% as many observations as W3, and W2 has 28% as many (see Table A2). The disparities arise in large part from the fact that the WISE detectors and exposure times were not intended only for asteroid observation; instead WISE is a general-purpose survey mission, which by necessity requires compromises (Wright et al., 2010).</ce:para>The strategy of discarding data that does not conform to the 40% rule would, if applied to the entire FC mission data set, eliminate both W1 and W2 entirely. Conversely, those asteroids for which data counts in W1 or W2 exceed 40% of the W3 count are by definition atypical. The use and implications of epochs and the 40% rule are described and discussed more fully in Appendix Section 12.1 and SI Section 4.</ce:para>A further consequence of the decision to restrict analysis to WISE epochs is that the NEOWISE analysis does not benefit from pooling data across mission phases. An asteroid that is observed with a small number of counts in the W1 and W2 bands during the FC mission could, in principle, be supplemented with observations in those bands in the subsequent 3B, PC, and Re mission phases, the last of which continues to the present. The current NEOWISE analysis can only use PC and Re mission data only as new epochs that are analyzed separately.</ce:para>The current state of WISE/NEOWISE data utilization can be viewed as an opportunity for planetary science. The application of new analytical techniques that are able to handle pooled data across epochs, and even across mission phases, would allow significant improvements to our knowledge of asteroids. It may be challenging to do this analysis, but should such techniques be developed, they could more than double the number of asteroids that have the highest-quality modeling approach with all four bands of data and greatly increase the number of asteroids analyzed by full thermal modeling in the sense of Table 4 (i.e., with both W3 and W4 present).</ce:para>- Publication:
-
Icarus
- Pub Date:
- November 2018
- DOI:
- 10.1016/j.icarus.2018.05.004
- Bibcode:
- 2018Icar..314...64M
- Keywords:
-
- Asteroids;
- Near-Earth objects;
- NEATM;
- WISE;
- NEOWISE