Element Distinctness, Frequency Moments, and Sliding Windows
Abstract
We derive new timespace tradeoff lower bounds and algorithms for exactly computing statistics of input data, including frequency moments, element distinctness, and order statistics, that are simple to calculate for sorted data. We develop a randomized algorithm for the element distinctness problem whose time T and space S satisfy T in O (n^{3/2}/S^{1/2}), smaller than previous lower bounds for comparisonbased algorithms, showing that element distinctness is strictly easier than sorting for randomized branching programs. This algorithm is based on a new time and space efficient algorithm for finding all collisions of a function f from a finite set to itself that are reachable by iterating f from a given set of starting points. We further show that our element distinctness algorithm can be extended at only a polylogarithmic factor cost to solve the element distinctness problem over sliding windows, where the task is to take an input of length 2n1 and produce an output for each window of length n, giving n outputs in total. In contrast, we show a timespace tradeoff lower bound of T in Omega(n^2/S) for randomized branching programs to compute the number of distinct elements over sliding windows. The same lower bound holds for computing the loworder bit of F_0 and computing any frequency moment F_k, k neq 1. This shows that those frequency moments and the decision problem F_0 mod 2 are strictly harder than element distinctness. We complement this lower bound with a T in O(n^2/S) comparisonbased deterministic RAM algorithm for exactly computing F_k over sliding windows, nearly matching both our lower bound for the slidingwindow version and the comparisonbased lower bounds for the singlewindow version. We further exhibit a quantum algorithm for F_0 over sliding windows with T in O(n^{3/2}/S^{1/2}). Finally, we consider the computations of order statistics over sliding windows.
 Publication:

arXiv eprints
 Pub Date:
 September 2013
 DOI:
 10.48550/arXiv.1309.3690
 arXiv:
 arXiv:1309.3690
 Bibcode:
 2013arXiv1309.3690B
 Keywords:

 Computer Science  Computational Complexity
 EPrint:
 arXiv admin note: substantial text overlap with arXiv:1212.4372