GPU-based minwise hashing

Ping Li, Anshumali Shrivastava, Arnd Christian König

Research output: Chapter in Book/Report/Conference proceedingConference contribution

13 Scopus citations


Minwise hashing [1] is a standard technique for efficient set similarity estimation in the context of search. The recent work of b-bit minwise hashing [3] provided a substantial improvement by storing only the lowest b bits of each hashed value. Both minwise hashing and b-bit minwise hashing require an expensive preprocessing step for applying κ (e.g., κ = 500) permutations on the entire data in order to compute κ minimal values as the hashed data. In this paper, we developed a parallelization scheme using GPUs, which reduced the processing time by a factor of 20 ∼ 80. Reducing the preprocessing time is highly beneficial in practice, for example, for duplicate web page detection (where minwise hashing is a major step in the crawling pipeline) or for increasing the testing speed of online classifiers (when the test data are not preprocessed). Copyright is held by the author/owner(s).

Original languageEnglish (US)
Title of host publicationWWW'12 - Proceedings of the 21st Annual Conference on World Wide Web Companion
Number of pages2
StatePublished - 2012
Externally publishedYes
Event21st Annual Conference on World Wide Web, WWW'12 - Lyon, France
Duration: Apr 16 2012Apr 20 2012

Publication series

NameWWW'12 - Proceedings of the 21st Annual Conference on World Wide Web Companion


Other21st Annual Conference on World Wide Web, WWW'12

All Science Journal Classification (ASJC) codes

  • Computer Networks and Communications


  • GPU
  • Hashing
  • Large-scale learning


Dive into the research topics of 'GPU-based minwise hashing'. Together they form a unique fingerprint.

Cite this