BLOG

An Efficient Incremental Indexing Mechanism for Extracting Top-k Representative Queries Over Continuous Data-streams

by Sysco LABS Data & Analytics 21 January 2016

The annual ACM/IFIP/USENIX Middleware conference is a major forum for the discussion of innovations and recent advances in the design, construction and use of middleware systems.  The scope of the conference is the design, implementation, deployment, and evaluation of distributed system platforms and architectures for computing, storage, and communication environments. Highlights of the conference includes a high quality single-track technical program, invited speakers, an industrial track, panel discussions involving academic and industry leaders, poster and demonstration presentations, a doctoral symposium, and workshops.

Yasanka Horawalavithana, Software Engineer from CAKE LABS presented his research, titled “An Efficient Incremental Indexing Mechanism for Extracting Top-k Representative Queries Over Continuous Data-streams” at ARM workshop, collocated with ACM/IFIP/USENIX Middleware 2015, Vancover, Canda. Top-k publish/subscribe (pub/sub) models have gained traction as an expressive alternative to extend the binary notion of matching. In Yasanka’s study, the focus is on the problem of extracting the k-most representative set of publications in the dynamic case where the results are updated over a stream of matching publications. This can be observed as the minimum independent dominating set problem in graph theory, when streaming publications are represented as dynamic graph spaces. Due to the inherent complexity of solving this problem over continuous data, an incremental indexing mechanism is proposed for handling a stream of publications. The proposed mechanism is based on Locality Sensitive Hashing (LSH) to avoid the overhead of recalculating neighborhoods over consecutive sliding windows. The experimental results show that the incremental version of LSH indexing mechanism reduces the computational cost of naive greedy approach significantly, while producing Top-k representative results at 70% accuracy compared to the naive optimal method. Further this project has won the Best Computer Science Thesis Award at the University of Colombo convocation last year, sponsored by the Council for Information Technology (CINTEC), Sri Lanka.

View the presentation at:

10 Comments for “An Efficient Incremental Indexing Mechanism for Extracting Top-k Representative Queries Over Continuous Data-streams”

  1. I have been exploring for a little bit for any high quality articles or weblog posts on this sort
    of area . Exploring in Yahoo I finally stumbled upon this site.
    Studying this info So i’m glad to convey that I’ve a very excellent uncanny feeling
    I discovered exactly what I needed. I most undoubtedly will make certain to don?t disregard this website and provides it a look on a continuing basis.

  2. Julia says:

    Most database systems provide mechanisms for exporting or unloading data from the internal database format into flat files. Extracts from mainframe systems often use COBOL programs, but many databases, as well as third-party software vendors, provide export or unload utilities.

  3. It’s great that you are getting thoughts from this piece of writing as well as from our discussion made
    at this time.

  4. Thank you for this article It is indeed very useful

  5. Tremendous issues here. I’m very happy to look your article.

    Thank you so much and I am having a look forward to touch you.
    Will you please drop me a e-mail?

  6. Stephan says:

    Hello, just wanted to say, I loved this post. It was practical.
    Keep on posting!

  7. Hi there, this weekend is good in favor of me, as this time i am reading this
    wonderful educational paragraph here at my house.

  8. Greetings I am so glad I found your site, I really found you by accident,
    while I was browsing on Bing for something else,
    Nonetheless I am here now and would just like to say kudos for a
    fantastic post and a all round interesting blog (I also love
    the theme/design), I don’t have time to read it all at the moment but
    I have bookmarked it and also added your RSS feeds, so when I have time I will be back to read a lot more, Please do keep up the great job.

  9. Bettie says:

    Normally I don’t read post on blogs, but I wish to say that this write-up
    very compelled me to try and do it! Your writing style has been surprised me.
    Thank you, quite great post.

  10. WOW just what I was searching for. Came here by searching for Data analysis
    Middleware
    Top-k

Leave a Comment