Monday, December 28

Dean and Ghemawat Strike Back on MapReduce

Jeff Dean and Sanjay Ghemawat wrote an article for the January edition of CACM, MapReduce: A Flexible Data Processing Tool. In the article, they refute the findings of A Comparison of Approaches to Large-Scale Data Analysis. On their blog, the authors also wrote a post bashing MapReduce: MapReduce, A major step backwards. The post is no longer available, but thankfully Greg had good coverage.

In the article Dean and Ghemawat address the paper and attempt debunk its claims, although they lack the benchmarks to back it up. In the process, they inform you about the right way to run M/R jobs efficiently:
  1. Avoid starting processes for each new job, reuse workers.
  2. Careful data shuffling, avoid O(M*R) disk seeks
  3. Beware of text storage formats.
  4. Use natural indices like timestamps on files.
  5. Do not merge reducer output.
They present some good M/R lessons in their refutation. You should be using a binary serialization system like Avro or Protocol Buffers and storing your data in a format that provides efficient access, using a natural file structure or using a database system like HBase.

23 comments:

  1. The "MapReduce is backwards" article moved after some time and now has completely disappeared. I suspect it has something to do with Vertica recently partnering with Cloudera and supporting Hadoop connectivity within the Vertica product. Still, doesn't make Stonebraker's writings any less unbiased.

    ReplyDelete
  2. I love your job jobs efficiently: Avoid starting processes for each new job, reuse workers.
    Careful data shuffling, avoid O(M*R) disk seeks
    Beware of text storage formats.
    Use natural indices like timestamps on files.
    Do not merge reducer output.
    It will Be very helpful for me.
    Delhi Escorts Service

    ReplyDelete
  3. Faith Fully Sweet Independent Delhi Escorts Provide Best Escorts Services in Delhi city. Meet our Delhi escorts, escorts in Delhi.
    Delhi Escorts Service
    http://www.delhihotservices.com

    ReplyDelete
  4. Day by day the market of escort is too much popular but just because of money most of the escort company offer cheap class escort services and it regret you from escort service and to make it more advanced Delhi escort added some advanced technique with full of pleasure.
    Delhi escort services

    ReplyDelete
  5. This post is very good for us. It has got a lot of benefit from us.I hope that you will be writing this post again Escorts Service In Mahipalpur

    ReplyDelete
  6. We say that you welcome in Delhi, one of the best cities in the world, which is a fully weird and great city. This is not the clerical center of India, but the people city is the main city of India. Woman from outer space click here for acting as well as for state grits
    Independent Sexy Female Delhi Escorts
    Escorts Service in Delhi
    High Profile Delhi Escorts
    High Class Delhi Escorts Service Independent
    Independent Sexy Call Girls Delhi Escorts Service

    ReplyDelete
  7. Helo,High Class Udaipur Escorts Agency, the luxurious Udaipur Escort Agency where you can find the most beautiful Escorts girls in Udaipur.Female Escorts in Udaipur Independent Escorts in Udaipur

    ReplyDelete
  8. Are you a newbie in love making and don’t know how to start and when to end, than our escort service Delhi could be the best teacher for you.Get in touch with us shortly.
    Escort services in Delhi

    ReplyDelete
  9. This post is very good for us. It has got a lot of benefit from us.I hope that you will be writing this post again.
    Nehru Place Call Girls

    ReplyDelete
  10. Much obliged to you for setting aside the opportunity to distribute this data extremely helpful!
    Escorts in Jaipur
    Chandigarh Escorts

    ReplyDelete
  11. If you have come from the West and you are captivated by the Asian beauty, in that case, we make sure that you get the best Call Girls in Mahipalpur to help you taste the Asian flavor in the best way. At Mahipalpur, we make sure that you are getting one heck of an experience with our girls. So, no matter whether it is official meeting or bachelor party, we are there for all your needs. Visit:- Here

    ReplyDelete
  12. Do you need a Call Girls in Gurgaon to give you the real pleasure of your life, well, if you revert in affirmative, in that case, you are at the right place. We are a stupendous track record of providing the best girls that would completely take the evening of yours by storm, so to avail them all you need to do is just call and we will provide the best at the right price. Visit:- Here

    ReplyDelete
  13. Absolutely wonderful it has lots of thing to choose and very entertaining blog thanks.Send Gifts To pakistan

    ReplyDelete