Wednesday, June 10

Hadoop Summit Coverage: State of Hadoop

Presented by Owen O'Malley and Eric Baldeschwieler (Yahoo!)

Owen gave a brief overview of the history of Hadoop and the Hadoop ecosystem.

Yahoo! Hadoop distribution

The big news is that on stage Eric announced the release of the Yahoo! distribution, the distribution that Yahoo! uses internally. It's not new software, but a re-packaging of publicly available releases that have been tested internally on Yahoo!'s clusters. They're starting with the 0.20 release of Hadoop as an Alpha release and this will continue to grow and stabilize.

Yahoo now has dozen's of clusters with 25,000+ nodes. The largest cluster has about 4,000 nodes.

Webmap
Contains content and metadata that powers Yahoo! search.

In 2008
  • 70 hours runtime
  • 300 TB shuffling
  • 200 TB output
In 2009
  • 73 hours
  • 490 TB shuffling
  • 280 TB output
  • 55%+ hardware
Cluster stats
In 2008
  • 2000 nodes
  • 6 PB raw disk
  • 16 TB RAM
  • 16k CPUs
In 2009
  • 4000 nodes
  • 16 PB disk
  • 64 TB RAM
  • 32k CPUs (40% faster cpus)
Major features coming to Hadoop
Core
  • Backwards compatibility (0.21 will make the last big API changes)
  • Append, sync, and flush support
  • Scheduling - Capacity and Fairshare
  • Continuous integration - easier to build and test Hadoop distributions
Pig - "Make pigs fly"
  • Support for SQL and metadata
  • Column oriented storage access layer (a new column-oriented storage view, not services like HBase)
  • Multi-query optimizations
Oozie is a new workflow and scheduling system

There was a question about cluster management. Eric recommend that people use Chukwa and Ganglia as open-source tools for large-scale cluster management.

60 comments:

  1. Hi admin, i went through article. As we all know, .net is most popular programming language and it offer huge career prospects for talented professionals. It’s totally awesome, keep on updating your blog with such awesome information. .Net Training in Chennai

    ReplyDelete
  2. I am reading your post from the beginning, it was so interesting to read & I feel thanks to you for posting such a good blog, keep updates regularly.
    Regards,
    Python Training in Chennai|Python Training Institutes in Chennai

    ReplyDelete
  3. Thanks for sharing this informative blog. If you are interested in taking .net in professional carrier visit this website.Dot Net Training in Chennai

    ReplyDelete
  4. Excellent post!!!. The strategy you have posted on this technology helped me to get into the next level and had lot of information in it.
    salesforce training in chennai | salesforce training institute in chennai

    ReplyDelete
  5. Really informative post for the candidates who seeking this precious information.
    Really,
    Salesforce Training in Chennai|Salesforce Training

    ReplyDelete
  6. Thank you for this valuable information. Get your business to the next level in simple steps. erp in chennai | erp for automotive industry chennai

    ReplyDelete
  7. Thanks for your informative post!!! After completing my graduation, i am confused whether to choose web design as my career. Your article helped me to make a right choice. Web Designing Training Institute in Chennai | Web Designing Training Institute in Velachery.

    ReplyDelete
  8. Interesting and informative article.. very useful to me.. thanks for sharing your wonderful ideas.. please keep on updating..

    PHP Training in chennai | Web Designing Training in chennai

    ReplyDelete

  9. Really an amazing post..! By reading your blog post i gained more information.
    SEO MOBILE MARKETING

    ReplyDelete
  10. Thanks for sharing such a informative guide. Your blog gives me detailed information. Android Training Institute in Chennai | IOS Training Institute in Chennai | Best Java Training in Chennai

    ReplyDelete
  11. Thanks for sharing informative article. Download Windows 7 ultimate for free from getintopc. It helps you to explore full functionality of windows operating system.

    ReplyDelete
  12. Needed to compose you a very little word to thank you yet again regarding the nice suggestions you’ve contributed here.selenium training in bangalore

    ReplyDelete
  13. Thanks for splitting your comprehension with us. It’s really useful to me & I hope it helps the people who in need of this vital information.
    MCA Project Center in Chennai | MCA Project Center in Velachery

    ReplyDelete
  14. I wish to show thanks to you just for bailing me out of this particular
    trouble.As a result of checking through the net and meeting
    techniques that were not productive, I thought my life was done.

    java training in chennai


    java training in bangalore



    java training in bangalore

    ReplyDelete
  15. It has been simply incredibly generous with you to provide openly what exactly many individuals would’ve marketed for an eBook to end up making some cash for their end, primarily given that you could have tried it in the event you wanted.big data training in Chennai

    ReplyDelete
  16. I ‘d mention that most of us visitors are endowed to exist in a fabulous place with very many wonderful individuals with very helpful things.
    dotnet training in bangalore

    ReplyDelete
  17. Thank you a lot for providing individuals with a very spectacular possibility to read critical reviews from this site.
    hadoop-training-in-porur

    ReplyDelete
  18. Very interesting article about Hadoop. Thanks for sharing such a wander full post. Best Cloud Solution Services| Austere Technologies

    ReplyDelete
  19. thanks for giving good information and our massiltechnologies also providing the best services mobile application development and read more details for visit our site Mobile application development Services

    ReplyDelete
  20. Hi, Very good informative article. Thanks for sharing such nice article, keep on up dating such good articles.

    NO.1 APP DEVELOPMENT SERVICES | MASSIL TECHNOLOGIES

    ReplyDelete
  21. VERY HAPPY TO SEE YOUR BLOG. IT'S VERY HELP FULL INFORMATION ,THANKS FOR SHARING.

    ORACLE CLOUD SERVICES FOR APPLICATION DEVELOPMENT | MASSIL TECHNOLOGIES

    ReplyDelete
  22. Great information on this blog, many useful and new ideas i'm learning here.It's very helpful.
    Best Photoshop Training Institute in Chennai | No.1 Adobe Photoshop Training Institute in Velachery

    ReplyDelete
  23. Great and interesting blog..Thanks for sharing this informative article..
    ME/M.Tech Project Center in Chennai | ME/M.Tech Project Center in Velachery

    ReplyDelete
  24. Extraordinary blog. you put Good stuff. All the themes were clarified briefly.Thanks for sharing that post.Keep in blogging…
    Web Designing Training Institute in Chennai | Web Designing Training Institute in Velachery

    ReplyDelete
  25. Nice post.Thanks for your informative post. Your article helped me a lot to understand the future of blog.
    Best AWS Training Institute in Chennai | Best AWS Training Institute in Velachery

    ReplyDelete
  26. Very good informative article. Thanks for sharing such nice article, keep on up dating such good articles.
    MCA Project Center in Chennai | MCA Project Center in Velachery

    ReplyDelete
  27. Your article is awesome! How long does it take to complete this article? I have read through other blogs, but they are cumbersome and confusing. I hope you continue to have such quality articles to share with everyone! I believe there will be many people who share my views when they read this article from you!
    Best CorelDraw Training Institute in Chennai | Photoshop Training Institute in Chennai | No.1 Graphics Designing Training Institute in Velachery

    ReplyDelete
  28. Thanks for sharing such a great blog post. I really like your blog post which is so informative.
    white label website builder

    mobile website builder

    ReplyDelete
  29. wow...Great blog, very help full information. Thanks for sharing, keep updating.

    Best IT Security Services | Austere Technologies

    ReplyDelete
  30. Your article is detailed, thanks to it I solved the problem I am entangled. I will regularly follow your writers and visit this site daily.
    Excellent Summer Courses in Guindy | Best Summer Classes in Chennai

    ReplyDelete
  31. Appreciation for really being thoughtful and also for deciding on
    certain marvelous guides most people really want to be aware of.

    white label website builder

    ReplyDelete
  32. Very interesting , good job and thanks for sharing such a good blog...
    Good Vacation Classes in Chennai | Best Summer Course in Porur

    ReplyDelete
  33. Now a days Hadoop is most significant to learn for future maintenance for large volume of data.

    Hadoop Training in Chennai
    Big Data Training in Chennai

    ReplyDelete
  34. Informative blog... Thanq for sharing your thoughts with us ... Nice work .. keep going

    Selenium training | Selenium training in chennai

    ReplyDelete
  35. I am reading your post from the beginning, it was so interesting to read & I feel thanks to you for posting such a good blog, keep updates regularly.
    Best IT Training Institute in Chennai|Best IT Training Institute in Velachery

    ReplyDelete
  36. Very good informative article. Thanks for sharing such nice article, keep on up dating such good articles.
    Best Commerce College in Hyderabad | Avinash College Of Commerce

    ReplyDelete
  37. Great blog. I have read your blog it's very interesting and informative. Keep sharing...
    CorelDraw Training Institute in Chennai | CorelDraw Courses in Velachery

    ReplyDelete
  38. The best thing is that your blog really informative thanks for your great information!
    InDesign Certification Center in Chennai | No.1 InDesign Training in Keelkattalai

    ReplyDelete
  39. Nice post. IT has clear statement about Hadoop Summit coverage. Thank you.

    Dot Net Training in Chennai
    Dot Net Training Institute in Chennai

    ReplyDelete
  40. Great post! Its give more new information. Thanks for sharing with us.

    Selenium Training in Chennai

    ReplyDelete
  41. Thanks for one marvelous posting! I enjoyed reading it;Great post.The information was very useful...
    Android Certifications Course in Chennai | No.1 Android Exam in Guindy

    ReplyDelete
  42. Thank you for the sharing good knowledge and information its very helpful and understanding.. as we are looking for this information since long time.
    Certified Ethical Hacking Courses in Chennai | No.1 Ethical Hacking in Velachery

    ReplyDelete
  43. Hi. Thank you for Your valuable information. Keep posting useful information.

    Java Course in Chennai
    Java Training Institute in Chennai

    ReplyDelete