OpenSFS News

Seagate’s Apache Hadoop on Lustre Connector

By jfranklin

664px-Hadoop_logo.svg

In November last year, Seagate announced their contribution of the “Apache Hadoop on Lustre Connector” as part of their continued commitment to open source communities. Running Apache Hadoop jobs has been made even easier as the Apache Hadoop on Lustre Connector eliminates the need to copy data to the Hadoop Distributed File System (HDFS). The Hadoop on Lustre Connector allows both Hadoop and HPC Lustre clusters to use exactly the same data without having to move data between file systems or storage devices. And Hadoop ecosystem tools like Mahout, Hive and Pig are now able to utilize the Lustre File System.

Importantly, HPC customers are supported through Seagate’s release of source code for a patch to Hadoop that will enable Hadoop to share data with HPC architectures that use Lustre for storage. HPC customers, especially in the Life Science and Energy field, are increasingly using Hadoop and Lustre together as part of their data analysis workflows.

From the press release:
“Seagate believes that direct involvement enabling core capabilities as well as fostering the addition of new application environments is critical to open source community vitality, especially for Lustre which is a foundation for much of the success of high performance computing among science, government and business community leaders. Our work with OpenStack Swift, the Open Compute Project (OCP), OpenSFS, EOFS and now Hadoop is just the beginning,” said Ken Claffey, Vice President of ClusterStor, Seagate Cloud Systems and Solutions. “We are committed to driving open source innovation and partnering with open source communities as they develop cutting-edge enabling technologies that are foundational for the entire industry.”

Seagate’s continued involvement and commitment as an OpenSFS Promoter level member includes being an active board member.

LUG 2015: Presentation Submission Deadline is February 27

By jfranklin

OpenSFS_EOFS_theme-banner_final

The deadline to submit a presentation for the 13th annual Lustre® User Group (LUG) conference is quickly approaching. Don’t miss your chance to share your company’s best practices, case studies, or business experiences with over 200+ Lustre-focused attendees.

OpenSFS relies on the real world experience of the open source Lustre community members to provide valuable sessions to LUG attendees, and we want to hear from you! All LUG presentations will be 30 minutes, with a total of 25 speaking opportunities available.

Submission deadline Extended to February 27, 2015!

To submit an abstract:

OpenSFS is also excited to include panel discussions and poster sessions in the LUG agenda:

  • Suggest a topic for a panel discussion – or be part of a panel and debate the future direction of the industry with leading developers, vendors, and users of Lustre. OpenSFS Administration welcomes your suggestions!

With the ongoing growth and outstanding progress of the Lustre community, LUG 2015 promises to be an exciting and informative event! Registration will open in February and additional event logistics are included on the LUG event page.

Please contact [email protected]   if you have any questions.

OpenSFS Board Meetings Open To Community

By jfranklin

Wondering what goes on inside OpenSFS decision-making? Got an issue or a question? The OpenSFS board meets weekly on Tuesdays and is open to community participation. Openness has been an on-going commitment by OpenSFS. We try to make sure that our decision-making process is transparent. Our technical working groups have long been open to outside participation. Moving into 2015, we are ensuring that board meetings are open to the community as well.

To find out details, dial-in information, and more, please see the Board Meetings and Activities page.

Call for Papers: ORNL-sponsored International Workshop on Lustre Ecosystem

By jfranklin

Oak Ridge National Laboratory (ORNL) is calling for abstracts for the International Workshop on The Lustre Ecosystem: Challenges and Opportunities, March 3-4, 2015 in Annapolis, Maryland, USA.

The workshop is seeking “contributions that explore improvements in the performance and flexibility of the Lustre file system for supporting diverse workloads. This will be a great opportunity for the Lustre community to discuss the challenges associated with enhancing Lustre for diverse applications, the technological advances necessary, and the associated ecosystem.”

Specific topics of interest (so far):

— Workload Characterization
— Adaptability and Scalability of Lustre for Diverse Workloads
— Resilience and Serviceability of Lustre
— Knowledge Provenance in Lustre
— Application-driven Lustre Benchmarking
— Integrating Big Data Technologies with Lustre
— Performance Monitoring Tools for Lustre

Extended abstracts due: January 23, 2015
Acceptance notification: February 12, 2015
Technical presentations: March 4, 2015
Full papers due: March 13, 2015

For more information, please visit the official International Workshop on The Lustre Ecosystem: Challenges and Opportunities page.

We can’t wait to see your submissions!

LWG Meeting Jan 22-23 2015 in Livermore, CA

By jfranklin

The OpenSFS Lustre Working Group (LWG) would like to invite all Lustre software developers to register and attend the two day Lustre Developer Project Meeting in Livermore, CA on January 22-23.

This meeting is by, and for,  for Lustre software developers.  All topics concerning Lustre software developement are welcome.  The agenda will be finalized with input from you.

> Details

SC14 BoF Announced

By jfranklin

Supercomputing 2014 – November 17-20 – New Orleans

Going to  SC’14? So are we! Come join us November 17-20 as we bring supercomputing to New Orleans. OpenSFS will be  hosting a booth on the show floor (#2524) as well as planning additional SC activities. Stay tuned for additional details or contact OpenSFS Administration.

The Birds of a Feather (BoF) session is scheduled for Tuesday November 18, 12:15 to 1:15pm
At this year’s Lustre Community BOF the worldwide community of Lustre developers, administrators, and solution providers will gather to discuss new challenges and corresponding opportunities emerging within HPC and Big Data and how Lustre can continue to evolve to support them.

Tue Nov 18 12:15-13:15  –  Room 275-76-77
Lustre Community BOF: At the heart of HPC and Big Data 

The abstract is the following:

Lustre is the leading open source file system for HPC.  Lustre is the only open-source community developed file system in the top 10 of the top 500 HPC systems. As a result, Lustre is now at the heart of many HPC and Big Data infrastructures supporting multiple sectors from financial services, oil and gas, advanced manufacturing, advanced web services, to basic science. At this year’s Lustre Community BOF the worldwide community of Lustre developers, administrators, and solution providers will gather to discuss new challenges and corresponding opportunities emerging within HPC and Big Data and how Lustre can continue to evolve to support them.

Participants:
Eric Barton, Intel; Chris Morrone, Lawrence Livermore National Laboratory; Nathan Rutman, Seagate Technology; Galen Shipman, Oak Ridge National Laboratory; Hugo Falter, EOFS

http://sc14.supercomputing.org/program/birds-feather-sessions-bofs