Category

Blog

LF AI Partnering with ITU AI/ML in 5G Challenge

By Blog

The LF AI Foundation (LF AI) is excited to be part of the ITU AI/ML in 5G Challenge as a promotion partner. The challenge is focused on finding solutions to relevant problems in 5G through the use of Artificial Intelligence (AI) and Machine Learning (ML). The global challenge theme is “How to apply ITU’s ML architecture in 5G networks”. 

A few words from ITU on the challenge, “ITU invites you to participate in the ITU Artificial Intelligence/Machine Learning in 5G Challenge, a competition which is scheduled to run from now until the end of the year. Participation in the Challenge is free of charge and open to all interested parties from countries that are a member of ITU”. LF AI looks forward to following the challenge and encourages your participation. Learn more by visiting the ITU AI/ML in 5G Challenge website, and submit interest from now until April 30th.

ITU AI/ML in 5G Challenge Resources

LF AI Resources

LF AI Day ONNX Community Virtual Meetup – Silicon Valley 2020

By Blog

IBM, ONNX, and the LF AI Foundation are pleased to sponsor the upcoming LF AI Day* – ONNX Community Virtual Meetup – Silicon Valley 2020, to be held via Zoom on April 9th from 9am-12pm PT.

ONNX, an LF AI Foundation Graduated Project, is an open format to represent deep learning models. With ONNX, AI developers can more easily move models between state-of-the-art tools and choose the combination that is best for them. 

The virtual meetup will cover ONNX Community updates, partner/end-user stories, and SIG/WG updates. Check out the full agenda here. If you are using ONNX in your services and applications, building software or hardware that supports ONNX, or contributing to ONNX, you should attend! This is a great opportunity to connect with and hear from people working with ONNX across many companies. 

Registration is now open and the event is free to attend. Capacity will be 500 attendees. For up to date information on this virtua meetup, please visit the event website

Want to get involved with ONNX? Be sure to join the ONNX Announce and ONNX Technical-Discuss mailing lists to join the community and stay connected on the latest updates. 

Note: In order to ensure the safety of our event participants and staff due to the Novel Coronavirus situation (COVID-19) the ONNX Steering Committee decided to make this a virtual-only event via Zoom.

*LF AI Day is a regional, one-day event hosted and organized by local members with support from LF AI and its Projects. Learn more about the LF AI Foundation here.

ONNX Key Links

LF AI Resources

Pyro 1.3 Now Available!

By Blog

Pyro, an LF AI Foundation Incubation Project, has released version 1.3 and we’re thrilled to see another great release from the community. Pyro is a universal probabilistic programming language (PPL) written in Python and supported by PyTorch on the backend. Pyro enables flexible and expressive deep probabilistic modeling, unifying the best of modern deep learning and Bayesian modeling.

In version 1.3, Pyro adds a variety of improvements, including: 

  • A forecasting module for multivariate hierarchical heavy-tailed time series
  • An AutoNormalizingFlow guide
  • Subsample-compatible AutoNormal and AutoDelta guides
  • A pyro.subsample primitive
  • Four new tutorials
  • A NeuTra example
  • And more…

To learn more about the Pyro 1.3 release, check out the full release notes. Want to get involved with Pyro? Be sure to join the Pyro Announce and Pyro Technical-Discuss mailing lists to join the community and stay connected on the latest updates. 

Congratulations to the Pyro team and we look forward to continued growth and success as part of the LF AI Foundation! To learn about hosting an open source project with us, visit the LF AI Foundation website.

Pyro Key Links

LF AI Resources

LF AI Foundation New Member Welcome – RStudio + inwinStack + ISSIP

By Blog

We are thrilled to welcome three new members to the LF AI Foundation – RStudio, The International Society of Service Innovation Professionals, and inwinStack. Learn a bit more about these organizations:

RStudio 

RStudio joins LF AI as a General member. RStudio open source and enterprise-ready, professional software combines robust and reproducible data analysis with tools to effectively share data products. The flagship professional products RStudio Server Pro, RStudio Connect, and RStudio Package Manager equip professional data science teams to develop and share their work at scale.

inwinSTACK

inwinSTACK joins LF AI as a General member. inwinSTACK is a leading provider of IT solutions based on open source cloud-native softwares. They offer enterprise-grade services of consulting, training, IaaS integration and deployment, system maintenance and support that ensure clients’ success on their digital transformation journey and ability to stay ahead in the rapidly evolving data-driven world of smart, innovative technologies.

ISSIP

ISSIP joins LF AI as an Associate member. The International Society of Service Innovation Professionals, ISSIP (pronounced iZip), is a 501 (C) (3) professional association co-founded by IBM, Cisco, HP and several Universities with a mission to promote Service Innovation for our interconnected world. Their purpose is to help institutions and individuals to grow and be successful in our global service economy.

We look forward to partnering with these new LF AI Foundation members to help support open source innovation and projects within the artificial intelligence (AI), machine learning (ML), and deep learning (DL) space. Welcome to our new members!

Interested in joining LF AI as a member community? Learn more here

LF AI Resources

sparklyr joins LF AI as its Newest Incubation Project – Scaling data science and machine learning workflows using Apache Spark and R

By Blog

The LF AI Foundation (LF AI), the organization building an ecosystem to sustain open source innovation in artificial intelligence (AI), machine learning (ML), and deep learning (DL), today is announcing sparklyr as its newest Incubation Project. sparklyr is an R Language package that lets you analyze data in Apache Spark, the well-known engine for big data processing, while using familiar tools in R. The R Language is widely used by data scientists and statisticians around the world and is known for its advanced features in statistical computing and graphics. 

sparklyr makes using and extending Apache Spark more accessible by providing access to core functionality like installing, connecting and managing Spark and using Spark’s MLlib, Spark Structured Streaming, and Spark Pipelines from R. sparklyr supports connecting to local and remote Apache Spark clusters, provides an interface to Spark’s built-in machine learning algorithms, supports for executing custom R code across Spark clusters, and building multiple extensions to use Spark from R like H2O, XGBoost, GraphFrames, MLeap and more.

“We are extremely pleased to welcome sparklyr to LF AI. Connecting key tools and functionality is critical to growing the open source AI ecosystem, as well as supporting them into long term sustainability under a neutral, vendor-free, and open governance,” said Dr. Ibrahim Haddad, Executive Director of LF AI. “We look forward to fostering new collaborations and possible integrations with existing LF AI projects, such as enabling support for Horovod in R with sparklyr and enabling support to export models and pipelines with ONNX.”

LF AI provides a wide range of services to projects, and the first step is starting as an Incubation Project. Full details on why you should host your open source AI project with us are available here.

“We are very excited to have sparklyr join LF AI and to renew our ongoing commitment to the project. Hosting sparklyr with LF AI within the Linux Foundation enables more organizations to contribute to sparklyr, which benefits the R community by bringing additional talent, ideas, and shared components from other Linux Foundation projects like Delta Lake, Horovod, ONNX, and so on.” said Javier Luraschi, Software Engineer at RStudio, Inc. “Joining LF AI is a big step forward for sparklyr and a great time for you to join this community, contribute, and help sparklyr grow in 2020 and beyond!

sparklyr supports a complete backend for dplyr, a popular tool for working with data frame objects both in memory and out of memory. This enables you to use dplyr code to analyze large datasets in Apache Spark without having to rewrite R code.

For more information on getting involved immediately with sparklyr, please see the following resources.

LF AI Resources

LF AI 2019 Year in Review

By Blog

As we look back to 2019, it’s clear to see that The LFAI Foundation (LF AI) had a year of great momentum and growth, and we couldn’t be more excited for the continued success in the coming year. Check out the 2019 year in review highlights below and if you’re not already involved, we’d love to have you join the LF AI Community!

Members

LF AI members grew by 8 companies who have hit the ground running in their engagement across the Premier, General, and Associate membership levels. We’ve seen a diverse group of companies getting involved across various industries and we welcome those interested in contributing to the support of open source projects within the artificial intelligence (AI), machine learning (ML), and deep learning (DL) space. You can think of LF AI as a greenhouse growing and sustaining open source AI, ML, and DL projects from seed to fruition. Interested in joining LF AI as a member? Learn more here

Projects

The LF AI portfolio of projects increased by 5 new projects, 1 at the Graduate level and 4 at the Incubation level. Projects included ONNX, Pyro, Adlik, sparklyr, and Marquez. We were thrilled to also see Incubation Project Angel move to Graduate level as well. Interested in hosting your open source project with LF AI? Learn more here.

You can also explore how LF AI projects, among many others, fit into the space of open source AI, ML, and DL projects by visiting the LF AI Interactive Landscape

Initiatives

Through the great contributions across the LF AI Community we launched two very important initiatives, an ML Workflow Working Group and a Trusted AI Committee. Both of these initiatives are open for participation and we encourage anyone interested to join the conversations by joining the mail lists or attending an upcoming meeting. 

The ML Workflow Working Group kicked off within the LF AI Technical Advisory Council (TAC) with the goal of defining an ML Workflow and to promote cross project integration. This working group meets bi-weekly on Thursdays, from 7:30-8:00 am PT. Please join the mail list and visit the ML Workflow Wiki to learn more about participating. 

The Trusted AI Committee was also launched and we are seeing continued growth of interest in what is a very hot topic in the AI, ML, and DL space. The committee goals are to create policies, guidelines, tooling, and use cases by industry; among others. This committee meets bi-weekly on Thursdays from 7:00-7:45 am PT. Please join the mail list and visit the Trusted AI Wiki to learn more about participating. 

Community 

Last year we rebranded the foundation from LF Deep Learning to LF AI Foundation and continued efforts throughout the year to increase communication and collaboration within the LF AI Community. If you haven’t connected with us across these channels please do so!

Thank you to all that helped make 2019 such a successful year, it would not have been possible without the broad contributions across the LF AI Community. 

Cheers to wrapping up 2019 and on to 2020!

LF AI Resources

Horovod Version 0.19.0 Now Available!

By Blog

Horovod, an LF AI Foundation Incubation Project, has released version 0.19.0 and we’re thrilled to see the results of their hard work. Horovod is a distributed deep learning framework that improves the speed, scale, and resource utilization of deep learning training.

In version 0.19.0, Horovod adds tighter integration with Apache Spark, including a new high-level Horovod Spark Estimator framework and support for accelerator-aware task-level scheduling in the upcoming Spark 3.0 release. With Horovod Spark Estimators, you can train your deep neural network directly on your existing Spark DataFrame, leveraging Horovod’s ability to scale to hundreds of workers in parallel without any specialized code for distributed training. This enables deep learning frameworks to integrate seamlessly with ETL jobs, allowing for more streamlined production jobs, with faster iteration between feature engineering and model training. 

This release also contains experimental new features including a join operation for PyTorch and the ability to launch Horovod jobs programmatically from environments like notebooks using a new interactive run mode

With the new join operation, users no longer need to worry about how evenly their dataset divides when training. Just add a join step at the end of each epoch, and Horovod will train on any extra batches without causing the waiting workers to deadlock.

Using Horovod’s new interactive mode, users can launch distributed training jobs in a single line of Python. Define the distributed training function, execute it with multiple parallel processes, then return the results as a Python list of objects. This new API mirrors horovod.spark, but can run on any nodes you would normally use with horovodrun.

Full release notes for Horovod version 0.19.0 available here. Curious about how Horovod can make your model training faster and more scalable? Check out these new updates and try out the framework. And be sure to join the Horovod Announce and Horovod Technical-Discuss mailing lists to join the community and stay connected on the latest updates. 

Congratulations to the Horovod team and we look forward to continued growth and success as part of the LF AI Foundation! To learn about hosting an open source project with us, visit the LF AI Foundation website here.

Horovod Key Links

LF AI Resources

LF AI Foundation Announces Graduation of Angel Project

By Blog

Distributed machine learning platform has evolved into a full stack machine learning platform, ready for large scale deployment

SAN FRANCISCO – December 19, 2019 – The LF AI Foundation, the organization building an ecosystem to sustain open source innovation in artificial intelligence (AI), machine learning (ML) and deep learning (DL), is announcing today that hosted project Angel is moving from an Incubation to a Graduation Level Project. This graduation is the result of Angel demonstrating thriving adoption, an ongoing flow of contributions from multiple organizations, and a documented and structured open governance process. Angel has achieved a Core Infrastructure Initiative Best Practices Badge, and demonstrated a strong commitment to community.

Angel is a distributed machine learning platform based on parameter server. It was open sourced by Tencent, the project founder, in July 2017 and then joined LF AI as an Incubation Project in August 2018. The initial focus of Angel was on sparse data and big model training. However, Angel now includes feature engineering, model training, hyper-parameter tuning and model serving, and has evolved  into a full stack machine learning platform.

“With Angel, we’ve seen impressive speed in adding new features and rollout in large corporations at scale. With the 3.0 release of Angel, we have witnessed excellent progress in features, adoption and contributions in a short period of time,” said Dr. Ibrahim Haddad, Executive Director of the LF AI Foundation. “This is a big step forward signaling to the market a maturing open source technology ready for large scale deployment. Congratulations, Angel!”

More than 100 companies or institutions use Angel in products or inside the firewall. The extensive list of implementations includes well-known names like Weibo, Huawei, Xiaomi, Baidu, DiDi, and many more. 

“We are excited to move from Incubation to Graduate Level Project in LF AI, and we see that as just another important milestone in the process, not the end goal. We need to continue to push both technically and with community outreach, to increase momentum, adoption and encourage additional contributions. We will continue to aim for lofty goals,” said Fitz Wang, Senior Researcher at Tencent, Angel Technical Project Lead. “We will be deeply involved in LF AI events in 2020 and present at several events under the LF AI booth. If you’d like to contribute to Angel, please reach out to us via our mailing lists and visit the LF AI booth at any of the LF events.”

Feature Roadmap for 2020

  • Version 3.2 – Graph Computing, adding more algorithms
    • Traditional graph algorithms: Closeness, HyperANF, more
    • Graph Embedding algorithms: Node2Vec, DeepWalk
    • Graph neural network: GraphSAGE
  • Version 3.3 – Federated Learning

Angel Project Resources

LF AI Resources

About LF AI Foundation

The LF AI Foundation, a Linux Foundation project, accelerates and sustains the growth of Artificial Intelligence (AI), Machine Learning (ML) and Deep Learning (DL) open source projects. Backed by many of the world’s largest technology leaders, LF AI is a neutral space for harmonization and ecosystem engagement to advance AI, ML and DL innovation. To get involved with the LF AI Foundation, please visit https://lfai.foundation.

About Linux Foundation 

Founded in 2000, the Linux Foundation is supported by more than 1,000 members and is the world’s leading home for collaboration on open source software, open standards, open data, and open hardware. Linux Foundation projects like Linux, Kubernetes, Node.js and more are considered critical to the development of the world’s most important infrastructure. Its development methodology leverages established best practices and addresses the needs of contributors, users and solution providers to create sustainable models for open collaboration. For more information, please visit us at linuxfoundation.org.

# # #

LF AI Foundation Welcomes ZILLIZ as Premier Member

By Blog

LF AI continues fast pace of membership and project portfolio growth

GPU hardware-accelerated Analytics Platform for Massive-Scale Geospatial and Temporal Data

SAN FRANCISCO – December 17, 2019 – The LF AI Foundation, the organization building an ecosystem to sustain open source innovation in artificial intelligence (AI), machine learning (ML) and deep learning (DL), is announcing today that ZILLIZ has joined the Foundation as a Premier member.

ZILLIZ was founded in 2016 with its headquarter in Shanghai. With the vision of “Reinvent Data Science”, ZILLIZ aligns itself on developing open source data science software leveraging new generation heterogenous computing technologies. Milvus, a high-performance vector search engine for deep learning applications open sourced by ZILLIZ recently, is gathering momentum in the open source AI community.

“We are pushing forward a globalization strategy that fully incorporates global open source communities. We believe open development leads to greater implementation and greater good for all,” said ZILLIZ Founder and CEO Charles Xie. “We think the most critical data challenges today are processing unstructured data which are explosively growing. And even for structured data, we also need new approaches while 5G/IoT applications are gaining dominance in the next decade. We believe open source and open collaboration will foster more innovations to address these challenges.”

“As a pioneer of data science software embracing heterogenous hardware, ZILLIZ is enabling enterprises to transform the unstructured data from digital contents to data assets, which is essential in building high quality AI systems and services.” said Dr. Ibrahim Haddad, Executive Director of the LF AI Foundation. “We are pleased to welcome ZILLIZ as a Premium Member of LF AI and excited to support their contributions connected with the open source AI community including their Milvus project.”

LF AI Project Portfolio Growth

2019 has been a growth year for LF AI, seeing the foundation quickly adding to its portfolio of hosted projects. LF AI currently hosts the following projects: Acumos, Angel, Elastic Deep Learning, Horovod, Pyro, Adlik and ONNX. Two more projects will be joining in December and will be announced at a later date. 

To learn more about hosting a project in LF AI and the benefits, please visit https://lfai.foundation/ and explore the “Projects” main menu item.

A full list of the LF AI hosted projects is available here: https://lfai.foundation/projects/

LF AI Resources

About LF AI Foundation

The LF AI Foundation, a Linux Foundation project, accelerates and sustains the growth of Artificial Intelligence (AI), Machine Learning (ML) and Deep Learning (DL) open source projects. Backed by many of the world’s largest technology leaders, LF AI is a neutral space for harmonization and ecosystem engagement to advance AI, ML and DL innovation. To get involved with the LF AI Foundation, please visit https://lfai.foundation.

About Linux Foundation 

Founded in 2000, the Linux Foundation is supported by more than 1,000 members and is the world’s leading home for collaboration on open source software, open standards, open data, and open hardware. Linux Foundation projects like Linux, Kubernetes, Node.js and more are considered critical to the development of the world’s most important infrastructure. Its development methodology leverages established best practices and addresses the needs of contributors, users and solution providers to create sustainable models for open collaboration. For more information, please visit us at linuxfoundation.org.

# # #

Thank you! LF AI Day Shanghai Summary

By Blog

Organizer: LF AI Foundation
Co-organizer: Huawei, Tencent, Baidu, Alibaba, DiDi, WeBank, Tesra Sponsor: Huawei, Tencent

From Jessica Kim, LF AI Outreach Committee Chairperson: “With China’s first commercial deployment into 5G, the real era of intelligence has arrived, but we still have a lot of technical issues that need to be explored and solved in a practical way, and people from different industries and different technical fields need to work together.

On September 17th, 2019, at the first LF AI Day in China, held at the Huawei Research Institute in Shanghai, senior technical experts from Huawei, Tencent, Baidu, Alibaba Cloud, DiDi, Tesra and Webank gathered to share the applications and practices of AI. Online live-streaming viewing rates exceeded more than 1500 viewers. For folks who missed the whole day live event, Huawei Editor prepared the event summary, to be able to review the wonderful moments of the General Assembly!”