Gradient Flow #18: Forecasting & Groupthink, Interpreting NLP, Ray Ecosystem

Subscribe • Previous Issues

This edition has 640 words which will take you about 4 minutes to read.

“If you want to go fast, go alone. If you want to go far, go together.” - African Proverb.

Data Exchange podcast

  • Using machine learning to modernize medical triage and monitoring systems  Kira Radinsky is Chairwoman & Chief Technology Officer at Diagnostic Robotics, a startup using AI to build a medical-grade triage and clinical-predictions platform. She was one of the pioneers in using alternative data sources to augment forecasting models. Her earlier work includes models to predict social unrest as well as disease outbreaks. Diagnostic Robotics was already building tools for clinical triage and they very quickly were able to tune their systems for the battle against COVID-19.  

  • Connecting Reinforcement Learning to Simulation Software   Max Pumperla, deep learning engineer at Pathmind and a contributor to many open source projects in data science and machine learning (including Hyperopt and Keras). At Pathmind, he is helping bring reinforcement learning to software used for simulations. Many companies already use simulation software and RL lets them tackle more complex real-world scenarios.

[Image: reykjavik, harpa, hall from pikist]

Machine Learning tools and infrastructure


FREE Virtual Conferences

  • Horovod on Ray   Initially released in 2017, Horovod has become one of the most popular open source packages out of Uber. It’s used for scaling deep learning training, specifically data-parallel distributed training.  As of the latest release, Ray can now be used to execute Horovod jobs “without needing to coordinate the workers by hand”. Travis Addair who leads the Horovod project at Uber, will give a talk at the Ray Summit on Distributed Deep Learning with Horovod on Ray.  Horovod and Seldon are among a growing number of popular libraries that have integrations with Ray for distributed execution.

  • Interpreting Natural Language Processing Models   In this recent Simons Institute talk, Yonatan Belinkov of Technion describes much needed new interpretability tools in an age when end-to-end learning using neural models dominates most NLP tasks. He goes through recent research into how linguistic structures (syntax, POS, morphology, …) can be used to understand neural language models. Reminiscent of work in explaining computer vision and speech models, layers of deep language models appear to correspond to a hierarchy of language properties (higher layers correspond to more sophisticated linguistic units). For more on recent research in NLP, the upcoming NLP Summit features progress in speech recognition (Bo Li of Google) and new tools for testing NLP models (Marco Túllio Ribeiro of MSR).


Work and Hiring

[Image by Hillary Wimsatt from Pixabay]

Recommendations


If you enjoyed this newsletter please support our work by encouraging your friends and colleagues to subscribe:


Ben Lorica edits the Gradient Flow newsletter. He is co-chair of the Ray Summit, chair of the NLP Summit, and host of the Data Exchange podcast. You can follow him on Twitter @BigData.  This newsletter is produced by Gradient Flow.

Free Report: 2020 NLP Industry Survey Results

Check out the free report on the 2020 NLP Industry Survey:

The report presents the findings of an online survey which ran for 41 days (July 5 to August 14, 2020). A total of 571 respondents from more than 50 countries completed the survey. A quarter of all respondents hold technical leadership roles. 

To learn more about how organizations are using natural language technologies, what tools they are using, and the challenges they face, download your copy of the report today:

DOWNLOAD


Ben Lorica edits the Gradient Flow newsletter. He is the co-chair of the Ray Summit, chair of the NLP Summit, and host of the Data Exchange podcast. You can follow him on Twitter @BigData

Gradient Flow #17: RL for Recommenders, AI Assurance, Traffic Prediction

Subscribe • Previous Issues

This edition has 880 words which will take you about 5 minutes to read.

“Juggling is sometimes called the art of controlling patterns, controlling patterns in time and space.”  - Ron Graham

Data Exchange podcast

  • What is AI Assurance?  Ofer Razon and Superwise are part of a community in the early stages of building tools and best practices for scaling AI operations. The goal is to help multiple stakeholders build the necessary solutions to evaluate models, receive alerts and troubleshoot on time, and gather insights to improve efficiency.  AI assurance will ultimately bring together different parts of an organization including business, data science and operational teams, legal and compliance, and privacy and security. 

  • Using machine learning to detect shifts in government policy   Weifeng Zhong is the core maintainer of the open source Policy Change Index (PCI), a framework that uses machine learning and NLP to “process and read” large amounts of text to discern government priorities and policies. The initial PCI uncovers major policy shifts in China by text mining the People’s Daily.

[Image: National Library of China from Wikimedia]

Machine Learning tools and infrastructure


FREE Virtual Conferences

  • Learn about Multi Armed Bandits and RL-based Recommender Systems   This is a tutorial I’ve long advocated for.  Earlier this year I wrote a post on enterprise applications of reinforcement learning and one of the areas I highlighted was recommendation and personalization systems. This industry first tutorial, takes place at the Ray Summit and will be led by one of my favorite teachers, Paco Nathan.

  • How to accelerate NLP deep learning model training across multiple GPUs   New language models like BERT are very expensive to train. It cost OpenAI over $12M to train GPT-3! So instead of training from scratch, most companies focus on fine-tuning these models for their specific applications. A team at Determined AI explains how you can reduce the time it takes to fine-tune BERT from 7 hours to 30 minutes, with very minor changes to your model code.  CTO Neil Conway will be speaking on this topic at the upcoming NLP Summit, the industry gathering for people interested in natural language technologies.

  • Call for Speakers (Data+AI Summit Europe) closes Sep 13th. End 2020 by being part of what will be a very strong roster of speakers.


Work and Hiring

[Image: Wildfire smoke, view from Bernal Hill in San Francisco, 2020-09-09; by Ben Lorica]

Recommendations


If you enjoyed this newsletter please support our work by encouraging your friends and colleagues to subscribe:


Ben Lorica edits the Gradient Flow newsletter. He is the Program Chair of the Spark+AI Summit, co-chair of the Ray Summit, chair of the NLP Summit, and host of the Data Exchange podcast. You can follow him on Twitter @BigData.  This newsletter is produced by Gradient Flow.

Gradient Flow #16: Conversational Assistants, Model Compression, Cloud Native

Subscribe • Previous Issues

This edition has 710 words which will take you about 4 minutes to read.

“You may not get rich by using all the available information, but you surely will become poor if you don’t.”  - Jack Treynor

Data Exchange podcast

  • Best practices for building conversational AI applications   Alan Nichol is co-founder and CTO of Rasa, the startup behind the popular open source framework for building conversational AI applications.  We talked about the state of developer tools, as well as software engineering best practices for building chatbots and related applications.

  • Tools for scaling machine learning   Our special correspondent Jenn Webb organized a mini-panel composed of myself and Paco Nathan, author, teacher, and founder of Derwen.ai, a boutique consulting firm specializing in Data, machine learning (ML), and AI.

[Image: Danish Architecture - Aarhus by Alex Berger]

Machine Learning tools and infrastructure

  • One Simple Chart: which industry sectors are using reinforcement learning

  • Compression of Deep Learning Models for Text: A Survey    A few months ago a friend wowed me with the speech recognition model on his Google Pixel phone. It was amazingly accurate, its latency was incredibly low, and it was delivering this in the middle of a loud SF coffee shop.  As much as we laud recent progress in language models, deploying gigantic models is challenging even for high-end servers. This new paper from MSR India surveys state-of-the-art model compression strategies for NLP models.   

  • Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores   A VLDB paper that describes some of the core technological challenges and solutions that set the lakehouse apart from the data lake. The lakehouse pattern was first described in a blog post earlier this year.

  • Practical Data Ethics  This new course from Fast.ai. was originally taught in-person at the University of San Francisco Data Institute at the start of 2020.

  • 5 Levels of Conversational AI   A few years ago I asked the founders of Rasa to explain conversational assistants to a broad audience by devising a framework similar to the “levels of driver assistance technology” used by the NHTSA and the car industry. The original post was widely read within chatbot circles and Alan Nichol just updated it to reflect current technologies and the market as it stands today.


FREE Virtual Conferences

  • Trends in AI and Python Scalability  Dean Wampler and I do a deep dive into two trends that influenced how we put together the Ray Summit program. First, there’s been steady progress towards simplification, efficiency, and lower costs: think of cloud computing, microservices, serverless, and cloud native infrastructure. Second, we were also cognizant of the growing importance of machine learning, particularly of deep learning and reinforcement learning, techniques that bring a host of challenges for developers.

  • NLP: the most important field in machine learning   A short presentation by Clément Delangue, CEO of Hugging Face, whose open-source framework (Transformers) has been downloaded millions of times.  Clément will be giving a keynote at the upcoming NLP Summit.

  • Bias in AI Workshop   This interesting panel discussion was part of a recent workshop organized by the National Institute of Standards and Technology (which is part of the US Department of Commerce).


Work and Hiring

[Image: Exhibit in the Ethnological Museum from Wikimedia]

Recommendations

  • Code for The Economist’s election forecasting model    The US elections are 68 days away, so more data scientists involved in election models are updating their predictions, and some are even publishing their source code.  For a nuanced comparison of predictions from The Economist and FiveThirtyEight, see Andrew Gelman’s recent post.

  • ETL for voter rolls   Data pipelines are everywhere and most organizations are just beginning to put tools and best practices in place to tame them. If a data pipeline breaks down or throws off errors, applications and systems that depend on it are heavily impacted. This article profiles citizens who scrutinize state voter files to help minimize errors during the cleanup process. Using simple data query techniques they have identified thousands of voters mistakenly purged from voter lists. More resources should be allocated to lowering the error rates of voter roll cleanup tools used by state governments. Alas, more attention is placed on electoral fraud which is negligible in the US (see our recent video). 

  • How an AI grading system ignited a national controversy in the U.K.

  • China hires over 100 TSMC engineers in push for chip leadership   If Chinese companies are forbidden from buying critical semiconductors, they can and will build their own local ecosystem. 

  • Asynchronous Reinforcement Learning   A team from Intel Labs and USC introduce an efficient high-throughput reinforcement learning architecture for training agents on a single machine. The goal of this project is to democratize deep RL and make it possible to train "whole populations of agents on billions of environment transitions using widely available commodity hardware".

  • Kintsugi: Japanese art of repairing broken pottery    A beautiful short video from the BBC.


If you enjoyed this newsletter please support our work by encouraging your friends and colleagues to subscribe:


Ben Lorica edits the Gradient Flow newsletter. He is the Program Chair of the Spark+AI Summit, co-chair of the Ray Summit, chair of the NLP Summit, and host of the Data Exchange podcast. You can follow him on Twitter @BigData.  This newsletter is produced by Gradient Flow.

Gradient Flow #15: Technology Adoption, Bias in Speech, Fizz Buzz

Subscribe • Previous Issues

This edition has 710 words which will take you about 4 minutes to read.

“Most people who have the data are in power. And most people who are powerless do not have data.” - Cathy O’Neil

Data Exchange podcast

[Image: The Golden Spiral by IPBri]

Machine Learning tools and infrastructure


Virtual Conferences

  • NLP Summit  The preliminary program is out! Featured speakers include Clément Delangue (CEO at Hugging Face), Piero Molino (creator of Ludwig), Dirk Groeneveld (of AllenNLP), Joel Grus, Kira Radinsky, Amy Heineike and more. Paco Nathan will give a keynote on the results of our NLP Industry Survey. Marco Túllio Ribeiro (of Microsoft Research) will give a talk on a recent project which won a Best Paper Award at ACL 2020.

  • Applications of RL to business process simulation, automation, and optimization    A great overview by Max Pumperla, engineer at Pathmind and maintainer of Hyperopt.

  • Comprehensive Review of Deep Reinforcement Learning Methods and Applications in Economics   This 2020 survey paper lists an amazing number of use cases in stock pricing and investing, insurance, auctions, banking and online markets, macroeconomics, and financial risk management. The upcoming Ray Summit (a Free, virtual conference) has numerous sessions from financial services companies, including a keynote by Manuela Veloso (Head of J.P. Morgan AI Research). Manuela will describe how they use RL in electronic trading models.


Work and Hiring

[Image: Chocolate Hills from Wikimedia.]

Recommendations

  • A Thousand Cuts   This brilliant new documentary about social media disinformation centers around events in the Philippines and attacks against award winning journalist Maria Ressa and her team at Rappler. [Bonus: Official theme song by Ruby Ibarra]

  • Calling Bullshit: The Art of Skepticism in a Data-Driven World   A timely book in the era of increasingly sophisticated disinformation and gaslighting. As the authors describe it “New-school bullshit uses the language of math and science and statistics to create the impression of rigor and accuracy”. This is a must-read for those who have to attend meetings at large organizations.

  • Question → NLP → SQL  Previous Natural Language database query tools only seem to work well during demos. Hopefully this GPT-3 based demo is the start of something better.

  • Bias in machine learning ... speech recognition edition   A new PNAS paper analyzed five state-of-the-art automatic speech recognition models (from Amazon, Apple, Google, IBM, and Microsoft) and found all of them exhibited substantial racial disparities.

  • Most popular YouTuber in each country   A set of graphics that list top YouTube personalities and estimates their annual take (millions of dollars each year for the most popular personalities).


If you enjoyed this newsletter please support our work by encouraging your friends and colleagues to subscribe:


Ben Lorica edits the Gradient Flow newsletter. He is the Program Chair of the Spark+AI Summit, co-chair of the Ray Summit, chair of the NLP Summit, and host of the Data Exchange podcast. You can follow him on Twitter @BigData.  This newsletter is produced by Gradient Flow.

Loading more posts…