Virtual Assistant Infrastructure

Cow: Transforming Almond (Jestin Ma, Sawyer Birmbaum)

We investigate the application of Google's "Transformer" model to semantic parsing problems, comparing it with the traditional RNN-based model used to parse Almond commands. Our initial results are promising; the transformer model performs at and, in some case, even slightly above the existing Almond parser while significantly reducing training times.

Exhibit: Data-driven Approach for Generating Automatic GUI Layouts (Richard Yang, Yuguan Xing)

In the pursuit of automatically generating GUIs for virtual assistants, an important component of this task is to identify the locations of where GUI elements should be. Our project attempts to leverage a labeled dataset of existing GUIs and use machine learning models to learn how to predict the locations of elements.

Motif: Simple, Codeless, Demonstration-Based Programming for Creating Customized APIs for the Web (Allan Jiang, Albert Feng)

Motif Automation enables an end-to-end, codeless system to create APIs for any set of web pages, which can later be used from a variety of interfaces, such as a voice-controlled personal assistant. The system has three stages: 1) recording a demonstration, 2) annotating the demonstration, and 3) executing the workflow as an API. Motif converts a typically arduous and manual task into a single API call, which can be created in a couple minutes via demonstration, rather than a couple hours of scripting and coding.

Domain Specific Assistants

Sharpei: Your Go-To Shopper (Jeyla Aranjo)

An automated service for finding dresses at Macys.com, given keywords and price points.

Dora (Eni Asebiomo, Kaylee Bunner, Meghana Rao)

Don't know what to do when visiting a city? Dora can help you. Dora is a travel virtual assistant that plans your day in a new location given a price parameter!

WeSchedule (Tuan Caraballo)

WeSchedule is a platform to optimize and automate administrative operations at outpatient clinics. It leverages AI to compute optimal and realistic medical staff schedules and fair distributions of tasks across all team members. We aim to replace schedulers in the healthcare industry as well reduce the work of clinic managers by at least 50%.

Almondify - Integrating Spotify into Almond by Hemanth Kini, Gabby Wright

Our project provides best-in-class support for natural language control of Spotify. We enable advanced music playback control, playlist modification, and information retrieval features, and demonstrate that Almond can provide superior control of a third-party service over commercial voice assistants.

Friendhub (Matt Millican, Michael Araya, Silei Xu)

Do we really need to rely on Facebook or Twitter to stay connected with our local social groups? We think not. We introduce FriendHub, a concept for a series of user-programmable social hubs curated by users to reinforce the real-world communities they belong to.

Locus (Sam Premutico, Matthew Stewart, Oluremi Oso)

Locus ingests, analyzes, and reports your Google location history. With Locus, you can interface with your Google location data via natural language to learn about your location history, trends, and habits.

Hiya: Managing Construction Project Workflows (Haiyin Wang)

Hiya is a tool enabling construction project managers to handle task workflows between multiple parties and dependency relationships. Its main features include a natural language interface for establishing task dependency based notifications, access control for tasks, and a web application interface to relate tasks to 3D model components of a construction project.