Week 20: First implementations attempts

Following last week's research, the team attempted to create basic interactions using both openAI (GPT-3) and Bert.

OpenAI Implementation

As mentioned in the last post, OpenAI is not open source, meaning it is not possible to train the neural network any further. Even though, it does offer some interesting ways to tune it through prompts. Here's a simple example taken from OpenAI's website:

The following is a list of companies and the categories they fall into:
Apple, Facebook, Fedex  
Apple Category:

And OpenAI correctly identifies what it is being asked to do, answering:

Technology  

Facebook Category: 
Social Media  

Fedex Category: 
Delivery

Following this example, we trained OpenAI through the prompt by giving it some examples:

Why was my consumption reduced?;explain; consumption"
Should I buy a solar panel?;reccomend; solar panel
...
<sentence>;<intent>;<details>

With only 16 examples, OpenAI was able to properly detect the intent and extra information of over 20 tests, including examples that were never shown, using synonyms and different sentence structures.

The team was shocked by how good the results were in proportion to the amount of work put into it and as such met with the teachers and decided to focus on the Bert implementation, and then comparing its results to OpenAI's.

Bert Implementation

Even though Bert is also a pretrained model, it can't be told what to do via a prompt, and as such needs to be trained. The team is experiment with the BIO slot filling, in order to identify different intents and keywords. This approach does need tens of times more than the 16 train queries OpenAI needed. This way, the group split the work that needs to be done between the following:

Dataset creation

datasets with the respective slot filling - through the use of synonyms maps and paraphrasing libraries, the team is able to generate around 100 examples from each phrase provided.

This still takes a long time to generate and the actual quality of the datasets are yet to be tested.

Bert base implementation

The first iteration of the model that will be trained with the generated datasets. This will be able to receive a query and extract its intents and keywords.

Conversation flow and dialog managing

This is the interface between the language interpretation model and the actual system. Bert is great at identifying intents and keywords, but that information still need to be validated and actually trigger an action.

This module tackles issues such as handling missing information, communicating with the system and making the interaction smoother.