How we built a vector-powered fashion assistant

18 September 2020 - 6 mins reading time

We recently challenged ourselves to build a real-world example of Vector AI in-action. Sometimes you have to eat your own dog food.

Vector AI currently powers over 50 million monthly requests. However, we often don't get to be one of those 50 million.

So we kicked off an internal hackathon!

In only three hours we created an AI powered fashion assistant capable of recommending you outfits, all powered by Vector AI. You can check it out here: fashionfiesta.me

Here's how we did it...

Figuring out what to do

The first challenge was to decide what to build.

Our resident frontend and design pro mentioned that his friend really wanted to be able to get outfit recommendations as he never knows what to wear.

So this got us thinking, if we could find data of curated outfits we could use Vector AI’s API to get the items of clothing that it best matched and the items that went with it.

We gave ourselves 3 hours and it seemed like a huge undertaking to be able to build a web app that would understand the piece of clothing the user has and determine what would go with it.

Doing it traditionally we’d have to:

  • build keywords for every type of clothing and map it to whether it’s headwear or footwear
  • build rules based on which colours go together
  • what combinations of clothing are appropriate (we need to know that we shouldn’t recommend you to wear dress shoes with a pair of tracksuit pants)

This has turned into an impossible challenge.

Enter Vector AI. Vectorisation allows us to easily encode the characteristics of each item (colour, category etc.) into numbers ("vectors").

Then we can train a model to predict whether two items go together using a dataset of "matching items" to train our deep learning models.

Vector AI helps us do all of this through a series of API calls. We divided the challenge into three steps:

  1. Getting the data
  2. Designing the app
  3. Using Vector AI

Getting the data

To recommend outfits we needed to find a curated list of clothes, mapped with what goes together with each item of clothing. We figured we'll visit one of our favourite Australian online retailers, The Iconic, and see if they match products that go together.

Screenshot from The Iconic showing their "Wear it with" section (https://www.theiconic.com.au/linen-frill-neck-smock-dress-1116558.html)

On certain products, they have a "Wear it with" section which links to other items that the model is wearing.

We imagine The Iconic has professionals to put these outfits together so we can be comfortable knowing that whatever is in this section is an outfit worth recommending.

So there you go, a library of publicly accessible data. We can also then link the products to the retailer so users can purchase any items they may be missing.

Designing the app

There are a lot of ways we could approach this but considering that selecting an idea and sourcing the data has now taken us 1.5 hours we decided to optimise for time efficiency. The ideal variant would be to vectorise the entire product catalog so that Vector AI can learn what makes outfits go together.

However, we opted for a simpler option where we would focus on the name and description of the product to build associations. Our goal is for a user to enter a query like "blue jeans", and get back products that will go with it.

The flow would look like this:

  1. Allow a user to input a search query (i.e. "blue jeans")
  2. Encode the input as a vector
  3. Send the vector to Vector AI to get the outfits
  4. Visualise an outfit

Using Vector AI

Vector AI makes working with vectors as simple as interacting with an API. Data scientists commonly use vectors to represent complex data objects and interact with them. To learn more about vectors, stay tuned for our blog post on it - or feel free to request access to be the first to know.

To get started we have to insert the data we collected earlier into Vector AI. To do this, we used the Vector AI python SDK. With the SDK we took our data from a JSON file and inserted it into a new collection specifying the fields to be encoded.

from vectorai import ViClient
from vectorai.models.deployed import ViText2Vec
import json

vi_client = ViClient(username='MY_USERNAME', api_key='MY_APIKEY')
text_encoder = ViText2Vec('MY_USERNAME', 'MY_APIKEY')

def insertData():
    data = []
    with open('DATA.json') as json_file:
        data = json.load(json_file)

    for d in data:
        d['_id'] = d['sku']
    
    vi_client.insert_documents(MY_COLLECTION, data, models={'name': text_encoder.encode, 'short_description': text_encoder.encode})
    
insertData()

Once that's done, we can now query our collection. We encode our query into a vector and send a search request. We also apply a filter to ensure we only include data with recommendations.

query = {
    "text": {
        "vector": text_encoder.encode(INPUT_QUERY),
        "fields": [
            "name_vector_",
            "short_description_vector_"
        ]
    }
}

filter = { 
    "field": "hasLooks", 
    "filter_type": "match", 
    "condition": "==", 
    "condition_value": "true" 
}

data = vi_client.advanced_search(MY_COLLECTION, query, filters=[filter], page_size=20)

Putting it all together

We used Vue for the frontend. Our Javascript SDK isn't ready yet, so we will interact with the API directly.

We whipped together a quick UI, even quicker logo and came up with an admittedly catchy name: Fashion Fiesta. Spicy.

With Vector AI we were able to solve a difficult problem using powerful vector-based technology.

And to be honest, it felt great.

We're opening Vector AI for early access. If you're interested in seeing how it can make a difference for you, feel free to request access  - we aim to get back to you within 12 hours.