AddMaple Blog

A Deep Dive into Ngrams, Filters, and Contextual Analysis

Before you roll your eyes at the mention of word clouds, please let us explain why ours are useful with the addition of a few extra tricks. We can all agree that analyzing unstructured text is difficult. And we can agree that free text data could contain hidden insights. For example that open ended survey question might have brought something to your attention you didn't know to ask about in the multiple choice questions. The review data waiting to be explored might help you understand how your customers view your product compared to your competitors. And getting a handle on support data could help you prioritize the next feature because you have a greater understanding of that customer need in the first place

Exploring unstructured free text (e.g. free text responses to a survey question) is difficult. At AddMaple we’ve explored a few different approaches and would like to share why we think interactive word clouds are an extremely powerful tool.

There is a lot of evidence that “conversational surveys” or allowing users to enter free text answers is valuable. Rather than restricting users to choosing a preselected list of answers (which by definition will be limited to what we know at the time), free-text answers allow you to discover what users really think about a topic. As well as surveys, analyzing customer support messages or customer reviews is extremely valuable. Oftentimes a decision is made to leave out open questions because of the analytical challenges they bring but dear reader, we have made this much simpler to tackle. 

At AddMaple we are big believers in thematic analysis (and the role that AI can play in it), however, in this post, we will explain a simpler and quicker way to explore a large free text dataset. We’ll explore n-grams, stemming, and interactive word clouds and we’ll explain why we made the choices that we did.

N-grams are commonly used in natural language processing, here is a brief explanation of what they are:

N-grams: A contiguous sequence of 'n' items from a given text sample. In text, an item often means a word. The 'n' in n-gram denotes how many words are grouped together, aiding in understanding language structure.

Unigrams: Single words. In the phrase "I love ice cream", unigrams are "I", "love", "ice", and "cream".

Bigrams: Pairs of words. From the same phrase, the bigrams are: "I love", "love ice", and "ice cream".

Trigrams: Sequences of three words. Using the aforementioned example, trigrams are: "I love ice" and "love ice cream".

By using n-grams, we can capture linguistic patterns, with unigrams highlighting individual words, while bigrams and trigrams offer contextual insight by examining words in pairs or groups of three.

N-grams, Unigrams, Bigrams, and Trigrams Explained

Traditionally, many analysts have championed the use of bigrams and trigrams when trying to understand the thematic structure of a text. These combinations of two or three words can provide context that a single word - or unigram - might lack. For instance, the phrase "climate change" encapsulates a specific concept, while the words "climate" and "change" separately might not capture the full essence.

But here lies a challenge: What if crucial words that are conceptually linked don't sit right next to each other in a sentence? Take, for instance, a statement like "Climate is experiencing drastic changes due to global activities." The words "climate" and "changes" are separated, and thus wouldn’t be included together in bigrams or trigrams.

Combining trigrams, bigrams, and unigrams using a scoring approach that ranks trigrams the highest, followed by bigrams and finally unigrams. The purpose of this technique is to highlight common bigrams and trigrams as well as unigrams. In the above example, this would have resulted in a bigram of “Stack Overflow”. This approach has some issues, however:
undefinedundefinedundefined

Allowing the user to select between unigrams, bigrams, and trigrams. This gives the user control, however, it still has issues:
undefinedundefined

After exploring the above (and other approaches), we settled on using unigrams but combined with interactive filters.

Word clouds that use unigrams and interactive filters are the most useful in our experience. Here's how it works: when you click on a word, the cloud can adjust to show you other words most frequently associated with your chosen term. This interactivity means that, regardless of whether the words are immediately adjacent to each other or several words apart in the original text, you can gain insights into their contextual relationships.

Imagine being able to tap on "climate" and immediately seeing words like "change," "impact," and "global" becoming more prominent, thus giving you a clear snapshot of the surrounding discourse. With the flexibility to filter down on two or more words, users enjoy the benefits that bigrams or trigrams traditionally offer, but with the added advantage of broader context and real-time exploration.

AddMaple is optimized to apply filters within milliseconds for datasets of up to a million records. The user experience improvement of instant filters is remarkable. It allows quick analysis and exploration of datasets in a way that is just not possible when each filter takes a few seconds to apply. 

 After settling on unigrams, we still had the issue of stemming to consider. 

Why we choose not to use stemming in our interactive word clouds

Stemming has long been a cornerstone in the realm of textual analysis. At its core, stemming aims to reduce a word to its base or root form, allowing words like "answer" and "answers" to be processed as one entity. By consolidating such variations, analysts can gain a clearer picture of the frequency and prominence of core themes within a body of text.

However, like many tools, stemming is not without its complications. For the uninitiated, a word cloud that displays only stems might seem alien and incoherent. Let's take the word "computing" for example. Its stem, "comput", would baffle many, as it doesn't correspond to any commonly used term in English. While "answer" and "answers" are aggregated as "answer", not all words are so accommodating.

To avoid the confusion of only including the stem, we tried displaying the most frequent variant of a stemmed word in the word cloud. On the surface, this seemed like a viable solution. Yet it introduces a few issues. Firstly there are many situations where you do want to see variants of a stem, for example you may not want the noun “researcher” and the verb “researching” to be linked together. In addition it creates these issues when filtering:

Filtering solely on the most frequent variant misses the other variants of the stem

Filtering on the stem alone catches unrelated words, e.g. the stem "operat" could pull in "operation", "operator", and "operate", but could also unintentionally capture words like "operative".

Cleverly filtering on all variants of a stemmed word works but makes text filtering more complex both from a filtering perspective and for the UI. It breaks the simple connection between the text filter box and the text filtering.

In light of these challenges, our approach at AddMaple has been to side-step stemming in favor of harnessing the power of interactive filtering in word clouds. When users can swiftly sift through terms and observe relationships in real-time, the distinction between "answer" and "answers" isn’t a problem. The intuitive nature of this method, coupled with its speed, makes it a good choice for exploring and understanding free text data.

In addition to our decision not to use stemming in our interactive word clouds, we also chose to forgo lemmatization. Lemmatization, like stemming, aims to reduce words to a base or dictionary form. However, it considers the context and converts the word to its meaningful base form, unlike stemming, which often chops off word endings. For instance, "better" is lemmatized to "good" and "running" to "run". While this sounds helpful, especially for linguistic accuracy, it introduces complexities in a word cloud context. The transformed words can sometimes deviate significantly from their original form, potentially confusing users who are trying to connect the visualized words back to their raw data. 

Additionally, the computational demands of lemmatization, especially for large datasets, can impact the responsiveness and simplicity of the user interface, a critical aspect of our design ethos at AddMaple. Thus, while acknowledging the linguistic sophistication of lemmatization, we chose to prioritize user experience and immediacy of insight, steering clear of any processes that could complicate or obfuscate the intuitive understanding of data.

After exploring more complex natural language processing, and observing how these options impacted on the ease with which a user is able to grasp the meaning of a body of text, and the user-interface itself, we settled on keeping things simple. 

The design of user interfaces is as much an art as it is a science. While advanced functionalities are appealing technically, they can also obfuscate what the user is trying to do, muddying functionality, leading to unintended confusion. It's a delicate balance between equipping users with powerful tools and ensuring clarity and intuitiveness. Overcomplicate the interface, and you risk alienating users; oversimplify, and you may not deliver on the promise of deep insights.

This is why at AddMaple, we have centered our approach on the principle of intuitive interaction. We’ve designed our word clouds to look good, be intuitive to work with and be quick to dive into the details and explore connections between words.

We also don’t hide the actual raw free text responses from users - once a dataset has been filtered down using our interactive word clouds it is easy to switch to a table view focussed on the column being analyzed.

In our word clouds, we exclude "stop words," which are common but less informative words like 'the', 'is', and 'on'. This helps highlight more meaningful words, making the data analysis clearer and more relevant. Currently we only support English stop words, but let us know and we can add support for your language.

Load the Data: Begin by loading your dataset into AddMaple. You can load from CSV or SAV files or bring your data in directly from Typeform, Survey Monkey or Google Drive. AddMaple will automatically detect your column types and prepare text columns for unigram based analysis.

Instant Word Cloud Visualization: For each text column, a word cloud is auto-generated. Crafted from unigrams, this cloud disregards stop words and captures the essence of the responses by highlighting the most frequently used terms. Whether respondents are discussing generic concepts or mentioning specific product names, you'll be able to see the key words used at a glance.

Interactive Exploration: Dive deeper into the data by clicking around. Employ the filtering options to narrow down topics or themes. The beauty of AddMaple's interface is the fluidity it offers; see a word you are interested in, click it and instantly see related words. 

Deep Dive with Table View: After honing in on a subset of data through filters, explore the raw data in our table view. By simply clicking on the 'table' link, you're presented with a neatly organized table showcasing the raw responses related to your filters. Each filtered word is highlighted to make it easy to see how they were actually used in the raw data. This focused approach, devoid of other distracting columns, makes it easy to explore lots of raw text responses in an efficient manner.

AddMaple's intuitive design takes you on a journey from overarching themes to the nuanced sentiments of individual respondents. This gradient of insights ensures that while you grasp the bigger picture, you never lose touch with the unique perspectives that often hold the key to understanding your audience better.

If you think we should approach this problem another way, we’d love to hear from you. Designing a data analysis tool that is powerful, but intuitive is a big challenge. Hopefully this post explains the why behind our word cloud interface. The next time you have a dataset with free text responses try exploring it with AddMaple.

Explore how interactive word clouds can be a useful tool for analyzing unstructured text data. In this article we disccuss the technical choices we made in building this feature from ngrams to stop words, stemming and lemmatization.

The Power of Interactive Word Clouds

We're all used to streamed text content from LLM chat APIs. With the release of OpenAI's text-to-speech API we wanted to explore how to stream audio at the same time as text.

A lot of chat APIs use server-sent events. This works well for text based data and there are nice primitives to use in the browser to handle these event streams.

However we wanted to stream text and binary audio data at the same time. One option could have been to use base64 encoding, but this felt hacky and inefficient. It involves extra processing on both the server and the client.

With the help of GPT4 we discovered the multipart/x-mixed-replace content type. This used to be natively supported in browsers for streaming images, e.g. from webcams. Most browsers no longer support it directly for image/video data but you can still use fetch and handle the stream manually.

This is perfect for our use case as it means we can send different content types in a single stream. For GPT4 this can include text, images, audio and JSON.

This article is a walk through of implementing this in a NextJS app - full code here.

This is a quick demo of the code in action, with full duplex audio and multiple simultaneous tool (function) calls.

Demo open ai.mp4

multipart/x-mixed-replace what does it look like?

It is actually simple. You define a boundary string and then separate each part of the stream with that boundary. Each part has a content type header followed by the data (in whatever format you want).

We wanted to start playing audio as soon as text was being streamed back from the chat completions API (rather than wait for the full response). A quick implementation might send each chunk of text to the TTS API. However this would be inefficient and result in very unnnatural audio as often OpenAI will send a single word at a time. So we wrote a simple function that sends a minumum of 30 characters and only full sentences to the TTS API.

Each returned audio stream is then pushed to the client. The end result is a mixed stream that contains both text and audio data. Audio is sent a sentence at a time, while text is streamed a word at a time with new partitions every 5 words. The result is a stream that could look like this:

----MultipartStreamBoundary
Content-Type: text/plain

How can I help you
----MultipartStreamBoundary
Content-Type: text/plain

today?
----MultipartStreamBoundary
Content-Type: audio/mpeg

���� BINARY DATA ����

A longer response would have the text and audio interspersed and contain 100s of text and audio sections.

Sending the boundary string and content-type header after every 5 words of text isn't efficient. Seeing as we are in control of both the client and the server we could probably do something cleverer, however for a demo this works well enough.

Each time the client receives a chunk of data it checks for the encoded boundary string. When it finds a boundary it processes the data buffered so far. It first checks for a content-type header, if it is text data it updates the messages array with the content, if its audio data it queues it up to play and if its JSON data it runs the associated functions.

Here is the (GPT4 generated and lightly edited) code to handle the stream, splitting it at each boundary string. Notice that because we're expecting binary and text data we keep the processing at the binary level rather than converting to text.

const encoder = new TextEncoder()
const boundary = encoder.encode(BOUNDARY)
const response = await fetch(url, {body, headers, method: "POST"})
const reader = response.body.getReader()
let {value, done} = await reader.read()
let buffer = value
if (done) {
  handlePart()
}

while (!done) {
  // Search for the boundary in the buffer
  let boundaryIndex = indexOfUint8Array(buffer, boundary)

  while (boundaryIndex === -1 && !done) {
    // We haven't found the boundary, read more data
    ;({value, done} = await reader.read())
    if (value) {
      buffer = concatUint8Arrays(buffer, value)
      boundaryIndex = indexOfUint8Array(buffer, boundary)
    }
  }

  if (boundaryIndex >= 0) {
    // Process the part we have found
    const part = buffer.slice(0, boundaryIndex)
    handlePart(part)
    // Remove the processed part from the buffer
    buffer = buffer.slice(boundaryIndex + boundary.length + 2) // +2 for the \r\n
  }
}

All the code for processing and handling this data is here.

The actual processing of each chunk of data then looks like this:

const {contentType, content} = getContent(part)
if (contentType === "text/plain; role=user") {
  // transcribed text from an audio recording
  addMessage({role: "user", content, id})
} else if (contentType === "text/plain") {
  addMessage({role: "assistant", content, id}) // handles concatening streamed text
} else if (contentType === "audio/mpeg") {
  const audioBlob = new Blob([content], {type: "audio/mpeg"})
  enqueueAudio(audioBlob)
} else if (contentType === "application/json") {
  const results = await Promise.all(content.map(handleFunction))
  addMessage([
    {role: "assistant", id, content: null, tool_calls: content},
    ...results.map((result, idx) => ({
      role: "tool",
      content: result,
      send: true,
      tool_call_id: content[idx].id,
    })),
  ])
}

In the above code we are dealing with 4 different types of content in just a few lines of code. Compare this to the code that handles receiving function/tool responses in vercel's ai library.

Remarkably playing the audio in the browser doesn't take many lines of code.

const audioBlob = new Blob([rawBytes], {type: "audio/mpeg"})
const audioUrl = URL.createObjectURL(audioBlob)
const audio = new Audio(audioUrl)
audio.play()

We ended up with a fairly simple hook that implements an audio queue where each audio chunk is played sequentially. The end result sounds like a single audio stream.

The newer OpenAI models support calling multiple functions (now called tools) in a single response. Now that we have a way of sending text and audio from the server in a single stream it made sense to also send JSON. There is no point streaming the individual parts of a function argument in the way that they are received from OpenAI. So we decided to buffer up any tool responses and send them as an application/json chunk. This makes the client implementation simpler as it now only receives complete JSON. Arguably its semantically better as well - our stream can now send 3 different types of data each with their own content type.

This demo shows off the power of multiple tools by defining separate functions to change the styling of a circle, square and triangle. It is a lot more powerful being able to ask the LLM to do multiple things at the same time.

Currently DALL-E isn't integrated into the chat completions API, but apparently thats coming soon. When it does there will be another content type we can add to our flexible multipart/x-mixed-replace stream 😊

After finding how easy playing audio is in modern browsers we decided to implement recording as well - complete with silence detection and waveform rendering.

The code needed for recording and sending to the server is super simple. Full code is here, but here is a simplified extract:

const stream = await navigator.mediaDevices.getUserMedia({audio: true})
const recorder = new MediaRecorder(stream, {mimeType: "audio/webm"})
recorder.ondataavailable = (event) => {
  const body = new FormData()
  body.append("file", event.data, "audio.webm")
  fetch(url, {body})
}

This was a fun experiment that we think could be useful for many projects. It seems highly likely that as a tech community we are going to be building a lot of software on top of LLM APIs. It also seems highly likely that more LLMs will become multi-modal AND that no matter the speed improvements there will be many use-cases where you want to stream the data to your application rather than wait for the full response.

Maybe its time for the return of multipart/x-mixed-replace!

ps we're regularly launching AI based features into smart data analysis platform - AddMaple

Exploring audio and text streaming from chat APIs, the use of the multipart/x-mixed-replace content type with multi-modal chat models enables efficient delivery of mixed content types like text, images, audio, and JSON in a single stream.

(ab)using multipart/x-mixed-replace for multi-modal chat streams

There is a treasure trove of interesting data available from the Pew Research Center

From the American Trends Panel Datasets to surveys exploring how we interact and relate to technology.

Some Pew Research data is available for free (not their most recent datasets) but the catch is that they are only available in the .sav (spss) format. This format is designed for IBM SPSS software, which is expensive and hard to use. But fret not. 

If you are keen to explore a SAV file, such as what you can get from Pew, but don't have IBM SPSS, then this guide is for you. We'll show you how to open, analyze, explore and create shareable charts and reports from Pew Research data in just a few clicks with AddMaple.

You might want to bolster your research portfolio, or use a report as a case study that requires you to have a reputable data source. Well, Pew Research Center is considered by many to be a trusted source for data measuring a broad range of issues, attitudes and trends shaping our world. It is a nonpartisan fact tank. Their research spans numerous topics, from politics and media to science and society. Their commitment to public service and open access means that many of their data sets are freely available for download. However, while the data is available, accessing its insights isn't always straightforward for the layperson. That's where AddMaple comes into play.

First things first, why should you use Pew Research data?

Step-by-Step Guide: From Download to Insight

Navigate to Pew Research's Data Download section

Browse the available datasets and choose one that sparks your interest

Sign in or create an account if you're a new user 

In the AddMaple dashboard, select "New Project

Select the .sav file you downloaded from Pew Research and let AddMaple work its magic

Once uploaded, AddMaple will automatically detect the column types of your dataset

Navigate through the provided summaries, charts, and visuals that AddMaple has created for you

Customize and dig deeper into specific data points, filter information, or perform comparative analyses

If you've found an intriguing insight or pattern, use AddMaple's sharing features to send your findings to colleagues or embed them in presentations. Or add them to a report you can publish with AddMaple. 

Creating and sharing interesting charts is quick and intuitive in AddMaple.

For example, we downloaded the Jan 2021 Core Trends Survey from Pew and within a few clicks created an "Age vs Internet Usage" interactive chart.

After creating the chart, we clicked "Share Chart" to get a sharable url to an interactive version of the chart. 

The link to the chart can be embedded in tools like Notion, or shared on Slack with automatic image previews.

Below is an actual embedded version of the interactive chart created above. Click on the kebab icon for interactive elements. Toggle between count view and percentage view. 

We've only scratched the surface of the type of analysis that you can do in AddMaple.

For a more detailed guide to analyzing surveys, check out our full guide.

Learn how to use AddMaple to analyze datasets from Pew Research Center. No need for SPSS or other advanced statistics tools. AddMaple gives you an exploratory chart summary dashboard you can pivot, segment, share, add to a report and more.

How to explore Pew Research datasets that are free to download

There’s a lot we could talk about but I’ll try to keep it brief. We’re closely tracking OpenAI’s advancements and currently use OpenAI in AddMaple to summarize data containing free text (unstructured data) among other things. We have a few additional features coming soon and are convinced that AI coupled with programming genius can fundamentally improve data analysis, making it more efficient, intuitive, and even enjoyable. But why stop there? Why not innovate beyond data analysis and improve how we write and read, or engage with reports too? More on that below. In short, OpenAI’s first DevDay developer conference didn’t unveil GPT-5, but it brought super exciting opportunities.

Longer context support
This is something we’ve grappled with at AddMaple. If you want a LLM to summarize a lot of free text, then the best way is to send it everything in one chunk. However GPT3.5 and 4 maxed out at 32,000 tokens (about 24,000 words). This is still a lot, but wasn’t enough for larger surveys or datasets containing long free-text data, e.g. interviews. While there are existing ways of splitting the text up and creating multiple summaries and then summarizing the summaries, this was slow and introduced other problems.

The new GPT4 Turbo (vroom 🏎 )now supports about 96,000 words or 128,000 tokens. This is more than a typical novel which contains about 80,000 words. While pretty expensive, it unlocks many more use cases (and kills Anthropic’s main USP…)

GPTs and the Assistants API with knowledge retrieval and persistent threads
A large number of AI startups were effectively threatened yesterday with this feature. Without getting too much into the weeds of the tech, OpenAI has created their first version of “AI Agents”. These agents are slightly different to the ChatGPT interface where you ask it a question and it generates a response. Agents can essentially perform more advanced tasks.

Here are a few ideas of what can be created with this feature (generated by ChatGPT of course…)

A Custom Family Cooking Assistant: rather than stop with ChatGPT’s ability to generate recipe ideas, you could have a more advanced cooking assistant that is tailored to your family’s dietary preferences, what you have in the fridge and even the kitchen gadgets that you own/enjoy using.

An Intelligent DJ That Gets Your Music Taste: We use Spotify A LOT at home, especially their auto-generated playlists. But there are 2 huge problems. Firstly, it sometimes suggests styles of music that I really don’t like, and I can’t easily stop Spotify from playing those. And secondly, Spotify sometimes loses track of, tracks I’ve heard on different playlists within a short period. I also don’t really want to hear a song more than 2 or 3 times in a week regardless of playlist. It should be fairly easy to create an assistant that can fix these issues but can still help me discover more music.

A Legal Precedent Researcher: Were you to be in possession of a hefty database containing previous legal cases, you could benefit from a Legal Assistant that can help you find relevant cases containing similar precedents to the one you’re working on currently.

From a business perspective, OpenAI is really interesting. They are going after enormous markets simultaneously and by all accounts are succeeding in all of them!

D2C: ChatGPT which is already boasting nearly $1 billion annual revenue

Custom Models: which allows enterprises and well-funded startups to train their own models with pricing starting between $2–3 million: https://openai.com/form/custom-models

API Platform: OpenAI keep reducing their API pricing, enticing developers from big and small companies alike to build on top of their services. Some of the new features they’ve offered start bringing an element of “lock-in” which will make it even harder for their competitors.

Revenue-sharing and Incentivization: OpenAI announced its launch of a marketplace for AI assistants within ChatGPT, offering developers a slice of the revenues they generate.

I’m interested in what you took away from the announcements yesterday?

For data analysis and AddMaple specifically these are some of the ideas we are exploring:

An Insights Assistant to support users with their data analysis:

Our current capabilities— AddMaple detects column types and converts raw data into chart summaries automatically without requiring upfront cleaning/preparation. Users can near-instantly segment/pivot and share these charts individually or add them to a report for their audience to explore.

But with the updates to OpenAI, an Insights Assistant could interpret survey results from AddMaple’s visualizations and could provide natural language summaries of key findings to speed up report writing. For instance, once AddMaple visualizes customer satisfaction across different regions or products, the assistant could automatically generate a text summary detailing which regions/products received the highest and lowest satisfaction.

Once AddMaple reads the dataset provided and provides in-browser summaries, we could allow users to could ask a Query Assistant specific questions about their survey data, such as, “Which features received the highest satisfaction ratings?”. The assistant could process this query, retrieve the relevant visualizations from AddMaple, and could then further assist the user by providing additional verbal or written summaries of the charts. Our published reports are currently explorable too (this report for Hacker News allows us, the authors, to select columns that our readers can segment/pivot/share. With OpenAI’s announcement, our reports could be queried using natural language assistants too. Not everyone reading a report wants to read. What if they could query the report and the published columns using natural language? Sounds like a great opportunity to me.

There’s a lot we could talk about but I’ll try to keep it brief. We’re closely tracking OpenAI’s advancements and currently use OpenAI in AddMaple to summarize data containing free text (unstructured data) among other things.

OpenAI’s first developer conference happened. Here’s what we thought of DevDay

Read the latest news and updates from the AddMaple team. We'll share our thoughts on data visualization, data analysis, and everything in between.