Automating Trend Research: A Technical Deep Dive of Next Trend
Discovering Trends using NLP and Social Proof
Setting Context
Next Trend is a mobile app that helps content creators, entrepreneurs, business owners, and marketers stay updated with their industry's current and potential future trends.
To accomplish this, we developed an end-to-end solution for automating social media monitoring and trend analysis. We use natural language processing (NLP) to analyze high volumes of social media posts to curate the hottest trends on the internet.
Intelligent content analysis helps users leverage social proof for their products and services and improve brand perception. It allows them to validate the real value of their offering and identify and preempt competitive challenges. It also provides actionable insights into their market, potential new markets, and what thought leaders and influencers focus on and predict in your product space.
TLDR; Handpicked tools and techniques to extract viral business, technology, and public interest trends from social media posts. The components in our tech stack have been chosen to facilitate super-fast data extraction, intelligent categorization, and real-time notifications.
Architecture Diagram
Before diving into this architecture, here is a brief overview of our tech stack.
Tech Stack
Next Trend is built on a modern stack that allows for speedy development cycles and is flexible and scalable. This flexibility allows us to experiment freely while developing a system that quickly responds to trend shifts.
Mobile App built using Flutter.
Google’s cross-platform Flutter SDK helped us easily create a user-friendly UI that renders uniformly across devices and OSs and does not rely on native code.
Scraping Infrastructure built using FastAPI, Redis Queue, Redis Cache, and Postgres.
To enhance performance, we chose lightning-quick FastAPI for the web layer, Redis because it’s lightweight and functions as a message broker and cache, and Postgres for storing time-series data. We used the web-queue-worker architecture to decouple long-running workflows like scraping and categorization to run on worker nodes.
NLP API built using spaCy and FastAPI
spaCy is one of the most popular NLP frameworks on the market. It has enabled us to successfully train our own Named Entity Recognizer (NER) model. Our NER can extract the most relevant entities from scraped data. spaCy also offers powerful word vector capabilities for the discovery of similar keywords.
Admin Panel for Content Moderation built using Vue.js.
Vue.js is a complete framework with essential built-in features and awesome style guidelines, allowing us to add new features and functionality on demand.
Fast Reliable Document Storage on Firestore
No-ops Firestore scales automatically, so it is ideal for storing all our trend data. Real-time updates and the ability to handle mobile data across OSs are critical for keeping our customers up-to-date. We also rely on Firestone’s offline support to mitigate the potentially detrimental effects of latency and poor internet connections.
Synchronization using Google Cloud Functions
Google Cloud Functions enable us to create relevant and synchronal notifications for customers about both interesting trends and critical events, enriching the streaming data experience in real-time. We also use cloud functions to synchronize our Firestore database with Algolia.
Github Actions for CI/CD and Automated Releases
We use Github Actions to automate our development, testing, and deployment workflows and support our internal environment for Agile development. Github Actions assists us in testing our codebase and automatically building and deploying our apps and services.
Algolia for Full-Text Search
Algolia addresses some shortcomings associated with database full-text search. Algolia is faster than many similar tools and is typo-tolerant, suggesting trends as the user types in their search. Algolia also allows us to customize criteria for ranking relevance, which helps us curate trends and suggest personalized trend recommendations.
Notifications using Firebase Cloud Messaging (FCM)
We used FCM to enable push notifications for our users. We also created topics that allow users to personalize the type of notifications they receive.
Overall Architecture
Next Trend’s architecture is designed around enabling four core functionalities:
- Extraction
- Categorization
- Moderation
- Notification
These processes provide a completely automated workflow for intelligent data extraction, predictive analysis, and custom reporting for trends found on social media platforms.
All without human intervention.
Extraction and Categorization
In today’s world, it’s tough to tell what the next big trend will be. Trends come and go with surprising regularity, and it can be difficult to sift through the noise and determine which keywords are likely to be major winners. This is one of the reasons why automating extraction and categorization was important for us from the beginning.
Extraction
The central hub of the Next Trend system is the Scraper Worker process. Its main function is to extract keywords and their data and pass it along to the Categorization Worker process. This data is collected from social media platforms, complemented by third-party SEO data and additional information from topic-specific news and blogs. Previously stored data provides insights into historical changes and trend patterns over time.
Sequence Diagram for Automated Keyword Research
We deep dive into extraction in this post:
Categorization
The Categorization Worker’s function is to parse and categorize relevant data passed from the Scraper Worker and filter out winning trends using social media metrics and search interest. These workers also perform housekeeping tasks (for example, storage and backup), define system workflows, and effect a series of deep content analysis processes. We employ sentiment analysis methods and use social media indicators to filter users’ tone and intent intelligently.
Content Moderation and Reporting
Screenshot From our Admin Panel
Newly scraped trends are pushed in real-time to the Next Trend administration interface, where our editors can customize trend data and add editorial comments.
Notifications
The final piece of the puzzle is to send push notifications to users, for example, a new trend in their business niche. Users can customize their notification experience by setting the threshold for receiving notifications and the frequency and category of trends.
Closing Remarks
As a University student, Next Trend was the most challenging project I’ve worked on by far because of the sheer number of things I had to learn along the way. However, it’s also been so rewarding seeing it all come together in the end. I hope this deep dive provides some insight into the bigger picture of how we’re building Next Trend into a complete trend research platform.
If you’re curious about anything I’ve mentioned in this post, feel free to connect with me on LinkedIn and shoot me a message!
