Introduction

I have been thinking about the topic of trend research for some time. How could it be possible to detect deep trends from a scientific perspective, to catch early signs that could be an indicator for trends with big economic or societal potential? First i decided to use some common and modern technologies to build a dashboard that worked the following way:

I used a Windmill instance to build scripts that fetched articles, scientific papers and news from multiple sources. I normalised them via a script and created embeddings (1536) for each resource, then saved the embeddings inside a Qdrant vector database. The next challenge was how to extract meaningful information out of the collected data, so i decided to use an algorithm to create clusters based on semantic meaning. The algorithm i used was HDBSCAN. Based on the semantic meaning, my script started creating the clusters every day, while the resources were collected every four hours.

Info: there were a lot more steps in between. This is just a short description.

The result was a dashboard that showed some interesting clusters and a good semantic connection. But the main problem with the dashboard was its aimless purpose. I had a lot of data and thought the more data i used, the more accuracy i would get, but the opposite was the case. In this v1 trend dashboard i mixed a lot of different sources like Hacker News, OpenAlex and Polymarket, and retrospectively this was the fundamental problem: too many sources that weren't aligned with the purpose of the dashboard. So i decided to build version 2, but this time scientifically and with a targeted focus.

Trend Dashboard v2

The Dashboard's Purpose

I decided to set up a set of 3 purposes, or call them goals, that define the dashboard's purpose.

1. Early Detection

The dashboard should catch a trend while it is still purely scientific, while it only exists in research papers, before it shows up in patents, funding rounds or the news. The earlier a trend can be spotted, the more valuable the signal is. This is the current idea for a core radar function of the dashboard.

2. The Gap Between Science and Market

For every trend the dashboard should show how big the gap is between the scientific stage and the economic stage right now. Some topics were discovered by science years ago but the market hasn't picked them up yet. Others are moving fast on both sides at the same time. This gap, the Lag Score, is the actual insight the dashboard is built to deliver, not just a list of clusters.

3. Signal, Not Noise

Not every cluster of similar papers is a meaningful trend. The dashboard should only surface topics that show real and sustained growth, not one off spikes or things that are just cyclically popular. This purpose is what v1 was missing, a layer that decides what is actually worth showing.

These three purposes now act as a filter for every decision i make in v2. If a data source, a feature or a metric doesn't serve at least one of these three goals, it doesn't belong in the dashboard, this is the discipline that was missing in v1. Sources like Hacker News or Polymarket, which seemed interesting in v1, wouldn't survive this filter today, they don't help with early detection of scientific signals, they don't map onto the science to market gap, and they don't help separate signal from noise in a scientific sense.

Main Dimensions

I was also convinced that it is extremely important to define the main source dimensions to make the development line more precise.

1. Scientific Dimension

This dimension covers the raw scientific output itself. It includes research papers, preprints and academic publications across all natural science disciplines. This is where a trend is born, often years before anyone outside academia notices it.

2. Patent Dimension

This dimension covers the moment scientific knowledge starts turning into applied, protectable technology. Patent filings are the first strong signal that a company or inventor sees commercial potential in a scientific idea. This dimension sits between pure science and the open market.

3. MarketCap Dimension

This dimension covers the economic and public facing side of a trend. It includes funding rounds, company valuations, stock movements and public attention. This is where a trend becomes visible to people outside science and business insiders, and where the Lag Score finally becomes measurable end to end.

These three dimensions aren't just defined by myself, they are also grounded in scientometrics, the research field that studies and measures scientific literature, publication patterns and their connections to patents and technology. More on this in the next chapter.

Scientific Foundation

To be trustworthy, especially when showing analyses or evaluations, reliability is much more than just important. To solve this problem i decided to build the whole dashboard project using common scientific best practices grounded in scientometrics and technology forecasting. In this context it is recommended to set up some concrete scientific questions to solve.

One of the papers i used as a foundation for this project is [1], a well known work in the field of technology forecasting. One point they make became very important for how i think about v2. They point out that patents alone are not enough to understand where a technology is heading, patent data needs to be combined with an understanding of actual market needs before it can be trusted as a signal [1]. In other words, a single source of data, no matter how good, is not reliable enough on its own to draw real conclusions from.

This is exactly the lesson i already learned the hard way in v1. Back then i trusted one method, semantic clustering on a mixed pile of sources, to tell me what a meaningful trend was. It looked convincing, but it wasn't grounded in anything that could actually be checked or validated.

For v2 i decided to build reliability into the dashboard from the start, not add it afterwards. This means combining multiple independent signals instead of trusting one. Scientific publications, patents and market data each tell a different part of the story, and only when they agree, or when the gap between them can be clearly measured, does a trend become trustworthy enough to show. This idea of combining bibliometric and patent analysis to forecast a technology's development is directly supported by [1], and the underlying method of detecting meaningful clusters and bursts in scientific literature is grounded in the work of [2] and [3].

[1] also points out that even established forecasting methods still require further validation through real case studies before their results can be fully trusted. I take the same approach for v2. Before i present any trend as reliable, i plan to test the dashboard against known historical breakthroughs, technologies where we already know how science, patents and market attention developed over time. If the dashboard can correctly identify these known cases early, that becomes the actual proof that the Lag Score works, not just a claim.

Research Delimitation

Before presenting the methodology, it is necessary to define the boundaries of what this dashboard claims to do, and equally important, what it does not claim to do. A trend detection system that overstates its own certainty undermines the reliability it is trying to establish.

What the dashboard claims:

The dashboard identifies correlational signals, not causal relationships. A short lag between scientific publication activity and market attention indicates a statistical pattern, not proof that the science caused the market movement, or that the market movement will follow the same pattern for future topics.

The Lag Score is a relative, comparative metric, not an absolute prediction. It allows topics to be ranked and compared against one another (topic A currently shows a larger science-to-market gap than topic B), but it does not predict a specific date, timeframe, or probability that a given topic will reach market maturity.

The dashboard surfaces candidates for further investigation, not final conclusions. A trend flagged by the dashboard is a starting point for human judgment and deeper research, not a standalone investment or research recommendation.

The scope is limited to natural science disciplines and their documented outputs (publications, patents, funding, public attention). Trends driven primarily by social, political or non-scientific factors fall outside what the dashboard is designed to detect.

What the dashboard does not claim:

It does not claim to predict which specific trends will succeed commercially, only that a gap between scientific and market activity currently exists and can be measured.

It does not claim completeness. Coverage is limited by the data sources connected to the system (OpenAlex, patent databases, funding data), and any trend not represented in these sources cannot be detected, regardless of its real-world significance.

It does not claim to replace domain expertise. The dashboard is a screening and prioritization tool, not a substitute for expert evaluation of a given field.

It does not claim real-time certainty. Scientific and patent data sources have inherent publication delays (the time between research completion and public indexing), so the dashboard's signals are always a reflection of the most recently indexed data, not the true current state of a field.

References

[1] T. U. Daim, G. Rueda, H. Martin, and P. Gerdsri, "Forecasting emerging technologies: Use of bibliometrics and patent analysis," Technological Forecasting and Social Change, vol. 73, no. 8, pp. 981–1012, 2006.

[2] C. Chen, "CiteSpace II: Detecting and visualizing emerging trends and transient patterns in scientific literature," Journal of the American Society for Information Science and Technology, vol. 57, no. 3, pp. 359–377, 2006.

[3] H. Small, K. W. Boyack, and R. Klavans, "Identifying emerging topics in science and technology," Research Policy, vol. 43, no. 8, pp. 1450–1467, 2014.

Tools that where used to write the article: Claude Science, Claude Sonnet.