A team MIT scientists have created an algorithm that they claim can predict Twitter trending topics with 95 percent accuracy before they even happen.
MIT associate professor Devavrat Shah and his student Stanislav Nikolov claim that their tool will predict which topics will trend on an average of an hour and a half before Twitter’s own algorithm, and at times, even hours before Twitter. The reason they say, is that their tool is unlike typical trending tools which only look at how small trends evolve over time. These guys just look at the data.
The MIT’s blog explains what this means:
In the standard approach to machine learning, Shah explains, researchers would posit a “model” — a general hypothesis about the shape of the pattern whose specifics need to be inferred. “You’d say, ‘Series of trending things … remain small for some time and then there is a step,’” says Shah, the Jamieson Career Development Associate Professor in the Department of Electrical Engineering and Computer Science. “This is a very simplistic model. Now, based on the data, you try to train for when the jump happens, and how much of a jump happens.
“The problem with this is, I don’t know that things that trend have a step function,” Shah explains. “There are a thousand things that could happen.” So instead, he says, he and Nikolov “just let the data decide.”