60,000 duplicate feeds removed
May 12th, 2008
Things may have seemed a bit quiet recently, but we have been working on backend improvements that lay the groundwork for a new swell of features to be rolled out in the coming weeks.
One such improvement relates to duplicate entries. We have taken great steps to remove duplicate feeds from the system that clog up our recommendation and matching algorithms. We have eliminated over 60,000 duplicate feeds in the system!
Many sites have multiple feeds that all point to the same content, many of these feeds are either old, are URL masked and redirect to the actual feed or particular feed readers tweak them in some way. This presents a challenge when we compare users who both read TechCrunch but they each have different versions of the feed. We needed to make our system more robust to handle these cases.
We have done just that.
We have applied what we learned and have added ’scrubbing bubbles’ to our feed importation to do all the magic needed to keep the algorithms and profile pages from getting clogged. Please leave a comment if you find a feed we missed!
