Problem

Some feeds are updated rarely. For these, however often jaiku wants to poll them is just fine. Other feeds, though, like last.fm feeds, can be updated very often. Jaiku seems to poll my last.fm feed every 10 to 12 hours (or maybe it just feels like that long, but it’s often several hours), which means lots of songs never get sent posted to my presence stream. Some twitter-users are very prolific, as well, which means that when you see their updates it’s always bunched together in “and 19 more” form.

Solution?

It seems to me that you could set up an adaptive structure like this:

• Define some range of poll rates (say, minimum once per day and maximum once per hour, or something)
• When you poll a feed, note whether there was anything new to post
  • if there was something new 3 of the last 5 times, then move to the next higher poll rate
  • if not, then move to the next lower poll rate
• repeat, adjusting the poll rate to track the update rate of the feeds

So feeds that change a lot will get polled more often, while feeds that do not change will get polled less often. And, most importantly, jaiku would adapt to feeds that are updated in spurts. So, for instance, twitter and last.fm feeds will naturally update a lot more often when people are awake, and then go dormant when people are asleep.

Maybe a different algorithm would be better to capture the irregularity of the update rate:

• When a feed has updates in it, bump this feed up to the highest poll rate
• When a feed has no updates twice in a row, move to the next lower poll rate

So, in this scenario, say a last.fm feed has no updates all night. So, over the course of the night, jaiku polls their feed less and less often. When the user wakes up and starts listening to music, their feed will have updates again. The first time jaiku pulls the feed, it sees the changes and bumps the feed up to the “once per hour” poll rate. Each hour, there continue to be updates, since the user is awake. Once the user stops listening, jaiku will poll the following two hours, and not see updates. Then it will start polling every 2 hours, then every 3 hours, until an update is spotted again, and the cycle starts over.

For feeds (like most people’s netflix queues, for instance) which are usually updated once a day at most, the algorithms would usually not poll the feed very often. It only really affects the feeds that do get constant updates.

I haven’t done any research or modelling… probably there are some adaptive algorithms out there that would work better than these two ideas. But, it seems like it would improve the user experience…

Update: further ideas

Perhaps in addition to the above: when a person makes a jaiku post for the first time after a long break, schedule a person’s feeds for a couple test polls starting 30 minutes after that. The idea is, “this person maybe just woke up, or came back from some offline break, and is now active again.” This would help catch the start of an active period for their feeds quickly, since the poll rate might have gone down quite a bit during the downtime (which would mean a lag of up to 3 or 4 hours before jaiku would notice their feeds are active again).