The Mystery of the Changing Task Status

Know how in a horror movie, the protagonist closes the water faucet in the kitchen, and it starts gushing when they turn their back to it?

One of our customers, let’s call them Lindsay Brooks, went through something similar…

Lindsay reached out to us on what was, until then, a usual day. Before getting into the story, let me back up a bit and set some context: Lindsay uses Rocketlane, which is a collaborative customer onboarding platform. It combines project management, communication, file sharing, and reporting capabilities into a single platform that both you and your customers can use for a smooth onboarding experience at both ends. An integral part of customer onboarding projects (any project, really) is task management. Rocketlane allows you to add tasks, and ‘sub tasks’ under those tasks, assign them to people, and more.

And this task management is where Lindsay and her team were facing a puzzling issue: their tasks' statuses were changing of their own accord, immediately after one of the team updated a task status. For example, Lindsay would change her task status from, say, ‘To do’ to ‘In Progress’, or create a new task and assign it a status, only for the status to change to ‘Blocked’ a minute later. 

Our initial assumption was that this had to do with a security issue. We got on a call with the customer and went through the usual checks: Did someone have access to the customer's login credentials? Was their device compromised? We looked at the security audit logs and ruled out the possibility of any devices being compromised.

Next, we explored the possibility of a single user being able to trigger more than one action at a time. This wasn’t a possibility, too, as Lindsay and team told us these actions - the task status changes - were happening outside of their working hours!

We moved on to check the activity logs. It got curiouser and curiouser: the first instance of task status change and the one that followed it immediately were from two different IP addresses (let’s call them Good IP and Bad IP). What’s more, the second task change actions were always from the same IP address (Bad IP).

This wasn’t possible unless Lindsay and her team were using a VPN. They confirmed that they weren’t using VPN. We dug further into our logs.

Now, Rocketlane sends actionable email notifications when task statuses are changed. This means you can update task statuses right from your inbox, with the click of a button.

We found that the emails weren’t opened, but the buttons to change task statuses had been clicked. And the clicks from unopened emails were always done from the Bad IP. Actions from the Good IP (verified by Lindsay’s team as theirs) had involved opening Rocketlane, or the email, and then changing the task status.

(At this point, we were very tempted to enlist the services of an exorcist, but we pressed on).

We dug further, and learnt that the IPs belonged to AWS and we couldn’t get more information on them. We went back to the unopened-emails part of the problem. Now, usual human behavior would be to click the links/buttons that will help them take the action they want to. You don’t click every link in an email! But that’s exactly what was happening: for all users on Lindsay’s team, the emails were unopened, but all links had been clicked!

The engineering team tracked the domain to which these emails were sent. The issue had occurred for the customer’s domain only, and not on other domains (say, gmail.com). And then, it all came together.

We figured that the customer could be using a spam checker. And we were right. The spam checker clicks all links in an email automatically to know if it is safe, and that is exactly what it was doing with our email notifications!

Having proven to Lindsay that our platform wasn’t possessed, we decided we’d save the trouble of having to deal with this mystery for all our customers—we requested them to whitelist our email.

Lindsay stopped worrying about spooky task changes, we preempted similar problems for the rest of our customers, and the icing on the cake: we got our own vanilla ice cream car moment!

Read more

1. Rocketlane Named a 2022 Gartner® Cool Vendor in Technology Go-to-Market
2. Rocketlane advances to SOC 2 Type 2 Compliance and Certification
3. Rocketlane highest rated client onboarding platform in G2 Fall Report

Want to know what Rocketlane can do for you? Book a demo