How Not to Handle a Security Breach

A few months ago I wrote about how I had gained access to 82 separate hotel IPTV systems within an hour or so.

What I didn't mention in that post was that I actually discovered the issue around 6 months before that post was published, but had spent the intervening 6 months attempting to get the relevant parties to fix the problem.  I took a 2-pronged approach to doing this - reaching out to both the vendor that provides the system, and the hotel that I had discovered the problem in.

The system itself is provided by an Australia company called Ezestream, which also appears to go by the name Movielink.  Despite multiple emails and even attempts to communicate to a few of their people via LinkedIn I was unable to even get a response from Ezestream, so I gave up on that fairly quickly.

The hotel property themselves were far more responsive, and very quickly put me in touch with someone from the Marriott security team, although from there things went downhill quickly.

Marriott's initial response was that before they would look at the issue I had to join their Bug Bounty program on HackerOne and submit the issue there, and that after I did so they would "approve it" (whatever that means?). I pushed back initially on the grounds that their HackerOne program only covers issues on their website, which this was not, but they basically refused to communicate with me or take any action unless I did.

I eventually gave in and submitted the issue via HackerOne, including the full draft version of the blog post, as well as specifics not including in the post such as IP address, usernames, passwords, screenshots of exploits, and a few other random vulnerabilities in the system that I had found.

2 weeks later I was contacted again and told that the issue was fixed!  In fact, it was actually fixed a week previously, so only about a week after it was reported!  Great News!  I was also told that they couldn't confirm it was fixed as I hadn't provided IP addresses of the systems in question (which I had), so they were simply willing to take the word of the vendor that it had been fixed.  The report was closed, and the issue marked as resolved.

Except, of course, the issue wasn't fixed. The relevant systems were still accessible from the internet.  The exact exploits that I had previously provided (as simple one-liner curl statements no less) still worked as before. The root-level SSH keys that had been publicly available were still in use across all 80+ properties.  It did appear that the non-password protected Samba share had been removed, and a few default passwords for the web interface had been changed (but the bug that allowed authentication to be bypassed hasn't been fixed), but little more seemed to have changed.

Over the next few months Marriott made multiple claims that they were still looking into the issues and, working with the vendor, but after that they went quiet and haven't made any updates since August!

Fast forward to almost 6 months after the problem was first reported and it did appear that the issue had finally been resolved!  None of the systems appear to be accessible via the internet, and although I hadn't received any confirmation of a fix from Marriott (nor for that matter, any updated in the past few months) I decided to finally publish the blog post describing the issue which had been sitting in draft for a few days short of 6 months!

A weeks or so later  I happened to be in Singapore - a city where a few hotels that used the Ezestream system were located - so I decided to pick one of those properties to stay in for a night whilst I was there.

Sure enough, the TV system in use was from Ezestream, and not surprisingly connecting my laptop to the Ethernet cable going to the back of the TV gave full access to the local system, with all of the same bugs that had previously been reported (including those that allowed bypassing authentication). The root-level SSH keys that had previously been used were still active. In short, the security of the local system didn't appear to have been improved at all in the preceding 7 months.

What's more, the links to the Ezestream server and all other properties were all still active, and still using the same keys. I was trivially able to jump over to the server in the properly in New Zealand where I had originally discovered the issue, or any other property in the system.  It appears that in the end their fix was not to stop access to their servers, but to simply to limit which IP's can access them to those where their systems are located. This might be a half-way valid solution, if not for the fact that the networks in the properties running these systems are publicly accessible by virtue of the fact that they run to the network port on the back of the TVs in every room!

As I said at the start of my previous post, "Security is hard to get right".  It's easy to see how a chain like Marriott could fail to notice security issue like these on a 3rd-party managed system running inside multiple of their properties.  But there is absolutely no excuse for not taking quick and decisive action once they are notified of such issues. Taking over 5 months to make even rudimentary changes that don't actually solve the issue is unacceptable, as is taking the word of a vendor that such a problem has been fixed without confirming it (or even better, doing a full audit/pen-test on the system) - especially given that this was reported to them right around the same time that they were being fined US$124 million for a previous security issue!

Even today, over 7 months after the issue was reported, the HackerOne incident is still only in the "Triaged/Open" state, with no progress for several months...