A few weeks ago whilst playing with Boost Mobile's "BoostOne" app I stumbled across a security flaw that allowed access to the account information (including phone number, email, physical address and more), payment details/history, and bill information for any Boost Mobile user. More importantly, the flaw also allowed changing the users PIN, which would allow anyone to port out or "sim swap" the users account!  All that is needed to access this information is either the users phone number or their email address!

This vulnerability also appears to have affected customers of Dish Network's new Project Genesis 5G network.

I reached out to Dish Network to report this issue, and whilst their response was far from stellar (that'll be the subject of a future blog post!), it does appear that they fixed the issue within about 10 days of me first reaching out to them. At this point they still haven't confirmed to me that it's been fixed, however the problem no longer exists so it appears that it has been.

Enough intro, lets dig into the vulnerability...

Dish/Boost Mobile use a standard OAUTH-style authentication mechanism, where a username and password are used to generate a long-expiry "Refresh Token", which in turn is used to generate a short-lived "Access Token".  In this example we have an refresh token that corresponds to the user 'scott@doc.net.au', and we're using this to request an access token.

For simplicity, I've saved this access token in a variable, and the same access token is then used for all future requests. Remember, this access token corresponds to the 'scott@doc.net.au' account, so we would only expect it to have access to data for that account.

Using this token, we can make a request for account information. This returns numerous details for the account, including the owners name, address, phone number, email address, etc.  Of course as these details are for my own account there's no security issue here - this is information that I would expect to have access to.

What is interesting about this request is the actual request URL itself - it includes the username that the information is being requested for. As the only information returned should be for the user that is logged in it doesn't seem that there should be a need for that parameter to be passed.

If we try the same call without the username parameter, we get an error :

So seemingly the username is being used for the query, it's not just extraneous information in the request.

So what happens if we tried and request information for a different user? Obviously the correct action would be to deny that request.  Unfortunately, that's not at all what happened!

In this case we've request account details for a different username (xxx@wow.com), whilst still using the authentication credentials for the initial account.  This request should have been denied, but it was NOT!  Instead we are given full account details for the other user! This data includes name, email address, physical address, phone number, phone details/IMEI and more.

(Note that in this case both of the accounts being used here are my accounts, however they are completely separate/unrelated accounts. They have separate email address, physical addresses, phone numbers, and even a slightly different surname between the two. Boost Mobile has no way of knowing that these two accounts belong to the same person - for all due purpose these accounts belong to completely separate people)

This type of issue is known as a "Broken Object Level Authorization" (BOLA) vulnerability. The website is correctly enforcing authentication (you require a valid token to access it), however it is not enforcing authorization at the object-level so data for all users can be accessed, not just the user that is authenticated. BOLA is listed at #1 on the OWASP API Security list.

The attack above would require us to first know the email address for the user whos account we were trying to gain information on, but it turns out we don't even need that, as the username parameter can also be passed a phone number.

Once again, this was using a phone number that does not correspond to the authentication token that we're using to make the request.

In addition to the data returned by the 'account' request, we can also request additional details around billing, payments, etc.  These use a URL that includes the account ID for the account - which is included in the output above.  ("id":"223xxxxxx066").

"Bill" contains details of the users upcoming bill.

"Payment Method" shows the details for the credit card setup to make this next payment - although thankfully only the last 4 digits plus the expiry and not the full number, although the "token" field could also be of interest.

"Payment History" shows details of prior payments, including URLs pointing to further details of those payments including the card used to make them (again, only the last 4 digits/expiry).

Changing Data!

To this point we have only been querying information from the API, however there are also multiple fields that can be updated - including the users address, their password (which requires the current password so seemingly can't be abused), and their PIN code!  This PIN code is what is used when authenticating to a Boost agent via website chat, social media or the phone, as well as being what is used to port a phone number out of Boost.

Note that while this response says the PIN has been "set up" successfully, it works the same even if a PIN has already been set on the account. Unlike the password, no additional authentication is required to set/change the PIN.

The impact of this flaw is significant.  Starting with only an email address or a phone number we are able to reset the PIN number and then port out the phone number - what is commonly known as a SIM Swap Attack, and is commonly used by criminals to bypass 2 factor authentication mechanisms.

Only Boost Mobile?

Dish Network, the parent company behind Boost Mobile, actually owns/runs multiple phone providers in the US, including their new 5G network "Project Genesis"

All of the API URLs above included a "tenant" parameter which was set to "BOOST". Changing this to something else gives an Invalid tenant error, implying that this field is indeed being used to set the provider.

Changing the tenant to "GENESIS" did NOT give this invalid tenant error, but instead simply returned an empty set of results - implying that this issue existed not just for Boost Mobile customers, but also for Project Genesis customers. Unfortunately I don't have access to a Project Genesis phone number/email which would allow me to confirm this, but it does appear that these customers are affected as well.

The empty response is the same as the response that is returned when querying the "Boost" tenant with a phone number (username) that is not a Boost customer - so it's an indication that there is no account for that phone number in the tenant being queried.  

Although I have not tested enough to be sure, it does not appear that there is any rate-limiting on this query, meaning it would be feasible to brute-force/enumerate phone numbers using this query looking for valid Boost Mobile customers.

Other Issues with the Boost Mobile website/API

Although the issue described above is by far the most critical issue I've found, there are a number of other issues with the Boost website/API.  I'll keep those for another day...

References

• OWAWP API Top 10 list, with Broken Object Level Authorization at #1 on the list.
• "T-Mobile customer data plundered thanks to bad API" - ARS Technica
• "A Deep Dive On The Most Critical API Vulnerability — BOLA" - Inon Shkedy

What I didn't mention in that post was that I actually discovered the issue around 6 months before that post was published, but had spent the intervening 6 months attempting to get the relevant parties to fix the problem.  I took a 2-pronged approach to doing this - reaching out to both the vendor that provides the system, and the hotel that I had discovered the problem in.

The system itself is provided by an Australia company called Ezestream, which also appears to go by the name Movielink.  Despite multiple emails and even attempts to communicate to a few of their people via LinkedIn I was unable to even get a response from Ezestream, so I gave up on that fairly quickly.

The hotel property themselves were far more responsive, and very quickly put me in touch with someone from the Marriott security team, although from there things went downhill quickly.

Marriott's initial response was that before they would look at the issue I had to join their Bug Bounty program on HackerOne and submit the issue there, and that after I did so they would "approve it" (whatever that means?). I pushed back initially on the grounds that their HackerOne program only covers issues on their website, which this was not, but they basically refused to communicate with me or take any action unless I did.

I eventually gave in and submitted the issue via HackerOne, including the full draft version of the blog post, as well as specifics not including in the post such as IP address, usernames, passwords, screenshots of exploits, and a few other random vulnerabilities in the system that I had found.

2 weeks later I was contacted again and told that the issue was fixed!  In fact, it was actually fixed a week previously, so only about a week after it was reported!  Great News!  I was also told that they couldn't confirm it was fixed as I hadn't provided IP addresses of the systems in question (which I had), so they were simply willing to take the word of the vendor that it had been fixed.  The report was closed, and the issue marked as resolved.

Except, of course, the issue wasn't fixed. The relevant systems were still accessible from the internet.  The exact exploits that I had previously provided (as simple one-liner curl statements no less) still worked as before. The root-level SSH keys that had been publicly available were still in use across all 80+ properties.  It did appear that the non-password protected Samba share had been removed, and a few default passwords for the web interface had been changed (but the bug that allowed authentication to be bypassed hasn't been fixed), but little more seemed to have changed.

Over the next few months Marriott made multiple claims that they were still looking into the issues and, working with the vendor, but after that they went quiet and haven't made any updates since August!

Fast forward to almost 6 months after the problem was first reported and it did appear that the issue had finally been resolved!  None of the systems appear to be accessible via the internet, and although I hadn't received any confirmation of a fix from Marriott (nor for that matter, any updated in the past few months) I decided to finally publish the blog post describing the issue which had been sitting in draft for a few days short of 6 months!

A weeks or so later  I happened to be in Singapore - a city where a few hotels that used the Ezestream system were located - so I decided to pick one of those properties to stay in for a night whilst I was there.

Sure enough, the TV system in use was from Ezestream, and not surprisingly connecting my laptop to the Ethernet cable going to the back of the TV gave full access to the local system, with all of the same bugs that had previously been reported (including those that allowed bypassing authentication). The root-level SSH keys that had previously been used were still active. In short, the security of the local system didn't appear to have been improved at all in the preceding 7 months.

What's more, the links to the Ezestream server and all other properties were all still active, and still using the same keys. I was trivially able to jump over to the server in the properly in New Zealand where I had originally discovered the issue, or any other property in the system.  It appears that in the end their fix was not to stop access to their servers, but to simply to limit which IP's can access them to those where their systems are located. This might be a half-way valid solution, if not for the fact that the networks in the properties running these systems are publicly accessible by virtue of the fact that they run to the network port on the back of the TVs in every room!

As I said at the start of my previous post, "Security is hard to get right".  It's easy to see how a chain like Marriott could fail to notice security issue like these on a 3rd-party managed system running inside multiple of their properties.  But there is absolutely no excuse for not taking quick and decisive action once they are notified of such issues. Taking over 5 months to make even rudimentary changes that don't actually solve the issue is unacceptable, as is taking the word of a vendor that such a problem has been fixed without confirming it (or even better, doing a full audit/pen-test on the system) - especially given that this was reported to them right around the same time that they were being fined US$124 million for a previous security issue!

Even today, over 7 months after the issue was reported, the HackerOne incident is still only in the "Triaged/Open" state, with no progress for several months...

I've blogged before about some of my digging into Hotel IP-based TV systems, where with minimal effort I was able to obtain everything from basic access right up to full, remote access to the hotels entertainment systems.  (4 years later that particular system is still up and running, and still remotely accessible with default passwords)

However when staying in a hotel in New Zealand recently I came across what could only be described as the motherlode - full root access not just to that hotels TV systems, but also to the equivalent systems in 81 other hotels.  And all of it accessible over the internet!

IPTV

I always love walking into a hotel room for the first time and seeing "Hello Mr Howard" on the powered-on TV screen.  It means that the hotel is using some form of intelligent TV system - one that can turn the TV on or off on-demand, and display custom content.  In the case of this particular hotel in New Zealand, there was no Set-top box, but just an IP-based TV.  In general these TV's themselves are fairly secure and uninteresting, which normally limits what can be found, but still leaves the potential for an insecure server driving them.

After a few seconds of network snooping I had the IP address of the server, and from the network snooping alone I could tell this was a system I'd come across before and not had any luck with, so I wasn't expecting much.  A nmap on the server showed a myriad of open ports - everything from the expected 22 (SSH) and 80/443 (used by the TV, if nothing else), through to a number of more interesting things like 3306 (MySQL), 2049 (NFS) and 139/445 (CIFS), as well as a number of seemingly random ports like 81, 82 and 5555.

Ports 80 and 443 as expected both had web servers running on them, but with no default page and no indication of valid paths other than those that I'd see on the network capture before (which were just TV-Server communication). MySQL and NFS also turned out to be uninteresting (no access and nothing shared respectively), but scanning CIFS showed a number of shares available :

Including one that allowed guest access :

Not surprisingly given the "back" in the name of the share, this contained a number of dated directories that contained backup copies of what appeared to be the entire server.  However very surprisingly, it didn't contain backups for the server in this hotel, but instead about 70 other unrelated hotels around the APAC region!

Digging through one of these backups turned out to be a goldmine in terms of understanding the system, and gave URL paths to access the IPTV system itself, to a copy of phpMyAdmin (a web-based MySQL admin tool), password files (both system and application level), as well many other interesting looking things.

phpMyAdmin, with default MySQL credentials of course, gave access to the MySQL data, and some tables that included user credentials.  To their credit, these were hashed. Not so much to their credit, most of the hashes were trivially reversed with a simple Google search - not surprising given that they were the same as the username in most cases...

These passwords could then be used to access the two main web GUI systems on the box - one called 'NTV-Admin' and a second called 'ADS'.

NTV-Admin

NTV-admin turned out to be a fairly typical Hotel TV admin GUI, and the password gathered above gave full admin access to it. This allowed access to a number of things such as hotel occupancy (128 rooms checked in) :

Listings of all of the movies available, plus the ability to add/delete movies, change prices, etc (not as interesting as it seems given that all movies were free to start with!) :

View details of which rooms were checked in, who was checked into them, and even when they had checked in :

And even a historic record for each room, including all guests checking in and out for the past >12 months - this time even with first names (masked) :

Plus the ability to remotely control the TV, or even send a message to be displayed on a single TV, or all TVs in the hotel.

ADS

However it is the 'ADS' system that turned out to be far more interesting.  I'm still not entirely sure what 'ADS' stands for, but it was the first few items on the menu that looked most interesting - "Root options" and "Cross-site Management" :

Root options

You can can probably already guess what an option called "Linux Console" under a menu with "Root" in it does, but for completeness...

From 1 to 81

A quick look at the Linux level showed a number of ssh processes being used to open TCP tunnels between this system and a master system at the softwares vendor, as well as some scripts to facilitate backing up between different customers - which explains the backup files that started it all.  This also showed that it was possible to jump between the systems at various customer sites via the vendors central system.  Not surprisingly, the systems all seemed to use the same passwords and the same root SSH keys for access, with all of the relevant private keys being stored on the system with no passphrase.

ie, access to one system gives full access to ALL of the other systems, located at other hotels around the world!

Coming back to the 'ADS' web interface and going to "Cross-site Management" gives us a list of all of those systems - with a total of 82 systems showing a "cloud_online" date within the last few hours, implying they were actively connected (there were a dozen others that had not connected for some time, including some demo systems) :

Remember those TCP tunnels I mentioned above?  Well clicking on the links in the table above takes us to a URL that points to the vendors system, with a unique port for each system - which then forwards over that SSH-forwarded TCP tunnel back to the relevant system.  ie, every single one of these systems is accessible via the internet, with the traffic being forwarded by the system vendor! What's more, after attempting to access the system they show up in the "Recent" list, where in addition to the forwarded URL being listed, for many systems so is a "direct_access" URL which provides access to the system directly without having to route through the vendor.

Summary

There's so much wrong here it's hard to know where to start.  There's the obvious security issues such as :

• Default passwords used everywhere
• Same SSH keys used on all systems
• Many systems directly accessible via the Internet
• ALL systems accessible over the internet via the vendors system
• All access over unencrypted HTTP.
• No additional authentication to obtain root access on the systems.
• CIFS shares containing backup data (including passwords/shadow files/etc) with guest access

And that's really just scratching the surface.  However there's the potentially bigger issues that each customers system seem to be being used as backup targets for others customers - many of which would be competitors. This data, which is stored without any form of encryption and (in at least the case of this system) zero authentication, could be mined to find out things like occupancy rates for a competing hotel given it contains checkin/checkout data.

I wrote previously about some of the changes to Snapshots in X2.

A number of corresponding changes were also made to the REST API, in particular from version 6.1 which introduced version 3 (/v3/) of the REST API.  In particular, a new 'command' concept was added for many snapshot operations, where operations like taking a new snapshot are now done as a PUT to the relevant /command/ URL, rather than as a POST on the snapshot object.

As previously, the REST API guide is still the best place to look for all available options, but be aware that there's 2 distinct API guides - one for API version 2.x, and a second for version 3.x.  In order to find the commands below - which are the ones you should be using on an X2 array - you'll need to look in the version 3.x guide.

Creating a Repurpose Copy

In the v2 API, creating a repurpose copy was done by making a POST operation against the snapshots object.  This changes in v3 to being a PUT operation against the  URI /api/json/v3/commands/consistency-groups/create-repurpose-copy. As previously, the details of what you want to take a copy of, new object names, etc, are all passed as options in the body of the request.

For example, to take a new snapshots of the Consistency Group "OracleProd", putting the new snapshots into a Linked Consistency Group called "OracleSnap1" we would use :

[scott ~]$ curl -X PUT -d '{"from-consistency-group-id":"OracleProd","new-consistency-group-name":"OracleSnap1"}' -u admin:Xtrem10 -k -s https://xms/api/json/v3/commands/consistency-groups/create-repurpose-copy

{
    "content": {
        "volumes": [
            {
                "href": "https://xms/api/json/v3/types/volumes/75b5c5ec2e114e2d972ff7709826b97a",
                "index": 57,
                "guid": "75b5c5ec2e114e2d972ff7709826b97a",
                "name": "OracleProd1.1559727279785"
            },
[...  trimmed ...]
        ],
        "consistency-group": {
            "href": "https://xms/api/json/v3/types/consistency-groups/02a8c28800d44f63990e97d63c3f9d80",
            "index": 1,
            "guid": "02a8c28800d44f63990e97d63c3f9d80",
            "name": "OracleSnap1"
        }
    }
}

The response includes details of both the Linked Consistency Group that is created, as well as entries for each of the new snapshot volume(s) created by the operation.

Creating a Protection Copy

Taking a Protection Copy (a read-only snapshot) is fundamentally the same as the above only using a different URI - /api/json/v3/commands/consistency-groups/create-protection-copy

Refreshing a Consistency Group

Refreshing a CG (or Linked CG), either from another CG or from a Snapshots Set, uses a PUT request to the URI /api/json/v3/commands/consistency-groups/refresh-data, once again as a PUT request with the relevant options passed in as the request body.

eg, to refresh the Linked CG we created above, we would use :

[scott ~]$ curl -X PUT -d '{"from-consistency-group-id":"OracleProd","to-consistency-group-id":"OracleSnap1","no-backup":"true"}' -u admin:Xtrem10 -k -s https://xms/api/json/v3/commands/consistency-groups/refresh-data
[scott ~]$

Note that this time there is no response - the fact that the query completes successfully is our indication that the refresh completed.

Other Operations

Other operations, such as adding volumes to a Consistency Group, are similar.  The details of the command URLs for these are all covered in the REST API guide.

Snapshot Types

Previously when taking a snapshot you could elect to take either a "read-only" or "read-write" snapshot. The result was basically the same, with the obvious difference that the snapshot volumes in one were read-only, whilst the others were read-write.
In X2 these terms have changed, and we now have a "Protection Copy" (equivalent of a read-only snapshot), and a "Repurpose Copy" (read-write snapshot).  To some extent this is just a name change, however there have also been related changes to the management concepts for these snapshots in order to both simply them as well as make them more flexible.

Snapshot Groupings

Other than the new names, the biggest change with X2 is the introduction of a new grouping concept, the "Linked Consistency Group".  The existing groupings of "Consistency Group" and "Snapshot Set" still exist, although the use of the latter has changed a little.

In X2, these three concepts are used as follows :

• Consistency Group - A manually created group of volume (or potentially snapshots) that is used as the SOURCE of a snapshot operation.  This is basically the same as previously, although a few operations have changed to add more flexibility.
• Linked Consistency Group - A system-created group of snapshots that is the result of a "Repurpose Copy" operation (ie, a read-write snapshot).
• Snapshot Set - A system-created group of snapshots that is the result of a "Protection Copy" operation (ie, a read-only snapshot).

So basically the difference from X1 is that when creating a read-write (Repurpose) copy, instead of a Snapshots Set being created, you now get a Linked Consistency Group.  This Linked Consistency Group functions basically the same as a standard Consistency Group, with the "Linked" part of the name referring to the fact that there is a relationship between it and the CG from which it was created which allows refer operations to occur between them.

Creating a read-only (Protection) copy still creates a snapshots set, which allows you to manage these separately from copies that you intend to actively use.

Refreshing Snapshots

Refreshing snapshots remains mostly unchanged, except that most refresh operations now will be between Consistency Groups (rather than Snapshot Sets).  It is still possible to refresh from a Snapshot Set (when going from a read-only Protection Copy to a CG), however in the GUI you'll need to click on "Enable Advanced Mode" to see that as an option.

Restoring Snapshots

Restoring snapshots remains unchanged, however as this is sometimes a misunderstood concept it's worth restating how "restore" works.
Restore is basically a limited version of Refresh.  Where Refresh allows you to refresh from any copy of a snapshot, Restore only allows you to restore from a copy that is both :

• Read Only (ie, a "Protection Copy"), AND
• A direct snapshot from the volume being restored

Having these additional restrictions allows you to know that what you're restoring is an immutable (unchanged) copy of the volume that you're restoring to - not something that has potentially been modified (as a read-write snapshot could have been), or that is actually a copy of a different snapshot that has itself been modified.

If you need to "restore" from something that doesn't meet these rules (eg, maybe you accidentally took a read-write Repurpose Copy when you meant to take a read-only Protection Copy), then you can still do it - you just need to instead use a 'Refresh' operation which will give exactly the same result, but without the additional restrictions.

Adding Volumes to a Consistency Group

Adding volumes to an existing CG has changed significantly from X1, based on customer request.  In X1, you could easily add a volume to a CG, however there was no way to create the equivalent snapshot other than refreshing the snapshot set - which might not be what you wanted to do at that stage.

In X2, we've made adding the snapshot to the Linked CG a manual operation, which gives you to flexibility to do it whenever you want.  Thus to add a new volume to a CG the steps are :

1. Add the volume itself to the CG. In the GUI this can be done from the CG screen by selecting the CG and then going to Manage -> Add/Remove Volumes
2. Select the Linked Consistency Group that you want to add a snapshot of the volume to, and then go to Manage -> Add Paired Volume
3. Select the Reference Consistency Group. ie, the one you added the new volume to above
4. Select the new volume that you added to that reference CG
5. Select 'Create New Pair Volume'.  This will automatically create the new snapshot volume for you.  If you'd prefer, you could have created the snapshot manually first, and then used "Pair Existing Volume" instead - but in most cases that just added extra effort for no benefit!
6. If required, map the newly created snapshot to the host that the Linked CG is mapped to so that it's available once you do a refresh.

Note that it's not possible to add a new volume to a Snapshot Set (ie, a Protection Copy) - and there's no need to do so!  Snapshot sets are a point in time copy, so adding a new volume simply isn't needed.  When the next Protection Copy is taken, it will automatically include all of the volumes in the CG at that time, which will include the new volume.

CLI Changes

I generally don't recommend doing Snapshot operations from the CLI - it's easier to use the GUI, or better to use the RestAPI if you're automating something - however for completeness there are a number of new snapshot commands in the CLI that replace the old commands.  (The old commands still exist, but should not be used with X2 - they are only there for backwards compatibility!).  These commands include :

• create-protection-copy
• create-repurpose-copy
• refresh-data
• restore-data
• add-volume-to-consistency-group
• add-copy-volume-to-consistency-groups

Rest API Changes

The Rest API has also been changed fairly significantly - more on that in a future post...

Around 4 years ago United Airlines launched a "Bug Bounty" program. Bug bounty programs are becoming more and more common as a way of companies rewarding people for reporting security issues that they discover on their website. Those rewards generally take the form of a cash payment, or sometimes just recognition for having discovered and reported the flaw.

United's Bug Bounty program is unique in that instead of paying cash for problems detected, they pay awards in United Mileage Plus frequent flyer miles - with payments of between 25,000 and 1,000,000 miles depending on the severity of the bug discovered.

Bug Bounty Submission 1

A little over 2 years after United launched their Bug Bounty program I came across a flaw in their website that I considered worthy of the program, and subsequently submitted it to them. (You can find details of that issue in a separate blog post).  After a few weeks, United denied my submission, claiming that the website was operating correctly.

"After looking into the issue we have determined this finding is not a valid security concern. The values apparently are required for other service calls. They are transmitting the information securely" - Ben from United's Bug Bounty Program

Anyone with any idea of security would know that this response made no sense as the issue clearly is a security flaw, however I decided not to bother following up any further, and vowed to write a blog post instead describing the issue.

Bug Bounty Submission 2

It wasn't until around 18 months later that I finally got around to writing a post describing the issue I'd discovered and submitted.  Given that so much time had passed I decided I should first give United another chance to acknowledge that this was a bug, so after confirming that the problem still existed I made another submission to the Bug Bounty program, including both this same issue as well as a second separate but related issue.  To my surprise, they admitted that both of my submissions were valid bugs, but that the first issue I'd (re-)submitted was not covered by their Bug Bounty program as it was a "previously-known bug that the development team is currently addressing".

Even when I pointed out that I had previously reported this bug over 18 months ago - presumably well before it was a known issue - they refused to budge and claimed that my submission was not valid.

Public Disclosure

Given that over 18 months had passed since I had submitted the bug and it had not been fixed (their own T&C's state that "Our desired timeframe to remediate each valid submission is within 90 days ...") - and because I had already written the blog post describing the issue - I decided to publicly disclose the information leakage issue, whilst not disclosing the second issue they had accepted as a new/previously unknown issue.  To their credit, United did act quickly and fixed the issue within 24 hours of me posting it.

A few days later I received an email from United's Bug Bounty team pointing out that as per their T&C's, publicly disclosing any issues discovered would disqualify me from receiving any miles as payment for discovering the issue - with the clear implication being that I should remove my blog post.  Apparently the fact that they had already denied my submission and thus I wouldn't be receiving any miles anyway didn't occur to them at this stage...

Change of Face

After hearing nothing for a few weeks, out of the blue I received an email stating that as a result of the issue I'd reported now being fixed, as a "valid submitter" I was going to receive a prize of 100,000 United Miles.  However before they could pay the prize I would have to take down my blog post describing the (now fixed!) issue.

It's hard to know what to make of this.  On one hand I can see this as them coming to their senses and recognizing that my original submission was valid and deserved to be awarded - even though they have never actually stated that fact. That being the case, the T&C's of the bug bounty program around disclosure obviously need to be followed, and thus the blog post should go.

However it's just as easy to look on this as a bribe to remove my blog post. Take down the details of the flaw, and we'll pay you 100,000 miles...

The REAL Problem with United's Bug Bounty

Despite the issues described above, the real problem I have with United's Bug Bounty isn't their process, but their payment.  The novelty of using "Miles" as payment sounds like a cool idea for an Airline, but ignores one major problem - the tax consequences.

In addition to getting 100,000 miles, United would also be supplying me with a 1099-MISC tax form, valuing the miles at 2 cents per mile, or $2,000 total.  As a California resident, that amounts to around a $900 tax liability for me, so in effect my "free" 100,000 miles would actually be costing just under 1 cent each.

Of course, paying tax on Bug Bounty payments isn't uncommon.  The difference is that in most cases the payment itself would be in cash, so the tax liability could be offset against the payment, and the result would always be net positive.  With miles, that's not possible - the IRS won't allow me to give up 45,000 of my miles as payment of my resulting tax bill - they'll want cold hard cash!

What's a Mile Worth?

So the question becomes, what is a "Mile" really worth, and is it worth paying $900 to get 100,000 of them.  It's a difficult question to answer given that miles don't have a clear cash value. There are a number of ways to use miles, from flights to merchandise to gift cards, so lets have a look at each of those.

Gift Cards

Starting with the simplest one, and the one nearest to cash value - Gift cards.  United has an entire website where you can buy gift cards with your miles. The prices vary a little depending on which shop the gift cards is for, but in general a $100 gift card will cost you around 15,600 miles, meaning that 100,000 miles would give you around $640 in gift cards.

Let me say that again.  100,000 miles, which would result in a tax bill of over $900, could be turned into gift cards worth around $640, leaving me $260 out of pocket.  Clearly not a good option.

Merchandise

United also provides the ability to purchase merchandise with Miles - an option they send me a catalog in the mail for at least 3-4 times a year.

Their current "best selling" product is apparently a pair of Bose QC35 headphones, which are available for only 48,100 miles. These headphones have an RRP from Bose of $350, valuing 100,000 miles at just over $700 - still around $200 less than the tax bill. Other items are all similar values - such as an Amazon Echo for 13,600 miles ($99.99 on Amazon, or 0.74 cents/mile)

Flights

Calculating the value of miles for flights is difficult, as it can vary dramatically depending on where and when you're flying.  For example, I recently flew from San Francisco to Palm Springs - a ticket that would have cost $264 if I'd paid cash, but instead cost me only 10,000 miles - giving a value of 2.64 cents/mile (or around $2,640 for 100,000 miles).

In this case I was lucky that a "cheap" redemption option was available - and even then it was only available because I have status with United.  If I'd booked a day later, or if I hadn't had United status, the price would have been 32,500 miles for that same $264 flight, giving a valuation of only 0.8 cents/mile ($800 for 100,000 miles - back to less than the tax bill!)

It's certainly possible to get some great value for flights using Miles, especially when flying Business or First Class on 'Saver' awards  - although again it's hard to put a value that especially given that such flights can be very difficult to come across.  Even picking a relatively expensive route such as San Francisco to Sydney return in Business Class, booked a month or so in advance will cost around $5000+tax when paying cash, or 400,000 miles - still only giving a value of about 1.25 cents/mile.

However for me personally, using miles presents a further challenge which is that as a very frequent flyer, I already have over 2 million miles on United and their partner airlines - enough to last me 5 to 10 (or more) years of flying.  Accepting an additional 100,000 miles is going to give me a tax burden today, for miles that I realistically won't be using for another 10 or more years!

And all of this ignores the fact that United reserves the right to clear out your Mileage balance for any one of a number of reasons including  simply not earning/using your miles for 18 months - giving those 100,000 miles a value of exactly $0 (but still with a $900 tax bill!)

Fixing The Problem

The simple fact is that the "novelty" value of United's Bug Bounty payment method turns what is otherwise a reasonably strong program into what for many people will be at best a farce, and potentially even a very expensive experience!

In the Terms and Conditions for the program, United does call out the potential for tax implications, but even there they get it wrong :

The term "at a rate of 2% per mile added to your annual earnings" simply makes no sense - this is likely a typo for "2 cents per mile", but if it is then it's a typo that has existed since the start of the program despite others having called them out on it.

United needs to either determine a way to provide a non-cash prize without tax implications, or at least give the option of a cash alternative.  Being offered $2,000 cash instead of miles that are apparently worth $2,000 resolves all of the issues described above, as it allows the tax implications to be paid out of the prize itself, rather than being an additional burden on the recipient.

Ever left a boarding pass in the seat-back pocket on your flight?  Or perhaps dropped it whilst on the shuttle to the rental car center?  How about the baggage label on your checked bag - ever thrown that in the bin without a second thought?

I'm sure most of us have at some time, but would you do that if you knew that boarding pass or baggage tag allowed someone to access all of your Mileage Plus personal information, such as your home address, phone number, email address and even your Mileage Plus point balance, date you opened your Mileage Plus account, and whether or not you own a United-branded Chase credit card?

Of course, none of this information is included on the boarding pass, but due to an information disclose bug on one of United's partners websites, it's possible to access all of this information using only 2 pieces of information that are on every boarding pass - a surname, and a frequent flyer number.

(There is a second issue that allows you to also obtain this information using only the surname and ticket confirmation number, however United has at least committed to fixing this issue. As the confirmation number is also on the baggage tag it's possible to use that to get this information as well).

To their credit, several years ago United Airlines did start "masking" the frequent flyer number on boarding passes (eg, writing it as AB-***123), however every boarding pass still contains a large 2D barcode, which contains all sorts of interesting information about your ticket, including your full Frequent Flyer number (plus your surname, but that is printed in clear on the boarding pass anyway).

Armed with the Surname and Mileage Plus number, it's then off to one of United's branded websites that is actually run by points.com, buymiles.mileageplus.com  As the name implies, this website allows you to buy additional Mileage Plus miles, or transfer them to family members.  In order to "authenticate" to this website, the only details you need to enter are - you guessed it - a surname and a Mileage Plus number.  Of course, this wouldn't be a problem if all you could do would be to purchase Miles (using your own credit card!) for the person you're logged in as - however as a part of the "login" process the website actually leaks information about the user whose details have been entered.

The first two piece of information are blatantly displayed on the post-login page - your current Mileage Plus balance, and a masked email address.  Not good, but not too bad either.

The issue isn't so much with what is shown, as what isn't.  During the authentication process, a number of other details are passed to and from the websites involved VIA THE BROWSER.  This means it's simply a matter of looking at the data being passed to determine multiple other details about the account :

So very quickly we've gone from having only the users surname and Mileage Plus number (or even just their boarding pass!) to having their home address, phone number, whether or not they have a United credit card, the date their account was created, mileage balance, and their United Mileage Plus status (4=1K, 3=Platinum, etc).

If nothing else, this information would leave the the user open to a very targeted phishing attack - especially given that we also have details of a recent flight they took (based on the boarding pass).

Disclosure to United Airlines

This issue was reported to United via their "Bug Bounty" program in July of 2017. After spending around a week investigating the issue they responded that :

"After looking into the issue we have determined this finding is not a valid security concern. The values apparently are required for other service calls. They are transmitting the information securely."

It is now 18 months later and the issue has not been fixed.

The information disclosed is data that would be considered Personally Identifiable Information (PII) in most jurisdictions, and in particular would fall under GDPR in the EU, so it's a little hard to understand why United considered this not to be a "valid security concern".

XtremIO X1 Snapshots

First, some history. As per the blog post listed above, taking a snapshot on X1 always resulted in a "Snapshot Set" being created.  This Snapshot Set could then be used to create a new snapshot, or as a part of a snapshot refresh.  A "Consistency Group" in X1 was only used as the source of a snapshot or refresh.  Refresh was allowed between Consistency Group and Snapshot Sets, or between Snapshots Sets and other Snapshot Sets (as long as they were both taken from the same CG tree).

When taking a snapshot with X1, you had the choice of a read-only or a read-write snapshot.  The two acted basically the same, except that a read-only snapshot couldn't be refreshed (and obviously, was read-only when presented to a host).

XtremIO X2 Snapshots

In X2, the general terms remain the same.  We still have "Consistency Groups" and we still have "Snapshot Sets", but their use is a little different.

A Consistency Group is still a group of one or more volumes.  It's now used for more than just snapshots (such as Native Replication and QoS), but fundamentally it's similar to in X1.

Snapshot Sets still exist, but their use has changed a little, and they are now generally only used for "Protection" copies.

The most obviously change when you come to take a snapshot of a Consistency Group is that the term "snapshot" has been removed from the GUI, and instead you have the option of a "Protection Copy" or a "Repurpose Copy", which correspond to a read-only snapshot and a read-write snapshot respectively.  However there are more differences between the two than just that...

Protection Copy

As the name implies, a protection copy creates a copy of your volumes that can be used for "protection" - a read-only "backup"-style copy of your volumes.  As with in previous versions, taking a protection copy of a CG will result in a Snapshot Set being created for the resulting volumes, and you will be able to specify the name of that Snapshot Set during the process.

As we're taking a Protection Copy, the resulting volumes will be read-only (displayed as "Read-Access" in the GUI)

As this is a read-only copy, there is no option to refresh the snapshot - refreshing really only make sense when the volumes have been mapped to a host and you want to update the data on them, which generally won't be a how a protection copy is used.  If you want to update a protection copy, you can just take a new protection copy of the volumes.

Repurpose Copy

A Repurpose Copy is a copy of data you are going to "repurpose" for a new use, such as a development copy. As a result, it's a full read-write copy of the volumes, and is exactly equivalent (at the volume/snapshot level) to taking a read-write snapshot in previous versions.

The big difference between Repurpose Copy and a read-write snapshot is that instead of the resulting snapshots being grouped into a Snapshot Set as in previous versions, they are instead put into a new Consistency Group, or what is sometimes referred to as a "Linked Consistency Group" on the grounds that it is "linked" to the original CG that it was taken from.

After creating the Repurpose Copy we will have 2 Consistency groups - the one we took the snapshot of (OracleProd), and the new one (OracleDev1) containing the newly created read-write snapshots.

Refreshing a Snapshot

Repurpose copies can be refreshed just like read-write snapshot sets could be previously - just select the Consistency Group you want to refresh, and select "Refresh Selected" from the Repurpose menu.

When doing a refresh, you will by default only be given the option to refresh FROM another Consistency Group - either the original source CG, or another repurpose copy.  If instead you want to refresh from a read-only Protection Copy it's just matter of selecting the "Enable advance mode" checkbox during the wizard at which point you'll also be given the option of refreshing from a Snapshot Set.

In X2 when refreshing a Consistency Group, the name of the CG is NOT changed as a part of the process.  This is different to X1, where the name of the snapshot set was changed with each refresh, and additional steps - such as renaming the resulting snapshots set - were often required to keep the name the same.  This makes programmatically refreshing a snapshot much easier than before, as you will always be refreshing the same objects, without the need to use tags or rename the resulting objects as was required with X1.

CLI

New CLI commands have been added to match the above operations :

create-protection-copy - Creates a new read-only Protection Copy, along with a new Snapshot Set

create-repurpose-copy - Creates a new repurpose copy, along with a new (Linked) Consistency Group

refresh-data - Refreshes a read-write snapshot (Consistency Group) from either a another consistency group (original volume or another repurpose copy) or from a snapshots set (protection copy)

REST API

There have been a number of changes to the REST API, so I'll cover those in another post (coming soon...)

Backwards Compatibility

As a part of these changes, we wanted to maintain backwards compatibility for any CLI or REST API scripts that people were using on X1 systems.

In order to do this, X2 systems will act in a 'backwards compatible' mode if the older X1 CLI or REST API commands are used to create or manipulate snapshots.  For example, the X1 'create-snapshots' CLI command continues to exist and acts exactly as it did in X1 - a new snapshots set will be created even when creating a read-write snapshot (and NOT a new consistency group as would be created when using the new create-repurpose-copy command).  The same is true for the 'create-snapshot-and-reassign' command that was previously used for refreshing snapshots, which continues to work exactly as it did previously.

In order to get this backwards compatibility, it's important to use ONLY the old-style commands with these copies, and not jump between the old and new mechanisms.  Where this can be a little difficult is when creating the initial snapshot.  Many people will use the GUI to create a snapshot, but then use the CLI or REST API to refresh it (normally as a part of an automated workflow). This will NOT work with X2, as the GUI will follow the new mechanism for creating the snapshot (resulting in a CG being created rather than a snapshots set).  The workaround is simply to create the original snapshot using the create-snapshot CLI command rather than using the GUI, in order to get the old-style functionality.

(If you're not already famaliar with using the XtremIO REST API, then I'd suggest reading my series on using it first)

XtremIO has a very powerful API for accessing performance data from the XMS, but it can be a little difficult to get your head around. The REST API Guide available on support.emc.com, does a good job of documenting the options, but not really how to use them.

Understanding XtremIO Performance Data

Before accessing the data from the API, it helps to understand a little about how performance data is collected and stored by XtremIO.

Every 5 seconds the XMS attempts to collect hundreds of performance metrics from the array - not just at the array level, but also at the level of things like Targets, Initiators, and Volumes.

This is obviously a lot of data, so over time the data is consolidated to less frequent intervals. The initial "raw data" is kept of 3 days. One minute data is kept for 7 days, ten minute data for 30 days, and so on. (See the XtremIO Users Guide for full details)

What is interesting is that this consolidation is done soon after the data is collection, not once the data reaches the expiry time for the previous timeframe (eg, 3 days for the 5 second raw data). Thus for (say) data collected yesterday the system will have data available at each of "raw data" (5 seconds), "1 minute", "10 minute", "1 hour" and "1 day" granularity, After 3 days, the raw data will be deleted, but the other points will return. After 7 days, the 1 minute data will be deleted, and so on.

As we'll see shortly, when querying data we can specify which of the granularities we want to use, or we can let the system decide for us.

When the consolidation occurs, the system will maintain 3 data points for each time period - a minimum, a maximum, and an average. However for the initial raw data it will have only a single value (which is in itself generally an average over the 5 seconds collection iterval)

Querying the Data

All performance queries are made using a single URI - /api/json/v2/types/performance

At a minimum you need to pass an "entity" parameter to specify what you which type of object you want performance data for. The list of possible entities is contained in the REST API guide, but you can also get a list by passing in an invalid entity (eg, .../performance?entity=A) which will return a response containing the full list :

{
    "message": "Command Syntax Error: entity property must have one of the following values: [SnapshotGroup, Initiator, Target, VolumeTag, XEnv, DataProtectionGroup, Volume, Cluster, Tag, InitiatorGroup, InitiatorGroupTag, SSD, TargetGroup, Xms]", 
    "error_code": 400
}

Given that many of these entities return a LOT of data by default, lets start with one of the more simple ones - "XEnv" which will return performance data for the "X Environments", otherwise know as the CPUs!

Sending a request for /api/json/v2/types/performance?entity=XEnv (note that the entity name IS CASE SENSITIVE, so in this case the X and E need to be upper case!) returns a JSON response which basically consists of 2 sections. The first is a number of "counter" entries that look like :

"counters": [
    [
        1513555200000, 
        "529d231911a84b0a879a557c250abcd4", 
        "X1-SC1-E1", 
        1, 
        2.2815487587808998
    ], 
    [
        1513555200000, 
        "33ba4bf1e2ea4175905098bdc9a825c2", 
        "X1-SC1-E2", 
        2, 
        2.1723940081082902
    ], 

Secondly is a "members" entry :

"members": [
    "timestamp", 
    "guid", 
    "name", 
    "index", 
    "avg__cpu_usage"
], 

We need to use the two of these together - the multiple "counters" entries include the actual data, whilst the (single) members entry provides the order of the fields in the counters.

Thus for the example above we've got a data point with a timestamp of 1513555200000, for the XEnv with the name "X1-SC1-E1" (X-brick 1, Storage Controller 1, Environment/CPU 1), for which the performance counter "avg__cpu_usage" has a value of 2.2815487587808998

We've also got a second data point with the same timestamp, but for the XEnv "X1-SC1-E2" (CPU2), with an "avg__cpu_usage" value of 2.1723940081082902. There were of course hundreds of other data points returned - in part because we didn't specify a time frame for the query!

The name of the data field "avg__cpu_usage" actually tells us that we were NOT seeing the raw data in this result, but instead one of the consolidated values, and specifically the "average" value (not minimum or maximum for that consolidated period). We'll see below how to get the raw data or the min/max values. The double-underscore in the name is used to indicate the "avg" comes from the XMS's consolidation of the data, and not an average value from the array itself.

Limiting the Results

By default, the number of data points returned can be very large - especially if you're not specifying a time range (which we'll cover in a moment).

You can filter the returned results in 2 separate ways - either by the entities (eg, one or more specific volumes), or by the properties returned (eg, bandwidth, iops, latency, etc).

To filter by entity you use the "entity-name=XXX" option on the URL. Multiple of these can be used in the same query to return data for multiple entities.

eg, to get data just for the volume "MyVol1" you could use :

/api/json/v2/types/performance?entity=Volume&entity-name=MyVol1

To get data for three separate volumes, simply list them all :

/api/json/v2/types/performance?entity=Volume&entity-name=MyVol1&entity-name=MyVol2&entity-name=MyVol3

Even with only a few entities the results can still be very large. For example, "Volume" entities return around 24 different performance metrics for each time point, including bandwidth (avg__bw), read bandwidth (avg__rd_bw), write bandwidth (avg__wr_bw), IOPS, read IOPS, write IOPS, etc

Most likely you only need a few of those, so you can specify which properties you want returned using the "prop=XXX" option. The best way to use this is to do a query without it to see the full list of properties, and then specify just the ones you want. As before, you can pass multiple "prop" options to get multiple properties returned.

eg, to get the average bandwidth and IOPS for both MyVol1 and MyVol2 :

/api/json/v2/types/performance?entity=Volume&entity-name=MyVol1&entity-name=MyVol2&prop=avg__bw&prop=avg__iops

Note that there are a few properties returned regardless of whether you ask for them or not, such as timestamp, name, index and guid.

Filtering by Time

As mentioned above, the XMS consolidates data over time, with data being available for up to 2 years (although at a low granularity - one point per day!), or as frequently as every 5 seconds.

When it comes to selecting what data we're interest in there's a few different options that play a part.

The first is "time-frame", which can be one of four specific ranges "last_hour", "last_day", "last_week" or "last_year", or alternatively "real_time" or "custom_time".

When using custom_time, either or both of "from-time" and "to-time" can be specified to control the range - if either (or both) parameters are skipped then the time of the oldest (from-time) or newest (to-time) data available will be used.

from/to-time are specified in GMT time, in the format "YYYY-MM-DD hh:mm:ss". Note that's a space in the middle, which is not a valid character to have in a URL. If the software you're using to make the request doesn't automatically do so, you should replace it with "%20" which is the encoded equivalent of a space (Space is ASCII code 32 decimal, which is 20 hex).

In addition to "time-frame" you can also specify which level of data consolidation, or "granularity" you want the data from. ie, whether you want the "raw" data (5 second), "one_minute", "ten_minute", "one_hour" or "one_day" values. Of course, not all data is available for all time ranges, so if you ask for a "time_frame" of "one_year" with a "granularity" of "raw", you'll actually end up getting only the past 3 days worth of data - as that is all that is available for the "raw" setting.

It's also possible to pass a "granularity" of "auto", in which case the system will automatically determine the granularity based on the time range you've specified.

Aggregation Type

The final parameter you may want to use is "aggregation-type". By default, when querying data that has been consolidated (ie, everything except "raw" data) you will be given the average over the time period, and the name of any fields that have been aggregated with be prefaced with "avg__" (eg, avg__iops). You can instead specifically specify that you want the "min", "max" or "avg" values - and as with the other options you can specify multiple if needed.

/api/json/v2/types/performance?entity=Volume&entity-name=MyVol1&aggregation-type=min&aggregation-type=max&aggregation-type=avg

This will give you results that include all of min__bw, max__bw and avg__bw (and of course the same for every other property returned!)

This also leads to one of the quirks of the performance API. If you end up querying the "raw" data - either because you specifically specify it (with granularity), or because you specify a time-frame where the system picks that data automatically (eg, last-hour), then the data isn't aggregated. This means that there are no min/max/avg values, but more importantly it means that the property names returned do NOT include the avg__ at the start like they do when querying other time frames.

eg, if you query XEnv performance data for the past hour, the response will have the following entries :

"members": [
    "timestamp", 
    "guid", 
    "name", 
    "index", 
    "cpu_usage"
],

But if you instead query for the data for the past day (with no other changes), you will receive :

"members": [
    "timestamp", 
    "guid", 
    "name", 
    "index", 
    "avg__cpu_usage"
],

There's 2 potential ways to handle this - either explicitely specify the granularity of the data you want to receive, or programatically handle the avg__ prefix on the results if it exists.

"null" values

There is one other thing to be aware of when using "raw" data, which is that although the system attempts to collect data every 5 seconds, for various reasons it occasionally fails. eg, there might be a network issue between the XMS and the array that causes the data collection to fail.

In this case, when querying the raw data you will still receive entries, however the value will be "null". For example, for a XEnv query where no data is available you'll see :

   [
        1527861190000, 
        "529d231911a84b0a879a557c250abcd4", 
        "X1-SC1-E1", 
        1, 
        null
    ], 

There is a second occasion that "null" will be returned. If a volume or a snapshot has never been mapped to a host (ie, it doesn't have an NAA assigned), then the array doesn't generate performance data for it - as it can't have any! In this case, the data for both the raw data and the consolidated values will be "null"

[
        1527938480000, 
        "a3ea0523b56a49c4a200eb04070bb210", 
        "NewVol1", 
        4, 
        null, 
        null, 
        null, 
        null, 
        null,

These entries should generally be ignored as invalid (or at best, uninteresting) data points.

Putting it all together

Lets say I wanted to get the average IOPS and Bandwidth for my two Oracle volumes for the first week of May 2018, with one data point per hour. So we've got :
entity=volume because I'm looking for data on a volume
entity-name=Oracle1&entity-name=Oracle2 as those are my 2 Oracle volume names
prop=avg__bw&prop=avg__iops because these are the only two properties I'm interested in
time-frame=custom_time&from-time=2018-05-01 00:00:00&to-time=2018-05-07 23:59:59 (Don't forget to replace the spaces with %20 if needed!)
granularity=one_hour

Giving me a full query of

/api/json/v2/types/performance?entity=volume&entity-name=Oracle1&entity-name=Oracle2&prop=avg__bw&prop=avg__iops&time-frame=custom_time&from-time=2018-05-01%2000:00:00&to-time=2018-05-07%2023:59:59&granularity=one_hour

If I wanted to get both the maximum AND average values, then I'd need to add :
prop=max__bw&prop=max__iops (In addition to the existing prop entries)
aggregation-type=max&aggregation-type=avg in order to get BOTH minimum and average

Giving :

/api/json/v2/types/performance?entity=volume&entity-name=Oracle1&entity-name=Oracle2&prop=avg__bw&prop=avg__iops&prop=max__bw&prop=max__iops&time-frame=custom_time&from-time=2018-05-01%2000:00:00&to-time=2018-05-07%2023:59:59&granularity=one_hour&aggregation-type=max&aggregation-type=avg

Final Thoughts

Whilst this may seem complex, once you get your head around the options it's actually fairly simple. As with all REST API queries, the best option is to simply play around with some queries and see what you get back - either with a tool like Postman or HTTPRequester in Firefox, or even using something like cURL. Try different options, and see what you get back. Use prop and entity-name to limit the number of entries.

There's also another option I haven't mentioned which is "limit=X" which can be used to limit the number of results returned. Using this in a final query probably isn't a good idea, but it can be useful when learning and testing to make sure you don't get too much data back from a query.

In general, when a Storage Controller is shutting down for any reason, XtremIO X1 will send a "Logical Unit Not Supported" SCSI response code (0x5/0x25) in response to any requests received by that storage controller from a connected host. This could be just a single SC shutdown (eg, during an upgrade, where SC's are rebooted one at a time), or during an entire array shutdown (eg, where power has been lost and all Storage Controllers are running on battery).

For most operating systems, this gives the behavior that we want - the Multipathing software on the host will mark that one path as down, and continue to use the other paths. For a complete array shutdown (eg, power outage), the host will end up marking all of the paths down, but will continue to try using them and will recover the paths once the array is available again.

However VMware ESXi attempts to be a little more intelligent when it looses connectivity to a LUN in situations like this, and can actually put the device into one of two different states when it loses all paths to that device - "All Paths Down" (APD), or "Permanent Device Loss" (PDL). VMware have a KB article and a blog post that cover some of the differences between these two states, but simplistically the difference is that with APD you expect the LUN to return at a later stage, whilst with PDL the expectation is that it will not return (as the name implies!)

In general, the only way to recover from a PDL state is to reboot the VMware host, which can obviously be a fairly intrusive action. With APD, the devices will generally recover once at least one path to the device is recovered.

APD option in XtremIO

Given the potential impact and extended recovery from LUNs going into PDL state, in XtremIO XIOS 4.0.10 we added the ability to change the SCSI codes that the array sends to VMware hosts during a Storage Controller shutdown. Instead of sending a SCSI "Logical Unit Not Supported" response like we do for most OS'es, we instead send a SCSI "Transport Disrupted" response, which has the same immediate impact on the VMware side (path is taken offline), however if all paths fail, VMware will put the devices into "All Paths Down" state, rather than PDL.

For backwards compatibility reasons we didn't change the default behavior, but instead added a new configuration setting to allow admins to select if they wanted to old (PDL) or new (APD) behavior. The current setting can be viewed/changed using the "show-clusters-parameters" and "modify-clusters-parameters" commands from the XMS CLI :

As these settings are for VMware ESXi only, they only apply to initiators that have been configured with an Operating System of "ESX". Initiators configured with any other OS, including the default of "Other" will still follow the old "PDL" behavour, regardless of what the clusters ESX Device Connectivity Mode is set to.

Default Behavior

As mentioned above, when we added this setting we did not change the default behavior. All existing and new systems kept the "PDL" behavior unless it was explicitly changed.

In XIOS version 4.0.25-22 (and later) we changed the default behavior, but only for newly installed clusters. Existing clusters still retain the PDL behavior unless they are changed, however any clusters installed with 4.0.25 or later will default to the APD behavior.

VMware Multipathing Bugs

These changes were originally only really relevant for situations where the entire XtremIO cluster was shutdown, such as a planned outage or a power outage. However over the past few years VMware ESXi Native Multipathing has had two bugs that have made these changes relevant for other situations as well. The first of these was in ESXi 6.0u2, whilst the second was in specific patches across multiple versions of ESXi.

In both of these situations, ESXi acts incorrectly when it receives a "Logical Unit Not Supported" response, with the end result that it can incorrectly fail all paths to the storage, resulting in a host-side Data Unavailability situation - even though the array is fully functional. These issues can be triggered by many different arrays (generally Active/Active multipath arrays), including XtremIO X1.

Although these are bugs on the ESXi side, and they can be avoided by applying VMware patches (at the time of writing, the patches for the second issue are still pending), changing the ESXi path policy will also avoid triggering them as we will no longer send a "Logical Unit Not Supported" which is the only SCSI response that triggers this issue.

Best Practice

In general, the best practice for this setting would be the configure it to "apd".

If you're running one of the affected VMware ESXi versions and are planning an XtremIO upgrade, then there's basically no choice - without changing this setting the upgrade will result in ESXi incorrectly deciding that all paths to the storage are down, and result in a host-side data unavailability state.

If you're not running one of those versions, or not planning an upgrade, it's still generally recommended to change to "apd".

As this setting only affects what happens when a storage controller is being shutdown, there's no impact on the operation of the array when making that change. If the "Operating System" setting for the initiators is not correctly set to "ESX" within XtremIO you'll also need to change that - which can also be done on the fly without any impact (similar to the APD/PDL setting, this setting only has impact when an SC is shutting down for some reason).

How about X2?

XtremIO X2 acts differently when shutting down a storage controller. For unplanned outages like power failure, the Storage Controller will immediately stop responding (rather than running on BBU's during the shutdown process like XtremIO X1 did). As a result, the host will simple see the path drop, and will act accordingly.

For planned Storage Controller shutdowns (including upgrades), the behavior is similar - the array will drop the port, which the host will detect and multipathing will offline the path.

In both cases, this results in an 'APD-style' behavior on ESXi, so whilst the mechanism is different, the end result is the same as the new behavior on XtremIO X1.

Recently there has been a lot of talk around a method of saving on airfares called "Hidden City Ticketing". Whilst Hidden City Ticketing can save money, there are a number of catches when it comes to using it - catches that can cause extreme difficulties and extra cost if they are not fully understood.

If you are considering using Hidden City Ticketing then I would suggest reading and fully understanding the details below before you do to avoid getting badly burnt...

So What Is It?

Lets say you wanted to fly from St Louis (STL) to Atlanta (ATL).

Only one airline flies that route directly - Delta, as Atlanta is one of their hubs. Other airlines will of course allow you to fly from St Louis to Atlanta, but they will require you to have a stop somewhere. For example, with American Airlines you will need to fly via Charlotte, or with United Airlines via Chicago (ORD)

In a simplistic world, airlines would just charge for their flights based on the distance of the flight - but we all know that's not how airline prices work.

From a pricing perspective, Delta has the ability to charge a price premium for this flight. The passengers have the advantage that they don't need to have a stopover anywhere, and thus will arrive earlier - and thus they will likely be able to charge more for that flight. Thus if Delta was to charge $200 for that flight, United and American might need to charge $150 in order to get passengers to fly their more roundabout, and longer, routing.

But what if you don't actually want to go from St Louis to Atlanta, but you actually want to go to Chicago? This is a similar situation as above, only reversed - Delta don't fly that route directly (only via Atlanta), whilst United Airlines and American have direct flights. This time UA and AA get to charge the premium price (lets say $200 again), whilst Delta have to discount (say $180) in order to compensate for the stopover.

So now you've got 3 options for flying from St Louis to Chicago.

1. You can fly on United, and pay the $200 fare

2. You can fly on Delta, via Atlanta, at the discounted $180 fare

3. Or, you could book a ticket with United to fly from St Louis to Atlanta via Chicago for $150, but simply get off the plane in Chicago and not board your connecting flight!

Option 3 is what is known as "Hidden City Ticketing - you've purchased a ticket from St Louis to Atlanta, but your real destination is the "hidden city" along the route, Chicago!

You've managed to get the best routing (direct, with no stops!) for a price that was even cheaper than the indirect route with a stop. Not surprisingly the airline isn't going to be too happy with you as you've managed to get a flight they wanted to charge you $200 for, but only paid $150 for it - but what can they do, right?

The Downside of Hidden City Ticketing

Turns out, there are a number of things they can do - and some of them can end very badly for you.

Future Flights

The first thing most airlines will do when you fail to board a flight is that they will cancel the rest of the flights on your itinerary - including any return flights. In your efforts to get a cheap STL-ORD return flight you actually booked STL-ORD-ATL, then ATL-ORD-STL on the same booking, then the moment you fail to board your ORD-ATL flight they will cancel not only that flight (which you never intended to take anyway!), but also the ATL-ORD-STL flights for the return - leaving you stranded in Chicago with no return flight (and no refund!)

The workaround for this is fairly simple - book everything as a one-way flight - with your outbound and return trips on separate bookings. This can sometimes change the pricing dramatically (airlines often charge more for one-way flights than for returns), but it's really the only option.

This rule also means that you must be flying TO the hidden city, not FROM it. If you're trying to fly from Chicago (ORD) to St Louis, then booking ATL-ORD-STL won't help you as once you fail to board the ATL-ORD flight the airline will cancel your ORD-STL booking as well!

Baggage

When you check bags for a flight, the airline will automatically send them to your final destination. If you fail to board your second flight, one of two things will happen

For domestic flights in the US, your bags will carry on to the final airport - even if you don't. As a result, if you checked bags for your STL-ORD-ATL flight but didn't go beyond Chicago, your bags will still fly to Atlanta - and then it'll be your responsibility (and cost!) to get them back. You can't expect any sympathy from the airline here, as they will likely know exactly what has happened...

For most other flights, there are laws that if you fail to board your flight then your bags must be removed. Whilst this does mean your bags will stop in the intermediate airport (Chicago, in our example - even though that's not technically relevant as this would be a US domestic flight), in doing so the airline will have had to delay the flight to remove the bags - at a real cost to them, and an annoyance to everyone on the flight. As a result, they are not going to do you any favors in actually getting your bags back to you, and at a minimum you can expect to be in for a significant delay whilst they do so.

So to put it simply, you can't check bags when using hidden city ticketing - but can only travel with carry-on bags.

If you travel light this might not sound like a big disadvantage, but there are countless things that can go wrong. If the airline decides your bags are too large/heavy, or if the plane runs out of space in the overhead bins, then you might be forced to check your bag even if you don't want to. At best, you will need to be very careful to make sure that this doesn't happen.

There used to be a workaround for this issue, which was known as "short checking", where the airline would allow you to check your bag only as far as an intermediate airport. Thus even though you were (apparently) flying to Atlanta, you could ask them to only check your bag as far as Chicago. In recent years most airlines have removed the ability to short check a bag - mainly because people were using it as a way to get around the baggage problem with hidden city ticketing! Whilst you can still ask, it's an almost certainty that you will be refused.

IRROPS

IRROPS (short for IRregular OPerationS) is airline speak for something going wrong. It could be a flight being canceled or delayed due to bad weather, a mechanical issue, or even an oversold flight that they need to move passengers off.

When you buy a ticket from St Louis to Atlanta via Chicago, the airlines obligation is to get you to Atlanta. There is no requirement for them to actually fly you on the route you've originally booked - although obviously they generally would if everything is going to plan.

If your St Louis to Chicago flight is cancelled, the airline might simply move you to a later flight on the same route - or they might move you to a completely different routing - such as St Louis to Newark to Atlanta, or even move you to another airlines flight such as putting you on the direct Delta St Louis to Atlanta flight.

Of course, as you were actually only headed to Chicago this leaves you with a problem - but it's one that you really can't do much about! Whilst you can certainly ask to be moved back to a St Louis->Chicago->Atlanta routing, there's absolutely no requirement for the airline to comply, and even if they do they will likely give you far lower priority than any passengers actually booked to Chicago so it could be hours or even days before you can actually get on a flight!

Legalities

For the most part, there is nothing illegal about using Hidden City Ticketing. It is generally against the contract you have with the airline (the one that came along with your ticket but which of course nobody ever bothers to read!), most of which state that you can't buy a ticket you don't intend to fly. Of course, it's near impossible for the airline to prove you never intended to take the second flight (and didn't simply change plans halfway through), so unless you become a serial offender you're likely not going to have problems there.

That said, there are situations where there could be legal implications when travelling internationally. For example, say you had bought a ticket from the US to New Zealand via Australia, with an intent to stay in Australia. As you were booked as a transit passenger through Australia, the various Australian Arrival taxes would not have been paid with your ticket, as these are not changed for transit passengers - but they would be due as you are no longer transiting to New Zealand. This could potentially cause difficulties for either you or the airline.

Visas

Another issue that can occur with international itineraries is that of visa requirements. For example, it's not uncommon to find cheap hidden city flights from the US to Europe, with Russia being the final destination (for example, San Francisco -> Paris -> Moscow).

However in order to be allowed board even the first leg of this flight, you will need to have a visa for the final destination - Russia. It doesn't matter that you are planning to only fly only as far as Paris, you will not be allowed board the plane in San Francisco without a Russian visa.

International-to-Domestic

One of the few routes where you can generally avoid the issues above are some international flights, connecting to a domestic flight, where your real destination (the hidden city) is the first point of arrival from the international flight.

For example, say you want to fly from Singapore to San Francisco. Adding an additional flight to somewhere in the US could cause the price to drop (eg, Singapore -> San Francisco -> Reno)

As is the case in many (but not all!) countries, the US requires you to collect your bags at the first point of entry into the country - in this case San Francisco. Despite the fact your baggage is tagged all the way to your final destination, you would normally need to collect it in San Francisco and then give it back to the airline for your connecting flight - but in general there is nothing stopping you simply taking your luggage and leaving the airport.

However even then you need to be careful as the rules can and do vary from country-to-country, and even airport-to-airport. For example, whilst it's possible to do what is described above in San Francisco, it is NOT possible at Washington Dullas airport, as connecting passengers (and their bags) are sent to a separate immigrations/customs area where there is no choice but to re-check your bag!

So Is It Worth It?

Obviously different people will have different answers to this, but in my opinion, Hidden City Ticketing is NOT worth the effort or trouble it can cause. Yes, there is the opportunity to save some money, but in most cases the risk of something going wrong outweigh the potential savings.

If you know you'll be travelling with only a small amount of carry-on luggage (ie, something that can fit under the seat in-front of you), on a route that has a low chance of IRROPS/delays, and are willing to fight to get what you want if something does go wrong, then it might be worth your effort - but in most cases the potential for problem outweighs the advantages.

I've stated previously that using curl for accessing the XtremIO REST API (or any API!) isn't ideal as it's difficult to handle errors returned by the API.

Below is a BASH function that can help with this. It takes the options for a REST API call, and returns an error if either the curl/network/etc side of things fails, OR if the REST API returns an error.

Using it is as simple as calling the function with the relevant parameters, and then checking the return value to see if the API call was successful or not.

eg :

Response=$(API admin Xtrem10 GET https://xms.example.com/api/json/v2/types/volumes)

The full API function, along with an example, is included below. You can also get it from here

#!/bin/bash

#
# The API function will initiate a REST API call to the XtremIO XMS with
# error checking.
#
# Usage :
#    API username password req_type URL req_body
#
#      username - The username to use to authenticate to the XMS (eg, admin)
#      password - The password to use to authenticate to the XMS
#      req_type - One of GET, POST, PUT or DELETE
#      URL      - The full URL to the resources
#      req_body - For POST/PUT, the JSON-formatted body of the request
#
# The function returns 0 if the request is successful, and the JSON response
# from the array (if any) is send to stdout, otherwise it returns non-zero.
#
API() {

        local TMPFILE=`mktemp`
        local BODY=""

        if [ $# -eq 5 ]; then
                BODY="-d ${5}"
        elif [ $# -ne 4 ]; then
                echo Invalid number of arguments past to API
                return 99
        fi

        local RESP=$(curl -s --output $TMPFILE --write-out "%{http_code}" -X ${3} ${BODY} -u ${1}:${2} -k -s ${4})

        if [ $? -ne 0 ]; then
                RESP=$?
        elif [ $RESP -ge 200 ] && [ $RESP -le 299 ];  then
                RESP=0
        else
                RESP=1
        fi

        cat $TMPFILE
        rm $TMPFILE

        return $RESP
}

Response=$(API admin Xtrem10 POST https://xms.example.com/api/json/v2/types/snapshots '{"from-consistency-group-id":"CG1","to-snapshot-set-id":"SS1","no-backup":"true"}')

if [ $? -eq 0 ]; then
        echo SUCCESS.  Response was :
        echo $Response
else
        echo ERROR.  Error was :
        echo $Response
fi

It's been 20 years almost to the day since I started using Veritas Volume Manager, or Sun Enterprise Volume Manager (SEVM) as the re-branded version from Sun was called at the time. SEVM used to be included free with all Sun SPARCstorage Array 110 model arrays (an external proprietary fiber-connected JBOD array capable of supporting up to 30x 2Gbyte or 4Gbyte disks!)

The good news is that in almost 20 years, very little has changed, so when a customer asked me to do some testing with Veritas InfoScale (as it's now called) on XtremIO X2 it was amazingly easy to get things up and running - despite it being almost 10 years since I've used VXVM!

The specific purpose of the testing was to show refreshing of an XtremIO snapshot on a Linux (CentOS 7) host running VXVM. Before we do a refresh, we need to actually take the snapshot, and make it available on the host...

The Setup

To do this I setup a clean CentOS 7.4 host, installed Veritas InfoScale Enterprise Storage 7.3.1, and then presented 2 LUNs from an XtremIO array to the host. As I was using VMware, the LUNs were mapped to the host as RDMs.

Using vxdiskadm I then added these disks to a new diskgroup ("proddg"), created a 200GB volume ("prodvol1") in that diskgroup, and mounted it up a new a new VXFS filesystem mounted under /prod1 :

In order to track different copies, I created a file with the current time/date in this filesystems :

Taking the Snapshot

With all of that done, it's then back to the array to create the initial snapshot of these volumes. The initial 2 volumes were put in an XtremIO Consistency Group called "Vol1cg", which I then created a repurpose copy (ie, writable snapshot) of called "Vol1cg-copy1" :

After presenting the new snapshot LUNs to the VMware host, I then mapped them to the same Linux guest as the original LUNs. This adds an extra complexity to the process, as the snapshot LUNs have the exact same VXVM signature on them as the primary LUN, so as far as VXVM is concerned they are the same disks.

Thankfully VXVM handles this cleanly, detecting that the serial number of the disk has changed, resulting in a "udid_mismatch". This is a good thing, as it causes VXVM to ignore those disks and not try and treat them as a part of the existing diskgroup.

(xtremio0_0 and _1 are the existing disks. xtremio0_2 and _3 are the snapshot LUNs)

If we had presented these snapshot LUNs to a different host, at this point we could have simply imported the diskgroup (optionally renaming it in the process) and started using it. However because we've got both the original and copy LUNs on the same host, we need to do an extra step to tell Veritas exactly which LUNs we're wanting to work on.

To do this, we add a "tag" to the LUNs that we want to operate on. The actual tag we use isn't important, as long as we're consistent - in this case I tagged the disks with "devcopy" :

Next we can import the copied diskgroup with a new important flags :

• -n devdg - The new name for the diskgroup (so we don't have two diskgroups with the same name)
• -o useclonedev=on - This is the magic flag that tells VXVM that we're importing a hardware-cloned LUN, and allows it to ignore a number of things that would otherwise be an issue (eg, the diskgroup marked as already being in use)
• -o tag=devcopy - Defines which copy of the disks we want to using - being those that we tagged with the "devcopy" tag above
• -o updateid - Tells VXVM to create new disk identifiers for the disks, so they no longer conflict with the originals.

Note that when mounting the filesystem it did a journal replay - that's because the filesystem was still mounted when we took the snapshot. For non-vxfs filesystems extra steps like an fsck might be required.

Finally we can check the file we created on the filesystem and confirm that it's an exact copy of the original :

By changing the file on the original volume we can confirm that it does NOT change on the copy - which is what we would expect given that it's a completely independent copy of the volume/diskgroup :

Refreshing the Snapshot

Refreshing the snapshot on the array is a simple, single-step process - but before we can do that we need to make sure the host is no longer using the volumes we're going to refresh. The refresh will change the data on these LUNs without the filesystem/volume managers knowledge, which will result in perceived data corruption of the copy if the host is still accessing it.

For VXVM/VXFS, that's just a matter of unmounting the filesystem and deporting the disk group. At that point "vxdisk list" will show the disks as being a part of the diskgroup, but with parentheses around the diskgroup name to show that it is not imported :

Refreshing the snapshot could be done via the XtremIO GUI, but it can also be easily done via the REST API - here we are refreshing the "Vol1cg-copy1" consistency group from the production "Vol1cg" CG :

Veritas does not continually scan LUNs for changes, which means that even after the refresh, vxdisk list shows the old data for the disk names - the same as we saw before the refresh occurred. In order to fix this, we need to tell VXVM to rescan the disks and re-read their private regions by running "vxdctl -f enable" :

At this stage we're at the same point we were above with the snapshot LUNs being an exact copy of the production LUNs, and thus having the same VXVM identifiers, so we can run through the exact same steps (tag, vxdg import, mount) as before to import them as a new diskgroup :

As you can see, the time in the file has change to the new one - showing that the contents of the LUNs were refresh!

In Part 1 of Using the XtremIO REST API I covered how to access the API, and how to read data from it. Part 2 covered how to create and modify objects, Part 3 covered Snapshots, Part 4 was some Best Practices. Next up is an example using the API from Perl.

Using the REST API from Perl

My language-of-choice for scripting is Perl, and whilst it's not necessarily the best language for RESP API's, there are a few modules that can make it relatively easy, including REST::Client and JSON.

If you're using Red Hat/CentOS/etc then the majority of the modules you need can be installed using the following :

yum install perl-libwww-perl perl-JSON perl-Crypt-SSLeay

However at least for verison 6, the REST::Client module itself isn't available in the default RHEL/CentOS repositories, but it is available in EPEL. After configuring access to EPEL ("yum install epel-release" on CentOS, a little more difficult on RHEL) you can install it using :

yum install perl-REST-Client

Alternatively you can download the REST::Client module from CPAN and either compile/install it, or simply grab the lib/REST directory and put it somewhere in your Perl library path (eg, the currently directory!)

Next grab this relatively simple example script - show-vols - which will query the API and show basic details of all volumes configured on an array. You'll need to configure the details near the top of the script (username/password/XMS hostname or IP), but otherwise this script shouldn't need any changes.

Firstly we load in the Perl modules we're going to use. REST::Client is obviously used to make the REST requests, whilst JSON is needed to help us parse the results returned from the API, and MIME::Base64 is used to encode the username/password into the request. Data::Dumper isn't technically required, but it can be useful when trying to see the details of what is being returned by the API.

use REST::Client;
use MIME::Base64;
use JSON;
use Data::Dumper;

Next is a fairly simple function called checkerr() that can be used after each REST call to make sure the the call succeeded. In this case all errors will be fatal and the script will immediately print an error and exit, but of course that could be modified to act differently if required.

The XtremIO REST API requires very minimal configuration within the REST::Client module - we simply need to set the Content-Type to be application/json, and configure the standard HTTP Authentication header (note that whilst the password appears to be sent in clear-text, all communication is over HTTPS, so it's actually fully encrypted) :

my $client = REST::Client->new();
my $headers = {Authorization => "Basic ".encode_base64($username.":".$password), "Content-Type" => 'application/json'};

Now we can make our API call. REST::Client has a few ways of doing this, but the easiest is to just use the GET, POST, PUT and DELETE methods. So to issue a GET request to get the list of volumes we simply have :

$client->GET("https://$xms/api/json/v2/types/volumes", $headers);

The output from the above command will be a block of JSON formatted code, like :

{
    "volumes": [
        {
            "href": "https://xtremio4.scxtremiolab.com/api/json/v2/types/volumes/344",
            "name": "MyVol1",
            "sys-name": "mycluster"
        },
        {
            "href": "https://xtremio4.scxtremiolab.com/api/json/v2/types/volumes/345",
            "name": "MyVol2",
            "sys-name": "mycluster"
        },

The Perl JSON module (specifically the from_json() function) can trivially parse this, returning a tree of hashes and arrays depending on the actual data. The easiest way to see how it has actually formatted the resulting data is using the Data::Dumper module. Using print Dumper($resp) gives us :

$VAR1 = {
          'volumes' => [
                         {
                           'href' => 'https://xtremio4.scxtremiolab.com/api/json/v2/types/volumes/344',
                           'name' => 'MyVol1',
                           'sys-name' => 'mycluster'
                         },
                         {
                           'sys-name' => 'mycluster',
                           'name' => 'MyVol2',
                           'href' => 'https://xtremio4.scxtremiolab.com/api/json/v2/types/volumes/345'
                         },
[...]

So we have a hash (with at least one key called "volumes"), containing values that are an array (due to the []'s displayed), with each entry in the array being a hash with at 3 entries for name (the name of the volume), sys-name (the cluster name) and the unique URI for that volume. The URI is what will allow us to get more information for each of those volumes, so we can then walk through the array and grab just the href values for each of the volumes :

foreach my $v (@{$resp->{volumes}}) {
	push @Volumes, $v->{href};
}

Next it's simply a matter of looping through each of the href values , doing a GET for the URI to get the full details for that volume, and then printing the results.

Of course once again we'll get back a JSON string for the results, so again it's best to use Data::Dumper to start with to learn the format of the response :

$VAR1 = {
          'content' => {
                         'creation-time' => '2017-11-06 15:30:23',
                         'acc-size-of-rd' => '0',
                         'small-rd-bw' => '0',
                         'wr-latency' => '0',
[...etc...]
                         'name' => 'MyVol1',
                         'acc-num-of-small-rd' => '0'
                       }
        };

And from there we can find the fields we want and display them :

print << "EOT"
Volume name : $resp->{content}->{name}
Volume size : $resp->{content}->{"vol-size"}

The easier way

The example above gets the list of volumes and then loops through each one to get the full details. There's actually an easier way to do this, which is to append ?full=1 to the URL. eg, instead of requesting

https://xms.example.com/api/json/v2/types/volumes

which returns the name/cluster/href for each volume and requires a subsequent call for each volume to get the actual details, we can request

https://xms.example.com/api/json/v2/types/volumes?full=1

which will return all of the available data for all volumes.

You can even take this a step further and specify exactly which options you want returned, rather than the full list. In the script linked above we are only interetsed in the vol-size, logical-space-in-use and creation-time properties (plus name, which is always included) so we could use the following request :

https://xms.example.com/api/json/v2/types/volumes?full=1&prop=vol-size&prop=logical-space-in-use&prop=creation-time

which will return only those fields (plus a few default ones) :

{
    "volumes": [
        {
            "index": 344,
            "creation-time": "2017-11-06 15:30:23",
            "name": "MyVol1",
            "vol-size": "1048576",
            "guid": "e44cdfc10d914d7cba85de1a90af6d95",
            "logical-space-in-use": "0"
        },

Making changes

You can just as readily use REST::Client to make changes to the array as well, using POST, PUT and DELETE functions. Take a look at the create-vol script for a simple example of using a POST call to create a new volume.

I generally don't recommend people use the XtremIO CLI for programmatic access - it's just so much easier and better to use the REST API.

However there are occasions where using the CLI may make sense. Access to the CLI can be automated via SSH, and sending commands is relatively trivial - right up until it becomes time to parse the output from those commands.

$ ssh scott@xms 'add-volume vol-name="MyVol1" vol-size="1t"'
Added Volume MyVol1 [29]

The normal output from CLI commands that return data is a series of columns, which at first might appear easy to parse. For example, if I pick one of the more simple commands as an example we see something like this :

xmcli (admin)> show-bricks
Brick-Name Index Cluster-Name Index State
X1         1     xtremio4     1     in_sys
X2         2     xtremio4     1     in_sys

5 columns, each of them with a single entry in them, and each what appears to be a fixed width. The problem is that none of those assumptions are necessarily true! Looks at what happens if I rename my cluster to have a longer name containing a space :

xmcli (admin)> show-bricks
Brick-Name Index Cluster-Name      Index State
X1         1     XtremIO Cluster 4 1     in_sys
X2         2     XtremIO Cluster 4 1     in_sys

As the entry in the field increases, the size of the field grows to accommodate it. Although not shown here, there's an additional issue that fields where there is no value will simply show as blank, so between that and the spaces within the values we can't use field location to parse the entries. This rules out basically every traditional way of parsing such output!

Thankfully there is a pattern to the output, and thus a way to parse it. Although there may be spaces in the fields themselves, the field headings do NOT have spaces within them - and there's guaranteed always to be at least one (and frequently more) spaces between the headings. That number of spaces can change over time as the values change - there will always be at least one, and the columns below them will always align with the first character in the heading.

Taking the second show-bricks output above and looking at it's header line we have :

Brick-Name Index Cluster-Name      Index State
^          ^     ^                 ^     ^
1          12    18                36    42

We can then uses the values to parse the data below it into the relevant fields.

A relatively easy way to do this is to use the block of "gawk" code below. Not easy because it's easy to understand - just easy because I've already written it for you!

The part of the command in red (including the "NR>1") should be left unchanged - this is the part of the code that splits the fields based on the header field lengths.

The part in blue is the actual action - in this case it's matching fields from the output - confirming that the field "Created-From-Volume" is non-zero, and that the field "Volume-Name" contains the substring from the variable "contains". It then prints the value of the "Volume-Name" field if it matches.

The green section at the start sets the variable "contains" to "Sol". In this case I could have just as easily hard-coded that value, but if you're passing in shell variables it's much easier doing it as it's shown here - set a gawk variable to the value you want, and then use that within the gawk script.

The above obviously isn't very easy to read, so here's an easier version to read :

gawk -v contains="$backup" '
  NR==1 {
    a=$0;
    while (a) {
      l=match(a, "( [A-Z]|$)");
      flen=(flen l " ");
      n=substr(a, 1, l);
      sub(" +$", "", n);
      field[n]=++fn;
      a=substr(a, l+1);
    }
    FIELDWIDTHS=flen;
  }

  NR>1 {
    for (i=1; i<=NF; i++) {
      sub(" *$", "", $i);
    }
  }

  NR>1 && $field["Created-From-Volume"] && $field["Volume-Name"]~$contains {
    print $field["Volume-Name"]
  } '

Note that the above REQUIRES GNU Awk (gawk). It uses features that are not available in standard "awk" (specifically the FIELDWIDTHS variable, if I recall correctly).

Obviously this is still fairly complex, and that's even without adding in any error checking - but if you must automate things via the CLI it at least makes it possible...