Flash testing tools - SQLIO

One of the mistakes people frequently make when attempting to test a flash array is their selection of test tools.

This is completely understandable, given that many (most?) of the tools that exist for testing storage were designed for disk-based environments, which react very differently under many situations than flash-based storage.

One of the major differences between a disk array and any flash-based array is the way that they handle blocks of zeros. If asked to write a block consisting of all zeros, most disk arrays will actually write that data to the disk - thus the performance of writing zeros is exactly the same as writing real data blocks.

However on a flash-based array, these zero blocks will be optimized out, and not actually written to the flash media. Depending on the product this could occur within the array controller (especially for arrays that supports thin provisioning), or it could happen at the flash media itself where a block of zeros will be treated as an UNMAP/TRIM request to mark the block as unused in order to assist with garbage collection.

Regardless of where this optimization occurs, the end result is the same - the performance of writing entire blocks of zeros will be different to, and normally significantly better than, the performance of writing non-zero blocks. Thus any performance testing tool that writes only zeros will not be suitable for use on any product that contains flash media.

The problem thus becomes how to tell if a testing tool is writing zeros, or if it's writing non-zeros.

SQLIO

One tool that comes up frequently is Microsoft's SQLIO. Based on the description of what this tool does it sounds like a perfect tool for testing storage performance, especially in a Windows/SQL environment. However the lack of options related to the content of the data used for testing, especially dedup and compression settings, is a red flag that it may not be a good tool for testing flash arrays.

To test SQLIO, I ran a quick test using fairly standard options. This test runs a "write" test that will run for 30 seconds, creating a default-sized 32MB file to run the test on :

C:\Program Files (x86)\SQLIO>sqlio.exe -kW -t2 -s30 -b8 -BN c:\tmp\testfile.dat
sqlio v1.5.SG
2 threads writing for 30 secs to file c:\tmp\testfile.dat
        using 8KB IOs over 512KB stripes with 64 IOs per run
        buffering set to not use file nor disk caches (as is SQL Server)
size of file c:\tmp\testfile.dat needs to be: 33554432 bytes
current file size:      0 bytes
need to expand by:      33554432 bytes
expanding c:\tmp\testfile.dat ... done.
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 13788.09
MBs/sec:   107.71

After the test, we're left with the 32MB file that was created in c:\tmp\testfile.dat :
10/11/2015 01:09 PM 33,554,432 testfile.dat

As a very simple test of how “random” the test data is, we can simply attempt to compress the file, preferably with 7zip. Doing this gives us :
10/11/2015 01:10 PM 4,928 testfile.7z

So our 32MB of test data had been compressed to slightly under 5KB - that's a compression ratio of 6800:1. This doesn't tell if the file is zeros or not, but it does tell us that it's clearly highly redundant, and thus may not be a good test tool for flash.

The easiest way to tell if the file is zeros or not is to open it in a text editor. In this case I'm using Frhed, but any hex editor/viewer will work :

Fairly clearly the resulting file is completely composed of zeros. From this we can easily tell that SQLIO is not a good tool for testing any storage with flash media. Any results it gives will be unrealistic and potentially even an order of magnitude or more higher than for a real-world situation.