Enabling flash cache compression on Exadata X3-2

I have a plan to write a few posts about the new features in the storage software that was released in December 2013, first out is this one on enabling flash cache compression.

What you need to enable this feature

• An Oracle Exadata of generation X3 or X4
• Storage software 11.2.3.3 or 12.1.
• Licenses for the Advanced Compression Option

Flash cache compression will allow the storage cells to compress and decompress data in the flash cards on the fly. The compression is done in the flash cards themselves, so it does not use CPU on the cell hosts and it is completely transparent to the database.

This procedure is explained in the release notes for the Storage cell software, for 11.2.3.3 that is doc id 1487339.1. I'm doing this with downtime, so all databases are down and the GI is down. See MOS if you want to do this rolling with the databases and GI up.

To enable the compression we need to drop the flash cache, the area reserved for smart flash logging and then all flashdisks. This is done as root on the first cellserver

[root@dm01cel01 ~]# cellcli -e alter flashcache all flush
Flash cache dm01cel01_FLASHCACHE altered successfully
[root@dm01cel01 ~]# cellcli -e drop flashcache all
Flash cache dm01cel01_FLASHCACHE successfully dropped
[root@dm01cel01 ~]# cellcli -e drop flashlog all
Flash log dm01cel01_FLASHLOG successfully dropped
[root@dm01cel01 ~]# cellcli -e drop celldisk all flashdisk
CellDisk FD_00_dm01cel01 successfully dropped
CellDisk FD_01_dm01cel01 successfully dropped
...

Because this is an X3-2 system we must enable support for compression and then enable the compression.

[root@dm01cel01 ~]# cellcli -e alter cell "flashCacheCompX3Support=TRUE"
Cell dm01cel01 successfully altered
[root@dm01cel01 ~]# cellcli -e "alter cell flashCacheCompress=TRUE"
Cell dm01cel01 successfully altered

Recreating celldisks, then the cache for smart flash logging and finally the flash cache itself.

[root@dm01cel01 ~]# cellcli -e "create celldisk all flashdisk"
CellDisk FD_00_dm01cel01 successfully created
CellDisk FD_01_dm01cel01 successfully created
...
[root@dm01cel01 ~]# cellcli -e "create flashlog all size=512M"
Flash log dm01cel01_FLASHLOG successfully created
[root@dm01cel01 ~]# cellcli -e create flashcache all
Flash cache dm01cel01_FLASHCACHE successfully created

This needs to be repeated on all storage cells. That's it! On a quarter rack this was done within half an hour.

[root@dm01db01 ~]#  dcli -g ~/cell_group -l root cellcli -e "list cell detail"|grep flash
dm01cel01: flashCacheMode:       WriteBack
dm01cel01: flashCacheCompress:   TRUE
dm01cel02: flashCacheMode:       writeback
dm01cel02: flashCacheCompress:   TRUE
dm01cel03: flashCacheMode:       writeback
dm01cel03: flashCacheCompress:   TRUE

Update:

@db_magnus Works fabulous on the X4-2 too. Did you notice that the flash cache doubled in size after enabling compression?
— Carol Dacko (@cdacko) February 10, 2014

Doubling of cache size? Sure thing! I didn't check this before the change, so I had to double-check the X3 datasheet, but it should have 1,6TB of flash pr storage cell.

[root@dm01cel01 ~]# cellcli -e list flashcache detail|grep -i Size
         effectiveCacheSize:     2978.75G
         size:                   2978.75G