Fusing blobs in Azure.......

If you're reading this as a DBA you might be thinking i've written some plsql code to combine to blob column values into one - you'd be sadly mistaken what I'm actually going to talk about is this thing

https://azure.microsoft.com/en-us/blog/linux-fuse-adapter-for-blob-storage/

What this enables is the mounting of azure blob storage directly as a unix 'filesystem'. Now azure file storage can already be done using the tried and trusted smb protocol (samba) and we already have experience of that working (albeit a little slowly) on existing servers. However blob storage could not be used in this way and all interactions would be via rest calls or using some sort of api (az/azcopy etc)

What interested me about this is that it would perhaps give me another option for being able to directly write oracle backups to azure storage without having some kind of intermediate step ( I can't see Oracle ever releasing an api for rman to enable this as they dont want people putting oracle in Azure - only oracle cloud will do.......). Experience with azcopy utilities had shown some incredible performance and i wanted to see if this was somehow replicated.

So i went about doing some testing of this.

As a prereq i created a blob storage account in the portal - smoe quick screen grabs of that just for reference - so go through the wizard

create a container inside it

And find the access keys we need later

The software itself is released by microsoft and you can see it all in github here https://github.com/Azure/azure-storage-fuse

I followed the install instructions from here  https://github.com/Azure/azure-storage-fuse/wiki/Installation Which in my case just equated to running

rpm -Uvh https://packages.microsoft.com/config/rhel/7/packages-microsoft-prod.rpm

followed by

yum install blobfuse

Now continuing with the instructions i create a temporary location that it requires

mkdir /mnt/resource/blobfusetmp
chown oracle:dba /mnt/resource/blobfusetmp

Then i create a config file which contains the details and secret key for my azure storage i created earlier

vi /etc/azstorconfig.cfg

this contains

accountName oraclebkptest
accountKey blahblahblah==
containerName backups

Now i create an actual mount point where i want the thing visible

mkdir /fusebackups

So now we're ready to mount - so lets try it

blobfuse /fusebackups --tmp-path=/mnt/resource/blobfusetmp -o attr_timeout=240 -o entry_timeout=240 -o negative_timeout=120 --config-file=/etc/azstorconfig.cfg
Failed to connect to the storage container. There might be something wrong about the storage config, please double check the storage account name, account key and container name. errno = 1600


And we fail miserably - with not much to go on - thankfully pretty much the only thing i found solves the problem - I am forcing https in my storage setup but the default is http - so i just need to add the flag --use-https=true

blobfuse /fusebackups --tmp-path=/mnt/resource/blobfusetmp -o attr_timeout=240 -o entry_timeout=240 -o negative_timeout=120 --config-file=/etc/azstorconfig.cfg --use-https=true

And that runs without error - and indeed i can see it in df-k output as this

blobfuse                              32895696   2146344  29055304   7% /fusebackups

The permissions look a little odd but it should be useable

 ls -ld /fusebackups
drwxrwx---. 2 32602 bin 4096 Aug 29  4439189 /fusebackups

however when i come to switch to oracle and take a look i see the following.....


d??????????   ? ?      ?       ?            ? fusebackups

Now thats a new one on me and i was a little baffled - but a quick look at the man pages for fuse and some guesswork i added the following mount options to the mount command and tried again

blobfuse /fusebackups --tmp-path=/mnt/resource/blobfusetmp -o attr_timeout=240 -o entry_timeout=240 -o negative_timeout=120 --config-file=/etc/azstorconfig.cfg --use-https=true -o uid=1001 -o gid=1002 -o allow_other


After this as the oracle user i see this - so looking good

drwxrwx---.   2 oracle dba  4096 Jan  1  1970 fusebackups

So we're ready for a backup now i think - so lets try it.......

RMAN> backup format '/fusebackups/%U' database;

which throws out this

RMAN-03009: failure of backup command on ORA_DISK_1 channel at 12/06/2017 19:09:51
ORA-19504: failed to create file "/fusebackups/5sslfaii_1_1"
ORA-27044: unable to write the header block of file
Linux-x86_64 Error: 1: Operation not permitted
Additional information: 4

hmm - thats not that useful - lets try just manually copying a file to the location

cp file /fusebackups

this runs without error but seems a little slow - in fact timing it for a 20MB file takes 20 seconds.....

time cp file/fusebackups
real    0m20.193s
user    0m0.000s
sys     0m0.006s

hmmmmmm

lets do an ls of whats there

ls
.
.
.
.
hang hang hang

And it never comes back - lets try again

and we get

ls: reading directory .: Machine is not on the network

double hmmmm

lets look in this cache directory that was mentioned right at the start


cd /mnt/resource/blobfusetmp
 find .
.
./root
./root/GOLDDB_hot_lv1_20171205_184_5oslcp91_1_1

And we see the file - but it seems to be in this limbo cache area and nowhere else - indeed the blob storage screen in the portal just shows this complete lack of files also

So at that point i gave up and decided maybe this isn't really production ready.........not sure that the rman interface would even work at all even if the basics of this were working - maybe worth revisiting next year as could be useful for some use cases when its more robust.