Yay, another Google Cloud Storage Python Library


There’s yet another Python library for Google Cloud Storage library out there, mine ;-). It’s called gcs-client and it’s a somewhat mix of Google’s Api Python Client (without the discovery) and Google’s App Engine GCS Client.

google cloud storage

A couple of weeks ago I wanted to play with Google’s Cloud Storage using Python and found that none of the big libraries fit my profile. On one had I had the Google Api Python Client which is really neat, but it is too generic, the upload functionality is somewhat limited when you have dynamic data and it’s not very pythonic not to mention that it doesn’t retry on 429 errors (too many requests) as Google documentation recommends, on the other had I had Google’s App Engine GCS Client which has everything I wanted but is meant to be used inside Google’s App Engine.

So I decided to create yet another library, this one specific to Google’s Cloud Storage and a little more pythonic. And while it is still on the early stages of development and missing a lot of features, it still has the most basic functionality:

  • Creating buckets
  • Deleting buckets
  • Listing buckets in a project
  • Getting default bucket for a project
  • Getting bucket attributes
  • Listing objects in a bucket
  • Getting objects attributes
  • Deleting objects
  • Reading object contents
  • Writing an object
  • Configurable retries with Truncated Exponential Backoff

The library can be easily installed with from pypi: pip install gcs-client.

Here’s an example of the library usage that prints the buckets of a project called ‘project_name’ using ‘private_key.json’ credentials, display some info on one of the buckets and list the objects inside that bucket (or creates one if it’s empty).

 import gcs_client

 credentials = gcs_client.Credentials('private_key.json')
 project = gcs_client.Project('project_name', credentials)

 # Print buckets in the project
 buckets = project.list()
 print 'Buckets:\n\t- ','\n\t- '.join(map(str, buckets))

 # Print some information from first bucket
 bucket = buckets[0]
 print 'Bucket %s is located in %s with storage class %s' % (bucket, bucket.location, bucket.storageClass)

 # List the objects in the bucket
 objects = bucket.list()
 if not objects:
     print 'There are no objects, creating one'
     filename = '/tmp/my_file.txt'
     with bucket.open(filename, 'w') as f:
         f.write('this is a test file\n' * 100)
     objects = [gcs_client.Object(bucket.name, filename, credentials=credentials)]

 if objects:
     print '\t','\n\t'.join(map(lambda o: o.name + ' has %s bytes' % o.size, objects))
     # Read the contents from the first file
     with objects[0].open() as obj:
         print 'Contents of file %s are:\n' % obj.name, obj.read()
 else:
     print 'There are no objects, nothing to do'

You can read more about the library on read the docs and look at the source code on git

Leave a comment

Your email address will not be published. Required fields are marked *