Standalone Cinder: The definitive SDS 6


Are you looking for the best Software Defined Storage in the market? Look no further, Standalone Cinder is here! Let’s have an overview of the Standalone Cinder service, see some specific configurations, and find out how to make requests with no other OpenStack service is deployed.

standalone

Cinder

Until not so long ago Cinder was always mentioned in an OpenStack context, but for some time now you could hear conversations where Cinder was standing on its own and was discussed as a independent service out of the OpenStack ecosystem to function as an SDS provider for non OpenStack environments.

When looking at Cinder many people only saw it as one more OpenStack services and didn’t see all of its possibilities, but those with more foresight realized a while back of the possibilities the OpenStack block storage service had to offer to the outside world, and started working on ways to use Cinder without Nova for other OpenStack projects at first, and then on its own as an independent SDS.

There are many projects out there requiring access and control of heterogeneous storage solutions, and when engineers on these projects start considering the implications of developing a solution that supports storage devices from different vendors the most common reaction is an acute head pain just thinking about abstraction design, multiple transport protocols support, proprietary control protocols, access to hardware for testing, keeping up to date with new solutions… The list just never ends!

So to avoid reinventing the wheel the normal approach is to look in the market for an existing solution that can leveraged, and you really can’t find a better solution than Cinder with its:

  • REST API interface
  • No vendor lock-in
  • The myriad drivers
  • Broad market adoption
  • Rich and diverse community
  • Direct involvement from vendor in driver development

When trying to develop a solution that requires access to storage solutions from different vendors most people’s head will start spinning just thinking about the implications, abstraction design, multiple protocol support, access to hardware for testing, being up to date with new solutions…

Unfortunately when people start looking at Cinder for an SDS solution they are usually overwhelmed by the lack of information in this regard and the implication in all available documentation that Cinder is tightly coupled to other OpenStack projects like Nova, Glance, Keystone. So facing this situation without any prior OpenStack experience can be a daunting experience, so it is my hope that this short post will help alleviate this pain.

Overview

To keep the post brief and to the point we won’t be going over all the configuration details to get a working standalone Cinder service, since that’s already documented and it’s common to all Cinder deployments, so we’ll only be addressing the ten thousand feet view of the requirements and the interesting specifics that may require more effort to discover.

When deploying Cinder in an OpenStack environment will be comprised of the API, Volume, Scheduler, and Backup Cinder services, will interact with Horizon (dashboard), Nova (compute), Keystone (identity), Glace (image), Heat (orchestration), Ironic (Bare Metal), and Ceilometer (data collection) services, and will require a message broker, a DBMS, and a storage device.

Those are the services that one must usually considers when looking at the Cinder service, but when looking at Cinder as a standalone service we will most likely want to strip it of all unnecessary surrounding services to reduce it to the bare minimum required to serve as a pure SDS; and these are the API, Volume, and Scheduler Cinder services together with all the supporting services, namely the message broker, the DBMS, and the storage device/s that Cinder will be managing.

For the message broker that Cinder uses in the method invocation between the API, Scheduler, and Volume services, we can choose -through the Oslo Messaging library- between RabbitMQ, Qpid, and ZeroMQ. For additional information on their specific configurations please look into Oslo messaging and the configuration reference 2.

NOTE: ZeroMQ was not properly supported in earlier Cinder releases, but it is now in the upcoming Cinder Ocata release thanś to Michał Dulko’s efforts

A SQL database is required for the DBMS, and both MySQL and PostgreSQL are supported through SQLAlchemy. More information can be found in the OpenStack Database documentation and installation database example.

For the storage device/s, any of the supported drivers can be used, even LVM file based using a loopback device, although this is obviously not recommended.

Cinder configuration

Basic Cinder configuration is beyond the point of this short post and should be well covered in the Cinder documentation and vendor’s drivers documentation.

From Cinder’s perspective it’s quite easy to do without most of the OpenStack services since the service is mainly acting as a provider and doesn’t call any of the services of it’s own accord. This is the case of Horizon, Nova, Heat, and Ironic services. So we can just go ahead and ignore all those as if they didn’t even exist.

NOTE: Without Nova, special feature assisted snapshots will not be available.

Ceilometer isn’t a problem either since Cinder doesn’t require it to be present as Oslo messaging’s notifier will not send anything to the message broker if the notification transport mechanism is not configured. Although nothing gets actually sent, the actual mechanism is still unnecessarily wasteful as it is now, since we go through the trouble of preparing the actual message -sometimes this entails querying the DB-, so I created a Launchpad bug and proposed a patch to short-circuit this mechanism when there’s no notification transport mechanism configured in Cinder.

Now that we know we can ignore Horizon, Nova, Heat, Ironic, and Ceilometer, that leaves us with with Keystone and Glace.

Keystone is used not only as the authentication provider but also for the service catalog, which means that if we want to remove Keystone in our deployment we’ll have to know where our cinder service is to make our requests, which doesn’t seem like a problem, and to disable Keystone authentication.

Disabling Keystone authentication is quite simple, all we need to do is to change the auth_strategy configuration option in our cinder configuration file from keystone to noauth under the [DEFAULT] section. And since we won’t be using Keystone that means that we won’t be needing any [keystone_authtoken] section in the configuration.

If disabling authentication is not an option for our use case we can always add custom Middleware to deal with our own authentication system. How the authentication mechanism works and how to add a custom authentication mechanism are out of the scope of this post, but I’ll to do a write up on it in the future.

We are almost done, now we only have to look at Glance. As the image provider service, Glance is a service frequently used by Cinder, so if we don’t deploy it in our environment that means that any request we make to Cinder that requires of this service will fail after Cinder fails to contact Keystone to get the location of Glance from the catalog, or to connect directly to Glance if we have set it up that way.

Some of the operations that require the usage of Glance and that will not be available are creating a volume from an image, and uploading a volume to an image. For the upload to image functionality we can prevent it from being accidentally used using the policy mechanism, but for the volume creation using an image as the source we currently have no mechanism to disable it, so I proposed one.

To disable creating image from a volume we have to make sure our JSON Cinder policy -usually in /etc/cinder/policy.json– for that action is set accordingly:

   "volume_extension:volume_actions:upload_image": "!",

More information on OpenStack policies in the documentation and the configuration reference.

If we want/need to have those features available we can always deploy Glance without authorization just like we are doing with Cinder.

Using Cinder

Now that we have deployed Cinder without authentication it is time to use it. So we know that Cinder has a REST API interface, and that we have disabled authentication, so the first thing we’ll try to do is to do a sanity check of our service with a request using curl.

CURL

As a first step we go to the Cinder API documentation and look for one of the simplest REST API request and we pick the volume listing endpoint, and see that we require a tenant_id parameter for that – /v3/{tenant_id}/volumes/detail – so we decide to call the tenant admin_tenant and we make the query, but to our surprise it doesn’t work as expected:

user@localhost:$ curl -i http://192.168.121.207:8776/v3/admin_tenant/volumes/detail
HTTP/1.1 204 No Content
Content-Length: 0
X-Auth-Token: admin:admin
X-Server-Management-Url: http://192.168.121.207:8776/v3/admin_tenant/volumes/detail/admin
Content-Type: text/plain; charset=UTF-8
X-Openstack-Request-Id: req-2c957c59-b7da-4b8b-8359-9e76fbc7f151
Date: Wed, 18 Jan 2017 23:36:40 GMT

So what happened here? Well, if you think about it you’ll see it makes sense, you have disabled authentication, but the authorization mechanism is still in place, and since Cinder is a multi-tenant service, you’ll have to specify the user and project/tenant to use on your requests, even if they are not going to be authenticated.

Not all projects handle this in the same way, but Cinder and Nova both use the X-Auth-Token header field to pass the user and project’s ids separated by a colon. Cinder will accept any arbitrary string up to 255 characters in length as user and project, they don’t need to be a valid UUID. So now that we know this we can try again:

user@localhost:$ curl -i http://192.168.121.207:8776/v3/admin_tenant/volumes/detail -H "Accept: application/json" -H "X-Auth-Token: admin_user:admin_tenant"
HTTP/1.1 200 OK
X-Compute-Request-Id: req-6a772a89-f1cc-4415-bb1f-900a07e1e3ce
Content-Type: application/json
Content-Length: 15
Openstack-Api-Version: volume 3.0
Vary: OpenStack-API-Version
X-Openstack-Request-Id: req-328fe1c6-ef3b-4631-b959-fd09bff6c00c
Date: Wed, 18 Jan 2017 23:48:03 GMT

{"volumes": []}

Success!! And no surprise here, we get an empty list since we haven’t created any volumes.

If we want the user id and project id to be the same we can just provide one value in the X-Auth-Token header and it will used for both:

user@localhost:$ curl -i http://192.168.121.207:8776/v3/admin_tenant/volumes/detail -H "Accept: application/json" -H "X-Auth-Token: admin_tenant"
HTTP/1.1 200 OK
X-Compute-Request-Id: req-d60fbe3b-bf58-4222-ba0c-66d7ccb7d14a
Content-Type: application/json
Content-Length: 15
Openstack-Api-Version: volume 3.0
Vary: OpenStack-API-Version
X-Openstack-Request-Id: req-d2006042-32ce-4515-a5f1-85ca721c0e5d
Date: Thu, 19 Jan 2017 00:05:55 GMT

{"volumes": []}

NOTE: Please be aware that the tenant in the URL and the project in the X-Auth-Token header must match or we’ll get a 400 malformed request url error

Since we have disabled authentication Cinder will believe any user and tenant id we send and it will treat all request as made by an admin user. Having users and tenants is still useful, as we’ll be able to use all Cinder features normally, like filtering volumes by user and not getting all volumes mixed up when we list them.

Even though it’s not supported by Cinder, you can make some small modifications to the code -or create a custom middleware authenticator- if you want to always use the same user and not require the X-Auth-Token header in your REST API calls, or you can even set all users as non admin users unless the user parameter is “admin”. To do this you just need to change the NoAuthMiddleware class in cinder/api/middleware/auth.py.

For example, to always use admin user and project you’ll need to change the class to something like this:

class NoAuthMiddleware(base_wsgi.Middleware):
    """Return a fake token if one isn't specified."""

    @webob.dec.wsgify(RequestClass=wsgi.Request)
    def __call__(self, req):
        token = req.headers.get('X-Auth-Token', 'admin')
        user_id, _sep, project_id = token.partition(':')
        project_id = project_id or user_id
        remote_address = getattr(req, 'remote_address', '127.0.0.1')
        if CONF.use_forwarded_for:
            remote_address = req.headers.get('X-Forwarded-For', remote_address)
        ctx = context.RequestContext(user_id,
                                     project_id,
                                     is_admin=True,
                                     remote_address=remote_address)

        req.environ['cinder.context'] = ctx
return self.application

And if we want to only make the context as admin when the user is “admin” we just have to change is_admin=True, with is_admin=user_id=='admin',.

For API version 3 one must remember that Cinder uses microversioning to support introducing breaking changes within the same major version, so we’ll also need to send header 'OpenStack-API-Version with the version we want to use, for example for version 3.10 we would do:

user@localhost:$ curl -i http://192.168.121.207:8776/v3/admin_tenant/volumes/detail -H "Accept: application/json" -H "X-Auth-Token: admin_tenant" -H "OpenStack-API-Version: volume 3.10"
HTTP/1.1 200 OK
X-Compute-Request-Id: req-8b85cae1-fbe0-46b2-89e9-47f9bc4bc9b4
Content-Type: application/json
Content-Length: 15
Openstack-Api-Version: volume 3.10
Vary: OpenStack-API-Version
X-Openstack-Request-Id: req-e27a59d2-f4da-4a3a-a936-b2e2b8bb02b1
Date: Thu, 19 Jan 2017 00:16:03 GMT

{"volumes": []}

Cinder client

We are now up to a great start as we can already send commands, but there are so many operations and parameters available in Cinder -more so now than ever with the introduction of microversions- that it’s hard to find the ones we want, and sometimes we just want to do a quick test, or want to see a specific example of the query we need to send, or we just don’t want to do the API queries ourselves; and this is where the Cinder client can be a great help for us since it has support to almost all possible operations, can work as a CLI tool or a a python library to use in your own code, and can present the exact curl queries that have been used to make the call.

The cinder client is the official client for Cinder, together with the OpenStack client, but it currently doesn’t have support for noauth and Ivan Kolodyazhny is working on fixing that, but in the meantime I have created a hack that will get the job done and support noauth.

That patch is unlikely to get merged since it is just a hack that may not be up to Cinder’s standards, but we can still use it installing from master and pulling the patch from gerrit:

user@localhost:$ virtualenv .venv
New python executable in /home/vagrant/.venv/bin/python
Installing setuptools, pip, wheel...done.


user@localhost:$ . .venv/bin/activate

user@localhost:$ pip install git+https://github.com/openstack/python-cinderclient.git
Collecting git+https://github.com/openstack/python-cinderclient.git
Cloning https://github.com/openstack/python-cinderclient.git to /tmp/pip-IUpNW1-build
...
Successfully installed Babel-2.3.4 PrettyTable-0.7.2 debtcollector-1.11.0 func ...

user@localhost:$ curl https://review.openstack.org/changes/425277/revisions/714ceeb94248c202944e2461fbf060603e23af1c/patch?zip | gunzip -c - | patch -d .venv/lib/python2.7/site-packages/cinderclient
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 3489 0 3489 0 0 5094 0 --:--:-- --:--:-- --:--:-- 5093
patching file client.py
Hunk #1 succeeded at 605 (offset -4 lines).
Hunk #2 succeeded at 655 (offset -4 lines).
patching file shell.py

NOTE: the download URL will change as new patches are submitted to resolve merge conflicts, so the patch should be checked for the latest download link.

So with that patch in place we can now instruct the Cinder client to work with noauth by passing the --os-auth-system=noauth parameter in the command line or setting the OS_AUTH_SYSTEM=noauth environment variable.

When working with noauth the client will require at least 2 parameters, the user id and the cinder URL. The project id is optional as will default to the user id if not provided:

  • User-id: –os-user-id=uuid_or_name or OS_USER_ID=uuid_or_name
  • URL: –os-cinder-url=http://127.0.0.1:8776 or OS_CINDER_URL=http://127.0.0.1:8776
  • Project-id: –os-project-id=uuid_or_name or OS_PROJECT_ID=uuid_or_name
user@localhost:$ cinder --os-auth-system=noauth --os-cinder-url="http://192.168.121.207:8776" --os-user-id=admin list
+----+--------+------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+----+--------+------+------+-------------+----------+-------------+
+----+--------+------+------+-------------+----------+-------------+

Now we can see what the actual curl query is using the --debug option:

user@localhost:$ cinder --os-auth-system=noauth --os-cinder-url="http://192.168.121.207:8776" --os-user-id=admin --debug list
REQ: curl -i http://192.168.121.207:8776/v2/admin/volumes/detail -X GET-H 'X-Auth-Project-Id: admin'-H 'User-Agent: python-cinderclient'-H 'Accept: application/json'-H 'X-Auth-Token: {SHA1}c759eaf09e4638954f63ace0ce1b53b40f62ccb7'

DEBUG:cinderclient.client:
REQ: curl -i http://192.168.121.207:8776/v2/admin/volumes/detail -X GET-H 'X-Auth-Project-Id: admin'-H 'User-Agent: python-cinderclient'-H 'Accept: application/json'-H 'X-Auth-Token: {SHA1}c759eaf09e4638954f63ace0ce1b53b40f62ccb7'

RESP: [200] {'Content-Length': '15', 'X-Compute-Request-Id': 'req-9cd9f7d4-80e4-4cbd-a8b4-4f2c1d42ee38', 'Connection': 'keep-alive', 'Date': 'Thu, 19 Jan 2017 01:28:41 GMT', 'Content-Type': 'application/json', 'X-Openstack-Request-Id': 'req-09473d96-1083-4db9-839a-c8528f219121'}
RESP BODY: {"volumes": []}

DEBUG:cinderclient.client:RESP: [200] {'Content-Length': '15', 'X-Compute-Request-Id': 'req-9cd9f7d4-80e4-4cbd-a8b4-4f2c1d42ee38', 'Connection': 'keep-alive', 'Date': 'Thu, 19 Jan 2017 01:28:41 GMT', 'Content-Type': 'application/json', 'X-Openstack-Request-Id': 'req-09473d96-1083-4db9-839a-c8528f219121'}
RESP BODY: {"volumes": []}

GET call to None for http://192.168.121.207:8776/v2/admin/volumes/detail used request id req-09473d96-1083-4db9-839a-c8528f219121
DEBUG:cinderclient.client:GET call to None for http://192.168.121.207:8776/v2/admin/volumes/detail used request id req-09473d96-1083-4db9-839a-c8528f219121
+----+--------+------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+----+--------+------+------+-------------+----------+-------------+
+----+--------+------+------+-------------+----------+-------------+

NOTE: Remember that your tenants will still be subject to existing quota restrictions.

Now we’ll be able to do all operations supported by the Cinder client, but there are some operations that are not supported by the client because they were only called by Nova and didn’t make sense to be be exposed to outside users. But please don’t misunderstand it, the REST API is open and can be accessed by anyone, it’s just that the client doesn’t expose those operations.

There is an ongoing effort to expose these operations through a Cinder client extension and you can read more about it in one of Ivan’s posts and in the brief documentation available.

Recapitulation

Cinder as a standalone SDS service is the perfect solution to seamlessly access a wide array of heterogeneous vendor storage devices through it’s REST API support and broad range of operations.

The service can be accessed programmatically using REST API requests, or using the Cinder client as a library or as a CLI tool.

We have recently started an Etherpad to gather use cases and potential work that we would like to do to cover all user cases. If you have any specific use case, functionality, or bug, please don’t hesitate to add it to the Etherpad so it can be taken into consideration.


Picture: “ms. independent” by darwin Bell is licensed under CC BY-NC 2.0


Leave a comment

Your email address will not be published. Required fields are marked *

6 thoughts on “Standalone Cinder: The definitive SDS

  • Arpit

    Hey, Need you help.
    I have following cinder.conf.
    cinder.conf
    My cinder.conf looks like :
    [database]
    sql_connection=mysql://cinder:*CINDER_DBPASS*@localhost/cinder

    [DEFAULT]
    auth_strategy = noauth
    rootwrap_config = /etc/cinder/rootwrap.conf
    api_paste_confg = /etc/cinder/api-paste.ini
    iscsi_helper = tgtadm
    volume_name_template = volume-%s
    volume_group = cinder-volumes
    verbose = True
    state_path = /var/lib/cinder
    lock_path = /var/lock/cinder
    volumes_dir = /var/lib/cinder/volumes
    volume_driver = cinder.volume.drivers.lvm.LVMISCSIDriver
    iscsi_protocol = iscsi

    and setting these env variables:
    export OS_TENANT_NAME=default
    export OS_USERNAME=user
    export OS_PASSWORD=password
    export OS_AUTH_STRATEGY=noauth
    #export OS_AUTH_URL=http://127.0.0.1:8776/v2/
    export OS_AUTH_SYSTEM=noauth
    After this I am getting :
    “ERROR: HTTPConnectionPool(host=’127.0.0.1′, port=8776): Max retries exceeded with url: /v2 (Caused by NewConnectionError(‘: Failed to establish a new connection: [Errno 111] Connection refused’,))”

    It would be really helpful if you help me on this..thanks!

    • geguileo Post author

      Hi Arpit,

      I don’t know why it’s failing, but looking at the cinder-api logs and displaying client’s debug logs (with “–debug”) should help shed some light into it.

      You could also try using “curl” to make sure there’s nothing going on with the cinder client.

      Cheers,
      Gorka.