Manual validation of Cinder A/A patches 2

 

In the Cinder Midcycle I agreed to create some sort of document explaining the manual tests I’ve been doing to validate the work on Cinder’s Active-Active High Availability -as a starting point for other testers and for the automation of the tests- and writing a blog post was the most convenient way for me to do so, so here it is.

checklist

Scope

The Active-Active High Availability work in Cinder is formed by a good number of specs and patches, and most of them have not yet been merged and some have not even been created, yet we are at a point were we can start testing things to catch bugs and performance bottlenecks as soon as possible.

We have merged in master -Newton cycle- most of the DLM work and all of the patches that form the foundation needed for the new job distribution and cleanup mechanisms, but we decided not to include in this cycle any patches that changed the way we do the job distribution or the cleanup since those also affect non clustered deployments; we wanted to be really sure we are not introducing any bugs in normal deployments.

The scope of the tests I’m going to be discussing in this post is limited to the job distribution and cleanup mechanism using the Tooz library with local file locks instead of a DLM. This way we’ll be able to follow a classic crawl-walk-run approach where we first test these mechanisms ignoring the DLM variable from the equation, together with the potential configuration and communication issues it brings. Later we’ll add the DLM as well as connection failures simulation.

Since the two mechanisms to test are highly intertwined we’ll be testing them both at the same time.

The explanation provided in this post is not only useful for testing existing code, but it’s also interesting for driver maintainers, as they can start testing their drivers to confirm they are ready for the Active-Active work. It is true that this would be a very basic check -since proper tests require of a DLM and having the services deployed in different hosts- but it would allow them to get familiar with it and catch the most obvious issues.

It is important that all driver maintainers are able to start working on this at the same time to ensure fair treatment for all, otherwise driver maintainers working on the A/A feature would have an unfair advantage.

Deployment

For the initial tests we want to keep things as simple as possible, so we’ll only have 1 cinder-api service -so no HAProxy configuration is needed-, only 1 scheduler (it’s really easy to add more, but it’ll make it harder to debug, so only 1 for now), we’ll not be using a DLM like we said earlier, and we will also be using local storage using the LVM driver -yes you read it right- to simplify our deployment. For this we’ll be using an all in one deployment using DevStack, as it will reduce configuration requirements since we don’t need services deployed in one host -or VM- to communicate with another host -or VM-. I know this sounds counter-intuitive, but this is good enough for now as you’ll see and in the near future we’ll expand this configuration to do more real tests.

To run 2 cinder-volume services in a clustered configuration under the same devstack deployment all you really need is to pull the latest patch in the HA A/A series from Gerrit and then run both services with the same cluster and different host configuration options. But since we are going to perform some additional tests it will be good to configure a little bit more.

Our DevStack configuration will do these things:

  • Download the Cinder component from the latest patch in the A/A branch (at the moment of this writing it’s the 6th patch for Scheduler’s Cosmetic Changes first patch for Add remaining operations to cluster).
  • Set over subscription ration to 10.0 (since we won’t be really writing anything in most cases).
  • Configure the host parameter instead of using the default value.
  • Create 2 LVM backends of 5GB each
  • Set backends to use thin provisioning

So first we must edit the local.conf and make sure we have included these lines:

# Retrieve Cinder code from gerrit's A/A work
CINDER_REPO=https://review.openstack.org/p/openstack/cinder
CINDER_BRANCH=refs/changes/68/355968/1

# 5GB LVM backends
VOLUME_BACKING_FILE_SIZE=5125M

# 2 backends
CINDER_ENABLED_BACKENDS=${CINDER_ENABLED_BACKENDS:-lvm:lvmdriver-1,lvm:lvmdriver-2}

[[post-config|$CINDER_CONF]]
[DEFAULT]
# Don't use default host name
host = host1
[lvmdriver-1]
lvm_type = thin
lvm_max_over_subscription_ratio = 10.0
[lvmdriver-2]
lvm_type = thin
lvm_max_over_subscription_ratio = 10.0

For reference this is the local.conf file I use. If you want to use other storage backends you just need to adapt above configuration to the backend driver.

We didn’t configure the cluster option on purpose, don’t worry, we’ll do it later after we’ve done some tests.

You’ll need to create 2 files both with the cluster option -same value on both- and only one of them with the host option.

These were the commands I run after updating my VM and made sure it had git installed, they basically cloned devstack, downloaded my devstack configuration, deployed a devstack, created the 2 configuration files for later, created a new window with logging, left the command I’ll need to run to start the second service, and attach to the stack screen session.

user@localhost:$ git clone https://git.openstack.org/openstack-dev/devstack

user@localhost:$ cd devstack

user@localhost:$ curl -o local.conf http://gorka.eguileor.com/files/cinder/manual_ha_aa_local.conf

user@localhost:$ ./stack.sh

user@localhost:$ echo -e "[DEFAULT]\ncluster = mycluster" > /etc/cinder/host1.conf

user@localhost:$ echo -e "[DEFAULT]\ncluster = mycluster\nhost = host2" > /etc/cinder/host2.conf

user@localhost:$ screen -S stack -X screen -t c-vol2

user@localhost:$ screen -S stack -p c-vol2 -X logfile /opt/stack/logs/c-vol2.log

user@localhost:$ screen -S stack -p c-vol2 -X log on

user@localhost:$ touch /opt/stack/logs/c-vol2.log

user@localhost:$ screen -S stack -p c-vol2 -X stuff $'/usr/bin/cinder-volume --config-file /etc/cinder/cinder.conf --config-file /etc/cinder/host2.conf & echo $! >/opt/stack/status/stack/c-vol2.pid; fg || echo "c-vol failed to start" | tee "/opt/stack/status/stack/c-vol2.failure"'

user@localhost:$ screen -x stack -p 0

Code changes

There are some cases were Cinder may perform operations too fast for us to act on them, be it to check that something has occurred or to make Cinder do something, so we’ll be making some small modifications to cinder’s code to create some delays that will give us some leeway.

Required changes to the code are:

cinder/volume/utils.py

def introduce_delay(seconds, operation='-', resource_id='-'):
    for __ in range(seconds):
        time.sleep(1)
        LOG.debug(_('Delaying %(op)s operation on %(id)s.'),
                  {'op': operation, 'id': resource_id})

And then we need to introduce calls to it from cinder/volume/flows/manager/create_volume.py and cinder/volume/manager.py in create volume, delete volume, create snapshot and delete snapshot, so that we have a 30 seconds delay before actually performing the operation and in the case of doing a volume creation from an image, the delay should be right after we have changed the status to “downloading”.

We can do these code changes manually or we can just change the CINDER_BRANCH inside local.conf to point to a patch that I specifically created to introduce these delays for my manual tests.

CINDER_BRANCH=refs/changes/69/353069/3

If you are using my configuration you are already pointing to that patch.

1. Non clustered tests

My recommendation is to split the screen session so we can see multiple windows at the same time as it will allow us to execute commands and follow the flow from the API, SCH, and VOL nodes.

We all have our preferences, but when I’m working on these tests I usually have the screen session horizontally split in at least 5 sections -command line, c-api, c-sch, c-vol, c-vol2- and I tend to reorder my windows so the cinder windows are at the beginning in the same order as I listed going from 0 to 4, and having the c-back window as number 5, number 6 a mysql connection, and number 7 with a vim editor.

The reason why we didn’t add the cluster configuration when deploying DevStack is because we wanted to check the non clustered deployment first.

1.0 – Sanity checks

The first thing we should do, now that we have a DevStack running and before doing any tests, is do some checks that will allow us to do some basic sanity checks once we run 2 services in the same cluster:

  • Check that there are no clusters:
user@localhost:$ cinder --os-volume-api-version 3.11 cluster-list --detail
+------+--------+-------+--------+-----------+----------------+----------------+-----------------+------------+------------+
| Name | Binary | State | Status | Num Hosts | Num Down Hosts | Last Heartbeat | Disabled Reason | Created At | Updated at |
+------+--------+-------+--------+-----------+----------------+----------------+-----------------+------------+------------+
+------+--------+-------+--------+-----------+----------------+----------------+-----------------+------------+------------+

  • Check services and notice that the Cluster field is empty for all services:
user@localhost:$ cinder --os-volume-api-version 3.11 service-list
+------------------+-------------------+------+---------+-------+----------------------------+---------+-----------------+
| Binary | Host | Zone | Status | State | Updated_at | Cluster | Disabled Reason |
+------------------+-------------------+------+---------+-------+----------------------------+---------+-----------------+
| cinder-backup | host1 | nova | enabled | up | 2016-08-08T17:33:26.000000 | - | - |
| cinder-scheduler | host1 | nova | enabled | up | 2016-08-08T17:33:23.000000 | - | - |
| cinder-volume | host1@lvmdriver-1 | nova | enabled | up | 2016-08-08T17:33:30.000000 | - | - |
| cinder-volume | host1@lvmdriver-2 | nova | enabled | up | 2016-08-08T17:33:31.000000 | - | - |
+------------------+-------------------+------+---------+-------+----------------------------+---------+-----------------+

  • Check there’s no RabbitMQ cluster queue:
user@localhost:$ sudo rabbitmqctl list_queues name | grep cinder-volume.mycluster

user@localhost:$
  • Check that the workers table is empty:
user@localhost:$ mysql cinder -e 'select * from workers;'

user@localhost:$

1.1 – Creation

The most basic thing we need to test is that we are able to create a volume and that we are creating the workers table entry:

user@localhost:$ cinder create --name mydisk 1; sleep 3; mysql cinder -e 'select * from workers;'
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2016-08-08T17:52:47.000000 |
| description | None |
| encrypted | False |
| id | 16fcca48-8729-44ab-b024-ddd5cfd458a4 |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | mydisk |
| os-vol-host-attr:host | None |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 2e00c4d79a5f49708438f8d3761a6d3d |
| replication_status | disabled |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| updated_at | None |
| user_id | f78b297498774851a758b08385e39b77 |
| volume_type | lvmdriver-1 |
+--------------------------------+--------------------------------------+
+---------------------+---------------------+------------+---------+----+---------------+--------------------------------------+----------+------------+
| created_at | updated_at | deleted_at | deleted | id | resource_type | resource_id | status | service_id |
+---------------------+---------------------+------------+---------+----+---------------+--------------------------------------+----------+------------+
| 2016-08-08 17:52:47 | 2016-08-08 17:52:48 | NULL | 0 | 2 | Volume | 16fcca48-8729-44ab-b024-ddd5cfd458a4 | creating | 3 |
+---------------------+---------------------+------------+---------+----+---------------+--------------------------------------+----------+------------+

user@localhost:$

We can see that we have a new entry in the workers table for the volume that is being created, and this operation is being performed by the service #3.

It is important that once the volume has been created we check that the workers table table is empty.

user@localhost:$ cinder list
+--------------------------------------+-----------+--------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+--------+------+-------------+----------+-------------+
| 16fcca48-8729-44ab-b024-ddd5cfd458a4 | available | mydisk | 1 | lvmdriver-1 | false | |
+--------------------------------------+-----------+--------+------+-------------+----------+-------------+

user@localhost:$ mysql cinder -e 'select * from workers;'

user@localhost:$

1.2 – Deletion

Now we proceed to delete the newly created volume and make sure that we also have the workers DB entry while the operation is undergoing and that it is removed once it has completed.

user@localhost:$ cinder delete mydisk; sleep 3; mysql cinder -e 'select * from workers;'
Request to delete volume mydisk has been accepted.
+---------------------+---------------------+------------+---------+----+---------------+--------------------------------------+----------+------------+
| created_at | updated_at | deleted_at | deleted | id | resource_type | resource_id | status | service_id |
+---------------------+---------------------+------------+---------+----+---------------+--------------------------------------+----------+------------+
| 2016-08-08 18:26:16 | 2016-08-08 18:26:16 | NULL | 0 | 3 | Volume | 16fcca48-8729-44ab-b024-ddd5cfd458a4 | deleting | 3 |
+---------------------+---------------------+------------+---------+----+---------------+--------------------------------------+----------+------------+

user@localhost:$ cinder list
+----+--------+------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+----+--------+------+------+-------------+----------+-------------+
+----+--------+------+------+-------------+----------+-------------+

user@localhost:$ mysql cinder -e 'select * from workers;'

user@localhost:$

1.3 – Cleanup

We are going to test that basic cleanup works when the node dies, for this we’ll create a volume that we’ll attach to a VM, create another volume and start creating a snapshot, start creating a volume from an image, start creating a volume, and start deleting another volume.

So in the end we’ll have the following cleanable volume statuses:

  • “in-use”
  • “creating”
  • “deleting”
  • “downloading”

Snapshot:

  • “creating”

The sequence of commands and results would look like this:

user@localhost:$ nova boot --flavor m1.nano --image cirros-0.3.4-x86_64-uec myvm
+--------------------------------------+----------------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | |
| OS-EXT-SRV-ATTR:host | - |
| OS-EXT-SRV-ATTR:hostname | myvm |
| OS-EXT-SRV-ATTR:hypervisor_hostname | - |
| OS-EXT-SRV-ATTR:instance_name | instance-00000001 |
| OS-EXT-SRV-ATTR:kernel_id | 4c1a9ce2-a78e-43ec-99e3-5b532359d62c |
| OS-EXT-SRV-ATTR:launch_index | 0 |
| OS-EXT-SRV-ATTR:ramdisk_id | 7becc8ae-0153-44be-9b81-9c94e5c7849a |
| OS-EXT-SRV-ATTR:reservation_id | r-qy2z7bpp |
| OS-EXT-SRV-ATTR:root_device_name | - |
| OS-EXT-SRV-ATTR:user_data | - |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| adminPass | Xk7g2tbzVNjg |
| config_drive | |
| created | 2016-08-09T10:43:23Z |
| description | - |
| flavor | m1.nano (42) |
| hostId | |
| host_status | |
| id | 74fa7147-a0c5-4b17-b9b3-ce4ebfbea911 |
| image | cirros-0.3.4-x86_64-uec (432c9a2b-8ed2-4957-8d12-063217f26a3f) |
| key_name | - |
| locked | False |
| metadata | {} |
| name | myvm |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| security_groups | default |
| status | BUILD |
| tags | [] |
| tenant_id | 83e1beb749d74956b664ef58c001af29 |
| updated | 2016-08-09T10:43:23Z |
| user_id | c21ca8dae0644e52afe624a518e5e8f2 |
+--------------------------------------+----------------------------------------------------------------+


user@localhost:$ cinder create --name attached 1
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2016-08-09T10:43:31.000000 |
| description | None |
| encrypted | False |
| id | a9102b47-37ff-4fd2-a76c-44e50c00e1fd |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | attached |
| os-vol-host-attr:host | None |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 83e1beb749d74956b664ef58c001af29 |
| replication_status | disabled |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| updated_at | None |
| user_id | c21ca8dae0644e52afe624a518e5e8f2 |
| volume_type | lvmdriver-1 |
+--------------------------------+--------------------------------------+


user@localhost:$ nova volume-attach myvm a9102b47-37ff-4fd2-a76c-44e50c00e1fd
+----------+--------------------------------------+
| Property | Value |
+----------+--------------------------------------+
| device | /dev/vdb |
| id | a9102b47-37ff-4fd2-a76c-44e50c00e1fd |
| serverId | 74fa7147-a0c5-4b17-b9b3-ce4ebfbea911 |
| volumeId | a9102b47-37ff-4fd2-a76c-44e50c00e1fd |
+----------+--------------------------------------+


user@localhost:$ cinder create --name deleting_vol 1
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2016-08-09T12:25:06.000000 |
| description | None |
| encrypted | False |
| id | 36fcb60b-83fc-420b-94cb-1f8f7979ea9d |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | deleting_vol |
| os-vol-host-attr:host | None |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 83e1beb749d74956b664ef58c001af29 |
| replication_status | disabled |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| updated_at | None |
| user_id | c21ca8dae0644e52afe624a518e5e8f2 |
| volume_type | lvmdriver-1 |
+--------------------------------+--------------------------------------+


user@localhost:$ cinder create --name snapshot_vol 1
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2016-08-09T12:11:33.000000 |
| description | None |
| encrypted | False |
| id | 6a12f169-c6a7-4de5-85f2-c8259cbd6924 |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | snapshot_vol |
| os-vol-host-attr:host | None |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 83e1beb749d74956b664ef58c001af29 |
| replication_status | disabled |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| updated_at | None |
| user_id | c21ca8dae0644e52afe624a518e5e8f2 |
| volume_type | lvmdriver-1 |
+--------------------------------+--------------------------------------+


user@localhost:$ mysql cinder -e 'select * from workers;'


user@localhost:$ cinder list
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+
| 6a12f169-c6a7-4de5-85f2-c8259cbd6924 | available | snapshot_vol | 1 | lvmdriver-1 | false | |
| 36fcb60b-83fc-420b-94cb-1f8f7979ea9d | available | deleting_vol | 1 | lvmdriver-1 | false | |
| a9102b47-37ff-4fd2-a76c-44e50c00e1fd | in-use | attached | 1 | lvmdriver-1 | false | 74fa7147-a0c5-4b17-b9b3-ce4ebfbea911 |
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+


user@localhost:$ cinder create --name downloading --image-id cirros-0.3.4-x86_64-uec 1; cinder create --name creating 1; cinder snapshot-create snapshot_vol --name creating_snap; cinder delete deleting_vol; sleep 3; kill -9 -- -`cat /opt/stack/status/stack/c-vol.pid`
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2016-08-09T12:26:06.000000 |
| description | None |
| encrypted | False |
| id | 58d2c5aa-9334-46a1-9246-0bc893196454 |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | downloading |
| os-vol-host-attr:host | None |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 83e1beb749d74956b664ef58c001af29 |
| replication_status | disabled |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| updated_at | None |
| user_id | c21ca8dae0644e52afe624a518e5e8f2 |
| volume_type | lvmdriver-1 |
+--------------------------------+--------------------------------------+
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2016-08-09T12:26:08.000000 |
| description | None |
| encrypted | False |
| id | a7443e99-b87a-4e0a-bb44-6b63bdef477b |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | creating |
| os-vol-host-attr:host | None |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 83e1beb749d74956b664ef58c001af29 |
| replication_status | disabled |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| updated_at | None |
| user_id | c21ca8dae0644e52afe624a518e5e8f2 |
| volume_type | lvmdriver-1 |
+--------------------------------+--------------------------------------+
+-------------+--------------------------------------+
| Property | Value |
+-------------+--------------------------------------+
| created_at | 2016-08-09T12:26:10.353392 |
| description | None |
| id | acc8b408-2148-4de7-9774-ccb123650244 |
| metadata | {} |
| name | creating_snap |
| size | 1 |
| status | creating |
| updated_at | None |
| volume_id | 6a12f169-c6a7-4de5-85f2-c8259cbd6924 |
+-------------+--------------------------------------+
Request to delete volume deleting_vol has been accepted.


user@localhost:$ mysql cinder -e 'select * from workers;'
+---------------------+---------------------+------------+---------+----+---------------+--------------------------------------+-------------+------------+
| created_at | updated_at | deleted_at | deleted | id | resource_type | resource_id | status | service_id |
+---------------------+---------------------+------------+---------+----+---------------+--------------------------------------+-------------+------------+
| 2016-08-09 12:26:06 | 2016-08-09 12:26:08 | NULL | 0 | 45 | Volume | 58d2c5aa-9334-46a1-9246-0bc893196454 | downloading | 3 |
| 2016-08-09 12:26:08 | 2016-08-09 12:26:09 | NULL | 0 | 46 | Volume | a7443e99-b87a-4e0a-bb44-6b63bdef477b | creating | 3 |
| 2016-08-09 12:26:10 | 2016-08-09 12:26:10 | NULL | 0 | 47 | Snapshot | acc8b408-2148-4de7-9774-ccb123650244 | creating | 3 |
| 2016-08-09 12:26:11 | 2016-08-09 12:26:11 | NULL | 0 | 48 | Volume | 36fcb60b-83fc-420b-94cb-1f8f7979ea9d | deleting | 3 |
+---------------------+---------------------+------------+---------+----+---------------+--------------------------------------+-------------+------------+


user@localhost:$ cinder list
+--------------------------------------+-------------+--------------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-------------+--------------+------+-------------+----------+--------------------------------------+
| 36fcb60b-83fc-420b-94cb-1f8f7979ea9d | deleting | deleting_vol | 1 | lvmdriver-1 | false | |
| 58d2c5aa-9334-46a1-9246-0bc893196454 | downloading | downloading | 1 | lvmdriver-1 | false | |
| 6a12f169-c6a7-4de5-85f2-c8259cbd6924 | available | snapshot_vol | 1 | lvmdriver-1 | false | |
| a7443e99-b87a-4e0a-bb44-6b63bdef477b | creating | creating | 1 | lvmdriver-1 | false | |
| a9102b47-37ff-4fd2-a76c-44e50c00e1fd | in-use | attached | 1 | lvmdriver-1 | false | 74fa7147-a0c5-4b17-b9b3-ce4ebfbea911 |
+--------------------------------------+-------------+--------------+------+-------------+----------+--------------------------------------+


user@localhost:$ cinder snapshot-list
+--------------------------------------+--------------------------------------+----------+---------------+------+
| ID | Volume ID | Status | Name | Size |
+--------------------------------------+--------------------------------------+----------+---------------+------+
| acc8b408-2148-4de7-9774-ccb123650244 | 6a12f169-c6a7-4de5-85f2-c8259cbd6924 | creating | creating_snap | 1 |
+--------------------------------------+--------------------------------------+----------+---------------+------+

We can see how we didn’t have any entry in the workers table before we execute all the operations and killed the service, and that at the end we have expected entries in it now and that they match the status of the volumes and snapshots. This part is crucial, because if they don’t match they will not be cleaned up.

This would be a simulation of a service that is abruptly interrupted in the middle of some operations and that will need to recover on the next restart.

Before we check the restart of the service we want to remove the iSCSI target to make sure that it gets recreated on service start, but take notice that you need to change the UUID in the command by the UUID of the volume that was attached to the instance:

user@localhost:$ sudo tgt-admin --force --delete iqn.2010-10.org.openstack:volume-c632fd5d-bd05-4eda-a146-796136376ece

user@localhost:$ sudo tgtadm --lld iscsi --mode target --op show

user@localhost:$

And now we can restart the c-vol service we just killed -going to the c-vol window, pressing Ctrl+p, and hitting enter- and check that the service is actually doing what we expect it to do, which is reclaim the workers entries -changing the updated_at field since the service_id is already his- and doing the cleanup. Since most cleanups are just setting the status field to “error” we won’t see them in the workers table anymore, and only the delete operation remains until it is completed.

user@localhost:$
user@localhost:$ mysql cinder -e 'select * from workers;'
+---------------------+---------------------+------------+---------+----+---------------+--------------------------------------+----------+------------+
| created_at | updated_at | deleted_at | deleted | id | resource_type | resource_id | status | service_id |
+---------------------+---------------------+------------+---------+----+---------------+--------------------------------------+----------+------------+
| 2016-08-09 12:26:11 | 2016-08-09 12:32:52 | NULL | 0 | 48 | Volume | 36fcb60b-83fc-420b-94cb-1f8f7979ea9d | deleting | 3 |
+---------------------+---------------------+------------+---------+----+---------------+--------------------------------------+----------+------------+


user@localhost:$ cinder list
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+
| 36fcb60b-83fc-420b-94cb-1f8f7979ea9d | deleting | deleting_vol | 1 | lvmdriver-1 | false | |
| 58d2c5aa-9334-46a1-9246-0bc893196454 | error | downloading | 1 | lvmdriver-1 | false | |
| 6a12f169-c6a7-4de5-85f2-c8259cbd6924 | available | snapshot_vol | 1 | lvmdriver-1 | false | |
| a7443e99-b87a-4e0a-bb44-6b63bdef477b | error | creating | 1 | lvmdriver-1 | false | |
| a9102b47-37ff-4fd2-a76c-44e50c00e1fd | in-use | attached | 1 | lvmdriver-1 | false | 74fa7147-a0c5-4b17-b9b3-ce4ebfbea911 |
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+


user@localhost:$ cinder snapshot-list
+--------------------------------------+--------------------------------------+--------+---------------+------+
| ID | Volume ID | Status | Name | Size |
+--------------------------------------+--------------------------------------+--------+---------------+------+
| acc8b408-2148-4de7-9774-ccb123650244 | 6a12f169-c6a7-4de5-85f2-c8259cbd6924 | error | creating_snap | 1 |
+--------------------------------------+--------------------------------------+--------+---------------+------+


user@localhost:$ sudo tgtadm --lld iscsi --mode target --op show
Target 1: iqn.2010-10.org.openstack:volume-a9102b47-37ff-4fd2-a76c-44e50c00e1fd
System information:
Driver: iscsi
State: ready
I_T nexus information:
I_T nexus: 2
Initiator: iqn.1994-05.com.redhat:d434849ec720 alias: localhost
Connection: 0
IP Address: 192.168.121.80
LUN information:
LUN: 0
Type: controller
SCSI ID: IET 00010000
SCSI SN: beaf10
Size: 0 MB, Block size: 1
Online: Yes
Removable media: No
Prevent removal: No
Readonly: No
SWP: No
Thin-provisioning: No
Backing store type: null
Backing store path: None
Backing store flags:
LUN: 1
Type: disk
SCSI ID: IET 00010001
SCSI SN: beaf11
Size: 1074 MB, Block size: 512
Online: Yes
Removable media: No
Prevent removal: No
Readonly: No
SWP: No
Thin-provisioning: No
Backing store type: rdwr
Backing store path: /dev/stack-volumes-lvmdriver-1/volume-a9102b47-37ff-4fd2-a76c-44e50c00e1fd
Backing store flags:
Account information:
MoYuxFwQJQvaWJNmz47H
ACL information:
ALL

And after 30 seconds or so the volume named “deleting_vol” will finish deleting and we won’t have the workers table entry anymore:

user@localhost:$ cinder list
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+
| 58d2c5aa-9334-46a1-9246-0bc893196454 | error | downloading | 1 | lvmdriver-1 | false | |
| 6a12f169-c6a7-4de5-85f2-c8259cbd6924 | available | snapshot_vol | 1 | lvmdriver-1 | false | |
| a7443e99-b87a-4e0a-bb44-6b63bdef477b | error | creating | 1 | lvmdriver-1 | false | |
| a9102b47-37ff-4fd2-a76c-44e50c00e1fd | in-use | attached | 1 | lvmdriver-1 | false | 74fa7147-a0c5-4b17-b9b3-ce4ebfbea911 |
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+

user@localhost:$ mysql cinder -e 'select * from workers;'

user@localhost:$

2. Clustered tests

One thing to remember -as it will help us know where we can expect operations to go- is that clustered operations are scheduled using round-robin. So if we create 2 volumes one will be created in each service, and if we don an attach, each service will perform one part of the attachment, since we have the reservation and the connection initiation.

It’s important to remember that we already have some resources -volumes and snapshots- in our backend, and we should check the contents of the DB for the volumes to confirm that they don’t belong to any cluster.

user@localhost:$ mysql cinder -e 'select display_name, id, status, host, cluster_name from volumes where not deleted;'
+--------------+--------------------------------------+-----------+-------------------------------+--------------+
| display_name | id | status | host | cluster_name |
+--------------+--------------------------------------+-----------+-------------------------------+--------------+
| downloading | 58d2c5aa-9334-46a1-9246-0bc893196454 | error | host1@lvmdriver-1#lvmdriver-1 | NULL |
| snapshot_vol | 6a12f169-c6a7-4de5-85f2-c8259cbd6924 | available | host1@lvmdriver-1#lvmdriver-1 | NULL |
| creating | a7443e99-b87a-4e0a-bb44-6b63bdef477b | error | host1@lvmdriver-1#lvmdriver-1 | NULL |
| attached | a9102b47-37ff-4fd2-a76c-44e50c00e1fd | in-use | host1@lvmdriver-1#lvmdriver-1 | NULL |
+--------------+--------------------------------------+-----------+-------------------------------+--------------+

Now we have to stop the c-vol service pressing Ctr+c in the c-vol window and then press Ctrl+p to bring back the command that started the service so we can modify the command and add --config-file /etc/cinder/host1.conf right after --config-file /etc/cinder/cinder.conf before running the command. With this we are effecitvely starting the service in the cluster. The command would look like this:

user@localhost:$ usr/bin/cinder-volume --config-file /etc/cinder/cinder.conf --config-file /etc/cinder/host1.conf & echo $! >/opt/stack/status/stack/c-vol.pid; fg || echo "c-vol failed to start" | tee "/opt/stack/status/stack/c-vol.failure"

Now we go to the c-vol2 window and run the command that is already written there.

2.0 – Sanity checks

We now have 2 services running in the same cluster.

  • Check cluster status
user@localhost:$ cinder --os-volume-api-version 3.11 cluster-list --detail
+-----------------------+---------------+-------+---------+-----------+----------------+----------------------------+-----------------+----------------------------+------------+
| Name | Binary | State | Status | Num Hosts | Num Down Hosts | Last Heartbeat | Disabled Reason | Created At | Updated at |
+-----------------------+---------------+-------+---------+-----------+----------------+----------------------------+-----------------+----------------------------+------------+
| mycluster@lvmdriver-1 | cinder-volume | up | enabled | 2 | 0 | 2016-08-09T13:42:20.000000 | - | 2016-08-09T13:41:13.000000 | |
| mycluster@lvmdriver-2 | cinder-volume | up | enabled | 2 | 0 | 2016-08-09T13:42:20.000000 | - | 2016-08-09T13:41:13.000000 | |
+-----------------------+---------------+-------+---------+-----------+----------------+----------------------------+-----------------+----------------------------+------------+

  • Check services status:
user@localhost:$ cinder --os-volume-api-version 3.11 service-list
+------------------+-------------------+------+---------+-------+----------------------------+-----------------------+-----------------+
| Binary | Host | Zone | Status | State | Updated_at | Cluster | Disabled Reason |
+------------------+-------------------+------+---------+-------+----------------------------+-----------------------+-----------------+
| cinder-backup | host1 | nova | enabled | up | 2016-08-09T13:42:45.000000 | - | - |
| cinder-scheduler | host1 | nova | enabled | up | 2016-08-09T13:42:47.000000 | - | - |
| cinder-volume | host1@lvmdriver-1 | nova | enabled | up | 2016-08-09T13:42:50.000000 | mycluster@lvmdriver-1 | - |
| cinder-volume | host1@lvmdriver-2 | nova | enabled | up | 2016-08-09T13:42:50.000000 | mycluster@lvmdriver-2 | - |
| cinder-volume | host2@lvmdriver-1 | nova | enabled | up | 2016-08-09T13:42:46.000000 | mycluster@lvmdriver-1 | - |
| cinder-volume | host2@lvmdriver-2 | nova | enabled | up | 2016-08-09T13:42:46.000000 | mycluster@lvmdriver-2 | - |
+------------------+-------------------+------+---------+-------+----------------------------+-----------------------+-----------------+

  • Check RabbitMQ cluster queue:
user@localhost:$ sudo rabbitmqctl list_queues name | grep cinder-volume.mycluster
cinder-volume.mycluster@lvmdriver-2
cinder-volume.mycluster@lvmdriver-1

  • Check existing volumes were moved to the cluster when the service was started
user@localhost:$ mysql cinder -e 'select display_name, id, status, host, cluster_name from volumes where not deleted;'
+--------------+--------------------------------------+-----------+-------------------------------+-----------------------------------+
| display_name | id | status | host | cluster_name |
+--------------+--------------------------------------+-----------+-------------------------------+-----------------------------------+
| downloading | 58d2c5aa-9334-46a1-9246-0bc893196454 | error | host1@lvmdriver-1#lvmdriver-1 | mycluster@lvmdriver-1#lvmdriver-1 |
| snapshot_vol | 6a12f169-c6a7-4de5-85f2-c8259cbd6924 | available | host1@lvmdriver-1#lvmdriver-1 | mycluster@lvmdriver-1#lvmdriver-1 |
| creating | a7443e99-b87a-4e0a-bb44-6b63bdef477b | error | host1@lvmdriver-1#lvmdriver-1 | mycluster@lvmdriver-1#lvmdriver-1 |
| attached | a9102b47-37ff-4fd2-a76c-44e50c00e1fd | in-use | host1@lvmdriver-1#lvmdriver-1 | mycluster@lvmdriver-1#lvmdriver-1 |
+--------------+--------------------------------------+-----------+-------------------------------+-----------------------------------+

2.1 – Volume creation

The most basic thing we need to test is that we are able to create a volume in a cluster and that we are creating the workers table entry. To see that we are really sending it to the cluster we’ll just create 2 volumes instead of one. It is useful to have c-vol and c-vol2 windows open to see in the logs how each one is processing one of the creations.

user@localhost:$ cinder create --name host1 1; cinder create --name host2 1; sleep 3; mysql cinder -e 'select * from workers;'
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2016-08-09T14:04:15.000000 |
| description | None |
| encrypted | False |
| id | a592ff26-d70c-4a0e-92a3-ad3f5b8ac599 |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | host1 |
| os-vol-host-attr:host | None |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 83e1beb749d74956b664ef58c001af29 |
| replication_status | disabled |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| updated_at | None |
| user_id | c21ca8dae0644e52afe624a518e5e8f2 |
| volume_type | lvmdriver-1 |
+--------------------------------+--------------------------------------+
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2016-08-09T14:04:16.000000 |
| description | None |
| encrypted | False |
| id | 271aac9a-e9c2-4a89-87d2-c6fd13d81a5d |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | host2 |
| os-vol-host-attr:host | None |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 83e1beb749d74956b664ef58c001af29 |
| replication_status | disabled |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| updated_at | None |
| user_id | c21ca8dae0644e52afe624a518e5e8f2 |
| volume_type | lvmdriver-1 |
+--------------------------------+--------------------------------------+
+---------------------+---------------------+------------+---------+----+---------------+--------------------------------------+----------+------------+
| created_at | updated_at | deleted_at | deleted | id | resource_type | resource_id | status | service_id |
+---------------------+---------------------+------------+---------+----+---------------+--------------------------------------+----------+------------+
| 2016-08-09 14:04:15 | 2016-08-09 14:04:15 | NULL | 0 | 53 | Volume | a592ff26-d70c-4a0e-92a3-ad3f5b8ac599 | creating | 3 |
| 2016-08-09 14:04:16 | 2016-08-09 14:04:17 | NULL | 0 | 54 | Volume | 271aac9a-e9c2-4a89-87d2-c6fd13d81a5d | creating | 5 |
+---------------------+---------------------+------------+---------+----+---------------+--------------------------------------+----------+------------+

We can see that we have 2 new entries in the workers table for the volumes we are creating, and service 3 and service 5 are performing these operations.

It is important that once the volume has been created we check that the workers table is empty and both volumes are in “available” status.

user@localhost:$ cinder list
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+
| 271aac9a-e9c2-4a89-87d2-c6fd13d81a5d | available | host2 | 1 | lvmdriver-1 | false | |
| 58d2c5aa-9334-46a1-9246-0bc893196454 | error | downloading | 1 | lvmdriver-1 | false | |
| 6a12f169-c6a7-4de5-85f2-c8259cbd6924 | available | snapshot_vol | 1 | lvmdriver-1 | false | |
| a592ff26-d70c-4a0e-92a3-ad3f5b8ac599 | available | host1 | 1 | lvmdriver-1 | false | |
| a7443e99-b87a-4e0a-bb44-6b63bdef477b | error | creating | 1 | lvmdriver-1 | false | |
| a9102b47-37ff-4fd2-a76c-44e50c00e1fd | in-use | attached | 1 | lvmdriver-1 | false | 74fa7147-a0c5-4b17-b9b3-ce4ebfbea911 |
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+

user@localhost:$ mysql cinder -e 'select * from workers;'

user@localhost:$

If we check the DB contents for the new volumes we’ll have an unexpected surprise:

user@localhost:$ mysql cinder -e 'select display_name, id, status, host, cluster_name from volumes where display_name in ("host1", "host2");'
+--------------+--------------------------------------+-----------+-------------------------------+-----------------------------------+
| display_name | id | status | host | cluster_name |
+--------------+--------------------------------------+-----------+-------------------------------+-----------------------------------+
| host2 | 271aac9a-e9c2-4a89-87d2-c6fd13d81a5d | available | host1@lvmdriver-1#lvmdriver-1 | mycluster@lvmdriver-1#lvmdriver-1 |
| host1 | a592ff26-d70c-4a0e-92a3-ad3f5b8ac599 | available | host1@lvmdriver-1#lvmdriver-1 | mycluster@lvmdriver-1#lvmdriver-1 |
+--------------+--------------------------------------+-----------+-------------------------------+-----------------------------------+

As you can see both have the host field set to “host1@lvmdriver-1#lvmdriver-1” even though host2 was created in host2. You may think that this is a mistake, but it’s not, it’s what we can expect, since the scheduler doesn’t know which host from the cluster will be taking the job it just assigns one host that is up.

This will not be an issue for the cleanup and it’s only relevant for operations that have not cluster aware yet, as they will all be going to the same host instead of distributed among all the hosts as we’ll see in the next section.

2.2 – Snapshot creation

Snapshot creation is not yet cluster aware, so it will still use the host DB field to direct the job. This will soon change, but it’s a good opportunity to illustrate what I meant in the previous test. Creating 2 snapshots from the volumes we created in the previous section we’ll see how they are both handled by “host1”.

user@localhost:$ cinder snapshot-create host1 --name host1_snap; cinder snapshot-create host2 --name host2_snap; sleep 3; mysql cinder -e 'select * from workers;'
+-------------+--------------------------------------+
| Property | Value |
+-------------+--------------------------------------+
| created_at | 2016-08-09T14:21:28.497009 |
| description | None |
| id | 7d0923dd-c666-41df-ab12-2887e6a04bc3 |
| metadata | {} |
| name | host1_snap |
| size | 1 |
| status | creating |
| updated_at | None |
| volume_id | a592ff26-d70c-4a0e-92a3-ad3f5b8ac599 |
+-------------+--------------------------------------+
+-------------+--------------------------------------+
| Property | Value |
+-------------+--------------------------------------+
| created_at | 2016-08-09T14:21:30.478635 |
| description | None |
| id | 6d2124b8-2cdd-48ad-b525-27db18470587 |
| metadata | {} |
| name | host2_snap |
| size | 1 |
| status | creating |
| updated_at | None |
| volume_id | 271aac9a-e9c2-4a89-87d2-c6fd13d81a5d |
+-------------+--------------------------------------+
+---------------------+---------------------+------------+---------+----+---------------+--------------------------------------+----------+------------+
| created_at | updated_at | deleted_at | deleted | id | resource_type | resource_id | status | service_id |
+---------------------+---------------------+------------+---------+----+---------------+--------------------------------------+----------+------------+
| 2016-08-09 14:21:28 | 2016-08-09 14:21:28 | NULL | 0 | 55 | Snapshot | 7d0923dd-c666-41df-ab12-2887e6a04bc3 | creating | 3 |
| 2016-08-09 14:21:30 | 2016-08-09 14:21:30 | NULL | 0 | 56 | Snapshot | 6d2124b8-2cdd-48ad-b525-27db18470587 | creating | 3 |
+---------------------+---------------------+------------+---------+----+---------------+--------------------------------------+----------+------------+

We can see that we have 2 new entries in the workers table for the volumes we are creating, and both are being executed by service 3 (“host1”).

As usual, we check the results:

user@localhost:$ cinder snapshot-list
+--------------------------------------+--------------------------------------+-----------+---------------+------+
| ID | Volume ID | Status | Name | Size |
+--------------------------------------+--------------------------------------+-----------+---------------+------+
| 6d2124b8-2cdd-48ad-b525-27db18470587 | 271aac9a-e9c2-4a89-87d2-c6fd13d81a5d | available | host2_snap | 1 |
| 7d0923dd-c666-41df-ab12-2887e6a04bc3 | a592ff26-d70c-4a0e-92a3-ad3f5b8ac599 | available | host1_snap | 1 |
| acc8b408-2148-4de7-9774-ccb123650244 | 6a12f169-c6a7-4de5-85f2-c8259cbd6924 | error | creating_snap | 1 |
+--------------------------------------+--------------------------------------+-----------+---------------+------+


user@localhost:$ mysql cinder -e 'select * from workers;'

user@localhost:$

2.3 – Deletion

Volume, Snapshot, Consistency Group, and Consistency Group Snapshot deletions are cluster aware, so even though both snapshots were created in “host1”, deletion will be spread between the 2 hosts, as we can see in the logs.

user@localhost:$ cinder snapshot-delete host1_snap; cinder snapshot-delete host2_snap; cinder delete downloading; sleep 3; mysql cinder -e 'select * from workers;'
equest to delete volume downloading has been accepted.
+---------------------+---------------------+------------+---------+----+---------------+--------------------------------------+----------+------------+
| created_at | updated_at | deleted_at | deleted | id | resource_type | resource_id | status | service_id |
+---------------------+---------------------+------------+---------+----+---------------+--------------------------------------+----------+------------+
| 2016-08-09 15:40:53 | 2016-08-09 15:40:53 | NULL | 0 | 59 | Volume | 58d2c5aa-9334-46a1-9246-0bc893196454 | deleting | 3 |
+---------------------+---------------------+------------+---------+----+---------------+--------------------------------------+----------+------------+


user@localhost:$ cinder list
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+
| 271aac9a-e9c2-4a89-87d2-c6fd13d81a5d | available | host2 | 1 | lvmdriver-1 | false | |
| 6a12f169-c6a7-4de5-85f2-c8259cbd6924 | available | snapshot_vol | 1 | lvmdriver-1 | false | |
| a592ff26-d70c-4a0e-92a3-ad3f5b8ac599 | available | host1 | 1 | lvmdriver-1 | false | |
| a7443e99-b87a-4e0a-bb44-6b63bdef477b | error | creating | 1 | lvmdriver-1 | false | |
| a9102b47-37ff-4fd2-a76c-44e50c00e1fd | in-use | attached | 1 | lvmdriver-1 | false | 74fa7147-a0c5-4b17-b9b3-ce4ebfbea911 |
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+


user@localhost:$ cinder snapshot-list
+--------------------------------------+--------------------------------------+--------+---------------+------+
| ID | Volume ID | Status | Name | Size |
+--------------------------------------+--------------------------------------+--------+---------------+------+
| acc8b408-2148-4de7-9774-ccb123650244 | 6a12f169-c6a7-4de5-85f2-c8259cbd6924 | error | creating_snap | 1 |
+--------------------------------------+--------------------------------------+--------+---------------+------+


user@localhost:$ mysql cinder -e 'select * from workers;'

user@localhost:$

If you are wondering why we don’t have workers table entries for the deletion, that’s because snapshot deletion is not cleanable in existing code. So it’s something we’ll probably want to add, but it’s not specific to the High Availability Active-Active work.

2.4 – Attach volume

We can attach one of the newly created volumes to our existing VM and see in the logs how it’s handled by the 2 services.

user@localhost:$ nova volume-attach myvm a592ff26-d70c-4a0e-92a3-ad3f5b8ac599
+----------+--------------------------------------+
| Property | Value |
+----------+--------------------------------------+
| device | /dev/vdc |
| id | a592ff26-d70c-4a0e-92a3-ad3f5b8ac599 |
| serverId | 74fa7147-a0c5-4b17-b9b3-ce4ebfbea911 |
| volumeId | a592ff26-d70c-4a0e-92a3-ad3f5b8ac599 |
+----------+--------------------------------------+

2.5 – Detach volume

Now we’ll stop the “host1” service to confirm that it doesn’t matter, because even though the host field is set to “host1”, the “host2” can handle the detach operation as well since it’s in the same cluster:

Go to the c-vol window and stop the service using Ctrl+c, execute the following command while looking at the c-vol2 log:

user@localhost:$ cinder --os-volume-api-version 3.11 cluster-list --detail

After a little bit we can see how the service is no longer considered to be alive in the cluster and is reported as being down:

user@localhost:$ cinder --os-volume-api-version 3.11 cluster-list --detail
+-----------------------+---------------+-------+---------+-----------+----------------+----------------------------+-----------------+----------------------------+------------+
| Name | Binary | State | Status | Num Hosts | Num Down Hosts | Last Heartbeat | Disabled Reason | Created At | Updated at |
+-----------------------+---------------+-------+---------+-----------+----------------+----------------------------+-----------------+----------------------------+------------+
| mycluster@lvmdriver-1 | cinder-volume | up | enabled | 2 | 1 | 2016-08-09T15:52:55.000000 | - | 2016-08-09T13:41:13.000000 | |
| mycluster@lvmdriver-2 | cinder-volume | up | enabled | 2 | 1 | 2016-08-09T15:52:55.000000 | - | 2016-08-09T13:41:13.000000 | |
+-----------------------+---------------+-------+---------+-----------+----------------+----------------------------+-----------------+----------------------------+------------+

Future manual tests

These are just some manual tests to illustrate how things work and how we can test things, giving what I think is a good sample of different scenarios. I have tested more cases and manually debugged the code to ensure that specific paths are being followed in some corner cases, but I think these are a good start and can serve as a guide for people to test the other cluster aware operations.

I have tested more cluster aware operations than the ones I have included here, but I think these give a decent sample on the tests and expected results, so more tests can be easily done in this setup, and we can also easily add another cinder-volume service that is out of the cluster to this setup to confirm they can peacefully coexist.

I’ll be doing more tests on all operations that are already cluster aware, current and new, and even more once new features are added, like triggering service cleanups from the API.

Current cluster aware operations


– Manage existing
– Get pools
– Create volume
– Retype
– Migrate volume to host
– Create consistency group
– Update service capabilities
– Delete volume
– Delete snapshot
– Delete consistency group
– Delete consistency group snapshot
– Attach volume
– Detach volume
– Initialize connection
– Terminate connection
– Remove export

With the latest patches all operation in the Cinder volume service are cluster aware.

On Automatic Tests

I believe that to be reasonably certain that the Active-Active implementation is reliable we’ll need some sort of Fault Injection mechanism for Cinder intended not for real deployments, like Tempest is, but designed for testing deployments.

The reason for this is that you cannot automatically create a real life workload, make it fail, and then check the results without a really knowing what specific part of the code was actually being run at the moment the failure occurred. Some failures can be externally simulated, but the simultaion of others present their own challenges.

Again we’ll take the crawl-walk-run approach beginning with manual tests, then some kind of automated tests, then add multi-node CI jobs, and finally -hopefully- introducing the Fault Injection mechanism to add additional tests.


Picture: “Checklist” by [Claire Dela Cruz](https://pixabay.com/en/users
/ellisedelacruz-2310550/) is licensed under CC0 1.0


Cinder Active-Active HA – Newton mid-cycle

 

Last week took place the OpenStack Cinder mid-cycle sprint in Fort Collins, and on the first day we discussed the Active-Active HA effort that’s been going on for a while now and the plans for the future. This is a summary of that session.

Just like in previous mid-cycles the Cinder community did its best to accommodate remote attendees and make them feel included in the sessions with hangouts, live video streaming, IRC pings as reminders, and even moving the microphone around the room so they could hear better, and it’s an effort that I for one appreciate.

Above video includes 2 sessions, the first one is the discussion about the mascot, and the second is the one related to Active-Active, which begins 23 minutes and 34 seconds into the video, you can skip it manually or just jump right in there here.

We had a clear agenda on the Etherpad prior to the meeting, and even if we didn’t follow the order closely, we did cover everything in it and some more:

  • Status of patches
  • Status of testing
  • Help with reviewing
  • Set goals for Newton: DB changes

Getting the code in

During the Austin summit we reached an agreement related to the job distribution for clusters, and the patches were changed accordingly, so we are now ready to start merging them; but people are having problems reviewing patches because there is too much to process.

Even if the Cinder A-A blueprint has links to all specs and patches, it’s hard sometimes to understand what each patch of the series is trying to accomplish in the grand scheme of things.

This is understandable because with the new job distribution approach now patches from different specs are interleaved in the chain of patches. That’s why I try to keep specs up to date with any change or additional information I deem relevant while coding the feature, and one must pay attention to the commit message on the review to see to which one of the specs the patch is related to.

But this may not be enough, and maybe even after having read the specs there are things that aren’t clear in the patches, and that would mean that either the specs lack information or are not clear enough and needed more work, or the problem lay in the patches and they required more or better comments, or refactoring to make them easier to follow. In any case, we agreed that reviewers should not be coy about things that are not clear and I would make the appropriate changes to specs or code to fix it.

There were a couple of controversial patches at the beginning of the chain of patches that would delay the merging of the rest of the patches until an agreement could be reached. So Dulek suggested to remove them from the chain and rework the other patches as necessary, if this wasn’t too much work, to move things along. Since this mostly meant adding some more code to a couple of patches seemed reasonable and we agreed to have them separated from the chain.

We want to review and merge as many patches as possible in Newton, but we want to keep the risk of disruption low, so a list of low risk patches that can be reviewed and merged in N is necessary to focus our efforts.

Testing

Quoting Bruce Eckel, “If it’s not tested, it’s broken”, so we are going to need some multinode automated tests at the gate, and the Tempest patch for this is under review, but that won’t be enough, because Tempest is meant for API validation, not negative tests, and much less the kind of tests that are required to verify the correctness of an Active-Active cluster on failure.

To have proper tests we’ll need a fault injection mechanism, but that’s a big effort that probably will not be in time for the tech preview release of Active-Active HA, so for now we’ll go with manual testing forcing corner case error conditions as well as normal error conditions and non error conditions.

We agree that the tempest tests together with this manual testing will be enough for the tech-preview, but the manual process we follow will need to be properly documented so people can see which test cases are being checked -managing their expectations accordingly and allowing them to suggest new tests- and serve as a starting point for the automated tests.

In order to coordinate testing, resolve doubts, and create focused reviews, Scottda suggests creating a Working Group that will meet weekly on IRC, and we all vote in favor.

Drivers

We discussed multiple topics related to drivers and Active-Active configurations, the first one was that they need to be ready for the feature if they don’t want to be at a disadvantage for not being able to support this feature. And they can already start testing their own drivers today with existing patches, although there is not documentation to help them in the process.

Szymon pointed out that there is an issue with drivers inheriting from the same base when the base class uses external locks, because that means that changing the base class to use distributed locks changes all drivers at the same time, and that’s not ideal as some drivers may not want or need them. The case presented was the RemoteFS driver and inheriting drivers

Proposed approach is creating 2 different classes, one that leaves all drivers as they are, using local locks, and another class that uses Tooz abstraction layer for drivers that want to use distributed locks. To this issue I suggested an alternative approach changing the base class to use Tooz abstraction layer which uses the default local file locks, equivalent to the ones the drivers are using, and moving the Tooz configuration options from the DEFAULT section to the driver specific section, that way we can have a multibackend deployment where one driver uses local locks and the other uses distributed locks.

Review sprint

On Friday, after all the sessions, the Cinder team decided to give a big push to Active-Active patches, so we all synchronized -those in Fort Collins and those attending remotely- for a reviewing sprint where I would be updating patches and replying to gerrit and IRC comments as they were going through the patches. And it was a great success, as we were able to go through the 12 patches -many of considerable size- that we wanted to get merged in Newton.

Some of the patches haven’t merged yet, but they are in the process of merging and being verified by Zuul at this moment.

Actions

During the meeting we agreed upon the following actions:

  • Remove the 2 controversial patches from the chain and update other patches ➞ Done
  • Create a list of low risk patches that can merge in N ➞ First patch in the series that shouldn’t merge is https://review.openstack.org/303021
  • Create a document with manual tests
  • Look into fault injection
  • Pick a time for the Work Group’s weekly meeting ➞ Will be held on #openstack-cinder on Tuesdays at 16:00 UTC, as announced on the dev mailing list

Other

Session notes were taken in Etherpad as usual and they are split according to the day they were help, but they are all linked in the mid-cycle Cinder Etherpad and are available for reference.

All videos from the mid-cycle are also available at the openstack-cinder YouTube channel.


VBox & VMware in SecureBoot Linux (2016 Update) 2

 

Kernel driver not installed (rc=-1908) The VirtualBox Linux kernel driver (vboxdrv) is either not loaded or there is a permission problem with /dev/vboxdrv. Please reinstall the kernel module by executing 'etc/init.d/vobxdrv setup' as root. If it is available in your distribution, you should install the DKMS package first. This package keeps track of Linux kernel changes and recompiles the vboxdrv kernel module if necessary.

If you have a Linux system running in Secure Boot and you install VirtualBox or VMware player you will see, with some frustration, that you won’t be able to run any VMs.

This post also applies if you are running your system with module signature verification enabled (CONFIG_MODULE_SIG) even if it’s not running in Secure Boot.

This is an old issue, and I’ve already written about it in another post almost 2 years ago, but at this point some degree of imagination is needed to succeed following that guide, so I have finally decided to update it. The reason why it took me so long to update the post is that I haven’t had VirtualBox or VMware player installed for quite a long time.

To install VirtualBox we’ll use the repository:

user@localhost:$ sudo dnf install gcc kernel-devel VirtualBox

Earlier picture shows what you’ll see from the GUI, but if you run it from the console you’ll see:

user@localhost:$ virtualbox
WARNING: The vboxdrv kernel module is not loaded. Either there is no module
available for the current kernel (4.5.7-202.fc23.x86_64) or it failed to
load. Please recompile the kernel module and install it by

sudo /sbin/rcvboxdrv setup

You will not be able to start VMs until this problem is fixed.

Now we have to compile the Kernel Modules for VirtualBox

user@localhost:$ sudo akmods
Checking kmods exist for 4.5.7-202.fc23.x86_64 [ OK ]

If we try to load any of these modules we’ll see the main problem:

user@localhost:$ sudo modprobe -v vboxdrv
insmod /lib/modules/4.5.7-202.fc23.x86_64/misc/vboxdrv.ko
modprobe: ERROR: could not insert 'vboxdrv': Required key not available

And then you’ll realize what the problem is, modprobe is complaining about required key not being available. Which actually means that the module is not signed and therefore cannot be loaded.

Now that you know what the problem is, the solution is quite simple; you just need to sign the module and make sure that the system recognizes the key as valid.

If you already have a X.509 key you can skip the key creation part and go directly to signing the module and enrolling the key. But if you don’t, you’ll need to generate a key to sign any third party module you want to install or any custom module you use.

Creating an X.509 Key Pair to sign the driver is easy:

user@localhost:$ openssl req -new -x509 -newkey rsa:2048 -keyout MOK.priv -outform DER -out MOK.der -nodes -days 36500 -subj "/CN=Akrog/"

In the above command, replace MOK with the name of the file you want for the key and Akrog with the Common Name you want to use. It’s usually the organization that signs it, but you can write whatever you like, although I recommend a significant name as it will be inserted into the system’s key ring.

VirtualBox uses multiple kernel modules and we need to sign them all. In my previous post I went into a little bit more detail, but I think it’s enough to say that we need to get the location of the modules using modinfo and then signing them using sign-file script like this:

user@localhost:$ for f in $(dirname $(modinfo -n vboxdrv))/*.ko; do echo "Signing $f"; sudo /usr/src/kernels/$(uname -r)/scripts/sign-file sha256 ./MOK.priv ./MOK.der $f; done

If you don’t fee confortable with that script you can do it manually, but be careful, as they may add or remove modules from dirname $(modinfo -n vboxdrv) directory:

user@localhost:$ sudo /usr/src/kernels/$(uname -r)/scripts/sign-file sha256 ./MOK.priv ./MOK.der $(modinfo vboxdrv)

user@localhost:$ sudo /usr/src/kernels/$(uname -r)/scripts/sign-file sha256 ./MOK.priv ./MOK.der $(modinfo vboxguest)

user@localhost:$ sudo /usr/src/kernels/$(uname -r)/scripts/sign-file sha256 ./MOK.priv ./MOK.der $(modinfo vboxnetadp)

user@localhost:$ sudo /usr/src/kernels/$(uname -r)/scripts/sign-file sha256 ./MOK.priv ./MOK.der $(modinfo vboxnetflt)

user@localhost:$ sudo /usr/src/kernels/$(uname -r)/scripts/sign-file sha256 ./MOK.priv ./MOK.der $(modinfo vboxpci)

user@localhost:$ sudo /usr/src/kernels/$(uname -r)/scripts/sign-file sha256 ./MOK.priv ./MOK.der $(modinfo vboxsf)

user@localhost:$ sudo /usr/src/kernels/$(uname -r)/scripts/sign-file sha256 ./MOK.priv ./MOK.der $(modinfo vboxvideo)

Modinfo no longer displays module signed information, so we’ll have to trust that this has worked.

To enroll the public key in the MOK (Module owned Key) your UEFI partition must have MokManager.efi installed. You can check this running

user@localhost:$ sudo find /boot -name MokManager.efi

Now we have to manually add the public key to shim’s MOK list and we’ll be asked for a password that will be used during the UEFI boot to enroll the new key, so make sure you remember it at least for a minute ;-):

user@localhost:$ mokutil --import MOK.der
input password:
input password again:
Failed to write MokAuth
Failed to unset MokNew

Despite the errors I got it seems to work just fine.

What we’ve done with this is request the MOK manager to insert a new key, but we haven’t inserted it yet, so we need to reboot for that and follow the enrolling process that is quite straight forward: Press a key to start the process if you are asked to, then select “Enroll MOK”, then “Continue”, and then “Yes”; and the key has been inserted. This is a persistent operation, so you’ll only need to do this once.

When you have finished booting you can easily check that the key is in the system key ring using the CN we used when creating the X.509 key:

user@localhost:$ keyctl list %:.system_keyring
112560593: ---lswrv 0 0 asymmetric: Fedora kernel signing key: e948c9015e04bd4cd5879fe2f9230a1d70859c7d
489921950: ---lswrv 0 0 asymmetric: Fedora Secure Boot CA: fde32599c2d61db1bf5807335d7b20e4cd963b42
98641885: ---lswrv 0 0 asymmetric: Akrog: d5d3e2008907a7cebc8914780bd29b03fecc214b
525156767: ---lswrv 0 0 asymmetric: Microsoft Corporation UEFI CA 2011: 13adbf4309bd82709c8cd54f316ed522988a1bd4
1001714488: ---lswrv 0 0 asymmetric: Microsoft Windows Production PCA 2011: a92902398e16c49778cd90f99e4f9ae17c55af53

And that it was EFI who loaded it:

user@localhost:$ dmesg | grep 'EFI: Loaded cert'
[ 0.456158] EFI: Loaded cert 'Microsoft Windows Production PCA 2011: a92902398e16c49778cd90f99e4f9ae17c55af53' linked to '.system_keyring'
[ 0.456194] EFI: Loaded cert 'Microsoft Corporation UEFI CA 2011: 13adbf4309bd82709c8cd54f316ed522988a1bd4' linked to '.system_keyring'
[ 0.457111] EFI: Loaded cert 'Akrog: d5d3e2008907a7cebc8914780bd29b03fecc214b' linked to '.system_keyring'
[ 0.457768] EFI: Loaded cert 'Fedora Secure Boot CA: fde32599c2d61db1bf5807335d7b20e4cd963b42' linked to '.system_keyring'

Now you should be able to run VirtualBox VMs without problem.

For a more detailed description of the process of signing kernel modules you can check Red Hat’s documentation here.

If you read VirtualBox ticket regarding this issue you’ll see they wash their hands on the matter saying: “This is not really a VirtualBox bug. Oracle cannot sign kernel modules using the Fedora key”.
I for one believe that this is a bug in the installation, as they could easilly see if the installation is running on a BIOS or EFI/UEFI system (checking for /sys/firmware/efi directory) and whether Secure Boot is enabled or not (checking the efivar SecureBoot) and if it’s enable request a key to sign the driver or ask you if you want to create one and have it inserted it in the MOK automatically.

VMware signing should be the very similar, unfortunately I cannot check it because the installer won’t work when you have KVM running, as is my case, and even when I disabled KVM modules temporarily and was able to install it, the module compilation using vmware-modconfig wouldn’t work as it should. So I’ll assume we have the VMware player installed and the kernel modules compiled.

For those fighting the VMware installation, it’s worth mentioning that in the older post Pipio was having trouble with VMware and this post in the message board helped in the resolution of the problem.

The last time I run VMware without signed kernel modules the error that was displayed was this:
Error when trying to run VM:    Could not open /dev/vmmon: No such file or directory.    Please make sure that the kernel module `vmmon' is loaded.

So for VMware the signing would be like for VirtualBox, but replacing vboxdrv with vmmon like this:

user@localhost:$ for f in $(dirname $(modinfo -n vmmon))/*.ko; do echo "Signing $f"; sudo /usr/src/kernels/$(uname -r)/scripts/sign-file sha256 ./MOK.priv ./MOK.der $f; done

PS: I used Fedora 23, but the signing process should work on any Linux using EFI shim for UEFI boot.