During Mitaka we introduced some big changes in Cinder that have a great impact for developers working on new and existing functionality. These new features include, but are not limited to, API microversions, support for Rolling Upgrades, and conditional DB update functionality to remove API races. So we decided to have Recap Sessions during the OpenStack Summit in Austin.
The idea behind this was to facilitate adoption and reduce bugs by focusing on explaining how the functionality should be used, what special considerations should be taken into account, resolving questions and concerns, as well as looking into the future related work, instead of focusing on the implementation details.
We had 3 recap sessions:
- API microversions, lead by Scott DAngelo
- Rolling Upgrades, lead by Michal Dulko
- Conditional Updates, lead by me
When the session was originally proposed there was not much information about these functionalities, but by the time the recap session took place we had already added some great devref documents as well as some patches that illustrated each functionality and could serve as reference for other patches.
The sessions were live streamed and recorded for those who couldn’t attend and for future reference, although due to the position of the camera displayed screen is not visible:
API microversions
This is one of those functionalities that has no immediate user visible impact when it gets implemented but that enables a broad range of possibilities and provides great flexibility for almost everything else we do.
I’m so glad this has been added to Cinder, now we’ll be able to make breaking changes to the API without actually breaking the API!!
The session lead by Scott DAngelo starts at minute 35 and 5 seconds of the video and can be accessed directly here.
The developer’s reference documentation is a great help for anyone working with the Cinder REST API, be it on the server or client side.
Actions resulting from the session:
- Tempest test changes – Example patch for tempest at https://review.openstack.org/300639/
- Functional tests
- Communicate to deployers and other client users to move to the V3 endpoint.
Additional videos on this topic are available from previous work sessions:
Rolling Upgrades
This feature is not only huge -it affects almost every part of Cinder- but it is also complex and therefore requires a good knowledge for development and reviewing patches as it has implications on the API, RCP calls, data serialization, DB model, service start, signal handling by services, DB migrations…
This session was lead by Michal Dulko and starts at minute 70 and 7 seconds and can be accessed directly here.
The developer’s reference documentation is a great help for anyone approaching this for the first time, as it gives examples of how to do changes requiring multiple releases to maintain interoperability between one release and the next in order to support rolling upgrades.
Even though the slides cannot be seen in the video Michal shared them online.
Actions resulting from the session:
- Create CI job to tests rolling upgrades
Conditional Updates
Cinder suffered from races when checking on the DB the conditions for actions requested to the API nodes, so we implemented a conditional update mechanism that would facilitate doing compare and swap from the API while preserving abstraction from the DB implementation as much as possible.
Due to a scheduling conflict with another talk, I couldn’t attend the second half of the recap session, so my colleagues did me the favor of letting me go first, so this recap session is at the beginning of the video.
Even though the slides cannot be seen in the video, they are available online.
This was quite a productive session for me, because people raised interesting concerns like the possibility of not catching multi table updates at the gate if the code path was not executed for some reason, the lack of stress tests for these new conditional updates to ensure that the API have actually been removed, and why MySQL and PostgreSQL don’t do UPDATE in the same way and if we are safe with current code.
To address this last concern I had to resort to Mike Bayer’s expertise on the matter, and he replied with a very detailed and clear response that was not only illuminating -improving my knowledge in the matter- but also confirmed that we were safe with our code and we could use all the features of the conditional update mechanism. If you are interested in reading his full reply it is available on the session’s Etherpad, but basically SQL specs are very clear on how an UPDATE should operate on the unmodified row (like PostgreSQL does) and MySQL has documented their non standard behavior (uses changing row) and therefore is the expected behavior of any MySQL engine, present or future.
Actions resulting from the session:
- Prevent multi table updates even on DB engines that support it: Done, now up for review https://review.openstack.org/310656
- Create devref document: Done, now available at http://docs.openstack.org/developer/cinder/devref/api_conditional_updates.html
- Confirm that conditional update is retrying on DB deadlocks: It does, we are using a decorator for the retries.
- Investigate UPDATE differences: We are safe, as mentioned above and in the Etherpad.
- Functional stress tests: Ivan Kolodyazhny offered to do Rally tests.
Other
Notes for all sessions were taken in the Recap Session Etherpad and are available for reference.
Other design session videos from the Summit are also available at the openstack-cinder YouTube channel.
