OpenStack Day 6: Deploying the Overcloud

Exciting times... after much prep work we'll be deploying the overcloud. While we are arguably using OpenStack already on the director (because we are), it's not the same as being able to point to a multi-node deployment and spin up some instances. The Overcloud is what we'll actually be using in production. The director is largely a management tool geared toward the result which is the Overcloud.

When we deploy the overcloud we know that Ironic is going to power up our nodes which will, in turn, PXE boot the initial image from the director. But then what happens? That's where almost all of the magic of TripleO comes into play.

When we deploy the overcloud we will be using OpenStack tooling (Mistral, Heat and Ironic) along with Puppet for configuration management. Each of these is also a potential entry point into customizing our deployments. Red Hat provides a good overview for understanding the Heat Orchestration Templates (HOT): Understanding Heat Templates (It is a bit Red Hat specific and some command won't work if you aren't using Red Hat's variant of openstack... in particular the 'openstack management' command).

Explore a little with 'openstack workbook list' and 'openstack workflow list'.

The Heat template collection for the overcloud is stored in /usr/share/openstack-tripleo-heat-templates/. There's a lot in here and bits of it are pretty interesting to see what the overcloud deployment is doing. We can customize templates and environment files to customize our deployment.

Puppet is the next big contributor to the Overcloud deployment. Puppet (www.puppet.com) is configuration management software. It will allow us to tell which packages to install, how those packages can be configured and how the OS should be configured. It's pretty obvious how this contributes to getting the nodes to have the right software and settings. Modifying the puppet manifests is another entry point should we need to customize our deployment.

Since this is our first deployment we aren't going to tweak anything. Just let it do it's thing and see what shakes out. The background is interesting because we may want to observe the deployment and understand what's happening.

Ok. Maybe not just push the button. TripleO provides us with a bit of last minute validation. Sometimes this just finds weirdness but sometimes it finds something useful. Plus it gives a bit of practice. We're going to launch the pre-deployment validation workflow. You can find templates at /usr/share/openstack-tripleo-validations/ .

$ openstack workflow execution create tripleo.validations.v1.run_groups '{"group_names": ["pre-deployment"]}'
+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| ID                | 187856d5-b5f5-4504-9e52-f1a0943a1885 |
| Workflow ID       | 2eadb3f2-98c1-4eeb-9c06-3b6b1c7b2b9a |
| Workflow name     | tripleo.validations.v1.run_groups    |
| Description       |                                      |
| Task Execution ID | <none>                               |
| State             | RUNNING                              |
| State info        | None                                 |
| Created at        | 2017-01-09 22:23:40.809719           |
| Updated at        | 2017-01-09 22:23:40.815409           |
+-------------------+--------------------------------------+

That's the last result that will be pushed to you. It's running. Getting actual results is a tad more complicated. First we have to wait until no more processes are running. We can see the process status by doing 'openstack task execution list'. For our purposes we really don't care about success, we only care about failure. So...

Let's see if anything is running:

$ openstack task execution list | grep RUNNING

| d166b19e-6647-4e7a-979e-e279b9ff3348 | run_validation_group | tripleo.validations.v1.run_groups | 7139ead2-f6ff-443b-a1b9-c0c1aab9e5f3 | RUNNING | None |

| 1c1fc57b-aaa2-4d26-8da2-a6185a8d5138 | notify_running | tripleo.validations.v1.run_validation | 944af79b-9d1d-46cf-aada-cf1d9ec627d1 | RUNNING | None |

| a4c7df7b-f622-4479-b299-5c482e3c3c4d | notify_running | tripleo.validations.v1.run_validation | 35a00874-91eb-493f-aba6-cc66c780f47d | RUNNING | None |

| e336cfbe-3573-4a13-b9c1-035c684c1f90 | notify_running | tripleo.validations.v1.run_validation | d58a0a39-c039-4957-a5f5-dc142b0224a3 | RUNNING | None |

| 1f80671e-45ce-4572-b7b9-5bdf628f1527 | notify_running | tripleo.validations.v1.run_validation | 5a455516-70c9-436e-92f2-bc171b9b8171 | RUNNING | None |

| 232891c0-73bd-4190-96fd-9b2f6d63515e | notify_running | tripleo.validations.v1.run_validation | 087d2ac7-fa71-42fc-8349-7e75d0e58ea6 | RUNNING | None |


There is so we just repeat that command every few seconds until there isn't. At that point we are interested in the errors... and more we are only interested in errors that occurred during run_validation.

$ openstack task execution list | grep ERROR | grep run_validation
| 377cdc82-fca0-48e4-9f2b-4893c9d01ff4 | run_validation                    | tripleo.validations.v1.run_validation            | 6c2d8b09-49f0-4238-bca9-d62e6cc331e7 | ERROR   | {u'stderr': u'', u'stdout... |

| ec233d8d-0a6d-44e7-a12e-f54a21f19dc2 | run_validation                    | tripleo.validations.v1.run_validation            | 6d354722-a9c3-4481-8772-d2c0561211e7 | ERROR   | {u'stderr': u'', u'stdout... |

So... 2 errors this pass. Yep. This pass. Some validations stop when they encounter an error so we'll do this multiple times. Also pay attention to the ID of the last error so you know not to check that one again on subsequent passes.

For each error we will run

$ mistral task-get-result {ID} where {ID} is the grouping in the first column.

$ mistral task-get-result 377cdc82-fca0-48e4-9f2b-4893c9d01ff4
{
    "stderr": "",
    "stdout": "
Task 'Check the services for debug flag' failed:\nHost: localhost\nMessage: The key 'debug' under the section 'DEFAULT' in file /etc/nova/nova.conf has the value: 'True'\n\n
Task 'Check the services for debug flag' failed:\nHost: localhost\nMessage: The key 'debug' under the section 'DEFAULT' in file /etc/neutron/neutron.conf has the value: 'True'\n\n
Task 'Check the services for debug flag' failed:\nHost: localhost\nMessage: The key 'debug' under the section 'DEFAULT' in file /etc/heat/heat.conf has the value: 'True'\n\n
Task 'Check the services for debug flag' failed:\nHost: localhost\nMessage: The key 'debug' under the section 'DEFAULT' in file /etc/ironic/ironic.conf has the value: 'True'\n\nFailure! The validation failed for the following hosts:\n* localhost\n"
}

Looks like we need to edit /etc/nova/nova.conf, /etc/neutron/neutron.conf, /etc/heat/heat.conf and /etc/ironic/ironic.conf to fix the debug key. You might wonder why that needs to be fixed since we never manually changed it. Yeah... I expect this is a case where the validation suite hasn't kept pace with progress. Or maybe they are all test driven development and the validation points out work that needs to happen. Hard to say. For now we'll just make the change and quiet the validation. Or if your comfortable with this setting just leave it alone and move to the next error. Will having logging set to debug cause problems?

Repeat this process until you are error free or at least comfortable with the errors.

Once your content with the validation status, launch it. The deploy has a lot of possible command line options. We're being simple. You can see all the options with 'openstack help overcloud deploy'.

--templates - Create the overcloud using the heat template collection located in /usr/share/openstack-tripleo-heat-templates

by default 1 compute and 1 control node will be deployed which is exactly what we want so no other options need be specified.

In addition to watching the output, run the occasional 'openstack baremetal node list' in another window and watch the provisioning state change. Console on the nodes is also interesting. This is going to take a while... you can watch the heat part, sort of, by opening another window to the director and issuing 'openstack stack list' and then 'openstack stack show {ID}' and 'openstack stack output show {ID} --all'.

$ openstack overcloud deploy --templates
Removing the current plan files
Uploading new plan files
Started Mistral Workflow. Execution ID: b1585580-0b58-469a-b4ed-5246322268b6
Plan updated
Deploying templates in the directory /tmp/tripleoclient-zqUDul/tripleo-heat-templates
Started Mistral Workflow. Execution ID: 0f0da3b6-28ce-463b-a55f-a86475f95690
2017-01-09 22:46:12Z [overcloud]: CREATE_IN_PROGRESS  Stack CREATE started
...
 Stack overcloud CREATE_COMPLETE
Started Mistral Workflow. Execution ID: a06adb46-e9bf-4506-a8ba-5492a3de5560
Overcloud Endpoint: http://192.168.50.103:5000/v2.0
Overcloud Deployed

Woot!I have an overcloud. A small one, but an overcloud. Next up: How do I access it? Will I be able to spin up an instance that I can also access? This requires a lot of pieces to have worked properly, especially networking. The very fact that the Overcloud Controller is on my private network means I either need to do something clever or attach my workstation to that network so that I can get to the Horizon dashboard. For that matter, why doesn't the Undercloud have a dashboard? It does??? Stay tuned...