So the Linux Foundation has released a number of it's training courses for free on EdX. They are introductory and overview type classes to just wet your appetite.
LFS151 - Introduction to Cloud Infrastructure Technologies. This one doesn't go deep (Introduction after all) but it goes pretty wide and I think does a good job of touching on all of the current/popular Cloud Infrastructure options. Unless your well versed in all of them it's worth your time to watch it at faster than normal speed.
LFS152 - Introduction to OpenStack. I'm about 70% through this one. Again, it's an Introduction but it does give you the information and tools necessary to start playing with OpenStack in a directed fashion.
The other two are:
LFS101 - Introduction to Linux
LFS161 - Introduction to DevOps: Transforming and Improving Operations.
Haven't watched them but I'd guess they will be of similar quality to the others. If your new to DevOps, Linux and/or Cloud I'd recommend them with no hesitation. Will watch them over the weekend and post a retraction if I'm wrong :)
Thursday, January 12, 2017
Tuesday, January 10, 2017
OpenStack Day 6.5 - A Brief Detour with the Triple-O UI
We now have a two node OverStack consisting of one compute node and one controller node.
What's Tuskar? "Tuskar gives administrators the ability to control how and where OpenStack services are deployed across the datacenter.". Mmmmkay. The gist is that with Tuskar we can create resource classes and then Tuskar will use performance monitoring, health statistics, and other metrics to aid us in capacity planning.
Tuskar-UI is built on top of Horizon and provides a UI into the Tuskar data. So at this point we're pretty sure there's a GUI running on the undercloud. The TripleO docs say to create an SSH tunnel pointing port 8080 to port 80 on localhost. I have doubt. It also talks of editing /etc/openstack-dashboard/local_settings also which isn't something that even exists. I expect these docs might be a little... dated.
If we look at /etc/httpd/conf.d we see '25-tripleo-ui.conf'. This says to listen on port 3000 of our internal IP. So... First pass, lets SSH tunnel to port 3000 and see what shakes out.
$ netstat -an | grep 3000
tcp 0 0 192.168.50.1:3000 0.0.0.0:* LISTEN
It's only listening on the management interface so we can use port 3000 on our external interface.
$ ssh -g -N -L {EXT_IP}:3000:192.168.50.1:3000 root@192.168.50.1
root@192.168.50.1's password:
Then we point our browser at http://{EXT_IP}:3000/
Taking the simple way out and just attaching a workstation to the 192.168.50.0/24 network. The top of our DHCP range is .250 so I'm taking .254 for my workstation testing. Pointing my browser at http://192.168.50.1:3000/ got the login. The login worked perfectly... and there's a lot of interesting information there including login information to my overcloud (which is apparently parked at 192.168.50.103). The right hand side also appears to be stocked with validation information. So the short answer is that it's a networking issue.
So this bit of networking and lack of information doesn't surprise me at all. When I hand rolled an instance I had to create a separate gateway machine to allow external access to Horizon. Pretty annoying and it makes me wonder if I'm doing something wrong. Accessing the GUIs just shouldn't be a difficult thing.
In my hand rolled deployment there was not a director. Since we've said the director is the gateway for the management network that means we would need to do our inbound gateway there as well since there will be NAT and it'd get weird if we didn't send it through a single mapping. I'm forced to ponder how the floating IPs are going to work at this point as well. Those should be handled by our controller which will do the bridging and security via Neutron.
One thing at a time. The obvious way would be to do a bit of port forwarding using iptables or such. Sadly this will fail. In /var/www/openstack-tripleo-ui/dist/tripleo_ui_config.js it defines keystone as http://192.168.50.1:5000/v2.0 ... this makes its way to your browser and your browser will try to access that directly. It will fail.
Rather than trying to change all of the configuration to play nice with me the answer is probably a web proxy on that network (squid or nginx). Most of OpenStack is RESTful interfaces so this should just work. (yep, famous last words... let's see).
I'm more familiar with squid so...
# yum install squid squid-sysvinit
# vi /etc/squid/squid.conf
Change my browsers proxy settings to use the external IP of the director on port 3128 and tell it to use that proxy for all ports.
So what? Well the UI gives us another data point to help us confirm or reject some of our assumptions. It also provides a way to see some possible weirdness before it's an issue. Also you just may find it easier to manage Nodes and roles from this interface.
Triple-O UI Detour
Before we go to the OverCloud, lets ponder the dashboard and some GUI on the undercloud. The TripleO docs list two 'Basic Deployment' options: CLI (which is the way I went) and GUI. Apparently "Part of the Undercloud installation is also Tuskar-UI which you can use to drive deployment". Tuskar-UI is one of the two components that make up Tuskar. Tuskar is built with Triple-O in mind so this shouldn't even be surprising.What's Tuskar? "Tuskar gives administrators the ability to control how and where OpenStack services are deployed across the datacenter.". Mmmmkay. The gist is that with Tuskar we can create resource classes and then Tuskar will use performance monitoring, health statistics, and other metrics to aid us in capacity planning.
Tuskar-UI is built on top of Horizon and provides a UI into the Tuskar data. So at this point we're pretty sure there's a GUI running on the undercloud. The TripleO docs say to create an SSH tunnel pointing port 8080 to port 80 on localhost. I have doubt. It also talks of editing /etc/openstack-dashboard/local_settings also which isn't something that even exists. I expect these docs might be a little... dated.
If we look at /etc/httpd/conf.d we see '25-tripleo-ui.conf'. This says to listen on port 3000 of our internal IP. So... First pass, lets SSH tunnel to port 3000 and see what shakes out.
$ netstat -an | grep 3000
tcp 0 0 192.168.50.1:3000 0.0.0.0:* LISTEN
It's only listening on the management interface so we can use port 3000 on our external interface.
$ ssh -g -N -L {EXT_IP}:3000:192.168.50.1:3000 root@192.168.50.1
root@192.168.50.1's password:
Then we point our browser at http://{EXT_IP}:3000/
Progress. The username and password can be found in the stackrc file:
OS_USERNAME=admin
OS_PASSWORD=${sudo hiera admin_password}
Hiera is part of Puppet which we touched on briefly last post.
$ sudo hiera admin_password
d32545245ea237fd144218a0b6d91278af259339
d32545245ea237fd144218a0b6d91278af259339
If I drop this info in and hit the login button I'm presented with 'Connection Error: Connection to Keystone is not available'. *sigh* The question becomes do I need to forward the keystone port (5000) also or is it some other issue?
So this bit of networking and lack of information doesn't surprise me at all. When I hand rolled an instance I had to create a separate gateway machine to allow external access to Horizon. Pretty annoying and it makes me wonder if I'm doing something wrong. Accessing the GUIs just shouldn't be a difficult thing.
In my hand rolled deployment there was not a director. Since we've said the director is the gateway for the management network that means we would need to do our inbound gateway there as well since there will be NAT and it'd get weird if we didn't send it through a single mapping. I'm forced to ponder how the floating IPs are going to work at this point as well. Those should be handled by our controller which will do the bridging and security via Neutron.
One thing at a time. The obvious way would be to do a bit of port forwarding using iptables or such. Sadly this will fail. In /var/www/openstack-tripleo-ui/dist/tripleo_ui_config.js it defines keystone as http://192.168.50.1:5000/v2.0 ... this makes its way to your browser and your browser will try to access that directly. It will fail.
Rather than trying to change all of the configuration to play nice with me the answer is probably a web proxy on that network (squid or nginx). Most of OpenStack is RESTful interfaces so this should just work. (yep, famous last words... let's see).
I'm more familiar with squid so...
# yum install squid squid-sysvinit
# vi /etc/squid/squid.conf
- Add my IP t the localnet src ACL
- Comment out the 'http_access deny ~Safe_ports' line (we could have added keystone to the safe ports maybe)
- Comment out 'http_access deny CONNECT ~SSL_ports'
Change my browsers proxy settings to use the external IP of the director on port 3128 and tell it to use that proxy for all ports.
And Voila... my windows workstation can now access the Undercloud UI.
So what? Well the UI gives us another data point to help us confirm or reject some of our assumptions. It also provides a way to see some possible weirdness before it's an issue. Also you just may find it easier to manage Nodes and roles from this interface.
OpenStack Day 6: Deploying the Overcloud
Exciting times... after much prep work we'll be deploying the overcloud. While we are arguably using OpenStack already on the director (because we are), it's not the same as being able to point to a multi-node deployment and spin up some instances. The Overcloud is what we'll actually be using in production. The director is largely a management tool geared toward the result which is the Overcloud.
When we deploy the overcloud we know that Ironic is going to power up our nodes which will, in turn, PXE boot the initial image from the director. But then what happens? That's where almost all of the magic of TripleO comes into play.
When we deploy the overcloud we will be using OpenStack tooling (Mistral, Heat and Ironic) along with Puppet for configuration management. Each of these is also a potential entry point into customizing our deployments. Red Hat provides a good overview for understanding the Heat Orchestration Templates (HOT): Understanding Heat Templates (It is a bit Red Hat specific and some command won't work if you aren't using Red Hat's variant of openstack... in particular the 'openstack management' command).
Explore a little with 'openstack workbook list' and 'openstack workflow list'.
The Heat template collection for the overcloud is stored in /usr/share/openstack-tripleo-heat-templates/. There's a lot in here and bits of it are pretty interesting to see what the overcloud deployment is doing. We can customize templates and environment files to customize our deployment.
Puppet is the next big contributor to the Overcloud deployment. Puppet (www.puppet.com) is configuration management software. It will allow us to tell which packages to install, how those packages can be configured and how the OS should be configured. It's pretty obvious how this contributes to getting the nodes to have the right software and settings. Modifying the puppet manifests is another entry point should we need to customize our deployment.
Since this is our first deployment we aren't going to tweak anything. Just let it do it's thing and see what shakes out. The background is interesting because we may want to observe the deployment and understand what's happening.
Ok. Maybe not just push the button. TripleO provides us with a bit of last minute validation. Sometimes this just finds weirdness but sometimes it finds something useful. Plus it gives a bit of practice. We're going to launch the pre-deployment validation workflow. You can find templates at /usr/share/openstack-tripleo-validations/ .
$ openstack workflow execution create tripleo.validations.v1.run_groups '{"group_names": ["pre-deployment"]}'
+-------------------+--------------------------------------+
| Field | Value |
+-------------------+--------------------------------------+
| ID | 187856d5-b5f5-4504-9e52-f1a0943a1885 |
| Workflow ID | 2eadb3f2-98c1-4eeb-9c06-3b6b1c7b2b9a |
| Workflow name | tripleo.validations.v1.run_groups |
| Description | |
| Task Execution ID | <none> |
| State | RUNNING |
| State info | None |
| Created at | 2017-01-09 22:23:40.809719 |
| Updated at | 2017-01-09 22:23:40.815409 |
+-------------------+--------------------------------------+
| 1c1fc57b-aaa2-4d26-8da2-a6185a8d5138 | notify_running | tripleo.validations.v1.run_validation | 944af79b-9d1d-46cf-aada-cf1d9ec627d1 | RUNNING | None |
| a4c7df7b-f622-4479-b299-5c482e3c3c4d | notify_running | tripleo.validations.v1.run_validation | 35a00874-91eb-493f-aba6-cc66c780f47d | RUNNING | None |
| e336cfbe-3573-4a13-b9c1-035c684c1f90 | notify_running | tripleo.validations.v1.run_validation | d58a0a39-c039-4957-a5f5-dc142b0224a3 | RUNNING | None |
| 1f80671e-45ce-4572-b7b9-5bdf628f1527 | notify_running | tripleo.validations.v1.run_validation | 5a455516-70c9-436e-92f2-bc171b9b8171 | RUNNING | None |
| 232891c0-73bd-4190-96fd-9b2f6d63515e | notify_running | tripleo.validations.v1.run_validation | 087d2ac7-fa71-42fc-8349-7e75d0e58ea6 | RUNNING | None |
There is so we just repeat that command every few seconds until there isn't. At that point we are interested in the errors... and more we are only interested in errors that occurred during run_validation.
$ openstack task execution list | grep ERROR | grep run_validation
| 377cdc82-fca0-48e4-9f2b-4893c9d01ff4 | run_validation | tripleo.validations.v1.run_validation | 6c2d8b09-49f0-4238-bca9-d62e6cc331e7 | ERROR | {u'stderr': u'', u'stdout... |
| ec233d8d-0a6d-44e7-a12e-f54a21f19dc2 | run_validation | tripleo.validations.v1.run_validation | 6d354722-a9c3-4481-8772-d2c0561211e7 | ERROR | {u'stderr': u'', u'stdout... |
So... 2 errors this pass. Yep. This pass. Some validations stop when they encounter an error so we'll do this multiple times. Also pay attention to the ID of the last error so you know not to check that one again on subsequent passes.
For each error we will run
$ mistral task-get-result {ID} where {ID} is the grouping in the first column.
$ mistral task-get-result 377cdc82-fca0-48e4-9f2b-4893c9d01ff4
{
"stderr": "",
"stdout": "
Task 'Check the services for debug flag' failed:\nHost: localhost\nMessage: The key 'debug' under the section 'DEFAULT' in file /etc/nova/nova.conf has the value: 'True'\n\n
Task 'Check the services for debug flag' failed:\nHost: localhost\nMessage: The key 'debug' under the section 'DEFAULT' in file /etc/neutron/neutron.conf has the value: 'True'\n\n
Task 'Check the services for debug flag' failed:\nHost: localhost\nMessage: The key 'debug' under the section 'DEFAULT' in file /etc/heat/heat.conf has the value: 'True'\n\n
Task 'Check the services for debug flag' failed:\nHost: localhost\nMessage: The key 'debug' under the section 'DEFAULT' in file /etc/ironic/ironic.conf has the value: 'True'\n\nFailure! The validation failed for the following hosts:\n* localhost\n"
}
Looks like we need to edit /etc/nova/nova.conf, /etc/neutron/neutron.conf, /etc/heat/heat.conf and /etc/ironic/ironic.conf to fix the debug key. You might wonder why that needs to be fixed since we never manually changed it. Yeah... I expect this is a case where the validation suite hasn't kept pace with progress. Or maybe they are all test driven development and the validation points out work that needs to happen. Hard to say. For now we'll just make the change and quiet the validation. Or if your comfortable with this setting just leave it alone and move to the next error. Will having logging set to debug cause problems?
Repeat this process until you are error free or at least comfortable with the errors.
Once your content with the validation status, launch it. The deploy has a lot of possible command line options. We're being simple. You can see all the options with 'openstack help overcloud deploy'.
--templates - Create the overcloud using the heat template collection located in /usr/share/openstack-tripleo-heat-templates
by default 1 compute and 1 control node will be deployed which is exactly what we want so no other options need be specified.
In addition to watching the output, run the occasional 'openstack baremetal node list' in another window and watch the provisioning state change. Console on the nodes is also interesting. This is going to take a while... you can watch the heat part, sort of, by opening another window to the director and issuing 'openstack stack list' and then 'openstack stack show {ID}' and 'openstack stack output show {ID} --all'.
$ openstack overcloud deploy --templates
Removing the current plan files
Uploading new plan files
Started Mistral Workflow. Execution ID: b1585580-0b58-469a-b4ed-5246322268b6
Plan updated
Deploying templates in the directory /tmp/tripleoclient-zqUDul/tripleo-heat-templates
Started Mistral Workflow. Execution ID: 0f0da3b6-28ce-463b-a55f-a86475f95690
2017-01-09 22:46:12Z [overcloud]: CREATE_IN_PROGRESS Stack CREATE started
...
Stack overcloud CREATE_COMPLETE
Started Mistral Workflow. Execution ID: a06adb46-e9bf-4506-a8ba-5492a3de5560
Overcloud Endpoint: http://192.168.50.103:5000/v2.0
Overcloud Deployed
Woot!I have an overcloud. A small one, but an overcloud. Next up: How do I access it? Will I be able to spin up an instance that I can also access? This requires a lot of pieces to have worked properly, especially networking. The very fact that the Overcloud Controller is on my private network means I either need to do something clever or attach my workstation to that network so that I can get to the Horizon dashboard. For that matter, why doesn't the Undercloud have a dashboard? It does??? Stay tuned...
When we deploy the overcloud we know that Ironic is going to power up our nodes which will, in turn, PXE boot the initial image from the director. But then what happens? That's where almost all of the magic of TripleO comes into play.
When we deploy the overcloud we will be using OpenStack tooling (Mistral, Heat and Ironic) along with Puppet for configuration management. Each of these is also a potential entry point into customizing our deployments. Red Hat provides a good overview for understanding the Heat Orchestration Templates (HOT): Understanding Heat Templates (It is a bit Red Hat specific and some command won't work if you aren't using Red Hat's variant of openstack... in particular the 'openstack management' command).
Explore a little with 'openstack workbook list' and 'openstack workflow list'.
The Heat template collection for the overcloud is stored in /usr/share/openstack-tripleo-heat-templates/. There's a lot in here and bits of it are pretty interesting to see what the overcloud deployment is doing. We can customize templates and environment files to customize our deployment.
Puppet is the next big contributor to the Overcloud deployment. Puppet (www.puppet.com) is configuration management software. It will allow us to tell which packages to install, how those packages can be configured and how the OS should be configured. It's pretty obvious how this contributes to getting the nodes to have the right software and settings. Modifying the puppet manifests is another entry point should we need to customize our deployment.
Since this is our first deployment we aren't going to tweak anything. Just let it do it's thing and see what shakes out. The background is interesting because we may want to observe the deployment and understand what's happening.
Ok. Maybe not just push the button. TripleO provides us with a bit of last minute validation. Sometimes this just finds weirdness but sometimes it finds something useful. Plus it gives a bit of practice. We're going to launch the pre-deployment validation workflow. You can find templates at /usr/share/openstack-tripleo-validations/ .
$ openstack workflow execution create tripleo.validations.v1.run_groups '{"group_names": ["pre-deployment"]}'
+-------------------+--------------------------------------+
| Field | Value |
+-------------------+--------------------------------------+
| ID | 187856d5-b5f5-4504-9e52-f1a0943a1885 |
| Workflow ID | 2eadb3f2-98c1-4eeb-9c06-3b6b1c7b2b9a |
| Workflow name | tripleo.validations.v1.run_groups |
| Description | |
| Task Execution ID | <none> |
| State | RUNNING |
| State info | None |
| Created at | 2017-01-09 22:23:40.809719 |
| Updated at | 2017-01-09 22:23:40.815409 |
+-------------------+--------------------------------------+
That's the last result that will be pushed to you. It's running. Getting actual results is a tad more complicated. First we have to wait until no more processes are running. We can see the process status by doing 'openstack task execution list'. For our purposes we really don't care about success, we only care about failure. So...
Let's see if anything is running:
$ openstack task execution list | grep RUNNING
| d166b19e-6647-4e7a-979e-e279b9ff3348 | run_validation_group | tripleo.validations.v1.run_groups | 7139ead2-f6ff-443b-a1b9-c0c1aab9e5f3 | RUNNING | None || 1c1fc57b-aaa2-4d26-8da2-a6185a8d5138 | notify_running | tripleo.validations.v1.run_validation | 944af79b-9d1d-46cf-aada-cf1d9ec627d1 | RUNNING | None |
| a4c7df7b-f622-4479-b299-5c482e3c3c4d | notify_running | tripleo.validations.v1.run_validation | 35a00874-91eb-493f-aba6-cc66c780f47d | RUNNING | None |
| e336cfbe-3573-4a13-b9c1-035c684c1f90 | notify_running | tripleo.validations.v1.run_validation | d58a0a39-c039-4957-a5f5-dc142b0224a3 | RUNNING | None |
| 1f80671e-45ce-4572-b7b9-5bdf628f1527 | notify_running | tripleo.validations.v1.run_validation | 5a455516-70c9-436e-92f2-bc171b9b8171 | RUNNING | None |
| 232891c0-73bd-4190-96fd-9b2f6d63515e | notify_running | tripleo.validations.v1.run_validation | 087d2ac7-fa71-42fc-8349-7e75d0e58ea6 | RUNNING | None |
There is so we just repeat that command every few seconds until there isn't. At that point we are interested in the errors... and more we are only interested in errors that occurred during run_validation.
$ openstack task execution list | grep ERROR | grep run_validation
| 377cdc82-fca0-48e4-9f2b-4893c9d01ff4 | run_validation | tripleo.validations.v1.run_validation | 6c2d8b09-49f0-4238-bca9-d62e6cc331e7 | ERROR | {u'stderr': u'', u'stdout... |
| ec233d8d-0a6d-44e7-a12e-f54a21f19dc2 | run_validation | tripleo.validations.v1.run_validation | 6d354722-a9c3-4481-8772-d2c0561211e7 | ERROR | {u'stderr': u'', u'stdout... |
So... 2 errors this pass. Yep. This pass. Some validations stop when they encounter an error so we'll do this multiple times. Also pay attention to the ID of the last error so you know not to check that one again on subsequent passes.
For each error we will run
$ mistral task-get-result {ID} where {ID} is the grouping in the first column.
$ mistral task-get-result 377cdc82-fca0-48e4-9f2b-4893c9d01ff4
{
"stderr": "",
"stdout": "
Task 'Check the services for debug flag' failed:\nHost: localhost\nMessage: The key 'debug' under the section 'DEFAULT' in file /etc/nova/nova.conf has the value: 'True'\n\n
Task 'Check the services for debug flag' failed:\nHost: localhost\nMessage: The key 'debug' under the section 'DEFAULT' in file /etc/neutron/neutron.conf has the value: 'True'\n\n
Task 'Check the services for debug flag' failed:\nHost: localhost\nMessage: The key 'debug' under the section 'DEFAULT' in file /etc/heat/heat.conf has the value: 'True'\n\n
Task 'Check the services for debug flag' failed:\nHost: localhost\nMessage: The key 'debug' under the section 'DEFAULT' in file /etc/ironic/ironic.conf has the value: 'True'\n\nFailure! The validation failed for the following hosts:\n* localhost\n"
}
Looks like we need to edit /etc/nova/nova.conf, /etc/neutron/neutron.conf, /etc/heat/heat.conf and /etc/ironic/ironic.conf to fix the debug key. You might wonder why that needs to be fixed since we never manually changed it. Yeah... I expect this is a case where the validation suite hasn't kept pace with progress. Or maybe they are all test driven development and the validation points out work that needs to happen. Hard to say. For now we'll just make the change and quiet the validation. Or if your comfortable with this setting just leave it alone and move to the next error. Will having logging set to debug cause problems?
Repeat this process until you are error free or at least comfortable with the errors.
Once your content with the validation status, launch it. The deploy has a lot of possible command line options. We're being simple. You can see all the options with 'openstack help overcloud deploy'.
--templates - Create the overcloud using the heat template collection located in /usr/share/openstack-tripleo-heat-templates
by default 1 compute and 1 control node will be deployed which is exactly what we want so no other options need be specified.
In addition to watching the output, run the occasional 'openstack baremetal node list' in another window and watch the provisioning state change. Console on the nodes is also interesting. This is going to take a while... you can watch the heat part, sort of, by opening another window to the director and issuing 'openstack stack list' and then 'openstack stack show {ID}' and 'openstack stack output show {ID} --all'.
$ openstack overcloud deploy --templates
Removing the current plan files
Uploading new plan files
Started Mistral Workflow. Execution ID: b1585580-0b58-469a-b4ed-5246322268b6
Plan updated
Deploying templates in the directory /tmp/tripleoclient-zqUDul/tripleo-heat-templates
Started Mistral Workflow. Execution ID: 0f0da3b6-28ce-463b-a55f-a86475f95690
2017-01-09 22:46:12Z [overcloud]: CREATE_IN_PROGRESS Stack CREATE started
...
Stack overcloud CREATE_COMPLETE
Started Mistral Workflow. Execution ID: a06adb46-e9bf-4506-a8ba-5492a3de5560
Overcloud Endpoint: http://192.168.50.103:5000/v2.0
Overcloud Deployed
Woot!I have an overcloud. A small one, but an overcloud. Next up: How do I access it? Will I be able to spin up an instance that I can also access? This requires a lot of pieces to have worked properly, especially networking. The very fact that the Overcloud Controller is on my private network means I either need to do something clever or attach my workstation to that network so that I can get to the Horizon dashboard. For that matter, why doesn't the Undercloud have a dashboard? It does??? Stay tuned...
Monday, January 9, 2017
OpenStack Day 5: Introspection and Flavors
I kind of feel a little bad about punting on the IPMI issue previously. The TripleO install (and I assume the MAAS based install) are very useful. However I think they could also be useful for home labs and such if they were not so IPMI dependent. Not everyone has a stack of servers laying around. I fully intend to revisit this down the road a bit...
For now though, let's move onward. Our nodes are registered to Baremetal/Ironic.
The next step is "Introspection". This will cause each node to power up and it will PXE boot kernel and ramdisk images from the director. The purpose of this initial boot is for this image to scan the hardware and report back what it finds with regards to ram, disk space, CPU (quantity and capability), etc...
In a small network you may know all the things and will know what purpose you want to put the node toward. In larger networks we can define that certain types of nodes (compute, swift, etc...) require certain physical parameters. These are the "Flavors" that we'll look at later. Using these flavors we can auto-assign nodes to certain roles.
# openstack baremetal introspection bulk start
Setting nodes for introspection to manageable...
Starting introspection of manageable nodes
Started Mistral Workflow. Execution ID: 6f2a291e-d756-4fd9-af20-bb70b683b221
Waiting for introspection to finish...
Introspection for UUID 663d5d3a-b7ca-4d82-96cf-970df5802ca4 finished successfully.
Introspection for UUID 10cf5d91-447d-4c2e-b3fb-40ef80b79d61 finished successfully.
Introspection for UUID 561c884c-c5a3-4bd5-a3df-e96adc588a7a finished successfully.
Introspection for UUID 66e5a5ba-c486-45a7-a0ea-9ccb2a0743f9 finished successfully.
Introspection for UUID be784a1f-a2f1-4c2d-92e7-21c5ae52738a finished successfully.
Introspection completed.
Setting manageable nodes to available...
Started Mistral Workflow. Execution ID: 836a2e6d-b33a-481a-be93-8d9a9d828dbf
Had there been a problem the nodes would have been left in the 'manageable' state. In that case you'd need to do a bit of troubleshooting to figure out what the problem was, fix it and then re-run the introspection.
You can monitor the progress of the introspection if you wish (and should if there were errors and you are re-running). I'd open another terminal window so as not to interfere with the introspection window. In my case I initially got a bad introspection because I had the same IP Address on IPMI of two nodes. Fixed. [Note: That explains why I have 5 instead of 6 nodes even though 6 pieces of hardware made the initial cut after IPMI Testing... the 6th had the same IP as one of the other nodes and I just unplugged it. I'll add it back in later as an example of adding a node after the overcloud has been deployed.]
It is also possible to introspect nodes individually which will be useful as you add single nodes or modify hardware in existing nodes.
You can look at what introspection found by examining an individual node.
Here we can see that there are 24G of RAM, 930G local storage, 8 CPUs, etc...
And here we take a little bit of a side journey. We need to figure out what our nodes are going to be used for. To do that we have to understand the possibilities and the minimum, and recommended, requirements for these possibilities. In some cases we may need to make a few tweaks such as adding RAM or reconfiguring the hard drive layouts. Arguably we should have done this prior to introspection. However, knowing that we can just re-run the introspection it seemed better to wait until we knew that we were going to successfully get to this point with the hardware before investing any time in individual nodes.
So let's look at our various node options and their requirements.
Compute - these nodes will be running the virtual instances. They need to have enough CPU, RAM and disk (*) to do this. The processor must be 64-bit x86 and must support the AMD-V or Intel VT hardware virtualization extensions. 4 Cores is the recommended minimum. It will need at least 6GB of RAM and more if you want to provide more to your instances. We'll want at least 40 GB of disk space and at least one 1-Gbps Network Interface Card, two would be better and three better still if your going to get fancy with the networking and want to avoid a bit of VLAN-ing. Finally it needs to support IPMI which we know already and we'll be revisiting the word "need". The HD layout should just be one large disk/array. By default other disks won't be used.
Controller - responsible for hosting all of the core services: Horizon, Keystone, databases, HA, etc.. Requirements are very much the same as Compute except two (2) Network Interface Cards is a requirement (and like Compute, three would be even better).
Ceph Storage: 64-bit x96 processor. Memory requirements vary depending on the amount of storage. The rule of thumb is 1GB of memory per 1TB of storage space. At least one NIC is required, but two is better and since storage gets hit heavily it's worth considering 10GB interface cards. Ceph nodes will run the Ceph Storage OSD (object storage daemon) but the controllers will be acting as the controller. It is recommended to pre-layout the disk as follows. The director isn't going partition the journal disk, that's a manual step for later. ditto the GPT disk labels ('parted [device] mklabel gpt').
For now though, let's move onward. Our nodes are registered to Baremetal/Ironic.
The next step is "Introspection". This will cause each node to power up and it will PXE boot kernel and ramdisk images from the director. The purpose of this initial boot is for this image to scan the hardware and report back what it finds with regards to ram, disk space, CPU (quantity and capability), etc...
In a small network you may know all the things and will know what purpose you want to put the node toward. In larger networks we can define that certain types of nodes (compute, swift, etc...) require certain physical parameters. These are the "Flavors" that we'll look at later. Using these flavors we can auto-assign nodes to certain roles.
# openstack baremetal introspection bulk start
Setting nodes for introspection to manageable...
Starting introspection of manageable nodes
Started Mistral Workflow. Execution ID: 6f2a291e-d756-4fd9-af20-bb70b683b221
Waiting for introspection to finish...
Introspection for UUID 663d5d3a-b7ca-4d82-96cf-970df5802ca4 finished successfully.
Introspection for UUID 10cf5d91-447d-4c2e-b3fb-40ef80b79d61 finished successfully.
Introspection for UUID 561c884c-c5a3-4bd5-a3df-e96adc588a7a finished successfully.
Introspection for UUID 66e5a5ba-c486-45a7-a0ea-9ccb2a0743f9 finished successfully.
Introspection for UUID be784a1f-a2f1-4c2d-92e7-21c5ae52738a finished successfully.
Introspection completed.
Setting manageable nodes to available...
Started Mistral Workflow. Execution ID: 836a2e6d-b33a-481a-be93-8d9a9d828dbf
You can see that introspection for my five nodes completed successfully and nodes were set to 'available'. This process can take a long time. Don't interrupt it.
$ openstack baremetal node list
| 10cf5d91-447d-4c2e-b3fb-40ef80b79d61 | TAG-203 | None | power off | available | False |
| be784a1f-a2f1-4c2d-92e7-21c5ae52738a | TAG-201 | None | power off | available | False |
| 663d5d3a-b7ca-4d82-96cf-970df5802ca4 | TAG-183 | None | power off | available | False |
| 561c884c-c5a3-4bd5-a3df-e96adc588a7a | TAG-206 | None | power off | available | False |
| 66e5a5ba-c486-45a7-a0ea-9ccb2a0743f9 | TAG-202 | None | power off | available | False |
| 10cf5d91-447d-4c2e-b3fb-40ef80b79d61 | TAG-203 | None | power off | available | False |
| be784a1f-a2f1-4c2d-92e7-21c5ae52738a | TAG-201 | None | power off | available | False |
| 663d5d3a-b7ca-4d82-96cf-970df5802ca4 | TAG-183 | None | power off | available | False |
| 561c884c-c5a3-4bd5-a3df-e96adc588a7a | TAG-206 | None | power off | available | False |
| 66e5a5ba-c486-45a7-a0ea-9ccb2a0743f9 | TAG-202 | None | power off | available | False |
You can monitor the progress of the introspection if you wish (and should if there were errors and you are re-running). I'd open another terminal window so as not to interfere with the introspection window. In my case I initially got a bad introspection because I had the same IP Address on IPMI of two nodes. Fixed. [Note: That explains why I have 5 instead of 6 nodes even though 6 pieces of hardware made the initial cut after IPMI Testing... the 6th had the same IP as one of the other nodes and I just unplugged it. I'll add it back in later as an example of adding a node after the overcloud has been deployed.]
$ sudo journalctl -l -u openstack-ironic-discoverd -u openstack-ironic-discoverd-dnsmasq -u openstack-ironic-conductor -f
$ ironic node maintenance set [NODE UUID] $ openstack baremetal introspection start [NODE UUID] $ ironic node maintenance unset [NODE UUID]
$ openstack baremetal node show 10cf5d91-447d-4c2e-b3fb-40ef80b79d61...| properties | {u'memory_mb': u'24576', u'cpu_arch': u'x86_64', u'local_gb': u'930', u'cpus': u'8', u'capabilities': u'cpu_hugepages:true,cpu_txt:true,boot_option:local,cpu_aes:true,cpu_vt:true,cpu_hugepages_1g:true'}...
And here we take a little bit of a side journey. We need to figure out what our nodes are going to be used for. To do that we have to understand the possibilities and the minimum, and recommended, requirements for these possibilities. In some cases we may need to make a few tweaks such as adding RAM or reconfiguring the hard drive layouts. Arguably we should have done this prior to introspection. However, knowing that we can just re-run the introspection it seemed better to wait until we knew that we were going to successfully get to this point with the hardware before investing any time in individual nodes.
So let's look at our various node options and their requirements.
Compute - these nodes will be running the virtual instances. They need to have enough CPU, RAM and disk (*) to do this. The processor must be 64-bit x86 and must support the AMD-V or Intel VT hardware virtualization extensions. 4 Cores is the recommended minimum. It will need at least 6GB of RAM and more if you want to provide more to your instances. We'll want at least 40 GB of disk space and at least one 1-Gbps Network Interface Card, two would be better and three better still if your going to get fancy with the networking and want to avoid a bit of VLAN-ing. Finally it needs to support IPMI which we know already and we'll be revisiting the word "need". The HD layout should just be one large disk/array. By default other disks won't be used.
Controller - responsible for hosting all of the core services: Horizon, Keystone, databases, HA, etc.. Requirements are very much the same as Compute except two (2) Network Interface Cards is a requirement (and like Compute, three would be even better).
Ceph Storage: 64-bit x96 processor. Memory requirements vary depending on the amount of storage. The rule of thumb is 1GB of memory per 1TB of storage space. At least one NIC is required, but two is better and since storage gets hit heavily it's worth considering 10GB interface cards. Ceph nodes will run the Ceph Storage OSD (object storage daemon) but the controllers will be acting as the controller. It is recommended to pre-layout the disk as follows. The director isn't going partition the journal disk, that's a manual step for later. ditto the GPT disk labels ('parted [device] mklabel gpt').
- /dev/sda - root disk where the director will put the overcloud image
- /dev/sdb - Journal disk, ideally SSD to improve performance.
- /dev/sdc ... /dev/sdN - the OSD disks.
Cinder Storage - See Ceph Storage
Swift Storage - See Ceph Storage
Ok. The Hard disk requirement of 40G is pretty laughable these days. Unless your running on a USB stick your hard disk is going to be bigger than that. Even low end SSD drives are all larger now. So we'll take that one as "have a hard drive". We will discuss some of the finer points of local storage on compute and controller nodes in the future as there are some optimizations and interesting tidbits. But for now: have a hard disk.
On my first deployment I'm not going to bother with Ceph, Cinder or Swift. These are all optional roles. I fully intend to completely destroy my environment a few times before I'm happy. The only requirement for an overcloud are one controller and one compute node. That's where I intend to start and go so far as to launch and access an instance to make sure I've got all the pieces figured out. Then we burn it all down and re-deploy based on the knowledge learned to that point. Iterate.
In this first iteration I'm not going to get crazy with the networking either. In fact my application may not require any advanced networking ever. Redhat provides a pretty excellent page on Planning Networks. From that perspective I'm shooting for a 'Basic Environment'.
My final storage layout is going to have to be more complex. I really don't want my images stored on the controller and I will need some persistent instance storage. My initial thinking was I'd use Ceph for this but I've seen a rumor that Ceph doesn't support QCOW2 meaning I'd have to use RAW format to boot from Glance if it was Ceph backed. Not sure how I feel about that yet... and it may only apply if you are booting it as a guest disk (but there are good reasons to do just that).
The Director is capable of creating Cinder nodes, swift nodes and ceph nodes. The ceph nodes are capable of backing Images (via Glance), Volumes (Cinder) and Guest Disks which allow booting inside Ceph w/out using Cinder.
Here I take a brief pause to move around any RAM or hard drives and to configure any RAID that may be necessary before moving onto flavorings.
Let's take a look at our flavors.
According to the TripleO docs: "The undercloud will have a number of default flavors created at install time. In most cases these flavors do not need to be modified, but they can if desired. By default, all overcloud instances will be booted with the baremetal flavor, so all baremetal nodes must have at least as much memory, disk, and cpu as that flavor.". Okay. That's a mouth full. Let's break it down.
$ openstack flavor list
+--------------------------------------+---------------+------+------+-----------+-------+-----------+
| ID | Name | RAM | Disk | Ephemeral | VCPUs | Is Public |
+--------------------------------------+---------------+------+------+-----------+-------+-----------+
| 1bbdc277-9b08-465a-9e4e-4e0e0d949e85 | control | 4096 | 40 | 0 | 1 | True |
| 55ef3815-ee57-4ce7-829e-0d9f5afdc127 | compute | 4096 | 40 | 0 | 1 | True |
| 6d33b344-4e96-4b24-835f-4c7c8fe76373 | ceph-storage | 4096 | 40 | 0 | 1 | True |
| c26e60eb-ed4f-407c-b838-5864cb67990a | baremetal | 4096 | 40 | 0 | 1 | True |
| ca2d58c7-afd1-4f5e-9aa6-b729bf4bf4da | swift-storage | 4096 | 40 | 0 | 1 | True |
| e7e4a151-0a23-4503-a482-098e368eaf44 | block-storage | 4096 | 40 | 0 | 1 | True |
+--------------------------------------+---------------+------+------+-----------+-------+-----------+
+--------------------------------------+---------------+------+------+-----------+-------+-----------+
| ID | Name | RAM | Disk | Ephemeral | VCPUs | Is Public |
+--------------------------------------+---------------+------+------+-----------+-------+-----------+
| 1bbdc277-9b08-465a-9e4e-4e0e0d949e85 | control | 4096 | 40 | 0 | 1 | True |
| 55ef3815-ee57-4ce7-829e-0d9f5afdc127 | compute | 4096 | 40 | 0 | 1 | True |
| 6d33b344-4e96-4b24-835f-4c7c8fe76373 | ceph-storage | 4096 | 40 | 0 | 1 | True |
| c26e60eb-ed4f-407c-b838-5864cb67990a | baremetal | 4096 | 40 | 0 | 1 | True |
| ca2d58c7-afd1-4f5e-9aa6-b729bf4bf4da | swift-storage | 4096 | 40 | 0 | 1 | True |
| e7e4a151-0a23-4503-a482-098e368eaf44 | block-storage | 4096 | 40 | 0 | 1 | True |
+--------------------------------------+---------------+------+------+-----------+-------+-----------+
Here we see the "number of default flavors" that were created in the undercloud at install time. These are all set to the same requirements: 4GB of RAM, 40G of Disk, 1 CPU. Pretty small and not entirely useful requirements.
By default all nodes are flavored as "baremetal" so they need at least the minimum requirements specified for the "baremetal" flavor. This is the default and doesn't show up when we look at the node properties.
| properties | {u'memory_mb': u'24576', u'cpu_arch': u'x86_64', u'local_gb': u'930', u'cpus': u'8', u'capabilities': u'cpu_hugepages:true,cpu_txt:true,boot_option:local,cpu_aes:true,cpu_vt:true,cpu_hugepages_1g:true'}
However if we change the flavor of a node:
$ ironic node-update 10cf5d91-447d-4c2e-b3fb-40ef80b79d61 replace properties/capabilities=cpu_hugepages:true,cpu_txt:true,boot_option:local,cpu_aes:true,cpu_vt:true,cpu_hugepages_1g:true,profile:control
So this is essentially the same properties/capabilities section with the addition of profile:compute. No when we do 'openstack baremetal node show 10cf5d91-447d-4c2e-b3fb-40ef80b79d61' we can see the change in capabilities:
| properties | {u'memory_mb': u'24576', u'cpu_arch': u'x86_64', u'local_gb': u'930', |
| | u'cpus': u'8', u'capabilities': u'cpu_hugepages:true,cpu_txt:true,boot_o |
| | ption:local,cpu_aes:true,cpu_vt:true,cpu_hugepages_1g:true,profile:contr |
| | ol'} |
|
| | u'cpus': u'8', u'capabilities': u'cpu_hugepages:true,cpu_txt:true,boot_o |
| | ption:local,cpu_aes:true,cpu_vt:true,cpu_hugepages_1g:true,profile:contr |
| | ol'} |
|
We can see that this will use the 'control' profile. To set it back to the default we can just re-do the node-update without that addition.
An alternate method:
$ ironic node-update 10cf5d91-447d-4c2e-b3fb-40ef80b79d61 add properties/capabilities='profile:compute,boot_option:local'
However this doesn't do what it should really. It doesn't add to 'properties/capabilties' instead it adds a 'properties/capabilities' section to the node and, in the process, overwrites that section if it already exists. For our purposes 'add' and 'replace' are the same in this specific case.
An alternate method:
$ ironic node-update 10cf5d91-447d-4c2e-b3fb-40ef80b79d61 add properties/capabilities='profile:compute,boot_option:local'
However this doesn't do what it should really. It doesn't add to 'properties/capabilties' instead it adds a 'properties/capabilities' section to the node and, in the process, overwrites that section if it already exists. For our purposes 'add' and 'replace' are the same in this specific case.
Once a profile is assigned to a flavor, nova will only deploy it on ironic nodes with the same profile. Your overcloud deployment will fail if not enough ironic nodes are tagged with a profile.
I'm using a limited amount of hardware and it's capabilities vary pretty widely. Because of this it's not worth my effort to setup any fashion of automatic flavoring. However it is worth my effort to manually flavor nodes since some are better suited to a task by virtue of RAM, CPU or HD space.
Since my initial implementation will be two nodes I will tag one as control and the other as compute using the method above.
One last bit of configuration before we deploy the overcloud.
It is recommended to set a nameserver so that the nodes can resolve DNS.
$ neutron subnet-list
| ae23dfb5-cb87-4475-adc9-ec82062db8af | | 192.168.50.0/24 | {"start": "192.168.50.100", "end": "192.168.50.250"} |
[stack@ostack-director ~]$ neutron subnet-update ae23dfb5-cb87-4475-adc9-ec82062db8af --dns-nameserver 8.8.8.8
Updated subnet: ae23dfb5-cb87-4475-adc9-ec82062db8af
| ae23dfb5-cb87-4475-adc9-ec82062db8af | | 192.168.50.0/24 | {"start": "192.168.50.100", "end": "192.168.50.250"} |
[stack@ostack-director ~]$ neutron subnet-update ae23dfb5-cb87-4475-adc9-ec82062db8af --dns-nameserver 8.8.8.8
Updated subnet: ae23dfb5-cb87-4475-adc9-ec82062db8af
Subscribe to:
Posts (Atom)