The Full Stacky: OpenStack Day 5: Introspection and Flavors

I kind of feel a little bad about punting on the IPMI issue previously. The TripleO install (and I assume the MAAS based install) are very useful. However I think they could also be useful for home labs and such if they were not so IPMI dependent. Not everyone has a stack of servers laying around. I fully intend to revisit this down the road a bit...

For now though, let's move onward. Our nodes are registered to Baremetal/Ironic.

The next step is "Introspection". This will cause each node to power up and it will PXE boot kernel and ramdisk images from the director. The purpose of this initial boot is for this image to scan the hardware and report back what it finds with regards to ram, disk space, CPU (quantity and capability), etc...

In a small network you may know all the things and will know what purpose you want to put the node toward. In larger networks we can define that certain types of nodes (compute, swift, etc...) require certain physical parameters. These are the "Flavors" that we'll look at later. Using these flavors we can auto-assign nodes to certain roles.

# openstack baremetal introspection bulk start
Setting nodes for introspection to manageable...
Starting introspection of manageable nodes
Started Mistral Workflow. Execution ID: 6f2a291e-d756-4fd9-af20-bb70b683b221
Waiting for introspection to finish...
Introspection for UUID 663d5d3a-b7ca-4d82-96cf-970df5802ca4 finished successfully.
Introspection for UUID 10cf5d91-447d-4c2e-b3fb-40ef80b79d61 finished successfully.
Introspection for UUID 561c884c-c5a3-4bd5-a3df-e96adc588a7a finished successfully.
Introspection for UUID 66e5a5ba-c486-45a7-a0ea-9ccb2a0743f9 finished successfully.
Introspection for UUID be784a1f-a2f1-4c2d-92e7-21c5ae52738a finished successfully.
Introspection completed.
Setting manageable nodes to available...
Started Mistral Workflow. Execution ID: 836a2e6d-b33a-481a-be93-8d9a9d828dbf

You can see that introspection for my five nodes completed successfully and nodes were set to 'available'. This process can take a long time. Don't interrupt it.

Had there been a problem the nodes would have been left in the 'manageable' state. In that case you'd need to do a bit of troubleshooting to figure out what the problem was, fix it and then re-run the introspection.

You can monitor the progress of the introspection if you wish (and should if there were errors and you are re-running). I'd open another terminal window so as not to interfere with the introspection window. In my case I initially got a bad introspection because I had the same IP Address on IPMI of two nodes. Fixed. [Note: That explains why I have 5 instead of 6 nodes even though 6 pieces of hardware made the initial cut after IPMI Testing... the 6th had the same IP as one of the other nodes and I just unplugged it. I'll add it back in later as an example of adding a node after the overcloud has been deployed.]

$ sudo journalctl -l -u openstack-ironic-discoverd -u openstack-ironic-discoverd-dnsmasq -u openstack-ironic-conductor -f

It is also possible to introspect nodes individually which will be useful as you add single nodes or modify hardware in existing nodes.

$ ironic node maintenance set [NODE UUID]
$ openstack baremetal introspection start [NODE UUID]
$ ironic node maintenance unset [NODE UUID]

You can look at what introspection found by examining an individual node.


$ openstack baremetal node show 10cf5d91-447d-4c2e-b3fb-40ef80b79d61

...

| properties             | {u'memory_mb': u'24576', u'cpu_arch': u'x86_64', u'local_gb': u'930', u'cpus': u'8', u'capabilities': u'cpu_hugepages:true,cpu_txt:true,boot_option:local,cpu_aes:true,cpu_vt:true,cpu_hugepages_1g:true'}

...

Here we can see that there are 24G of RAM, 930G local storage, 8 CPUs, etc...

And here we take a little bit of a side journey. We need to figure out what our nodes are going to be used for. To do that we have to understand the possibilities and the minimum, and recommended, requirements for these possibilities. In some cases we may need to make a few tweaks such as adding RAM or reconfiguring the hard drive layouts. Arguably we should have done this prior to introspection. However, knowing that we can just re-run the introspection it seemed better to wait until we knew that we were going to successfully get to this point with the hardware before investing any time in individual nodes.

So let's look at our various node options and their requirements.

Compute - these nodes will be running the virtual instances. They need to have enough CPU, RAM and disk (*) to do this. The processor must be 64-bit x86 and must support the AMD-V or Intel VT hardware virtualization extensions. 4 Cores is the recommended minimum. It will need at least 6GB of RAM and more if you want to provide more to your instances. We'll want at least 40 GB of disk space and at least one 1-Gbps Network Interface Card, two would be better and three better still if your going to get fancy with the networking and want to avoid a bit of VLAN-ing. Finally it needs to support IPMI which we know already and we'll be revisiting the word "need". The HD layout should just be one large disk/array. By default other disks won't be used.

Controller - responsible for hosting all of the core services: Horizon, Keystone, databases, HA, etc.. Requirements are very much the same as Compute except two (2) Network Interface Cards is a requirement (and like Compute, three would be even better).

Ceph Storage: 64-bit x96 processor. Memory requirements vary depending on the amount of storage. The rule of thumb is 1GB of memory per 1TB of storage space. At least one NIC is required, but two is better and since storage gets hit heavily it's worth considering 10GB interface cards. Ceph nodes will run the Ceph Storage OSD (object storage daemon) but the controllers will be acting as the controller. It is recommended to pre-layout the disk as follows. The director isn't going partition the journal disk, that's a manual step for later. ditto the GPT disk labels ('parted [device] mklabel gpt').

/dev/sda - root disk where the director will put the overcloud image
/dev/sdb - Journal disk, ideally SSD to improve performance.
/dev/sdc ... /dev/sdN - the OSD disks.

Cinder Storage - See Ceph Storage

Swift Storage - See Ceph Storage

Ok. The Hard disk requirement of 40G is pretty laughable these days. Unless your running on a USB stick your hard disk is going to be bigger than that. Even low end SSD drives are all larger now. So we'll take that one as "have a hard drive". We will discuss some of the finer points of local storage on compute and controller nodes in the future as there are some optimizations and interesting tidbits. But for now: have a hard disk.

On my first deployment I'm not going to bother with Ceph, Cinder or Swift. These are all optional roles. I fully intend to completely destroy my environment a few times before I'm happy. The only requirement for an overcloud are one controller and one compute node. That's where I intend to start and go so far as to launch and access an instance to make sure I've got all the pieces figured out. Then we burn it all down and re-deploy based on the knowledge learned to that point. Iterate.

In this first iteration I'm not going to get crazy with the networking either. In fact my application may not require any advanced networking ever. Redhat provides a pretty excellent page on Planning Networks. From that perspective I'm shooting for a 'Basic Environment'.

My final storage layout is going to have to be more complex. I really don't want my images stored on the controller and I will need some persistent instance storage. My initial thinking was I'd use Ceph for this but I've seen a rumor that Ceph doesn't support QCOW2 meaning I'd have to use RAW format to boot from Glance if it was Ceph backed. Not sure how I feel about that yet... and it may only apply if you are booting it as a guest disk (but there are good reasons to do just that).

The Director is capable of creating Cinder nodes, swift nodes and ceph nodes. The ceph nodes are capable of backing Images (via Glance), Volumes (Cinder) and Guest Disks which allow booting inside Ceph w/out using Cinder.

Here I take a brief pause to move around any RAM or hard drives and to configure any RAID that may be necessary before moving onto flavorings.

Let's take a look at our flavors.

According to the TripleO docs: "The undercloud will have a number of default flavors created at install time. In most cases these flavors do not need to be modified, but they can if desired. By default, all overcloud instances will be booted with the baremetal flavor, so all baremetal nodes must have at least as much memory, disk, and cpu as that flavor.". Okay. That's a mouth full. Let's break it down.

$ openstack flavor list
+--------------------------------------+---------------+------+------+-----------+-------+-----------+
| ID                                   | Name          | RAM | Disk | Ephemeral | VCPUs | Is Public |
+--------------------------------------+---------------+------+------+-----------+-------+-----------+
| 1bbdc277-9b08-465a-9e4e-4e0e0d949e85 | control       | 4096 |   40 |         0 |     1 | True      |
| 55ef3815-ee57-4ce7-829e-0d9f5afdc127 | compute       | 4096 |   40 |         0 |     1 | True      |
| 6d33b344-4e96-4b24-835f-4c7c8fe76373 | ceph-storage | 4096 |   40 |         0 |     1 | True      |
| c26e60eb-ed4f-407c-b838-5864cb67990a | baremetal     | 4096 |   40 |         0 |     1 | True      |
| ca2d58c7-afd1-4f5e-9aa6-b729bf4bf4da | swift-storage | 4096 |   40 |         0 |     1 | True      |
| e7e4a151-0a23-4503-a482-098e368eaf44 | block-storage | 4096 |   40 |         0 |     1 | True      |
+--------------------------------------+---------------+------+------+-----------+-------+-----------+

Here we see the "number of default flavors" that were created in the undercloud at install time. These are all set to the same requirements: 4GB of RAM, 40G of Disk, 1 CPU. Pretty small and not entirely useful requirements.

By default all nodes are flavored as "baremetal" so they need at least the minimum requirements specified for the "baremetal" flavor. This is the default and doesn't show up when we look at the node properties.

| properties | {u'memory_mb': u'24576', u'cpu_arch': u'x86_64', u'local_gb': u'930', u'cpus': u'8', u'capabilities': u'cpu_hugepages:true,cpu_txt:true,boot_option:local,cpu_aes:true,cpu_vt:true,cpu_hugepages_1g:true'}

However if we change the flavor of a node:

$ ironic node-update 10cf5d91-447d-4c2e-b3fb-40ef80b79d61 replace properties/capabilities=cpu_hugepages:true,cpu_txt:true,boot_option:local,cpu_aes:true,cpu_vt:true,cpu_hugepages_1g:true,profile:control

So this is essentially the same properties/capabilities section with the addition of profile:compute. No when we do 'openstack baremetal node show 10cf5d91-447d-4c2e-b3fb-40ef80b79d61' we can see the change in capabilities:

| properties       | {u'memory_mb': u'24576', u'cpu_arch': u'x86_64', u'local_gb': u'930',    |
|                        | u'cpus': u'8', u'capabilities': u'cpu_hugepages:true,cpu_txt:true,boot_o   |
|                        | ption:local,cpu_aes:true,cpu_vt:true,cpu_hugepages_1g:true,profile:contr |
|                        | ol'}                                                                                                            |
|

We can see that this will use the 'control' profile. To set it back to the default we can just re-do the node-update without that addition.

An alternate method:

$ ironic node-update 10cf5d91-447d-4c2e-b3fb-40ef80b79d61 add properties/capabilities='profile:compute,boot_option:local'

However this doesn't do what it should really. It doesn't add to 'properties/capabilties' instead it adds a 'properties/capabilities' section to the node and, in the process, overwrites that section if it already exists. For our purposes 'add' and 'replace' are the same in this specific case.

Once a profile is assigned to a flavor, nova will only deploy it on ironic nodes with the same profile. Your overcloud deployment will fail if not enough ironic nodes are tagged with a profile.

I'm using a limited amount of hardware and it's capabilities vary pretty widely. Because of this it's not worth my effort to setup any fashion of automatic flavoring. However it is worth my effort to manually flavor nodes since some are better suited to a task by virtue of RAM, CPU or HD space.

Since my initial implementation will be two nodes I will tag one as control and the other as compute using the method above.

One last bit of configuration before we deploy the overcloud.

It is recommended to set a nameserver so that the nodes can resolve DNS.

$ neutron subnet-list
| ae23dfb5-cb87-4475-adc9-ec82062db8af | | 192.168.50.0/24 | {"start": "192.168.50.100", "end": "192.168.50.250"} |
[stack@ostack-director ~]$ neutron subnet-update ae23dfb5-cb87-4475-adc9-ec82062db8af --dns-nameserver 8.8.8.8
Updated subnet: ae23dfb5-cb87-4475-adc9-ec82062db8af

And with that we are ready for our initial deployment.

The Full Stacky

Monday, January 9, 2017

OpenStack Day 5: Introspection and Flavors

No comments:

Post a Comment