Support for NVMe devices?

[tompsku]

Is there plan to support aws series-5 and t3 nodes that used NVMe devices? As current system does not support device naming scheme used by these nodes.

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/nvme-ebs-volumes.html#identify-nvme-ebs-device

[bmoyles]

We handle this similar to what you'd find on an Amazon Linux AMI -- there are udev rules in place that set up equivalent /dev/xvd and /dev/sd links to the NVMe devices.
I'd suggest launching an Amazon Linux AMI and checking out their rules and helper scripts. On Nitro instances, the real requested block device name is embedded in the NVMe device metadata, for example, which makes mapping an externally requested /dev/xvdN to the correct NVMe device possible.

[tompsku]

Thanks for the udev hint, currently using pure centos7 ami i.e. not amazon ami.
After udev rules and some tuning I debug this more first failure comes in util/linux.py in native_device_prefix method as it tries to find “common” device name prefix and it uses os.listdir('/sys/block') for it. Naturally this fails as udev rules do not create new block device and should not do that. After fixing this too next problem is how devices are named, root device used for “common” device name comes as xvda, but volumes attached by aminator gets sdf. This cause new udev rule fix and fake those dev-link names. After all of these changes and changes to aminator code I finally got it running. So, if you use those standard udev rules from Amazon Linux AMI, I think you need to have some other magic in config files or other places …

[bmoyles]

Try this set of files
https://gist.github.com/bmoyles/755ffd13847bb6a6b50fbe014962595f

It's near what Amazon provides but with some tweaks we made internally, specifically in 70-ec2-nvme-devices.rules and ec2nvme-id

It ensures that if you ask for a volume at /dev/xvdb, for example, and you're on a Nitro instance where you will get something like /dev/nvme1n1, that both /dev/sdb and /dev/xvdb symlinks are created for the device so there's no ambiguity.

It's kind of funky, but Amazon will allow one to ask for a device at /dev/xvdb and another at /dev/sdc. It even allows you to ask for /dev/xvdb and then /dev/sdb -- the latter call will not fail but the volume will be stuck in attaching (or that was the case when I last tried this a few months back and re-worked some of the udev foo).

ec2nvme-id as we modified it can be a handy debugging tool, too, both as a python library and a command line tool. We have it symlinked into /sbin so it lands on the PATH and is easily accessible

# /sbin/ec2nvme-id -h
usage: ec2nvme-id [-h] [-v | -b | -u] device

Reads EBS or instance-store information from NVMe devices.

positional arguments:
  device                Device to query

optional arguments:
  -h, --help            show this help message and exit

Output Options:
  Options may not be combined. Default is to return both volume and block
  device

  -v, --volume          Return volume-id
  -b, --block-dev-aliases
                        Return block device aliases
  -u, --udev            Output block device aliases in format suitable for
                        udev rules

# /sbin/ec2nvme-id /dev/nvme0n1 
Volume ID: vol-04b08c43961e1b62e
Block Device Aliases: /dev/sdb /dev/xvdb

# /sbin/ec2nvme-id -v /dev/nvme0n1 
Volume ID: vol-04b08c43961e1b62e

# /sbin/ec2nvme-id -b /dev/nvme0n1
Block Device Aliases: /dev/sdb /dev/xvdb

# /sbin/ec2nvme-id -u /dev/nvme0n1
_EBS_ALIAS_SD=sdb
_EBS_ALIAS_XVD=xvd

[bmoyles]

The other thing you might need is configuration for the block device plugin.
The default is
https://github.com/Netflix/aminator/blob/master/aminator/plugins/blockdevice/default_conf/aminator.plugins.blockdevice.linux.yml

The default configuration is still wired up such that it assumes you're baking on an old paravirtual instance that allows you to attach a volume at a minor device

Per https://github.com/Netflix/aminator/blob/master/aminator/default_conf/aminator.yml the config root is, by default, /etc/aminator. Try creating /etc/aminator/plugins/aminator.plugins.blockdevice.linux.yml with contents:

enabled: true
device_prefixes: [xvd, sd]
device_letters: 'jklmnopqrstuvwxyz'
use_minor_device_numbers: false

in conjunction with the udev config in the previous post.

[bmoyles]

After digging a bit more and after an unrelated internal discussion, I see where this breaks down on Nitro instances and w/ NVMe, and my suggestions above are incomplete. The udev stuff is likely still a necessity, but as you noted, the naive look at /sys/block to determine the "native" device prefix is flawed. We haven't moved our aminator workers to Nitro instances so it hasn't blown up on us yet but it absolutely will and we'll get a fix out for this soon.

cc @asher -- this is somewhat related to the discussion on the PR for SPIN-3856