Okay, so maybe hiring that guy who looks like Snidely Whiplash was a poor choice.
Starting with ONTAP 9.11.1 you can enable further secure ONTAP by setting up multi-admin verification, or MAV for short. MAV can prevent system administrators from taking certain data disruptive actions without getting approval from at least one other administrator.
How it Works
MAV works by creating a rule set of commands or GUI operations that require multiple users to approve before it can be executed. Alongside are administrator groups which can provide authentication for those actions.
When an action is triggered that’s a part of that rule set, say the deletion of a volume, the process is put into a pending state. Once another MAV group administrator approves the action the original requestor can then execute what they were trying to do.
Rule sets can be built from one or multiple of the follow actions (as of 9.11.1)(sorted by the order in which they show up in System Manager):
- cluster peer delete
- vserver peer delete
- volume snapshot delete
- volume delete
- volume flexcache delete
- secuirty login password*
- security login unlock*
- event config modify*
- set -privilege diagnostic
- security login create
- security login modify
- system mode run
- system node systemshell
- volume snapshot autodelete modify
- volume snapshot restore
- volume snapshot policy create
- volume snapshot policy modify
- volume snapshot add-schedule
- volume snapshot modify-schedule
- volume snapshot remove schedule
The ones with an * behind are rules created automatically with MAV configuration. I guess you can remove them after the fact but that seems to me like opening a big security hole.
Throughout this doc and others you might see administrator and admin used interchangeably. To clarify, MAV isn’t just for personnel with full admin rights, but for any system administrator with appropriate access privileges.
It’s pretty simple to set up and configure. It’s also highly customizable allowing you create multiple user groups and multiple rule sets. In a smaller organization you can simply set one of each just to cross check potentially disruptive actions across two administrators. A larger org could delineate responsibilities, say creating a compliance group who’s responsibility is just managing the deletion of data.
Oh, shit. I need to make changes and one of my admins is in Cancun!
Good news is that it seems pretty easy to disable MAV without having secondary approval. The bad news is that it seems pretty easy to disable MAV without having secondary approval…
Okay, so MAV is good for a lot of scenarios. And as noted before its not without its flaws and SnapLock is the only way to get real security here. While a properly configured SnapLock environment won’t let anyone bypass restrictions, MAV can be disabled with a few commands via accessing the system via direct console access and diagnostic level commands.
The commands are readily available via a NetApp KB article but only visible to NetApp employees and verified partners. I won’t share those commands here but if encounter an issue and need to bypass MAV call up support and reference that KB.
MAV vs SnapLock
You might ask yourself, “why would I pay for a SnapLock license when I can use mutli-admin verification for free?” The simple answer is that MAV is defeatable, whether through colluding admins or an intentional backdoor (covered later in this article).
While folks will talk about MAV as a way to prevent rogue administrators everyone knows that someone really looking to cause harm has way more ability to do so than we want to admit. SnapLock is the only way to really prevent the deletion of data. Not only does it protect snapshots but it also protects the underlying aggregates. With SnapLock Compliance enabled the only way to delete data out of turn is to physically destroy the drives.
In my mind MAV is best for preventing someone from making stupid mistakes. New guy on the team? Want to be extra cautious so no one accidentally deletes a volume? MAV is a great way to add that safety net. Preventing a compromised account? I get it, but frankly I haven’t seen that happen without everything else having gone to hell in a hand basket first. And that rogue administrator? If someone really wanted to fuck things up MAV isn’t going to stop them.
Prerequisites & Considerations
As of 9.12.1 MAV is set up cluster wide. This means core security actions, like changing passwords for other users, is globally MAV protected. You can configure general MAV rules to be specific to operations at a SVM level.
Before getting started make sure you have SNMP configured so that the system can email administrators when there’s a request that requires their attention. This is not a hard requirement from what I can tell, but it’s definitely best practice. While you’ll probably want to reach out to teammates directly to get verification, rather than waiting for them to respond to an email, it at least makes for a nice external audit log.
Speaking of best practices I think it would be best to review the available rule sets to plan out policies before setting things up.
Also, don’t be a dummy like me and make sure that all users are accounted for and able to log in. Changing passwords will require MAV approval (wait, is that like saying ATM Machine?). If you only have two administrators, and one has forgotten how to log in… well that’s why there’s a workaround.
On top of that, don’t create any approval situations where you have more requirements than actual users. I didn’t test every scenario, it currently being 2:30 on a Saturday, but System Manager seemed to have no problem with creating approval requirements from more users than I actually had configured.
Configuring MAV via CLI
As with all things CLI, setting up MAV is pretty straightforward.
First step is to create a MAV approval group:
security multi-admin-verify approval-group create -vserver <admin SVM name> -name <MAV group name> -approvers <comma separated list of admin accounts> -email <comma separated list notification emails>
Second step is to enable MAV:
security multi-admin-verify modify -approval-groups <approval group(s)> -enabled true
There are a bunch of other flags on this command that allow you to further refine the config:
-required-approvers sets a number of individual approvers required to commit an action
-approval-expiry [#h][#m][#s] sets a time limit for administrative approval (default is 1 hour)
-execution-expiry [#h][#m][#s] sets a time limit for the requesting user to commit the requested action (default is 1 hour)
The documentation only shows setting hours, minutes, and seconds, but in the GUI you can configure days with the max documented as being 14 days.
Third step is to define the approval rules:
security multi-admin-verify rule create -operation <"protected_operation">
“Protected_operation” is one of those rules mentioned above (such as volume delete). There’s no rule grouping that I can see, so it’s just a stack of independent rules, thus you’d need to run that command multiple times for each action.
To this you can also add -query <operation_subset> which can further define command flags such as when removing a snapshot with the -force true flag. The command can also be extended with additional flags including -vserver, -query, -required-approvers, -approval-groups, -execution-expiry, and -approval-expiry.
CLI Approval Process
When you try and run a command that’s MAV protected via CLI you’ll get a message stating…
Warning: This operation requires multi-admin verification. To create a verification request use "security multi-admin-verify request create". Would you like to create a request for this operation? (y|n):
Hitting y results in the following response…
Error: command failed: The security multi-admin-verify request (index #) is auto-generated and requires approval.
The index # is going to be the job reference ID for your fellow administrators to reference.
Shout across the cubicle and ask your friend to run…
security multi-admin-verify request show <index #>
The output will show you details of the request including the operation, operation specifics, who requested it, etc etc.
If you don’t have the specific index number you can run:
security multi-admin-verify request show-pending
To take action run:
security multi-admin-verify request approve/veto/delete <index #>
As you might imagine approve, well, approves the request. Veto keeps it in the queue until there’s enough approvals. Delete removes the request from the available actions.
Note that request approval doesn’t automatically result in that action being taken. The original requestor needs to go back in and repeat the action a second time, this time it executes without passing through the verification process.
Configuring MAV via System Manager
If you’re setting up MAV with an eye for simplicity then System Manager is probably the way to do it. Yeah, yeah, I want to rag on SM and there are plenty of GUI issues to nitpick here, but for a simple MAV configuration you can’t beat it.
To configure MAV, expand Cluster in the left-hand navigation menu then go to Settings and scroll down until you find the Multi-Admin Approval tile. Click the gear to bring up the setup wizard.
The GUI is a bit wonky so get ready for some nested scrolling, cutoff text, and oddly shaped input boxes.
In the first section click + Add to create a group and give it a name, select the various user accounts that can grant approval, email addresses for notifications to be sent to, and check the Default Group (for at least one group).
In the second section click + Add a second time and continue to do so to add operational rule sets. Click add, then go to the operation drop down, select one of the rules, rinse and repeat.
You can define query specifics, the required number of approvers required, and even specific approval groups if you created different ones in the previous step. None of these are required. Unfortunately you need to click + Add for every. single. operation. No way to just select all.
Under Advanced you can further fine tune some of the operating parameters and even configure SMTP if you haven’t already.
Unfortunately there’s very little error correction in this wizard so you might encounter an error message after hitting submit and have to do it all over again.
After configuration hitting the gear icon on the widget will take you to a proper configuration page where you can tweak the overall MAV config.
System Manager Approval Process
Actions can be approved by qualified administrators by going to Events & Jobs in the left hand navigation menu then clicking on Mult-Admin Requests. Here you will see a list of past and current requests. Select one of the requested actions and Delete, Approve, and Reject (for some reason they didn’t use “veto” like in the CLI) at the top.
If there are additional approvers required for an action, and you change your mind, you can once again click and change your vote, but only from approve to reject. Once rejected the request must be recreated.
As with the CLI workflow, after a request has been approved the original administrator will need to back and take that action a second time.
- Oct 29th, 2022 – Original Posting (with/without proofreading)
- Oct 30th, 2022 – Some sort of rudimentary proofreading