Getting your cloud ready for the Hadean Platform
First time administrative setup
The Hadean Platform requires that an administrator for your cloud provider go through an automated one time setup, so that clusters created by the platform have the necessary permissions for dynamic scaling.
The advantage of this approach is that the permissions of your clusters are managed, and separate from your user permissions. For example, if a user with admin permissions deploys a cluster, that cluster does not then inherit admin permissions as well. It only gets the specific managed set of permissions we create here.
If you have not done this setup, either provisioning may fail (resulting in clusters appearing in a "Failed" status in the cluster list) or if provisioning succeeds, attempting to run any application on the cluster will fail with an "Out of resources" error message.
If you plan to self-manage your cloud resources or need to run on physical machines, cloud providers other than Azure or AWS, or on-premises VMs, then you do not need to follow the steps in this section. See the Configuring pre-provisioned infrastructure section.
You can find a readme that will explain how to run the scripts we provide, and the scripts for your cloud providers first time setup in the
~/.hadean/deployment/init directory.The readme's exact path is
~/.hadean/deployment/init/readme.md and can be viewed either as raw text, or with a markdown renderer.Ensure you have the
az CLI installed. Then, simply open PowerShell and run the script provided, specifying a location.~/.hadean/deployment/init/azure/deploy.ps1 uksouth
The location argument must be a valid Azure location name. You can find a list of these in our docs. This location specifies where the resource group for the resources we create will be deployed. You do not need to do this per location, and the location you choose will not impact the behaviour or the performance of the platform. Simply pick the location that you are likely to use the most, or whatever location is appropriate for your use of Azure.
The script automates the entire setup process. Once run, the following resources will exist:
- 1.
- 2.
- 3.
The managed identity can then be assigned by users to the cluster scheduler. The cluster scheduler will login using
az login --identity, and use managed service identity to create and destroy dynamic resources.If you need more restricted permissions, those must be created by your administrator as a role and assigned to the managed identity instead of the contributor role.
There are three steps to setting up AWS for use with the platform:
- 1.Deploy Policies
- 2.Assign User Policies
- 3.Hosted Zone Registration
You can deploy the AWS CloudFormation template that we provide either through the CLI or through the AWS Console.
To use the CLI, ensure you have the
aws CLI installed. Then, simply create a new stack from our template:aws cloudformation deploy --template-file ~/.hadean/deployment/init/aws/permissions.cloudformation.yaml --stack-name hadeanpermissions
The following resources are created:
- 1.A role that contains a policy and assume policy document called
HadeanPlatformClusterRolewhich clusters will use for their permissions. The policy in this role is tightly constrained to the specific permissions that the platform uses. - 2.An instance profile called
HadeanPlatformClusterProfilewhich has the role we just created assigned to it. - 3.A managed policy called
HadeanPlatformClusterUserPolicywhich when assigned to a user gives them the ability to associate the instance profile with an EC2 instance and pass its role to that instance.
Once you have deployed the CloudFormation template, you must assign the
HadeanPlatformClusterUserPolicy policy to the groups or users that are going to use the platform. This will give those users the ability to use the instance profile to setup the credentials for the scheduler, so that it can dynamically scale.The last step for AWS setup is to ensure you have a hosted zone in Route53 that your users can list and create Route53 records for. More information on this requirement is available here. This step is not optional.
We recommend setting up a hosted zone specifically for creating clusters within. This must be a public hosted zone that is internet resolvable for the Let's Encrypt ACME protocol to work.
Once created, you need to create a policy for your groups/users to be able to create record sets within that hosted zone.
- 1.Ensure that users can perform all read actions on the hosted zone:
GetHostedZone,GetHostedZoneCount,GetHostedZoneLimit,ListHostedZones,ListHostedZonesByName,ListHostedZonesByVPC. - 2.Ensure that users can perform the following write and read actions on record sets under that hosted zone:
ChangeResourceRecordSets,GetChange,ListResourceRecordSets.
You can restrict these permissions to the specific hosted zone through the
Resource part of the policy, by setting it to the ARN of the hosted zone you created.Once you are done, users can specify the hosted zone they want to create a cluster within by using the
--domain-name argument to the hadean cluster create command, specifying the name used by the hosted zone. For example, if you've registered hadean.com., then your users must use --domain-name hadean.com when creating clusters.