Hints, Tips, and Troubleshooting
Useful knowledge about using the platform
If you want to get debug logs, simply pass the
--debug flag to run. This will set RUST_LOG to debug. Right now, more specific logging configurations are not yet available, but stay tuned!The
hadean crate comes with docstrings based on the rustdoc format. As with any Rust crate, run cargo doc once you're inside the crate directory (e.g. ~/.hadean/sdk/crates/hadean) to compile the API documentation. The generated documentation can be found inside the target directory of the crate. Alternatively, you can use cargo doc --open to compile and open the documentation using your default browser.Currently, multiple arguments are passed to user binaries in remote clusters as a complete string, as one argument. In the future arguments will be provided as multiple strings and specified as an array in our configuration. For now, you must code against this in your application by either specifying the defaults you want in code, or by re-parsing the complete string provided.
When using the freemium version of the SDK, In some situations, when under load and
standby-machines is configured to 1 or more, it is possible that the dynamic backend will create additional machines (more than 4). However, these additional machines will not run hadean processes on them. Once scaled up, the stop command will take longer, since it will need to down-scale.This does not impact the licensed version of the platform, since that has no restriction on the number of machines in use.
When building, if you build with a different toolchain to the SDK's toolchain you will get an error such as:
= note: /home/.../.hadean/sdk/lib/libhadean_foreign.a(std-363511a59ca6fedb.std.2k3kczsj-cgu.0.rcgu.o): In function `__rdl_alloc':
(.text.__rdl_alloc+0x0): multiple definition of `__rdl_alloc'
/home/.../.rustup/toolchains/nightly-2021-06-07-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-363511a59ca6fedb.rlib(std-363511a59ca6fedb.std.2efebab1c8fd4763-cgu.0.rcgu.o):/rustc/35fff69d043b1c0f5c29894e7f4b0da8b039c131//library/std/
src/alloc.rs:350: first defined here
/home/.../.hadean/sdk/lib/libhadean_foreign.a(std-363511a59ca6fedb.std.2k3kczsj-cgu.0.rcgu.o): In function `__rdl_alloc_zeroed':
(.text.__rdl_alloc_zeroed+0x0): multiple definition of `__rdl_alloc_zeroed'
/home/.../.rustup/toolchains/nightly-2021-06-07-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-363511a59ca6fedb.rlib(std-363511a59ca6fedb.std.2efebab1c8fd4763-cgu.0.rcgu.o):/rustc/35fff69d043b1c0f5c29894e7f4b0da8b039c131//library/std/
src/alloc.rs:382: first defined here
Similar errors can also be provoked by running with the wrong version of GLIBC. This is because the build you produce is dynamically linked and the version of GLIBC on the remote machine must match the version you linked with locally.
Currently, the supported version of GLIBC is
Ubuntu GLIBC 2.31-0ubuntu9.2 and you can check what version you have using ldd --version.If you find when running locally that you get an error such as:
hadean run ~/hadean/examples/rust/hello/target/release/hello
[INFO ] Starting application...
[ERROR] [W][2022-01-25T13:26:06+0000][2486] pid_t subproc::runChild(nsjconf_t*, int, int, int, int)():486 Received error message from the child process before it has been executed
[ERROR] [F][2022-01-25T13:26:06+0000][2] pid_t subproc::runChild(nsjconf_t*, int, int, int, int)():466 Launching child process failed
[ERROR] [E][2022-01-25T13:26:06+0000][2486] int nsjail::standaloneMode(nsjconf_t*)():272 Couldn't launch the child process
[INFO ] trying to connect 1. time
[INFO ] trying to connect 2. time
[INFO ] trying to connect 3. time
[INFO ] trying to connect 4. time
[INFO ] trying to connect 5. time
[ERROR] error while handling a request or response: Channel(Io(Os { code: 111, kind: ConnectionRefused, message: "Connection refused" })), None, Please contact Hadean and provide your log file.
Then you may not have
fuse-overlayfs installed. On an install of Ubuntu 20.04 this is normally installed, however, the WSL distribution of Ubuntu 20.04 is missing this package. Simply install it:sudo apt update
sudo apt install fuse-overlayfs -y
If you are using the platform from WSL, then you may find that DNS can stop working. This causes all internet services to fail to resolve from your WSL instance, while Windows will still have a functional internet connection. For example, if you run the command
ping www.google.com from WSL, you won't get a successful response.This can be fixed in a number of ways. The most permanent fix is to overwrite the nameservers that WSL uses.
For example, you could set your nameservers to the Cloud Flare nameservers. First, close any open work you have in WSL. Then, open a powershell terminal. Finally, run the following script:
$Distro = "Ubuntu-20.04"
wsl --terminate $Distro
wsl -d $Distro -- bash -c "echo $'[network]\ngenerateResolvConf = false' | sudo tee /etc/wsl.conf"
wsl --terminate $Distro
wsl -d $Distro -- bash -c "sudo rm -rf /etc/resolv.conf; echo $'nameserver 1.1.1.1\nnameserver 1.0.0.1' | sudo tee /etc/resolv.conf"
wsl --terminate $Distro
wsl -d $Distro -- bash -c "sudo chattr +i /etc/resolv.conf"
This will prevent WSL from regenerating its nameserver configuration and overwrite the nameserver configuration with the Cloud Flare DNS IP addresses. You could also pick any other nameserver you prefer, including your own ISPs nameservers. Simply replace the IP addresses in the script with the primary and secondary nameservers you prefer.
WSL has been known to sometimes have its clock run out of sync with the system clock. If this happens, some operations on WSL will stop working. You can fix this with the command ntpdate command:
sudo ntpdate pool.ntp.org
This message is printed just as the dynamic backend begins scaling. Nothing has gone wrong, and the application should scale up in the next 5 to 10 minutes. In future versions we will suppress this message. Please see Deploying to the cloud for a play-by-play of the runtime logs for an example.
When running the create command, you may get this error message:
[2021-09-13T13:36:55Z ERROR] terraform plan failed:
Error: building AzureRM Client: obtain subscription() from Azure CLI: Error parsing json result from the Azure CLI: Error waiting for the Azure CLI: exit status 1: ERROR: Please run 'az login' to setup account.
with provider["registry.terraform.io/hashicorp/azurerm"],
on versions.tf line 16, in provider "azurerm":
16: provider "azurerm" {
This can be fixed by setting up credentials with your cloud providers CLI as described in Prerequisites. In order to deploy to the cloud you need to be logged into your account so the resources can be set up using your subscription.
When you run your application for the first time we need to create a backend storage account in Azure that terraform will store its state in.
This error happens when the backend storage account we created for terraform in Azure or AWS was not ready yet when terraform started to use it.
Simply repeat the command, running the application again and you won't see this error the second time around.
gzip: stdin: unexpected end of file tar: Child returned status 1 tar: Error is not recoverable: exiting now
This is due to the command expiring as it only has a lifetime of 10 minutes. To obtain a new download command simply refresh the page.
You can find us on our Discord channel if you run into any problems or just want to discuss the Hadean Platform SDK!