Developing with WATcloud

Here, we discuss setting up WATcloud to be used for software development in the Autonomous Software Division.

Why WATcloud? What is WATcloud?

Due to the high computational requirements of many aspects of the ASD stack, WATO has a large server infrastructure for remote development WATcloud (opens in a new tab). In this section, you will learn to connect to WATcloud on VS code. Connecting to a server to do remote development is not only a crucial aspect of software development at WATonomous, but is also a very common practice in the industry.

A Look from Afar

How does WATcloud share compute resources fairly?

WATcloud relies heavily on a resource management tool known as SLURM (opens in a new tab). SLURM ensures that all resources in WATcloud are shared in a fair and well-managed manner.

For the everyday developer, you can imagine SLURM as a "build your own computer" tool. You specify to SLURM what compute resources you want (CPU, RAM, GPU, memory, time limit, etc.) and SLURM will build a compute node with the resources it has on hand.

So how does remote development actually work?

Remote development for a WATonomous member typically consists of a local machine, host machine, SLURM node, and a docker container. They are defined below:

• Local Machine Your personal computer.
• Host Machine The computer you connect to. In the case of WATcloud, this is the SLURM login node.
• SLURM Node Used to manage compute resources. It creates SLURM Jobs according to your needs.
• SLURM Job An "imaginary computer" that is created by WATcloud. You specify to WATcloud what compute you need by running commands in the SLURM login node.
• Docker Container An isolated coding environment.

To do remote development in the Autonomous Software Division, the process can be summed up by the image below:

Job Scheduling vs Interactive Development

Use job scheduling when you want to run a command for a very long time (>1 day long). Use interactive development when you are actively making changes to your code and testing it.

For most WATonomous members, you would use job scheduling for tasks like training neural networks, large data processing, numerical optimization, etc. On the other hand, you would use interactive development when you are coding/testing ROS2 nodes, interacting with / visualizing live data, making code changes in general, etc.

Setting up WATcloud for ASD

Dealing with SSH can be quite foreign to alot of new developers. Thankfully, we provide a series of helper scripts that will make setup for WATcloud easier on you.

General Setup

This section is required so that you have proper access to our server cluster.

git clone [email protected]:WATonomous/wato_asd_tooling.git

touch ~/.ssh/config

cd wato_asd_tooling
bash ssh_helpers/generate_ssh_config.sh

ssh tr-ubuntu3
ssh derek3-ubuntu2
ssh delta-ubuntu2

cd wato_asd_tooling
bash ssh_helpers/configure_agent_forwarding.sh

Set-Service -Name ssh-agent -StartupType Automatic; Start-Service ssh-agent

ssh -T [email protected]

Setup for Interactive Development

Unlike job scheduling, SLURM was not built to handle interactive development. Luckily we have a team of very talented individuals, and we managed to make interactive development work nonetheless :).

Creating an interactive development environment entails starting an SSH server inside the SLURM node, some wacky SSH key sharing, a netcast proxycommand, as well as pointing docker to a persistent filesystem. You don't have to do that though. You just need to do the following.

One-time Setup

Follow these steps if you are setting up the SLURM dev sessions for the first time, or you were using past solutions for SLURM that WATO provided.

your_local_machine$ cat ~/.ssh/your_key.pub (copy the output of this)
your_local_machine$ ssh [tr-ubuntu3 or derek3-ubuntu2]
watcloud_slurm_node$ touch ~/.ssh/authorized_keys
watcloud_slurm_node$ nano ~/.ssh/authorized_keys (paste you rpublic key into here)

watcloud_slurm_node$ nano ~/.bashrc

if [[ "$(hostname)" == *"slurm"* ]]; then 
    export XDG_RUNTIME_DIR=${XDG_RUNTIME_DIR:-/tmp/run}
    export XDG_CONFIG_HOME=${XDG_CONFIG_HOME:-/tmp/config}
    export DOCKER_HOST="unix://${XDG_RUNTIME_DIR}/docker.sock"
fi

cd wato_asd_tooling
nano session_config.sh

cd wato_asd_tooling
bash start_interactive_session.sh

cd wato_asd_tooling
bash ssh_helpers/setup_slurm_ssh.sh

Starting a SLURM Dev Session Regularly

Once you have done the one-time setup, connecting to a SLURM Dev Session is easy.

cd wato_asd_tooling
nano session_config.sh

cd wato_asd_tooling
bash start_interactive_session.sh

Setup for Job Scheduling

There is no setup. Creating an SLURM job is really easy. It was what SLURM was designed for. You can view docs on SLURM in the WATcloud documentation (opens in a new tab).