You may have seen an earlier guide: Workaround for instances with networking bug on Ubuntu 18.04 in relation to a networking bug on 18.04 causing intermittent connectivity issues. This guide is similar but an updated version that allows the use of the true private networking by disabling Netplan altogether. This guide walks you through the steps required in order to replace Netplan with the older
ifupdown package on Ubuntu Bionic Beaver (18.04).
If you’re not already a Civo user and want to try this out safely in the cloud first. Civo offers $70 free credit to new users if accepted into the KUBE100 beta: Apply here. This will be enough to run a Medium-sized 3 node cluster.
IMPORTANT: IF YOU HAVE CONFIGURED THE VM TO BE ACCESSIBLE VIA SSH KEY ONLY (NO PASSWORD ACCESS), IF YOU MAKE A MISTAKE IN THE CONFIGURATION THAT CAUSES AN ISSUE WITH THE NETWORKING, WE WILL BE UNABLE TO ACCESS THE MACHINE... PERMANENTLY. WHILST SETTING PASSWORDS FOR SSH IS OFTEN INADVISABLE, IN THIS CASE IT IS HIGHLY RECOMMENDED IN ORDER TO ENSURE ACCESS TO THE VM VIA THE CONSOLE, SHOULD A MISCONFIGURATION OCCUR. PLEASE ENSURE YOU FOLLOW THE STEPS CAREFULLY AND CHECK / TEST WHERE INDICATED TO MAKE SURE YOU DO NOT LOSE YOUR DATA.
If you would like us to do this for you, please get in touch and we will run through these steps. We will however need access to the machine in order to do so.
If at all possible, we would recommend one or more of the following before applying the workaround:
- Backup your data.
- Migrate data onto another Civo instance (not using 18.04).
- Mount a volume to the instance following this guide and move your data onto it.
- Snapshot the VM.
If you have not rebooted your machine and / or still have SSH access, you can skip this step and move onto Step 2.
If you have already rebooted and are no longer able to access the VM continue reading Step 1 (this step).
If you set up your machine with a password, you should be good to move onto Step 4. If you only have SSH key access to the machine, get in touch with us and we will try and update your NAT address to point to the other interface therefore allowing access. Once you have done this, you can move onto Step 3.
There are two paths you can take, depending on whether your instance was configured to use an SSH key at creation or a password. If you created your instance with an SSH key, move onto Step 3. If you created your instance with a password move onto Step 4.
YOU DO NOT HAVE TO PERFORM THIS STEP. HOWEVER, IF YOU MAKE A MISTAKE IN THE NETWORK CONFIGURATION THAT PREVENTS SSH ACCESS, YOU WILL NOT BE ABLE TO RECOVER THE MACHINE. EVER.
Log into your instance via SSH. Then create a new user
sudo adduser civorescue
When prompted, set a password for the user. Ensure you set a strong password:
- Use a combination of letters (upper and lower case), numbers and symbols.
- It should be at least 8 characters long (if not longer).
- Avoid using a single dictionary word like "workaround", even if you replace letters with numbers or symbols such as "W0rk4r0und".
- Ensure that you keep it somewhere safe. A password manager like 1Password, LastPass, KeePassX etc is ideal.
- Ensure you use a password that you have not used elsewhere in case another site is compromised and malicious users gain access to your password.
- Even if you do not wish to keep the password after setting it in this guide, you must still ALWAYS use strong passwords.
Enter new UNIX password: Retype new UNIX password:
For each of the following, hit return:
Enter the new value, or press ENTER for the default Full Name : Room Number : Work Phone : Home Phone : Other :
Y when asked if the information is correct:
Is the information correct? [Y/n] Y
Add the user to the sudo group
sudo usermod -aG sudo civorescue
Now move onto Step 4.
This step is important. Do not assume console access works. It is well worth the effort to confirm you have console access now, rather than be sorry later.
Navigate to the instance in the Civo UI and locate the console button (computer screen icon) from the action buttons on the top right of the instance. This will launch a spice terminal that will prompt you for a username and password.
If successful, also confirm that your user has
sudo access by running:
This should prompt you for your password and then switch to the
root user. Exit the
root user with:
If this all worked, then we are good to move onto Step 5.
Find the IP address of the
This should output something similar to:
ens3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450 inet 10.0.0.2 netmask 255.255.255.0 broadcast 10.0.0.255 ether 00:00:00:00:00:00 txqueuelen 1000 (Ethernet) RX packets 5 bytes 1224 (1.2 KB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 14 bytes 1564 (1.5 KB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
The part you are interested in is
inet 10.0.0.2, which is the address you need for the next step. In this case it is
10.0.0.2. Yours will likely be different but will be an address from the
Move onto Step 6
Remove the incorrect gateway, replacing
<GATEWAY_IP> with the address from the previous step:
sudo route del -net 0.0.0.0 gw <GATEWAY_IP> netmask 0.0.0.0 dev ens3
Using the address from our example, this should look something like this:
sudo route del -net 0.0.0.0 gw 10.0.0.2 netmask 0.0.0.0 dev ens3
If you are accessing the VM via the console, then this should have resumed SSH connectivity which will make things easier for you as you can now use normal terminal functionality such as copy and paste. Close the console and SSH into the VM if you haven't already. Then move onto Step 7.
Next install the
sudo apt-get update sudo apt-get install -y ifupdown
Delete all of the Netplan configuration files:
sudo rm -rf /etc/netplan/*.yml
Open the file
/etc/network/interfaces and copy and paste the following:
auto lo iface lo inet loopback auto ens3 iface ens3 inet dhcp auto ens4 iface ens4 inet dhcp up ip route change default via 172.31.255.254 source /etc/network/interfaces.d/*.cfg
You can now reboot the instance.
Now move onto Step 10.
Rebooting the instance should hopefully have worked and you should still have SSH access to the machine. You can confirm this by SSHing into the instance. Once in the instance, running:
Should now only show one default route:
default via 172.31.255.254 dev ens4 proto dhcp metric 100 ...
Move onto Step 11 only if you normally access the VM with an SSH key and you created a username and password in Step 3 that you wish to disable.
IF YOU REMOVE THE ABILITY TO LOG IN WITH A PASSWORD AND DO NOT HAVE AN SSH KEY OR OTHERWISE RELY ON PASSWORD ACCESS VIA SSH, YOU WILL NOT BE ABLE TO ACCESS THE VM AND ANY ACCESS SSH, CONSOLE OR OTHERWISE WILL BE LOST AND UNRECOVERABLE AS WILL ANY DATA ON THE VM. DO NOT RUN THIS STEP ON ANY USER THAT YOU NEED TO LOG IN WITH USING A PASSWORD.
From the VM run:
passwd civorescue --lock
This will prepend a ! to the encrypted password in
/etc/shadow meaning it will result in a value that will match no possible encrypted value.