IPv4 addresses are a problem, a big problem. We're running out, everyone is running out. Because of this, we need to be very careful with how we allocate them at least until IPv6 becomes commonplace. For a hosting provider like ourselves, this is especially true. Therefore as we grow we need to make some changes and adaptations to how we provide our networking capabilities.

One thing we need to do is add more IP addresses and ranges to the pool for Civo as well as revise how they are currently allocated and routed. The last part we are going to be starting work on in the New Year and this will mean some changes. As with our policy of transparency, we'd like to share with you what we are planning, how it will affect you and some detail on how and why.

TL;DR:

Currently you have a single private IP address on your instances. If you have a public IP for the instance, it is NATed (passed through) to your private IP. Going forward you will have one or two network interfaces on each instance. One entirely private-only looking like 10.x.x.x and one that starts with 172.x.x.x which will have a NATed public IP attached to it. This means you'll be able to configure your software to listen on the 10.x.x.x interface and it will always be 100% private. If you want it to listen publicly, you configure it to listen on the 172.x.x.x instance.

Basic Floating IP Mechanics within Civo

The IP addresses on your instances are allocated via DHCP. On Ubuntu for example, you can see this here:

/etc/network/interfaces.d/50-cloud-init.cfg

auto lo
iface lo inet loopback

auto ens3
iface ens3 inet dhcp # <- This line here

That means when your instance boots, it will try to find a DHCP server on the network and when it does, that server responds with an IP address allocation (as well as other network config) and a lease for that IP address. In Civo, this will be an IP address from your private subnet(s) (a /24 from 10.0.0.0/8). A router is then created to connect the public internet facing network to your private network. Next, we allocate a floating IP address and configure that to NAT through to your instance.

The following diagram illustrates this process:

Current.png

You will be able to see this if you run ifconfig or ip a on one of your instances. You will notice that your NICs do not have your public address on them and instead an address from your private range.

# ifconfig
ens3      Link encap:Ethernet  HWaddr aa:aa:aa:aa:aa:aa  
          inet addr:10.0.0.2  Bcast:10.0.0.255  Mask:255.255.255.128
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:5318 errors:0 dropped:0 overruns:0 frame:0
          TX packets:520 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:357100 (357.1 KB)  TX bytes:124116 (124.1 KB)

Why Does This Need to Happen?

This is due to the fact that the IPs need to be "floating", i.e. moveable from one host to another. For two instances on the internet (or any IP network) to be able to communicate with one another, they both must have unique IP addresses. If there is a conflict, it will prevent the machines involved from communicating with one another and any other device on the network. DHCP will tell the instances on the network which IP address they should have, but it then becomes the responsibility of the instance to set its IP and stay in communications with the DHCP server, renewing the lease when it expires. Leases are usually hours long and so instigating a change outside of the instance itself becomes problematic. One can allocate static IPs but this becomes tedious, is done from within the instnace itself and requires resetting and / or updating networking services and other management overhead.

This is where NAT (Network Address Translation) comes in. In a nutshell, what NAT does is takes packets destined for one address and tranparently re-routes them to another. This is how your router at home works. Each device on your home network will be allocated an address from a private range, then when you make a request to the internet, your router will receive communications on the private network and "translate" them onto the public network. Likewise for the response but in reverse. Otherwise each device on your network would need to have its own internet facing IP address.

Our use of NAT has a slightly different purpose. Let's say you have an instance running your web server and you have set up DNS to point your domain to that IP address. If you wanted to move the IP from that instance onto another, you would not be able do so if you had a statically allocated IP address within OpenStack (or most, if not all clouds for that matter). Static allocation is certainly possible, but if you delete your instance, you will lose your IP, most likely for good (it returns to a pool where anyone might be allocated it at a later time).

What is Going to Change?

Currently, each user has a router created for their account and all instances connect to one another and the outside world through that router.

Current multiple.png

What we plan to change this to is a setup where all instances that will be externally accessible, will have a NAT only network address (172.16.0.0/22 in the diagram) and at least one private address (10.0.0.0/24). Instances that are not to be externally accessible (such as a DB server for example) will only have a private address. Any instances configured to be private only, will never be externally accessible directly and instead will have to be accessed via another node that is externally accessible such as a jump box.

Going forward.png

Why Do This?

In short, to conserve and reclaim IP addresses whilst maintaining floating IP functionality and fully private networks for our customers. The current configuration works well and isn't particularly wasteful, but as we come out of beta and expand our business, we need to be more efficient in all areas.

Feedback

As you know, we are dedicated to being the number one developer cloud and so if you have any feedback, thoughts or suggestions we'd love to hear from you on this proposal via the chat window at the bottom right.