That's not entirely true. While Rackspace does provide a shared private network for intra-DC communication, it also provides the Cloud Networks product that is capable of creating tenant specific networks. Think VLAN tagging for Cloud.
On that private network, you can use your own addressing, use multi-cast, etc. Much less limited and more secure than a shared private network. It's also free.
The last time I looked in the Rackspace docs, it looked like this was in the process of being rolled out ("production ready but will be available to customers in a phased release"). Is Cloud Networks considered fully supported now?
RackConnect is a product that allows us to link cloud servers in our public Cloud environment with servers in a dedicated configuration.
We are currently using RackConnect 2.0 which achieves this by attaching the shared private network to the dedicated environment and configuring the cloud servers network stacks to use the dedicated load balancer and firewall as their default gateway, so that all traffic flows through the dedicated config. Incoming traffic (or traffic from the dedicated configuration) will be routed out to the Cloud servers by the dedicated load balancer.
RackConnect 3.0 (coming soon) will provide the same service, but the connection from the cloud servers to the dedicated configuration will be provided by Cloud Networks, our SDN product. This simplifies the configuration and provides additional security to the traffic.
>Most other providers do not restrict private network either
EC2 has security groups, and the default group would be that non-tenant machines could not access your services (this is external to your image, at presumably the hypervisor or networking level. What you do in ufw/iptables is above and beyond this). I don't see any similar mechanism in the Digital Ocean world.
AWS supports VPC (virtual private network). It lets you setup sub groups of VMs that are only network accessible to each other with explicit endpoints open (ex: just HTTP open to an ELB). It's recommended for all new deployments. We use it in our cloud deployment of JackDB and it's really pleasant to use. Plus it makes it really easy to setup a bastion SSH proxy as an end point (vs. having all your instances publicly accessible).
Most other providers do not restrict private network either, I'm talking about the big ones like RackSpace, Amazon and others.
What you're talking about is dedicated private VLAN's or private subnet's and that is not common especially in a cloud environment.