How to connect 1 million devices over Layer 3 IP-protocol

Question

I have edited the question after reading about MAC-to-MAC protocol (RMII/MII) here: https://community.nxp.com/thread/316374.

My IOT device looks like a USB stick. I am building a datacenter for 1 million IOT devices. Each datacenter is a large room with 100 server racks. Each server rack will contain 10,000 devices. I have full design control over the IOT device - MCU/MPU, firmware, and so on.

I need to design a network technology that will allow me to connect 1M IOT devices to a local network, so that I can address (select) each device and talk to it from a central application server, and each device can reply to the application server.

Here are some considerations:

1. Each device connects to a PCB board inside a 2U drawer inside the server rack. I can select any physical connector for this, like USB, or microSD (note that this is the connector - not the protocol). Each drawer can have maybe 400 devices (20x20). The full rack will contact perhaps 10 drawers.
2. The entire network will reside inside this datacenter. There is no need to connect to the outside world.
3. The IoT devices don't need to talk to each other. My requirement is only to be able to address (select) a device and talk it from an application server, and for each device to reply to the application server.
4. I do not have any specific bandwidth requirements. Bandwidth is not critical.
5. I need to keep the per-device cost low, so cannot put Ethernet on each device.

This is my idea:

I can skip Layer 1 (PHY), and use Layer-2 (MAC) and Layer-3 (IP) to create a network. I would put a TCP/IP stack into the IoT device and then connect 100 of them in each drawer, via PCB connections, to one or more Layer-2 switches implemented in an integrated circuit (ASIC or FPGA), using MAC-to-MAC protocol (RMII/MII). Then connects these switches from the drawers together among them, in the same way, so each server rack ends up being a small LAN.

Then I would connect all rack servers to another switch (or router?), and end up with one big LAN. Would this work?

I am OK with designing and building my own PCB circuits/FPGA for the switches, routers, etc , whatever is necessary.

Answer 1

I feel your approach is flawed by your 5th point: "I need to keep the per-device cost low, so cannot put Ethernet on each device."

Look at it versus "I'll reinvent a layer 1 protocol and all the needed hardware to handle it" and think if it really lower your overall cost.

BTW the main problem on a network is the collisions of informations from multiples devices talking to each others on the medium (copper here if I got it right), which get worse with the number of devices using the same 'link' to talk.

I.e when device 1 from drawer 4 in rack 7 try to talk to device 8 on drawer 5 of rack 21 you'll have somewhere a shared link between the drawers and racks, that's what ethernet switches aim to avoid, knowing on which port a specific MAC address reside avoid broadcasting the information on all ports and preventing other communication on those ports at the same time.

What ethernet also does is listening on the link to know if it can send data or not, that's the layer 2 part of ethernet. Sadly the fact ethernet has been a mix of layer 1 and layer 2 blur the lines between physical medium and the layer 2 communication.

If you want to use TCP/IP, you'll need a layer 2 protocol, reinventing one in place of ethernet would be costly at best.

So what I would do is keep the ethernet on the devices, use whatever you want as layer 1 protocol to exchange between the objects and the board and use standard ethernet cards/switches for the inter-board/inter-rack/app-to-object communications.

The conception costs will be mainly on the object and the board, the rest of network being well know technologies would probably be cheaper than reinventing one with all the caveats of massive communication (just the DHCP addressing for that much devices will be something to care about along with lease durations etc.).