I have edited the question after reading about MAC-to-MAC protocol (RMII/MII) here: https://community.nxp.com/thread/316374.

My IOT device looks like a USB stick. I am building a datacenter for 1 million IOT devices. Each datacenter is a large room with 100 server racks. Each server rack will contain 10,000 devices. I have full design control over the IOT device - MCU/MPU, firmware, and so on.

I need to design a network technology that will allow me to connect 1M IOT devices to a local network, so that I can address (select) each device and talk to it from a central application server, and each device can reply to the application server.

Here are some considerations:

  1. Each device connects to a PCB board inside a 2U drawer inside the server rack. I can select any physical connector for this, like USB, or microSD (note that this is the connector - not the protocol). Each drawer can have maybe 400 devices (20x20). The full rack will contact perhaps 10 drawers.
  2. The entire network will reside inside this datacenter. There is no need to connect to the outside world.
  3. The IoT devices don't need to talk to each other. My requirement is only to be able to address (select) a device and talk it from an application server, and for each device to reply to the application server.
  4. I do not have any specific bandwidth requirements. Bandwidth is not critical.
  5. I need to keep the per-device cost low, so cannot put Ethernet on each device.

This is my idea:

I can skip Layer 1 (PHY), and use Layer-2 (MAC) and Layer-3 (IP) to create a network. I would put a TCP/IP stack into the IoT device and then connect 100 of them in each drawer, via PCB connections, to one or more Layer-2 switches implemented in an integrated circuit (ASIC or FPGA), using MAC-to-MAC protocol (RMII/MII). Then connects these switches from the drawers together among them, in the same way, so each server rack ends up being a small LAN.

Then I would connect all rack servers to another switch (or router?), and end up with one big LAN. Would this work?

I am OK with designing and building my own PCB circuits/FPGA for the switches, routers, etc , whatever is necessary.

  • 3
    Being a consultant, I try and get the clients to focus on the desired end state. You mention that you don't want Ethernet on all million devices, but then talk about putting an TCP/IP stack on every device...so conflicting information. You talk about connecting a number of devices to a 'board' using something like USB, so why do you need TCP/IP at all?? Where is the data from the IoT devices going? What kind of data is coming from the IoT device? Is it data that can be aggregated or averaged so that you could 'proxy' sets of IoT sensors as just one data point?
    – JD Allen
    Jun 24, 2018 at 0:11
  • 1
    Keep in mind too, that 1 million devices would require at least 16 Class B subnets (or 1 Class A subnet, like 10.x.x.x/8) with a number of hubs/routers required to do this right.
    – JD Allen
    Jun 24, 2018 at 0:21
  • 1
    You can't just ignore Layer 1 & 2, you still need something for the IP to be carried over (especially if you have some sort of switch and want to do DHCP).
    – hardillb
    Jun 24, 2018 at 11:28
  • 1
    @JDAllen, TCP/IP does not require Ethernet (which is a layer 2 and 1. TCP/IP runs on anything (e.g. WiFi). So the information is not conflicting. As for the reference to USB, I clarified it is in reference to the size (as big as a USB stick) and to the physical connector (USB-connector, microSD connector, etc) - this is different from the USB-protocol (again, a layer 2/1 protocol). You did not understand the question correctly. Finally, I need TCP/IP because it is good network technology (for layer 3 and 4). You can suggest a better one if you know one. I think TCP/IP is good enough. Jun 24, 2018 at 18:45
  • 1
    @JDAllen as for the "16 Class B subnets" or "1 Class A subnet", I don't understand whether you mean that this is an issue. The network will be self-contained inside a building. Is this a problem? I am happy to use switches/routers as necessary - but not off-the-shelf. I prefer to use ICs built on PCB to provide switching/routing as necessary. This is the information I am looking for. Jun 24, 2018 at 18:47

1 Answer 1


I feel your approach is flawed by your 5th point: "I need to keep the per-device cost low, so cannot put Ethernet on each device."

Look at it versus "I'll reinvent a layer 1 protocol and all the needed hardware to handle it" and think if it really lower your overall cost.

BTW the main problem on a network is the collisions of informations from multiples devices talking to each others on the medium (copper here if I got it right), which get worse with the number of devices using the same 'link' to talk.

I.e when device 1 from drawer 4 in rack 7 try to talk to device 8 on drawer 5 of rack 21 you'll have somewhere a shared link between the drawers and racks, that's what ethernet switches aim to avoid, knowing on which port a specific MAC address reside avoid broadcasting the information on all ports and preventing other communication on those ports at the same time.

What ethernet also does is listening on the link to know if it can send data or not, that's the layer 2 part of ethernet. Sadly the fact ethernet has been a mix of layer 1 and layer 2 blur the lines between physical medium and the layer 2 communication.

If you want to use TCP/IP, you'll need a layer 2 protocol, reinventing one in place of ethernet would be costly at best.

So what I would do is keep the ethernet on the devices, use whatever you want as layer 1 protocol to exchange between the objects and the board and use standard ethernet cards/switches for the inter-board/inter-rack/app-to-object communications.

The conception costs will be mainly on the object and the board, the rest of network being well know technologies would probably be cheaper than reinventing one with all the caveats of massive communication (just the DHCP addressing for that much devices will be something to care about along with lease durations etc.).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.