Using 400G QSFP cables with Tenstorrent P150 hardware

Tenstorrent blackhole officially supports only 800G QSFP cables. As these are 3x the price than 400G cables, and I don’t expect high traffic over the link with sensible sharding when doing tensor-parallel inference, I figured I just get some cheap 10gtek 400G cables.

Getting this to work with tt-metal / ttnn requires a little bit of work. tt-metal has a hard-coded list of supported topologies in x_mesh_graph_descriptor.yaml files, and it uses various heuristics to pick which one to use. Each file lists how devices are connected, including the number of ethernet links used.

When using a QSFP-DD 800G cable, the cards establish 4 ethernet links, while 400G cable has it establishing only 2. This means that the selected mesh graph would declare more links than actually exist, and fail.

RuntimeError: TT_FATAL @ /home/yiding/projects/tt/tt-metal/tt_metal/fabric/fabric_host_utils.cpp:313: topology_info.corners.size() == 2
info:
Expected 2 corners for 1D mesh, got 0.

We can fix this by using a custom mesh graph descriptor. Here’s an example for 2 p150 connected by a single 400g cable. The notable change here is changing ChipSpec.ethernet_ports from 4 to 2:

# p150_x2_400g_mesh_graph_descriptor.yaml
ChipSpec:
  arch: blackhole
  ethernet_ports:
    N: 2
    E: 2
    S: 2
    W: 2

Board:
  - name: P150
    type: Mesh
    topology: [1, 2]

Mesh:
  - id: 0
    board: P150
    device_topology: [1, 2]
    host_topology: [1, 1]
    host_ranks: [[0]]

Graph: []

Then set the environment variable TT_MESH_GRAPH_DESC_PATH to point to that path of that file to force tt-metal to use it.