> For the complete documentation index, see [llms.txt](https://csp.gitbook.io/genai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://csp.gitbook.io/genai/foss/tyle.md).

# TYLE

## TYLE Format Specification — v0.1 (draft)

**TYLE** — *Tyled Inference Layout Encoding* A CPU-native, single-file model format for local LLM inference. GPU is a supported-but-secondary consumer via load-time repack.

File extension: `.tyle` Magic bytes: `TYLE` (0x54 0x49 0x4C 0x45) Endianness: little-endian throughout.

> Status: v0.2 draft. The LUT and AMX layouts below are verified against source — T-MAC `preprocess_weights` (`python/t_mac/weights.py`) and llama.cpp AMX `mmq.cpp`/`common.h`. Byte-exact packing is captured in §4.3–4.4. Three v0.1 assumptions were corrected on inspection (TYLE\_K=32 not 64; per-quant-type packing; tuning-dependent LUT layout). The container, metadata, and section framing are stable enough to build a loader against.

***

### Design thesis

Every layout fork that GGUF resolves in favor of GPU-neutrality, TYLE resolves in favor of the CPU. Weights are stored in the shape the CPU kernel consumes — no dequant-then-multiply repack, no load-time tiling pass. GPU support is an optional repack performed during the host→device copy that already has to happen.

Non-goals: GPU-optimal throughput, training, fine-tuning in-place.

***

### 1. Container structure

```
┌─────────────────────────────┐
│ Header (fixed 64 bytes)     │
├─────────────────────────────┤
│ Metadata block (KV, varlen) │
├─────────────────────────────┤
│ Tensor directory            │
├─────────────────────────────┤
│ Schedule hints block        │
├─────────────────────────────┤
│ NUMA shard map (optional)   │
├─────────────────────────────┤
│ Tensor data                 │  ← laid out in execution order
│   (mmap-aligned, tyled)     │
└─────────────────────────────┘
```

All section offsets are absolute from file start. Tensor data begins on a page boundary (4096-byte aligned) so `mmap` + page-cache prefetch is clean.

***

### 2. Header (64 bytes, fixed)

| Offset | Size | Field                | Notes              |
| ------ | ---- | -------------------- | ------------------ |
| 0      | 4    | magic                | `TYLE`             |
| 4      | 2    | version\_major       | `0`                |
| 6      | 2    | version\_minor       | `1`                |
| 8      | 8    | metadata\_offset     |                    |
| 16     | 8    | tensor\_dir\_offset  |                    |
| 24     | 8    | schedule\_offset     | 0 if absent        |
| 32     | 8    | numa\_map\_offset    | 0 if absent        |
| 40     | 8    | tensor\_data\_offset | page-aligned       |
| 48     | 4    | tensor\_count        |                    |
| 52     | 4    | flags                | bitfield, see §2.1 |
| 56     | 8    | reserved             | must be 0          |

#### 2.1 Header flags bitfield

| Bit  | Name              | Meaning                                       |
| ---- | ----------------- | --------------------------------------------- |
| 0    | EXEC\_ORDER       | tensor data is laid out in forward-pass order |
| 1    | HAS\_LUT          | file contains LUT-encoded tensors             |
| 2    | HAS\_AMX\_TYLES   | file contains AMX-tyle-shaped tensors         |
| 3    | HAS\_SPARSITY     | activation-sparsity metadata present          |
| 4    | NUMA\_SHARDED     | NUMA shard map present and populated          |
| 5    | GPU\_REPACK\_HINT | per-tensor GPU offload hints present          |
| 6–31 | reserved          | must be 0                                     |

***

### 3. Metadata block (key-value)

Self-describing KV store, GGUF-style — this part is deliberately unoriginal because it works.

```
u32   kv_count
repeat kv_count:
  u16  key_len
  u8   key[key_len]        # UTF-8
  u8   value_type          # enum below
  ...  value               # type-dependent
```

**value\_type enum:** `0 u8, 1 i8, 2 u16, 3 i16, 4 u32, 5 i32, 6 u64, 7 i64, 8 f32, 9 f64, 10 bool, 11 string (u32 len + bytes), 12 array (u8 elem_type + u32 count + elems)`

**Required keys (v0.1):**

| Key                     | Type           | Notes                                                                    |
| ----------------------- | -------------- | ------------------------------------------------------------------------ |
| `tile.arch`             | string         | model architecture, e.g. `llama`, `qwen`                                 |
| `tile.name`             | string         | model display name                                                       |
| `tile.context_length`   | u32            |                                                                          |
| `tile.embedding_length` | u32            |                                                                          |
| `tile.block_count`      | u32            | number of transformer blocks                                             |
| `tile.head_count`       | u32            |                                                                          |
| `tile.head_count_kv`    | u32            |                                                                          |
| `tile.target_isa`       | array\[string] | e.g. `["avx512_vnni","amx_int8"]` — ISAs this file's layout is tuned for |
| `tile.tokenizer.model`  | string         | tokenizer family                                                         |
| `tile.tokenizer.tokens` | array\[string] | vocab                                                                    |
| `tile.tokenizer.merges` | array\[string] | if BPE                                                                   |

Tokenizer + config live *inside* the file. Single-file self-containment is retained from GGUF — that was never the problem.

***

### 4. Tensor directory

One entry per tensor. Fixed-size entries for fast scan, with variable name pool appended.

```
repeat tensor_count:
  u32   name_offset        # into name pool
  u32   name_len
  u8    n_dims
  u64   dims[n_dims]       # logical shape
  u8    encoding           # §4.1
  u8    quant_bits         # 1,2,3,4,5,6,8 ; 0 = not quantized (f16/f32/bf16)
  u8    tile_layout        # §4.2
  u8    reserved
  u64   data_offset        # from tensor_data_offset, page-aware
  u64   data_len_bytes
  u32   exec_order_index   # position in forward pass (0-based)
  u32   gpu_offload_score  # 0=keep on CPU … 255=strongly prefer GPU
```

#### 4.1 encoding enum

| Val | Name      | Meaning                                                                           |
| --- | --------- | --------------------------------------------------------------------------------- |
| 0   | RAW       | f32 / f16 / bf16, no quant                                                        |
| 1   | GROUPWISE | per-group scale+zero, GGUF-style Q\_K analogue                                    |
| 2   | LUT       | T-MAC-style: bit-plane-major weight indices into a runtime-built LUT (§4.3)       |
| 3   | CODEBOOK  | non-uniform / centroid codebook                                                   |
| 4   | AMX\_INT8 | int8 weights pre-packed into 16×32 (TILE\_N×TILE\_K) tiles, per-quant-type (§4.4) |

#### 4.2 tile\_layout enum

Describes the physical blocking of the data so the loader knows the kernel can consume it directly.

| Val | Name          | Meaning                                                             |
| --- | ------------- | ------------------------------------------------------------------- |
| 0   | LINEAR        | row-major, no special tiling                                        |
| 1   | SIMD\_BLOCKED | blocked to a SIMD register width (see `tile.target_isa`)            |
| 2   | AMX\_TILE     | 16×32 (TILE\_N×TILE\_K) VNNI-packed B-matrix, per-quant-type (§4.4) |
| 3   | LUT\_INDEXED  | bit-plane index planes, sign-folded (§4.3)                          |

#### 4.3 LUT encoding detail (T-MAC `preprocess_weights`, verified against source)

Source: `python/t_mac/weights.py::preprocess_weights`. This is the offline transform; TILE stores its **output** so no preprocessing happens at load. Verified facts (defaults in parens):

**Input convention.** Weights arrive as `uint8` shape `(M, K)`, where the original signed int1/2/3/4 value has had a bias of `2^(bits-1)` added to make it unsigned (int2: −2,−1,0,1 → 0,1,2,3). Store the biased-uint convention explicitly — a loader must know weights are pre-biased, not two's-complement.

**The transform, in order:**

1. **Bit-plane split.** Each weight is split into `bits` planes: `w = stack([(w >> ib) & 1 for ib in range(bits)])`. Confirms the earlier claim — storage is bit-plane-major, one plane per bit.
2. **LUT-group fold.** Planes are reshaped to `(M//bits, bits, K//g, g)` and the `g` weights in each group are folded into a single index via `sum((w[...,ig] << ig))`. So each stored element is a `g`-bit LUT index, **not** a raw weight.
3. **SIMD interleave.** A multi-stage reshape/transpose interleaves along `simd_n_in` (16) and `simd_n_out` (8) and packs `ngroups_per_elem = 8//g` groups into each byte via `sum((w[...,ng] << (ng*g)))`. Final logical shape `(M//bm, K//g, bm//ngroups_per_elem)`.

**Parameters to store per-tensor** (these are the real names/defaults, replacing the guessed ones):

| Field            | Default | Meaning                                  |
| ---------------- | ------- | ---------------------------------------- |
| `lut.bits`       | 4       | 1/2/3/4                                  |
| `lut.g`          | 4       | LUT group size (weights per table index) |
| `lut.bm`         | 512     | tuned M-tiling block                     |
| `lut.kfactor`    | 16      | tuned K-tiling block                     |
| `lut.simd_n_in`  | 16      | inner SIMD width                         |
| `lut.simd_n_out` | 8       | outer SIMD width (fp16 lane count)       |
| `lut.group_size` | K/ncols | quant scale group size                   |
| `lut.has_zeros`  | bool    | asymmetric (zeros present) vs symmetric  |

**Scales/zeros** are permuted in parallel to shape `(M//bm, K//group_size, -1)`; if `zeros` present they're stacked with scales. Store both alongside the index planes.

**Correction to v0.1:** `bm`, `kfactor`, `simd_n_in/out` are *tuning-dependent* — the packed layout is only valid for the tile shape it was compiled for. This means a `.tile` LUT tensor is **tile-shape-specific**, so the header must record the tuning params and a loader must reject a mismatch (or the format ships one canonical tuning per ISA class). This is a real constraint the v0.1 draft missed.

> Still runtime-built, not stored: the LUT table itself (activation-dependent, `tbl`/`pshuf` at inference).

#### 4.4 AMX tile encoding detail (llama.cpp `mmq.cpp`, verified against source)

Source: `ggml/src/ggml-cpu/amx/{common.h,mmq.cpp}`. **Three corrections to v0.1:**

1. **TILE\_K is 32, not 64.** `common.h`: `TILE_M 16, TILE_N 16, TILE_K 32, VNNI_BLK 4, AMX_BLK_SIZE 32`. The 64 figure was the raw hardware register byte-width; llama.cpp's *software* tile is 16×32 because it feeds AVX-512 VNNI (`_mm512_dpbusd_epi32`) as well as TMUL, and VNNI works in blocks of 4 int8 (`VNNI_BLK 4`). Store `tile_k = 32`.
2. **Packing is per-quant-type, not one universal int8 tile.** `pack_B` is specialized for `q4_0, q4_1, q8_0, q4_K, q5_K, q6_K, iq4_xs`. Each has its own `packed_B` layout. TILE's `AMX_INT8` encoding therefore needs a `amx.source_quant` field naming which block format was packed.
3. **The exact `packed_B` layout is documented in the source comments.** For `q4_K`:

   ```
   quants {8, TILE_N, 16}  uint8scales {8, TILE_N}      uint8mins   {8, TILE_N}      uint8d      {TILE_N}         ggml_half
   ```

   For `q8_0`: `quants {TILE_N, TILE_K} int8`, then `d0 {TILE_N} ggml_half`, then an `int32` **compensation** term per column (`s8s8_compensation`, computed via `_mm512_dpbusd_epi32` against a 0x80 offset vector — this handles the signed×signed → unsigned×signed conversion VNNI requires).

**The driver:** `convert_B_packed_format(packed_B, B, N, K)` tiles over `NB = N/TILE_N` and `KB = K/BLOCK_K`, calling `pack_B` per (n,k) tile with weights transposed so `B[n0*KB + k]` lands column-major within the tile. Weights are stored **column-major within each 16-wide N tile** — this transpose is the repack TILE eliminates by storing post-transpose.

**Store per-tensor:** `amx.tile_n=16`, `amx.tile_k=32`, `amx.vnni_blk=4`, `amx.source_quant` (enum), `amx.has_compensation` (bool — true for s8s8 types like q8\_0).

> The `s8s8_compensation` int32 column is required data, not derivable at load without recomputation — TILE stores it inline so the kernel skips the `s8s8_compensation()` call entirely.

***

### 5. Schedule hints block (optional)

Removes per-machine autotune cold-start. Advisory — a runtime may ignore it.

```
u32   entry_count
repeat entry_count:
  u16  machine_class_len
  u8   machine_class[..]   # e.g. "avx512_8c", "amx_xeon_sp", "neon_rpi5"
  u8   recommended_threads
  u8   tile_m              # gemm tile dims
  u8   tile_n
  u8   tile_k
  u8   flags               # bit0: prefer LUT path; bit1: enable fast-aggregation
```

***

### 6. NUMA shard map (optional)

The current llama.cpp NUMA story is BIOS hacks and forks. TILE expresses sharding in-file so a multi-socket loader can map the right bytes to the right node without duplication guesswork.

```
u32   socket_count
u32   sharded_tensor_count
repeat sharded_tensor_count:
  u32  tensor_index        # into tensor directory
  u8   shard_axis          # 0=row, 1=col
  repeat socket_count:
    u64  shard_offset      # from tensor_data_offset
    u64  shard_len_bytes
```

A single-socket loader ignores this block and reads each tensor whole via its directory entry. A NUMA-aware loader mmaps each shard on its owning node.

***

### 7. Tensor data

* Begins page-aligned; each tensor's `data_offset` chosen so hot tensors don't straddle pages badly.
* When `EXEC_ORDER` is set, tensors appear in forward-pass order → sequential page-cache prefetch during a forward pass.
* LUT payloads follow §4.3 (bit-plane index arrays; table is runtime-built, not stored). AMX\_TILE payloads follow §4.4 (16×32 VNNI-packed B-matrix tiles, layout per source quant type). Both load with zero repack — that is the point of the format.

***

### 8. GPU-as-nice-to-have contract

* No GPU-packed duplicate is stored. `gpu_offload_score` + `GPU_REPACK_HINT` tell the runtime which tensors are worth repacking onto the device during the host→device copy that already occurs.
* The format promises: best-in-class CPU inference, and it *uses* your GPU if present. It does **not** promise GPU-optimal throughput — that is explicitly out of scope.

***

### 9. Open items before v0.2

1. ✅ **Resolved.** LUT layout verified against `preprocess_weights` — bit-plane split → g-bit LUT-index fold → SIMD interleave, output shape `(M//bm, K//g, bm//ngroups_per_elem)`. See §4.3.
2. ✅ **Resolved.** AMX layout verified against `mmq.cpp` — per-quant-type `packed_B` (quants + scales + mins + d + optional s8s8 compensation), TILE\_N=16, TILE\_K=32. See §4.4.
3. **New constraint (from §4.3):** LUT packed layout is *tuning-specific* (`bm`, `kfactor`, `simd_n_in/out`). Decide: ship one canonical tuning per ISA class, or store tuning params in-header and let loaders reject mismatches. This is the biggest open design question now.
4. **New constraint (from §4.4):** AMX packing is per-source-quant-type, not universal. Either enumerate supported source quants or define a canonical TILE int8 tile and convert at pack time.
5. Decide single-layout (runtime repack for GPU) vs. optional dual-layout escape hatch. v0.1 assumes single-layout.
6. Activation-sparsity metadata block — reserved via `HAS_SPARSITY` flag, not yet specified.
7. Alignment policy for straddle-avoidance under 2MB huge pages.
8. Versioning/compat rules for adding new `encoding` / `tile_layout` values without breaking loaders.

### 10. Strategic note

The kernel wins (LUT, AMX) are already *demonstrated* by T-MAC and llama.cpp — this de-risks the performance thesis but means the moat is not novelty. It is being the first **native, zero-repack, single-file** container that unifies these proven CPU kernels without the repack tax, the toolchain fragility (AMX needs clang + AVX512\_VNNI, is Linux-only in practice), and the BIOS-hack NUMA story. This is a packaging-and-coherence play. Adoption will therefore hinge on shipping (or being adopted by) a runtime people use for reasons *other* than the format — the same dynamic that carried GGUF on llama.cpp's back.

***

The README gives the LUT algorithm clearly, which is enough to define the on-disk layout. The key technical facts that constrain your `LUT_INDEXED` encoding:

The T-MAC method: group one-bit weights (e.g., into groups of 4), precompute all possible partial sums, and store them in a LUT; use shift-and-accumulate to scale bits from 1 to 4; use tbl/pshuf instructions for the lookup; and reduce table size from 2^n to 2^(n-1) using a sign bit. Two structural consequences for your format:

1. **Weights are decomposed into bit-planes, not stored as packed integers.** T-MAC processes int1/2/3/4 by treating each bit position separately and shift-accumulating. So `LUT_INDEXED` storage is *bit-plane-major*: for a b-bit weight, b separate index planes, each addressing a LUT. This is fundamentally different from GGUF's packed-nibble Q4 layout — which is exactly why it can't be consumed without repack.
2. **The LUT itself is activation-dependent, computed at runtime — not stored.** The table holds partial sums of *activations* over the 2^(g-1) sign-folded combinations of a g-weight group. Activations aren't known until inference, so the LUT is built per-forward-pass. What you store on disk is the **weight→group-index mapping** (which precomputed-sum slot each weight group points to), plus the group size `g` and the sign metadata. The table is transient.

One important caveat before you finalize: T-MAC doesn't define its own container — it rides GGUF, converting HF models into an i2 GGUF variant via a patched convert script with a kcfg.ini kernel config, using group size 128 and activation group size 64 in the example (`-gs 128 -ags 64`). So there's no "T-MAC file format" to copy; there's a packing convention layered onto GGUF plus a separate kernel-config file. Your opportunity is real — you'd be first to make it a native on-disk layout — but it means the exact byte packing lives in their TVM-generated kernels, not in a spec document. To pin it precisely you'd need to read the `convert-hf-to-gguf-t-mac.py` packing and the generated kernel's expected tensor shape.

That's more than v0.1 needs. Here's what I can lock into the spec now with confidence:

* `LUT_INDEXED` = bit-plane-major storage: `quant_bits` planes, each an index array into a runtime-built LUT
* Store `lut_group_size` (weights per LUT group, e.g. 4) and `act_group_size` (e.g. 64) as per-tensor metadata
* Store a `sign_folded` flag (the 2^n → 2^(n-1) optimization)
* The LUT table is explicitly *not* stored — mark it runtime-constructed

Let me get the AMX tile shape, then I'll update the spec with both TBDs closed to this level of precision.That's enough to close the AMX TBD precisely. The Intel/ONNX documentation gives the exact TMUL tile constraints, and the llama.cpp discussion confirms the practical limit.

The hard constraints on `AMX_TILE` layout, from Intel's own TMUL definition: TMUL computes C\[M]\[N] += A\[M]\[K] \* B\[K]\[N]. A tile has 1–16 rows and 1–MAX\_TILE\_K columns; B tile has 1–MAX\_TILE\_K rows and 1–16 columns; C tile is up to 16×16. MAX\_TILE\_K = 64/sizeof(type), so MAX\_TILE\_K = 64 for int8 and 32 for bf16. And the physical register limit is confirmed independently: the amx\_tile instruction is limited to a 16×64-byte tile size.

So the layout facts your spec must encode:

* **int8 weight tiles are 16×64** (rows × cols), bf16 tiles are **16×32**. This is fixed by hardware, not a tunable.
* **The B matrix (weights) must be pre-packed** in a specific interleaved order so `tile_loadd` streams it directly into a TMM register. llama.cpp/ONNX pack weights into a "PackedBType" with TILE\_K-strided blocks; the kernel loop iterates `k -= TILE_K` over pre-packed A and B tiles. That pre-pack is exactly the repack tax you're eliminating by storing weights already in this shape.
* One important reality check from the same discussion: even Intel gives little info on TMUL and no longer maintains the demo code, and llama.cpp's AMX path doesn't do optimal large-block tiled int8 — it's constrained to the 16×64 register tile. So storing AMX-native tiles is a genuine, unfilled gap, but the ecosystem support is thin, which cuts both ways: less competition, but also fewer reference implementations to validate against.

Let me update the spec to close both TBDs to this precision.Now let me add the two new detail subsections and rewrite the tensor-data and open-items sections.Both TBDs are now closed to the algorithmic/geometric level — what remains is byte-exact packing verification, which needs reading actual source, not more searching.

What got pinned down:

**LUT (§4.3)** — The key structural insight is that T-MAC stores weights as **bit-plane index arrays**, not packed integers, and the lookup table itself is *never stored* — it's built per-forward-pass from activations. So your format stores the weight→slot indices plus group-size/sign-fold metadata, and marks the table runtime-constructed. This is genuinely incompatible with GGUF's packed-nibble layout, which is exactly why a native format is justified. The standout property: latency scales linearly with bit-width, so 2-bit and 1-bit get progressively faster — something dequant-based kernels can't do.

**AMX (§4.4)** — Hardware-fixed, not tunable: int8 tiles are **16×64**, bf16 are **16×32**, because `MAX_TILE_K = 64/sizeof(type)`. Weights store as the pre-packed B-matrix in K-strided tile order that streams straight into a TMM register. The repack you're eliminating is precisely the pack step llama.cpp/ONNX do at load.

Two honest findings worth keeping in view:

* **The AMX ecosystem is thin.** Intel no longer maintains reference TMUL demo code, and llama.cpp's AMX path is constrained to the 16×64 register tile with no optimized large-block tiling. Less competition, but fewer implementations to validate against — I flagged this in §4.4.
* **The moat is packaging, not novelty** (new §10). The kernels are proven, which de-risks your performance claims but means TILE wins by being the first coherent zero-repack container, not by inventing anything. Adoption still hinges on the runtime-gravity problem from earlier.

The two `[VERIFY]` items — LUT index-plane byte order and the AMX B-matrix interleave — both require reading source (`convert-hf-to-gguf-t-mac.py` and llama.cpp's `ggml-cpu/amx/` pack code).&#x20;

***

## TYLE Format Specification — v0.1 (draft)

**TYLE** — *Tyled Inference Layout Encoding* A CPU-native, single-file model format for local LLM inference. GPU is a supported-but-secondary consumer via load-time repack.

File extension: `.tyle` Magic bytes: `TYLE` (0x54 0x49 0x4C 0x45) Endianness: little-endian throughout.

> Status: v0.2 draft. The LUT and AMX layouts below are verified against source — T-MAC `preprocess_weights` (`python/t_mac/weights.py`) and llama.cpp AMX `mmq.cpp`/`common.h`. Byte-exact packing is captured in §4.3–4.4. Three v0.1 assumptions were corrected on inspection (TYLE\_K=32 not 64; per-quant-type packing; tuning-dependent LUT layout). The container, metadata, and section framing are stable enough to build a loader against.

***

### Design thesis

Every layout fork that GGUF resolves in favor of GPU-neutrality, TYLE resolves in favor of the CPU. Weights are stored in the shape the CPU kernel consumes — no dequant-then-multiply repack, no load-time tiling pass. GPU support is an optional repack performed during the host→device copy that already has to happen.

Non-goals: GPU-optimal throughput, training, fine-tuning in-place.

***

### 1. Container structure

```
┌─────────────────────────────┐
│ Header (fixed 64 bytes)     │
├─────────────────────────────┤
│ Metadata block (KV, varlen) │
├─────────────────────────────┤
│ Tensor directory            │
├─────────────────────────────┤
│ Schedule hints block        │
├─────────────────────────────┤
│ NUMA shard map (optional)   │
├─────────────────────────────┤
│ Tensor data                 │  ← laid out in execution order
│   (mmap-aligned, tyled)     │
└─────────────────────────────┘
```

All section offsets are absolute from file start. Tensor data begins on a page boundary (4096-byte aligned) so `mmap` + page-cache prefetch is clean.

***

### 2. Header (64 bytes, fixed)

| Offset | Size | Field                | Notes              |
| ------ | ---- | -------------------- | ------------------ |
| 0      | 4    | magic                | `TYLE`             |
| 4      | 2    | version\_major       | `0`                |
| 6      | 2    | version\_minor       | `1`                |
| 8      | 8    | metadata\_offset     |                    |
| 16     | 8    | tensor\_dir\_offset  |                    |
| 24     | 8    | schedule\_offset     | 0 if absent        |
| 32     | 8    | numa\_map\_offset    | 0 if absent        |
| 40     | 8    | tensor\_data\_offset | page-aligned       |
| 48     | 4    | tensor\_count        |                    |
| 52     | 4    | flags                | bitfield, see §2.1 |
| 56     | 8    | reserved             | must be 0          |

#### 2.1 Header flags bitfield

| Bit  | Name              | Meaning                                       |
| ---- | ----------------- | --------------------------------------------- |
| 0    | EXEC\_ORDER       | tensor data is laid out in forward-pass order |
| 1    | HAS\_LUT          | file contains LUT-encoded tensors             |
| 2    | HAS\_AMX\_TYLES   | file contains AMX-tyle-shaped tensors         |
| 3    | HAS\_SPARSITY     | activation-sparsity metadata present          |
| 4    | NUMA\_SHARDED     | NUMA shard map present and populated          |
| 5    | GPU\_REPACK\_HINT | per-tensor GPU offload hints present          |
| 6–31 | reserved          | must be 0                                     |

***

### 3. Metadata block (key-value)

Self-describing KV store, GGUF-style — this part is deliberately unoriginal because it works.

```
u32   kv_count
repeat kv_count:
  u16  key_len
  u8   key[key_len]        # UTF-8
  u8   value_type          # enum below
  ...  value               # type-dependent
```

**value\_type enum:** `0 u8, 1 i8, 2 u16, 3 i16, 4 u32, 5 i32, 6 u64, 7 i64, 8 f32, 9 f64, 10 bool, 11 string (u32 len + bytes), 12 array (u8 elem_type + u32 count + elems)`

**Required keys (v0.1):**

| Key                     | Type           | Notes                                                                    |
| ----------------------- | -------------- | ------------------------------------------------------------------------ |
| `tyle.arch`             | string         | model architecture, e.g. `llama`, `qwen`                                 |
| `tyle.name`             | string         | model display name                                                       |
| `tyle.context_length`   | u32            |                                                                          |
| `tyle.embedding_length` | u32            |                                                                          |
| `tyle.block_count`      | u32            | number of transformer blocks                                             |
| `tyle.head_count`       | u32            |                                                                          |
| `tyle.head_count_kv`    | u32            |                                                                          |
| `tyle.target_isa`       | array\[string] | e.g. `["avx512_vnni","amx_int8"]` — ISAs this file's layout is tuned for |
| `tyle.tokenizer.model`  | string         | tokenizer family                                                         |
| `tyle.tokenizer.tokens` | array\[string] | vocab                                                                    |
| `tyle.tokenizer.merges` | array\[string] | if BPE                                                                   |

Tokenizer + config live *inside* the file. Single-file self-containment is retained from GGUF — that was never the problem.

***

### 4. Tensor directory

One entry per tensor. Fixed-size entries for fast scan, with variable name pool appended.

```
repeat tensor_count:
  u32   name_offset        # into name pool
  u32   name_len
  u8    n_dims
  u64   dims[n_dims]       # logical shape
  u8    encoding           # §4.1
  u8    quant_bits         # 1,2,3,4,5,6,8 ; 0 = not quantized (f16/f32/bf16)
  u8    tyle_layout        # §4.2
  u8    reserved
  u64   data_offset        # from tensor_data_offset, page-aware
  u64   data_len_bytes
  u32   exec_order_index   # position in forward pass (0-based)
  u32   gpu_offload_score  # 0=keep on CPU … 255=strongly prefer GPU
```

#### 4.1 encoding enum

| Val | Name      | Meaning                                                                           |
| --- | --------- | --------------------------------------------------------------------------------- |
| 0   | RAW       | f32 / f16 / bf16, no quant                                                        |
| 1   | GROUPWISE | per-group scale+zero, GGUF-style Q\_K analogue                                    |
| 2   | LUT       | T-MAC-style: bit-plane-major weight indices into a runtime-built LUT (§4.3)       |
| 3   | CODEBOOK  | non-uniform / centroid codebook                                                   |
| 4   | AMX\_INT8 | int8 weights pre-packed into 16×32 (TILE\_N×TILE\_K) tiles, per-quant-type (§4.4) |

#### 4.2 tile\_layout enum

Describes the physical blocking of the data so the loader knows the kernel can consume it directly.

| Val | Name          | Meaning                                                             |
| --- | ------------- | ------------------------------------------------------------------- |
| 0   | LINEAR        | row-major, no special tiling                                        |
| 1   | SIMD\_BLOCKED | blocked to a SIMD register width (see `tile.target_isa`)            |
| 2   | AMX\_TILE     | 16×32 (TILE\_N×TILE\_K) VNNI-packed B-matrix, per-quant-type (§4.4) |
| 3   | LUT\_INDEXED  | bit-plane index planes, sign-folded (§4.3)                          |

#### 4.3 LUT encoding detail (T-MAC `preprocess_weights`, verified against source)

Source: `python/t_mac/weights.py::preprocess_weights`. This is the offline transform; TILE stores its **output** so no preprocessing happens at load. Verified facts (defaults in parens):

**Input convention.** Weights arrive as `uint8` shape `(M, K)`, where the original signed int1/2/3/4 value has had a bias of `2^(bits-1)` added to make it unsigned (int2: −2,−1,0,1 → 0,1,2,3). Store the biased-uint convention explicitly — a loader must know weights are pre-biased, not two's-complement.

**The transform, in order:**

1. **Bit-plane split.** Each weight is split into `bits` planes: `w = stack([(w >> ib) & 1 for ib in range(bits)])`. Confirms the earlier claim — storage is bit-plane-major, one plane per bit.
2. **LUT-group fold.** Planes are reshaped to `(M//bits, bits, K//g, g)` and the `g` weights in each group are folded into a single index via `sum((w[...,ig] << ig))`. So each stored element is a `g`-bit LUT index, **not** a raw weight.
3. **SIMD interleave.** A multi-stage reshape/transpose interleaves along `simd_n_in` (16) and `simd_n_out` (8) and packs `ngroups_per_elem = 8//g` groups into each byte via `sum((w[...,ng] << (ng*g)))`. Final logical shape `(M//bm, K//g, bm//ngroups_per_elem)`.

**Parameters to store per-tensor** (these are the real names/defaults, replacing the guessed ones):

| Field            | Default | Meaning                                  |
| ---------------- | ------- | ---------------------------------------- |
| `lut.bits`       | 4       | 1/2/3/4                                  |
| `lut.g`          | 4       | LUT group size (weights per table index) |
| `lut.bm`         | 512     | tuned M-tiling block                     |
| `lut.kfactor`    | 16      | tuned K-tiling block                     |
| `lut.simd_n_in`  | 16      | inner SIMD width                         |
| `lut.simd_n_out` | 8       | outer SIMD width (fp16 lane count)       |
| `lut.group_size` | K/ncols | quant scale group size                   |
| `lut.has_zeros`  | bool    | asymmetric (zeros present) vs symmetric  |

**Scales/zeros** are permuted in parallel to shape `(M//bm, K//group_size, -1)`; if `zeros` present they're stacked with scales. Store both alongside the index planes.

**Correction to v0.1:** `bm`, `kfactor`, `simd_n_in/out` are *tuning-dependent* — the packed layout is only valid for the tile shape it was compiled for. This means a `.tile` LUT tensor is **tile-shape-specific**, so the header must record the tuning params and a loader must reject a mismatch (or the format ships one canonical tuning per ISA class). This is a real constraint the v0.1 draft missed.

> Still runtime-built, not stored: the LUT table itself (activation-dependent, `tbl`/`pshuf` at inference).

#### 4.4 AMX tile encoding detail (llama.cpp `mmq.cpp`, verified against source)

Source: `ggml/src/ggml-cpu/amx/{common.h,mmq.cpp}`. **Three corrections to v0.1:**

1. **TILE\_K is 32, not 64.** `common.h`: `TILE_M 16, TILE_N 16, TILE_K 32, VNNI_BLK 4, AMX_BLK_SIZE 32`. The 64 figure was the raw hardware register byte-width; llama.cpp's *software* tile is 16×32 because it feeds AVX-512 VNNI (`_mm512_dpbusd_epi32`) as well as TMUL, and VNNI works in blocks of 4 int8 (`VNNI_BLK 4`). Store `tile_k = 32`.
2. **Packing is per-quant-type, not one universal int8 tile.** `pack_B` is specialized for `q4_0, q4_1, q8_0, q4_K, q5_K, q6_K, iq4_xs`. Each has its own `packed_B` layout. TILE's `AMX_INT8` encoding therefore needs a `amx.source_quant` field naming which block format was packed.
3. **The exact `packed_B` layout is documented in the source comments.** For `q4_K`:

   ```
   quants {8, TILE_N, 16}  uint8scales {8, TILE_N}      uint8mins   {8, TILE_N}      uint8d      {TILE_N}         ggml_half
   ```

   For `q8_0`: `quants {TILE_N, TILE_K} int8`, then `d0 {TILE_N} ggml_half`, then an `int32` **compensation** term per column (`s8s8_compensation`, computed via `_mm512_dpbusd_epi32` against a 0x80 offset vector — this handles the signed×signed → unsigned×signed conversion VNNI requires).

**The driver:** `convert_B_packed_format(packed_B, B, N, K)` tiles over `NB = N/TILE_N` and `KB = K/BLOCK_K`, calling `pack_B` per (n,k) tile with weights transposed so `B[n0*KB + k]` lands column-major within the tile. Weights are stored **column-major within each 16-wide N tile** — this transpose is the repack TILE eliminates by storing post-transpose.

**Store per-tensor:** `amx.tile_n=16`, `amx.tile_k=32`, `amx.vnni_blk=4`, `amx.source_quant` (enum), `amx.has_compensation` (bool — true for s8s8 types like q8\_0).

> The `s8s8_compensation` int32 column is required data, not derivable at load without recomputation — TILE stores it inline so the kernel skips the `s8s8_compensation()` call entirely.

***

### 5. Schedule hints block (optional)

Removes per-machine autotune cold-start. Advisory — a runtime may ignore it.

```
u32   entry_count
repeat entry_count:
  u16  machine_class_len
  u8   machine_class[..]   # e.g. "avx512_8c", "amx_xeon_sp", "neon_rpi5"
  u8   recommended_threads
  u8   tile_m              # gemm tile dims
  u8   tile_n
  u8   tile_k
  u8   flags               # bit0: prefer LUT path; bit1: enable fast-aggregation
```

***

### 6. NUMA shard map (optional)

The current llama.cpp NUMA story is BIOS hacks and forks. TILE expresses sharding in-file so a multi-socket loader can map the right bytes to the right node without duplication guesswork.

```
u32   socket_count
u32   sharded_tensor_count
repeat sharded_tensor_count:
  u32  tensor_index        # into tensor directory
  u8   shard_axis          # 0=row, 1=col
  repeat socket_count:
    u64  shard_offset      # from tensor_data_offset
    u64  shard_len_bytes
```

A single-socket loader ignores this block and reads each tensor whole via its directory entry. A NUMA-aware loader mmaps each shard on its owning node.

***

### 7. Tensor data

* Begins page-aligned; each tensor's `data_offset` chosen so hot tensors don't straddle pages badly.
* When `EXEC_ORDER` is set, tensors appear in forward-pass order → sequential page-cache prefetch during a forward pass.
* LUT payloads follow §4.3 (bit-plane index arrays; table is runtime-built, not stored). AMX\_TILE payloads follow §4.4 (16×32 VNNI-packed B-matrix tiles, layout per source quant type). Both load with zero repack — that is the point of the format.

***

### 8. GPU-as-nice-to-have contract

* No GPU-packed duplicate is stored. `gpu_offload_score` + `GPU_REPACK_HINT` tell the runtime which tensors are worth repacking onto the device during the host→device copy that already occurs.
* The format promises: best-in-class CPU inference, and it *uses* your GPU if present. It does **not** promise GPU-optimal throughput — that is explicitly out of scope.

***

### 9. Open items before v0.2

1. ✅ **Resolved.** LUT layout verified against `preprocess_weights` — bit-plane split → g-bit LUT-index fold → SIMD interleave, output shape `(M//bm, K//g, bm//ngroups_per_elem)`. See §4.3.
2. ✅ **Resolved.** AMX layout verified against `mmq.cpp` — per-quant-type `packed_B` (quants + scales + mins + d + optional s8s8 compensation), TILE\_N=16, TILE\_K=32. See §4.4.
3. **New constraint (from §4.3):** LUT packed layout is *tuning-specific* (`bm`, `kfactor`, `simd_n_in/out`). Decide: ship one canonical tuning per ISA class, or store tuning params in-header and let loaders reject mismatches. This is the biggest open design question now.
4. **New constraint (from §4.4):** AMX packing is per-source-quant-type, not universal. Either enumerate supported source quants or define a canonical TILE int8 tile and convert at pack time.
5. Decide single-layout (runtime repack for GPU) vs. optional dual-layout escape hatch. v0.1 assumes single-layout.
6. Activation-sparsity metadata block — reserved via `HAS_SPARSITY` flag, not yet specified.
7. Alignment policy for straddle-avoidance under 2MB huge pages.
8. Versioning/compat rules for adding new `encoding` / `tile_layout` values without breaking loaders.

### 10. Strategic note

The kernel wins (LUT, AMX) are already *demonstrated* by T-MAC and llama.cpp — this de-risks the performance thesis but means the moat is not novelty. It is being the first **native, zero-repack, single-file** container that unifies these proven CPU kernels without the repack tax, the toolchain fragility (AMX needs clang + AVX512\_VNNI, is Linux-only in practice), and the BIOS-hack NUMA story. This is a packaging-and-coherence play. Adoption will therefore hinge on shipping (or being adopted by) a runtime people use for reasons *other* than the format — the same dynamic that carried GGUF on llama.cpp's back.

***

I'll pull the actual packing code. Let me clone both repos and read the LUT conversion and AMX pack source directly.Got it. Let me read the AMX tile constants and the pack function.Important correction to the spec. llama.cpp uses `TILE_K 32`, not 64, and `VNNI_BLK 4`. Let me read the actual pack function to understand why and get the exact interleave.This is the real packing logic. Let me read the core `pack_B`, `pack_qs`, and `convert_B_packed_format` functions to nail the exact layout.This is exactly the ground truth — the code comments spell out the packed\_B layout. Let me capture the `convert_B_packed_format` driver and confirm the full picture.I have the ground truth for AMX. Now let me grab the T-MAC conversion side. The T-MAC repo is heavy (TVM submodule), so I'll fetch just the convert script and kernel config rather than clone recursively.That's the exact T-MAC weight transform. Let me read the full function with its parameter docs to capture the layout precisely.I now have both packing layouts from source. Both TBDs are fully closed, and there are three corrections to the spec's earlier assumptions. Let me update it.Now I'll rewrite §4.3 and §4.4 with the exact source-derived layouts and fix the three wrong assumptions.Now fix the enum lines and status that still reference the wrong tile\_k=64, and update open items.Both TBDs are closed against actual source, and reading the code corrected three assumptions the earlier draft got wrong from documentation alone.

**LUT (from `t_mac/weights.py::preprocess_weights`)** — The real transform is bit-plane split → fold `g` weights into one `g`-bit LUT index → a multi-stage SIMD interleave, final shape `(M//bm, K//g, bm//ngroups_per_elem)`. The real parameter names and defaults are now in the spec (`bits=4, g=4, bm=512, kfactor=16, simd_n_in=16, simd_n_out=8`). Also captured: weights are stored **pre-biased to unsigned** (add `2^(bits-1)`), which a loader must know.

**AMX (from `mmq.cpp` + `common.h`)** — The source comments literally document each `packed_B` layout, so this is exact, not inferred.

The three corrections matter:

1. **TILE\_K is 32, not 64.** The 64 I gave you earlier was the raw hardware register width; llama.cpp's software tile is 16×32 because it also feeds AVX-512 VNNI, which works in blocks of 4 int8. Getting this wrong would have made every stored tile the wrong shape.
2. **Packing is per-quant-type**, not one universal int8 tile — `q4_0, q8_0, q4_K, q5_K, q6_K, iq4_xs` each have distinct layouts. So `AMX_INT8` needs a `source_quant` field.
3. **The LUT layout is tuning-specific** (`bm`, `kfactor`). This is the important one for your design: a `.tile` LUT tensor is only valid for the tile shape it was compiled for, so you either ship one canonical tuning per ISA class or record tuning params in-header and reject mismatches. That's now flagged as the biggest open design question.

This is exactly the kind of thing that only surfaces by reading code — documentation would have let all three errors through into a v0.2 that produced unloadable files.

The spec is at v0.2 with a clean open-items list. The two new constraints (LUT tuning-specificity, AMX per-quant-type) are design decisions, not research gaps — you can settle them yourself. Natural next step is the reference loader: a Python reader that parses the header, tensor directory, and metadata, and correctly interprets an LUT and an AMX tensor per §4.3–4.4. That would validate the container design against real byte layouts.&#x20;

***

## TILE Format Specification — v0.1 (draft)

**TILE** — *Tiled Inference Layout Encoding* A CPU-native, single-file model format for local LLM inference. GPU is a supported-but-secondary consumer via load-time repack.

File extension: `.tile` Magic bytes: `TILE` (0x54 0x49 0x4C 0x45) Endianness: little-endian throughout.

> Status: v0.2 draft. The LUT and AMX layouts below are verified against source — T-MAC `preprocess_weights` (`python/t_mac/weights.py`) and llama.cpp AMX `mmq.cpp`/`common.h`. Byte-exact packing is captured in §4.3–4.4. Three v0.1 assumptions were corrected on inspection (TILE\_K=32 not 64; per-quant-type packing; tuning-dependent LUT layout). The container, metadata, and section framing are stable enough to build a loader against.

***

### Design thesis

Every layout fork that GGUF resolves in favor of GPU-neutrality, TILE resolves in favor of the CPU. Weights are stored in the shape the CPU kernel consumes — no dequant-then-multiply repack, no load-time tiling pass. GPU support is an optional repack performed during the host→device copy that already has to happen.

Non-goals: GPU-optimal throughput, training, fine-tuning in-place.

***

### 1. Container structure

```
┌─────────────────────────────┐
│ Header (fixed 64 bytes)     │
├─────────────────────────────┤
│ Metadata block (KV, varlen) │
├─────────────────────────────┤
│ Tensor directory            │
├─────────────────────────────┤
│ Schedule hints block        │
├─────────────────────────────┤
│ NUMA shard map (optional)   │
├─────────────────────────────┤
│ Tensor data                 │  ← laid out in execution order
│   (mmap-aligned, tiled)     │
└─────────────────────────────┘
```

All section offsets are absolute from file start. Tensor data begins on a page boundary (4096-byte aligned) so `mmap` + page-cache prefetch is clean.

***

### 2. Header (64 bytes, fixed)

| Offset | Size | Field                | Notes              |
| ------ | ---- | -------------------- | ------------------ |
| 0      | 4    | magic                | `TILE`             |
| 4      | 2    | version\_major       | `0`                |
| 6      | 2    | version\_minor       | `1`                |
| 8      | 8    | metadata\_offset     |                    |
| 16     | 8    | tensor\_dir\_offset  |                    |
| 24     | 8    | schedule\_offset     | 0 if absent        |
| 32     | 8    | numa\_map\_offset    | 0 if absent        |
| 40     | 8    | tensor\_data\_offset | page-aligned       |
| 48     | 4    | tensor\_count        |                    |
| 52     | 4    | flags                | bitfield, see §2.1 |
| 56     | 8    | reserved             | must be 0          |

#### 2.1 Header flags bitfield

| Bit  | Name              | Meaning                                       |
| ---- | ----------------- | --------------------------------------------- |
| 0    | EXEC\_ORDER       | tensor data is laid out in forward-pass order |
| 1    | HAS\_LUT          | file contains LUT-encoded tensors             |
| 2    | HAS\_AMX\_TILES   | file contains AMX-tile-shaped tensors         |
| 3    | HAS\_SPARSITY     | activation-sparsity metadata present          |
| 4    | NUMA\_SHARDED     | NUMA shard map present and populated          |
| 5    | GPU\_REPACK\_HINT | per-tensor GPU offload hints present          |
| 6–31 | reserved          | must be 0                                     |

***

### 3. Metadata block (key-value)

Self-describing KV store, GGUF-style — this part is deliberately unoriginal because it works.

```
u32   kv_count
repeat kv_count:
  u16  key_len
  u8   key[key_len]        # UTF-8
  u8   value_type          # enum below
  ...  value               # type-dependent
```

**value\_type enum:** `0 u8, 1 i8, 2 u16, 3 i16, 4 u32, 5 i32, 6 u64, 7 i64, 8 f32, 9 f64, 10 bool, 11 string (u32 len + bytes), 12 array (u8 elem_type + u32 count + elems)`

**Required keys (v0.1):**

| Key                     | Type           | Notes                                                                    |
| ----------------------- | -------------- | ------------------------------------------------------------------------ |
| `tile.arch`             | string         | model architecture, e.g. `llama`, `qwen`                                 |
| `tile.name`             | string         | model display name                                                       |
| `tile.context_length`   | u32            |                                                                          |
| `tile.embedding_length` | u32            |                                                                          |
| `tile.block_count`      | u32            | number of transformer blocks                                             |
| `tile.head_count`       | u32            |                                                                          |
| `tile.head_count_kv`    | u32            |                                                                          |
| `tile.target_isa`       | array\[string] | e.g. `["avx512_vnni","amx_int8"]` — ISAs this file's layout is tuned for |
| `tile.tokenizer.model`  | string         | tokenizer family                                                         |
| `tile.tokenizer.tokens` | array\[string] | vocab                                                                    |
| `tile.tokenizer.merges` | array\[string] | if BPE                                                                   |

Tokenizer + config live *inside* the file. Single-file self-containment is retained from GGUF — that was never the problem.

***

### 4. Tensor directory

One entry per tensor. Fixed-size entries for fast scan, with variable name pool appended.

```
repeat tensor_count:
  u32   name_offset        # into name pool
  u32   name_len
  u8    n_dims
  u64   dims[n_dims]       # logical shape
  u8    encoding           # §4.1
  u8    quant_bits         # 1,2,3,4,5,6,8 ; 0 = not quantized (f16/f32/bf16)
  u8    tile_layout        # §4.2
  u8    reserved
  u64   data_offset        # from tensor_data_offset, page-aware
  u64   data_len_bytes
  u32   exec_order_index   # position in forward pass (0-based)
  u32   gpu_offload_score  # 0=keep on CPU … 255=strongly prefer GPU
```

#### 4.1 encoding enum

| Val | Name      | Meaning                                                                           |
| --- | --------- | --------------------------------------------------------------------------------- |
| 0   | RAW       | f32 / f16 / bf16, no quant                                                        |
| 1   | GROUPWISE | per-group scale+zero, GGUF-style Q\_K analogue                                    |
| 2   | LUT       | T-MAC-style: bit-plane-major weight indices into a runtime-built LUT (§4.3)       |
| 3   | CODEBOOK  | non-uniform / centroid codebook                                                   |
| 4   | AMX\_INT8 | int8 weights pre-packed into 16×32 (TILE\_N×TILE\_K) tiles, per-quant-type (§4.4) |

#### 4.2 tile\_layout enum

Describes the physical blocking of the data so the loader knows the kernel can consume it directly.

| Val | Name          | Meaning                                                             |
| --- | ------------- | ------------------------------------------------------------------- |
| 0   | LINEAR        | row-major, no special tiling                                        |
| 1   | SIMD\_BLOCKED | blocked to a SIMD register width (see `tile.target_isa`)            |
| 2   | AMX\_TILE     | 16×32 (TILE\_N×TILE\_K) VNNI-packed B-matrix, per-quant-type (§4.4) |
| 3   | LUT\_INDEXED  | bit-plane index planes, sign-folded (§4.3)                          |

#### 4.3 LUT encoding detail (T-MAC `preprocess_weights`, verified against source)

Source: `python/t_mac/weights.py::preprocess_weights`. This is the offline transform; TILE stores its **output** so no preprocessing happens at load. Verified facts (defaults in parens):

**Input convention.** Weights arrive as `uint8` shape `(M, K)`, where the original signed int1/2/3/4 value has had a bias of `2^(bits-1)` added to make it unsigned (int2: −2,−1,0,1 → 0,1,2,3). Store the biased-uint convention explicitly — a loader must know weights are pre-biased, not two's-complement.

**The transform, in order:**

1. **Bit-plane split.** Each weight is split into `bits` planes: `w = stack([(w >> ib) & 1 for ib in range(bits)])`. Confirms the earlier claim — storage is bit-plane-major, one plane per bit.
2. **LUT-group fold.** Planes are reshaped to `(M//bits, bits, K//g, g)` and the `g` weights in each group are folded into a single index via `sum((w[...,ig] << ig))`. So each stored element is a `g`-bit LUT index, **not** a raw weight.
3. **SIMD interleave.** A multi-stage reshape/transpose interleaves along `simd_n_in` (16) and `simd_n_out` (8) and packs `ngroups_per_elem = 8//g` groups into each byte via `sum((w[...,ng] << (ng*g)))`. Final logical shape `(M//bm, K//g, bm//ngroups_per_elem)`.

**Parameters to store per-tensor** (these are the real names/defaults, replacing the guessed ones):

| Field            | Default | Meaning                                  |
| ---------------- | ------- | ---------------------------------------- |
| `lut.bits`       | 4       | 1/2/3/4                                  |
| `lut.g`          | 4       | LUT group size (weights per table index) |
| `lut.bm`         | 512     | tuned M-tiling block                     |
| `lut.kfactor`    | 16      | tuned K-tiling block                     |
| `lut.simd_n_in`  | 16      | inner SIMD width                         |
| `lut.simd_n_out` | 8       | outer SIMD width (fp16 lane count)       |
| `lut.group_size` | K/ncols | quant scale group size                   |
| `lut.has_zeros`  | bool    | asymmetric (zeros present) vs symmetric  |

**Scales/zeros** are permuted in parallel to shape `(M//bm, K//group_size, -1)`; if `zeros` present they're stacked with scales. Store both alongside the index planes.

**Correction to v0.1:** `bm`, `kfactor`, `simd_n_in/out` are *tuning-dependent* — the packed layout is only valid for the tile shape it was compiled for. This means a `.tile` LUT tensor is **tile-shape-specific**, so the header must record the tuning params and a loader must reject a mismatch (or the format ships one canonical tuning per ISA class). This is a real constraint the v0.1 draft missed.

> Still runtime-built, not stored: the LUT table itself (activation-dependent, `tbl`/`pshuf` at inference).

#### 4.4 AMX tile encoding detail (llama.cpp `mmq.cpp`, verified against source)

Source: `ggml/src/ggml-cpu/amx/{common.h,mmq.cpp}`. **Three corrections to v0.1:**

1. **TILE\_K is 32, not 64.** `common.h`: `TILE_M 16, TILE_N 16, TILE_K 32, VNNI_BLK 4, AMX_BLK_SIZE 32`. The 64 figure was the raw hardware register byte-width; llama.cpp's *software* tile is 16×32 because it feeds AVX-512 VNNI (`_mm512_dpbusd_epi32`) as well as TMUL, and VNNI works in blocks of 4 int8 (`VNNI_BLK 4`). Store `tile_k = 32`.
2. **Packing is per-quant-type, not one universal int8 tile.** `pack_B` is specialized for `q4_0, q4_1, q8_0, q4_K, q5_K, q6_K, iq4_xs`. Each has its own `packed_B` layout. TILE's `AMX_INT8` encoding therefore needs a `amx.source_quant` field naming which block format was packed.
3. **The exact `packed_B` layout is documented in the source comments.** For `q4_K`:

   ```
   quants {8, TILE_N, 16}  uint8scales {8, TILE_N}      uint8mins   {8, TILE_N}      uint8d      {TILE_N}         ggml_half
   ```

   For `q8_0`: `quants {TILE_N, TILE_K} int8`, then `d0 {TILE_N} ggml_half`, then an `int32` **compensation** term per column (`s8s8_compensation`, computed via `_mm512_dpbusd_epi32` against a 0x80 offset vector — this handles the signed×signed → unsigned×signed conversion VNNI requires).

**The driver:** `convert_B_packed_format(packed_B, B, N, K)` tiles over `NB = N/TILE_N` and `KB = K/BLOCK_K`, calling `pack_B` per (n,k) tile with weights transposed so `B[n0*KB + k]` lands column-major within the tile. Weights are stored **column-major within each 16-wide N tile** — this transpose is the repack TILE eliminates by storing post-transpose.

**Store per-tensor:** `amx.tile_n=16`, `amx.tile_k=32`, `amx.vnni_blk=4`, `amx.source_quant` (enum), `amx.has_compensation` (bool — true for s8s8 types like q8\_0).

> The `s8s8_compensation` int32 column is required data, not derivable at load without recomputation — TILE stores it inline so the kernel skips the `s8s8_compensation()` call entirely.

***

### 5. Schedule hints block (optional)

Removes per-machine autotune cold-start. Advisory — a runtime may ignore it.

```
u32   entry_count
repeat entry_count:
  u16  machine_class_len
  u8   machine_class[..]   # e.g. "avx512_8c", "amx_xeon_sp", "neon_rpi5"
  u8   recommended_threads
  u8   tile_m              # gemm tile dims
  u8   tile_n
  u8   tile_k
  u8   flags               # bit0: prefer LUT path; bit1: enable fast-aggregation
```

***

### 6. NUMA shard map (optional)

The current llama.cpp NUMA story is BIOS hacks and forks. TILE expresses sharding in-file so a multi-socket loader can map the right bytes to the right node without duplication guesswork.

```
u32   socket_count
u32   sharded_tensor_count
repeat sharded_tensor_count:
  u32  tensor_index        # into tensor directory
  u8   shard_axis          # 0=row, 1=col
  repeat socket_count:
    u64  shard_offset      # from tensor_data_offset
    u64  shard_len_bytes
```

A single-socket loader ignores this block and reads each tensor whole via its directory entry. A NUMA-aware loader mmaps each shard on its owning node.

***

### 7. Tensor data

* Begins page-aligned; each tensor's `data_offset` chosen so hot tensors don't straddle pages badly.
* When `EXEC_ORDER` is set, tensors appear in forward-pass order → sequential page-cache prefetch during a forward pass.
* LUT payloads follow §4.3 (bit-plane index arrays; table is runtime-built, not stored). AMX\_TILE payloads follow §4.4 (16×32 VNNI-packed B-matrix tiles, layout per source quant type). Both load with zero repack — that is the point of the format.

***

### 8. GPU-as-nice-to-have contract

* No GPU-packed duplicate is stored. `gpu_offload_score` + `GPU_REPACK_HINT` tell the runtime which tensors are worth repacking onto the device during the host→device copy that already occurs.
* The format promises: best-in-class CPU inference, and it *uses* your GPU if present. It does **not** promise GPU-optimal throughput — that is explicitly out of scope.

***

### 9. Open items before v0.2

1. ✅ **Resolved.** LUT layout verified against `preprocess_weights` — bit-plane split → g-bit LUT-index fold → SIMD interleave, output shape `(M//bm, K//g, bm//ngroups_per_elem)`. See §4.3.
2. ✅ **Resolved.** AMX layout verified against `mmq.cpp` — per-quant-type `packed_B` (quants + scales + mins + d + optional s8s8 compensation), TILE\_N=16, TILE\_K=32. See §4.4.
3. **New constraint (from §4.3):** LUT packed layout is *tuning-specific* (`bm`, `kfactor`, `simd_n_in/out`). Decide: ship one canonical tuning per ISA class, or store tuning params in-header and let loaders reject mismatches. This is the biggest open design question now.
4. **New constraint (from §4.4):** AMX packing is per-source-quant-type, not universal. Either enumerate supported source quants or define a canonical TILE int8 tile and convert at pack time.
5. Decide single-layout (runtime repack for GPU) vs. optional dual-layout escape hatch. v0.1 assumes single-layout.
6. Activation-sparsity metadata block — reserved via `HAS_SPARSITY` flag, not yet specified.
7. Alignment policy for straddle-avoidance under 2MB huge pages.
8. Versioning/compat rules for adding new `encoding` / `tile_layout` values without breaking loaders.

### 10. Strategic note

The kernel wins (LUT, AMX) are already *demonstrated* by T-MAC and llama.cpp — this de-risks the performance thesis but means the moat is not novelty. It is being the first **native, zero-repack, single-file** container that unifies these proven CPU kernels without the repack tax, the toolchain fragility (AMX needs clang + AVX512\_VNNI, is Linux-only in practice), and the BIOS-hack NUMA story. This is a packaging-and-coherence play. Adoption will therefore hinge on shipping (or being adopted by) a runtime people use for reasons *other* than the format — the same dynamic that carried GGUF on llama.cpp's back.

***

I'll build a reference loader with a matching writer so it's testable end-to-end. Let me check the frontend/code conventions aren't needed here (pure Python lib), then write it.Now a test that builds a realistic small model — a RAW tensor, a LUT tensor with T-MAC params, and an AMX tensor with q8\_0 packing — writes it, reads it back, and verifies round-trip fidelity including the packed-shape reconstruction.All checks pass. The reference loader works end-to-end and — importantly — the LUT tensor is built with the *actual* T-MAC `preprocess_weights` transform, so this validates the container against real byte layouts, not a toy stand-in.

What the test proves:

* **RAW** — exact float round-trip.
* **LUT (§4.3)** — a real T-MAC-preprocessed weight (M=512, K=1024, 2-bit) packs to shape `(2, 256, 256)` = 131072 bytes, and the loader's `lut_logical_shape()` reconstructs that exact shape from stored metadata alone. The "kernel-view reshape" line confirms the contract: a consumer reshapes `t.data` to the packed shape with **zero repack**.
* **AMX (§4.4)** — the q8\_0 packed\_B payload sizes correctly at 622592 bytes for a 32×32 tile grid, including the fp16 scales and the int32 s8s8-compensation column that the source revealed is required. `source_quant` and `has_compensation` survive round-trip.
* **Schedule, NUMA, exec-order** — all serialize and read back; tensor data lands page-aligned; EXEC\_ORDER flag drives layout ordering.

One honest limitation worth stating: this validates *structure and sizing*, not numerical kernel correctness. The AMX payload is fabricated bytes of the right shape, not a real quantization, and neither kernel is executed. Proving a real model produces correct logits needs the actual T-MAC/AMX kernels wired to consume these buffers — that's the next tier of work, and it's where the "zero-repack" claim gets its final benchmark.

Files:You now have a working reference implementation of TILE grounded in real kernel layouts.

The three files:

* **`tileformat.py`** — the reader/writer library. Header, KV metadata, tensor directory, LUT/AMX per-tensor metadata, schedule hints, NUMA shard map, page-aligned execution-order data. Well-commented, correctness over speed.
* **`test_tile.py`** — the round-trip test, which builds its LUT tensor with the actual T-MAC `preprocess_weights` transform so it validates against genuine byte layouts.
* **`tile-format-spec-v0.1.md`** — the v0.2 spec the code implements.

Where this leaves the project: the format design is validated at the container-and-layout level, and the two hardest technical claims (LUT bit-plane packing, AMX per-quant-type tiles) are pinned to source, not guesswork. The honest gap between here and a real format is executing the kernels — wiring T-MAC and llama.cpp's AMX gemm to consume these buffers and confirming a real model's logits match, which is also where the zero-repack speed claim gets benchmarked.

Two design decisions are still yours to make, both surfaced by reading the source (open items 3 and 4 in the spec): whether LUT layout ships as one canonical tuning per ISA class or stores tuning params and rejects mismatches, and whether AMX enumerates supported source quants or defines a single canonical TILE tile. Neither needs more research — they're judgment calls about how much flexibility to trade for simplicity.

***
