Learned Distance Metric

Overview

Euclidean and cosine distances treat all embedding dimensions equally. A learned distance metric passes the difference vector and its element-wise square through a linear transformation, then computes the squared norm. This approximates a Mahalanobis-like distance but with nonlinear capacity from the squared terms, learning which dimensions and interactions matter for similarity.

Quick Start

import torch
import torch.nn as nn

class LearnedMetric(nn.Module):
    def __init__(self, emb_dim=64):
        super().__init__()
        self.linear = nn.Linear(emb_dim * 2, emb_dim * 2, bias=False)

    def forward(self, v1, v2):
        d = v1 - v2
        d2 = d.pow(2)
        x = self.linear(torch.cat([d, d2], dim=-1))
        return x.pow(2).sum(dim=-1)

metric = LearnedMetric(emb_dim=128)
dist = metric(embedding_a, embedding_b)

Workflow

1. Train embedding model with a fixed or learned metric
2. Initialize LearnedMetric matching embedding dimension
3. Train jointly with contrastive or triplet loss
4. At inference, use the learned metric for nearest-neighbor retrieval

Key Decisions

• No bias: bias-free linear keeps distance symmetric and zero for identical inputs
• Concat [d, d^2]: captures both linear (Mahalanobis-like) and nonlinear distance components
• Joint training: train metric alongside the embedding model end-to-end
• vs Mahalanobis: Mahalanobis is linear-only; this adds nonlinear capacity with minimal parameters

References

• Similarity DenseNet121 [0.805LB]