Torchcode Pytorch Interview Practice

作者 reason-machines2384a003145a無授權條款83 個星標收錄於 2026年10月8日更新於 2026年10月8日儲存庫3 個月前更新

LeetCode-style PyTorch interview practice environment with auto-grading for implementing softmax, attention, GPT-2 and more from scratch.

僅含說明Learning & EducationSoftware Development
AI 產生的概覽

附自動評分的 PyTorch 面試練習，包含 40 道從零實作題。

功能
此技能帶領機器學習工程師完成 40 道 LeetCode 風格的 PyTorch 題目，從 ReLU、softmax 到 LayerNorm、多頭注意力與 GPT-2 元件。它提供 notebook 範本、參考解答、提示與實作範例，並透過 torch_judge 套件對照 PyTorch 參考實作檢查正確性、驗證梯度並測量耗時。內容也涵蓋線上、pip、Docker 與本機建置等安裝方式，以及評分與自動微分常見問題的排解。
適用情境
適合準備 PyTorch 或機器學習程式面試、想進行有結構且可評分練習時使用。適用於從零實作算子或 Transformer 元件、執行 torchcode 評分器、檢查 PyTorch 實作等請求。也適合學習數值穩定性、初始化以及訓練/評估模式等模式。
執行需求
需要安裝 PyTorch 與 torch-judge 套件的 Python 環境以使用評分 API；notebook 可在 Jupyter、Colab 或 Docker 映像中執行。部分安裝方式需要網路連線以下載套件、複製儲存庫或拉取容器映像。此技能不附帶指令碼，僅為說明與程式範例。

TorchCode — PyTorch Interview Practice

Skill by ara.so — Daily 2026 Skills collection.

TorchCode is a Jupyter-based, self-hosted coding practice environment for ML engineers. It provides 40 curated problems covering PyTorch fundamentals and architectures (softmax, LayerNorm, MultiHeadAttention, GPT-2, etc.) with an automated judge that gives instant pass/fail feedback, gradient verification, and timing — like LeetCode but for tensors.

Installation & Setup

Option 1: Online (zero install)

Option 2: pip (for use inside Colab or existing environment)

bash
pip install torch-judge

Option 3: Docker (pre-built image)

bash
docker run -p 8888:8888 -e PORT=8888 ghcr.io/duoan/torchcode:latest# Open http://localhost:8888

Option 4: Build locally

bash
git clone https://github.com/duoan/TorchCode.gitcd TorchCodemake run# Open http://localhost:8888

make run auto-detects Docker or Podman and falls back to local build if the registry image is unavailable (common on Apple Silicon/arm64).

Judge API

The torch_judge package provides the core API used in every notebook.

python
from torch_judge import check, status, hint, reset_progress
# List all 40 problems and your progressstatus()
# Run tests for a specific problemcheck("relu")check("softmax")check("layernorm")check("attention")check("gpt2")
# Get a hint without spoilershint("softmax")
# Reset progress for a problemreset_progress("relu")

check() return values

  • Colored pass/fail per test case
  • Correctness check against PyTorch reference implementation
  • Gradient verification (autograd compatibility)
  • Timing measurement

Problem Set Overview

Difficulty levels: Easy → Medium → Hard

#ProblemKey Concepts
1ReLUActivation functions, element-wise ops
2SoftmaxNumerical stability, exp/log tricks
3Linear Layery = xW^T + b, Kaiming init, nn.Parameter
4LayerNormNormalization, affine transform
5Self-AttentionQKV projections, scaled dot-product
6Multi-Head AttentionHead splitting, concatenation
7BatchNormBatch vs layer statistics, train/eval
8RMSNormLLaMA-style norm
16Cross-Entropy LossLog-softmax, logsumexp trick
17DropoutTrain/eval mode, inverted scaling
18EmbeddingLookup table, weight[indices]
19GELUtorch.erf, Gaussian error linear unit
20Kaiming Initstd = sqrt(2/fan_in)
21Gradient ClippingNorm-based clipping
31Gradient AccumulationMicro-batching, loss scaling
40Linear RegressionNormal equation, GD from scratch

Working Through a Problem

Each problem notebook has the same structure:

templates/  01_relu.ipynb       # Blank template — your workspace  02_softmax.ipynb  ...solutions/  01_relu.ipynb       # Reference solution (study after attempt)

Typical notebook workflow

python
# Cell 1: Import judgefrom torch_judge import check, hintimport torchimport torch.nn as nn
# Cell 2: Your implementationdef my_relu(x: torch.Tensor) -> torch.Tensor:    # TODO: implement ReLU without using torch.relu or F.relu    raise NotImplementedError
# Cell 3: Run the judgecheck("relu")

Real Implementation Examples

ReLU (Problem 1 — Easy)

python
def my_relu(x: torch.Tensor) -> torch.Tensor:    return torch.clamp(x, min=0)    # Alternative: return x * (x > 0)    # Alternative: return torch.where(x > 0, x, torch.zeros_like(x))

Softmax (Problem 2 — Easy, numerically stable)

python
def my_softmax(x: torch.Tensor, dim: int = -1) -> torch.Tensor:    # Subtract max for numerical stability (prevents overflow)    x_max = x.max(dim=dim, keepdim=True).values    x_shifted = x - x_max    exp_x = torch.exp(x_shifted)    return exp_x / exp_x.sum(dim=dim, keepdim=True)

LayerNorm (Problem 4 — Medium)

python
def my_layer_norm(    x: torch.Tensor,    weight: torch.Tensor,   # gamma (scale)    bias: torch.Tensor,     # beta (shift)    eps: float = 1e-5) -> torch.Tensor:    mean = x.mean(dim=-1, keepdim=True)    var = x.var(dim=-1, keepdim=True, unbiased=False)    x_norm = (x - mean) / torch.sqrt(var + eps)    return weight * x_norm + bias

RMSNorm (Problem 8 — Medium, LLaMA-style)

python
def rms_norm(x: torch.Tensor, weight: torch.Tensor, eps: float = 1e-6) -> torch.Tensor:    rms = torch.sqrt((x ** 2).mean(dim=-1, keepdim=True) + eps)    return (x / rms) * weight

Scaled Dot-Product Self-Attention (Problem 5 — Medium)

python
import torch.nn.functional as Fimport math
def scaled_dot_product_attention(    Q: torch.Tensor,  # (B, heads, T, head_dim)    K: torch.Tensor,    V: torch.Tensor,    mask: torch.Tensor = None) -> torch.Tensor:    d_k = Q.size(-1)    scores = torch.matmul(Q, K.transpose(-2, -1)) / math.sqrt(d_k)    if mask is not None:        scores = scores.masked_fill(mask == 0, float('-inf'))    attn_weights = F.softmax(scores, dim=-1)    return torch.matmul(attn_weights, V)

Multi-Head Attention (Problem 6 — Medium)

python
class MyMultiHeadAttention(nn.Module):    def __init__(self, d_model: int, num_heads: int):        super().__init__()        assert d_model % num_heads == 0        self.num_heads = num_heads        self.head_dim = d_model // num_heads        self.d_model = d_model
        self.W_q = nn.Linear(d_model, d_model)        self.W_k = nn.Linear(d_model, d_model)        self.W_v = nn.Linear(d_model, d_model)        self.W_o = nn.Linear(d_model, d_model)
    def forward(self, x: torch.Tensor, mask: torch.Tensor = None) -> torch.Tensor:        B, T, C = x.shape
        def split_heads(t):            return t.view(B, T, self.num_heads, self.head_dim).transpose(1, 2)
        Q = split_heads(self.W_q(x))        K = split_heads(self.W_k(x))        V = split_heads(self.W_v(x))
        attn_out = scaled_dot_product_attention(Q, K, V, mask)        # (B, heads, T, head_dim) -> (B, T, d_model)        attn_out = attn_out.transpose(1, 2).contiguous().view(B, T, C)        return self.W_o(attn_out)

Cross-Entropy Loss (Problem 16 — Easy)

python
def cross_entropy_loss(logits: torch.Tensor, targets: torch.Tensor) -> torch.Tensor:    # logits: (B, C), targets: (B,) with class indices    # Use logsumexp trick for numerical stability    log_sum_exp = torch.logsumexp(logits, dim=-1)  # (B,)    log_probs = logits[torch.arange(len(targets)), targets]  # (B,)    return (log_sum_exp - log_probs).mean()

Dropout (Problem 17 — Easy)

python
class MyDropout(nn.Module):    def __init__(self, p: float = 0.5):        super().__init__()        self.p = p
    def forward(self, x: torch.Tensor) -> torch.Tensor:        if not self.training or self.p == 0:            return x        mask = torch.bernoulli(torch.ones_like(x) * (1 - self.p))        return x * mask / (1 - self.p)  # inverted scaling

Kaiming Init (Problem 20 — Easy)

python
def kaiming_init(weight: torch.Tensor) -> torch.Tensor:    fan_in = weight.size(1)    std = math.sqrt(2.0 / fan_in)    with torch.no_grad():        weight.normal_(0, std)    return weight

Gradient Clipping (Problem 21 — Easy)

python
def clip_grad_norm(parameters, max_norm: float) -> float:    params = [p for p in parameters if p.grad is not None]    total_norm = torch.sqrt(sum(p.grad.data.norm() ** 2 for p in params))    clip_coef = max_norm / (total_norm + 1e-6)    if clip_coef < 1:        for p in params:            p.grad.data.mul_(clip_coef)    return total_norm.item()

Gradient Accumulation (Problem 31 — Easy)

python
def train_with_accumulation(model, optimizer, dataloader, accumulation_steps=4):    optimizer.zero_grad()    for i, (inputs, targets) in enumerate(dataloader):        outputs = model(inputs)        loss = criterion(outputs, targets) / accumulation_steps  # scale loss        loss.backward()
        if (i + 1) % accumulation_steps == 0:            optimizer.step()            optimizer.zero_grad()

Common Patterns & Tips

Numerical stability pattern

Always subtract the max before exp():

python
# WRONG — can overflow for large valuesexp_x = torch.exp(x)
# CORRECT — numerically stableexp_x = torch.exp(x - x.max(dim=-1, keepdim=True).values)

Causal attention mask (for GPT-style models)

python
def causal_mask(T: int, device) -> torch.Tensor:    return torch.tril(torch.ones(T, T, device=device)).unsqueeze(0).unsqueeze(0)

nn.Module skeleton (used in many problems)

python
class MyLayer(nn.Module):    def __init__(self, ...):        super().__init__()        self.weight = nn.Parameter(torch.empty(...))        self.bias = nn.Parameter(torch.zeros(...))        self._init_weights()
    def _init_weights(self):        nn.init.kaiming_uniform_(self.weight)
    def forward(self, x: torch.Tensor) -> torch.Tensor:        ...

Train vs eval mode pattern

python
def forward(self, x):    if self.training:        # use batch statistics        mean = x.mean(dim=0)        var = x.var(dim=0, unbiased=False)        # update running stats        self.running_mean = (1 - self.momentum) * self.running_mean + self.momentum * mean        self.running_var = (1 - self.momentum) * self.running_var + self.momentum * var    else:        # use running statistics        mean = self.running_mean        var = self.running_var    return (x - mean) / torch.sqrt(var + self.eps) * self.weight + self.bias

Project Structure

TorchCode/├── templates/          # Blank notebooks for each problem (your workspace)│   ├── 01_relu.ipynb│   ├── 02_softmax.ipynb│   └── ...├── solutions/          # Reference solutions (study after attempting)│   └── ...├── torch_judge/        # Auto-grading package│   ├── __init__.py     # check(), status(), hint(), reset_progress()│   └── tasks/          # Per-problem test cases├── Dockerfile├── Makefile└── pyproject.toml      # torch-judge package definition

Troubleshooting

Docker image not available for Apple Silicon (arm64)

bash
# make run auto-falls back to local build, or force it:make buildmake start

check() not found in Colab

bash
!pip install torch-judge# then restart runtime

Notebook reset to blank template

Use the toolbar "Reset" button in JupyterLab to reset any notebook to its original blank state — useful for re-practicing a problem.

Gradient check fails but output is correct

Ensure your implementation uses PyTorch operations (not NumPy) so autograd works:

python
# WRONG — breaks autogradimport numpy as npresult = np.exp(x.numpy())
# CORRECT — autograd compatibleresult = torch.exp(x)

Viewing reference solution

After attempting a problem, open the matching file in solutions/:

solutions/02_softmax.ipynb

Key Concepts Tested

ConceptProblems
Numerical stabilitySoftmax, Cross-Entropy, LogSumExp
Autograd / nn.ParameterLinear, LayerNorm, all nn.Module problems
Train vs eval behaviorBatchNorm, Dropout
BroadcastingLayerNorm, RMSNorm, attention masking
Shape manipulationMulti-Head Attention (view, transpose, contiguous)
Weight initializationKaiming Init, Linear Layer
Memory-efficient trainingGradient Accumulation, Gradient Clipping

來源與署名

來源：reason-machines/trending-skills位於skills/torchcode-pytorch-interview-practice提交2384a00

授權條款： 無授權條款

內容歸原作者所有。SourceWeft 從公開儲存庫中收錄這些內容。

檢舉或申請下架

更多來自 reason-machines/trending-skills 的技能

Yourvpndead Vpn Detection

reason-machines

介紹一款 Android 應用程式，用來偵測 VPN 用戶端並探測本機 SOCKS5 與 Clash API 連接埠是否外洩。

Security833 個月前更新

X Mentor Skill Nuwa

reason-machines

提供 X（Twitter）內容策略指引：推文與長推撰寫、鉤子評論、成長策略與帳號診斷。

Marketing & Sales833 個月前更新

Wildworld Dataset

reason-machines

介紹 WildWorld 動作條件 ARPG 世界建模資料集，並提供預期的載入、篩選與基準評測程式碼。

Data & Analytics833 個月前更新

Understand Anything Knowledge Graph

reason-machines

為程式碼庫建立可互動的知識圖譜，並支援視覺化探索、搜尋與提問。

Software Development833 個月前更新

Webphysics Avbd Engine

reason-machines

指導將實驗性 WebGPU AVBD 剛體與軟體物理引擎整合到瀏覽器專案中。

Software Development833 個月前更新

Ui Ux Pro Max Skill

reason-machines

依產品描述產生 UI/UX 設計系統，包含頁面結構、風格、配色、字體搭配與檢查清單。

Design & Creative833 個月前更新

更多Learning & Education技能

Jupyter Notebook

openai

精選

使用內建範本與輔助指令碼建立或編輯用於實驗或教學的 Jupyter 筆記本。

包含腳本
Learning & Education27K4 週前更新

Wiki Onboarding

microsoft

精選

產生四份針對不同讀者的程式碼庫入門指南：貢獻者、資深工程師、高階主管和產品經理。

Learning & Education3K今天更新

Azuresql Db Faq

microsoft

精選

解答 Azure SQL 資料庫容器（個人預覽版）支援哪些功能，以及為何與雲端服務不同。

Learning & Education182 週前更新

Teach

mattpocock

透過多次工作階段教使用者學習某個主題，在工作區中產出 HTML 課程、參考文件與學習紀錄。

Learning & Education280K今天更新

Scaffold Exercises

mattpocock

建立課程練習目錄結構，包含 problem、solution、explainer 子目錄與 readme 佔位檔案，並通過 lint 檢查。

Learning & Education280K今天更新

Growth Log

affaan-m

指導撰寫成長日誌條目，從已完成的工作中萃取可移轉的模式，而不是敘述事件。

Learning & Education275K3 天前更新