Torchcode Pytorch Interview Practice

作者 reason-machines2384a003145a无许可证83 个星标收录于 2026年10月8日更新于 2026年10月8日仓库3个月前更新

LeetCode-style PyTorch interview practice environment with auto-grading for implementing softmax, attention, GPT-2 and more from scratch.

仅含说明Learning & EducationSoftware Development
AI 生成的概览

带自动评分的 PyTorch 面试练习，包含 40 道从零实现题。

功能
该技能引导机器学习工程师完成 40 道 LeetCode 风格的 PyTorch 题目，从 ReLU、softmax 到 LayerNorm、多头注意力和 GPT-2 组件。它提供 notebook 模板、参考答案、提示和实现示例，并通过 torch_judge 包对照 PyTorch 参考实现检查正确性、验证梯度并测量耗时。内容还涵盖在线、pip、Docker 和本地构建等安装方式，以及评分与自动求导常见问题的排查。
适用场景
适合准备 PyTorch 或机器学习编码面试、希望进行有结构且可评分的练习时使用。适用于从零实现算子或 Transformer 组件、运行 torchcode 评分器、检查 PyTorch 实现等请求。也适合学习数值稳定性、初始化以及训练/评估模式等模式。
运行要求
需要安装 PyTorch 和 torch-judge 包的 Python 环境以使用评分 API；notebook 可在 Jupyter、Colab 或 Docker 镜像中运行。部分安装方式需要联网以下载包、克隆仓库或拉取容器镜像。该技能不附带脚本，仅为说明与代码示例。

TorchCode — PyTorch Interview Practice

Skill by ara.so — Daily 2026 Skills collection.

TorchCode is a Jupyter-based, self-hosted coding practice environment for ML engineers. It provides 40 curated problems covering PyTorch fundamentals and architectures (softmax, LayerNorm, MultiHeadAttention, GPT-2, etc.) with an automated judge that gives instant pass/fail feedback, gradient verification, and timing — like LeetCode but for tensors.

Installation & Setup

Option 1: Online (zero install)

Option 2: pip (for use inside Colab or existing environment)

bash
pip install torch-judge

Option 3: Docker (pre-built image)

bash
docker run -p 8888:8888 -e PORT=8888 ghcr.io/duoan/torchcode:latest# Open http://localhost:8888

Option 4: Build locally

bash
git clone https://github.com/duoan/TorchCode.gitcd TorchCodemake run# Open http://localhost:8888

make run auto-detects Docker or Podman and falls back to local build if the registry image is unavailable (common on Apple Silicon/arm64).

Judge API

The torch_judge package provides the core API used in every notebook.

python
from torch_judge import check, status, hint, reset_progress
# List all 40 problems and your progressstatus()
# Run tests for a specific problemcheck("relu")check("softmax")check("layernorm")check("attention")check("gpt2")
# Get a hint without spoilershint("softmax")
# Reset progress for a problemreset_progress("relu")

check() return values

  • Colored pass/fail per test case
  • Correctness check against PyTorch reference implementation
  • Gradient verification (autograd compatibility)
  • Timing measurement

Problem Set Overview

Difficulty levels: Easy → Medium → Hard

#ProblemKey Concepts
1ReLUActivation functions, element-wise ops
2SoftmaxNumerical stability, exp/log tricks
3Linear Layery = xW^T + b, Kaiming init, nn.Parameter
4LayerNormNormalization, affine transform
5Self-AttentionQKV projections, scaled dot-product
6Multi-Head AttentionHead splitting, concatenation
7BatchNormBatch vs layer statistics, train/eval
8RMSNormLLaMA-style norm
16Cross-Entropy LossLog-softmax, logsumexp trick
17DropoutTrain/eval mode, inverted scaling
18EmbeddingLookup table, weight[indices]
19GELUtorch.erf, Gaussian error linear unit
20Kaiming Initstd = sqrt(2/fan_in)
21Gradient ClippingNorm-based clipping
31Gradient AccumulationMicro-batching, loss scaling
40Linear RegressionNormal equation, GD from scratch

Working Through a Problem

Each problem notebook has the same structure:

templates/  01_relu.ipynb       # Blank template — your workspace  02_softmax.ipynb  ...solutions/  01_relu.ipynb       # Reference solution (study after attempt)

Typical notebook workflow

python
# Cell 1: Import judgefrom torch_judge import check, hintimport torchimport torch.nn as nn
# Cell 2: Your implementationdef my_relu(x: torch.Tensor) -> torch.Tensor:    # TODO: implement ReLU without using torch.relu or F.relu    raise NotImplementedError
# Cell 3: Run the judgecheck("relu")

Real Implementation Examples

ReLU (Problem 1 — Easy)

python
def my_relu(x: torch.Tensor) -> torch.Tensor:    return torch.clamp(x, min=0)    # Alternative: return x * (x > 0)    # Alternative: return torch.where(x > 0, x, torch.zeros_like(x))

Softmax (Problem 2 — Easy, numerically stable)

python
def my_softmax(x: torch.Tensor, dim: int = -1) -> torch.Tensor:    # Subtract max for numerical stability (prevents overflow)    x_max = x.max(dim=dim, keepdim=True).values    x_shifted = x - x_max    exp_x = torch.exp(x_shifted)    return exp_x / exp_x.sum(dim=dim, keepdim=True)

LayerNorm (Problem 4 — Medium)

python
def my_layer_norm(    x: torch.Tensor,    weight: torch.Tensor,   # gamma (scale)    bias: torch.Tensor,     # beta (shift)    eps: float = 1e-5) -> torch.Tensor:    mean = x.mean(dim=-1, keepdim=True)    var = x.var(dim=-1, keepdim=True, unbiased=False)    x_norm = (x - mean) / torch.sqrt(var + eps)    return weight * x_norm + bias

RMSNorm (Problem 8 — Medium, LLaMA-style)

python
def rms_norm(x: torch.Tensor, weight: torch.Tensor, eps: float = 1e-6) -> torch.Tensor:    rms = torch.sqrt((x ** 2).mean(dim=-1, keepdim=True) + eps)    return (x / rms) * weight

Scaled Dot-Product Self-Attention (Problem 5 — Medium)

python
import torch.nn.functional as Fimport math
def scaled_dot_product_attention(    Q: torch.Tensor,  # (B, heads, T, head_dim)    K: torch.Tensor,    V: torch.Tensor,    mask: torch.Tensor = None) -> torch.Tensor:    d_k = Q.size(-1)    scores = torch.matmul(Q, K.transpose(-2, -1)) / math.sqrt(d_k)    if mask is not None:        scores = scores.masked_fill(mask == 0, float('-inf'))    attn_weights = F.softmax(scores, dim=-1)    return torch.matmul(attn_weights, V)

Multi-Head Attention (Problem 6 — Medium)

python
class MyMultiHeadAttention(nn.Module):    def __init__(self, d_model: int, num_heads: int):        super().__init__()        assert d_model % num_heads == 0        self.num_heads = num_heads        self.head_dim = d_model // num_heads        self.d_model = d_model
        self.W_q = nn.Linear(d_model, d_model)        self.W_k = nn.Linear(d_model, d_model)        self.W_v = nn.Linear(d_model, d_model)        self.W_o = nn.Linear(d_model, d_model)
    def forward(self, x: torch.Tensor, mask: torch.Tensor = None) -> torch.Tensor:        B, T, C = x.shape
        def split_heads(t):            return t.view(B, T, self.num_heads, self.head_dim).transpose(1, 2)
        Q = split_heads(self.W_q(x))        K = split_heads(self.W_k(x))        V = split_heads(self.W_v(x))
        attn_out = scaled_dot_product_attention(Q, K, V, mask)        # (B, heads, T, head_dim) -> (B, T, d_model)        attn_out = attn_out.transpose(1, 2).contiguous().view(B, T, C)        return self.W_o(attn_out)

Cross-Entropy Loss (Problem 16 — Easy)

python
def cross_entropy_loss(logits: torch.Tensor, targets: torch.Tensor) -> torch.Tensor:    # logits: (B, C), targets: (B,) with class indices    # Use logsumexp trick for numerical stability    log_sum_exp = torch.logsumexp(logits, dim=-1)  # (B,)    log_probs = logits[torch.arange(len(targets)), targets]  # (B,)    return (log_sum_exp - log_probs).mean()

Dropout (Problem 17 — Easy)

python
class MyDropout(nn.Module):    def __init__(self, p: float = 0.5):        super().__init__()        self.p = p
    def forward(self, x: torch.Tensor) -> torch.Tensor:        if not self.training or self.p == 0:            return x        mask = torch.bernoulli(torch.ones_like(x) * (1 - self.p))        return x * mask / (1 - self.p)  # inverted scaling

Kaiming Init (Problem 20 — Easy)

python
def kaiming_init(weight: torch.Tensor) -> torch.Tensor:    fan_in = weight.size(1)    std = math.sqrt(2.0 / fan_in)    with torch.no_grad():        weight.normal_(0, std)    return weight

Gradient Clipping (Problem 21 — Easy)

python
def clip_grad_norm(parameters, max_norm: float) -> float:    params = [p for p in parameters if p.grad is not None]    total_norm = torch.sqrt(sum(p.grad.data.norm() ** 2 for p in params))    clip_coef = max_norm / (total_norm + 1e-6)    if clip_coef < 1:        for p in params:            p.grad.data.mul_(clip_coef)    return total_norm.item()

Gradient Accumulation (Problem 31 — Easy)

python
def train_with_accumulation(model, optimizer, dataloader, accumulation_steps=4):    optimizer.zero_grad()    for i, (inputs, targets) in enumerate(dataloader):        outputs = model(inputs)        loss = criterion(outputs, targets) / accumulation_steps  # scale loss        loss.backward()
        if (i + 1) % accumulation_steps == 0:            optimizer.step()            optimizer.zero_grad()

Common Patterns & Tips

Numerical stability pattern

Always subtract the max before exp():

python
# WRONG — can overflow for large valuesexp_x = torch.exp(x)
# CORRECT — numerically stableexp_x = torch.exp(x - x.max(dim=-1, keepdim=True).values)

Causal attention mask (for GPT-style models)

python
def causal_mask(T: int, device) -> torch.Tensor:    return torch.tril(torch.ones(T, T, device=device)).unsqueeze(0).unsqueeze(0)

nn.Module skeleton (used in many problems)

python
class MyLayer(nn.Module):    def __init__(self, ...):        super().__init__()        self.weight = nn.Parameter(torch.empty(...))        self.bias = nn.Parameter(torch.zeros(...))        self._init_weights()
    def _init_weights(self):        nn.init.kaiming_uniform_(self.weight)
    def forward(self, x: torch.Tensor) -> torch.Tensor:        ...

Train vs eval mode pattern

python
def forward(self, x):    if self.training:        # use batch statistics        mean = x.mean(dim=0)        var = x.var(dim=0, unbiased=False)        # update running stats        self.running_mean = (1 - self.momentum) * self.running_mean + self.momentum * mean        self.running_var = (1 - self.momentum) * self.running_var + self.momentum * var    else:        # use running statistics        mean = self.running_mean        var = self.running_var    return (x - mean) / torch.sqrt(var + self.eps) * self.weight + self.bias

Project Structure

TorchCode/├── templates/          # Blank notebooks for each problem (your workspace)│   ├── 01_relu.ipynb│   ├── 02_softmax.ipynb│   └── ...├── solutions/          # Reference solutions (study after attempting)│   └── ...├── torch_judge/        # Auto-grading package│   ├── __init__.py     # check(), status(), hint(), reset_progress()│   └── tasks/          # Per-problem test cases├── Dockerfile├── Makefile└── pyproject.toml      # torch-judge package definition

Troubleshooting

Docker image not available for Apple Silicon (arm64)

bash
# make run auto-falls back to local build, or force it:make buildmake start

check() not found in Colab

bash
!pip install torch-judge# then restart runtime

Notebook reset to blank template

Use the toolbar "Reset" button in JupyterLab to reset any notebook to its original blank state — useful for re-practicing a problem.

Gradient check fails but output is correct

Ensure your implementation uses PyTorch operations (not NumPy) so autograd works:

python
# WRONG — breaks autogradimport numpy as npresult = np.exp(x.numpy())
# CORRECT — autograd compatibleresult = torch.exp(x)

Viewing reference solution

After attempting a problem, open the matching file in solutions/:

solutions/02_softmax.ipynb

Key Concepts Tested

ConceptProblems
Numerical stabilitySoftmax, Cross-Entropy, LogSumExp
Autograd / nn.ParameterLinear, LayerNorm, all nn.Module problems
Train vs eval behaviorBatchNorm, Dropout
BroadcastingLayerNorm, RMSNorm, attention masking
Shape manipulationMulti-Head Attention (view, transpose, contiguous)
Weight initializationKaiming Init, Linear Layer
Memory-efficient trainingGradient Accumulation, Gradient Clipping

来源与署名

来源：reason-machines/trending-skills位于skills/torchcode-pytorch-interview-practice提交2384a00

许可证： 无许可证

内容归原作者所有。SourceWeft 从公开仓库中收录这些内容。

举报或申请下架

更多来自 reason-machines/trending-skills 的技能

Yourvpndead Vpn Detection

reason-machines

介绍一款 Android 应用，用于检测 VPN 客户端并探测本机 SOCKS5 与 Clash API 端口是否泄露。

Security833个月前更新

X Mentor Skill Nuwa

reason-machines

提供 X（Twitter）内容策略指导：推文与长推写作、钩子点评、增长策略与账号诊断。

Marketing & Sales833个月前更新

Wildworld Dataset

reason-machines

介绍 WildWorld 动作条件 ARPG 世界建模数据集，并给出预期的加载、筛选与基准评测代码。

Data & Analytics833个月前更新

Understand Anything Knowledge Graph

reason-machines

为代码库构建可交互的知识图谱，并支持可视化探索、搜索和提问。

Software Development833个月前更新

Webphysics Avbd Engine

reason-machines

指导将实验性 WebGPU AVBD 刚体与软体物理引擎集成到浏览器项目中。

Software Development833个月前更新

Ui Ux Pro Max Skill

reason-machines

根据产品描述生成 UI/UX 设计系统，包括页面结构、风格、配色、字体搭配与检查清单。

Design & Creative833个月前更新

更多Learning & Education技能

Jupyter Notebook

openai

精选

使用内置模板和辅助脚本创建或编辑用于实验或教程的 Jupyter 笔记本。

包含脚本
Learning & Education27K4周前更新

Wiki Onboarding

microsoft

精选

生成四份面向不同读者的代码库入门指南：贡献者、资深工程师、高管和产品经理。

Learning & Education3K今天更新

Azuresql Db Faq

microsoft

精选

解答 Azure SQL 数据库容器（个人预览版）支持哪些功能，以及为何与云端服务不同。

Learning & Education182周前更新

Teach

mattpocock

通过多次会话教用户学习某个主题，在工作区中生成 HTML 课程、参考文档和学习记录。

Learning & Education280K今天更新

Scaffold Exercises

mattpocock

搭建课程练习目录结构，包含 problem、solution、explainer 子目录与 readme 占位文件，并通过 lint 检查。

Learning & Education280K今天更新

Growth Log

affaan-m

指导撰写成长日志条目，从已完成的工作中提炼可迁移的模式，而不是叙述事件。

Learning & Education275K3天前更新