GELU

Gaussian error linear unit, torch.erf

Easy Fundamentals

Problem Description

Implement the GELU (Gaussian Error Linear Unit) activation.

$$\text{GELU}(x) = x \cdot \Phi(x) = x \cdot 0.5 \cdot (1 + \text{erf}(x / \sqrt{2}))$$

def my_gelu(x: Tensor) -> Tensor: ...

• Do NOT use F.gelu, nn.GELU, or torch.nn.functional.gelu

• Use torch.erf for the exact version

Implement the function below. Use only basic PyTorch operations.

# ✏️ YOUR IMPLEMENTATION HERE

def my_gelu(x):
    pass

Use this code to debug before submitting.

# 🧪 Debug
x = torch.tensor([-2., -1., 0., 1., 2.])
print('Output:', my_gelu(x))
print('Ref:   ', torch.nn.functional.gelu(x))

Try solving it yourself first! Click below to reveal the solution.

# ✅ SOLUTION

def my_gelu(x):
    return 0.5 * x * (1.0 + torch.erf(x / math.sqrt(2.0)))

For interactive practice with auto-grading, run TorchCode locally:
pip install torch-judge then use check("gelu")

Gaussian error linear unit, torch.erf

Embedding Kaiming Init

Description Template Test Solution Tips