YouTipPytorch Torch Tanh
## PyTorch torch.tanh
The `torch.tanh` function in PyTorch is used to compute the element-wise hyperbolic tangent (tanh) of an input tensor. It is a widely used activation function in deep learning, particularly in recurrent neural networks (RNNs) like LSTMs and GRUs, as well as in various feedforward neural networks.
---
## Introduction to Hyperbolic Tangent (tanh)
The hyperbolic tangent function maps any real-valued input to a value between **-1 and 1**. Mathematically, it is defined as:
$$\tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}} = \frac{e^{2x} - 1}{e^{2x} + 1}$$
### Key Properties:
* **Output Range:** $(-1, 1)$
* **Zero-Centered:** Unlike the sigmoid function (which outputs between $0$ and $1$), the tanh function is zero-centered (its output at $x=0$ is $0$). This property often helps neural networks converge faster during training.
* **Saturating Gradients:** For very large positive or negative inputs, the gradient of the tanh function approaches zero, which can sometimes lead to the vanishing gradient problem.
---
## Syntax & Parameters
### Function Signature
```python
torch.tanh(input, *, out=None) -> Tensor
```
### Parameters
* **`input`** *(Tensor)*: The input tensor containing the values to compute.
* **`out`** *(Tensor, optional)*: The alternative output tensor in which to write the result. It must have the same shape as the input.
### Return Value
* Returns a new tensor with the same shape and data type as `input`, containing the element-wise hyperbolic tangent values.
---
## Code Examples
### Example 1: Basic Usage with a 1D Tensor
This example demonstrates how to compute the hyperbolic tangent of a simple 1D tensor.
```python
import torch
# Create a 1D tensor with float values
x = torch.tensor([-1.0, 0.0, 1.0])
# Compute the hyperbolic tangent
result = torch.tanh(x)
print("Input Tensor:")
print(x)
print("\nOutput Tensor (tanh):")
print(result)
```
**Output:**
```text
Input Tensor:
tensor([-1.0000, 0.0000, 1.0000])
Output Tensor (tanh):
tensor([-0.7616, 0.0000, 0.7616])
```
### Example 2: In-place Operation (`tanh_`)
If you want to modify the tensor in-place to save memory, you can use `torch.tanh_()` or the tensor method `.tanh_()`.
```python
import torch
# Create a 2D tensor
x = torch.tensor([[-2.0, -0.5], [0.5, 2.0]])
# Apply in-place tanh
x.tanh_()
print("In-place modified tensor:")
print(x)
```
**Output:**
```text
In-place modified tensor:
tensor([[-0.9640, -0.4621],
[ 0.4621, 0.9640]])
```
---
## Considerations & Best Practices
1. **Vanishing Gradient:** While `tanh` is zero-centered and generally performs better than the sigmoid function in hidden layers, it still suffers from the vanishing gradient problem for highly positive or negative inputs. For very deep networks, activation functions like `ReLU` or `GELU` are often preferred.
2. **Data Type Support:** `torch.tanh` supports floating-point and complex data types. If you pass integer tensors, PyTorch will automatically promote them to floating-point types to perform the calculation.