unilm_mask 注意力矩阵生成

unilm v1注意力矩阵生成 =============== [caption id="attachment\_5072" align="alignnone" width="1080"]![unilm v1注意力矩阵](https://terrychanorg.oss-accelerate.aliyuncs.com/2021/10/202110211…

缩放的点乘注意力ScaledDotProductAttention

缩放的点乘注意力ScaledDotProductAttention 就是Transformer中使用的注意力 ``` import math import torch import torch.nn as nn import torch.nn.functional as F from torch import Tensor import numpy …

DotProductAttention

用所有值计算查询的点积，并应用 softmax 函数来获得值的权重 ``` import math import torch import torch.nn as nn import torch.nn.functional as F from torch import Tensor import numpy as np from typing imp…