April 13, 2022
DotProductAttention
用所有值计算查询的点积，并应用 softmax 函数来获得值的权重 import math import torch import torch.nn as nn import torch.nn.functional as F from...
注意力
April 13, 2022
缩放的点乘注意力ScaledDotProductAttention
缩放的点乘注意力ScaledDotProductAttention 就是Transformer中使用的注意力 import math import torch import torch.nn as nn import torch.nn.functional as F from torch...
October 21, 2021
unilm_mask 注意力矩阵生成
unilm v1注意力矩阵生成 def unilm_mask(inputs, s): idxs = torch.cumsum(s, dim=1) mask = idxs[:, None, :] <= idxs[:,...