前言
本文介绍了倒置残差移动块（iRMB）及其在YOLOv11中的结合应用。现有轻量级CNN模型和Transformer模型在移动设备上存在局限性，iRMB融合了CNN和Transformer架构的优点，具有简单高效、资源消耗优化等创新点。它通过卷积多层感知器生成查询和键矩阵，利用膨胀卷积生成值矩阵，实施窗口级自注意力操作，还采用深度可分离卷积建模局部特征。基于iRMB构建的高效模型（EMO）在多个基准上表现出色。我们将iRMB集成进YOLOv11，实验证明其在性能上超越了先进方法，在参数、效率和准确度上取得良好权衡。
文章目录： YOLOv11改进大全：卷积层、轻量化、注意力机制、损失函数、Backbone、SPPF、Neck、检测头全方位优化汇总
专栏链接: YOLOv11改进专栏
@[TOC]
介绍
摘要
本论文旨在开发现代、高效、轻量的密集预测模型，并在参数、浮点运算次数与性能之间寻求平衡。虽然倒置残差块（IRB）是轻量级卷积神经网络（CNN）的重要基础，但在基于注意力的研究中尚缺类似的构件。本研究从统一视角出发，结合高效IRB和有效的Transformer组件，重新考虑轻量级基础架构。我们将基于CNN的IRB扩展到基于注意力的模型，并提出了一种单残差元移动块（MMB）用于轻量级模型设计。基于简单而有效的设计原则，我们推出了一种新型的倒置残差移动块（iRMB），并以此为基础构建了一个类似于ResNet的高效模型（EMO），适用于下游任务。在ImageNet-1K、COCO2017和ADE20K基准上的大量实验表明，我们的EMO在性能上超越了最先进的方法，例如，EMO-1M/2M/5M在Top-1准确率上分别达到了71.5、75.1和78.4，超过了同等级别的CNN-/基于注意力的模型，同时在参数、效率和准确度上取得了良好的权衡：在iPhone14上运行速度比EdgeNeXt快2.8-4.0倍。
创新点
iRMB (Inverted Residual Mobile Block) 的创新点在于其结合了CNN和Transformer架构的优点，为移动端应用设计了一个简单而高效的模块。这一设计旨在解决移动设备上对存储和计算资源限制下的高效模型需求，同时克服了现有轻量级CNN模型和Transformer模型的局限性。以下是iRMB的主要创新点：
-
融合CNN与Transformer的优点：iRMB吸收了CNN在建模短距离依赖方面的高效性，以及Transformer在动态建模长距离交互方面的能力。这种融合提供了一个均衡的解决方案，使得模型既能捕捉局部特征，也能理解全局上下文。
-
简单高效的设计：通过精心设计的Meta Mobile Block概念，iRMB通过级联的多头自注意力(MHSA)和卷积运算实现了高效的特征提取和信息流动。这种设计不仅保持了模型的高效性，而且简化了模型结构，便于移动端应用的部署和优化。
-
优化的资源消耗：相对于传统的Transformer模型，iRMB通过特定的设计减少了参数量和计算量(FLOPs)，使其更适合在资源受限的移动设备上运行。这一点通过在ImageNet-1K、COCO2017和ADE20K等基准测试中展示了其相对于同类模型的优越性能。
-
实现特定的技术突破：iRMB的设计克服了传统CNN模型由于其静态归纳偏差而导致的性能瓶颈，同时也解决了Transformer模型在移动设备上部署时由于参数和计算量大导致的问题。通过这种设计，iRMB为移动端高效模型的开发提供了新的思路。
-
灵活性和泛化能力：iRMB不仅在图像分类任务上表现出色，还在目标检测和语义分割等多个计算机视觉任务中展现了其优异的性能。这证明了iRMB不仅是一个高效的模块，而且具有良好的泛化能力，可以适用于多种视觉处理任务。
文章链接
论文地址：论文地址
代码地址：代码地址
基本原理
本文作者重新审视了轻量化CNN的反向残差模块和Transformer中的自注意力模块以及前馈神经网络模块，并从中归纳抽象出一个新的元结构——Meta-Mobile Block。本文作者认为之前的模块都是此元结构的一个实例化，并表明具体的实例化对模型性能非常重要。Meta-Mobile Block的公式如下所：
iRMB吸收了CNN在建模短距离依赖方面的高效性，以及Transformer在动态建模长距离交互方面的能力。这种融合提供了一个均衡的解决方案，使得模型既能捕捉局部特征，也能理解全局上下文。
如图2所示，在左侧部分，首先通过一个卷积多层感知器(CMLP)生成查询(Q)和键(K)矩阵。随后，利用膨胀卷积生成值(V)矩阵。接着，通过对Q、K、V实施窗口级别的自注意力操作，以实现远程信息的交互。此过程之后，使用深度可分离卷积(DWConv)对局部特征进行建模。最终，通过一个压缩卷积步骤调整通道数至原始水平，并将此输出与输入执行元素级加法操作以融合信息。特别指出，由于膨胀卷积和自注意力机制主要涉及矩阵乘法运算，因此可在执行膨胀卷积前计算自注意力机制。此优化策略旨在降低浮点操作次数(FLOPs)，同时保持操作的等效性，提高整体计算效率。
核心代码
import math
import torch
import torch.nn as nn
import torch.nn.functional as F
from functools import partial
from einops import rearrange
from timm.models._efficientnet_blocks import SqueezeExcite
from timm.models.layers import DropPath
inplace = True
class LayerNorm2d(nn.Module):
def __init__(self, normalized_shape, eps=1e-6, elementwise_affine=True):
super().__init__()
self.norm = nn.LayerNorm(normalized_shape, eps, elementwise_affine)
def forward(self, x):
x = rearrange(x, 'b c h w -> b h w c').contiguous()
x = self.norm(x)
x = rearrange(x, 'b h w c -> b c h w').contiguous()
return x
def get_norm(norm_layer='in_1d'):
eps = 1e-6
norm_dict = {
'none': nn.Identity,
'in_1d': partial(nn.InstanceNorm1d, eps=eps),
'in_2d': partial(nn.InstanceNorm2d, eps=eps),
'in_3d': partial(nn.InstanceNorm3d, eps=eps),
'bn_1d': partial(nn.BatchNorm1d, eps=eps),
'bn_2d': partial(nn.BatchNorm2d, eps=eps),
# 'bn_2d': partial(nn.SyncBatchNorm, eps=eps),
'bn_3d': partial(nn.BatchNorm3d, eps=eps),
'gn': partial(nn.GroupNorm, eps=eps),
'ln_1d': partial(nn.LayerNorm, eps=eps),
'ln_2d': partial(LayerNorm2d, eps=eps),
}
return norm_dict[norm_layer]
def get_act(act_layer='relu'):
act_dict = {
'none': nn.Identity,
'relu': nn.ReLU,
'relu6': nn.ReLU6,
'silu': nn.SiLU
}
return act_dict[act_layer]
class ConvNormAct(nn.Module):
def __init__(self, dim_in, dim_out, kernel_size, stride=1, dilation=1, groups=1, bias=False,
skip=False, norm_layer='bn_2d', act_layer='relu', inplace=True, drop_path_rate=0.):
super(ConvNormAct, self).__init__()
self.has_skip = skip and dim_in == dim_out
padding = math.ceil((kernel_size - stride) / 2)
self.conv = nn.Conv2d(dim_in, dim_out, kernel_size, stride, padding, dilation, groups, bias)
self.norm = get_norm(norm_layer)(dim_out)
self.act = get_act(act_layer)(inplace=inplace)
self.drop_path = DropPath(drop_path_rate) if drop_path_rate else nn.Identity()
def forward(self, x):
shortcut = x
x = self.conv(x)
x = self.norm(x)
x = self.act(x)
if self.has_skip:
x = self.drop_path(x) + shortcut
return x
class iRMB(nn.Module):
def __init__(self, dim_in, dim_out, norm_in=True, has_skip=True, exp_ratio=1.0, norm_layer='bn_2d',
act_layer='relu', v_proj=True, dw_ks=3, stride=1, dilation=1, se_ratio=0.0, dim_head=8, window_size=7,
attn_s=True, qkv_bias=False, attn_drop=0., drop=0., drop_path=0., v_group=False, attn_pre=False):
super().__init__()
self.norm = get_norm(norm_layer)(dim_in) if norm_in else nn.Identity()
dim_mid = int(dim_in * exp_ratio)
self.has_skip = (dim_in == dim_out and stride == 1) and has_skip
self.attn_s = attn_s
if self.attn_s:
assert dim_in % dim_head == 0, 'dim should be divisible by num_heads'
self.dim_head = dim_head
self.window_size = window_size
self.num_head = dim_in // dim_head
self.scale = self.dim_head ** -0.5
self.attn_pre = attn_pre
self.qk = ConvNormAct(dim_in, int(dim_in * 2), kernel_size=1, bias=qkv_bias, norm_layer='none',
act_layer='none')
self.v = ConvNormAct(dim_in, dim_mid, kernel_size=1, groups=self.num_head if v_group else 1, bias=qkv_bias,
norm_layer='none', act_layer=act_layer, inplace=inplace)
self.attn_drop = nn.Dropout(attn_drop)
else:
if v_proj:
self.v = ConvNormAct(dim_in, dim_mid, kernel_size=1, bias=qkv_bias, norm_layer='none',
act_layer=act_layer, inplace=inplace)
else:
self.v = nn.Identity()
self.conv_local = ConvNormAct(dim_mid, dim_mid, kernel_size=dw_ks, stride=stride, dilation=dilation,
groups=dim_mid, norm_layer='bn_2d', act_layer='silu', inplace=inplace)
self.se = SqueezeExcite(dim_mid, rd_ratio=se_ratio, act_layer=get_act(act_layer)) if se_ratio > 0.0 else nn.Identity()
self.proj_drop = nn.Dropout(drop)
self.proj = ConvNormAct(dim_mid, dim_out, kernel_size=1, norm_layer='none', act_layer='none', inplace=inplace)
self.drop_path = DropPath(drop_path) if drop_path else nn.Identity()
def forward(self, x):
shortcut = x
x = self.norm(x)
B, C, H, W = x.shape
if self.attn_s:
# padding
if self.window_size <= 0:
window_size_W, window_size_H = W, H
else:
window_size_W, window_size_H = self.window_size, self.window_size
pad_l, pad_t = 0, 0
pad_r = (window_size_W - W % window_size_W) % window_size_W
pad_b = (window_size_H - H % window_size_H) % window_size_H
x = F.pad(x, (pad_l, pad_r, pad_t, pad_b, 0, 0,))
n1, n2 = (H + pad_b) // window_size_H, (W + pad_r) // window_size_W
x = rearrange(x, 'b c (h1 n1) (w1 n2) -> (b n1 n2) c h1 w1', n1=n1, n2=n2).contiguous()
# attention
b, c, h, w = x.shape
qk = self.qk(x)
qk = rearrange(qk, 'b (qk heads dim_head) h w -> qk b heads (h w) dim_head', qk=2, heads=self.num_head,
dim_head=self.dim_head).contiguous()
q, k = qk[0], qk[1]
attn_spa = (q @ k.transpose(-2, -1)) * self.scale
attn_spa = attn_spa.softmax(dim=-1)
attn_spa = self.attn_drop(attn_spa)
if self.attn_pre:
x = rearrange(x, 'b (heads dim_head) h w -> b heads (h w) dim_head', heads=self.num_head).contiguous()
x_spa = attn_spa @ x
x_spa = rearrange(x_spa, 'b heads (h w) dim_head -> b (heads dim_head) h w', heads=self.num_head, h=h,
w=w).contiguous()
x_spa = self.v(x_spa)
else:
v = self.v(x)
v = rearrange(v, 'b (heads dim_head) h w -> b heads (h w) dim_head', heads=self.num_head).contiguous()
x_spa = attn_spa @ v
x_spa = rearrange(x_spa, 'b heads (h w) dim_head -> b (heads dim_head) h w', heads=self.num_head, h=h,
w=w).contiguous()
# unpadding
x = rearrange(x_spa, '(b n1 n2) c h1 w1 -> b c (h1 n1) (w1 n2)', n1=n1, n2=n2).contiguous()
if pad_r > 0 or pad_b > 0:
x = x[:, :, :H, :W].contiguous()
else:
x = self.v(x)
x = x + self.se(self.conv_local(x)) if self.has_skip else self.se(self.conv_local(x))
x = self.proj_drop(x)
x = self.proj(x)
x = (shortcut + self.drop_path(x)) if self.has_skip else x
return x
def autopad(k, p=None, d=1): # kernel, padding, dilation
"""Pad to 'same' shape outputs."""
if d > 1:
k = d * (k - 1) + 1 if isinstance(k, int) else [d * (x - 1) + 1 for x in k] # actual kernel-size
if p is None:
p = k // 2 if isinstance(k, int) else [x // 2 for x in k] # auto-pad
return p
class Conv(nn.Module):
"""Standard convolution with args(ch_in, ch_out, kernel, stride, padding, groups, dilation, activation)."""
default_act = nn.SiLU() # default activation
def __init__(self, c1, c2, k=1, s=1, p=None, g=1, d=1, act=True):
"""Initialize Conv layer with given arguments including activation."""
super().__init__()
self.conv = nn.Conv2d(c1, c2, k, s, autopad(k, p, d), groups=g, dilation=d, bias=False)
self.bn = nn.BatchNorm2d(c2)
self.act = self.default_act if act is True else act if isinstance(act, nn.Module) else nn.Identity()
def forward(self, x):
"""Apply convolution, batch normalization and activation to input tensor."""
return self.act(self.bn(self.conv(x)))
def forward_fuse(self, x):
"""Perform transposed convolution of 2D data."""
return self.act(self.conv(x))
class Bottleneck(nn.Module):
"""Standard bottleneck."""
def __init__(self, c1, c2, shortcut=True, g=1, k=(3, 3), e=0.5):
"""Initializes a bottleneck module with given input/output channels, shortcut option, group, kernels, and
expansion.
"""
super().__init__()
c_ = int(c2 * e) # hidden channels
self.cv1 = Conv(c1, c_, k[0], 1)
self.cv2 = Conv(c_, c2, k[1], 1, g=g)
self.add = shortcut and c1 == c2
self.iRMB = iRMB(c2, c2)
def forward(self, x):
"""'forward()' applies the YOLO FPN to input data."""
return x + self.iRMB(self.cv2(self.cv1(x))) if self.add else self.iRMB(self.cv2(self.cv1(x)))
class C2f_iRMB(nn.Module):
"""Faster Implementation of CSP Bottleneck with 2 convolutions."""
def __init__(self, c1, c2, n=1, shortcut=False, g=1, e=0.5):
"""Initialize CSP bottleneck layer with two convolutions with arguments ch_in, ch_out, number, shortcut, groups,
expansion.
"""
super().__init__()
self.c = int(c2 * e) # hidden channels
self.cv1 = Conv(c1, 2 * self.c, 1, 1)
self.cv2 = Conv((2 + n) * self.c, c2, 1) # optional act=FReLU(c2)
self.m = nn.ModuleList(Bottleneck(self.c, self.c, shortcut, g, k=((3, 3), (3, 3)), e=1.0) for _ in range(n))
def forward(self, x):
"""Forward pass through C2f layer."""
y = list(self.cv1(x).chunk(2, 1))
y.extend(m(y[-1]) for m in self.m)
return self.cv2(torch.cat(y, 1))
def forward_split(self, x):
"""Forward pass using split() instead of chunk()."""
y = list(self.cv1(x).split((self.c, self.c), 1))
y.extend(m(y[-1]) for m in self.m)
return self.cv2(torch.cat(y, 1))
if __name__ == "__main__":
# Generating Sample image
image_size = (1, 64, 640, 640)
image = torch.rand(*image_size)
# Model
model = iRMB(64, 64)
out = model(image)
print(len(out))
实验
脚本
import warnings
warnings.filterwarnings('ignore')
from ultralytics import YOLO
if __name__ == '__main__':
# 修改为自己的配置文件地址
model = YOLO('/root/ultralytics-main/ultralytics/cfg/models/11/yolov11-iRMB.yaml')
# 修改为自己的数据集地址
model.train(data='/root/ultralytics-main/ultralytics/cfg/datasets/coco8.yaml',
cache=False,
imgsz=640,
epochs=10,
single_cls=False, # 是否是单类别检测
batch=8,
close_mosaic=10,
workers=0,
optimizer='SGD',
amp=True,
project='runs/train',
name='iRMB',
)
结果
