为了提高10万封邮件的同步效率,我们可以考虑如下方法:
- 异步批量处理:邮件同步过程中,使用异步批量处理,分段接收邮件,减少一次性拉取的负载。
- 多线程并发:使用多线程,提高批量处理时的并发效率。
- 分页加载与缓存:将邮件分批存储到数据库,缓存处理过的数据,减少重复同步请求。
- WebSocket 实时反馈:通过WebSocket通知用户实时同步进度。
基于此设计一个场景:
场景描述:
用户首次使用我们的应用时,需要将之前的10万封历史邮件从服务器同步到本地应用。用户点击“同步”按钮后,系统会逐步、分批加载邮件,用户界面实时显示同步进度。我们使用Spring Boot来开发这个邮件同步系统。
项目结构
.
├── controller
│ └── MailSyncController.java // 控制器:启动同步操作并返回进度
├── service
│ ├── MailSyncService.java // 服务层:管理同步和进度
│ └── MailFetchService.java // 服务层:邮件获取
├── repository
│ └── MailRepository.java // 数据访问层:持久化邮件
├── model
│ └── Mail.java // 模型层:邮件实体
└── config
└── WebSocketConfig.java // 配置WebSocket
1. 邮件实体类 Mail.java
package com.example.demo.model;
import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.GenerationType;
import javax.persistence.Id;
@Entity
public class Mail {
@Id
@GeneratedValue(strategy = GenerationType.IDENTITY)
private Long id;
private String sender;
private String receiver;
private String subject;
private String content;
private String receivedDate;
// getters and setters
}
2. 数据访问层 MailRepository.java
package com.example.demo.repository;
import com.example.demo.model.Mail;
import org.springframework.data.jpa.repository.JpaRepository;
import org.springframework.stereotype.Repository;
@Repository
public interface MailRepository extends JpaRepository<Mail, Long> {
}
3. 邮件获取服务 MailFetchService.java
模拟从远程邮箱服务器获取邮件数据,加入多线程和批处理逻辑:
package com.example.demo.service;
import com.example.demo.model.Mail;
import org.springframework.stereotype.Service;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.Executors;
import java.util.concurrent.ThreadPoolExecutor;
@Service
public class MailFetchService {
private static final int BATCH_SIZE = 1000; // 每次同步1000封
private ThreadPoolExecutor executor = (ThreadPoolExecutor) Executors.newFixedThreadPool(10);
public List<Mail> fetchMails(int batchNumber) {
// 模拟从服务器批量获取邮件
List<Mail> mails = new ArrayList<>();
for (int i = 0; i < BATCH_SIZE; i++) {
Mail mail = new Mail();
mail.setSender("sender@example.com");
mail.setReceiver("receiver@example.com");
mail.setSubject("Subject " + (batchNumber * BATCH_SIZE + i));
mail.setContent("Content of the mail " + (batchNumber * BATCH_SIZE + i));
mail.setReceivedDate("2023-01-01");
mails.add(mail);
}
return mails;
}
public CompletableFuture<List<Mail>> fetchMailsAsync(int batchNumber) {
return CompletableFuture.supplyAsync(() -> fetchMails(batchNumber), executor);
}
}
4. 邮件同步服务 MailSyncService.java
管理邮件批量存储和进度更新,并使用WebSocket通知用户同步进度。
package com.example.demo.service;
import com.example.demo.model.Mail;
import com.example.demo.repository.MailRepository;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.messaging.simp.SimpMessagingTemplate;
import org.springframework.stereotype.Service;
import java.util.List;
import java.util.concurrent.atomic.AtomicInteger;
@Service
public class MailSyncService {
private static final int TOTAL_BATCHES = 100; // 总批次,10万封,每次1000封
private AtomicInteger progress = new AtomicInteger(0);
@Autowired
private MailFetchService mailFetchService;
@Autowired
private MailRepository mailRepository;
@Autowired
private SimpMessagingTemplate messagingTemplate;
public void startSync() {
for (int i = 0; i < TOTAL_BATCHES; i++) {
final int batchNumber = i;
mailFetchService.fetchMailsAsync(batchNumber).thenAccept(mails -> {
mailRepository.saveAll(mails);
int currentProgress = progress.incrementAndGet();
messagingTemplate.convertAndSend("/topic/progress", currentProgress * 100 / TOTAL_BATCHES);
});
}
}
}
5. WebSocket 配置 WebSocketConfig.java
配置 WebSocket,使前端能接收到同步进度更新。
package com.example.demo.config;
import org.springframework.context.annotation.Configuration;
import org.springframework.messaging.simp.config.MessageBrokerRegistry;
import org.springframework.web.socket.config.annotation.EnableWebSocketMessageBroker;
import org.springframework.web.socket.config.annotation.StompEndpointRegistry;
import org.springframework.web.socket.config.annotation.WebSocketMessageBrokerConfigurer;
@Configuration
@EnableWebSocketMessageBroker
public class WebSocketConfig implements WebSocketMessageBrokerConfigurer {
@Override
public void configureMessageBroker(MessageBrokerRegistry config) {
config.enableSimpleBroker("/topic");
config.setApplicationDestinationPrefixes("/app");
}
@Override
public void registerStompEndpoints(StompEndpointRegistry registry) {
registry.addEndpoint("/ws").setAllowedOrigins("*").withSockJS();
}
}
6. 控制器 MailSyncController.java
启动同步并返回初始状态。
package com.example.demo.controller;
import com.example.demo.service.MailSyncService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RestController;
@RestController
public class MailSyncController {
@Autowired
private MailSyncService mailSyncService;
@PostMapping("/startSync")
public String startSync() {
mailSyncService.startSync();
return "Sync started";
}
}
7. 前端(示例 HTML)
通过WebSocket实时显示同步进度。
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Mail Sync Progress</title>
<script src="https://cdn.jsdelivr.net/sockjs/1.1.4/sockjs.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/stompjs@2.3.3/lib/stomp.min.js"></script>
</head>
<body>
<button onclick="startSync()">Start Sync</button>
<p>Progress: <span id="progress">0</span>%</p>
<script>
let stompClient = null;
function connect() {
const socket = new SockJS('/ws');
stompClient = Stomp.over(socket);
stompClient.connect({}, function () {
stompClient.subscribe('/topic/progress', function (message) {
document.getElementById('progress').innerText = message.body;
});
});
}
function startSync() {
fetch('/startSync', { method: 'POST' });
connect();
}
</script>
</body>
</html>
说明
- 分批处理:通过分批获取和存储邮件,控制同步速度和服务器负载。
- 异步执行:异步调用
fetchMailsAsync
,避免阻塞主线程,提升同步效率。 - WebSocket进度更新:实时推送同步进度到前端,改善用户体验。
以上代码就是一个简单的邮件同步场景案例。