一、需求背景
先说下客户的需求,客户是一个中学理科教师,备课时经常需要在word或者ppt里输入一些数学公式。
该客户的痛点是,如果使用word自带的公式输入器是比较费时的,尤其是复杂的公式,因为要寻找各种符号,如下图所示
于是该教师想着现在的OCR技术这么先进,应该有更快的速度获取公式的办法,是的,没错。最省力的办法就是手写输入公式或者拍照,软件快速识别出来公式,把公式复制过来到word中。vue3和微信小程序的画布就能实现手写输入,本文就是实现uniapp的canvas实现画布输入手写的公式,然后把画好的图片发给后端识别,返回latex公式。效果如下图所示:
二、UI设计
从上往下由三部分组成
- 画布
- 操作按钮:清空、撤销、识别
- 识别结果
- 完整代码在最后
三、逻辑业务
-
1.页面初始化
获取画布上下文,并获取画布的长和宽,用于后期清空时重绘画布用到,关键代码如下:
// 生命周期钩子,相当于原来的onReady
onMounted(() => {
ctx.value = uni.createCanvasContext('sign')
ctx.value.setStrokeStyle(lineColor.value)
ctx.value.setLineWidth(lineWidth.value)
// 获取canvas的宽高
const query = uni.createSelectorQuery().in(instance)
query.select('#sign').boundingClientRect()
query.exec((res) => {
if (res && res.length) {
width.value = res[0].width
height.value = res[0].height
background()
save()
}
})
})
// 设置背景色方法,对应原来的background
const background = (color = '#f8f8f8') => {
ctx.value.fillStyle = color
ctx.value.fillRect(0, 0, width.value, height.value)
ctx.value.draw()
stack_file.value.splice(0, stack_file.value.length)
formula_ketex.value.splice(0, formula_ketex.value.length)
}
其中background函数主要是绘制一个矩形,根据画布的长和宽 并利用css控制其高度;save()函数用于保存当前的画布图片到栈内,便于实现撤销操作
.draw-page {
height: 100vh;
display: flex;
flex-direction: column;
}
2.实现画布手写输入
主要依靠canvas的touchmove事件,实时绘制画笔在屏幕上移动的路径线段,关键代码如下:
// 触摸移动方法,对应原来的touchmove
const touchmove = (e) => {
if (e.touches && e.touches.length) {
const t = e.touches[0]
pmouseX.value = mouseX.value
pmouseY.value = mouseY.value
mouseX.value = t.x
mouseY.value = t.y
draw()
}
}
// 绘制方法,对应原来的draw
const draw = () => {
ctx.value.moveTo(pmouseX.value, pmouseY.value)
ctx.value.lineTo(mouseX.value, mouseY.value)
ctx.value.stroke()
ctx.value.draw(true)
}
3.撤销操作
其原理是当画笔结束绘制后,把当前画布里的图片保存到栈内,需要撤销时,从栈顶取出图片,重新绘制到画布里,即实现了撤销操作。关键代码如下: // 恢复操作
const undo = () => {
if (stack_file.value.length > 0) {
const lastSavedUrl = stack_file.value.pop() // 取出栈顶(上次保存的)元素
const canvasWidth = width.value
const canvasHeight = height.value
// 设置图片绘制的起始坐标等,可按需调整,这里从坐标原点开始绘制覆盖整个canvas
const x = 0
const y = 0
ctx.value.drawImage(lastSavedUrl, x, y, canvasWidth, canvasHeight)
ctx.value.draw()
}
}
4.识别公式
该部分的逻辑是把栈顶的图片取出来,发送给后端,以base64格式发送(这是后端的要求格式),后端返回latex公式,后端用python写的,这里就不展示了。有兴趣的留下邮箱。postFormulaOcrAPI(params)是后端接口的封装,关键代码如下:
// 发送图片给后台
const postImage = async () => {
const params = {
base64_str: base64_str.value,
}
console.log('params:', params)
const res = await postFormulaOcrAPI(params)
console.log('提交图片', res)
if (res.code === 200) {
uni.showToast({ icon: 'success', title: '识别成功' })
formula_visible.value = true
formula_result.value = res.data.results
formula_ketex.value.push('$$' + formula_result.value + '$$')
} else {
uni.showToast({ icon: 'error', title: '识别错误' })
}
}
5.下载word文件
当latex识别成功后,前端把latex作为参数发给后端,后端返回word文件,/questions/latex_2_word/是后端的接口,关键代码如下:
// 请求word文件
const postLatexToWord = () => {
if (formula_result.value == ''){
return 0
}
const params = {
latex_str: formula_result.value,
}
uni.request({
url: '/questions/latex_2_word/',
method: 'POST',
data: params,
responseType: 'arraybuffer',
success: (res) => {
uni.hideLoading()
console.log('服务器返回结果', res)
if (res.statusCode === 200) {
const fs = uni.getFileSystemManager()
fs.writeFile({
filePath: wx.env.USER_DATA_PATH + '/download.docx',
data: res.data,
encoding: 'binary',
success(res) {
uni.openDocument({
showMenu: true,
filePath: wx.env.USER_DATA_PATH + '/download.docx',
success: function (res) {
console.log('打开文档成功!')
},
})
},
})
}
},
fail: (err) => {
uni.hideLoading()
console.log(err)
uni.showToast({
icon: 'none',
mask: true,
title: '失败请重新下载',
})
},
})
}
完整代码
<template>
<view class="draw-page">
<view class="draw-content">
<canvas
style="width: 100%; height: 100%"
ref="sign"
canvas-id="sign"
id="sign"
disable-scroll
@touchstart="touchstart"
@touchmove="touchmove"
@touchend="touchend"
></canvas>
</view>
<view class="actions">
<button @click="background()" type="warn" size="mini">清空</button>
<button @click="undo" type="primary" size="mini">恢复</button>
<button @click="submit" type="warn" size="mini">识别</button>
</view>
<view class="formula" v-show="formula_visible">
<input
class="uni-input"
ref="formula_result"
:value="formula_result"
placeholder="latex识别结果"
style="width: 100%;"
@input="inputLatex"
/>
<MathCanvas :mathFormula="formula_ketex"></MathCanvas>
<button @click="postLatexToWord" type="primary" size="mini">下载word</button>
</view>
</view>
</template>
<script setup>
import { onMounted, ref, getCurrentInstance, computed } from 'vue'
import { postFormulaOcrAPI } from '@/services/home'
import MathCanvas from '@/components/MathCanvas.vue'
// 定义响应式数据
const lineColor = ref('#000') // 线条颜色
const lineWidth = ref(2) // 线条宽度
const width = ref(0)
const height = ref(0)
const ctx = ref(null)
const mouseX = ref(0)
const mouseY = ref(0)
const pmouseX = ref(0)
const pmouseY = ref(0)
const instance = getCurrentInstance()
const stack_file = ref([])
const base64_str = ref('')
const formula_visible = ref(false)
const formula_result = ref('')
const formula_ketex = ref([])
// 输入latex,修改ketex的值动态修改公式
const inputLatex = (e) => {
formula_result.value = e.detail.value
console.log('formula_result', formula_result.value)
formula_ketex.value[formula_ketex.value.length - 1] = '$$' + formula_result.value + '$$'
}
// 生命周期钩子,相当于原来的onReady
onMounted(() => {
ctx.value = uni.createCanvasContext('sign')
ctx.value.setStrokeStyle(lineColor.value)
ctx.value.setLineWidth(lineWidth.value)
// 获取canvas的宽高
const query = uni.createSelectorQuery().in(instance)
query.select('#sign').boundingClientRect()
query.exec((res) => {
if (res && res.length) {
width.value = res[0].width
height.value = res[0].height
background()
save()
}
})
})
// 提交给后台进行识别
const submit = () => {
if (stack_file.value.length > 0) {
formula_ketex.value = formula_ketex.value.splice(0, formula_ketex.value.length)
const fileTempPath = stack_file.value[stack_file.value.length - 1]
console.log('栈顶元素:', fileTempPath)
uni.getFileSystemManager().readFile({
filePath: fileTempPath,
encoding: 'base64',
success: (data) => {
base64_str.value = data.data
console.log('base64_str:', base64_str.value)
postImage()
},
fail: (err) => {
console.log('error:', err)
return 0
},
})
}
}
// 发送图片给后台
const postImage = async () => {
const params = {
base64_str: base64_str.value,
}
console.log('params:', params)
const res = await postFormulaOcrAPI(params)
console.log('提交图片', res)
if (res.code === 200) {
uni.showToast({ icon: 'success', title: '识别成功' })
formula_visible.value = true
formula_result.value = res.data.results
formula_ketex.value.push('$$' + formula_result.value + '$$')
} else {
uni.showToast({ icon: 'error', title: '识别错误' })
}
}
// 恢复操作
const undo = () => {
if (stack_file.value.length > 0) {
const lastSavedUrl = stack_file.value.pop() // 取出栈顶(上次保存的)元素
const canvasWidth = width.value
const canvasHeight = height.value
// 设置图片绘制的起始坐标等,可按需调整,这里从坐标原点开始绘制覆盖整个canvas
const x = 0
const y = 0
ctx.value.drawImage(lastSavedUrl, x, y, canvasWidth, canvasHeight)
ctx.value.draw()
}
}
// 保存方法,对应原来的save
const save = () => {
uni.canvasToTempFilePath({
canvasId: 'sign',
success(res) {
// h5 tempFilePath是base64
const url = res.tempFilePath
stack_file.value.push(url)
// console.log(url)
},
fail(err) {
console.error(err)
},
})
}
// 设置背景色方法,对应原来的background
const background = (color = '#f8f8f8') => {
ctx.value.fillStyle = color
ctx.value.fillRect(0, 0, width.value, height.value)
ctx.value.draw()
stack_file.value.splice(0, stack_file.value.length)
formula_ketex.value.splice(0, formula_ketex.value.length)
}
// 触摸开始方法,对应原来的touchstart
const touchstart = (e) => {
if (e.touches && e.touches.length) {
const t = e.touches[0]
mouseX.value = t.x
mouseY.value = t.y
pmouseX.value = mouseX.value
pmouseY.value = mouseY.value
}
}
// 触摸结束,当前画面保存到栈里面
const touchend = (e) => {
// console.log(e)
save()
}
// 触摸移动方法,对应原来的touchmove
const touchmove = (e) => {
if (e.touches && e.touches.length) {
const t = e.touches[0]
pmouseX.value = mouseX.value
pmouseY.value = mouseY.value
mouseX.value = t.x
mouseY.value = t.y
draw()
}
}
// 绘制方法,对应原来的draw
const draw = () => {
ctx.value.moveTo(pmouseX.value, pmouseY.value)
ctx.value.lineTo(mouseX.value, mouseY.value)
ctx.value.stroke()
ctx.value.draw(true)
}
// 请求word文件
const postLatexToWord = () => {
if (formula_result.value == ''){
return 0
}
const params = {
latex_str: formula_result.value,
}
uni.request({
url: '/questions/latex_2_word/',
method: 'POST',
data: params,
responseType: 'arraybuffer',
success: (res) => {
uni.hideLoading()
console.log('服务器返回结果', res)
if (res.statusCode === 200) {
const fs = uni.getFileSystemManager()
fs.writeFile({
filePath: wx.env.USER_DATA_PATH + '/download.docx',
data: res.data,
encoding: 'binary',
success(res) {
uni.openDocument({
showMenu: true,
filePath: wx.env.USER_DATA_PATH + '/download.docx',
success: function (res) {
console.log('打开文档成功!')
},
})
},
})
}
},
fail: (err) => {
uni.hideLoading()
console.log(err)
uni.showToast({
icon: 'none',
mask: true,
title: '失败请重新下载',
})
},
})
}
</script>
<style lang="scss">
.draw-page {
height: 100vh;
display: flex;
flex-direction: column;
}
.draw-content {
flex: 1;
}
.actions {
display: flex;
button {
flex: 1;
}
}
.formula {
// height: 25vh;
width: 100%;
display: flex;
flex-direction: column;
}
</style>