五种方法比较
sessionStorage 只对当前页面有效,关闭页面自动清理数据,但是可以通过恢复标签页找回。 localStorage 是持久存储,但存在大小限制。 indexDB 是数据库,但会在存储空间不足时自动清空。 书签名称适合用来存储简短的数据,比如存放当前页码,“书签”更名符其实。 外部数据库适合大量持久存储数据,类似indexedDB但永不删除,支持多索引排序。 油猴脚本自带的 GM_setValue 的后端是 levelDB,适合大量键值对的存放与读取。
其中,sessionStorage 虽然是临时数据,但保持标签开启就不会消失,如果配合tab持久化扩展,那么使用效果更佳。临时不临时,全在用户一念之间。
localStorage、indexDB 虽说是存储到磁盘,但都不太靠谱。GM_setValue 背后的 levelDB 是谷歌开发的开源项目。 跨浏览器不太方便。
“书签存储”由用户管理,新建书签代表需要存储页码,删除书签,那么页码数等据随之而去。书签需要全部读取到内存,所以书签很多时,导致新建删除书签有延时。外部数据库由完整 sqlite 数据库支持,适合自定扩展逻辑,比如还可用于下载音视频图片、导出某网站内部的历史记录等等,性能稳定,安全有效。 可跨浏览器使用。
“书签存储”、外部数据库是我发明的黑科技,专治浏览器各种不服。
以下就是 chrome 浏览器油猴脚本超长时间持久存储临时数据的办法五种的精彩内容,永不过期 ——
一、sessionStorage
方法一,sessionStorage,会话存储。和tab与网址关联,刷新、前进后退、重启、关闭标签页不会丢失数据,打开新的标签页或站点才开启新数据。
// 存储数据
sessionStorage.setItem('key', 'value');
// 获取数据
var data = sessionStorage.getItem('key');
// 删除数据
sessionStorage.removeItem('key');
二、localStorage、indexedDB
方法二,localStorage,本地存储。还有 indexedDB。需要注意的是,根据浏览器的不同,localStorage 存在大小限制,而 indexedDB 可能会在c盘空间不足时,自动清除全部数据,故此此类存储需要仔细测试后,才能放给用户使用。
// 存储数据
localStorage.setItem('key', 'value');
// 获取数据
var data = localStorage.getItem('key');
// 删除数据
localStorage.removeItem('key');
三、书签名称
方法三,将数据存储在书签名称之中。用户脚本没有相关api,因为用户脚本需要兼容各个浏览器,而浏览器扩展的书签api可能大相径庭。
那么如何在脚本中将一些短的数据永久存储在书签之中呢?可配合相关chrome扩展使用,涉及chrome 扩展背景页面与内容页面的通信,需要通过sendMessage与窗口监听器实现。需要注意的是,sendMessage 传的数据必须可序列化。
// content 向背景页面发送消息
chrome.runtime.sendMessage({ data: 'value' });
// 在背景页面监听消息
chrome.runtime.onMessage.addListener(function(message, sender, sendResponse) {
// 处理接收到的消息
// 各种处理,然后通过 sendResponse ,或 dispatchEvent 回调
});
content.js 运行于每个页面,可以直接在其中调用 扩展api, 向背景页面发送数据,然后回调。
content.js :
window.addEventListener("my_bookmark_event", function(evt){
chrome.runtime.sendMessage("my-event-xxx", function(data) {
if(e.detail && e.detail.cb) {
var backEvt = new CustomEvent(e.detail.cb, {detail : data});
window.dispatchEvent(backEvt, function(data) { });;
}
});
});
content.js :
window.addEventListener("my_bookmark_handler", function(evt){
console.log("my_bookmark_handler:", evt, evt.detail);
chrome.runtime.sendMessage({
url:evt.detail.url
,name:evt.detail.name
,update:evt.detail.update
,data:evt.detail
,get:evt.detail.get
,type:"bookmark-handler"
}, function(data) { });
});
后台消息处理示例:
// 后台消息处理
chrome.runtime.onMessage.addListener(function(message, sender, sendResponse) {
if (message && message.type) {
sender.pageUrl = sender.url;
if(message.type == 'bookmark-handler'){
if(message.get) { // 搜寻已有书签,并回调
chrome.bookmarks.search({'url':message.url},
function(bookmark_array) {
// 如果有结果,
……
// 回调方法一
sendResponse(...);
// 回调方法二
chrome.tabs.executeScript(sender.tab.id, {
code: `var backEvt = new CustomEvent("${message.data.cb}", {detail : ${JSON.stringify(e)}});
window.dispatchEvent(backEvt, function(data) { });`
}, () => {});
}
);
} else if(message.update) {
// 同理,当已存在书签时,更新而非重复创建。
} else {
// 同理,chrome允许创建重复url的书签。
}
}
}
});
四、外部数据库
比如java,就可以用jdbc驱动sqlite数据库,然后开启微型服务器,持久化存储信息。
以下是我造的轮子,名之曰FFDB:
/** Sqlite Database helper */
public class FFDB {
static FFDB instance;
final Connection database;
Statement statement;
HashMap<String, PreparedStatement> preparedStatements = new HashMap<>();
static final HashMap<String, FHDB> FHDBs = new HashMap<>();
ExecutorService threadPool = Executors.newFixedThreadPool(5);
boolean bUniqueTable;
public static FFDB getInstance() {
if (instance==null) {
synchronized (FFDB.class) {
try {
if (instance==null)
instance = new FFDB();
} catch (SQLException e) {
CMN.Log(e);
}
}
}
return instance;
}
FFDB() throws SQLException {
SQLiteConfig config = new SQLiteConfig();
config.setSharedCache(true);
database = DriverManager.getConnection("jdbc:sqlite:D:/sample.db", config.toProperties());
statement=database.createStatement();
database.setAutoCommit(false);
}
void getInitMap(JSONObject json, StringBuilder sb) {
Object obj;
int idx=0;
Set<String> keys = json.keySet();
Iterator<String> iter = keys.iterator();
String key;
boolean ftd = false;
while (iter.hasNext()) {
key=iter.next();
obj = json.get(key);
if (ftd) {
sb.append(" , ");
} else {
ftd = true;
}
idx++;
if (key.equals("rowId")) {
sb.append("id INTEGER PRIMARY KEY AUTOINCREMENT");
}
else if (obj instanceof Integer
|| obj instanceof Long
|| obj instanceof Short
) {
sb.append(key).append(" INTEGER");
if (IU.parsint(obj+"", -1)!=-1) {
sb.append(" DEFAULT ").append(obj);
}
}
else if (obj instanceof String) {
sb.append(key).append(" TEXT");
if (!obj.toString().equals("-1")) {
sb.append(" DEFAULT ").append("'").append(obj).append("'");
}
}
else {
CMN.Log("getInitMap::不支持::", idx, obj);
}
}
}
public void openTable(String table, JSONObject map, JSONObject indexed) throws SQLException {
StringBuilder sb = new StringBuilder();
sb.append("CREATE TABLE IF NOT EXISTS ")
.append(table)
.append("(");
getInitMap(map, sb);
String sql = sb.append(")").toString();
CMN.Log("openTable::sql::", sql);
statement.execute(sql);
if (indexed!=null) {
Iterator<String> iter = indexed.keySet().iterator();
String key;
boolean ftd = false;
while (iter.hasNext()) {
key=iter.next();
sql="CREATE "+(IU.parsint(indexed.getInteger(key), 0)==1?" UNIQUE ":"")+"INDEX if not exists "+table+"_"+key+"_index ON "+table+"("+key+")";
statement.execute(sql);
}
}
}
void getValNames(JSONObject json, StringBuffer sb, boolean kuohao) {
Set<String> keys = json.keySet();
Iterator<String> iter = keys.iterator();
String key;
if(kuohao) sb.append("(");
boolean ftd = false;
while (iter.hasNext()) {
key=iter.next();
if (ftd) {
sb.append(",");
} else {
ftd = true;
}
sb.append(key);
}
if(kuohao) sb.append(")");
}
void getValMaps(JSONObject json, StringBuffer sb) {
Set<String> keys = json.keySet();
Iterator<String> iter = keys.iterator();
String key;
boolean ftd = false;
while (iter.hasNext()) {
key=iter.next();
if (ftd) {
sb.append(" & ");
} else {
ftd = true;
}
sb.append(key)
.append("=");
Object obj = json.get(key);
if (true) {
sb.append("?");
} else {
if (obj instanceof Integer
|| obj instanceof Long
|| obj instanceof Short
) {
sb.append(obj);
}
else if (obj instanceof String) {
sb.append("'").append(obj).append("'");
}
else {
CMN.Log("setValues::不支持::", key, obj);
}
}
}
}
void getValues(ResultSet set, JSONObject json) throws SQLException {
Iterator<String> iter = json.keySet().iterator();
Object obj;
int idx=0;
String key;
while (iter.hasNext()) {
key=iter.next();
idx++;
json.put(key, set.getObject(idx));
}
}
void setValues(PreparedStatement prepared, JSONObject json) throws SQLException {
prepared.clearParameters();
Collection<Object> vals = json.values();
Iterator<Object> iter = vals.iterator();
Object obj;
int idx=0;
while (iter.hasNext()) {
obj=iter.next();
idx++;
if (obj instanceof Integer) {
prepared.setInt(idx, (Integer) obj);
}
else if (obj instanceof Long) {
prepared.setLong(idx, (Long) obj);
}
else if (obj instanceof String) {
prepared.setString(idx, (String) obj);
}
else if (obj instanceof Blob) {
prepared.setBlob(idx, (Blob) obj);
} else {
CMN.Log("setValues::不支持::", idx, obj);
}
}
}
public void putBatch(String table, JSONArray array) {
try {
//statement.execute("insert into TEST(rid, fav) VALUES("+rid+", "+fav+")");
JSONObject json = array.getJSONObject(0);
StringBuffer sb = new StringBuffer();
sb.append("REPLACE INTO ").append(table);
getValNames(json, sb, true);
sb.append(" VALUES(");
boolean ftd = false;
for (int i = 0; i < json.size(); i++) {
if (ftd) {
sb.append(",");
} else {
ftd = true;
}
sb.append("?");
}
sb.append(")");
String sql = sb.toString();
synchronized (database) {
PreparedStatement prepared = preparedStatements.get(sql);
if (prepared==null || prepared.isClosed()) {
preparedStatements.put(sql, prepared=database.prepareStatement(sql));
}
int cc= 0;
try {
prepared.clearBatch();
cc = 0;
for (int i = 0; i < array.size(); i++) {
json = array.getJSONObject(i);
prepared.clearParameters();
setValues(prepared, json);
prepared.addBatch();
if (i%1024==0) {
database.commit();
cc += prepared.executeBatch().length;
prepared.clearBatch();
}
}
cc += prepared.executeBatch().length;
} catch (SQLException e) {
CMN.Log(e);
database.rollback();
}
CMN.Log("executeBatch::", cc, array.size());
}
} catch (Exception e) {
CMN.Log(e);
}
}
public void put(String table, JSONObject json) {
try {
//statement.execute("insert into TEST(rid, fav) VALUES("+rid+", "+fav+")");
StringBuffer sb = new StringBuffer();
sb.append("REPLACE INTO ").append(table);
getValNames(json, sb, true);
sb.append(" VALUES(");
boolean ftd = false;
for (int i = 0; i < json.size(); i++) {
if (ftd) {
sb.append(",");
} else {
ftd = true;
}
sb.append("?");
}
sb.append(")");
String sql = sb.toString();
synchronized (database) {
PreparedStatement prepared = preparedStatements.get(sql);
if (prepared==null || prepared.isClosed()) {
preparedStatements.put(sql, prepared=database.prepareStatement(sql));
}
try {
setValues(prepared, json);
prepared.execute();
database.commit();
} catch (SQLException e) {
CMN.Log(e);
database.rollback();
}
}
} catch (Exception e) {
CMN.Log(e);
}
}
public JSONObject get(String table, JSONObject ret, JSONObject where) {
//try {
// //ResultSet set = statement.executeQuery("select fav from TEST where rid="+rid+" limit 1");
// synchronized (database) {
// PreparedStatement prepared = database.prepareStatement("select fav from TEST where rid=? limit 1");
// prepared.setInt(1, rid);
// ResultSet set = prepared.executeQuery();
// if (set.next()) {
// defVal = set.getInt(1);
// }
// set.close();
// }
//} catch (Exception e) {
// CMN.Log(e);
//}
//return defVal;
try {
//ResultSet set = statement.executeQuery("select fav from TEST where rid="+rid+" limit 1");
StringBuffer sb = new StringBuffer();
sb.append("select ");
getValNames(ret, sb, false); // 模板
sb.append(" FROM ").append(table).append(" WHERE ");
getValMaps(where, sb);
sb.append(" limit 1");
String sql = sb.toString();
CMN.Log("get::sql::", sql);
synchronized (database) {
PreparedStatement prepared = preparedStatements.get(sql);
if (prepared==null || prepared.isClosed()) {
prepared=database.prepareStatement(sql);
}
setValues(prepared, where);
ResultSet set = prepared.executeQuery();
if (set.next()) {
getValues(set, ret);
}
set.close();
}
} catch (Exception e) {
CMN.Log(e);
}
return ret;
}
public JSONArray getBatch(String table, JSONObject temp, JSONArray array) {
JSONObject where = array.getJSONObject(0);
JSONArray ret = new JSONArray();
try {
//ResultSet set = statement.executeQuery("select fav from TEST where rid="+rid+" limit 1");
StringBuffer sb = new StringBuffer();
sb.append("select ");
getValNames(temp, sb, false); // 模板
sb.append(" FROM ").append(table).append(" WHERE ");
getValMaps(where, sb);
sb.append(" limit 1");
String sql = sb.toString();
CMN.Log("getBatch::sql::", sql);
synchronized (database) {
PreparedStatement prepared = preparedStatements.get(sql);
if (prepared==null || prepared.isClosed()) {
prepared=database.prepareStatement(sql);
}
for (int i = 0; i < array.size(); i++) {
where = array.getJSONObject(i);
setValues(prepared, where);
ResultSet set = prepared.executeQuery();
if (set.next()) {
JSONObject templet = (JSONObject) temp.clone();
getValues(set, templet);
ret.add(templet);
}
set.close();
}
}
} catch (Exception e) {
CMN.Log(e);
}
CMN.Log("getBatch::", ret.size(), array.size());
return ret;
}
public void downloadUnique(JSONObject temp) {
CMN.Log("downloadUnique::", temp);
// filename: ""
// finalUrl: "https://"
// referrer: "https://"
String filename = temp.getString("filename");
String finalUrl = temp.getString("finalUrl");
int idx = finalUrl.indexOf("?");
if (idx>0) finalUrl = finalUrl.substring(0, idx);
String referrer = temp.getString("referrer");
if (!bUniqueTable) {
try {
openTable("files", FFDB.parseObject("{url:'', name:'', fav:0}"), FFDB.parseObject("{url:1}"));
bUniqueTable = true;
} catch (SQLException ignored) { }
}
JSONObject ret = new JSONObject();
ret.put("name", null);
get("files", ret, FFDB.parseObject("{url:'"+finalUrl+"'}"));
if (ret.get("name") != null) {
CMN.Log("已经下载过了!");
} else {
CMN.Log("下载…");
ret.put("name", filename);
ret.put("url", finalUrl);
ret.put("fav", 0);
threadPool.execute(new Runnable() {
@Override
public void run() {
try {
DownloadInfo info = new DownloadInfo(new URL(temp.getString("finalUrl")));
info.setReferer(new URL(temp.getString("referrer")));
String path = "C:\\Users\\Admin\\Downloads\\video";
String fn = filename;
String fix = ".mp4";
int idx = fn.lastIndexOf(".");
if (idx>0) {
fix = fn.substring(idx);
fn = fn.substring(0, idx);
}
File file = new File(path+fn+fix);
int cc=0;
while (file.exists()) {
file = new File(path+fn+"."+(++cc)+fix);
}
WGet wGet = new WGet(info, file);
wGet.download();
put("files", ret);
} catch (Exception e) {
CMN.Log(e);
}
}
});
}
}
public static JSONObject parseObject(String text) {
if (text==null) {
return null;
}
boolean nord = (JSON.DEFAULT_PARSER_FEATURE & Feature.OrderedField.getMask())==0;
JSON.DEFAULT_PARSER_FEATURE |= Feature.OrderedField.getMask();
JSONObject ret = JSONObject.parseObject(text);
if (nord) {
JSON.DEFAULT_PARSER_FEATURE &= ~Feature.OrderedField.getMask();
}
return ret;
}
static class FHDB{
final String name;
final HashSet<String> FHash = new HashSet<>();
final StringBuilder FStr = new StringBuilder();
boolean FHashPrepared;
FHDB(String name) {
this.name = name;
}
public void prepareHashset() {
if (!FHashPrepared) {
synchronized (FHash) {
if (!FHashPrepared) {
FHashPrepared = true;
try {
String key;
BufferedReader reader = new BufferedReader(new FileReader("D:\\"+name+".txt"));
while ((key=reader.readLine())!=null) {
FStr.append(key).append("\n");
FHash.add(key);
}
reader.close();
} catch (Exception e) {
CMN.Log(e);
}
}
}
}
}
public void putHashSet(JSONArray array) {
prepareHashset();
ArrayList<String> toSave = new ArrayList<>(array.size());
for (int i = 0; i < array.size(); i++) {
String key = array.getString(i);
if (!FHash.contains(key)) {
toSave.add(key);
}
}
if (toSave.size()>0) {
synchronized (FHash) {
try {
BufferedOutputStream output = new BufferedOutputStream(new FileOutputStream("D:\\"+name+".txt", true));
for (String key:toSave) {
FHash.add(key);
FStr.append(key).append("\n");
output.write(key.getBytes(StandardCharsets.UTF_8));
output.write("\r\n".getBytes(StandardCharsets.UTF_8));
}
output.close();
} catch (Exception e) {
CMN.Log(e);
}
}
}
}
public String getHashSet() {
prepareHashset();
return FStr.toString();
}
}
public static Response handleRequest(HTTPSession session) {
if (Method.POST.equals(session.getMethod())) {
try {
session.parseBody(null);
//SU.Log("DB.jsp::", session.getHeaders());
SU.Log("DB.jsp::", session.getParameters(), session.getMethod());
FFDB db = FFDB.getInstance();
// String text = session.getParameter("data");
String text = session.getParms().get("data");
//SU.Log("text::", text, text.length());
//text = URLDecoder.decode(text);
Objects.requireNonNull(text);
JSONObject data;
if (text.startsWith("[")) {
data = JSONArray.parseArray(text).getJSONObject(0);
} else {
data = FFDB.parseObject(text);
}
if (data.containsKey("dwnld")) {
getInstance().downloadUnique(data);
} else {
JSONArray fSet = data.getJSONArray("fSet");
String tableName = data.getString("table");
if (fSet != null) {
if (tableName==null) {
tableName = "sample";
}
if (tableName.startsWith("sample") && !tableName.contains("\\") && !tableName.contains("/")) {
FHDB fSetter;
synchronized (FHDBs) {
fSetter = FHDBs.get(tableName);
if (fSetter==null) {
fSetter = new FHDB(tableName);
}
}
if (fSet.size() == 0) {
return newFixedLengthResponse(fSetter.getHashSet()) ;
} else {
fSetter.putHashSet(fSet);
}
}
}
else {
//CMN.Log(data);
JSONObject json = data.getJSONObject("json");
JSONArray batch = data.getJSONArray("batch");
JSONObject where = data.getJSONObject("where");
JSONObject indexed = data.getJSONObject("indexed");
if (json==null && batch!=null) {
json = batch.getJSONObject(0);
}
CMN.Log(tableName, json, where, indexed);
Objects.requireNonNull(tableName);
Objects.requireNonNull(json);
if (indexed != null) { // open
db.openTable(tableName, json, indexed);
}
else if (where != null) { // get
String ret;
if (batch != null) {
JSONArray result = db.getBatch(tableName, json, batch);
ret = result.toString();
CMN.Log("getBatch::", result.size(), json.size());
} else {
ret = db.get(tableName, json, where).toString();
}
return newFixedLengthResponse(ret);
}
else { // set
if (batch != null) {
db.putBatch(tableName, batch);
} else {
db.put(tableName, json);
}
}
}
}
} catch (Exception e) {
CMN.Log(e);
}
}
return emptyResponse;
}
}
还包括了一个简易下载器,同样存储下载过的文件名到数据库,避免重复下载。
nanohttpd中使用:
@Override
public Response handle(HTTPSession session) throws IOException {
String uri = session.getUri();
...
if(key.startsWith("\\DB.jsp"))
return app.handleFFDB(session);
}
五、用户脚本自带功能 GM_setValue
GM_setValue("my_key", {my_val:1});
var myVal = GM_getValue("my_key", {});
需要权限 @require GM_setValue。数据存储由脚本管理器管理,一般是 LevelDB ,但也可能是 sqlite。