Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
Q
qg-dbc-spark
Project
Project
Details
Activity
Releases
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Charts
Wiki
Wiki
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
data-spider
qg-dbc-spark
Commits
8a338355
Commit
8a338355
authored
Dec 27, 2019
by
data爬虫-冯 军凯
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
transactionLog 数据清洗 main类修改1234
parent
d7877d2f
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
2 additions
and
6 deletions
+2
-6
CleanningTransactionLogMain.java
...dbc/spark/transactionlog/CleanningTransactionLogMain.java
+2
-6
No files found.
src/main/java/cn/quantgroup/dbc/spark/transactionlog/CleanningTransactionLogMain.java
View file @
8a338355
package
cn
.
quantgroup
.
dbc
.
spark
.
transactionlog
;
package
cn
.
quantgroup
.
dbc
.
spark
.
transactionlog
;
import
cn.quantgroup.dbc.utils.JdbcExecuters
;
import
cn.quantgroup.dbc.utils.JdbcExecuters
;
import
cn.quantgroup.dbc.utils.PagingUtil
;
import
com.alibaba.fastjson.JSON
;
import
com.alibaba.fastjson.JSON
;
import
org.apache.spark.sql.Dataset
;
import
org.apache.spark.sql.Dataset
;
import
org.apache.spark.sql.SparkSession
;
import
org.apache.spark.sql.SparkSession
;
...
@@ -9,9 +8,6 @@ import org.apache.spark.sql.SparkSession;
...
@@ -9,9 +8,6 @@ import org.apache.spark.sql.SparkSession;
import
java.sql.Timestamp
;
import
java.sql.Timestamp
;
import
java.text.SimpleDateFormat
;
import
java.text.SimpleDateFormat
;
import
java.util.ArrayList
;
import
java.util.ArrayList
;
import
java.util.Arrays
;
import
java.util.List
;
import
java.util.Map
;
/**
/**
* @Author fengjunkai
* @Author fengjunkai
...
@@ -49,7 +45,7 @@ public class CleanningTransactionLogMain {
...
@@ -49,7 +45,7 @@ public class CleanningTransactionLogMain {
transactionLog
.
setUpdated_at
(
timestamp
);
transactionLog
.
setUpdated_at
(
timestamp
);
if
(
transactionLogs
.
size
()
!=
0
&&
transactionLogs
.
size
()
%
200
==
0
)
{
if
(
transactionLogs
.
size
()
!=
0
&&
transactionLogs
.
size
()
%
200
==
0
)
{
//
JdbcExecuters.prepareBatchUpdateExecuteTransactionid(sql, transactionLogs);
JdbcExecuters
.
prepareBatchUpdateExecuteTransactionid
(
sql
,
transactionLogs
);
System
.
out
.
println
(
"执行sql集合: "
+
JSON
.
toJSONString
(
transactionLogs
));
System
.
out
.
println
(
"执行sql集合: "
+
JSON
.
toJSONString
(
transactionLogs
));
transactionLogs
.
clear
();
transactionLogs
.
clear
();
}
}
...
@@ -58,7 +54,7 @@ public class CleanningTransactionLogMain {
...
@@ -58,7 +54,7 @@ public class CleanningTransactionLogMain {
e
.
printStackTrace
();
e
.
printStackTrace
();
}
}
});
});
//
JdbcExecuters.prepareBatchUpdateExecuteTransactionid(sql, transactionLogs);
JdbcExecuters
.
prepareBatchUpdateExecuteTransactionid
(
sql
,
transactionLogs
);
});
});
ss
.
stop
();
ss
.
stop
();
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment