-
-
Notifications
You must be signed in to change notification settings - Fork 11.4k
提升失败任务日志的扫描效率 #3443
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
提升失败任务日志的扫描效率 #3443
Conversation
|
考虑到长耗时任务,失败任务id不是绝对递增的,不可以简单通过最小id过滤。 可以通过时间过滤,一会扫描近期增量数据,比如3天内。 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR introduces cursor-based pagination to improve the scanning efficiency of failed task logs by adding a lastJobLogId parameter to avoid repeatedly scanning the entire log table (which can contain millions of records with 30-day retention).
Key Changes:
- Added
lastJobLogIdparameter to filter queries, scanning only logs withid > lastJobLogId - Modified the monitoring thread to track the last processed log ID across iterations
- Whitespace cleanup in XML mapper file
Reviewed Changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| xxl-job-admin/src/main/resources/mybatis-mapper/XxlJobLogMapper.xml | Added conditional id > lastJobLogId filter to findFailJobLogIds query for cursor-based pagination; minor whitespace cleanup |
| xxl-job-admin/src/main/java/com/xxl/job/admin/dao/XxlJobLogDao.java | Updated findFailJobLogIds method signature to accept lastJobLogId parameter |
| xxl-job-admin/src/main/java/com/xxl/job/admin/core/thread/JobFailMonitorHelper.java | Introduced lastFailLogId tracking variable and passed it to the DAO query for incremental scanning |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| try { | ||
|
|
||
| List<Long> failLogIds = XxlJobAdminConfig.getAdminConfig().getXxlJobLogDao().findFailJobLogIds(1000); | ||
| List<Long> failLogIds = XxlJobAdminConfig.getAdminConfig().getXxlJobLogDao().findFailJobLogIds(lastFailLogId,1000); |
Copilot
AI
Nov 19, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[nitpick] Missing space after comma in the method call. Should be findFailJobLogIds(lastFailLogId, 1000) instead of findFailJobLogIds(lastFailLogId,1000) for consistency with Java code style conventions.
| List<Long> failLogIds = XxlJobAdminConfig.getAdminConfig().getXxlJobLogDao().findFailJobLogIds(lastFailLogId,1000); | |
| List<Long> failLogIds = XxlJobAdminConfig.getAdminConfig().getXxlJobLogDao().findFailJobLogIds(lastFailLogId, 1000); |
What kind of change does this PR introduce? (check at least one)
The description of the PR:
提升失败任务日志的扫描效率。
Other information:
在实际业务重任务量很大,log 表只存 30 天也有几百万条记录,定时扫描导致 mysql cpu 一直占满