Skip to content

Conversation

@ddavidzhang
Copy link

@ddavidzhang ddavidzhang commented May 2, 2024

What kind of change does this PR introduce? (check at least one)

  • Bugfix
  • Feature
  • Code style update
  • Refactor
  • Build-related changes
  • Other, please describe:

The description of the PR:
提升失败任务日志的扫描效率。

Other information:
在实际业务重任务量很大,log 表只存 30 天也有几百万条记录,定时扫描导致 mysql cpu 一直占满

@xuxueli
Copy link
Owner

xuxueli commented Nov 19, 2025

考虑到长耗时任务,失败任务id不是绝对递增的,不可以简单通过最小id过滤。 可以通过时间过滤,一会扫描近期增量数据,比如3天内。

Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR introduces cursor-based pagination to improve the scanning efficiency of failed task logs by adding a lastJobLogId parameter to avoid repeatedly scanning the entire log table (which can contain millions of records with 30-day retention).

Key Changes:

  • Added lastJobLogId parameter to filter queries, scanning only logs with id > lastJobLogId
  • Modified the monitoring thread to track the last processed log ID across iterations
  • Whitespace cleanup in XML mapper file

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File Description
xxl-job-admin/src/main/resources/mybatis-mapper/XxlJobLogMapper.xml Added conditional id > lastJobLogId filter to findFailJobLogIds query for cursor-based pagination; minor whitespace cleanup
xxl-job-admin/src/main/java/com/xxl/job/admin/dao/XxlJobLogDao.java Updated findFailJobLogIds method signature to accept lastJobLogId parameter
xxl-job-admin/src/main/java/com/xxl/job/admin/core/thread/JobFailMonitorHelper.java Introduced lastFailLogId tracking variable and passed it to the DAO query for incremental scanning

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

try {

List<Long> failLogIds = XxlJobAdminConfig.getAdminConfig().getXxlJobLogDao().findFailJobLogIds(1000);
List<Long> failLogIds = XxlJobAdminConfig.getAdminConfig().getXxlJobLogDao().findFailJobLogIds(lastFailLogId,1000);
Copy link

Copilot AI Nov 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nitpick] Missing space after comma in the method call. Should be findFailJobLogIds(lastFailLogId, 1000) instead of findFailJobLogIds(lastFailLogId,1000) for consistency with Java code style conventions.

Suggested change
List<Long> failLogIds = XxlJobAdminConfig.getAdminConfig().getXxlJobLogDao().findFailJobLogIds(lastFailLogId,1000);
List<Long> failLogIds = XxlJobAdminConfig.getAdminConfig().getXxlJobLogDao().findFailJobLogIds(lastFailLogId, 1000);

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants