fix(pipelines/pingcap/ticdc): update TiCDC Kafka integration test command syntax
Fix the command syntax in the TiCDC Kafka integration test to properly handle the bash shell invocation by adding a double dash to separate tini options from the command (commit: 090ec21)
feat(prow-jobs): add master label to all Jenkins jobs (#4080)
This commit adds the `master: "0"` label to all Jenkins jobs across various repositories to ensure proper resource allocation and scheduling. The change affects multiple presubmit and postsubmit configurations in repositories like enterprise-extensions, tidb-test, docs, tidb-tools, tidb, tiflash, tiflow, tiproxy, copr-test, migration, and pd. The label helps in identifying and managing jobs that should run on master nodes. (commit: c060369)
This release contains the following breaking API changes:
- fix!: Change Org usage report Quantity to float64 ([#​3862](https://redirect.github.com/google/go-github/issues/3862)) BREAKING CHANGE: `UsageItem.Quantity` is now type `float64`. - chore!: Remove `PullRequestRuleParameters.AutomaticCopilotCodeReviewEnabled` field ([#​3866](https://redirect.github.com/google/go-github/issues/3866)) BREAKING CHANGE: `PullRequestRuleParameters.AutomaticCopilotCodeReviewEnabled` is now removed. - feat!: Implement Enterprise SCIM - Provision Groups & Users ([#​3852](https://redirect.github.com/google/go-github/issues/3852)) BREAKING CHANGE: `SCIMEnterpriseDisplayReference.Ref` is now of type `*string`. - fix!: Change copilot\_code\_review field names to match GitHub API ([#​3874](https://redirect.github.com/google/go-github/issues/3874)) BREAKING CHANGE: `CopilotCodeReviewRuleParameters.ReviewNewPushes` is now `ReviewOnPush`. - fix!: Add support for missing attributes in ProjectsV2 API ([#​3888](https://redirect.github.com/google/go-github/issues/3888)) BREAKING CHANGE: `AddProjectItemOptions` fields have changed types. - feat!: Use omitzero for BypassActors to support handling empty arrays ([#​3891](https://redirect.github.com/google/go-github/issues/3891)) BREAKING CHANGE: `UpdateRepositoryRulesetClearBypassActor`, `UpdateRepositoryRulesetClearBypassActor`, `UpdateRulesetClearBypassActor`, and `UpdateRulesetNoBypassActor` have been removed as they are no longer needed.
...and the following additional changes:
- Bump go-github from v79 to v80 in /scrape ([#​3855](https://redirect.github.com/google/go-github/issues/3855)) - feat: Add support for Enterprise Team APIs ([#​3861](https://redirect.github.com/google/go-github/issues/3861)) - feat: Implement Enterprise SCIM - Set Groups or Users ([#​3858](https://redirect.github.com/google/go-github/issues/3858)) - build(deps): Bump golang.org/x/term from 0.37.0 to 0.38.0 in /example ([#​3865](https://redirect.github.com/google/go-github/issues/3865)) - build(deps): Bump golang.org/x/sync from 0.18.0 to 0.19.0 in /tools ([#​3864](https://redirect.github.com/google/go-github/issues/3864)) - build(deps): Bump actions/checkout from 6.0.0 to 6.0.1 in the actions group ([#​3863](https://redirect.github.com/google/go-github/issues/3863)) - feat: Add advanced\_search parameter to SearchOptions ([#​3868](https://redirect.github.com/google/go-github/issues/3868)) - feat: Implement Enterprise SCIM - Delete Groups or Users ([#​3856](https://redirect.github.com/google/go-github/issues/3856)) - feat: Add support for copilot\_code\_review rule type ([#​3857](https://redirect.github.com/google/go-github/issues/3857)) - feat: Add support for Enterprise Team Members APIs ([#​3873](https://redirect.github.com/google/go-github/issues/3873)) - chore: Simplify JSON marshaling for RepositoryRulesetRules ([#​3875](https://redirect.github.com/google/go-github/issues/3875)) - feat: Add support for Enterprise Team Organizations APIs ([#​3876](https://redirect.github.com/google/go-github/issues/3876)) - docs: Clarify CreateTree semantics and TreeEntry usage ([#​3877](https://redirect.github.com/google/go-github/issues/3877)) - docs: Reformulate deprecation notice for Commits field in `PushEvent` ([#​3880](https://redirect.github.com/google/go-github/issues/3880)) - build(deps): Bump actions/cache from 4.3.0 to 5.0.1 in the actions group ([#​3883](https://redirect.github.com/google/go-github/issues/3883)) - build(deps): Bump golang.org/x/net from 0.47.0 to 0.48.0 in /scrape ([#​3882](https://redirect.github.com/google/go-github/issues/3882)) - build(deps): Bump codecov/codecov-action from 5.5.1 to 5.5.2 ([#​3884](https://redirect.github.com/google/go-github/issues/3884)) - build(deps): Bump golang.org/x/crypto from 0.45.0 to 0.46.0 in /example ([#​3885](https://redirect.github.com/google/go-github/issues/3885)) - chore: Rename 'opt' to 'opts' in multiple methods ([#​3887](https://redirect.github.com/google/go-github/issues/3887)) - feat: Add support for remaining Codespaces APIs ([#​3886](https://redirect.github.com/google/go-github/issues/3886)) - feat: Handle omitzero in structfield linter ([#​3881](https://redirect.github.com/google/go-github/issues/3881)) - feat: Implement Enterprise SCIM - Get a Group or User ([#​3889](https://redirect.github.com/google/go-github/issues/3889)) - feat: Add support for Codespace Machines APIs ([#​3890](https://redirect.github.com/google/go-github/issues/3890)) - feat: Add support for URL custom property value type ([#​3879](https://redirect.github.com/google/go-github/issues/3879)) - feat: Add UploadReleaseAssetFromRelease convenience helper ([#​3851](https://redirect.github.com/google/go-github/issues/3851)) - Update openapi\_operations.yaml ([#​3895](https://redirect.github.com/google/go-github/issues/3895)) - feat: Add support for Codespaces organizations APIs ([#​3892](https://redirect.github.com/google/go-github/issues/3892)) - feat: Add support for Copilot usage metrics reports API ([#​3893](https://redirect.github.com/google/go-github/issues/3893)) - feat: GetSignaturesProtectedBranch returns ErrBranchNotProtected when branch is not protected ([#​3896](https://redirect.github.com/google/go-github/issues/3896)) - Bump version of go-github to v81.0.0 ([#​3899](https://redirect.github.com/google/go-github/issues/3899))
</details>
---
### Configuration
📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.
♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
🔕 **Ignore**: Close this PR and you won't be reminded about this update again.
---
- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box
---
This PR was generated by [Mend Renovate](https://mend.io/renovate/). View the [repository job log](https://developer.mend.io/github/PingCAP-QE/ci).
This pull request removes the CDC integration test jobs for both Kafka and MySQL from the Tiflow project's CI/CD pipeline. It deletes the associated Jenkins job DSL definitions, pipeline scripts, and Kubernetes pod templates, as well as updates the Prow job configuration to stop referencing these periodic jobs. This effectively disables the periodic CDC integration tests for the `latest` branch of Tiflow.
**Removal of CDC Integration Test Jobs:**
* Deleted the Jenkins job DSL files for periodic CDC integration tests for both Kafka and MySQL (`periodics_cdc_integration_kafka_test.groovy` and `periodics_cdc_integration_mysql_test.groovy`). [[1]](diffhunk://#diff-2bcf3c8848ae05f14bc177f1e87adc168569d4672f98298c20b34f9b3f239215L1-L38) [[2]](diffhunk://#diff-5017bfbbf1bb1db27392316f87a26f40a3cfb6182451ccf0354dc1b21fa112d8L1-L38) * Removed the corresponding pipeline scripts that defined the test logic and stages for Kafka and MySQL CDC integration tests (`periodics_cdc_integration_kafka_test.groovy` and `periodics_cdc_integration_mysql_test.groovy` under `pipelines/pingcap/tiflow/latest/`). [[1]](diffhunk://#diff-2ddbe297da568f7227daeca36ca37d88518927cbd63cc72af5a436756bbf6d6eL1-L170) [[2]](diffhunk://#diff-9707c40fa4184f8e31164b04234f040090c54427c22294a17f06fa4d8bfcfa0aL1-L170)
**Cleanup of Kubernetes Resources:**
* Deleted the Kubernetes pod templates used for running the CDC integration tests for Kafka and MySQL (`pod-periodics_cdc_integration_kafka_test.yaml` and `pod-periodics_cdc_integration_mysql_test.yaml`). [[1]](diffhunk://#diff-2225eae0c6cc4d4773aced5fec91e0f712d4c6697e08d5b704fca839df6c09acL1-L182) [[2]](diffhunk://#diff-6bf89057fafc295ee9d2bcabc63cba16a4ebdb99406e53bd3ea81456c33236e2L1-L39)
**Prow Job Configuration Update:**
* Updated `prow-jobs/kustomization.yaml` to remove the reference to `pingcap_tiflow_latest-periodics.yaml`, which included the removed periodic jobs. (commit: c14bb55)
TiCDC: Reduce resource allocations in TiCDC test pipelines to optimize cost efficiency (#4086)
## Reduce Resource Allocation for TiCDC Integration Test Pipelines
This PR adjusts the CPU and memory resource allocations across multiple TiCDC integration test pipelines to optimize resource utilization and reduce costs. The changes are based on performance analysis and aim to maintain test reliability while using fewer resources.
### Why? Recent performance evaluations have shown that the existing resource allocations for these integration tests are higher than necessary. By reducing the CPU and memory limits, we can: - Decrease infrastructure costs associated with running these pipelines - Improve resource availability for other concurrent jobs - Maintain the same level of test coverage and reliability - Align resource allocations with actual requirements observed during test execution
### Changes Made: - **Build containers** (`pod-build.yaml` files): - Reduced CPU from 12 to 6 cores - Reduced memory from 16Gi to 8Gi
- **Test containers** (`pod-test.yaml` and `pod.yaml` files): - For heavy integration tests: Reduced CPU from 12 to 6 cores and memory from 32Gi to 16Gi - For light integration tests: Reduced memory from 16Gi to 8Gi (CPU already at appropriate levels)
### Affected Pipelines: 1. **Kafka Integration Tests** (both heavy and light variants, current and next-gen): - `pull_cdc_kafka_integration_heavy` - `pull_cdc_kafka_integration_heavy_next_gen` - `pull_cdc_kafka_integration_light` - `pull_cdc_kafka_integration_light_next_gen`
2. **MySQL Integration Tests** (both heavy and light variants, current and next-gen): - `pull_cdc_mysql_integration_heavy` - `pull_cdc_mysql_integration_heavy_next_gen` - `pull_cdc_mysql_integration_light` - `pull_cdc_mysql_integration_light_next_gen`
3. **Pulsar Integration Tests** (light variants, current and next-gen): - `pull_cdc_pulsar_integration_light` - `pull_cdc_pulsar_integration_light_next_gen`
4. **Storage Integration Tests** (both heavy and light variants, current and next-gen): - `pull_cdc_storage_integration_heavy` - `pull_cdc_storage_integration_heavy_next_gen` - `pull_cdc_storage_integration_light` - `pull_cdc_storage_integration_light_next_gen`
### Impact: These changes are expected to reduce resource consumption by approximately 50% for CPU and 50-75% for memory across all affected pipelines, while maintaining test reliability and coverage. The adjustments have been validated against recent test runs to ensure they don't compromise test stability or introduce flakiness.
fix(pipelines/pingcap/ticdc/latest): add Docker authentication to TiCDC integration pipelines (#4088)
Update all TiCDC integration pipelines to include Docker authentication configuration using the tidbx-docker-config credentials. This ensures proper access to private Docker registries during the pipeline execution. (commit: 9b870e4)
fix(pipelines/pingcap/ticdc/latest): increase memory limits for TiCDC integration test pods to prevent out-of-memory failures (#4087)
## Increase memory resources for TiCDC integration test pipelines
### Why this change? Recent test runs have shown that some TiCDC integration tests are experiencing out-of-memory (OOM) issues and performance degradation, particularly during heavy workload scenarios. To ensure test stability and reliability, we need to increase the memory allocation for both build and test pods across various integration test pipelines.
### Changes: - **Test pods**: Memory allocation adjusted based on test intensity: - **Light tests**: Memory increased from 8Gi to 16Gi - MySQL integration light
- **Next-gen pipelines**: Memory allocations updated consistently with their legacy counterparts - Build pods: 8Gi → 16Gi
### Impact: - Improved test stability by preventing OOM failures - Better performance for memory-intensive integration tests - Consistent resource allocation across all TiCDC integration test pipelines - No changes to CPU allocations - all remain at current levels
These changes will help ensure our integration tests run reliably and provide accurate results for pull request validation.
TiCDC: Increase memory limits for TiCDC integration test pods to improve stability (#4090)
## Increase Memory Limits for TiCDC Integration Test Pipelines
This PR updates the memory resource limits for several TiCDC integration test pipelines to improve stability and prevent out-of-memory (OOM) failures during test execution.
### Why this change? Recent test runs have shown intermittent failures due to memory exhaustion, particularly in heavier integration scenarios. Increasing the memory allocation ensures tests have sufficient resources to complete reliably, reducing flakiness and improving overall CI/CD pipeline stability.
### Changes Made: - **Kafka Integration Light Tests**: Memory increased from 8Gi to 16Gi for both `pull_cdc_kafka_integration_light` and `pull_cdc_kafka_integration_light_next_gen`. - **MySQL Integration Heavy Tests**: Memory increased from 16Gi to 32Gi for both `pull_cdc_mysql_integration_heavy` and `pull_cdc_mysql_integration_heavy_next_gen`. - **Storage Integration Light Tests**: Memory increased from 8Gi to 16Gi for both `pull_cdc_storage_integration_light` and `pull_cdc_storage_integration_light_next_gen`.
### Impact: - CPU limits remain unchanged across all pipelines. - These adjustments align with observed resource usage patterns and are expected to eliminate OOM-related test failures without over-provisioning.
